Questions

1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

• Executing a trusted shell

The first task is to execute a trusted shell and to set the PATH environment variable equal to dot (.). This will decrease the chances that we accidentally execute untrusted commands that are in the compromised system's PATH.

• Checking integrity of all files

After running a trusted shell, we have copied the md5 file downloaded from the honeynet web site (linux-suspended-md5s.gz) , we gunzip it, and we copy it into a floppy disk. In order to reduce our trust in the infected system, we copy an "md5sum" command from an clean system into the floppy, then we run it against the infected system using to following command :

md5sum -c linux-suspended-md5s | gzip -9 > /mnt/floppy/modifiedfiles.gz

The output result is gzipped to be able to be stored in the floppy, in order to minimize the impact on the infected system.

We transfer the floppy content into a production system, we extract its content, and we select the name of the modified or deleted files using the following commands :

gunzip modifiedfiles.gz cat  modifiedfiles  | grep FAILED | awk '{print $1}' | sort > modifiedresults.txt

• Determining the Running Processes

A snapshot of all the running processes is taken using the standard ps (process status) command (see ps-output) :

ps -aux

The suspected processes retrieved from the output of ps command are the following :

ID	%CPU	%MEM	SIZE	RSS	TTY	STAT	START	TIME	COMMAND
root	25239	0.0	0.3	1880	336	p3 S	15:32	0:00	/lib/.x/s/xopen -q -p
root	25241	0.0	0.7	1888	672	?  S	15:32	0:00	/lib/.x/s/xopen -q -p
root	25247	0.0	0.7	1668	732	?  S	15:32	0:00	/lib/.x/s/lsn
root	25542	0.0	1.1	2116	1064	?  S	15:32	0:00	/lib/.x/s/xopen -q -p
apache	21510	0.0	0.4	1476	392	?  S	15:28	0:00	./bash
apache	21511	0.0	1.1	2188	1124	p3 S	15:28	0:00	sh -i
apache	23289	0.0	0.3	1376	296	p3 S	15:30	0:00	/dev/shm/k
apache	23292	0.0	0.0	0	0	p3 Z	15:30	0:00	[k <zombie>]
apache	23302	0.0	0.0	0	0	p3 Z	15:30	0:00	[k <zombie>]
root	14642	0.0	1.1	2184	1132	p3 R	15:53	0:00	/bin/sh
root	23306	0.0	1.1	2184	1132	p3 S	15:30	0:00	/bin/sh

These bizarre entries in the process listing, expect for command "/bin/sh" may be executed by the attacker. The START field of the ps command output is extremely helpful when the time of the attack is isolated. We can identify suspect processes merely by the time they were executed.

• Detecting Loadable Kernel Module Rootkits

Rootkits are collection of commonly trojaned system processes and scripts that automate many of the actions attackers take when they compromise a system. The most advanced rootkits are Loadable Kernel Modules (LKMs). LKMs can hide processes, and create illicit back doors on a system. In the suspended image, there are many traces (see trace-rootkits) that demonstrate the existence of the following rootkits :

• Adore
• LKM
  

kstat is a very useful tool for detecting rootkit modules.
The analyzed system is infected by the RST.b (Remote Shell Trojan b). The origin of this infection is adore which is infected by this kind of virus. The following signs detected in result-vmdk file confirm that the analyzed system is infected :

• Trace contains the following command : “GET /~telcom69/gov.php HTTP/1.0”
• Calling ptrace to prevent analysis and debugging

RST.b initiates a virus-like self replication process that 

infects additional executable binaries in the current working directory and in the  /bin directory.

• Determining Open Ports and Listening Applications

We have used a clean netstat command with -p option that maps the name of the application and its process ID to the open ports. The command used is the following :

netstat -anp

The netstat command output (see netstat-output has chown that there are suspected applications that are responsible for the open network sockets. This applications are presented in the answer to the question 3.

To list only the processes that have opened network sockets, we have used the following command line :

lsof -i

The output of this command contains the following suspected processes (see lsof-output) :

smbd          3137     root    6u   IPv4   4571      TCP *:cfinger (LISTEN)
smbd          3137     root   16u  IPv4    976       TCP *:https (LISTEN)
smbd          3137     root   17u  IPv4    977       TCP *:http (LISTEN)
(swapd)      3153     root   16u  IPv4    976       TCP *:https (LISTEN)
(swapd)      3153     root   17u  IPv4    977       TCP *:http (LISTEN)
initd          15119     root    3u   IPv4  15617     TCP *:65336 (LISTEN)
initd          15119     root    5u   IPv4  15619     TCP *:65436 (LISTEN)
xopen       25239     root    8u   IPv4   9972       UDP *:3049
xopen       25239     root   16u  IPv4    976       TCP *:https (LISTEN)
xopen       25239     root   17u  IPv4    977       TCP *:http (LISTEN)
xopen       25241     root    8u   IPv4  12302     TCP *:squid (LISTEN)
xopen       25241     root   16u  IPv4    976       TCP *:https (LISTEN)
xopen       25241     root   17u  IPv4    977       TCP *:http (LISTEN)
lsn            25247     root   16u  IPv4    976       TCP *:https (LISTEN)
lsn            25247     root   17u  IPv4    977       TCP *:http (LISTEN)

• Discovering Sniffers on the compromised system

Before we use a clean ifconfig command to view interfaces configuration, we have found that the interface "eth0" is running in a promiscious mode. This information is displayed when loading the VMWare suspended image. To confirm this information, we have queried all network adapters using the following command :

ifconfig -a

The output of this command (see ifconfig-output.txt) has not shown (option not availabe for Redhat 7.2 ) the flag PROMISC indicating that a sniffer is currently running on the system. But, the combination of lsof and ps command has confirmed that. Also, another confirmation is provided by the tool added by the hacker which has switched the eth0 interface to the promiscious mode. The lsn tool is located on the following path :

/lib/.x/s/

A sniffer running on the system means that the attacker had root-level access.

• Checking for deleted files
• Locate command

We note that the locate binary command was not affected when the machine was hacked. This command deliver results based on the last updates content of its database. We directly think that probably this database content was not changed after the machine compromization. For example the /var/log/dmesg file doesn't exist and was probably deleted, however the locate command mention that this file exist.
 We decide to compare the previous and the actually content in order to discover the deleted and the new inserted files. We use the following command :

locate / > /mnt/floppy/oldfiles

In order to reduce the impact on the infected system, we write the locate command results into the floppy disk. We unmount the floppy, and we copy the content into an other production system. Then, back to the compromized system, and we update the database by using the updatedb command.

updatedb

We apply again the locate command :

locate / > /mnt/floppy/newfiles

 We unmount the floppy, copy its content to the production system, and we try to compare the newfile and oldfile content.

diff newfiles oldfiles | grep \< | awk '{print $2}' > insertedfiles diff newfiles oldfiles | grep \> | awk '{print $2}' > deletedfiles

 

• Reviewing the /Proc File System

The /proc file system is a pseudo-file system that is used as an interface to kernel data structures. Each process has a subdirectory in /proc that correponds to its process ID. Therefore, each running process will have a numerical subdirectory structure. We have investigated this directory to get process information.

When reviewing this directory, we have found that there are important files deleted by the attacker.

The list of deleted files is obtained using the following command :

ls -lR /proc | grep deleted | awk '{print $11}' | sort | uniq

• Reviewing mails sent

When analyzing result-vmdk content, we have found that there are mails send by the attacker. These mails are sent to the following email addresses :

• mybabywhy@yahoo.com
• buskyn17@yahoo.com
• tuiqoitu039t09q3@bigfoot.com
• bnadfjg9023@hotmail.com
• mki62969o@yahoo.com
  

These mails contain information on the actual configuration of the compromised host and services availables. The amount of information collected are available in  infoSent.txt

All what has been detected when analysing the system are an important signs that the system is compromised.

• Others actions detected

          There are other signs that indicate the compromise of the system  :

• Download of rootkits, log cleaner and install (mass2.tgz, rootkit.tar, vanish2.gz, ttt.tgz, a.tgz, er.tgz)
• Redirect command output and files (messages, wtmp, ...) content to /dev/null
• Delete logs
  

2. Explain the impact that your actions had on the running system

We have decided to run strings command on the suspended image before loading it using VMWare to get the maximum informations related to the compromise system. This is useful because simple actions may alter the analyzed system. For example, simply viewing a file or a directory entry causes information on the system to be changed. Also, the volatile data may be lost if not retrieved before powering down the system. Volatile data includes currently open sockets, running processes, the content of a system RAM and the location of unlinked files (files marked for deletion when processes that access it terminate). Actions taken on the compromised system have a impact on the swap content and  memory content (respectively /dev/shm and /proc/kcore). We have take attention to not write on the compromized partition to avoid altering informations related to the attack.

3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports)

From the clean netstat command output, we have identified the following processes that had a suspect ports open :

PID	Process	Port	Tcp service
3137	smbd -D	80	http
		443	https
		2003	cfinger
25241	xopen	3128	squid
15119	initd	65336	psybnc
		65436
850	nmbd	137	netbios-ns
		138	netbios-dgm

4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for ?

Three TCP connections were established from an to the compromised box using the following command :

/usr/lib/libshtift/netstat -tuap | grep ESTABLISHED

5. How many instances of an SSH server were installed and at what times ?

6. Which instances of the SSH servers from question 5 were run ?

7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected ?

8. Which system executables (if any) were trojaned and what configuration files did they use ?

9. How and from where was the system likely compromised ?

The analyze of the content of result-vmdk and result-vmss (see prelimiary tasks for details) has shown that there are several attack attempts. The attack attempt that has succeeded has exploited a buffer overflow in Samba which gives the attacker remote root access. The description related  to this vulnerability is available here (CAN-2003-0201). The installed samba version (2.2.1a-4) is affected by this vulnerability (versions affected < Samba 2.2.8a, <= Samba 2.0.10 ).  The trace of this attack has been detected from the result-vmdk file. Some keywords that indicate the success of this attack are :

• Data overflow in trans2
• Data overflow in cli_receive_trans
• Out of memory in reply_trans2
  
• Gain_root
• Security context stack overflow
  
  

All the attack trace is available here.
The attacker ip address is 213.154.118.200. When googling to search some informations related to this ip class, i have found that there a message posted from romania and which has the same network address (213.154.118) (see message).

Bonus Questions

• IP address
• Language used for irc
• Exploits used are in romanian language for example mass2.