Abstract
This is a submission to the Honeynet Project September 2003 Scan of the Month. This Scan of the Month requires the analysis of a RedHat Linux machine that was compromised on August 10, 2003. The uniqueness of this challenge lies in the fact that a completely live analysis of the compromised machine is possible thanks to the magic of machine virtualization.
During my analysis I find that the attacker, possibly a script kiddie from Bucharest, Romania, broke in through a whole in the Washington University FTP server program (wu-ftpd), and that after breaking in the kiddie installed an IRC proxy as well as some trojaned executables (ls, ps, netstat, ifconfig, and top) and some backdoors (SSH servers) and malware.
The analysis I've done is in some parts very detailed. If you are a grader, have several submissions to read and can't go over all the details, or are just a casual reader, feel free to jump directly to Section 1, “Questions”, which contains the answers to all the questions of this challenge. The questions (and answers) provide a good summary of the most important aspects of this challenge. However, just reading the answers is not the best way to understand some of the details nor the process I followed to in my forensics analysis of the compromised machine, so I encourage you to read the whole submission.
Table of Contents
- 1. Questions
- 1.1. Question 1
- 1.2. Question 2
- 1.3. Question 3
- 1.4. Question 4
- 1.5. Question 5
- 1.6. Question 6
- 1.7. Question 7
- 1.8. Question 8
- 1.9. Question 9
- 1.10. Bonus Question
- 2. Finding Out What Happened
- 2.1. File Analysis
- 2.1.1. MD5 Hashes Analysis
- 2.1.2. Timeline Analysis
- 2.1.3. Interesting Files
- 2.2. Processes Analysis
- 2.3. Network Analysis
- 3. Recommendations and Conclusions
- A. Files
- B. References
- C. Work Methodology
- D. Thanks
Q: Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.
A: To determine that the host was compromised I used two very unobtrusive processes. First, while logged into the possible compromised host I generated MD5 hashes for all the files in the host's file system. Then I compared the results with the MD5 hashes of all the files before the host was possibly compromised. The comparison was done using the diff Unix command, which allowed me to see quickly and easily what files were added to the system, which ones were deleted, and which ones were changed.
The second method I used was to perform a port scan from another machine. This port scan showed that ports that are not normally open were open.
If the above two findings does not convince you that your machine has been compromised then perhaps you should consider finding another hobby or a career change since you are not paranoid enough.
Q: Explain the impact that your actions had on the running system.
A: The initial analysis that I did had zero impact since no services where shut down, and the machine was not rebooted or disconnected from the network. While all the analysis I did was done online, with the machine up and running (even the Sleuth Kit analysis), I don't recommend that this is done this way in a production system since while the investigator is performing forensics on the machine, the attacker could be DoS'ing somebody, using the compromised machine for illegal purposes, or just rm -Rf / to get rid of evidence. I recommend that at least the machine be disconnected from the network while performing forensics.
Q: List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).
A: Please take a look at the network analysis that I did (Section 2.3, “Network Analysis”). I used the output from a non-trojaned netstat as well as a port scan from another machine.
Summarizing, processes keeping suspicious ports open have the following PIDs: 25239 (xopen, port UDP 3049), 25241 (xopen, port TCP/3128), 15119 (initd, ports TCP/65436 and TCP/65336), and 3137 ("smbd -D", port TCP/2003).
Q: Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?
A: Yes, there were three active network connections. Two of them were IRC sessions, from the compromised machine to IRC servers, and the other one from the IP of the attacking machine (213.154.118.200, a.k.a. sanido-08.is.pcnet.ro, a machine in Bucharest, Romania) to the compromised machine, TCP port 65336, which has the psyBNC IRC proxy listening on it. See Section 2.3, “Network Analysis” for the gory details.
Q: How many instances of an SSH server were installed and at what times?
A: There are more instances of a SSH than are necessary. Seems like our kiddie was having problems or didn't know what he was doing. I counted at least three SSH servers: /usr/lib/sp0, /lib/.x/s/xopen, and "/usr/bin/smbd -D". I believe sp0 was installed around 21:30, "smbd -D" around 21:33, and xopen around 22:32. See Section 2.1.2, “Timeline Analysis” for more information on the timeline analysis.
Q: Which instances of the SSH servers from question 5 were run?
A: sp0 segfaults at startup. The kiddie obviously tried this one but it didn't work. I found both "smbd -D" and xopen running. See Section 2.2, “Processes Analysis” for details on the running processes I found running.
Q: Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?
A: The "smbd -D" SSH server appears to have been modified to collect user IDs, passwords and source IPs used when logging in. The xopen SSH server appear to have been modified to collect the output from the ls, netstat and ps commands when run with different combinations of switches. I did not find examples of these types of information collected.
Q: Which system executables (if any) were trojaned and what configuration files did they use?
A: ls, ps, top, netstat and ifconfig were trojaned. The trojaned versions hide certain information when run. For example, the ps command won't display any process with a name of "smbd -D", the ifconfig command won't show that an interface is in promiscuous mode, the ls command won't show some /usr/lib/ice* files, etc.
The configuration files for these trojaned system files are:
/dev/ttyoa: list of IP networks and TCP/UDP ports to hide when netstat is run.
/dev/ttyof: list of files to hide when ls is run.
/dev/ttyop: list of process names to hide when ps or top are run.
There are 3 ways of figuring this out: 1) run strings on the trojaned files and look for suspicious files. Correlate this with the new files installed after the break in (via the MD5 analysis.) 2) run the trojaned commands under strace and see what files the trojan tries to open. 3) see what new files were installed by the kiddie, and use common sense (it's easy to see that the above files contain port numbers, network addresses, and process names.) I used #3 followed by #1 to confirm.
Q: How and from where was the system likely compromised?
A: I believe this system was compromised through a hole in the Washington University FTP server program (wu-ftpd). The specific information about this hole can be found here. While I did not find conclusive proofs about the IP of the machine that compromised this system I believe the attack came from 213.154.118.200, a machine in Bucharest, Romania , according to Sarangworld Traceroute Project.
Q: What nationality do you believe the attacker(s) to be, and why?
A: I believe our kiddie is Romanian. I believe so because the psyBNC log file at /etc/opt/psybnc/log/psybnc.log shows connections from an IP network that the Sarangworld Traceroute Project puts in Bucharest, Romania. Also, when I started the analysis of the compromised system there was a proxy IRC connection from the same IP network.
To find out what happened with the compromised host I performed three main analyses. Each one of these analyses focused on different aspects of the compromised system. While some of these analyses are stronger that the others, none of them is conclusive, which means that for a full understanding of what really happened all of them must be performed.
The file analysis consisted in looking at the files in the compromised host. This analysis had three parts: 1) analysis of the MD5 hashes of each file in the system, 2) a time analysis of all files, in particular during August 10, 2003, which is when the machine was compromised, and 3) a look at some interesting files that the attacker left behind.
The analysis of the MD5 hashes is perhaps the most important of all the analysis that I did of the compromised machine. This analysis is of capital importance because it tells us what files were added to the file system since the original MD5 hashes were calculated, what files were deleted, and what files were changed. In addition, this analysis is very unobtrusive, which means that can be performed while the machine is still up and we are assessing the scope of the compromise to decide whether the machine needs to be brought down to perform a more thorough analysis, reinstall the operating system, or just undo what the hackers did to the system. Finally, as I will show, the MD5 hashes analysis is done very quickly, which makes it the ideal starting point of the analysis of the compromised host. The only drawback: you need to have calculated the MD5 checksums of all files in the system before the incident. Fortunately, the Honeynet Project did this, and not only that; they also provided the MD5s for this challenge. Without these MD5 checksums, analyzing the compromised host would have been a lot harder.
The file provided by the Honeynet Project for this challenge looks like this:
4a51ff671301fdbe47be03e194f77a8e /dev/MAKEDEV 8831d78a68d5308c3f024b6d1ca8ec94 /var/lib/rpm/Packages 98e4e1fea4530c03923d06c3b5068fce /var/lib/rpm/Basenames 523809658d032fa3efbc052cc84ae81d /var/lib/rpm/Name [...] |
The first column is the MD5 hash, and the second is the file name. The idea here is two generate a similar file, with the MD5 checksums of all files in the system after the compromise. The MD5 checksums in this new file will then be compared to the MD5 checksums we were given. Using the diff we will be able to tell exactly what changed. The exact procedure is as follows:
Use the awk and sort commands to generate a file that is more appropriate for comparing files. What we want to do is to exchange the first and second columns so the file name is first. We then sort the file so all lines are sorted by file name:
$ awk '{print $2 " " $1}' linux-suspended-md5s | \ sort > linux-suspended-md5s.sortedWe then proceed to generate the MD5 hashes of the files in the system after the incident. We use the find command and then use the same awk | sort trick we used before:
$ find / -type f -exec md5sum {} \; | \ awk '{print $2 " " $1}' new-md5s | sort > new-md5s.sortedFinally, once we have the linux-suspended-md5s.sorted and the new-md5s.sorted files we just run the diff on them:
$ diff -U 0 linux-suspended-md5s.sorted new-md5s.sorted --- linux-suspended-md5s.sorted 2003-09-05 08:01:44.000000000 -0400 +++ md5sums.txt.sorted 2003-09-09 00:22:15.000000000 -0400 @@ -1,0 +2 @@ +/.bash_history 58a1787224e531e42880e5af2bd2ac69 @@ -37 +38 @@ -/bin/ls 3e743c6bfa1e34f2f2164c6a1f1096d0 +/bin/ls 9e7165f965254830d0525fda3168fd7d @@ -45 +46 @@ -/bin/netstat 0ea03807e53e90b147c4309573ebc76a +/bin/netstat c0e8b6ff00433730794eda274c56de3f @@ -47,0 +49 @@ +/bin/pico 8b0afccbb9fec255fe38bdb6d0776ba7 @@ -49 +51 @@ -/bin/ps 881c7af31f6f447e29820fb73dc1dd9a +/bin/ps a71c756f78583895afe7e03336686f8b @@ -89,0 +92,2 @@ [...]
After performing the three steps listed above we end up with a file that shows in a very concise way what files changed (new, modified or deleted) after the incident. The resulting file is large, so I am including it in Appendix A, Files (follow this to go there directly). However, it is important to note that a big part of the analysis of the compromised system is based on the contents of this file.
As an example of how to interpret the output from the diff command, above we can see that the file /bin/ls was replaced (or modified) because its MD5 hash is different after the incident. We can also see the file /.bash_history is a new file that did not exist in the system prior to the incident.
It is important to note that since chances are the system has been compromised we can't use any of the executables in the system for our forensics analysis, since anything could have been trojaned. For example, the md5sum command could have been trojaned to lie about the MD5 checksum of the file /bin/ls so we don't notice that it has been changed. For this reason I recommend that all used tools are run from a CD-ROM of floppy, and are copied there from trusted sources. This is just a small precaution that is not bullet-proof: the kernel could have been trojaned too, in which case all bets are off, and the only way to verify the MD5 checksums is by rebooting the machine with a trusted kernel (like a Knoppix CD.)
The MD5 analysis I did in the previous sections allows to easily determine what files were trojaned by the attacker. The MD5 hashes of the following files changed after the machine was compromised:
/bin/ls: modified to prevent a system administrator from seeing files associated with the break in. For example, this ls command will not show the files /usr/include/{icekey.h, iceconf.h, icepid.h}.
/bin/ps and /usr/bin/top: modified to prevent a system administrator from seeing processes associated with the break in. For example, this ps command will not show the processes "smbd -D" and "swapd".
/bin/netstat: probably modified to prevent a system administrator from seeing certain network connections. This version of netstat does not accept the -p switch to see PIDs and process names associated with a socket.
/sbin/ifconfig: modified to prevent a system administrator from seeing that a network interface is in promiscuous mode.
The original versions of all these files were saved to the directory /usr/lib/libshtift/.
The file /etc/rc.d/init.d/functions was modified by the attacker so the program /usr/bin/crontabs is executed when the machine is booted. This crontabs program only executes the program "/usr/bin/smbd -D".
/etc/rc.d/rc.sysinit was modified (probably by the attacker) to execute the program kflushd at boot time. However, I found no file with this name in the system.
The attacker added a user with the login ID "admin" to /etc/shadow. This user has the user ID 0, i.e. root equivalent.
"/usr/bin/smbd -D" (note that the file name is not just "smbd", it is "smbd -D"): this seems to be a SSH server (based on strings's output).
/usr/bin/crontabs: small program to just runs the program "/usr/bin/smbd -D. More information on this later.
"/usr/bin/(swapd)": a sniffer. Sends its output to standard output.
/dev/ttyo{a,f,p}: configuration files for the trojaned system executables (ls, ps, netstat, ifconfig and top.)
/usr/bin/sl2: a quick look with a disassembler tells me that this is a tool to send custom network traffic, most probably TCP traffic. This might be a Denial of Service (DoS) tool, although the ELF header seems to be corrupt and I could not run it to make sure.
/usr/lib/adore.o: this is the Adore Linux Kernel Module rootkit. I found no signs of this rootkit running on the machine. References to information on the Adore LKM in Appendix B, References.
/usr/bin/sense: a parser for log files generated by the above sniffer.
/usr/lib/sp*: (yet) another SSH server with its configuration, random seed and private key files (I'm sick of seeing hacked/trojaned SSH servers being installed by this kiddie.)
/usr/include/ice*: configuration, PID, random seed and private key files for a SSH server.
/bin/pico: probably this kiddie doesn't know how to use a real text editor like vi so he decided to bring with him his own lesser editor. In any case, this script kiddie is an idiot because this program doesn't run because it needs libncurses.so.4, which is not installed in the system.
/etc/opt/{psyBNC2.3.1.tar.gz,psybnc/}: an IRC bouncer. The .tar.gz contains the original sources. The attacker untarred them in the psybnc/ directory and build the program there.
/root/sslstop/: this directory contains two C source files and one Makefile. Two programs are generated after running make. One program just disables Apache's HTTPS support, and the other changes the SSL port from the 443 default to something else. The intruder ran these programs.
/lib/.x/*: there are several interesting things here:
There's a directory called /lib/.x/s/ that contains a SSH server renamed to xopen, the configuration files for this SSH server, one sniffer called lsn and its log file (called /lib/.x/s/mfs.
sk: this is the SuCKIT rootkit. This rootkit allows to hide PIDs and files from the look of the system administrator. Links to more information about SuCKIT can be found in Appendix B, References.
log: this is SuCKIT's login program (normally called login after a sk build from pristine sources.) This program is used to log into a remote host running suckitd (a.k.a. sk).
cl: this is a log file cleaner. strings shows the usage of this command. It can clean lines from log files that have specific strings.
.boot: this is an installation script. This script:
Starts the SSH server /lib/.x/s/xopen.
Starts the sniffer /lib/.x/s/lsn.
Runs the SuCKIT binary to turn on PID and file hiding. The output from the sk is sent to the file /lib/.x/reboot.log. Apparently all invocations of the SuCKIT binary failed with the following error message, which is a relief:
##################################################### # SucKIT version 1.3b by Unseen <unseen@broken.org> # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
This is probably because of a kernel SuCKIT does not understand.
E-mails to skiZophrenia_sick@yahoo.com the log file from the sniffer as well as vital machine information that would allow the attacker to re-gain access to the machine (backdoor port, etc.)
Runs the script /lib/.x/hide to hide, using sk all processes that have the strings "lsn", "xopen" and "/lib/.x" in their process names.
Deletes from log files all lines that have the strings "o.tgz", "suckit", "xopen" and "promisc". Also deletes from /var/log/maillog lines that include the strings "yahoo" and from /var/log/secure those that have the string "promisc".
/lib/.x/inst: this file is a shell script that creates the sk command in /lib/.x/. The binary is coded in ASCII inside log cleaner.
For the timeline analysis I used the Sleuth Kit (http://www.sleuthkit.org/). I also used the Sleuth Kit to do analysis of deleted files, but didn't find anything conclusive there.
I did not run the Sleuth Kit on the compromised machine. What I did was to use the dd to transfer an image of the root partition of the compromised machine to a trusted machine where I had the Sleuth Kit installed. I tried to mount a remote NFS share so I could send dd's output directly to the remote machine but portmap was not available in the compromised machine so that didn't work. I ended up using the smbmount to mount locally a remote SMB share. Then I sent dd's output to a file in the mounted directory.
The Sleuth Kit generated a big file. I used grep to filter some uninteresting things. The whole file can be seen here. Here's what I found in this timeline analysis (all times are UTC, even thought the compromised machine is in the US Pacific time zone):
In my opinion, the break in started on August 10, 2003 at 19:27. At this time the FTP server, in.ftpd is executed. I believe the attacker got in via a hole in wu-ftpd (http://www.wu-ftpd.org/). The following extract from the timeline generated by the Sleuth Kit shows the first file touched after the execution of in.ftpd. The timeline shows (see the big file in Appendix A, Files that the file access before 19:27 happened at 11:23 (8 hours before.)
Sun Aug 10 2003 19:27:36 172668 .a. -/-rwxr-xr-x bin bin 62582 /mnt/usr/sbin/in.ftpd Sun Aug 10 2003 20:30:00 26780 .a. -/-rwxr-xr-x root root 45661 /mnt/bin/date 0 .a. -rw-r--r-- root root 77647 <sda1-dead-77647> Sun Aug 10 2003 20:33:19 74 .a. -/-rwxr-xr-x root root 92024 /mnt/dev/ttyop 4060 .a. -/-rwxr-xr-x root root 92009 /mnt/usr/bin/sense 36692 .a. -/-rwxr-xr-x root root 92022 /mnt/bin/ls 32756 .a. -/-rwxr-xr-x root root 92011 /mnt/bin/ps 48856 .a. -/-rwxr-xr-x root root 92017 /mnt/usr/bin/top 59 .a. -/-rwxr-xr-x root root 92025 /mnt/dev/ttyofHalf an hour later in.ftpd is executed, at 20:30, we see that the attacker starts to poison the system by installing his trojaned crap. In the extract above we can see ls, ps, and top.
At 20:33 the (swapd) is installed and run for the first time (apparently it is killed and rerun again at 03:34.) "smbd -D" is also installed at this time. After 20:33 there is no activity until 21:13. At 21:30 more malware is installed (the Adore LKM rootkit, the sp0 SSH server I mentioned before, etc.) Also at this time, /etc/rc.d/rc.sysinit is modified to run /usr/bin/crontabs at boot time. Log files are also deleted at this time, and /root/.bash_history is symlinked to /dev/null.
At 22:32 the /lib/.x/.boot is apparently run, /lib/.x/sk is created, as well as the rest of the files in /lib/.x/.
At 22:40, sslstop.tar.gz is downloaded and compiled. sslport and ssltop are run and /etc/httpd/conf/httpd.conf is modified to not run SSL and to change the SSL port.
At 22:57, our good friend psyBNC2.3.1.tar.gz is downloaded. At 23:01 the build of psyBNC is started. The build finishes at 23:02, leaving the /etc/opt/psybnc/initd executable ready. The IRC bouncer is started at this time and there isn't much file activity until 03:34. At this time psyBNC is apparently built again.
At 03:35 /lib/.x/.boot is apparently run again. Also at this time, the SSH server /usr/lib/sp0 is run, but it crashes with a segfault. /lib/.x/s/xopen (another SSH server) is then run. The /lib/.x/s/lsn sniffer is also run at this time.
At 03:36 the machine is disconnected and I come in to start the forensics analysis. I change root's password at 03:54. The rest of the timeline file shows some of the things I did when I got in.
The file /usr/bin/crontabs, as I mentioned before, just executes the program /usr/bin/"smbd -D". crontabs is run at boot time. I found out what this program does by decompiling it. It is a very simple program (coded by a beginner):
void exec_u1(void)
{
system("\"smbd -D\"");
}
int main(void)
{
exec_u1();
return 0;
}
|
While it is easy to escape spaces and dashes from the shell (which would be needed to execute a program with a name like "smbd -D") my guess is that this program was written to make it hard to figure out how the rootkit is being started at boot time.
/lib/x/.sk is another interesting file. This is the SuCKIT rootkit. It allows to hide/unhide PIDs and files. It also provides an interesting backdoor for the attacker to get back in. While the installation of SuCKIT failed, I took some time to learn its capabilities and how it works. The source code can be easily found on the web.
The sk binary has an embedded password that is set by the attacker at build time. This password is then used to authenticate the attacker when he/she logs in through the backdoor. I wrote a quick password cracker to try to guess the password that our kiddie used. I wanted to see if I could break into his Yahoo Mail account. Anyway, I used John the Ripper to generate passwords that I later fed to my password cracker. The bad news is that the kiddie apparently chose a strong password since after a couple of hours running, John the Ripper tried all 5-character combinations and didn't find the password. But in any case this was a good exercise. The cracker can be found here.
The idea with an analysis of the running processes is to figure out what foreign processes are running on the compromised machine. We start this analysis by running a trusted copy of the ps command:
The output from ps is long, but we can immediately filter out lots of entries if we realize that the host was compromised on August 10th, and that there are lots of entries that have Aug09 in the START field. In other words, processes that started on August 9 were started before the machine was compromised, which means we safely can skip their analysis and focus on only a few processes. Most of the information in the following paragraphs was obtained by running the strings and lsof commands. lsof in particular is very useful because it tells me the exact location of the executable, any network sockets the process is listening on, and any open files (like log files) the process has open. In short, this command is essential to analyze a live system.
 | As we saw in Section 2.1, “File Analysis”, the command smbd -D is located in the /usr/bin/ directory. Running the strings command on this file we see that this process is probably a modified SSH server. The configuration file for this SSH server is hardcoded as /usr/include/iceconf.h. This file references /usr/include/icekey.h, /usr/include/icepid.h and /usr/include/iceseed.h as the host private key, the PID file and the random seed respectively. The strings command also returns a couple of interesting strings ("+- User Login Incoming ]-" and "username: %s password: %s%s hostname: %s") that lead me to think that this process is sniffing passwords. I will talk more about this process in Section 2.3, “Network Analysis”. | ||
 | The (swapd) is also located in /usr/bin/. The strings returns some interesting strings that lead me to think that this process is just a sniffer.
lsof -p 3153 (the PID of this process) shows that the sniffer is logging to the file /usr/lib/libice.log. Here's a sample of the contents of this file:
The script /usr/bin/sense is a parser for this type of file. One comment in this script tells us that (swapd) is a sniffer called LinSniffer, which can be found easily on the Internet. | ||
 | The syslogd command was not trojaned, as we saw in Section 2.1, “File Analysis”. The fact that this process shows as started on August 10 is probably because the attacker cleaned log files after breaking in and installing his rootkit. | ||
 | Same thing with the klogd command: this executable was not trojaned, and was probably restarted after cleaning of the log files to hide the attacker's actions. the | ||
 |
| ||
 | The xopen command is located in the "hidden" directory /lib/.x/s/, as the output from ps auxw shows. The strings strings command shows that this is yet another SSH server (apparently this kiddy has an obsession with SSH servers.) This SSH server listens for connections on port TCP 3128, and the configuration file is /lib/.x/s/sshd_config. Other required files (host private and public keys, random seed, PID file) are also in /lib/.x/s/. There are two instances of this program. Maybe our kiddie got impatient and ran it twice for some reason. | ||
 | The lsn command is located in /lib/.x/s/. The strings only tells us that this executable has been compressed with UPX (the Ultimate Packer of eXecutables). Using the upx on Linux I was able to unpack the program. Then strings showed some strings that again lead me to think that this is yet another network sniffer (this is our kiddy's second obsession.) lsof -p 25247 shows that this process has open the file /lib/.x/s/mfs. This is the file where the sniffer is logging its output. | ||
 | The initd is located in /etc/opt/psybnc/. We don't need to spend any time at all analyzing this process since we know, just by looking at the name, that this is the ultimate script kiddie companion: an IRC bouncer (proxy). No script kiddie attack is complete without leaving behind an IRC bouncer so they can connect to IRC networks using compromised machines as proxies. All the psyBNC stuff is in the directory above, including the configuration and log files. |
As soon as I got to the console of the compromised machine I ran netstat -an (using a copy of netstat that I trust, of course) and obtained the following output:
As can be seen, the machine has non-standard ports open, like TCP 2003, 3128, 65336 and 65436 and UDP 3049. We can also see that there are three connections in the ESTABLISHED states. The details about these connections are:
| | 213.154.118.200 resolves to sanido-08.is.pcnet.ro, a machine in Romania. This IP address is allocated to an ISP in that country. |
| | 64.62.96.42 does not resolves to anything, but the destination port (TCP 6667) suggests that there was an IRC server running on that machine, and that our compromised machine had a connection open to that server. |
| | 199.184.165.133 resolves to undernet.irc.rcn.net. The destination port (TCP 6667) also suggests that our compromised machine had an established IRC session with that host. |
Now, as I've said before, even though the copy of netstat that I used comes from a trusted source and I know is not trojaned, it is possible that the running kernel has been trojaned via a Linux Kernel Module (LKM). To double check that the netstat is accurate a port scan from a another machine is called for.
peloy@pumpin:~$ sudo nmap -A -p 1- 192.168.1.79 Starting nmap 3.45 ( http://www.insecure.org/nmap/ ) at 2003-09-27 21:30 EDT Interesting ports on 192.168.1.79: (The 65523 ports scanned but not shown below are in state: closed) PORT STATE SERVICE VERSION 21/tcp open ftp? 22/tcp open ssh OpenSSH 2.9p2 (protocol 1.99) 23/tcp open telnet? 79/tcp open finger Linux fingerd 80/tcp open http? 113/tcp open ident OpenBSD identd 139/tcp open netbios-ssn Samba smbd (workgroup: MYGROUP) 443/tcp open https? 2003/tcp open ssh 3128/tcp open squid-http? 65336/tcp open unknown 65436/tcp open irc-proxy psyBNC 2.3.1 Device type: general purpose Running: Linux 2.4.X|2.5.X OS details: Linux Kernel 2.4.0 - 2.5.20 Uptime 1.266 days (since Fri Sep 26 15:09:53 2003) Nmap run completed -- 1 IP address (1 host up) scanned in 134.287 seconds peloy@pumpin:~$ |
As we can see, the port scan matches the output from netstat, i.e. the port scan shows as open the same ports that netstat shows.
Now, it is important to know what processes are responsibles for keeping these ports open. Ways of finding this information include the lsof, fuser and netstat commands. Here's part of the output from the netstat command with the -p switch:
The above output reveals several important details:
| | A process called xopen is responsible for the open TCP port 3128 and for the open UDP port 3049. |
| | The smbd -D process I mentioned before in Section 2.2, “Processes Analysis” is listening on several TCP ports: standard ports HTTP (80) and HTTPS (443), and the non-standard port 2003. Now, it is important to note that there is something strange going on here: I killed the smbd -D process and was expecting the ports 80 and 443 to be closed. However, a second run of netstat -anp showed that xopen was now the one listening on 80 and 443. |
| | A process called initd is responsible for the open TCP ports 65436 and 65336. |
The remaining open ports are handled by standard programs that are normally installed in a Unix server: xinetd, smbd (the real one this time), nmbd, sshd, sendmail and identd.
While security and computer forensics purists would recommend that this machine is brought down and reinstalled from scratch, my opinion is that this machine can be saved by reversing the actions performed by the attacker. The kernel was not trojaned, as the MD5 hashes show. An attempt to load a kernel rootkit (SucKIT) failed. While the log files were deleted, the actions performed by the attacker are very well understood thanks to the analysis of the MD5 hashes.
So, I would recommend that the trojaned system files (see Section 2.1.1.1, “Trojaned Files”) be restored from a backup, or from the directory /usr/lib/libshtift if there are no backups; that the files /etc/rc.d/init.d/functions and /etc/rc.d/rc.sysinit be edited to remove the calls to the malware installed by the attacker; that all files that were not present before the compromise be deleted; and that the machine is be rebooted. Finally, the machine should be patched with the latest updates to close the hole used to break in. This should bring the machine back to life with minimum disruptions.
A. Files
The following files where generated during this Scan of the Month:
md5-diffs.txt: this file contains the output from the diff command when run on a sorted version of the file provided by the Honeynet Project (that contains MD5 hashes of all files in the system before the honeypot was compromised), and a similar file but with hashes of all files in the system after the compromise. The exact procedure followed to generate this file is given in Section 2.1.1, “MD5 Hashes Analysis”.
crack-sk.c: Brute force password cracker for passwords embedded in SuCKIT rootkits.
XML sources for this document: this HTML document was generated using DocBook XML. This directory contains all the files used in the generation of this document.
B. References
Linux Kernel Rootkits: http://la-samhna.de/library/rootkits/index.html.
Excellent introduction to Linux kernel rootkits.
Exploiting Loadable Kernel Modules, by Michael Reiter: http://138pc222.sshunet.nl/honeypot/extra/expl_lkm.html.
Good information about the Adore Linux Kernel Module rootkit.
Adore LKM, by Stealth. 1999/2000. http://www.team-teso.net/releases/adore-0.31.tar.gz
Linux on-the-fly kernel patching without LKM: http://www.phrack.org/show.php?p=58&a=7.
Excellent article on Phrack Magazine that discusses the SucKIT rootkit the attacker attempted to install.
The Sarangworld Traceroute Project.
Nice tool to find possible geographical location of an IP address.
Excellent machine virtualization software.
C. Work Methodology
The submission was prepared in emacs as DocBook XML, and then converted to HTML using Tim Waugh's xmlto XML converter. VMware was obviously used a lot. I wish there was an open source alternative.
D. Thanks
Thanks to ...
Chapu, for somehow putting up with me while I worked on this.
To the one that sent Daniela into our lives.
As always, the Honeynet Project for these nice, entertaining, and educational challenges.