Honeynet Scan 15 (May) Challenge
John Edward Scott <jes@daedalus-soln.demon.co.uk>
Abstract
On 15 March 2001, A Linux honeypot was successfully compromised, a rootkit was downloaded to the / partition and then deleted from the system. The purpose of this paper is to show step by step how the rootkit was recovered and identified from the the system and which files make up the rootkit. If possible, it will be shown whether the rootkit was actually installed on the system.
Tools used for the investigation
A variety of tools were used in this investigation ranging from commonly used Unix commands to custom tools
- The Unix strings command was used for viewing human-readable text from both binary files and the entire / partition image
- The Unix file command was used for determining file types of recovered files
- The Coroners Toolkit by Dan Farmer and Wietse Venema
Accessing the Partition
In order to view the / partition we need to mount it onto a "known" "clean" machine, after successfully downloading, untarring and confirming that the image is the one that was provided by Honeynet (via the MD5 checksum), the following command was used to mount the partition
mount -o ro,loop,nodev,noexec honeypot.hda8.dd /mnt/honeynet
We now have the / partition mounted at /mnt/honeynet. Note that the options used with the mount command were VERY important, since we don't accidently want to be able to execute any of the commands from the compromised partition and accidently infect our "clean" system (since at this point we don't know what we're dealing with).
Resurrecting the Dead
Using tools from the Coroners Toolkit, we can reconstruct any free but non-empty inodes from the disk image. Whilst in the directory containing the honeypot.hda8.dd image, the following commands were executed
We now have (in the hda8 subdirectory) the free but non-empty inodes resurrected for us to look through.
Using the Unix file command we can determine the filetype of each of the recovered files, just for ease of use (and permanent record), the following command was used whilst in the hda8 subdirectory
file * > /tmp/recovered.filetypes
Looking through this file we can see that there are a variety of recovered files, including empty files, files containing ASCII text, shell scripts and data. There are also a number of recovered binary files.
The recovered files
I won't discuss all of the recovered files here, only the more "interesting" ones.
- 16110 - This appears to be the modified
- 2041 - This appears to be a shell script used for installing the Rootkit. The banner at the top of the file is hardly discrete and reading through the script it's not difficult to follow what it does. A check is made to determine if ssh is used on the machine. Trojaned versions of ifconfig, netstat, ps and top are installed. Then some control files are created (namely /dev/rpm and /dev/last) which are used by the trojaned programs. Hidden directories are then created in "/dev/ida/.drag-on" and "/dev/ida/.. " (note the trailing space). Linsniffer (a commonly used network sniffer) and its various control files are then copied into these two hidden directories (why two?). Modified versions of inetd.conf and services are then copied to the /etc directory. A trojaned copy of lsattr is then copied to /usr/bin and a line is added to /etc/rc.d/rc.sysinit in order to start the program on system startup. It's not clear what the modified version of lsattr will do, however a quick check shows that the options used to start the program in /etc/rc.d/rc.sysinit do not exist in the regular lsattr program. A few backdoors are then added to any existing webserver and details of the rootkit install are mailed back to a couple of email addresses (last@linuxmail.org, bidi_damm@yahoo.com). The script then finally clears up, deleting the last, lk.tgz, computer and lk.tar.gz files
- 2043 - This script cleans up the logfiles to erase any traces of the intrusion
- 2044 - This is the deleted copy of the inetd.conf file
- 2045 - This script is used to start the mkxfs and linsniffer tools
- 2046 - This is the deleted copy of the services file
- 2047 - A perl script used to sort the output from the Linsniffer tool
- 2048 - An ssh client config file
- 2050 - Appears to be a key file for ssh, note the user listed in here root@dil2.datainfosys.net
- 2052 - The ssh server config file, note that this uses keys stored in the /dev/ida/.drag-on/ directory, a clear sign that this is part of the rootkit. Also note that it WILL allow root logins.
- 2060 - Another ssh server config file. This time to run on a different port (5)
- 23 - GZiped Compressed data. Unzipped and untarring this file produces the rootkit (Hurrah!). There are trojaned versions of the binaries along with an install script. We can do a diff between the 2041 file and the install script and see that they are identical. We can also do a diff between the trojaned binaries and the files they would replace to see that they are identical, indicating that the trojaned binaries have been installed. We can also use diff to match up the recovery binary files with the binary files in the rootkit.
What was contained in the Rootkit?
By using the Coroners Toolkit we were able to reconstruct the deleted files. This allowed us to recover the original tarball containing the Rootkit, the contents of which are shown below
- cleaner - Log file cleaner
- ifconfig - Trojan
- inetd.conf - Replacement for /etc/inetd.conf
- install - Installation script
- last.cgi - Backdoor for webserver
- linsniffer - Network sniffer
- logclear - Script to start the sniffer
- lsattr - Script to start linsniffer and mkxfs
- mkxfs - Trojan
- netstat - Trojan
- pidfile - Control file
- ps - Trojan
- s - SSH control file
- sense - Linsniffer utility script
- services - Replacement for /etc/services
- sl2 - Trojan
- ssh - SSH Executable
- ssh_config - SSH control file
- ssh_host_key - SSH key file
- ssh_host_key.pub - SSH key file
- ssh_random_seed - SSH key file
- sshd_config - SSH control file
- top - Trojan
Was the Rootkit installed on the system?
Looking around the filesystem we have already discovered that the directories "/dev/ida/.drag-on" and "/dev/ida/.. " exist. This would correspond with the Rootkit Installation script having been executed. Both directories also contain the files that we would expect the Installation script to have copied there. However we need to check that all of the rootkit was installed (there's always a chance that the install script could have failed for some reason).
To do this, we can compare the Rootkit binaries with the "in-situ" versions of the binaries. As a double precaution we use both the Unix diff command and MD5 checksums. The following command was executed from the directory containing the Rootkit binaries (after untarring the tarball).
diff ps /mnt/honeynet/bin/ps
and it produced no output (indicating the files were the same). Comparing the MD5 checksums gives us
md5 ps - gives 7728c15d89f27e376950f96a7510bf0f
md5 /mnt/honeynet/bin/ps - gives 7728c15d89f27e376950f96a7510bf0f
Therefore the MD5 checksums are also equal. To further prove the point we can compare the size of both binaries and they are indeed equal. Performing these actions on all the binaries contained in the Rootkit confirms that all the trojaned binaries have been copied into the relevant directories.
We can also see if we examine the /etc/rc.d/rc.sysinit file that the very last line is the line that has been added by the Rootkit install script (namely /usr/bin/lsattr -t1 -X53 -p
There is another trick we can perform using the incredibly useful Coroners Toolkit, by executing the following command
unrm honeypot.hda8.dd > unrm.txt
Now we can search through this file looking for evidence that the emails that should have been produced by the install script were actually created. Using a pager (since the file is huge) we can search for the string roote (this was one of the email address recipients specified, we do indeed find that the email has been created (and presumably sent).
We have now confirmed in a number of different ways that the Rootkit was indeed installed.