========================================================================== Question 1: Show step by step how you identify and recover the deleted rootkit from the / partition. ========================================================================== Summary: -------- Browsing the file system image with Autopsy and looking for deleted files, there shows up /lk.tgz, stored in inode 23. Retrieving data from the inode with "icat", uncompressing and untaring it, finally reveals typical parts of a rootkit: trojaned tools, a sniffer and an installation script, which attempts to replace some system programs with their trojaned counterparts. To back up this supposition, modification, access and creation timestamps were obtained from the mounted filesystem image, for both undeleted and deleted files. About 8 minutes after /lk.tgz has been modified for the last time, some system utilities like /bin/netstat and /bin/ps were first accessed, then overwritten. Files were created in /dev/ida/.drag-on and '/dev/ida/.. '. Sendmail configuration files were accessed, so possibly a mail has been sent. The suspicious file, /lk.tgz, has been deleted. Replacement of some system binaries indicates malicious activities. This seems to be related with lk.tgz. The analysis of MAC-timestamps resembles what has to be expected based on the installation script found in lk.tgz in inode 23. Conclusion: inode 23 contains the rootkit package, named /lk.tgz. Details: -------- I took this challenge as an opportunity to try out Autopsy, a tool to visualize data produced by The Coroner's Toolkit (TCT) and TCTUTILS. Autopsy 1.00 and TCTUTILS 1.00 are available from www.cerias.purdue.edu/homes/carrier/forensics First, the image is retrieved, unpacked and verified: # wget http://project.honeynet.org/scans/scan15/honeynet.tar.gz # md5sum honeynet.tar.gz 0dff8fb9fe022ea80d8f1a4e4ae33e21 honeynet.tar.gz # tar xzf honeynet.tar.gz # cd honeynet # md5sum honeypot.hda8.dd 5a8ebf5725b15e563c825be85f2f852e honeypot.hda8.dd Then, Autopsy is configured as explained in its README. As the timezone of the compromised system is unknown, the time difference is set to 0. (Investigator's TZ is GMT+02) The image honeypot.hda8.dd is loaded into Autopsy. Selecting "All Deleted Files" displays some files. Three of them have been deleted at the same time: /etc/mtab~ : appears to be a LRK4/5-style configuration file /lk.tgz : based on its name and size of half an MB this could be the rootkit /last : a directory While looking at the directory tree presented by Autopsy, there's a suspicious directory named ".drag-on" hidden in /dev/ida. This directory contains sshd configuration files, a sniffer and some utilities. This may be an installed rootkit. Further examination of lk.tkz is needed. Autopsy shows this file stored in inode 23 (please see also enclosed "exhibit-1.txt"). The filesystem image is mounted, then inode 23 is copied into a file "lk.tgz" using "icat" of The Coroner's Toolkit: # mount -o ro,loop,nodev,noexec honeypot.hda8.dd /mnt/t # icat /dev/loop0 23 > /var/forensics/scan15/lk.tgz # md5sum lk.tgz 115f438631de8d0a7c03c9d458eb7257 lk.tgz This matches the checksum given by Autopsy. The contents of lk.tgz is discussed in answer 2, MAC-timestamp analysis will be performed in the bonus round. ========================================================================== Question 2: What files make up the deleted rootkit? ========================================================================== Summary: -------- The rootkit contains trojaned replacement files for some system binaries, attempting to hide files and data communication. However, this attempt fails, because configuration files aren't named properly. Further the kit contains a version 1.2.27 sshd with a backdoor; password is "Frunza14" (w/o the quotes). The configuration file used by the blackhat instructs the daemon to store its pidfile in a non-existing directory. By chance, the daemon refuses to run. Also provided is linsniffer with some utility scripts and sl2, the well known DoS-tool Slice2. A cgi-backdoor isn't installed properly. An install script creates additional files and attempts to send to mails, beside replacing system files with modified or trojaned ones. Details: -------- The retrieved archive is unpacked and examined using tar: # tar tvzf lk.tgz drwxr-xr-x 1031/users 0 2001-02-26 21:40:30 last/ tar: Archive contains future timestamp 2002-02-08 14:08:13 -rwxr-xr-x 1031/users 611931 2002-02-08 14:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 16:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-03 04:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 16:22:50 last/linsniffer -rwxr-xr-x 1031/users 1345 1999-09-09 17:57:11 last/cleaner -rw-r--r-- 1031/users 3278 2001-01-27 16:11:32 last/inetd.conf -rwxr-xr-x 1031/users 79 2001-02-26 16:28:40 last/lsattr -rw-r--r-- 1031/users 11407 2001-01-27 16:11:44 last/services -rwxr-xr-x 1031/users 4060 2001-02-26 16:22:55 last/sense -rw-r--r-- 1031/users 880 2000-10-22 21:29:44 last/ssh_config -rw------- 1031/users 540 2000-10-22 21:29:44 last/ssh_host_key -rw-r--r-- 1031/users 344 2000-10-22 21:29:44 last/ssh_host_key.pub -rw------- 1031/users 512 2000-10-22 21:29:44 last/ssh_random_seed -rw-r--r-- 1031/users 688 2001-02-26 16:29:51 last/sshd_config -rwx------ 1031/users 8268 2001-02-26 16:22:59 last/sl2 -rwxr-xr-x 1031/users 4620 2001-02-26 16:23:10 last/last.cgi -rwxr-xr-x 1031/users 33280 2001-02-26 16:23:33 last/ps -rwxr-xr-x 1031/users 35300 2001-02-26 16:23:42 last/netstat -rwxr-xr-x 1031/users 19840 2001-02-26 16:23:47 last/ifconfig -rwxr-xr-x 1031/users 53588 2001-02-26 16:23:55 last/top -rwx------ 1031/users 75 2001-02-26 16:24:03 last/logclear -rw-r--r-- root/root 708 2001-03-03 04:05:12 last/s -rwxr-xr-x 1031/users 632066 2001-02-26 15:46:04 last/mkxfs # tar xzf lk.tgz This creates a directory "last" in the current working directory. # cd last # md5sum * 12e8748c19abe7a44e67196c22738e9b cleaner 086394958255553f6f38684dad97869e ifconfig b63485e42035328c0d900a71ff2e6bd7 inetd.conf 964db5da8cf89810a54659b6fdb81958 install 202a51b16ac8d1b4dc75de89e7344ed4 last.cgi 6c0f96c1e43a23a21264f924ae732273 linsniffer 5f22ceb87631fbcbf32e59234feeaa5b logclear dfb2eeea2a5ba23eb6a2b9d0cff9d82f lsattr 18a2d7d3178f321b881e7c493af72996 mkxfs 2b07576213c1c8b942451459b3dc4903 netstat 68b329da9893e34099c7d8ad5cb9c940 pidfile 7728c15d89f27e376950f96a7510bf0f ps 06d04fa3c4941b398756d029de75770e s 464dc23cac477c43418eb8d3ef087065 sense 54e41f035e026f439d4188759b210f07 services 4cfae8c44a6d1ede669d41fc320c7325 sl2 21ed3ca31a9c9b51a757f1644e26f2f7 ssh 5fd2ce512e0eba4d090191e8a1518808 ssh_config c2c1b08498ed71a908c581d634832672 ssh_host_key e76cd5baaab7b4f28c999946a9cb4dcc ssh_host_key.pub ad265d3c07dea3151bacb6930e0b72d3 ssh_random_seed 312de877e5180678cd54606e1c25af40 sshd_config 8ff0939cd49a0b2ef3156c7876afca4b top Determine the file types (output slightly edited for clarity) # file * cleaner: Bourne-Again shell script text ifconfig: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped inetd.conf: English text install: Bourne shell script text last.cgi: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped linsniffer: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped logclear: ASCII text lsattr: Bourne shell script text mkxfs: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped netstat: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped pidfile: PCX image data, version 2.5 ps: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped s: ASCII text sense: perl script text services: English text sl2: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped ssh: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped ssh_config: English text ssh_host_key: data ssh_host_key.pub: ASCII text ssh_random_seed: data sshd_config: ASCII text top: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped Analysis of the files: cleaner: -------- A comment at the beginning identifies this shell script as "sauber - by socked [11.02.99]". This script attempts to remove indications of the blackhat from utmp, wtmp and logfiles in /var/log. ifconfig: --------- Modified version of the well known tool, which doesn't indicate if an interface is in promiscous mode or not (it doens't contain the string PROMISC ususally found in ifconfig). inetd.conf: ----------- A stripped down configuration for the internet super-server, only telnet and pop3 are serviced by inetd. install: -------- The installation script, which moves the trojaned file into place and creates neccessary configuration files. It first checks, if the following regular files exist: /usr/bin/make /usr/bin/gcc /usr/sbin/sshd/ (sic!) The test for sshd usually fails due to the trailing slash, but the script doesn't care anyway. Next, it replaces the following programs with trojaned versions: /sbin/ifconfig /bin/netstat /bin/ps /usr/bin/top Then it copies the trojaned ssh daemon "mkxfs" to /usr/sbin/. Afterwards it creates two files, controlling the behaviour of the trojaned ifconfig, netstat, ps and top. --- /dev/rpm --- 3 sl2 3 sshdu 3 linsniffer 3 smurf 3 slice 3 mech 3 muh 3 bnc 3 psybnc ----------------- --- /dev/last --- 1 193.231.139 1 213.154.137 1 193.254.34 3 48744 3 3666 3 31221 3 22546 4 48744 4 2222 ----------------- The script creates the directories /dev/ida/.drag-on and /dev/ida/".. " (dot-dot-space) First it copies the following files to /dev/ida/.drag-on: linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed Then, it copies the same files in the very same order to /dev/ida/".. ". Afterwards, it deletes these files from the installation directory /last. By touching, install creates the following two files /dev/ida/.drag-on/tcp.log /dev/ida/".. "/tcp.log It replaces /etc/inetd.conf and /etc/services with it's own versions and restarts inetd. It replaces /usr/sbin/lsattr (used to show ext2fs sile attributes) with its own version (which starts the ssh daemon "mkxfs" and the sniffer) and appends a call to lsattr to /etc/rc.d/rc.sysinit. It changes the mode of lsattr to 500 and makes the file, if stored on an ext2fs, immutable. Afterwards, it calls lsattr itself, initializing linsniff and the ssh daemon. Next, install checks for some common cgi-bin directories and drops "last.cgi", an http backdoor, there. It neglects to change its mode, so last.cgi won't run suid root, but with privileges of httpd. It then mails a status report to last@linuxmail.org and bidi_damm@yahoo.com. The report covers - OS version information (uname -a) - FQDN (hostname -f) - ip addresses bound to the interfaces - uptime - CPU vendor id (from /proc/cpuinfo) - CPU model (from /proc/cpuinfo) - CPU speed (from /proc/cpuinfo) - Bogomips (from /proc/cpuinfo) - available disk space (df -h) After all, it deletes the archive, the directory "/last" holding the extracted files and the temporary file "computer" used for mailing the status report. last.cgi: --------- Backdoor, used to execute commands via http. linsniffer: ----------- A well known password sniffer. Based on a string inside the executable, output by default goes to tcp.log (which is common). logclear: --------- This script stops linsniffer, removes the log and launches the sniffer again. lsattr: ------- This shell script attempts to start the trojaned ssh daemon "mkxfs" with config file "s" and linsnifer. mkxfs: ------ Based on strings output, this is a version 1.2.27 ssh daemon. Interesting is the string "Frunza14". Execution in a controlled environment proves this to be a password opening a backtoor to root access and supressing access logging. netstat: -------- Trojaned version of the well known utility. I expected to see a reference to /dev/last. Instead netstat expects its configuration in /dev/caca. So it won't behave as expected by the blackhat. pidfile: -------- pid of the sshd. According to sshd_config, the daemon creates this file as needed in /dev/ida/.drag-on/, so it's useless here. ps: --- This file appears to be a trojaned version of ps. Again, the blackhat messed up the configuration: as it can be seen by seraching strings output for /dev/, the trojaned ps want's to read its configuration from /dev/dsx. The install scripts writes the config to /dev/rpm instead. s: -- Just another sshd configuration. The daemon will bind to port 5, the pidfile is now stored in /dev/ida/.inet - which does not exist! Does this affect the daemon? sense: ------ This perl script sorts and condenses the output of linsniffer. This is version 0.03 [BETA] by Mike Edulla services: --------- The blackhat's replacement for /etc/services. How does it differ from the original version? At least it doesn't contain an entry for port 5/tcp as it may be exptected (see "s" above). sl2: ---- This seems to be the SYN flooder Slice2. ssh: ---- ssh client ssh_config: ----------- Configuration file for ssh. ssh_host_key: ------------- ssh private key. ssh_host_key.pub: ----------------- ssh public key. ssh_random_seed: ---------------- This file gets written by the ssh daemon. sshd_config: ------------ The original ssh daemon configuration. The blackhat doesn't use this config! (see "s") top: ---- A trojaned version of "top". Again, it depends on /dev/dsx, while the blackhat stores the configuration in /dev/rpm. ========================================================================== Bonus Question: Was the rootkit ever actually installed on the system? How do you know? ========================================================================== Summary: -------- Yes, the rootkit was installed. System files are identical to those contained in the rootkit archive or have been altered accordingly. MAC-timestamp analysis documents one run of the install script. Linsniffer has logged two logins (one telnet, one ftp). Two mails have been sent, containing information about the compromised system. Details: -------- As already mentioned, the install script overwrites system files, modifies and creates other ones. To determine whether install has been run on the system, files found in the filesystem are compared with those found in the reconstructed rootkit archive. /sbin/ifconfig: match 086394958255553f6f38684dad97869e /mnt/t/sbin/ifconfig (given filesystem) 086394958255553f6f38684dad97869e ifconfig (reconstructed rootkit) /bin/netstat: match 2b07576213c1c8b942451459b3dc4903 /mnt/t/bin/netstat 2b07576213c1c8b942451459b3dc4903 netstat /bin/ps: match 7728c15d89f27e376950f96a7510bf0f /mnt/t/bin/ps 7728c15d89f27e376950f96a7510bf0f ps /etc/services: match 54e41f035e026f439d4188759b210f07 /mnt/t/etc/services 086394958255553f6f38684dad97869e ifconfig /etc/inetd.conf: match b63485e42035328c0d900a71ff2e6bd7 /mnt/t/etc/inetd.conf b63485e42035328c0d900a71ff2e6bd7 inetd.conf Checks on files located in /usr could not be performed likewise, because this part of the filesystem is not available. Both - in fact unused - rootkit configuration files, /dev/rpm and /dev/last exist. They contain data as expected by looking at "install". The last line of the system startup script /etc/rc.d/rc.sysinit contains the call to /usr/bin/lsattr. Both directories /dev/ida/.drag-on and /dev/ida/".. " exist and contain files identical to those from the rootkit archive: # md5sum /mnt/t/dev/ida/.drag-on/* 6c0f96c1e43a23a21264f924ae732273 /mnt/t/dev/ida/.drag-on/linsniffer 5f22ceb87631fbcbf32e59234feeaa5b /mnt/t/dev/ida/.drag-on/logclear 18a2d7d3178f321b881e7c493af72996 /mnt/t/dev/ida/.drag-on/mkxfs 06d04fa3c4941b398756d029de75770e /mnt/t/dev/ida/.drag-on/s 464dc23cac477c43418eb8d3ef087065 /mnt/t/dev/ida/.drag-on/sense 4cfae8c44a6d1ede669d41fc320c7325 /mnt/t/dev/ida/.drag-on/sl2 c2c1b08498ed71a908c581d634832672 /mnt/t/dev/ida/.drag-on/ssh_host_key 97df00e63da3efd62fac4f84802370f0 /mnt/t/dev/ida/.drag-on/ssh_random_seed 7754ec4105e2b56a4ec6721f522d580d /mnt/t/dev/ida/.drag-on/tcp.log (not copied) # md5sum /mnt/t/dev/ida/".. "/* 6c0f96c1e43a23a21264f924ae732273 /mnt/t/dev/ida/.. /linsniffer 5f22ceb87631fbcbf32e59234feeaa5b /mnt/t/dev/ida/.. /logclear 18a2d7d3178f321b881e7c493af72996 /mnt/t/dev/ida/.. /mkxfs 06d04fa3c4941b398756d029de75770e /mnt/t/dev/ida/.. /s 464dc23cac477c43418eb8d3ef087065 /mnt/t/dev/ida/.. /sense 4cfae8c44a6d1ede669d41fc320c7325 /mnt/t/dev/ida/.. /sl2 c2c1b08498ed71a908c581d634832672 /mnt/t/dev/ida/.. /ssh_host_key ad265d3c07dea3151bacb6930e0b72d3 /mnt/t/dev/ida/.. /ssh_random_seed d41d8cd98f00b204e9800998ecf8427e /mnt/t/dev/ida/.. /tcp.log (not copied) Only the sniffer in /dev/ida/.drag-on is activated. It recorded a ftp login from cr272065-a.wlfdle1.on.wave.home.com and a telnet login from ns2.giant.net. To recover the mails sent, the filesystem is searched for the string "Spatiu Liber". Autopsy finds it in blocks 90417 (exhibit-2.txt) and 90418 (exhibit-3.txt). Another approach would use "unrm" to extract slack space from the filesystem and let lazarus identify the blocks as mail data. As already shown, files in the saved filesystem are matching those from the reconstructed rootkit. However, this could be done by hand at different times. So in addition the timeline of events has to be reconstructed. At first, the MAC timestamps from undeleted files are analysed with "mactimes" of TCT: #./tct/bin/mactime \ -Rd /mnt/t \ -g /mnt/t/etc/group \ -p /mnt/t/etc/passwd \ 03.15.2001 \ > /var/forensics/scan15/existing.mac Likewise, an analysis is performed for deleted files, using "fls" from TCTUTILS: getting MAC times from deleted files # ./tctutils/bin/fls \ -m /mnt/t \ /var/forensics/scan15/honeypot.hda8.dd 2 \ > /var/forensics/scan15/deleted.mac Both timelines are merged with "mac_merge" from TCPUTILS: mergeing both files # ./tctutils/bin/mac_merge \ /var/forensics/scan15/existing.mac \ /var/forensics/scan15/deleted.mac \ > /var/forensics/scan15/all.mac.txt The merged timeline is provided for your convenience. Because fls doesn't convert uid/gid numbers to names, while mactime was instructed to do so, you'll be able to distinguish deleted from undeleted files at a glance. At "Mar 16 01 02:36:48" (again investigator's timezone is GMT+02) a possible transfer or build of /lk.tgz has been completed. Eight minutes later, /bin/netstat and /bin/ps are accessed. Possibly the blackhat tries out his rootkit and is dissatisfied by the results. So, he decides to reinstall it. But why are no traces of tar and gzip visible at this point? Because they are used later (Mar 16 01 15:47:01), which updates the access timestamp. Expected occurances of tests for make and gcc are missing, because both executables are located under /usr, which has not been analysed. The invocation of /last/install is missing, though. The reconstructed timeline shows the creation of both directories hidden under /dev/ida within a second as well as the modification of /etc/services, /etc/inetd.conf and /etc/rc.d/rc.sysinit. It also displays the gathering of system information (df, ifconfig), the access to "mail" and sendmail configuration files sendmail.cf and sendmail.cw. During the installation of rootkit files, /etc/mtab~ gets deleted. This file can be reconstructed using Autopsy (exhibit-4.txt) and contains the same data as /dev/rpm. There is no reference to this file in the installation script. Possiblly the blackhat deleted this file by hand during the execution of install, or closely before or after. Though coincidence is not perfect, the timeline reflects the behaviour to be expected from "install".