From ocrow@io.com Fri May 25 18:32:48 2001 Date: Fri, 25 May 2001 15:31:05 -0500 From: Owen Crow To: project@honeynet.org Subject: Scan of the Month 15 Analysis Scan of the Month 15 Analysis ________________________________________________________________________________ SHOW STEP BY STEP HOW YOU IDENTIFY AND RECOVER THE DELETED ROOTKIT FROM THE / PARTITION. Note: Most of this is verbatim from Dave's analysis of the forensics challenge. Thanks Dave. 1. Downloaded and extracted the honeypot.hda8.dd image. Verified md5sums for archive and extracted file 2. Created a mount.sh script to mount the image: #!/bin/sh mkdir /t 2>/dev/null mount -o ro,loop,nodev,noexec honeypot.hda8.dd /t 3. Downloaded the latest version of the Coroner's Toolkit and compiled. 4. Added the new binaries to my PATH. 5. Ran ils since the file I'm looking for was deleted: # ils honeypot.hda8.dd | ils2mac > body-deleted Recovered all of the deleted files using icat and this script: #!/bin/sh -ex mkdir files for file in `perl -ne 'print "$1\n" if /<(.*)>/' body-deleted` do icat honeypot.hda8.dd ${file#honeypot.hda8.dd-dead-} \ > files/$file done 6. 7. Looked for compressed or non-compressed archive files in the recovered files: # cd files # file * | grep -v empty honeypot.hda8.dd-dead-23: gzip compressed data, deflated, last modified: Fri Mar 2 21:09:06 2001, os: Unix honeypot.hda8.dd-dead-2039: ELF 32-bit LSB executable, Intel 80386, version 1 honeypot.hda8.dd-dead-2040: ASCII text honeypot.hda8.dd-dead-2041: Bourne shell script text honeypot.hda8.dd-dead-2042: ELF 32-bit LSB executable, Intel 80386, version 1, stripped honeypot.hda8.dd-dead-2043: Bourne-Again shell script text honeypot.hda8.dd-dead-2044: English text honeypot.hda8.dd-dead-2045: Bourne shell script text honeypot.hda8.dd-dead-2046: English text honeypot.hda8.dd-dead-2047: perl commands text honeypot.hda8.dd-dead-2048: English text honeypot.hda8.dd-dead-2049: data honeypot.hda8.dd-dead-2050: ASCII text honeypot.hda8.dd-dead-2051: data honeypot.hda8.dd-dead-2052: ASCII text honeypot.hda8.dd-dead-2053: ELF 32-bit LSB executable, Intel 80386, version 1 honeypot.hda8.dd-dead-2054: ELF 32-bit LSB executable, Intel 80386, version 1, stripped honeypot.hda8.dd-dead-2058: ELF 32-bit LSB executable, Intel 80386, version 1, stripped honeypot.hda8.dd-dead-2059: ASCII text honeypot.hda8.dd-dead-2060: ASCII text honeypot.hda8.dd-dead-2061: ELF 32-bit LSB executable, Intel 80386, version 1 honeypot.hda8.dd-dead-16110: ASCII text honeypot.hda8.dd-dead-30188: ELF 32-bit LSB executable, Intel 80386, version 1, stripped honeypot.hda8.dd-dead-30191: ELF 32-bit LSB executable, Intel 80386, version 1, stripped honeypot.hda8.dd-dead-48284: ELF 32-bit LSB executable, Intel 80386, version 1, stripped honeypot.hda8.dd-dead-56231: ASCII text honeypot.hda8.dd-dead-8100: English text 7. Checked the most likely file: # tar tzvf honeypot.hda8.dd-dead-23 drwxr-xr-x 1031/users 0 2001-02-26 14:40:30 last/ tar: Archive contains future timestamp 2002-02-08 07:08:13 -rwxr-xr-x 1031/users 611931 2002-02-08 07:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 09:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-02 21:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 09:22:50 last/linsniffer -rwxr-xr-x 1031/users 1345 1999-09-09 10:57:11 last/cleaner -rw-r--r-- 1031/users 3278 2001-01-27 09:11:32 last/inetd.conf -rwxr-xr-x 1031/users 79 2001-02-26 09:28:40 last/lsattr -rw-r--r-- 1031/users 11407 2001-01-27 09:11:44 last/services -rwxr-xr-x 1031/users 4060 2001-02-26 09:22:55 last/sense -rw-r--r-- 1031/users 880 2000-10-22 14:29:44 last/ssh_config -rw------- 1031/users 540 2000-10-22 14:29:44 last/ssh_host_key -rw-r--r-- 1031/users 344 2000-10-22 14:29:44 last/ssh_host_key.pub -rw------- 1031/users 512 2000-10-22 14:29:44 last/ssh_random_seed -rw-r--r-- 1031/users 688 2001-02-26 09:29:51 last/sshd_config -rwx------ 1031/users 8268 2001-02-26 09:22:59 last/sl2 -rwxr-xr-x 1031/users 4620 2001-02-26 09:23:10 last/last.cgi -rwxr-xr-x 1031/users 33280 2001-02-26 09:23:33 last/ps -rwxr-xr-x 1031/users 35300 2001-02-26 09:23:42 last/netstat -rwxr-xr-x 1031/users 19840 2001-02-26 09:23:47 last/ifconfig -rwxr-xr-x 1031/users 53588 2001-02-26 09:23:55 last/top -rwx------ 1031/users 75 2001-02-26 09:24:03 last/logclear -rw-r--r-- root/root 708 2001-03-02 21:05:12 last/s -rwxr-xr-x 1031/users 632066 2001-02-26 08:46:04 last/mkxfs 8. Created a link (ln honeypot.hda8.dd-dead-23 last.tar.gz) for my info. 9. Extracted the files with: # tar xzf last.tar.gz WHAT FILES MAKE UP THE DELETED ROOTKIT? Ran the file(1) command to get an idea of what the files were: # file * cleaner: Bourne-Again shell script text ifconfig: ELF 32-bit LSB executable, Intel 80386, version 1, stripped inetd.conf: English text install: Bourne shell script text last.cgi: ELF 32-bit LSB executable, Intel 80386, version 1, stripped linsniffer: ELF 32-bit LSB executable, Intel 80386, version 1 logclear: ASCII text lsattr: Bourne shell script text mkxfs: ELF 32-bit LSB executable, Intel 80386, version 1 netstat: ELF 32-bit LSB executable, Intel 80386, version 1, stripped pidfile: ASCII text ps: ELF 32-bit LSB executable, Intel 80386, version 1, stripped s: ASCII text sense: perl commands text services: English text sl2: ELF 32-bit LSB executable, Intel 80386, version 1 ssh: ELF 32-bit LSB executable, Intel 80386, version 1 ssh_config: English text ssh_host_key: data ssh_host_key.pub: ASCII text ssh_random_seed: data sshd_config: ASCII text top: ELF 32-bit LSB executable, Intel 80386, version 1, stripped WAS THE ROOTKIT EVER ACTUALLY INSTALLED ON THE SYSTEM? HOW DO YOU KNOW? Yes, at least partially. 1. Calculated an MD5 checksum on all the files in the mounted filesystem image with: #!/bin/sh -ex find /t -type f | while read file do md5sum "$file" done > t.md5 (I used xargs at first, but there were files with whitespace. After reading up on GNU xargs, I could have used -0, so now I know.) 2. Checked to see if any of the files in the rootkit were in the filesystem with: #!/bin/sh for file in files/last/* do echo "$file: " sum=`md5sum "$file" | awk '{print $1}'` grep -h $sum t.md5 done Which showed: files/last/cleaner: files/last/ifconfig: 086394958255553f6f38684dad97869e /t/sbin/ifconfig files/last/inetd.conf: b63485e42035328c0d900a71ff2e6bd7 /t/etc/inetd.conf files/last/install: files/last/last.cgi: files/last/linsniffer: 6c0f96c1e43a23a21264f924ae732273 /t/dev/ida/.drag-on/linsniffer 6c0f96c1e43a23a21264f924ae732273 /t/dev/ida/.. /linsniffer files/last/logclear: 5f22ceb87631fbcbf32e59234feeaa5b /t/dev/ida/.drag-on/logclear 5f22ceb87631fbcbf32e59234feeaa5b /t/dev/ida/.. /logclear files/last/lsattr: files/last/mkxfs: 18a2d7d3178f321b881e7c493af72996 /t/dev/ida/.drag-on/mkxfs 18a2d7d3178f321b881e7c493af72996 /t/dev/ida/.. /mkxfs files/last/netstat: 2b07576213c1c8b942451459b3dc4903 /t/bin/netstat files/last/pidfile: 68b329da9893e34099c7d8ad5cb9c940 /t/etc/at.deny files/last/ps: 7728c15d89f27e376950f96a7510bf0f /t/bin/ps files/last/s: 06d04fa3c4941b398756d029de75770e /t/dev/ida/.drag-on/s 06d04fa3c4941b398756d029de75770e /t/dev/ida/.. /s files/last/sense: 464dc23cac477c43418eb8d3ef087065 /t/dev/ida/.drag-on/sense 464dc23cac477c43418eb8d3ef087065 /t/dev/ida/.. /sense files/last/services: 54e41f035e026f439d4188759b210f07 /t/etc/services files/last/sl2: 4cfae8c44a6d1ede669d41fc320c7325 /t/dev/ida/.drag-on/sl2 4cfae8c44a6d1ede669d41fc320c7325 /t/dev/ida/.. /sl2 files/last/ssh: files/last/ssh_config: files/last/ssh_host_key: c2c1b08498ed71a908c581d634832672 /t/dev/ida/.drag-on/ssh_host_key c2c1b08498ed71a908c581d634832672 /t/dev/ida/.. /ssh_host_key files/last/ssh_host_key.pub: files/last/ssh_random_seed: ad265d3c07dea3151bacb6930e0b72d3 /t/dev/ida/.. /ssh_random_seed files/last/sshd_config: files/last/top: 3. Here are brief comments on each file: Kit fileInstalled Location & Notes cleaner used during installation and removed (shows up as first half of file 8510.t.txt in grave-robber output) ifconfig /t/sbin/ifconfig probably used to hide promiscuous interfaces (original file is probably 165802.x.txt in grave-robber output) inetd.conf /t/etc/inetd.conf adds pop-3 service that points to /usr/cyrus/bin/pop3d. Might be another backdoor, but the installer program doesn't seem to install the binary to complete the job... install used during installation and removed (shows up as binary file 8499.p.txt in the grave-robber output) last.cgi probably in /var/www/cgi-bin which is not available (shows up as binary file 8547.x.txt in grave-robber output) linsniffer /t/dev/ida/.drag-on/linsniffer "/t/dev/ida/.. /linsniffer" the standard rootkit packet/password sniffer judging by the name (shows up as grave-robber file 8503.x.txt) logclear /t/dev/ida/.drag-on/logclear "/t/dev/ida/.. /logclear" short script to stop linsniffer, remove the logfile, and restart (shows up at the very end of grave-robber file 8601...txt) lsattr probably in /usr/bin which is not available (shows up as grave-robber file 8516...txt) mkxfs /t/dev/ida/.drag-on/mkxfs "/t/dev/ida/.. /mkxfs" from the references to sshd in the strings(1) output, this is probably a backdoor ssh server (seems to be partially contained in grave-robber file 8738.!.txt) netstat /t/bin/netstat probably modified to hide promiscuous interfaces, etc. (the original or trojan version seems to be grave-robber file 114994...txt) pidfile incorrectly identified /t/etc/at.deny, but interesting since both files contain a single LF (\0A) ps /t/bin/ps appears to be modified to ignore files running out of /dev/dsx, but that's just a guess. Another guess is that by placing the rootkit files in /dev/ida, the rootkit will lose some of it's stealth? This is probably a bad guess as this version seems to differ a lot from the stock Redhat binary. (original appears to be grave-robber file 115050...txt) s /t/dev/ida/.drag-on/s "/t/dev/ida/.. /s" a sshd_config file for mkxfs (probably) sense /t/dev/ida/.drag-on/sense "/t/dev/ida/.. /sense" from the top of the file: "Sorts the output from LinSniffer" services /t/etc/services looks like the standard Redhat services file. Probably included just in case the local services file was modified as many services don't work unless they are defined. sl2 /t/dev/ida/.drag-on/sl2 "/t/dev/ida/.. /sl2" I've found references that say sl2 is a DoS tool, but I didn't investigate further. ssh probably in /usr/bin or /usr/local/bin which are not available ssh_config possibly in /usr/local/etc which is not available ssh_host_key /t/dev/ida/.drag-on/ssh_host_key "/t/dev/ida/.. /ssh_host_key" The host file for sshd. ssh_host_key.pub possibly in /usr/local/etc ssh_random_seed "/t/dev/ida/.. /ssh_random_seed" sshd_config possibly in /usr/local/etc which is not available top probably in /usr/bin which is not available. Like ps, it has references to /dev/dsx probably for hiding processes. EXTRA Running john the ripper on the two interesting shadow entries yields: root:asdfjkl;:11396:0:99999:7:-1:-1:134540356 jjs:whatever:11396:0:99999:7:-1:-1:134540380 I'm not sure if jjs is an artifact of the break-in. COMMENTS I have relied on descriptions of the rootkit files from the files themselves. If this were a real investigation, I would spend the time to test the binaries, to verify they do what they claim. Much more analysis could be done to the grave-robber output, etc., but I think that goes beyond the scope of the Challenge. ________________________________________________________________________________ ocrow@io.com (Owen Crow) [ Part 2, Application/PGP-SIGNATURE 240bytes. ] [ Unable to print this part. ]