Scan of the month May submission by Jim Gray +*+ 1) Show step by step how you identify and recover the deleted rootkit from the / partition. * First step is to get the partiton mounted using loopback mechanism. # mount -o ro,loop,nodev,noexec /usr/local/forensics/morgue/honeypot.hda8.dd /mnt/forensics * Next step is to run graverobber against the mounted image. Note, a lot of the information gathered will not be relevant as it will pertain to the machine the image is mounted on. This will however generate a body file for the mounted image # /usr/local/tct/bin/grave-robber -c /mnt/forensics -m -d /usr/local/forensics/morgue/ -o LINUX2 * Next step is to generate a body file that consists of the unallocated inodes on the mounted image. # /usr/local/tct/bin/ils honeypot.hda8.dd | /usr/local/tct/extras/ils2mac > body-deleted * Create a body file that has mactimes for both free and claimed inodes # cat body body-deleted > body-full * Generate a mactime report using the body file that was just created. This report references the passwd and group files from the image. # /usr/local/tct/bin/mactime -p /mnt/forensics/etc/passwd -g /mnt/forensics/etc/group -b body-full 03/14/2001 >mactime.txt * Oops. The mactime script threw this error and needed a quick fix. In string, @GET_PASSWD now must be written as \@GET_PASSWD at /usr/local/tct-1.06/lib/pass.cache.pl line 313, near "$PASSWD) info via @GET_PASSWD * This time it runs. # /usr/local/tct/bin/mactime -p /mnt/forensics/etc/passwd -g /mnt/forensics/etc/group -b body-full 03/14/2001 >mactime.txt * Next run unrm. # bin/unrm /usr/local/forensics/morgue/honeypot.hda8.dd > /usr/local/forensics/morgue/unrm_output * Then run lazarus to sort it out. # ../../tct/bin/lazarus -h -D ./lazarus_output/blocks -w ./lazarus_output/www -H ./lazarus_output unrm_output * Next run fls on the root of the image. # ../../tctutils/bin/fls -m /mnt/forensics honeypot.hda8.dd 2 > fls-m.txt # ../../tctutils/bin/fls -al honeypot.hda8.dd 2 > fls-al.txt # ../../tctutils/bin/fls -r honeypot.hda8.dd 2 > fls-recursive.txt * Looking at the output of these fls commands my main question is what is this file named lk.tgz at inode 23? Recovered it using icat. Also interesting is the deleted directory named /last. # ../../tct/bin/icat -f ext2fs honeypot.hda8.dd 23 > lk.tgz +*+ 2) What files make up the deleted rootkit? * Got ms5sum of lk.tgz, gunzipped, created a manifest listing and untarred the thing. # md5sum lk.tgz > lk.tgz.md5sum 115f438631de8d0a7c03c9d458eb7257 lk.tgz # tar tvf lk.tar > lk.tar.manifest (with warning: tar: Archive contains future timestamp 2002-02-08 08:08:13) drwxr-xr-x 1031/users 0 2001-02-26 15:40:30 last/ -rwxr-xr-x 1031/users 611931 2002-02-08 08:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 10:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-02 22:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 10:22:50 last/linsniffer -rwxr-xr-x 1031/users 1345 1999-09-09 11:57:11 last/cleaner -rw-r--r-- 1031/users 3278 2001-01-27 10:11:32 last/inetd.conf -rwxr-xr-x 1031/users 79 2001-02-26 10:28:40 last/lsattr -rw-r--r-- 1031/users 11407 2001-01-27 10:11:44 last/services -rwxr-xr-x 1031/users 4060 2001-02-26 10:22:55 last/sense -rw-r--r-- 1031/users 880 2000-10-22 15:29:44 last/ssh_config -rw------- 1031/users 540 2000-10-22 15:29:44 last/ssh_host_key -rw-r--r-- 1031/users 344 2000-10-22 15:29:44 last/ssh_host_key.pub -rw------- 1031/users 512 2000-10-22 15:29:44 last/ssh_random_seed -rw-r--r-- 1031/users 688 2001-02-26 10:29:51 last/sshd_config -rwx------ 1031/users 8268 2001-02-26 10:22:59 last/sl2 -rwxr-xr-x 1031/users 4620 2001-02-26 10:23:10 last/last.cgi -rwxr-xr-x 1031/users 33280 2001-02-26 10:23:33 last/ps -rwxr-xr-x 1031/users 35300 2001-02-26 10:23:42 last/netstat -rwxr-xr-x 1031/users 19840 2001-02-26 10:23:47 last/ifconfig -rwxr-xr-x 1031/users 53588 2001-02-26 10:23:55 last/top -rwx------ 1031/users 75 2001-02-26 10:24:03 last/logclear -rw-r--r-- root/root 708 2001-03-02 22:05:12 last/s -rwxr-xr-x 1031/users 632066 2001-02-26 09:46:04 last/mkxfs * Looks like a rootkit. * Copied tar file to directory rootkit then unpacked it. # tar -xvf lk.tar * generated md5sums for all files in rootkit # cd last # md5sum `ls` > ../../rootkit.files.md5sums 12e8748c19abe7a44e67196c22738e9b cleaner 086394958255553f6f38684dad97869e ifconfig b63485e42035328c0d900a71ff2e6bd7 inetd.conf 964db5da8cf89810a54659b6fdb81958 install 202a51b16ac8d1b4dc75de89e7344ed4 last.cgi 6c0f96c1e43a23a21264f924ae732273 linsniffer 5f22ceb87631fbcbf32e59234feeaa5b logclear dfb2eeea2a5ba23eb6a2b9d0cff9d82f lsattr 18a2d7d3178f321b881e7c493af72996 mkxfs 2b07576213c1c8b942451459b3dc4903 netstat 68b329da9893e34099c7d8ad5cb9c940 pidfile 7728c15d89f27e376950f96a7510bf0f ps 06d04fa3c4941b398756d029de75770e s 464dc23cac477c43418eb8d3ef087065 sense 54e41f035e026f439d4188759b210f07 services 4cfae8c44a6d1ede669d41fc320c7325 sl2 21ed3ca31a9c9b51a757f1644e26f2f7 ssh 5fd2ce512e0eba4d090191e8a1518808 ssh_config c2c1b08498ed71a908c581d634832672 ssh_host_key e76cd5baaab7b4f28c999946a9cb4dcc ssh_host_key.pub ad265d3c07dea3151bacb6930e0b72d3 ssh_random_seed 312de877e5180678cd54606e1c25af40 sshd_config 8ff0939cd49a0b2ef3156c7876afca4b top +*+ Bonus) Was the rootkit ever actually installed on the system? How do you know? * Yes. The rootkit was installed because in the lazarus output you can see the email messages that the install script sent.(See ~lazarus_output/blocks/84383.m.txt) There is also evidence in the mactime report. This section matches up well with the sequence of events in the install script. At the end it checks the sendmail.cf and sendmail.cw files as it sends off the emails. Mar 15 01 20:36:48 520333 m.. -rw-r--r-- root root Mar 15 01 20:44:50 35300 .a. -rwxr-xr-x root root /mnt/forensics/bin/netstat 33280 .a. -rwxr-xr-x root root /mnt/forensics/bin/ps 611931 .a. -rwxr-xr-x root root 1 .a. -rw-r--r-- root root 1345 .a. -rwxr-xr-x root root 880 .a. -rw-r--r-- root root 344 .a. -rw-r--r-- root root 688 .a. -rw-r--r-- root root 520333 .a. -rw-r--r-- root root Mar 15 01 20:45:02 2048 m.c drwxr-xr-x root root /mnt/forensics/bin 11952 .a. -rwxr-xr-x root root /mnt/forensics/bin/chown 35300 ..c -rwxr-xr-x root root /mnt/forensics/bin/netstat 33280 ..c -rwxr-xr-x root root /mnt/forensics/bin/ps 34816 m.c drwxr-xr-x root root /mnt/forensics/dev 12288 m.c drwxrwxr-x root root /mnt/forensics/dev/ida 7165 mac -rwx------ root root /mnt/forensics/dev/ida/.. /linsniffer 75 mac -rwx------ root root /mnt/forensics/dev/ida/.. /logclear 632066 .a. -rwxr-xr-x root root /mnt/forensics/dev/ida/.. /mkxfs 4060 mac -rwxr-xr-x root root /mnt/forensics/dev/ida/.. /sense 8268 mac -rwx------ root root /mnt/forensics/dev/ida/.. /sl2 7165 m.c -rwx------ root root /mnt/forensics/dev/ida/.drag-on/linsniffer 75 mac -rwx------ root root /mnt/forensics/dev/ida/.drag-on/logclear 632066 m.c -rwxr-xr-x root root /mnt/forensics/dev/ida/.drag-on/mkxfs 708 m.c -rw-r--r-- root root /mnt/forensics/dev/ida/.drag-on/s 4060 mac -rwxr-xr-x root root /mnt/forensics/dev/ida/.drag-on/sense 8268 mac -rwx------ root root /mnt/forensics/dev/ida/.drag-on/sl2 540 m.c -rw------- root root /mnt/forensics/dev/ida/.drag-on/ssh_host_key 87 mac -rw-r--r-- root root /mnt/forensics/dev/last 71 mac -rw-r--r-- root root /mnt/forensics/dev/rpm 3072 m.c drwxr-xr-x root root /mnt/forensics/sbin 19840 ..c -rwxr-xr-x root root /mnt/forensics/sbin/ifconfig 4060 .a. -rwxr-xr-x root root 8268 .a. -rwx------ root root 53588 .ac -rwxr-xr-x root root 75 .a. -rwx------ root root 66736 ..c -rwxr-xr-x root root 60080 ..c -r-xr-xr-x root root 42736 ..c -rwxr-xr-x root root Mar 15 01 20:45:03 33392 .a. -rwxr-xr-x root root /mnt/forensics/bin/cp 5760 .a. -rwxr-xr-x root root /mnt/forensics/bin/sleep 1024 m.c drwxr-xr-x root root /mnt/forensics/dev/ida/.. 632066 m.c -rwxr-xr-x root root /mnt/forensics/dev/ida/.. /mkxfs 708 mac -rw-r--r-- root root /mnt/forensics/dev/ida/.. /s 540 mac -rw------- root root /mnt/forensics/dev/ida/.. /ssh_host_key 512 mac -rw------- root root /mnt/forensics/dev/ida/.. /ssh_random_seed 0 mac -rw-r--r-- root root /mnt/forensics/dev/ida/.. /tcp.log 1024 m.c drwxr-xr-x root root /mnt/forensics/dev/ida/.drag-on 7165 .a. -rwx------ root root /mnt/forensics/dev/ida/.drag-on/linsniffer 632066 .a. -rwxr-xr-x root root /mnt/forensics/dev/ida/.drag-on/mkxfs 708 .a. -rw-r--r-- root root /mnt/forensics/dev/ida/.drag-on/s 540 .a. -rw------- root root /mnt/forensics/dev/ida/.drag-on/ssh_host_key 512 .a. -rw------- root root /mnt/forensics/dev/ida/.drag-on/ssh_random_seed 138 .a. -rw-r--r-- root root /mnt/forensics/dev/ida/.drag-on/tcp.log 3072 m.c drwxr-xr-x root root /mnt/forensics/etc 3278 mac -rw-r--r-- root root /mnt/forensics/etc/inetd.conf 13708 m.c -rwxr-xr-x root root /mnt/forensics/etc/rc.d/rc.sysinit 11407 m.c -rw-r--r-- root root /mnt/forensics/etc/services 17 .a. lrwxrwxrwx root root /mnt/forensics/lib/libcom_err.so.2 -> libcom_err.so.2.0 8465 .a. -rwxr-xr-x root root /mnt/forensics/lib/libcom_err.so.2.0 13 .a. lrwxrwxrwx root root /mnt/forensics/lib/libe2p.so.2 -> libe2p.so.2.3 17713 .a. -rwxr-xr-x root root /mnt/forensics/lib/libe2p.so.2.3 16 .a. lrwxrwxrwx root root /mnt/forensics/lib/libext2fs.so.2 -> libext2fs.so.2.4 85856 .a. -rwxr-xr-x root root /mnt/forensics/lib/libext2fs.so.2.4 3278 .a. -rw-r--r-- root root 79 .a. -rwxr-xr-x root root 11407 .a. -rw-r--r-- root root 4060 ..c -rwxr-xr-x root root 540 .ac -rw------- root root 512 .ac -rw------- root root 8268 ..c -rwx------ root root 75 ..c -rwx------ root root 708 .ac -rw-r--r-- root root 632066 .ac -rwxr-xr-x root root Mar 15 01 20:45:05 24816 .a. -rwxr-xr-x root root /mnt/forensics/bin/df 62384 .a. -rwxr-xr-x root mail /mnt/forensics/bin/mail 51 .a. -rw-r--r-- root root /mnt/forensics/etc/conf.modules 112 .a. -rw-r--r-- root root /mnt/forensics/etc/mail.rc 17 .a. lrwxrwxrwx root root /mnt/forensics/lib/ld-linux.so.1 -> ld-linux.so.1.9.5 25386 .a. -rwxr-xr-x root root /mnt/forensics/lib/ld-linux.so.1.9.5 788401 .a. -rwxr-xr-x root root /mnt/forensics/lib/libdb-2.1.3.so 14 .a. lrwxrwxrwx root root /mnt/forensics/lib/libdb.so.3 -> libdb-2.1.3.so 44108 .a. -rwxr-xr-x root root /mnt/forensics/lib/libproc.so.2.0.6 28633 .a. -rw-r--r-- root root /mnt/forensics/lib/modules/2.2.14-5.0/modules.dep 19840 .a. -rwxr-xr-x root root /mnt/forensics/sbin/ifconfig 6 .a. lrwxrwxrwx root root /mnt/forensics/sbin/modprobe -> insmod 0 mac drwxr-xr-x 1031 users 611931 ..c -rwxr-xr-x root root 1 ..c -rw-r--r-- root root 3713 .ac -rwx------ root root 796 mac -rw-r--r-- root root 1345 ..c -rwxr-xr-x root root 3278 ..c -rw-r--r-- root root 79 ..c -rwxr-xr-x root root 11407 ..c -rw-r--r-- root root 880 ..c -rw-r--r-- root root 344 ..c -rw-r--r-- root root 688 ..c -rw-r--r-- root root 4620 .ac -rwxr-xr-x root root 520333 ..c -rw-r--r-- root root Mar 15 01 20:46:09 34181 .a. -rw-r--r-- root root /mnt/forensics/etc/sendmail.cf 59 .a. -rw-r--r-- root root /mnt/forensics/etc/sendmail.cw