Scan of the Month May 2001 -------------------------- All time stamps are in GMT+0200. The analysis was performed on a Linux system; tools used include The Coroner's Toolkit and standard system utilities. Shell commands are interpreted by bash. Thomas Roessler ### Show step by step how you identify and recover the deleted rootkit from ### the / partition. 1. Confirm the MD5 Checksum This validates the integrity of the disk image. # md5 honeynet.tar.gz 0dff8fb9fe022ea80d8f1a4e4ae33e21 honeynet.tar.gz # tar -zxvf honeynet.tar.gz # md5 honeynet/honeypot.hda8.dd 5a8ebf5725b15e563c825be85f2f852e honeynet/honeypot.hda8.dd 2. Mounting the evidence We loopback-mount the dd'ed disk image on the analysis system: # losetup /dev/loop0 honeypot.hda8.dd # mount -o ro,nodev,nosuid,noexec /dev/loop0 /mnt This way, we can access the honeypot's root file system as /mnt on the analysis box. 3. Getting familiar with the system In order to get a very rough first impression, we run find on the honeypot's file system, looking for inode changes which happened after the system's installation was complete: # find . -cnewer tmp/install.log -print0 | xargs -0 ls -lncad \ > files In the output, we note several interesting details: - /bin/netstat, /bin/ps, /etc/ftpaccess, /etc/inetd.conf, /etc/services, /etc/rc.d/rc.sysinit, /root/.bash_history, and /sbin/ifconfig were changed on Mar 16 02:45 +0200. (Note: The attacker did not adjust ctimes!) - Directories '/dev/ida/.. ' and '/dev/ida/.drag-on' were created at 02:45 +0200. Both directories' content looks like an attacker's tool chest. - /dev/ida/.drag-on/tcp.log isn't empty, and was last changed late on Mar 16. Most likely, linsniffer (which usually stores output in tcp.log) was run on the system after the attack. Investigating the ifconfig binary with strings, we see that it can't report promiscuous interfaces - a typical root kit signature. Thus, we can assume that the attacker installed a root kit and packet sniffer on the system, and we have an idea of the point of time of his activities. Of course, for a complete analysis of the system, we'd have to dive into these directories, into root's bash_history, etc. into more detail. 4. Looking for the deleted root kit In order to find the deleted root kit, we generate a time line of the deleted, but intact, inodes using tools from The Coroner's Toolkit by Wietse Venema and Dan Farmer: $ ~/tct-1.06/bin/ils -r honeypot.hda8.dd | ~/tct-1.06/extras/ils2mac > body $ ~/tct-1.06/bin/mactime -b body -g /mnt/etc/group -p /mnt/etc/passwd 'Mar 16' > timeline $ cat timeline Feb 08 02 14:08:13 611931 m.. -rwxr-xr-x root root Mar 16 01 02:36:48 520333 m.. -rw-r--r-- root root Mar 16 01 02:44:50 611931 .a. -rwxr-xr-x root root 1 .a. -rw-r--r-- root root 1345 .a. -rwxr-xr-x root root 880 .a. -rw-r--r-- root root 344 .a. -rw-r--r-- root root [...] In the output generated, we first look for large, non-executable files: We hope that we can restore the root kit's tar ball which was downloaded at some point. Investigating the output, we note, in particular, : The file's content was last modified at Mar 16 2001 02:36:48 (the end of a download?), the file was last read at 02:44:50 (that is, shortly before the files and directories we found above were installed), and the file was removed at 02:45:05. The size - about 500 kB - also seems reasonable for a root kit tar ball. Thus, inode 23 is our prime suspect for the root kit tar ball. We extract it from the dd image using icat from The Coroner's Toolkit: $ ~/tct-1.06/bin/icat honeypot.hda8.dd 23 > 23 $ file 23 23: gzip compressed data, deflated, last modified: Sat Mar 3 04:09:06 2001, os: Unix $ tar tvzf 23 drwxr-xr-x 1031/users 0 2001-02-26 21:40:30 last/ tar: Archive contains future timestamp 2002-02-08 14:08:13 -rwxr-xr-x 1031/users 611931 2002-02-08 14:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 16:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-03 04:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 16:22:50 last/linsniffer [...] ### What files make up the deleted rootkit? The file's content heavily looks like a root kit. We unpack it and investigate the install script. Going through this script step by step, we find that it explains the timestamps we found in section 3. Thus, we conclude that the root kit was indeed installed. Finally, we batch-reconstruct the deleted but intact inodes from the disk image: $ ~/tct-1.06/bin/ils -r honeypot.hda8.dd | \ awk -F '|' '($2 == "f") { print $1 }' | \ while read i ; do ~/tct-1.06/bin/icat \ honeypot.hda8.dd $i > reconstr/$i ; done We compare the files in the unpacked root kit directory with the deleted inodes we restored: $ for f in last/* ; do for g in reconstr/* ; do if cmp $f $g \ 2> /dev/null > /dev/null ; then echo -e "$f\t$g" ; fi; \ done ; done last/cleaner reconstr/2043 last/inetd.conf reconstr/2044 last/install reconstr/2041 last/last.cgi reconstr/2054 last/logclear reconstr/2059 last/lsattr reconstr/2045 last/mkxfs reconstr/2061 last/pidfile reconstr/2040 last/s reconstr/2060 last/sense reconstr/2047 last/services reconstr/2046 last/sl2 reconstr/2053 last/ssh reconstr/2039 last/ssh_config reconstr/2048 last/ssh_host_key reconstr/2049 last/ssh_host_key.pub reconstr/2050 last/ssh_random_seed reconstr/2051 last/sshd_config reconstr/2052 last/top reconstr/2058 Note that this list does not contain all the files from the root kit tar ball. This may be because the install script uses mv to install various binaries. Finally, we look for files on the system which are identical to root kit files. This could be done by creating MD5 checksums of all files in the root kit and all files on the system. We go a simpler, but less scalable way, and just compare files: # cat chkrk.sh #!/bin/sh rkdir=last for f ; do for g in $rkdir/* ; do if cmp "$f" "$g" 2> /dev/null > /dev/null ; then echo -e "'$f'\t'$g'" fi done done # find /mnt -type f -print0 | xargs -0 ./chkrk.sh '/mnt/dev/ida/.drag-on/linsniffer' 'last/linsniffer' '/mnt/dev/ida/.drag-on/logclear' 'last/logclear' '/mnt/dev/ida/.drag-on/sense' 'last/sense' '/mnt/dev/ida/.drag-on/sl2' 'last/sl2' '/mnt/dev/ida/.drag-on/mkxfs' 'last/mkxfs' '/mnt/dev/ida/.drag-on/s' 'last/s' '/mnt/dev/ida/.drag-on/ssh_host_key' 'last/ssh_host_key' '/mnt/dev/ida/.. /linsniffer' 'last/linsniffer' '/mnt/dev/ida/.. /logclear' 'last/logclear' '/mnt/dev/ida/.. /sense' 'last/sense' '/mnt/dev/ida/.. /sl2' 'last/sl2' '/mnt/dev/ida/.. /mkxfs' 'last/mkxfs' '/mnt/dev/ida/.. /s' 'last/s' '/mnt/dev/ida/.. /ssh_host_key' 'last/ssh_host_key' '/mnt/dev/ida/.. /ssh_random_seed' 'last/ssh_random_seed' '/mnt/etc/services' 'last/services' '/mnt/etc/at.deny' 'last/pidfile' '/mnt/etc/inetd.conf' 'last/inetd.conf' '/mnt/bin/netstat' 'last/netstat' '/mnt/bin/ps' 'last/ps' '/mnt/sbin/ifconfig' 'last/ifconfig' ### Bonus Question: Was the rootkit actually installed? Our earlier suspicion concerning the netstat, ps, and ifconfig binaries on the system is thus confirmed.