Scan 15 1.Show step by step how you identify and recovered the deleted rootkit from the / partition. trinity% md5sum honeypot.hda8.dd 5a8ebf5725b15e563c825be85f2f852e honeypot.hda8.dd The checksum matches the one given on the honeynet web page. Let's start out by looking for deleted inodes using debugfs. trinity% echo lsdel | debugfs honeypot.hda8.dd > lsdel.out trinity% cat lsdel.out debugfs: 29 deleted inodes found. Inode Owner Mode Size Blocks Time deleted 56231 0 100644 33135 13/ 13 Thu Mar 15 05:17:36 2001 16110 0 100644 239 1/ 1 Thu Mar 15 05:20:25 2001 2058 0 100755 53588 54/ 54 Thu Mar 15 19:45:02 2001 30188 0 100755 66736 67/ 67 Thu Mar 15 19:45:02 2001 30191 0 100555 60080 60/ 60 Thu Mar 15 19:45:02 2001 48284 0 100755 42736 43/ 43 Thu Mar 15 19:45:02 2001 2047 0 100755 4060 4/ 4 Thu Mar 15 19:45:03 2001 2049 0 100600 540 1/ 1 Thu Mar 15 19:45:03 2001 2051 0 100600 512 1/ 1 Thu Mar 15 19:45:03 2001 2053 0 100700 8268 9/ 9 Thu Mar 15 19:45:03 2001 2059 0 100700 75 1/ 1 Thu Mar 15 19:45:03 2001 2060 0 100644 708 1/ 1 Thu Mar 15 19:45:03 2001 2061 0 100755 632066 622/ 622 Thu Mar 15 19:45:03 2001 23 0 100644 520333 512/ 512 Thu Mar 15 19:45:05 2001 2039 0 100755 611931 602/ 602 Thu Mar 15 19:45:05 2001 2040 0 100644 1 1/ 1 Thu Mar 15 19:45:05 2001 2041 0 100700 3713 4/ 4 Thu Mar 15 19:45:05 2001 2042 0 100644 796 1/ 1 Thu Mar 15 19:45:05 2001 2043 0 100755 1345 2/ 2 Thu Mar 15 19:45:05 2001 2044 0 100644 3278 4/ 4 Thu Mar 15 19:45:05 2001 2045 0 100755 79 1/ 1 Thu Mar 15 19:45:05 2001 2046 0 100644 11407 12/ 12 Thu Mar 15 19:45:05 2001 2048 0 100644 880 1/ 1 Thu Mar 15 19:45:05 2001 2050 0 100644 344 1/ 1 Thu Mar 15 19:45:05 2001 2052 0 100644 688 1/ 1 Thu Mar 15 19:45:05 2001 2054 0 100755 4620 5/ 5 Thu Mar 15 19:45:05 2001 2038 1031 40755 0 1/ 1 Thu Mar 15 19:46:09 2001 8097 0 40700 0 1/ 1 Fri Mar 16 04:03:12 2001 8100 0 100644 16329 177/ 177 Fri Mar 16 04:03:12 2001 It looks like out of 29 deleted inodes, 27 of them were deleted on March 15th which happens to be the same day of the compromise. 24 of those 27 files were all presumably owned by root and all deleted at roughly the same time. Next, let's dump the contents of each inode we found was deleted. trinity% cat lsdel.out| awk '/^ *[0-9]/ { print "dump <"$1"> \ recovered."$1 }' | debugfs honeypot.hda8.dd debugfs 1.18, 11-Nov-1999 for EXT2 FS 0.5b, 95/08/09 debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: debugfs: trinity% ls README recovered.2043 recovered.2052 recovered.30191 honeypot.hda8.dd recovered.2044 recovered.2053 recovered.48284 lsdel.out recovered.2045 recovered.2054 recovered.56231 recovered.16110 recovered.2046 recovered.2058 recovered.8097 recovered.2038 recovered.2047 recovered.2059 recovered.8100 recovered.2039 recovered.2048 recovered.2060 recovered.2040 recovered.2049 recovered.2061 recovered.2041 recovered.2050 recovered.23 recovered.2042 recovered.2051 recovered.30188 After running the file command on the data from the recovered inodes, it becomes evident the 24 files mentioned above are part of a rootkit. trinity% file recovered* recovered.16110: ASCII text recovered.2038: empty recovered.2039: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped recovered.2040: PCX image data, version 2.5 recovered.2041: Bourne shell script text recovered.2042: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped recovered.2043: Bourne-Again shell script text recovered.2044: English text recovered.2045: Bourne shell script text recovered.2046: English text recovered.2047: perl script text recovered.2048: English text recovered.2049: data recovered.2050: ASCII text recovered.2051: data recovered.2052: ASCII text recovered.2053: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped recovered.2054: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped recovered.2058: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped recovered.2059: ASCII text recovered.2060: ASCII text recovered.2061: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped recovered.23: gzip compressed data, deflated, last modified: Fri Mar 2 21:09 :06 2001, os: Unix recovered.30188: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped recovered.30191: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped recovered.48284: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped recovered.56231: ASCII text recovered.8097: empty recovered.8100: English text Let's take a closer look at recovered.23, what the file command says is compressed data. trinity% mv recovered.23 foo.tgz trinity% gunzip foo.tgz trinity% tar -tvf foo.tar drwxr-xr-x 1031/users 0 2001-02-26 14:40:30 last/ tar: Archive contains future timestamp 2002-02-08 07:08:13 -rwxr-xr-x 1031/users 611931 2002-02-08 07:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 09:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-02 21:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 09:22:50 last/linsniffer -rwxr-xr-x 1031/users 1345 1999-09-09 10:57:11 last/cleaner -rw-r--r-- 1031/users 3278 2001-01-27 09:11:32 last/inetd.conf -rwxr-xr-x 1031/users 79 2001-02-26 09:28:40 last/lsattr -rw-r--r-- 1031/users 11407 2001-01-27 09:11:44 last/services -rwxr-xr-x 1031/users 4060 2001-02-26 09:22:55 last/sense -rw-r--r-- 1031/users 880 2000-10-22 14:29:44 last/ssh_config -rw------- 1031/users 540 2000-10-22 14:29:44 last/ssh_host_key -rw-r--r-- 1031/users 344 2000-10-22 14:29:44 last/ssh_host_key.pub -rw------- 1031/users 512 2000-10-22 14:29:44 last/ssh_random_seed -rw-r--r-- 1031/users 688 2001-02-26 09:29:51 last/sshd_config -rwx------ 1031/users 8268 2001-02-26 09:22:59 last/sl2 -rwxr-xr-x 1031/users 4620 2001-02-26 09:23:10 last/last.cgi -rwxr-xr-x 1031/users 33280 2001-02-26 09:23:33 last/ps -rwxr-xr-x 1031/users 35300 2001-02-26 09:23:42 last/netstat -rwxr-xr-x 1031/users 19840 2001-02-26 09:23:47 last/ifconfig -rwxr-xr-x 1031/users 53588 2001-02-26 09:23:55 last/top -rwx------ 1031/users 75 2001-02-26 09:24:03 last/logclear -rw-r--r-- root/root 708 2001-03-02 21:05:12 last/s -rwxr-xr-x 1031/users 632066 2001-02-26 08:46:04 last/mkxfs These files all have matching checksums when compared to those recovered from the deleted inodes. We have recovered the deleted rootkit from the / partition. 2. What files make up the deleted rootkit? ssh: an ssh client, maybe backdoored? pidfile: pidfile for sshd install: shell script to install the rootkit linsniffer: sniffer used to log passwords cleaner: shell script used to erase log file entries inetd.conf: copy of inetd.conf with only telnet and pop3 turned on lsattr: shell scripted used to start the sniffer and mkxfs services: copy of the services file sense: perl script used to sort the output from the sniffer ssh_config: ssh config file ssh_host_key: ssh config file ssh_host_key.pub: ssh config file ssh_random_seed: ssh config file sshd_config: config file for sshd that reads data from /dev/ida/.drag-on/ sl2: used for generating syn packets from a forged address last.cgi: cgi script used to run programs on the local system ps: backdoored version of ps that tries to read /dev/dsx netstat: backdoored version of netstat that tries to read /dev/caca ifconfig: backdoored version of ifconfig (might try to hide promiscuous mode?) top: backdoored version of top that tries to read /dev/dsx logclear: kills the sniffer, removes the log files, and starts the sniffer again s: a config file for sshd mkxfs: a copy of sshd Bonus Question: Yes, it is likely the rootkit was installed on this system. The files that would have been installed by the rootkit and the files located on the root partition have matching MD5 checksums.