################################################################ # Getting Started # ################################################################ For the lack of a better place to start, I used the standard strings and grep commands on the image as follows (the -t x option returns the offset of the string): # strings -t x honeypot.hda8.dd | grep "kit" Which returned a couple of lines, however the first was very convincing: 8df41f echo "********* Instalarea Rootkitului A Pornit La Drum *********" This is visibly a line from a shell script or similar script. I don't know a lick of Romanian, but I recognized that it's the installation script for a rootkit. I figured there must be more text around it. I had two options at this point: use dd or a hex editor to recover more lines. I had Adam Rogoyski's Hexedit 0.9.3 (libcurses) available to get to that offset and poke around. However, dd is widely available and will do the job just nicely as well. Making the assumption that the filesystem was using 1024-char blocks, I did a hex integer (truncated) division by 0x400 and converted to decimal (9085d). I made the assumption that the script would be less than 1k to start with: # dd if=honeypot.hda8.dd bs=1024 skip=9085 count=1 of=first.out It proved to require 4 1024-char blocks to recover the whole file, but it was complete. This gave me some ideas where to root (no pun intended) around on the filesystem for later. Next, I mounted the image read-only with the loopback mechanism and had a look around given my discoveries in the installation script. I poked around for a while and found some rootkit-installed files...more on this later. # mount /tmp/honeynet/honeypot.hda8.dd -t ext2 -o loop -o ro /mnt/honeynet ################################################################ # Question 1: Show step by step how you identify and recover # # the deleted rootkit from the / partition. # ################################################################ This step turned out to be a good deal more straightforward than I expected. I referred to the Linux Ext2fs Undeletion mini-HOWTO by Aaron Crane and, following his suggestion, used the debugfs tool to list the deleted inodes on the filesystem. # debugfs honeypot.hda8.dd debugfs 1.17, 26-Oct-1999 for EXT2 FS 0.5b, 95/08/09 debugfs: lsdel 29 deleted inodes found. Inode Owner Mode Size Blocks Time deleted 56231 0 100644 33135 13/ 13 Thu Mar 15 06:17:36 2001 16110 0 100644 239 1/ 1 Thu Mar 15 06:20:25 2001 2058 0 100755 53588 54/ 54 Thu Mar 15 20:45:02 2001 30188 0 100755 66736 67/ 67 Thu Mar 15 20:45:02 2001 30191 0 100555 60080 60/ 60 Thu Mar 15 20:45:02 2001 48284 0 100755 42736 43/ 43 Thu Mar 15 20:45:02 2001 2047 0 100755 4060 4/ 4 Thu Mar 15 20:45:03 2001 2049 0 100600 540 1/ 1 Thu Mar 15 20:45:03 2001 2051 0 100600 512 1/ 1 Thu Mar 15 20:45:03 2001 2053 0 100700 8268 9/ 9 Thu Mar 15 20:45:03 2001 2059 0 100700 75 1/ 1 Thu Mar 15 20:45:03 2001 2060 0 100644 708 1/ 1 Thu Mar 15 20:45:03 2001 2061 0 100755 632066 622/ 622 Thu Mar 15 20:45:03 2001 23 0 100644 520333 512/ 512 Thu Mar 15 20:45:05 2001 2039 0 100755 611931 602/ 602 Thu Mar 15 20:45:05 2001 2040 0 100644 1 1/ 1 Thu Mar 15 20:45:05 2001 2041 0 100700 3713 4/ 4 Thu Mar 15 20:45:05 2001 2042 0 100644 796 1/ 1 Thu Mar 15 20:45:05 2001 2043 0 100755 1345 2/ 2 Thu Mar 15 20:45:05 2001 2044 0 100644 3278 4/ 4 Thu Mar 15 20:45:05 2001 2045 0 100755 79 1/ 1 Thu Mar 15 20:45:05 2001 2046 0 100644 11407 12/ 12 Thu Mar 15 20:45:05 2001 2048 0 100644 880 1/ 1 Thu Mar 15 20:45:05 2001 2050 0 100644 344 1/ 1 Thu Mar 15 20:45:05 2001 2052 0 100644 688 1/ 1 Thu Mar 15 20:45:05 2001 2054 0 100755 4620 5/ 5 Thu Mar 15 20:45:05 2001 2038 1031 40755 0 1/ 1 Thu Mar 15 20:46:09 2001 8097 0 40700 0 1/ 1 Fri Mar 16 05:03:12 2001 8100 0 100644 16329 177/ 177 Fri Mar 16 05:03:12 2001 Since there were only 29 deleted inodes, I had the luxury of being able to scrutinize each individually. I ended up recovering three files: first a Perl script for parsing output from linsniffer (part of the rootkit), an ELF binary that I believe was the pre-rootkit ps command on the system, and finally the GZipped tarball containing the rootkit itself. Using debugfs' stat command I was able to find the starting inode of each chain and proceeded to use the dump command to write them out to files. I ran the file command on each file to determine it's contents. Here's a little snippet of the action from the debugfs prompt: debugfs: stat <23> Inode: 23 Type: regular Mode: 0644 Flags: 0x0 Version/Generation: -130 User: 0 Group: 0 Size: 520333 File ACL: 0 Directory ACL: 0 Links: 0 Blockcount: 1024 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x3ab17021 -- Thu Mar 15 20:45:05 2001 atime: 0x3ab17012 -- Thu Mar 15 20:44:50 2001 mtime: 0x3ab16e30 -- Thu Mar 15 20:36:48 2001 dtime: 0x3ab17021 -- Thu Mar 15 20:45:05 2001 BLOCKS: 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 TOTAL: 512 debugfs: dump <23> /tmp/dump.3.out Now, at the console: # file /tmp/dump.3.out /tmp/dump.3.out: gzip compressed data, deflated, last modified: Fri Mar 2 22:09:06 2001, os: Unix # mv dump.3.out lk.tgz The MD5SUM for lk.tgz (the rootkit archive) is: 115f438631de8d0a7c03c9d458eb7257 I noted from the final line of the recovered installation script that it was indeed a GZipped tarball and renamed the file as such before moving it to a subdirectory for unpacking. We can tell that it was named lk.tgz as opposed to lk.tar.gz by observing the directory table in the image. At this point, our discussion leads nicely into the second question... ################################################################ # Question 2: What files make up the deleted rootkit? # ################################################################ After recovering the tarball, I unpacked it and found a subdirectory 'list' containing the following files: total 1472 -rwxr-xr-x 1 xxxxx users 611931 Feb 8 2002 ssh -rwx------ 1 xxxxx users 3713 Mar 2 22:08 install -rw-r--r-- 1 xxxxx users 708 Mar 2 22:05 s -rw-r--r-- 1 xxxxx users 1 Feb 26 10:29 pidfile -rw-r--r-- 1 xxxxx users 688 Feb 26 10:29 sshd_config -rwxr-xr-x 1 xxxxx users 79 Feb 26 10:28 lsattr -rwx------ 1 xxxxx users 75 Feb 26 10:24 logclear -rwxr-xr-x 1 xxxxx users 53588 Feb 26 10:23 top -rwxr-xr-x 1 xxxxx users 19840 Feb 26 10:23 ifconfig -rwxr-xr-x 1 xxxxx users 35300 Feb 26 10:23 netstat -rwxr-xr-x 1 xxxxx users 33280 Feb 26 10:23 ps -rwxr-xr-x 1 xxxxx users 4620 Feb 26 10:23 last.cgi -rwx------ 1 xxxxx users 8268 Feb 26 10:22 sl2 -rwxr-xr-x 1 xxxxx users 4060 Feb 26 10:22 sense -rwx------ 1 xxxxx users 7165 Feb 26 10:22 linsniffer -rwxr-xr-x 1 xxxxx users 632066 Feb 26 09:46 mkxfs -rw-r--r-- 1 xxxxx users 11407 Jan 27 10:11 services -rw-r--r-- 1 xxxxx users 3278 Jan 27 10:11 inetd.conf -rw-r--r-- 1 xxxxx users 880 Oct 22 2000 ssh_config -rw------- 1 xxxxx users 540 Oct 22 2000 ssh_host_key -rw-r--r-- 1 xxxxx users 344 Oct 22 2000 ssh_host_key.pub -rw------- 1 xxxxx users 512 Oct 22 2000 ssh_random_seed -rwxr-xr-x 1 xxxxx users 1345 Sep 9 1999 cleaner This combination of files matched no rootkits that I was able to find on the Internet. It appears to be leaner and cruder than those in the Linux Rootkit series among others. It looks to me that it's pieces of other kits and tools patched together...a real homebrew. ################################################################ # Bonus: Was the rootkit ever actually installed on the # # system? How do you know? # ################################################################ It was at least partially installed, as I found the SSH files in the /dev/ida/.drag-on and /dev/ida/".. " subdirectories. The md5sums of the binaries matched those in the rootkit package. Five files in /etc were modified around the time of the rootkit installation. One of which was inetd.conf, which matches the one found in the rootkit. The others were sendmail.cf, sendmail.cw, conf.modules, and mail.rc. I'm sure that other partitions would contain the CGI backdoor and other files copied to /usr and/or /var. ################################################################# # Final Thoughts/Observations/Statistics # ################################################################# In retrospect it would have been better to start this project with debugfs rather than the string search. However, finding the configuration script right off the bat was a big help. I also found in root's .bash_history that somebody had installed the mech IRC bot and cleaned out /etc/ftpaccess. The incidents are probably related. The telnet and attempted ftp connections in the sniffer log are also of some interest in the general incident analysis, however not so important to the specific task of recovering the rootkit. Time spent: 6 hours (2 hours recovering files, 2 hours searching in vain for rootkits online, 2 hours writing it up) Experience Level: EE/CS Undergrad Student and Jr Admin