1. Show step by step how you identify and recover the deleted rootkit from the
   / partition. 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here we go.

Phase #0:

$ wget http://project.honeynet.org/scans/scan15/honeynet.tar.gz
$ md5sum honeynet.tar.gz
0dff8fb9fe022ea80d8f1a4e4ae33e21  honeynet.tar.gz 	// OK
$ tar -zxf honeynet.tar.gz
$ cd honeynet; md5sum honeypot.hda8.dd
5a8ebf5725b15e563c825be85f2f852e  honeypot.hda8.dd	// OK
$ mkdir root
$ su
# mount -o loop,noexec,nodev,rw honeypot.hda8.dd root/
/home/miv/honeynet/honeypot.hda8.dd on /home/miv/honeynet/root type ext2
(ro,noexec,nodev,loop=/dev/loop0)
# cd root


Phase #1:

My first thought: hmm, who does compromise honeynet hosts most often ?? Answer:
Romanian script-kiddies ;) Let's go along this path. We'll try to find some
hidden directories:

# find . -name .\* -exec echo \"{}\" \;
"."
"./tmp/.font-unix"
"./dev/ida/.drag-on"			// looks interesting ...
"./dev/ida/.. "				// ... this one even more
"./etc/.pwd.lock"
"./etc/skel/.emacs"
"./etc/skel/.bash_logout"
"./etc/skel/.bash_profile"
"./etc/skel/.bashrc"
"./etc/skel/.screenrc"
"./lib/modules/2.2.14-5.0/.rhkmvtag"
"./root/.Xdefaults"
"./root/.bash_logout"
"./root/.bash_profile"
"./root/.bashrc"
"./root/.cshrc"
"./root/.tcshrc"
"./root/.bash_history"			// will be good to take a look into

# ls -la dev/ida/.drag-on
total 660
drwxr-xr-x    2 root     root         1024 May 16 18:51 .
drwxrwxr-x    4 root     root        12288 Mar 16 02:45 ..
-rwx------    1 root     root         7165 Mar 16 02:45 linsniffer
-rwx------    1 root     root           75 Mar 16 02:45 logclear
-rwxr-xr-x    1 root     root       632066 Mar 16 02:45 mkxfs
-rw-r--r--    1 root     root          708 Mar 16 02:45 s
-rwxr-xr-x    1 root     root         4060 Mar 16 02:45 sense
-rwx------    1 root     root         8268 Mar 16 02:45 sl2
-rw-------    1 root     root          540 Mar 16 02:45 ssh_host_key
-rw-------    1 root     root          512 Mar 16 15:45 ssh_random_seed
-rw-r--r--    1 root     root          138 Mar 16 17:28 tcp.log

# ls -la dev/ida/".. "
total 659
drwxr-xr-x    2 root     root         1024 Mar 16 02:45 .
drwxrwxr-x    4 root     root        12288 Mar 16 02:45 ..
-rwx------    1 root     root         7165 Mar 16 02:45 linsniffer
-rwx------    1 root     root           75 Mar 16 02:45 logclear
-rwxr-xr-x    1 root     root       632066 Mar 16 02:45 mkxfs
-rw-r--r--    1 root     root          708 Mar 16 02:45 s
-rwxr-xr-x    1 root     root         4060 Mar 16 02:45 sense
-rwx------    1 root     root         8268 Mar 16 02:45 sl2
-rw-------    1 root     root          540 Mar 16 02:45 ssh_host_key
-rw-------    1 root     root          512 Mar 16 02:45 ssh_random_seed
-rw-r--r--    1 root     root            0 Mar 16 02:45 tcp.log

Wow! Good shot. I bet this stuff isn't packaged with standard RH6.2 distro ...
Files names speak for themselves. We won't dig more because we have other task:
find _deleted_ rootkit (these toys obviously aren't deleted ;).

Let's take a look into root's history file:

# cat root/.bash_history
exec tcsh	// probably to avoid leaving tracks in bash's history file
ls
mkdir /var/...
ls
cd /var/...
ftp ftp.home.ro
tar -zxvf emech-2.8.tar.gz
cd emech-2.8
./configure
y
make
make
make install
mv sample.set mech.set
pico mech.set
./mech
cd /etc
pico ftpaccess	// I'm not sure what for. etc/ftpaccess is 0 in size ...
ls
exit

Our friend had made something nasty using tcsh shell and then installed emech
(IRC bot) from ftp.home.ro. I've even visited www.home.ro in hope that I can
learn something about attacker. Unfortunately it is in Romanian :( The only
word I could understand was: "dupa", which is widely used in some Eastern
European countries including Poland ;))

OK, we have some rootkit's files in dev/ida/.drag-on and dev/ida/".. " so far.
Still no trace of deleted rootkit. We have to go low-level way.


Phase #2:

How is a file deleted in ext2 filesystem ? Simply. File's entry is erased from
directory and file's inode is set to unused (links count == 0, and dtime !=0).
File's blocks stay untouched until there's some other data to write on disk.

Our task is to find inodes of deleted files. Do we need to write new tool ??
Someone has already written one. It's called debugfs. There's copy of it on
hda8 image (sbin/debugfs) in case you don't have it on your system..

# debugfs /dev/loop0
debugfs 1.18, 11-Nov-1999 for EXT2 FS 0.5b, 95/08/09
debugfs: help
...
list_deleted_inodes, lsdel
                         List deleted inodes
...
debugfs: lsdel
29 deleted inodes found.
 Inode  Owner  Mode    Size    Blocks    Time deleted
 56231      0 100644  33135   13/  13 Thu Mar 15 12:17:36 2001
 16110      0 100644    239    1/   1 Thu Mar 15 12:20:25 2001
  2058      0 100755  53588   54/  54 Fri Mar 16 02:45:02 2001
 30188      0 100755  66736   67/  67 Fri Mar 16 02:45:02 2001
 30191      0 100555  60080   60/  60 Fri Mar 16 02:45:02 2001
 48284      0 100755  42736   43/  43 Fri Mar 16 02:45:02 2001
  2047      0 100755   4060    4/   4 Fri Mar 16 02:45:03 2001
  2049      0 100600    540    1/   1 Fri Mar 16 02:45:03 2001
  2051      0 100600    512    1/   1 Fri Mar 16 02:45:03 2001
  2053      0 100700   8268    9/   9 Fri Mar 16 02:45:03 2001
  2059      0 100700     75    1/   1 Fri Mar 16 02:45:03 2001
  2060      0 100644    708    1/   1 Fri Mar 16 02:45:03 2001
  2061      0 100755 632066  622/ 622 Fri Mar 16 02:45:03 2001
    23      0 100644 520333  512/ 512 Fri Mar 16 02:45:05 2001
  2039      0 100755 611931  602/ 602 Fri Mar 16 02:45:05 2001
  2040      0 100644      1    1/   1 Fri Mar 16 02:45:05 2001
  2041      0 100700   3713    4/   4 Fri Mar 16 02:45:05 2001
  2042      0 100644    796    1/   1 Fri Mar 16 02:45:05 2001
  2043      0 100755   1345    2/   2 Fri Mar 16 02:45:05 2001
  2044      0 100644   3278    4/   4 Fri Mar 16 02:45:05 2001
  2045      0 100755     79    1/   1 Fri Mar 16 02:45:05 2001
  2046      0 100644  11407   12/  12 Fri Mar 16 02:45:05 2001
  2048      0 100644    880    1/   1 Fri Mar 16 02:45:05 2001
  2050      0 100644    344    1/   1 Fri Mar 16 02:45:05 2001
  2052      0 100644    688    1/   1 Fri Mar 16 02:45:05 2001
  2054      0 100755   4620    5/   5 Fri Mar 16 02:45:05 2001
  2038   1031  40755      0    1/   1 Fri Mar 16 02:46:09 2001
  8097      0  40700      0    1/   1 Fri Mar 16 11:03:12 2001
  8100      0 100644  16329  177/ 177 Fri Mar 16 11:03:12 2001

Woah ! We have all we need. How to recover all these files ? Debugfs has 'dump'
command which saves whole file pointed by given inode.

debugfs: dump <8100> 8100.rec   # 

... and so on with all 29 files. We can do it with this ugly oneliner:

#  echo lsdel|debugfs /dev/loop0 2>/dev/null|awk '$1 ~ /[0-9]+/          \
>  {c=sprintf("echo dump \\<%s\\> %s.rec|debugfs /dev/loop0",$1,$1);  \
>  print c;system(c)}'

It dumps all deleted files to current directory with names of the form: 
inode-nr.rec

Note: we can recover all files using manual 'mi' in debugfs on all 29 inodes,
resetting dtime to 0 and refcount to !=0 and then running fsck on /dev/loop0.
We should get all files linked to /lost+found. Don't ask me why I chose the
other way.

Mission #1 accomplished, sir! Proceeding to Mission #2.


2. What files make up the deleted rootkit?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We are free to take a look at all these treasures. After some research we
come to conclusion that 2038 and 8097 are inodes for directories (stat
<inode-nr> in debugfs) and the other are regular files. Now we'll try to
recover deleted directories contents.

# debugfs -w /dev/loop2  // -w enables read-write mode
debugfs: stat <2038>
Inode: 2038   Type: directory    Mode:  0755   Flags: 0x0   Version/Generation: -130400047
User:  1031   Group:   100   Size: 0
File ACL: 0    Directory ACL: 0
Links: 0   Blockcount: 2
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x3ab17021 -- Fri Mar 16 02:45:05 2001
atime: 0x3ab17021 -- Fri Mar 16 02:45:05 2001
mtime: 0x3ab17021 -- Fri Mar 16 02:45:05 2001
dtime: 0x3ab17061 -- Fri Mar 16 02:46:09 2001
BLOCKS:
8481
TOTAL: 1

debugfs: mi <2038>
                          Mode    [040755]
                       User ID    [1031]
                      Group ID    [100]
                          Size    [0] 1024
                 Creation time    [984707105]
             Modification time    [984707105]
                   Access time    [984707105]
                 Deletion time    [984707169]
                    Link count    [0]
                   Block count    [2]
                    File flags    [0x0]
                      File acl    [0]
                 Directory acl    [0]
              Fragment address    [0]
               Fragment number    [0]
                 Fragment size    [0]
               Direct Block #0    [8481]
               Direct Block #2    [0]
               Direct Block #1    [0]
               Direct Block #3    [0]
               Direct Block #4    [0]
               Direct Block #5    [0]
               Direct Block #6    [0]
               Direct Block #7    [0]
               Direct Block #8    [0]
               Direct Block #9    [0]
              Direct Block #10    [0]
              Direct Block #11    [0]
                Indirect Block    [0]
         Double Indirect Block    [0]
         Triple Indirect Block    [0]

We changed only size to 1024 (according to lsdel output this directory file
used 1 block = 1204 bytes). Now we can dump directory contents:

dumpfs: dump <2038> 2038.rec

# strings 2038.rec
pidfile
install
computerer
cleaner
inetd.conf
lsattr
services
sense
ssh_config
ssh_host_key
ssh_host_key.pub
ssh_random_seed
sshd_config
last.cgi
netstat
ifconfig
logclear
mkxfs

We do the same with second directory (inode 8097). It seems that there was
only one file named 'w'. 

OK, there are inode->name mappings inside directory files. We can use 'em
to rename our *.rec files to something more meaningful.

It's a bit difficult to extract inode numbers from directories. We can
accomplish the same if we notice (using 'file' utility) that 23.rec is .tgz
file. After uncompressing it we get something like this:

# mv 23.rec 23.tgz
# tar -zxf 23.tgz
# ls -l last
total 1425
-rwxr-xr-x    1 1031     users        1345 Sep  9  1999 cleaner
-rwxr-xr-x    1 1031     users       19840 Feb 26 16:23 ifconfig
-rw-r--r--    1 1031     users        3278 Jan 27 16:11 inetd.conf
-rwx------    1 1031     users        3713 Mar  3 04:08 install
-rwxr-xr-x    1 1031     users        4620 Feb 26 16:23 last.cgi
-rwx------    1 1031     users        7165 Feb 26 16:22 linsniffer
-rwx------    1 1031     users          75 Feb 26 16:24 logclear
-rwxr-xr-x    1 1031     users          79 Feb 26 16:28 lsattr
-rwxr-xr-x    1 1031     users      632066 Feb 26 15:46 mkxfs
-rwxr-xr-x    1 1031     users       35300 Feb 26 16:23 netstat
-rw-r--r--    1 1031     users           1 Feb 26 16:29 pidfile
-rwxr-xr-x    1 1031     users       33280 Feb 26 16:23 ps
-rw-r--r--    1 root     root          708 Mar  3 04:05 s
-rwxr-xr-x    1 1031     users        4060 Feb 26 16:22 sense
-rw-r--r--    1 1031     users       11407 Jan 27 16:11 services
-rwx------    1 1031     users        8268 Feb 26 16:22 sl2
-rwxr-xr-x    1 1031     users      611931 Feb  8  2002 ssh
-rw-r--r--    1 1031     users         880 Oct 22  2000 ssh_config
-rw-------    1 1031     users         540 Oct 22  2000 ssh_host_key
-rw-r--r--    1 1031     users         344 Oct 22  2000 ssh_host_key.pub
-rw-------    1 1031     users         512 Oct 22  2000 ssh_random_seed
-rw-r--r--    1 1031     users         688 Feb 26 16:29 sshd_config
-rwxr-xr-x    1 1031     users       53588 Feb 26 16:23 top

Bingo! It's rootkit package of course. Now we can map *.rec files to rootkit
files simply comparing file sizes and/or md5sums. We should get something like
this:

23.rec		lk.tgz or lk.tar.gz (last line in install script)
2038.rec	'last' directory
2039.rec	ssh
2040.rec	pidfile
2041.rec	install
2042.rec        Not so easy. My scenario: this inode was once used by
                linsniffer. Then install script rm'ed linsniffer and created
                'computer' file (to send some info via mail) which took
		linsniffer's inode. That's why this file is only 765 bytes
		long (linsniffer has 7165 bytes).
2043.rec	cleaner
2044.rec	inetd.conf
2045.rec	lsattr
2046.rec	services
2047.rec	sense
2048.rec	ssh_config
2049.rec	ssh_host_key
2050.rec	ssh_host_key.pub
2051.rec	ssh_random_seed	
2052.rec	sshd_config
2053.rec	sl2
2054.rec	last.cgi
2058.rec	top
2059.rec	logclear
2060.rec	s
2061.rec	mkxfs

We found most of rootkit files between *.rec files. We didn't found: ifconfig,
linsniffer, ps, netstat and top. Where did they disappear ? The answer is
simple.  Just take a look at rootkit's install script. Netstat, ifconfig, ps
and top were moved over original utils while linsniffer was moved to
/dev/ida/.drag-on and to /dev/ida/".. "

What about unmapped *.rec's ? We still don't know anything about inodes:
30188, 30191, 48284, 8097, 8100, 16110 and 56231.

These 3 files are IMHO unrelated to system compromise:
8097.rec	some directory (I bet that's /tmp/whatisPID)
8100.rec	makewhatis temporary file linked to 8097 directory
16110.rec	some file from /etc/pam.d

While these are original system tools deleted by install script:
30188.rec	original netstat
30191.rec 	original ps 
48284.rec 	original ifconfig
56231.rec 	I first thought it is original top overwritten by some crap
		but it can't be - top is located in /usr/bin (another
                partition). Still have no idea what's that ...

Uff... We have all 29 recovered files if I count right.

Accomplished !

3. Bonus Question:
   Was the rootkit ever actually installed on the system? How do you
   know?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, it was. How do I know ? I can say that looking at rootkit install script
and on some files at filesystem:
- five basic system utilities (netstat, top, ps, ifconfig, lsattr) were removed
  and replaced by trojanized binaries
- two files were added to /dev directory (/dev/rpm and /dev/last)
- two directories were created: /dev/ida/.drag-on and /dev/ida/".. "
- original /etc/inetd.conf and /etc/services were replaced 
- "/usr/bin/lsattr -t1 -X53 -p" line was added to /etc/rc.d/rc.sysinit
- probably /home/httpd/cgi-bin/last.cgi and /usr/sbin/mkxfs were added (no way
  to check this out because /home partition isn't provided)


Accomplished !

Timings:
-- Recovering files: 30m
-- Writing paper   :  6h ??!!