Scan 15 - Eric Severance <esev@esev.com>

  1. Show step by step how you identify and recover the deleted rootkit from the / partition.

    The first step in finding the deleted rootkit is to find a utility that allows the user to examine of the details of an ext2 filesystem. One such program is debugfs. debugfs can be used to display numerous information about the provided filesystem. debugfs can be started with the following command:

    > /sbin/debugfs ./honeypot.hda8.dd
    debugfs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
    debugfs:

    The first task for debugfs is to locate the inodes of any files which have been recently deleted. The command lsdel is used for this purpose. Executing lsdel gives the following:

    debugfs: lsdel
    29 deleted inodes found.
    InodeOwnerModeSize BlocksTime deleted
    56231010064433135 13/13Thu Mar 15 06:17:36 2001
    1611001006442391/ 1Thu Mar 15 06:20:25 2001
    205801007555358854/ 54Thu Mar 15 20:45:02 2001
    3018801007556673667/ 67Thu Mar 15 20:45:02 2001
    3019101005556008060/ 60Thu Mar 15 20:45:02 2001
    4828401007554273643/ 43Thu Mar 15 20:45:02 2001
    2047010075540604/ 4Thu Mar 15 20:45:03 2001
    204901006005401/ 1Thu Mar 15 20:45:03 2001
    205101006005121/ 1Thu Mar 15 20:45:03 2001
    2053010070082689/ 9Thu Mar 15 20:45:03 2001
    20590100700751/ 1Thu Mar 15 20:45:03 2001
    206001006447081/ 1Thu Mar 15 20:45:03 2001
    20610100755632066622/ 622 Thu Mar 15 20:45:03 2001
    230100644520333512/ 512Thu Mar 15 20:45:05 2001
    20390100755611931602/ 602Thu Mar 15 20:45:05 2001
    2040010064411/ 1Thu Mar 15 20:45:05 2001
    2041010070037134/ 4Thu Mar 15 20:45:05 2001
    204201006447961/ 1Thu Mar 15 20:45:05 2001
    2043010075513452/ 2Thu Mar 15 20:45:05 2001
    2044010064432784/ 4Thu Mar 15 20:45:05 2001
    20450100755791/ 1Thu Mar 15 20:45:05 2001
    204601006441140712/ 12Thu Mar 15 20:45:05 2001
    204801006448801/ 1Thu Mar 15 20:45:05 2001
    20500100644344 1/ 1Thu Mar 15 20:45:05 2001
    205201006446881/ 1Thu Mar 15 20:45:05 2001
    2054010075546205/ 5Thu Mar 15 20:45:05 2001
    203810314075501/ 1Thu Mar 15 20:46:09 2001
    809704070001/ 1Fri Mar 16 05:03:12 2001
    8100010064416329177/ 177Fri Mar 16 05:03:12 2001
    debugfs:

    Now that we have a listing of all the inodes that have been recently deleted we need to determine which inode contains file information for the rootkit. There isn't any straight forward way of locating that inode, the best you can to is form an educated guess. I came up with two methods for locating the rootkit: 1) Look for inodes that are at the beginning of a sequence; 2) Knowing that rootkits are normally tar-gzipped and not executable, look for a large file with 644 permissions. Using these two methods, the most logical choice is inode 23. The dump command can be used within debugfs to recover a deleted inode. I will dump inode 23 into the file test.tar.gz:

    debugfs: dump <23> test.tar.gz
    debugfs: quit

    Now, on the command line, lets see if my guess was correct:

    >tar -xzvf test.tar.gz
    last/
    tar: Archive contains future timestamp 2002-02-08 08:08:13
    last/ssh
    last/pidfile
    last/install
    last/linsniffer
    last/cleaner
    last/inetd.conf
    last/lsattr
    last/services
    last/sense
    last/ssh_config
    last/ssh_host_key
    last/ssh_host_key.pub
    last/ssh_random_seed
    last/sshd_config
    last/sl2
    last/last.cgi
    last/ps
    last/netstat
    last/ifconfig
    last/top
    last/logclear
    last/s
    last/mkxfs

    Sure enough, the rootkit has now been recovered from inode 23 on the compromised filesystem.

  2. What files make up the deleted rootkit?

    The files associated with this rootkit are listed below along with their inodes and an MD5 hash:

    inodefilenameMD5
    2058last/top8ff0939cd49a0b2ef3156c7876afca4b
    2047last/sense464dc23cac477c43418eb8d3ef087065
    2049last/ssh_host_keyc2c1b08498ed71a908c581d634832672
    2051last/ssh_random_seedad265d3c07dea3151bacb6930e0b72d3
    2053last/sl24cfae8c44a6d1ede669d41fc320c7325
    2059last/logclear5f22ceb87631fbcbf32e59234feeaa5b
    2060last/s06d04fa3c4941b398756d029de75770e
    2061last/mkxfs18a2d7d3178f321b881e7c493af72996
    23lk.tar.gz -or- lk.tgz115f438631de8d0a7c03c9d458eb7257
    2039last/ssh21ed3ca31a9c9b51a757f1644e26f2f7
    2040last/pidfile68b329da9893e34099c7d8ad5cb9c940
    2041last/install964db5da8cf89810a54659b6fdb81958
    2043last/cleaner12e8748c19abe7a44e67196c22738e9b
    2044last/inetd.confb63485e42035328c0d900a71ff2e6bd7
    2045last/lsattrdfb2eeea2a5ba23eb6a2b9d0cff9d82f
    2046last/services54e41f035e026f439d4188759b210f07
    2048last/ssh_config5fd2ce512e0eba4d090191e8a1518808
    2050last/ssh_host_key.pube76cd5baaab7b4f28c999946a9cb4dcc
    2052last/sshd_config312de877e5180678cd54606e1c25af40
    2054last/last.cgi202a51b16ac8d1b4dc75de89e7344ed4
    2038last
    2057last/ifconfig086394958255553f6f38684dad97869e
    last/linsniffer6c0f96c1e43a23a21264f924ae732273
    2056last/netstat2b07576213c1c8b942451459b3dc4903
    2055last/ps7728c15d89f27e376950f96a7510bf0f


  3. Bonus Question: Was the rootkit ever actually installed on the system? How do you know?

    To answer this question one must discover what this rootkit actually installs. To do this, take a look at the last/install script. Below is a decomposition of what the install script modifies.

    FileDescriptionEvidenceInstalled?
    /sbin/ifconfig

    -and-

    /bin/netstat

    -and-

    /bin/ps    		 The script removes the original versions of these files and replaces them with a modified version. 1. MD5(sbin/ifconfig)= 086394958255553f6f38684dad97869e
    MD5(bin/netstat)= 2b07576213c1c8b942451459b3dc4903
    MD5(bin/ps)= 7728c15d89f27e376950f96a7510bf0f

    2. The inodes for these files are in the same sequence as the rootkit.    		 Yes
    /usr/bin/top The script removes the original file and replaces it with a modified version. There is no evidence of this because we do not have the /usr partition. Unknown
    /usr/bin/lsattr The script removes the original file and replaces it with a modified version. The script then executes this new version. By examining the modified lsattr it can be seen that this file appends the output of linsniffer to /dev/ida/.drag-on/tcp.log Looking inside this file we find:

    cr272065-a.wlfdle1.on.wave.home.com => asdf1 [21]

    ----- [Timed Out]

    ns2.giant.net => asdf1 [23]
    da#da,~daO~daO~daU~ #'da[~dac~!dan~da~?    		 Yes
    /dev/rpm The script creates a /dev/rpm file and inserts the following data into that file:

    3 sl2
    3 sshdu
    3 linsniffer
    3 smurf
    3 slice
    3 mech
    3 muh
    3 bnc
    3 psybnc    		 > cat dev/rpm
    3 sl2
    3 sshdu
    3 linsniffer
    3 smurf
    3 slice
    3 mech
    3 muh
    3 bnc
    3 psybnc    		 Yes
    /dev/last The script creates a /dev/last file and inserts the following data into that file:

    1 193.231.139
    1 213.154.137
    1 193.254.34
    3 48744
    3 3666
    3 31221
    3 22546
    4 48744
    4 2222    		 > cat dev/last
    1 193.231.139
    1 213.154.137
    1 193.254.34
    3 48744
    3 3666
    3 31221
    3 22546
    4 48744
    4 2222    		 Yes
    /dev/ida/.drag-on

    -and-

    "/dev/ida/.. "    		 These two directories are created and the following files (from the rootkit) are placed within them: linsniffer, logclear, sense, sl2, mkxfs, s, ssh_host_key, ssh_random_seed. Also a file named tcp.log is created within the directories MD5(dev/ida/.drag-on/linsniffer)=
        6c0f96c1e43a23a21264f924ae732273
    MD5(dev/ida/.drag-on/logclear)=
        5f22ceb87631fbcbf32e59234feeaa5b
    MD5(dev/ida/.drag-on/mkxfs)=
        18a2d7d3178f321b881e7c493af72996
    MD5(dev/ida/.drag-on/s)=
        06d04fa3c4941b398756d029de75770e
    MD5(dev/ida/.drag-on/sense)=
        464dc23cac477c43418eb8d3ef087065
    MD5(dev/ida/.drag-on/sl2)=
        4cfae8c44a6d1ede669d41fc320c7325
    MD5(dev/ida/.drag-on/ssh_host_key)=
        c2c1b08498ed71a908c581d634832672
    MD5(dev/ida/.drag-on/ssh_random_seed)=
        97df00e63da3efd62fac4f84802370f0
    MD5(dev/ida/.drag-on/tcp.log)=
        7754ec4105e2b56a4ec6721f522d580d
    MD5(dev/ida/.. /linsniffer)=
        6c0f96c1e43a23a21264f924ae732273
    MD5(dev/ida/.. /logclear)=
        5f22ceb87631fbcbf32e59234feeaa5b
    MD5(dev/ida/.. /mkxfs)=
        18a2d7d3178f321b881e7c493af72996
    MD5(dev/ida/.. /s)=
        06d04fa3c4941b398756d029de75770e
    MD5(dev/ida/.. /sense)=
        464dc23cac477c43418eb8d3ef087065
    MD5(dev/ida/.. /sl2)=
        4cfae8c44a6d1ede669d41fc320c7325
    MD5(dev/ida/.. /ssh_host_key)=
        c2c1b08498ed71a908c581d634832672
    MD5(dev/ida/.. /ssh_random_seed)=
        ad265d3c07dea3151bacb6930e0b72d3
    MD5(dev/ida/.. /tcp.log)=
        d41d8cd98f00b204e9800998ecf8427e    		 Yes
    /etc/inetd.conf

    -and-

    /etc/services    		 These two files are replaced by the files from the rootkit MD5(etc/inetd.conf)= b63485e42035328c0d900a71ff2e6bd7
    MD5(etc/services)= 54e41f035e026f439d4188759b210f07    		 Yes
    /etc/rc.d/rc.sysinit The script adds the following line to the end of /etc/rc.d/rc.sysinit: /usr/bin/lsattr -t1 -X53 -p > tail -2 etc/rc.d/rc.sysinit
    /usr/bin/lsattr -t1 -X53 -p
    		 Yes

    Conclusion: Based on the evidence shown in the table, the rootkit has been installed on this system.