Scan of the Month 15 Analysis

Show step by step how you identify and recover the deleted rootkit from the / partition.

Note: Most of this is verbatim from Dave's analysis of the forensics challenge. Thanks Dave.

  1. Downloaded and extracted the honeypot.hda8.dd image. Verified md5sums for archive and extracted file
  2. Created a mount.sh script to mount the image: 
    #!/bin/sh
mkdir /t 2>/dev/null
mount -o ro,loop,nodev,noexec honeypot.hda8.dd /t
  3. Downloaded the latest version of the Coroner's Toolkit and compiled.
  4. Added the new binaries to my PATH.
  5. Ran ils since the file I'm looking for was deleted: 
    # ils honeypot.hda8.dd | ils2mac > body-deleted
    6. Recovered all of the deleted files using icat and this script: 
    #!/bin/sh -ex
mkdir files
for file in `perl -ne 'print "$1\n" if /<(.*)>/' body-deleted`
do icat honeypot.hda8.dd ${file#honeypot.hda8.dd-dead-} \
   > files/$file
done
  6. 7. Looked for compressed or non-compressed archive files in the recovered files: 
    # cd files
# file * | grep -v empty
honeypot.hda8.dd-dead-23:    gzip compressed data, deflated, last modified: Fri Mar  2 21:09:06 2001, os: Unix
honeypot.hda8.dd-dead-2039:  ELF 32-bit LSB executable, Intel 80386, version 1
honeypot.hda8.dd-dead-2040:  ASCII text
honeypot.hda8.dd-dead-2041:  Bourne shell script text
honeypot.hda8.dd-dead-2042:  ELF 32-bit LSB executable, Intel 80386, version 1, stripped
honeypot.hda8.dd-dead-2043:  Bourne-Again shell script text
honeypot.hda8.dd-dead-2044:  English text
honeypot.hda8.dd-dead-2045:  Bourne shell script text
honeypot.hda8.dd-dead-2046:  English text
honeypot.hda8.dd-dead-2047:  perl commands text
honeypot.hda8.dd-dead-2048:  English text
honeypot.hda8.dd-dead-2049:  data
honeypot.hda8.dd-dead-2050:  ASCII text
honeypot.hda8.dd-dead-2051:  data
honeypot.hda8.dd-dead-2052:  ASCII text
honeypot.hda8.dd-dead-2053:  ELF 32-bit LSB executable, Intel 80386, version 1
honeypot.hda8.dd-dead-2054:  ELF 32-bit LSB executable, Intel 80386, version 1, stripped
honeypot.hda8.dd-dead-2058:  ELF 32-bit LSB executable, Intel 80386, version 1, stripped
honeypot.hda8.dd-dead-2059:  ASCII text
honeypot.hda8.dd-dead-2060:  ASCII text
honeypot.hda8.dd-dead-2061:  ELF 32-bit LSB executable, Intel 80386, version 1
honeypot.hda8.dd-dead-16110: ASCII text
honeypot.hda8.dd-dead-30188: ELF 32-bit LSB executable, Intel 80386, version 1, stripped
honeypot.hda8.dd-dead-30191: ELF 32-bit LSB executable, Intel 80386, version 1, stripped
honeypot.hda8.dd-dead-48284: ELF 32-bit LSB executable, Intel 80386, version 1, stripped
honeypot.hda8.dd-dead-56231: ASCII text
honeypot.hda8.dd-dead-8100:  English text
  7. Checked the most likely file: 
    # tar tzvf honeypot.hda8.dd-dead-23
drwxr-xr-x 1031/users        0 2001-02-26 14:40:30 last/
tar: Archive contains future timestamp 2002-02-08 07:08:13
-rwxr-xr-x 1031/users   611931 2002-02-08 07:08:13 last/ssh
-rw-r--r-- 1031/users        1 2001-02-26 09:29:58 last/pidfile
-rwx------ 1031/users     3713 2001-03-02 21:08:37 last/install
-rwx------ 1031/users     7165 2001-02-26 09:22:50 last/linsniffer
-rwxr-xr-x 1031/users     1345 1999-09-09 10:57:11 last/cleaner
-rw-r--r-- 1031/users     3278 2001-01-27 09:11:32 last/inetd.conf
-rwxr-xr-x 1031/users       79 2001-02-26 09:28:40 last/lsattr
-rw-r--r-- 1031/users    11407 2001-01-27 09:11:44 last/services
-rwxr-xr-x 1031/users     4060 2001-02-26 09:22:55 last/sense
-rw-r--r-- 1031/users      880 2000-10-22 14:29:44 last/ssh_config
-rw------- 1031/users      540 2000-10-22 14:29:44 last/ssh_host_key
-rw-r--r-- 1031/users      344 2000-10-22 14:29:44 last/ssh_host_key.pub
-rw------- 1031/users      512 2000-10-22 14:29:44 last/ssh_random_seed
-rw-r--r-- 1031/users      688 2001-02-26 09:29:51 last/sshd_config
-rwx------ 1031/users     8268 2001-02-26 09:22:59 last/sl2
-rwxr-xr-x 1031/users     4620 2001-02-26 09:23:10 last/last.cgi
-rwxr-xr-x 1031/users    33280 2001-02-26 09:23:33 last/ps
-rwxr-xr-x 1031/users    35300 2001-02-26 09:23:42 last/netstat
-rwxr-xr-x 1031/users    19840 2001-02-26 09:23:47 last/ifconfig
-rwxr-xr-x 1031/users    53588 2001-02-26 09:23:55 last/top
-rwx------ 1031/users       75 2001-02-26 09:24:03 last/logclear
-rw-r--r-- root/root       708 2001-03-02 21:05:12 last/s
-rwxr-xr-x 1031/users   632066 2001-02-26 08:46:04 last/mkxfs
  8. Created a link (ln honeypot.hda8.dd-dead-23 last.tar.gz) for my info.
  9. Extracted the files with: 
    # tar xzf last.tar.gz

What files make up the deleted rootkit?

Ran the file(1) command to get an idea of what the files were: 
# file *
cleaner:          Bourne-Again shell script text
ifconfig:         ELF 32-bit LSB executable, Intel 80386, version 1, stripped
inetd.conf:       English text
install:          Bourne shell script text
last.cgi:         ELF 32-bit LSB executable, Intel 80386, version 1, stripped
linsniffer:       ELF 32-bit LSB executable, Intel 80386, version 1
logclear:         ASCII text
lsattr:           Bourne shell script text
mkxfs:            ELF 32-bit LSB executable, Intel 80386, version 1
netstat:          ELF 32-bit LSB executable, Intel 80386, version 1, stripped
pidfile:          ASCII text
ps:               ELF 32-bit LSB executable, Intel 80386, version 1, stripped
s:                ASCII text
sense:            perl commands text
services:         English text
sl2:              ELF 32-bit LSB executable, Intel 80386, version 1
ssh:              ELF 32-bit LSB executable, Intel 80386, version 1
ssh_config:       English text
ssh_host_key:     data
ssh_host_key.pub: ASCII text
ssh_random_seed:  data
sshd_config:      ASCII text
top:              ELF 32-bit LSB executable, Intel 80386, version 1, stripped

Was the rootkit ever actually installed on the system? How do you know?

Yes, at least partially.
  1. Calculated an MD5 checksum on all the files in the mounted filesystem image with: 
    #!/bin/sh -ex
find /t -type f | while read file
do
  md5sum "$file"
done > t.md5

    (I used xargs at first, but there were files with whitespace. After reading up on GNU xargs, I could have used -0, so now I know.)

  2. Checked to see if any of the files in the rootkit were in the filesystem with: 
    #!/bin/sh
for file in files/last/*
do
  echo "$file:  "
  sum=`md5sum "$file" | awk '{print $1}'`
  grep -h $sum t.md5 
done
    Which showed: 
    files/last/cleaner:
files/last/ifconfig:
086394958255553f6f38684dad97869e  /t/sbin/ifconfig
files/last/inetd.conf:
b63485e42035328c0d900a71ff2e6bd7  /t/etc/inetd.conf
files/last/install:
files/last/last.cgi:
files/last/linsniffer:
6c0f96c1e43a23a21264f924ae732273  /t/dev/ida/.drag-on/linsniffer
6c0f96c1e43a23a21264f924ae732273  /t/dev/ida/.. /linsniffer
files/last/logclear:
5f22ceb87631fbcbf32e59234feeaa5b  /t/dev/ida/.drag-on/logclear
5f22ceb87631fbcbf32e59234feeaa5b  /t/dev/ida/.. /logclear
files/last/lsattr:
files/last/mkxfs:
18a2d7d3178f321b881e7c493af72996  /t/dev/ida/.drag-on/mkxfs
18a2d7d3178f321b881e7c493af72996  /t/dev/ida/.. /mkxfs
files/last/netstat:
2b07576213c1c8b942451459b3dc4903  /t/bin/netstat
files/last/pidfile:
68b329da9893e34099c7d8ad5cb9c940  /t/etc/at.deny
files/last/ps:
7728c15d89f27e376950f96a7510bf0f  /t/bin/ps
files/last/s:
06d04fa3c4941b398756d029de75770e  /t/dev/ida/.drag-on/s
06d04fa3c4941b398756d029de75770e  /t/dev/ida/.. /s
files/last/sense:
464dc23cac477c43418eb8d3ef087065  /t/dev/ida/.drag-on/sense
464dc23cac477c43418eb8d3ef087065  /t/dev/ida/.. /sense
files/last/services:
54e41f035e026f439d4188759b210f07  /t/etc/services
files/last/sl2:
4cfae8c44a6d1ede669d41fc320c7325  /t/dev/ida/.drag-on/sl2
4cfae8c44a6d1ede669d41fc320c7325  /t/dev/ida/.. /sl2
files/last/ssh:
files/last/ssh_config:
files/last/ssh_host_key:
c2c1b08498ed71a908c581d634832672  /t/dev/ida/.drag-on/ssh_host_key
c2c1b08498ed71a908c581d634832672  /t/dev/ida/.. /ssh_host_key
files/last/ssh_host_key.pub:
files/last/ssh_random_seed:
ad265d3c07dea3151bacb6930e0b72d3  /t/dev/ida/.. /ssh_random_seed
files/last/sshd_config:
files/last/top:
  3. Here are brief comments on each file:
    Kit fileInstalled Location & Notes
    cleaner used during installation and removed (shows up as first half of file 8510.t.txt in grave-robber output)
    ifconfig
    /t/sbin/ifconfig
    probably used to hide promiscuous interfaces (original file is probably 165802.x.txt in grave-robber output)
    inetd.conf 
    /t/etc/inetd.conf
    adds pop-3 service that points to /usr/cyrus/bin/pop3d. Might be another backdoor, but the installer program doesn't seem to install the binary to complete the job...
    install used during installation and removed (shows up as binary file 8499.p.txt in the grave-robber output)
    last.cgi probably in /var/www/cgi-bin which is not available (shows up as binary file 8547.x.txt in grave-robber output)
    linsniffer 
    /t/dev/ida/.drag-on/linsniffer "/t/dev/ida/.. /linsniffer"
    the standard rootkit packet/password sniffer judging by the name (shows up as grave-robber file 8503.x.txt)
    logclear 
    /t/dev/ida/.drag-on/logclear "/t/dev/ida/.. /logclear"
    short script to stop linsniffer, remove the logfile, and restart (shows up at the very end of grave-robber file 8601...txt)
    lsattr probably in /usr/bin which is not available (shows up as grave-robber file 8516...txt)
    mkxfs 
    /t/dev/ida/.drag-on/mkxfs "/t/dev/ida/.. /mkxfs"
    from the references to sshd in the strings(1) output, this is probably a backdoor ssh server (seems to be partially contained in grave-robber file 8738.!.txt)
    netstat 
    /t/bin/netstat
    probably modified to hide promiscuous interfaces, etc. (the original or trojan version seems to be grave-robber file 114994...txt)
    pidfile incorrectly identified /t/etc/at.deny, but interesting since both files contain a single LF (\0A)
    ps 
    /t/bin/ps
    appears to be modified to ignore files running out of /dev/dsx, but that's just a guess. Another guess is that by placing the rootkit files in /dev/ida, the rootkit will lose some of it's stealth? This is probably a bad guess as this version seems to differ a lot from the stock Redhat binary. (original appears to be grave-robber file 115050...txt)
    s 
    /t/dev/ida/.drag-on/s "/t/dev/ida/.. /s"
    a sshd_config file for mkxfs (probably)
    sense 
    /t/dev/ida/.drag-on/sense "/t/dev/ida/.. /sense"
    from the top of the file: "Sorts the output from LinSniffer"
    services 
    /t/etc/services
    looks like the standard Redhat services file. Probably included just in case the local services file was modified as many services don't work unless they are defined.
    sl2 
    /t/dev/ida/.drag-on/sl2 "/t/dev/ida/.. /sl2"
    I've found references that say sl2 is a DoS tool, but I didn't investigate further.
    ssh probably in /usr/bin or /usr/local/bin which are not available
    ssh_config possibly in /usr/local/etc which is not available
    ssh_host_key 
    /t/dev/ida/.drag-on/ssh_host_key "/t/dev/ida/.. /ssh_host_key"
    The host file for sshd.
    ssh_host_key.pub possibly in /usr/local/etc
    ssh_random_seed 
    "/t/dev/ida/.. /ssh_random_seed"
    sshd_config possibly in /usr/local/etc which is not available
    top probably in /usr/bin which is not available. Like ps, it has references to /dev/dsx probably for hiding processes.

    Extra

    Running john the ripper on the two interesting shadow entries yields: 
    root:asdfjkl;:11396:0:99999:7:-1:-1:134540356
jjs:whatever:11396:0:99999:7:-1:-1:134540380
    I'm not sure if jjs is an artifact of the break-in.

    Comments

    I have relied on descriptions of the rootkit files from the files themselves. If this were a real investigation, I would spend the time to test the binaries, to verify they do what they claim.

    Much more analysis could be done to the grave-robber output, etc., but I think that goes beyond the scope of the Challenge.

    ocrow@io.com (Owen Crow)