Analysis for Scan of the Month 15 by Albert Bendicho

Analysis for Scan of the Month 15

By Albert Bendicho, May 2001

mail: bendi#at#redestb.es

Introduction

What we want to do

The purpose of our work is to recover the rootkit on the disk. We will also look at the rootkit and study if it was installed. In the conclusion section we will summarize all the information while answering the questions of the challenge.

How do we do it

First on "Steps to recover the toolkit" we explain how we find deleted files by using "ils" and "icat". This utility is part of "The Coroner's Toolkit" (TCT) by Dan Farmer and Wietse Venema.

Next we analyze the files found through this process to try to find the rootkit. Giving a fast look at those files we find what seems to be the rootkit.

Then on "Analysis done on the rootkit" we analyze the contents of the rootkit and its behavior.

In "Analyzing the other files" we do some analysis on the other recovered files while taking advantage of what we know from the rootkit so far.

Next on "Analysis done on the disk" we mount the disk and look for more traces of the installation of the rootkit.

Finally on "Advanced research" we show some extra recovered information that even that's not needed for the purpose of our task (recovering the toolkit and checking if it was installed) it gives us some extra information.

What we need

All we need is the image to analyze and TCT on a Linux machine. We are using RedHat 7.0 with a 2.2.16-22 kernel (using a 2.4 kernel gave problems trying to mount honeypot.hda8.dd).

Note: It would have been better to use RedHat 6.2, since it was the system installed on the honeypot, but I had none at hand. This would have helped to compare files.

Steps to recover the rootkit

Execute "`ils`" to find free inodes on honeypot.hda8.dd

We do that by executing;
/path/to/TCT/bin/ils honeypot.hda8.dd > deleted-info.txt
A quick look at deleted-info.txt tells us that we can recover several files, so we try to do it with "icat"

Execute "`icat`" on all the files

We can do it by issuing the following command;

awk -F'|' 'BEGIN {print "#!/bin/sh"} /^[0-9]+/ {print "echo " $1; print "/path/to/TCT/bin/icat honeypot.hda8.dd " $1 " > recovered" $1 ".txt" }' deleted-info.txt > recoverall.sh

We give execution privileges to recoverall.sh by issuing;

chmod u+x recoverall.sh

and then we execute it;

./recoverall.sh

we get

23
: other inodes recovered
12107
/home/bendicho/forensic/tct-1.06/bin/icat: read (512@1807052486656): Success
16110
20883
/home/bendicho/forensic/tct-1.06/bin/icat: read (512@1807052486656): Success
22103
: other inodes recovered
28172
/home/bendicho/forensic/tct-1.06/bin/icat: read (512@1807052486656): Success
30188
: other inodes recovered

So it seems that all but inodes 12107, 20883 and 18172 have been recovered. A look at the corresponding "recovered###.txt" files shows that these three inodes have created a file with a 0 length. We won't look further to those files unless needed.

Next we try to see what have we recovered. We execute "file" on all of those files;

file recovered*

We get;

recovered12107.txt: empty
recovered16110.txt: ASCII text
recovered2038.txt:  empty
recovered2039.txt:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
recovered2040.txt:  ASCII text
recovered2041.txt:  Bourne shell script text
recovered2042.txt:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
recovered2043.txt:  Bourne-Again shell script text
recovered2044.txt:  English text
recovered2045.txt:  Bourne shell script text
recovered2046.txt:  English text
recovered2047.txt:  perl commands text
recovered2048.txt:  English text
recovered2049.txt:  data
recovered2050.txt:  ASCII text
recovered2051.txt:  data
recovered2052.txt:  ASCII text
recovered2053.txt:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
recovered2054.txt:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
recovered2058.txt:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
recovered2059.txt:  ASCII text
recovered2060.txt:  ASCII text
recovered2061.txt:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
recovered20883.txt: empty
recovered22103.txt: empty
recovered22104.txt: empty
recovered22105.txt: empty
recovered22106.txt: empty
recovered22107.txt: empty
recovered22108.txt: empty
recovered23.txt:    gzip compressed data, deflated, last modified: Sat Mar  3 04:09:06 2001, os: Unix
recovered28172.txt: empty
recovered30188.txt: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
recovered30191.txt: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
recovered48284.txt: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
recovered56231.txt: ASCII text
recovered8097.txt:  empty
recovered8100.txt:  English text

The most interesting file here is "recovered23.txt". We gunzip it by issuing;

cp recovered23.txt recovered23.gz 
gunzip recovered23

We then look at the resulting file with

file recovered23

and we get

recovered23: GNU tar archive

So we untar it with;

tar -xvf recovered23

And we get

last/
tar: Archive contains future timestamp 2002-02-08 14:08:13
last/ssh
last/pidfile
last/install
last/linsniffer
last/cleaner
last/inetd.conf
last/lsattr
last/services
last/sense
last/ssh_config
last/ssh_host_key
last/ssh_host_key.pub
last/ssh_random_seed
last/sshd_config
last/sl2
last/last.cgi
last/ps
last/netstat
last/ifconfig
last/top
last/logclear
last/s
last/mkxfs

So it seems we got the rootkit!!

Analysis done on the rootkit

Files that form the rootkit

ssh; It seems to be secure shell version 1.3.0. To confirm we download and compile the original ssh 1.3.0 and compare the resulting binaries. The size is quite similar (611,931 bytes against 614,300) and comparing the output of "strings" applied to both gives no big difference. It can be both, the original or a trojaned version. We don't look further.
pidfile; Probably a file needed by some of the other programs on the rootkit or a file that was there accidentally when the tarfile was created.
install; A shell script to install the rootkit. Seems to be written by "gata". Comments are in Romanian, so we can suspect a romanian hacker created (or modified from another rootkit) this script. We can try to reach the hacker by trying to get more info from the two e-mail addresses that appear on the script.
linsniffer; A simple network sniffer. Not a full blown sniffer but very good for password sniffing and similar things.
cleaner; It's "Sauber" ("cleanly" in German), a shell script done by "socked" to clean system logs at /var/log
inetd.conf; An "inetd.conf" configuration file (usually found on /etc) with the ssh protocol ready.
lsattr; A shell script that uses a directory named /dev/ida/.drag-on and executes a LinSniffer and a program called "mkxfs" (usually a program to create an xiafs file system, but we see that it's part of the rootkit).
services; A "services" file (usually found on /etc). Compared with the one on the system we use (a RH7.0, not a RH 6.2) reveals that doesn't have nothing unusual.
sense; A perl script that sorts out the output of LinSniffer. More specifically it can give transcripts of telnet, FTP and imap sessions as well as other TCP "dialogs".
ssh_config; Default configuration file for SSH.
ssh_host_key; File for SSH.
ssh_host_key.pub; File for SSH.
ssh_random_seed; File for SSH.
sshd_config; File for SSHD.
sl2; Doing a "strings" on it we find some strings that we can try to find at an internet search engine that lets us recognize this program as probably being "Syn Flooder by Zakath".
last.cgi; Doing a "strings" on it we find some strings that we can try to find at an internet search engine that lets us recognize it as probably pertaining to "becys.cgi" used on "Scan of the Month 13". A CGI script that allows execution of any command giving the output of the command through http.
ps; A "ps" replacement.
netstat; A "netstat" replacement.
ifconfig; An "ifconfig" replacement.
top; A "top" replacement.
logclear; A script to clear sniffed stuff with linsniffer and restart it.
s; File for SSHD. Very similar to "sshd_config".
mkxfs; A "strings" on it reveals that it looks like a ssh server (sshd).

We take a look at the "install" file to know what we have to expect when we analyze the other recovered files and the mounted disk. We get an idea of what gets removed, copied, moved or created during the installation.

Analyzing the other files

Let's see what have we found with "ils" and "icat". We, obviously, won't cover the empty files and will only comment the ones that seem related to the rootkit. We check the file size and compare with the files on the "last" directory and then we compare them to make sure they're the same. Let's see the first file;

recovered2039.txt; Size 611.931 bytes. Same size as last/ssh. With a;

cmp recovered2039.txt last/ssh

and see that the files are identical. We proceed with all the other files and get the following relations;

`recovered2039.txt`	`last/ssh`
`recovered2040.txt`	`last/pidfile`
`recovered2041.txt`	`last/install`
`recovered2043.txt`	`last/cleaner`
`recovered2044.txt`	`last/inetd.conf`
`recovered2045.txt`	`last/lsattr`
`recovered2046.txt`	`last/services`
`recovered2047.txt`	`last/sense`
`recovered2048.txt`	`last/ssh_config`
`recovered2049.txt`	`last/ssh_host_key`
`recovered2050.txt`	`last/ssh_host_key.pub`
`recovered2051.txt`	`last/ssh_random_seed`
`recovered2052.txt`	`last/sshd_config`
`recovered2053.txt`	`last/sl2`
`recovered2054.txt`	`last/last.cgi`
`recovered2058.txt`	`last/top`
`recovered2059.txt`	`last/logclear`
`recovered2060.txt`	`last/s`
`recovered2061.txt`	`last/mkxfs`

We can see that we have a sequence of inode numbers that relates exactly to the order of extraction of the "recovered23" file with the following exceptions;

recovered2042.txt should correspond to "last/linsniffer", but a comparison gives differences. If we look at the file deleted-info.txt created at the beginning we can see that the "st_block0" of the entry for inode 2042 is the same as for entry 2053. Now if we do a "cmp" with "recovered2053.txt" and "recovered2042.txt";

cmp recovered2053.txt recovered2042.txt

we get

cmp: EOF on recovered2042.txt

confirming that they point to the same blocks (just 2042 is shorter).

We're missing "recovered2055.txt", "recovered2056.txt" and "recovered2057.txt". This would correspond to the files

last/ps
last/netstat
last/ifconfig

But if we look at the installation script we see that these files are moved, not copied, to the new location. We leave as a task to be done later to find out if the inode where /bin/ps, /bin/netstat and /sbin/ifconfig are 2055, 2056 and 2057 respectively.

Finally we check what we have on the other recovered files. We find;

recovered8100.txt is part of a text file. Looks like the output of an "apropos" command.

recovered16110.txt is a PAM configuration file. It has to be noted that allows null passwords (nullok). No further investigation done.

recovered30188.txt is an executable. Doing a "strings" on it it looks like "netstat"

recovered30191.txt is an executable. Doing a "strings" on it it looks like "ps"

recovered48284.txt is an executable. Doing a "strings" on it it looks like "ifconfig"

recovered56231.txt is a file filled with 0's

The "netstat", "ps" and "ifconfig" we're seeing are, with a high probability, the orignal ones that were replaced by the versions on the rootkit.

Analysis done on the disk

Looking for traces of the rootkit

We have seen most of the parts of the kits on the recovered files of the disk, but we're going to analyze now the mounted disk. We mount it with;
mkdir/t
mount -o nodev,loop,noexec,ro honeypot.hda8.dd /t
Checking for the presence of files related to the rootkit we can still find (considering that we only have the / partition and are missing many other partitions);
/t/dev/ida/.drag-on directory with it's rootkit components
/t/dev/ida/".. " directory with it's rootkit components
/t/sbin/ifconfig
/t/bin/netstat
/t/bin/ps
/t/etc/inetd.conf
/t/etc/services
Most of these files have the creation time of "Mar 16 02:45"
All these files are still the same the rootkit installs except for;

/t/dev/ida/.drag-on/ssh_random_seed, changed contents and date "Mar 16 15:45"
/t/dev/ida/.drag-on/tcp.log, has some content and date of "Mar 16 17:28"

Verifying inodes

Doing an
ls -i /t/bin/ps /t/bin/netstat /t/sbin/ifconfig 
We get
2056 /t/bin/netstat 2055 /t/bin/ps 2057 /t/sbin/ifconfig
Confirming that they're on inodes 2055, 2056 and 2057

Advanced research

So far we've done everything to answer completely the questions, but we can still dig a little further. By using "unrm" and we can try to find more things. For example we can recover the mails that the install script sent to the hacker. Here they are;

To: last@linuxmail.org
Subject: placinte
* Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown
* Hostname : asdf1
* IfConfig :           inet addr:127.0.0.1  Bcast:127.255.255.255  Mask:255.0.0.0
          inet addr:172.16.1.108  Bcast:172.16.1.255  Mask:255.255.255.0
* Uptime :   7:45pm  up  8:23,  0 users,  load average: 0.00, 0.00, 0.00
* Cpu Vendor ID : vendor_id     : GenuineIntel
* Cpu Model : model             : 4
model name      : Pentium MMX
* Cpu Speed: cpu MHz            : 200.457171
* Bogomips: bogomips    : 399.77
* Spatiu Liber: Filesystem            Size  Used Avail Use% Mounted on
/dev/hda8             251M   33M  205M  14% /
/dev/hda1              23M  2.4M   19M  11% /boot
/dev/hda6             1.6G  2.1M  1.5G   0% /home
/dev/hda5             1.6G  367M  1.2G  23% /usr
/dev/hda7             251M  5.3M  232M   2% /var

To: bidi_damm@yahoo.com
Subject: roote
* Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown
* Hostname : asdf1
* IfConfig :           inet addr:127.0.0.1  Bcast:127.255.255.255  Mask:255.0.0.0
          inet addr:172.16.1.108  Bcast:172.16.1.255  Mask:255.255.255.0
* Uptime :   7:45pm  up  8:23,  0 users,  load average: 0.00, 0.00, 0.00
* Cpu Vendor ID : vendor_id	: GenuineIntel
* Cpu Model : model		: 4
model name	: Pentium MMX
* Cpu Speed: cpu MHz		: 200.457171
* Bogomips: bogomips	: 399.77
* Spatiu Liber: Filesystem            Size  Used Avail Use% Mounted on
/dev/hda8             251M   33M  205M  14% /
/dev/hda1              23M  2.4M   19M  11% /boot
/dev/hda6             1.6G  2.1M  1.5G   0% /home
/dev/hda5             1.6G  367M  1.2G  23% /usr
/dev/hda7             251M  5.3M  232M   2% /var

But "unrm" in conjunction with "lazarus" can also help in other circumstances. For example suppose that we don't have the information provided by "ils" (that could happen if the hacker hacked that information to try to cover himself or if for some reason this information is trashed). Then "unrm" and "lazarus" can recover the rootkit too. We include a description of a way to recover the rootkit by using only "unrm" and "lazarus".

Another advanced research we can do is execute "mactime" (part of TCT) to have a better feeling of what happened. With;

/path/to/TCT/bin/mactime -Rd /t 01/01/1999 >mactimeoutput.txt

We get a first impression of what happened. And it seems (at first glance) that on March 16th, at 02:45 the hacker began the installation of the rootkit. The same day at 15:39 there's some activity until 15:53, then from 17:22 until 17:28 and finally at 18:20. There's some activity at 11:02 and 18:01, but seems to be generated by the "cron" or "anacron" daemons. We will not investigate further because not having the other partitions (/home, /usr and /var) would make it too hard and because it's not our goal.

Conclusion

1.Show step by step how you identify and recover the deleted rootkit from the / partition.

We've explained it on the "Steps to recover the toolkit" section.

2.What files make up the deleted rootkit?

last/ssh
last/pidfile
last/install
last/linsniffer
last/cleaner
last/inetd.conf
last/lsattr
last/services
last/sense
last/ssh_config
last/ssh_host_key
last/ssh_host_key.pub
last/ssh_random_seed
last/sshd_config
last/sl2
last/last.cgi
last/ps
last/netstat
last/ifconfig
last/top
last/logclear
last/s
last/mkxfs

Look at the "Analysis done on the rootkit" to have a description of all these files.

Bonus Question:

Was the rootkit ever actually installed on the system? How do you know?

Yes, We've seen some of the necessary files still on their place as seen in the "Analysis done on the disk" section while most of the unnecessary ones have been found on the recovered files found by "ils" and "icat" as explained on the "Analyzing the other files" section.
It's also important to mention that the sequence of inodes of the found files perfectly matches the behaviour that could be expected of the uncompression/untaring of the rootkit.
On the "Advanced research" section we've seen the actual mails sent to the hacker by the installation script. From the analysis on the "mactime" output we can also suggest that the hacker loged in to the server and had some activity. More investigation could be done to confirm this specially with the other volumes available.