All analysis work is done in one directory, which contains the following subdirectories:
The image is mounted in loopback mode on the mnt directory:
# mount -o ro,loop,nodev,noexec images/honeypot.hda8.dd mnt
The MD5 hash of all allocated files in the image is saved:
# find mnt -type f -exec md5sum {} \; > data/md5.all
The following forensics tools are used for the analysis:
The images directory is used as the morgue directory for Autopsy. The fsmorgue file contains the following:
# cat images/fsmorgue
honeynet.tar.gz       /
During a forensic investigation, it is important to verify that the images have not been tampered with. The MD5 hash of the compressed and uncompressed files will be compared to the values given on the Honeynet site.
# md5sum images/honeynet.tar.gz
0dff8fb9fe022ea80d8f1a4e4ae33e21 images/honeynet.tar.gz
# md5sum images/honeypot.hda8.dd
5a8ebf5725b15e563c825be85f2f852e images/honeypot.hda8.dd
The goal of this analysis is to identify deleted files from a rootkit. The difficulty of this task depends greatly on what OS and file system type was used. As this analysis is being performed on an EXT2FS image from a Redhat Linux system, the task maybe easy.
When Linux deletes a file, the file name and inode value are not deleted in the directory entry list and the block pointers are not deleted from the inode contents. Other systems, such as Solaris, delete the inode value from the directory entry and delete the block pointers from the inode. Therefore, recovering a recently deleted file is much easier on a Linux system than on a Solaris system.
The first step is to make a timeline from the unallocated inodes and unallocated directory entries, using TCT and TCTUTILs. The timeline details will be explored using the Autopsy Forensic Browser.
The timeline is created by listing the unallocated inodes, using ils (TCT), and converting them to the mactime format, using ils2mac (TCT). mactime (TCT) is used to create the timeline, and fls (TCTUTILs) and mac_merge (TCTUTILs) are then used to add deleted file names to it.
> 03/14/2001 > data/mactime.tmp
There is a slight error in the timeline because there is an entry for
February 8, 2002 at the top of the timeline, which is out of order.
This will cause mac_merge to not work, so open the timeline
and move it
to the end of the timeline. Next, run fls and mac_merge
to integrate deleted file names.
# fls -m "mnt/" images/honeypot.hda8.dd 2 > data/hda8.fls
# mac_merge data/mactime.tmp data/hda8.fls > data/mactime.txt
The timeline now contains entries for all unallocated inodes (from ils) and some deleted file names (from fls). Inodes that were unallocated due to regular system maintenance can be identified easily. For example, at 06:19:37 (CST) the following 3 unallocated inodes and 3 fls entries can be correlated and there is no need to examine them right now.
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-12107>
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-20883>
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-28172>
16 ma. lrwxrwxrwx 0     0     mnt/etc/rc.d/rc4.d/K83ypbind (deleted)
16 ma. lrwxrwxrwx 0     0     mnt/etc/rc.d/rc2.d/K83ypbind (deleted)
16 ma. lrwxrwxrwx 0     0     mnt/etc/rc.d/rc5.d/K83ypbind (deleted)
The timeline contains many administrative file deletions until 20:36:48 (CST). At this point, we see a modification time for inode 23, which is identified by fls as /lk.tgz. Based on the name and size (500kB), this is probably the installation file for the rootkit. This file is recovered using icat (TCT).
# icat images/honeypot.hda8.dd 23 > recover/i23.lk.tar.gz
At 20:44:50, there are six unallocated inodes that were accessed (2039, 2040, 2043, 2048, 2050, 2052), in addition to inode 23. The inode contents will be examined using the inode browsing mode of Autopsy.
When inode 2039 is entered into Autopsy, the inode and block details are listed for the 600kB file (screen shot). The view as file option is selected and it is obvious that this file is an executable because it starts with "ELF". By selected the strings option, the file is run through the strings(1) UNIX utility and is much easier to read. A quick scan of the output indicates that this is an ssh client (screen shot). It is unknown at this point if this was the original binary that was replaced by a trojan version or if this is a copy of a trojan version. It will be recovered for further analysis.
# icat images/honeypot.hda8.dd 2039 > recover/i2039.ssh.elf
A similar process is performed on the other inodes listed in the timeline. Each executable file is run through strings to identify what its original function was. The following is found:
inode | fls Name | Type | Description (from strings) |
---|---|---|---|
23 | /lk.tgz | gzip tar | Rootkit installation file |
2038 | /last | directory | |
2039 | ELF | ssh client | |
2040 | All zeros | ||
2041 | sh script | installs rootkit | |
2042 | ELF | unknown | |
2043 | bash script | sauber: removes log entries | |
2044 | text | /etc/inetd.conf file | |
2045 | sh script | runs mkxfs and linsniffer | |
2046 | text | /etc/services file | |
2047 | perl script | sorts output of linsniffer | |
2048 | text | ssh client config file | |
2049 | binary | SSH private key for root@dil2.datainfosys.net | |
2050 | binary | ssh key file | |
2051 | binary | unknown data | |
2052 | text | ssh server config file | |
2053 | ELF | port scanner | |
2054 | ELF | executes commands via an http server | |
2058 | ELF | top | |
2059 | script | restarts linsniffer | |
2060 | text | ssh server config file | |
2061 | ELF | sshd | |
30188 | ELF | netstat | |
30191 | ELF | ps | |
48284 | ELF | ifconfig | |
56231 | Nothing, all zeros |
Each of the above inodes is recovered and saved to the recover directory. This set of inodes represents all unallocated inodes with a MAC value after March 14. Although, there maybe additional deleted files whose inode has been reallocated. Further analysis will be performed to determine if this happened.
The lk.tgz file is assumed to be the rootkit installation file. It will be opened to identify the files that are contained in it.
# md5sum recover/i23.lk.tar.gzmd5sum
115f438631de8d0a7c03c9d458eb7257 recover/i23.lk.tar.gz
# gunzip recover/i23.lk.tar.gz
# md5sum recover/i23.lk.tar
a68523a575b6e6b35738103d6d120b94 recover/i23.lk.tar
# cd recover
# tar xf i23.lk.tar
# cd ..
will be used to correlate the unallocated inodes with files in the rootkit. The MD5 value of the recovered inodes and the MD5 value of the rootkit files will be generated and uniq will be used to identify the files that were deleted but not in the rootkit and the files that were in the rootkit but not deleted.
# md5sum recover/* recover/last/* | sort | uniq -u -w 32
086394958255553f6f38684dad97869e recover/last/ifconfig
2b07576213c1c8b942451459b3dc4903 recover/last/netstat
5e1725f2734365fef9e55398785f3033 recover/i30191.ps.exe
6c0f96c1e43a23a21264f924ae732273 recover/last/linsniffer
7728c15d89f27e376950f96a7510bf0f recover/last/ps
928c5f9a4b4068a5db47dfdc65ea6cde recover/i2042.exe
a68523a575b6e6b35738103d6d120b94 recover/i23.lk.tar
f174e862d00d0998c3fa4ccd632019b5 recover/i30188.netstat.exe
This shows that many of the recovered inodes were part of the rootkit. Inodes 30191, 2042, and 30188 were not part of the lk.tgz file. From the rootkit files that were not found in the unallocated inodes and the strings analysis, it can be assumed that inode 30188 is the original copy of netstat, which was replaced by the rootkit version. Similarly, inode 30191 is the original version of ps, which was replaced by the rootkit version.
This assumption will be verified by comparing the MD5 value of the /bin/netstat, and /bin/ps binaries with the ones from the rootkit installation. /sbin/ifconfig will also be compared.
# md5sum mnt/bin/netstat recover/last/netstat
2b07576213c1c8b942451459b3dc4903 mnt/bin/netstat
2b07576213c1c8b942451459b3dc4903 recover/last/netstat
# md5sum mnt/bin/ps recover/last/ps
7728c15d89f27e376950f96a7510bf0f mnt/bin/ps
7728c15d89f27e376950f96a7510bf0f recover/last/ps
# md5sum mnt/sbin/ifconfig recover/last/ifconfig
086394958255553f6f38684dad97869e mnt/sbin/ifconfig
086394958255553f6f38684dad97869e recover/last/ifconfig
Therefore, the rootkit was installed and the above binaries were replaced with trojan versions.
To identify if other system files were replaced by the rootkit, md5sum will again be used. As shown in Section 1.1, the MD5 hash values of all allocated files were saved to data/md5.all. The following commands search the file of MD5 hashes for ones that match those in the rootkit.
# for i in recover/last/*
> do echo $i;
> grep `md5sum $i` data/md5.all;
> done;
recover/last/cleaner
recover/last/ifconfig
data/md5.all:086394958255553f6f38684dad97869e mnt/sbin/ifconfig
recover/last/inetd.conf
data/md5.all:b63485e42035328c0d900a71ff2e6bd7 mnt/etc/inetd.conf
recover/last/install
recover/last/last.cgi
recover/last/linsniffer
data/md5.all:6c0f96c1e43a23a21264f924ae732273 mnt/dev/ida/.drag-on/linsniffer
data/md5.all:6c0f96c1e43a23a21264f924ae732273 mnt/dev/ida/.. /linsniffer
recover/last/logclear
data/md5.all:5f22ceb87631fbcbf32e59234feeaa5b mnt/dev/ida/.drag-on/logclear
data/md5.all:5f22ceb87631fbcbf32e59234feeaa5b mnt/dev/ida/.. /logclear
recover/last/lsattr
recover/last/mkxfs
data/md5.all:18a2d7d3178f321b881e7c493af72996 mnt/dev/ida/.drag-on/mkxfs
data/md5.all:18a2d7d3178f321b881e7c493af72996 mnt/dev/ida/.. /mkxfs
recover/last/netstat
data/md5.all:2b07576213c1c8b942451459b3dc4903 mnt/bin/netstat
recover/last/pidfile
data/md5.all:68b329da9893e34099c7d8ad5cb9c940 mnt/etc/at.deny
recover/last/ps
data/md5.all:7728c15d89f27e376950f96a7510bf0f mnt/bin/ps
recover/last/s
data/md5.all:06d04fa3c4941b398756d029de75770e mnt/dev/ida/.drag-on/s
data/md5.all:06d04fa3c4941b398756d029de75770e mnt/dev/ida/.. /s
recover/last/sense
data/md5.all:464dc23cac477c43418eb8d3ef087065 mnt/dev/ida/.drag-on/sense
data/md5.all:464dc23cac477c43418eb8d3ef087065 mnt/dev/ida/.. /sense
recover/last/services
data/md5.all:54e41f035e026f439d4188759b210f07 mnt/etc/services
recover/last/sl2
data/md5.all:4cfae8c44a6d1ede669d41fc320c7325 mnt/dev/ida/.drag-on/sl2
data/md5.all:4cfae8c44a6d1ede669d41fc320c7325 mnt/dev/ida/.. /sl2
recover/last/ssh
recover/last/ssh_config
recover/last/ssh_host_key
data/md5.all:c2c1b08498ed71a908c581d634832672 mnt/dev/ida/.drag-on/ssh_host_key
data/md5.all:c2c1b08498ed71a908c581d634832672 mnt/dev/ida/.. /ssh_host_key
recover/last/ssh_host_key.pub
recover/last/ssh_random_seed
data/md5.all:ad265d3c07dea3151bacb6930e0b72d3 mnt/dev/ida/.. /ssh_random_seed
recover/last/sshd_config
recover/last/top
This data shows that the rootkit files were also copied to '/dev/ida/.. /' and '/dev/ida/.drag-on/'. After this step, all recovered inodes except 2042 have been identified.
Inode 2038 is identified by fls as the deleted /last directory. This directory will be examined for further information. When a directory is deleted, Linux sets the size to 0 and therefore icat can not be used to view the directory contents. ils though, will always show the first two blocks that an inode uses.
# ils images/honeypot.hda8.dd 2038
[...]
2038|f|1031|100|984707105|984707105|984707105|984707169|40755|0|0|8481|0
Block 8481 contains the directory entry contents. Its contents can be viewed in Autopsy, using Block Browsing mode (screen shot). For the sake of this document, bcat (TCTUTILs) will be used on the command line.
# bcat -h images/honeypot.hda8.dd 8481 512[...]
0 f6070000 0c000102 2e000000 02000000 .... .... .... .... 16 f4030202 2e2e0000 f7070000 0c000301 .... .... .... .... 32 73736800 f8070000 10000701 70696466 ssh. .... .... pidf 48 696c6500 f9070000 10000701 696e7374 ile. .... .... inst 64 616c6c00 fa070000 14000801 636f6d70 all. .... .... comp 80 75746572 65720000 fb070000 10000701 uter er.. .... .... 96 636c6561 6e657200 fc070000 14000a01 clea ner. .... .... 112 696e6574 642e636f 6e660000 fd070000 inet d.co nf.. .... 128 10000601 6c736174 74720000 fe070000 .... lsat tr.. .... 144 20000801 73657276 69636573 ff070000 ... serv ices .... 160 10000501 73656e73 65000000 00080000 .... sens e... .... 176 28000a01 7373685f 636f6e66 69670000 (... ssh_ conf ig.. 192 01080000 14000c01 7373685f 686f7374 .... .... ssh_ host 208 5f6b6579 02080000 30001001 7373685f _key .... 0... ssh_ 224 686f7374 5f6b6579 2e707562 03080000 host _key .pub .... 240 18000f01 7373685f 72616e64 6f6d5f73 .... ssh_ rand om_s 256 65656400 04080000 fc020b01 73736864 eed. .... .... sshd 272 5f636f6e 66696700 05080000 0c000301 _con fig. .... .... 288 736c3200 06080000 dc020801 6c617374 sl2. .... .... last 304 2e636769 07080000 2c000201 70730000 .cgi .... ,... ps.. 320 08080000 20000701 6e657473 74617400 .... ... nets tat. 336 09080000 10000801 6966636f 6e666967 .... .... ifco nfig 352 0a080000 0c000301 746f7000 0b080000 .... .... top. .... 368 10000801 6c6f6763 6c656172 0c080000 .... logc lear .... 384 84020101 73000000 0d080000 78020501 .... s... .... x... 400 6d6b7866 73000000 00000000 00000000 mkxf s... .... .... 416 00000000 00000000 00000000 00000000 .... .... .... ....
This block contains a list of directory entry structures. Each structure contains the inode value, the size, and the file name. By parsing the structures (and converting hex to decimal), the following data is determined:
File Name | Inode |
---|---|
ssh | 2039 |
pidfile | 2040 |
install | 2041 |
computer | 2042 |
cleaner | 2043 |
inetd.conf | 2044 |
lsattr | 2045 |
services | 2046 |
sense | 2047 |
ssh_config | 2048 |
ssh_host_key | 2049 |
ssh_host_key.pub | 2050 |
ssh_random_seed | 2051 |
sshd_config | 2052 |
sl2 | 2053 |
last.cgi | 2054 |
ps | 2055 |
netstat | 2056 |
ifconfig | 2057 |
top | 2058 |
logclear | 2059 |
s | 2060 |
mkxfs | 2061 |
When this list is compared to the results found in Section 2.2, many of the original assumptions are verified. According to this data, inode 2042 was allocated by the /last/computer file. It would seem that all inodes have been accounted for and the analysis is over. Upon rootkit examination however, this is not true. The install file shows that the computer file should be ASCII text, but the previous strings analysis shows that it is an ELF executable. Therefore, either the inode or block was reallocated to a new file. Further analysis shows that inode 2042 points to block 9124, as does inode 2053. Inode 2053 has been identified as the sl2 executable.
1. The recovery steps are shown above.
2. The following files make up the deleted rootkit (lk.tgz from inode 23):
The installation of the rootkit creates a file named computer. While this file could not be recovered, its contents were found in blocks 90417 and 90418 by performing a search in Autopsy for the string "Bogomips".
The analysis also found the original ps, netstat, and ifconfig executables.
The bonus question is if the rootkit was actually installed. The answer is yes, based on the existence of the '/dev/ida/.drag-on/' and '/dev/ida/.. /' directories, the recovered computer file, the replaced binaries (ps, netstat, and ifconfig), and the replaced /etc/ files.
Brian Carrier [carrier@cerias.purdue.edu]