From goober@ksp.sk Fri May 25 11:48:53 2001 Date: Fri, 25 May 2001 12:30:17 +0200 (CEST) From: Peter Kosinar To: project@honeynet.org Subject: Scan 15 -= [ Scan of the Month 15 ] =- I've treated this SotM similarly to the 'Forensic Challenge' (except that I put much less time into it), so you may find information which is completely unrelated to the questions. I tried to explain my proceeding most of the time, but in some places the detailed explanation would need a complete article, so it's left as an exercise for the reader (these parts will be enclosed in [* *]). Note: Through the analysis, I'll use the words 'attack' and 'rootkit installation' as synonyms. -=[ Preparation for the analysis ]=- First, the partition image was extracted and mounted to /mnt/test using the loopback interface (for safety reasons, it was mounted read-only). # mount honeypot.hda8.dd /mnt/test -oloop,ro In order to coordinate the timestamps, I've reconfigured my system to use the time-zone of the compromised machine. This was done by replacing my /etc/localtime with the one from the machine. -=[ Locating modified files ]=- There are several standard approaches used to identify modifications of the filesystem caused by the attacker. First, it might be useful to find some information about the filesystem itself. For this purpose, we use the utility `dumpe2fs' (part of e2fsprogs package) (irrelevant data removed): # dumpe2fs honeypot.hda8.dd Last mount time: Thu Mar 15 11:22:23 2001 Last checked: Thu Mar 15 05:09:26 2001 If we want to find the files altered by the attacker, it might be helpful to know the approximate time of the attack. If we assume that the attacker had not rebooted the machine, the attack must take place after 11:22:23 (you can verify that this assumption was correct). Furthermore, we can skip some time needed to finish the start-up, hoping that the attacker was not quick enough to intrude the system immediately after boot-up. So I'll assume that the intrusion happened after 12:00. Knowing this, we are ready to locate all the modified or changed files (although these words sound like synonyms, they refer to two different timestamps used on ext2 filesystem). I've omitted the files _accessed_ after that time intentionally, because there are usually many false positives among them and this case is not an exception. Of course, during a complete forensic analysis, one should not ignore them. # touch -t 031512002001.00 /tmp/ref1 # find \( -newer /tmp/ref1 -or -cnewer /tmp/ref1 \) -ls > /tmp/list1 Another standard (although more time-consuming) method for detecting the altered files is checking their inode numbers. On most fresh installations, files from the same package have similar inode numbers. Doing a simple `ls -ai' in /bin shows that almost all files have inode number around 30100, with the exception of `ps' (inode 2055) and `netstat' (2056). Similar check in /sbin yields `ifconfig' (2057). Cross-checking them with our first list shows 100% full match. Quick look at the files in /tmp/list1 shows that most of them had been modified around 19:45, which gives us the probable time of the incident. So far we have found some files that have been altered during the attack. There is a high probability that most of them belong to the rootkit installed by the attacker. On the first glance, the rootkit seems to be hiding itself in /dev/ida/.{drag-on,. } directories. However, before we proceed to prove this, we should check the deleted files too. -=[ Recovering the deleted files ]=- Undeleting files on ext2 partition can be done in three different ways -- easy, medium and difficult :-). The easy way consists of running Midnight Commander, which supports undeleting (Command/Undelete files). Inserting honeypot.hda8.dd as a filesystem name you can quickly recover all the deleted files from the disk. The medium way consists of running debugfs (another useful utility from e2fsprogs) and dumping all the deleted files manually. And finally, the difficult way is writing your own program for dumping the used inodes (for an example, you can have a look at my Forensic Challenge submission[1]). The advantage of your own program is that it may be able to find more information than `mc' or `debugfs'. On the other hand, in this case (for the demonstration purposes) I used the easiest approach (copying the files under midnight commander). The undeleted files were stored in /tmp/deleted. # ls -l /tmp/deleted -rw-r--r-- 1 honeypot honeypot 239 Mar 15 05:20 16110:1 -rwxr-xr-x 1 honeypot honeypot 611931 Mar 15 19:45 2039:602 -rw-r--r-- 1 honeypot honeypot 1 Mar 15 19:45 2040:1 -rwx------ 1 honeypot honeypot 3713 Mar 15 19:45 2041:4 -rw-r--r-- 1 honeypot honeypot 796 Mar 15 19:45 2042:1 -rwxr-xr-x 1 honeypot honeypot 1345 Mar 15 19:45 2043:2 -rw-r--r-- 1 honeypot honeypot 3278 Mar 15 19:45 2044:4 -rwxr-xr-x 1 honeypot honeypot 79 Mar 15 19:45 2045:1 -rw-r--r-- 1 honeypot honeypot 11407 Mar 15 19:45 2046:12 -rwxr-xr-x 1 honeypot honeypot 4060 Mar 15 19:45 2047:4 -rw-r--r-- 1 honeypot honeypot 880 Mar 15 19:45 2048:1 -rw------- 1 honeypot honeypot 540 Mar 15 19:45 2049:1 -rw-r--r-- 1 honeypot honeypot 344 Mar 15 19:45 2050:1 -rw------- 1 honeypot honeypot 512 Mar 15 19:45 2051:1 -rw-r--r-- 1 honeypot honeypot 688 Mar 15 19:45 2052:1 -rwx------ 1 honeypot honeypot 8268 Mar 15 19:45 2053:9 -rwxr-xr-x 1 honeypot honeypot 4620 Mar 15 19:45 2054:5 -rwxr-xr-x 1 honeypot honeypot 53588 Mar 15 19:45 2058:54 -rwx------ 1 honeypot honeypot 75 Mar 15 19:45 2059:1 -rw-r--r-- 1 honeypot honeypot 708 Mar 15 19:45 2060:1 -rwxr-xr-x 1 honeypot honeypot 632066 Mar 15 19:45 2061:622 -rw-r--r-- 1 honeypot honeypot 520333 Mar 15 19:45 23:512 -rwxr-xr-x 1 honeypot honeypot 66736 Mar 15 19:45 30188:67 -r-xr-xr-x 1 honeypot honeypot 60080 Mar 15 19:45 30191:60 -rwxr-xr-x 1 honeypot honeypot 42736 Mar 15 19:45 48284:43 -rw-r--r-- 1 honeypot honeypot 12288 Mar 15 05:17 56231:13 -rw-r--r-- 1 honeypot honeypot 16329 Mar 16 04:03 8100:177 Using `strings' and `file', a bit of common sense and two passes through the list of deleted files, one can figure out following correspondence: [* By using additional data coming from analysis performed by `dumper' (see [1]), I was able to find out the original locations of these files *] File Original location ; Comment Pass #1: 16110:1 <-> /etc/pam.d/passwd ; w/o `md5' and `shadow' options, this ; file was probably replaced by newer ; version during the installation 30188:67 <-> /bin/netstat ; original (clean) version 30191:60 <-> /bin/ps ; original (clean) version 48284:43 <-> /sbin/ifconfig ; original (clean) version 8100:177 <-> /tmp/?/w ; temporary file created by `makewhatis' ; which is regularly called by cron 23:512 <-> /lk.tgz ; complete rootkit archive; Using ; the information from this archive, ; we can match most of the remaining ; files with their original names 56231:13 - full of zeroes (->contains no useful information), so we can forget it Pass #2 (using the information from lk.tgz): 2039:602 <-> /last/ssh 2040:1 <-> /last/pidfile 2041:4 <-> /last/install 2042:1 <-> /last/linsniffer ; damaged copy of linsniffer 2043:2 <-> /last/cleaner 2044:4 <-> /last/inetd.conf 2045:4 <-> /last/lsattr 2046:12 <-> /last/services 2047:4 <-> /last/sense 2048:1 <-> /last/ssh_config 2049:1 <-> /last/ssh_host_key 2050:1 <-> /last/ssh_host_key.pub 2051:1 <-> /last/ssh_random_seed 2052:1 <-> /last/sshd_config 2053:9 <-> /last/sl2 2054:5 <-> /last/last.cgi 2058:54 <-> /last/top 2059:1 <-> /last/logclear 2060:1 <-> /last/s 2061:622 <-> /last/mkxfs Notice the gap between 2054 and 2058. Recalling that /bin/{ps,netstat} and /sbin/ifconfig were exactly inodes 2055-2057, it is obvious that these three files come from the rootkit and were replaced during its installation. Now we have all the deleted and modified files (or at least we hope so :-)), so we can proceed to the rootkit analysis. -=[ Rootkit analysis ]=- At first, we shall have a look at the contents of lk.tgz. Viewing the files (strings, grep and less are always your friends), in some cases comparing them with known (clean) versions (or with other rootkits), [* and possibly running them in controlled environment *], etc. gives us following results (I included a few words of explanation in the case of binary files where the functionality was not so obvious): last/cleaner Purpose: According to the header, it is `sauber' (German word meaning `clean'), written by `socked'. As its name suggests, it is a log cleaner, which simply removes all lines containing specified word from almost all /var/log/* files (except for compressed logs, utmp, wtmp and lastlog). Location: deleted inode <2043> last/ifconfig Purpose: Trojaned ifconfig that doesn't display PROMISC flag. How do I know that? : Most known rootkits come with modified ifconfig which hides PROMISC flag. Why should this one be an exception ? :-) Verifying this statement is left as an exercise to the reader. Location: /sbin/ifconfig last/inetd.conf Purpose: Modified version of /etc/inetd.conf, with telnet and pop-3 enabled. POP3 server is ran from a strange location (/usr/cyrus/bin/pop3d) under a strange username (cyrus). The /usr partition was not available for analysis, but I personally doubt there is any file (or even the directory) with such name. Location: deleted inode <2044>, /etc/inetd.conf last/last.cgi Purpose: CGI working as a remote control server. How do I know it? : The file extension suggests that the program is a CGI and `strings' strengthen this suggestion. Checking this can be done simply - you can either put this cgi into your cgi-bin directory (again ONLY on a machine dedicated for this purpose) or you can just set the environment variable QUERY_STRING to the arguments you'd have given to the CGI ('id' or 'ps' might be a good idea). Location: deleted inode <2054>, probably also /home/httpd/cgi-bin/last.cgi (according to last/install). Unfortunately, the /home partition was not available for analysis. last/linsniffer Purpose: Well-known and widely used simply Ethernet sniffer for Linux. How do I know it? : Again, this file is integral part of most rootkits. Current location: deleted inode <2042> (damaged), /dev/ida/.{drag-on,. }/linsniffer last/logclear Purpose: Simple script for restarting linsniffer. Location: deleted inode <2059>, /dev/ida/.{drag-on,. }/logclear last/lsattr Purpose: Shell script which runs backdoored sshd and linsniffer at startup. Looking at the way this script is called from /etc/rc.d/rc.sysinit, it is apparent that its original name was `hdparm'. Location: deleted inode <2045>, /usr/bin/lsattr (probably) (this one is special, because it has `immutable' bit set, thus it'll survive even re-installation of the e2fsprogs package. last/mkxfs Purpose: Backdoored sshd 1.2.27 with the `universal password' feature. How do I know it? : First, `strings' revealed that it looks like a ssh server. The attackers usually want to leave a backdoor in the system and sshd is very good place to do so. `Universal password' (or `global password', ...) is very common for trojaned sshd's (another common feature is password logging). This `feature' allows anyone knowing the correct password to log-in as any user. [* This file was not even stripped off its symbols, which makes search for the backdoor features even easier. A quick session in gdb reveals the variable `rkpass' that holds the universal password, which is, by the way 'Frunza14' *] Location: deleted inode <2061>, /dev/ida.{drag-on,. }/mkxfs, probably also in /usr/sbin/mkxfs last/netstat Purpose: Trojaned netstat 1.19, which hides certain connections. Backdoor configuration is read from /dev/caca. How do I know it? : Usual trick used by the attackers to hide their presence in the system is to modify the utilities that show active connections (=netstat), in order to prevent administrator from seeing them connecting to the system. Finding the configuration file for this `feature' can be a bit tricky, but as it is usually stored in common places, a simple command like: # strings /bin/netstat | grep '^/' | grep -v proc may be a good idea. [* Another nice command is `strace', which (if used properly) shows which files are opened by the program during the execution. As the program _must_ read the configuration at some moment, you only have to figure out which file is the right one. However, this should be done ONLY in controlled environment, because you don't know how the trojaned application behaves. *] Location: /bin/netstat last/pidfile Purpose: Original intention was probably to serve as sshd pidfile. Location: deleted inode <2040> last/ps Purpose: Trojaned ps 1.01, hides certain processes. Backdoor configuration is read from /dev/dsx. How do I know it? : see last/netstat Location: /bin/ps last/s Purpose: SSHD configuration file. The server listens on port 5, HostKey and RandomSeed are read from /dev/ida/.drag-on/. One mistake in this configuration is the PidFile location - /dev/ida/.inet/pid, which is (in my humble opinion) a reminiscence from the original rootkit. Location: deleted inode <2060>, /dev/ida/.{drag-on,. }/s last/sense Purpose: Linsniffer output sorter by Mike Edulla Location: deleted inode <2047>, /dev/ida/.{drag-on,. }/sense last/services Purpose: /etc/services Location: deleted inode <2046>, /etc/services last/sl2 Purpose: DoS attack tool, sometimes called `slice' How do I know it? : `strings' and gdb show that this tool is a using the network. Running it against loopback while monitoring the traffic (tcpdump) is usually a good idea (once again, I have to repeat myself - the computer MUST not be connected to the network). Location: deleted inode <2053>, /dev/ida/.{drag-on,. }/sl2 last/ssh Purpose: Standard ssh client, version 1.2.30 How do I know it? : A quick look at the executable shows that it looks like a ssh client. Location: deleted inode <2039> last/ssh_config Purpose: SSH configuration file Location: deleted inode <2048> last/ssh_host_key Purpose: SSH secret host key. From the contents of this file it seems that it was created by root@dil2.datainfosys.net. Location: deleted inode <2049>, /dev/ida/.{drag-on,. }/ssh_host_key last/ssh_host_key.pub Purpose: SSH public host key. Corresponds to the last/ssh_host_key mentioned above. Location: deleted inode <2050> last/ssh_random_seed Purpose: SSH random seed (what else ? :-) ) Location: deleted inode <2051>, /dev/ida/.{drag-on,. }/ssh_random_seed last/sshd_config Purpose: A bit broken sshd configuration file Location: deleted inode <2052> last/top Purpose: Trojaned top 1.01, doesn't display specific processes. Reads its (backdoor) configuration from /dev/dsx. How do I know it? : see last/netstat Location: deleted inode <2058>, probably also /usr/bin/top (according to last/install), but this remains unproven, because of lack of data. last/install Purpose: Main installation script. This one will be our main target, because it shows which files are parts of the rootkit, where shall we look for the rootkit files and how to get rid of them. I'll have a look at this script in the next paragraph. Location: deleted inode <2041> [* According to atime record, the archive (lk.tgz) was last time accessed at 19:44:50. All these files: last/{ssh, pidfile, cleaner, ssh_config, ssh_host_key.pub, sshd_config} have the same atime, so it is highly probable that they were just deleted after their extraction from the archive (and thus are redundant). *] Now, let's have a look at the installation script. At the beginning, it checks for presence of `make', `gcc' and `ssh' (although the latter is not checked properly). Afterwards, it deletes /sbin/ifconfig, /usr/bin/top and /bin/{netstat,ps}. All the files are replaced with their trojaned versions. Three of them (ps,ifconfig,netstat) have been effectively moved from one directory to another, but still on the same filesystem -> that's why we haven't found them among the deleted files (and that's also the reason why their inode numbers were different from the rest of the directory). The last one (top) was moved between two different partitions, so it was deleted on the / partition and (probably) created on /usr. mkxfs is also copied to /usr/sbin. Following this, two backdoor configuration files are created (/dev/last and /dev/rpm) containing a few process names, IP addresses and ports. However, neither of these two files is ever read by the backdoored executables (they read their configuration from /dev/dsx and /dev/caca). The next step is creating /dev/ida/.{drag-on,. } directories and copying linsniffer, logclear, sense, sl2, mkxfs, s, ssh_host_key and ssh_random_seed into both of them (following this, tcp.log is created in both directories). Then, inetd.conf and services are copied to /etc. However, this time they _overwrite_ the original files, so we wouldn't find them by comparing their inode numbers. Inetd is sent a HUP signal (forcing it to reload inetd.conf). /usr/bin/lsattr is deleted and replaced with the mentioned script, this script is set to be `immutable' and a line is inserted into /etc/rc.d/rc.sysinit, which calls /usr/bin/lsattr at every boot (with a few strange switches, -t1 -X53 -p, that look pretty much like hdparm switches). Finally, the last.cgi is moved to a suitable location (web-server's cgi-bin directory). At last, a mail containing basic information about the computer is sent to two addresses (last@linuxmail.org and bidi_damm@yahoo.com). At the end, the installation script removes the rootkit package. -=[ Summary and Answers ]=- The programs in this rootkit show very high level of similarity to those from Scan 13 (rootkit package xzibit.tar.gz). In some cases, they are exact copies, in the others, this script kiddie (I'm sorry if I offend someone, but the behavior looks exactly like a kiddie) modified some scripts to look more l33t :-) without knowing how they really work. As a result, he (or maybe she) installed half-a-dozen trojaned utilities that do nothing more than their regular versions (as there is no configuration file for them, they behave as if they were clean). Furthermore, if the attacked system had not supported libc.5 (luckily for the kiddie, this one did - according to /etc/ld.so.cache), the programs would haven't even worked. Q1: Show step by step how you identify and recover the deleted rootkit from the / partition. A1: The steps have been shown in previous paragraphs. Q2: What files make up the deleted rootkit? A2: The whole rootkit package can be found in one deleted file (lk.tgz). Besides that, another 20 deleted files are programs coming from the rootkit (all of them are mentioned above). List of files that have been added/modified and are parts of the rootkit (there are a few files that were modified, but are not parts of the rootkit - some of them had been modified by the attacker manually) is here (coming from /tmp/list1 which was mentioned above): /dev/ida /dev/ida/.drag-on /dev/ida/.drag-on/linsniffer /dev/ida/.drag-on/logclear /dev/ida/.drag-on/sense /dev/ida/.drag-on/sl2 /dev/ida/.drag-on/mkxfs /dev/ida/.drag-on/s /dev/ida/.drag-on/ssh_host_key /dev/ida/.drag-on/ssh_random_seed /dev/ida/.drag-on/tcp.log /dev/ida/..\ /dev/ida/..\ /linsniffer /dev/ida/..\ /logclear /dev/ida/..\ /sense /dev/ida/..\ /sl2 /dev/ida/..\ /mkxfs /dev/ida/..\ /s /dev/ida/..\ /ssh_host_key /dev/ida/..\ /ssh_random_seed /dev/ida/..\ /tcp.log /dev/rpm /dev/last /etc/services /etc/rc.d/rc.sysinit /etc/inetd.conf /etc/ftpaccess /bin/netstat /bin/ps /sbin/ifconfig It is highly probable that following files were created/altered too: /home/httpd/cgi-bin/last.cgi /usr/bin/top /usr/bin/lsattr /usr/sbin/mkxfs QB: Was the rootkit ever actually installed on the system? How do you know? AB: As far as I can say, the rootkit _was_ installed successfully. At least three system binaries had been replaced with their trojaned versions, start-up script was modified, ... References: [1] - http://project.honeynet.org/challenge/results/submissions/peter/programs/dumper.c The referenced utilities are available in almost all Linux distributions. ----------------------------- Peter Kosinar