From canacar@eee.metu.edu.tr Fri May 25 11:48:37 2001 Date: Fri, 25 May 2001 10:46:45 +0300 From: Can Erkin Acar To: project@honeynet.org Subject: Scan of the month Scan of the Month #15 - May Identify and recover a deleted rootkit from a compromised Linux system. The analysis took about 4 hours, plus the writeup, which took another 4 hours. The rootkit is very similar to previously analyzed kits, with some configuration changes that (un)fortunately make the kit almost useless. Question 1: Show step by step how you identify and recover the deleted rootkit from the / partition. * download & verify checksum: $ ftp http://project.honeynet.org/scans/scan15/honeynet.tar.gz $ md5 honeynet.tar.gz MD5 (honeynet.tar.gz) = 0dff8fb9fe022ea80d8f1a4e4ae33e21 * extract & verify checksum: $ tar xzf honeynet.tar.gz $ cd honeynet $ md5 honeypot.hda8.dd MD5 (honeypot.hda8.dd) = 5a8ebf5725b15e563c825be85f2f852e * use ils (from tct [1]) to list deleted inodes: $ ils -f ext2fs honeypot.hda8.dd >ils.out $ cat ils.out st_ino|st_alloc|st_uid|st_gid|st_mtime|st_atime|st_ctime|st_dtime|st_mode|st_nlink|st_size|st_block0|st_block1 23|f|0|0|984706608|984707090|984707105|984707105|100644|0|520333|307|308 2038|f|1031|100|984707105|984707105|984707105|984707169|40755|0|0|8481|0 2039|f|0|0|1013173693|984707090|984707105|984707105|100755|0|611931|8482|8483 2040|f|0|0|983201398|984707090|984707105|984707105|100644|0|1|9084|0 ... 22108|f|0|0|984754076|984754076|984754076|984754076|100644|0|0|0|0 28172|f|0|0|984655177|984655177|984655225|984655225|120777|0|16|1764699694|779381102 30188|f|0|0|952425102|984677103|984707102|984707102|100755|0|66736|126628|126629 30191|f|0|0|952452206|984677352|984707102|984707102|100555|0|60080|126695|126696 48284|f|0|0|952425102|984677122|984707102|984707102|100755|0|42736|199330|199331 56231|f|0|0|984655056|984655056|984655056|984655056|100644|0|33135|229685|229686 * use icat (from tct) to extract files. Files with mode 120777 are symlinks and need to be filtered out (thus the awk) $ mkdir extract $ awk -F\| '{if (int($1) && $9!=120777) printf "extract.sh %s\n",$1}' ils.out |sh -s where extract.sh is a shell script that extracts its inode argument to a file using the command: icat -f ext2fs honeypot.hda8.dd $1 > extract/inode.$1 * use 'file' to determine filetypes: $ cd extract $ file * inode.16110: ASCII text inode.2038: empty inode.2039: ELF 32-bit LSB executable, Intel 80386, version 1 inode.2040: ASCII text inode.2041: Bourne shell script text inode.2042: ELF 32-bit LSB executable, Intel 80386, version 1 inode.2043: Bourne-Again shell script text inode.2044: English text inode.2045: Bourne shell script text inode.2046: English text inode.2047: perl commands text inode.2048: English text inode.2049: data inode.2050: ASCII text inode.2051: data inode.2052: ASCII text inode.2053: ELF 32-bit LSB executable, Intel 80386, version 1 inode.2054: ELF 32-bit LSB executable, Intel 80386, version 1 inode.2058: ELF 32-bit LSB executable, Intel 80386, version 1 inode.2059: ASCII text inode.2060: ASCII text inode.2061: ELF 32-bit LSB executable, Intel 80386, version 1 inode.22103: empty inode.22104: empty inode.22105: empty inode.22106: empty inode.22107: empty inode.22108: empty inode.23: gzip compressed data, deflated, last modified: Sat Mar 3 05:09:06 2001, os: Unix inode.30188: ELF 32-bit LSB executable, Intel 80386, version 1 inode.30191: ELF 32-bit LSB executable, Intel 80386, version 1 inode.48284: ELF 32-bit LSB executable, Intel 80386, version 1 inode.56231: ASCII text inode.8097: empty inode.8100: English text * inode.23 contains our rootkit. Other files are deleted rootkit files, and replaced system binaries which were deleted by the installation script before being replaced. $ tar xvzf inode.23 last last/ssh last/pidfile last/install last/linsniffer last/cleaner last/inetd.conf last/lsattr last/services last/sense last/ssh_config last/ssh_host_key last/ssh_host_key.pub last/ssh_random_seed last/sshd_config last/sl2 last/last.cgi last/ps last/netstat last/ifconfig last/top last/logclear last/s last/mkxfs Question 2: What files make up the deleted rootkit? $ cd last $ file * cleaner: Bourne-Again shell script text log cleaner script: # sauber - by socked [11.02.99] ifconfig: ELF 32-bit LSB executable, Intel 80386, version 1 modified ifconfig (does not show PROMISC) inetd.conf: English text inetd.conf replacement, all entries are commented out except: telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd pop-3 stream tcp nowait cyrus /usr/cyrus/bin/pop3d pop3d install: Bourne shell script text echo "********* Instalarea Rootkitului A Pornit La Drum *********" echo "********* Mircea SUGI PULA ********************************" does some checks, removes and replaces /sbin/ifconfig, /bin/netstat, /bin/ps /usr/bin/top with trojaned versions, copies mkxfs to /usr/sbin creates /dev/rpm and /dev/last files containing trojan configuration creates /dev/ida/.drag-on and /dev/ida/".. " directories, and copies files linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed and creates an empty file tcp.log to both directories. replaces /etc/inetd.conf and /etc/services with rootkit versions (that disable most services) and signals inetd to reread its configuration. this is probably just to close off the holes used to exploit this machine. replaces /usr/bin/lsattr, and modifies /etc/rc.d/rc.sysinit to run it at startup, also flags it immutable, mode 500 and runs it. searches for possible cgi-bin directory locations and copies last.cgi each of these directories. finally, collects host information (model, cpu, disk, network) and mails the results to last@linuxmail.org and bidi_damm@yahoo.com addresses. last.cgi: ELF 32-bit LSB executable, Intel 80386, version 1 a cgi command shell (executes commands and displays output) linsniffer: ELF 32-bit LSB executable, Intel 80386, version 1 password sniffer logclear: ASCII text script fragment to clear sniffer logs and re-start linsniffer lsattr: Bourne shell script text start-up script. changes to /dev/ida/.drag-on and runs mkxfs (sshd) and linsniffer mkxfs: ELF 32-bit LSB executable, Intel 80386, version 1 sshd binary, possibly trojaned netstat: ELF 32-bit LSB executable, Intel 80386, version 1 trojaned netstat - does NOT include the strings /dev/rpm or /dev/last which are supposed to be configuration files (instead contains /dev/caca) thus it is useless! pidfile: ASCII text empty ps: ELF 32-bit LSB executable, Intel 80386, version 1 trojaned ps (as in netstat, reads config from /dev/dsx instead of /dev/rpm) s: ASCII text sshd configuration file, setup for listening on port 5 sense: perl commands text # Sorts the output from LinSniffer 0.03 [BETA] by Mike Edulla services: English text /etc/services replacement (no obvious misinformation) sl2: ELF 32-bit LSB executable, Intel 80386, version 1 slash - ddos tool [2] ssh: ELF 32-bit LSB executable, Intel 80386, version 1 ssh client ssh_config: English text client configuration file ssh_host_key: data private key file for sshd ssh_host_key.pub: ASCII text public key for sshd ssh_random_seed: data random seed for sshd sshd_config: ASCII text normal sshd_config top: ELF 32-bit LSB executable, Intel 80386, version 1 trojaned top binary (uses /dev/dsx for configuration) Bonus Question: Was the rootkit ever actually installed on the system? How do you know? * mount and examine / partition: # mkdir mnt # vnconfig -c svnd0 honeypot.hda8.dd # mount -t ext2fs /dev/svnd0c mnt # ls -ald mnt/dev/ida/.* drwxr-xr-x 2 root wheel 1024 Mar 16 03:45 mnt/dev/ida/.. drwxr-xr-x 2 root wheel 1024 Mar 16 03:45 mnt/dev/ida/.drag-on # ls -al mnt/dev/ida/.drag-on total 1320 drwxr-xr-x 2 root wheel 1024 Mar 16 03:45 . drwxrwxr-x 4 root wheel 12288 Mar 16 03:45 .. -rwx------ 1 root wheel 7165 Mar 16 03:45 linsniffer -rwx------ 1 root wheel 75 Mar 16 03:45 logclear -rwxr-xr-x 1 root wheel 632066 Mar 16 03:45 mkxfs -rw-r--r-- 1 root wheel 708 Mar 16 03:45 s -rwxr-xr-x 1 root wheel 4060 Mar 16 03:45 sense -rwx------ 1 root wheel 8268 Mar 16 03:45 sl2 -rw------- 1 root wheel 540 Mar 16 03:45 ssh_host_key -rw------- 1 root wheel 512 Mar 16 16:45 ssh_random_seed -rw-r--r-- 1 root wheel 138 Mar 16 18:28 tcp.log # tail -2 mnt/etc/rc.d/rc.sysinit /usr/bin/lsattr -t1 -X53 -p It is clear that the install script was run and the rootkit is installed tcp log contains some entries, possibly belonging to the hacker: cr272065-a.wlfdle1.on.wave.home.com => asdf1 [21] ----- [Timed Out] ns2.giant.net => asdf1 [23] However, the rootkit did NOT function as intended due to misplaced configuration files. It is obvious that it is a re-packaging of rootkit components by an unskilled person (kiddies?). Final Note: Observant readers may have noticed that the analysis was done on an OpenBSD machine (OpenBSD 2.9). The main reason is that I do not have a linux machine with enough bandwidth and diskspace available (no I am not complaining). I had to perform an additional step of persuading tct filesystem tools (ils and icat) to read an ext2fs filesystem on a non-linux machine. Although it was a quick-and-dirty solution (barrowing the ext2fs.h header file from Linux and hacking at the sources until they work) it proved to be stable. I will post the details to the forensics list and anyone interested can contact me personally. References: [1] Dan Farmer and Wietse Venema, The Coroners Toolkit, http://www.porcupine.org/forensics/tct.html [2] sl2: a similar executable was found in Honeynet project 'Scan of the month #13: auto rooter'. The writeups contain enough information to identify the tool. http://project.honeynet.org/scans/scan13/