Question #1
Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.
The first signs that something is amiss is the line "eth0: Promiscuous mode enabled." This indicates that a sniffer program has probably been run.
Just to get my bearings on this new system I do the following:
id uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) pwd /root ls -al total 52 drwxr-x--- 5 root root 4096 Aug 10 20:29 . drwxr-xr-x 18 root root 4096 Aug 10 15:54 .. -rw-r--r-- 1 root root 1126 Aug 23 1995 .Xresources lrwxrwxrwx 1 root root 9 Aug 10 15:30 .bash_history -> /dev/null -rw-r--r-- 1 root root 24 Jun 10 2000 .bash_logout -rw-r--r-- 1 root root 234 Jul 5 2001 .bash_profile -rw-r--r-- 1 root root 176 Aug 23 1995 .bashrc -rw-r--r-- 1 root root 210 Jun 10 2000 .cshrc drwx------ 2 root root 4096 Aug 6 11:13 .links drwx------ 2 root root 4096 Aug 6 11:51 .ssh -rw-r--r-- 1 root root 196 Jul 11 2000 .tcshrc -rw-r--r-- 1 root root 1257 Jul 14 13:55 anaconda-ks.cfg drwxrwxr-x 2 500 500 4096 Aug 10 15:54 sslstop -rw-r--r-- 1 root root 1627 Jul 4 14:09 sslstop.tar.gz |
I see another sign that something is afoul. The .bash_history file has been changed to a symlink to /dev/null. This will cause bash to stop saving the command history to a file and stop us from getting the very useful information that it would provide. This symlink was created at Aug 10 15:30. The sslstop directory was created 24 minutes later. This appears to be a program to change some config options for apache.
Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 845/smbd tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 732/xinetd tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3137/smbd -D tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 677/identd tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 732/xinetd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 699/sshd tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 732/xinetd tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN 15119/initd tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 759/sendmail: accep tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3137/smbd -D tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN 15119/initd tcp 0 2528 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED 15119/initd udp 0 0 192.168.1.79:137 0.0.0.0:* 850/nmbd udp 0 0 0.0.0.0:137 0.0.0.0:* 850/nmbd udp 0 0 192.168.1.79:138 0.0.0.0:* 850/nmbd udp 0 0 0.0.0.0:138 0.0.0.0:* 850/nmbd udp 0 0 0.0.0.0:3049 0.0.0.0:* 25239/xopen Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 2 [ ACC ] STREAM LISTENING 943 778/gpm /dev/gpmctl unix 4 [ ] DGRAM 7984 3247/syslogd /dev/log unix 2 [ ] DGRAM 15679 732/xinetd unix 2 [ ] DGRAM 7993 3252/klogd unix 2 [ ] DGRAM 1078 893/login -- root unix 2 [ ] DGRAM 990 820/crond unix 2 [ ] DGRAM 924 759/sendmail: accep unix 2 [ ] DGRAM 834 677/identd unix 2 [ ] DGRAM 804 657/apmd unix 2 [ ] STREAM CONNECTED 417 1/init Active IPX sockets Proto Recv-Q Send-Q Local Address Foreign Address State |
There are several suspicious items in this output, which I will discuss in the answer to question #3 below.
To see which files have been modified, I need to generate a new set of checksums:
for i in `cat /mnt/files`
do
md5sum $i
done >/mnt/newsums.md5
Unmount the floppy:
umount/mnt
On the trusted the Linux system, I perform a diff of the checksum files to see which files have changed.
mount /dev/fd0 /mnt
diff host79-2003-08-06.md5 /mnt/newsums.md5 >sums.diff
Checksum diff output
Suspicious items are marked in red.
Many files will have changed during the normal operation of the system. I will ignore these. The changed files that do concern me are:
These are files that are frequently trojaned by a rootkit. (Now I am glad that I have trusted versions on the floppy.) I will discuss these further in the answer to question 8 below.
This process has shown me which files have been modified, but it does not show me which files have been added.
To get this information I need a list of files in a format to compare to the list I created above.
I create this list and remove the files from the /proc, /dev, and /mnt directories:
mount /dev/fd0 /mnt
find / | grep -v ^/proc/ | grep -v ^/dev/ | grep -v ^/mnt/ >files2
To see which files have been added:
diff /mnt/files files2 >files.diff
Files diff output
Suspicious items are marked in red.
Again, many files will have been added during the normal operation of the system.
I look for suspicious additions and find several.
The one that gets my attention first is /lib/.x, so I take a closer look at that:
/mnt/ls -alR /lib/.x
/lib/.x: total 172 drwxr-xr-x 3 root root 4096 Aug 10 15:32 . drwxr-xr-x 8 root root 4096 Aug 10 15:32 .. -rwxr-xr-x 1 apache apache 1223 Mar 20 15:53 .boot -rwxr-xr-x 1 apache apache 17931 Jan 8 2003 cl -rwxr-xr-x 1 apache apache 303 Dec 23 2002 hide -rw-r--r-- 1 root root 222 Aug 10 15:32 hide.log -rwxr-xr-x 1 apache apache 59137 Mar 22 09:00 inst -rw-r--r-- 1 root root 2442 Aug 10 15:32 install.log -rw-r--r-- 1 root root 1 Aug 10 15:32 ip -rwxr-xr-x 1 apache apache 25795 Jan 8 2003 log drwxrwxrwx 2 root root 4096 Aug 10 15:32 s -rwxr-xr-x 1 root root 28632 Aug 10 15:32 sk /lib/.x/s: total 264 drwxrwxrwx 2 root root 4096 Aug 10 15:32 . drwxr-xr-x 3 root root 4096 Aug 10 15:32 .. -rwxrwxrwx 1 root root 5192 Nov 4 2000 lsn -rw-r--r-- 1 root root 1224 Aug 10 16:04 mfs -rw-r--r-- 1 root root 6 Aug 10 15:32 pid -r--r--r-- 1 root root 5 Aug 10 15:32 port -rw------- 1 root root 512 Aug 10 16:32 r_s -rwxrwxrwx 1 root root 536 Dec 28 2002 s_h_k -rwxrwxrwx 1 root root 340 Dec 28 2002 s_h_k.pub -rwxrwxrwx 1 root root 669 Dec 28 2002 sshd_config -rwxrwxrwx 1 root root 217667 Dec 28 2002 xopen |
Notice that several of the files are owned by the apache user.
This is unusual and leads me to believe that they were placed there by an apache exploit.
I investigate these files using the commands file, less, and strings to discover the following.
Output from these commands
============================================================ Time: Sun Aug 10 15:40:47 Size: 100 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:40:50 Size: 80 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:40:56 Size: 60 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:41:08 Size: 40 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 15:41:32 Size: 20 Path: 192.168.1.79 => 63.99.224.38 [21] ------------------------------------------------------------ ============================================================ Time: Sun Aug 10 16:04:13 Size: 44 Path: proxyscan.undernet.org => 192.168.1.79 [23] ------------------------------------------------------------ k |
I continue looking through the list of added files.
The directory /etc/opt/psybnc has been added.
Lets have a look at it:
/mnt/ls -alR /etc/opt/psybnc
psybnc directory listing
This is the psyBNC program, which is used to hold open an IRC connection and to proxy IRC in order to hide the user's IP address. There are three instances of this program currently running, and one has an active connection. This is discussed in the answers below.
The only language files included with this version are English, Italian, and German. The intruder probably speaks one or more of these languages.
Now we know that the intruder likes to use IRC, so lets search memory for any clues we might find.
The file /proc/kcore is mapped to the kernel core memory.
strings /proc/kcore | less
I use the search function of less to look for MSG, JOIN, and other common IRC strings. I find that the intruder has connected to fairfax.va.us.undernet.org and joined the channels #RedCode, #Ef~#Wurst, and #AiaBuni (which has a topic of "a bunch of hacked comps and the hackers")
Two snippets of conversation were found. One looks like the admin of this system talking to the intruder:
PRIVMSG decedatu :<NeThUs`> If you want a shell all you needed to was ask. If you don't mind please PRIVMSG decedatu :<NeThUs`> let me know what you changed the root password to and which exploit you PRIVMSG decedatu :<NeThUs`> used. I don't keep that box very secure as you now know. If you wish to PRIVMSG decedatu :<NeThUs`> have a shell let me know those two pieces of information and I will PRIVMSG decedatu :<NeThUs`> hook PRIVMSG decedatu :<NeThUs`> it up for you. PRIVMSG decedatu :<NeThUs`> -Mike |
Another is the intruder talking to a cohort.
:decedatu!~decedatu@193.231.34.1 PRIVMSG [[[kgb]]] :<BotikMik> uai cnd intra sick ii spunetzi ca am spus io ca il fuck...daca vretzi :decedatu!~decedatu@193.231.34.1 PRIVMSG [[[kgb]]] :<killme> ...totzi murim...dar nu azi...zbye |
Since many users of IRC talk in "chat shorthand" style, it can sometimes be difficult to identify the actual language. However, I do not believe that this is German or Italian. I am guessing that it is Romanian. To test this theory, I do Google searches for spunetzi, vretzi, and totzi. The results for all three searches were overwhelmingly Romanian. This guess is further backed up by the discovery (from the search of kcore, see above) of several systems involved in this incident, which are Romanian. Using the whois command I find the following.
Since the apache logs have been deleted, I search the hard drive for any remnants left in the free space.
strings /dev/sda1 | grep HTTP
All I find are the following entries.
[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): / [Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): / [Sun Aug 10 13:40:51 2003] [error] System: No such fil213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-" 213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-" 213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-" |
The time stamp is about two hours before the /lib/.x files were created, so I will assume that this is the intruder poking around on the web server. The IP address 213.154.118.219 (extreme-service-11.is.pcnet.ro) is another system in Romania. This is consistent with the IP addresses discovered earlier.
Question #2
Explain the impact that your actions had on the running system.
The actions taken to analyze the system will have a small impact on the system. Various log files will be modified and some swap space and disk space will be overwritten. Since this could destroy some evidence of the compromise, best practices recommend that minimal investigation be performed on a live system. The normal practice would be to create an image of the disk using dd or some other imaging tool and perform the analysis on that image. Since this is a virtual machine and we already have a suspended image, I will ignore this step. In fact, since I can easily restore the system back to the state it was in when suspended, I am free to analyze the live system with little concern about the impact of my actions (aside from isolating it from the network.) I can always return to the original suspended image for a more thorough forensic examination.
Question #3
List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).
The trusted copy of the netstat command (see above) showed several suspicious ports open.
I used the /proc directory to get the full path for the suspicious processes.
The /proc directory shows many details for each active process in a directory with the PID as the name.
For example, the file /proc/15119/exe will be a symbolic link to the actual executable file.
See this example
What I found:
Question #4
Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?
The trusted netstat command (see output above) shows an active connection on port 65336 from a system with IP address 213.154.118.200 (sanido-08.is.pcnet.ro), which is an ISP in Romania. The process used for this connection is psybnc, which is a tool for holding open IRC connections and for proxying IRC connections to hide your IP address. We can deduce from this that the service is most likely IRC or IRC DCC protocol.
Question #5
How many instances of an SSH server were installed and at what times?
Question #6
Which instances of the SSH servers from question 5 were run?
By examining the /proc directory I discovered:
Question #7
Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information?
If so, was any information collected?
Once again, I look at the /proc directory to get some further information on a suspicious process.
(PID=25247 See the output)
This process has the same files open as the other instances of the xopen program.
Strangely, though, its exe link does not show up. I think this may be caused by the program
overlaying itself with the lsn program, which is compressed.
It has one additional file open: /lib/.x/s/mfs, which I found earlier to contain password sniffer output.
Therefore, I conclude that this program has been modified to capture passwords.
The file shows several FTP connections and one Telnet connection, however, no passwords were recorded.
This was most likely caused by none being entered.
Question #8
Which system executables (if any) were trojaned and what configuration files did they use?
The diff of the checksums showed that several files were replaced with trojan versions. Examining these files with the strings command hinted at which configuration files they used.
This behavior is consistent with the operation of several common root kits.
Question #9
How and from where was the system likely compromised?
The running apache version is 1.3.20, which has a well-known vulnerability in the SSLv2 handshake. This is the vulnerability exploited by the "Slapper" worm and may be the exploit used to compromise this system. I believe that this or some other apache exploit was used to compromise this system. The files created in the /lib/.x directory which are owned by the apache user lead me to this conclusion.
The search for apache log file remnants showed which system was accessing the web server shortly before the compromise occurred. This leads me to believe that the compromise was most likely executed from the same address: 213.154.118.219 (extreme-service-11.is.pcnet.ro).
Bonus Question:
What nationality do you believe the attacker(s) to be, and why?
Romanian. The IP address seen in the apache log remnant was Romanian.
The active network connection was from Romania.
Several other Romanian IP addresses were encountered in this investigation.
But, most convincingly, the IRC chat conversation discovered appeared to be in Romanian.