Submitted by

Orlando F. S. Bordoni

(orlandobordoni@bol.com.br)

Honeynet.org Challenge - Scan 19

The Chalenge

On September 16th a Redhat Linux 6.2 honeypot was compromised. This is the 3rd time this system was compromised by the same intruder. The honeynet is VMware based and uses a modified bash to log to syslog. Syslog is remotly logging to 0.0.0.0 (remote syslog server IP has been replaced). The compromised system has an IP of 192.168.1.102. After successfully breaking into the box, the attacker ended up using 3 modes of connecting and running commands (some of which is encrypted). The attacker also tried to hide some of his edits from the MAC times.

1. Which vulnerability did the intruder exploit?

2. What ways, and in what order, did the intruder use to connect and run commands on the system?

3. How did the intruder try to hide his edits from the MAC times?

4. The intruder downloaded rootkits, what were they called? Are they new/custom rootkits?

5. Recover (tell how you did it too) the rootkits from the snort binary capture

6. What does the rootkit do to hide the presence of the attacker on the system?

7. What did you learn from this exercise?

8. How long did this challenge take you?

Bonus Questions:
Based on this challenge, write an example letter of notification to the source owner that attacked the system. Include any evidence or logs that you feel important.

The analysis

Introduction

First of all it is necessary to download the file snort-0315@0005.log.tar.gz (or the .zip version) and check its integrity (md5 checksum) by the command:

md5 snort-0315@0005.log.tar.gz

After checking and unzipping it (if possible ;-), we get two snort log files: the “newdat3.log”, which has all the information about what went on in the honeypot from 00:06 up to 23:09; and the “slog2.log” file, in which we can find the comunications from the 15th September at 23:13 till the 16th at 22:07 between honeypot’s syslog and the server’s syslog (despite recording the system logs in disk, the honeypot’s syslog sends them to the server).

From then on, to make it easier, it is necessary to extract important data which is recorded in the logs. Therefore, we’re supposed to choose what programs to use. To my mind Snort (www.snort.org) and Ethereal (www.ethereal.com) are pretty useful for this.

Several hints about Snort and Ethereal

You can check out several hints about using Snort and Ethereal to be able to analyze more easily the log files. Supposing you’re an expert already, just jump to the next part! ;-)

Snort:

Using Snort we’re allowed to do several types of data extraction. To answer the questions we’ll do an analisys of suspicious material as exploits and portscans, which will give us the exploit that the attacker used to compromise the honeypot; and make a dump of all ASCII conections’ texts, excluding the packet’s headers, in order to see all that was done (including the typed commands and the server’s answers) in telnet or ftp connections, for instance.

To analyze the suspicious packets we should download the updated Snort rules (www.snort.org). Then, we unzip them in snort’s directory and run the command (supposing the newdat3.log file is in snort’s directory):

snort -r newdat3.log -l ./log

After executing Snort, we can find several interesting directories and files at “./log”. One of them is “alert” which contains a list of suspicious packets that may represent breaking danger, including postcans and exploits.

To dump the conections’ ACSII texts from newdat3.log it is necessary to use this snort.conf and execute the same command just mentioned. You may verify in “/log/” the text dump of all conections sorted by IP. For instance in the file “SESSION_1243-21” inside the 207.35.251.172 directory we can find all ASCII text of the conection between 207.35.251.172 (port 1243) and 192.168.1.102 (port 21 – FTP).

Ethereal:

With Ethereal you can do a complete analyzis of the packets. You may filter the archive (for example looking up at the connections made on port 23 from IP 217.156.93.166), as well isolate all the packets of a connection, dumping its ASCII text (if it is a binary archive download, you can even recover it), using the function Follow TCP Stream. To make read efortless, we may select a different color for each kind of packet. You may find this and many other features in Ethereal.

In order to make an easier analysis of the logs it may be found interesting to list all the TCP valid connections in the newdat3.log. Just open the archive and add the following filter:

tcp.flags.syn == 1 and tcp.flags.ack == 1

This way, Etheral will list up all the TCP packets in which the bits syn and ack are set, meaning to show the second package of each TCP connection (the server accepted connection).

Right clicking on one packet and chosing the Follow TCP Stream option will make Ethereal list only the packets of the same connection, farther an dump of all ASCII texts of that connection.

Pay attention that some of these connections are only postcans that connect to the port and then simply disconnect, and not the hacker trying to make a login.

An overview of the atack

To make the intrusion clearer, the following table represents the hacker main steps and the IP’s he used to connect to the system This table was constructed joining data from newdat3.log and slog2.log files. A weird fact is that at 4:03 we can see on slog2.log two telnet connections terminating, but we can't see it's corresponding entries on newdat3.log file. One possible explanation for this is that the honeynet guys for some unknow reason needed to filter these connections; they did it on newdat3.log but forgot to do it on slog2.log (ins't this strange?).

And last but not the least, come the answers!

1. Which vulnerability did the intruder exploit?

As it may be seen on the table above, at 00:06:05 the honeypot is exploited by the “rpc.statd Remote Format String” exploit (for further information’s access www.whitehats.com/info/IDS442) which was originated from IP 210.114.220.46. This exploit ran a shell code on this machine. We can't tell surely whether the exploit succeeded or not, but probably it didn’t work, as we shall see later (perhaps the statd version wasn’t vulnerable to the exploit).

Later on, at 20:52:52, the intruder tries to login, unsuccessfully, on telnet with user nobody, using the following passwords “virus” and “ultravirus”. He may have executed exploit in statd (by 00:06), and returned to install trojans in the honeypot. A minute later, he connects to the honeypot’s FTP and places another exploit, the “Wu-Ftpd Remote Format String Stack Overwrite” (for further information, check up http://www.whitehats.com/info/IDS453), this time successfully. From then on, he can run any command as root through the ftp server.

2. What ways, and in what order, did the intruder use to connect and run commands on the system?

At first FTP, then telnet and finaly ssh (24 port). Checking the table we can see the ways the intruder compromised the system. After the intruder managed to execute commands as root using honeypot’s ftp server (at 20:55:45), he removed user’s nobody password. Some seconds later, he login through telnet (now the intruder is keeping two open connections: a telnet’s and a ftp’s), verifies that he is the only user in the system (command “w”) and disconnect. Then, he turns to type commands on FTP session, creating a dns user (without password), with the same root's permissions (uid=0).

Now the hacker turns up to connect through telnet, logs in as super user, makes downloads of the rootkits and installs them in the system. Using the rootkit ZerO, our intruder makes up a ssh server, listening on port 24. Finally, at 21:51:58, he logs in into ssh shell and installs a rootkit copy.

3. How did the intruder try to hide his edits from the MAC times?

Before answering this question, we shall have a brief introduction of what MAC times are and how a hacker uses it.

MAC times are dates and times of Modification, Acess and Creation that every archive has. (When a file is modified, its modification date is updated with the system’s date at the moment; when we run “Is-I”, the dates we see in front of the files and directories are the dates of the last modification.)

These file dates can be modified by a hacker (using the touch command) in order to change files MAC times, thus the admin don’t notice the invasion. The hacker records the date of the last change of the file he’s gonna modifie and later restores the original file data as if nothing had happened.

In our challenge, the intruder tries to do exactly this, unsuccessfully (probably he should be simply copying and pasting commands without knowing for sure what the hell he was doing): In the ftp section he'd ran the exploit (20:55 till 21:56), he executed the command "passwd nobody -d" and remembered he was supposed to have saved the modification date of the passwd file before changing it, so the administrator wouldn’t notice the modifications. He also ran the following commands (in fact he copied and pasted, ‘cause it would be impossible for him to type them in less than a second):

mkdir -p /etc/X11/applnk/Internet/.etc

mkdir -p /etc/X11/applnk/Internet/.etcpasswd

touch -acmr /etc/passwd /etc/X11/applnk/Internet/.etcpasswd

touch -acmr /etc /etc/X11/applnk/Internet/.etc

passwd nobody -d

/usr/sbin/adduser dns -d/bin -u 0 -g 0 -s/bin/bash

passwd dns -d

touch -acmr /etc/X11/applnk/Internet/.etcpasswd /etc/passwd

touch -acmr /etc/X11/applnk/Internet/.etc /etc

As the hacker copied and pasted the commands, the system replied to all of them in one time:

Changing password for user nobody

Removing password for user nobody

passwd: Success

Changing password for user dns

Removing password for user dns

passwd: Success

By this way, the intruder saved the modified date of file /etc/passwd and of dir /etc in temporary files, removed user nobody’s password and added user dns (no password). Later he restored the original modification date of /etc and /etc/passwd.

Our intruder didn’t manage to hide his steps while changing system passwords for two reasons:

• He modified the password of the user nobody (passwd nobody -d) before saving the /etc/passwd and /etc MAC times, so when he tried to fix his mistake, playing commands in order to save MAC times, it was too late, for the files dates were already changed.

• He forgot to backup the MAC times of other files (shadow, passwd-, shadow-). He finds this out after listing the directories / and /etc/ with the comand “ls -lt” (which lists files by modification order):

ls -lt /

total 66

drwxr-xr-x 2 root root 2048 Sep 16 04:40 bin

drwxr-xr-x 29 root root 3072 Sep 16 04:31 etc

drwxrwxrwt 3 root root 1024 Sep 16 04:26 tmp

drwxr-x--- 2 root root 1024 Sep 12 18:01 root

drwxr-xr-x 6 root root 34816 Sep 12 17:53 dev

drwxr-xr-x 2 root root 1024 Sep 12 17:52 boot

drwxr-xr-x 5 root root 1024 Sep 12 14:12 home

drwxr-xr-x 3 root root 3072 Sep 12 14:10 sbin

drwxr-xr-x 19 root root 1024 Sep 12 14:10 var

drwxr-xr-x 4 root root 3072 Sep 12 14:02 lib

drwxr-xr-x 20 root root 1024 Sep 12 13:53 usr

dr-xr-xr-x 41 root root 0 Sep 12 13:51 proc

drwxr-xr-x 4 root root 1024 Sep 12 13:35 mnt

drwxr-xr-x 2 root root 12288 Sep 12 13:34 lost+found

drwxr-xr-x 2 root root 1024 Aug 23 1999 opt

ls -lt /etc

total 1088

-r-------- 1 root root 678 Sep 16 04:40 shadow

-rw------- 1 root root 5 Sep 16 04:40 group.lock

-rw------- 1 root root 5 Sep 16 04:40 gshadow.lock

-rw-r--r-- 1 root root 728 Sep 16 04:40 passwd-

-r-------- 1 root root 667 Sep 16 04:40 shadow-

-rw-r--r-- 1 root root 728 Sep 16 04:31 passwd

-rw-r--r-- 1 root root 64 Sep 12 17:53 issue

-rw-r--r-- 1 root root 63 Sep 12 17:53 issue.net

-rw-r--r-- 1 root root 20480 Sep 12 17:52 aliases.db

-rw------- 1 root root 60 Sep 12 17:52 ioctl.save

-rw-r--r-- 1 root root 90 Sep 12 17:52 mtab

-rw-r--r-- 1 root root 4 Sep 12 17:51 HOSTNAME

-rw-r--r-- 1 root root 1018 Sep 12 16:54 syslog.conf

(...)

Some of the modified files and directories (in which the intruder forgot to backup MAC times) have the modified date 04:40 and others 04:31 (in which he made the backup wrongly).

Remark: Note the date 04:40 of the compromised system is the 21:21 of snort’s log. We must consider this conversion for all the analysis made in this file.

4. The intruder downloaded rootkits, what were they called? Are they new/custom rootkits?

The intruder, through a telnet session with honeypot, connects to the ftp site teleport.go.ro (at 21:41), as user “teleport” (password: gunoierul) and does the following rootkits downloads:”Zer0.tar.gz”, “copy.tar.gz” and “ooty.tar.gz”. After several analysis made in the rootkits, we conclude that the rootkit Zer0 is the modified version of “t0rnkit” made by Viruzzel, a Romenian cracker. The two other files, copy.tar.gz e ooty.tar.gz, are remote and local exploit collections respectively. Our intruder may be Viruzzel himself, as well as may have copied and used his rootkits (to my mind, almost certainly the second one).

5. Recover (tell how you did it too) the rootkits from the snort binary capture

To recover any file (including binaries) from a connection, you may use Ethereal’s Follow TCP Stream.

The intruder did three downloads from FTP site teleport.go.ro, and therefore you only need to look for three connections between the honeypot and 193.231.236.42 on port 20 (ftp-data). For each of them, we choose any packet of connection, right click on it and choose the option Follow TCP Stream. Ethereal will show in a window the whole file and, then, you need to save it with the correct extension: “.tar.gz” (we may see the original file name if we check the ASCII text between honeypot and the FTP site on port 21). The recovered files are: ”Zer0.tar.gz”, “copy.tar.gz” e “ooty.tar.gz”.After this you may unzip them in order to see if the recover was successful.

6. What does the rootkit do to hide the presence of the attacker on the system?

After recovering and unzipping the Zer0.tar.gz file, we may analyze the Go script and find out what the rootkit did to hide the hacker’s presence in honeypot.

Firstly the script turns the syslog off so nothing can be recorded. After doing this, it checks if there is any remote logs server and tells the hacker that the machine is logging to the 000.000.00.000 IP (the real IP doesn’t appear ‘cause the guys from honeynet.org hided, for clear reasons, their syslog server IP):

checking for remote logging...

holy guacamole batman

REMOTE LOGGING DETECTED

I hope you can get to these other computer(s): 000.000.00.000

cuz this computer is LOGGING to it...

From then on, our intruder knows he won’t be able to hide the invasion in case he doesn’t manage to compromise the syslog server as well (in this case this is pretty impossible ‘cause guys from honeynet made it VERY safe! ;-). Now the script run some commands to hide several directories:

./ava h /usr/X11R6/bin/.,

./ava h /usr/info/.t0rn

./ava h /dev/rd/sdc0

./ava h /dev/rd/nscd.init

./ava h /etc/rc.d/rc3.d/S50inet

./ava h /usr/X11R6/lib/X11/.~

After this, it cleans all the systems logs (/var/log/) lines that contain words such as login, ftp and dns commanding:

./vrssb login

* sauber by socked [13.03.2k+1]*

*Cleaning logs.. This may take a bit depending on the size of the logs.

Cleaning boot.log (0 lines)... lines removed!

Cleaning boot.log.1 (133 lines)... lines removed!

Cleaning cron (8 lines)... lines removed!

Cleaning cron.1 (599 lines)... lines removed!

Cleaning dmesg (70 lines)... lines removed!

Cleaning htmlaccess.log (0 lines)... lines removed!

Cleaning maillog (0 lines)... lines removed!

Cleaning maillog.1 (24 lines)... lines removed!

Cleaning messages (0 lines)... lines removed!

Cleaning messages.1 (383 lines)... lines removed!

Cleaning netconf.log (0 lines)... lines removed!

Cleaning secure (0 lines)... lines removed!

Cleaning secure.1 (52 lines)... lines removed!

Cleaning sendmail.st (0 lines)... lines removed!

Cleaning spooler (0 lines)... lines removed!

Cleaning spooler.1 (0 lines)... lines removed!

Cleaning xferlog (0 lines)... lines removed!

Cleaning xferlog.1 (0 lines)... lines removed!

syslogd: no process killed * Alles sauber mein Meister !'Q%&@

./vrssb ftp

* sauber by socked [13.03.2k+1]*

*Cleaning logs.. This may take a bit depending on the size of the logs.

Cleaning boot.log (0 lines)... lines removed!

Cleaning boot.log.1 (133 lines)... lines removed!

Cleaning cron (8 lines)... lines removed!

Cleaning cron.1 (599 lines)... lines removed!

Cleaning dmesg (70 lines)... lines removed!

Cleaning htmlaccess.log (0 lines)... lines removed!

Cleaning maillog (0 lines)... lines removed!

Cleaning maillog.1 (24 lines)... lines removed!

Cleaning messages (0 lines)... lines removed!

Cleaning messages.1 (377 lines)... lines removed!

Cleaning netconf.log (0 lines)... lines removed!

Cleaning secure (0 lines)... lines removed!

Cleaning secure.1 (42 lines)... lines removed!

Cleaning sendmail.st (1 lines)... lines removed!

Cleaning spooler (0 lines)... lines removed!

Cleaning spooler.1 (0 lines)... lines removed!

Cleaning xferlog (0 lines)... lines removed!

Cleaning xferlog.1 (0 lines)... lines removed!

syslogd: no process killed * Alles sauber mein Meister !'Q%&@

./vrssb dns

* sauber by socked [13.03.2k+1]*

*Cleaning logs.. This may take a bit depending on the size of the logs.

Cleaning boot.log (0 lines)... lines removed!

Cleaning boot.log.1 (133 lines)... lines removed!

Cleaning cron (8 lines)... lines removed!

Cleaning cron.1 (599 lines)... lines removed!

Cleaning dmesg (70 lines)... lines removed!

Cleaning htmlaccess.log (0 lines)... lines removed!

Cleaning maillog (0 lines)... lines removed!

Cleaning maillog.1 (24 lines)... lines removed!

Cleaning messages (0 lines)... lines removed!

Cleaning messages.1 (376 lines)... lines removed!

Cleaning netconf.log (0 lines)... lines removed!

Cleaning secure (0 lines)... lines removed!

Cleaning secure.1 (16 lines)... lines removed!

Cleaning sendmail.st (1 lines)... lines removed!

Cleaning spooler (0 lines)... lines removed!

Cleaning spooler.1 (0 lines)... lines removed!

Cleaning xferlog (0 lines)... lines removed!

Cleaning xferlog.1 (0 lines)... lines removed!

syslogd: no process killed * Alles sauber mein Meister !'Q%&@

Next he deletes the bash history (file where you can find the history of all typed commands) and transform the file into a link to /dev/null (disabling the history):

rm -rf /bin/.bash_history

ln -s /dev/null /bin/.bash_history

Finaly he recovers the MAC times of several directories:

touch -acmr /tmp/.dir1 /bin

touch -acmr /tmp/.dir2 /usr/X11R6/bin

touch -acmr /tmp/.dir3 /etc/rc.d/rc3.d

touch -acmr /root /etc

rm -rf /tmp/.dir*

7. What did you learn from this exercise?

Most times hackers use a system they broke previously as a launch pad (they run portscans, exploits and break into others systems through the launch pad) so when an admin discover his system hacked, he will never identify the real hacker, only the launch pad. These launch pads are usually under responsibility of administrators that don’t check their systems logs periodically and don’t care about installing security updates. Therefore it is a good act to send an e-mail telling those admins about the problem they've got.

While breaking systems down, sometimes, a hacker may forget to hide several traces and the administrator may use this to find out if the system was broken or not (in the case of this intruder the administrator could have found the breaking out listing the directory /etc). As things don’t occur always as this, administrators should install a very good log system so a breaking doesn’t occur without any noticing. For example, had our intruder hidden the changes in the right way, there were no remote syslog nor Snort logging everything that was going on, it would have been more difficult to discover the invasion.

8. How long did this challenge take you?

It took me some 3 hours searching/analysing and more 5 hours building up the answer and translating it to English.

Bonus Question: Based on this challenge, write an example letter of notification to the source owner that attacked the system. Include any evidence or logs that you feel important.

Searching the whois databases you can find information about the owners of the IP’s that broke in the honeypot system:

• IP 207.35.251.172 (executed exploit and had unauthorized access to honeypot):

B-Line Technical Services (NETBLK-B-LINE-CA)

800 Rene Levesque, Flr.3

Montreal, Quebec H3B 1X9

CA

Netname: B-LINE-CA

Netblock: 207.35.251.160 – 207.35.251.191

Coordinator:

Daoust, Philippe (PD135-ARIN) noc@in.bell.ca

1-800-450-7771 +1 (416) 215-5423

• IP 210.114.220.46 (executed statd exploit):

IP Address : 210.114.220.0-210.114.221.255

Network Name : YEOMYONGCABLEBROAD

Connect ISP Name : SHINBIRO

Connect Date : 20000131

Registration Date : 20000714

[ Organization Information ]

Orgnization ID : ORG129891

Org Name : Yeomyong Cable Broadcasting

State : PUSAN

Address : 140-94 Myongjang-dong Tongrae-ku

Zip Code : 607-110

[ Admin Contact Information]

Name : Bookeun Lee

Org Name : Yeomyong Cable Broadcasting

State : PUSAN

Address : 140-94 Myongjang-dong Tongrae-ku

Zip Code : 607-110

Phone : +82-51-523-2345

Fax : +82-51-532-6464

E-Mail : ip@mgate.shinbiro.com

[ Technical Contact Information ]

Name : Bookeun Lee

Org Name : Yeomyong Cable Broadcasting

State : PUSAN

Address : 140-94 Myongjang-dong Tongrae-ku

Zip Code : 607-110

Phone : +82-51-523-2345

Fax : +82-51-532-6464

E-Mail : ip@mgate.shinbiro.com

• IP 217.156.93.166 (obtained unauthorized access to honeypot):

inetnum: 217.156.93.0 - 217.156.93.255

netname: MIDO-IMPEX

descr: S.C. MIDO IMPEX S.R.L.

descr: 9 C 5 MARASESTI STREET, 5500, BACAU, ROMANIA

country: RO

admin-c: RD5445-RIPE

tech-c: RD5445-RIPE

status: ASSIGNED PA

mnt-by: AS3233-MNT

notify: domain-admin@listserv.rnc.ro

changed: cristih@rnc.ro 20010515

source: RIPE

person: ROMULUS DOGARU

address: 9 C 5 MARASESTI ST

address: 5500, BACAU

address: ROMANIA

phone: +40-34-171060

fax-no: +40-34-171080

e-mail: romi@mido.ro

nic-hdl: RD5445-RIPE

notify: domain-admin@listserv.rnc.ro

mnt-by: AS3233-MNT

changed: cristih@rnc.ro 20001124

source: RIPE

All IP’s seem to be of firms which are being used as launching pads to other breaking (what makes their identity secret). We can send an e-mail to each of them, with several honeypots logs, warning them that one of their IP’s is practicing ilicity acts. Next you may see an example of notification letter to the responsible of 207.35.251.172 IP, Philippe Daoust noc@in.bell.ca):

Dear mr. Philippe Daoust,

Last 16/09/2001, at 20:55, our system had an unauthorized acess originated from IP 207.35.251.172 (which is under your responsibility) as you may check in the coming logs:

Snort’s log:

[**] [1:338:1] FTP EXPLOIT format string [**]
[Classification: Attempted User Privilege Gain]
[Priority: 8]
09/16-20:55:52.235847 207.35.251.172:2243 ->
192.168.1.102:21
TCP TTL:48 TOS:0x0 ID:16648 IpLen:20 DgmLen:76 DF
***AP*** Seq: 0xCF7869CC Ack: 0xEBCD7EC0 Win: 0x7D78
TcpLen: 32
TCP Options (3) => NOP NOP TS: 237391678 29673183
[Xref => http://www.whitehats.com/info/IDS453]


Syslog:

(20:55:52) ftpd[5609]: ANONYMOUS FTP LOGIN FROM 207.35.251.172 [207.35.251.172], mozilla@

Later on, the computer which has this IP commanded several actions which compromised the integrity of our system. I believe this computer is being used by a hacker as a launch pad to break other systems. I’m writing in order to ask for the necessary providences in order that the mencioned computer is checked and recovered so it won’t harm other systems connected to Internet.

For further information related to the breaking, please contact me.

Sincerely,
John Doe – net administrador

Acknowledgements

I’d like to thank my dear friend Tatiana Al-Chueyr Pareira Martins (Tati), without whom I wouldn’t have managed to translate this stuff.

References

Ethereal (www.ethereal.com)

Snort (www.snort.org)

Whitehats (www.whitehats.com)

SecurityFocus (www.securityfocus.com)

The SANS Institute (www.sans.org)