Scan of the Month 28 - The analysis

The analysis

Starting it all

The challenge came with two files: day1.log.gz and day3.log.gz. I downloaded them both and made sure I got them right:

   $ md5 *
   MD5 (day1.log.gz) = 79e5871791542c8f38dd9cee2b2bc317
   MD5 (day3.log.gz) = af8ab95f41530fe3561b506b422ed636

Both files were correctly downloaded, so I could untar them with 'tar'.
To determine what kind of files these were I used the tool 'file':

   $ file *.log
   day1.log: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 1514)
   day3.log: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 1514)

Okay, so they're tcpdump files. Now I know what I got, let's get things started.

Question 1

Let me first tell you I didn't really use the questions as starting-point. Instead, I jumped into the files given and logged everything I saw. Soon I had some questions already answered and some ideas spawned in my mind for answering the others.

When looking at the first log with 'tcpdump -r day1.log -n', I found a lot of DNS-queries coming from and going to the honeypot. This led me to believe it was being used as DNS-server. I found out the honeypot was sending ICMP destination port unreachable packets to incoming UDP packets for port 137. Most Windows-based systems would have something listening (and answering) on port 137 UDP. This probably wasn't a Windows-based system.

After a while I saw a connection to port 6112 being made. I didn't know what port that was, so I looked into it with 'tcpdump -r day1.log -n -s1514 -xX port 6112'. During this connection the honeypot sent the string "zoberius:SunOS:5.8:sun4u" to the peer. This has got to be a SunOS box. But I digged deeper to get some more certainty.

In the same connection something which looked like shellcode was sent. It was padded with '801c 4011', which could be NOOPs (I knew 0x90 is a NOP on x86 systems, but on Sparc this would be different). Anyway, the shellcode contained commands for *nix systems, such as SunOS. I still had some things to find out. First: port 6112. Using google I found out that port 6112 is used for the Subprocess Control Server (dtspcd) which comes with CDE. CDE comes with some UNIXes, Solaris 8 being one of them. I soon found an alert for a buffer-overflow in dtspcd. It contained a reference to ARACHNIDS signature IDS353, "SHELLCODE sparc NOOP", which arose my interest since I had the hunch '801c 4011' could be a Sparc NOOP. Looking at the arachNIDS database at whitehats.com I found out that I was right all along, '801c 4011' is a Sparc NOOP. Going further in the logfile I saw a connection to a root-shell on port 1524 being made. The command 'uname -a' is run, which returns: "SunOS zoberius 5.8 Generic_108528-09 sun4u sparc SUNW,Ultra-5_10".

With all this information it's chrystal clear: the honeypot, called 'zoberius', is a Sun Ultra (presumably an U5 or U10) running SunOS 5.8.

Question 2

This question is almost answered already. The attacker used an exploit for a buffer-overflow in the Subprocess Control Server (dtspcd) to gain root privileges on the system. The advisory for this problem can be found here. The exploit runs the following commands after gaining root:

   /bin/ksh -c echo "ingreslock stream tcp nowait root /bin/sh sh -i">/tmp/x;/usr/sbin/inetd -s /tmp/x;sleep 10;/bin/rm -f /tmp/x

This means a one-line inetd configuration file is made in /tmp/x, inetd is run with that configuration file and after sleeping 10 seconds the file is deleted. Inetd would spawn a '/bin/sh' shell when someone'd connect to port 'ingreslock'.

I didn't know the port 'ingreslock' from my head, but '/etc/services' helps:

   $ grep ingreslock /etc/services 
   ingreslock      1524/tcp    #ingres
   ingreslock      1524/udp    #ingres

Now I knew the attacker would have a root-shell when connecting to port 1524.

Question 3

I already found the system running the exploit: 61.219.90.180, apparantly a machine in Taiwan. I knew there was a rootshell on port 1524. The attacker would most probably connect to that port soon after the break-in. 'tcpdump -r day1.log -n -s1514 -xX port 1524' gave me what I was looking for:

  17:36:25.573445 61.219.90.180.56709 > 192.168.100.28.1524: S 2149411790:2149411790(0) win 5840  (DF)
  17:36:25.573445 192.168.100.28.1524 > 61.219.90.180.56709: R 0:0(0) ack 2149411791 win 0 (DF)

The attacker (or his/her tool) was in a bit of a hurry, the rootshell wasn't bound on port 1524 yet. However, 12 seconds later the connection attempt succeeds, from the same address: 61.219.90.180.

During this session the attacker creates a directory '/usr/share/man/man1/.old'. Using 'ftp' the attacker downloads some files from 62.211.66.16, a system belonging to tin.it, an Italian ISP. The files 'wget', 'dlp', 'solbnc' and 'ipv6sun' are downloaded (manually, at first the attacker made a typo: 'iupv6sun' instead of 'ipv6sun'). I then got a hunch, perhaps question 6 meant IPv6, the next generation IP standard. 'ipv6sun' could contain some IPv6-related tools.

After setting the newly downloaded tools executable, the attacker downloads a file called 'sol.tar.gz' from 62.211.66.53. Again, a tin.it system. This file contains a rootkit, which the attacker installs after untarring with a simple './setup' (don't you like those easy-to-use rootkits?).

During the process of installing the rootkit the attacker is asked to enter two port numbers: one for a trojan SSH-daemon and one for a IRC-bouncer called PsyBNC. The trojan SSH-daemon gets port 5001, whereas PsyBNC is free to bind() on port 7000. On day 1 nothing connects to port 5001, but PsyBNC is used. Several connections from 80.117.14.44 are made. That address is part of a ADSL IP-pool of Telecom Italia. In my experience with honeypots crackers tend to connect from their home to freshly installed PsyBNCs on freshly cracked systems. I reckon this attacker is no exception, although there is of course always a possibility it's not his/her own system.

Question 4

Now I had a decent overview I could create a diagram showing what happened during the attack. Using 'tcpdump -r I filled the following table.

Time	From	To	What
17:36:25 - 17:36:37	61.219.90.180:56399	192.168.100.28:6112	Buffer-overflow in dtscpd exploited, root-shell bound to port 1524
17:36:37 - 18:00:00	61.219.90.180:56712	192.168.100.28:1524	Some tools and a rootkit downloaded from 2 tin.it systems. Rootkit installed, logs cleaned, some daemons patched
17:42:42 - 17:45:13	192.168.100.28:32783	62.211.66.16:21	'wget', 'dlp', 'solbnc' and 'ipv6sun' are downloaded
17:45:29 - 17:52:40	192.168.100.28:32789	62.211.66.53:80	'sol.tar.gz' downloaded
17:54:25 - 17:58:32	192.168.100.28	sunsolve.sun.com:21	Two downloads of official Sun patches for a couple of daemons
18:04:07 - 18:12:40	80.117.14.44:3934	192.168.100.28:7000	The attacker configuring the bouncer, letting it connect with irc.stealth.net (an IRCnet server in the US), letting it join #agropoli2
18:12:39 - 19:01:17	80.117.14.44:3935	192.168.100.28:7000	Sending invalid commands to the IRC-server, idling in #agropoli2 and #bobz

From 22:54:21 till 22:54:35 and from 04:17:02 till 04:17:18 the attacker connected a few times to the IRC-bouncer too. I didn't include this in the diagram because it wasn't really interesting, honostly.

Question 5

When I read this question the word 'stacheldraht' immediately popped up in my head, wavering a bit in pointy, painful ways before my eyes. This was because I've once taken apart the stacheldraht sources to see what it does, how badly it was coded and where it's weaknesses lay. No tools needed for this question.

The ICMP packets containing 'skillz' are sent once every 10 seconds by stacheldraht agents to so-called 'handlers'. By doing this the handlers are informed of the existence of the agents. When the handler receives the packet it updates it's list of agents, keeping it up-to-date.

Stacheldraht is a 3-tier DDoS tool, consisting of a client, which can connect over TCP to handlers, also called masterservers. Those connections are encrypted. An attacker (AKA DDoS-kiddie) can then send commands to stacheldraht like:

   stacheldraht(status: a!1 d!0)>.mack 192.168.100.29

The handler relays those commands to the agents, possibly a few thousand of them. The above command means 'mass ack', or in English: a massive TCP ACK flood. There are quite a few attack-possibilities: TCP SYN, TCP ACK, UDP, ICMP echo request, IP with random flags/data, TCP with random flags/data, etc.

Originally stacheldraht agents sent ICMP packets with an ICMP ID 666 and contents 'skillz' (padded with 0-bytes) to the handlers. This one is a bit different. It still uses 'skillz', but not ICMP ID 666, instead, it uses 6666. I found out by looking at tcpdump output of a stacheldraht analysis where the ID was still '666'. Then I looked at my own tcpdump output and converted the '1a 0a' I found there to decimal, giving me 6666. I grepped for '6666' in the agent's source of two different versions, both yielding the same result:

   /* contact the masterserver */
   send_connect(inet_addr(server),6666,"skillz");

I think this change might have been done by 'yps'; from a 'mserv.c' (the source for the handler):

   stacheldraht 1.666+antigl+yps by randomizer

<SNIP>

   changed ids in client, removed remote update,remote kill and detection
 
   those changes by yps

For a detailed analysis, see this article.

Question 6

Soon after I started looking at day3.log with tcpdump I found I needed another tool. No way I was going to browse through a DDoS-attack by holding my PgDn key minute after minute. I first didn't realize the traffic to 195.130.233.20 was a packetflood, the "ip 0 (DF) [tos 0x7,CE]" confused me a bit. I thought that might have been the protocol question 6 is asking for. But ethereal helped me out.

Sliding past the DDoS-attack I could see some activity on port 5001, the port the trojan SSH-daemon was listening on. Apparantly the attacker was doing something. Then I saw an "AUTH.CRIT" syslog being sent to the syslog server. According to syslogd, the attacker rebooted the system. That's pretty suspicious. Not much further I saw something more interesting: IPv6 traffic.

IPv6 traffic encapsulated in IPv4 traffic even. Encapsulation is done using protocol 41:

  $ grep 41 /etc/protocols 
  ipv6    41      IPV6            # ipv6

Ethereal shows this encapsulation very elegantly. It first shows the lowest layer: ethernet. Then comes IPv4. The 'Protocol'-field shows '0x29' here. Hexadecimal 0x29 is 41 in decimal. The next layer is IPv6. Ethereal nicely shows the source and destination addresses, along with the other header-fields.

The attacker has created an IPv6 tunnel to 163.162.170.173, a system at the Telecom Italia network. The attacker uses 2001:6b8:0:400::5d0e as IPv6 address. This address is part of 2001:6b8::/48, which is assigned to Telecom Italia for the 'ngnet initiative'. Ngnet allows people to create IPv6 tunnels to let them get some experience in usage and configuration of IPv6 itself and applications using IPv6. There are quite a few tunnelbrokers around. I've configured and use two tunnels myself. The attacker uses IPv6 for IRC (nice how ethereal shows this). The server (s)he's connecting to is irc6.edisontel.it, according to this page. This server is connected to IRCnet. Most crackers using IPv6 for IRC do so for reasons such as hiding their IPv4 address or creating 'cool' hostnames. Because there are so many IPv6 adresses and the DNS for reverse lookups are mostly in their own control, it's easier for crackers to have 'cool' hostnames for IPv6 addresses than for IPv4 addresses.

Question 7

From day1.log it was already pretty clear the attacker is Italian. He downloaded his/her tools from Italian systems. The IRC-client sent Italian info to the IRC-server. There was some Italian talk on the IRC-channels, albeit not much. Day 3 showed this much better. The following (sanitized) excerpt shows some Italian:

  <-PaKi-_> ancora con bacardi?
  <bobz`> ihhihi
  <-PaKi-_> uhuhuhuhuu
  <bobz`> nn sapevo ke mettere:D

Too bad I don't understand it, could've been funny.. Another excerpt I found with 'strings day3.log':

   <bobz> anni?
   <|AnDr34z|> 17 tu vez?
   <bobz> 15
   <|AnDr34z|> non ci credo
   <bobz> :D

Apparantly our cracker 'bobz' is 15 years old. Nice to know you're 'trusting' your Sun machines to 15-year-olds..

Bonus question 1

IPv6 is different from IPv4. IDS-tools written for IPv4 should be updated to be able to understand IPv6 packets. They also should be able to understand encapsulation of IPv6 packets inside IPv4 packets. For IPv6 the control-messages have changed too, ICMP6 isn't the same as IPv4 ICMP. Basically it shouldn't be that hard to implement the needed changes: when it comes to TCP, UDP and the application protocols, nothing has changed. It's very important for the Intrusion Detection industry to adapt their tools, since the cracker-community is already using IPv6, perhaps passing through IDS's undetectedly.

Bonus question 2

I used ethereal to read the logs of day 3. Ethereal can decode IPv6 packets, even when they're encapsulated in IPv4.

The other tool I used, tcpdump, can decode the packets too:

  08:18:53.918514 192.168.100.28 > 163.162.170.173: 
                  2001:6b8:0:400::5d0e.32780 > 2001:750:2:0:202:a5ff:fef0:aac7.6667: P 14464:14466(2) ack 103750 win 25560 (DF)

snort isn't officialy IPv6-capable yet, as far as I know. I downloaded the CVS snapshot of Thu May 8th, which didn't appear to be IPv6-capable. However, I heard there's been done some work on this.

Tools I used

I used the following tools for the analysis:

Tool	Description
md5	Creates a MD5 sum of a file. Used it to check if the logfiles were downloaded correctly.
tcpdump	A network sniffer. Used by the honeynet project to write logfiles, I used it to read them back again.
ethereal	A network protocol analyser, can sniff a live network or read captured data. I used it to be able to read day3.log a bit easier than with tcpdump.
strings	Shows readable ASCII-text from (binary) files. Used it to see the IRC traffic in day3.log a bit easier than with ethereal.
Phoenix	Lightweight browser, AKA Mozilla browser and Firebird. Used it to read up on dtscpd, ngnet.it and stacheldraht.
joe	Joe's Own Editor, a ASCII-text screen editor I used to create these HTML files.
whois	With whois I looked up the IP-addresses I came across during the analysis.

Index