Scan of the Month #28
Executive Summary
On November 29th 2002, the honeypot system zoberius.example.net.mx was
compromised using a well known exploit for which patches exist. After gaining
administrative access to the system, the attacker installed a rootkit to hide
his presence on the system and also installed a set of tools typical of a
blackhat IRC operator. This is all fairly typical activity, with little new
information to be discovered.
However, on December 1st 2002, the attacker configured IPv6 on the system and
proceeded to connect to an IPv6 IRC network. This network appears to be
legitimate. While the activity after that point is not out of the ordinary,
the enabling and subsequent use of IPv6 is (to this point) atypical and
indicates that the blackhats have added a new tool to their arsenal.
Analysis
This analysis was conducted on a Linux system (Redhat 7.2) with the
following tools:
Ethereal - A graphical front end for tcpdump. Ideal for sifting through large
packet dumps quickly.
tcpdump - A command line program for capturing packets and manipulating
packet dumps.
snort - The definitive freeware network IDS
ngrep - A libpcap based pattern matcher - grep for network traffic.
file - A program that identifies file type
strings - A program for extracting printable strings from a file. Very
useful in working with binaries for which the source is not
readily available.
vi - A basic text editor. All around useful, but the binary editing
mode can be helpful in stripping ASCII protocol headers off of
binary files.
Babelfish - A language translation service for those of us who don't speak
all languages.
md5sum - A program that generates MD5 checksums. Useful for checking the
integrity of files.
The data for this month's scan comes in two log files day1.log.gz and
day3.log.gz. The first order of business after downloading the files is
to check the MD5 checksums. One has to be careful with honeypot operators:
bash-2.05$ md5sum day1.log.gz day3.log.gz
79e5871791542c8f38dd9cee2b2bc317 day1.log.gz
af8ab95f41530fe3561b506b422ed636 day3.log.gz
The files are tcpdump packet captures which have been compressed. Given the
format of the logs, the best starting point for the analysis will most
likely be Ethereal as it has good protocol decoding capabilities and enables
you to look through packet captures very quickly. Before working with the
files, I moved them to a safe location and created working copies. True, you
can always download another pristine copy of either of the files, but why be
sloppy?
bash-2.05$ mkdir pristine
bash-2.05$ mv *.log.gz pristine
bash-2.05$ cp pristine/* .
Having moved the source files to a safe location, the analysis now begins:
bash-2.05$ ethereal day1.log.gz #Ethereal can read the compressed versions directly
Pass 1
Day 1 (November 29th, 2002)
A quick scan of the packets in day1.log reveals a lot of the usual network
stuff (PINGs, name lookups, occasional scans). After a bit the activity
picks up and we see the first hints of our intruder. In particular, the
hosts 61.219.90.180 and 62.211.66.16 jump out due to the amount of traffic
early on. Since most of the traffic to 62.211.66.16 is FTP related, the
deed is most likely done, so we'll focus on 61.219.90.180 first. Setting the
display filter in Ethereal hides the other packets:
(ip.addr eq 61.219.90.180)
Right away some interesting things pop out. In the info column, the TCP
destination port 6112 is the home of a well known vulnerability. My first
thought was rpc.statd, but a quick search reveals it's typically dtspcd
(although bnetd also lives there and it's not a priviledged port, so it
could potentially be anything).
There are actually several connections from the attack host to the honeypot.
Using Ethereal's Follow TCP Stream feature I was able to look at each stream
individually to understand the sequence of events.
There's an initial connection which is immediately closed - recon. Note
that there was no effort to avoid any intrusion detection systems; it was
a normal TCP connection. Next, there's an immediate connection to ingreslock
(TCP port 1524)- more recon. This appears to be the exploit trying to
determine if the system has already been exploited. The next connection
appears to be a normal dtspcd protocol request (perhaps to confirm that
it's actually dtspcd and not something else listening on that port) - still
more recon. Having confirmed that it is dtspcd, the exploit is launched and
access is gained. For a great deal more detail, see the Scan of the Month #20.
At this point, the attacker takes over the machine.
Day 3
With a general idea of how things looked on day 1, I took a look at day 3.
Day 3 has lots of stuff going on. My first reaction on reading the
questions was that the attacker had most likely turned on IPv6 (why else
specify IPv4 in the question). A quick perusal of the file with ethereal
confirmed that. Unfortunately, the attacker did nothing more interesting
than start an IRC bot over the connection. However, there's a lot more
going on.
Pass 2
At this point, I had a a pretty rough idea of the outline of the attack.
However, there was a lot of fine detail missing. To help fill in the blanks
a little bit and to help me deal with the large amount of data, I broke
out snort to take a look at the log files:
bash-2.05$ snort -A full -c /etc/snort/snort.conf -l . -N -r day1.log
bash-2.05$ mv alert day1.snort.def
bash-2.05$ snort -A full -c /etc/snort/snort.conf -l . -N -r day3.log
bash-2.05$ mv alert day3.snort.def
I used a default installation of snort 1.9.1 (setting HOME_NET to be the
local network of the honeypot) to see what it would catch without any
tuning. For this project, ACID seemed like overkill, so I used the
universal text database query tool:
bash-2.05$ grep "\[\*\*\]" alert | sort | uniq -c
This pulls out the alert description lines, sorts them, and provides a
count of unique lines. It's not going to win any awards for analysis
tool of the year, but it will get the job done.
Surprisingly, snort did not catch the dtspcd exploit itself. However, it
did identify the SPARC NOOP slide, and identified the ICMP packets with
skillz in the payload as Stacheldraht packets (day 1 alerts, day 3 alerts).
A quick check of the configuration file revealed that a number of rulesets
are turned off by default. Those rulesets appear to be potentially "noisy"
rulesets which may generate large numbers of false positives. After
turning those on, we get some different output (day 1 alerts, day 3 alerts).
On day1, snort reveals very little that we didn't know about already.
Day 3, gets a little more interesting with reports of port scans and DDoS
SYN floods and all sorts of ICMP weirdness. It still doesn't specifically
catch the dtspcd exploit, but it at least points out an apparent DDoS
attack and gives a few useful leads.
Questions
1. What is the operating system of the honeypot? How did you
determine that? (see day1)
The honeypot is running Solaris 8 (SPARC, 64 bit). The primary
source of this information was the kernel revision in the uname
output from the rootkit installation session (Generic_108528-09).
This is corroborated by the following:
The exploit was Solaris SPARC dtspcd exploit code.
The ethernet MAC vendor ID is Sun.
The rootkit was targeted at Solaris and the binaries are SPARC.
The hardware is SPARC. In fact, from the syslogged reboot
messages (day 3: packet #117184), the hardware is:
A 360MHz Sun Ultra (UltraSPARC-IIi) with 128 MB RAM,
8.4GB Ultra ATA hard drive (Seagate ST38410A), and a
10/100 Ethernet interface running at 10Mbps (most likely
an Ultra 5). It also appears to be running BIND 8.2.2-P5.
2. How did the attacker(s) break into the system? (see day1)
The attacker gained administrative access to the system using the
an exploit for the dtspcd vulnerability. This attack (at least the
exploit phase) was similar to Scan of the Month #20.
3. Which systems were used in this attack, and how?(see day1)
61.219.90.180 - This is the host that exploited the dtspcd
bug and installed the back door
62.211.66.16 - Serves tools (FTP) - wget, dlp, solbnc, ipv6sun
62.211.66.53 - Serves tools (HTTP) - sol.tar.gz (rootkit)
61.134.3.11 - Stacheldraht handler
80.117.14.222 - IRC Node (likely a DHCP assigned address for one machine)
80.117.14.44 - IRC Node (likely a DHCP assigned address for one machine)
206.252.192.195 - IRC Node (irc-1.stealth.net)
163.162.170.173 - IPv4 address of tunnel to irc6 peer node
192.168.100.28 - The honeypot itself - used as an IRC bouncer,
port scanner, and DDoS agent.
62.101.108.086 - Connections to the rootkit SSH port from
this host
4. Create a diagram that demonstrates the sequences involved
in the attack. (see day1)
The following diagram is an illustration of the events
on day 1 (the actual exploit) and is more concise than
the full time line.
Reconnaisance
+---------------+ +----------------+
| Attacker | Probe for dtspcd--->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<----------Listening +----------------+
+---------------+ +----------------+
| Attacker | Check ingreslock--->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<------Not listening +----------------+
+---------------+ +----------------+
| Attacker | Verify it's dtspcd->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<--------It's dtspcd +----------------+
Exploit
+---------------+ +----------------+
| Attacker | Arbitrary code----->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<--------Shell(1524) +----------------+
+---------------+ +----------------+
| Attacker | Check ingreslock--->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<---Here's the shell +----------------+
you ordered
+---------------+ +----------------+
| Attacker | Commands----------->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<--------Shell(1524) +----------------+
Take Ownership
+---------------+ +----------------+
| Attacker |<-----Get Tools(FTP) | Honeypot |
| 62.211.66.16 |---------------------| 192.168.100.20 |
+---------------+ All yours---------->+----------------+
+---------------+ +----------------+
| Attacker |<----Get Tools(HTTP) | Honeypot |
| 62.211.66.53 |---------------------| 192.168.100.20 |
+---------------+ Rootkit------------>+----------------+
+---------------+ +----------------+
| Attacker | Install rootkit---->| Honeypot |
| 62.219.90.180 |---------------------| 192.168.100.20 |
+---------------+<--------Shell(1524) +----------------+
Operations
IRC & DDoS
5. What is the purpose/reason of the ICMP packets with 'skillz' in
them? (see day1)
The ICMP packets are from a Stacheldraht DDoS agent looking for
its handler. See Dave Dittrich's analysis of Stacheldraht for
details.
The proper response to skillz is ficken (if you're a Stacheldraht
handler). We begin to see these responses on day 3 from
61.134.3.11. 217.116.38.10 does not respond as far as we can tell
from these log files.
This is appears to be a variant. At the very least the ID field in
the ICMP packets is different. In the original Stacheldraht, the ID
field from the agent was 666 and the ID field in the handler response
was 667. In our case, the ID field from the client is 0x1a0a. The
handler still increments the client supplied ID by one to 0x1a1b.
The file solsch from the installed rootkit contains the Stacheldraht
agent as does the file solbnc.
6. Following the attack, the attacker(s) enabled a unique protocol that
one would not expect to find on an IPv4 network. Can you identify
that protocol and why it was used? (see day3)
On Day 3, the attacker enabled IPv6 - the next generation of the IP
protocol. Although IPv6 is supported at some level by most routers,
since native IPv6 support is not yet a MUST in the broader internet
it is typically tunneled through IPv4 to ensure routability. In this
case, the tunnel enabled the attacker to connect to an Italian ISP's
IPv6 IRC network: irc6.edisontel.it
7. Can you identify the nationality of the attacker? (see day3)
It appears that the attacker is Italian. Most of the IRC connections
originate from machines in Italy, and the IPv6 irc6 net is
hosted there. That in and of itself doesn't mean much, however,
information gleaned from the rootkit indicates it is of Italian
origin (or at least modification). In addition, the dialog in the
IRC chat sessions decodes best with Italian -> English translation
(via Babelfish).
Bonus Question:
* What are the implications of using the unusual IP protocol to the
Intrusion Detection industry?
Currently, IPv6 is typically tunneled through IPv4 if no native
IPv6 connectvity exists. It is possible that attacks could be
obfuscated if IDS systems don't decode the protocol, or decode it
incorrectly.
IPv6 also uses many concepts that are different from the well
understood IPv4 protocol and as such may be confusing to people
expecting to see IPv4 conventions. Furthermore, IPv6 protocols may
be exploitable in entirely new ways - the ID community needs to
consider what those might be and create new tools or adapt old
tools to the task of detecting IPv6 specific exploits.
* What tools exist that can decode this protocol?
Ethereal can decode this protocol as can tcpdump and most of the
rest of the libpcap based packages. There also appear to be
several commercial tools that purport to decode IPv6.
Conclusion
This month's scan was a fairly typical example of an intrusion. It starts
off with the typical search/exploit/trojan cycle and then follows the
pattern of an IRC warrior. On day 3, they started an IPv6 tunnel. While
this is certainly a new twist, the activities that take place through the
tunnel remain typical of a machine that has been taken over for IRC use.
For this attack, there wasn't any significanlty new activity beyond the IPv6
tunnel. However, this does serve as notification that the blackhats are
paying attention, and that our packet analysis and Intrusion Detection tools
must now be ready to deal with IPv6 as well.
greetz to vlad and jesse for their proof reading skillz!!