Scan 28 - May 2003

Author: Diego González Gómez
dggomez@users.sourceforge.net

Table of contents:

Analysis:

First, I downloaded and checked the binary files:

$ wget http://www.honeynet.org/scans/scan28/day1.log.gz
$ wget http://www.honeynet.org/scans/scan28/day3.log.gz
$ echo '79e5871791542c8f38dd9cee2b2bc317  day1.log.gz' > day1.md5 ; md5sum -c day1.md5
day1.log.gz: OK
$ echo 'af8ab95f41530fe3561b506b422ed636  day3.log.gz' > day3.md5 ; md5sum -c day3.md5
day3.log.gz: OK

After that, I extracted the content of the files to begin the analysis process.

Day 1

The first file analyzed was day1.log file. This tcpdump log file is supposed to have the break in activity.

General Approach

I used tethereal tool to retrieve protocol statistics.

$ tethereal -nr day1.log -z "io,phs" > day1_stats.txt

Looking at the protocol tree, we can see that all the traffic in the log file, under the Ethernet protocol, is constructed using IP. The next protocols under IP tree are TCP, ICMP and UDP. There are not too many different protocols.

After examining the protocol information, I decided to get the IO-users activity. It summarizes the most active communications, and the IP addresses involved.

$ tethereal -nr day1.log -z "io,users,ip" >> day1_stats.txt

I used those results to start the next stage of the analysis.

Attack

I opened day1.log file with ethereal to analyze in detail the most popular IP addresses (the ones with the most activity). I began with the most popular IP addresses in the statistics file obtained before.

206.252.192.195
It seems to be an IRC server. It was used by the honeypot to open an IRC session on port 6667 at trace number 8405, and then on port 5555 at trace number 8564. This is interesting but it is not representative of an attack that I'm looking for. I will describe the IP activity later, at the end of this section.

61.219.90.180
This IP address makes its first appearance at trace number 561.
It sucessfully opened TCP 6112 port on the honeypot. After that, it tried to open TCP 1524 TCP (trace 564) but the port was closed.

It is necessary to explain that the TCP 6112 port corresponds to dtspcd, as described by IANA [1].

A small search reveals much information about security flaws related to this service. I found a known vulnerability of CDE, described by CERT®, which could apply to this situation [2]. The Common Desktop Environment (CDE) is an integrated graphical user interface that runs on UNIX and Linux operating systems. The CDE Sub process Control Service (dtspcd) is a network daemon that accepts requests from clients to execute commands and launch applications remotely. On systems running CDE, Internet services daemon (typically inetd or xinetd) spawn dtspcd in response to a CDE client request. dtspcd is typically configured to run on TCP 6112 port with root privileges.

On the other hand, the scan of TCP port 1524 port had a specific motive - it wasn't an arbitrary choice. Its well-known that that port is widely used by the script kiddie community to implement a backdoor for most automated attacks. This probably happens because the attackers just copy the new shell code for buffer overflows (shell code is very difficult to write), into their scripts [3]. Therefore, the remote IP address made an attempt to connect to that port to check if the honeypot was already compromised.

The next packets are very interesting. The following octets correspond to the trace number 571, a packet sent from the honeypot to the remote IP address. In the data field, we can observe the probable name of the operating system of the honeypot (SunOS: 5.8):

571 33016.113820 192.168.100.28 -> 61.219.90.180 TCP 6112 > 56710 [PSH, ACK] Seq=3124564266 Ack=2140233551 Win=24616 Len=70 TSV=113867452 TSER=48509986

(stripped)
0x0040 34 22 30 30 30 30 30 30 30 30 31 34 30 30 33 32 4"00000000140032
0x0050 30 30 30 31 20 20 33 20 00 2f 2f 2e 53 50 43 5f 0001 3 .//.SPC_
0x0060 41 41 41 56 54 61 71 44 64 00 31 30 30 30 00 7a AAAVTaqDd.1000.z
0x0070 6f 62 65 72 69 75 73 3a 53 75 6e 4f 53 3a 35 2e oberius:SunOS:5.
0x0080 38 3a 73 75 6e 34 75 00                         8:sun4u.       

Several packets later, we can observe the beginning of a buffer overflow attack, described by CERT®. The following information corresponds to trace number 580:

580 33016.563790 61.219.90.180 -> 192.168.100.28 TCP 56711 > 6112 [ACK] Seq=2143411080 Ack=3124882182 Win=5840 Len=1448 TSV=48510034 TSER=113867474

0x0000 08 00 20 d1 76 19 00 07 ec b2 d0 0a 08 00 45 00 .. .v.........E.
0x0010 05 dc ef bd 40 00 2c 06 10 ba 3d db 5a b4 c0 a8 ....@.,...=.Z...
0x0020 64 1c dd 87 17 e0 7f c1 db 88 ba 41 eb 06 80 10 d..........A....
0x0030 16 d0 61 5f 00 00 01 01 08 0a 02 e4 34 52 06 c9 ..a_........4R..
0x0040 7a d2 30 30 30 30 30 30 30 32 30 34 31 30 33 65 z.0000000204103e
 0x0050 30 30 30 33 20 20 34 20 00 00 00 31 30 00 80 1c 0003  4 ...10...
0x0060 40 11 80 1c 40 11 10 80 01 01 80 1c 40 11 80 1c @...@.......@...
0x0070 40 11 80 1c 40 11 80 1c 40 11 80 1c 40 11 80 1c @...@...@...@...
(stripped)

The server has a 4K (0x1000) internal buffer. The value 0x103e indicates the number of bytes in the packet to copy into the buffer. Since 0x103e is greater than 0x1000, the last 0x3e bytes of the packet will overwrite memory after the end of the 4K buffer.

Following Procedures

This is only the beginning of the compromise of the honeypot. As we can see, the buffer overflow attack was successful. Only 10 seconds later, at trace number 588, the attacker successfully opens a TCP session on port 1524 (the backdoor). I have recorded in a file the TCP stream of that session because it contains valuable information about the attacker's activity. I am going to describe and analyze the actions taken in that session.

The first commands launched by the attacker were:

# uname -a; ls -l /core /var/dt/tmp/DTSPCD.log; PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/ccs/bin:/usr/gnu/bin; export PATH; echo "BD PID(s): "`ps -fed|grep ' -s /tmp/x'|grep -v grep|awk '{print $2}'`

The answer of the honeypot to the 'uname -a' command was:

"SunOS zoberius 5.8 Generic_108528-09 sun4u sparc SUNW, Ultra-5_10"

So now, we have detailed data about the characteristics of the honeypot's platform. We can confirm which it's operating system is.

The next steps taken by the attacker in the TCP session do not need explanation, because they are very explicit. Anyway, the most important actions are summarized below:

  1. Unset HISTFILE and DISPLAY environmental variables.

  2. Create /usr/share/man/man1/.old directory, and change to that directory.

  3. Try to use wget tool, but it didn't exist in the honeypot. Therefore, he started an FTP session on 62.211.66.16 IP address, and used get command to retrieve wget, dlp, solbnc, and ipv6sun files. The username and password used were bobzz/joka.

  4. Use wget to obtain http://62.211.66.53:80/bobzz/sol.tar.gz file.

  5. Download and install ftp://sunsolve.sun.com:21/pub/patches/111085-02.zip, ftp://sunsolve.sun.com:21/pub/patches/108949-07.zip, and 111606-02.zip files. The IP address appeared in the log file for sunsolve.sun.com was 192.18.99.122. Those files are patches for Sun Solaris Systems. With this action, the attacker patched the vulnerabilities exploited by him, avoiding the possibility of another attacker to compromise the honeypot in the same way that he did.

  6. Configure and start psyBNC 2.2.1 (the BNC tool mentioned before) listening on port 7000, to load automatically on the next reboot. A 'bnc' is short for a 'bouncer'. A 'bnc' acts as a proxy for IRC, allowing the real IP address to be hidden and using a vanity host (like 'this.is.a.hp28t.vhost.com'). The attacker uses it later to open his IRC sessions. [4]

  7. Run solbnc tool. At that time, the honeypot begins to send ICMP packets with the word 'skillz' in the data filed (trace number 8332). I detail this behaviour in the next section (Suspects ICMP Packets).

  8. Run dlp tool deleting all /var/log files again.

Before analyzing in detail the suspicious ICMP packets, I wish to explain a large IRC session I found from IP address 80.117.14.44  on the recently opened TCP 7000 port (PsyBNC) of the honeypot. That session began at trace number 8362. It is assumed that this IP address belongs to the attacker. He opened a session on the installed BNC port after compromising the system.

During TCP port 7000 sessions, the attacker (address 80.117.14.44) used PsyBNC tool to open some IRC sessions to IP address 206.252.192.195. The TCP sessions extracted indicates that the attacker writes in Italian.

Suspects ICMP Packets

A closer look at the ICMP packets gives an opportunity to discover more detailed information. I used ethereal to filter ICMP traffic of day1.log file. There were two main types of ICMP packets, after the compromise. The first type was similar to the following packet:

8332 34422.398454 192.168.100.28 -> 217.116.38.10 ICMP Echo (ping) reply

0x0000 00 07 ec b2 d0 0a 08 00 20 d1 76 19 08 00 45 00 ........ .v...E.
0x0010 04 14 40 5b 40 00 ff 01 87 f9 c0 a8 64 1c d9 74 ..@[@.......d..t
0x0020 26 0a 00 00 9c a3 1a 0a 00 00 00 00 00 00 00 00 &...............
0x0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 73 6b ..............sk
0x0040 69 6c 6c 7a 00 00 00 00 00 00 00 00 00 00 00 00 illz............
0x0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
(stripped)

I made a search in the 'Snort Signature Database' [5] using that information, and I found that those ICMP packets were originated by a variant of 'Stacheldraht DDOS' tool. It is a program used to mount distributed denial of service attacks, normally found on Sun Solaris machines.

The honeypot was used as a DDoS agent. As described in the Stacheldraht DDOS document, written by David Dittrich [6]: "Once the agent has determined a list of potential handlers, it then starts at the beginning of the list of handlers and sends an ICMP_ECHOREPLY packet with an ID field containing the value 6666 (the original Stacheldraht DDOS tool uses 666) and data field containing the string "skillz". If the handler receives this packet, it sends back an ICMP_ECHOREPLY packet with an ID field containing the value 6667 (667 in the original DDoS tool) and data field containing the string "ficken"."

I did not observe those ICMP_ECHOREPLY packets in day1.log file. Nevertheless, I identified the potential handlers as addresses 61.134.3.11 and 217.116.38.10.

The second type of ICMP packets were very different than the first type. They did not have the word 'skillz', and they were a smaller size. Those packets were series of ICMP_ECHOREQUEST sent by different IP addresses, and replied to by the honeypot. They were identified by snort as 'Speedera ICMP PINGs': Usually benevolent pings used by SpeedEra.net to find closest cache for the host. They did not seem to be a menace.

Day 3

The second tcpdump log file analyzed was day3.log file. This day, the honeypot captured some unique protocol activity.

General Approach

The same tethereal commands used in the General Approach of Day 1 produced this statistical information.

From the first moment we can observe that the number of protocols appeared is greater than before. There are new protocols under IP tree such as radius, srvloc, slimp3, or dlsw. None of them has significant activity. However, it is necessary to emphasize that there is a protocol over layer two (Ethernet) that in day 1 was not used: IPv6.

IP version 6 (IPv6) is a new version of the Internet Protocol, designed as a successor to IP version 4 (IPv4) [7]. The changes from IPv4 to IPv6 fall primarily into the following categories:

Summarize important activity

Before analyzing the unusual protocol activity, it is important to describe the more interesting actions that occurred in day3.log file.

Unusual Protocol

At trace number 117614, the honeypot (2001:6b8:0:400::5d0e IPv6 address) was used to open an IRC session.  He opened a TCP session on port number 6667 to 163.162.170.173 IP address (2001:750:2:0:202:a5ff:fef0:aac7 IPv6), through IPv6 tunneled traffic.

The IPv6 protocol was used to open an IRC session on an Italian IRC server (irc6.edisontel.it). The welcome message of the servers confirmed that it has IPv6 support ('IPv6 I-lines are only for Italian pTLA') and indicated its new IPv6 address, written above. The TCP stream of the session confirmed that the attacker talks in Italian.

Questions:

1. What is the operating system of the honeypot? How did you determine that? (see day1)
The operating system of the honeypot is:

SunOS zoberius 5.8 Generic_108528-09 sun4u sparc SUNW,Ultra-5_10

I found the string 'SunOS 5.8' in the data field of a packet sent from the honeypot, with TCP 6112 source port (trace number 571). After that point, I could get more data from the TCP Stream from the TCP session over port 1524 just after the attack (trace number 588).

Though it is possible to get information from the file day1.log there is more detailed information within the file day3.log, when the honeypot sends several Syslog Protocol packets to another local machine (at trace number 117204):

SunOS Release 5.8 Version Generic_108528-09 64-bit
Sun Ultra 5/10 UPA/PCI (UltraSPARC-IIi 360 Mhz)
128 MB RAM / HD ST38410A

2. How did the attacker(s) break into the system? (see day1)
The attacker exploited a known vulnerability in the 'CDE Subprocess Control Service' dtspcd (an integrated graphical user interface that runs on UNIX and Linux operating systems) described by CERT® Advisory CA-2002-01 [2]. dtspcd typically runs on TCP port 6112.

The attacker made a successful buffer overflow attack against TCP port 6112 (trace number 580), and opened a remote shell with root privileges on port 1524. After that he downloaded several tools from different IP addresses using FTP and HTTP sessions. He used those tools to take control of the compromised system..

3. Which systems were used in this attack, and how? (see day1)
 I will describe the hosts involved in the attack and their more remarkable actions using the data found in day1.log file:

  1. 61.219.90.180: The first host involved in the attack had the IP address 61.219.90.180. It first scanned TCP ports 6112 (dtspcd) and 1524. Script kiddies frequently use the port 1524 to install backdoors; therefore, I think that the attacker was verifying if the honeypot was already compromised before attacking it. After checking that the 6112 port was open and the 1524 closed, the attacker made a successful buffer overflow attack [2] and installed a backdoor on TCP port number 1524. After that, he used the backdoor to take the following actions.

  2. 62.211.66.16: The attacker downloaded into the compromised honeypot several tools (wget, dlp, solbnc, and ipv6sun) from 62.211.66.16 using FTP.

  3. 62.211.66.53: The attacker downloaded one file (sol.tar.gz) from 62.211.66.53 to the honeypot using an HTTP session. He used that file to automate several tasks such as deleting /var/log files, installing a SSH Rootkit on port 5001, or installing a Rootkit PsyBNC on port 7000.

  4. 192.18.99.122 - sunsolve.sun.com: The attacker opened an FTP session to sunsolve.sun.com and downloaded several patches into the honeypot, to fix the vulnerabilities he exploited.

  5. 80.117.14.44: This host was used by the attacker to open a TCP session on port number 7000 (Rootkit PsyBNC Port) of the honeypot (trace number 8632).

  6. 61.134.3.11 and 217.116.38.10: Since the attacker installed a DDoS tool in the compromised system, the honeypot began to send several ICMP_ECHOREPLY packets with the word 'skillz' to these addresses, but they didn't respond in day1.log file. In day3.log file, the honeypot received response activity from the address 61.134.3.11. You can find a detailed description in my answer to question 5.

4. Create a diagram that demonstrates the sequences involved in the attack. (see day1)
The following simplified diagram could describe the general lines of sequence of the attack, and the hosts involved.

        61.219.90.180           62.211.66.53            80.117.14.44            61.134.3.11
 honeypot            62.211.66.16           192.18.99.122           217.116.38.10
==============================================================================================
    |    1    |          |          |           |           |           |           |
    |<------->|          |          |           |           |           |           |
    |    2    |          |          |           |           |           |           |
    |<--------|          |          |           |           |           |           |
    |    3    |          |          |           |           |           |           |
    |<--------|          |          |           |           |           |           |
[(3)|         |    4     |          |           |           |           |           |
[(3)|<------------------>|          |           |           |           |           |
[(3)|         |    5     |          |           |           |           |           |
[(3)|<----------------------------->|           |           |           |           |
[(3)|         |    6     |          |           |           |           |           |
[(3)|<----------------------------------------->|           |           |           |
[(3)|         |          |          |           |           |           |           |
    |         |    7     |          |           |           |           |           |
    |<------------------------------------------------------|           |           |
    |         |    8     |          |           |           |           |           |
    |------------------------------------------------------------------>|           |
    |         |    9     |          |           |           |           |           |
    |------------------------------------------------------------------------------>|
    |         |    9b    |          |           |           |           |           |
    |<------------------------------------------------------------------------------|
    |         |          |          |           |           |           |           |
==============================================================================================

1. Scan of TCP ports 6112 and 1524 of the honeypot.
2. Successful buffer overflow attack to honeypot's TCP 6112 (dtspcd) port.
3. Open a TCP session on the recently opened port 1524 (backdoor).
4. Download several files using an FTP session.
5. Download tool using HTTP protocol, to automate tasks that will compromise the honeypot.
6. Download several patch files using HTTP protocol, and fix honeypot vulnerabilities.
7. The attacker opened a TCP session on port 7000 (psyBNC: bouncer installed by him). He
   uses this session to establish IRC sessions.
8. The honeypot sends ICMP_ECHOREPLY packets (with the word 'skillz' in the data field).
9. The honeypot sends ICMP_ECHOREPLY packets (with the word 'skillz' in the data field).
9b.In day3.log file the honeypot receives response from one 'handler', and it is used to
   mount several DDoS attacks.

5. What is the purpose/reason of the ICMP packets with 'skillz' in them? (see day1)
 The ICMP packets with the string 'skillz' in their data field presents symptoms of the use of a DDoS (distributed denial of service) attack tool called "stacheldraht" or one of its variants [6].

The following diagram represents a standard stacheldraht network:

                       +--------+
                . . .  | client |  . . .
                       +--------+ 
                           |
        . . . --+----------+-----------+-- . . .
                |                      |
          +-----------+          +-----------+
          |  handler  |          |  handler  |
          +-----------+          +-----------+
                |                      |
. . . ---+------+-----+------------+---+--------+-- . . .
         |            |            |            |
     +-------+    +-------+    +-------+    +-------+
     | agent |    | agent |    | agent |    | agent |
     +-------+    +-------+    +-------+    +-------+

As described in the document written by David Dittrich, the ICMP_ECHOREPLY packets with 'skillz' are sent by an agent (honeypot) to an internal list of handlers (61.134.3.11, 217.116.38.10).

The analysis process revealed that in this scenario, the attacker used a variant of that attack tool. The ICMP packets had an id of 6666 rather than 666 as noted in the Dittrich document.

6. Following the attack, the attacker(s) enabled a unique protocol that one would not expect to find on an IPv4 network. Can you identify that protocol and why it was used? (see day3)
I think that the protocol asked for is Internet Protocol version 6. IPv6 is a new version of the Internet Protocol, designed as a successor to IP version 4 (IPv4) [7]. The changes from IPv4 to IPv6 fall primarily into the following categories:

This protocol is not widely adopted, and it is mainly used in Asian countries, such as Japan. The attacker probably took advantage of this characteristic to avoid detection and security mechanisms. He enabled an IPv6 tunnel over IPv4 almost at the end of the day3.log file to establish an IRC session to 163.162.170.173 (irc6.edisontel.it) at trace number 117614.

7. Can you identify the nationality of the attacker? (see day3)
A search for ip address 80.117.14.44 (in day1.log file) and for 80.117.14.222 (in day3.log file) in the ripe.net database [8] returns these results:

inetnum:      80.117.0.0 - 80.117.255.255
netname:      TINIT-ADSL-LITE
descr:        Telecom Italia
descr:        Accesso ADSL BBB
country:      IT

This tells that in the attacks were involved two different IP address (probably the same machine with dynamic IP address) from an ADSL connection from an Italian ISP.

On the other hand, the IRC sessions extracted from both tcpdump log files demonstrates that the attacker talks with other people in Italian. One of those sessions was registered in this file.

Therefore, I am almost sure that the attacker (Dj`bobz`) comes from Italy.

Bonus Questions

What are the implications of using the unusual IP protocol to the Intrusion Detection industry?
The use of new protocols has an important impact on the Intrusion Detection tools. The IDSs must support each protocol to identify signs of misuse or anomaly behaviours. The logging mechanisms and detection engines must be capable to recognize the format and operating mode of the new protocol.

The appearance of new protocols affects the NIDS (Network based IDS) tools. Almost no NIDS products can decode IPv6. In addition, even in strictly IPv4 networks, attacks can enable IPv6 tunneling within IPv4, blinding detection technologies.

The honeypot does not care about which IP protocol an attacker uses, because it will detect, capture and log all IP activity.

One way to solve the problem of NIDS with the use of new protocols consists in adding recognition features for the new protocols. This requires a suitable NIDS design, and important time and money consuming activity. [9]

What tools exist that can decode this protocol?
There are a number of applications that can support IPv6. Many of them are described at ipv6.org web site. [10]
 

Conclusion:

Two remarkable issues have been involved in this SotM: DDoS attacks, and unusual protocols (IPv6). As described in the challenge statement, what is interesting is not how the attackers broke in, but what they did afterwards.

The attacker compromised the honeypot and used it to mount DDoS attacks. These types of attacks are usually very dangerous and devastating. Among the victims of these attacks we may find several large and powerful organizations, with plenty of resources and bandwidth. Finding the communications between the entities before the attack is not an easy task, because it usually consumes little bandwidth, and can be hidden in ICMP communications. In fact, in this challenge the clients (agents) sent ICMP_ECHOREPLY packets to communicate with their controllers (handlers). Understanding the general operation mode of these attacks and the tools used to perform them is essential in order to prevent and block them. Several IDS tools are capable to correctly identify them.

The use of unusual protocols, such as IPv6 affects the Intrusion Detection Systems. These systems must have suitable knowledge before analyzing them. If not, they cannot detect any activity handled by the protocol. This issue can be used by the attackers to perform attacks, circumventing detection methods. The problem gets worse with IPv6 because in strictly IPv4 networks, someone can enable IPv6 tunneling within IPv4. Although IPv6 is not widely adopted, (it is mainly used in Asia) it is necessary to add support to recognize it in the present intrusion detection products. However, this requires a scalable product design and important efforts in research area.

References:

  1. IANA - Internet Assigned Numbers Authority. <http://www.iana.org/assignments/port-numbers>.

  2. CERT® Advisory CA-2002-01 Exploitation of Vulnerability in CDE Subprocess Control Service. January 14, 2002. <http://www.cert.org/advisories/CA-2002-01.html>.

  3. Insecure.org. Spitzner, Lance. Security Incidents: Re: increase of scans against port 1524. Jun 05, 2002. <http://lists.insecure.org/lists/incidents/2002/Jun/0028.html>.

  4. Jestrix, jestrix@jestrix.net. An Introduction to psyBNC 2.3.1. Updated on Mar 9, 2003 <http://www.jestrix.net/tuts/psy.html>, <mirror>.

  5. Snort.org. Snort Signature Database. <http://www.snort.org/snort-db/sid.html>.

  6. Dittrich, David. The "stacheldraht" distributed denial of service attack tool. University of Washington. December 31, 1999. <http://staff.washington.edu/dittrich/misc/stacheldraht.analysis>, <mirror>.

  7. IPv6.org. IPv6: The Next Generation Internet! Updated on Apr 14, 2003 <http://www.ipv6.org/>.

  8. RIPE NCC Homepage. Frequently updated <http://www.ripe.net/>.

  9. Spitzner, Lance. Honeypots: Simple, Cost-Effective Detection. Last updated April 30, 2003 <http://www.securityfocus.com/infocus/1690>.

  10. IPv6.org. IPv6 Enabled Applications. Updated on March 21, 2003 <http://www.ipv6.org/v6-apps.html>.