The following tools & references were used to perform this analysis.
1. The Win32 port of Snort 1.8 from http://www.snort.org
2. The Win32 version of Ethereal from http://www.ethereal.com
3. The Cygwin utilities from http://sources.redhat.com/cygwin
4. Various whitepapers at the Honeynet Project
5. TCPFlow, available here
6. TCPShow, available here
7. The Security Focus Vulnerability database
First, download the log file and verify the MD5 Sum:
bash-2.04$ md5sum scan19.zip
c065797b3c2ddfad3396e3d4542ed8a7 *scan19.zip
bash-2.04$ md5sum scan19.tar.gz
11e0be295d138df14111796a7733a5d2 *scan19.tar.gz
The signatures match, so proceed with the analysis.
The Snort logs are rather large, so the initial identification of the attacker is quite difficult. As the syslogs are smaller, it is quicker to search them for clues. I used TCPShow to analyse the Snort syslogs
tcpshow < slog2.log > udp.txt
The first interesting traffic shows an attempt to use the rpc.statd bug to gain access to this server. However, a quick look at this traffic in short shows that the attempt failed, and was the only traffic from that particular host.
The next interesting traffic shows a failed login attempt using the "nobody" account, from the host 217.156.93.166:
Packet 15
TIME: 23:53:05.761806 (14:26.781117)
UDP: 0.0.0.0.syslog -> 0.0.0.0.syslog
DATA: <33>PAM_pwdb[5608]: get passwd; pwdb: request not recognized
---------------------------------------------------------------------------
Packet 16
TIME: 23:53:07.193248 (1.431442)
UDP: 0.0.0.0.syslog -> 0.0.0.0.syslog
DATA: <37>login[5608]: FAILED LOGIN 1 FROM 217.156.93.166 FOR nobody, Authentication service cannot retrieve authentication info.
A second failed attempt from the same IP follows, and then 20 minutes later, a successful login to the nobody account. The IP address from the successful attempt is not recorded, but this is unusual activity. This gives a first starting point for the analysis.
TCPFlow was used to provide a trace of all the sessions from this host:
tcpflow -r newdat3.log 'host 217.156.93.166'
This showed the failed logins, and then a successful login, with no apparent change in between. What followed was the download and installation of some rootkits. This lead me to believe a second machine was involved in the initial compromise.
To identify the second machine, I took a chance on the only interesting entry in the syslog trace between the failed and successful logins. This was a ftp session from the host 207.35.251.172.
Packet 20
TIME: 23:55:52.196786 (01:51.402778)
UDP: 0.0.0.0.syslog -> 0.0.0.0.syslog
DATA: <94>ftpd[5609]: ANONYMOUS FTP LOGIN FROM 207.35.251.172 [207.35.251.172], mozilla@
Once again, tcpflow was used to analyse traffic from this host:
tcpflow -r newdat3.log 'host 207.35.251.172'
My chance paid off on this. The trace shows the ftp session beginning, and the execution of the SITE EXEC commands, with large inputs. This eventually leads to a shell session, with root level access. See the attached file wuftpdexploit.txt for the full session. (This file was generated by loading the trace into Ethereal and applying the filter
"ip.addr eq 207.35.251.172 and tcp.port eq 21" and then choosing the Follow TCP Stream option.) A quick search of the Security Focus vulnerabilities database shows that wu-ftpd 2.6.0 has an input validation problem that allows use of the SITE EXEC commands to gain root level access to a machine.
I also used Snort to dump out the information from the log into a separate directory to see if any further information could be gathered.
snort -dv -r newdat3.log -l C:\sotm\attacker host 207.35.251.172
This dumped 3,273 files into the directory, and a glance at these files showed that the machine had first performed a port scan of the compromised host, before compromising the wu-ftp daemon. It would appear that at least the scan and initial compromise was scripted.
The attacking machines had been identified as 217.156.93.166 and 207.35.251.172. With this information I had enough to proceed on to analysing the activity.
Using the information in wuftpdexploit.txt, I could follow how the system was initially compromised. It was likely a scripted compromise (the input data is too complicated to enter by hand), followed by the entering of commands manually by the attacker (mistakes are made in some commands). Once the shell session was established, the attacker checked out the contents of some directories, and then proceeded to remove the password on the nobody account:
passwd nobody -d
Changing password for user nobody
Removing password for user nobody
passwd: Success
The attacker then created two directories, and copied the current MAC times from /etc/passwd and /etc to these directories using the touch command. The password on nobody was then removed again, and a user named "dns" was created, with uid=0 (root equivalent). The password on the dns account was removed, and the MAC times copied back to the original files. This was obviously to hide the fact that /etc/passwd and /etc had been modified.
cd /
mkdir -p /etc/X11/applnk/Internet/.etc
mkdir -p /etc/X11/applnk/Internet/.etcpasswd
touch -acmr /etc/passwd /etc/X11/applnk/Internet/.etcpasswd
touch -acmr /etc /etc/X11/applnk/Internet/.etc
passwd nobody -d
/usr/sbin/adduser dns -d/bin -u 0 -g 0 -s/bin/bash
passwd dns -d
touch -acmr /etc/X11/applnk/Internet/.etcpasswd /etc/passwd
touch -acmr /etc/X11/applnk/Internet/.etc /etc
Changing password for user nobody
Removing password for user nobody
passwd: Success
Changing password for user dns
Removing password for user dns
passwd: Success
There was little interesting activity after this, and the session was closed.
Now that the attacker had an account with no password, and a root level account with no password, a telnet session was established. This is the successful login observed in the syslog trace. For some reason, using the Show TCP Stream option in Ethereal wouldn't give me the complete session, so I used the information from the original tcpflow output. I've included the file "192.168.001.102.00023-217.156.093.166.61216" as reference. This file is the full telnet session. The following is a summary of the activity in this session.
1. Login as nobody
2. su to dns (root level account created during the ftp session)
3. Ftp to teleport.go.ro, a Romanian ftp server
4. Download three files: Zer0.tar.gz (139711 bytes), copy.tar.gz (265189 bytes) and ooty.tar.gz (14847 bytes)
5. Unpack Zer0.tar.gz and run the file "Go" from this archive, passing it the command line parameter "24". This installs a rootkit, including a SSH daemon listening on port 24.
The rootkit installation proceeds along the following lines:
1. Attempts to kill syslogd and fails
2. Detects that the machine is logging to a remote machine and warns the installer
3. Extracts some files
4. Installs a version of SSH on the port specfied in the command line (24)
5. Modifies the IPChains configuration
6. Installs and configures something called "Adore"
7. Adore hides a few files, executes some of the files included in the rootkit
8. The log files are cleaned
9. The installation is complete.
Once the attacker had the rootkit installed, the telnet session was ignored, and an SSH session was started instead. During this time, the attacker also opened a new telnet session, and two other SSH sessions. This was observed using the Snort dumps, the SSH sessions show the SSH version in plain text before the encrypted traffic.
1. Which vulnerability did the intruder exploit?
As detailed above, the intruder exploited an input validation bug in wu-ftpd with SITE EXEC. Using this bug, the attacker was able to execute commands as root, and create a root user account for later use.
2. What ways, and in what order, did the intruder use to connect and run commands on the system?
Ignoring the initial failed attempts at login, the intruder used the following methods to connect and run commands: First, the ftp session with the initial compromise. Second, a telnet session from another host. Finally, SSH sessions, combined with a telnet session (presumably testing the changes being made to the system).
3. How did the intruder try to hide his edits from the MAC times?
As detailed above, the intruder used "touch" to copy the MAC times from /etc and /etc/passwd to temporary files, make changes and then copy the times back, making the files appear as though they hadn't been modified.
4. The intruder downloaded rootkits, what were they called? Are they new/custom rootkits?
Rootkits are called Zer0.tar.gz, copy.tar.gz and ooty.tar.gz. Zer0.tar.gz seems to be a renamed version of the torn rootkit (I found this out by digging around in the rootkit files)
Copy.tar.gz contains mj2 & mj3 which are copies of the ELF/blitz SYN flooder. I was alerted to this by InoculateIT denying me access. I also used "strings" on some of the other files, and found zxploit by Team Teso which is used for compromising wu-ftpd, wu-scan by Narrow, smrf5 is (papa)smurf.c bt TFreak and suu.tgz which lead me to http://www.securityfocus.com/cgi-bin/vulns-item.pl?section=exploit&id=1634 (via Google of course). This would appear to be a custom archive of preexisting tools.
ooty.tar.gz (14847 bytes)seems to be a packaged version of a pre-existing rootkit, which I couldn't find the name of but could find various references to it on non-English language sites (a search on google for "Marchew bighole" (strings found in perl.x in the archive)
5. Recover (tell how you did it too) the rootkits from the snort binary capture
As I learned in last months challenge, this is fairly easy. I used Ethereal to load the capture and used the filter tcp.port eq 20 and not tcp.checksum_bad. I then chose the first packet, used the Follow TCP Stream option and saved the file as Zer0.tar.gz (the first file downloaded). I also noted the last number of the packet, and reset the filter back to my original one, and went to the next packet after the final packet in that file and repeated the process. I then confirmed the sizes of each file with the sizes noted in the FTP download. I haven't attached the files to the scan as it was triggering virus scanners (both locally and remotely :)
6. What does the rootkit do to hide the presence of the attacker on the system?
The rootkit does several things:
1. Attempts to kill the syslog daemon before the script runs.
2. Copies some of the MAC times to temporary files before making changes, and then copies the MAC times back to preserve the timestamps of files.
3. One of the executables that is installed (ava) is used to hide files and processes
4. One of the other executables is used to clean log entries from various log files.
7. What did you learn from this exercise?
I've learned that you can't assume that attacks will be coming from one machine only, and hackers will use whatever machines they have at their disposal to attack you. This could be because different machines are restricted in which protocols they can send out on the Internet by firewalls or packet filtering.They will also try to hide their tracks as often as possible. I've also learned that even the most innocous seeming system binaries can be turned against you - in this case touch.
8. How long did this challenge take you?
This challenge took around twelve hours. The majority of this time was floundering around trying to identify the compromise. Once I figured this out, it only took two hours of analysis and two hours to write it up.
Bonus Question: Based on this challenge, write an example letter of notification to the source owner that attacked the system. Include any evidence or logs that you feel important.
See attached Letter 1 & Letter 2
This analysis was carried out by Stuart Fox (mail me at work or personally).