To aid me in analysing the log files, I imported both of them into their own Excel spreadsheet and clean up the columns with some formatting/merging. Combined both log files into one Excel spreadsheet using the time column as the index. Once that is done, translated all the hostnames in the firewall log into IP addresses. For some reason, some of the hostname are no longer resolvable by DNS, and these had to be resolved by cross-examining the log entries from the IDS log (and some whois queries). Now, we finally have something that we could examine to figure out what had happened.
To help understanding the events better, additional entries were added into the spreadsheet to include the dates when each box was brought online and/or rebuilt. And then comb through the spreadsheet categorizing the different type of events, and notes were made to highlight any interesting activities (different type of scans and what they are looking for, Buffer-Overflow attacks launched and repeat visitors). Colour (yes, this is how we spell color in Canada) coding the IDS events base on the machine types also enhanced the readability of the combined log files.
After some quick lookups to www.whitehats.com on the details of some of the IDS codes mentioned in the logs, the picture is somewhat clear now on what had taken place…
After reading the recently published honeynet.org white paper on log statistics analysis, I wrote up a bourne shell script that give the me the count per day statistics for each events. However, events for RPC-related activities were broken down into IDS7 (a source port connection to RPC from port 53), IDS13 events (portmap-request-mountd), IDS15 events (portmap-request-status), RPC-Info-Queries and a whole bunch of stealth/SYN scans. Therefore, I quickly modified my script to present the count-per-day statistics for each destination service instead. The 3-day moving average is then calculated and plotted for analysis. However, due to the limited size of the data set, the graph didn’t look too helpful … Modified my script once again, this time the data is presented in an hourly basis instead daily. Plot the new graph base on the new counts and 3-day moving averages, and now this graph is showing something interesting…
Another quick visit back to the honeynet.org website and located the white paper documenting the “incidents”, and located the 4 IP addresses where those worms came from (216.191.92.10, 216.234.204.69, 210.111.145.180 & 216.23.6.24). However, there are no entries in either log to suggest if those hosts had tried non-NBT base attacks/probes on any of the honeynet boxes during this timeframe.
For your reference, a HTML formatted copy of the spreadsheet with the combined log file is also attached along with this submission along with the diagram plotted for the analysis by the Statistical Process Control methodology.
Question #1: What trends did you identify?
Scans, both the stealth and the non-stealth ones, are purpose-oriented. No unnecessary blind/exhaustive port scans were performed. This is demonstrated by the fact that most scans were focus around the RPC and the FTP services (for which several attacks were launched against).
The intruder typically do their scanning and proceed with their hacking attempts shortly after. Very few scans/attacks were done from the same IP address outside of the 24 hour-day the firewall employ to generate its unique inbound access log. The “repeat” visitors are listed below:
c871553-b.jffsn1.mo.home.com 8Nov2000 7:31:15 telnet
c871553-b.jffsn1.mo.home.com 10Nov2000 10:37:31 ssh
c871553-b.jffsn1.mo.home.com 14Nov2000 0:58:36 ftp
cr449893-a.rchrd1.on.wave.home.com 11Nov2000 21:25:06 sunrpc
cr449893-a.rchrd1.on.wave.home.com 16Nov2000 16:49:51 http
sim-home-2-3.urbanet.ch 16Nov2000 20:48:43 ssh
sim-home-2-3.urbanet.ch 17Nov2000 14:17:10 ssh
Question #2: What does this activity tell us about the
blackhat community
The crackers come knocking on the door with their mind set on what they plan to do.
They scanned for a host that might have this particular vulnerability they are looking to exploit, once a potential victim is found, they move in and attack immediately. Also, from looking at these scans that didn’t lead to an attack, it is likely that some crackers scripted blind scans of a whole bunch of networks from one IP address, and after the result of these scans were retrieved and analyzed, they moved in for the kill from a different IP address.
By focusing on only one type of attack, they tripped off the minimum amount of alarms necessary for they to launch the attack (and thus minimized their chance of being noticed). And they moved quickly to reduce the likelihood of the admin got on their trail and build up defences for the vulnerability they are about to exploit.
For those “repeat” visitors (ones that return to the honeynet days afterwards from the same IP address), they seemed like they were trying to crack the system the old fashion way (by brute-forcing the telnet, ssh and ftp services). But this is just my personal intuition, since we don’t have any information on what they were doing with those authentication-based services.
Question #3: What, if anything, happened in the
firewall and IDS logs that gave us a clue of what was coming? Could any of the
attacks been predicted ahead of time. If so, how?
The only clue that the honeynet was being scoped for the attack is the concentration/focus of the probes/scans. Apart from some minor noise, the majority of the probes were focused on FTP and RPC, and indeed several buffer-overflow attacks were launched against vulnerability in these two services.
Now, there were three incidents of the same type of attack (IDS128) that came with no warnings whatsoever, and there’s pretty much nothing you could do about these attacks other than being very diligent in keeping the systems up-to-date and the fingers crossed.
Now, for the RPC-related activities (since this is one we have the most information on), we are able to apply the Statistical Process Control methodology into the analysis. As shown in the attached graph, the 3-day moving average line raised across the control limit line at three separate occasions. One of these occasions (around mid-day of Nov. 6th, 2000) was just about a day and half prior to the second buffer-overflow attack/attempt on the RPC service. If a program is written to analyze the snort log file on a hourly basis, we might be able to give the sysadmin a good indication that attacks are imminent.
If real-time alerts were in place to alert the admin of any probes, the admin might be able to research into the vulnerabilities being posted on BUGTRAQ on the service being probed and build up his defences before the actual attacks comes.
Or, the admin could weight his risk and impact and decided to shut down the service(s) probed to buy himself time before he could figure out what type of attack the cracker is planning on.
However, one should not configure automatic blocking of those services as a scripted response for the port scan (e.g. the OPSEC-based response available in certain IDS), since they can be abused into another sort of DoS to the victim hosts.
Question #4: What data did you find more valuable, the
Snort alerts or the firewall logs of unique scans? Why?
I found the combination of both the firewall logs of unique inbound activities and the snort logs being the most useful.
Neither one of them was able to provide the complete picture on what’s going on. While both of them had missed some connections, the firewall was able to identify the services being probed better than Snort. Snort merely logged that a non-stealth port-scanning has taken place, but did not log the service that was probed. The firewall log, on the other hand, provided too little information for statistical analysis. However, a more verbose firewall logging system might be all that we need for statistical analysis.
Now, with information from both logs combined, we would be able to gather enough data (from either hourly or daily statistics) to apply the Statistical Process Control methodology to look for the sudden increase in the amount of traffic to a particular type of service. This type of “advanced-warning” system is especially valuable in a production environment, where it’s difficult to identify the hostile probes from the friendly visitors.
With this being said, Snort logs are valuable in providing the forensics to what had taken place in the network. Without the IDS logs, one might have not known whether the network/systems security was comprised and/or how it was compromised.
Question #5: What lesson did you learn from this?
Apart from monitoring the appropriate InfoSec newgroup/mailing lists carefully, one could also get advance warnings on potential intrusions base on statistical analysis of both the firewall and the IDS logs.
While there’s nothing to replace a diligent InfoSec officer that makes sure all his network device/hosts were patched up-to-date and were configured properly for security, the advance warnings allow the InfoSec officer to gather his troops and be on alert for the potential problems.
Question #6: How long did this challenge take you?
It took me a total of around 8 hours for this challenge. However, this might not be accurate as the work done were scattered throughout a couple of weeks.
First, for ease of data analysis, imported both logs into the MS Excel and spent the next three hours trying combined the data from both logs into a new log completed with the dates each box went online. All DNS names were resolved into IP addresses, special notation were made to indicate the source of those IP addresses (some from nslookup, and some have to be derived from the firewall log). This took about 30-45 minutes.
Spent the next hour or so studying the log entries and highlight anything of interest.
Spent another 10-15 minutes looking up the vulnerabilities being mentioned within the IDS log.
Spent another 15-20 minutes looking over honeynet.org for that white paper mentioned in the bonus question and digested the content of that white paper.
Spent another 1 ½ - 2 hours studying over the new honeynet.org white paper on statistical analysis of the log entries and try to apply the principal to the log data supplied.
Spent the next 2-3 hours reviewing the notes and composing the write-up.
Bonus Question:
Both the Snort alerts and the inbound firewall logs
missed a successful attack. The only reason the Honeynet Project detected this
successful attack was because the compromised system attempted an outbound
connection (by definition this means a system was compromised). The Honeynet
Project did a writeup on this incident, can you identify the attack and why the
Honeynet failed to alert to the attack (though the attack was captured). Hint:
the attack is written up and posted somewhere on our site.
As posted in the white paper titled “Know Your Enemy: Wors at War” on www.honeynet.org, this attack was done by several variant of worms swamping the Internet looking vulnerable Windows9x system to win its master the trophy of a contest at the distributed.net. These worms plant themselves within the windows system directories and the registry. Apart from installing the distributed.net client (to help its author to win the contest), it also goes out to the Internet looking for more victims. The worms disguise themselves by names that would otherwise be mistaken for a typical Microsoft executables that comes with any Windows9x system.
The IDS did catch all the network traffic from the probing of the NBT ports, to the transfer of those files, to the eventual attempted posting of the processed data and even the worms’ attempt to infect other systems on the Internet. It failed to alert the admin at honeynet.org of the attack simply because there isn’t a signature/rule (in Snort) that would identify these type of traffic pattern as something to cause concern over. This is the biggest weakness of any IDS out there today. It’s like the anti-virus war that has been fought all these years between the crackers/pranksters and the anti-virus software vendor.
The traffic generated by these worms is essentially a sequence of file copying and the modification of the file win.ini. Just as with those VB-script based viruses, there’s no easy way to prevent the infection other than to disable the underlying mechanism of the operating system utilized by the worm for its delivery and/or functions. In this case, that would be to disable the file-sharing feature of the vulnerable Windows9x machines.
References: