Significant malicious events:
The data was analyzed in daily batches in the beginning each day, using nFX OSP software.
Q2: Was the system compromised? How do you know? If yes, how many times and by how many attackers? What would you consider the most compelling evidence of the compromise available if you find that the system was indeed compromised?
A:
The system was indeed compromised at least twice. Here we define "compromise" as a successful attack, followed by the some follow-up activity.
The most compelling evidence of compromise (in our opinion, as honeypot owners) was the outbound IRC communication. While 'wget' and 'lynx' malicious software download attempts provide fairly compelling evidence, IRC implies that the intrusion succeeded, the attacker has some degree of control over the machine and that he managed to install his/her own software (an IRC client or bot). In this case, we observe an abundance of IRC activity (channel joins, nick changes, operator activity, multiple TCP 6667 connections, etc) detected by Snort and reported by iptables
Here are some of the IRC traces from the snort logs:
Q3: If this were the evidence from a production system, how would you learn that the machine was compromised, given the data available? For this question, assume you do not have the honeynet-specific data streams, such as sebek2 or bash logger, just like in this challenge.
A: Even for production environment, outbound IRC initiated by a web server is a 100% reliable indication that the successful attack has taken place (IDS logs needed). We cannot reliably conclude a compromise from syslog messages. We can make an educated guess that a machine was compromised from Apache logs (access_log) since it responded with a code 200 to an 'awstat.pl' attack request (web server logs). Iptables log indicating outbound connection from a server can also provide basis for a fairly educated guess (iptables logs).
Q4: What else was going on at the system at the same time? What times of "Internet noise" can you categorize, given the data? Is there anything out of the ordinary with the noise levels? What attack and probe types observed actually had a chance of affecting the target?
A: We can briefly summarize the types of noise by Snort alarm or by destination port (left as an exercise to the reader).
Q5: Do you think that the time was synchronized between the various monitoring systems (where Snort and iptables logs were collected) and a victim system(where syslog and Apache logs were collected)?
A:
1. The time between the monitoring box ('bastion') and the gateway ('bridge') was synchronized thru NTP. Thus, iptables and Snort records are in sync.
2. On the other hand, the time between those systems above and the victim server ('combo') was not synchronized. In fact, there was a huge gap of 4:47 hours, due to the carelessness of the honeypot operator (i.e. me :-)) However, such situation is very common is production environments and thus provided a fun challenge for the participants. It is not easy to keep the honeynet time-synchronized with the analysis systems, since no direct communication is established. However, the honeynet was later configured to talk to a global time server and so were the analysis systems (independently).
Q6: Describe the procedures and tools of that you used to analyze all the distinct log sources together.
A: Logs were all imported into netForensics nFX OSP SIM software. The system converted them to the uniform format and the built-in tools were then used to summarize, visualize and correlate logs as well as track attacker's activities.
Conclusion Logs from multiple sources (host and network), combined together, provide tremendous value during the compromised system analysis and allow a complete incident reproduction even without using honeypot specific tools such as Sebek
Last modified: Fri May 20 00:51:01 Eastern Standard Time 2005