Results of the Forensic Challenge
Index
Summary
In all, we received 13 submissions from around the world to the
Challenge. An "official" analysis by Dave Dittrich (with assistance
from Wietse Venema) was done as well. All analyses (including the
official one) were done without any data from the IDS, nor with any
tools or techniques from other analyses. Many entrants (and some who
contacted us who couldn't make the deadline) had no idea
how much time this analysis would take, and it took a
lot (as you will see.) Most finished when they ran out of
time, not when they felt they were done.
Overall, the efforts put in by those submitting entries are very
thorough and professional, a step above the incident reports you
often see on mailing lists that gives the most basic "at first glance"
facts and asks more questions than it answers. I anticipate that this
will begin to change, as anyone in the security community can now take an art
historian style view of 13 different paintings (14 if you count the Honeynet
analysis) of the same landscape.
Each submission, even within the rules/guidelines for the Challenge,
took a slightly different angle. Nearly every entrant found at least
one thing that the others did not (me included, both in finding things
and missing them.) We tried to comment as much as possible on each
entry, but even the judges had time limitations and a deadline. We
want to thank everyone who participated for contributing to the
project, and hope they gain from it as well.
The average time spent in investigation turned out to be about 34
hours per person. That's a standard week's worth of work to clean up
and deal with the mess left by an intruder in about a half an hour.
That's about a 60:1 ratio! Using a standard upper-mid range annual
salary figure of US$70,000 per investigator, that works out to be a
cleanup cost of over US$2000 for a single incident. It is very likely
one of dozens, if not hundreds, of intrusions just like it. As you
will see when you read the analyses, this wasn't the first
time this intruder did this.
"But all it takes to re-install Red Hat is 30 minutes. How do you
come up with US$2000 damage?" Simple. For the same reasons cited
in
i.only.replaced.index.html.txt
(and then some, since this is more than just a web page defacement.)
When a system is compromised, and the data on it and its network are
compromised, it is not simple to determine the extent of the damage
without a lot of work. We do not know if the blackhat stold peoples
passwords, hacked other systems, has implemented sniffers, etc.
This argues for strong prevention, defense in
depth (including monitoring in depth), and trained responders. If all
the administrator does is re-install the OS, they are doing a wholly
inadequate job of responding to a security incident, as the extent of
damage may be far greater then a single system.
Crackers commonly deride system administrators for shoddy security, so why do
they then feel justified in claiming they did "no damage" by
suggesting the system administrator should do a
similarly shoddy job of incident response? Make no mistake.
Computer system intrusions have a cost.
That is not to suggest that every intrusion warrants a
complete forensic investigation, but in some circumstances it is
entirely appropriate and needs to be done quickly (and correctly).
Consider if this were a military site, or a government contractor
doing classified work (e.g., as occured recently
with Sandia National Labs). Those responding to such an intrusion
do so under the assumption that the intruder is a foreign intelligence
or military attacker, not just some teenage kid in their bedroom. I
wouldn't want them to respond any other way, in case it IS a military
threat. The 104 hours spent by Teo's team would not be entirely
unreasonable in that case (although I believe the cost of criminal
investigation should be separated from that of incident response and
cleanup, and "intellectual property" and other losses should only be
allowed if such losses can actually be proven, unlike for example the
Steve Jackson
Games case where a 911 document which could be purchased for some
US$30 was valued at US$79,449 for purposes of estimating damages.)
Time/Cost Analysis
Here are the time/cost breakdowns for this incident (see the
"costs.txt" files for more detailed data):
| Submitter |
Time spent |
Years security/sysadmin exp. |
Investigators |
| teo | 104 hours | 7/3/1/1 years | 4 |
| david | 80 hours | 8/5/4 | 3 |
| royans | 80 hours | 5 years | 1 |
| brian-carrier | 64 hours | 6 years | 1 |
| brian | 48 hours | 5 years | 1 |
| peter | 48 hours | 3 years | 1 |
| addam | 40 hours | 3 years | 1 |
| marco | 40 hours | 8 years | 1 |
| roessler | 37 hours | 8 years | 1 |
| knut | 15 hours | 11 years | 1 |
| andy | 10 hours | Not stated | 1 |
| tye | 10 hours | Not stated | 1 |
| john | Not stated. | Not stated | NA* |
| TOTAL | 576 hours | | 17 |
(* John is not included in the cost analysis because he did not state
time spent or experience.)
Average time spent per investigation (/12): 48.0 hours
Average time spent per investigator (/17): 33.9 hours
Average cost per investigation @ US$33.65/hr: US$2,067.46 +/-
US$310.12
Minimum investigation cost: US$430.72 +/- US$64.61
Maximum investigation cost: US$4,479.50 +/- US$671.92
Assuming the incident was investigated by an independant consulting
firm, charging US$300.00 per billable hour (benefit costs included in this
rate), and using the five most extensive investigations for estimating
the average (75.4 hours), the "damage" escalates significantly.
Average cost per investigation @ US$300.00/hr: US$22,620.00 +/-
US$3,393.00
One thing is for certain. It is much harder and takes
more skill to figure out what was damaged than to do the damage. Take
a very close look at the top submissions and you'll see what I
mean.
Rankings
Entries were judged by a panel that include Wietse Venema, Tan, Lance
Spitzner, and Dave Dittrich, with assistance by Dan Farmer, Kevin
Mandia, and T Elam. Judges operated under guidelines that used a point
system for ranking each entry. The maximum score that an entry could
receive was 45 points. The combined scores of all judge's were then
averaged.
The Honeynet Project would like to thank all thirteen members who submitted
a writeup. For many of these people, this was there first such attempt
at a forensic analysis. Even though some of the individuals knew they
would not place in the top three, they submitted their entries anyways
to both improve themselves and the security community.
In gratitude for the hard work of all these individuals, every member
will be awarded a 2nd Edition of "Hacking Exposed." In addition,
the Top Three submissions will also be receiving a prestigious
Honeynet Project team shirt. (You can see the submissions by
following the link from the submitter's name. You can see the
comments of the judges by following the link from the points.)
The End?
The Forensic Challenge judging is now over, but this sub-project is
going to live on in several ways.
- The results will be here for quite some time for people to learn
from. There are some new tools that came out of the Challenge, some
interesting techniques, and some good examples of how to do a
forensic analysis of a compromised system. If you have the time
(a couple hours per submission), you will benefit from reading
all of the analyses and seeing how each differs from the others.
You will learn something, guaranteed.
- The Honeynet Project members will be carrying on the learning
with talks and courses based on the lessons learned in the Challenge.
We will be assembling some "best practices" documents and guidelines
to help move forward the state of the art in forensic analysis. Look
for more details about what was learned from the Forensic Challenge in
a talk at CanSecWest '01 March
28-30 in Vancouver, BC, Canada, and talks/courses at upcoming BlackHat
and SANS conferences.
- We are working with members of the United States Department of
Justice and the King County Prosecutor's Office to make sure
that the examples and best practices fit the needs of local, state and
federal law enforcement agencies in understanding and assessing
computer crime cases and pursuing suspects. Detailed technical
analyses are great for geeks, but hard for judges, prosecutors,
and criminal investigators to digest and understand quickly.
This is the first time a learning situation like this has existed,
where members of law enforcement can speak openly about the analyses
of a real intrusion without fear of compromising an actual --
and quite costly -- criminal case. Hopefully this will help
bridge some gaps and smooth the road between computer security
professionals and the law enforcement community.
(Note that there will be no prosecutions of anyone involved in
this intrusion. This is not about catching the person who did
this intrusion, but rather about what can be learned from it. Whoever
did this is veeerrrrry lucky its working out this way.
This time. ;)
If you have any suggestions, questions, or comments on the Challenge,
feel free to contact us at <challenge@honeynet.org>
Shouts and greetz
Thanks once again to everyone who has assisted or participated in the
challenge, including all the submitters listed above, Lance Spitzner,
Dan Farmer, Wietse Venema, Tan, Kevin Mandia, T Elam, Rik Farrow,
Kevin Manson, Steve Schroeder, Floyd Short, Richard Murray, Ivan
Orton, and Alisha Ritter (if I left anyone out, I apologize - blame it
on lack of sleep.)
Dave Dittrich
