HoneyNet.org Scan of the Month September 2003 (Scan 29)

www.honeynet.org/scans/scan29/

Michael Conners (michael_conners@SPAM-CONTROL.hotmail.com)
Ray Strubinger  (rstrubinger@SPAM-CONTROL.hotmail.com)

This challenge offered two files for download, a suspended VMware image and and
MD5 listing of all files on the system prior to compromise.  An MD5 checksum of
the suspended image was provided and used to verify the integrity of the
suspended image after download.  The MD5 checksum of the downloaded image is
computed by running the command (under linux) md5sum <image name>.  After a few
moments a value will be produced that should match the value shown on the honeynet page
for that file.  If the values match, then the suspended image is an exact copy
of the image posted by the honeynet team.

General Overview

The analysis of a live system is probably one of the most stressful types of
analysis that a forensics examiner conducts.  A live system presents a few
challenges, and a few opportunities that aren't normally experienced when
dealing with a "dead" or powered down system.  When dealing with a live system
the examiner has to be aware of the impact his actions (or sometimes, inactions)
will have on the system.  A live system may be booby trapped by the attacker.
Such booby traps may destroy vital information that would aid the investigation.
A live system also presents the examiner with the opportunity to capture
applications the attacker may have left running.  This can be useful in proving
that some application was actually executed.  A live system also introduces the
idea of an "order of volatility" meaning that some aspects of the system are
more stable than others.  This should be kept in mind when conducting the exam
or when acquiring evidence from the system.

An investigator should be aware of the impact of his actions and the way those
actions might alter system files or file meta data.  When dealing with a system,
especially a live system, the investigator should be prepared to explain their
actions, the impact of those actions, and the justification of the actions.
Detailed notes and a "two-man" rule are also recommended to reduce the chance of
errors.

The preceding paragraph may have seemed ominous but a suspended VMware session
presents the best of both worlds to the examiner.  A VMware session allows the
examiner to do things they probably would not normally do if the system were
really live because the examiner can always restore the system to a known good
state.  A VMware session allows the examiner to view the system many times from
a known state or to go back and re-examine something that may have been missed
the first time.

Much more analysis was done than what was suggested by the questions (in fact
even more analysis could be done.)  Included after the final question is a brief
outline of additional findings relating to the type of rootkit, the tools used
to clean the logs, and various pieces of information obtained from the RPM
database of the compromised machine.  A complete assessment of this compromise
could easily take 60 to 100 pages and consume two or three months of part time
effort.

Tools used for this analysis:

VMware 4.0 		  - www.vmware.com
Chkrootkit-0.42b 	  - www.chkrootkit.org
Autopsy-1.74	 	  - www.sleuthkit.org
SleuthKit-1.65	 	  - www.sleuthkit.org
Red Hat 8 Installation CD - www.redhat.com
Work station running Red Hat Linux 7.3
An incident response CD with statically linked binaries (ifconfig, netstat, ls,
ps, etc...)



Questions

1. Describe the process you used to confirm that the live host was compromised
while reducing the impact to the running system and minimizing your trust in the
system.

Background information:  We were told that a Red Hat Linux 7.2 system running
within VMware was compromised on August 10, 2003.  We were told that the system
was suspended while a root console window was open (resuming this image will
drop us into the root console in this case.)  We were provided
with a set of known good hashes that were created from the system's files prior
to the compromise.  During the analysis phase it was determined that the machine
was created on the afternoon of July 14, 2003.

Normally the investigator would interview the system administrator and anyone
else involved in the discovery of the intrusion in order to gain an
understanding of the system and the circumstances surrounding the discovery of
the compromise.

To begin the confirmation of the compromise, the investigator changed into the
linux-2 directory that was created when the suspended VMware session was
unpacked.  The command, vmware linux.vmx was executed to start VMware and resume
the session.  The opening screen showed that the process swapd the the eth0
interface in promiscuous mode (see enclosed screen shot, startup-screen.png.)  This is usually a good indication that the wire is being sniffed.  We now assume the worst; that we are dealing with a compromised system running a rootkit that will hide various
pieces of information such as running processes or interface status from the
investigator.  To counter malicious system applications that may have been left
behind by the attacker, an incident response CD-ROM containing statically
compiled utilities was used.  This CD was inserted into the CD-ROM drive then
mounted.  The investigator changed directory to the location where the incident
response CD was mounted and used the utilities found there to confirm the
compromise.

After mounting the CD a trusted copy of bash was executed.  This was done to
provide the examiner with a clean shell.

A trusted copy of the date utility was executed to determine the current system
time.  Capturing this information is important in the overall documentation of
the analysis process.  The date command produced the following output:

Sunday August 10 20:29:21 PDT 2003


The command, ps auxwww, executed from the incident response CD, immediately
revealed something suspicious, a hidden directory within a lib
directory.  The process listing showed a binary located in the hidden
directory was running.

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.5  1412  520 ?        S    Aug09   0:05 init
root         2  0.0  0.0     0    0 ?        SW   Aug09   0:00 [keventd]
root         3  0.0  0.0     0    0 ?        SW   Aug09   0:00 [kapm-idled]
root         4  0.0  0.0     0    0 ?        SWN  Aug09   0:00 [ksoftirqd_CPU0]
root         5  0.0  0.0     0    0 ?        SW   Aug09   0:00 [kswapd]
root         6  0.0  0.0     0    0 ?        SW   Aug09   0:00 [kreclaimd]
root         7  0.0  0.0     0    0 ?        SW   Aug09   0:00 [bdflush]
root         8  0.0  0.0     0    0 ?        SW   Aug09   0:00 [kupdated]
root         9  0.0  0.0     0    0 ?        SW<  Aug09   0:00 [mdrecoveryd]
root        17  0.0  0.0     0    0 ?        SW   Aug09   0:04 [kjournald]
root        92  0.0  0.0     0    0 ?        SW   Aug09   0:00 [khubd]
root       657  0.0  0.5  1396  524 ?        S    Aug09   0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript
ident      677  0.0  0.9 26932  944 ?        S    Aug09   0:00 identd -e -o
ident      685  0.0  0.9 26932  944 ?        S    Aug09   0:00 identd -e -o
ident      686  0.0  0.9 26932  944 ?        S    Aug09   0:00 identd -e -o
ident      695  0.0  0.9 26932  944 ?        S    Aug09   0:00 identd -e -o
ident      696  0.0  0.9 26932  944 ?        S    Aug09   0:00 identd -e -o
root       699  0.0  1.3  2676 1272 ?        S    Aug09   0:00 /usr/sbin/sshd
root       732  0.0  1.0  2264  956 ?        S    Aug09   0:00 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid
root       759  0.0  2.1  5296 1984 ?        S    Aug09   0:00 sendmail:accepting connections
root       778  0.0  0.5  1440  496 ?        S    Aug09   0:00 gpm -t ps/2 -m /dev/mouse
root       820  0.0  0.6  1584  660 ?        S    Aug09   0:00 crond
root       845  0.0  1.2  3256 1172 ?        S    Aug09   0:00 smbd -D
root       850  0.0  1.1  2416 1084 ?        S    Aug09   0:00 nmbd -D
root       893  0.0  1.1  2320 1076 tty1     S    Aug09   0:00 login -- root
root       894  0.0  0.4  1384  448 tty2     S    Aug09   0:00 /sbin/mingetty tty2
root       895  0.0  0.4  1384  448 tty3     S    Aug09   0:00 /sbin/mingetty tty3
root       896  0.0  0.4  1384  448 tty4     S    Aug09   0:00 /sbin/mingetty tty4
root       899  0.0  0.4  1384  448 tty5     S    Aug09   0:00 /sbin/mingetty tty5
root       900  0.0  0.4  1384  448 tty6     S    Aug09   0:00 /sbin/mingetty tty6
root       901  0.0  1.3  2452 1296 tty1     S    Aug09   0:00 -bash
root      3137  0.0  0.7  1900  716 ?        S    13:33   0:03 smbd -D
root      3153  0.0  0.7  1664  704 ?        S    13:33   0:00 (swapd)
root      3247  0.0  0.6  1472  592 ?        S    13:33   0:00 syslogd -m 0
root      3252  0.0  1.1  1984 1096 ?        S    13:33   0:00 klogd -2
root     25239  0.0  0.3  1880  336 ?        S    15:32   0:00 /lib/.x/s/xopen -q -p 3128
root     25241  0.0  0.7  1888  672 ?        S    15:32   0:00 /lib/.x/s/xopen -q -p 3128
root     25247  0.0  0.7  1668  732 ?        S    15:32   0:00 /lib/.x/s/lsn
root     15119  0.0  1.3  2296 1240 ?        S    16:02   0:00 initd
root     15361  0.1  1.4  2552 1376 tty1     S    20:33   0:00 ./bash
root     15390  0.0  0.7  2636  728 tty1     R    20:35   0:00 ./ps auxwww
root     15391  0.0  1.4  2552 1376 tty1     R    20:35   0:00 ./bash



A trusted copy of ifconfig was executed.  This revealed that eth0 was in
promiscuous mode.  (Recall that we already had an indication that the interface was in
promiscuous mode from the first screen presented once the VMware session was
restored.)  Since we've not been able to interview the system
administrator, we don't know if this is a normal state for the interface.  (Some
security programs actually put interfaces into promiscuous mode.)


Using the ls -ltr command on the directory showed the file times were modified
on August 10th -- the day of the compromise.  Running ls -ltr within root's home
directory showed that .bash_history had been linked to /dev/null -- a typical
technique employed by crackers in an attempt to conceal the commands used while
on the system.

At this point if this were an actual live system and not a system that presented
the opportunity to restore its state this investigator would pull the plug and
obtain forensic images of the partitions.  The fact that this VMware system can
be restored to the state in which it was origionally found allows the
investigator to do things that normally would not or should not be done on a
typical live system.

Chkrootkit was executed from secure media against the compromised
system to gain additional information.  The output from chkrootkit can be found
in Appendix A.  Amoung other things, Chkrootkit indicated that several system
utilities had been compromised (infected is the term used by the tool) and that
several processes were hidden.


2. Explain the impact that your actions had on the running system.

This is one of the classic problems in Computer Forensics.  Touching the system
changes the system, doing nothing also changes the system.  This dichotomy has
often been compared to paramedics working in a crime scene.  They have to
provide aid to the victim yet exercise care to avoid destroying the crime scene
in the process of providing aid.  This is a similar situation faced by the
computer forensic examiner.  When examining a live system the concept of
Locard's Exchange Principal comes into effect.  This principal, initially
advanced by a early 20th century criminal forensic analyst basically states that
if you enter or leave a scene you alter the scene and the scene alters you.  To
put it another way, you leave small traces behind and take small traces with you
when you enter and exit a room.  Because this investigation revolves around a
suspended VMware session we have an unusual situation that our actions on the
system have the result of only inhibiting our ability to confirm the compromise.
Because the VMware session can be restored, we always have the ability to go
back and redo a technique, procedure, or recover something that may have been
missed.

In this case, the mount command was executed from a bash shell to mount an
incident response CD-ROM.  This action altered the access time of the mount
binary and the bash shell.  The MAC times on the mount point (/mnt/cdrom) were
also altered by the action.



3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non
Red Hat 7.2 default ports).

The netstat command presents an easy way to determine which ports
are open and which processes are associated with those ports.  The entire output
of the command, netstat -anp is shown below.

The output from the netstat command is shown below:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State PID/Program name
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN	845/smbd
tcp        0      0 0.0.0.0:79              0.0.0.0:*               LISTEN 	732/xinetd
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN	3137/smbd -D
tcp        0      0 0.0.0.0:113             0.0.0.0:*               LISTEN	677/identd
tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN	3137/smbd -D
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN	732/xinetd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN	699/sshd
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN	732/xinetd
tcp        0      0 0.0.0.0:65336           0.0.0.0:*               LISTEN	15119/initd
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN	25241/xopen
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN	759/sendmail: accep
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN	3137/smbd -D
tcp        0      0 0.0.0.0:65436           0.0.0.0:*               LISTEN	15119/initd
tcp        0      0 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED	15119/initd
tcp        0      9 192.168.1.79:1149       64.62.96.42:6667        ESTABLISHED	15119/initd
tcp        0      0 192.168.1.79:1146       199.184.165.133:6667    ESTABLISHED	15119/initd
udp        0      0 192.168.1.79:137        0.0.0.0:*	850/nmbd
udp        0      0 0.0.0.0:137             0.0.0.0:*		850/nmbd
udp        0      0 192.168.1.79:138        0.0.0.0:*	850/nmbd
udp        0      0 0.0.0.0:138             0.0.0.0:*		850/nmbd
udp        0      0 0.0.0.0:3049            0.0.0.0:*		25239/xopen
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name	Path
unix  2      [ ACC ]     STREAM     LISTENING     943    778/gpm	/dev/gpmctl
unix  4      [ ]         DGRAM                    7984   3247/syslogd	/dev/log
unix  2      [ ]         DGRAM                    15679  732/xinetd
unix  2      [ ]         DGRAM                    7993   3252/klogd
unix  2      [ ]         DGRAM                    1078   893/login -- root
unix  2      [ ]         DGRAM                    990    820/crond
unix  2      [ ]         DGRAM                    924    759/sendmail: accep
unix  2      [ ]         DGRAM                    834    677/identd
unix  2      [ ]         DGRAM                    804    657/apmd
unix  2      [ ]         STREAM     CONNECTED     417    1/init
Active IPX sockets
Proto Recv-Q Send-Q Local Address              Foreign Address            State


Starting from the top and working down:
Port 2003 process ID 3137 is unusual because it's associated with a program name
that makes it appear it's related to samba.

Port 65336 process ID 15119 is unusual because it's a very high port that's in a listen state on the compromised system.

Port 3128 process ID 25241 is unusual because the port is not found in /etc/services and the xopen program was found in the in ital analysis to be running from a hidden directory.

Port 65436 process ID 15119 is unusual because it's a very high port that's not
found in /etc/services and it's also in a listen state.  Its association with
initd is also suspect as it is not a default Red Hat port.

Port 65336 process ID 15119 is connected to IP address 213.154.118.200

Port 1149 process ID 15119 is connected to IP 64.62.96.42 on port 6667 - a port typically associated with IRC.

Port 1146 process ID 15119 is connected to IP 199.184.165.133 port 6667 and is unusual for previously mentioned reasons.

Port 3049 process ID 25239 is unusual because it's not in /etc/services and is associated with a suspect program - xopen.

Process 3137 on port 443 was unusual not because of the port, but because of its  name, smbd -D.

These processes have been included with this analysis.


The lsof utility was also used to capture the open ports and files on the
system.  The lsof (list open files) utility is nearly ideal when dealing with a
system that has a rootkit installed.  The September 2003 edition of SysAdmin
magazine feature an article on rootkit detection by Eric Cole.  In the article
Cole states that most rootkits that are known today are unable to
deal with the way lsof works.  The output from the lsof command is
available in Appendix B due to its length.  When the
output is lsof is parsed (with grep) for the word "LISTEN" a slightly different
picture emerges of the ports that are open that are unusual.

Output from lsof | grep LISTEN

COMMAND     PID USER   FD   TYPE     DEVICE     SIZE       NODE NAME

identd      677 root    4u  IPv4        836                 TCP *:auth (LISTEN)
identd      685 root    4u  IPv4        836                 TCP *:auth (LISTEN)
identd      686 root    4u  IPv4        836                 TCP *:auth (LISTEN)
identd      695 root    4u  IPv4        836                 TCP *:auth (LISTEN)
identd      696 root    4u  IPv4        836                 TCP *:auth (LISTEN)
sshd        699 root    3u  IPv4        860                 TCP *:ssh (LISTEN)
xinetd      732 root    3u  IPv4        881                 TCP *:finger(LISTEN)
xinetd      732 root    4u  IPv4        882                 TCP *:telnet(LISTEN)
xinetd      732 root    5u  IPv4        883                 TCP *:ftp (LISTEN)
sendmail    759 root    4u  IPv4        925                 TCP localhost.localdomain:smtp (LISTEN)
smbd        845 root    9u  IPv4       1015                 TCP *:netbios-ssn(LISTEN)
smbd       3137 root    6u  IPv4       4571                 TCP *:cfinger(LISTEN)
smbd       3137 root   16u  IPv4        976                 TCP *:https (LISTEN)
smbd       3137 root   17u  IPv4        977                 TCP *:http (LISTEN)
(swapd)    3153 root   16u  IPv4        976                 TCP *:https (LISTEN)
(swapd)    3153 root   17u  IPv4        977                 TCP *:http (LISTEN)
initd     15119 root    3u  IPv4      15617                 TCP *:65336 (LISTEN)
initd     15119 root    5u  IPv4      15619                 TCP *:65436 (LISTEN)
xopen     25239 root   16u  IPv4        976                 TCP *:https (LISTEN)
xopen     25239 root   17u  IPv4        977                 TCP *:http (LISTEN)
xopen     25241 root    8u  IPv4      12302                 TCP *:squid (LISTEN)
xopen     25241 root   16u  IPv4        976                 TCP *:https (LISTEN)
xopen     25241 root   17u  IPv4        977                 TCP *:http (LISTEN)
lsn       25247 root   16u  IPv4        976                 TCP *:https (LISTEN)
lsn       25247 root   17u  IPv4        977                 TCP *:http (LISTEN)


Output of lsof | grep UDP

COMMAND     PID USER   FD   TYPE     DEVICE     SIZE       NODE NAME

nmbd        850 root    6u  IPv4       1025                 UDP *:netbios-ns
nmbd        850 root    7u  IPv4       1026                 UDP *:netbios-dgm
nmbd        850 root    8u  IPv4       1028                 UDP 192.168.1.79:netbios-ns
nmbd        850 root    9u  IPv4       1029                 UDP 192.168.1.79:netbios-dgm
xopen     25239 root    8u  IPv4       9972                 UDP *:3049





4. Were there any active network connections? If so, what address(es) was the
other end and what service(s) was it for?

There were several active network connections that were connected to remote IP
addresses.

The IP address, 213.154.118.200 port 1188 was connected to 192.168.1.79 port
65336.

The IP address, 64.62.96.42 port 6667 was connected to 192.168.1.79 port 1149.

The IP address, 199.184.165.133 port 6667 was connected to 192.168.1.79 port 1146.

The IP address 192.168.1.79 is the IP address of the compromised system.

The IP address, 213.154.118.200 was run through nslookup which resolved the IP
address to sanido-08.is.pcnet.ro.  RO is the country code for Romania.
A whois query was run against the pcnet.ro domain.  The results of the whois query are located in Appendix C.  The domain was found to be associated with a Romanian ISP.

One address shown in the netstat display was, 213.154.118.200:1188.  (The number following the colon is the port number on the foreign machine.)  The address on the other end
of 213.154.118.200 was 192.168.1.79:65336.  The 192.168.1.79 address
corresponds to the IP address of the compromised system while the port 65336
corresponds to the port where the psybnc process was listening.  Googling for
psybnc revealed that it is an IRC bot used to maintain ops on an IRC server.


Nslookup was unable to resolve the IP 64.62.96.42.  Port 6667 which was used by
the IP address is often associated with IRC.  To gain additional
information on this IP address a traceroute was executed with the last several
hops shown below.

gar1-p370.phmaz.ip.att.net (12.123.142.25)  55.239 ms  55.333 ms  62.860 ms
ge1-3-1000m.cr1.brdr.phx.puregig.net (12.127.141.70)  68.162 ms  67.950 ms 71.219 ms
ge1-1-1000m.jc-01.phx1.puregig.net (140.99.96.102)  68.413 ms  67.834 ms  67.917ms
64.62.96.6 (64.62.96.6)  70.387 ms  68.129 ms  67.862 ms
* * *
* * *

The asterisks could indicate that the traceroute tried to traverse a device such
as a firewall that was not returning TTL information.

The address was also run through ARIN to determine who owned the netblock.  The
results of the query are shown below.


OrgName:    Axient Communications, Inc.
OrgID:      AXNT
Address:    8936 N. Central Avenue
City:       Phoenix
StateProv:  AZ
PostalCode: 85020
Country:    US

NetRange:   64.62.0.0 - 64.62.127.255
CIDR:       64.62.0.0/17
NetName:    AXIENT-1
NetHandle:  NET-64-62-0-0-1
Parent:     NET-64-0-0-0-0
NetType:    Direct Allocation
NameServer: DNS.LAX.LLNW.NET
NameServer: DNS.PHX1.LLNW.NET
Comment:    ADDRESSES WITHIN THIS BLOCK ARE NON-PORTABLE
RegDate:    2000-03-14
Updated:    2002-06-19

TechHandle: WP5-ARIN
TechName:   Petrisko, William
TechPhone:  +1-602-850-3089
TechEmail:  billp@wjp.net



The IP address 199.184.165.133 also used port 6667.  This IP address was
resolved to, undernet.irc.rcn.net.
The rcn.net domain was run through whois which revealed that the domain was
maintained by entities on the east coast of the US.  The full output from the
whois query can be found in Appendix F.



5. How many instances of an SSH server were installed and at what times?

The command, rpm -qa --last was run then grepped for the word ssh.  The full
output from the rpm query can be found in Appendix G.  This
indicates that the original ssh server was installed on July 14, 2003.

openssh-server-2.9p2-7                        Mon 14 Jul 2003 01:54:37 PM PDT

There were four ssh servers running on the system; three of which were installed
by the attacker.  The attacker ran the servers on various ports gave the
processes different names in a effort to avoid detection. The attacker installed
their ssh servers on August 10, 2003 around 15:32 PDT.




6. Which instances of the SSH servers from question 5 were run?

There were two instances of xopen and one instance of a process named smbd -D.
Cat and netcat were used to extract PIDs 25241, 25239, and 3137 from the running
system.

A strings output on each binary revealed that they were ssh servers.

The binaries have been included with this analysis.


7. Did any of the SSH servers identified in question 5 appear to have been
modified to collect unique information? If so, was any information collected?

A file named mfs was found in the directory /lib/.x/s/.  The log appeared to
contain only information about scan and connection attempts.


8. Which system executables (if any) were trojaned and what configuration files
did they use?

This question was interpreted in terms of a rootkit to mean which system
executables have or may have been replaced by a rootkit.

Chkrootkit identified five system files as being trojaned:  ifconfig, ls,
netstat, ps, and top.  Three files were used as configuration files for the
trojaned binaries:  /dev/ttyoa, /dev/ttyob, and /dev/ttyop.

A combination of cat and netcat were used to remove the binaries from the
compromised system while it was running.  These files were later processed
through strings to determine which files they used for configuration.  The three
tty files were found to contain commands, partial IP addresses, and what
appeared to be port numbers or process IDs.

Strings found the following relationship between the tty files and the trojaned
binaries:
ps and top used ttyop
ls used ttyof
netstat used ttyoa
ifconfig did not appear to use any of the the tty files.

These trojaned processes and config files are included in the analysis.

9. How and from where was the system likely compromised?

Using the timeline features of Autopsy/Sleuthkit the system appears to have been
compromised by exploiting a vulnerable wu-ftp
server.  The output from the command,

rpm -qa --last | grep -i ftp

revealed that wu-ftpd-2.6.1-18 was installed on Monday 14 July 2003 at 1:54:43
PM PDT.

The phrase, "wu-ftpd-2.6.1-18 cve" was used with the Google search engine to
locate exploits.  The search led to this page:

http://www.securityfocus.com/bid/8315/info/

Included at the above URL were links to exploit code that could be used to
execute arbitrary commands on a vulnerable system.

Examining /etc/xinetd.d/wu-ftpd confirmed that the ftp server was configured to
run as root.  A buffer overflow exploit used against this ftp server could have
presented the attacker a root shell.  Attackers will often bind the
root shell to a port then connect to the port and have the ability to execute
commands as the root user.

A binary for carrying out a ptrace exploit was found in /dev/shm/k.  This
exploit would send an email to a Yahoo! account once executed.

The psybnc.log file found under /etc/opt/psybnc/log showed connections from
Romanian and US hosts.

The attacker linked /root/.bash_history to /dev/null, a common technique used by
attackers in an effort to avoid leaving a command history behind.  Another copy
of the .bash_history file was located at / and contained the domain, apu.edu.
From the context in the bash history file it appears that this domain was where
the honeypot was deployed.  The whois command was run on
this domain, the output can be found in Appendix D.

Also found in the .bash_history file was the domain, izolam.net.  A whois lookup
was run on izolam.net, the full output can be found in Appendix E.  The
attacker used wget to download a file that was used to modify the webserver.

The attacker also removed /var/mail/root and killed the running apache
processes.  The attacker may have been inspired to remove the mail file after
viewing bounced emails from Yahoo! or from the crontab reports that the sa file
was missing.



Bonus Question:
What nationality do you believe the attacker(s) to be, and why?
The attackers appear to be Russian or Romanian.  The IP addresses found in
various files on the compromised host all point back to addresses in the
European Union, specifically, Romania.  The other evidence that points toward
Russian or Romanian (or other formerly communist country) is that the usernames
created on the IRC bot that was installed on the compromised system all had a
theme -- kgb, red code, and sic (short for sickle perhaps?)


Additional information:

Although chkrootkit did not detect it, a strings analysis of the sk binary found
on the compromised system suggests that it is the suckit rootkit version 1.3b.
In a recent Sysadmin article, Eric Cole stated that suckit was one of the most
sophisticated rootkits known because of the techniques it uses to avoid
detection.

The attacker made some effort to interfere with the system's logging.  The same
care was not exercised on the files and tools the attacker left behind.  The
attacker also left the RPM database intact which, when queried with rpm -a
--verify (Appendix H) showed files that had been tampered with as indicated by
mismatches in the md5 checksums.

Autopsy showed that vanish2.tgz had been deleted.  A Google search led to

http://packetstormsecurity.nl/UNIX/penetration/log-wipers/vanish2.tgz

From the Packetstorm web site:
Vanish is a log wiper that cleans WTMP, UTMP, lastlog, messages, secure,
xferlog, maillog, warn, mail, httpd.access_log, and httpd.error_log. Tested
under SuSE. Changes: Now uses the rootkit fix program to preserve ctimes and log
ownership/permissions. By Neo

Most of the log files had been scrubbed, deleted, or linked to /dev/null.

The file, /var/log/secure was found to contain the IP address 193.109.122.5
which nslookup resolved to the host, proxyscan.undernet.org Google had no
information on this particular host but if the hostname is an indication, it is
a machine that can be used to scan the Internet.
A traceroute was run in an effort to approximately locate this host.  A partial
result of the traceroute to proxyscan.undernet.org is shown here:

TE2-1-pos030.nl.lambdanet.net (217.71.96.102)
Business-Internet-Trends-BV-BIT.nl.lambdanet.net (217.71.99.38)
jun1.kelvin.network.bit.nl (213.136.31.6)
proxyscan.undernet.org (193.109.122.5)

From the www.undernet.org website:
The Undernet is one of the largest realtime chat networks in the world, with
approximately 45 servers connecting over 35 countries and serving more than
1,000,000 people weekly.

I've excluded most of the details of the use of Autopsy, the Sleuthkit, and the techniques used to obtain  forensically sound images  from the VMware session.  In short, the VMware machine can be configured to boot from CD-ROM -- that's what the Red Hat 8 CD was for.  Then dd and netcat, run from the CD were used to extract /dev/sda1 and /dev/sda2 (the swap partition) for analysis.  Explaining those tasks could have easily added another ten pages.  


References

Cole, Eric.  SysAdmin. September 2003. "Detecting Rootkits"

http://www.redhat.com/archives/enigma-list/2002-October/msg00432.html  -- page
describing a compromise similar to this one.

http://hysteria.sk/sd/sk/ -- web site where the SuckIT root kit can be obtained.









Appendix A

Output from chkrootkit from the compromised system

ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not infected
Checking `gpm'... not infected
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... INFECTED
Checking `inetd'... not tested
Checking `inetdconf'... not found
Checking `identd'... not infected
Checking `init'... not infected
Checking `killall'... not infected
Checking `ldsopreload'... not infected
Checking `login'... not infected
Checking `ls'... INFECTED
Checking `lsof'... not found
Checking `mail'... not infected
Checking `mingetty'... not infected
Checking `netstat'... INFECTED
Checking `named'... not found
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... INFECTED
Checking `pstree'... not infected
Checking `rpcinfo'... not infected
Checking `rlogind'... not infected
Checking `rshd'... not infected
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... not found
Checking `tcpdump'... not infected
Checking `top'... INFECTED
Checking `telnetd'... not infected
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `vdir'... not infected
Checking `w'... not infected
Checking `write'... not infected
Checking `aliens'...
/dev/ttyop /dev/ttyoa
Searching for sniffer's logs, it may take a while... nothing found
Searching for HiDrootkit's default dir... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... nothing found
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/5.6.0/i386-linux/.packlist /lib/.x /lib/.x/.boot
/lib/.x
Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for Sadmind/IIS Worm... nothing found
Searching for MonKit... nothing found
Searching for Showtee... nothing found
Searching for OpticKit... nothing found
Searching for T.R.K... nothing found
Searching for Mithra... nothing found
Searching for OBSD rk v1... nothing found
Searching for LOC rootkit ... nothing found
Searching for Romanian rootkit ... nothing found
Searching for HKRK rootkit ... nothing found
Searching for Suckit rootkit ... nothing found
Searching for Volc rootkit ... nothing found
Searching for Gold2 rootkit ... nothing found
Searching for TC2 Worm default files and dirs... nothing found
Searching for Anonoying rootkit default files and dirs... nothing found
Searching for ZK rootkit default files and dirs... nothing found
Searching for ShKit rootkit default files and dirs... nothing found
Searching for anomalies in shell history files... Warning:
`//root/.bash_history' is linked to another file
Checking `asp'... not infected
Checking `bindshell'... INFECTED (PORTS:  3049)
Checking `lkm'... You have     4 process hidden for ps command
Warning: Possible LKM Trojan installed
Checking `rexedcs'... not found
Checking `sniffer'... Checking `w55808'... not infected
Checking `wted'... nothing deleted
Checking `scalper'... not infected
Checking `slapper'... not infected
Checking `z2'... not infected


Appendix B

Output from lsof -i from the compromised system

COMMAND    PID USER   FD   TYPE DEVICE SIZE NODE NAME
identd     677 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     685 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     686 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     695 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     696 root    4u  IPv4    836       TCP *:auth (LISTEN)
sshd       699 root    3u  IPv4    860       TCP *:ssh (LISTEN)
xinetd     732 root    3u  IPv4    881       TCP *:finger (LISTEN)
xinetd     732 root    4u  IPv4    882       TCP *:telnet (LISTEN)
xinetd     732 root    5u  IPv4    883       TCP *:ftp (LISTEN)
sendmail   759 root    4u  IPv4    925       TCP localhost.localdomain:smtp
(LISTEN)
smbd       845 root    9u  IPv4   1015       TCP *:netbios-ssn (LISTEN)
nmbd       850 root    6u  IPv4   1025       UDP *:netbios-ns
nmbd       850 root    7u  IPv4   1026       UDP *:netbios-dgm
nmbd       850 root    8u  IPv4   1028       UDP 192.168.1.79:netbios-ns
nmbd       850 root    9u  IPv4   1029       UDP 192.168.1.79:netbios-dgm
smbd      3137 root    6u  IPv4   4571       TCP *:cfinger (LISTEN)
smbd      3137 root   16u  IPv4    976       TCP *:https (LISTEN)
smbd      3137 root   17u  IPv4    977       TCP *:http (LISTEN)
(swapd)   3153 root   16u  IPv4    976       TCP *:https (LISTEN)
(swapd)   3153 root   17u  IPv4    977       TCP *:http (LISTEN)
initd    15119 root    3u  IPv4  15617       TCP *:65336 (LISTEN)
initd    15119 root    5u  IPv4  15619       TCP *:65436 (LISTEN)
initd    15119 root    6u  IPv4  16157       TCP
192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd    15119 root   11u  IPv4  18904       UDP
192.168.1.79:1030->192.168.1.1:domain
xopen    25239 root    8u  IPv4   9972       UDP *:3049
xopen    25239 root   16u  IPv4    976       TCP *:https (LISTEN)
xopen    25239 root   17u  IPv4    977       TCP *:http (LISTEN)
xopen    25241 root    8u  IPv4  12302       TCP *:squid (LISTEN)
xopen    25241 root   16u  IPv4    976       TCP *:https (LISTEN)
xopen    25241 root   17u  IPv4    977       TCP *:http (LISTEN)
lsn      25247 root   16u  IPv4    976       TCP *:https (LISTEN)
lsn      25247 root   17u  IPv4    977       TCP *:http (LISTEN)


Appendix C

[whois.rotld.ro]

% whois.rotld.ro :
%
% Rights restricted by copyright.
%
% Specifically, this data MAY ONLY be used for Internet operational
%   purposes. It may not be used for targeted advertising or any
%   other purpose.
%
% Este INTERZISA folosirea datelor de pe acest server in oricare
%   alt scop decat operarea retelei. In special este INTERZISA
%   folosirea lor in scopuri publicitare.
%


domain-name: pcnet.ro
description: PC-NET Data Network
admin-contact: MB51-ROTLD
technical-contact: AN160-ROTLD
zone-contact: AB494-ROTLD
nameserver:  ns1.pcnet.ro 213.154.128.1
nameserver:  ns2.pcnet.ro 213.154.128.2
nameserver:  ns3.pcnet.ro 213.154.128.3
info:        object maintained by ro.rnc local registry
info:        Register your .ro domain names at www.rotld.ro
notify:      domain-admin@listserv.rnc.ro
object-maintained-by: ROTLD-MNT
mnt-lower:   ROTLD-MNT
updated:     domain-admin@listserv.rnc.ro 19970519
updated:     ciprian@rnc.ro 19990601
updated:     ciprian@rnc.ro 19991207
updated:     cristih@rnc.ro 20000829
source:      ROTLD

person:      Mihai Batraneanu
address:     PC-NET Data Network S.A.
address:     Splaiul Unirii 10, bl B5, sc2, et 1
address:     Bucharest, Romania
phone:       +40-21-330 28 01
fax-no:      +40-21-330 28 42
e-mail:      mihai@pcnet.ro
nic-hdl:     MB51-ROTLD
info:        object maintained by ro.rnc local registry
notify:      domain-admin@listserv.rnc.ro
object-maintained-by: ROTLD-MNT
updated:     danacorb@sunu.rnc.ro 19970901
updated:     ciprian@rnc.ro 19991207
source:      ROTLD

person:      Alina-Mihaela Nemes
address:     PCNET DATA NETWORK SA
address:     Bd. Mircea Eliade, nr. 18
address:     Bucharest, Romania
phone:       +40-21-2080460
fax-no:      +40-21-2080461
e-mail:      alina@pcnet.ro
nic-hdl:     AN160-ROTLD
info:        object maintained by ro.rnc local registry
notify:      domain-admin@listserv.rnc.ro
object-maintained-by: ROTLD-MNT
updated:     danacorb@sunu.rnc.ro 19970901
updated:     ciprian@rnc.ro 19991207
updated:     imanea@rnc.ro 19991207
source:      ROTLD

person:      Adrian Batraneanu
address:     PC-NET Data Network S.A.
address:     Splaiul Unirii 10, bl B5, sc2, et 1
address:     Bucharest, Romania
phone:       +40-21-330 28 01
fax-no:      +40-21-330 28 42
e-mail:      adi@pcnet.ro
nic-hdl:     AB494-ROTLD
info:        object maintained by ro.rnc local registry
notify:      domain-admin@listserv.rnc.ro
object-maintained-by: ROTLD-MNT
updated:     danacorb@sunu.rnc.ro 19970901
updated:     ciprian@rnc.ro 19991207
source:      ROTLD



Appendix D

Whois output from the apu.edu domain

Running the command, whois apu.edu produced the following:

   Domain Name: APU.EDU
   Registrar: EDUCAUSE
   Whois Server: whois.educause.net
   Referral URL: http://www.educause.edu/edudomain
   Name Server: CBRU.BR.NS.ELS-GMS.ATT.NET
   Name Server: CMTU.MT.NS.ELS-GMS.ATT.NET
   Name Server: NS.APU.EDU
   Status: ACTIVE
   Updated Date: 04-may-2003
   Creation Date: 03-may-1994
   Expiration Date: 03-may-2004


--------------------------

Domain Name: APU.EDU

Registrant:
   Azusa Pacific University
   PO Box 7000
   Azusa, CA 91702-7000
   UNITED STATES

Contacts:

   Administrative Contact:
   John Reynolds
   Chief Information Officer
   Azusa Pacific University
   PO Box 7000
   Azusa, CA 91702-7000
   UNITED STATES
   (626) 969-3434
   jreynolds@apu.edu


   Technical Contact:
   James Stoker
   Network Administrator
   Azusa Pacific University
   PO Box 7000
   Azusa, CA 91702-7000
   UNITED STATES
   (626) 969-3434
   jstoker@apu.edu


Name Servers:
   NS.APU.EDU                   199.184.237.168
   CBRU.BR.NS.ELS-GMS.ATT.NET
   CMTU.MT.NS.ELS-GMS.ATT.NET

Domain record activated:    03-May-1994
Domain record last updated: 13-Aug-2002


Appendix E

Output from a whois lookup run against a domain found in a copy of .bash_history

Whois output from izolam.net:

   Domain Name: IZOLAM.NET
   Registrar: NETWORK SOLUTIONS, INC.
   Whois Server: whois.networksolutions.com
   Referral URL: http://www.networksolutions.com
   Name Server: NS6.READYHOSTING.COM
   Name Server: NS5.READYHOSTING.COM
   Status: ACTIVE
   Updated Date: 10-may-2003
   Creation Date: 10-may-2003
   Expiration Date: 10-may-2004


Registrant:
ReadyHosting, Inc, ReadyHosting, Inc (LIKTBZWGKD)
   6127 Green Bay Rd
   Suite 400
   Kenosha, WI 53142
   US

   Domain Name: IZOLAM.NET

   Administrative Contact:
      ReadyHosting, Inc, ReadyHosting, Inc  (COKGNLAFEI) sales@readyhosting.com
      6127 Green Bay Rd
      Suite 400
      Kenosha, WI 53142
      US
      262-652-7640
   Technical Contact:
      Ready Hosting Inc.  (TWHBUQCCZO)          sysadmin@readyhosting.com
      6127 Green Bay Road
      Suite 400
      Kenosha, WI 53142
      US
      262-652-7640 fax: 262-652-7650

   Record expires on 10-May-2004.
   Record created on 10-May-2003.

   Domain servers in listed order:

   NS5.READYHOSTING.COM         63.99.209.103
   NS6.READYHOSTING.COM         63.99.209.104


Apppendix F

Whois output from rcn.net

[whois.crsnic.net]

Whois Server Version 1.3

Domain names in the .com and .net domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

   Domain Name: RCN.NET
   Registrar: NETWORK SOLUTIONS, INC.
   Whois Server: whois.networksolutions.com
   Referral URL: http://www.networksolutions.com
   Name Server: AUTH3.DNS.RCN.NET
   Name Server: AUTH1.DNS.RCN.NET
   Name Server: AUTH2.DNS.RCN.NET
   Name Server: AUTH4.DNS.RCN.NET
   Status: ACTIVE
   Updated Date: 12-jun-2002
   Creation Date: 02-may-1995
   Expiration Date: 03-may-2011


Registrant:
Residential Communications Network (RCN-DOM)
   105 Carnegie Center
   Princeton, NJ 08540
   US

   Domain Name: RCN.NET

   Administrative Contact:
      RCN  (ETS3-ORG)		abuse@RCN.COM
      7921 WOODRUFF CT
      SPRINGFIELD, VA 22151-2108
      US
      703-321-8000 fax: 703-321-8316
   Technical Contact:
        (EROLS-NOC)		domreg@RCN.COM
      RCN
      7921 Woodruff Court
      Springfield, VA 22151
      US
      703-321-8000 fax: 123 123 1234

   Record expires on 03-May-2011.
   Record created on 08-Oct-2002.
   Database last updated on 27-Sep-2003 17:18:49 EDT.

   Domain servers in listed order:

   AUTH1.DNS.RCN.NET            207.172.3.20
   AUTH3.DNS.RCN.NET            207.172.3.21
   AUTH4.DNS.RCN.NET            207.172.3.22
   AUTH2.DNS.RCN.NET            207.172.3.20



Appendix G

Output from the command rpm -qa --last

dev86-0.15.5-1                                Wed 16 Jul 2003 10:18:43 AM PDT
gcc-2.96-98                                   Wed 16 Jul 2003 10:17:49 AM PDT
glibc-devel-2.2.4-13                          Wed 16 Jul 2003 10:17:45 AM PDT
binutils-2.11.90.0.8-9                        Wed 16 Jul 2003 10:17:35 AM PDT
kernel-headers-2.4.7-10                       Wed 16 Jul 2003 10:16:08 AM PDT
cpp-2.96-98                                   Wed 16 Jul 2003 10:16:06 AM PDT
tcpdump-3.6.2-9                               Mon 14 Jul 2003 01:54:50 PM PDT
mod_ssl-2.8.4-9                               Mon 14 Jul 2003 01:54:48 PM PDT
mod_dav-1.0.2-6                               Mon 14 Jul 2003 01:54:48 PM PDT
apache-1.3.20-16                              Mon 14 Jul 2003 01:54:47 PM PDT
mm-1.1.3-1                                    Mon 14 Jul 2003 01:54:46 PM PDT
curl-7.8-1                                    Mon 14 Jul 2003 01:54:46 PM PDT
wu-ftpd-2.6.1-18                              Mon 14 Jul 2003 01:54:43 PM PDT
anonftp-4.0-9                                 Mon 14 Jul 2003 01:54:43 PM PDT
samba-client-2.2.1a-4                         Mon 14 Jul 2003 01:54:42 PM PDT
samba-2.2.1a-4                                Mon 14 Jul 2003 01:54:40 PM PDT
samba-common-2.2.1a-4                         Mon 14 Jul 2003 01:54:39 PM PDT
telnet-server-0.17-20                         Mon 14 Jul 2003 01:54:38 PM PDT
sysstat-4.0.1-2                               Mon 14 Jul 2003 01:54:38 PM PDT
rsh-server-0.17-5                             Mon 14 Jul 2003 01:54:37 PM PDT
openssh-server-2.9p2-7                        Mon 14 Jul 2003 01:54:37 PM PDT
links-0.96-2                                  Mon 14 Jul 2003 01:54:37 PM PDT
finger-server-0.17-9                          Mon 14 Jul 2003 01:54:37 PM PDT
wget-1.7-3                                    Mon 14 Jul 2003 01:54:36 PM PDT
traceroute-1.4a12-1                           Mon 14 Jul 2003 01:54:36 PM PDT
telnet-0.17-20                                Mon 14 Jul 2003 01:54:36 PM PDT
stunnel-3.19-1                                Mon 14 Jul 2003 01:54:35 PM PDT
rsh-0.17-5                                    Mon 14 Jul 2003 01:54:34 PM PDT
pidentd-3.0.14-1                              Mon 14 Jul 2003 01:54:34 PM PDT
openssh-clients-2.9p2-7                       Mon 14 Jul 2003 01:54:33 PM PDT
openssh-2.9p2-7                               Mon 14 Jul 2003 01:54:33 PM PDT
nmap-2.54BETA22-3                             Mon 14 Jul 2003 01:54:33 PM PDT
ncftp-3.0.3-6                                 Mon 14 Jul 2003 01:54:32 PM PDT
gnupg-1.0.6-3                                 Mon 14 Jul 2003 01:54:32 PM PDT
ftp-0.17-12                                   Mon 14 Jul 2003 01:54:31 PM PDT
finger-0.17-9                                 Mon 14 Jul 2003 01:54:31 PM PDT
bind-utils-9.1.3-4                            Mon 14 Jul 2003 01:54:31 PM PDT
zip-2.3-10                                    Mon 14 Jul 2003 01:54:30 PM PDT
xinetd-2.3.3-1                                Mon 14 Jul 2003 01:54:30 PM PDT
make-3.79.1-8                                 Mon 14 Jul 2003 01:54:30 PM PDT
pspell-0.12.2-3                               Mon 14 Jul 2003 01:54:29 PM PDT
vixie-cron-3.0.1-63                           Mon 14 Jul 2003 01:54:28 PM PDT
libtool-libs-1.4-8                            Mon 14 Jul 2003 01:54:28 PM PDT
anacron-2.3-17                                Mon 14 Jul 2003 01:54:28 PM PDT
timeconfig-3.2.2-1                            Mon 14 Jul 2003 01:54:27 PM PDT
quota-3.01pre9-3                              Mon 14 Jul 2003 01:54:27 PM PDT
pciutils-2.1.8-23                             Mon 14 Jul 2003 01:54:27 PM PDT
lokkit-0.50-6                                 Mon 14 Jul 2003 01:54:26 PM PDT
kernel-2.4.7-10                               Mon 14 Jul 2003 01:54:13 PM PDT
iptables-1.2.3-1                              Mon 14 Jul 2003 01:54:08 PM PDT
ipchains-1.3.10-10                            Mon 14 Jul 2003 01:54:07 PM PDT
apmd-3.0final-34                              Mon 14 Jul 2003 01:54:06 PM PDT
initscripts-6.40-1                            Mon 14 Jul 2003 01:54:05 PM PDT
util-linux-2.11f-9                            Mon 14 Jul 2003 01:54:04 PM PDT
rpm-4.0.3-1.03                                Mon 14 Jul 2003 01:54:03 PM PDT
zlib-1.1.3-24                                 Mon 14 Jul 2003 01:54:02 PM PDT
SysVinit-2.78-19                              Mon 14 Jul 2003 01:54:01 PM PDT
sendmail-8.11.6-3                             Mon 14 Jul 2003 01:53:56 PM PDT
openldap-2.0.11-13                            Mon 14 Jul 2003 01:53:54 PM PDT
krb5-libs-1.2.2-13                            Mon 14 Jul 2003 01:53:54 PM PDT
sh-utils-2.0.11-5                             Mon 14 Jul 2003 01:53:53 PM PDT
passwd-0.64.1-7                               Mon 14 Jul 2003 01:53:53 PM PDT
kudzu-0.99.23-1                               Mon 14 Jul 2003 01:53:52 PM PDT
gpm-1.19.3-20                                 Mon 14 Jul 2003 01:53:52 PM PDT
cyrus-sasl-plain-1.5.24-20                    Mon 14 Jul 2003 01:53:52 PM PDT
cyrus-sasl-md5-1.5.24-20                      Mon 14 Jul 2003 01:53:51 PM PDT
cyrus-sasl-1.5.24-20                          Mon 14 Jul 2003 01:53:51 PM PDT
authconfig-4.1.19-1                           Mon 14 Jul 2003 01:53:51 PM PDT
pam-0.75-14                                   Mon 14 Jul 2003 01:53:50 PM PDT
cracklib-dicts-2.7-12                         Mon 14 Jul 2003 01:53:49 PM PDT
words-2-17                                    Mon 14 Jul 2003 01:53:48 PM PDT
which-2.12-3                                  Mon 14 Jul 2003 01:53:48 PM PDT
vim-minimal-5.8-7                             Mon 14 Jul 2003 01:53:48 PM PDT
vim-common-5.8-7                              Mon 14 Jul 2003 01:53:47 PM PDT
utempter-0.5.2-6                              Mon 14 Jul 2003 01:53:44 PM PDT
tmpwatch-2.8-2                                Mon 14 Jul 2003 01:53:43 PM PDT
time-1.7-14                                   Mon 14 Jul 2003 01:53:43 PM PDT
mouseconfig-4.23-1                            Mon 14 Jul 2003 01:53:43 PM PDT
mkbootdisk-1.4.2-3                            Mon 14 Jul 2003 01:53:43 PM PDT
mount-2.11g-5                                 Mon 14 Jul 2003 01:53:42 PM PDT
mkinitrd-3.2.6-1                              Mon 14 Jul 2003 01:53:42 PM PDT
lilo-21.4.4-14                                Mon 14 Jul 2003 01:53:42 PM PDT
dev-3.2-5                                     Mon 14 Jul 2003 01:53:39 PM PDT
textutils-2.0.14-2                            Mon 14 Jul 2003 01:53:23 PM PDT
tcsh-6.10-6                                   Mon 14 Jul 2003 01:53:23 PM PDT
tar-1.13.19-6                                 Mon 14 Jul 2003 01:53:22 PM PDT
sysklogd-1.4.1-4                              Mon 14 Jul 2003 01:53:22 PM PDT
slocate-2.6-1                                 Mon 14 Jul 2003 01:53:21 PM PDT
kbdconfig-1.9.14-1                            Mon 14 Jul 2003 01:53:21 PM PDT
console-tools-19990829-36                     Mon 14 Jul 2003 01:53:21 PM PDT
sed-3.02-10                                   Mon 14 Jul 2003 01:53:19 PM PDT
rootfiles-7.2-1                               Mon 14 Jul 2003 01:53:19 PM PDT
redhat-release-7.2-1                          Mon 14 Jul 2003 01:53:19 PM PDT
readline-4.2-2                                Mon 14 Jul 2003 01:53:18 PM PDT
raidtools-0.90-23                             Mon 14 Jul 2003 01:53:18 PM PDT
psmisc-20.1-2                                 Mon 14 Jul 2003 01:53:17 PM PDT
procps-2.0.7-11                               Mon 14 Jul 2003 01:53:16 PM PDT
procmail-3.21-1                               Mon 14 Jul 2003 01:53:16 PM PDT
openssl-0.9.6b-8                              Mon 14 Jul 2003 01:53:15 PM PDT
man-1.5i2-6                                   Mon 14 Jul 2003 01:53:15 PM PDT
less-358-21                                   Mon 14 Jul 2003 01:53:14 PM PDT
gzip-1.3-15                                   Mon 14 Jul 2003 01:53:14 PM PDT
grub-0.90-11                                  Mon 14 Jul 2003 01:53:14 PM PDT
dhcpcd-1.3.18pl8-13                           Mon 14 Jul 2003 01:53:13 PM PDT
ash-0.3.7-2                                   Mon 14 Jul 2003 01:53:13 PM PDT
grep-2.4.2-7                                  Mon 14 Jul 2003 01:53:12 PM PDT
gawk-3.1.0-3                                  Mon 14 Jul 2003 01:53:12 PM PDT
findutils-4.1.7-1                             Mon 14 Jul 2003 01:53:11 PM PDT
at-3.1.8-20                                   Mon 14 Jul 2003 01:53:10 PM PDT
fileutils-4.1-4                               Mon 14 Jul 2003 01:53:09 PM PDT
ed-0.2-21                                     Mon 14 Jul 2003 01:53:09 PM PDT
info-4.0b-3                                   Mon 14 Jul 2003 01:53:08 PM PDT
diffutils-2.7.2-2                             Mon 14 Jul 2003 01:53:08 PM PDT
cpio-2.4.2-23                                 Mon 14 Jul 2003 01:53:08 PM PDT
ncurses-5.2-12                                Mon 14 Jul 2003 01:53:07 PM PDT
modutils-2.4.6-4                              Mon 14 Jul 2003 01:53:04 PM PDT
MAKEDEV-3.2-5                                 Mon 14 Jul 2003 01:53:04 PM PDT
logrotate-3.5.9-1                             Mon 14 Jul 2003 01:53:03 PM PDT
groff-1.17.2-3                                Mon 14 Jul 2003 01:53:03 PM PDT
libstdc++-2.96-98                             Mon 14 Jul 2003 01:53:01 PM PDT
iproute-2.2.4-14                              Mon 14 Jul 2003 01:53:01 PM PDT
hotplug-2001_04_24-11                         Mon 14 Jul 2003 01:53:01 PM PDT
crontabs-1.10-1                               Mon 14 Jul 2003 01:53:00 PM PDT
bzip2-1.0.1-4                                 Mon 14 Jul 2003 01:53:00 PM PDT
bash-2.05-8                                   Mon 14 Jul 2003 01:52:59 PM PDT
termcap-11.0.1-10                             Mon 14 Jul 2003 01:52:58 PM PDT
libtermcap-2.0.8-28                           Mon 14 Jul 2003 01:52:58 PM PDT
syslinux-1.52-2                               Mon 14 Jul 2003 01:52:57 PM PDT
setuptool-1.8-2                               Mon 14 Jul 2003 01:52:57 PM PDT
ntsysv-1.2.24-1                               Mon 14 Jul 2003 01:52:57 PM PDT
netconfig-0.8.11-7                            Mon 14 Jul 2003 01:52:57 PM PDT
slang-1.4.4-4                                 Mon 14 Jul 2003 01:52:56 PM PDT
shadow-utils-20000902-4                       Mon 14 Jul 2003 01:52:56 PM PDT
setserial-2.17-4                              Mon 14 Jul 2003 01:52:56 PM PDT
newt-0.50.33-1                                Mon 14 Jul 2003 01:52:56 PM PDT
reiserfs-utils-3.x.0j-2                       Mon 14 Jul 2003 01:52:55 PM PDT
pwdb-0.61.1-3                                 Mon 14 Jul 2003 01:52:55 PM PDT
popt-1.6.3-1.03                               Mon 14 Jul 2003 01:52:55 PM PDT
perl-5.6.0-17                                 Mon 14 Jul 2003 01:52:54 PM PDT
pcre-3.4-2                                    Mon 14 Jul 2003 01:52:48 PM PDT
parted-1.4.16-8                               Mon 14 Jul 2003 01:52:48 PM PDT
net-tools-1.60-3                              Mon 14 Jul 2003 01:52:48 PM PDT
mktemp-1.5-11                                 Mon 14 Jul 2003 01:52:47 PM PDT
mingetty-0.9.4-18                             Mon 14 Jul 2003 01:52:47 PM PDT
mailx-8.1.1-22                                Mon 14 Jul 2003 01:52:47 PM PDT
losetup-2.11g-5                               Mon 14 Jul 2003 01:52:47 PM PDT
ksymoops-2.4.1-1                              Mon 14 Jul 2003 01:52:47 PM PDT
iputils-20001110-6                            Mon 14 Jul 2003 01:52:47 PM PDT
hdparm-4.1-2                                  Mon 14 Jul 2003 01:52:46 PM PDT
glib-1.2.10-5                                 Mon 14 Jul 2003 01:52:46 PM PDT
gdbm-1.8.0-10                                 Mon 14 Jul 2003 01:52:46 PM PDT
file-3.35-2                                   Mon 14 Jul 2003 01:52:46 PM PDT
eject-2.0.9-2                                 Mon 14 Jul 2003 01:52:45 PM PDT
e2fsprogs-1.23-2                              Mon 14 Jul 2003 01:52:45 PM PDT
dosfstools-2.7-1                              Mon 14 Jul 2003 01:52:45 PM PDT
db3-3.2.9-4                                   Mon 14 Jul 2003 01:52:45 PM PDT
db2-2.4.14-7                                  Mon 14 Jul 2003 01:52:44 PM PDT
db1-1.85-7                                    Mon 14 Jul 2003 01:52:44 PM PDT
cracklib-2.7-12                               Mon 14 Jul 2003 01:52:44 PM PDT
glibc-2.2.4-13                                Mon 14 Jul 2003 01:52:43 PM PDT
chkconfig-1.2.24-1                            Mon 14 Jul 2003 01:52:43 PM PDT
bzip2-libs-1.0.1-4                            Mon 14 Jul 2003 01:52:43 PM PDT
bdflush-1.5-17                                Mon 14 Jul 2003 01:52:43 PM PDT
filesystem-2.1.6-2                            Mon 14 Jul 2003 01:52:39 PM PDT
basesystem-7.0-2                              Mon 14 Jul 2003 01:52:39 PM PDT
setup-2.5.7-1                                 Mon 14 Jul 2003 01:52:38 PM PDT
redhat-logos-1.1.3-1                          Mon 14 Jul 2003 01:52:38 PM PDT
mailcap-2.1.6-1                               Mon 14 Jul 2003 01:52:37 PM PDT
indexhtml-7.2-1                               Mon 14 Jul 2003 01:52:37 PM PDT
glibc-common-2.2.4-13                         Mon 14 Jul 2003 01:52:36 PM PDT


Appendix H

Output from the command rpm -a --verify

.M...... g /var/spool/at/.SEQ
SM5....T   /bin/ps
SM5....T   /usr/bin/top
S.5....T c /etc/issue
S.5....T c /etc/issue.net
.M......   /dev/shm
.M....G.   /dev/tty2
.M....G.   /dev/tty3
.M....G.   /dev/tty4
.M....G.   /dev/tty5
.M....G.   /dev/tty6
S.5....T c /etc/pam.d/system-auth
.......T c /etc/openldap/ldap.conf
.....UG. c /var/lib/rpm/__db.001
.....UG. c /var/lib/rpm/__db.002
S.5....T c /etc/rc.d/init.d/functions
S.5....T c /etc/rc.d/rc.sysinit
.M....G. g /var/log/wtmp
.M......   /dev/shm
missing    /var/log/sa
S.5....T   /boot/kernel.h-2.4.7
..5....T c /etc/mime.types
missing    /var/log/lastlog
S.5....T   /bin/netstat
S.5....T   /sbin/ifconfig
S.5....T   /bin/ls
.......T c /etc/krb5.conf
S.5....T c /etc/mail/statistics
S.5....T c /etc/xinetd.d/chargen
S.5....T c /etc/xinetd.d/chargen-udp
S.5....T c /etc/xinetd.d/daytime
S.5....T c /etc/xinetd.d/daytime-udp
S.5....T c /etc/xinetd.d/echo
S.5....T c /etc/xinetd.d/echo-udp
S.5....T c /etc/xinetd.d/time
S.5....T c /etc/xinetd.d/time-udp
S.5....T c /etc/xinetd.d/finger
S.5....T c /etc/xinetd.d/rexec
S.5....T c /etc/xinetd.d/rlogin
S.5....T c /etc/xinetd.d/rsh
S.5....T c /etc/xinetd.d/telnet
missing    /var/log/samba
S.5....T c /etc/xinetd.d/wu-ftpd
..5....T c /etc/httpd/conf/httpd.conf
missing    /var/log/httpd
S.5....T c /var/www/html/index.html
missing    /var/cache/ssl_gcache_data.dir
missing    /var/cache/ssl_gcache_data.pag
missing    /var/cache/ssl_gcache_data.sem