Project Honeynet Scan of the Month 29

September 2003

Matthew Richard mrichard91@crocker.com

Table of Contents

From http://www.honeynet.org/scans/scan29/:

“On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. Note, this image was recovered from VMware Workstation 4.0, it will not work in older versions. You can download an evaluation copy.”

After downloading the image file and the file of MD5 hashes the first step was to verify them against the published MD5's.

The process of powering on the suspended Vmware image highlighted several differences between the environment in which the Vmware image had been created.  Initially the image was restored on an AMD Athlon processor running Redhat Linux 9.0 and Vmware v4.0.  Unfortunately a live Vmware image will only restore on a similar CPU architecture as that which it was booted in, in this case an Intel CPU.   The image was then restored on an Intel P4 running Windows XP and Vmware v4.0.  The Vmware software has certain configuration settings for devices such as the floppy, cdrom and NIC that map each to a host operating system resource.  For example the Linux version of Vmware maps the cdrom drive to /dev/cdrom whereas the Windows version maps it to d:.  These settings must be changed before the Vmware machine can be restored.  In this investigation the cdrom, floppy and NIC were all changed to the local Windows devices d:, a:, and vmnet1.

All output and images were copied from the Windows XP host to the analysis host running Redhat Linux 9.0.

Questions

Upon starting the Vmware machine several messages appear on the console. Since this evidence was likely to disappear quickly a screenshot was taken using the Vmware “Capture Screen” option. The following initial messages were very suspicious:

The messages were suspicious since swapd does not normally use network functions such as PF_INET or SOCK_PACKET. Also the fact that eth0 had entered promiscuous mode indicated the probable presence of a sniffer.

The next step was to mount a response CD containing trusted system binaries. The binaries on the CD were statically compiled to minimize the trust placed on the possibly compromised system. The first priority is to gather and save volatile system information. The CD was mounted using “mount /dev/cdrom.”

For the purposes of collecting data on a trusted system and not relying on the possibly compromised host operating system, the Vmware host was configured to collect the output of the tools. This environment was created by connecting the Vmware guest OS's network connection to a newly created Vmware host-only network. The host-only network was configured to use the 192.168.1.0/24 network to allow it to communicate with the vmware guest without changes to the guest. As an added protection against possible contamination between the possibly compromised guest OS and the host OS, firewall settings on the host OS were configured to statefully allow only incoming port 50000. For each command run a netcat listener was created on the host to take the incoming network data and redirect it to a text file on the host. This was done using the command “nc -l -p 50000 -s 192.168.1.1 > outputfile.txt”. After each command on the guest was run a new listener was setup on the host.

The volatile data gathered was the output of the commands “lsof”, “lsof -i”, “ifconfig”, “netstat -anp” and “kern_check”. Each command was run using the statically compiled binary from the cdrom in the format of “/mnt/crom/unixforensics/response_kit/linux_x86_static/command | /mnt/crom/unixforensics/response_kit/linux_x86_static/nc 192.168.1.1 50000”.

With the output of these commands saved to a trusted host a brief analysis on the trusted host showed that the live host was probably compromised. This conclusion was based upon the following data:

• The network adapter had been placed in promiscuous mode. This was noted on the console as well as the output of ifconfig.
• The live host had numerous processes listening on non-standard ports. For more information see question 3.
• The output of kern_check showed that the kernel function sys_ni_syscall had been remapped, possibly by a rootkit.

At this point a system administrator would be forced to make a decision to either image the system while it is live or power down the system and image it while offline. This decision would likely be influenced by several factors such as how critical uptime is on the system. For the sake of this exercise it was chosen to power off the system and create an image offline. This decision was made due to the unique circumstances presented by this challenge. In this challenge the compromised host was a suspended image of a live machine. Since the image was suspended in Vmware and is easily copied, it is possible to both restore the system to production and perform an offline image of the filesystem. This would not normally be possible with a production system thus forcing a choice. The method of imaging the filesystem offline is preferable since no data can be changed during the imaging process.

The system was imaged by booting the system with a Redhat 8.0 installation disk. The linux rescue mode was used and the filesystem was mounted in read-only mode. An md5sum of the disk was performed to insure image integrity later using the command “md5sum /dev/sda1”. The host OS was configured to receive the image from the guest using netcat “nc -l -p 50000 > forensic_dd.img”. The guest was imaged using the command “dd if=/dev/sda1 | nc 192.168.1.1 50000”. The image was later mounted on the analysis machine and the md5 checksums were successfully compared. On the analysis machine the image was mounted read-only using the following command “sudo mount -o ro,loop,nodev,noexec,noatime forensic_dd.img /mnt/sepchallenge”. The mounted path is referred to in other sections of this document.

The restoring of the Vmware image and the mounting of the forensics CD generated several log messages due to incompatabilities between the Windows and Linux versions of vmware. The log messages were sent to /var/log/messages which was redirected to /dev/null, however the log data was able to be retrieved from the unallocated disk space on the hard drive image. The log messages may have alerted the attacker to the presence of an investigation.

The act of mounting the cd would also have updated the atime for /bin/mount, /etc/fstab and /dev/cdrom.

In gathering the volatile data from the system the statically compiled binaries used live system resources such as tty, pipes, and network sockets. Details of the resources used can be seen in the lsof output as it transferred its output to the host system.

The statically compiled binaries avoided using any system resources and thus trusting the system except the nc binary. For an unknown reason the nc binary access several system libraries. In this case, since nc was responsible for delivering data across the network, it may have been possible for one of libraries to have been trojaned in memory and thus alter data. An md5sum of the output of the tools could have been done to validate that no data was altered in transit, however this was not done during the investigation.

Answer

PIDs with suspect ports:

• 15119
• 3137
• 25241
• 25239

Method

Using the statically linked netstat binary from the response CD the “netstat -anp” command was executed. The output showed several ports that are not open by default on Redhat 7.2. The TCP ports 65436, 65336, 3128 and 2003 are not default Redhat 7.2 ports. UDP port 2049 is normally used by NFS and is not a default Redhat 7.2 port. A complete listing of the netstat output can be found in Appendix B.

• tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN 15119/initd
• tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN 15119/initd
• tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D
• tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3137/smbd -D
• tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3137/smbd -D
• tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen
• udp 0 0 0.0.0.0:3049 0.0.0.0:* 25239/xopen

Answer

There was an active network connection from 213.154.118.200:1188 to the honeypot on port 65336. This port was identified as suspect in Question 3. The service on this port was “psybnc”. Psybnc is an IRC proxy that helps hide the real IP address of an IRC client.

Description of psybnc from Net Knowledge Base http://www.netknowledgebase.com/tutorials/psybnc.html:

If you know nothing about bncs, a bnc is short for a 'bouncer.' A bnc acts as a proxy for irc, allowing you to hide your real IP address and use a vhost (vanity host - something like 'this.is.a.l33t.vhost.com'). What are the advantages of this? Well, mainly there's just one important one: It'll stop stupid packet kiddies from trying to knock you off the network. Everyone hates getting disconnected, and with a bnc on a decent shell, you should be pretty immune. Remember though: the kiddies can still nuke you, but it is assumed that the shell provider has a high-bandwidth line that allows it to withstand the numerous packets. If your shell is on a 56.6, you'll still be screwed.

Method

The connection is seen in both the netstat output (Appendix B) and the output of “lsof -i” (Appendix C). By viewing the output of lsof (Appendix D) and looking for process 15119 it can be seen that the process does indeed have an active network connection. It also appears that the process has a file named psybnc.log open.

The psybnc log file (Appendix E) shows that psynbc was bound to port 65336.

Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336

Further analysis shows of the psybnc log file shows that the last client connectoin to the psybnc server was sanido-08.is.pcnet.ro. As of 7:30pm on 9/21/2003 sanido-08.is.pcnet.ro resolved to 213.154.118.200.

Answer

4 instances of the SSH server were installed.

1. /usr/sbin/sshd – Sun Aug 10 16:33:57 2003
2. /usr/bin/smbd -D – Sun Aug 10 16:33:33 2003
3. /usr/lib/sp0 – Sun Aug 10 18:30:54 2003
4. /ilb/.x/s/xopen – Sun Aug 10 18:32:16 2003

Method

To find all of the instances of SSH server installed a signature for sshd was needed. The signature could be used to search the filesystem for files containing that signature, similar to a virus scanner. To find a good signature a look through the sshd source was needed. The sshd source contains a usage function with the following line:

fprintf(stderr, "sshd version %s\n", SSH_VERSION);

Using the “sshd version” signature a search of the all files on the system is performed:

Of course this method could be fooled if the sshd daemon was compiled without the usage function.

To find the installation times the creation time of the file inodes are checked. To do this “ls -i” is run on each file. Using the inode number from the ls output debugfs is run to view the ctime of the inode.

Answer

The following SSH servers from question 5 were run:

• /lib/.x/s/xopen
• /usr/bin/smbd -D
• /usr/sbin/sshd

Method

To determine if each instance of the SSH servers from question 5 were run it was necessary to analyze the MAC times of each file. The MAC times for each of the SSH server binaries was obtained using “ls -ial filename” and 'debugfs -R “stat <inode #>” forensics_dd.img'.

The /usr/sbin/sshd binary was run at the last system startup which occurred at 17:34 8/9/2003. The attacker appears to have modified the inodes of a large number of system binaries at 16:33 8/10/2003 which modified the ctime of the binary.

The /usr/lib/sp0 binary was not run. The atime of the binary occurred before the creation time of the binary. This apparent anomaly occurred due the manner in which the mv command updates MAC times. The mv command updates only the ctime of the moved file but not the atime. Because of this the file appears to have been accessed before it was created. See Appendix G for a listing of the MAC times for all 4 binaries.

In addition to the MAC time evidence there was no other evidence that the binary had been run. There was no process running as evidenced by the lack of the PID file defined in the binary's configuration file /usr/lib/sp0_cfg.

Here is a excerpt from the adore installation script, the entire script can be found in Appendix F.

Both /lib/.x/s/xopen and /usr/bin/smbd -D binaries were run and were running at the time the Vmware image was suspended. The xopen binary had 2 copies running with process id's 25239 and 25241. The smbd -D binary had 1 copy running with PID 3137.

Answer

The “/usr/bin/smbd -D” binary appears to have been modified to collect username and password information. It does not appear that any information was collected.

Method

The strings command was run on all of the SSH servers identified in question 5. Several interesting items were discovered in the output of “/usr/bin/smbd -D”.

Excerpt from “strings /usr/bin/smbd\ -D”:

In the process of running strings on the 3 trojaned versions of SSH server it appeared that all 3 had information collecting components. All 3 versions had the following strings:

Since only one of the trojaned SSH servers had the username and password collection routine more research was required. A Google search for “SSH ps laxww” revealed a posting to security focus about possibly trojaned SSH server http://archives.neohapsis.com/archives/sf/sun/2001-q3/0174.html. Konrad Reich writes:

Answer

The following binaries were trojaned:

• /usr/bin/top
• /bin/netstat
• /bin/ps
• /sbin/ifconfig

The following configuration files were used by the trojaned binaries:

• /dev/ttyop
• /dev/ttyoa
• /dev/ttyof

Method

Using the known good MD5 hashes provided by the Honeynet Project I compared them to the MD5 hashes from the compromised filesystem.

These trojaned binaries can are also found in /usr/lib/libshtift:

When comparing the original list of MD5's to the list created from the compromised filesystem 5 new files show up in /dev

A quick find also reveals that these are the only non-device files in the /dev directory besides MAKEDEV.

A look at these files reveals suspicious looking content that could be filters for trojaned binaries. See Appendix A.

Going on this suspicion I searched the trojaned binaries for references to these files. This also tells which configuration files are used by which binary.

Answer

The system appears to have been compromised through a buffer overflow in openssl by the host extreme-service-11.is.pcnet.ro with IP address 213.154.118.219 at 13:24 on 8/10/2003.

The system appears to have had an upatched version of openssl that was not updated with an updated package per the Redhat Errata described in https://rhn.redhat.com/errata/RHSA-2002-155.html.

Method

Using a system MAC timeline the first signs of suspicious activity seem to occur at Aug 10 2003 15:27:36. The MAC timeline was generated using the “mac-daddy.pl” tool written by Rob Lee. The events at 15:27 indicate access to files used by the wu-ftpd package. All previous activity seemed to be related to innocent system activity such as cron jobs.

A search of the Redhat errata for wu-ftpd on Redhat 7.2 shows that there was an updated package to fix a remotely exploitable vulnerability https://rhn.redhat.com/errata/RHSA-2003-245.html. However in the errata notes Redhat states:

Also a search of the web for wu-ftpd exploits that would work on Redhat 7.2 turned up nothing. At this point it seems probable that the FTP daemon was the compromise vector but may have been used after infection.

The attacker had been smart enough to redirect /var/log/messages to /dev/null and delete most other relevant system log information. The next logical step would be to look for log data in unallocated disk space on the system. To obtain all data located in unallocated space the dls tool from the The @stake Sleuth Kit (TASK) is used. The command “/usr/local/task/bin/dls forensic_dd.img >forensic_dd.dls”. From this file only text data would be good for finding deleted log file data. The strings command was run against the dls output.

Since any log data in the unallocated space would contain the date and time a search was performed for the string “Aug 10”. The search revealed the following interesting log messages:

A Google search for “ OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED” revealed several sites that indicated this log message was related to exploitation of the openssl vulnerability. Specifically the AUSCERT site http://www.auscert.org.au/render.html?it=2409&cid=1 describes the presence of these log messages as a sign of possible compromise.

What nationality do you believe the attacker(s) to be, and why?

Answer

The attacker(s) appear to be Romanian.

Method

Several artifacts on the compromised machine indicate that the attackers are Romanian.

First, the IP address that launched the initial attack, 213.154.118.219, belongs to the Pcnet ISP network in Romania (see Appendix H).

Second, the IP address from question 4 that had an active connection, 213.154.118.200 , also belongs to the Pcnet ISP network in Romania.

Finally, all mail artifacts found in the unallocated disk space contain Romanian text. For an example email in Romanian from the unallocated space see Appendix I.  The text from the email was determined to be Romanian through trial and error.  Initially the Google translators for Spanish, Italian, German, French and Portuguese all returned no valid translations for word in the email.  Assuming that the attackers were truly from Romania as their IP addresses suggested a Romanian-English translator was tried.  The dictionary at http://www.dictionare.com/dictionaries/dictionary.htm was able to successfully translate all of the words in the email in Appendix I

Contents of /dev/ttyop:

Contents of /dev/ttyof:

Contents of /dev/ttyoa:

Appendix F – Adore rootkit installation script

% This is the RIPE Whois server.
% The objects are in RPSL format.
%
% Rights restricted by copyright.
% See http://www.ripe.net/ripencc/pub-services/db/copyright.html

inetnum:      213.154.96.0 - 213.154.127.255
netname:      PCNET
descr:        PCNET Data Network S.A.
descr:        PROVIDER ADSL Network
country:      RO
admin-c:      BT17-RIPE
tech-c:       PDNN1-RIPE
status:       ASSIGNED PA
notify:       tudor@pcnet.ro
mnt-by:       AS8503-MNT
changed:      tudor@pcnet.ro 20030704
source:       RIPE

route:        213.154.116.0/22
descr:        PCNET
origin:       AS8503
notify:       tudor@pcnet.ro
mnt-by:       AS8503-MNT
changed:      tudor@pcnet.ro 20020912
source:       RIPE

role:         PCNET Data Network NOC
address:      Splaiul Unirii, nr. 10
address:      Bucharest, ROMANIA
phone:        +40 1 330 86 61
phone:        +40 1 330 35 23
fax-no:       +40 1 675 49 99
e-mail:       tudor@pcnet.ro
trouble:      +40 9 325 18 84
admin-c:      BT17-RIPE
tech-c:       BT17-RIPE
tech-c:       AP158-RIPE
tech-c:       CM3059-RIPE
tech-c:       CN19-RIPE
tech-c:       IG20-RIPE
tech-c:       CR60-RIPE
nic-hdl:      PDNN1-RIPE
remarks:      ----------
remarks:      abuse: abuse@pcnet.ro
remarks:      ----------
remarks:      for escaladation please directly call the
remarks:      technical manager
notify:       tudor@pcnet.ro
mnt-by:       AS8503-MNT
changed:      tudor@pcnet.ro 20011008
source:       RIPE

person:       Bogdan Tudor
remarks:      Technical Manager
remarks:      PCNET Data Network S.A.
address:      Bucharest, Romania
phone:        +40 9 325 18 84
phone:        +40 1 330 86 61
phone:        +40 1 330 35 23
fax-no:       +40 1 675 49 99
nic-hdl:      BT17-RIPE
mnt-by:       BT17-RIPE-MNT
notify:       tudor@pcnet.ro
e-mail:       tudor@pcnet.ro
changed:      tudor@pcnet.ro 20011009
source:       RIPE

Appendix J – Screenshot of console