This report shows the analysis of the Honeynet Scan of the Month #29 for September 2003. The scenario of the challenge was a Redhat 7.2 system that was deployed as a Virtual Honeypot in a VMWare session. An attacker compromised the Redhat system and the system was suspended within VMWare. We released the suspended image to the public along with nine questions to answer.
This report shows how I analyzed the system and gives the "official" answers to the questions.
Before I started the incident response and analysis of the system, I had to prepare the environment. The image was downloaded from: http://www.honeynet.org/scans/scan29/ and the MD5 has of the image was calculated and verified. The MD5 hash database of the honeypot before it was deployed was also downloaded and verified.
This challenge was going to require me to verify that the system had been compromised, so I needed to prepare a CD of trusted tools that could be used on the live honeypot. To create this CD, tools were collected and compiled statically when possible. The following tools were added to the CD from trusted systems:
A CD with The Penguin Sleuth Kit was also created so that the suspect system could be booted from it if the system was found to be compromised. All of the above steps could have been done way before the system was ever suspected of being compromised.
The final preparation step was to create an evidence server on the same network as the suspect system so that I could save the tool output to it. In this case, the evidence server was the Linux system that was running VMWare. Also, because the VMWare session potentially had malicious contents, the Linux system that was running VMWare was removed from all networks.
The incident was going to be verified by looking for traces of new tools and rootkits. The following key ideas were used:
The honeypot was unsuspended in VMWare and it gave me a root shell and a message about (swapd) being in promiscuous mode. The CD of trusted tools was placed in the system and it was mounted on /mnt. I changed to that directory and executed the trusted version of ifconfig. It showed that the system had an IP address of 192.168.1.79 and that the network card was in promiscuous mode (which was a potential sign of compromise and made sense based on the console message). The IP address of the Linux system running VMWare (the evidence server) was set to 192.168.1.100.
The trusted version of date was run on the system and it reported Sun Aug 10 20:33:57 PDT 2003. The trusted version of mount was run to identify which partitions were used by the system. The output showed that there was one partition (/dev/sda1) mounted at /. The fls tool from The Sleuth Kit was run on the partition to collect file meta information and to create a timeline:
# ./fls -f linux-ext3 -r -m / /dev/sda1 | ./nc -w 3 192.168.1.100 9000
And the evidence server was running the following:
# nc -l -p 9000 > fls.rm
The fls.rm file on the evidence server contained data about every allocated file on the system, including those hidden by kernel module rootkits and files hidden by trojan version of /bin/ls. This process also did not modify any of the access times on the system's files except /dev/sda1. The MD5 hash of the file was calculated on the evidence server.
I used the fls.rm file to confirm the incident. The first step was to look for non-special files in the /dev/ directory. The entries in the fls.rm file for the /dev/ directory were removed using grep:
# grep "|\/dev\/" fls.rm > fls.rm.dev
This created a file with over 13,800 entries for the /dev/ directory and the entries were sorted with the 'mactime' tool to create a timeline of all activity in that directory. It also provides an easy way to view the file types (which is what I wanted).
# mactime -b fls.rm.dev > timeline.dev
Analysis of the timeline file showed entries that had regular file types. The regular files have a '-' as the type instead of a 'c' or 'b'. The following suspect files and directories:
The MAKEDEV file is standard in all Linux systems and was not suspect. The other files were suspect though and they had Change times for August 10, 2003.
Another way to verify the incident was to look for files that began with a '.'. This was again done with 'grep' and 'mactime'.
# grep "\/\." fls.rm > fls.rm.dot
# mactime -b fls.rm.dot > timeline.dot
The timeline file showed many files that started with a '.' that were legit. The /lib/.x/ directory was identified though with several suspect file names in it and a M-time and C-time of August 10, 2003.
At this point, I knew that the ifconfig showed that the network device was in promiscuous mode and that there were files that were suspected of being part of a rootkit. It was decided to proceed with the acquisition and volatile data collection, where the incident could be further verified using other techniques.
On the honeypot system, I executed the trusted version of netstat to identify the open ports.
# ./netstat -na | ./nc -w 3 192.168.1.100 9000
The netstat output shows the following suspicious ports:
The output also showed that there was a connection from TCP 65336 to 213.154.118.200:1188. Other connections are for the 'netcat' sessions of our evidence collection on port 9000. (NOTE: Other submissions to the challenge reported two additional IRC connections. I think they timed out by the time that I ran 'netstat' after analyzing the 'fls' output)
The system was also port scanned from the evidence server. This would show if there were any open ports that were being hidden from the netstat binary from the kernel.
# nmap -sS -p 1- 192.168.1.79
The nmap output showed no additional suspect ports.
Lastly, the lsof tool was used to collect process and open socket information.
# ./lsof -n | ./nc -w 3 192.168.1.100 9000
The lsof output is examined to identify the processes that the suspect ports open and identify the processes that had any of the files in the suspected rootkit directory open. The following are found:
The contents of each of the above processes were saved using the pcat tool from The Coroner's Toolkit.
# ./pcat 3137 | ./nc -w 3 192.168.1.100 9000
Using all of the above information, I concluded that the system was likely compromised and powered it off. Because the shutdown process could have been trojaned, it was a safer bet to just power the system off.
Before the disk was acquired, the MD5 hash was first calculated:
# md5sum /dev/sda
f273a4f65222b4413a36452ae2d68827
Next, the network address was assigned and the entire disk was acquired using dd and netcat.
# ifconfig eth0 192.168.1.79
# dd if=/dev/sda bs=4k | nc -w 3 192.168.1.100 9000
On the evidence server, the hash of the disk image was calculated to verify that dd copied all sectors.
# md5sum sda.dd
f273a4f65222b4413a36452ae2d68827 sda.dd
Note that all submissions to the challenge will have different hash values for this step. This hash value is based on the changes that you made to the system during the incident verification and response.
I powered off the honeypot and closed VMWare. On the evidence server, I next extracted the partitions from the disk image using mmls, from The Sleuth Kit, and dd.
# mmls -t dos sda.dd
DOS Partition Table
Units are in 512-byte sectors
Slot
Start
End
Length
Description
00: ----- 0000000000 0000000000 0000000001 Primary Table (#0)
01: ----- 0000000001 0000000031 0000000031 Unallocated
02: 00:00 0000000032 0001884159 0001884128 Linux (0x83)
03: 00:01 0001884160 0002097151 0000212992 Linux Swap / Solaris x86 (0x82)
We see that there are two partitions. The root mounting point and the swap space.
# dd if=sda.dd skip=32 count=1884128 of=sda1.dd
# md5sum sda1.dd
54682b1cd7b1a4f8c97a8ea3e0959554 sda1.dd
# dd if=sda.dd skip=1884160 count=212992 of=sda2.dd
# md5sum sda2.dd
764ce8994d21e3483db04d97ff8d7f01 sda2.dd
I next examined the file system images with a Linux system running The Sleuth Kit v1.65 and Autopsy v1.74. The analysis process uses techniques that allow us to find "Quick Hits" and then we will conduct a more thorough search to fill in the blanks. This investigation (1) uses the MD5 hashes, (2) examines the suspect files identified from the hashes, and (3) fills in the blanks.
During the investigation, I use the Event Sequencer mode of Autopsy to sort the file activity as I find it. This makes it easier to identify where there are holes in the theory.
A case was created in Autopsy named sotm_29 and a host named host1 was also added with a timezone of PST8PDT. The MD5 hash database was added as a 'Known Good' database that contains hashes that can be ignored.
The file system and swap space images were added to the host along with their MD5 values so that they could be later verified. The Host Gallery can be seen here. Before the hash database can be used, it must be indexed. This can be done by pressing the "Hash Databases" button and choosing the "Index DB" button in the 'Ignore Database' section.
To reduce the amount of data that we will need to look at, we should utilize the hash database that was generated before the honeypot was deployed and identify the files that have been created or have been changed since then. The 'File Type' mode of Autopsy is used to do this. The process takes quite a while because it examines over 35,000 entries. While Autopsy is doing this, other analysis can be performed (examining /lib/.x/ for example).
The output summary file shows that over 17,000 of the entries were for non-files (such as devices and directories), over 17,000 files were ignored because they were in the hash database, and only 617 files had been added or changed since the honeypot was deployed. That is 1.7% of the original number of files.
The compress category shows us /etc/opt/psyBNC2.3.1.tar.gz and /root/sslstop.tar.gz. The other three entries are deleted file names that previously used the inodes that the above two files are using. A full analysis should be done of these archive files.
The data category contains all of the files that do not have an internal structure that the 'file' command knows about. Useful entries in here include /etc/opt/psybnc/ files, /usr/lib/sp0_key, /usr/lib/sp0_seed, /usr/include/icekey.h, /usr/include/iceseed.h, /lib/.x/s/s_h_k, and /lib/.x/s/r_s. Because these files have an unknown structure, there might not be much useful information in them. Depending on time, other file types should probably be looked at first.
The exec category contains executable files and libraries. It shows us files in the /etc/opt/psybnc directory and the startup scripts /etc/rc.d/init.d/functions and /etc/rc.d/rc.sysinit. For system executables, we find /usr/bin/top, /usr/bin/sense, /usr/bin/sl2, /usr/bin/crontabs, /usr/bin/smbd -D, /usr/bin/(swapd), /usr/lib/adore.o, /usr/lib/cleaner.o, /usr/lib/sp0, /bin/netstat, /bin/ps, /bin/ls, /bin/pico, /lib/.x/hide, /lib/.x/inst, /lib/.x/log , /lib/.c/cl, /lib/.x/s/xopen, /lib/.x/s/lsn , /lib/.x/sk, /lib/.x/.boot, /root/sslstop/sslstop, /root/sslstop/sslport, and /sbin/ifconfig. All of these should be examined to identify what they do. The startup scripts will show what needs to be started whenever the system is booted. Note that we saw the (swapd), smbd -D, and xopen processes when we responded to the system.
The text category shows all text documents, including many logs. The interesting files that have been added since the system was deployed include /dev/ttyop, /dev/ttyoa, /dev/ttyof (which we saw during the verification), /etc/opt/psybnc/ files, /usr/bin/logclear, /usr/lib/libsss, /usr/lib/libice.log, /usr/lib/sp0_cfg, /usr/include/iceconf.h, /usr/include/icepid.h, /lib/.x/ files, /root/sslstop/ files, and /.bash_history.
Having all of the files sorted and looked up in the hash database takes some processing time, but it was well worth it in this case. We have quickly identified the areas that must be focused on:
The /lib/.x/ was a good place to start investigating because it was a possible rootkit directory that was identified during the incident response and by the hash database lookups. This could be investigated while the files were being sorted or after. (screen shot).
The .boot file is a script that appears to start an SSH server located in /lib/.x/s/xopen, executes /lib/.x/s/lsn and /lib/.x/sk, sends email to skiZophrenia_sick@yahoo.com, and executes /lib/.x/cl with several arguments including "suckit", "xopen", and "promisc".
The cl executable has a usage message that shows that it removes strings and usernames from logs. This was called from the .boot script.
The hide script looks at the running processes via ps and executes /lib/.x/sk for each process that has the string "/lib/.x", "xopen", or "lsn" in the name. All output is saved to lib/.x/hide.log.
The hide.log file has a log entry about SucKIT v 1.3b and an error about not finding the sys_call_table[].
The inst script creates the sk file (used by hide) and creates /sbin/init13996 from /sbin/init. The user is told to execute sk to install.
The install.log hash log entries for SucKIT v 1.3b like the hide.log did, but more of them.
The log executable is for SucKIT v1.3b. Most likely the client, based on the strings.
The sk executable is for SucKIT v1.3b, most likely the server that hides the data.
The s/lsn executable is packed with the UPX executable packer and therefore there are few useful strings except "promiscuous" and this could be a sniffer. Fortunately, we have the process memory that was saved during the initial response. Process 25247 was for the "lsn" process. The memory contents show the ASCII format of a network sniffer log file, the string "Sun Aug 10 20:35:23 :User sic ([[[kgb]]]) disconnecting from stoned server", and a log from the port scan that I did of the system before the process memory was grabbed. The IP address 63.99.224.38 is also in the process memory. This IP address is registered to Ready Hosting. Recall that this process had ports 80 and 443 open and is started by .boot
The s/mfs file has the same format as the lsnmfs has entries for 63.99.224.38, proxyscan.undernet.org, and the evidence server network sessions.
The s/pid file has 25241, which was the PID for the xopen process.
The s/port file has 3128, which was the port for the xopen process.
The s/r_s file has no useful strings (see sshd_config).
The s/s_h_k and s/s_h_k.pub files are an SSH key pair.
The s/sshd_config file is an SSHD config file that uses the /lib/.x/s/r_s file as a random seed and the s_h_k key files.
The s/xopen executable is an SSH server. There are two possible MD5 hashes that could be passwords "d4377659532b8cd242de551fbbb45adc" and "5ff282cefe8fde3cc3ed6a9c42a7234d". No obvious password and login logging was found. This process had port 3128 open.
psybnc is a common IRC bouncer (or proxy). We can look at the log and configuration files in this directory for some clues about who the attacker was. The log/psync.log file shows the user 'sic' and 'redcode' connecting to 'fairfax.va.us.undernet.org' and 'mesa.az.us.undernet.org'. The user is coming in from 'sanido-09.is.pcnet.ro' and 'sanido-08.is.pcnet.ro'.
The directory shows a makefile, two .c files, two executables, and some deleted swap files. Examining the source code of sslport.c shows that it is a program that changes the default SSL port in apache in the configuration file to another port. It creates a temporary file /tmp/.httpd.tmp. The program was compiled and accessed 23 seconds after being compiled.
The sslstop.c file edits the apache configuration file to remove SSL support. It also uses a temporary file /tmp/.httpd.tmp. This program was compiled and accessed 2 minutes later.
The /.bash_history file shows the attacker downloading this file from izolam.net and then killing 'apache' processes.
The /usr/lib/ directory had several files that were identified from the hash databases. I examined those next. The adore.o file has ASCII strings for hiding processes and such, so it is likely the kernel module for the Adore rootkit. The cleaner.o is also likely associated with the Adore rootkit.
The libice.log file appears to be a log from network activity and includes an entry from 'proxyscan.undernet.org' and entries from the portscan that I did of the system to detect open ports. It is unknown which process created this file. The libsss file is only 2 bytes in size and only has a quotation mark and new line.
The sp0 file is an executable file with strings for an SSH server. It has two possible MD5 hashes for backdoor passwords 'f25a2fc72690b780b2a14e140ef6a9e0' and '5ff282cefe8fde3cc3ed6a9c42a7234d'. Note that the latter MD5 was also found in the SSH server in the /lib/.x/s/xopen. The sp0_cft file is an SSH server configuration file that uses port 345 and /usr/lib/sp0_key as the host key, /usr/lib/sp0_seed as the seed file, and /usr/lib/sp0.pid as the PID file. The sp0_key and sp0_seed files exist in the directory, but the sp0.pid file does not. As no process was running on port 345 when we responded to the system and there is no PID file, this SSH server was likely not executed.
+-[ User Login Incoming ]----------- --- --- - -
| username: %s password: %s%s hostname: %s
+----------------------------------- ----- --- -- -- -
It is not obvious where the passwords would be saved to, but the previous ASCII string was "2xvu2ole2olevkorj". I noticed that this string had some order to it and it could be a simple obfuscation of a file name. If we assume that '2' was substituted for a '/', then we can look at the ASCII chart and see that '/' is 47 and '2' is 50. Therefore, the algorithm could be two add three to the original ASCII value. Continuing with this, we find that 'x' translates to 'u', 'v' to 's', and 'u' to 'r'. This process can continue to reveal "/usr/lib/libshlog". This file does not exist, so it is likely that no passwords were recorded (which make sense as this is a honeypot).
The /etc/rc.d/init.d/functions file has an entry added at the bottom to execute /usr/bin/crontabs -t1 -X53 -p. That will be analyzed next.
The usr/bin/(swapd) file has strings for setting promiscuous mode and refers to /usr/lib/libice.log. We previously saw that and it had some network log entries. Therefore, this is likely a network sniffer. A process by this name was running as PID 3153 and had TCP ports 80 and 443 open.
The /usr/bin/crontabs file has a reference to smbd -D and pthread strings. Recall that this is the program that was called by /etc/rc.d/init.d/functions.
The /usr/bin/sense file is a perl script that sorts the output of "LinSniffer" (according to the header).
The /usr/bin/sl2 file has strings that appears to be for a port scanner. It refers to src and destination addresses and high and low ports.
The /usr/bin/smbd -D file was already identified as an SSH server.
The /usr/bin/top file refers to /dev/ttyop, which was identified as a regular file in the /dev/ directory. Therefore, this likely hides processes that are in that file. We will confirm that in the next section (note that in reality I checked it at the same time, but it is easier to document this way).
The /bin/ls file references /dev/ttyof, which was identified as a text file. Therefore, this has likely been modified to hide certain files.
The /bin/netstat file references /dev/ttyoa, which was identified as a regular text file in the /dev/ directory. Therefore, this has likely been trojaned to hide certain network connections.
The /bin/pico file is for the pico text editor. A brief analysis shows that it does not appear to be modified.
The /bin/ps file references /dev/ttyop and likely hides processes. This was the same config file for the top file.
The /dev/ttyoa file was referenced by netstat and it contains a list of IP addresses that probably are hidden when netstat is run.
The /dev/ttyof file was referenced by ls and contains file names such as "psbnc" and "smbd". These are likely file names that should be hidden by ls.
The /dev/ttyop file was referenced by top and it contains a list of names that are the same as file names that we have encountered. Therefore, these names are likely the ones that should be hidden by top.
The hdx1 and hdx2 files that were identified have a size of 0.
For August 10, 2003 we see activity at 13:33 that creates the modified system binaries, the SSH server with smbd -D, modifies the functions startup script, installs the (swapd) network sniffer, and sl2 port scanner.
All is quiet until 15:30 when Adore is installed and the sp0 SSH server. This included the addition of files to rc.sysinit that did not exist on the system. This installation must have failed.
Shortly after, at 15:31, another installation occurs. This one installs the files into /lib/.x/ and includes the SucKIT rootkit. This installation includes the xopen SSH server.
At 15:51 we see that the random seed for the xopen SSH server was accessed and the sslstop programs were compiled. Some of the other files are accessed and the random seed for the xopen server is written to at 16:32.
At 20:35 we see the network programs responding to my response and my port scan.
We are missing many of the install scripts, we haven't checked the logs, and we can make a timeline to collect more information.
The /var/log/boot.log file has entries starting at Aug 10 13:33:57. The entries are for syslog starting and stopping. At 15:52:12 there are entries for httpd shutting down and failing to start up again.
The /var/log/cron file has entries starting at Aug 10 13:40:00. There are no entries that appear to be related to the incident.
The /var/log/maillog file has entries starting at Aug 10 13:33:56 and has mail being sent to the following addresses:
Note that many rootkit installations send email and the first two are for the user 'apache', which should not have permission to install a rootkit. So, those might have failed.
The /var/log/secure file has entries for Aug 10 16:04:14 for the 'xinetd' daemon starting a telnet session with 193.109.122.5 (proxyscan.undernet.org). The network sniffer logs all had references to this address as well. There is also an entry at 18:58:33 for 'sshd' and a message from 202.85.165.46. 202.85.165.46 is registered to iAdvantage Limited in Hong Kong.
The string "User Login Incoming" was searched for to find other files where 'smbd -D' could have saved password information to. Only the 'sbmd -D' binary was found to have that string.
The email addresses were searched next to find any install scripts. The "jijeljijel" string found two hits. The mail log and an installation script that starts in fragment 39396 and ends in 39398. The first part of the script appears to have been overwritten by another file. The install script is for "Sonkeriki Root". It creates several files and directories that were not found on this system. Therefore, this was likely not run and this is confirmed by the email that was sent from the 'apache' user. The attacker likely got access via an apache exploit and tried to install the rootkit using that account.
A search for "newptraceuser" results in 13 hits. All are the actual email that are queued up to be sent or the logs that we already saw. The script that sent the mail could not be found.
A search for "skiZophrenia_siCk" results in 4 hits. These hits include the actual email message that was sent and the log message that was already seen. The script that sent the mail was not found.
To find the script that installed the first rootkit, the "smbd -D" string was searched for. It found 54 hits. The first is for a log file whose inode is allocated, but the file name no longer exists. This is likely from a log file that the attacker deleted but it remained allocated until the OS closed it. The file from inode 45307 can be found here. This shows us that 'smbd -D' started to listen at 13:33:33. Connections were made to it at 14:14:41 from 213.154.118.218 (extreme-service-10.is.pcnet.ro) and the login to root failed. The login was repeated at 14:14:08 and 14:18:00 and they all failed. At 15:30:30 the Ethernet device went into promiscuous mode. Several other bind errors occurred for port 2003 and the root user performed an 'su' at 15:56:11.
Another string from that rootkit is also searched for. The rootkit configuration file, /dev/ttyoa, is searched for. This generates 9 hits. Fragment 201083 contains the beginning of an installation script that spans 2 fragments. The script does a 'chattr' on all of the system binaries, replaces some of the system binaries with trojan versions, installs '(swapd)', adds the entry for 'crontabs' to the init.d/functions file, and deletes the logs. Email messages are sent to "mybabywhy@yahoo.com" and "buskyn17@yahoo.com". The script appears to be in Romanian.
To find out about the second rootkit, which installed 'sp0', we search for "sp0" and get 57 hits. One of the hits is for fragment 36691, which is an installation script that installs adore into /tmp/rk/, which we do not see on this system. Fragment 112752 has another installation script that installs 'sp0' into /bin/. This script compiles adore first and then installs it. The binaries are not found in the locations where this script places them so the compilation could have failed. This does add "kflushd" to the end of /etc/rc.d/rc.sysinit, which we previously observed.
To help find relevant log files, the IP addresses that have been seen so far were searched for. We have seen "213.154.118.218", "213.154.118.200", and "213.154.118.201". So, the string "213.154" was searched for and resulted in 16 hits. The last hit shows a log file in fragment 114422 for Apache and a request from 213.154.118.219 for the URL "/sumthin" at 13:23:17. This URL is common with scanning tools for the OpenSSL vulnerability (http://www.webmasterworld.com/forum11/2100.htm).
To identify additional logs for that time period the string "Aug 10 13:2" is searched for to find entries between 13:20 - 13:29. This identifies fragment 114400, which is the Apache log. The fragment is allocated to inode 46935, which likely belongs to the log file that was deleted by the attacker but still open by the apache process. This shows 7 errors during the suspect time frame from the suspect IP:
[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin
[Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:32:38 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
These log messages are consistent with those listed in http://www.auscert.org.au/render.html?it=2409&cid=1 for an attack against a vulnerable OpenSSL system, which RedHat 7.2 was (https://rhn.redhat.com/errata/rh72-errata-security.html).
The .bash_history file showed the attacker using wget to download some tools, so a search for that could result in some previous history files or commands. The search for "wget" in unallocated space results in 39 hits. Fragment 17036 shows the commands for downloading several rootkits and trying to install them. Fragment 39350 shows output from downloading a tool called xl.tgz and a tool that appears to scan systems and overflow a buffer. Other hits returned scripts that had already been found.
Email addresses were found in the logs and additional Romanian hosts were identified. A deleted rootkit installation script was also found to be in Romanian.
The updated Event Sequencer can be found here.
We have answered the questions and found a lot of evidence. There are still some holes in the theory about exactly what happened and what was downloaded. Examining each of the tools at the URLS that were found in unallocated space could reveal additional information.
1. I verified the incident by running a statically-compiled and trusted version of fls from The Sleuth Kit and nc. They collected data from the system and saved it to an evidence server. Using the data, the regular files in /dev/ were found and the /lib/.x directory was found. The port scan also showed suspect ports.
2. My actions impacted the system by mounting the CD of trusted tools, accessing the raw device for the hard disk, and executing the 'fls' and 'nc' tools that were statically compiled. Other system tools were executed to collect volatile data after the incident was verified and those were not statically compiled, so they relied on shared libraries on the system and modified the access time.
3. From the lsof output, the 'smbd' (PID 3137) process was identified to have port 2003 open, the 'initd' (PID 15119) process was identified to have ports 65336 and 65436 open, and the 'xopen' (PID 25241) process was identified to have port 3128 open.
4. From both the netstat and lsof output, there was an active connection from port 65336 (the 'initd' process) to 213.154.118.200:1188.
5. 3 new ssh servers were found in addition to the original server that was not modified. The new servers were located in /usr/lib/sp0 (c-time: Aug 10, 2003 15:30:54), /usr/bin/smbd -D (c-time: Aug 10, 2003 13:33:33), /lib/.x/s/xopen (c-time: Aug 10, 2003 15:32:16).
6. The xopen and smbd -D servers were found running when the system was examined live. The PID file for sp0 was not found, so it was likely not run.
7. The smbd -D version of the SSH server appeared to save login and password information to the /usr/lib/libshlog file. No information was collected. The other versions of SSHD likely had backdoor passwords as identified above.
8. The following were found to have been modified by the attacker and a likely configuration file was found for each:
| Program | Configuration File |
|---|---|
| /usr/bin/top | /dev/ttyop |
| /bin/ls | /dev/ttyof |
| /bin/netstat | /dev/ttyoa |
| /bin/ps | /dev/ttyop |
| /sbin/ifconfig | None - hides promiscuous mode |
9. The system was likely compromised from 213.154.118.219 using an OpenSSL vulnerability. We did not find evidence of the method used to gain root access from the 'apache' user, but the system was vulnerable to several attacks, including the 'ptrace' vulnerability. Downloading the tools from the sites identified in the recovered scripts may show what tools were used.
Bonus: Nationality was likely Romanian based on the addresses that the attacker came from and the language of the installation script.
