Scan 16:

1. Identify the encryption algorithm used and decrypt

The first task in any crypto analysis is a byte count. If the file is
crackable the algorithm is usually very simple. Often each byte is
XORed with a constant value and a byte count will show up any patterns
which could give the algorithm away.

The following Python code was used:

import sys
bytes = {}
data = sys.stdin.read ()
for x in range (0, 256):
    bytes[x] = 0
for x in data:
    bytes[ord(x)] += 1
for x in range (0, 256):
    print str(x) + " " + str(bytes[x])

The script was run as:
% cat somefile | python2 count.py > data
% gnuplot
gnuplot> plot 'data' with lines

The ASCII-art version of the graph is:
 60 ++----------+-----------+-----------+-----------+-----------+----------++
    +           +           +           +           +         'data' ****** +
    |                                                                       |
 50 ++                                *                                    ++
    |                                 *                                     |
    |                                 *               *                     |
    |                                 *               *                     |
 40 ++                                *               *                    ++
    |                                **               *                     |
    |                                **               *                     |
 30 ++                               ***              **                   ++
    |                                ***              **                    |
    |                                *****            **                    |
 20 ++                               *****            **       *           ++
    |                                *****            **       *            |
    |                                ******        *  **       *            |
    |                                ******        * ***       *            |
 10 ++                               ******        *****       *           ++
    |                                **** *       ******       *            |
    +           +           +        **** **      ******       *+           +
  0 *********************************-*-*-***********-************---------++
    0          50          100         150         200         250         300


The pattern given shows a number of things. Firstly - it is a weak
cipher. If the cipher were strong (e.g. DES, TwoFish and the like) the
distribution of the bytes would be uniform. (A uniform distribution
does not make a good cipher - but all good ciphers have
them). Secondly, it's almost certainly not a Caeser cipher (which adds
a constant value to the bytes) because there are definite areas in the
output which do not look like the distribution of text. Thus it is
likely to be a constant XOR based system.

There are well known methods for breaking XOR ciphers (the interested
reader is referred to [1]) but they were not needed in this case. Since
the cipher looks so primitive, it was worth a shot at it being an XOR
with a 1-byte key as they are so simple to code.

The following Python code was used to decrypt the file with every key
assuming a 1-byte constant XOR:

import sys
data = sys.stdin.read ()
for y in range (0, 256):
    print "**********************************************"
    print str(y)
    for x in data:
        sys.stdout.write (chr(ord(x) ^ y))

This generates a lot of output, but the human brain is very good at
picking out text. However it turns out that the actual text comes out
at the very bottom with a key of 255.

This means that the cipher is even more primitive than was thought at
first. XORing with 255 is the same as NOTing each byte. So the actual
cipher is simply the NOT of each byte.

The decrypted file is thus:
[file]
find=/dev/pts/01/bin/find
du=/dev/pts/01/bin/du
ls=/dev/pts/01/bin/ls
file_filters=01,lblibps.so,sn.l,prom,cleaner,dos,uconf.inv,psbnc,lpacct,USER

[ps]
ps=/dev/pts/01/bin/psr
ps_filters=lpq,lpsched,sh1t,psr,sshd2,lpset,lpacct,bnclp,lpsys
lsof_filters=lp,uconf.inv,psniff,psr,:13000,:25000,:6668,:6667,/dev/pts/01,sn.l,prom,lsof,psbnc

[netstat]
netstat=/dev/pts/01/bin/netstat
net_filters=47018,6668

[login]
su_loc=/dev/pts/01/bin/su
ping=/dev/pts/01/bin/ping
passwd=/dev/pts/01/bin/passwd
shell=/bin/sh

su_pass=l33th4x0r

2. The purpose of the file

The file is plainly a root kit file. A google search for some of the
strings in the file turned up [2]. In [2] it becomes clear that this
file is used to configure wrappers for a root kit

The file is in sections, one for each wrapper and each section
contains
	1) The location(s) of the real binary(s)
	2) Some filter information

Taking, for example, one the filter lines:
lsof_filters=lp,uconf.inv,psniff,psr,:13000,:25000,:6668,:6667,/dev/pts/01,sn.l,prom,lsof,psbnc

This is configuring the wrapper for lsof not to show any open fds for
a number of utilities and ports. One would suspect that the cracker
would install back doors on some of these ports and use the lsof filter
to hide them.

Interestingly - the filters for netstat do not hide the same
ports. One port is common to both (6668) and is a common IRC
port. This suggests that the attacker wanted to hide a connection to
an IRC server. Many DDoS systems use IRC to coordinate the actions of
the zombies (as detailed in [3]) so a DDoS zombie could be expected on
the system.

3. Lessons and Why It Was Encrypted

Firstly - although the cipher is too poor to even be considered a
cipher (technically it is a code, not a cipher, because it has no key)
it is enough to hide the configuration from utilities like strings which is
how the administrator in [2] found it. This makes it perfectly clear
that strings is no replacement for md5sum and should never be relied
on for finding hostile binarys. Hiding from strings is now common
place and code like TFN uses strings too short ( < 3 letters) for
`strings` to pickup by default, for the same effect.

Playing cat and mouse and making strings try simple ciphers is not a
solution as using much more powerful ciphers is very easy.

4. Time Taken

About 10 minutes (not including writing it up)

Bonus Question:

> This encryption method and file are part of a security toolkit. Can
  you identify this toolkit?

[2] suggests that it is part of "SunOS Rootkit v2.5 (C) 1997-2001
Tragedy/Dor" . A search for this turns up
78795fed5abb0aaed98b41a62cafb393  rootkitSunOS.tgz

Which does not seem to have anything to do with it.

However the code seems to be related to the rootkit described in
detail in [4]. A little more searching didn't turn up anything
conclusive.

Adam Langley
agl@linuxpower.org
Freenet Project

References:
[1] Applied Cryptography, Bruce Schneier, Section 1.5
[2] http://archives.neohapsis.com/archives/sf/sun/2001-q2/0088.html
[3] Gibson, The strange tale of the DOS attacks against GRC.COM
    http://grc.com/dos/grcdos.htm
[4] http://www.sans.org/y2k/the_compromise.htm