==================================================================== INITIAL ANALYSIS ==================================================================== After downloading the file "somefile.zip" and checking its MD5 signature, I unzipped it. The first step in determining the encryption method is to open the file with a hex-editor. Most encryption programs, actually almost all programs, leave an easily recognizable header in saved files which makes identification trivial. With "somefile", this is not the case. A property of "somefile" that is extremely obvious when you look at it with a hex editor is the fact that all bytes are larger than 0x80. This implies that the encryption algorithm that was used is most probably very weak. The simplest form of encryption is a monoalphabetic substitution, where a single character is replaced by another character. To investigate this option, I made a single character frequency analysis of "somefile". The characters that have nonzero frequencies are listed in the following table: char freq 86 1 87 1 88 2 89 11 8a 7 8b 28 8c 52 8d 11 8e 1 8f 34 90 10 91 29 92 2 93 28 96 24 97 6 98 3 99 14 9a 24 9b 18 9c 12 9d 14 9e 9 a0 6 a2 4 a4 4 aa 1 ac 1 ad 1 ba 1 c2 14 c5 4 c7 3 c8 2 c9 9 ca 1 cb 2 cc 3 cd 2 ce 13 cf 18 d0 45 d1 5 d3 30 f5 22 There are several ways to solve a monoalphabetic substitution after making an initial single character frequency analysis. These are for instance, guessing high frequency letters (like the space and the letter "e"), or looking at two-or three letter frequencies (usually called bigram and trigram statistics). In our case, since all characters are >0x80, and the letters that occur actually occur in blocks (0x86-0xa4, 0xaa-0xad, 0xc2-0xd3, 0xf5), we expect a more direct mapping of the encoded characters and the regular ASCII set, namely an exclusive or with a fixed byte (I will call this the EXOR byte). After guessing this to be the encryption algorithm, the EXOR byte can be found by trial and error, but looking a bit closer also gives the required result. First of all, the leading bit of the EXOR byte should be 1 in order to decode to printable ASCII. Furthermore, and this actually solves the encryption, we make the observation that the 0xf5 byte, which actually occurs quite often in the encrypted file, should map to a character that is relatively far off from the other decrypted characters. Of course, this can only be decrypted to the end-of-line character 0x0a. The EXOR byte therefore is 0xff. ==================================================================== DECODING "somefile" ==================================================================== Running a simple C program actually yields the following sensible text after decryption: [file] find=/dev/pts/01/bin/find du=/dev/pts/01/bin/du ls=/dev/pts/01/bin/ls file_filters=01,lblibps.so,sn.l,prom,cleaner,dos,uconf.inv,psbnc,lpacct ,USER [ps] ps=/dev/pts/01/bin/psr ps_filters=lpq,lpsched,sh1t,psr,sshd2,lpset,lpacct,bnclp,lpsys lsof_filters=lp,uconf.inv,psniff,psr,:13000,:25000,:6668,:6667,/dev/pts /01,sn.lprom,lsof,psbnc [netstat] netstat=/dev/pts/01/bin/netstat net_filters=47018,6668 [login] su_loc=/dev/pts/01/bin/suping=/dev/pts/01/bin/pingpasswd=/dev/pts/01/bi n/passwdshell=/bin/shsu_pass=l33th4x0r ==================================================================== INTERPRETATION OF CONTENTS ==================================================================== The content of the file is probably a file that is used by (trojaned commands in) a rootkit that apparently was installed on the machine. In the [file] section we see that trojaned versions of find, du, and ls are installed in /dev/pts/01/bin/, and these filter all outputs containing the strings in the file_filters line. For instance, the filter on 01 will render the directory /dev/pts/01 invisible to the find, du and ls commands. In the [ps] section we see that a trojaned version of ps is installed as /dev/pts/01/bin/psr, and as in the files section, the trojaned version will not display processes with names containing strings in the ps_filters line. Moreover the lsof_filters line will make lsof not display any open files with strings contained in the list lsof_filters. In the [netstat] section we see that a trojaned version of netstat is installed as /dev/pts/01/bin/netstat. The trojaned netstat will not list lines with ports 47018 and 6668. In the [login] section we see that trojaned versions of su_loc, ping, and passwd are installed in /dev/pts/01/bin/, and a hard coded superuser password is given as l33th4x0r. ==================================================================== POSSIBLE EXPLANATIONS ==================================================================== Finding a text file on a machine containing the lines above warns even the less security conscious system administrators. Therefore, this file is encrypted (actually, since no password is required, I would prefer the term "scrambled"). Considering the fact that the machine is running Solaris, and seeing the properties of the rootkit installed, makes me guess that the machine was victim to the snmpXdmid exploit. This exploit, CA-2001-05 (March 30, 2001) gives an intruder, using a buffer overflow, root access. If this exploit is indeed what happened, then extrapolating the findings in de decrypted file to the things that were reported on systems that were affected by the snmpXdmid exploit, explains the following items from the file somefile: 1. port 47018 is probably an ssh backdoor 2. port 6668 is probably used by an IRC proxy, installed as /var/lp/lpacct/lpacct 3. /usr/lib/lpset is probably a sniffer ==================================================================== LESSON LEARNED ==================================================================== The lesson learned from this challenge is that: 1. (for the administrator) relying on searching for suspicious text files or strings in executables can be useless if the file is a configuration file is used that is scrambled 2. (for the hacker) using simple encryption techniques will not give any protection against an attentive administrator. However, since the configuration file needs prompt decoding by the trojaned commands, it is impossible to use (secret) password based encryption, and scrambling is all you can hope for. ==================================================================== THE TIME IT TOOK ==================================================================== Discovering the encryption method, including a program to do the frequency counts, took about 10 minutes. Writing a C-program to decrypt the file took another 5 minutes. Interpreting the contents of the file and recalling the CA-2001-5 took about half an hour. ==================================================================== THE ROOTKIT THAT IS USED ==================================================================== The encryption method used and the contents of the file resembles closely the file "uconf.inv" that is used as a reference file by trojaned versions of ls, ps and netstat in the Adore rootkit.