Scan of the Month 16 (June 2001): XOR Isn't Encryption Dino Dai Zovi 1. Identify the encryption algorithim used to encrypt the file. The file is "encrypted" by XOR'ing (exclusive or) every byte of the file with the hexadecimal value 0xFF. 2. How did you determine the encryption method? I suspected that the encryption method would not be too advanced, given the file's origin. Notably, as part of a rootkit, its decryption would probably have to be automated, so the file would most probably have been obfuscated instead of being truly encrypted. Also, given the size of the file, it is to small to be an executable, so it is most probably an ASCII text file of some sort. To determine this, I tested the byte value distribution of the file. If the file was encrypted with a decent algorithm, there would be a relatively even distribution of byte values. To test this, I wrote a small C program to count the occurances of byte values (bytedist.c) and determine the percentage of the bytes that were ASCII and the percentage of bytes that were printable. This yielded the fact that 211 byte values were not present in the file and none of the bytes used were ASCII. In contrast, a small test file (txt, about 536 bytes) encrypted with the standard UNIX 'bdes' utility only had 39 unused bytes with 52% ASCII characters (41% printable ASCII). The uneven and limited distribution of values in 'somefile' suggest that the byte values were individually altered. I suspected that they were XORed with a fixed value. I amended 'bytedist' to generate its statistics based on the bytes after they were XORed with a given value. Using a rough binary search algorithm by hand, I determined that with values from 0x80 - 0xFF (128 - 255), 100% of the bytes in 'somefile' were ASCII characters. Next, I did a similar rough binary search over the range 0x80 - 0xFF to determine where the highest percentage of printable characters was. I found that between 0xF1 and 0xFF about 96% of the characters were printable. I now just needed to try decoding the file with byte values in this range. My next tool, xor.c, XORs its input with a given byte. Using this, I tried decoding the file with each of the bytes from 0xF1 - 0xFF. XORing with 0xFF yielded the original file. 3. Decrypt the file, be sure to explain how you decrypted the file. The decrypted file, inserted below, was decrypted by XORing each byte with the value 0xFF (as described above) with xor.c. ---- BEGIN somefile.txt ---- [file] find=/dev/pts/01/bin/find du=/dev/pts/01/bin/du ls=/dev/pts/01/bin/ls file_filters=01,lblibps.so,sn.l,prom,cleaner,dos,uconf.inv,psbnc,lpacct,USER [ps] ps=/dev/pts/01/bin/psr ps_filters=lpq,lpsched,sh1t,psr,sshd2,lpset,lpacct,bnclp,lpsys lsof_filters=lp,uconf.inv,psniff,psr,:13000,:25000,:6668,:6667,/dev/pts/01,sn.l,prom,lsof,psbnc [netstat] netstat=/dev/pts/01/bin/netstat net_filters=47018,6668 [login] su_loc=/dev/pts/01/bin/su ping=/dev/pts/01/bin/ping passwd=/dev/pts/01/bin/passwd shell=/bin/sh su_pass=l33th4x0r ---- END somefile.txt ---- 4. Once decrypted, explain the purpose/function of the file and why it was encrypted. The file appears to be the configuration for a rootkit. The "[file]" block defines the paths to either the real or altered versions of 'find', 'du', and 'ls'. The 'file_filters' variable appears to define files that are to be hidden by these utilities. Similarly, the 'ps', 'netstat', 'su_loc', 'ping', 'passwd', and 'shell' variables point to the real or altered versions of these utilities. The 'ps_filters' variable lists process names that are not to be displayed by 'ps'. 'lsof_filters' defines some processes that are not to be displayed ('lp', 'psniff', 'psr', 'prom', 'psof', 'psbnd'), files not to be listed ('uconf.inv', 'sn.l'), directories not to be listed ('/dev/pts/01'), and port numbers not to be listed (13000, 25000, 6668, 6667). Finally, it also defines a creative alternate root password for 'su'. 5. What lesson did you learn from this challenge? Statistical analysis can be used to determine the strength of the encryption used on a file. In the case of XOR encoding, it can also be used to easily determine the byte value used to encode the file. 6. How long did this challenge take you? The byte value distribution analysis took roughly 20 minutes. It took around 5 minutes to write xor.c, test it, and successfully decrypt 'somefile'. In total, the analysis and decryption of 'somefile' took around half an hour. Of course, cleaning up the code, documentation, and writing the report took longer. I spent about two and a half hours doing these tasks. BONUS. This encryption method and file are part of a security toolkit. Can you identify this toolkit? In a message to the SecurityFocus FOCUS-SUN mailing list (http://www.securityfocus.com/archive/92/178904), Christopher Wong described this rootkit, which he found on his system. He quoted the README, which is inserted below: ---- BEGIN README ---- This is: SunOS Rootkit v2.5 (C) 1997-2001 Tragedy/Dor If you find this file, most likely your host has been hacked by a user of this rootkit. If you want information about this tool, removal instructions or such, please email bert.smith@mbox.bol.bg The author takes NO RESPONSIBILITY for anyone who misuses this tool. ... ---- END README ---- -Dino Dai Zovi