1.Identify the encryption algorithm used to encrypt the file. 

The decryption/encryption function is a bijective function f:B->B with 
x->x XOR 0xFF. Where "B" stands for a single Byte or Z2^8.


2.How did you determine the encryption method?

The highest bit in every Byte was set. Every modern encryption function is
build to have a good entropy.
The setting of the highest Bit in every Byte could be an artifact of the
encryption function and the plaintext(highly unlikely), or hints at
something "easier" than a modern Blockcipher. So something along the lines
of a Hill cipher or even more basic is suspected. With the length of the
file "somefile" and the difference of the Bytes after the removal of the
highest bit in mind it is assumed, that the plaintext is a 7-bit ASCII
text. Construction a Hill cipher that would result in a ciphertext with the
mentioned attributes seemed too much trouble for something this obvious. So
something along the lines of +,-,*,/,XOR,shift or rotation is suspected. Due to
the byte value of the readable characters *,/ are rejected. A shift to the
right for 1 bit with 1 as a filler is also rejected, as this is not a
bijective function. Rotation also seems highly unlikely, as the highest Bit
is always set, and constructing a function, that will rotate the bytes in a
way that the highest bit is always set looked like more effort than the
aforementioned Hill cipher. Since the subtraction of 128 (deleting of the
highest Bit) did not result in anything very meaningful, and the
difference between the Bytes was to high to result in anything meaningful
if subtractions with a Value significantly lesser or higher than 128
(except for a function of the form f:B->B x-> 256-x. This was not tested, as
this is the equal to the function mentioned in 1.). With the ruling out of
subtraction the addition was also ruled out. Next to try was the XOR
function. To XOR an 7-bit ASCII text with something that results in the
ciphertext (f:B->B x->x XOR y) the highest bit of y has to be set. The
first thing that comes to mind when XORing is the complement (XOR 0xFF).
This was tried and resulted in the hopefully correct decryption of the
file.


3.Decrypt the file, be sure to explain how you decrypted the file. 

The decryption was done by the following program written in C:
-- Begin of program test

#include <stdio.h>


int     main(){
	int	x;
	while ((x=getchar()) != EOF ) {
		putchar(x ^ 0xFF);
	}
}

-- End of program test

The file was decrypted with "./test < somefile > somefile2"

For the process resulting in this program please refer to 2.

-- Beginning of decrypted output

breiholz@terrorthrone:~$ cat somefile2 
[file]
find=/dev/pts/01/bin/find
du=/dev/pts/01/bin/du
ls=/dev/pts/01/bin/ls
file_filters=01,lblibps.so,sn.l,prom,cleaner,dos,uconf.inv,psbnc,lpacct,USER

[ps]
ps=/dev/pts/01/bin/psr
ps_filters=lpq,lpsched,sh1t,psr,sshd2,lpset,lpacct,bnclp,lpsys
lsof_filters=lp,uconf.inv,psniff,psr,:13000,:25000,:6668,:6667,/dev/pts/01,sn.l,prom,lsof,psbnc

[netstat]
netstat=/dev/pts/01/bin/netstat
net_filters=47018,6668

[login]
su_loc=/dev/pts/01/bin/su
ping=/dev/pts/01/bin/ping
passwd=/dev/pts/01/bin/passwd
shell=/bin/sh

su_pass=l33th4x0r

-- End of decrypted output


4.Once decrypted, explain the purpose/function of the file and why it 
  was encrypted.

Since I have no experience with Solaris (see Bonus Question) I am not sure
about the nature of the security toolkit in question, so everything
regarding the purpose/function of this decrypted file is derived from the
information given by it (or pure speculation if you want to see it this
way:)

The first paragraph ("[file]") looks like it tells the toolkit in question
where to look for basic files it uses (ls, du etc.). According to the path
of these files I assume that these are files modified versions of the
original binaries, designed to hide the intruders  sniffing/intrusion 
etc. "toolkit". 

The next paragraph ("[ps]") looks like it tells the security toolkit what
program to use for "ps" output. In this case I assume it is a rewritten ps
(according to the path) program designed to hide some things. The
"ps_filters" line seems to indicate the processes that should not be
displayed. This is assumed, because the process "psr" ( the utility named
by the "ps=" line) is in the filter line. Also everything called from
"/dev/pts/01" should be hidden (this is assumed to be the place, where the
intruder installed some of its "rootkit" files). As seen in the
"lsof_filters" line I assume that the intruder tries to hide other stuff
with this. The data ":6668" looks like it is a port number, where the
binaries of the intruder are listening for a connection, as it is also in
the "net_filters" line (So it is assumed, that this is done to hide
connections from the ports mentioned in this line. This is mostly derived
from the mentioning of the tool "psniff" which seems to be an packet sniffer
according to http://patrick.tuxfamily.org/release/psniff.c.).

The next paragraph ("[netstat]") looks like it follows the same
requirements above. It describes a path to a binary to be used ( in this
case a rewritten binary (assumed according to the path)) with a filter
variable that indicates the program what is not to be displayed. This is
mostly derived from the assumption that on port 6668 (see the paragraph on
"[ps]") is running a tool of the intruder.

The next paragraph ("[login]") looks like it sets the path to standard
programs, that can be used from this security toolkit. The "shell" variable
is pointing to a reasonable place for the real binary (as far as I can tell.
I am not really accustomed to Solaris (see Bonus Question)). The "su_loc"
and "passwd" variables are again pointing to binaries located in
/dev/pts/01/bin/. I assume that these are binaries build to capture
passwords and part of the "kit" the intruder installed. I find it hard to
think of any "real" reason for the intruder to remap the ping command
(perhaps this is done to collect data of the network. If a system
administrator is using ping to test various things in his network, the data
collected by this tool may be valuable for further intrusions and/or a 
topology mapping of the attacked network and affiliated networks.).

The line "su_pass=l33th4x0r" seems to define a standard ?override? password
for the su function implemented in the security toolkit.

Why is this file encrypted?

If the option "su_pass" is a normal option in the security toolkit some
means to hide the password from prying eyes are necessary. As this file is
part of a security toolkit I assume that the file is encrypted to hide the
su_pass password (probably the password to invoke the su command in the
toolkit, or for the toolkit to use root/user only functions) and to hide the
data what is monitored and what is exempt from monitoring (the "*filter"
lines). The encryption of this file could be done by the intruder, if
she/he replaced the security toolkit, to hide her/his passwords and kit 
from other hackers that successfully hit the same system.

Note: Personally I think the second case is highly unlikely, because of the
weak encryption, but not impossible. Also I personally think that hiding
processes like "lp" (?a grep expression for, or the printer daemon on
Solaris?) and others like that is a bit ?stupid?, because if these
processes are active in normal operational mode of this node another tool
that scans for halted/broken/disrupted etc. services should alert the admin
that a standard process is not running, as it cannot "see" it. (This may be
completely _false_ as I have no real knowledge of Solaris systems).


5.What lesson did you learn from this challenge?

I guess the most important lesson I learned from this challenge is:
If this is a security toolkit (and published as such, and not a mere
"hacking" toolkit") it has a "stupid" encryption for valuable passwords. 
If a "security toolkit" you use claims to store passwords/valuable data in
encrypted fashion, always have your brain a go on the encrypted file. You
know, what you typed in for information you want to keep private and an
attacker most of the time does not. If you can break the encryption in
reasonable time your passwords/valuable data is not really secure and you
better look for another toolkit implementing the same functionality as the
one you used, or if you have to use it, install it only when needed and
then uninstall it, or write a wrapper for it to use better encryption, or
rewrite parts of it to use better encryption.

Lesson learned: Not everything that claims to use encryption is using
"reasonable" encryption. Never trust a program you have not thoroughly
tested or reviewed.

Sometimes your brain is faster than the computer (well, I do not have any
sophisticated programs to do cryptography analysis but the brain process
was just faster this time :) :).

Next time I encounter a process that looks suspicious and demands a login I
will try l33th4x0r and/or try typing like an imbecile :)

I will definitely try to get some experience on Solaris OS and others. Grave
oversight on my part there to neglect them :). I am just a student, so lets
see what time and job opportunity will say :)


6. How long did this challenge take you? 

I needed about 25 minutes. 16 Minutes for figuring out the basics described
in 2. . the rest for coding various programs to test my hypothesis (
subtraction of 128 and other numbers, the final program) and some
paperwork while cursing the hexdisplay of the file. About 3 Minutes to find
a source code to psniff (see 4.). Some hours of fun browsing the web
searching for a Solaris security toolkit having the discovered attributes
(and reading other stuff :). About 1 hour to write this report.
In total it took me about 1 hour 30 minutes without the bonus question.


Bonus Question: 
This encryption method and file are part of a security toolkit. Can you
identify this toolkit? 

As said before I do not have any real experience with Solaris OS or
toolkits associated with it. After browsing the web for some time I found
very interesting documents but nothing to help me identify the security
toolkit. I will check back on Monday to see what the toolkit was in the
response of others :)


Ending Note:
Most of the things in this "report" are based on assumptions of usability,
difficulty of various tasks and experience (we used to encrypt stuff with
XOR 0xFF way back in school to taunt some people :). It is entirely
possible that the notes in 4. are totally wrong (see above). Also it is
possible to build an encryption that results in the mentioned plaintext
while resulting in another "meaningful" plaintext in another
encryption/decryption algorithm to obscure the real data, thus the
decrypted file could be totally of the mark, but I do not think so.
Also most of the assumptions made are not "correctly proven" in a
mathematical way, as I did not think it to be necessary.

	Henning Breiholz 22.6.2001
	Student and Sysadmin