When dealing with an unknown binary, the first step is to determine what kind of executable it is:
$ file the-binary
the-binary: ELF 32-bit LSB executable, Intel 80386, version 1,
statically linked, stripped
So, for now we know that
$ strings -n 8 the-binary | wc -l
368
There are quite many strings, but if we ignore those which can be found in standard libraries (using strings), we end up with:
9:uLAj;j 9:u!Aj;j [mingetty] /tmp/.hj237349 /bin/csh -f -c "%s" 1> %s 2>&1 /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin/:. HISTFILE /bin/csh -f -c "%s"So, first preliminary hypotheses:
The decryption is performed like this (D=decrypted, E=encrypted): Now, the second byte of the decrypted data is tested (in a structure
ressembling switch(...){...}). For now, I have tried to analyse only
certain types of packets (hopefully, these were the more important ones):
Packet Type 3 = Exec a command and return output
D[0]=E[0]-0x17
D[n+1]=E[n+1]-0x17-E[n]
The code is quite obfuscated by using sprintf()s instead of just storing
the data. Thus, if the decrypted data contain a null, the decryption
would probably fail. The inverse function (Encrypt) can be found at 0x804a194.
Basically, executes the command (which starts on byte 2 of the decrypted
packet) using a system("/bin/csh -f -c \"%s\" 1> %s 2>&1"), where the
first %s is replaced by the command and the second is "/tmp/.hj237349".
When the command finishes, this file is opened (using fopen()) and sent
using encrypted packets of similar structure (except for the packet type).
At the same time, the parent process sigkills the child (if it doesn't
finish in 10 seconds; timed using alarm()).