Root Password Compromised!

Posted this last night to https://www.centos.org/modules/newbb/viewtopic.php?topic_id=25614&start=0#forumpost103783 as it’s not a Virtualmin issue per se, but not had a response yet.

During a routine look at my servers logs I noticed on April 1st I had 3 successful login attempts to root that wasn’t me (I’m the only person that has access to the server).

The IP address for each attempt is from Turkey, checking the logs for 78.160 gives:

Mar 15 10:59:41 servername### sshd[23980]: reverse mapping checking getaddrinfo for dsl78.160-27118.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 15 11:00:35 servername### sshd[24004]: reverse mapping checking getaddrinfo for dsl78.160-27118.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 15 11:00:58 servername### sshd[24004]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=78.160.105.238 user=root
Mar 15 11:01:00 servername### sshd[24004]: Failed password for root from 78.160.105.238 port 4603 ssh2
Mar 15 11:02:59 servername### sshd[24474]: reverse mapping checking getaddrinfo for dsl78.160-27118.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 15 11:02:59 servername### sshd[24474]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=78.160.105.238 user=root
Mar 15 11:03:02 servername### sshd[24474]: Failed password for root from 78.160.105.238 port 4624 ssh2
Mar 15 11:03:15 servername### sshd[24475]: Connection closed by 78.160.105.238
Mar 15 11:03:15 servername### sshd[24474]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=78.160.105.238 user=root
Apr 1 02:21:16 servername### sshd[8266]: reverse mapping checking getaddrinfo for dsl78.160-30835.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 1 02:25:35 servername### sshd[8407]: reverse mapping checking getaddrinfo for dsl78.160-30835.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 1 02:25:35 servername### sshd[8407]: Accepted password for root from 78.160.120.115 port 4090 ssh2
Apr 1 02:53:17 servername### sshd[9232]: reverse mapping checking getaddrinfo for dsl78.160-29042.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 1 02:53:17 servername### sshd[9232]: Accepted password for root from 78.160.113.114 port 1567 ssh2
Apr 1 03:11:54 servername### sshd[10282]: reverse mapping checking getaddrinfo for dsl78.160-31884.ttnet.net.tr failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 1 03:11:54 servername### sshd[10282]: Accepted password for root from 78.160.124.140 port 2538 ssh2

You can see the 3 successful attempts and related break in attempts which apparently worked!

I guess that means I have a security hole, but I don’t know enough about server security to track it down.

Dedicated server runs Centos 5.4 and Virtualmin 3.77 GPL. I keep the server regularly up to date via Virtualmins built in upgrade facility.

I regularly change the root password, last time was early Feb (and just now) and it’s random with special characters (no way it would be guessed).

I’ve only just discovered this problem and nothing jumps out as changed on the server. Ran chkrootkit and don’t see anything that jumps out.

Recently had my FTP password acquired I believe through Filezilla storing the passwords in a plain text file (no longer do that!) and I was running an old version of Adobe which can apparently give access via an Internet Explorer plugin. Although a dozen of my sites were compromised it was easy to fix (a pain, but easy) and as they didn’t have root access there was nothing changed on the server per se (uploaded various Trojans and malicious code to the public folder of various domains that basically redirected my sites traffic). To be on the safe side, reinstalled every domain from scratch with new passwords but as there was no root access believed it was secure (I realise a server isn’t secure from a skilled hacker).

This does not appear to be the same sort of thing.

Looking for advice on what’s happened and what to do about it?

Thanks.

David

not sure what happened as you dont see anything changed on the server and cant detect a rootkit.
however to prevent anything like this, you’d probably want to deny root access with a password.
So you have to create a pair of keys and use that to log into the server.

A second thought is to create a user with some odd name and give that user root access.
In any hacking attempt, 50% is the username and we all know about root…
the new user should then only be able to log in with a key

It could very well be that the hacker got access to one of your users accont through some faulty script and tried to level to root access from there

Some thoughts in addition to what Ronald mentioned – it can be tough to figure out exactly how someone broke in without spending a lot of time doing forensics.

However, in addition to chkrootkit, you may also want to try running rkhunter, which does some further checks.

Also, make sure there aren’t any SSH keys setup in /root/.ssh/authorized_keys, as those can allow remove access even if you were to change a password.

You’ll also want to check if any cron jobs were added to roots crontab… you can run “crontab -l” and look through the output to make sure it all looks legitimate.

And lastly, just to be super-safe, you may want to run “yup update” from the command line to make sure everything is fully up to date.

-Eric

Thanks for the tips, working though them now. Server security isn’t my thing, so slow going.

I’ve been using a different port for SHH for a while and it does help, but the hackers do find it eventually, guess I need to change it more often. For anyone looking to change the SHH port it can de changed via Virtualmin:

In Virtualmin: Webmin/Servers/SHH Server “Edit Config”

change the line

Port 22

to

Port 1234

(where 1234 is an unused port).

Is there an easy way to choosing unused ports, been using various online port checkers that says a port isn’t used, but when I switch to that port I get a warning in the secure logs like:

Apr 4 00:39:40 ###### sshd[3668]: error: Bind to port #### on 0.0.0.0 failed: Address already in use.

It still works, but worried I’m going to choose an important used port (I’ve tried ports over 5,000 and over 50,000 with the same message) and break something.

Checked /root/.ssh/authorized_keys and the file doesn’t exist.
Checked cron jobs and no issues.

Hoping the hacker had not got around to doing anything bad on the server, though think I’ll get a new server (could do with more memory anyway) and when I’ve figured out how better to secure things on the old server, use the new one and close the old one.

Did find this file /root/.ssh/known_hosts is it important if you shouldn’t have other hosts?

I’ve got a single entry in that file associated with an old server, looks like when I used backups from the old server to install Virtual servers this was added.

The entry is

oldserverIP ssh-rsa longstringofrandomcharachters

Is it safe to delete this file completely or delete the contents of the file? I looked for a way to find this entry under Virtualmin/Webmin but couldn’t find a setting.

Didn’t know you could disable root access, but still login to root. Anyone know a thorough tutorial on this? I’ve been reading http://wiki.centos.org/HowTos/Network/SecuringSSH#head-9c01429983dccbf74ade8674815980dc6434d3ba “7. Use Public/Private Keys for Authentication” which I’m going to try.

To confirm, I create a set of keys, disable root password access and use the keys to access the server.?

Or do I do the above AND login via another user name and then su to root (not used su before)?

BTW if anyone reading this writes tutorials (instructions) for this sort of stuff, try not to assume we all know what you know (been reading a lot of confusing articles on server security recently): I write SEO tutorials/articles and it’s easy to forget what I consider obvious isn’t the case for everyone. So many tutorials assume the reader has a good level of knowledge and it makes understanding the instructions very slow going and it definitely puts me off trying some of this stuff (get it wrong, no access to the server!!!). Step by step instructions like run “crontab -l” is always appreciated as I wouldn’t have known to run that command for a quick list of all cron jobs :slight_smile:

Thanks again for the advice.

David

Well, I’m not sure rotating SSH ports is necessary… just keeping it off port 22 should keep most of the baddies away, but someone will always be able to find it.

As far as what port to use – if you’re picking one between 5,000 and 50,000, I’d suggest just not picking 10000 or 20000 (the Virtualmin and Usermin ports).

Hoping the hacker had not got around to doing anything bad on the server,

That’s doing a lot of hoping :slight_smile: You’d definitely be safer by moving to that new server you mentioned…

Did find this file /root/.ssh/known_hosts

You don’t need to worry about the known_hosts file, that’s not going to cause you problems.

To confirm, I create a set of keys, disable root password access and use the keys to access the server.?

That sounds about right, though, what I’d do is verify that the SSH keys work before disabling root password access :slight_smile:

Or do I do the above AND login via another user name and then su to root (not used su before)?

That’s up to you and how paranoid you’re trying to be :slight_smile:

-Eric

Few days ago the server became unstable, basically trying to use it was like wading through deep mud with an elephant on my back :slight_smile: Everything ran super slow and was unusable.

Don’t know what the problem was, one of my sites had a big traffic increase, but it was only an extra 10,000 visitors a day, so that shouldn’t have caused something this dramatic. It felt like a DDOS attack or I was trying to run the latest dedicated server software on a 386 with barely any memory!

Couldn’t keep my sites up so got a second server, (upgraded, double the memory, two HDs) installed Virtualmin and recovered the Virtualservers from the latest backups (couple of days old, so didn’t loose much).

I had the HD from the old server mounted as a second HD on the new server so I could recover the backups and eventually anything important lost that was added those two days between the backup.

I had a copy of the backups offline and had scanned them for viruses with Avast and there was nothing found.

Everything was running perfectly until yesterday.

Changed the server time and time zone (via Virtualmin/Webmin) and as it went to GMT time zone Virtualmin appeared to crash. First got a pid error (didn’t note it down), tried to reload Virtualmin, but it wouldn’t reload. Everything else was working fine on the server.

Assumed Virtualmin had crashed, planned to log into SHH and do a soft reboot, but as I logged in got the Host Identification Failed warning with info about intruders etc…

Didn’t log in to SHH.

Got the dedicated server company to hard reboot and the server came back online, but https access seamed to come online quite slow. I was trying to load Virtualmin and it didn’t load while http connections were fine. Within a few minutes I could access Virtualmin normally.

Tried a SHH connection and still had the Host Identification Failed warning, so haven’t logged in via SHH yet.

Checked the server time and time zone and it had changed to GMT, time was wrong, so fixed that.

Contacted the dedicated server company to see if they’d made any changes to the server over night and they said no.

Changing time zone wouldn’t change the host ID (right?), so don’t know why I’m getting this warning.

Obviously when I first logged into SHH to the new server I got this warning (it’s on the same IP as the old one) and I accepted the change as it was a new server. But I shouldn’t get this warning again.

Everything is working fine on the server, checking the secure logs I see the usual hacking attempts, but so far no one has compromised the server.

The servers running Centos 5.4 (fully updated via yum), installed Virtualmin via the install.sh file with no major problems and it’s updated to 3.76.gpl (not sure why it’s not 3.77).

I’ve not had time to make major server modifications as was getting all my sites back online etc…

The second HD I mounted was a pain to do. Both HDs had the same volume name so the backup one wouldn’t mount the easy way.

I found a fix that didn’t involve going into rescue mode and unplugging one HD… and mounted the second HD. Looking through the Webmin Filemanager module the backup HD is still on the system (and has the new volume name I gave it), but the files are not in the folder where I mounted them.

The HDs both had the label VolGroup00.

I worked out which HD was which and used the UUID to rename the old HD to VolGroup01 and could then mount it.

Used this commands

vgrename UUID VolGroup01

Where UUID was from the backup HD.

mount /dev/VolGroup/LogVol01 /mounthere/

and it worked without having to remove a HD or boot into rescue mode.

Recovered my domains from the latest backup and took a break.

After making the change above I don’t think I logged back into SHH, but like with the time zone change that wouldn’t change the host ID, right?

I’m far, far, far from an expert on servers (learn what I need as I need it) hence giving all the details above in case something above would result in a host ID change. If not that leaves a man in the middle attack!

If it is a man in the middle attack how do I confirm this?
And having since logged into Virtualmin and one virtualserver via SFTP (it was already connected as this happened and after the reboot automatically reconnected to finish an upload) have I already lost my root password and one virtualserver password?

I’m really beginning to hate dedicated servers!

David

Well, if someone had once broken in as root, they’d have had the opportunity to install a rootkit, which could potentially allow them access to your server.

SSH host keys don’t tend to change, even if SSH is updated/upgraded.

The fact that you’re seeing an error suggesting it has makes me quite suspicious :slight_smile:

Some rootkits do make modifications to SSH, and possibly the SSH host key…

So it’s possible you’re running into problems related to the breakin.

-Eric

This is a new server with the old HD as a backup which looks like it wasn’t mounted permanently (guessing the mount info wasn’t saved after the reboot).

So even if the old server was compromised with a rootkit how would a hacker access it on the backup HD?

All I’ve taken from the old HD are the virtualserver backups which I reinstalled one by one. So none of the Virtualmin/Centos 5.4 files per se.

I’d scanned an offline copy of the backups for viruses using Avast, no issues. I’m assuming Avast scans for rootkits, when I’ve had malicious code in virtualserver backups Avast has found it.

So Centos 5.4 is new, Virtualmin is new, so unless there was an exploit in one of the virtualserver backups I don’t see how a rookit could be accessed on the old HD? I mounted it in a new folder as well which without access to the server no one would know where it is.

I think what I need to do is rule out a man in the middle attack and I have no idea how to do that?

If I can’t find a compromise it suggests there’s nothing wrong and I’d have to accept the new host ID before connecting via SHH.

David

From my previous answer:

It could very well be that the hacker got access to one of your users account through some faulty script and tried to level to root access from there

If this indeed happened and you restored the domains, likely the rootkit is inside one of one of those domains. Or its not a rootkit but some script connecting to an outside server sending nasty commands to yours.

Do you trust your users? Do any of them have ssh access or other elevated rights?

Stumbled on some of what the hacker did when they got my root password on the old server.

They changed the Google AdSense publisher Id on at least one of my sites!

Looks like they got the ad revenue from about 30,000 ad impressions!

Contacted Google AdSense, so guess Ill get it back and the publisher ID that it was changed to will loose their account. Pretty stupid thing for a hacker to do as it takes 2 months for an AdSense payment and will only take one complaint like mine to get their entire account deleted and they won’t get a penny if the change is noticed within 8 weeks! Who is going to not notice a drop of 30,000 ad impressions over a week.

These changes did not show up in the FTP logs so they must have edited the files another way that left no obvious trace.

Still not figured out how to rule out a man in the middle attack attack, so still can’t safely log in via SHh.

David

did they really get root access or just hacked that website?

you can set up Kerberos which is not so sensitive for MIM attacks also enable SSH’s StrictHostKeyChecking option would lead to better security.

you may also want to use:
http://cannibal.mi.org/wwwnfr.html
to find out whats going on with your box.
More documentation:
http://www.usenix.org/publications/login/1998-5/lalonde.html

Yes they had root access on the old server.

The problem I have right now is I can’t login to SHH direct because I get the Host Identification Failed warning on the new server. Which could be a MIM attack, I need to rule it out.

I’ve been running commands through Virtualmin/Webmin.

Looking for a way to rule out an MIM attack without being able to log into the server with SHH.

On the new server not had any unauthorized SHH connections (so no root break ins as far as I can tell).

I’ve installed Chkrootkit and RKhunter on the new server (via Webmin Command Shell module) and nothing shows up. Is there a way to run those to search every file on a server (scan the home directory for example)? From what I can tell they don’t scan those files.

If there was a MIM attack would logging into Virtualmin leave the server open to attack or is it just a SHH connection that’s the issue?

David

First, it’s “ssh”, not “shh”… “shh” is what you tell someone if they’re talking too loudly :slight_smile:

Second, if you had been using the same name to access the old server as you are the new one, that’s a legitimate change that would cause ssh to raise a red flag. That is, the ssh client on your desktop doesn’t know you changed out the server.

In that circumstance, it’d warn you about a man in the middle attack. But if you just changed out an old server and replaced it with a new one, and it uses the same domain name as the last server, I wouldn’t be too worried about a man in the middle attack :slight_smile:

If you’re feeling paranoid, the thing to do would be to verify the fingerprint of the SSH host key on the server. There’s some explanations on how to do that if you poke around on Google, including this one:

http://cafe.elharo.com/security/verifying-ssh-host-fingerprints/

-Eric

Think I’ve found the problem.

Checked the keys the SSH client I use had stored and it looks like it’s using the wrong one!

I logged into the old server last with port 1234 (not literally).

I first logged into the new server with the default port 22 and saved the new host key.

I then changed the SSH port on the new server to 1234 (used the same port as didn’t have an easy method to find unused ports). I’m assuming I made the port change the day before the problems, but didn’t log into SSH after making the change (I don’t recall).

In the folder that stores the keys on my work PC there’s only one key associated with port 1234 and it is dated 3rd April which is 3 to 4 days before I got the new server (I changed the port of the old server around about then to 1234).

So my SSH client is checking the old servers key for port 1234!

The port 22 key is dated Apr 07 which is when I got the new server.

The new key with port 22 has the same key (looks the same, it’s 500+ characters in size so not easy to check) as that in the file /etc/ssh/ssh_host_dsa_key.pub on the server (I’m getting access to that file via Virtualmin).

So no MIM attack, just my SSH client loading the wrong key because I didn’t find a new port!

Confirmed all this by switching the port back to 22 and had no problems connecting to SSH.

Blinking typical hey.

Thanks for the help.

David