Operating system: CentOS7 OS version: CentOS Linux 7.9.2009
Morning everyone.
My provider blocked one of my servers yesterday as they suspect it has been compromised.
They gave me access this morning.
The info they gave me was that they received a report that this server has been carrying out malicious network activity, including attacks on servers elsewhere on the Internet.
They provided a log which had a lot of this (ip hidden by xx)
I have 6 wordpress sites on that server, so once I had access back this morning, I have scanned them and checked them all for issues. Wordfence flagged some on one of the sites, the rest were fine.
I have iptables configured just using the default settings (block all except ports used for virtual hosting on itnerface). Is this enough?
Is it possible to do a scan (malware) of the whole server?
Finally, what can I check for in my logs to see if it does specify a little more which virtual server is the cause?
You should begin with a manual scan of the log entries sent by your service provider; then (in case the suspected virtual server is still online) use ‘top’ and ‘atop’ to identify scripts which are taking up an inordinate amount of CPU cycles or RAM. Use a tool like Matomo to import and analyse your logs to find scripts which have been invoked very often recently.
You should do all this while others post things for you to do which I may have missed.
Regarding IPtables, is there a guide available somewhere which I can follow to strengthen what I have?
How do I do a scan of the whole server?
The service providor logs I received didn’t help a great deal to identify the virtual server itself. Just the IP of the whole server. I obviously have the log files within webmin/virtualmin that I could use?
I’m also seeing a LOT of this in the messages log file which I fear may be some cause for concern?
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns2.serveroffer.lt/A/IN’: 2001:678:6::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/AAAA/IN’: 2001:678:19::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns2.serveroffer.lt/A/IN’: 2001:678:4::4#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/A/IN’: 2001:678:6::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/AAAA/IN’: 2001:678:6::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/A/IN’: 2001:678:4::4#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns2.serveroffer.lt/A/IN’: 2001:678:8c::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns2.serveroffer.lt/AAAA/IN’: 2001:678:8c::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/AAAA/IN’: 2001:678:4::4#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/A/IN’: 2001:678:8c::1#53
Dec 5 11:14:05 hostnamereplaced named[23137]: network unreachable resolving ‘ns1.serveroffer.lt/AAAA/IN’: 2001:678:8c::1#53
Dec 5 11:14:06 hostnamereplaced named[23137]: network unreachable resolving ‘mone183.secundiarourous.com/A/IN’: 2603:5:2240::1#53
Dec 5 11:14:06 hostnamereplaced named[23137]: network unreachable resolving ‘mone183.secundiarourous.com/A/IN’: 2603:5:2140::1#53
Linux Malware Detect (LMD), also known as Maldet, is a malware scanner for Linux.
It is particularly effective for the detection of php backdoors, darkmailers and many other malicious files that can be uploaded on a compromised website.
rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. It does this by comparing SHA-1 hashes of important files with known good ones in online databases, searching for default directories (of rootkits), wrong permissions, hidden files, suspicious strings in kernel modules, and special tests for Linux.
Lynis is a battle-tested security tool for systems running Linux. It performs an extensive health scan of your systems to support system hardening and compliance testing.
You do realise, @cs10, that Virtualmin has been designed such that the entire system is not compromised when any single virtual server is hacked? All you need to do to fix your present problem is to identity the offending script or restore the compromised / suspect virtual server from a known good backup.
@cs10 hi, first of all download your current (suspected compromised) wp install including current database for scientific investigation and keep it somewhere zipped out of your server. As @calport said you can restore the clean last working backup you have, but that is just temporary as your attacker may come back and repeat the hack… I am not saying that it is going to happen but good possibility is there - because if there is somewhere way in and if it was compromised it will be done again and again - so to just restore backup and forgot about it is just ill and stupid advice. You should investigate what did happen and if it happen so you would know how to fix it and prevent it.
You should mainly harden your security for that site and perhaps apply your solution for other sites too… for wp-login.php that is not that hard… you can have combination like dual login with apache assistance or even set to allow apache to allow see admin site only by your ip etc. for malware and php shells there is nice solutions within php it self… I got some .php files, you upload one run it via browser, wait few seconds and you will see if there is anything dodgy.
also you could deploy fal2ban and write some small regex against brute force just for wp-admin it self + force wp-admin to be load only via ssl, there is also option to deploy your own ssl verification, basically using your own ssl will allow you to display page like wp-admin etc… otherwise the one who will not have your ssl in browser would not be even able to log in even if he would have password and user name, there is tons of options
if you really in panic mode you can set up email and or push notification for your phone when anyone log into your server via ssh, be it root or normal user or whatever… and please dont use ftp or ftps… use only ssh or sftp.
for investigation you would ideally need compare last working backup with hacked copy. Also once you restore clean site, make sure you create md5 check list of all wp site and keep it safe offline on your computer. If any change occur even in htaccess or you would have any suspicions something is not right, just re-upload the checksum list up and check it via terminal… if something was changed by anyone or anything terminal will tell you which file it was and it does this within second or two…
if you need any help more deeper with this, feel free to pm me or contact me via my profile.
@unborn is correct. My advice was incomplete and therefore stupid.
Restoring from the last good backup would solve the problem only temporarily; the permanent solution (and correct thing to do) would be to fix the vulnerability in the script and harden server security, as @unborn correctly went on to recommend.
I do apologize for the incomplete advice that I had offered and hope that this sets the record straight for future readers of this message.
I have nothing to add in terms of remediation, but in terms of prevention of future problems…
I make it a point to change the SSH port to some unused one, and set up CSF to block IP’s (and Class C’s in the case of hits by near neighbors) that hit 22 more than twice in 24 hours. I also have CSF block IP’s (and Class C’s in the case of hits by near neighbors) that hit 3389 even once.
The blocked IP’s are then added to a blocklist that I maintain, as well as submitted to AbuseIPDB. (This is my page there.) My blocklists are self-rehabilitating, so IP’s will automatically drop off after two or three days of good behavior.
I also have CSF configured to detect and block many other kinds malicious acts including hits on non-existent CMS login pages on hand-coded sites, repeated bad login attempts (including to services that don’t even exist on a given server), webmail spam, and form submissions on dedicated honeypot pages like this one.
CSF makes this very easy to do because of the BLOCK_REPORT functionality, which allows every block to trigger a script, and passes enough arguments to enable a sensible report.
But even without reporting, it also makes it easy to quickly block bad actors; and one of the most effective ways to do that is to change the SSH port and report attempts on both the standard SSH and RDP ports, essentially making 22 and 3389 into honeypots.
You can also do the same with other service ports to varying extents.
I don’t mean to imply that simply changing the SSH port is enough in terms of hardending security. But in conjunction with blocking (and reporting, if you want to go that far), it’s a good way to quickly neutralize bad actors.
