I’d look at the process list (using ps
), and the biggest CPU/memory users (using top
). If something is using a lot of CPU, see what user is doing it. That could be the compromised user (if a user is compromised, rather than root).
I’d check the maillog. A lot of hackers just want to be able to send spam. Maybe they do it through the local mail server (but maybe they do it some other way, which won’t show up in the maillog).
And I’d check the error_log and access_log for the compromised user. If the goal was to serve something from your webserver (like malware payloads), you might see it here.
Then, I might check network traffic for things that don’t belong using tshark
. Malware often self-replicates, so in addition to programs to hide itself and reinstall itself if it gets deleted/stopped, they often include a scanner to look for other old servers that can be compromised and added to the botnet.
I’d look for hidden files and directories (files that start with .
) in the compromised home directories. Something like find . -name ".*" /home/domainname
can maybe do that (I’m not sure about the .*
glob, it might need something more specific).
If you have not identified a compromised user after all of this, but you still think you’ve been compromised, it is pretty certainly a root-level exploit, and they may be able to completely hide themselves from you. The only way to see a root-level exploit with certainty is an outside observer. Either you boot from read-only media (like a rescue disk), and then mount up the system filesystem(s) read-only and poke around, or you sniff traffic from outside the server itself or both. Attackers want to do something with your system, so if you shut off all of the expected services but still have traffic, you may be able to at least prove it’s been compromised, but may not be able to determine how or to what extent (again, if they have root, you can’t trust the system).
I might, as a hail Mary, run rpm -Va >/root/package-verify.txt
and look through that for evidence of changed system files. Rootkits change system files so they can hide themselves, and they usually don’t do it with packages. So, an otherwise invisibly rooted system might still show you some clues because system files like passwd
, ps
, top
, etc. don’t match what’s in the package. If you did see that, it would confirm a root-level exploit, if a sloppy one.
You can try tools like chkrootkit
. But, if the attacker has root, running it won’t prove you don’t have a rootkit, because they can hide themselves from tools that detect rootkits. You have to run those kinds of tools from read-only clean boot media (like a rescue disk) in order to have any faith in their results.
There are also tools for detecting WordPress malware, and the like. I have no experience with those.
That’s high level. You need to get deeper than that, though, and it’s well past being a Virtualmin issue.