Unusual CPU Usage

SYSTEM INFORMATION
OS type and version Ubuntu Linux 20.04.5
Webmin version 2.001
Virtualmin version 7.3-1

Hello,
I’ve been observing unusual CPU usage for a few days on my server.
When I click on CPU it redirects to several WP plugins but plugins on different websites…

When I kill them the CPU usage drops but then back to 100%…

I am sharing the output of the “top” command with you.
Can you help me?
How can I get rid of this disease?

This is screenshot: https://i.imgur.com/tWgfE2w.png

What are cma_m_mailprefs and homeAug162010?

Whatever user those are running as has been exploited, I’d wager.

Those look like the randomized sort of names attackers use to make themselves harder to detect by automated scanners.

You need to update whatever apps that domain is running, and track down and cleanup all the various cronjobs and extra PHP scripts the attacker has put in place to reinstall itself if it gets shut down.

I don’t understand what you’re describing here. But, if you have multiple users running these processes, then more than one site has been compromised, presumably because they were running the same exploitable plugin or application version.

The same is true for 3 websites…
As I said, when I click on the CPU counter;

xyz123.com/public_html/wp-content/plugins/xyz/l.php
yyy.com/public_html/wp-content/plugins/yyy/l.php
zzz.com/public_html/wp-content/plugins/zzz/l.php

it’s actually running a file that doesn’t exist in that directory… I don’t know how it does it! After raising this topic here I disabled these sites and made these directories (public_html) CHMOD 000 with root user.

When I look now the CPU is 5% and everything is normal.
How can I find what triggered this l.php file?

l.php is in a directory under public_html so anyone in the world could trigger it by visiting the URL:
xyz123.com/public_html/wp-content/plugins/xyz/l.php

As Joe pointed out:

Your Virtualmin system is secure but some domains have been hacked. If you disable those domains, the Virtualmin server will function normally.

If you have a known good backup of the domains that have been hacked, you should restore them and then update WP and plugins and do whatever needs to be done to keep WP secure to prevent this from happening again.

The strange thing is that I deleted l.php from all directories.
I click on the CPU and kill the application doing this danger file.
It restarts after 10 minutes.
Where can I find out how started it?

Repeat: l.php is currently not available anywhere.

I know exactly what you are going through.

But this thread is about high CPU usage and the community has collectively diagnosed and offered a solution to that issue.

It is due to Virtualmin’s robust architecture, which has evolved to what it is today, that the hackers were not able to take control of your entire server after they got in through one of your domains. It now falls upon you to teach yourself how to best secure WordPress so that there is no strange behaviour on your server; that is perhaps not a topic directly related to the Virtualmin forum.

@tpnsolutions is an excellent consultant. Perhaps you could discuss it privately with him?

Of course it does. The attacker obtained access as the user the domain runs as. They were able to add cronjobs, other scripts with different names, etc.

Killing one process is not what cleaning up an exploited user account looks like.

As I said above:

Restoring from a known-good backup of the domain, and then immediately updating the apps and plugins and removing any you aren’t very confident you can trust, would be one way to get there. You probably want to backup the current database before doing that, so you don’t lose any data added since the attack occurred. But, beware that databases can also execute code and can run malware, and someone with domain level access could also find out passwords for databases and anything else stored in config files in the user home. I don’t think I’ve ever seen databases used as a reinfection vector, and it would require problematic application code to escape, but we already know you have problematic application code, since the attacker got into multiple sites. (For instance, if the site has a maintenance or queue processing cronjob, it may be exploitable by something in the database to run commands on the system.)

Attackers have “kits” of tools they install, which sets up various processes and cronjobs, and modified files within your app, to reinstall their malware whenever you remove any single file. The domain index.php and many other pages may have been modded to reinstall the malware, there may be cronjobs, a remote shell so they can login and poke around if the system stops phoning home, various files that look harmless but aren’t.

If you leave any of the attackers files, they can probably get back in. If you leave the application(s) in the state they were in, they can (probably) get back in (some attackers patch the exploit they came through so nobody else can take over their new botnet member). But, they also install some new backdoors of their own.

The only sure-fire way to clean up is to backup everything important, delete the whole domain, recreate it, reinstall the applications fresh from the upstream sources (installing WordPress with Virtualmin grabs it from the WordPress download servers), and restoring databases and data…but, not any of the application/plugin code from the exploited site (because none of it is trustworthy without going over it with a fine-toothed comb).

You’re fixated on this one file, l.php. That one file is not the problem. The problem is that one file proves the attacker was able to create executable files on your system. If they can do that, they can do anything the user can do.

The good news is that they probably did not escalate to root (root escalation is not usually easy, on an up to date Linux system), since if they did they would be able to hide all their files and CPU usage from you. You can only be (somewhat) sure about an absence of root-level exploit via external observation. Either booting from a trusted read-only filesystem and looking around, or observing network data in and out, since people exploit systems for reasons, usually mundane reasons like sending spam or participating in DDoS attacks or crypto mining.

You can’t trust anything in those domain’s home directories at this point, especially anything in executable directories.