250% CPU usage?

SYSTEM INFORMATION
OS type and version Ubuntu 20.04.6 LTS
Webmin version 2.105
Virtualmin version 7.10.0
Related packages Apache/MariaDB/PHP8.3

Hi all. Although I have a bunch of Webmin installs running perfectly fine on a variety of servers, I have one Virtualmin instance that has me scratching my head a bit. It’s running fine, as far as I can tell, but according to running processes I have one process reporting CPU usage of 198% which is obviously impossible.

The details are:
command [card0-crtc0]
parent process: /sbin/init
user/group: www-data/www-data
size: 2.12 Gib

The Ubuntu server is a virtual machine and has no need for video other than command line console.

I’d appreciate any ideas on what’s going on. It’s hard to monitor true CPU usage with this potentially phantom process showing an impossible figure. I don’t want to screw the pooch by digging into this without simply running it by the collective mind here first.

Thanks.

That’s what top reports from the console so it looks “legit” but makes no sense to me.

I don’t think, but will stand corrected, Webmin/Virtualmin also have no need for video. So where has it come from? What/Who installed it? Could it be a simple stop the process and remove whatever is starting it?

A command named “2c29ecac014485a” looks legit, to you?

That looks extremely suspicious to me. I would assume you have an exploited web app running a cryptominer, or some other nefarious something.

Also, it’s running as www-data, so I think you’ve installed mod_php, which you shouldn’t have done. The attacker has access to all of your websites, since Apache has access to all of your websites (the preferred execution modes would limit attackers to only the user that owns the app they exploit).

As I think it through this morning, it appears that this isn’t a virtualmin issue (specifically) but more of a general Linux issue. A month ago I had a web site get exploited via a WordPress plugin. I’m fairly certain all the malicious PHP files are gone but the fact www-data was accessing card0-crtc0 and then trying to access a since deleted file:

/home/userxyz/.wp-cli/2c29ecac014485a6a5504ad11a92fc2f (deleted)

Is worrisome. I’m an experienced IT systems admin/consultant but my linux expertise is, say, intermediate but getting better every day!

I responded to the previous post before seeing yours. I meant to say that the executable wasn’t uploaded code but that it was attempting to access a since deleted file. To your point, I don’t recall installing mod_php but it’s entirely possible that this happened during my install of PHP8? Crud. Okay. I’ll look to uninstall it. Without looking it up, I don’t know offhand what mod_php even does. Thanks. Truly appreciate of any pointers here.

Say no more, A common :hole: to fall into.

If you followed random docs on the web rather than ours, that’s pretty likely. The way the dependencies are setup in the Ubuntu/Debian PHP packages, mod_php is the preferred execution mode to be installed if another one isn’t specified…and some documentation explicitly includes installing mod_php, not merely accidentally doing so. Both are irresponsible and dangerous, but I have no control over anything other than our docs and our packages.

But, merely uninstalling mod_php won’t give you any assurance your system is no longer exploited/exploitable.

You’re chasing ghosts. The files are going to move around…new names, new paths. There’s probably a cronjob that reinstalls the exploit, probably a bunch of backdoors that they can query that have been inserted into the various WordPress files, again to reinstall their code. This account (and potentially this server) is toast. You can only ever be sure this account is secure by restoring a backup from before the exploit and updating WordPress and all plugins (hopefully that prevents it being exploited again).

You could try reinstalling all the WordPress files from known-good sources and making sure any extra PHP files not from WordPress and plugins get deleted. That may or may not catch everything (probably not).

Oh, but I should be clear, mod_php is not why your site was exploited. It helped after the fact. It’s a mistake to have mod_php installed, in the general case, not specific to your problem.

Hi Joe. I greatly (humbly and gratefully) appreciate your explanation. I have to ponder this for a bit on what the heck to do now. I was pulling my hair a few weeks ago out trying to clean the site(s) as I assumed that some exploit in a very common WP plugin across sites was the culprit. It’s something that just happens on occasion to customers over the years. This time, however, for a week or two, malicious php files kept popping up in random locations within the various account folders (as you suggested.) I think I’ve examined nearly every php file in every folder looking for any malicious code. Reinstalled every plugin and examined all WP core files.

Just so I understand correctly what might have happened. Flawed WP plugin allowed malicious code to execute. Malicious code somehow was able to use elevated permissions due to mod_php being enabled. Without objection do you mind me sending a short PM?

Thanks, again.
Matt

That’s one common way attackers get in.

mod_php means the code runs as the Apache user. It already has elevated permissions (not root, but more than a normal user in that Apache can read other user’s homes). mod_php has some protections and if the attacker wasn’t explicitly trying to gain access to other user homes, they may not have done anything in those other user homes. But, mod_php is known to be leaky, and because mod_php has been deprecated for many years and the PHP devs tell you not to use it (and there is never any reason to use, it is slower, bigger, and less secure than alternatives), it doesn’t get a lot of attention.

mod_php is not your big problem here. It is a problem. You shouldn’t have installed it, and you should uninstall it, and if you start over with a fresh server, I’d advise you make sure you don’t make that mistake again. But, it’s not how the attacker got in. If other users have been exploited, though, that’s possibly how it happened. Removing mod_php does nothing for your currently known exploited user, though, and manually going through and playing whack-a-mole with the files the exploit produces is probably not going to do anything useful. You have to prevent it from re-installing itself, and it probably has many mechanisms for doing that.

Attackers have toolkits they use to insure their exploits are sticky and hard to defeat. Once they’re in, they have a lot of mechanisms to make sure they stay in. It’s automatic and it’s probably quite fast. By the time you find one exploited file and fix it, some other file or process sees that change and puts it back (maybe with a randomized location and name, so you don’t immediately spot it again).

That’s why I’m saying the only way to be (mostly) sure would be a restore from a known-good backup (and, if you didn’t spot the exploit quickly, you may not know how far back you have to go to find a known good backup). Replacing all the files with known-good versions directly from the WordPress site or the plugin authors would maybe do the thing. You might have to make it all owned by root while you make those changes (because they may have cronjobs to bring their files back, they may have endpoints that allow them to trigger a reinstall).

Since you can see these processes means they probably don’t have root (if they had root, they could install la rootkit that would hide their cryptominer, or whatever, processes). So, you’ve got that going for you. It probably can be cleaned up, but it won’t happen by just deleting a few files that look suspicious.

I’m not able to provide private support with our Open Source software. I do the best I can to answer questions here in the public forum, given the time I have. And, I’ve told you what I know about the way attackers operate; I’m guessing without having looked at your system (and I’m not going to look at your system), but it’s guessing based on some experience with these kinds of exploits.

https://forum.virtualmin.com/guidelines

Fair enough. You’ve gone above and beyond as it is. A thousand thanks!

Spending all day on this, there’s a chance I’ve cleaned it up. Far too early to claim victory. There was a malicious cron job running under www-data. Combed through all cron jobs for all users. Stopped cron. Disabled job. Killed invading processes. Deleted some files and restarted services. I’ve been watching both netstat and processes all afternoon. Nothing looks amiss but it’s far too soon to have any confidence. I’d like to avoid the nuclear option if possible.

Anyway, I think It’s only right for me to convert to a subscription. I’ve been very impressed with Virtualmin and Webmin, and I can’t thank you enough for your assistance today. Cheers.

Matt.

Install wordfence, will protect WP.

Thanks. That was installed on day one but didn’t prevent the intrusion, unfortunately. This was an eye-opener for me in that even though customers have had experienced hacked web sites before, this was my first experience with an exploit breaking through to the OS. I don’t think it had free rein, as not all web sites had malicious php files created, based upon analysis of the data but I can’t be positive. Oddly, no php files were modified - only new files created. Regardless, it makes for an unpleasant situation. Really regret not understanding that mod_php wasn’t necessary or that it was such a security hole. Oh well… Thanks to all.

1 Like

I’ll repeat, mod_php is not how they got in. mod_php makes exploits more dangerous, but in this case, you had an exploitable web application, presumably something in WordPress or a plugin. You may still have exploitable web application, even if you cleaned up all the files.

mod_php is not a security vulnerability in and of itself (as far as I know, I am unaware of any current exploits), but because it runs an attackers scripts as the web server user, they potentially have more access than they would in a default Virtualmin deployment, which runs apps as the domain owner user, either with PHP-FPM or mod_fcgid+suexec (the former is recommended in all modern systems). And, because its use has been discouraged by the PHP developers for at least a decade (and we’ve never recommended it, going back ~20 years) it’s not getting a lot of development attention. If it has security problems, they don’t get attention.

So, you have two problems: Exploitable web app. mod_php installed. The two are not directly related.

1 Like

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.