It turns out that a Cron job was added which periodically added infected files to my server. This most likely was caused by the vulnerability in JCE Editor. Once the Cron job was removed and the infected files were cleaned, my website is back to normal.
One question: in order to see these Cron jobs, I needed to look at the cron.d, cron.daily, cron.hourly, cron.weekly, cron.monthly, cron.yearly directories in /etc. When I looked at the system-scheduled cron job tab, I don’t believe that I see all the jobs listed. I could be wrong here, but I don’t remember seeing a cron job for my affected server. I was signed in as the root user. Does this tool not list all the cron jobs? I am not sure I am reading this correctly. If it does not list all cron jobs running on the server, it is possible a malicious cron job could be inserted on our system.
When you installed the JCE editor you allowed to create the cron job.
Virtualmin/Wemin is not allowing malicious jobs to be scheduled without permission.
All software must be evaluated and continually updated , so sorry to say but the bottom line is you did not keep up2date on what you installed (or what someone else installed) and that is why you were vulnerable to an exploit.
Were there really cronjobs made by the attacker in those directories?
If so, you’ve been rooted. You can never trust your server again. They likely installed a rootkit, which will hide their other nefarious activities.
If it was a user cronjob, then it’s not the case. But, it really matters whether it was an administrator-created cronjob or a user cronjob. Users can create cronjobs, but not in e.g. /etc/cron.daily.
I don’t believe so. I am not sure where the Cron job was because it was removed by a 3rd party. However, that person did not have root access but only to one of my virtual servers, the affected one. It has now been 5 days since it was removed and there have not been further attacks. They were occurring daily but only involved a single virtual server. I might not have been clear. My question was only, are all cron jobs displayed in the tab–System/scheduled cron job or does it just display a subset of them? Because if something like this happens again, I like to know where to look. I have changed my root password since this happened.
There seem to be several ways the intruders used when making use of the jce flaw.
We managed to fix this (on a friend’s server) by:
as the vhost user in shell/via ssh: crontab -l - and if there was something - crontab -e and remove the corresponding line
restarting the php-fpm service of the corresponding PHP version. Where must have been some process running which recreated the files in public_html/tmp dir every day a new
we’ve got daily/weekly/monthly btrfs snapshots so we could always compare the contents of the public_html and tmp dirs with the one from some time ago
and restore the database from backup, if there were profiles created by the intruders
we did protect the site with an htaccess, too - but with basic auth, so after restoring DB and all files from the snapshot (using rsync ...--delete ...) the joomla admin could update the jce and other plugins
first, my friend did disable the whole vhost, but the side effect was: DNS zone was not reachable anymore. Would be nice if disabling (instead of removing) of e.g. apache2/nginx service for a vhost would be possible in Virtualmin.
I presume this is eluding to PHP ? if so you need to be careful of having 2 many restrictions as you may break some functionality that the dev has coded in.
for example a lot of the projects I code have only 1 file in the web root which takes all user input (either get or post) sanitises it, then passes it as arguments to the actual program which is residing elsewhere in the file system. The advantage of this method is the configuration files are never exposed to the internet (they are outside of the web root), all in coming data is handled by 1 file so from a security point of view there is only 1 file to maintain.
Everyone has a different perception on how to do security, I am merely pointing out that disabling certain functions can be a detriment also
I appreciate everyone’s help. Fortunately, I have been able to remove all the malicious code from my site, and I have better monitoring installed so hopefully I will not have these problems again.