Today I found a process called .X0-unix running in my server (CentOS 6 running Webmin/Virtualmin). This process was running from:
[root@server ~]# find / -name .X0-unix
/usr/libexec/webmin/.X0-unix
I cannot find anything in the webmin logs to explain how that file got there and there have not been any ssh/ftp transfers into my server. How the hell did they get that file into the Webmin service? As far as I can tell it is an irc bot called tsunami so no major damage done at the moment but I am very worried that webmin is compromised. The logs do not show any logins for any users (except when I logged in to check of course). A new flaw/backdoor into webmin?
How can I try to find out where the exploit is? I cannot shut down virtualmin completely as my users depend on it to administer their domains.
After some more digging I have also found more files that do not belong to webmin in /usr/libexec/webmin:
40 -rw-r–r-- 1 root root 38243 May 23 00:38 PK.ico
40 -rw-r–r-- 1 root root 38243 May 23 00:40 PK.ico.2
40 -rw-r–r-- 1 root root 38243 May 23 00:40 PK.ico.3
They are all irc scripts. Still have no idea how they got there. Still looking at all possibilities but I find it very suspicious that they are all in the webmin directory.
Update to Webmin 1.800. There’s a security bug in Authentic Theme as shipped with Webmin devel versions 1.794 and 1.794 (which we rolled out to Virtualmin users to address Ubuntu fixes and Let’s Encrypt enhancements).
It’s been fixed in 1.800. We learned of the bug about two hours ago. We’ll talk about specifics of the problem in a few days after folks have had a chance to update. You can open a private ticket in the issue tracker to discuss it in more detail, if you’d like us to assist with assessing the scope of the problem on your system.
are there any reasons why one of my centos server got updated to 1.795 a few months ago and now can’t see the 1.800 update. the other servers can see the update 1.800
all servers uses epel and official repo… just wondering.
I have seen the same thing on a few servers 3 days ago.
They seem to be using a shell exploit in the [edit: we’ll discuss the exact security issue later :-)], at first I though it was a shellshock hack (but of course those have been patched for years). It looked like it was escaping out some sort of login alert mail out routine.
If you reboot the server the bad processes will go away, and you can delete the junk files left behind in the webmin directory.
I have not found any permanent changes to the system files, so once the theme is updated the issue should be gone for good, however this exploit could have been very destructive since it runs as root.
I am very disappointed of Webmin team. Not because of getting our server hacked, but because it could be avoided easily. We got hacked almost 48h after the 1.800 patch went public, however the release did not get to the official Webmin mailing list, so I was unaware of the patch.
Another bad luck is that Centos 6.8 upgrade come in the same time, so spotting webmin between 135 other packages manually is not simple task.
On webmin website, all the forums and mailing list go through source forge. Are they update? Why are there forums there and here… Please centralize the information.
Please advise me how can I immediately know that there is a webmin/virtualmin update have their changelog.
Also, you could configure Virtualmin to send notifications about available updates. Now, it can be difficult to understand the severity of an issue from that, but it would be an opportunity to know that updates are waiting and available.
Another thing you could do is subscribe to the News Forum here, which would provide you with email updates for Webmin and Virtualmin related news:
These are the files that have been uploaded/compiled to webmin directory
.X0-unix
pass.h
pass.h.1
pass.h.2
pass.h.3
pass.h.4
pass.h.5
pass.h.6
PK.ico
PK.ico.1
PK.ico.10
PK.ico.11
PK.ico.12
PK.ico.13
PK.ico.14
PK.ico.15
PK.ico.16
PK.ico.17
PK.ico.2
PK.ico.3
PK.ico.4
PK.ico.5
PK.ico.6
PK.ico.7
PK.ico.8
PK.ico.9
These are the one in /tmp
kait.c
mg
pmabot.c
PwNzbot.c
mg seems to be the first file of the hack that downloads, compiles and runs the rest of the code.
The hack seems to be at least 3 different IRC controlled DDOS bots: Linux/Tsunami, Kaitan and PHP/PBot. The control server is irc.ugotownedz.org and he files were downloaded from ugotownedz.org.
It seems have attempted to do local password brute force.
Some tips for others since none of my machines where affected.
I was not using Authentic theme. (never liked the slowness, virtmin mobile is way better on mobile)
2, Changed port to something other than 10000. (any internet facing service should never be on the default port)
3, Only accept ssh from VPN ip or a specific IP, or at least don’t leave ssh on port 22.
Ossec bans for failed password attempts, for ssh, mail, http, etc. (install some type of active response monitor)
5 Ossec monitors /etc and other folders for changes.
2 and 3 will prevent most attacks.
4 and 5 will prevent the others if they get that far, or at least notify you so you can act.
This goes for any web hosting platform, not just webmin.
deploy fail2ban at least for ssh and setup ban for day or month as a minimum… or close port 22 to outside world
disable ssh logon with passwords and usernames globally and enable ssh keys logon only
close port 10000 to accessible from outside - connect to it via vpn or within your network - if that is not possible, you can setup ip access for webmin login page and change default port for webmin (that is not that helpful but it does slow attacks down)
limit how many times user can fail to logon to webmin and then ban it for day or so (the ip address or user or both)
check for any unknown user accounts on your system
change passwords for everyone - as well for root account too
check access logs - how they did came in
check ssh config (if any) for any amomalies
check start up scripts and remove anything suspicious
check cron jobs for any user - including for root and remove anything suspicious
check any unknown ssh keys in your authorized_keys file(s) - remove anything that its not yours perhaps best options would be to generate new file with just your keys…
disable ftp - if you run any, just use sftp which is ssh - if you must run it make sure you do deploy fail2ban (at least or something similar), fail 3 times logon = ban for day, month or year…
deploy notification script for ssh logins !! - on all srvs I’ve ever touched, I’ve deployed my own script which will send me google hangout message (aim - anything like icq, google hangouts, irc or xmpp) with ip address, username with time of logon at the very moment that user (any user using ssh) is logged in. It also sends those details to my client (owner of the srv) to email as well so we both are informed asap. This will allow you to see things in real time like who just connected from where…really great idea is this and I would greatly suggest to you to use this as well. Writing script like it is very easy! So if anything is happening on srv, be it attack or regular usage - you will know in advance as by this way ssh will not connect anyone before the quiet (discreet) notification is not send out to you (be it aim or email msg or both)…
…that should be enough to stop new hacks and you can continue to investigate from there… I strongly believe that you will be just fine with those points mentioned above perhaps its overkill, but from personal experience - this is what always worked at lest for me past 7 years…
This apply to virtualmin too, however if you will detect new hack it would be the time to revise your website scripts in a bit different manner… nothing hard to accomplish…
Move your SSH port to a new port location. Port scanners usually scans for port numbers below 1000, setting it up a little high will do but make sure you that port was unblocked on your firewall. Make sure that you can login on SSH on that port.
Yes you are correct, if logon by SSH keys are enabled and login by account is disabled, moving the port is pointless since all connections will be dropped.
But just moving from default port will cut down more than 90% of all bruteforce bots so i think is worth to change default port to something else (1024 < X).