Usermin randomly stops working across all Ubuntu 20.04 servers

Ubuntu 20.04

On all my Ubuntu 20.04 server Usermin stops working from time to time. It typically stops working the moment a user logs in. Fixing the problem when it occurs, is fairly trivial. Just restart Usermin. Knowing when it stops - what’s another problem but by port monitoring using PRTG and service monitoring using Spatie’s Server monitor we’re able to see it’s happing.

I’ve read this post which is unfortunately already closed:

In the case of Usermin, you’ll look in /var/usermin/miniserv.error and /var/usermin/miniserv.log

The fact is this is happening across all servers. I’ll give it my best shot to see if I can see anything in the logs but even on small light load new servers this is happening.

For now, maybe anyone got tips for a watchdog?

Can’t help but you did forgot to post versions?

If there are updates here changelogs
Webmin

Do changelog search then hmm - General Discussion - Virtualmin Community

Sometime problems with browser and security settings / plugins or tls versions are some we had in the past.
So for support browser and such also needed. Cache, sessions and and…

Thanks @jotst, valid point, more information, versions etc.

Across all my servers I have automatic updates enabled - both for security updates and application updates. On the client side I have Chrome for Linux latest version, but my brother uses Chrome on Mac and clients use random browsers and everyone has the same problem.

I am 100% sure I’m running “the latest” of everything. Hopefully that means I don’t have to scout change log?

Anyhow, to have a “watchdog” I’ve now googled monit but next problem is finding the PID file for Usermin. Any tips where to find the PID file for Usermin?

Check?

Is the version usermin Version 1.833 (4th December 2021) ?

Webmin Version 1.983 (4th December 2021) ?
virtualmin Version 6.17 (9th September 2021) ?

Also on rebooted servers?
Backup before.

Thanks so much. I think you are onto something:

Usermin: 1.823 [outdated]
Webmin: 1.981 [outdated]
Virtualmin: 6.17-3 [up to date]

Here I am having believed for more than a year even longer choosing the box below would also automatically update things like Webmin and Usermin. I can certainly see Virtualmin being updated and my servers are stock “Virtualmin” installs.

image

Lol and how proud was I thinking my servers are perfectly up to date.

Ok so next question, I guess I have to figure out why Webmin and Usermin hasn’t updated automatically? I have 6 servers which I now have to audit / change.

Thanks so much for your feedback thus far, I really appreciate it!

Any tips how to keep Usermin and Webmin up to date? I found these instructions for Debian 11 but they don’t work for Ubuntu even after substituting sarge for focal

echo "deb https://download.webmin.com/download/repository sarge contrib" | tee /etc/apt/sources.list.d/webmin.list
wget -qO - http://www.webmin.com/jcameron-key.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/jcameron-key.gpg

Please ask her Virtualmin Support while you can set repo’s active under same part as the updates are, but don’t know it is wise todo.

While some are in virtualmin repo i am not sure and then you can break , somewhere in forum i remember @joe writing about that.

Also i don’t know if update are solution :wink:

Thanks again for replying. I appreciate the replies.

don’t know it is wise todo
then you can break
i don’t know if update are solution

Right so from what I gather I’m moving into risky territory here:

  • Doesn’t look like automatic updates are supported for Ubuntu / Focal
  • Even if it was, maybe something is going to break
  • Interaction between Virtualmin should be kept in mind

Well let’s put it this way, it not worth risking the stable services for 100s of clients just to get automatic updates that might not fix this problem.

So I’m back to how to automatically start Usermin

I’ll start a new post: “How to restart Usermin with Monit if it stops” and I’ll use the Virtualmin forum so that I can reduce risk.

a link about break.

but don’t know about that enough to help also no knowledge here about Ubuntu ( should be TLS version is only thing i know)
Webmin update - is safety? - General Discussion - Virtualmin Community

Still you can search yourself in changelogs or github issue’s or one of the updates has changes for this. :wink:

Webmin / Usermin Bugs (sourceforge.net)

Issues · webmin/usermin · GitHub

I have tried lots of monitoring systems and have finally settled for the free remote monitoring system offered by Rackspace. For their minimum billing of usd 5 per month, Rackspace offers free uptime monitoring with realtime notification via email and SMS when my server is down. They also offer 50GB of free storage which works with Virtualmin backup.and they offer free DNS with a TTL which can be set to as low as 300. There are one or two additional freebies that come with the USD 5 monthly minimum billing - 50,000 bulk mail messages per month via MailGun and some cloudy services which I do not use and so cannot recall offhand the details.

But if you want real-time monitoring and notifications about your servers, I would recommend the USD 5 minimum monthly subscription of Rackspace Cloud.

@vander.host,

This is invalid information. You should NOT be updating Webmin from the Webmin repo when using Virtualmin. The versions you indicated as being outdated are also NOT outdated in terms of the versions made available to Virtualmin repo.

Virtualmin team does extra testing to ensure Webmin/Usermin work properly with the current version of Virtualmin.

1 Like

You are up to date with what is in the Virtualmin repos (intentionally). Where are you seeing “outdated”?

Do not add other repos, please, including the Webmin repos, for Virtualmin updates, unless explicitly told to do so by one of us (Staff).

That’s not true.

Services stopping “randomly” is almost always the OOM killer kicking in. Usermin doesn’t crash…so, something is stopping it.

How much free memory do you have?

That is because of me and my post , but i was on wrong leg because topic about usermin.

Then after it seems a virtualmin install did write not to have separate updates for usermin / webmin becuase it can break things.

So is misunderstanding sorry for that one. :wink:

But i also extra mentioned if bug / known problem he can find it probably somewhere in the updates with changelogs reading or issue’s.

Thanks guys for the incredible amount of tips and advice. Hopefully I can post enough information here to solve this problem.

What’s interesting this morning it happened again, and when it happens I notice that the service appears to be running:

# service usermin status
● usermin.service - LSB: web-based account administration interface for Unix systems
     Loaded: loaded (/etc/init.d/usermin; generated)
     Active: active (exited) since Thu 2021-11-18 08:56:46 UTC; 1 months 10 days ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 0 (limit: 2278)
     Memory: 0B
     CGroup: /system.slice/usermin.service

Warning: journal has been rotated since unit was started, output may be incomplete.

However the port closed the moment I tried accessing it:

root@buspage:~# telnet localhost 20000
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Then when I restart everything is fine again:

service usermin stop
service usermin start
telnet localhost 20000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]

Services stopping “randomly” is almost always the OOM killer kicking in

I don’t think it’s memory for but reference I paste the memory consumption here. Also I have many different servers and all of them have quite a bit of RAM, this one has the lowest though:

Real memory 792.92 MiB used / 980.54 MiB cached / 1.94 GiB total
Virtual memory 1.15 GiB used / 3.99 GiB total

Also I have to point out the service doesn’t stop “randomly”. It stops the moment any user tries to access port 20000. Then it works for a long time, but then stops again when a user tries to access it.

@calport thanks for the monitoring advice. I use PRTG extensively and Spatie’s network monitoring tools which has a Linux service checker. The problem of course as just mentioned is the service appears to just keep on running, but checking port 20000 always fails and then PRTG sends a push notification.

@vander.host,

Are you running a full stack on that system?

  • Web Server
  • Mail Server
  • DNS Server
  • ClamAV (antivirus)
  • SpamAssassin (antispam)
  • MySQL
  • etc

Yes it’s a stock / default Virtualmin install without any customizations whatsoever. All my installations are the same, stock installs without customizations. Around 6 of them, Ubuntu 20.04. All of them has the same issue. If I can figure out a watchdog or monit test that checks for port 20000 existence I can implement a temporary workaround.

@vander.host,

I have found that 2GB RAM is pushing things a bit, if you are running a full stack. Email software generally eats up a lot of resources on their own.

Most of our systems have a minimum of 4GB and even then I rarely put a full stack on the same machine in order to optimize and maximize resources (though I don’t expect everyone to follow suite on this front as it does require some extra work to maintain).

Mostly ClamAV. Everything else is quite small. SpamAssassin is the next biggest part of the mail stack, but it’s minuscule compared to clamd (which is over 1GB, by itself, and continues to grow faster every year). If ever you suspect memory is an issue, disable virus scanning and shut down clamd, and see how things look after. I’ve been tempted to stop installing ClamAV by default, as it’s just unrealistic to run it on most of the servers people are running Virtualmin on, and it is not intuitive that such a tiny part of the work the system does requires so much of the resources (even though the setup wizard is very clear about ClamAV being very large, I don’t think people always read it).

2 Likes