How do I investigate what closed the webmin service?

SYSTEM INFORMATION
OS type and version CentOS 7.9.2009
Webmin version 1.994
Virtualmin version 7.1
Related packages systemd

A few days ago I noticed the Webmin service is down on a CentOS 6 server (no Virtualmin)… An investigation didn’t bring up any suspects, so I had to let it go. (actually now I remembered that the systemd service was totally missing, and I couldn’t find out the how nor the why. A yum reinstall webmin fixed it)

Today on a CentOS 7 running Virtualmin and a handful of sites, a monitor notified me that the Webmin service went down.

[root@amadeus ~]# service webmin status
Redirecting to /bin/systemctl status webmin.service
● webmin.service - Webmin server daemon
   Loaded: loaded (/usr/lib/systemd/system/webmin.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2022-06-09 04:16:38 UTC; 10min ago
  Process: 2370 ExecStop=/etc/webmin/.stop-init (code=exited, status=0/SUCCESS)
  Process: 729 ExecStart=/etc/webmin/.start-init (code=exited, status=0/SUCCESS)
 Main PID: 2292 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/webmin.service
           └─9181 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf

This time I investigate more, but there’s nothing out of the ordinary in any log under /var/log.

Here’s a part of the messages log, showing the service stopping at Jun 9 04:16:38:

Jun  9 04:16:36 amadeus su: (to domainX.xyz-admin) root on none
Jun  9 04:16:36 amadeus systemd: Created slice User Slice of domainX.xyz-admin.
Jun  9 04:16:36 amadeus systemd: Started Session c36622 of user domainX.xyz-admin.
Jun  9 04:16:36 amadeus systemd: Removed slice User Slice of domainX.xyz-admin.
Jun  9 04:16:37 amadeus systemd: Reloading The Apache HTTP Server.
Jun  9 04:16:37 amadeus systemd: Reloaded The Apache HTTP Server.
Jun  9 04:16:38 amadeus named[1160]: validating www.domainY.xyz/AAAA: no valid signature found
Jun  9 04:16:38 amadeus named[1160]: validating www.domainY.xyz/A: no valid signature found
Jun  9 04:16:38 amadeus .stop-init: Stopping Webmin server in /usr/libexec/webmin
Jun  9 04:17:18 amadeus saslauthd[745]: do_auth         : auth failure: [user=tttt@domainZ.xyz] [service=smtp] [realm=domainZ.xyz] [mech=pam] [reason=PAM auth error]
Jun  9 04:17:25 amadeus saslauthd[749]: do_auth         : auth failure: [user=comet@domainZ.xyz] [service=smtp] [realm=domainZ.xyz] [mech=pam] [reason=PAM auth error]
Jun  9 04:17:40 amadeus saslauthd[748]: do_auth         : auth failure: [user=comet@domainZ.xyz] [service=smtp] [realm=domainZ.xyz] [mech=pam] [reason=PAM auth error]
Jun  9 04:17:45 amadeus saslauthd[747]: do_auth         : auth failure: [user=deer@domainZ.xyz] [service=smtp] [realm=domainZ.xyz] [mech=pam] [reason=PAM auth error]
Jun  9 04:18:01 amadeus saslauthd[746]: do_auth         : auth failure: [user=deer@domainZ.xyz] [service=smtp] [realm=domainZ.xyz] [mech=pam] [reason=PAM auth error]
Jun  9 04:18:01 amadeus systemd: Started Session 6509 of user root.
Jun  9 04:18:02 amadeus named[1160]: network unreachable resolving 'yandex.com/MX/IN': 2a02:6b8:0:1::1#53
Jun  9 04:18:02 amadeus named[1160]: network unreachable resolving 'yandex.com/MX/IN': 2a02:6b8::1#53
Jun  9 04:18:02 amadeus named[1160]: network unreachable resolving 'ns9.z5h64q92x9.net/AAAA/IN': 2001:2030:20::78#53
Jun  9 04:18:02 amadeus named[1160]: network unreachable resolving 'ns9.z5h64q92x9.net/A/IN': 2001:2030:20::78#53
Jun  9 04:18:02 amadeus named[1160]: network unreachable resolving 'ns9.z5h64q92x9.net/AAAA/IN': 2a02:6b8::1#53

There was about 800 MB of free RAM at the time I checked, and there’s a little doubt that it could have been an out of memory issue, but I could not find any OOM event in the logs. There’s a low RAM monitor and it wasn’t triggered. All other services seem to be reporting casually. Nothing suspicious.

Do you have a suggestion or some tip to look somewhere for the reason this might have happened?

Thank you

Right after posting this I noticed there’s a runaway process that’s eating RAM up to 99% in quick bursts.
It seems indeed an OOM condition, but it’s weird that nothing in the logs confirms this.

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.