Webmin Loses Connection, Quits Listening on 10000

Hi there,

Recently, without any significant changes that I’m aware of, I found that my webmin instance was not accessible. It simply said “connection refused” in chrome on the default port 10000. After checking the systemctl status, it appeared that the webmin service was still running (and in fact, all of my websites were still accessible), but using “netstat -tulpn” showed that nothing was listening on port 10000. Usermin on port 20000 was working just fine.

I ended up restarting the service, which worked, but less than 30 minutes later while I was doing normal things, I got a popup in webmin that said “lost connection.” Upon refresh, the same thing was happening; nothing was listening on port 10000, usermin works fine, websites work fine, etc.

It sounds to me like miniserv is crashing, however the miniserv.error logs show absolutely nothing around the time of the crash. You might also consider memory usage, but my server has 16GB and I rarely use more than 60%. dmesg shows nothing about killing any processes.

Where should I be looking to diagnose this problem?

Thanks

Howdy,

A common cause of that is resource issues… if your server is running out of RAM, it could start killing off processes such as Webmin in order to keep the server online.

What is the output of these commands:

free -m uptime dmesg | tail -30

Hey there, thanks for the response.

It just crashed again and it happened right after I went into the postfix config and clicked on the Relocated Mapping button. This error appeared on the page and then nothing was accessible anymore:

“Failed to query Postfix config command to get the current value of parameter relocated_maps:”

I have run the commands you mentioned, here are the outputs:

free -m
          total        used        free      shared  buff/cache   available

Mem: 16051 3224 12295 343 531 12223
Swap: 0 0 0

uptime

22:17:04 up 2:55, 2 users, load average: 0.25, 0.34, 0.35

dmesg | tail -30

[ 9666.701185] audit: type=1400 audit(1566705780.822:320): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=23837 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[ 9666.762450] audit: type=1400 audit(1566705780.882:321): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm=“imap” requested_mask=“ra” denied_mask=“ra” fsuid=1001 ouid=1001
[ 9666.763760] audit: type=1400 audit(1566705780.882:322): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=23837 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[ 9666.765036] audit: type=1400 audit(1566705780.886:323): apparmor=“ALLOWED” operation=“mknod” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm=“imap” requested_mask=“c” denied_mask=“c” fsuid=1001 ouid=1001
[ 9666.767414] audit: type=1400 audit(1566705780.886:324): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm=“imap” requested_mask=“rwc” denied_mask=“rwc” fsuid=1001 ouid=1001
[ 9666.767482] audit: type=1400 audit(1566705780.886:325): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=23837 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[ 9666.768074] audit: type=1400 audit(1566705780.886:326): apparmor=“ALLOWED” operation=“file_lock” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm=“imap” requested_mask=“wk” denied_mask=“wk” fsuid=1001 ouid=1001
[ 9666.768179] audit: type=1400 audit(1566705780.886:327): apparmor=“ALLOWED” operation=“file_lock” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm=“imap” requested_mask=“k” denied_mask=“k” fsuid=1001 ouid=1001
[ 9666.768203] audit: type=1400 audit(1566705780.886:328): apparmor=“ALLOWED” operation=“unlink” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm=“imap” requested_mask=“d” denied_mask=“d” fsuid=1001 ouid=1001
[10085.172472] audit: type=1400 audit(1566706199.295:329): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index.log" pid=26086 comm=“imap” requested_mask=“ra” denied_mask=“ra” fsuid=1001 ouid=1001
[10085.172831] audit: type=1400 audit(1566706199.295:330): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=26086 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[10085.418278] audit: type=1400 audit(1566706199.539:331): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm=“imap” requested_mask=“ra” denied_mask=“ra” fsuid=1001 ouid=1001
[10085.418613] audit: type=1400 audit(1566706199.539:332): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=26086 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[10085.419518] audit: type=1400 audit(1566706199.539:333): apparmor=“ALLOWED” operation=“mknod” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm=“imap” requested_mask=“c” denied_mask=“c” fsuid=1001 ouid=1001
[10085.421396] audit: type=1400 audit(1566706199.543:334): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm=“imap” requested_mask=“rwc” denied_mask=“rwc” fsuid=1001 ouid=1001
[10085.421479] audit: type=1400 audit(1566706199.543:335): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=26086 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[10085.422260] audit: type=1400 audit(1566706199.543:336): apparmor=“ALLOWED” operation=“file_lock” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm=“imap” requested_mask=“wk” denied_mask=“wk” fsuid=1001 ouid=1001
[10085.422322] audit: type=1400 audit(1566706199.543:337): apparmor=“ALLOWED” operation=“file_lock” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm=“imap” requested_mask=“k” denied_mask=“k” fsuid=1001 ouid=1001
[10085.422338] audit: type=1400 audit(1566706199.543:338): apparmor=“ALLOWED” operation=“unlink” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm=“imap” requested_mask=“d” denied_mask=“d” fsuid=1001 ouid=1001
[10386.739392] audit_printk_skb: 3 callbacks suppressed
[10386.739394] audit: type=1400 audit(1566706500.859:340): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index.log" pid=27887 comm=“imap” requested_mask=“ra” denied_mask=“ra” fsuid=1001 ouid=1001
[10386.739801] audit: type=1400 audit(1566706500.863:341): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=27887 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[10386.802458] audit: type=1400 audit(1566706500.923:342): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm=“imap” requested_mask=“ra” denied_mask=“ra” fsuid=1001 ouid=1001
[10386.803610] audit: type=1400 audit(1566706500.923:343): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=27887 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[10386.806576] audit: type=1400 audit(1566706500.927:344): apparmor=“ALLOWED” operation=“mknod” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm=“imap” requested_mask=“c” denied_mask=“c” fsuid=1001 ouid=1001
[10386.808546] audit: type=1400 audit(1566706500.931:345): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm=“imap” requested_mask=“rwc” denied_mask=“rwc” fsuid=1001 ouid=1001
[10386.808619] audit: type=1400 audit(1566706500.931:346): apparmor=“ALLOWED” operation=“open” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=27887 comm=“imap” requested_mask=“rw” denied_mask=“rw” fsuid=1001 ouid=1001
[10386.809092] audit: type=1400 audit(1566706500.931:347): apparmor=“ALLOWED” operation=“file_lock” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm=“imap” requested_mask=“wk” denied_mask=“wk” fsuid=1001 ouid=1001
[10386.809190] audit: type=1400 audit(1566706500.931:348): apparmor=“ALLOWED” operation=“file_lock” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm=“imap” requested_mask=“k” denied_mask=“k” fsuid=1001 ouid=1001
[10386.809218] audit: type=1400 audit(1566706500.931:349): apparmor=“ALLOWED” operation=“unlink” profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm=“imap” requested_mask=“d” denied_mask=“d” fsuid=1001 ouid=1001

Not sure what these audit things are, but there’s a whole lot of them in dmesg, even as far back as -100. Thanks!

MItch, did you get it figured out?

I am having the exact same issue. It’s really annoying.
I don’t want to keep having to restart the instance. And i really hope it’s not resources, as i have double the ram as my last vps.

No luck. It seems like it stays up until I start using it, then within 30 mins it will die randomly. It “feels” like a memory usage problem, but there are no traces of it being killed, I have 16GB of RAM, and this just randomly started happening without any changes to the server after it working for a year…so I have no idea

@Falcons34, you may wan to try what I mentioned in Comment #1 above to see if that shows anything.

@MutchTamadge, is this a dedicated server, or a VPS? If a VPS, what kind of VPS is it? I don’t see any obvious resource issues in the above, but you’re right that it does sound like a resource issue of some kind. I’m curious if it could be related to a VPS with certain types of resource restrictions, such as OpenVZ.

-Eric

Hi Eric, thanks for this

Yes, I ran those and it was showing out of memory errors and killing off clam processes.
So I’m assuming clam is the problem. I knew it was a memory hog but was hoping 4G of RAM would be enough.
Anyhow, is there any recommendation for a program to replace clam?

Maybe try boosting your php memory limit up some.

I’m having the same issue after doing to latest updates.

I solved my issues this: https://www.virtualmin.com/comment/818100#comment-818100

2 things I disabled afterward that really saved a lot of ram were dovecot and postfix.

I run dovecot and postfix on a 4GB RAM system and don’t have any problems.

Normally I don’t use Clamav in the foreground…however,

If I start clamav

-my RAM usage increases by about 20% (normally im running at about 22%, now 42%)…stats tell me /usr/sbin/clamd --foreground=true is using 1.01GB.

  • freshclam -d is using a further 151MB

-CPU does jump for a while then drops back to normal.

@adamjedgar
Nice site, what theme are you using?

Collin

WP 5.2.3 theme maxhost and revslider version… apache 2.4.25 … owl carousel, tweenmax , bootstrap 3.3.6 and so more you can read out of headers and source if you are meaning these things. also some mixed content , on mail ports from “postfix” i guess The expiration date of this certificate is 2019-08-04 09:28:17 UTC.

And Collin you are using WP 5.2.3 maybe raidiant theme parts, apache 2.4.29 Ubuntu , woocommerce 3.7.1 and so more Trigger The server does not support any cipher suites that provide forward secrecy. on thgose 2 dashboard.innovativeflare.com 54.70.248.63 scan.innovativeflare.com 34.209.57.169

on maindomain box : The server uses a 1024-bit Diffie-Hellman group., The server uses a 1024-bit RSA key. ,

and on all still also TLS 1.0