Webmin process stay here forever

SYSTEM INFORMATION
OS type and version debian 8,9 and10
Webmin version 1.990
os: Debian Linux 10
root: /usr/share/webmin
theme version: 19.85.1
virtualmin version: 6.17.gpl-3
webmin version: 1.990

hi,

i have a lot of webmin and usermin process that stay here forever never closing. If i stop webmin they are still here, only killing them by hand or restarting the machine remove them.

root 1202817 0.0 0.0 32276 24140 ? S 2021 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 499175 0.0 0.0 32272 23776 ? S 2021 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2044294 0.0 0.0 32296 24012 ? S Jan16 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 3611120 0.0 0.0 32292 25020 ? S Jan19 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 497782 0.0 0.0 32292 23888 ? S Jan30 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 1231041 0.0 0.0 32292 23584 ? S Feb10 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2717859 0.0 0.0 32292 23588 ? S Feb13 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 1078005 0.0 0.0 32276 23612 ? S Feb27 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 1359297 0.0 0.0 32320 24184 ? S Mar13 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2026412 0.0 0.0 32312 23992 ? Ss Mar16 3:24 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf

they all have connected sockets that seems to never close:

perl 2044294 root 9u IPv4 161437042 0t0 TCP 192.168.37.10:webmin->104.152.52.127:48050 (ESTABLISHED)

tcp 0 0 192.168.37.10:10000 104.152.52.127:48050 ESTABLISHED
tcp 0 0 192.168.37.10:10000 104.152.52.127:41528 ESTABLISHED

its a process that is here since Jan16 and i have logouttime=120.

how can i make those people that connect and never release the socket disapear and not make process pileup ?

best regards,
Ghislain.

What is the ouput of lsof -i:10000?

Edit: some context - I have been seeing a significant increase in connections to port 10000 (on Webmin servers that are publicly accessible), nearly always from tor exit nodes. Looks like they are trying to look for exploits in ndmp…but they keep connections open on Webmin like mad and it uses up a considerable amount of memory.

We expected that to be fixed already with Webmin 1.990. If you restart Webmin manually by running /etc/webmin/restart and waiting a bit, will it pileup the connections like this over again?

@Jamie, what do you think about it? Also, is this a typo here in miniserv.pl file on line 5314?

Should that be $config{'logouttimes'} instead on the line 5314?

No, this code is correct - logouttime is for the global session expiry time, and logouttimes is for per-user or per-group settings.

1 Like

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 2994 root 3u IPv4 2322546858 0t0 TCP 213.246.37.222:webmin->104.152.52.141:58274 (ESTABLISHED)
miniserv. 3747 root 8u IPv4 1737682298 0t0 TCP 213.246.37.222:webmin->104.152.52.184:49306 (ESTABLISHED)
perl 5104 root 8u IPv4 2231919681 0t0 TCP 213.246.37.222:webmin->104.152.52.139:48575 (ESTABLISHED)
perl 6072 root 8u IPv4 1820262093 0t0 TCP 213.246.37.222:webmin->104.152.52.130:54488 (ESTABLISHED)
perl 8221 root 3u IPv4 2321625945 0t0 TCP 213.246.37.222:webmin->197.253.232.70:25213 (ESTABLISHED)
perl 8394 root 3u IPv4 2321625979 0t0 TCP 213.246.37.222:webmin->197.253.206.164:4086 (ESTABLISHED)
perl 8504 root 3u IPv4 2321629847 0t0 TCP 213.246.37.222:webmin->197.253.206.164:26219 (ESTABLISHED)
perl 23071 root 5u IPv4 2221364180 0t0 TCP *:webmin (LISTEN)
perl 30862 root 8u IPv4 2000974582 0t0 TCP 213.246.37.222:webmin->192.3.251.168:44420 (ESTABLISHED)
perl 34190 root 8u IPv4 1937351081 0t0 TCP 213.246.37.222:webmin->104.152.52.131:59584 (ESTABLISHED)

for exemple. All those process are left with a tcp connexion open in the void.
Ghislain.

well until now yes it will take some times to pile up again now that i have restarted webmin. Will tell you if i see that again.

If I kill off those connections they always come back.

What makes you think that it is not getting closed due to inactivity? Most probably the connection is still active, for example due to brute-force attack (no session) or other background activity (if logged in)?

Take a closer look to those newly created PIDs using Webmin / System ⇾ Running Processes module - does it return any Trace Process information?

Additionally, this may be browser specific. For example, Chromium based browsers are keeping the connection opened for a way longer time and generate to create at least two PIDs, while Firefox closes connection immediately and generating one PID. @Jamie, did you notice that?

its because the process are not doing anything, strace show nothing is happening

/var/usermin/miniserv.error:[22/Mar/2022:21:26:59 +0100] [45.43.54.159] Timeout : Waited for 460 seconds for start of headers
/var/usermin/miniserv.error:[22/Mar/2022:21:27:02 +0100] [45.43.54.159] Timeout : Waited for 450 seconds for start of headers

it is mostly because of brute force on the passwords by Chinese or USA ips but i dont see why a webmin/usermin process should stay open and not even close on a service webmin restart if this is the case. The process is not anymore linked to the main process as parent and it should be blocked after 3 bad try like configured and ban for 90s, even if not doing anything we setup 120s timeout. So in any case it should not be there hanging and resisting restarts and be there month after. :slight_smile:
I dont see why linux let the tcp connection open after month, i have kernel tcp keepalive setup so i dont understand this.

its usermin/webmin both.

SO_KEEPALIVE 	Enable/disable keep connections alive.

perhaps adding this to setsockopt in miniserrv.pl ?

Would it be possible to capture the traffic by one of these clients using tcpdump ?

I’m interested to know if they are sending any traffic at all, or just headers, or some actual HTTP request?

well those are bot connections so hard to do a tcpdump at the connection.
Now on the stalled process they don’t do a thing

[~]: tcpdump -vv host 104.152.52.127
tcpdump: listening on veth16f007ea, link-type EN10MB (Ethernet), capture size 262144 bytes

then i ctrl-c because nothing ever happen :slight_smile:

best regards,
Ghislain

Hmmm … Webmin has various timeouts to terminate network connections that are left open by clients like this. However, it’s possible that a client may send some traffic and then stop, which we won’t necessarily detect.

i dont know if SO_KEEPALIVE on the soket detect those.

regards,
Ghislain.

Another question - do these connections trigger any entries in /var/webmin/miniserv.log ?

[~]: ps auxwf|grep /usr/share/webmin/miniserv.pl|grep Apr03|awk ‘{ print $2 }’|xargs -I% lsof -np %|grep TCP
miniserv. 109311 root 8u IPv4 3518428853 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:44010 (ESTABLISHED)
miniserv. 109325 root 8u IPv4 3518428893 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:50952 (ESTABLISHED)
miniserv. 109354 root 8u IPv4 3518428916 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58110 (ESTABLISHED)
miniserv. 109355 root 8u IPv4 3518428919 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58114 (ESTABLISHED)
miniserv. 109356 root 8u IPv4 3518428922 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58132 (ESTABLISHED)
miniserv. 109360 root 8u IPv4 3518428934 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58134 (ESTABLISHED)
miniserv. 109363 root 8u IPv4 3518428938 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58118 (ESTABLISHED)
miniserv. 109366 root 8u IPv4 3518428949 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:60178 (ESTABLISHED)
miniserv. 109377 root 8u IPv4 3518428976 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:35756 (ESTABLISHED)
miniserv. 109381 root 8u IPv4 3518428980 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:35986 (ESTABLISHED)
miniserv. 109384 root 8u IPv4 3518429016 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:36008 (ESTABLISHED)
miniserv. 109388 root 8u IPv4 3518429019 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:37580 (ESTABLISHED)
miniserv. 109441 root 8u IPv4 3518429031 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:42140 (ESTABLISHED)
miniserv. 109442 root 8u IPv4 3518429034 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:42138 (ESTABLISHED)
miniserv. 109443 root 8u IPv4 3518429037 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:43422 (ESTABLISHED)
miniserv. 109451 root 8u IPv4 3518429049 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:48618 (ESTABLISHED)
miniserv. 109452 root 8u IPv4 3518446780 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:48626 (ESTABLISHED)
miniserv. 109454 root 8u IPv4 3518446786 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:48912 (ESTABLISHED)
miniserv. 109533 root 8u IPv4 3518428098 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:51770 (ESTABLISHED)
miniserv. 109557 root 8u IPv4 3518428105 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:49996 (ESTABLISHED)
miniserv. 109564 root 8u IPv4 3518428108 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:55204 (ESTABLISHED)
miniserv. 109566 root 8u IPv4 3518428114 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:55442 (ESTABLISHED)
miniserv. 109582 root 8u IPv4 3518449684 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:35610 (ESTABLISHED)
miniserv. 109682 root 8u IPv4 3518431957 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:50020 (ESTABLISHED)

[~]: grep 118.193.36.188 /var/webmin/webmin.log
[~]: grep 118.193.36.188 /var/webmin/miniserv.log
Binary file /var/webmin/miniserv.log matches
[~]: grep -a 118.193.36.188 /var/webmin/miniserv.log
118.193.36.188 - - [03/Apr/2022:18:15:45 +0200] “GET / HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:46 +0200] “GET /other/codepay/js/codepay_util.js HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:51 +0200] “GET / HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:53 +0200] “GET / HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:57 +0200] “GET /otc/ HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:08 +0200] “GET /reg.php HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:09 +0200] “GET /logo.png HTTP/1.1” 404 345
118.193.36.188 - - [03/Apr/2022:18:16:18 +0200] “GET /getLocale HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:28 +0200] “GET /m HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:28 +0200] “GET /api/pc/configure HTTP/1.1” 401 4892

so not in webmin but some in miniserv.pl.

best regards,
Ghislain.

Interesting, looks like the client never sent a request.

How long do these connections hang around for?

see here, this has been around for ages

the processes stay for month, i clear them regulary when i login but i can find some that have 6month at least.

i can find process like this here since 2019 !!
this is on several machine, webmin and usermin

root 1204 0.0 0.2 137664 73260 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 13024 0.0 0.2 137668 73548 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 13092 0.0 0.0 12280 6436 ? S 2019 0:00 _ /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 14535 0.0 0.2 137664 73260 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 17397 0.0 0.2 137664 73132 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 20905 0.0 0.2 137668 73304 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 23339 0.0 0.0 12280 7524 ? S 2019 0:00 _ /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 23489 0.0 0.0 12280 6436 ? S 2019 0:00 _ /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 3281 0.0 0.0 12280 8056 ? Ss 2019 14:10 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 11212 0.0 0.4 140692 72532 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 11214 0.0 1.1 97588 73568 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 11496 0.0 0.2 138216 73796 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 13423 0.0 0.1 80804 20648 ? S 2020 0:00 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 134885 0.0 0.2 138236 71072 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf

regards,
Ghislain.

Wait, 2019? Has this machine not been rebooted for 3 years? :slight_smile: