gadnet
March 30, 2022, 12:37pm
1
SYSTEM INFORMATION
OS type and version
debian 8,9 and10
Webmin version
1.990
os: Debian Linux 10
root: /usr/share/webmin
theme version: 19.85.1
virtualmin version: 6.17.gpl-3
webmin version: 1.990
hi,
i have a lot of webmin and usermin process that stay here forever never closing. If i stop webmin they are still here, only killing them by hand or restarting the machine remove them.
root 1202817 0.0 0.0 32276 24140 ? S 2021 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 499175 0.0 0.0 32272 23776 ? S 2021 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2044294 0.0 0.0 32296 24012 ? S Jan16 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 3611120 0.0 0.0 32292 25020 ? S Jan19 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 497782 0.0 0.0 32292 23888 ? S Jan30 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 1231041 0.0 0.0 32292 23584 ? S Feb10 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2717859 0.0 0.0 32292 23588 ? S Feb13 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 1078005 0.0 0.0 32276 23612 ? S Feb27 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 1359297 0.0 0.0 32320 24184 ? S Mar13 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2026412 0.0 0.0 32312 23992 ? Ss Mar16 3:24 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
they all have connected sockets that seems to never close:
perl 2044294 root 9u IPv4 161437042 0t0 TCP 192.168.37.10:webmin->104.152.52.127:48050 (ESTABLISHED)
tcp 0 0 192.168.37.10:10000 104.152.52.127:48050 ESTABLISHED
tcp 0 0 192.168.37.10:10000 104.152.52.127:41528 ESTABLISHED
its a process that is here since Jan16 and i have logouttime=120.
how can i make those people that connect and never release the socket disapear and not make process pileup ?
best regards,
Ghislain.
What is the ouput of lsof -i:10000
?
Edit: some context - I have been seeing a significant increase in connections to port 10000 (on Webmin servers that are publicly accessible), nearly always from tor exit nodes. Looks like they are trying to look for exploits in ndmp…but they keep connections open on Webmin like mad and it uses up a considerable amount of memory.
Ilia
March 30, 2022, 11:24pm
3
We expected that to be fixed already with Webmin 1.990. If you restart Webmin manually by running /etc/webmin/restart
and waiting a bit, will it pileup the connections like this over again?
@Jamie , what do you think about it? Also, is this a typo here in miniserv.pl
file on line 5314 ?
Should that be $config{'logouttimes'}
instead on the line 5314 ?
Jamie
March 31, 2022, 2:37am
4
No, this code is correct - logouttime
is for the global session expiry time, and logouttimes
is for per-user or per-group settings.
1 Like
gadnet
March 31, 2022, 7:19am
5
pixel_paul:
lsof -i:10000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 2994 root 3u IPv4 2322546858 0t0 TCP 213.246.37.222:webmin->104.152.52.141:58274 (ESTABLISHED)
miniserv. 3747 root 8u IPv4 1737682298 0t0 TCP 213.246.37.222:webmin->104.152.52.184:49306 (ESTABLISHED)
perl 5104 root 8u IPv4 2231919681 0t0 TCP 213.246.37.222:webmin->104.152.52.139:48575 (ESTABLISHED)
perl 6072 root 8u IPv4 1820262093 0t0 TCP 213.246.37.222:webmin->104.152.52.130:54488 (ESTABLISHED)
perl 8221 root 3u IPv4 2321625945 0t0 TCP 213.246.37.222:webmin->197.253.232.70:25213 (ESTABLISHED)
perl 8394 root 3u IPv4 2321625979 0t0 TCP 213.246.37.222:webmin->197.253.206.164:4086 (ESTABLISHED)
perl 8504 root 3u IPv4 2321629847 0t0 TCP 213.246.37.222:webmin->197.253.206.164:26219 (ESTABLISHED)
perl 23071 root 5u IPv4 2221364180 0t0 TCP *:webmin (LISTEN)
perl 30862 root 8u IPv4 2000974582 0t0 TCP 213.246.37.222:webmin->192.3.251.168:44420 (ESTABLISHED)
perl 34190 root 8u IPv4 1937351081 0t0 TCP 213.246.37.222:webmin->104.152.52.131:59584 (ESTABLISHED)
for exemple. All those process are left with a tcp connexion open in the void.
Ghislain.
gadnet
March 31, 2022, 7:20am
6
well until now yes it will take some times to pile up again now that i have restarted webmin. Will tell you if i see that again.
If I kill off those connections they always come back.
Ilia
March 31, 2022, 10:34am
8
What makes you think that it is not getting closed due to inactivity? Most probably the connection is still active, for example due to brute-force attack (no session) or other background activity (if logged in)?
Take a closer look to those newly created PIDs using Webmin / System ⇾ Running Processes module - does it return any Trace Process information?
Additionally, this may be browser specific. For example, Chromium based browsers are keeping the connection opened for a way longer time and generate to create at least two PIDs, while Firefox closes connection immediately and generating one PID. @Jamie , did you notice that?
gadnet
March 31, 2022, 12:41pm
9
its because the process are not doing anything, strace show nothing is happening
/var/usermin/miniserv.error:[22/Mar/2022:21:26:59 +0100] [45.43.54.159] Timeout : Waited for 460 seconds for start of headers
/var/usermin/miniserv.error:[22/Mar/2022:21:27:02 +0100] [45.43.54.159] Timeout : Waited for 450 seconds for start of headers
it is mostly because of brute force on the passwords by Chinese or USA ips but i dont see why a webmin/usermin process should stay open and not even close on a service webmin restart if this is the case. The process is not anymore linked to the main process as parent and it should be blocked after 3 bad try like configured and ban for 90s, even if not doing anything we setup 120s timeout. So in any case it should not be there hanging and resisting restarts and be there month after.
I dont see why linux let the tcp connection open after month, i have kernel tcp keepalive setup so i dont understand this.
its usermin/webmin both.
gadnet
March 31, 2022, 1:04pm
10
SO_KEEPALIVE Enable/disable keep connections alive.
perhaps adding this to setsockopt in miniserrv.pl ?
Jamie
April 1, 2022, 5:22am
11
Would it be possible to capture the traffic by one of these clients using tcpdump
?
I’m interested to know if they are sending any traffic at all, or just headers, or some actual HTTP request?
gadnet
April 1, 2022, 1:30pm
12
well those are bot connections so hard to do a tcpdump at the connection.
Now on the stalled process they don’t do a thing
[~]: tcpdump -vv host 104.152.52.127
tcpdump: listening on veth16f007ea, link-type EN10MB (Ethernet), capture size 262144 bytes
then i ctrl-c because nothing ever happen
best regards,
Ghislain
Jamie
April 2, 2022, 4:41am
13
Hmmm … Webmin has various timeouts to terminate network connections that are left open by clients like this. However, it’s possible that a client may send some traffic and then stop, which we won’t necessarily detect.
gadnet
April 4, 2022, 12:54pm
14
i dont know if SO_KEEPALIVE on the soket detect those.
regards,
Ghislain.
Jamie
April 5, 2022, 4:00am
15
Another question - do these connections trigger any entries in /var/webmin/miniserv.log
?
gadnet
April 5, 2022, 7:59am
16
[~]: ps auxwf|grep /usr/share/webmin/miniserv.pl|grep Apr03|awk ‘{ print $2 }’|xargs -I% lsof -np %|grep TCP
miniserv. 109311 root 8u IPv4 3518428853 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:44010 (ESTABLISHED)
miniserv. 109325 root 8u IPv4 3518428893 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:50952 (ESTABLISHED)
miniserv. 109354 root 8u IPv4 3518428916 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58110 (ESTABLISHED)
miniserv. 109355 root 8u IPv4 3518428919 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58114 (ESTABLISHED)
miniserv. 109356 root 8u IPv4 3518428922 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58132 (ESTABLISHED)
miniserv. 109360 root 8u IPv4 3518428934 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58134 (ESTABLISHED)
miniserv. 109363 root 8u IPv4 3518428938 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:58118 (ESTABLISHED)
miniserv. 109366 root 8u IPv4 3518428949 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:60178 (ESTABLISHED)
miniserv. 109377 root 8u IPv4 3518428976 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:35756 (ESTABLISHED)
miniserv. 109381 root 8u IPv4 3518428980 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:35986 (ESTABLISHED)
miniserv. 109384 root 8u IPv4 3518429016 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:36008 (ESTABLISHED)
miniserv. 109388 root 8u IPv4 3518429019 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:37580 (ESTABLISHED)
miniserv. 109441 root 8u IPv4 3518429031 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:42140 (ESTABLISHED)
miniserv. 109442 root 8u IPv4 3518429034 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:42138 (ESTABLISHED)
miniserv. 109443 root 8u IPv4 3518429037 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:43422 (ESTABLISHED)
miniserv. 109451 root 8u IPv4 3518429049 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:48618 (ESTABLISHED)
miniserv. 109452 root 8u IPv4 3518446780 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:48626 (ESTABLISHED)
miniserv. 109454 root 8u IPv4 3518446786 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:48912 (ESTABLISHED)
miniserv. 109533 root 8u IPv4 3518428098 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:51770 (ESTABLISHED)
miniserv. 109557 root 8u IPv4 3518428105 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:49996 (ESTABLISHED)
miniserv. 109564 root 8u IPv4 3518428108 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:55204 (ESTABLISHED)
miniserv. 109566 root 8u IPv4 3518428114 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:55442 (ESTABLISHED)
miniserv. 109582 root 8u IPv4 3518449684 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:35610 (ESTABLISHED)
miniserv. 109682 root 8u IPv4 3518431957 0t0 TCP xx.xx.xx.xx:webmin->118.193.36.188:50020 (ESTABLISHED)
[~]: grep 118.193.36.188 /var/webmin/webmin.log
[~]: grep 118.193.36.188 /var/webmin/miniserv.log
Binary file /var/webmin/miniserv.log matches
[~]: grep -a 118.193.36.188 /var/webmin/miniserv.log
118.193.36.188 - - [03/Apr/2022:18:15:45 +0200] “GET / HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:46 +0200] “GET /other/codepay/js/codepay_util.js HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:51 +0200] “GET / HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:53 +0200] “GET / HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:15:57 +0200] “GET /otc/ HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:08 +0200] “GET /reg.php HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:09 +0200] “GET /logo.png HTTP/1.1” 404 345
118.193.36.188 - - [03/Apr/2022:18:16:18 +0200] “GET /getLocale HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:28 +0200] “GET /m HTTP/1.1” 401 4892
118.193.36.188 - - [03/Apr/2022:18:16:28 +0200] “GET /api/pc/configure HTTP/1.1” 401 4892
so not in webmin but some in miniserv.pl.
best regards,
Ghislain.
Jamie
April 6, 2022, 3:53am
17
Interesting, looks like the client never sent a request.
How long do these connections hang around for?
jimr1
April 7, 2022, 6:41am
18
see here , this has been around for ages
gadnet
April 7, 2022, 12:22pm
19
the processes stay for month, i clear them regulary when i login but i can find some that have 6month at least.
i can find process like this here since 2019 !!
this is on several machine, webmin and usermin
root 1204 0.0 0.2 137664 73260 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 13024 0.0 0.2 137668 73548 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 13092 0.0 0.0 12280 6436 ? S 2019 0:00 _ /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 14535 0.0 0.2 137664 73260 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 17397 0.0 0.2 137664 73132 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 20905 0.0 0.2 137668 73304 ? S 2019 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 23339 0.0 0.0 12280 7524 ? S 2019 0:00 _ /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 23489 0.0 0.0 12280 6436 ? S 2019 0:00 _ /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 3281 0.0 0.0 12280 8056 ? Ss 2019 14:10 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 11212 0.0 0.4 140692 72532 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 11214 0.0 1.1 97588 73568 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 11496 0.0 0.2 138216 73796 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 13423 0.0 0.1 80804 20648 ? S 2020 0:00 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 134885 0.0 0.2 138236 71072 ? S 2020 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
regards,
Ghislain.
Jamie
April 8, 2022, 4:07am
20
Wait, 2019? Has this machine not been rebooted for 3 years?