Cloudmin can't connect to Webmin (RPC Protocol)

SYSTEM INFORMATION (CLOUDMIN) (VIRTUALMIN)
OS type and version CentOS 7.9 (EOL I know) Rocky Linux 8.10
Webmin version 2.202 2.202
Cloudmin/Virtualmin version 9.8 Connect 7.30.2

Hi, I have a bunch of managed virtualmin servers on cloudmin with no problem.
I’ve been re-creating my virtualmin servers with Rocky Linux. They are all individually working (I can access virtualmin panel online and use it normally).
But when I try to add them on Cloudmin to be managed, Webmin connection fails. It says:

Checking SSH login ..
.. logged in OK.

Checking Webmin login ..
.. Webmin login failed : Connection failed (Port 10001 is firewalled. Ports 10001 to 10010 used by Webmin's RPC protocol)

I checked my virtualmin firewall config many times, even disabling it but I get the same message. I also checked if there are any other firewalls in between them but I’m certain there is none.
SELinux is off, I checked netstat to see if webmin is listening to those ports, but it doesnt appear to be, even on the servers that work with cloudmin, so I guess these ports only listen when needed…
Anyway, any suggestions on how I could debug and solve this?

P.S.: My firewalld rules

[root@xxxxx ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: dhcpv6-client dns dns-over-tls ftp http https imap imaps mdns pop3 pop3s smtp smtp-submission smtps ssh
  ports: 10000-10100/tcp 20000/tcp 49152-65535/tcp 20/tcp 2222/tcp 10000-10100/udp
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
        rule family="ipv4" source address="xxx.xxx.xxx.xxx/32" service name="ftp" accept
        rule family="ipv4" source address="xxx.xxx.xxx.xxx/32" service name="ftp" accept
        rule family="ipv4" source address="xxx.xxx.xxx.xxx/32" port port="3306" protocol="tcp" accept

Is there an external firewall?
I don’t use cloudmin but ‘assume’ it would open the correct ports.

If you’re at a cloud host, I assume they have their own firewall and you need to configure it appropriately, or use a private subnet for your VMs and allow them all to connect to each other “locally”. Most clouds offer that kind of thing, but you’ll need to consult the cloud provider docs to figure out how.

1 Like

They are VPS (Cloudmin on OVH Canada, and Virtualmin on Hostinger Brazil), and I made sure to check for any host provided firewalls, they are all disabled. Plus I don’t have any problems connecting to the other non-standard ports and services, thats why I wanted to know if there is a different way I can test for those RPC connections, to debug them on localhost, localnet and then externally

end to end

mit@~:tracepath -p 10000 tadmin.com
 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway                                              0.393ms 
 1:  _gateway                                              0.637ms 
 2:  syn-222-254-157-021.inf.hickup.com                 20.885ms 
 3:  no reply
 4:  lag-25.pltsohae01r.netops.charter.com                12.894ms 
 5:  lag-30.rcr01clmkohpe.netops.charter.com              15.710ms 
 6:  lag-1-100.rpr01cleyohdh.netops.charter.com           21.588ms asymm  7 
 7:  no reply
 8:  no reply
 9:  4.1.48.242                                           53.796ms asymm 14 
10:  lo-0.rc-b.slr.lxa.us.net.iguana.com                   55.157ms asymm 15 
11:  lo-0.gw-distdh-a.slr.lxa.us.placeholder.net            54.508ms asymm 15 
12:  ae-1.gw-prtr-r5-1a.slr.lxa.us.net.iguana.com          53.439ms asymm 17 
13:  fred.tadmin.com                                 53.733ms !H
     Resume: pmtu 1500
[root@cloudmin ~]# tracepath -p 10000 virtualmin.server.redacted
 1?: [LOCALHOST]                                         pmtu 1500
 1:  gateway                                               0.136ms 
 1:  gateway                                               0.118ms 
 2:  192.168.143.254                                       0.083ms 
 3:  149.56.60.126                                         0.256ms 
 4:  10.98.242.225                                         0.210ms 
 5:  10.196.145.44                                         0.255ms 
 6:  10.74.9.238                                           0.256ms 
 7:  10.95.81.10                                           1.182ms 
 8:  no reply
 9:  10.200.3.133                                         11.200ms 
10:  edgeuno.nyiix.net                                     9.696ms asymm 12 
11:  ae1255.0.edge8.gru1.as7195.net                      130.434ms asymm 16 
12:  ae0.0.edge7.gru1.as7195.net                         128.802ms asymm 15 
13:  200.25.58.93                                        128.092ms asymm 16 
14:  153.92.2.182                                        137.624ms asymm 17 
15:  153.92.3.13                                         137.517ms asymm 18 
16:  193.203.175.187                                     130.196ms asymm 19 
17:  xxxxxxxx.hstgr.cloud                               128.932ms reached
     Resume: pmtu 1500 hops 17 back 20 

[root@cloudmin ~]# tracepath -p 10001 virtualmin.server.redacted
 1?: [LOCALHOST]                                         pmtu 1500
 1:  gateway                                               0.193ms 
 1:  gateway                                               0.086ms 
 2:  192.168.143.254                                       0.087ms 
 3:  149.56.60.126                                         0.177ms 
 4:  10.98.242.225                                         0.168ms 
 5:  10.196.145.52                                         0.246ms 
 6:  10.74.9.234                                           0.187ms 
 7:  10.95.81.10                                           1.256ms 
 8:  no reply
 9:  10.200.3.133                                          9.917ms 
10:  edgeuno.nyiix.net                                     8.992ms asymm 12 
11:  ae1255.0.edge8.gru1.as7195.net                      130.448ms asymm 16 
12:  ae0.0.edge7.gru1.as7195.net                         129.349ms asymm 15 
13:  200.25.58.93                                        128.095ms asymm 16 
14:  153.92.2.182                                        137.776ms asymm 17 
15:  153.92.3.13                                         137.674ms asymm 18 
16:  193.203.175.187                                     130.110ms asymm 19 
17:  xxxxxxxx.hstgr.cloud                               127.774ms reached
     Resume: pmtu 1500 hops 17 back 20

Reachable on both ports.
If i curl https://virtualmin.server.redacted:10000/ on the cloudmin server i get the login page html normally. https cert is valid and all that…

9 more ports to go? There is a range used. 10000 -10010

Both? You’ve undercounted. Webmin fast RPC uses ports 10001-10100 (rarely using more than a handful of those ports, but it could use any of them).

It expanded to 10000-10100 a few years back.

My firewall rule goes from 10000 to 10100, but ok I’ll try more of them.
What if they are all reachable? What could make Cloudmin fail to use them?

Edit:
I made a script to test from 10000 to 10100.
Its reachable from 10000 to 10082. I can’t see a reason for them to not be reachable other than nothing listening on those ports.
What other tests could I do?