Let’s encrypt certificate renewal not working for virtualmin admin page

SYSTEM INFORMATION
OS type and version Debian Linux 12
Webmin version 2.610
Virtualmin version 7.50.2 GPL
Webserver version Apache version 2.4.65
Related packages Cerbot 2.1.0

I have 12 vhosts controlled by virtualmin and I am trying to renew SSL certificates using Let’s encrypt with web-based authentication (virtualmin/my server does not control DNS).

  • The 11 real domains/vhosts I made succeeds.
  • The 1 out-of-the-box vhost for the virtualmin admin page fails though (It seems the docroot doesnt actually serve files)

working vhost without subdomains (only serverAlias)

cmd tried

sudo virtualmin generate-letsencrypt-cert --domain rueth.online --renew --web

=> no error, all succeeds

config

<VirtualHost *:80>
    SuexecUserGroup #1006 #1006
    ServerName rueth.online
    ServerAlias www.rueth.online
    ServerAlias mail.rueth.online
    ServerAlias webmail.rueth.online
    ServerAlias admin.rueth.online
    DocumentRoot /home/ruethonline/public_html
    ErrorLog /var/log/virtualmin/rueth.online_error_log
    CustomLog /var/log/virtualmin/rueth.online_access_log combined
    ScriptAlias /cgi-bin/ /home/ruethonline/cgi-bin/
    ScriptAlias /awstats /home/ruethonline/cgi-bin/awstats.pl
    DirectoryIndex index.php index.htm index.html
    <Directory /home/ruethonline/public_html>
        Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    </Directory>
    <Directory /home/ruethonline/cgi-bin>
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    </Directory>
    ProxyPass /.well-known !
    RewriteEngine on
    RewriteCond %{HTTP_HOST} =webmail.rueth.online
    RewriteRule ^/(?!.well-known)(.*)$ https://rueth.online:20000/ [R]
    RewriteCond %{HTTP_HOST} =admin.rueth.online
    RewriteRule ^/(?!.well-known)(.*)$ https://rueth.online:10000/ [R]
    RemoveHandler .php
    RemoveHandler .php8.2
    RedirectMatch ^/awstats$ /awstats/
    <FilesMatch \.php$>
        SetHandler proxy:unix:/run/php/1759001099121642.sock|fcgi://127.0.0.1
    </FilesMatch>
    <Files awstats.pl>
        AuthName "rueth.online statistics"
        AuthType Basic
        AuthUserFile /home/ruethonline/.awstats-htpasswd
        require valid-user
    </Files>
    RewriteRule ^/(?!.well-known)(.*)$ https://%{HTTP_HOST}/$1 [R]
</VirtualHost>
<VirtualHost *:443>
    SuexecUserGroup #1006 #1006
    ServerName rueth.online
    ServerAlias www.rueth.online
    ServerAlias mail.rueth.online
    ServerAlias webmail.rueth.online
    ServerAlias admin.rueth.online
    DocumentRoot /home/ruethonline/public_html
    ErrorLog /var/log/virtualmin/rueth.online_error_log
    CustomLog /var/log/virtualmin/rueth.online_access_log combined
    ScriptAlias /cgi-bin/ /home/ruethonline/cgi-bin/
    ScriptAlias /awstats /home/ruethonline/cgi-bin/awstats.pl
    DirectoryIndex index.php index.htm index.html
    <Directory /home/ruethonline/public_html>
        Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    </Directory>
    <Directory /home/ruethonline/cgi-bin>
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    </Directory>
    ProxyPass /.well-known !
    RewriteEngine on
    RewriteCond %{HTTP_HOST} =webmail.rueth.online
    RewriteRule ^/(?!.well-known)(.*)$ https://rueth.online:20000/ [R]
    RewriteCond %{HTTP_HOST} =admin.rueth.online
    <FilesMatch \.php$>
        SetHandler proxy:unix:/run/php/1759001099121642.sock|fcgi://127.0.0.1
    </FilesMatch>
    RewriteRule ^/(?!.well-known)(.*)$ https://rueth.online:10000/ [R]
    RemoveHandler .php
    RemoveHandler .php8.2
    SSLEngine on
    SSLCertificateFile /etc/ssl/virtualmin/1759001099121642/ssl.cert
    SSLCertificateKeyFile /etc/ssl/virtualmin/1759001099121642/ssl.key
    SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
    RedirectMatch ^/awstats$ /awstats/
    <Files awstats.pl>
        AuthName "rueth.online statistics"
        AuthType Basic
        AuthUserFile /home/ruethonline/.awstats-htpasswd
        require valid-user
    </Files>
    SSLCACertificateFile /etc/ssl/virtualmin/1759001099121642/ssl.ca
</VirtualHost>

testing also gets http 200

sudo mkdir -p /home/ruethonline/public_html/.well-known/acme-challenge/
echo OK | sudo tee /home/ruethonline/public_html/.well-known/acme-challenge/test
sudo namei -l /home/ruethonline/public_html/.well-known/acme-challenge/test
curl -Il http://rueth.online/.well-known/acme-challenge/test

bad vhost (virtualmin admin)

cmd tried

sudo virtualmin generate-letsencrypt-cert --domain ns3119878.ip-51-38-181.eu --renew --web

=> error from tool:

.. failed : Web-based validation failed : Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for ns3119878.ip-51-38-181.eu

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: ns3119878.ip-51-38-181.eu
  Type:   unauthorized
  Detail: 51.38.181.35: Invalid response from http://ns3119878.ip-51-38-181.eu/.well-known/acme-challenge/3rvb1PTw0zBo2167LQ6-1HaygOP-11NCNNw_Fw5B8XY: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

config

<VirtualHost *:80>
    SuexecUserGroup #1001 #1001
    ServerName ns3119878.ip-51-38-181.eu
    DocumentRoot /home/._hostname/public_html
    ErrorLog /var/log/virtualmin/ns3119878.ip-51-38-181.eu_error_log
    CustomLog /var/log/virtualmin/ns3119878.ip-51-38-181.eu_access_log combined
    ScriptAlias /cgi-bin/ /home/._hostname/cgi-bin/
    DirectoryIndex index.php index.htm index.html
    <Directory /home/._hostname/public_html>
        Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
        AddType text/plain .php
    </Directory>
    <Directory /home/._hostname/cgi-bin>
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    </Directory>
    ProxyPass /.well-known !
    RemoveHandler .php
    <FilesMatch \.php$>
        SetHandler None
        AddType text/plain .php
    </FilesMatch>
</VirtualHost>
<VirtualHost *:443>
    SuexecUserGroup #1001 #1001
    ServerName ns3119878.ip-51-38-181.eu
    DocumentRoot /home/._hostname/public_html
    ErrorLog /var/log/virtualmin/ns3119878.ip-51-38-181.eu_error_log
    CustomLog /var/log/virtualmin/ns3119878.ip-51-38-181.eu_access_log combined
    ScriptAlias /cgi-bin/ /home/._hostname/cgi-bin/
    DirectoryIndex index.php index.htm index.html
    <Directory /home/._hostname/public_html>
        Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
        AddType text/plain .php
    </Directory>
    <Directory /home/._hostname/cgi-bin>
        Require all granted
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    </Directory>
    <FilesMatch \.php$>
        SetHandler None
        AddType text/plain .php
    </FilesMatch>
    ProxyPass /.well-known !
    RemoveHandler .php
    SSLEngine on
    SSLCertificateFile /etc/ssl/virtualmin/175897603948843/ssl.cert
    SSLCertificateKeyFile /etc/ssl/virtualmin/175897603948843/ssl.key
    SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
    SSLCACertificateFile /etc/ssl/virtualmin/175897603948843/ssl.ca
</VirtualHost>

testing also gets http 404

sudo mkdir -p /home/._hostname/public_html/.well-known/acme-challenge/
echo OK | sudo tee /home/._hostname/public_html/.well-known/acme-challenge/test
sudo namei -l /home/._hostname/public_html/.well-known/acme-challenge/
curl -Il http://ns3119878.ip-51-38-181.eu/.well-known/acme-challenge/test

That sounds like yet another variant of “the wrong site shows up”: Troubleshooting Websites | Virtualmin — Open Source Web Hosting Control Panel

If the .hostname VirtualHost is an * (and it is, as shown in your config above) and any others are an IP, the .hostname will never be served.