Troubleshooting HTTP for Let's Encrypt

Virtualmin
1
SYSTEM INFORMATION
OS type and version Rocky Linux 9.1
Webmin version 2.011
Virtualmin version 7.5 Pro

I’m not quite sure where I went wrong, but I can’t get Let’s Encrypt to work anymore. I can place a file and go to hxxps://www[dot]foo-bar[dot]com/.well-known/test.html but when I try hxxps://www[dot]foo-bar[dot]com/.well-known/test.html I get a 404 error.

Is there a way to determine the workflow of a web request to output on the server? If I could figure out the process that is causing port 80 to deny the file instead of showing it, then hopefully I can get Let’s Encrypt working again.

I’ve been racking my brain for days, and can’t seem to find a handle on this.

2

Directives listed below:

ServerName foo-bar[dot]com
ServerAlias www.foo-bar[dot]com
ServerAlias mail.foo-bar[dot]com
ServerAlias webmail.foo-bar[dot]com
ServerAlias admin.foo-bar[dot]com
ServerAlias 
DocumentRoot /home/hop/public_html
ErrorLog /var/log/virtualmin/foo-bar[dot]com_error_log
CustomLog /var/log/virtualmin/foo-bar[dot]com_access_log combined
ScriptAlias /cgi-bin/ /home/hop/cgi-bin/
ScriptAlias /awstats/ /home/hop/cgi-bin/
DirectoryIndex index.php index.php4 index.php5 index.htm index.html
<Directory /home/hop/public_html>
    Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch 
    Require all granted
    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>
<Directory /home/hop/cgi-bin>
    Require all granted
    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    SetHandler proxy:unix:/var/fcgiwrap/1672948756764787.sock/socket|fcgi://localhost
    ProxyFCGISetEnvIf true SCRIPT_FILENAME "/home/hop%{reqenv:SCRIPT_NAME}"
</Directory>
ProxyPass /.well-known !
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.foo-bar[dot]com
RewriteRule ^(?!/.well-known)(.*) https://www[dot]foo-bar[dot]com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.foo-bar[dot]com
RewriteRule ^(?!/.well-known)(.*) https://www[dot]foo-bar[dot]com:10000/ [R]
RemoveHandler .php
RemoveHandler .php8.0
RemoveHandler .php7.4
<FilesMatch \.php$>
    SetHandler proxy:unix:/var/php-fpm/1672948756764787.sock|fcgi://127.0.0.1
</FilesMatch>
<Files awstats.pl>
    AuthName "foo-bar[dot]com statistics"
    AuthType Basic
    AuthUserFile /home/hop/.awstats-htpasswd
    require valid-user
</Files>
RewriteCond %{HTTPS} off
RewriteRule ^/(?!.well-known)(.*)$ https://%{HTTP_HOST}/$1 [R]

ServerName foo-bar[dot]com
ServerAlias www.foo-bar[dot]com
ServerAlias mail.foo-bar[dot]com
ServerAlias webmail.foo-bar[dot]com
ServerAlias admin.foo-bar[dot]com
ServerAlias 
DocumentRoot /home/hop/public_html
ErrorLog /var/log/virtualmin/foo-bar[dot]com_error_log
CustomLog /var/log/virtualmin/foo-bar[dot]com_access_log combined
ScriptAlias /cgi-bin/ /home/hop/cgi-bin/
ScriptAlias /awstats/ /home/hop/cgi-bin/
DirectoryIndex index.php index.php4 index.php5 index.htm index.html
<Directory /home/hop/public_html>
    Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch 
    Require all granted
    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>
<Directory /home/hop/cgi-bin>
    Require all granted
    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    SetHandler proxy:unix:/var/fcgiwrap/1672948756764787.sock/socket|fcgi://localhost
    ProxyFCGISetEnvIf true SCRIPT_FILENAME "/home/hop%{reqenv:SCRIPT_NAME}"
</Directory>
ProxyPass /.well-known !
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.foo-bar[dot]com
RewriteRule ^(?!/.well-known)(.*) https://www[dot]foo-bar[dot]com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.foo-bar[dot]com
RewriteRule ^(?!/.well-known)(.*) https://www[dot]foo-bar[dot]com:10000/ [R]
RemoveHandler .php
RemoveHandler .php8.0
RemoveHandler .php7.4
<FilesMatch \.php$>
    SetHandler proxy:unix:/var/php-fpm/1672948756764787.sock|fcgi://127.0.0.1
</FilesMatch>
SSLEngine on
SSLCertificateFile /etc/ssl/virtualmin/1672948756764787/ssl.combined
SSLCertificateKeyFile /etc/ssl/virtualmin/1672948756764787/ssl.key
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
<Files awstats.pl>
    AuthName "foo-bar[dot]com statistics"
    AuthType Basic
    AuthUserFile /home/hop/.awstats-htpasswd
    require valid-user
</Files>
3

so everything showing ok at mxtoolbox.com?

4

Yes. DNS lookup records are correct. Let’s Debug is showing “All OK” with the site as well. Something is catching the request at port 80 and redirecting it to a 404 error on the web server. If I look up the same file using port 443, it shows it without problem.

5

have you got a .htaccess file doing redirects?

6

No, there aren’t any .htaccess files in the directory. So here’s something weird:

I can renew the Webmin SSL certificate just fine (Webmin > Webmin Configuration > SSL Encryption > Let’s Encrypt > Request Certificate), but it didn’t update the Usermin certificate (Webmin > Usermin Configuration > SSL Encryption > Current Certificate). I had to do that manually by selecting the Copy Certificate From Webmin button.

But…any of the other domains in Virtualmin redirect to that 404 error.

7

Weird, got me stumped. Under the VM Server Configuration → Website Directs I have a redirect to ssl that maybe set as well. but you shouldn’t get 404

8

I’ve got that as well for both the default Webmin site and the VM sites.

redirect
redirect838×136 28.3 KB

9

Thats different to mine, I have as destination https://yoursite.com/$1

10

That was put in by the server when I first set up the site and I haven’t changed it.

And to further muddy the water, since Webmin uses a subdomain, I decided to try and request a new cert for the main domain. It gives me this error:

Validating configuration for foo-bar[dot]com …
… errors were found, which will prevent Let’s Encrypt from issuing a certificate :

Apache website : An IPv6 DNS record with address dddd:cccc:cbbb::aaaa:d363 exists, but this virtual server does not have IPv6 enabled

I use Cloudflare, and that’s why I’m getting an IPv6 address, but if I don’t want to use IPv6 on the server.

11

mxtoolbox has nothing useful here.

12

If you can’t browse to .well-known on the server filesystem, obviously LE can’t validate. You have to fix that. When you proxy or redirect, you must exclude .well-known from that and allow it to be served from the filesystem.

13

Joe, there shouldn’t be any proxying or redirecting whatsoever. That’s where I’m having the problem.

14

After some further digging, I found that I can browse a new test.html file I placed in the .well-known folder of the main Webmin server.

When I tried to access the original test.html from my first post, which had different content, I was immediately redirected to the main Webmin server’s file that I had just created.

Since the content was different, then that was the only clue I had. Now just to figure out why the redirect is occurring.

15

Even if there is no htaccess file in that directory, the htaccess file the parent directory will apply recursively.

You should provide @stefan1959 an unqualified answer to his very valid question.

16

@Joe @stefan1959

There are no .htaccess files upstream either.

17

Have you tried turning off the cloudflare proxy while troubleshooting?
You mentioned that you are using cloudflare.
Turn off the proxy to show direct. Perhaps there is a conflict?

18

Good call, but I haven’t turned it on yet for that subdomain. Just DNS only with no proxy.

19

Did you manually enter this?

20

I did not. All of the new servers I create have that automatically in the configuration.