Certificate renewals failing on subdomains that are reverse proxied

Virtualmin
1

I’m no expert here, but I suspect the reason I’m getting the following errors is because there’s a configuration problem in my apache conf file for this subdomain. Here is the error:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Reusing existing private key from /etc/letsencrypt/live/bitwarden.jongriffith.com/privkey.pem.
Performing the following challenges:
http-01 challenge for bitwarden.jongriffith.com
Using the webroot path /home/jongriffith.com/domains/bitwarden.jongriffith.com/public_html for all unmatched domains.
Waiting for verification...
Challenge failed for domain bitwarden.jongriffith.com
http-01 challenge for bitwarden.jongriffith.com
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: bitwarden.jongriffith.com
   Type:   unauthorized
   Detail: 68.15.186.195: Invalid response from
   http://bitwarden.jongriffith.com/.well-known/acme-challenge/D6urxXqAnJLCJ-SC_lxJ68ubIWVT8mFeUc4xhjv8QIQ:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

and:

Performing the following challenges:
dns-01 challenge for bitwarden.jongriffith.com
Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl
Waiting for verification...
Challenge failed for domain bitwarden.jongriffith.com
dns-01 challenge for bitwarden.jongriffith.com
Cleaning up challenges
Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl
Some challenges have failed.
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: bitwarden.jongriffith.com
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up TXT for
   _acme-challenge.bitwarden.jongriffith.com - check that a DNS record
   exists for this domain

QUESTION:

Is the path http://bitwarden.jongriffith.com/.well-known/acme-challenge/ created on the fly and then deleted after the challenge is completed? There is no path like this on neither the document root (shown above) nor the root of the site on the proxied host.

I’m guessing that the creation of this folder on the fly is the problem because it’s on a proxied host, but I don’t know how to solve the problem.

I’ll post the conf file to the apache2 site here and maybe someone can help me with the configuration so it works. Also not sure why renewals were going through until recently on these proxied sites.

<VirtualHost 192.168.1.200:80 [fe80::41:f6ff:fe68:9051]:80>
    SuexecUserGroup "#1002" "#1002"
    ServerName bitwarden.jongriffith.com
    #ServerAlias www.bitwarden.jongriffith.com
    #ServerAlias mail.bitwarden.jongriffith.com
    #ServerAlias webmail.bitwarden.jongriffith.com
    #ServerAlias admin.bitwarden.jongriffith.com
    DocumentRoot /home/jongriffith.com/domains/bitwarden.jongriffith.com/public_html
    ErrorLog /var/log/virtualmin/bitwarden.jongriffith.com_error_log
    CustomLog /var/log/virtualmin/bitwarden.jongriffith.com_access_log combined
    ScriptAlias /cgi-bin/ /home/jongriffith.com/domains/bitwarden.jongriffith.com/cgi-bin/
    DirectoryIndex index.html index.htm index.php index.php4 index.php5
    <Directory /home/jongriffith.com/domains/bitwarden.jongriffith.com/public_html>
        Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
        allow from all
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
        Require all granted
        AddType application/x-httpd-php .php
        AddHandler fcgid-script .php
        AddHandler fcgid-script .php7.4
        AddHandler fcgid-script .php8.2
        FCGIWrapper /home/jongriffith.com/domains/bitwarden.jongriffith.com/fcgi-bin/php7.4.fcgi .php
        FCGIWrapper /home/jongriffith.com/domains/bitwarden.jongriffith.com/fcgi-bin/php7.4.fcgi .php7.4
        FCGIWrapper /home/jongriffith.com/domains/bitwarden.jongriffith.com/fcgi-bin/php8.2.fcgi .php8.2
    </Directory>
    #<Directory /home/jongriffith.com/domains/bitwarden.jongriffith.com/cgi-bin>
    #    allow from all
    #    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    #    Require all granted
    #</Directory>
    ProxyPreserveHost On
    ProxyRequests off
    ProxyVia On
    #ProxyPass /api/websocket ws://192.168.1.101:7277/api/websocket
    #ProxyPassReverse /api/websocket ws://192.168.1.101:7277/api/websocket
    ProxyPass / http://192.168.1.101:7277/
    ProxyPassReverse / http://192.168.1.101:7277/
    RewriteEngine on
    #RewriteCond %{HTTP:Upgrade} =websocket [NC]
    #RewriteRule /(.*) ws://192.168.1.101:7277/$1 [P,L]
    #RewriteCond %{HTTP:Upgrade} !=websocket [NC]
    #RewriteRule /(.*) http://192.168.1.101:7277/$1 [P,L]
    #RewriteEngine on
    #RewriteCond %{HTTP_HOST} =webmail.bitwarden.jongriffith.com
    #RewriteRule ^(?!/.well-known)(.*) https://bitwarden.jongriffith.com:20000/ [R]
    #RewriteCond %{HTTP_HOST} =admin.bitwarden.jongriffith.com
    #RewriteRule ^(?!/.well-known)(.*) https://bitwarden.jongriffith.com:10000/ [R]
    RemoveHandler .php
    RemoveHandler .php7.4
    RemoveHandler .php8.2
    FcgidMaxRequestLen 1073741824
    RedirectMatch ^/(?!.well-known)(.*)$ https://bitwarden.jongriffith.com/$1
    IPCCommTimeout 31
</VirtualHost>
<VirtualHost 192.168.1.200:443 [fe80::41:f6ff:fe68:9051]:443>
    SuexecUserGroup "#1002" "#1002"
    ServerName bitwarden.jongriffith.com
    #ServerAlias www.bitwarden.jongriffith.com
    #ServerAlias mail.bitwarden.jongriffith.com
    #ServerAlias webmail.bitwarden.jongriffith.com
    #ServerAlias admin.bitwarden.jongriffith.com
    DocumentRoot /home/jongriffith.com/domains/bitwarden.jongriffith.com/public_html
    ErrorLog /var/log/virtualmin/bitwarden.jongriffith.com_error_log
    CustomLog /var/log/virtualmin/bitwarden.jongriffith.com_access_log combined
    #ScriptAlias /cgi-bin/ /home/jongriffith.com/domains/bitwarden.jongriffith.com/cgi-bin/
    DirectoryIndex index.html index.htm index.php index.php4 index.php5
    <Directory /home/jongriffith.com/domains/bitwarden.jongriffith.com/public_html>
        Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
        allow from all
        AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
        Require all granted
        AddType application/x-httpd-php .php
        AddHandler fcgid-script .php
        AddHandler fcgid-script .php7.4
        AddHandler fcgid-script .php8.2
        FCGIWrapper /home/jongriffith.com/domains/bitwarden.jongriffith.com/fcgi-bin/php7.4.fcgi .php
        FCGIWrapper /home/jongriffith.com/domains/bitwarden.jongriffith.com/fcgi-bin/php7.4.fcgi .php7.4
        FCGIWrapper /home/jongriffith.com/domains/bitwarden.jongriffith.com/fcgi-bin/php8.2.fcgi .php8.2
    </Directory>
    #<Directory /home/jongriffith.com/domains/bitwarden.jongriffith.com/cgi-bin>
    #    allow from all
    #    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
    #    Require all granted
    #</Directory>
    ProxyPreserveHost On
    ProxyRequests off
    ProxyVia On
    #ProxyPass /api/websocket ws://192.168.1.101:7277/api/websocket
    #ProxyPassReverse /api/websocket ws://192.168.1.101:7277/api/websocket
    ProxyPass / http://192.168.1.101:7277/
    ProxyPassReverse / http://192.168.1.101:7277/
    RewriteEngine on
    #RewriteCond %{HTTP:Upgrade} =websocket [NC]
    #RewriteRule /(.*) ws://192.168.1.101:7277/$1 [P,L]
    #RewriteCond %{HTTP:Upgrade} !=websocket [NC]
    RewriteRule /(.*) http://192.168.1.101:7277/$1 [P,L]
    #RewriteEngine on
    #RewriteCond %{HTTP_HOST} =webmail.bitwarden.jongriffith.com
    #RewriteRule ^(?!/.well-known)(.*) https://bitwarden.jongriffith.com:20000/ [R]
    #RewriteCond %{HTTP_HOST} =admin.bitwarden.jongriffith.com
    #RewriteRule ^(?!/.well-known)(.*) https://bitwarden.jongriffith.com:10000/ [R]
    RemoveHandler .php
    RemoveHandler .php7.4
    RemoveHandler .php8.2
    FcgidMaxRequestLen 1073741824
    SSLEngine on
    SSLCertificateFile /home/jongriffith.com/domains/bitwarden.jongriffith.com/ssl.cert
    SSLCertificateKeyFile /home/jongriffith.com/domains/bitwarden.jongriffith.com/ssl.key
    SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
    SSLCACertificateFile /home/jongriffith.com/domains/bitwarden.jongriffith.com/ssl.ca
    #IPCCommTimeout 31
</VirtualHost>

Any insight would be greatly appreciated. Thanks.

2

Yes.

You need to exclude that path from your proxy rules.

ProxyPass /.well-known/ !

https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass

3

This did not work. What I’m more concerned about is that all three of the sub-domains that I have that are reverse proxied were renewing without problems until my subscription was renewed last month.

It can’t be possible that 3 conf files are suddenly incorrect, unless something changed them in an update…or … I don’t know…

All I know is that I didn’t make any changes to the server in the time since the last renewals and now.

4

Okay, so I tried adding the ProxyPass /.well-known/ ! line to the conf, then I created the .well-known folder and dropped in a hello world index.html file to see if it would be served.

I get a 404. If this is working correctly, shouldn’t the index.html file I added be served?

5

Yep.

Look in the access_log and error_log for that domain for clues about why it’s failing.