I’ve got 2 AlmaLinux vpses that are configered relatively similar. The sites configured on host2 have host1 as their secondary name server and the sites on host1 have host2 as the secondary name server. The sites are having vhosts served by Apache.
All worked well until a few months ago. The certificate renewal fails on host2 while the sites on host1 have still proper renewals. They both use the functionality from Virtualmin out of the box.
When I go to the Manage Virtual Server > Manage SSL Certificates > SSL Providers and click request certificate (e.g. just for www.domain.tld and domain.tld) it gives me the following output:
Renewal failed due to
Web-based validation failed :
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for domain.tld and www.domain.tld
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: domain.tld
Type: unauthorized
Detail: 31.220.44.xxx: Invalid response from http://domain.tld/.well-known/acme-challenge/x13qErfPDjt1dk0pPICQeEr9C34MJoWolWyiGdYr78k: 404
Domain: www.domain.tld
Type: unauthorized
Detail: 31.220.44.xxx: Invalid response from http://www.domain.tld/.well-known/acme-challenge/cF_RQjoxb3fLWc95HF1Wn0x9NV8NKC5jYYapCZr8QZo: 404
Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
DNS-based validation failed :
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for domain.tld and www.domain.tld
Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
Domain: domain.tld
Type: dns
Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.domain.tld - check that a DNS record exists for this domain
Domain: www.domain.tld
Type: dns
Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.domain.tld - check that a DNS record exists for this domain
Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Last successful renewal
04/06/2025 10:03 AM
Last failed renewal
06/20/2025 12:04 PM
When checking the logs, it looks like it can not find the _acme-challenge. Not via web based validation and not via DNS based validation (would prefer the latest to support wildcards as well).
I tried to request the certificate from the console using certbot.
certbot certonly --manual --preferred-challenges dns -d domain.tld -v
This generated the acme_challenge. When I add this manually as a dns record in Virtualmin DNS records and save it, certbot does generate the certificate. I can copy the certificate to the path of the virtual server and the accompanying private key as well and then the certificate is renewed successfully. (in the virtualmin Setup SSL certificate it shows ’ * SSL CA cert for Let’s Encrypt/R10 does not match the issuer of the SSL cert Let’s Encrypt/E5’ but it seems unrelated and it works fine, expiration after 89 days).
It looks like the hooks to the virtualmin perl scripts to create the dns records or to put the acme-challenge in place do not work for some reason. I am not sure why though. According to the logs it is being called but it does not log anything itself as far as I can see and it looks like it does not add the acme challenge.
I hope someone can give me some guidance on what could be misconfigured or what could have broken this process. How can I debug the letsencrypt-dns.pl script?
With regard to DNS and wildcards…I would guess it’s failing because Virtualmin is not managing your DNS (it thinks that it is, but I guess the glue records for your zone are not pointing to the Virtualmin server and its secondary). It wouldn’t try to use DNS validation or offer to create wildcards if it wasn’t configured to manage DNS.
If Virtualmin is not managing your DNS you should disable that feature in Features and Plugins or for the domains it is not managing.
Virtualmin is managing the dns. I also added the _acme-challenge in Virtualmin to make it work when calling certbot manually. My idea is that the webmin/vitualmin script called by certbot to add the acme-challenge to the virtual host as a file or to the DNS as a TXT record does not function properly (or maybe further down the chain, a command that is called by that script). It can not find the acme-challenge because it is not added. Certbot is able to retrieve the challenge and when manually added in the DNS (the bind server set up by virtualmin), validation does work and a valid certificate is generated. Thinks do not work automatically though. However it did work automatically a few months ago, so apparently something changed due to a software update or a changed setting.
I tried disabling the firewall for a moment when I renew the certificate and it did not make a difference. The site is reachable and serving visitors. I think the 404 is there because the acme-challenge is not put there. I think something goes wrong after fetching the challenge and before checking the challenge server by the webserver or dns. It looks like the problem is present for all sites on this vps and it used to work a few months ago. It also still works on the other vps (that has the other virtualmin dns server set up similar and serves some sites where this vps serves as the slave dns server).
is this another example of the www.domain.tld/.well-known/ directory not being accessible? just put a simple hello.html file there and browse to it. is there some .htaccess preventing it?
It has to be if you don’t have a cert https won’t work correctly so therefore certbot won’t be able to get there. As to why http is not available only the op knows . But at a guess there is something in the .htaccess file or apache config files or however nginx simulates .htaccess if this is an nginx server. Or this domain may have cloudflare enabled which could have a baring on it
The http site is served from port 8080 as I used to point the domain to a caching server that fetched it from 8080. Maybe that is a problem for the HTTP validation, but shouldn’t it then still sucessfully fall back to DNS based validation.
There is no AAAA record as I solely use IPv4 addresses.
The domain used for the virtualmin host names (and primary and secondary ns) has it’s DNS records externally (IBM NS1). A glue record at the registrar is not necessary in that case as far as I understand.
Clicking your last link the ,8080 one, always redirects to https so there something forcing https redirects. As you far away from a standard setup I would retrace all your steps to see if you can find the problem
It must be accessible over HTTP. That is not negotiable or configurable. What happens if your certificate is expired or invalid? You then wouldn’t be able to get a new certificate. You must configure any redirects or proxy rules to allow requests to .well-known to be served from the filesystem.
If Virtualmin is managing DNS for your zone (the domain name), then the glue records must point to the DNS servers managed by Virtualmin (itself and its secondary, probably, but there are other ways to configure things, Route 53 is supported, and there may be other cloud DNS providers involved if you’re using Virtualmin Pro). Let’s Encrypt DNS validation is proving you are the owner of the domain. If your glue records don’t point to the Virtualmin server, Virtualmin is not able to publish the necessary record to validate you own the domain.
If Virtualmin is managing the DNS servers authoritative for your domain (according to glue records), you’ll be able to issue Let’s Encrypt certs using DNS validation, and request wildcards (but you generally shouldn’t request wildcards, it’s very rarely a benefit and has some negative security implications).
This thread is kind of about two different problems and there’s a bunch of crosstalk about two unrelated issues (HTTP and DNS validation, two wholly separate subsystems with wholly separate solutions). so if you want to follow up about DNS validation and how DNS needs to be configured for DNS validation to work, make a new topic.
I believe your HTTP validation problem can be solved by simply excluding .well-known from whatever redirect or proxying you’re doing on http requests to your server. Those must be served from the fileystem.
In the DNS records hosted by IBM NS1 the host2.mydomain.tld point to the IP of my virtualmin VPS.
All other domains hosted on the vps have their DNS records managed by bind on the VPS. I think a glue record is needed when the host2.mydomain.tld would use the same name server to prevent circular references. I have been using glue records in the past, when I did have the domain DNS the VPS itself.
Furthermore if a glue records would have been the issue for DNS validation then it is odd that requesting a certificate with certbot and manually adding the record in Virtualmin DNS records would just work.
I am sorry if I misunderstood.
The VirtualHosts for non-ssl traffic do have this line.
If you’re not able to reach a file you put in /home/domainname/public_html/.well-known then obviously you have something else sucking up requests and you need to fix it.
Please make a new topic to discuss DNS issues. They are unrelated to HTTP validation, and I can’t deal with chaotic threads about many different unrelated problems. The cross-talk is too messy. This is a failing on my part, but I just can’t follow chaotic threads, so I ask everyone to stick to one problem per thread.
I will create a new ticket for the DNS validation. When changing the virtualhost to listen to both 8080 and 80 the web based validation works. I had it on port 8080 for a long time (to prevent users accessing the site and bypassing the caching server), certificate renewals weren’t an issue, not sure why it worked then. I actually thought that Virtualmin did create a temporary VirtualHost on port 80 during the renewal, but apparently it doesn’t.
Also I was assuming the problem was in the manual auth hook from certbot to webmin/virtualmin and that the acme challenge was received but not put in place. That could have been a common cause for both dns renewal and web based, but apparently these are different problems here.
Thanks a lot for your time and effort.
No. Virtualmin created a VirtualHost when the virtual server (domain) was created. It has no reason to believe it needs a temporary one. Everything works if you let it.
That’s not the right way to solve that problem. (New topic if you want to discuss the right way to solve it.)
There is no such hook. Virtualmin calls certbot, not the other way around (I mean, there isn’t a hook that certbot runs to do the magic one would expect in a Virtualmin installation, where the cert works in all the various services including Virtualmin/Webmin/Usermin, Dovecot, Postfix, web, etc.), which is why you have to be careful calling certbot manually, because that skips all the steps Virtualmin takes to make the cert available for all the various services.
A cert for a domain is either managed by Virtualmin or you manage it manually. You shouldn’t try to mix and match.