| SYSTEM INFORMATION |
Operating system Rocky Linux 9.4
Webmin version 2.202
Usermin version 2.102
Virtualmin version 7.20.2
Authentic theme version 21.20.7
Automatic renewal of Lets Encrypt certificate succeeded at 5:08 am and received email indicating this. Certificate is installed and shows as 90 days till expiration. But then at 5:09, a failure message indicates (domain and IP replaced with xs) :
Web-based validation failed : Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Requesting a certificate for xxxxxxxx.com and www.xxxxxxxx.com Performing the following challenges: [LISTS THE CHALLENGES]
Waiting for verification…
Challenge failed for domain [LISTS THE DOMAINS] Cleaning up challenges Some challenges have failed.
IMPORTANT NOTES:
The following errors were reported by the server:
Domain: xxxxxxxx.com
Type: unauthorized
Detail: x.x.x.x: Invalid response from
http://www.xxxxxxxx.com/.well-known/acme-challenge/ldnU2yY47UT7QOhYP9agUoa6W7L_FoIJm2bX1QvHi84:
404
A look in /var/log/letsencrypt/letsencrypt.log shows the successful request and installation but oddly there is nothing to match with the failure emails that follow. A failure email for the same domain shows up roughly every 60 minutes. At 6:14 am, 7:19, 8:24, 9:29, 10:34.
I could not discover what is triggering these failure emails. Looked in the scheduled cron jobs, running processes, but come up empty to try to discover the source. Since there is nothing in the letsencrypt.log file it almost seems like an actual attempt to get a certificate does not happen but a triggered email does.
Tried doing just a manual SSL certificate renew for this domain hoping that might clear this away. The manual renew succeeded (although didn’t send a success email but maybe it doesn’t since you’re sitting there watching it in real time.) But then got another failure email at 11:39.
So I am stumped. What is causing these failure emails? If the letsencrypt.log file has no info about a renewal failure, does that mean the failed renewal never happened?
The error indicates there is a issue with the https challenge.
Do you have a working website at the address in the error, you haven’t added a redirect or proxy?
There is a working website there and it has a current SSL certificate from Lets Encrypt from 5:08 am on 8/22. The failure emails started at 5:09 am (and still continue to arrive.) I just checked again and there is nothing in the /var/log/letsencrypt/letsencrypt.log regarding this domain since the successful renewals earlier today.
Something is triggering and generating this email and sending it out every 65 minutes. Whatever it is does not write anything into the log file noted above.
So just discovered that there are records in the apache access log for the failures even though nothing is recorded in the letsencrypt.log. Here is the successful series that generated the email at 5:08 am:
13.214.38.31 - - [22/Aug/2024:05:07:10 -0400] "GET /.well-known/acme-challenge/LfbrD_lRYJaiW9hRZhjfVvAvqbayHB96Rx7UAN3_Rbk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
47.129.170.61 - - [22/Aug/2024:05:07:10 -0400] "GET /.well-known/acme-challenge/93DNJzWN-t85BiPlWV8coS1C9vT-pfj4_MliUBTiip0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
13.60.83.207 - - [22/Aug/2024:05:07:10 -0400] "GET /.well-known/acme-challenge/LfbrD_lRYJaiW9hRZhjfVvAvqbayHB96Rx7UAN3_Rbk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
51.20.51.232 - - [22/Aug/2024:05:07:10 -0400] "GET /.well-known/acme-challenge/93DNJzWN-t85BiPlWV8coS1C9vT-pfj4_MliUBTiip0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
54.201.121.173 - - [22/Aug/2024:05:07:10 -0400] "GET /.well-known/acme-challenge/LfbrD_lRYJaiW9hRZhjfVvAvqbayHB96Rx7UAN3_Rbk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
54.201.121.173 - - [22/Aug/2024:05:07:09 -0400] "GET /.well-known/acme-challenge/93DNJzWN-t85BiPlWV8coS1C9vT-pfj4_MliUBTiip0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.144.221.106 - - [22/Aug/2024:05:07:09 -0400] "GET /.well-known/acme-challenge/LfbrD_lRYJaiW9hRZhjfVvAvqbayHB96Rx7UAN3_Rbk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.144.221.106 - - [22/Aug/2024:05:07:09 -0400] "GET /.well-known/acme-challenge/93DNJzWN-t85BiPlWV8coS1C9vT-pfj4_MliUBTiip0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.103 - - [22/Aug/2024:05:07:09 -0400] "GET /.well-known/acme-challenge/LfbrD_lRYJaiW9hRZhjfVvAvqbayHB96Rx7UAN3_Rbk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.100 - - [22/Aug/2024:05:07:09 -0400] "GET /.well-known/acme-challenge/93DNJzWN-t85BiPlWV8coS1C9vT-pfj4_MliUBTiip0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
Then every 65 minutes there is a series of requests that fail with http codes 301 and 404 and they always look like this:
23.178.112.109 - - [22/Aug/2024:05:08:32 -0400] "GET /.well-known/acme-challenge/ldnU2yY47UT7QOhYP9agUoa6W7L_FoIJm2bX1QvHi84 HTTP/1.1" 404 51611 "http://xxxxxxxx.com/.well-known/acme-challenge/ldnU2yY47UT7QOhYP9agUoa6W7L_FoIJm2bX1QvHi84" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.104 - - [22/Aug/2024:05:08:31 -0400] "GET /.well-known/acme-challenge/D7XfQT3PmIFDXtaL898rFs_kzWBVne6I9gRh_bo3CyY HTTP/1.1" 404 51617 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.109 - - [22/Aug/2024:05:08:31 -0400] "GET /.well-known/acme-challenge/ldnU2yY47UT7QOhYP9agUoa6W7L_FoIJm2bX1QvHi84 HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.104 - - [22/Aug/2024:05:08:09 -0400] "GET /.well-known/acme-challenge/hfmv0sqchyMiRlaCUg52j6kG5LotWL8_WCo0M2wDWVo HTTP/1.1" 404 51610 "http://xxxxxxxx.com/.well-known/acme-challenge/hfmv0sqchyMiRlaCUg52j6kG5LotWL8_WCo0M2wDWVo" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.104 - - [22/Aug/2024:05:08:08 -0400] "GET /.well-known/acme-challenge/s--aLwV5Bd8-RUadGpc5DxN73wEHz1IQ8xMmeS1N7fM HTTP/1.1" 404 51602 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.104 - - [22/Aug/2024:05:08:08 -0400] "GET /.well-known/acme-challenge/hfmv0sqchyMiRlaCUg52j6kG5LotWL8_WCo0M2wDWVo HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
Key point here though is that there is no reason for this to even happen. The first attempt to renew the certificate was successful and the certificate indicates that it expires in 90 (or 89 now) days. And obviously Virtualmin is making some request to Lets Encrypt because they are accessing the server looking for the challenge file but Virtualmin is not recording it in the letsencrypt.log file.
Yeah the error says there a issue.
test using this site, select http
https://letsdebug.net/
the error is reported by www.letsencrypt.org not virtualmin, virtualmin just uses there tools.
letsdebug.net says there is nothing wrong with the domain - and indeed there is nothing wrong. As the title here states, Lets Encrypt succeeded and installed a good certificate. It is Virtualmin that AFTERWARDS continues to think there is a need to renew the certificate. It is stuck in some process that does not do the request correctly as it does not record anything in the letsencrypt.log file.
It’s as if it made some mistake when it requested and succeeded like not recording the success correctly and now thinks it needs to retry every 65 minutes.
No idea, sounds weird. Maybe staff have an idea.
Yes, quite unusual. Other domains on this same Virtualmin setup have gone through Lets Encrypt renewals without issue. This single domain has the problem. Certainly appears to be a bug in Virtualmin maybe caused by some anomaly in the domain.
It’s been 24 hours now of 65 minute retries that fail. I’m about to restore this domain to a backup from a week ago to make this go away. Don’t want to get banned by Lets Encrypt for tagging them repeatedly with a malformed renewal request I can’t stop and also can’t investigate.
Left with no recourse to fix this, I’ve deleted and recreated this domain virtual server from a backup. I noticed that as part of this process, Virtualmin requested a certificate from Lets Encrypt which was successful. The SSL certificate on the backup was 28 days from expiring. So fingers crossed I don’t end up in this repeating failure again.
1 Like