Letsencrypt renewal fails after working for a few years - anyone else seeing this?

SYSTEM INFORMATION
OS type and version Ubuntu Linux 24.04.3
Webmin version 2.520
Virtualmin version 7.50.0 Professional
Webserver version Apache version 2.4.58
Related packages n/a

Failed while requesting a certificate for xxx.com, www.xxx.com from Let’s Encrypt ..
.. request failed, I cannot find the 'acme … ’ file

does anyone know where I should find the file created by certbot? Also the Webserver is actually apache, but he letsencrypt log seem to indicate it expect nginx

This problem just started 3 days ago, but has already renewed 6 different Domain certificates more than 5 times for each … nothing modified for the two domains that started failing in the last few days … I am looking for clues about why it happened and how to fix it.

Diagnostic info: Web-based validation failed :

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for xxx.com and www.xxx.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: xxx.com
Type: connection
Detail: IP4Address: Fetching https:///.well-known/acme-challenge/GOsZ5je2-aDkgalzb7wKgpLW6vGNksIkgTuy9jhg9Us: Invalid empty host in redirect target

Domain: www.xxx.com
Type: connection
Detail: IPAddrress: Fetching https:///.well-known/acme-challenge/WIdKAlqKmX96kfC1tLCgefxKHq77zp–nSNMU4G0OKU: Invalid empty host in redirect target

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

All challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

DNS-based validation failed :

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for xxx.com and www.xxx.com

Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
Domain: xxx.com
Type: dns
Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.xxx.com - check that a DNS record exists for this domain

Domain: www.xxx.com
Type: dns
Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.xxxcom - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to “certbot --help manual” and the Certbot User Guide.

All challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

letsencrypt log excerpt: Requesting a certificate for xxx.com, www.xxx.com from Let’s Encrypt ..
.. request failed : Web-based validation failed :

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for xxx.com and www.xxx.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: xxx.com
Type: connection
Detail: IP$ Address: Fetching https:///.well-known/acme-challenge/GOsZ5je2-aDkgalzb7wKgpLW6vGNksIkgTuy9jhg9Us: Invalid empty host in redirect target

Domain: www.xxx.com
Type: connection
Detail: IP$ Address: Fetching https:///.well-known/acme-challenge/WIdKAlqKmX96kfC1tLCgefxKHq77zp–nSNMU4G0OKU: Invalid empty host in redirect target

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

All challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

DNS-based validation failed :

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate for xxx.com and www.xxx.com

Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
Domain: xxx.com
Type: dns
Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.xxx.com - check that a DNS record exists for this domain

Domain: www.xxx.com
Type: dns
Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.xxx.com - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to “certbot --help manual” and the Certbot User Guide.

All challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

letsencrypt log file excerpt:
2025-11-25 15:41:55,662:DEBUG:certbot._internal.main:certbot version: 2.9.0
2025-11-25 15:41:55,662:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/letsencrypt
2025-11-25 15:41:55,663:DEBUG:certbot._internal.main:Arguments: [‘-a’, ‘webroot’, ‘-d’, ‘xxx.com’, ‘-d’, ‘xxx.com’, ‘–webroot-path’, ‘/home/xxx/public_html’, ‘–duplicate’, ‘–force-renewal’, ‘–non-interactive’, ‘–agree-tos’, ‘–config’, ‘/tmp/.webmin/300939_8878_1_letsencrypt.cgi’, ‘–rsa-key-size’, ‘2048’, ‘–cert-name’, ‘xxx.com’, ‘–no-autorenew’, ‘–reuse-key’, ‘–key-type’, ‘rsa’]
2025-11-25 15:41:55,663:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2025-11-25 15:41:55,675:DEBUG:certbot._internal.log:Root logging level set at 30
2025-11-25 15:41:55,676:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2025-11-25 15:41:55,676:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Saves the necessary validation files to a .well-known/acme-challenge/ directory within the nominated webroot path. A seperate HTTP server must be running and serving files from the webroot path. HTTP challenge only (wildcards not supported).
Interfaces: Authenticator, Plugin
Entry point: EntryPoint(name=‘webroot’, value=‘certbot._internal.plugins.webroot:Authenticator’, group=‘certbot.plugins’)
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x78195b665d30>
Prep: True
2025-11-25 15:41:55,676:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x78195b665d30> and installer None
2025-11-25 15:41:55,676:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2025-11-25 15:41:55,851:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri=‘https://acme-v02.api.letsencrypt.org/acme/acct/2139127305’, new_authzr_uri=None, terms_of_service=None), dc5deccf68d319b72d35d02461d2f21b, Meta(creation_dt=datetime.datetime(2024, 12, 28, 20, 44, 46, tzinfo=), creation_host=‘xxx.ca’, register_to_eff=None))>
2025-11-25 15:41:55,852:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2025-11-25 15:41:55,867:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2025-11-25 15:41:56,101:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 “GET /directory HTTP/1.1” 200 1063
2025-11-25 15:41:56,102:DEBUG:acme.client:Received response:
HTTP 200

Server: nginx <<< not Apache? Why?

Date: Tue, 25 Nov 2025 20:41:56 GMT
Content-Type: application/json
Content-Length: 1063
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“BVDsvoC2wg0”: “Adding random entries to the directory - API Announcements - Let's Encrypt Community Support”,
“keyChange”: “https://acme-v02.api.letsencrypt.org/acme/key-change”,
“meta”: {
“caaIdentities”: [
letsencrypt.org
],
“profiles”: {
“classic”: “Profiles - Let's Encrypt”,
“shortlived”: “Profiles - Let's Encrypt (not yet generally available)”,
“tlsclient”: “Profiles - Let's Encrypt”,
“tlsserver”: “Profiles - Let's Encrypt
},
“termsOfService”: “https://letsencrypt.org/documents/LE-SA-v1.6-August-18-2025.pdf”,
“website”: “https://letsencrypt.org
},
“newAccount”: “https://acme-v02.api.letsencrypt.org/acme/new-acct”,
“newNonce”: “https://acme-v02.api.letsencrypt.org/acme/new-nonce”,
“newOrder”: “https://acme-v02.api.letsencrypt.org/acme/new-order”,
“renewalInfo”: “https://acme-v02.api.letsencrypt.org/acme/renewal-info”,
“revokeCert”: “https://acme-v02.api.letsencrypt.org/acme/revoke-cert
}
2025-11-25 15:41:56,271:DEBUG:certbot._internal.display.obj:Notifying user: Renewing an existing certificate for xxx.com and www.xxx.com
2025-11-25 15:41:56,271:DEBUG:certbot.configuration:Var reuse_key=True (set by user).
2025-11-25 15:41:56,549:INFO:certbot._internal.client:Reusing existing private key from /etc/letsencrypt/live/xxx.com/privkey.pem.
2025-11-25 15:41:56,555:DEBUG:acme.client:Requesting fresh nonce
2025-11-25 15:41:56,555:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2025-11-25 15:41:56,610:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 “HEAD /acme/new-nonce HTTP/1.1” 200 0
2025-11-25 15:41:56,610:DEBUG:acme.client:Received response:
HTTP 200

Server: nginx <<< also why not Apache?
Date: Tue, 25 Nov 2025 20:41:56 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: zahUfOdGPpVlJojG19nFqwYWvDuzVSaLpAoZJcNNaYCB1jGFxt4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

2025-11-25 15:41:56,610:DEBUG:acme.client:Storing nonce: zahUfOdGPpVlJojG19nFqwYWvDuzVSaLpAoZJcNNaYCB1jGFxt4
2025-11-25 15:41:56,610:DEBUG:acme.client:JWS payload:
b’{\n “identifiers”: [\n {\n “type”: “dns”,\n “value”: “xxx.com”\n },\n {\n “type”: “dns”,\n “value”: “www.xxx.com”\n }\n ]\n}’
2025-11-25 15:41:56,615:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
“protected”: “eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMjEzOTEyNzMwNSIsICJub25jZSI6ICJ6YWhVZk9kR1BwVmxKb2pHMTluRnF3WVd2RHV6VlNhTHBBb1pKY05OYVlDQjFqR0Z4dDQiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciJ9”,
“signature”: “IafFX0BnpN2OK2ZYEZ6lmWg4BGMNd2I_GpACwetFtCbpXefhsSD-tWvOPhYeHFkeRhXPkNjGoD2k-Zog3D9OUYz63IiUkWaZWuxfjjlERhV9x9gf5rxgh_RmwQubQIm_q5RGWA4F_lHwP3owowPA-VC81uwYzhfSJIByed03DUruB-_pM5HXPYsQ21ENCTKV7bLVKMaNGVLfYc74S-5y3n1TUVuD2t8Uf7bSzo9UWku4XDWTLQxlRgPxcAB-kVURNvXSerwCCJRMuD49-zzBFglCW5FSHX9ruRsjF0RzuS42TofyU5CrQGkBi3FtFR5euGAdNlbTRH0vzcwsjNs1vg”,
“payload”: “ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInByaW50YWJsZXM0ZnJlZS5jb20iCiAgICB9LAogICAgewogICAgICAidHlwZSI6ICJkbnMiLAogICAgICAidmFsdWUiOiAid3d3LnByaW50YWJsZXM0ZnJlZS5jb20iCiAgICB9CiAgXQp9”
}
2025-11-25 15:41:56,699:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 “POST /acme/new-order HTTP/1.1” 429 568
2025-11-25 15:41:56,699:DEBUG:acme.client:Received response:
HTTP 429
Server: nginx
Date: Tue, 25 Nov 2025 20:41:56 GMT
Content-Type: application/problem+json
Content-Length: 568
Connection: keep-alive
Boulder-Requester: 2139127305
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: jkzIQkhdC1oeixN6Gb8V9RcEUMFC8iAlsVWWJZ049f4AiZWrsDc

{
“type”: “urn:ietf:params:acme:error:rateLimited”,
“detail”: “Your account is temporarily prevented from requesting certificates for xxx.com, www.xxx.com and possibly others. Please visit: Let's Encrypt - Portal”,
“status”: 429
}
2025-11-25 15:41:56,700:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File “/bin/letsencrypt”, line 33, in
sys.exit(load_entry_point(‘certbot==2.9.0’, ‘console_scripts’, ‘certbot’)())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 19, in main
return internal_main.main(cli_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/certbot/_internal/main.py”, line 1894, in main
return config.func(config, plugins)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/certbot/_internal/main.py”, line 1600, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/certbot/_internal/main.py”, line 131, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/usr/lib/python3/dist-packages/certbot/_internal/renewal.py”, line 399, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/certbot/_internal/client.py”, line 428, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/certbot/_internal/client.py”, line 478, in _get_order_and_authorizations
orderr = self.acme.new_order(csr_pem)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/acme/client.py”, line 137, in new_order
response = self._post(self.directory[‘newOrder’], order)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/acme/client.py”, line 365, in _post
return self.net.post(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/acme/client.py”, line 738, in post
return self._post_once(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/acme/client.py”, line 751, in _post_once
response = self._check_response(response, content_type=content_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/acme/client.py”, line 602, in _check_response
raise messages.Error.from_json(jobj)
acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Your account is temporarily prevented from requesting certificates for xxx.com, www.xxx.com and possibly others. Please visit: Let's Encrypt - Portal
2025-11-25 15:41:56,740:ERROR:certbot._internal.log:An unexpected error occurred:
2025-11-25 15:41:56,740:ERROR:certbot._internal.log:Your account is temporarily prevented from requesting certificates for xxx.com, www.xxx.com and possibly others. Please visit: Let's Encrypt - Portal?

Thats a clue, by the forum at LE you redirecting to a IP address, “LE doesn’t support redirecting to an IP”.

Check here

1 Like

Thank you for this clue … not sure what to do with it yet, but it does help. Neither of the two domains that now have this problem have been modified for almost a year, neither has the server been modified other than regular Ubuntu & Virtualmin updates … however one of the other domains has been going through significant re-writes and has implemented a few re-directs with HTML updates … I’ll look at the possibility that one of those went into the wrong domain files.

1 Like

Put a file in .well-known and browse to the URL. There’s no need to guess about what’s happening.

The URL will be: http://youdomain.tld/.well-known/the-file-name

If you can’t browse to the file you created in .well-known, Let’s Encrypt can’t validate, and you need to fix that. According to the error, you’re redirecting somewhere, so you need to not do that for .well-known.

1 Like

Exactly this. Create a small script to see what the url does or use a simple tool like: https://www.redirect-checker.org/
Worst case scenario you got some virus, but most likely there is some wrong rewriterule in .htaccess or something.

Well … some progress, but not resolved yet.

I did create the folders: /.well-known and /.well-known/acme-challenge … and put JPG file in each and was able to get a browser to display the images.

I did also look at the apache logs and confirm that letsencrypt actually is getting a a RC 301 when attempting to GET the hashed txt file that cerbot successfully created on the server.

I cannot find any evidence of valid redirect … nor any difference to the config files for 3 other domains that are working properly … now know that only 2 domains are failing … the oldest ones, orginally written more than 10 years ago (not sure if that matters).

I have however noticed something very strange when using the ‘preview website’ within the webconfig function … the two websites that fail with letsencrypt also fail to display, and when copying the link to view before pushing the preview button shows the following link: https://172.16.1.21:10000/virtual-server/link.cgi/192.168.14.6/http://www.yyy.yyy/ … but when clicking on that link it instead produces a ‘permantly moved’ message with a redirect link to URL: https:/// … the 3 websites that still work do work … but if I copy the URL from the preview website selection it produces a valid working URL: https://172.16.1.21:10000/virtual-server/link.cgi/192.168.14.6/http://www.xxx.xxx/

I do recall removing 2 or 3 websites from the server a few months ago, and checking on the ‘enforce SSL’ … but I checked it off today, and it did not change the problem.

So, I now suspect that whatever is producing the letsencrypt problem is also causing the Permanent 301 when using Webmin

Any idea where I should look next?

Does it redirect you to https or some other URL and then display the image? Because, if so, that’s what Let’s Encrypt is complaining about. You need to be able to answer for a domain on exactly http://domain.tld/.well-known - if you’re redirecting to something else and then serving, that’s not acceptable to Let’s Encrypt.

Check for .htaccess. Some web apps place one automatically when enabling fancy URLs or whatever.

Also check to be sure you’re not seeing a “the wrong site shows up” problem (though that seems unlikely if you’re able to retrieve a file that only exists in /home/domain/public_html/.well-known (and not in any other domain’s .well-known dir).

Update: Not resolved, but finding out there are several related issues. Now preparing to migrate websites to a freshly prepared Virtualmin (Ubuntu 24.04.3) server … which seems to be the least effort and lowest risk.

Other symptoms:

  1. http requests to two of 5 domains get return code 301 … with URL updated in browser to have FQDN website portion of URL removed … Apache log file shows RC 301 using http 1.1

  2. https request for the same URL ‘works’ except for the expired certificate issue … Apache log file shows RC 200.

  3. other websites continue to work without obvious issue … all have valid unexpired certificates

  4. letsencrypt fetch of encrypted files in .well-known/acme-challenge folder appears to work (see attached images) Apache log shows RC 200 for each file … but still produces failure to renew error.

  5. found a setting re: proxy that I do not understand, wonder if it has something to do with the problem (another image)

  6. there are way too many RC 301 entries in the Apache log files of the 2 websites that cannot renew certificates indicating that many other web pages that should have been working normally are also producing no output because of redirect to an invalid URL.

So, I don’t think I will be able to find the source of the problem because of lack of time to keep digging into it.

BTW there is an .htaccess file added into the .werll-known/acme-challenge folder … but it only appears to be intended to overlay any other .htaccess file that might have been leftover in this folder … it simply has an ‘allow all’ line.

one more possible clue, if you are interested.

there were 8 websites hosted by this virtualmin instance … all worked from 2024-Dec until about 2025-Nov-18 … then two had the ‘certificate fail to renew’ issue.

the websites that were deleted (late summer) to bring the total websites from 8 to 5 were lower in the listing of all websites than the 3 websites that continue to work properly, but earlier in the listing than the two websites that did fail .. one on Nov 18, the other on Nov 21.

I suspect the certificate renewal failure is directly related to RC 301 from the unexpected redirects to an incomplete URL (missing the FQDN portion) that were un-noticed until the certificate renewal issue was noticed.

Finally found the cause of the RC301 with the target URL stripped of the Domain name … it was caused by an .htaccess file in the webroot folder.

This is what caused the letsencrypt service to be unable to verify the hashed data string in the .well-know/acme-challenge/ folder and led to expired certificates that could not be renewed.

the BAD .htaccess contained the following 3 lines:
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www.(.)$ [NC]
RewriteRule ^(.
)$ https://%1/$1 [R=301,L]

the last line caused a 301 RC to most http requests … I deleted this file several times, and it kept coming back … then I discovered that someone else believed it was needed to cause http requests to be forced to https … and kept it in because of believing it was required to make https work properly .

I’m not sure where this .htaccess snippet came from, but suspect it was generated by one of the ‘AI’ sites in response to a question about how to migrate end users from http to https … even though there seems to be a much easier way to do this using ‘checkbox’ in virtualmin webconfig menus. I was puzzled about how it kept re-appearing after I deleted it … but now I believe I’ve got it fixed a bit more permanently now.