LetsEncrypt has stopped renewing certificates

dumorian · October 12, 2024, 4:45pm

SYSTEM INFORMATION
OS type and version	Rocky Linux 8.10
Usermin version	2.102
Virtualmin version	7.20.2
Theme version	21.20.7
Apache version	2.4.37
Package updates	All installed packages are up to date

LetsEncrypt has stopped renewing certs for all domains on this system. This system acts as a public mailserver.

As a test, I have set up one of my personal domains on this system as an example. LetsEncrypt does not succeed on new domains either.

Here are the A records in DNS.
mail A 208.67.39.33
webmail A 208.67.39.33

Here are the redirects created during the creation via Virtualmin:

Here is the saved request info for the domain:

Here is the link to a file I placed in a .well-known directory.
https://mail.johnhintonwoodturner.com/.well-known/test.html

Here is the contents from the /var/log/letsencrypt/letsencrypt.log
2024-10-12 12:52:17,799:DEBUG:certbot._internal.main:certbot version: 1.22.0
2024-10-12 12:52:17,801:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/letsencrypt
2024-10-12 12:52:17,801:DEBUG:certbot._internal.main:Arguments: [‘-a’, ‘webroot’, ‘-d’, ‘mail.johnhintonwoodturner.com’, ‘-d’, ‘webmail.johnhintonwoodturner.com’, ‘–webroot-path’, ‘/home/johnhintonwoodturner/public_html’, ‘–duplicate’, ‘–force-renewal’, ‘–non-interactive’, ‘–agree-tos’, ‘–config’, ‘/tmp/.webmin/213920_1025594_2_letsencrypt.cgi’, ‘–rsa-key-size’, ‘2048’, ‘–cert-name’, ‘mail.johnhintonwoodturner.com’]
2024-10-12 12:52:17,803:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2024-10-12 12:52:17,884:DEBUG:certbot._internal.log:Root logging level set at 30
2024-10-12 12:52:17,887:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2024-10-12 12:52:17,894:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: Authenticator, Plugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7fbe77b3ad30>
Prep: True
2024-10-12 12:52:17,895:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7fbe77b3ad30> and installer None
2024-10-12 12:52:17,895:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2024-10-12 12:52:17,921:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri=‘https://acme-v02.api.letsencrypt.org/acme/acct/96937123’, new_authzr_uri=None, terms_of_service=None), 3fcd6fbbb302cd9297c891639737c80d, Meta(creation_dt=datetime.datetime(2020, 9, 17, 21, 49, 31, tzinfo=), creation_host=‘mars.ew3d.com’, register_to_eff=None))>
2024-10-12 12:52:17,924:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2024-10-12 12:52:17,926:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2024-10-12 12:52:18,062:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 601, in urlopen
chunked=chunked)
File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 344, in _make_request
self._validate_conn(conn)
File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 844, in validate_conn
conn.connect()
File “/usr/lib/python3.6/site-packages/urllib3/connection.py”, line 358, in connect
ssl_context=context)
File "/usr/lib/python3.6/site-packages/urllib3/util/ssl.py", line 354, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File “/usr/lib64/python3.6/ssl.py”, line 365, in wrap_socket
_context=self, _session=session)
File “/usr/lib64/python3.6/ssl.py”, line 810, in init
self.do_handshake()
File “/usr/lib64/python3.6/ssl.py”, line 1070, in do_handshake
self._sslobj.do_handshake()
File “/usr/lib64/python3.6/ssl.py”, line 648, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File “/usr/lib/python3.6/site-packages/urllib3/util/retry.py”, line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory(Caused by SSLError(SSLError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)’),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/bin/letsencrypt”, line 11, in
load_entry_point(‘certbot==1.22.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3.6/site-packages/certbot/main.py”, line 19, in main
return internal_main.main(cli_args)
File “/usr/lib/python3.6/site-packages/certbot/_internal/main.py”, line 1632, in main
return config.func(config, plugins)
File “/usr/lib/python3.6/site-packages/certbot/_internal/main.py”, line 1473, in certonly
le_client = _init_le_client(config, auth, installer)
File “/usr/lib/python3.6/site-packages/certbot/_internal/main.py”, line 793, in _init_le_client
return client.Client(config, acc, authenticator, installer, acme=acme)
File “/usr/lib/python3.6/site-packages/certbot/_internal/client.py”, line 294, in init
acme = acme_from_config_key(config, self.account.key, self.account.regr)
File “/usr/lib/python3.6/site-packages/certbot/_internal/client.py”, line 59, in acme_from_config_key
client = acme_client.BackwardsCompatibleClientV2(net, key, config.server)
File “/usr/lib/python3.6/site-packages/acme/client.py”, line 875, in init
directory = messages.Directory.from_json(net.get(server).json())
File “/usr/lib/python3.6/site-packages/acme/client.py”, line 1236, in get
self._send_request(‘GET’, url, **kwargs), content_type=content_type)
File “/usr/lib/python3.6/site-packages/acme/client.py”, line 1174, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/usr/lib/python3.6/site-packages/requests/sessions.py”, line 535, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python3.6/site-packages/requests/sessions.py”, line 648, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python3.6/site-packages/requests/adapters.py”, line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)’),))
2024-10-12 12:52:18,067:ERROR:certbot._internal.log:An unexpected error occurred:
2024-10-12 12:52:18,067:ERROR:certbot._internal.log:requests.exceptions.SSLError: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)’),))

Hostname is mars.ew3d.com
Postfix $mydestination includes mars.ew3d.com

cyberndt · October 12, 2024, 5:06pm

I’ve done the honors for you and deleted all my post @Joe

dumorian · October 12, 2024, 6:16pm

Here is the apache config for that domain.

<VirtualHost 208.67.39.33:80>
ServerName johnhintonwoodturner.com
ServerAlias webmail.johnhintonwoodturner.com
ServerAlias mail.johnhintonwoodturner.com
ServerAlias autoconfig.johnhintonwoodturner.com
ServerAlias autodiscover.johnhintonwoodturner.com
DocumentRoot /home/johnhintonwoodturner/public_html
ErrorLog /var/log/virtualmin/johnhintonwoodturner.com_error_log
CustomLog /var/log/virtualmin/johnhintonwoodturner.com_access_log combined
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/johnhintonwoodturner/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted

<Directory /home/johnhintonwoodturner/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted

ProxyPass /.well-known !
RemoveHandler .php
RemoveHandler .php7.2
RemoveHandler .php8.1
<FilesMatch .php$>
SetHandler proxy:fcgi://127.0.0.1:8004

Redirect /mail/config-v1.1.xml /cgi-bin/autoconfig.cgi
Redirect /.well-known/autoconfig/mail/config-v1.1.xml /cgi-bin/autoconfig.cgi
ScriptAlias /AutoDiscover/AutoDiscover.xml /home/johnhintonwoodturner/cgi-bin/autoconfig.cgi
ScriptAlias /Autodiscover/Autodiscover.xml /home/johnhintonwoodturner/cgi-bin/autoconfig.cgi
ScriptAlias /autodiscover/autodiscover.xml /home/johnhintonwoodturner/cgi-bin/autoconfig.cgi
RewriteEngine on
RewriteCond %{HTTPS} off
RewriteRule ^/(?!.well-known)(.*)$ https://%{HTTP_HOST}/$1 [R]

<VirtualHost 208.67.39.33:443>
ServerName johnhintonwoodturner.com
ServerAlias webmail.johnhintonwoodturner.com
ServerAlias mail.johnhintonwoodturner.com
ServerAlias autoconfig.johnhintonwoodturner.com
ServerAlias autodiscover.johnhintonwoodturner.com
DocumentRoot /home/johnhintonwoodturner/public_html
ErrorLog /var/log/virtualmin/johnhintonwoodturner.com_error_log
CustomLog /var/log/virtualmin/johnhintonwoodturner.com_access_log combined
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/johnhintonwoodturner/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted

<Directory /home/johnhintonwoodturner/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted

ProxyPass /.well-known !
RemoveHandler .php
RemoveHandler .php7.2
RemoveHandler .php8.1
<FilesMatch .php$>
SetHandler proxy:fcgi://127.0.0.1:8004

Redirect /mail/config-v1.1.xml /cgi-bin/autoconfig.cgi
Redirect /.well-known/autoconfig/mail/config-v1.1.xml /cgi-bin/autoconfig.cgi
ScriptAlias /AutoDiscover/AutoDiscover.xml /home/johnhintonwoodturner/cgi-bin/autoconfig.cgi
ScriptAlias /Autodiscover/Autodiscover.xml /home/johnhintonwoodturner/cgi-bin/autoconfig.cgi
ScriptAlias /autodiscover/autodiscover.xml /home/johnhintonwoodturner/cgi-bin/autoconfig.cgi
SSLEngine on
SSLCertificateFile /home/johnhintonwoodturner/ssl.combined
SSLCertificateKeyFile /home/johnhintonwoodturner/ssl.key
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

When turning on SSL in Virtualmin, it failed to create the LetsEncrypt cert and fell back to creating a self signed cert. I can access that via the link given in the first post, using Firefox and a SSL exception for the self signed cert.

In the LetsEncrypt setup, I have used the ‘Only Update Renewal’ button to save my config before requesting a cert since it resets to default if using only the Request Certificate button.

LetsEncrypt worked perfectly on this system for some years. It stopped working a few weeks ago. At that time, I went away from using a wildcard Sectigo cert for the services run under ew3d.com. I was fighting with trying to create a cert for the hostname along with some others needed for the main domain. But, it has killed the renewals for all domains on this system

Thanks for the reply!.

cyberndt · October 12, 2024, 6:21pm

I’ve done the honors for you and deleted all my post @Joe

dumorian · October 12, 2024, 7:01pm

Thanks Cyberndt!

I did not know the generate LetsEncrypt cert option for hostname was available from the SSL template.

I turned it on and still it fails. Of note, the logs include
2024-10-12 14:37:36,152:DEBUG:certbot._internal.main:Arguments: [‘-a’, ‘webroot’, ‘-d’, ‘mars.ew3d.com’, ‘–webroot-path’, ‘/home/._default_hostname/public_html’, ‘–duplicate’, ‘–force-renewal’, ‘–non-interactive’, ‘–agree-tos’, ‘–config’, ‘/tmp/.webmin/153413_1077918_11_check.cgi’, ‘–rsa-key-size’, ‘2048’, ‘–cert-name’, ‘mars.ew3d.com’]
There is no /home/._default_hostname directory. Should this actually have this name or should it be labelled with the actual hostname?

LetsEncrypt logs are a bit lacking in detail.

I am unabled to renew nor generate a cert from a domain on this system with a direct login via Webmin. It is showing “Checking hostnames for resolvability …
… all hostnames can be resolved”

cyberndt · October 12, 2024, 7:03pm

I’ve done the honors for you and deleted all my post @Joe

dumorian · October 12, 2024, 7:07pm

The self signed cert succeeded.

Sorry. Was editing the last post when you responded. It shows the answer. I cannot generate nor renew a cert when logging in to Webmin/Virtualmin directly from a domain without a LetsEncrypt cert nor a domain due for renewal on that system.

cyberndt · October 12, 2024, 7:10pm

I’ve done the honors for you and deleted all my post @Joe

dumorian · October 12, 2024, 7:20pm

It does. Same errors. FYI, I created the self signed cert from the Webmin SSL configuration interface. I can login to the system using Firefox to override the self signed security issue.

cyberndt · October 12, 2024, 7:21pm

I’ve done the honors for you and deleted all my post @Joe

dumorian · October 12, 2024, 7:39pm

LOL! That’s where I have been driven. Nuts!

This system originally had a virtualhost for the hostname. It used LetsEncrpyt just perfectly. I deleted that and when trying to recreate it, found Virtualmin would no longer allow that. And I’m still stuck 3 weeks later.

I do not have a disk storage issue. Email is all working great, postfix and dovecot. Webmail is functioning for all the sites just fine. Only LetsEncrypt stopped. I suspect as I work with the rest of my systems the same problem will happen as they all have a virtualhost for the hostname.

In the Virtualmin SSL config, I tried both switches, Yes, and Yes and have it configurable. Both failed at the end of recheck configuration.

dumorian · October 12, 2024, 7:46pm

Yeah, I have done that for a few domains just pointing them to a different server here to generate the cert, and moving it back.

I’m not sure where a cert for hostname should live. It seems LetsEncrypt is looking for a /home/._default_hostname/public_html. I’m not sure if this is the real path or if hostname is replaced with the real hostname listed in Certbot’s configs. Either way, there are no dot files nor dot directories in my /home directory.

cyberndt · October 12, 2024, 7:51pm

I’ve done the honors for you and deleted all my post @Joe

dumorian · October 12, 2024, 8:21pm

No /home/. directories exist. I wonder if these options should still exist? Joe says no SSL cert is needed for the hostname. All I know is when I got rid of it LetsEncrypt blew up.

cyberndt · October 12, 2024, 8:49pm

I’ve done the honors for you and deleted all my post @Joe

Joe · October 12, 2024, 9:50pm

It does not! It 100% absolutely does not need a TLS certificate to connect to Let’s Encrypt servers.

Joe · October 12, 2024, 9:56pm

I’ve told you how to troubleshoot this in the other thread.

If you’ve been blocked for too many failures (which the errors you’ve posted indicate that is so), you need to wait a while (disable auto-renewal, since something is broken and it can’t succeed and it might keep you blocked). Then you need to get an error other than “Max retries exceeded”.

Or, you can do what I told you in the other thread:

Make sure DNS records exist for every name you are requesting a cert for. Don’t request certs for names that don’t resolve. (I covered this in more detail in the other thread, I recommend you review that.)
Make sure requests are making it to the filesystem for the .well-known path. Put a file in /home/domainname/public_html/.well-known/somefilename and try to download it. If you cannot download it, Let’s Encrypt will not be able to download its validation file, and it will fail. You need to fix that. Redirects or proxy rules are probably to blame.

Please do these things, so we can resolve this. It is not a hard problem. It should not take two topics and dozens of comments to figure it out.

Edit: Here’s that other comment in the other thread with a bit more detail about the common problems and their solutions: Trouble dealing with hostname and SSL certs - #16 by Joe

Tactikast · October 13, 2024, 12:46am

I copy paste what I said in the other topic (And you can follow the links I provided for wildcard with Let’s Encrypt):

If I understood, @dumorian doesn’t ask a certificate for “mars.ew3d.com". He ask it for “mail3.ew3d.com” or even “mail.johnhintonwoodturner.com” I didn’t see him anywhere asking it for “mars.ew3d.com” (Because according to me, he expect the redirection to be carried out by let’s encrypt. Because it’s the first time he use it, before he had a wildcard certificate with sectigo. But it doesn’t work the same way with a Sectigo wildcard certificate and a Let’s Encrypt certificate)

The wildcard from Sectigo carry out everything: *.ew3d.com (For beginner: the “*” means everything)`

While Let’s Ecrypt (In our situation) only carry: mail3.ew3d.com (You need to add every single subdomain)

According to me It was probably working before due to the SECTIGO WILDCARD you had.

So you could had everythingyouwant.ew3d.com because of the wildcard carrying out everything for you.

But know you are with let’s encrypt, free certificate. So the wildcard no longer apply. So you don’t have any certificate covering mars.ew3d.com because you never had any (it was the wildcard which was taking care of it). But you probably had a redirection setup somewhere (perhaps not even) in your server concerning the mars.ew3d.com url.

Still you need to ask directly a certificate to LE for this specific address. None of the redirection or previous rules you was using before can apply because it was the Sectigo Certificate which was (indirectly) allowing it.

So then as cyberndt said, you can also ask a wildcard certificate with let’s encrypt; But the setup, to get it, is different (ie: check the links I provided in the other topic).

Lastly as Joe and Stegan. None of the setup you did server side (neither the hostname) are used by Let’s Encrypt. It simply doesn’t work this way. You can not get SSL certificate for hostname; You can only get a certificate for an “A Record” clearly setup in your DNS provider.

Sorry if I said a mistake but I’m quite sure I’m right.

dumorian · October 13, 2024, 2:05pm

At the top of this post, I show my A records for this domain. Also, I created the .well-known path, uploaded a file which simply says hello. It all works. I also show the redirects from the Virtualmin interface. I do not see any problems in these areas.

I have also checked all three of my nameservers to verify the A records are correct and were transferred correctly.

dumorian · October 13, 2024, 2:09pm

As for the max retries, this happens after several seconds during the failure. At the moment, there is only one domain trying to renew its cert. It is all handled via Virtualmin. I assume the max retries is simply the failures being stopped after maybe 10 seconds.