Let's Encrypt broken, Debian 10, Webmin 1.999

[kitserve]

SYSTEM INFORMATION
OS type and version	Debian 10
Webmin version	1.999
Virtualmin version	7.1-1
Related packages	certbot 0.31.0-1+deb10u1

This server was originally set up with a much older version of Debian (which predated Let’s Encrypt). It’s been upgraded in place over the years following best practice (i.e. the official Debian guides) to Debian 10. Webmin, Virtualmin and Usermin were all installed from the official apt repos and have also been regularly upgraded in place. All the software on the server is currently up to date, and there aren’t any external repos other than Webmin and Virtualmin.

When Let’s Encrypt came out and Virtualmin started supporting it, we installed Let’s Encrypt via Virtualmin, which put it in /opt/eff.org/. This was prior to Debian supporting Let’s Encrypt in the repos. At some point over the last couple of years, we installed the Debian packaged version of certbot. This did cause some slightly confusing configuration overlap, but until now I’ve been able to fix any problems that came up.

Over the last couple of days we’ve started getting warnings like this:

An error occurred requesting a new certificate for (redacted) from Let's Encrypt : Web-based validation failed : Your system is not supported by certbot-auto anymore.
certbot-auto and its Certbot installation will no longer receive updates.
You will not receive any bug fixes including those fixing server compatibility
or security problems.
Please visit https://certbot.eff.org/ to check for other alternatives.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: (redacted), retry after 2022-08-16T16:15:08Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/
Please see the logfiles in /var/log/letsencrypt for more details.
DNS-based validation failed : Your system is not supported by certbot-auto anymore.
certbot-auto and its Certbot installation will no longer receive updates.
You will not receive any bug fixes including those fixing server compatibility
or security problems.
Please visit https://certbot.eff.org/ to check for other alternatives.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Renewing an existing certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: (redacted), retry after 2022-08-16T16:15:08Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/
Please see the logfiles in /var/log/letsencrypt for more details.

I removed /opt/eff.org/ in the hopes of resolving any potential confusion between different versions of certbot. Unfortunately some parts of Virtualmin still seem to be stuck thinking that they should call certbot-auto. For older sites on the server, when I go Server Configuration → SSL Certificate → Let’s Encrypt, I get this message at the bottom:

Renewal failed due to
Web-based validation failed :
Skipping bootstrap because certbot-auto is deprecated on this system.

If I click on Request Certificate, I get this message:

Requesting a certificate for (redacted) from Let's Encrypt ..
.. request failed : Web-based validation failed :
Skipping bootstrap because certbot-auto is deprecated on this system.
Your system is not supported by certbot-auto anymore.
Certbot cannot be installed.
Please visit https://certbot.eff.org/ to check for other alternatives.
   DNS-based validation failed :
Skipping bootstrap because certbot-auto is deprecated on this system.
Your system is not supported by certbot-auto anymore.
Certbot cannot be installed.
Please visit https://certbot.eff.org/ to check for other alternatives.

However, if I click on the same options for a newer site, the Let’s Encrypt section looks fine, no errors. From this I’m guessing that Virtualmin’s internal config for the older sites somehow has certbot-auto embedded in it, while newer sites were successfully set up using the system certbot.

How do I get Virtualmin to forget about certbot-auto and automatically use the system version of certbot for all sites, new and old? I’ve tried disabling and re-enabling SSL for one of the older sites, but Let’s Encrypt is still broken. I’m able to manually run certbot from the command line and generate certificates that way, but of course it doesn’t hook into Virtualmin, and I also keep receiving emails every day from the server reminding me that the certificate can’t be renewed. I’ve got a pretty solid understanding of Linux server admin and I can sort out this stuff out manually when it’s not managed by Virtualmin, but it’s a mystery to me where Virtualmin is getting its information. Without that I’m pretty stuck resolving the problem…

[kitserve]

As an aside: how do I configure where the warning emails go? I can’t find any settings related to them either. If I could send them to a different address while fixing this problem it would help avoid them burying other emails.

[pcmerc]

Setup a cron job that runs daily to renew certs.
Update the paths for the SSL certs in the /etc/webmin/miniserv.conf file
The vhost domains are prefixed with ipkey_ & ipcert_ which symlink to the certbot certificate paths /etc/letsencrypt/live/vhost.domain.com/fullchain.pem & /etc/letsencrypt/live/vhost.domain.com/privkey.pem.
I will also symlink the items in /etc/ssl/virtualmin/vhostID/certs* etc
I also update the /etc/postfix/sni_map & regenerate that with cd /etc/postfix && postmap -F sni_map
Recycle services using certificates in question

[kitserve]

Thanks for taking the time to reply. I’ve already got the standard system certbot cron job set up, and it’s working. The certificates for Webmin and other system services are fine, it’s just some of the virtual hosts that are broken. I checked in /etc/webmin/miniserv.conf and couldn’t see any certificate configuration that needed changing. I also couldn’t see any ipkey_ or ipcert_ symlinks on the system, despite running updatedb followed by locate ipkey_ and locate ipcert_. There is no directory /etc/ssl/virtualmin/ on this server. The postfix SNI map files are there, but they’re not so important, the real issue is Webmin not being able to use the system certbot for existing domains, particularly for Apache.

[pcmerc]

That makes sense after reading your initial post this system was setup initially on an older version of virtualmin.
For my clients what I usually do when OS’s change major revs, a new instance is built along side the current, newer OS/virtualmin etc. All current vhosts are then exported/backed-up & imported into the new system using virtualmin’s backup/restore. Each hosts possible kinks are ironed out & any data is then synced as needed up to the point of cut over.
I do not support OS upgrades. OS installers are not that intelligent.

[Ilia]

@kitserve

I’ve already got the standard system certbot cron job set up, and it’s working.

You shouldn’t setup any cron jobs for certbot manually and let Virtualmin do it for you!

[pcmerc]

I disagree. Improvise & do whatever you need to for the systems you manage.

[kitserve]

@pcmerc As I said in the original post, it’s Debian 10, but started out as (I think) Debian 7 and was upgraded in place several times over the years. It was initially set up with whatever version of Virtualmin was current at the time, using the official GPL apt repos supplied by the Webmin team.

@Ilia The currently active certbot cron job is the one that came with the system certbot package, I haven’t modified it. No idea how Virtualmin handles conflicts between system certbot and its own way of doing things, but since I understand that system certbot is currently recommended when it’s available, I’d expect it to use the cron job that comes with that rather than setting up another one.

[kitserve]

In case it helps anyone else, I managed to track down some more information related to this. First, file /etc/webmin/webmin/config contained an entry related to Let’s Encrypt on my system. I replaced line letsencrypt_cmd=/usr/local/share/letsencrypt/letsencrypt-auto with letsencrypt_cmd=/usr/bin/certbot and restarted Webmin. Second, I moved /usr/local/share/letsencrypt/letsencrypt-auto to /usr/local/share/letsencrypt/letsencrypt-auto.bak and then created a symlink from /usr/local/share/letsencrypt/letsencrypt-auto to /usr/bin/certbot. After that I was able to reissue certificates for the old domains. It feels like a bit of a messy fix since there’s still an extra copy of letsencrypt-auto sitting on the server, but at least it’s working again now and seemingly correctly using the system copy of certbot.

[Ilia]

After that I was able to reissue certificates for the old domains.

Thank you for providing a feedback. Although, Virtualmin system is not setup to use letsencrypt-auto command by default.

[kitserve]

That’s interesting, because /usr/share/webmin/webmin/letsencrypt-lib.pl seems to check for letsencrypt-auto as its first choice. Looking at the timestamps, letsencrypt-auto was added to the server back in early 2016. I don’t remember how it got there but I suppose I may have added it manually.