Anyone having problems with Let's Encrypt certbot and multiple SAN/domain renewals?

Here’s a fun error log from when I tried to use certbot to renew a certificate, all for domains hosted on the same box - although not all of them have web hosting (and some are mailserver subdomains so will never have web hosting).

It seems if you have a domain hosted but with no direct hosting, the integration isn’t sophisticated enough to use DNS challenge response in place of HTTP.

However what’s more confusing is that for some of the domains with hosting, the file was able to be created but Let’s Encrypt wasn’t able to view the resultant .well-known/acme-challenge file when checking.

Is there any reason certbot can’t use DNS-based challenge/response for bulk updates like this?

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for autoconfig.example1.com
http-01 challenge for autodiscover.example1.com
http-01 challenge for example2.coop
http-01 challenge for example1.com
http-01 challenge for mail.example2.coop
http-01 challenge for mail.example1.com
http-01 challenge for mail.example.co.uk
http-01 challenge for mail.example3.coop
http-01 challenge for mail.example4.
http-01 challenge for mail.example5.com
http-01 challenge for example.co.uk
http-01 challenge for pop.example2.coop
http-01 challenge for pop.example.co.uk
http-01 challenge for pop.example.com
http-01 challenge for pop.example3.coop
http-01 challenge for pop.example4.
http-01 challenge for pop.example5.com
http-01 challenge for example3.coop
http-01 challenge for example4.
http-01 challenge for example5.com
http-01 challenge for smtp.example2.coop
http-01 challenge for smtp.example1.com
http-01 challenge for smtp.example.co.uk
http-01 challenge for smtp.example.com
http-01 challenge for smtp.example3.coop
http-01 challenge for smtp.example4.
http-01 challenge for smtp.example5.com
http-01 challenge for www.example2.coop
http-01 challenge for www.example1.com
http-01 challenge for www.example.co.uk
http-01 challenge for www.example3.coop
http-01 challenge for www.example4.
http-01 challenge for www.example5.com
Using the webroot path /home/example-user1/public_html for all unmatched domains.
Waiting for verification...
Resetting dropped connection: acme-v02.api.letsencrypt.org
Challenge failed for domain example2.coop
Challenge failed for domain mail.example2.coop
Challenge failed for domain mail.example.co.uk
Challenge failed for domain mail.example3.coop
Challenge failed for domain mail.example4.
Challenge failed for domain mail.example5.com
Challenge failed for domain example.co.uk
Challenge failed for domain pop.example2.coop
Challenge failed for domain pop.example.co.uk
Challenge failed for domain pop.example.com
Challenge failed for domain pop.example5.com
Challenge failed for domain example3.coop
Challenge failed for domain example4.
Challenge failed for domain example5.com
Challenge failed for domain smtp.example2.coop
Challenge failed for domain smtp.example.co.uk
Challenge failed for domain smtp.example.com
Challenge failed for domain www.example2.coop
Challenge failed for domain www.example.co.uk
Challenge failed for domain www.example3.coop
Challenge failed for domain www.example4.
Challenge failed for domain www.example5.com
Challenge failed for domain pop.example4.
http-01 challenge for example2.coop
http-01 challenge for mail.example2.coop
http-01 challenge for mail.example.co.uk
http-01 challenge for mail.example3.coop
http-01 challenge for mail.example4.
http-01 challenge for mail.example5.com
http-01 challenge for example.co.uk
http-01 challenge for pop.example2.coop
http-01 challenge for pop.example.co.uk
http-01 challenge for pop.example.com
http-01 challenge for pop.example5.com
http-01 challenge for example3.coop
http-01 challenge for example4.
http-01 challenge for example5.com
http-01 challenge for smtp.example2.coop
http-01 challenge for smtp.example.co.uk
http-01 challenge for smtp.example.com
http-01 challenge for www.example2.coop
http-01 challenge for www.example.co.uk
http-01 challenge for www.example3.coop
http-01 challenge for www.example4.
http-01 challenge for www.example5.com
http-01 challenge for pop.example4.
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: example2.coop
   Type:   unauthorized
   Detail: Invalid response from
   http://example2.coop/.well-known/acme-challenge/xu2rLrYmgdXVRYJTjgP9O522xIPvvA8kvIkYH1dtm_8
   [176.126.240.161]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: mail.example2.coop
   Type:   unauthorized
   Detail: Invalid response from
   http://mail.example2.coop/.well-known/acme-challenge/wsyIBwU8yg9FD9xwZglkTgvaFYZtI8HPozHr9_yn4rA
   [176.126.240.161]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: mail.example3.coop
   Type:   unauthorized
   Detail: Invalid response from
   https://mail.example3.coop/.well-known/acme-challenge/LPyUYFDnoBiklPj6WNv97FEg5BHtIhLnGJL_99Bq79U
   [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: mail.example4.
   Type:   unauthorized
   Detail: Invalid response from
   http://mail.example4./.well-known/acme-challenge/ULffIJ6twC1nqnvDDQzEyj5-X5G6yd4eU2GCra7un4U
   [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: mail.example5.com
   Type:   unauthorized
   Detail: Invalid response from
   https://mail.example5.com/.well-known/acme-challenge/xXjK9QvKEAHIIlK-B7oFPCxM36MyWePZ7AU7XFHk4RE
   [dead:beef::1]: 404

   Domain: example.co.uk
   Type:   unauthorized
   Detail: Invalid response from
   http://example.co.uk/.well-known/acme-challenge/fGLiUHNOpsHnRZtiQi3bcWCPumBmn5ud5z-OoGoNpi4
   [2a07:7800::138]: "<html>\n<head>\n  <meta charset=\"UTF-8\">\n
   <meta name=\"viewport\" content=\"width=device-width,
   initial-scale=1.0\">\n  <meta http-equ"

   Domain: pop.example.co.uk
   Type:   unauthorized
   Detail: Invalid response from
   http://pop.example.co.uk/.well-known/acme-challenge/u0I6eBgE7KDiyABhEAX58foD9ek_GChnBfyHD1xYUWw
   [2a07:7800::138]: "<html>\n<head>\n  <meta charset=\"UTF-8\">\n
   <meta name=\"viewport\" content=\"width=device-width,
   initial-scale=1.0\">\n  <meta http-equ"

   Domain: example3.coop
   Type:   unauthorized
   Detail: Invalid response from
   https://example3.coop/.well-known/acme-challenge/tsTyVlZQ4-WzoKxu0hFdMBlDMK28XIW37taIj0uZaqQ
   [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: example4.
   Type:   unauthorized
   Detail: Invalid response from
   http://example4./.well-known/acme-challenge/beou57eEY_25cgF2lsiYglPXpIQTzx_B8Y2WlzQGbeY
   [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: example5.com
   Type:   unauthorized
   Detail: Invalid response from
   https://example5.com/.well-known/acme-challenge/hsAjQVD8P1vDkHz6mCirlxvBDSX0vfjzYqIJMWymMd4
   [dead:beef::1]: 404

   Domain: www.example2.coop
   Type:   unauthorized
   Detail: Invalid response from
   http://www.example2.coop/.well-known/acme-challenge/ObCMGxQMeddYJNunLvxunzt4s0gecaEBHJ2LudB91JE
   [176.126.240.161]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: www.example.co.uk
   Type:   unauthorized
   Detail: Invalid response from
   http://www.example.co.uk/.well-known/acme-challenge/DQ_yVNTQ2hl2wxs1mKbCxaY1cDkTmFLeENm-ji6NQQo
   [2a07:7800::138]: "<html>\n<head>\n  <meta charset=\"UTF-8\">\n
   <meta name=\"viewport\" content=\"width=device-width,
   initial-scale=1.0\">\n  <meta http-equ"

   Domain: www.example3.coop
   Type:   unauthorized
   Detail: Invalid response from
   https://www.example3.coop/.well-known/acme-challenge/4k0QfwYqecU5bZPUgCyqgCc5gBIUMXKwpmK3WLgmrns
   [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: www.example4.
   Type:   unauthorized
   Detail: Invalid response from
   http://www.example4./.well-known/acme-challenge/IRofBH6oXTykZwEJiITGUevlcuz5PxAjXJPvI5qZz4U
   [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: www.example5.com
   Type:   unauthorized
   Detail: Invalid response from
   https://www.example5.com/.well-known/acme-challenge/_e7Na766ip7jcvFmay8noKuSm4x9wF1Y1bmEViU_A4E
   [dead:beef::1]: 404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: mail.example.co.uk
   Type:   connection
   Detail: Fetching
   http://mail.example.co.uk/.well-known/acme-challenge/NifElUHOToKCp9aiakfyeWiPhiBDWzNBCT4Lx9iOV1c:
   Connection refused

   Domain: smtp.example.co.uk
   Type:   connection
   Detail: Fetching
   http://smtp.example.co.uk/.well-known/acme-challenge/D-lPSFk8zpH1E4oAjbtF0aXMSM48EM3BnYWj1ku8wVU:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: pop.example2.coop
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for pop.example2.coop
   - check that a DNS record exists for this domain

   Domain: pop.example.com
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for
   pop.example.com - check that a DNS record exists for this
   domain

   Domain: pop.example5.com
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for
   pop.example5.com - check that a DNS record exists for this
   domain

   Domain: smtp.example2.coop
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for smtp.example2.coop
   - check that a DNS record exists for this domain

   Domain: smtp.example.com
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for
   smtp.example.com - check that a DNS record exists for this
   domain

   Domain: pop.example4.
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for
   pop.example4. - check that a DNS record exists for this
   domain








IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mail.example.co.uk
   Type:   connection
   Detail: Fetching
   http://mail.example.co.uk/.well-known/acme-challenge/WKDrwIejE4x8PJvFxDvV3_oF5K9kDPYHrZ-_KMOAISw:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: example.co.uk
   Type:   unauthorized
   Detail: Invalid response from
   http://example.co.uk/.well-known/acme-challenge/2ag0ojrJv_6DqUeW4n9im1v6NY0nJV4fh3vNRi23I10
   [2a07:7800::138]: "<html>\n<head>\n  <meta charset=\"UTF-8\">\n
   <meta name=\"viewport\" content=\"width=device-width,
   initial-scale=1.0\">\n  <meta http-equ"

   Domain: www.example.co.uk
   Type:   unauthorized
   Detail: Invalid response from
   http://www.example.co.uk/.well-known/acme-challenge/qfHjrYD17OlbzOMjJ2Vz8vSDn4Z5XcoROi6qamZUL_k
   [2a07:7800::138]: "<html>\n<head>\n  <meta charset=\"UTF-8\">\n
   <meta name=\"viewport\" content=\"width=device-width,
   initial-scale=1.0\">\n  <meta http-equ"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.






DNS-based validation failed :
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for mail.example.co.uk
dns-01 challenge for example.co.uk
dns-01 challenge for www.example.co.uk
Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl
Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl
Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl
Waiting for verification...
Challenge failed for domain mail.example.co.uk
Challenge failed for domain example.co.uk
Challenge failed for domain www.example.co.uk
dns-01 challenge for mail.example.co.uk
dns-01 challenge for example.co.uk
dns-01 challenge for www.example.co.uk
Cleaning up challenges
Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl
Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl
Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl
Some challenges have failed.
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mail.example.co.uk
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up TXT for
   _acme-challenge.mail.example.co.uk - check that a DNS record
   exists for this domain
 - The following errors were reported by the server:

   Domain: example.co.uk
   Type:   unauthorized
   Detail: No TXT record found at _acme-challenge.example.co.uk

   Domain: www.example.co.uk
   Type:   unauthorized
   Detail: No TXT record found at
   _acme-challenge.www.example.co.uk

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
Re-loading Webmin ..
.. done