Here’s a fun error log from when I tried to use certbot to renew a certificate, all for domains hosted on the same box - although not all of them have web hosting (and some are mailserver subdomains so will never have web hosting).
It seems if you have a domain hosted but with no direct hosting, the integration isn’t sophisticated enough to use DNS challenge response in place of HTTP.
However what’s more confusing is that for some of the domains with hosting, the file was able to be created but Let’s Encrypt wasn’t able to view the resultant .well-known/acme-challenge file when checking.
Is there any reason certbot can’t use DNS-based challenge/response for bulk updates like this?
Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Renewing an existing certificate Performing the following challenges: http-01 challenge for autoconfig.example1.com http-01 challenge for autodiscover.example1.com http-01 challenge for example2.coop http-01 challenge for example1.com http-01 challenge for mail.example2.coop http-01 challenge for mail.example1.com http-01 challenge for mail.example.co.uk http-01 challenge for mail.example3.coop http-01 challenge for mail.example4. http-01 challenge for mail.example5.com http-01 challenge for example.co.uk http-01 challenge for pop.example2.coop http-01 challenge for pop.example.co.uk http-01 challenge for pop.example.com http-01 challenge for pop.example3.coop http-01 challenge for pop.example4. http-01 challenge for pop.example5.com http-01 challenge for example3.coop http-01 challenge for example4. http-01 challenge for example5.com http-01 challenge for smtp.example2.coop http-01 challenge for smtp.example1.com http-01 challenge for smtp.example.co.uk http-01 challenge for smtp.example.com http-01 challenge for smtp.example3.coop http-01 challenge for smtp.example4. http-01 challenge for smtp.example5.com http-01 challenge for www.example2.coop http-01 challenge for www.example1.com http-01 challenge for www.example.co.uk http-01 challenge for www.example3.coop http-01 challenge for www.example4. http-01 challenge for www.example5.com Using the webroot path /home/example-user1/public_html for all unmatched domains. Waiting for verification... Resetting dropped connection: acme-v02.api.letsencrypt.org Challenge failed for domain example2.coop Challenge failed for domain mail.example2.coop Challenge failed for domain mail.example.co.uk Challenge failed for domain mail.example3.coop Challenge failed for domain mail.example4. Challenge failed for domain mail.example5.com Challenge failed for domain example.co.uk Challenge failed for domain pop.example2.coop Challenge failed for domain pop.example.co.uk Challenge failed for domain pop.example.com Challenge failed for domain pop.example5.com Challenge failed for domain example3.coop Challenge failed for domain example4. Challenge failed for domain example5.com Challenge failed for domain smtp.example2.coop Challenge failed for domain smtp.example.co.uk Challenge failed for domain smtp.example.com Challenge failed for domain www.example2.coop Challenge failed for domain www.example.co.uk Challenge failed for domain www.example3.coop Challenge failed for domain www.example4. Challenge failed for domain www.example5.com Challenge failed for domain pop.example4. http-01 challenge for example2.coop http-01 challenge for mail.example2.coop http-01 challenge for mail.example.co.uk http-01 challenge for mail.example3.coop http-01 challenge for mail.example4. http-01 challenge for mail.example5.com http-01 challenge for example.co.uk http-01 challenge for pop.example2.coop http-01 challenge for pop.example.co.uk http-01 challenge for pop.example.com http-01 challenge for pop.example5.com http-01 challenge for example3.coop http-01 challenge for example4. http-01 challenge for example5.com http-01 challenge for smtp.example2.coop http-01 challenge for smtp.example.co.uk http-01 challenge for smtp.example.com http-01 challenge for www.example2.coop http-01 challenge for www.example.co.uk http-01 challenge for www.example3.coop http-01 challenge for www.example4. http-01 challenge for www.example5.com http-01 challenge for pop.example4. Cleaning up challenges Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server: Domain: example2.coop Type: unauthorized Detail: Invalid response from http://example2.coop/.well-known/acme-challenge/xu2rLrYmgdXVRYJTjgP9O522xIPvvA8kvIkYH1dtm_8 [176.126.240.161]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: mail.example2.coop Type: unauthorized Detail: Invalid response from http://mail.example2.coop/.well-known/acme-challenge/wsyIBwU8yg9FD9xwZglkTgvaFYZtI8HPozHr9_yn4rA [176.126.240.161]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: mail.example3.coop Type: unauthorized Detail: Invalid response from https://mail.example3.coop/.well-known/acme-challenge/LPyUYFDnoBiklPj6WNv97FEg5BHtIhLnGJL_99Bq79U [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: mail.example4. Type: unauthorized Detail: Invalid response from http://mail.example4./.well-known/acme-challenge/ULffIJ6twC1nqnvDDQzEyj5-X5G6yd4eU2GCra7un4U [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: mail.example5.com Type: unauthorized Detail: Invalid response from https://mail.example5.com/.well-known/acme-challenge/xXjK9QvKEAHIIlK-B7oFPCxM36MyWePZ7AU7XFHk4RE [dead:beef::1]: 404 Domain: example.co.uk Type: unauthorized Detail: Invalid response from http://example.co.uk/.well-known/acme-challenge/fGLiUHNOpsHnRZtiQi3bcWCPumBmn5ud5z-OoGoNpi4 [2a07:7800::138]: "<html>\n<head>\n <meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n <meta http-equ" Domain: pop.example.co.uk Type: unauthorized Detail: Invalid response from http://pop.example.co.uk/.well-known/acme-challenge/u0I6eBgE7KDiyABhEAX58foD9ek_GChnBfyHD1xYUWw [2a07:7800::138]: "<html>\n<head>\n <meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n <meta http-equ" Domain: example3.coop Type: unauthorized Detail: Invalid response from https://example3.coop/.well-known/acme-challenge/tsTyVlZQ4-WzoKxu0hFdMBlDMK28XIW37taIj0uZaqQ [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: example4. Type: unauthorized Detail: Invalid response from http://example4./.well-known/acme-challenge/beou57eEY_25cgF2lsiYglPXpIQTzx_B8Y2WlzQGbeY [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: example5.com Type: unauthorized Detail: Invalid response from https://example5.com/.well-known/acme-challenge/hsAjQVD8P1vDkHz6mCirlxvBDSX0vfjzYqIJMWymMd4 [dead:beef::1]: 404 Domain: www.example2.coop Type: unauthorized Detail: Invalid response from http://www.example2.coop/.well-known/acme-challenge/ObCMGxQMeddYJNunLvxunzt4s0gecaEBHJ2LudB91JE [176.126.240.161]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: www.example.co.uk Type: unauthorized Detail: Invalid response from http://www.example.co.uk/.well-known/acme-challenge/DQ_yVNTQ2hl2wxs1mKbCxaY1cDkTmFLeENm-ji6NQQo [2a07:7800::138]: "<html>\n<head>\n <meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n <meta http-equ" Domain: www.example3.coop Type: unauthorized Detail: Invalid response from https://www.example3.coop/.well-known/acme-challenge/4k0QfwYqecU5bZPUgCyqgCc5gBIUMXKwpmK3WLgmrns [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: www.example4. Type: unauthorized Detail: Invalid response from http://www.example4./.well-known/acme-challenge/IRofBH6oXTykZwEJiITGUevlcuz5PxAjXJPvI5qZz4U [dead:beef::1]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" Domain: www.example5.com Type: unauthorized Detail: Invalid response from https://www.example5.com/.well-known/acme-challenge/_e7Na766ip7jcvFmay8noKuSm4x9wF1Y1bmEViU_A4E [dead:beef::1]: 404 To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. - The following errors were reported by the server: Domain: mail.example.co.uk Type: connection Detail: Fetching http://mail.example.co.uk/.well-known/acme-challenge/NifElUHOToKCp9aiakfyeWiPhiBDWzNBCT4Lx9iOV1c: Connection refused Domain: smtp.example.co.uk Type: connection Detail: Fetching http://smtp.example.co.uk/.well-known/acme-challenge/D-lPSFk8zpH1E4oAjbtF0aXMSM48EM3BnYWj1ku8wVU: Connection refused To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided. - The following errors were reported by the server: Domain: pop.example2.coop Type: dns Detail: DNS problem: NXDOMAIN looking up A for pop.example2.coop - check that a DNS record exists for this domain Domain: pop.example.com Type: dns Detail: DNS problem: NXDOMAIN looking up A for pop.example.com - check that a DNS record exists for this domain Domain: pop.example5.com Type: dns Detail: DNS problem: NXDOMAIN looking up A for pop.example5.com - check that a DNS record exists for this domain Domain: smtp.example2.coop Type: dns Detail: DNS problem: NXDOMAIN looking up A for smtp.example2.coop - check that a DNS record exists for this domain Domain: smtp.example.com Type: dns Detail: DNS problem: NXDOMAIN looking up A for smtp.example.com - check that a DNS record exists for this domain Domain: pop.example4. Type: dns Detail: DNS problem: NXDOMAIN looking up A for pop.example4. - check that a DNS record exists for this domain IMPORTANT NOTES: - The following errors were reported by the server: Domain: mail.example.co.uk Type: connection Detail: Fetching http://mail.example.co.uk/.well-known/acme-challenge/WKDrwIejE4x8PJvFxDvV3_oF5K9kDPYHrZ-_KMOAISw: Connection refused To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided. - The following errors were reported by the server: Domain: example.co.uk Type: unauthorized Detail: Invalid response from http://example.co.uk/.well-known/acme-challenge/2ag0ojrJv_6DqUeW4n9im1v6NY0nJV4fh3vNRi23I10 [2a07:7800::138]: "<html>\n<head>\n <meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n <meta http-equ" Domain: www.example.co.uk Type: unauthorized Detail: Invalid response from http://www.example.co.uk/.well-known/acme-challenge/qfHjrYD17OlbzOMjJ2Vz8vSDn4Z5XcoROi6qamZUL_k [2a07:7800::138]: "<html>\n<head>\n <meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n <meta http-equ" To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. DNS-based validation failed : Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator manual, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Renewing an existing certificate Performing the following challenges: dns-01 challenge for mail.example.co.uk dns-01 challenge for example.co.uk dns-01 challenge for www.example.co.uk Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Waiting for verification... Challenge failed for domain mail.example.co.uk Challenge failed for domain example.co.uk Challenge failed for domain www.example.co.uk dns-01 challenge for mail.example.co.uk dns-01 challenge for example.co.uk dns-01 challenge for www.example.co.uk Cleaning up challenges Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server: Domain: mail.example.co.uk Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mail.example.co.uk - check that a DNS record exists for this domain - The following errors were reported by the server: Domain: example.co.uk Type: unauthorized Detail: No TXT record found at _acme-challenge.example.co.uk Domain: www.example.co.uk Type: unauthorized Detail: No TXT record found at _acme-challenge.www.example.co.uk To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Re-loading Webmin .. .. done