Using Cloudflare to complete Let's Encrypt DNS challenge?

ftoemboel · January 4, 2020, 12:16am

Hello,
I do not know whether it is possible at the moment; at least I was not able to find the following functionality:
When generating an SSL cert using certbot via the command line, it is possible to complete the DNS-01 challenge with Cloudflare like so:
certbot certonly --dns-cloudflare --dns-cloudflare-credentials API-Key -d example.com,*.example.com
Is it possible to do that automatically in Virtualmin instead of the HTTP-01 challenge?
Thanks a lot,
fred

Ilia · January 4, 2020, 10:23am

Hi, Fred!

There is no direct support in Virtualmin for Cloudflare, so far. I was thinking of a work arounds and here is what I got for you.

Create custom certbot-cf.sh (chmod +x) script under /usr/local/bin (should be on the PATH and usually is)
Add this script to Webmin module configuration:

image2880×882 237 KB

Add to the custom script file:

certbot --dns-cloudflare --dns-cloudflare-credentials API-Key

Tell us if it works. In case it doesn’t, please provide the exact command that works for you to get LE SSL certificate, while using Cloudflare, and I talk to Jamie about enrolling it.

It seems very easy to do.

ftoemboel · January 4, 2020, 1:51pm

Hi, Ilia!
Thank you for your reply, I too think it should be easy to integrate Cloudflare DNS (at least into LE SSL validation).
I now get this error:

The exact command that works for me is:

certbot certonly --dns-cloudflare --dns-cloudflare-credentials PATH_TO_FILE_CONTAINING_CREDENTIALS -d *.domain.tld

The referenced file contains the following information:

Cloudflare API credentials used by Certbot

dns_cloudflare_email = cloudflare@example.com
dns_cloudflare_api_key = 0123456789abcdef0123456789abcdef01234567

I got the info from here: Welcome to certbot-dns-cloudflare’s documentation! — certbot-dns-cloudflare 0 documentation

Best, fred

ftoemboel · January 4, 2020, 2:07pm

This is what I get using the proposed work-around when not requesting a wildcard cert:

This is the log file:

2020-01-04 14:55:50,081:DEBUG:certbot.main:certbot version: 0.27.0
2020-01-04 14:55:50,082:DEBUG:certbot.main:Arguments: [‘–dns-cloudflare’, ‘–dns-cloudflare-credentials’, ‘PATH_TO_CREDENTIALS_FILE’]
2020-01-04 14:55:50,082:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-01-04 14:55:50,090:DEBUG:certbot.log:Root logging level set at 20
2020-01-04 14:55:50,091:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-01-04 14:55:50,092:DEBUG:certbot.plugins.selection:Requested authenticator dns-cloudflare and installer None
2020-01-04 14:55:50,092:DEBUG:certbot.plugins.selection:Single candidate plugin: * dns-cloudflare
Description: Obtain certificates using a DNS TXT record (if you are using Cloudflare for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-cloudflare = certbot_dns_cloudflare.dns_cloudflare:Authenticator
Initialized: <certbot_dns_cloudflare.dns_cloudflare.Authenticator object at 0x7f314f751be0>
Prep: True
2020-01-04 14:55:50,093:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_dns_cloudflare.dns_cloudflare.Authenticator object at 0x7f314f751be0> and installer None
2020-01-04 14:55:50,093:INFO:certbot.plugins.selection:Plugins selected: Authenticator dns-cloudflare, Installer None
2020-01-04 14:55:50,115:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri=‘https://acme-v02.api.letsencrypt.org/acme/acct/75087857’, new_authzr_uri=None, terms_of_service=None), 2615a565d5a2d4e815382320219b15cf, Meta(creation_dt=datetime.datetime(2020, 1, 3, 18, 22, 25, tzinfo=), creation_host=‘example.domain.tld’))>
2020-01-04 14:55:50,116:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2020-01-04 14:55:50,118:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2020-01-04 14:55:50,646:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 “GET /directory HTTP/1.1” 200 658
2020-01-04 14:55:50,647:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 04 Jan 2020 13:55:50 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“eXkOJGoCPEg”: “Adding random entries to the directory - API Announcements - Let's Encrypt Community Support”,
“keyChange”: “https://acme-v02.api.letsencrypt.org/acme/key-change”,
“meta”: {
“caaIdentities”: [
“letsencrypt.org”
],
“termsOfService”: “https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf”,
“website”: “https://letsencrypt.org”
},
“newAccount”: “https://acme-v02.api.letsencrypt.org/acme/new-acct”,
“newNonce”: “https://acme-v02.api.letsencrypt.org/acme/new-nonce”,
“newOrder”: “https://acme-v02.api.letsencrypt.org/acme/new-order”,
“revokeCert”: “https://acme-v02.api.letsencrypt.org/acme/revoke-cert”
}
2020-01-04 14:55:50,648:DEBUG:certbot.display.ops:No installer, picking names manually
2020-01-04 14:55:50,655:INFO:certbot.main:Obtaining a new certificate
2020-01-04 14:55:50,730:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0007_key-certbot.pem
2020-01-04 14:55:50,732:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0007_csr-certbot.pem
2020-01-04 14:55:50,733:DEBUG:acme.client:Requesting fresh nonce
2020-01-04 14:55:50,733:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2020-01-04 14:55:50,864:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 “HEAD /acme/new-nonce HTTP/1.1” 200 0
2020-01-04 14:55:50,866:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 04 Jan 2020 13:55:50 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0102ygXcEclVLBxECyFrhVp61hSj2xbRx3sqGgAKY4hc1is
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

2020-01-04 14:55:50,866:DEBUG:acme.client:Storing nonce: 0102ygXcEclVLBxECyFrhVp61hSj2xbRx3sqGgAKY4hc1is
2020-01-04 14:55:50,866:DEBUG:acme.client:JWS payload:
b’{\n “identifiers”: [\n {\n “type”: “dns”,\n “value”: “a”\n }\n ]\n}’
2020-01-04 14:55:50,871:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
“protected”: “eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNzUwODc4NTciLCAibm9uY2UiOiAiMDEwMnlnWGNFY2xWTEJ4RUN5RnJoVnA2MWhTajJ4YlJ4M3NxR2dBS1k0aGMxaXMiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciJ9”,
“signature”: “LrOCC2pBJKfgGi_1iibPP7OPyo9hzyA_nux5m5oFds2oeVALLo7gQ5g7QSMwHSkdBy12fzsi5JOBgIC7Me0ElNP0pFBAqC5HU839EFpJGIXT7ChWUCkLmsCY7TNcm8YOMMTp6xH6BrNo33RX47MY4qI7_NtQLkJt5sjhshORrTNgavhb6oTvwPwmYwz6X701Wywszp20GslSI32gDHeLbWOwUgDvn8r8fCvNnvhuAZrehIXrgkGx_Inf8u3e6F3P-EGv4dViZ5gdb2JJfku3vulcezKpiWMRRlHQjjCbsTsNe6vZz7yqFUTChGUOefmyRS0Zx5lq8zuk3yNJKQVafA”,
“payload”: “ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImEiCiAgICB9CiAgXQp9”
}
2020-01-04 14:55:51,017:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 “POST /acme/new-order HTTP/1.1” 400 180
2020-01-04 14:55:51,017:DEBUG:acme.client:Received response:
HTTP 400
Server: nginx
Date: Sat, 04 Jan 2020 13:55:50 GMT
Content-Type: application/problem+json
Content-Length: 180
Connection: keep-alive
Boulder-Requester: 75087857
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 01025v91jD9NiQBHp3SBHD98aoV3Mdhp4Wgs3zanYgdwtvE

{
“type”: “urn:ietf:params:acme:error:rejectedIdentifier”,
“detail”: “Error creating new order :: Cannot issue for "a": Domain name needs at least one dot”,
“status”: 400
}
2020-01-04 14:55:51,018:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.27.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1364, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1254, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 120, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 391, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 334, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 366, in _get_order_and_authorizations
orderr = self.acme.new_order(csr_pem)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 889, in new_order
return self.client.new_order(csr_pem)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 672, in new_order
response = self._post(self.directory[‘newOrder’], order)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 96, in _post
return self.net.post(*args, **kwargs)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1204, in post
return self._post_once(*args, **kwargs)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1218, in _post_once
response = self._check_response(response, content_type=content_type)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1073, in _check_response
raise messages.Error.from_json(jobj)
acme.messages.Error: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for “a”: Domain name needs at least one dot
2020-01-04 14:55:51,019:ERROR:certbot.log:An unexpected error occurred:
2020-01-04 14:55:51,019:ERROR:certbot.log:Error creating new order :: Cannot issue for “a”: Domain name needs at least one dot

Ilia · January 4, 2020, 2:36pm

What do you set on the previous page? Can you share the screenshot?

ftoemboel · January 5, 2020, 7:48pm

Regardless of what I set, the outcome is always as above.

Ilia · January 5, 2020, 9:24pm

Oh, wait, you are not adding $@ to the end of the script line, to pass the other arguments from Webmin core?

certbot --dns-cloudflare --dns-cloudflare-credentials API-Key $@

ftoemboel · January 6, 2020, 12:52am

I now added $@ to the following result:

ftoemboel · January 6, 2020, 1:15am

I now traced down the problem why it is not working. It is now simply passing too many arguments to certbot (mainly the concomittant use of “–dns-cloudflare” and “- a webroot”). Any idea how to fix that?
‘–agree-tos’, ‘–dns-cloudflare’, ‘–dns-cloudflare-credentials’, ‘SOME_PATH’, ‘–register-unsafely-without-email’, ‘-a’, ‘webroot’, ‘-d’, ‘example.tld’, ‘-d’, ‘www.example.tld’, ‘-d’, ‘mail.example.tld’, ‘–webroot-path’, ‘/home/example/public_html’, ‘–duplicate’, ‘–force-renewal’, ‘–manual-public-ip-logging-ok’, ‘–config’, ‘/tmp/.webmin/SOMENUMBERS_letsencrypt.cgi’, ‘–rsa-key-size’, ‘2048’, ‘–cert-name’, ‘example.tld’]

Ilia · January 6, 2020, 1:18am

Yep. It was expected. I just wanted you to give it a try.

Okay, I will talk once more with Jamie, will see when we could added it.

Ilia · January 6, 2020, 1:20am

Wait. Can you try DNS validation on Webmin Configuration/SSL certificate page?

ftoemboel · January 6, 2020, 1:24am

Did not work:

ftoemboel · January 6, 2020, 1:25am

Thank you for your awesome work!

Ilia · January 6, 2020, 1:32am

When you requested certificate with your command in console, you didn’t do any manual modifications to DNS records on CloudFlare, as everything is done by cerbot command, right?

Can you provide successful output of such command (when run in console)?

ftoemboel · January 8, 2020, 1:04pm

Of course!

user@servername:~$ sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials PATH_TO_CREDENTIALS -m someone@example.tld -d *.example.tld
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for example.tldWaiting 10 seconds for DNS changes to propagate
Waiting for verification…
Cleaning up challenges

IMPORTANT NOTES:

Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/example.tld-0001/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/example.tld-0001/privkey.pem
Your cert will expire on 2020-04-07. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew all of your certificates, run
“certbot renew”
If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

Ilia · January 8, 2020, 1:23pm

We will most likely factor out Let’s Encrypt into a separate module, away from Webmin Configuration to make more sense, and then we will add more features into it.

For now, you could easily use custom built scripts, to make things work for you automatically.

@Jamie Do you think we could do it a bit sooner?

ftoemboel · January 8, 2020, 1:52pm

Separate module sounds great!