Openssl Verify Fails with Virtualmin Lets Encrypt: Verify return code: 21 (unable to verify the first certificate)

When verifying Lets Encrypt Certificate Chain for a virtual server in Virtualmin (newest updates), the following error occurs:

depth=0 /CN=mydomain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /CN=mydomain.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /CN=mydomain.com
verify error:num=21:unable to verify the first certificate
verify return:1

subject=/CN=mydomain.com
issuer=/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3

No client certificate CA names sent

SSL handshake has read 3280 bytes and written 712 bytes

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 4F0B5503C08676DF90941BFBBF2355665D9408C80F409947512E7048F720C0DA
Session-ID-ctx:
Master-Key: 56CCCC08C1CE3E5B477132D298D8074D6A1160C0B3F6FE07181F0E2526DFFFEEE0CE5AEEE0F371C2F0C154A74B2A32D3
Key-Arg : None
Start Time: 1465813969
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)

read:errno=0

This has implications on Wordpress sites using for instance “Jetpack Plugin” - http_request_failed SSL certificate problem: unable to get local issuer certificate]
or other Applications using OAUTH and site interconnects for instance REST APIs…

I suspect it has something to do with the Certificate is not generated as fullchain.pem but only ssl.cert for the virtual servers.
https://community.letsencrypt.org/t/cannot-verify-domain-with-openssl/11545

I detected two problems.

  1. The Centos 7 httpd is 2.4.6 so there can no chain file be in SSLCertificateFile Directive. It has to be a chain of Root and intermediate cert in the directive: SSLCertificateChainFile
  2. When adding the Directive to the domain of the virtual host domain “myvirthost.com”, nothing happens because
    openssl s_client -connect myvirthost.com:443
    defaults to the certificate of the MAINVIRTUALDOM.COM which is the one the host is named after and is used for admin.MAINVIRTUALDOM.COM and webmail.MAINVIRTUALDOM.COM as well as postfix, etc.

At least the Error goes away when adding the Directive
SSLCertificateChainFile /home/MAINVIRTUALDOM.COM/CHAIN.crt to httpd.conf with the intermediate crt and root crt copied together into this file:

CONNECTED(00000003)
depth=1 /C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0

Certificate chain
0 s:/CN= MAINVIRTUALDOM.COM
i:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
1 s:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1

Server certificate
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
subject=/CN= MAINVIRTUALDOM.COM
issuer=/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3

No client certificate CA names sent

SSL handshake has read 5851 bytes and written 712 bytes

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 641DA4CF4D8822720888E42EFB1CDF52D66D4F2852B676F848DF9FC568996A1B
Session-ID-ctx:
Master-Key: 5BB3CD953BEBC1E1406B1AE65A9DF05ECE10C652643830A2571F9F5DBD4752D59532F4A4CA69766F2CB095B9105BEC30
Key-Arg : None
Start Time: 1465817007
Timeout : 300 (sec)
Verify return code: 0 (ok)

read:errno=0

But the Error with Wordpress JETPACK doesn´t go away.

So I guess there are two problems:

1.) When Jetpack tests for ssl validity, it does not use the “servername” option:

openssl s_client -servername virtualdomain.com -connect virtualdomain.com:443

  1. The Virtualmin Apache Configuration lacks to include the SSLCertificateChainFile in every vhost directive which should be only applicable for apache >2.4.8 because in <=2.4.8 the complete chain should be in the SSLCertificateFile directive.

openssl s_client -servername virtdomain.com -connect virtdomain.com:443 validation fails:

CONNECTED(00000003)
depth=0 /CN=virtdomain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /CN=virtdomain.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /CN=virtdomain.com
verify error:num=21:unable to verify the first certificate
verify return:1

Certificate chain
0 s:/CN=virtdomain.com
i:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3

Verify return code: 21 (unable to verify the first certificate)

Problem and Solution:
Virtualmin does not include the SSL CA Chain of lets encrypt in the certificate file.

In apache > 2.4.8 there should be a file containing the intermediate AND root Certificate of LetsEncrypt https://letsencrypt.org/certificates/ in the httpd.conf Directive of every vhost: SSLCertificateChainFile /home/virtdomain.com/chain.crt additional to the SSLCertificateFile Directive.

In apache <=2.5.8 the SSLCertificateFile Directive should contain the Certificate chain in that order: {virtdomain CRT}{LE Intermediate CRT}{LE Root CRT} because the SSLCertificateChainFile is deprecated.

However, the Jetpack Problem is solved by disabling the xmlrpc block in the All in One WP Firewall, and add the $_SERVER 443 Directive into WP config like described here:
https://jetpack.com/support/getting-started-with-jetpack/known-issues/

Openssl verification problem still persists