Situation: Server with Webmin/Virtualmin hosting multiple virtual servers all correctly set up with Letsencrypt SSL certificates among which the default domain’s (main server identity) SSL certificate is also globally used by the email services (Dovecot and Postfix).
Everything used to work fine for the last few years up until Letsencrypt switched to the new R3 CA certificate. I actually pinpointed the exact source of my problem to that R3 CA thing after 4 days of investigating log files, searching the web and troubleshooting with different scenarios and apps.
Problem: Some email clients (not all) do not communicate with the email server anymore (no IMAP connection and no possibility to send emails). For example, the email client app from Samsung (even the latest update) silently stopped loading new emails while the same accounts set up on Thunderbird continued working flawlessly. Trying to configure those same email accounts with my usual parameters using Gmail android app for example displayed an SSL CA error and the only workaround was to allow the client to accept any certificates and same went with another third party android email client that I tried. So I had to accept all certificates for IMAP to connect and even disable STARTTLS and basically set the parameter to “NONE” for SMTP to pass the configuration. I also made a test with the new Microsoft Windows default email client on Windows 10 and let it automatically set my mailbox which surprisingly worked perfectly but I suspect it just decided to be more tolerant with the CA certificate in order to make it work.
On the server’s side, every time I try to connect using one of the above “non-working-anymore” clients, the mail.log file shows the errors: dovecot: imap-login: Error: SSL: Stacked error: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown: SSL alert number 46 dovecot: imap-login: Error: SSL: Stacked error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca: SSL alert number 48
and sometimes the following warnings: postfix/smtpd[xxxxx]: warning: TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:s3_pkt.c:1487:SSL alert number 48: postfix/smtpd[xxx]: warning: TLS library problem: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:640:
On the other hand, I have absolutely no problems on the https:// side. All my websites display without complaining on major browsers irrespective of them being under the old X3 CA or the new R3 CA.
What would be the best and safest solution (or even a temporary workaround or patch) to apply in this situation?
I read about the possibility to force the Letsencrypt client to use an alternate chain and get the old X3 CA anyway. I would like to understand how to do that and try it at least to verify my theory but then I’m not sure if it’s a viable solution for the long term and I’m also reluctant to mess around the way certificates are being obtained and updated in my server (I’m still using the built-in Virtualmin Letsencrypt client to manage and update my certificates).
I’m not comfortable with the tedious task of manually changing (or have users change) the parameters in each email client to a lower SSL standard just to circumvent this issue.
Thanks in advance for any help
On the other hand, I have absolutely no problems on the https:// side. All my websites display without complaining on major browsers irrespective of them being under the old X3 CA or the new R3 CA.
It seems that main, used by default Dovecot’s certificate wasn’t updated? Try doing it manually by clicking Copy to Dovecot button on SSL Certificate > Service Certificates page.
Thanks for your quick reply Ilia.
It’s really weird because I can’t find the necessary buttons on the Service Certificates page of the main domain in question. I’ve seen them in the past but never noticed when and why they disappeared from that page!
What’s even weirder is their presence in every other domains and subdomains of my server!
Screenshot for a random subdomain of the main domain
Also, during my troubleshooting, I did check the files /etc/ssl/certs/dovecot.pem and /etc/ssl/private/dovecot.pem right after manually requesting a new Letsencrypt certificate and they were both updated immediately.
But if those are not the certificate files to be updated for Dovecot, how can I update the necessary files manually while trying to figure why the “Copy to Dovecot” button disappeared?
Have you requested a certificate using certbot package?
But if those are not the certificate files to be updated for Dovecot, how can I update the necessary files manually while trying to figure why the “Copy to Dovecot” button disappeared?
You could try going to some other virtual server, when those buttons appear, click _Copy to Dovecot and then go back to the domain you wish to have it setup and do the same operation again. Do .ca files and .bundle and .everything contain R3 intermediate certificate or X3?
Alright now I understand that the buttons disappear from a domain’s page when used. Silly me!
Well I did just what you said by copying to dovecot from another domain then restored it from the main domain to no avail, the situation remains the same. The same error (Dec 23 14:39:22 ns dovecot: imap-login: Error: SSL: Stacked error: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown: SSL alert number 46) reappeared on the log file as soon as I tried syncing emails from the samsung android app.
Indeed I’m using Webmin 1.962 and Virtualmin 6.14 but I’m not using certbot to request the certificates, I’m still using the built-in Virtualmin mechanism to request certificates (I believe through the ACME v2 python script).
Now if you have a moment, please have a look on the openssl command’s result and tell me what you think:
CONNECTED(00000003)
depth=0 CN = myMainDomain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = myMainDomain.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=myMainDomain.com
i:/C=US/O=Let's Encrypt/CN=R3
---
Server certificate
-----BEGIN CERTIFICATE-----
M//
...
...
...
//==
-----END CERTIFICATE-----
subject=/CN=myMainDomain.com
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2204 bytes and written 447 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Session-ID-ctx:
Master-Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 //
......
......
......
0090 //
Start Time: 1608730574
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
Yep that did the trick!
The old X3 CA being used in the chain instead of the new R3 was indeed the node of the problem.
Thanks a bunch for helping me out with this annoying issue.
Keep up the good work guys, you’re the best
