I have a virtual server created for the server host name. It is one line above the one that overwrote my cert. It is possible that I clicked the wrong one when working on this a while back.
I purchase a wild card cert to use on all my systems. Particularly for Postfix, Dovecot as I don’t know how reliable LetsEncrypt certs are with respect to email client recognition.
So, apparently at least one of those buttons follows a path I set to my purchased wild card cert used by multiple services. How do I find which one or ones is doing this or how should I fix this. It seems I have a ticking time bomb here.
FYI, I shell in via console to update my main server cert files. I also find the interface to be a bit confusing when one first uses the command line to obtain and install a cert.
I now have had this happen on another CentOS 7 system. I had a manual creation to do. It did not replace my main server keys but it did leave Dovecot’s config broken and Dovecot down. I deleted all the entries at the bottom of dovecot.conf which were added by LetsEncrypt. This allowed Dovecot to start.
This has absolutely smashed my system. It took out more than just email…I have also had a client joomla website go down as well. For me it was two problems that just happened to coincide (one problem i have encounered before…(which is number 1 below)
the webmin update over wrote Virtualmin>Services>PHP-FPM Configuration “group” name for one of my client domains…it removed the “.com” at the end of the domain causing php5.6-fpm to enter a failed state (so it read group = domain when it should have read group = domain.com)
this SSL cert issue has really created a nightmare for me…Ive got complaints all over the place with clients saying their apps (such as outlook and apple mail) arent working
When i requested a new SSL cert for my main domain and server, it completely knocked webmin/virtualmin offline and i could not access my control panel. I had to restore the server from a backup. Totally screwed me this did.
Anyway, at least i know what is going on now.
I have had to roll back to only using webmin update 6.09. I cannot install 6.09-2 or it screws up my system.
I need to pick a quite time on my network before run the gauntlet again.
PHP-FPM is wholly unrelated to this conversation, please open a new topic (or maybe there’s already a topic for it).
I’ve been unable to reproduce the certificate overwrite issue, thus far, but I’ll talk to Jamie to see if he has any insight. It’d be really helpful for the discussion to be specific about which services are having problems. There are multiple independent key/certificate config locations (and it differs based on distro/version), so as long as we’re not specific about problem reports, it leaves a lot of variables undefined.
I see Dovecot mentioned by @dumorian, which is helpful; is the problem always with Dovecot (IMAP/POP3 client connections) or does it impact Postfix as well, or anything else? Dovecot is particularly complex because in some versions it supports multiple certificates on a single IP (Postfix does, too, in very new versions, but I don’t think we’re trying to handle that yet). A wildcard cert complexifies the situation, as well, and I think I want to narrow it down to one specific workflow that triggers the problem with a regular (non-wildcard) Let’s Encrypt certificate, assuming that is a case that happens. Me or Jamie needs to be able to see it happening to figure out why.
Joe,
it happened twice in two days on my server (you can have access if you wish)
as far as I see the domains autorenewed the lets Encrypt certificate and then overwrote the certs for the main domain on the server.
to get over it I twice ran the request Certificate for teh main domain.
Brian
I run a EV wildcard cert on all of my systems. I have Postfix, Dovecot, FTP and such set to use that cert which is stored in the default /etc/pki/tls directory structure for CentOS. All of my systems are fully updated except for ImageMagick which has a dependency issue last time I checked.
On one system, the first to have the problem, an update to clientdomain.com overwrote my EV cert and key in /etc/pki/tls. I thought maybe I was the problem as this domain was just one line down from myservername.myserverdomain.com. I though maybe I clicked on the wrong line.
Yesterday another cert renewed, it did the same thing. I’m certain I did not do anything aside from the initial creation of the cert.
Issue one: LetsEncrypt is overwriting my main cert files.
Last night, after renaming a wordpress.domain.com virtualserver, I ran LetsEncypt manually to get the cert for the new www.domain.com. I found out this morning that Dovecot was not running. When I tried to start it I got an error something to the effect that “ssl some key =” I didn’t see the particular line so I just deleted all the entries put into dovecot.conf by LetsEncrypt. I should have copied the error, but panic happens when email is not functioning. In this case my EV cert was not overwritten, but only the Dovecot problem.
Those are the three LetsEncypt updates which have occurred over the last couple of days.
Please be specific. For what services did you see this behavior? There are several: Dovecot (IMAP/POP, i.e. receiving mail), Postfix (SMTP, i.e. sending mail), ProFTPd (FTP), Webmin and Usermin.
They all use different key/cert locations and those locations can be different depending on distro/version. I’m trying to reproduce the problem, and just hearing that “it happened to me, too” doesn’t really add any information to help do that.
Surely only one virtual server should “own” a particular service. For example, using the “Copy To” button for Dovecot on a specific VS, that should then display:
“This SSL certificate is already being used by : Dovecot (host this-domain.com)”
But since the most recent upgrade a significant number of virtual servers on my system display this message (and I know for a fact that the “Copy to” button was never used for them, so I have no idea why this is so).
So essentially they are all fighting for control of the one service. Therefore, when one gets a certificate update it replaces the Dovecot certificate and throws everything into chaos.
The Virtualmin control panel has no means for us users to correct this. There is no button to STOP a VS from having control of the Dovecot certificate. You might say - just go to some VS that does display the “Copy to” button. Hit that, and that should reset everything. But it doesn’t. In fact in my experience the “Copy to” button for Dovecot simply does not work. Does it for you Joe?
OK, well try this then to reset it: Go to the server config “enabled features” and disable and then enable “SSL web site”. But then you enter a new fresh hell with this:
“Adding new SSL virtual website … … certificate file is not valid : Line 32 does not look like PEM format”.
I have found the only option then is to completely delete then restore the VS. & guess what? When you do, there it is again:
“This SSL certificate is already being used by : Dovecot (host this-domain.com)”
Joe
sorry about lack of info.
I should have taken more notice, but it was at least the Dovecot and I think Postix certs that were overwritten by the domain that wasn’t the main domain for my server.
Please note that I do NOT use LetsEncrypt certs for my mail server systems. They are not recognized and from what I have read will not become EV or OV certs so will not be accepted by many or maybe all email client softwares. That why I buy a cert.
I’m left with how are my EV cert keys getting overwritten when they are completely outside of any domain’s file structure?
At the moment, any client I suppose can log in and run a LetsEncrypt renewal which is a bad situation when it takes down smtp, imap and pop services, or at least throws them into using the wrong cert which email clients complain about and would require a scary override for them. “Who the heck is ‘somedomain’ and why are they into my email program?”
I did install I believe it was certbot when we had the last LetsEncrypt issue a few months ago. I was getting extremely close to needing renewals so I installed the following:
Description certbot is a free, automated certificate authority that aims
to lower the barriers to entry for encrypting all HTTP traffic on the internet. Package certbot Class Unspecified Version 1.3.0-1.el7 Vendor Fedora Project Architecture noarch Installed 03/30/2020 11:17:29 AM
I have that on the 2 affected systems. But again, so far only 3 certs needed updates which happen to all be on CentOS 7 systems.
I did not uninstall it as I didn’t know if Virtualmin may also need that package. That’s about the only fly in the soup that I can think of on these systems.
Joe
I raised a ticket with some information from my system.
I use Lets Encrypt and when a certificate auto renewes the SSL cert from that domain gets copied to main server ssl (at least for Dovecot)
On the first server where this problem occurred, I deleted the ssl entries within dovecot.conf for that first offending domain.
Results:
Before SSL cert said this cert was already being used by Dovecot.
Deleted the entries.
Dovecot would not restart. I believe lots of processes running maybe under these other/alternate certs?
Rebooted the system and Dovecot started fine.
The notice in SSL cert for the domain was gone.
Results:
Apparently on CentOS 7 systems the use of the cert is determined by looking at dovecot.conf? It seems like ssl.conf may have been a better place?
Rebooting the system is never fun during prime hours.
I still would like to know what sets a renewal to overwrite the main server keys in /etc/pki/tls
I haven’t found reference to this yet.
