SSL domain using self-signed CA works for MQTT but fails for HTTPS

Virtualmin
1
SYSTEM INFORMATION
OS type and version Ubuntu Linux 22.04.5
Usermin version 2.302
Virtualmin version 7.30.8
Theme version 24.02
Apache version 2.4.52
Package updates All installed packages are up to date

We have a OpenSSL self-signed Root and private key, which signs our server cert, server.CA and then client an intermediary certs

We have successfully set up mosquitto on virtualmin on a subdomain mqtt.domain2.com

However, when we try now HTTPS SSL site, it fails.
even openeing a browser at https://mqtt.domain2.com fails with ‘not secure’ message forcing HTTP vs HTTPS.
the cert does show its valid

Virtualmin with subdomain

Manage Virtual Server → Setup SSL certificate → CA Certificate is the self signed CA.crt that created the server and client certs

CN = mqtt.domain2.com

In Virtualmin for the subdomain

Web Configuration → Configure SSL Website → SSL options

, we see the SSL for the subdomain is correctly listed.

Under the SSL Directives, we see it correctly configured:

SuexecUserGroup #1005 #1002

ServerName mqtt. domain2.com

ServerAlias www.mqtt. domain2.com

ServerAlias mail.mqtt. domain2.com

ServerAlias webmail.mqtt. domain2.com

ServerAlias admin.mqtt. domain2.com

The SSL certificate is good also (which works for MQTT). From a web browser, for the subdomain (mqtt.Domain2.com), we see the SSL certificate:

Common Name (CN): mqtt.domain2.com

Organization (O): domain2.com

Organizational Unit (OU): server.domain2.com

Common Name (CN): root.domain2.com

Organization (O) : domain2 LLC

Organizational Unit (OU): domain2.com

2

You have presumably not told MQTT to check certificates to be sure they’re valid, or you have made your local CA cert available to the MQTT client. MQTT is a completely different protocol with different servers and clients
you can’t glean any useful information about anything related to HTTPS from what MQTT servers and clients are doing.

Any major browser will reject invalid certificates (which is any certificate that cannot be validated by the CAs found in the system CA bundle or the CA bundle included with the browser). It is possible to add a CA to a browser, but only your own.

For web servers, you should get a free Let’s Encrypt certificate (or one from any other certificate provider that will be considered valid by any browser). You should not try to use a self-signed certificate for a website.

3

Also, I know we’ve had this conversation in the past, but to reiterate: Your MQTT server does not need to share a certificate with your web server. They are completely unrelated services. You can get a Let’s Encrypt certificate for mqtt.yourdomain.tld and use it for web traffic, while also using a self-signed cert (presumably for cert-based authentication, since there’s not other reason to run your own CA) for your MQTT server on the same name (MQTT runs on a different port).

You should not try to use a self-signed certificate for your web server. You may have good reasons to use a self-signed cert and your own CA for MQTT, if you are using X.509 authentication for your MQTT clients.

4

The issue is for embedded devices, which require long SSL for the product life, 10-20 years for firmware updates over HTTPS. its a real issue that needs a solution. we solved this for MQTT.

5

If you don’t have a way to update the CA bundle on the device, you can’t solve it. But, I would assume you can update the CA bundle on the device.

If you haven’t updated the CA bundle on the device, then you have not solved it for MQTT, either, you’re just ignoring invalid TLS. A self-signed certificate is invalid unless the client knows how to validate it, which means it has a copy of the CA cert for verifying certs signed with your CA signing key.

I’ve told you how to solve this: Make your client devices aware of your private CA.

You can’t do that with browsers in the wild, but presumably if these are devices you control you can.

6

For example, on an EL Linux system (not your embedded devices, unless they run some EL Linux version) that’s done like this: How to configure your CA trust list in Linux

Or, for Debian, the last bit of these docs cover that: How To Set Up and Configure a Certificate Authority (CA) On Debian 11 | DigitalOcean

I don’t know what your devices run, so I don’t have any guesses about how to make them aware of your CA.

7

An alternative to TLS for updates is to sign the updates, instead of using encryption on the web server. That’s how much Linux updates are handled. apt and dnf repos do not need to run over HTTPS. The repo metadata is signed and there are hashes for the packages for apt, and rpm packages are signed along with signed metadata for dnf. I’m pretty sure most update systems are similar, as most update systems in wide use predate widespread SSL/TLS usage.

(But, you still probably ought to get TLS right for your MQTT connections. It isn’t right currently, even if it looks like it works, unless you have added the CA to your client devices. It just means it’s ignoring an invalid TLS cert, so it could be trivially spoofed and would allow MITM attacks.)

8

All the clients have the root CA. MQTT works, HTTPS fails.

to my knowledge, we are not using MQTT with the ‘insecure’ field
Our Mosquitto clients are NOT to explicitly ignore certificate validation errors by using the --insecure option, we do validate, so no idea.

here is the server conf file

Place your local configuration in /etc/mosquitto/conf.d/

A full description of the configuration file is at
/usr/share/doc/mosquitto/examples/mosquitto.conf.example
pid_file /run/mosquitto/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
log_dest file /var/log/mosquitto/mosquitto.log
listener 8883
cafile /etc/ssl/certs/ISRG_Root_X1.pem
certfile /etc/mosquitto/certs/server.pem
keyfile /etc/mosquitto/certs/server.key
password_file /etc/mosquitto/password_file
allow_anonymous false2
include_dir /etc/mosquitto/conf.d

9

This is the Let’s Encrypt CA cert. I thought you were using a self-signed cert?

What HTTPS client are you using on the client device?

10

appologies, i pasted the original, here is our conf file, and its our new ‘long term’ OpenSSL private SSL

pid_file /run/mosquitto/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
log_dest file /var/log/mosquitto/mosquitto.log
listener 8883

#is the root CA
cafile /etc/mosquitto/certs/ca.crt
#the ca-chain is a server.pem + root CA.pem
certfile /etc/mosquitto/certs/ca-chain.cert.pem
#is the server key
keyfile /etc/mosquitto/certs/server_new.key

password_file /etc/mosquitto/password_file
allow_anonymous false
include_dir /etc/mosquitto/conf.d

I have also uploaded the 2 x pdf of the actual debug messages of the TLS, one from MQTT, the other from HTTPS. they are the same for the first part, then you see the MQTT continues (whilst the HTTPS errors)
compare MQTT vs HTTPS TLS debug.zip (255.1 KB)

11

OK, so that explains how MQTT works. You’ve made it aware of your CA.

What HTTPS client are you using, and how are you configuring it to know about your CA?

12

the client is an embedded system running FreeRTOS, but most of the libraries for esp-idf are ported from Linux, so legitimate

and we use the exact same CA.pem file for both, and the client uses the same TLS for both!!

if I switch to HTTP, of course it works.

its only HTTPS that fails the TLS with 2700

13

So, your problem is with that HTTPS client. I have no experience with that, so I’m not going to be of any help.

14

Well, yes and no
I can pursue the vendor for the TLS library why its get the 2700 certificate reject

but its the same for the website with a browser - the certificate shows as ‘Not Secure’ and refuses the HTTPS and moves to HTTP
I would have thought that it would still accept the SSL as the cert + CA can validate the cert is good, hence understand a warning of ‘self-cert’, but reject seems harsh.

Seems MQTT is able to ‘deal’ with the issue

for HTTPS, seems i have 2 options

  1. use HTTP, but not secure
  2. download the new cert to the client each time we need to access mqtt.domain.com

so, how can we from the client, pull the SSL from the sever ?
or even an intermediary server to perfrom the task prior to a OTA HTTPS ?
i.e go to mqtt.domain.com, and pull the cert, which i think needs to be the CA ?
if we use Letsencrypt on the server (and this year moves to 47 day expiration), can we ‘pull’ the right cert to be authorized by the TLS hello ?

I know you can pull it with openssl:
openssl s_client -showcerts -connect mqtt.domain.com:443 < /dev/null | openssl x509 -outform PEM > certificate.pem

but how from a client without openssl installed ?

15

Have you added your CA to your sytsem CA bundle, or otherwise made the browser aware of your certificate authority?

You have configured it to be aware of your certificate authority.

I don’t know what this means. You mean the CA cert? You need to preinstall it. You can’t make browsers accept random CAs without user intervention
that defeats the purpose of TLS. It would allow anyone to pretend to be any website, and allow MITM attacks. TLS provides two things, encryption and validation you are talking to who you think you’re talking to. If the server could force the browser to trust a self-signed cert, the second thing would be removed. You could never trust a website to be who they say they are.

The CA cert needs to be preinstalled. For devices you control, you can do that. Just put the file on the device and configure the clients to know about it. I’ve provided links for how to do it on popular Linux versions, but I don’t know anything about your embedded devices. You’ll need to figure that one out.

But, the answer remains the same as it’s always been: Your clients need to know about the certificate authority and be configured to trust TLS connections verified by that CA. Unless/until you do that, you should not expect connections to be valid/verified. They’re not supposed to be.

16

yes, the same client has the ca.pem, it works for MQTT, does not work for HTTPS
will work with the vendor why the TLS HELLO does not accept the ca.pem

17

Considering you always connect using the hostname in the cert’s SAN, I would use an LE cert for HTTPS (webserver) and keep the private CA for MQTT, serving leaf and intermediate certs (ssl.cert and ssl.combined) without the root on both.

18

right, but that doesnt fix the problem with LE certs expiring, and the client then cannot complete the TLS HELLO process as its copy of the old cert has expired

so since HTTPS fails, and MQTT works with a self-signed cert, unless the vendor (esp-idf) comes back with a fix for the self-signed fail, then we are left with downloading the LE cert CA to the client event 47 days, or whenever it needs to access HTTPS.

19

They said they need it to last 10-20 years. Let’s Encrypt has changed their signing root CA at least three times in the past five years. Unless the OS CA bundle gets reliably updated regularly on those devices, Let’s Encrypt is not an option.

20

If updating the LE CA cert works, why on earth wouldn’t installing your own CA work? That doesn’t make sense. If you know how to add CAs and update CAs, Let’s Encrypt is reintroducing complexity you don’t want (because you have to update the CA whenever Let’s Encrypt changes their root signing cert, but you can make your CA never expire if you choose, so you can keep signing new certs forever and they’ll keep working on your devices in the field without need a new CA installed).