TLSA records contain SHA256 of empty string when using ECC/ECDSA certificates

Virtualmin
1

System Information:

Severity: Critical - this bug took down our entire production mail server. All incoming mail was rejected for hours by DANE-validating senders (Microsoft 365, Gmail, etc.) until we could diagnose the root cause and disable DNSSEC as an emergency workaround. Please prioritize.

Bug Description:
When TLSA records are generated for a domain using an ECC (ECDSA) Let’s Encrypt certificate, all generated TLSA records contain the SHA256 hash of an empty string instead of the actual public key hash:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

This is the well-known SHA256 of empty input (echo -n "" | sha256sum).

Root Cause:
In feature-dns.pl, function create_tlsa_dns_record() (around line 5108-5113), the public key extraction pipeline uses openssl rsa -pubin, which only handles RSA keys:

$hash = &backquote_command(
    "openssl x509 -in ".quotemeta($temp).
      " -pubkey -outform DER 2>/dev/null | ".
    "openssl rsa -pubin -outform DER 2> /dev/null | ".   # <-- only handles RSA
    "openssl sha256 2>/dev/null");

For ECC/ECDSA certificates, openssl rsa -pubin fails silently (stderr discarded), producing empty stdout. The next stage (openssl sha256) then hashes the empty stream, resulting in the empty-string hash. Exit code is 0 because of the discarded errors, so the bug is not detected by the existing error handling.

Reproduction:

  1. Create a Virtualmin domain with an ECC Let’s Encrypt cert (Hash type: ECC in SSL settings)
  2. Enable TLSA records: virtualmin modify-dns --domain example.com --enable-tlsa --tlsa-pubkey
  3. Check generated records: dig _25._tcp.mail.example.com TLSA +short
  4. All records contain e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Impact:
DANE-validating mail servers (Microsoft 365/Outlook, Gmail, etc.) reject incoming mail to such domains with errors like:

450 4.7.323 tlsa-invalid: The domain failed DANE validation

This effectively blocks all incoming email to affected domains until DNSSEC or TLSA is disabled, or the records are corrected manually. In our case this caused a multi-hour mail outage on our production server.

Proposed Fix:
Replace openssl rsa -pubin with openssl pkey -pubin. The pkey command handles RSA, ECC, Ed25519, and other key types uniformly:

$hash = &backquote_command(
    "openssl x509 -in ".quotemeta($temp).
      " -pubkey -outform DER 2>/dev/null | ".
    "openssl pkey -pubin -outform DER 2> /dev/null | ".
    "openssl sha256 2>/dev/null");

openssl pkey is available since OpenSSL 1.0.0 (released 2010), so this change has no compatibility downside.

Verified Working:
This fix was applied and tested on our production system (Debian 13, Webmin 2.621, Virtualmin 8.1.0.pro-1, OpenSSL 3.5.4). After patching feature-dns.pl with the one-line change above and restarting Webmin/Usermin, TLSA records are now generated correctly for both RSA and ECC certificates. We verified the resulting hashes against the public keys served by Postfix, Dovecot, Apache, and Webmin/Usermin - all match.

Disclosure: The diagnosis and fix were developed with assistance of an AI assistant (Claude). The actual code analysis, testing, and verification on a live production system were performed by me.

Secondary Issue: Stale TLSA records not cleaned up

While diagnosing the above, we also encountered a related issue that should be tracked separately if you prefer, but it compounds the impact of the main bug:

The Virtualmin GUI/CLI does not reliably remove old TLSA records when:

  • Disabling TLSA via virtualmin modify-dns --disable-tlsa
  • Re-syncing TLSA via --sync-tlsa
  • Renewing or replacing SSL certificates

Result: Multiple generations of stale TLSA records accumulate in the BIND zone file. Even after --disable-tlsa followed by --enable-tlsa, we found old 3 0 1 records from previous certificate generations alongside the newly-generated 3 1 1 records. Some of these stale records dated back several certificate renewal cycles.

These leftover records also cause DANE validation failures because none of them match the currently-served certificate. Manual cleanup of the zone file (sed-removal of all TLSA records, then re-enable via Virtualmin) was required to recover.

Suggested investigation:

  • Verify that sync_domain_tlsa_records() in feature-dns.pl correctly identifies and removes ALL existing TLSA records for the domain before adding new ones, not just records matching the current generation logic
  • Consider adding a --purge-tlsa flag that removes all TLSA records unconditionally regardless of what Virtualmin “remembers” having added

Combined with the ECC bug above, this means a system that started with RSA, switched to ECC, and renewed a few times can end up with a pile of broken TLSA records and no straightforward way to clean them up.

2

this is being dealt with here:

1 Like
3

I also use ECC and dont have this problem.

There is always a problem with mail for DNS renewal with virtualmin because it does 301 cert renevwals and not 311.

Some years back it was brought up, but I think it is still not changed.

Three line fix for DANE TLSA DNS records going out of date every 90 days with LE · Issue #803 · virtualmin/virtualmin-gpl

4

That makes sense if you’re still using the Yes, with full certificate TLSA option (3 0 1). This bug only affects the Yes, with just public key option (3 1 1), because that path extracts and hashes the public key from the certificate.

The setting can be changed under “Virtualmin → System Settings → Server Templates → Default / Sub-Server → SSL website for domain” page with “Create TLSA DNS records for SSL certificates” option.

5

Oh! I will switch to that public key option then when the fixes are all in. Thanks!

1 Like