Fixed: Postfix SNI path problem for tls_server_sni_maps OpenSSL 3.0 callback limitation

Webmin
,
1

After migrating to Debian 13 (Postfix 3.7+ / OpenSSL 3.0), I encountered a persistent SSL Alert 80 (Internal Error) when using tls_server_sni_maps with file paths. Even with correct permissions and valid ECC/RSA-hased certificates, the SNI handshake would fail during the identity swap.

The Issue: In some Postfix builds linked against OpenSSL 3.x, the TLS library fails to hot-swap certificate identities via file pointers in the SNI map (possibly only if the cryptographic providers (RSA vs ECC) weren’t perfectly primed at startup).

The Solution: Instead of using file paths in the map (e.g., example.com /etc/postfix/cert.pem and .example.com /etc/postfix/cert.pem or even example.com file:/etc/postfix/cert.pem), you must inline the raw certificate data as a Base64 string (and the leading dot can not be used anymore, as an exact match is required, so add your MX records like so mail.example.com tHeBaSeDhAsH..... This allows the OpenSSL callback to access the key material directly in memory.

Implementation:

  1. Create a script (I called mine /usr/local/bin/update_postfix_sni_maps.sh) to iterate through /etc/ssl/virtualmin/.
  2. Concatenate the Key, Cert, and CA into a single blob.
  3. Convert that blob to a single-line Base64 string: base64 -w 0.
  4. Format the map as: domain.com [BASE64_STRING] (one (sub)domain per line).
  5. Build the database: postmap hash:/etc/postfix/tls_server_sni_maps
  6. And apply the changes: postfix reload.

Automation Hook: ensure it only rebuilds the map when your certificates are renewed.

Add a post hook to certbot. See Adding --pre-hook and --post-hook for certbot? for the conversations on that. Note: this is not the best approach, something like a script in Virtualmin Configuration > Actions upon server and user creation > Command to run after making changes to a server using the $VIRTUALSERVER_ACTION variable to filter for SSL_DOMAIN (but I did not try this, as I only use certbot).

This resolved the “Internal Error 80” and allowed a mixed RSA/ECC environment to function perfectly with TLS 1.3.

Observation: When using MTA-STS in Enforce mode, an “Internal Error 80” or an SNI mismatch isn’t just a warning; it is a total mail block.

Data: TLS reporting showed a transition from 100% success to 100% failure due to the Postfix/OpenSSL 3.0 path-parsing issue.

Validation: Implementation of the Base64-inlined SNI map restored 100% deliverability and passed all DMARC/TLS checks.

Some useful info and commands:

  • postconf -e "smtpd_tls_loglevel = 3" && postfix reload will show you the actual errors.
  • echo "QUIT" | openssl s_client -connect 127.0.0.1:25 -starttls smtp -servername example.com -brief to test for failure or success
  • Set smtpd_tls_chain_files in main.cf to an /etc/ssl/virtualmin/?/ssl.everything, but not directly if you use a jailed smtpd, so make a copy that postfix can read (sudo -u postfix test -r /tmp/some.file && echo OK || echo NOT).
  • Test to see if the (sub)domain is in the map: postmap -q example.com hash:/etc/postfix/tls_server_sni_maps | cut -c 1-50

Wish: that virtualmin gets a post hook for provider-independent certificate renewal and keeps the domain certificates up to date for the TLS SNI map.

Related posts to cross-link: