Virtualmin overwrites virtual_alias_map generated by Mailman 3

Virtualmin
1
SYSTEM INFORMATION
OS type and version Ubuntu Linux 24.04.2
Webmin version 2.303
Virtualmin version 7.30.8 (GPL)
Webserver version Apache 2.4.58
Mailserver version Postfix 3.8.6
Mailman version GNU Mailman 3.3.10 (Tom Sawyer)
Related packages Postfix, Mailman 3

Summary

When using Mailman 3 with Postfix and Virtualmin, the Mailman-managed alias map (/opt/mailman/mm/var/data/postfix_vmap) is overwritten or deleted whenever a mail user is edited or updated in Virtualmin. This breaks Mailman list delivery, as the map must be managed exclusively by Mailman.

My Setup

On the system detailed above, I have Mailman 3 installed within a virtualenv, following Virtualenv Installation — Mailman Suite 3.3 documentation. The mail server is Postfix, and there are users - managed through Virtualmin as system users - which have individual email-addresses in the same domain as the list addresses.

To get the delivery to work initially, I have setup a domain with alias in mailman, such that mailman has its own domain lists.example.org for setting up its local delivery aliases, while the mailing list addresses are of the form listname@example.org.

A side note (just in case it’s relevant): The domain primarily used for emails (example.org) is an alias server of the domain primarily used for web services (let’s call it xmpl.org). On the alias server’s “Email Settings” page, Mail aliases mode is set to “Copy aliases from target”.
This has not been any issue in the past.

Mailman’s side

Mailman then generates 3 files to be added to the Postfix config (and their corresponding .db files):

postfix_domains

Location: /opt/mailman/mm/var/data/postfix_domains

lists.example.org  example.org

postfix_lmtp

Location: /opt/mailman/mm/var/data/postfix_lmtp

# Aliases which are visible only in the @lists.example.org domain.
some-listname@lists.example.org                        lmtp:[127.0.0.1]:8042
some-listname-bounces@lists.example.org                lmtp:[127.0.0.1]:8042
some-listname-confirm@lists.example.org                lmtp:[127.0.0.1]:8042
some-listname-join@lists.example.org                   lmtp:[127.0.0.1]:8042
some-listname-leave@lists.example.org                  lmtp:[127.0.0.1]:8042
some-listname-owner@lists.example.org                  lmtp:[127.0.0.1]:8042
some-listname-request@lists.example.org                lmtp:[127.0.0.1]:8042
some-listname-subscribe@lists.example.org              lmtp:[127.0.0.1]:8042
some-listname-unsubscribe@lists.example.org            lmtp:[127.0.0.1]:8042
# ... equivalently for any other lists

Port 8042 (changed) is where Mailman listens to incoming mails, as defined in /etc/mailman3/mailman.cfg:

# ...
[mta]
lmtp_port: 8042
# ...

postfix_vmap

Location: /opt/mailman/mm/var/data/postfix_vmap

# Virtual mappings for the @example.org domain.
some-listname@example.org                         some-listname@lists.example.org
some-listname-bounces@example.org                 some-listname-bounces@lists.example.org
some-listname-confirm@example.org                 some-listname-confirm@lists.example.org
# ... etc

Postfix’ side

main.cf

To tie Mailman’s mapping files into Postfix, the relevant parts of it’s /etc/postfix/main.cf are:

alias_maps = hash:/etc/aliases
virtual_alias_maps = hash:/opt/mailman/mm/var/data/postfix_vmap,
    hash:/etc/postfix/virtual
transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp
local_recipient_maps = proxy:unix:passwd.byname,
    $alias_maps,
    hash:/opt/mailman/mm/var/data/postfix_lmtp,
    $virtual_alias_maps
relay_domains = hash:/opt/mailman/mm/var/data/postfix_domains

If I remember correctly (it’s been a while by now), the proxy:unix:passwd.byname-entry in local_recipient_maps was necessary to reinstate delivery to the user mailboxes.

Current behavior

Good: Email delivery works to individual users and to mailing lists.

Bad: Whenever I edit a user via Virtualmin, it apparently rewrites all files mentioned under virtual_alias_maps in Postfix’ main.cf. That means the changes to the user are reflected in /etc/postfix/virtual, but it also wipes out all entries from Mailman’s postfix_vmap.

Basically, I now need to run mailman aliases on the server whenever I as much as touch a user in Virtualmin to restore the postfix_vmap contents - hoping that no mail to a mailing list is sent in the meantime. I can still set up a cron job to do that regularly in case I forget it some day, but even that is not a really great option.

What I tried

Initially, I had the opposite order in virtual_alias_maps which made Virtualmin write new entries into Mailman’s postfix_vmap file (I’m not entirely sure whether is also wiped all other entries then or not). This is of course also not helpful because then Mailman will remove these new entries whenever it updates the files (periodically, after a change or with the mailman aliases command).

With some help of the usual kind/enthusiastic but ultimately unhelpful AI, I have tried:

Setting postfix_virtual_alias_maps=/etc/postfix/virtual in /etc/webmin/virtual-server/config had no effect. Virtualmin still parses and rebuilds all maps listed in main.cf.
The same happened when I tried setting postfix_virtual_maps=/etc/postfix/virtual instead - AI’s first suggestion but I figured since there was a name change in main.cf, either might be worth a shot.

I’ve tried to find any clue on the internet (and in this forum) how to prevent Virtualmin from overwriting one of the files in virtual_alias_maps to no avail.

I have not yet tried to symlink Mailman’s postfix_vmap file instead and configure the link location in the main.cf. I think that’s also a poor solution if it even works.

Any help will be appreciated. And I’m ready to give more infos if necessary or helpful to fix this issue.

Ultimately, if currently impossible, I might consider a feature request to add an option which (virtual) maps from Postfix Virtualmin should never touch.

Thanks for your time!
Elpy

PS: I’ve been using Virtualmin for probably over 10 years and I’m still in love with it 95% of the time!

2

Where is this located? All of my Mailman3 stuff is under transport mapping.

3

@ID10T:

This one /opt/mailman/mm/var/data/postfix_vmap which is included in Postfix’ virtual_alias_maps =

4

Would you be able to share more details?

Maybe my setup is just quirky and you’ve found a better way that I could adapt as well.

5

It’s been a couple years. It was the Debian packages at the time. I have no /opt/mailman. There was a quirk in that a default alias generated by Virtualmin seemed to be a catchall and kept the configuration from getting as far as the transport mapping though. Once I removed that alias, it worked. At least that’s kind of how I remember it but that was close to 2 years ago now.

1 Like
6

That’s a bug-like for sure. I’ll ask @Jamie to take a look, as the Postfix maps parsing code probably needs to understand that there can be multiple files and we only want to edit the “main” one (in /etc/postfix) and leave third party ones alone.

1 Like
7

Yeah, Virtualmin will look at all the map files when editing a user, even if there are multiple.

@ElpyDE what is the actual change Virtualmin makes to /opt/mailman/mm/var/data/postfix_vmap ? If it only contains entries related to mailing lists, these shouldn’t be touched when editing a user.

8

In the current configuration, all entries from that file are just deleted - curiously though, the comments at the top of the file remain.

All of the entries in that file are indeed only from Mailman3 - that is the idea I was going with, because that makes sense to me, and because Mailman3 generates it that way (periodically and as needed when lists change).

I should clarify that all the left-hand side entries (...@example.org) are in a Domain that Virtualmin typically controls. So it might be thrown off by that? The right-hand side entries lead to the “alias domain” @lists.example.org which Virtualmin has no clue about.

Another thing I noticed…

When the order in /etc/postfix/main.cf of the two paths in virtual_alias_maps was reversed (such that /etc/postfix/virtual comes first and Mailman’s /opt/mailman/mm/var/data/postfix_vmap is second), like this:

# main.cf
virtual_alias_maps = /etc/postfix/virtual, /opt/mailman/mm/var/data/postfix_vmap

then Virtualmin would - when saving a mail user - also (a) delete all entries created by Mailman in /opt/mailman/mm/var/data/postfix_vmap, (b) remove all entries related to that user from /etc/postfix/virtual, and (c) write all the entries for that user at the end of /opt/mailman/mm/var/data/postfix_vmap instead (including those deleted from /etc/postfix/virtual plusminus changes made on the “Edit User” page).

From this, I figured, I must keep /etc/postfix/virtual last in virtual_alias_maps for Virtualmin to put all changes there. But that didn’t stop it from wiping out Mailman’s /opt/mailman/mm/var/data/postfix_vmap - or more precisely, if I’m correct, remove all entries it doesn’t really know how to deal with.

9

Thanks, that’s useful info!

Also, what exact action in Virtualmin triggers this? Is it clicking on any user on the “Edit Users” page and then clicking Save?

Also, if you do edit a user and open the “Email settings” section, are any addresses related to your mailing lists shown?

Another question - is this domain with Mailman an alias domain in Virtualmin?

10

Good questions are also helpful :smile:

Let me maybe summarize my domain setup again as overview, because that might have been confusing to readers, myself and Virtualmin (and maybe not complete and fully accurate before):

example.org - The main domain for website and mail users, is a Virtual Server in Virtualmin.
example-with-dashes.org - The domain we use for email addresses, is an Alias Server of example.org in Virtualmin, with the Mail aliases mode is set to “Copy aliases from target”. :left_arrow: I suspect this might be where the issue comes in.
lists.example.org - This is for the Mailman web interface (proxy settings in Apache without Virtualmin’s involvement) and this is Mailman Alias-Domain for all lists.

So, users are created in example.org and get a name@example.org email address, plus the name@example-with-dashes.org alias created by Virtualmin. But for style/communication reasons, we encourage the use of name@example-with-dashes.org only.

For mailing lists, I have only used the listname@example-with-dashes.org form. This would then create entries (automatically generated by mailman):

In /opt/mailman/mm/var/data/postfix_vmap:

listname@example-with-dashes.org  listname@lists.example.org

And make sure those are delivered to Mailman’s process with LMTP, in /opt/mailman/mm/var/data/postfix_lmtp:

listname@lists.example.org lmtp:[127.0.0.1]:8042

As far as I can tell, editing any user in the main virtual server (example.org) triggers this. The same also happens when I don’t change anything and hit save. And it happens on user creation and user deletion as well.

It does not happen when I create/edit/delete a user in a different virtual server that is unrelated to the above mentioned. :left_arrow: Had to try this explicitly, because there aren’t other domains typically used for email on this server. But it makes me believe that Virtualmin removes all of the virtual aliases in example-with-dashes.org that have no matching entry in example.org

Maybe/probably because of the “Copy aliases from target”-setting, I’m not seeing any of the Mailman-related alias anywhere. That is, neither on any Edit User page, nor on the Mail Aliases page of example.org. The Mail Aliases page for example-with-dashes.org (by magic of typing in the address as it’s not in the menu) says “No mail aliases have been created in this domain yet.”

So, it seems that Virtualmin doesn’t recognize the Virtual Aliases for example-with-dashes.org, or deems them as not legitimate when updating. I understand that this is much easier for the “Copy aliases from target” mechanism - presumably just removing all virtual aliases for the Alias Server domain that can’t be explained by what’s found for the parent Virtual Server domain.

lists.example.org is not an alias domain known to Virtualmin. DNS just comes from the *.example.org wildcard entry, and the Apache config was added manually. At the time, I thought it would be good to keep Virtualmin “in the dark” about that domain.

Next step for me…

I will try to add a new list in Mailman for example.org and see what happens to that. I haven’t done that before, because I wanted all of the email addresses to be shown like listname@example-with-dashes.org and it doesn’t seem like Mailman supports it easily for lists to “be” in one domain but also “present” as if they’re in another one.

Again, thank you so much for your support! And sorry for the long post again as I’m trying to also understand what’s going on.

11

Ok this is the issue - when an alias domain has email aliases copied from the target, anything that doesn’t exist on the target domain (such as your special Mailman virtual aliases) will be over-written. The quick fix would be to turn off Copy aliases from target , although this does mean that other non-Mailman aliases won’t be synced from example.org

1 Like
12

That makes a lot of sense, unfortunately. At least I understand what’s going on now.

I have tried adding a mailing list “directly” for the main Virtual Server domain, example.org. Other than in the current setup, there is not Mailman Alias Domain (like lists.example.org) for this.

Mailman then only generates entries in /opt/mailman/mm/var/data/postfix_lmtp like

test123@example.org  lmtp:[127.0.0.1]:8042

Virtualmin does not show them, nor touch them. But it also doesn’t work - Postfix sends a “User unknown in virtual alias table” Error when I try sending to that mailing list. That’s probably not entirely surprising because in /etc/postfix/main.cf it’s only set as a transport map:

transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp

I guess I’ll have to reconsider how to change my domains around to make is work that mailing lists and regular users can live under the same domain.

Thanks for all the support. Glad it’s at least explained. When I find a good solution, I’ll post it here.

13

Ok thanks! Let us know what you find ..

14

@ElpyDE Was this a Ubuntu package install? I’m just curious. When I did my install the Ubuntu folks were waiting for some upstream change in Debian. As far as I know Debian doesn’t use the /opt directory.

15

No Ubuntu-provided packages put things in /opt. Third-party packages on EL distros often use /opt, and source installs of software may also use /opt (but /usr/local is also common in that case).

I generally recommend installing everything using packages from the native package manager whenever possible. When you install from source, you’re not on the hook for tracking new versions, keeping up with security updates, etc. I think it’s wise to outsource that to your OS vendor whenever possible. It’s free time, and time is the only thing you can’t buy more of.

16

I’ve not used the Ubuntu package but installed “manually” with a virtual environment - following Virtualenv Installation — Mailman Suite 3.3 documentation.

At the time, I just wasn’t sure how it would play with multiple domains when installed as system package - and that will be a project on another server. From what I’ve seen now, this might still be okay, once the mail mapping is sorted out nicely. Mailman and the Mailman web suite (running on Django) seem to be designed with multiple domains in mind.

17

For now, I’ve set up a script to regenerate Mailman’s alias tables whenever a change is detected. The directory /opt/mailman/mm/var/data, where all of Mailman’s alias tables live, is under git version control, so that’s relatively straight-forward.

It will also do a double-regenerate when there is a change in Mailman - because that will change the date in the comments in these files and then renew them once again before committing. So it’s not 100% clean, but rather living on the safe side.

This script runs as cron job every minute as user “mailman”:

#!/bin/bash
set -euo pipefail

# Configuration
DATA_DIR="/opt/mailman/mm/var/data"

# Change to the Git-tracked directory
cd "$DATA_DIR"

# Check for changes in postfix_vmap
if ! git diff --quiet; then
    echo "[$(date)] Detected change in postfix_vmap. Regenerating aliases..."

    # Re-generate aliases
    /opt/mailman/venv/bin/mailman aliases

    # Commit changes
    git add .
    git commit -m "Auto-update aliases on $(date -Iseconds)"
    echo "[$(date)] Changes committed."
fi

I’ve experimented a little bit with other solutions through Mailman (using some other domains and its alias domains, but with no success so far). In the long run, we might end up making the example-with-dashes.org domain, which is used as primary domain for email addresses, the main virtual domain in Virtualmin - and example.org an alias of that in Virtualmin. Maybe that will simplify the mapping and generation/deletion of virtual aliases.