Postfix virtual map escapes some @

SYSTEM INFORMATION
OS type and version AlmaLinux release 9.1 (Lime Lynx)
Virtualmin version 7.5
Postfix version 3.5.9
Roundcube version 1.6.0

It was a bit overdue. I’ve finally moved my virtualmin system running on centos to almalinux. During the move, I made some different configuration decisions from my old system.
One of those decisions is now giving me a headache.
On the previous system, account names were created as “USER.DOMAIN”. Because it was so confusing for my elderly parents to remember, I changed it to “USER@DOMAIN” on the new server. This way they only had to remember their e-mail address to login.

In preliminary testing after moving a couple of domains, I noticed I could not log in to roundcube, the webmail client we’re using. Username was correct, password was correct. Logging in through a different mail client (K9 on android, IMAPS + SMTP) works just fine.
So I tailed the maillog to see what error was being thrown. I got this:

Jan 22 07:54:53 SERVER dovecot[69792]: imap-login: Disconnected: Connection closed (auth failed, 1 attempts in 2 secs): user=<USER\@DOMAIN>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured, session=<C99bwtTymqp/AAAB>

I notice the @ is being escaped when attempting to login with imap. That can’t be right I thought. I checked the login form, sure enough. No escaping of the @ there.

I reviewed the roundcube configuration file and noticed the postfix virtual map is linked in there, so I go and have a look at the virtual map of postfix (/etc/postfix/virtual).
There I notice that for each user which has “Additional email addresses” (Virtualmin => edit users => user => E-mail settings => Additional email addresses), the destination (second field on the line) has an escaped @. All other accounts, who have only a single address, are fine.

so, of course, first thing I do is do a sed replace of all those escape chars and save the file, run “postmap /etc/postfix/virtual” and be over with it.
After this, I manage to log into roundcube. other imap clients were also fine. Sending mail to mail-tester gave splendid results. All seemed fine with the world.
So I continued and migrated all the remaining DNS and called it a day.
What I also did that day was copy some of the custom configuration I’d have made in postfix over the years over to the new postfix config. This included the following:
DISCLAIMER: DO NOT USE THIS, IT DOESN’T WORK!

smtpd_recipient_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination,
    reject_invalid_hostname,
    reject_unauth_pipelining,
    reject_non_fqdn_sender,
    reject_unknown_sender_domain,
    reject_non_fqdn_recipient,
    reject_unknown_recipient_domain,
    reject_rbl_client relays.ordb.org,
    reject_rbl_client blackholes.easynet.nl,
    reject_rbl_client cbl.abuseat.org,
    reject_rbl_client proxies.blackholes.wirehub.net,
    reject_rbl_client bl.spamcop.net,
    reject_rbl_client sbl.spamhaus.org,
    reject_rbl_client opm.blitzed.org,
    reject_rbl_client bl.spamcop.net,
    reject_rbl_client b.barracudacentral.org,
    reject_rbl_client dnsbl.njabl.org,
    reject_rbl_client list.dsbl.org,
    reject_rbl_client multihop.dsbl.org,
    check_policy_service unix:/var/spool/postfix/postgrey/socket

What I did not test until the next early morning was “receiving e-mails, does that work?”. I woke up to an empty mailbox. No spam. Strange, Perhaps the new server has better anti-spam (it does)? But no daily newsletters, no status mails. Something must be wrong.
I try to send myself an e-mail. Doesn’t arrive. Great.

Looking at the logs, first thing I notice is that all incoming mails are rejected. Mails from google and microsoft inclusive. I was puzzled. This seemed like an upside-down world.

Jan 21 06:16:18 SERVER postfix/smtpd[371935]: NOQUEUE: reject: RCPT from mail-ua1-f44.google.com[IP]: 554 5.7.1 Service unavailable; Client host [IP] blocked using sbl.spamhaus.org; Error: open resolver; https://www.spamhaus.org/returnc/pub/2400:cb00:114:1024::ac44:31d3; from=<USER@gmail.com> 
to=<USER@DOMAIN> proto=ESMTP helo=<mail-ua1-f44.google.com>

I commented out the rbl rejects in the configuration to avoid these kind of errors.

It did solve the rejects, but now I was presented with a whole new exciting error message:

Jan 21 08:29:44 SERVER postfix/error[381424]: 817D54021515: to=<USER@DOMAIN>, relay=none, delay=13, delays=13/0.01/0/0.01, dsn=5.1.1, status=bounced (User unknown in virtual alias table)

Splendid. User unknown in virtual alias table. I did grep for that particular username and can tell you, I got a positive return. Anyway, I had a feeling deep down that most of the changes I made to the postfix configuration could have broken things.
So I went back to the /etc/postfix/virtual file and looked at it again. No clue how to identify the accounts that need an escaped @ and the ones that don’t.
What I ended up doing is going in Virtualmin to all the accounts that had additional e-mail addresses and I added another bogus one for each account. This overrode the current file and all the escaped @s were back where they are supposed to be.

Looking at the logs, mails do now seem to find their way into the mailbox like they are supposed to. This unfortunately leaves me again with my original problem: I can’t log into roundcube.

I’d like to understand what’s happening. Why have some accounts their @ escaped in postfix virtual map and why do others not?

I have already figured out that roundcube is using the virtual map to select the account name for an e-mail account. Removing the virtual map references from the roundcube configuration resolves this, but also removes some of roundcube features. I’d prefer they would stay linked for identities.

1 Like

Hello,

Thank you for the excellent bug report and explanations!

I think you have accidentally removed resolve_dequoted_address set to no in your Postfix configuration. This can be set back in Servers ⇾ Postfix Mail Server: Address Rewriting and Masquerading page by setting Prevent use of @ in mailbox names? to No.

@Jamie, we should stop escaping \@ in Postfix virtual map file.

Hello @Ilia

Thank you for your quick reply!
It seems that this configuration is already as it should be.
I’m including a screenshot of the Address Rewriting and Masquerading page as I found it:

So I think we do need the @ to be escaped in the virtual file, or else Postfix will treat it as a forwarding address and probably create a mail loop. The real question is what Roundcube is doing in this case…

You’re working awfully hard to break stuff. That’s a lot of Postfix customizations that I’m unfamiliar with. :wink:

I don’t generally recommend use of reject_rbl_*. I put my RBL rules into SpamAssassin, so they can be a weight rather than a hard blacklist. Anything RBL you configure in Postfix you’d better be really confident the list is all spammers.

I don’t know why Roundcube would fail on valid email addresses…it’s not our software, but I don’t think anybody else has ever reported this problem, and I think email addresses with @ in them has been the default for a few years.

I think I’ve got a related problem. I just did a clean virtualmin install and changed one little thing in the Main.cf configuration file. I added:

smtpd_sender_login_maps = hash:/etc/postfix/virtual
smtpd_sender_restrictions = reject_sender_login_mismatch

I always used to do this on my old virtualmin servers (that didn’t use an @ in the e-mail usernames) to prevent an authenticated client from sending email from a MAIL FROM address that they do not explicitly own.

But now with the new Postfix, I got an error when trying to send a message:

Sender address rejected: not owned by user test@test.com

Could this also be caused by the escaped @ character in /etc/postfix/virtual ?

@Jamie I think roundcube reads out the postfix virtual map and tries to authenticate with the mailbox that’s used for the alias with which the user is trying to login. This would translate to user@domain

@Joe well, this configuration is from a few years (10?) back when I configured it. It used to work very well until I reinstalled the server to a new version of virtualmin (was on centos, it was a pain to upgrade or to convert it to alma, so did a reinstall)

I was not sure where to go to report this. I’m already very grateful with the support I’ve been receiving from the Virtualmin forums over time.
I’m not sure if the postfix virtual mapping is default in a roundcube installation. It was in the configuration file after I used the script to install.
– Edit –
These are the parameters I’m refering to by the way:

$rcmail_config['virtuser_file'] = '/etc/postfix/virtual';
$config['virtuser_file'] = '/etc/postfix/virtual';

@scriptex I don’t think this is related. Sounds to me you’ve migrated, just like me, from an old server to a new one and also changed the usernames in the process. Have a look in your passwd file and double check the usernames of the users you have installed. For me the posix user names were identical to the mail e-mail address (user@domain).
If you did a rsync or scp from your old server to your new server to retain the e-mails, you may need to update the ownership of the homes of these users.

@remibruggeman
I have not migrated. It is just a clean install on which I created a new domain.

The output of ‘cat /etc/postfix/virtual’ is :

test@test.com test\@test.com

For my old servers the ‘smtpd_sender_restrictions’ protection worked but now it does not work for this new server. I think that the server thinks that the mailboxes mismatch because of the backslash.

Rules have changed in newer versions of postfix.
Read here for updated rules: Sender address verification for all email

@Jamie, could you point to the Postfix documentation which suggests escaping @ with virtual file?

Based on plugin description:

/**
 * File based User-to-Email and Email-to-User lookup
 *
 * Add it to the plugins list in config.inc.php and set
 * path to a virtuser table file to resolve user names and e-mail
 * addresses
 */

We are the ones who is enabling this plugin. It’s disabled by default. I don’t really think that we need it actually.

The trick here could be is to use auth_username_translation to replace/remove \ from sent username.

The simplest solution would be is to edit roundcube/config/config.inc.php file and remove virtuser_file from plugins list.

1 Like

You can test this - try adding an entry to /etc/postfix/virtual like user@domain.com user@domain.com , and see if email gets delivered. For me, it just bounces…

I checked the docs… Everything around ‘reject_sender_login_mismatch’ stayed exactly the same I believe…

The problem is this:
When I authenticate as test@test.com and try to send mail from test@test.com, postfix always says 'Sender address rejected: not owned by user.’


Main.cf

smtpd_sender_login_maps = hash:/etc/postfix/virtual
smtpd_sender_restrictions = reject_sender_login_mismatch


/etc/postfix/virtual

test@test.com test\@test.com

I can confirm that it didn’t work for me either. I think we can just fix our install script and not add virtuser_file plugin.

Do you have an option resolve_dequoted_address = no added too?

:memo: I have just tried it and it worked for me with smtpd_sender_restrictions = reject_sender_login_mismatch option enabled.

Yes, I have this option added. See my whole main.cf below:

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
append_dot_mydomain = no
readme_directory = no
compatibility_level = 3.6

TLS parameters

smtpd_tls_cert_file = /etc/postfix/postfix.cert.pem
smtpd_tls_key_file = /etc/postfix/postfix.key.pem
smtpd_tls_security_level = may

smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level = dane
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

Main Config

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = mail1.domain.nl
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname, mail1.domain.nl, localhost.domain.nl, , localhost
relayhost = email-smtp.eu-central-1.amazonaws.com:587
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all
virtual_alias_maps = hash:/etc/postfix/virtual
sender_bcc_maps = hash:/etc/postfix/bcc
sender_dependent_default_transport_maps = hash:/etc/postfix/dependent
home_mailbox = Maildir/
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination
smtp_dns_support_level = dnssec
smtp_host_lookup = dns
allow_percent_hack = no
resolve_dequoted_address = no
tls_server_sni_maps = hash:/etc/postfix/sni_map
smtpd_tls_CAfile = /etc/postfix/postfix.ca.pem
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/smtp_sasl_password_map

Amazon SES CUSTOM

smtp_sasl_security_options = noanonymous
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes

SPAMExperts CUSTOM

smtpd_client_restrictions = check_client_access hash:/etc/postfix/access, permit_sasl_authenticated, permit_mynetworks, reject

Block cross-logins

smtpd_sender_login_maps = hash:/etc/postfix/virtual
smtpd_sender_restrictions = reject_sender_login_mismatch

And here is the content of /etc/postfix/virtual

domain@domain.nl domain
domain.nl domain.nl
hostmaster@domain.nl domain@domain.nl
abuse@domain.nl domain@domain.nl
postmaster@domain.nl domain@domain.nl
webmaster@domain.nl domain@domain.nl
test@domain.nl test\@domain.nl
other@domain.nl other\@domain.nl

Notice the backslash in the two records at the bottom… These are the users that I created via the virtualmin interface.

Btw, here’s the error I get when sending… Very strange…

<test@domain.nl>: Sender address rejected: not owned by user test@domain.nl.

We still need it when using regular format usernames though, right?

Okay guys, I am now 100% certain that my error is caused by the escaped characters in /etc/postfix/virtual.

What I tested:
I removed every backslash (the escaped @ character) from the file: /etc/postfix/virtual.
Then, I remapped the config file with command: postmap /etc/postfix/virtual

After that, I could send my e-mails like usually with the following restrictions in place:

smtpd_sender_login_maps = hash:/etc/postfix/virtual
smtpd_sender_restrictions = reject_sender_login_mismatch

If I put the backslash back, then I’ll receive the error again:

Sender address rejected: not owned by user test@test.com

So… Now I am lost… I am thinking about creating a Virtualmin Script (Actions upon server and user creation) that copies the /etc/postfix/virtual file to /etc/postfic/virtual_copy and removes every backslash from the copied file. Then I am able to use ‘smtpd_sender_restrictions’ like it is intended. But this is imho a dirty workaround that should not be necessary. If you have better ideas, please let me know.

Not really. I didn’t notice anything that would break aside from RoundCube couldn’t properly detect the domain name when sending mail. It sets From: address to something like username@localhost in case if only a username used for login (e.g. in case of a main domain user). Although, this can easily be fixed in RoundCube by properly defining your (user) email address. What else could break you think? RoundCube just sends and receives using credentials provided (i.e. uses SMTP and IMAP).

Well, yes, you could send mail but the problem is that mail is not getting received with this only change (i.e. replacing \@ with @ wouldn’t work for receiving mail).

That’s why I created a copy of the file to /etc/postfic/virtual_copy that’s only been used in smtpd_sender_login_maps = hash:/etc/postfix/virtual_copy.

Now I can send and receive. But like I said, this is a (dirty) workaround…

Instead, do you have any issues by disabling virtuser_file plugin completely by editing roundcube/config/config.inc.php file and removing virtuser_file from plugins list.