I run a Rocky Linux 8.8 server using Virtualmin (v 7.7). On this server I have a virtual server that has an email address that can receive and send emails - this has all been working fine for the last year, but sometime in the last ~week, emails to my address have bounced back to sender, with an error:
This is the mail system at host DOMAIN@SERVER.COM.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
<USER@DOMAIN> (expanded from
<user@VIRTUAL-SERVER>): can't create user output file
Checking the log file in the server at /var/log/procmail.log, I see:
Disk quota for username@DOMAIN-EXAMPLE.COM has been reached.
procmail: Program failure (73) of "/etc/webmin/virtual-server/lookup-domain.pl"
From test-email-sender@email.com Sat Jul 29 10:54:07 2023
Subject: #listentome
Folder: /dev/null 3316
Time:1690638847 From:test-email-sender@email.com To:user@virtual-server.com User:username@DOMAIN-EXAMPLE.COM Size:3317 Dest:/dev/null Mode:None
Which very clearly says “Disk quota for [user] has been reached.” BUT I cannot find any instance where the disk quota is exceeded in this server, at all.
\[root@server \~\]# quota -v username@DOMAIN-EXAMPLE.COM
Disk quotas for user username@DOMAIN-EXAMPLE.COM (uid 1007):
Filesystem blocks quota limit grace files quota limit grace
/dev/vda1 0 51200 51200 6 0 0
/dev/vdb1 196 0 0 42 0 0
\[root@server \~\]# quota -g username@DOMAIN-EXAMPLE.COM
quota: group username@DOMAIN-EXAMPLE.COM does not exist.
My understanding (and what is shown in Virtualmin!) is that the 0 in the quota and limits above means unlimited quota. Checking EVERY user and group using repquota -avug shows no users or groups are over quota. Using df -h and df -i (as noted in the answers of a StackOverflow question) show none of the disks are at or anywhere near 100%. I did find a similar issue from 2011 in the Virtualmin forums, but the comments didn’t really pertain to my issue - the root user and group doesn’t have any hard or soft quotas on the server (all “unlimited”).
For further context, there are MariaDB databases on this server and other virtual servers to produce websites. I’m the sys admin for this server in my department but have extremely limited Linux experience and am constantly learning things as I go with this system that I inherited.
if i use that quota command it shows 0 so its incorrect, please you VM tools to check quota’s
quota -v user@mydomain.au
Disk quotas for user user@mydomain.au (uid 1160):
Filesystem blocks quota limit grace files quota limit grace
/dev/vda1 136 51200 51200 33 0 0
if you like use command line use
virtualmin list-users --domain yourdomain.au --user username
In Webmin under Disk Quotas, that specific user (the user associated with the email address) indeed has “Unlimited” hard and soft limits.
In Virtualmin under that domain > edit users, indeed the same user shows Unlimited quota in the home directory. In fact, everything I see in Virtualmin and Webmin shows that no quotas are exceeded for any user or group.
I hadn’t tried using virtualmin via the command line yet, so I gave that a go for the specific mail user that I’m having issues with:
[root@server ~]# virtualmin list-users --domain domainname.ca --user username
User Real name Mail FTP DBs Quota
----------------- -------------------- ---- ---------- ---- ---------------
username Real Name Yes No No Unlimited
I’m still seeing emails being bounced back, with an error in /var/log/procmail.log of the following:
Disk quota for user@domain.ca has been reached.
procmail: Program failure (73) of "/etc/webmin/virtual-server/lookup-domain.pl"
From test-address@mail.com Mon Jul 31 20:15:04 2023
Subject: test email
Folder: /dev/null 3307
Time:1690845305 From:test-address@mail.com To:user@domain.ca User:user-domain.ca Size:3308 Dest:/dev/null Mode:None
Here’s a snippet of whats in /var/log/maillog - if this is useful in anyway, I can get a more complete log here?
Interesting - thanks for noticing that. Yes, the user was originally created via Virtualmin (at least, I’m pretty sure). The user was created roughly a year ago and has been working fine since then… until roughly the last week!
[I’ll come back here shortly if I can find any older maillog or procmail.log logs from when it WAS working earlier in July or before] * editing to add, I found /var/log/procmail.log from when it was working!:
That totally seems to be the issue - that /dev/null folder. Now… how do I fix that…?
The “Inbox mail file” seems to be set correctly in virtualmin, in the attached screenshot. This also very interestingly shows that the last email successfully received was July 27. Currently, the mailbox is empty.
Adding more context, thanks to stefan1959’s excellent find with that /dev/null folder: This does indeed seem to be a server-wide issue, not just a single-user issue. I have other email accounts on the server that I now realized are bouncing emails back, too! All of the /var/log/procmail.log entires are showing that same Folder: /dev/null issue instead of a reasonable path to a mail folder like /home/domain/homes/user/Maildir/new
My /etc/procmailrc file has not been changed since long before the mail stopped worked, and looks like this:
I see a small difference at the end of the file. Maybe someone on staff might know what might be going on. I sort of think its a variable name not working maybe. Please backup your virtual machines as a fresh reinstall might be the easiest solution. and seeing your using a older OS. If you can create a test machine and reinstall the backup and see if that fixes things. My procmailrc.
Ah, shoot, I’m REALLY hoping I won’t have to do a whole reinstall… Fingers crossed there’s some other brains that can take a look at this problem, too.
I’m currently trying to figure out exactly when the last email received was and then will try to “undo” and package updates from that point, seeing if whatever is broken may become fixed. Before a ~week ago, all was perfectly fine! I’ll update with relevant information when I can, though it may be in the morning.
Adding to this: emails can be forwarded through the email address where they bounce back. For example, if I send an email to username@domain.ca which also forwards to secondemail@anotherdomain.ca, the original sender gets a bounceback message AND that email gets successfully forwarded to secondemail@anotherdomain.ca.
I found the last email that was successfully received for certain was midday July 19, though I’m not sure if emails were received to any other email account after that one. I just did yum history rollback x to an update done on July 18. However, after restarting the server the emails are still not going through.
I created a new user - “testuseremail@domain.ca” and emails to that address get bounced back with a slightly different note(?) - I also had this “testuseremail@domain.ca” email forward to a microsoft-provided email address so maybe there’s an issue there? (I have not had ANY time to troubleshoot this specific thing, but I’ll paste it here - it’s currently 10pm and I’m at the end of 12 hours of work! Will return in the morning.)
This is the mail system at host server.ca.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
<testuseremail-domain.ca@domain.ca> (expanded from
<testuseremail@domain.ca>): mail forwarding loop for
testuseremail-domain.ca@server.ca
I haven’t yet done Recheck Configuration - I’m not sure where to find that (yet) but I will look into it in the morning. Thank you!
Oh my goodness gracious, I fixed the issue. It WAS a quota issue but it wasn’t showing up anywhere with “disk quotas” via command line or in Disk Quotas in virtualmin. I went to Virtualmin for the domain > Edit Virtual Server > Quotas and Limits.
The limit HERE under “Quotas and limits” was set to 1Gb and it had gone over that limit. Switched this quota to unlimited and voila, emails can be received again. Good grief. To think of how many hours I spent on this… Fixed now! Thank you very much to those that were helping me along.
