Since yesterday all local mail is failing to be delivered by procmail-wrapper on my Virtualmin system.
procmail-wrapper is up to date.
This is the error message in the logs:
Jul 17 16:25:57 hosting postfix/local[19974]: 11CB1C7A1F: to=<redacted>, orig_to=<redacted>, relay=local, delay=0.44, delays=0.43/0.01/0/0.01, dsn=5.3.0, status=bounced (Command died with status 127: "/usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME". Command output: sh: 1: /usr/bin/procmail-wrapper: not found )
There’s not been an update to procmail-wrapper and according to apt-get it’s all up to date.
The system is all up to date but I’m concerned something got updated that has caused this issue.
At the moment I’m getting no mail coming in at all.
Anyone got any ideas? The last virtualmin forum related post was related to an update of virtualmin in Jan 2022 but there seems no specific change that I can see here now.
It’s fairly urgent so I’d appreciate any insight anyone has.
And procmail is installed although it doesn’t look very recent:
root@hosting:~# procmail -v
procmail v3.23pre 2001/09/13
Copyright (c) 1990-2001, Stephen R. van den Berg <srb@cuci.nl>
Copyright (c) 1997-2001, Philip A. Guenther <guenther@sendmail.com>
Submit questions/answers to the procmail-related mailinglist by sending to:
<procmail-users@procmail.org>
And of course, subscription and information requests for this list to:
<procmail-users-request@procmail.org>
Locking strategies: dotlocking, fcntl()
Default rcfile: $HOME/.procmailrc
It may be writable by your primary group
Your system mailbox: /var/mail/root
root@hosting:~#
Basically this system has been working for a number of years and no recent changes (other than normal updates).
My assumption is that it’s something that procmail can’t find in the processing but we’ve made no changes and it just started doing this 24 hours ago.
Note that this is about the time the issue started but I’ve force re-installed procmail-wrapper using apt-get and the modify time on the binary is still the same.
Something clearly changed at that time though.
is there any command we can use to test the procmail-wrapper from the command line easily?
It’s not procmail-wrapper or procmail. It’s probably something procmail is trying to run that has changed.
Look in the procmail.log for clues. And, yes, there is a way to run it manually. Running procmail-wrapper requires very specific options (for security reasons, since it runs first as root and switches to the user), so it’s a little tricky and I’d have to look it up. But, testing procmail would probably find the problem just as readily (I don’t think the problem is either procmail-wrapper or procmail, so running procmail is almost certainly just as helpful as running it through procmail-wrapper). There are docs on the web about troubleshooting procmail. I’ll follow up when I have more time, if googling doesn’t get you there.
The weird thing is procmail.log has had nothing written to it since this started happening so it’s not getting as far as logging anything which suggests something is broken along the way … the log looks to be set early in the procmail script so this suggests something is not running properly.
Ok so I restored a copy of an old version of procmail-wrapper from another Virtualmin server and that has resolved the issue. The version isn’t the latest in the repo. I’m going to restore from a week old backup and see how that goes.
What’s weird is I did a reinstall of procmail-wrapper and it looked to have replace the file but obviously there was something else at play.
One of those mysteries though. At least we narrowed it down to the wrapper script. I’ll post what happens with the restore here later for completeness in case anyone else has this issue going forward.
Ok I restored a backup from a week ago and all ok. So something happened to the wrapper - no idea what but stopped it working. Anyway hopefully someone else finds this useful!
Thanks @Joe for the help - helped me to get to the right solution in the end.
Which version of procmail-wrapper do you now have? (dpkg -l procmail-wrapper)
The last update, and the only update in many years, to that package was a security update. You cannot safely run the old version, but there was a build of it that wouldn’t run on i386 architecture, maybe even with this error. But, the latest version found in all of our repos should be the correct version and it should run on both i386 and amd64/x86_64.
Edit: Oh, but you restored from a backup…if you just restored the specific binary without updating the package manager metadata, if may not know what version you have. That’s not recommended, as it leaves you unable to easily track versions of stuff you have installed (and that’s the only reasonable way to track what security issues you may need to address).
Ok so the restore is the od version (not the 2022 patched version). I’ve deleted and re-installed the update version and in all cases the latest wrapper does not work and logs this error.
The unpatched version works just fine. SO it must have been an update that happened at the weekend that caused this.
So any idea how I can work out how to get out the correct version? It installs fine and appears to match the amd64 platform of the server.
But clearly it doesn’t work.
So now I have mail but turns out the system is (and was) not secure
Oh! You’re on a several years EOL distro! You have a lot of security problems, not just this one.
For our packages, you can update to using newer repos. Grab the latest version of the install script from your software licenses page on our website, and run it with the --setup flag. Make sure you use a current version (not any old ones you have laying around…--setup is newish and old versions will just try to install Virtualmin, which you don’t want to happen).
e.g.
/bin/sh virtualmin-install.sh --setup
This will change from the EOL old Virtualmin 5 (and earlier) repos to the still-maintained Virtualmin 6 repos (Virtualmin 7 repos are not directly compatible with your old installlation, but the 6 repos will be maintained for a couple more years, and they get all the same packages as the 7 repo).
For all the other packages that are out of date and have unpatched security bugs, you’re on your own (you’ll either need to pay Ubuntu the $75/year or upgrade to a maintained Ubuntu version…20.04 is the oldest maintained version, I believe, as 18.04 reached EOL a few months ago).
Thanks @Joe yes well aware of the release situation. I’m planning to migrate to a clean build server rather than upgrade as things normally don’t survive the upgrade process and easier to manage on a domain by domain basis.
