| SYSTEM INFORMATION||
| Operating system | CentOS Linux 7.9.2009 |
| Webmin version | 2.100 |
| Usermin version | 2.000 |
| Virtualmin version | 7.1 |
| Authentic theme version | 21.02 |
I have been struggling with the Usermin email filtering for a few years now. I use the filters to move messages into other folders so that they can be managed more easily. One of the biggest is a ‘Bulk’ folder where generic messages go (that I scan quickly) that are then automatically deleted after a couple of days. Of course I have others that are more specific that are not automatically cleaned by the system.
These are pretty well all using the FROM header to filter and originally the ‘Starts with’ and ‘Contains’ portion of the filter worked. I could never get the ‘Ends with’ option to work. Recently however, the ‘Contains’ option no longer appears to function properly. This leaves me with having to create an exact match for each filter. I have also tried to edit the procmail file directly in order simplify the filters. This may work but in order to know they work I have to wait for an incoming message that matches. I don’t see any way to test the procmail filters like I can using the Usermin filter interface. Does anyone know how I can easily do this?
For several years the filters worked fine the way I had them set up but recently I began to notice that new filters I added no longer moved messages when they were delivered and when I went in and tested them manually using the Usermin interface they also reported no messages found.
I ended up cleaning out the whole filter list (hundreds of entries) and starting fresh thinking that maybe something in the file was corrupt. The first filter I added failed as before. It took me quite some time to figure out that in order for me to get it to work I had to first select the ‘Show Matching’ button. It showed the messages in the Inbox.
Then I had to go back into the filter and select the ‘Apply to’ Button. Only after doing those steps did it find and move the messages. I also noticed that a new incoming message that met the criteria was not flagged and moved. I have yet to see if it will be flagged after doing those other steps.
It certainly would save a lot of steps if these buttons were not inside each filter but were at the bottom of the list and were used in conjunction with the filter(s) that were selected.
So right now it appears to me that the ‘Apply to’ button does not work on its own and that there may be some coding that is missing with the ‘Apply to’ procedure since it does work after the "Show Matching’ button has been selected.
I will have to do more testing to see if incoming mail is actually being redirected based on the filters I have set up so far.
If there is a way to test a Procmail filter on the inbox please let me know.
Rather than Edit the original I will add my new findings here
New mail came in that matched my existing filters but the filters do not appear to be working unless I run them manually. No messages were filtered even though there were direct matches to existing filters.
I created a filter in the procmail file manually (through Usermin ‘Manually Edit Configuration’)
Added this entry
:0
* ^From:.*staples.ca.*$
$HOME/Maildir/.Bulk/
I tested the regex using an online tester against the source text of the email. It located the line without a problem.
When I view the filter using the Usemin filter option it appears to display the info properly
but does display the extra .* on the end of the line and changes to ends with. I assume the $ at the end changes it to “ends with” and that if I just used .*staples.ca.* it would display “Contains”.
However neither of these options produce any results when I run the filter on the Inbox.
I can only assume that my procmail setup is not working properly. Could an upgrade over the last year or so broken the setup?
How can I check my system for issues related to this?
Why does the manual run of the usermin filter work after I first run a search (Show Matching Email in Folder button) but finds nothing if I don’t run this first?
Why does only a direct match work for the filters and not partial matches?
That sounds like we’re wrapping something in quotes that shouldn’t be, so the regex is being collapsed into a string match (against a string that’ll never match because nobody titles their email /^From:.*staples.ca.*$/.
I could imagine one of our rounds of edits to make the code more friendly to strictures and warnings might have caused it.
Looks like the issue is that while Procmail supports regexps when searching for email, Usermin doesn’t (and possible Dovecot doesn’t either). So testing the filter doesn’t work as expected …
It appears to me that the Show Matching email applies a search and that is stored in the Search folder.
The Apply To Email button then applies the filter to the “Search Folder” only, and NOT the selected folder (in my case Inbox)
This explains why the Filter works only after selecting the Show Matching Email button before selecting the ‘Apply To Email’ Button and does not work when the first button has not been selected first.
Of course this is only on exact matches and still does not work on ‘Contains’ and Ends with’ (the standard regex is not working as before.)
Maybe the dropdown attached to the button ‘Apply to Email’ was never linked properly in code?
It might also explain why the filters do not work with new messages, IF this is the same mechanism used for incoming messages. If new messages use a different mechanism it would not explain the failure of the filters.
Have you tested the ‘Contains’ filter directly from the Usermin filter dialog? Mine does not find any message using ‘Contains’ it only works with ‘Starts With’
And the ‘Starts with’ filter will not move a message unless I first select the ‘Show Matching Email’ before I run the ‘Apply to Email’ button.
None of my filters are working outside of running them manually through the Usermin filter dialog. According to the Procmail logs, Procmail appears to still be working for redirecting messages to individual mailboxes and dealing with Spam etc, just not the local usermin filters.
Maybe I screwed something up in usermin?
That’s just it. I haven’t edited anything. I thought that maybe a server update might be the issue. Any chance you know what version of Perl you are currently running
Ok I found the cause of this issue … the problem is that the mail search supported by IMAP isn’t the same as procmail, which trips up Usermin when showing the messages that match a filter. We’ll fix this as best we can in the next Usermin release…
the only difference I have is not using a regex so I am using
contains and in the match box staples.ca and not using a regex this may be not what you want to do but it does move/delete all messages from the inbox containing the supplied string in this case staples.ca. However none of the buttons at the bottom of the form (Show Matching Emai/Apply to Email) only show 0 results, but the filter actually works
There are several characters that need to be escaped to be taken literally (at least outside char classes):
Brackets: []
Parentheses: ()
Curly braces: {}
Operators: *, +, ?, |
Anchors: ^, $
Others: ., \
In order to use a literal ^ at the start or a literal $ at the end of a regex, the character must be escaped.
Some flavors only use ^ and $ as metacharacters when they are at the start or end of the regex respectively. In those flavors, no additional escaping is necessary. It’s usually just best to escape them anyway.
Yes, the Procmail filters are all correct and they do work when put through Procmail. The real issue is within the Usermin filtering. Usermin appears to apply the filter correctly when it it used to search for filtered info (Show Matching Email in Folder) but fails when you try to actually apply the filter (Apply to Email in Folder.)
The issue is not a big one but it does make it more difficult to clean up any existing messages. The simple work around is to select ‘Show Matching’ first which places the found items into the ‘Search Results’ folder and then select ‘Apply’ which works on the found messages. It requires several steps to do this though and since its a browser interface the movement between screens and refreshes are a little slow.
It could be faster if these buttons were placed outside each filter dialog (at the bottom of the main filter screen) and worked on any selected filter.
Right now I am just waiting for a fix to the existing button.
Since I mostly use the Thunderbird Client I can live without it but it would be nice if Usermin fixed this because it is a pretty nice web based mail client.
I would also like a little more functionality added to its Automatic Folder Clearing. This is a quick and handy filter. Thunderbird has a similar function in its Folder Properties.
Something like the following would be a start but I’m sure it could be improved quite a bit. The ‘Read only’ is not very clear. It means do not delete messages that have not yet been marked Read.
I move spam messages into a specific folder but then have to manually go into Usermin and mark it as Spam and delete. An automated system would allow me to set it up and then use whatever client I want to do the task of moving the spam into the designated folder.
The Mark all messages as spam in folder would do the same job as the Delete Spam button currently does when a message is viewed (learns and deletes) or the Mark as Spam and Delete function from the message List works.
The Move to folder choice would allow you to move the existing messages rather than Delete them as discussed in other threads. My mock-up shows Inbox as a default but it would be better to make the Spam folder the default.
Yes, that was the idea but that wording is not very clear. I was too lazy to redo it.
Something like ‘Delete only messages that have been read’ would have been clearer or reverse it and say
Do NOT delete unread messages
Obviously it would require more thought if it were to be implemented.
