Dovecot setup incompatible with full text search

SYSTEM INFORMATION
OS type and version Ubuntu 24.04.1
Webmin version 2.202
Virtualmin version 7.30.4
Webserver version OpenLiteSpeed 1.8.2
Related packages Dovecot, FTS Xapian

I’ve been struggling for some time to get full text search working with Virtualmin, using FTS Xapian. See Which mail_uid and mail_gid · Issue #173 · grosjo/fts-xapian · GitHub

The developers have generally been very helpful but have concluded “…basically, emails are not handled by dovecot for reception, only for reading…Suggest you clean-up your setup to remove all the errors and inconsistencies, including webmin, so you have a proper setup…lmtp is not even configured on dovecot” which basically means they don’t understand how it is set up and they’ve given up.

Has anyone managed to get any sort of full text search working? Is there any chance of future support?

given the real and justifiable concerns over privacy I am not sure if and why we should be doing any search of emails? (or for that matter any user content) :male_detective: :secret:
… but I’m sure there must be some tool to perform such an inspection :eyeglasses:

The idea is that people can search their own email archives from their phones. It’s pretty essential functionality for some people, and it has always been possible from desktop clients like Outlook and Thunderbird but not from phones (due to limited storage space).

Gmail has allowed this for years, but I’m trying to move away from Gmail precisely because of the privacy concerns.

On the face of it, FTS Xapian does what I want, by generating an index for each user that is accessible via IMAP, but no-one can get it to work. :frowning:

Another solution would be to access webmail (Usermin) from a web browser but it’s much too slow.

Not sure about yours but set it up as you wish for YOUR purposes. Webmin/Virtualmin can’t/shouldn’t set up for plugins not used.

root@main:/etc/dovecot# grep -ir lmtp
conf.d/auth-system.conf.ext:  # LDA and LMTP needs to look up users only from the userdb. This of course
conf.d/10-mail.conf:# example LDA/LMTP while delivering large mails or zlib plugin for keeping
conf.d/10-director.conf:# Enable director for LMTP proxying:
conf.d/10-director.conf:protocol lmtp {
conf.d/10-master.conf:service lmtp {
conf.d/10-master.conf:  unix_listener lmtp {
conf.d/10-master.conf:  #inet_listener lmtp {
conf.d/10-master.conf:    # Avoid making LMTP visible for the entire internet
conf.d/90-quota.conf:  # LDA/LMTP allows saving the last mail to bring user from under quota to
conf.d/15-lda.conf:## LDA specific settings (also used by LMTP)
conf.d/15-lda.conf:# in LMTP replies. Default is the system's real hostname@domain.
conf.d/10-logging.conf:#  %{session_time} - How long LMTP session took, not including delivery_time
root@main:/etc/dovecot#

service lmtp {
  unix_listener lmtp {
    #mode = 0666
  }

Who, the xapian folk? :thinking:

Them and me.

Was this an apt install or from their github? I see Debian has it as a package but I got ‘inconsistent’ results searching for an Ubuntu package.

Also, I think by default I had to change WHERE the control files were stored. I think they defaulted to /var/… to keep folks from messing up their mailboxes. The server I was moving from had them already in the user directory so I kept them there. Is this a problem for the plugin?

I tried both the apt install and github, actually. I initially installed the apt package but it was intermittently executing as the wrong user and causing errors, which is why I joined the thread on github linked above. They guided me through installing the latest version from github, which got rid of the errors but also got rid of all the functionality!

I’m using the locations as in your screenshot.

Have you got it working? If so, that’s very exciting.

I don’t have it installed. Just probing for bits of info.

Kind of a problem now. Does the github version have an ‘uninstall’? Straying from maintained packages can be problematic later on. Might have been better to just chase down the ‘random’ user thing.

I know :frowning:

I tried testing on a VPS first but it was too hard so it’s actually on a production server. They said it would be OK because the fixed package would eventually overwrite the compiled version but that’s not going to happen now.

At least it’s fairly easy to disable the plugin and cron job, which I have done.

Searching email is an extremely reasonable thing to want to do. It’s a mandatory feature of modern email.

Webmin is not involved in processing mail at all, so you’re barking up the wrong tree if you (or they) are trying to pin this on Webmin.

Virtualmin configures some mail stuff as part of the installation. You are not required to use it that way, but if you want to use Virtualmin features, you probably need to stick with the stack as we set it up (procmail is used as the local delivery agent, we don’t use the Dovecot LDA).

It’s not their job to understand how it’s setup. It’s your server, so it’s your job. :wink:

We use an almost entirely default Dovecot configuration. We literally only make a handful of tweaks to enable protocols and such. None of it is secret, you can sit down and read and understand how the Virtualmin configured mail stack works: Virtualmin-Config/lib/Virtualmin/Config/Plugin/Dovecot.pm at master · virtualmin/Virtualmin-Config · GitHub

For local delivery, Postfix hands mail to procmail-wrapper, which hands it to procmail, which delivers it to the user’s Maildir. Dovecot provides remote access to that Maildir via POP3/IMAP/POP3S/IMAPS. In this configuration Dovecot does not perform any mail delivery or processing on the way to delivery, it is just the POP3/IMAP server.

There is nothing stopping you from configuring things to work differently. If you remove procmail-wrapper+procmail from the equation, you can no longer use Virtualmin and Usermin to manage stuff like forwarding, spam filtering, and AV filtering. That’s fine if you have other ideas about how you want to handle those things, but you’ll need to learn how the pieces fit together.

You can already search email, and Dovecot or Usermin will do some caching and indexing to make it faster.

It’s not clear to me what LMTP has to do with full-text search. I would think searching would happen on behalf of POP3 or IMAP users, rather than at time of delivery to a mailbox. But, using Dovecot for local delivery conflicts with the Virtualmin procmail-based delivery configuration. If using LMTP is a dependency of fts-xapian, you obviously need to enable lmtp in Dovecot and configure Postfix to deliver to the Dovecot lmtp socket. And, you’ll need to configure spam/AV and all that other stuff some other way.

Many thanks for the thorough response, all excellent points.

Protonmail for one deliberately prevents full-text searching in the name of privacy, so perhaps “mandatory” is a little too strong a word. Nevertheless I stopped using Protonmail precisely because it’s a tradeoff too far for me, so I agree.

As it happens, I hardly use procmail now due to a bunch of problems I encountered in the early days (forwarding unfiltered spam, backscatter, forwarding loops, clamav ineffective and using too much memory…). I’ve switched to amavis for spam filtering, banned forwarding and so on just to get wanted mail delivered. I do keep procmail though for people who want out of office replies or automatic filtering to folders other than the inbox so I would be a bit reluctant to lose it.

I wasn’t aware that Dovecot was uninvolved in message delivery. I don’t think it’s possible to hook fts-xapian into procmail. I think I would need a completely different full-text-search solution, or else a completely different way of filtering incoming mail, is that right?

I suspect the LMTP thing is about automatically updating the search index at the time of message delivery. I wasn’t aware Xapian needed that.

There are some search solutions out there but nothing I would describe as usable. Usermin is much too slow (many minutes). Roundcube only seems to search subject lines and is also slow. Some phone apps (e.g. Edison Mail) have a pretty good search facility but it only works on one folder at a time and misfiling something in the wrong folder is one of the main use cases for needing search. Gmail only works with google accounts and doesn’t search attachments.

Xapian is a generic full-text search library. It doesn’t need anything but text, it certainly doesn’t need Dovecot. But, you seem to be talking about a specific packaging of Xapian for mail (fts-xapian, which I have no familiarity with). There are many implementations of search that work with Dovecot, and there are multiple ways to trigger indexing. But, yeah, I guess the LMTP thing is about triggering indexing.

You’ll need to do some reading. But, if you’re already filtering spam via another mechanism, you’re most of the way to removing dependence on procmail. Maybe you want to keep going and removing it completely and switching to Dovecot for delivery. If you’re using Roundcube instead of Usermin, your users already don’t have us handling forwarding and auto-replies (though Virtualmin can manage them in the UI, too). But, if you do that, we’re not the best people to help you with problems. Everyone here is using procmail.

But, Dovecot does not need to be the delivery agent for full-text search (indexing can happen on searches or via other triggers). If fts-xapian requires it, that is their business, there are several ways to use the Dovecot fts feature without using LMTP to trigger indexing. So, I think somebody is mixing up how things work. Just don’t use LMTP to trigger indexing, and you should be fine with whatever full-text search you want to use.

It’s somewhat likely we’ll refactor mail at some point to use Dovecot with Sieve for processing rather than procmail, and we’d probably also look at improving search performance. But, that’s a pretty big undertaking with a lot of code, documentation, and UI work to be done to make it replace the current implementation, so I’m hesitant to commit to any schedule for that (recall we all work part time on this, mostly volunteering). Maybe it’ll be in Virtualmin 8, maybe not. Of course, it’s Open Source, and we welcome contributions. But, this one is a big undertaking, probably not something someone can drop in and implement as a one-off project. It would likely take us months, I assume it’d take someone new to Virtualmin development longer.

That seems like a limitation of Roundcube. You can search inside email without any additional third party search tools using Dovecot. I’m not saying it’s fast (it’s not), but an IMAP client can use the SEARCH command to search within email. If Roundcube is incapable of doing a SEARCH (I don’t know that, I’d be surprised if it is so) via IMAP, then making full-text search work in Dovecot won’t make it work for your Roundcube users. Usermin can do full text search, both with local Maildir access and via IMAP.

Thats prob because you don’t have body turned on in the search
Seem quick enough to be but I delete my mailbox out pretty regularly.

Thanks again, all very interesting points.

I’m aware of the “Body” option in Roundcube and there are people who claim to have used it successfully with fts-xapian, I just haven’t witnessed this myself. What I meant was when I select it with a standard Virtualmin installation I only see results that contain the search term in the subject line of emails rather than the body as requested. I think this is standard IMAP behaviour if there’s no additional backend support and Roundcube is probably OK but I’ve never seen it (or any other solution) doing a successful FTS via IMAP myself. I do hear reports of it working with cPanel and DirectAdmin but that way lies madness. :slight_smile:

I’ve also witnessed fts-xapian (the apt package version, not the latest compiled update from git) producing log entries that indicated that indexing of a mail folder had been triggered by the arrival of a message for that folder. If it’s completely relying on LMTP then I think that must therefore be a recent change.

I’ll just need patience, I think. I haven’t got the time or money to fix this myself, unfortunately.

For a user and their personal emails yes of obviously.
but for others (without a subpoena/court order) it is a breach of privacy.

Given today’s environment, I’d think this would be a nice feature.
https://doc.dovecot.org/2.3/configuration_manual/mail_crypt_plugin/

You might have to stick with the Dovecot search in that case though?
https://doc.dovecot.org/2.3/configuration_manual/fts/

1 Like

No one is talking about making email searchable by other users. This is about users searching their own email.

Sorry, somehow - I’m not sure where I got the idea that this was about a Virtualmin system admin tool to be able to search emails prior/during/after their being sent to to the actual recipient/user - what a user receives/does with their email is of course totally down to them.

As a system admin I have no interest in the content of anyone’s email!
and neither should anyone.

This is exactly the problem. I’d rather see support for OpenLiteSpeed and fixes for other things that are costing me money and time. Email is a relatively low priority because people are moving away from self-hosting that, precisely because it’s so difficult in a world full of spam. I love the open source model and want to support it as much as possible, but I find I’m using less and less of the default Virtualmin setup and becoming more tempted by panels like DirectAdmin that have far fewer features, less control and cost more but set up more of what I need out of the box. We’re being overtaken and the funding model is partly the problem.