Where are mail logs (right out of the box)?

[Ron_E_James_D.O]

I wanted to test something (else) and installed virtualmin from scratch on a fresh Ubuntu this morning. I sent a mail to a test account and I did get the mail, but I couldn’t find the mail log. I felt like I was missing something right in front of my face but /var/log/mail.log is not there.

I found another thread about this (No Mail log on fresh install) which says that virtualmin has been changed to use journalctl.

How can I get useful mail logs (dovecot and smtp mostly) out of the journal? That thread’s recommendation (journalctl -fu postfix) does not give smtp results for example

vs:

(which has the logs you’d expect from postfix in /var/log/mail.log)

What was the reason for the change?

Is there a way to revert/convert this to using /var/log/mail.log?

How long (and where) is journal archived and how easy is it to separate mail logs that I care about from the endless spinning system stuff that I don’t for storage purposes?

We keep mail logs for as long as possible. Occasionally we get a request like “so and so says they haven’t used the mail for 6 months and don’t want to pay, when is the last time they used mail?”. Or “so and so has a subpoena for mail logs for x-person for the last two years” etc… Since mail logs are ordinarily plain text (and contain all and only mail logs) we just zip them nightly, send them to storage, and generally forget them.

How would I emulate that with journal?

Ron

SYSTEM INFORMATION
OS type and version	Ubuntu 24.04
Virtualmin version	7.20.2

[Ron_E_James_D.O]

Okay well seemed suspiciously easy to fix on my own:

sudo nano /etc/postfix/main.cf

And append with:

maillog_file = /var/log/mail.log

Then:

sudo systemctl restart postfix

And:

sudo nano /etc/dovecot/dovecot.conf

And append with:

log_path = /var/log/mail.log

Then:

sudo systemctl restart dovecot

And:

sudo nano /etc/rsyslog.conf

And append with:

mail.* -/var/log/mail.log

Then:

sudo systemctl restart rsyslog

Anybody who “knows stuff” know whether this is a bad idea or will break something else?

Thanks.

Ron

[Joe]

@Ron_E_James_D.O

which says that virtualmin has been changed to use journalctl.

No. Virtualmin has nothing to do with it. Virtualmin, in almost every case, works with the system as provided by the vendor.

Your distribution, and the packages they provide, decides the default. You can choose do something other than the default, as well.

@Ron_E_James_D.O

What was the reason for the change?

Ask your OS vendor. You picked an OS, we respect that decision.

That respect looks like us doing everything we can to work in the way that an expert with that OS and version would expect, nothing should be a surprise. Services start/stop/status with the same commands, logs end up in the same place (though we don’t yet send our own logs from Webmin/Usermin to the journal, we will eventually), we use the same configuration files and locations, we use standard system packages for everything we can, etc. (Though, it happens that all of our supported operating systems now use the journal for most logs. Which is the natural result of systemd and the journal being better than what came before. But, you’re welcome to change back to text logs, if the journal is a problem. Though I think it’s worth learning your way around the journal, as it is better for most cases I can think of. We provide a UI for viewing and querying it.)

@Ron_E_James_D.O

How would I emulate that with journal?

You can choose how much data the journal is allowed to consume, which will roughly determine how long it’s kept locally.

For my day job, we ingest the entire journal from all of our 200+ devices into Loki on a central system, which is searchable in Grafana. Any log centralization setup could do the same. journald can also publish to syslog-compatible destinations. The journal is super flexible.

You can also just switch to writing a log, or you can configure journald to also write a log of the postfix unit. There are several ways to do this. Before I made everything use the journal at my day job and before we switched to Loki+Grafana, we had a few systemd units that would just pipe the journal for some units into logs we could ingest into Splunk.

It’s not specific to Virtualmin, any documentation or guides about journald, journalctl, and systemd in this area will be useful. We’re working with the system as it is, as best we can, unless and until you tell Virtualmin to do something else.

[Joe]

@Ron_E_James_D.O

Anybody who “knows stuff” know whether this is a bad idea or will break something else?

Nope, that’s fine, if that’s your preference. As far as I recall, that’s the way it worked on Ubuntu up until 20.04 or thereabouts, maybe 18.04. I dunno exactly when mail logs went to the journal by default.

[jimr1]

@Ron_E_James_D.O

How would I emulate that with journal?

save messing with config files why not extract the data from journalctl to a text file
as an example

journalctl -t postfix/smtpd -t dovecot --since today

note you could use the userid for postfix like this

journalctl _UID=<postfix userid> --since today

which will show everything the postfix user has written to the journal, not just smtpd entries, but nothing for dovecot.
journalctl is very flexible the --since can be a date, or something like “1 hour ago” the choice is yours.
you can also add as many -t items as you like, for example if you added
-t webmin to the above command the output will show failed logins to webmin
As a side note just installed a fresh Ubuntu 24.04.1 followed by a virtualmin install and mail.log is present and being logged to

[Ron_E_James_D.O]

Thanks a lot for this explanation. It’s weird to me that I installed from the same .iso as another Ubuntu which used mail.log out-of-the-box but maybe they just changed it (it does ask at the beginning of the install if I want to get the latest installer, which I always answer yes to).

I noticed several weeks ago and tested again with this fresh install that mail rate limiting is not working out-of-the-box (opened in another thread: Mail Rate Limiting broken out of the box (Ubuntu 24.04)). Neither thread got much traction. Should I assume that the mail rate limiting is also an Ubuntu/postfix/milter issue and not something in Virtualmin’s choice of configs and such at boxing time? (ie, so if I want it, like mail logs in mail.log, I should work out how to to do it).

Similarly (but I think the answer is different) you have responded on another thread I have about :20000 not assuming the-domain when only a username is entered out-of-the-box (the original reason for this test install: Can usermin assume @-the-domain-from-the-URL at :20000 for the logins? - #12 by Ron_E_James_D.O). That one IS a Virtualmin thing and not mine to fix though, right?

Thanks.

[Ron_E_James_D.O]

This really is pretty powerful. I wonder if I can have both somehow?

It is weird to me that you just did an Ubuntu and ended up with mail.log. I installed mine answering “yes” to the “Do you want the latest installer” and then pasted the line from Virtualmin’s install instructions. It was as out-of-the-box as it could be. (That was just on Friday, the one I was working with and wondering if I’d borked was installed generally the same way on Wednesday.)

That said an Ubuntu 24.04.1 from June 24 uses mail.log by default.

[jimr1]

@Ron_E_James_D.O

This really is pretty powerful. I wonder if I can have both somehow?

Whatever happens everything will be written to the journal assuming the application you are using writes to the journal (some don’t) but most base applications do. Just play with journalctl to get what you need and write a bash script to save it to a file. You can have both for the above reason, however I don’t get that a default ubuntu 24.04 does not also write to mail.log

[jimr1]

@Ron_E_James_D.O

That said an Ubuntu 24.04.1 from June 24 uses mail.log by default.

It wasn’t released at that point do you mean 24.04 ?

[Ron_E_James_D.O]

It seems that appending my .conf files with log_path = /var/log/mail.log (as mentioned) causes mail logs to not be in the journal.

[Ron_E_James_D.O]

I’ve kept with the updates, so that would be why my dashboard says 24.04.1 on that one as well (it is mail.log by default).

[jimr1]

@Ron_E_James_D.O

seems that appending my .conf files with log_path = /var/log/mail.log

What does this mean ? Can you expand on this

[Ron_E_James_D.O]

@Ron_E_James_D.O

Okay well seemed suspiciously easy to fix on my own:

sudo nano /etc/postfix/main.cf

And append with:

maillog_file = /var/log/mail.log

In my above post that begins:

"Okay well seemed suspiciously easy to fix on my own:

sudo nano /etc/postfix/main.cf

And append with:

maillog_file = /var/log/mail.log
"

I got it writing to mail.log (and left that instruction set in case some other confused person, or me in a few months who will have forgotten by then, needs it).

The result is this:

vs:

(demonstrating, I believe, that I have generated smtp logs today which are in mail.log but not in the journal).

[jimr1]

I really don’t get it … just ran up a new vm, installed ubuntu 24.04.1, this time I didn’t install virtualmin but installed just postfix & dovecot. Tested it and both mail.log and the journal are written to. If this is a vm that has been supplied from a supplier perhaps the install image has been changed some how, but I can still say that on a default install everything works as expected

[Ron_E_James_D.O]

I agree that it’s weird, my Virtualmin config selections are midway down here:

Can usermin assume @-the-domain-from-the-URL at :20000 for the logins?

I have now repeated this with a purpose-made out-of-the-box install (on Ubuntu 24.04). I took screenshots of every step of my configuration (in case any of the choices matter). I also took screenshots of creating the virtual machine and user but I’ll spare you most of those. I also repeated the login test on Edge (photo is Opera). If you are right that my first test (just ron) was supposed to work, and I am right that it is broken out of the box, does that imply that I need do nothing and th…

My .iso is from May (thus the June install I mentioned). Maybe May’s .iso + today’s installer updates is the key to “breaking” mine this way.

[jimr1]

Is this a home setup using wsl or something else ?

[Ron_E_James_D.O]

No, Proxmox.

[jimr1]

Never used /needed to use promox this drops me out of trying to help. I think @shoulders uses this so could advise you better

[Ron_E_James_D.O]

Proxmox shouldn’t have anything to do with the Ubuntu/Virtualmin. Proxmox is just VMWare for people who bailed on VMWare when Broadcom bought it and made it not free anymore.

(I could rerun the experiment on ESXi but there really isn’t any way that the outcome would be different unless “Update to the most recent install script” has changed since Friday.)

[jimr1]

Still never used it so I am not going to offer any more as I do not know what proxmox is or exactly what it does. The only answer I can give it works as expected using a standard linux system