I have found out the reason why Postfix keeps getting killed by OOM Killer

Virtualmin
1
SYSTEM INFORMATION
OS type and version AlmaLinux 9.5
Webmin version 2.202
Virtualmin version 7.30.4
Webserver version 2.4.62
Related packages Postfix

Subject: I have found out the reason why Postfix keeps getting killed by OOM Killer

Good day from Singapore,

On 1st March 2025 Saturday, my Postfix service stopped responding. When I executed the “systemctl status postfix” LINUX command, it shows that postfix was killed by the OOM Killer.

To solve the problem, I tried restarting the postfix service but it did not solve the problem.

Then I tried rebooting my Virtual Private Server (VPS) in Germany but it did not solve the problem.

As a last resort, I tried uninstalling and reinstalling postfix but it did not solve the problem as well.

My postfix service was consuming 6.2 GB of RAM before it got killed by the OOM Killer. The total amount of physical memory (RAM) my VPS has is only 8 GB. No wonder my postfix service keeps getting killed by the OOM Killer. Is this a Denial of Service (DOS) attack?

This morning, on 2nd March 2025 Sunday, I suddenly had an idea. I started checking the Mail Queue. There were 115 messages stuck in the Mail Queue! After some deliberation, I decided to clear the mail queue.

After clearing the mail queue, my postfix service resumed normal service and no longer gets killed by the OOM Killer. Clearing the mail queue actually solved the problem!

But until now I still have no idea why 115 messages stuck in the mail queue will cause postfix to consume ENORMOUS amounts of RAM and cause it to get killed by the OOM Killer. OOM = Out of Memory.

Could somebody explain why? Do you think my Virtualmin VPS server has been compromised by Advanced Persistent Threat (APT) hackers?

I am looking forward to your reply. The last time my postfix service stopped working due to a similar OOM condition was on 11 Jan 2025 Saturday, as posted earlier.

Thank you.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individuals in Singapore

2

Normally you really shouldn’t have much in the queue. Either the system is accepting mail it can’t deliver or it is delivering mail that isn’t being accepted elsewhere. Both if someone is trying to use you as a relay. You need to keep check on the queue and find out what’s going on.

Postfix probably tries to clear the queue on a periodic basis. If this is outgoing mail, then a remote system may be holding open the connection as an anti-spam tactic. Now, if Postfix runs the queue before the first attempt is completed, you start having multiple queue runs going a the same time.

3

The mail queue is not directly related to the size of the postfix process.

You need to look at the log, and look more deeply at what processes are taking memory.

I doubt Postfix was actually using 6GB of RAM. It was probably not resident. You should look more deeply into what processes are using memory. If you’re looking at, e.g. top you can sort by memory usage (press <shift>-M) and look at the RES field (not the VIRT field).

You need to look at the logs for the services you suspect of having problems. On a modern OS, the log is probably in the journal, so look at journalctl -fu postfix, if it turns out Postfix is the cause of trouble (but I doubt it is, I think you’re looking at the wrong thing, Postfix resident size is normally quite small).

As for why the queue would grow, it’s probably because sending mail is blocked in some way. Maybe your system was sending spam and your hosting provider blocked port 25. Maybe you have a bunch of messages for a specific destination that has blocked you or gone offline. You need to look at the queued messages to know. “Flushing” the mail queue causes mail delivery to be re-attempted immediately (it does not delete the mail queue). But, the mail queue is not the reason your system ran out of memory.

This was a terrible idea. Now you have more problems to solve.

Summary:

  1. The memory problem is almost certainly not Postfix. You need to look at actual memory usage to see who/what is using a bunch of memory. I’ll be shocked if Postfix is actually your biggest memory user or using more than a few hundred MB of resident memory.
  2. Don’t uninstall/reinstall software to try to fix it, especially before you even know what problem you’re trying to solve.
  3. A large mail queue does not cause Postfix to consume a lot of memory. There’s something about your problem you haven’t identified, so you need to keep digging into what’s actually using memory. Also, 115 messages is not a large queue. A heavily loaded mail server on a slow link might have thousands of messages queued, and that’s not a problem from a memory or performance perspective.
4

Dear Joe,

Please refer to the output of Linux command “systemctl status postfix”.


postfix.service - Postfix Mail Transport Agent
     Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; preset: disabled)
     Active: active (running) since Sat 2025-03-01 16:30:47 +08; 1min 58s ago
    Process: 697 ExecStartPre=/usr/sbin/restorecon -R /var/spool/postfix/pid (code=exited, status=0/SUCCESS)
    Process: 716 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
    Process: 787 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS)
    Process: 819 ExecStart=/usr/sbin/postfix start (code=exited, status=0/SUCCESS)
   Main PID: 1149 (master)
      Tasks: 69 (limit: 48939)
     Memory: 6.2G
        CPU: 2min 5.479s

As shown above, postfix was consuming 6.2 GB of RAM before it got killed by OOM Killer.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

5

Dear Joe,

Please refer to postfix debugging output below.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

6

Dear Joe,

Please refer to output of top.

https://www.mail-archive.com/postfix-users@postfix.org/msg105259.html

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

7

Dear Joe,

Output of “systemctl status”.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

8

My system is a small, personal server. But, for comparison:

● postfix.service - Postfix Mail Transport Agent
     Loaded: loaded (/lib/systemd/system/postfix.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2025-03-01 06:34:45 EST; 2 days ago
   Main PID: 3795996 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 19031)
     Memory: 0B
        CPU: 0
     CGroup: /system.slice/postfix.service

Mar 01 06:34:45 main.cisnetadmin.com systemd[1]: Starting Postfix Mail Transport Agent...
Mar 01 06:34:45 main.cisnetadmin.com systemd[1]: Finished Postfix Mail Transport Agent.

Is the mail queue growing again? If so, have you checked the messages to see if they indicate ‘bad behavior’? How many legit email users are on the system?

9

My postfix service was killed again. When I checked the Mail Queue, it grew to 113 messages again. After clearing the mail queue, postfix service resumed normal operation again.

I haven’t examine the messages in the Mail Queue in detail yet.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

10

You don’t need to let Postfix stop to do this. Just periodically look and see what is there and examine the messages.
I’m pretty sure you can do it from the interface. If the emails are ‘dead spam’, which I suspect, you will need to figure out why they are there. Look at the headers. They tell you were it originated and where it was supposed to go before becoming stuck. (if they are stuck) You didn’t mention how many legit users you have so we can’t guess as to IF you have enough resources on that machine.

image
image828×375 27.4 KB

11

There are no messages in the Mail Queue right now. I have just cleared the mail queue. And postfix is currently consuming 376.5 MB of RAM. Postfix is still running right now.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

12

I have just asked ChatGPT Artificial Intelligence (AI) for assistance.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

13

Dear Joe,

All my troubleshooting and debugging output are being posted here.

https://www.mail-archive.com/postfix-users@postfix.org/maillist.html

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

14

I looked briefly. Please don’t cross post. I’m pretty sure spamc IS the default but I haven’t set up a new server in over a year so I don’t remember.

I wondered about this possibility but since clamscan is the default/recommended, and you seem to not want to post anything relevant here, we couldn’t know that. But this has been discussed way too many times here so I’m not sure how people get this wrong.

|

image
image916×730 88.9 KB

15

Dear Joe,

People at the postfix-users mailing list are telling me to use clamd instead of clamdscan and clamscan because clamdscan and clamscan consume huge amounts of memory.

How do I disable ClamAV completely?

Maybe Virtualmin should use clamd instead of clamscan?

Please refer to the discussions here:

https://www.mail-archive.com/postfix-users@postfix.org/maillist.html

Thank you.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

16

I’m also not sure why Clam standalone would show up in memory usage for Postfix unless it is ‘waiting’?

17

Remove it from your system … simple but as you only seem to want joe to answer just wait for a reply rather then ignoring everyone else

18

Sorry for the misperception. I am not ignoring everybody else. I did reply to users here.

Many thanks.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore

19

I don’t know if it will go through but I just posted to the mailing-list that the misunderstanding of the differences between clamscan and clamd IS NOT a Virtualmin problem.

In the future, please take some personal responsibility for your system administration. It isn’t up to the internet to solve the problems you don’t want to dig into.

I guess someone has pointed this out:
"
https://www.mail-archive.com/postfix-users@postfix.org/msg105295.html

virtualmin already has options for these.

20

Thank you for alerting me. I will read the article when I wake up in the morning.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Singapore