Postfix down, and cannot be started

kolonelsander · December 28, 2016, 2:45am

Hello,

My Postfix service is down after restarting the server, and when i try to manually start postfix service via virtualmin webGUI, it comes back with these errors:

Failed to start service : find: '/var/spool/postfix/defer/7/722AFD7EA6D2': No such file or directory find: '/var/spool/postfix/defer/7/7BCA8D7E949C': No such file or directory find: '/var/spool/postfix/defer/7/79BFACBE6040': No such file or directory find: '/var/spool/postfix/defer/7/76C4BD7F2DD0': No such file or directory find: '/var/spool/postfix/defer/7/7287BD7E74B9': No such file or directory find: '/var/spool/postfix/defer/7/7B543D7EA6A8': No such file or directory find: '/var/spool/postfix/defer/7/74DB5CBE604B': No such file or directory find: '/var/spool/postfix/defer/7/78148D7ED98C': No such file or directory find: '/var/spool/postfix/defer/7/7F1F1D7EF9C3': No such file or directory find: '/var/spool/postfix/defer/7/7D6EFD7EF9F6': No such file or directory find: '/var/spool/postfix/defer/7/7076FD7F094C': No such file or directory find: '/var/spool/postfix/defer/7/73D09D7F0976': No such file or directory find: '/var/spool/postfix/defer/7/791CFD7F0978': No such file or directory find: '/var/spool/postfix/defer/7/7C579D7F2DD1': No such file or directory find: '/var/spool/postfix/defer/7/7B400D7FC751': No such file or directory find: '/var/spool/postfix/defer/7/7659ED80D640': No such file or directory find: '/var/spool/postfix/defer/7/76806D812905': No such file or directory find: '/var/spool/postfix/defer/7/725CFD846146': No such file or directory find: '/var/spool/postfix/defer/7/7A3E8D7E9474': No such file or directory find: '/var/spool/postfix/defer/7/7F99ED84E576': No such file or directory find: '/var/spool/postfix/defer/7/75614D860622': No such file or directory find: '/var/spool/postfix/defer/7/791EED812906': No such file or directory find: '/var/spool/postfix/defer/7/7AC76CC2C0B2': No such file or directory find: '/var/spool/postfix/defer/7/774C5D7F0977': No such file or directory find: '/var/spool/postfix/defer/7/76947D8473BA': No such file or directory find: '/var/spool/postfix/defer/7/745ACD894E90': No such file or directory find: '/var/spool/postfix/defer/7/797C7D85F5D9': No such file or directory find: '/var/spool/postfix/defer/7/767AFD7F3F0E': No such file or directory find: '/var/spool/postfix/defer/7/7D7B0D84A321': No such file or directory find: '/var/spool/postfix/defer/7/72C01D7E74BF': No such file or directory find: '/var/spool/postfix/defer/7/70B07D7EA6A6': No such file or directory find: '/var/spool/postfix/defer/7/73115D7ED9B7': No such file or directory find: '/var/spool/postfix/defer/7/7B7B1D7EF9EC': No such file or directory find: '/var/spool/postfix/defer/7/7033BD7F2DCE': No such file or directory find: '/var/spool/postfix/defer/7/7540FD826D79': No such file or directory find: '/var/spool/postfix/defer/7/76D44D854A4F': No such file or directory find: '/var/spool/postfix/defer/7/7D202D855C0A': No such file or directory find: '/var/spool/postfix/defer/7/70965D85AF1D': No such file or directory find: '/var/spool/postfix/defer/7/75E63D85F5DF': No such file or directory find: '/var/spool/postfix/defer/7/76AD3D889483': No such file or directory find: '/var/spool/postfix/defer/7/7F929D88A4DD': No such file or directory find: '/var/spool/postfix/defer/7/7A60FD8A20C2': No such file or directory find: '/var/spool/postfix/defer/7/775F2D892EED': No such file or directory find: '/var/spool/postfix/defer/7/77DC8D8B5BFB': No such file or directory find: '/var/spool/postfix/defer/7/74C84D8B9F96': No such file or directory find: '/var/spool/postfix/defer/7/7FB14D88600B': No such file or directory find: '/var/spool/postfix/defer/7/7EEC7D7E81D6': No such file or directory find: '/var/spool/postfix/defer/7/79027D7EE8DE': No such file or directory find: '/var/spool/postfix/defer/7/71480D7F2DC6': No such file or directory find: '/var/spool/postfix/defer/7/71842CC4F1C1': No such file or directory find: '/var/spool/postfix/defer/7/717ECD7F2DDB': No such file or directory find: '/var/spool/postfix/defer/7/76396D895F8E': No such file or directory find: '/var/spool/postfix/defer/7/7F04DD87A5C4': No such file or directory find: '/var/spool/postfix/defer/7/7B5A9D7EF9D6': No such file or directory find: '/var/spool/postfix/defer/7/74949D93F6F8': No such file or directory find: '/var/spool/postfix/defer/7/7D629D951355': No such file or directory find: '/var/spool/postfix/defer/7/76526D7EA6A7': No such file or directory find: '/var/spool/postfix/defer/7/7C0D6D8CE138': No such file or directory find: '/var/spool/postfix/defer/7/73B8AD7FB50F': No such file or directory find: '/var/spool/postfix/defer/7/7B4A9D90E9F9': No such file or directory find: '/var/spool/postfix/defer/7/716FAD914AC6': No such file or directory find: '/var/spool/postfix/defer/7/7E1FBD92EC68': No such file or directory find: '/var/spool/postfix/incoming/5A917CB9DD98': No such file or directory find: '/var/spool/postfix/incoming/5BEBFCB9DD99': No such file or directory find: '/var/spool/postfix/incoming/469F9CB9DDA1': No such file or directory find: '/var/spool/postfix/incoming/4E258CB9DDA3': No such file or directory find: '/var/spool/postfix/incoming/A3097CB9DDA4': No such file or directory find: '/var/spool/postfix/incoming/A0BC1CB9DDAB': No such file or directory find: '/var/spool/postfix/incoming/7864ACB9DDB1': No such file or directory find: '/var/spool/postfix/incoming/1F1F9CB9DDB6': No such file or directory find: '/var/spool/postfix/incoming/39EB2CB9DDB7': No such file or directory find: '/var/spool/postfix/incoming/43511CB9DDB8': No such file or directory find: '/var/spool/postfix/incoming/EAC71CB9F083': No such file or directory find: '/var/spool/postfix/incoming/80826CB9F088': No such file or directory find: '/var/spool/postfix/incoming/E2B9ECB9F08C': No such file or directory find: '/var/spool/postfix/incoming/A0F3ECB9F097': No such file or directory find: '/var/spool/postfix/incoming/1E3FFCB9F0A0': No such file or directory find: '/var/spool/postfix/incoming/8D9C3CB9F0A4': No such file or directory find: '/var/spool/postfix/incoming/CA82CCB9F0A5': No such file or directory find: '/var/spool/postfix/incoming/D3AA2CB9F0A6': No such file or directory find: '/var/spool/postfix/incoming/9BA90CB9F0BD': No such file or directory find: '/var/spool/postfix/incoming/BEC78CB9F0BE': No such file or directory find: '/var/spool/postfix/incoming/82D59CB9FF64': No such file or directory find: '/var/spool/postfix/incoming/94BEACB9FF65': No such file or directory find: '/var/spool/postfix/incoming/EE038CB9FF67': No such file or directory find: '/var/spool/postfix/incoming/08C94CB9FF6C': No such file or directory find: '/var/spool/postfix/incoming/163DCCB9FF6E': No such file or directory find: '/var/spool/postfix/incoming/2B31DCB9FF6F': No such file or directory find: '/var/spool/postfix/incoming/0DD68CB9FF7E': No such file or directory find: '/var/spool/postfix/incoming/2A04CCB9FF87': No such file or directory find: '/var/spool/postfix/incoming/4B84FCB9FF89': No such file or directory find: '/var/spool/postfix/incoming/5E305CB9FF99': No such file or directory find: '/var/spool/postfix/incoming/C63B2CB9FF9A': No such file or directory find: '/var/spool/postfix/incoming/0EE26CB9FF9E': No such file or directory find: '/var/spool/postfix/incoming/C7A50CBA0EA2': No such file or directory find: '/var/spool/postfix/incoming/20CEDCBA0EA7': No such file or directory find: '/var/spool/postfix/incoming/4CB29CBA0EAC': No such file or directory find: '/var/spool/postfix/incoming/00E7BCBA0EB2': No such file or directory find: '/var/spool/postfix/incoming/A405ACBA0EB5': No such file or directory find: '/var/spool/postfix/incoming/2E77ECBA0EB9': No such file or directory find: '/var/spool/postfix/incoming/3A641CBA0EBA': No such file or directory find: '/var/spool/postfix/incoming/5CF71CBA0EBC': No such file or directory find: '/var/spool/postfix/incoming/33312CBA0EC2': No such file or directory find: '/var/spool/postfix/incoming/BD1D0CBA0EC5': No such file or directory find: '/var/spool/postfix/incoming/D9B8BCBA0ECD': No such file or directory find: '/var/spool/postfix/incoming/5E26ECBA0ED2': No such file or directory find: '/var/spool/postfix/incoming/80362CBA0ED3': No such file or directory find: '/var/spool/postfix/incoming/9EAC1CBA0ED7': No such file or directory find: '/var/spool/postfix/incoming/10649CBA0EDE': No such file or directory find: '/var/spool/postfix/incoming/572AFCBA2186': No such file or directory find: '/var/spool/postfix/incoming/89BFCCBA2189': No such file or directory find: '/var/spool/postfix/incoming/EAC2ECBA218C': No such file or directory find: '/var/spool/postfix/incoming/07A14CBA218F': No such file or directory find: '/var/spool/postfix/incoming/EA956CBA219D': No such file or directory find: '/var/spool/postfix/incoming/DEBFACBA21B9': No such file or directory find: '/var/spool/postfix/incoming/846F6CBA31A5': No such file or directory find: '/var/spool/postfix/incoming/0CD33CBA31AE': No such file or directory find: '/var/spool/postfix/incoming/34164CBA31AF': No such file or directory find: '/var/spool/postfix/incoming/8C160CBA31B1': No such file or directory find: '/var/spool/postfix/incoming/AA8FBCBA31C1': No such file or directory find: '/var/spool/postfix/incoming/ABD72CBA31C2': No such file or directory find: '/var/spool/postfix/incoming/61BDFCBA31CC': No such file or directory find: '/var/spool/postfix/incoming/3FE1BCBA31DC': No such file or directory find: '/var/spool/postfix/incoming/B61E6CBA4121': No such file or directory find: '/var/spool/postfix/incoming/D2D2CCBA4129': No such file or directory find: '/var/spool/postfix/incoming/12793CBA412D': No such file or directory find: '/var/spool/postfix/incoming/70A20CBA4131': No such file or directory find: '/var/spool/postfix/incoming/66F27CBA4136': No such file or directory find: '/var/spool/postfix/incoming/7B0A5CBA4137': No such file or directory find: '/var/spool/postfix/incoming/91F79CBA4149': No such file or directory find:

I have no idea what to do with these errors, any help will be much appreciated

Thanks,

Diabolico · December 28, 2016, 5:00am

Did you check mail queue? You can try over Wm module or “postqueue -p” (or “mailq”).

kolonelsander · December 28, 2016, 9:06am

Thanks for replying diabolico

I’ve checked mail queue as per advice, and found out that one of my user sending a lot of spam email,
I’ve deleted that account, purge all postfix mail queue using postsuper -d all

But, I still encountering same error whenever start postfix from virtualmin webGUI

Any Idea?

Diabolico · December 28, 2016, 12:22pm

First purge all emails in queue and then try to restart postfix. If postfix wont start post the new logs.

Eric · December 29, 2016, 12:05am

Howdy,

You may want to try running the command “service postfix restart” on the command line, and then review the mail logs in /var/log/maillog or /var/log/mail.log to see if any errors exist.

You may also want to check and see if the directories “/var/spool/postfix/incoming” and “/var/spool/postfix/defer” exist.

-Eric

kolonelsander · December 29, 2016, 1:28am

Hi andreycheck

Thanks a lot for your suggestion!
both “/var/spool/postfix/incoming” and “/var/spool/postfix/defer” directories exist.
after poking around i think i found out what the problem was:

postfix.service - Postfix Mail Transport Agent Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled) Active: failed (Result: timeout) since Thu 2016-12-29 07:40:42 WIB; 1min 26s ago Process: 17101 ExecStart=/usr/sbin/postfix start (code=killed, signal=TERM) Process: 17099 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS) Process: 17094 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
Dec 29 07:40:42 vm1.radiant-iu.com systemd[1]: postfix.service operation timed out. Terminating.
Dec 29 07:40:42 vm1.radiant-iu.com systemd[1]: Failed to start Postfix Mail Transport Agent.
Dec 29 07:40:42 vm1.radiant-iu.com systemd[1]: Unit postfix.service entered failed state.

while report from /var/spool/maillog:

/var/log/maillog: line 1: syntax error near unexpected token newline' /var/log/maillog: line 1: Dec 26 03:42:16 vm1 postfix/cleanup[14016]: 1FD3AC0ED C0E: message-id=e6btamcguftbumd7vc9s9gd0.1604168226086@domain.com’

How to fix this?

Diabolico · December 29, 2016, 3:43am

Are you sure it was compromised just one account? Did you check how this happened? It could be that your server got hacked and thats the reason behind all the spam.

While you consider the option your server was hacked and not just one account can you do three things:

Use "systemctl daemon-reload " and see if anything changes
Restart your server and check if postfix is up and running
Open “/usr/lib/systemd/system/postfix.service” and change “ExecStart=/usr/sbin/postfix start” to “ExecStart=/usr/sbin/postfix -vv start”, use “systemctl daemon-reload” and then restart postfix. This will enable verbose logging. Check the logs and see if there is some new information to explain what is going on.

P.S. Just remember, once done with this problem return postfix.service to original state and make local copy of any file you intend to change or delete.

kolonelsander · December 29, 2016, 3:59am

I’m pretty sure my server isn’t hacked, because i’ve made rule in mikrotik that only allow connection to that server from local IP only (192.x.x.x) via SSH port 22, and WEB port 80, all port that isn’t necessary for sending/receiving email also blocked, – is it safe enough?

Let me try your suggestion first and i will report back

kolonelsander · December 29, 2016, 8:54am

I’ve tried the point one and point two, but there is nothing happen,
I tried point three but the report like this:

[root@vm1 ~]# bash /usr/lib/systemd/system/postfix.service

/usr/lib/systemd/system/postfix.service: line 1: [Unit]: command not found

/usr/lib/systemd/system/postfix.service: line 2: Mail: command not found

/usr/lib/systemd/system/postfix.service: line 3: network.target: command not found

/usr/lib/systemd/system/postfix.service: line 4: exim.service: command not found

/usr/lib/systemd/system/postfix.service: line 6: [Service]: command not found

/usr/lib/systemd/system/postfix.service: line 12: start: command not found

/usr/lib/systemd/system/postfix.service: line 13: reload: command not found

/usr/lib/systemd/system/postfix.service: line 14: stop: command not found

/usr/lib/systemd/system/postfix.service: line 16: [Install]: command not found

everything is “command not found”, it is normal?

Diabolico · December 29, 2016, 1:03pm

Hm i think i was too sleepy and made a little mistake when i was typing, it should be just one “v” instead of two. But in case this doesnt work you can try other two solutions:

In “/etc/postfix/master.cf” find the line and change to this
# ==========================================================================

service type private unpriv chroot wakeup maxproc command + args

(yes) (yes) (yes) (never) (100)

==========================================================================

smtp inet n - - - - smtpd -v

Stop postfix and then start with this command “/usr/sbin/postfix -v start”

Any of this two solutions should enable verbose logging for postfix. If you use any of this two solutions dont forget to revert back postfix.service to the original state.