Logrotate caused nginx to stop

OS: Ubuntu Linux 16.04.4
Webmin: 1.960
Virtualmin: 6.12

Server RAM = 1 GB

syslog:

Oct 25 06:25:01 web01 CRON[29660]: (root) CMD (/etc/webmin/status/monitor.pl)
Oct 25 06:25:01 web01 CRON[29661]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Oct 25 06:25:05 web01 systemd[1]: Stopping A high performance web server and a reverse proxy server...
Oct 25 06:25:05 web01 systemd[1]: Stopped A high performance web server and a reverse proxy server.
Oct 25 06:25:05 web01 systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 25 06:25:05 web01 nginx[29766]: nginx: [alert] mmap(MAP_ANON|MAP_SHARED, 524288000) failed (12: Cannot allocate memory)
Oct 25 06:25:05 web01 nginx[29766]: nginx: [alert] munmap(FFFFFFFFFFFFFFFF, 524288000) failed (22: Invalid argument)
Oct 25 06:25:05 web01 nginx[29766]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 25 06:25:05 web01 systemd[1]: nginx.service: Control process exited, code=exited status=1
Oct 25 06:25:05 web01 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Oct 25 06:25:05 web01 systemd[1]: nginx.service: Unit entered failed state.
Oct 25 06:25:05 web01 systemd[1]: nginx.service: Failed with result 'exit-code'.

First time this happened in the server. The only change I made this week is adding one more virtual server. The server now has 8 virtual servers.

I found this related issues:


System Settings -> Server Templates -> Default Settings -> Log file rotation, is Yes

Also, I just run:

virtualmin disable-feature --logrotate --all-domains
virtualmin enable-feature --logrotate --all-domains

but I don’t know if that could help.

The server has 1 GB of RAM and, as I said, I added one more virtual server last week. Is a low RAM the cause? or should I check some sever configuration?

Any help would be appreciated :grinning:

It happened again today :disappointed:

Oct 26 06:25:01 web01 CRON[7695]: (root) CMD (/etc/webmin/status/monitor.pl)
Oct 26 06:25:01 web01 CRON[7696]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Oct 26 06:25:06 web01 rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="1220" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Oct 26 06:25:06 web01 systemd[1]: Stopping A high performance web server and a reverse proxy server...
Oct 26 06:25:06 web01 systemd[1]: Stopped A high performance web server and a reverse proxy server.
Oct 26 06:25:06 web01 systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 26 06:25:06 web01 nginx[7856]: nginx: [alert] mmap(MAP_ANON|MAP_SHARED, 524288000) failed (12: Cannot allocate memory)
Oct 26 06:25:06 web01 nginx[7856]: nginx: [alert] munmap(FFFFFFFFFFFFFFFF, 524288000) failed (22: Invalid argument)
Oct 26 06:25:06 web01 nginx[7856]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 26 06:25:06 web01 systemd[1]: nginx.service: Control process exited, code=exited status=1
Oct 26 06:25:06 web01 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Oct 26 06:25:06 web01 systemd[1]: nginx.service: Unit entered failed state.
Oct 26 06:25:06 web01 systemd[1]: nginx.service: Failed with result 'exit-code'.

So, disable/enable logrotate did not fixed the issue.

I will remove the extra virtual server that I added, and I hope tomorrow this stops. However, the server was working fine for a few days with the extra virtual server.

I removed the extra virtual server and the issue did not happen again yet. So, it is just a RAM issue.

Nevertheless, nginx should not be stopped, never. I will search how to accomplish that.

@Joe by any chance, Virtualmin provides a way to set up processes/services that should never be stopped or automatically restarted in case of failure (nginx, PHP-FPM, MySQL, etc). To avoid these cases when a cron action takes a web server down. For instance, I prefer logrotate to fail rather than having all websites down.

log rotation should never take anything down. But, lack of log rotation will (logs don’t rotate, disk fills, apps break in potentially catastrophic data-destructive ways).

I’d recommend adding more swap memory so that nginx has what it needs. I’m not sure why it’s doing that big mmap, maybe you have a lot of files being served in the new domain? I’m not familiar with nginx memory management, though. Could be normal.

Edit: To be clear, what you’re asking for isn’t really possible. It’s kind of like asking for a “never let anything bad happen” feature. We generally can’t know if a service restart will fail (for some unpredictable reason like this) until a service restart is tried, and once it’s tried, the problem already exists. The only way we could do something like “don’t logrotate if it could cause a service to not restart successfully” would be to “never logrotate”, which is guaranteed to cause failure.