NGinx does not restart after log file rotate

SYSTEM INFORMATION
OS type and version Debian Linux 12
Webmin version 2.101

Just finished rebuilding Debian based server and it’s now been operational for 3 days.

The last two days the NGinx server has stopped working during the night. Yesterday I added it to “Monitors” and this email was sent at 5 seconds past midnight.

Monitor on apache-web-server.twin-peaks-video.com for ‘Nginx Webserver’ has detected that the service has gone down at 09/15/2023 12:05 AM
Running /usr/bin/nginx …

Which coincides with log file rotation. It’s very hard to tell if this happened during the rotation of the NGinx logs or the Virtualmin logs.

I added to Monitors " /usr/bin/nginx " (looked it up) but that did not seem to work. As soon as I started NGinx in Virtualmin, email arrived “up and running”.

I have set Rotation to 1 week rather than 1 day.

I have to ask why is anything on or about an apache webserver running on a nginx webserver. I thought it was not possible to run both on the same box?

Hmm… If it goes down because it can’t log then there would be no useful information in the logs? Debian 12 isn’t well tested but maybe nginx detects no log file, can’t open a new one and goes down? But can create one on start up? And that is only IF it is related to log rotation which hopefully changing the timing will detect.

I did check the .log files and all the new files are created in both /logs/nginx and /logs/virtualmin. Just checked them all and they are being written into. When I looked the timestamps were just after 12:00AM, not when I restarted NGinx this morning.

Just discovered I should be using /usr/sbin/nginx in monitor. << Would adding to monitor cause any issues with log file rotate?

So you don’t have Nginx as a Monitor Type like Apache? Monitors shouldn’t effect anything.

A monitor of Apache makes no sense and shouldn’t even be an option for a nginx webserver (why is it even possible)

Its a option in Virtualmin, its adds that monitoring. So if a server goes down you don’t want a warning?

Been a tad busy.

I added NGinx to be Monitored. Not Apache. That was in the list but disabled.

I see where the confusion came in. apache-web-server was apart of my machine name, first used many many years ago when I ran Apache. I haven’t change the name.

No, what I am saying is why should it even be an option. Why would anyone ever want to monitor or even care about some Apache server if they are using Nginx. Unless I am wrong and you really can run both servers on the same box. I don’t use apache and I certainly don’t care yet I am still informed that it is not running :person_shrugging:t2:

Sorry, I meant there should be a equivalent to the apache server in the dropdown for Nginx webserver if he is using Nginx. Misunderstanding sorry.

and yes there is an equivalent, but it doesn’t answer the fundamental question “Why” - perhaps I should raise it as a new topic.

As I said yesterday, I modified log file rotation for NGinx from every night and made it a week.

This morning NGinx is running perfectly. So what ever is causing NGinx to not restart correctly must have some thing to do with the rotate logs.

I ran the logfile rotate on the NGinx entry and that worked with no issues.

But when I ran it on the Virtualmin section, this caused NGinx to die.

Here is the error message that I received when rotating the VirtualMin log files:

Job for nginx.service failed.
See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.

/home/nigel# journalctl -xeu nginx.service
Subject: A start job for unit nginx.service has failed
Defined-By: systemd
Support: Debian -- User Support

A start job for unit nginx.service has finished with a failure.

systemctl status nginx.service
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: failed (Result: start-limit-hit) since Sat 2023-09-16 10:52:26 MDT; 20s ago
Duration: 6ms
Docs: man:nginx(8)
Process: 1072349 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 1072350 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 1072379 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 1072351 (code=exited, status=0/SUCCESS)
CPU: 125ms

Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy >
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: nginx.service: Start request repeated too quickly.
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: nginx.service: Failed with result ‘start-limit-hit’.
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy >
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: nginx.service: Start request repeated too quickly.
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: nginx.service: Failed with result ‘start-limit-hit’.
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy >
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: nginx.service: Start request repeated too quickly.
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: nginx.service: Failed with result ‘start-limit-hit’.
Sep 16 10:52:26 apache-web-server.twin-peaks-video.com systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy >lines 1-21/21 (END)

Sounds like I thing that can be controlled. Google it, so take answer with caution.
https://www.reddit.com/r/sysadmin/comments/j5gwlh/nginx_is_being_restarted_too_many_times_per_time/

do you get any output from nginx -t → which checks nginx for configuration errors. I still do not understand why you are starting apache as well as nginx. I know I asked in another post that it is actually possible but it still seems like a big no no unless you employ other services.

Let it go, @Stegan . OP explained why the word “apache” is there, and it has nothing to do with starting or stopping Apache. (It’s the hostname of the system.)

@stegan I read the article you posted, and it does look surprisingly similar to the issue I have, but this is an area of Linux I don’t know to well. I understand the principles, but not the workings. Basically I’m afraid of messing things up! @ID10T looks like he was correct when he mentioned timing being off, but as I said, I don’t know how to change it.

I did look through Log Rotate and the Action Script of NGinx, but do not see anything time related.

There’s usually an underlying cause for why it’s restarting a bunch…it’s not the timing of the thing that’s the problem, it’s almost always the symptom.

So, you almost certainly shouldn’t be trying to make it more forgiving of rapid restarts (because that isn’t going to fix whatever is making it fail to start in the first place, you’d just be wasting your time).

Look in the nginx logs in /var/log/nginx, probably. error.log is, I believe what you want. We need to know why it doesn’t start the first time systemd tries to start it, and we don’t care about the subsequent failures and the limit on how fast it is allowed to restart.

@Joe Here is the error.log

2023/09/16 10:52:26 [alert] 1068339#1068339: *208 open socket #54 left in connection 4
2023/09/16 10:52:26 [alert] 1068339#1068339: aborting
2023/09/16 12:58:18 [alert] 1110220#1110220: *33 open socket #33 left in connection 4
2023/09/16 12:58:18 [alert] 1110220#1110220: *34 open socket #35 left in connection 5
2023/09/16 12:58:18 [alert] 1110220#1110220: aborting

I hope there is something in there that helps.

you can add a debug, read here. Look complicated to make sense of the result though.

Is that all? I don’t know what to make of that.