Nginx Reboot problem (heads-up)

Virtualmin
21

As someone who is very naieve to all of this (and would guess many others might be) the discussion on who/what is to blame is very informative.

As for selecting an OS, that to many is often not in their direct control (is also bound up in a heap of other decisions).

I have to complement Virtualmin again here as it did provide me with the initial heads-up and solution and my initial post here was to give an additional heads-up to others. OK it was a user who initially screamed at me that the whole site was down and I did have to go searching to find out why.:frowning:

OK I would have preferred Virtualmin to have done the “screaming” - so it checks and lets me know (and gives me a restart) in Dashboard -> Servers Status but this is such a critical issue (as well as if Postfix was down) that a stronger alert would have been nicer. :hammer: :scream:

I admit it is really my fault. I simply assumed after performing the package updates that as Virtualmin was up and running then the websites would be, so I got on with other life. I should have checked the websites, email, etc first. :yawning_face:

I am hoping now that I have done @johnhe 's solution, all will be ok into the future. I also agree with @joe that we should not be expecting Virtualmin to correct the mistakes/whims of the box provider, the OS or Nginx/Apache or what other software we choose to throw into the mix but Virtualmin already does some checks so why not make the negative result more prominent. The Dashboard is great but it is not even the default page and even if it was user has to open Servers Status to know something so critical is amiss.

22

You can configure Webmin to email you, text you, etc. if a service goes down in the System and Server Status module. You can even configure it to restart the service if it’s down, but I generally recommend against that (as mentioned, if you aren’t careful you can end up making problems worse
unless the service literally is fragile and crashes for no reason sometimes, restarting just makes the problem harder to spot, harder to troubleshoot etc. and might even cause damage to data, depending on how carefully your apps are coded).

This is all configurable. I thought dashboard was the default already though. What page do you see when you first login as root?

And, you can configure any section in the dashboard to be open by default. And, I’m pretty sure if any service is down it opens automatically.

23

That’s wonderful - But have not fathomed it out yet. Nginx is not listed there and could not fathom how to add it. An email would be nice.

It is for Webmin but not for Virtualmin which defaults to List Virtual Servers I also get miffed at when the page at :10000 opens it seems to prefer Webmin (but That’s Life!)

24

No, that’s buggy. I see it, too. I don’t know what’s happening, as it’s not how it always behaved in the past.

@Ilia did something change to make the dashboard not the default page in Virtualmin?

25

It has been for years like this. But we can most certainly make the dashboard default in Virtualmin too.

26

We will change that, so the dashboard will be the default page for Virtualmin.

27

Or, Optional? Some might prefer it the way it is.

I don’t know how others use it but I have a browser tab open for each box (and one for the forum) open all day. I can then swap to see the operational state of each with ease. It is only first thing when the page loads that this is in any way irksome. In the scheme of things pretty trivial.

28

It is always optional and will continue to be optional. (You can already choose to see the dashboard as the default in theme configuration.)

The only thing we’re discussing is what is the default, and my confusion over dashboard not already being the default (because it used to be, at least with the old Virtualmin Framed Theme, and I’m baffled that it apparently hasn’t been the default for the past few years since we switched to Authentic
I can’t believe I didn’t notice, but I guess I just switched it on all of my servers and never thought through why I was changing it, I dunno).

29

I have found a scenario where the solution I proposed is not necessarily sufficient to allow nginx to start on boot.

IF
your server does not know its IPv4 and IPv6 address before starting to boot up, because it is using an ‘elastic IP service’ (such as from AWS)
THEN
the network-online.target solution is not guaranteed to work.

You may be using an ‘elastic IP service’ without even realising it or being specically told.

I have found the following works:
Add the following two lines into /etc/sysctl.conf

net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 1

Run command sysctl -p /etc/sysctl.conf

Having an ‘elastic IP service’, from a start-up perspective, is like an unexpected knock at the door.

In fact, this may be a sufficient alternative solution.

Some additional problems with nginx in the recently released Virtualmin 7.3, with fixes, are documented in

and in

31

I have taken a deep look into this issue and managed to solve this for all Virtualmin users!

But first things first!

First of all, editing unit files directly is not good practice, as any changes can be overwritten during a package upgrade.

Instead, it should be done by running:

systemctl edit nginx

and adding:

[Unit]
After=network.target nss-lookup.target network-online.target

Furthermore, adding network.target won’t really help in the case of an IPv6 network. All contemporary distros, including Debian 12 and Ubuntu 24.04, already have:

[Unit]
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target


yet Nginx still fails to restart on all the distros I tried, starting with Alma 9 and ending with Ubuntu 24.04.

Even though this is an upstream issue, and not a Virtualmin issue itself, we will still address it in the post-installation scripts by running:

systemctl edit nginx

and adding:

[Unit]
StartLimitBurst=2
StartLimitIntervalSec=15

[Service]
Restart=on-failure
RestartSec=5s
SuccessExitStatus=SIGKILL

This will attempt to restart Nginx once after the reboot. If Nginx fails to restart after one try, it will stop trying, and you’ll need to restart it manually. If Nginx is stopped by the OOM killer, it will not restart automatically either!

33

I don’t think we should do that automatically.

34

Unless there is a better solution, I don’t see why not?

I’d prefer to address this problem for our users one way or another.

35

I’m not sure what the right thing is. Maybe warn about it during config check.

It doesn’t affect most users, and it indicates a misconfiguration at the network level, doesn’t it?

36

I’m not sure what’s really causing it, but it doesn’t only affect Virtualmin users; there are dozens of posts regarding the same issue. I spent a considerable amount of time today trying to figure out what’s really going on—it’s a very obscure issue.

37

I have changed it to use a timer. It’s much cleaner, as it does exactly what we want without altering other service behavior.

For anyone willing to apply the fix manually, create a file /etc/systemd/system/nginx.timer with the following content:

[Unit]
Description=Start Nginx after boot
PartOf=nginx.service

[Timer]
OnActiveSec=10
Unit=nginx.service

[Install]
WantedBy=multi-user.target

After reboot Nginx should always be running!

38

I diagree with @Ilia that the issue is very obscure.

The reason for the problem is that ALL the Linux distors are not packaging Nginx properly. I pointed it out months ago.

The solution is clear and in official Nginx documentation. I provided a link. I would not call that obscure.

Ideally the distos should be notified. Does not mean they will fix it or it is worth the considerable personal effort of reporting it.

If Virtualmin does not want to correct distro errors, that is their choice.

Apart from that my current solution has changed from what I originally used and reported as a fix. I simply add two lines to /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 1

It works for me. It can cause other problems.

39

The sample code config provided by Nginx is suitable for a baseline server setup that does not get fed its IP addresses during boot up.

From Nginx’s point of view, why should they provide sample configs for every scenario, particularly when they have documented what is required?

Maybe we are being too idealistic and a more cynical approach can fix this.

Someone might request from Nginx that they add server config samples on a best practice approach for a server that does get IP addresses remotely, as it is now very common and it causing a lot of issues.

If this is done then distros can have the sample configs shoved in their faces and be embaresssed into fixing the problem.

40

Disagree or not, but what you’re suggesting, and what you referred to as a working solution, in fact, isn’t. Ubuntu 24.04 and others have already used the suggested configuration, yet the problem remains:

image
image1444×712 77.6 KB

image
image1920×282 54 KB

This allows your system to bind to IP addresses that are not currently assigned to any local network interface—of course, it “solves” the problem. However, it’s an unacceptable change for a stock Virtualmin system due to the potential security and operational risks.

Except it simply doesn’t work. It should in theory, but it doesn’t in practice.

So far, I think the best, simplest, and least invasive solution is to use a timer to start Nginx shortly after the boot is finished.