Nginx Reboot problem (heads-up)

[Stegan]

As someone who is very naieve to all of this (and would guess many others might be) the discussion on who/what is to blame is very informative.

As for selecting an OS, that to many is often not in their direct control (is also bound up in a heap of other decisions).

I have to complement Virtualmin again here as it did provide me with the initial heads-up and solution and my initial post here was to give an additional heads-up to others. OK it was a user who initially screamed at me that the whole site was down and I did have to go searching to find out why.

OK I would have preferred Virtualmin to have done the “screaming” - so it checks and lets me know (and gives me a restart) in Dashboard -> Servers Status but this is such a critical issue (as well as if Postfix was down) that a stronger alert would have been nicer.

I admit it is really my fault. I simply assumed after performing the package updates that as Virtualmin was up and running then the websites would be, so I got on with other life. I should have checked the websites, email, etc first.

I am hoping now that I have done @johnhe 's solution, all will be ok into the future. I also agree with @joe that we should not be expecting Virtualmin to correct the mistakes/whims of the box provider, the OS or Nginx/Apache or what other software we choose to throw into the mix but Virtualmin already does some checks so why not make the negative result more prominent. The Dashboard is great but it is not even the default page and even if it was user has to open Servers Status to know something so critical is amiss.

[Joe]

@Stegan

OK I would have preferred Virtualmin to have done the “screaming” - so it checks and lets me know (and gives me a restart) in Dashboard -> Servers Status but this is such a critical issue (as well as if Postfix was down) that a stronger alert would have been nicer.

You can configure Webmin to email you, text you, etc. if a service goes down in the System and Server Status module. You can even configure it to restart the service if it’s down, but I generally recommend against that (as mentioned, if you aren’t careful you can end up making problems worse…unless the service literally is fragile and crashes for no reason sometimes, restarting just makes the problem harder to spot, harder to troubleshoot etc. and might even cause damage to data, depending on how carefully your apps are coded).

@Stegan

The Dashboard is great but it is not even the default page and even if it was user has to open Servers Status to know something so critical is amiss.

This is all configurable. I thought dashboard was the default already though. What page do you see when you first login as root?

And, you can configure any section in the dashboard to be open by default. And, I’m pretty sure if any service is down it opens automatically.

[Stegan]

@Joe

You can configure Webmin to email you, text you, etc. if a service goes down in the System and Server Status module.

That’s wonderful - But have not fathomed it out yet. Nginx is not listed there and could not fathom how to add it. An email would be nice.

@Joe

I thought dashboard was the default already though. What page do you see when you first login as root?

It is for Webmin but not for Virtualmin which defaults to List Virtual Servers I also get miffed at when the page at :10000 opens it seems to prefer Webmin (but That’s Life!)

[Joe]

@Stegan

It is for Webmin but not for Virtualmin which defaults to List Virtual Servers I also get miffed at when the page at :10000 opens it seems to prefer Webmin (but That’s Life!)

No, that’s buggy. I see it, too. I don’t know what’s happening, as it’s not how it always behaved in the past.

@Ilia did something change to make the dashboard not the default page in Virtualmin?

[Ilia]

@Joe

@Ilia did something change to make the dashboard not the default page in Virtualmin?

It has been for years like this. But we can most certainly make the dashboard default in Virtualmin too.

[Ilia]

We will change that, so the dashboard will be the default page for Virtualmin.

[Stegan]

Or, Optional? Some might prefer it the way it is.

I don’t know how others use it but I have a browser tab open for each box (and one for the forum) open all day. I can then swap to see the operational state of each with ease. It is only first thing when the page loads that this is in any way irksome. In the scheme of things pretty trivial.

[Joe]

@Stegan

Or, Optional? Some might prefer it the way it is.

It is always optional and will continue to be optional. (You can already choose to see the dashboard as the default in theme configuration.)

The only thing we’re discussing is what is the default, and my confusion over dashboard not already being the default (because it used to be, at least with the old Virtualmin Framed Theme, and I’m baffled that it apparently hasn’t been the default for the past few years since we switched to Authentic…I can’t believe I didn’t notice, but I guess I just switched it on all of my servers and never thought through why I was changing it, I dunno).

[johnhe]

I have found a scenario where the solution I proposed is not necessarily sufficient to allow nginx to start on boot.

IF
your server does not know its IPv4 and IPv6 address before starting to boot up, because it is using an ‘elastic IP service’ (such as from AWS)
THEN
the network-online.target solution is not guaranteed to work.

You may be using an ‘elastic IP service’ without even realising it or being specically told.

I have found the following works:
Add the following two lines into /etc/sysctl.conf

net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 1

Run command sysctl -p /etc/sysctl.conf

Having an ‘elastic IP service’, from a start-up perspective, is like an unexpected knock at the door.

In fact, this may be a sufficient alternative solution.

Some additional problems with nginx in the recently released Virtualmin 7.3, with fixes, are documented in

BUG with Virtualmin 7.3 Nginx redirect. Too much Perl code has leaked into nginx.conf causing failure

SYSTEM INFORMATION OS type and version Rocky Linux 8.7 Webmin version 2.001 Usermin version 1.860 Virtualmin version 7.3 Theme version 20.01.1:5 Package updates All installed packages are up to date Below is the current redirect to SSL code in nginx.conf files that now does not work. \Q and \E (to interpret literally) and the double $1 does not make sense. It looks like too much Perl code has leaked into nginx.conf, where \Q \E and /%{HTTP_HOST} does make sense. …

and in

BUG: Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/^(http|https)://%{ <-- HERE HTTP_HOST}(/.*)$/ at /usr/share/webmin/virtual-server/list_redirects.cgi line 20

How do I get this fix as I don’t see it in my current version. Also my error path is different 500 - Unescaped left brace in regex is illegal here in regex; marked by ← HERE in m/^(http|https)://%{ ← HERE HTTP_HOST}(/.*)$/ at /usr/libexec/webmin/virtual-server/list_redirects.cgi line 20. SYSTEM INFORMATION OS type and version AlmaLinux 8.7 Webmin version 2.001 Usermin version 1.860 Virtualmin version 7.3 Pro Theme version 20.01.1:5 Package updates All installed packages ar…

[Ilia]

I have taken a deep look into this issue and managed to solve this for all Virtualmin users!

But first things first!

@johnhe

Confirmed problem fixed by editing
/lib/systemd/system/nginx.service
so as
After=network.target nss-lookup.target
instead reads
After=network.target nss-lookup.target network-online.target

First of all, editing unit files directly is not good practice, as any changes can be overwritten during a package upgrade.

Instead, it should be done by running:

systemctl edit nginx

and adding:

[Unit]
After=network.target nss-lookup.target network-online.target

Furthermore, adding network.target won’t really help in the case of an IPv6 network. All contemporary distros, including Debian 12 and Ubuntu 24.04, already have:

[Unit]
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

…yet Nginx still fails to restart on all the distros I tried, starting with Alma 9 and ending with Ubuntu 24.04.

Even though this is an upstream issue, and not a Virtualmin issue itself, we will still address it in the post-installation scripts by running:

systemctl edit nginx

and adding:

[Unit]
StartLimitBurst=2
StartLimitIntervalSec=15

[Service]
Restart=on-failure
RestartSec=5s
SuccessExitStatus=SIGKILL

This will attempt to restart Nginx once after the reboot. If Nginx fails to restart after one try, it will stop trying, and you’ll need to restart it manually. If Nginx is stopped by the OOM killer, it will not restart automatically either!

[Joe]

@Ilia

Even though this is an upstream issue, and not a Virtualmin issue itself, we will still address it in the post-installation scripts by running:

I don’t think we should do that automatically.

[Ilia]

@Joe

I don’t think we should do that automatically.

Unless there is a better solution, I don’t see why not?

I’d prefer to address this problem for our users one way or another.

[Joe]

I’m not sure what the right thing is. Maybe warn about it during config check.

It doesn’t affect most users, and it indicates a misconfiguration at the network level, doesn’t it?

[Ilia]

I’m not sure what’s really causing it, but it doesn’t only affect Virtualmin users; there are dozens of posts regarding the same issue. I spent a considerable amount of time today trying to figure out what’s really going on—it’s a very obscure issue.