not from me (is it really an Ubuntu issue?) possibly more nginx issue.
I have to say that when I checked /lib/systemd/system/nginx.service on several boxes (not only the Virtualmin boxes that the line was After=network.target not nss-lookup.target or network-online.target
Although I marked this as a solution perhaps it needs more package updates/time to tell if it really solves
The systemd unit file in a package on Ubuntu is generally maintained by Ubuntu folks, not upstream nginx folks. (But, not always. Sometimes projects include one. Though I suspect they always get tweaked by the distro vendor.)
We won’t change the system nginx package. That’d be a problem for the Ubuntu folks, which is why I asked if it was a known issue upstream. We’re pretty religious about respecting users choices…you chose an OS, so you’re going to get that OSes packages and ways of doing things (i.e. we split vhosts into their own dir in Ubuntu/Debian, and we handle enabling modules in a way that is compatible with the Ubuntu/Debian way of doing it…we do things differently on other distros).
I assume there must be a reason they made the systemd unit file behave this way, perhaps there’s some discussion about it in their issue tracker or something. I only found StackOverflow discussions about it in a quick search, but it’s apparently been a problem for some folks for some time. It might be that there is something else wrong on your system that is unrelated to nginx that breaks this. I dunno.
We don’t modify the systemd unit for nginx as part of our installation. If you have differences, they’d be because of different Ubuntu versions or because of manual modifications.
So, I think we’re still kinda in the dark about why this happens. nss-lookup and network-online are not entirely unreasonable requirements for a web server to start, unless something is broken in either of those target statuses for some reason. But, also, I’m not sure I understand why they are necessary for the web server to start. The web server will generally function locally without network online or resolution…one might even choose to do so for development purposes.
Some additional relevant information.
Nginx is instructed to listen for IP4 and IP6 addressed in my config. In my logs nginx complained about an IP6 address not been available at the time of startup of nginx. So nginx was correct not to start up and the correct solution was to add a condition to ensure all used routable IP addresses are up, which is what I did for my solution.
So the solution is correct for my conditions I and I assume the conditions of others.
It can be legitimately argued that Ubuntu and other distributions are wrong.
I don’t see any easy solution to correcting distributions.
I think Virtualmin should reconsider their decision not to adopt the solution. The solution does no harm and omitting the solution does do harm.
There is relevant information from the authors of systemd at https://systemd.io/NETWORK_ONLINE/.
network.targetindicates that the network management stack has been started. Ordering after it it has little meaning during start-up: whether any network interfaces are already configured when it is reached is not defined.
network-online.targetis a target that actively waits until the network is “up”, where the definition of “up” is defined by the network management software. Usually it indicates a configured, routable IP address of some kind. Its primary purpose is to actively delay activation of services until the network has been set up.
I have IP6 enabled but saw no complaints in the nginx logs. So not so convinced that IP6 is the issue here. Still of the opinion that nginx should have those extra additions in the nginx.service file by default (especially if they do no harm)
It looks like package maintainers, including Ubuntu, think they know better than Nginx on how to start nginx with systemd. It is hard not to laugh.
First, I will show a slight improvement in the edit required with Ubuntu.
After=network-online.target nss-lookup.target
is sufficient to fix the problem. That is just append ‘-online’ to ‘network’. All this does is tell nginx to wait until the network is up, whatever that is defined to mean (systemd words, not mine, in link https://systemd.io/NETWORK_ONLINE/).
Now here is the juicy bit. Guess what is in current nginx source code for Debian based distributions (including Ubuntu)? Yes! It is the ‘-online’ version!
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
The link is http://hg.nginx.org/pkg-oss/file/stable-1.22/debian/nginx.service
Maybe the less I say about the absurdity of all of this the better!
You chose an OS. We respect that decision.
We’re pretty religious about this…if there’s something that makes it impossible to provide a good shared hosting experience, we may make changes, but changing a systemd unit file that comes with an OS-provided package is not something we’d do lightly (as it’s not something we can do easily or reliably without replacing the entire package, which we definitely aren’t doing).
I should be clear that the corollary to “you chose an OS” is that if you don’t like the decisions made by your OS, you should consider other OSes. I don’t trust the Ubuntu folks to make good decisions, so I consistently choose Rocky, when I have a choice. That’s personal preference.
And, to go further, I don’t think this one is all that ridiculous of a decision on their part. All you need to do to avoid this problem is get your network configuration right.
Leaving aside policy issues, the implication here is that with a proper network configuration the Ubuntu systemd configuration for Nginx is correct, and that the suggested Nginx configuration for use of nginx with systemd is overkill designed to overcome problems.
Given what I have quoted from systemd, I don’t agree with this.
I believe the Ubuntu maintainers have misunderstood the difference between network.target and network-online.target and they are wrong to override the suggested configuration for Nginx. From our perspective, the first can be taken to mean ‘we are able to execute network startup commands’ and the second can be taken to mean ‘we have finished network start up commands’. Systemd lets the OS decide what starting and finishing means
So why are there not more complaints that Ubuntu has got it wrong? Ubuntu, as a server, is regarded as a beginner’s choice and a lot of beginners on servers use low cost and low core/thread VPSs with startup processes that hog a CPU during startup, preventing true parallel startup.
With real multicore parallel startup, telling a web server it can startup when core network functionality is not ready is wrong. No amount of ‘correct network configuration’ will fix this.
I wanted binary compatibility and wanted to avoid using virtual OS solutions with my desktop, so I choose Ubuntu as a server, having used Debian before. I now see this as a mistake. Ubuntu is great as a desktop and as far as I am concerned is a far better choice than the choices offered by Debian desktop, where restrictions on ‘non-free’ software use is just one among many annoyances. Not that Ububtu desktop is without annoying issues.
I have too many other problems with Ubuntu as a server Another example with Virtualmin is with fail2ban. I am going back to Debian for server use and will report if Debian does a better job with statup.
It’s possible I also misunderstand the implications of the change they’ve made. But, if they’re wrong, you’ll want to report it to them. I’m certainly willing to believe they’re wrong. Again, I don’t trust the Ubuntu devs judgement, in general…they’ve made too many weird and rushed decisions over the years for me to be comfortable with Ubuntu on a server, though I’m in the minority on this…Ubuntu is pretty much the most popular distro for everything these days.
I’ve given up on trying to convince folks to use something else. We just do the best we can with the circumstances we’re in. And, honestly, it’s still fine. Even with the questionable decisions. It has good package management, a lot of packages, and a very big community. That’s more important than being perfect. And, it’s probably still better than Debian, for a variety of reasons. One of the reasons is that Debian has a history of following Ubuntu into bad decisions. If Ubuntu makes a poor choice today, it’ll probably show up in Debian in six months to a year. Debian avoided the biggest hasty decisions (upstart, Unity), but still lots of other annoyances have made it in. And, Debian has a shorter lifecycle than LTS and smaller community…so, less suitable for beginners running servers.
Anyway, if it’s a bug or a suboptimal config, I encourage you to file an issue upstream. Maybe they’ll at least explain why they made this particular choice.
I have spun up two additional nginx Virtualmin VPSs, one with Debian 11 and the other with Rocky 9.1.
Unlike with Ubuntu 22.04, nginx starts from boot for both.
For Debian 11 the system ‘After’ config is the same as for Ubuntu 22.04. This config, as pointed out above, is wrong and so will potentially result in nginx start up failures if network startup is incomplete.
For Rocky 9.1 the systemd ‘After/Wants’ config is the same in the source code for nginx, shown above. This config is sound. Nginx won’t startup until network startup is complete.
As someone who is very naieve to all of this (and would guess many others might be) the discussion on who/what is to blame is very informative.
As for selecting an OS, that to many is often not in their direct control (is also bound up in a heap of other decisions).
I have to complement Virtualmin again here as it did provide me with the initial heads-up and solution and my initial post here was to give an additional heads-up to others. OK it was a user who initially screamed at me that the whole site was down and I did have to go searching to find out why.
OK I would have preferred Virtualmin to have done the “screaming” - so it checks and lets me know (and gives me a restart) in Dashboard -> Servers Status but this is such a critical issue (as well as if Postfix was down) that a stronger alert would have been nicer. 
I admit it is really my fault. I simply assumed after performing the package updates that as Virtualmin was up and running then the websites would be, so I got on with other life. I should have checked the websites, email, etc first. 
I am hoping now that I have done @johnhe 's solution, all will be ok into the future. I also agree with @joe that we should not be expecting Virtualmin to correct the mistakes/whims of the box provider, the OS or Nginx/Apache or what other software we choose to throw into the mix but Virtualmin already does some checks so why not make the negative result more prominent. The Dashboard is great but it is not even the default page and even if it was user has to open Servers Status to know something so critical is amiss.
You can configure Webmin to email you, text you, etc. if a service goes down in the System and Server Status module. You can even configure it to restart the service if it’s down, but I generally recommend against that (as mentioned, if you aren’t careful you can end up making problems worse…unless the service literally is fragile and crashes for no reason sometimes, restarting just makes the problem harder to spot, harder to troubleshoot etc. and might even cause damage to data, depending on how carefully your apps are coded).
This is all configurable. I thought dashboard was the default already though. What page do you see when you first login as root?
And, you can configure any section in the dashboard to be open by default. And, I’m pretty sure if any service is down it opens automatically.
That’s wonderful - But have not fathomed it out yet. Nginx is not listed there and could not fathom how to add it. An email would be nice.
It is for Webmin but not for Virtualmin which defaults to List Virtual Servers I also get miffed at when the page at :10000 opens it seems to prefer Webmin (but That’s Life!)
No, that’s buggy. I see it, too. I don’t know what’s happening, as it’s not how it always behaved in the past.
@Ilia did something change to make the dashboard not the default page in Virtualmin?
It has been for years like this. But we can most certainly make the dashboard default in Virtualmin too.
We will change that, so the dashboard will be the default page for Virtualmin.
Or, Optional? Some might prefer it the way it is.
I don’t know how others use it but I have a browser tab open for each box (and one for the forum) open all day. I can then swap to see the operational state of each with ease. It is only first thing when the page loads that this is in any way irksome. In the scheme of things pretty trivial.
It is always optional and will continue to be optional. (You can already choose to see the dashboard as the default in theme configuration.)
The only thing we’re discussing is what is the default, and my confusion over dashboard not already being the default (because it used to be, at least with the old Virtualmin Framed Theme, and I’m baffled that it apparently hasn’t been the default for the past few years since we switched to Authentic…I can’t believe I didn’t notice, but I guess I just switched it on all of my servers and never thought through why I was changing it, I dunno).
I have found a scenario where the solution I proposed is not necessarily sufficient to allow nginx to start on boot.
IF
your server does not know its IPv4 and IPv6 address before starting to boot up, because it is using an ‘elastic IP service’ (such as from AWS)
THEN
the network-online.target solution is not guaranteed to work.
You may be using an ‘elastic IP service’ without even realising it or being specically told.
I have found the following works:
Add the following two lines into /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 1
Run command sysctl -p /etc/sysctl.conf
Having an ‘elastic IP service’, from a start-up perspective, is like an unexpected knock at the door.
In fact, this may be a sufficient alternative solution.
Some additional problems with nginx in the recently released Virtualmin 7.3, with fixes, are documented in
and in
This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.