Nginx cannot be started after update

palomnik · April 1, 2024, 4:05am

SYSTEM INFO
OS type and version	Ubuntu Linux 22.04.4
Virtualmin version	Virtualmin version 7.10.0

At some point after installing updates, the Nginx server went down. I rebooted and it came back up, but then went down again and could not be restarted. The following error was returned:

Failed to start Nginx : Job for nginx.service failed because the control process exited with error code. See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.

The output of “journalctl” was as follows:

start job for unit nginx.service has begun execution.
░░
░░ The job identifier is 143.
Mar 31 01:02:17 host5 nginx[892]: nginx: [emerg] bind() to [2604:180:f4::30d]:443 failed (99: Unknown error)
Mar 31 01:02:17 host5 nginx[892]: nginx: configuration file /etc/nginx/nginx.conf test failed
Mar 31 01:02:17 host5 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStartPre= process belonging to unit nginx.service has exited.
░░
░░ The process’ exit code is ‘exited’ and its exit status is 1.
Mar 31 01:02:17 host5 systemd[1]: nginx.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit nginx.service has entered the ‘failed’ state with result ‘exit-code’.
Mar 31 01:02:17 host5 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
░░ Subject: A start job for unit nginx.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit nginx.service has finished with a failure.
░░
░░ The job identifier is 143 and the job result is failed.
Mar 31 02:58:19 host5 systemd[1]: Starting A high performance web server and a reverse proxy server…
░░ Subject: A start job for unit nginx.service has begun execution
░░ Defined-By: systemd

The output of nginx -t:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

The output of nginx -T:
all of the config files are displayed in order as expected

The latest entries in /var/log/nginx/error.log:

2024/03/31 17:36:16 [emerg] 6308#6308: open() “/usr/share/nginx/abstractionengineering.com.conf” failed (2: No such file or directory)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:42:44 [emerg] 8061#8061: still could not bind()
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 17:49:36 [emerg] 9682#9682: still could not bind()
2024/03/31 18:15:45 [emerg] 870#870: bind() to [2604:180:f4::30d]:443 failed (99: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:16:40 [emerg] 1790#1790: still could not bind()
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to 0.0.0.0:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: bind() to [::]:80 failed (98: Unknown error)
2024/03/31 18:17:01 [emerg] 2057#2057: still could not bind()

The first line is troubling - there are no config files in /usr/share/nginx, all are in /etc/nginx/sites-svailable as expected. the output of nginx -T demonstrates that nginx finds the files in the right place.

If I cd into /etc/nginx/sites-abailable, and run:

nginx -t -c filename.con, I also get a weird error about the file not being in /usr/share/nginx.

nginx -t -c ./domain.com.conf
nginx: [emerg] open() “/usr/share/nginx/./domain.com.conf” failed (2: No such file or directory)
nginx: configuration file /usr/share/nginx/./domain.com.conf test failed

I don’t see any references in the config files that reference the /usr/share directory, but for some reason, this problem is happening after the update. I have not made any changes to sites or config files. This makes me suspect that some change with the update is doing this.

All of my sites are down until I can find the issue!

jimr1 · April 1, 2024, 6:48am

This looks like an nginx bug look at the screen shot you will see I get the same error as you but as you can see nginx is running fine … so I would ignore this warning/error and look elsewhere, what output do you get from

nginx -t

palomnik · April 1, 2024, 7:04am

The output of nginx -t:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

jimr1 · April 1, 2024, 7:14am

looked at your log files something is already running on port 80 (unable to bind errors) you need to find what is using port 80 try

ss -ltpn |grep :80

a running system will show

to find out what is running on port 80

Joe · April 1, 2024, 7:14am

So, if the configuration is good (as nginx says it is), and if others see the same weirdness with it complaining about missing configs in a place configs shouldn’t be…I guess the problem is the bind error.

So, what’s squatting on port 80 and 443? Do you have another web server on one or more of the IPs on the system (you’re trying to bind to 0.0.0.0, so all IPs on the system). If you have other web servers sitting on those ports even if only on one IP, it would block nginx from binding.

Possibly nginx didn’t actually shut down, and you’re trying to start it but it’s already listening.

If you don’t know why the web service ports would be unavailable to bind, try rebooting. But, also look around for what’s actually listening on those ports (I assume something is). netstat -lnp

Joe · April 1, 2024, 7:15am

jinx, you beat me to it!

palomnik · April 1, 2024, 5:52pm

Good advice - I did:

ss -ltpn |grep :80
LISTEN 0      511                                   *:80               *:*    users:
(("apache2",pid=1051,fd=4),("apache2",pid=1023,fd=4),("apache2",pid=1020,fd=4))

wtf?

ps -fae |grep apache
root        1020       1  0 10:29 ?        00:00:00 /usr/sbin/apache2 -k start
www-data    1023    1020  0 10:29 ?        00:00:00 /usr/sbin/apache2 -k start
www-data    1051    1020  0 10:29 ?        00:00:00 /usr/sbin/apache2 -k start

WTF!!! I never installed apache2 - this system has been nginx from the start (–LEMP), and apache is deselected in features and plugins. If it somehow got installed anyway, it was never running until now. I removed apache2 and rebooted and everything was good - Still a mystery on how this could happen.

Thanks for your help guys, you saved the day!!!

system · April 9, 2024, 5:52pm

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.