PHP 8.0 - 500 server error after package upgrades

After an update to Webmin today at 1100 central, 5 out of 6 virtual domains hosted on my box no longer parse PHP files. This has left us in a production down situation.

I have tried to pay Virtualmin support to help, but have only gotten questions about why I would be running PHP 8 and criticism of the base configuration of this server instead of working the problem (we remove mail related software because no mail services are needed for example).

Whenever any php file is rendered, including a file with exactly “hi” in the file with zero PHP code, the result is a 500 server error with this in the logs:

[cgi:error] [pid 17175] [client x.x.x.x:53959] End of script output before headers: php8.0.cgi

Comparing the 1 working site to the 5 broken sites, the 1 working site is set to use mod_php for the PHP execution method. The 5 broken sites are set to use the CGI wrapper.

I believe they should all be set to mod_php, and mod_php is installed and working for the 1 working site, but mod_php is not listed in the virtualmin UI as an option for those 5 broken sites so I cannot change it. mod_php IS listed and selected for the 1 working site.

My guess is that the webmin update tripped over the virtual host or apache configs and removed the mod_php options - but I do not know how that works, I’m only guessing based on the before and after that I’m seeing. This server has been configured how it is and running for almost a year with daily updates and weekly reboots.

Is there anyone listening who might be able to help? I can restore the server to last night’s backup, but when the update runs again, I believe this will simply break again and need to understand the root cause of the problem to prevent it from occurring again.

SYSTEM INFORMATION
OS type and version: Ubuntu 18.04.5 LTS
Webmin version: 1.981
Virtualmin version: 6.16
Related products version: Apache 2.4.48, PHP 8.0.10, MySQL 5.7.35

PHP8 strikes again.

I have tried to pay Virtualmin support to help

This is not an option for now, and for the next few months, as we are pretty busy preparing a new release, alongside with post-migration hassle.

Including all other dependencies?

[cgi:error] [pid 17175] [client x.x.x.x:53959] End of script output before headers: php8.0.cgi

Have you tried changing mode for PHP files adding an exec bit?

the 1 working site is set to use mod_php for the PHP execution method.

It doesn’t mean you should fix to switch other 5 broken sites to use mod_php.

mod_php is a horrible solution for shared hosting environments. Besides mod_php is getting deprecated and removed upstream.

The right thing to do would be is to make sure that you have php8.0-fpm package installed and use PHP FPM or PHP FCGI mode. After installing PHP FPM package, you should re-run configuration check, and changing execution mode on PHP Options page.

mod_php IS listed and selected for the 1 working site.

Have a look at /etc/webmin/virtual-server/domains is where all virtual servers configs are held. Perhaps diffing working against non-working virtual server config file would give you more insights.

My guess is that the webmin update tripped over the virtual host

Webmin update doesn’t touch Virtualmin configuration, neither Apache – less likely.

I believe this will simply break again and need to understand the root cause of the problem to prevent it from occurring again.

It would be interesting to find out why – please share if you do, and even though it is not a difficult process to understand what exactly happens upon package upgrade, it can be very time consuming.

2 Likes

correct me if I’m wrong but PHP version on Ubuntu is I think 7.4 and that’s on 20.04 no?

well PHP 8 have some issues with backwards compatibility even for php7.4 and lower. if it’s not officially in distributions official repos I would assume you know how to troubleshoot it as you did installed via custom repositories for some mysterious reason.

I wouldn’t touch it. good luck.

1 Like

That is correct.

I think what our little Bo did was run out and get php8 to have the latest and greatest like so many others did and it broke his sites because half of what he’s running isn’t compatible with it yet.

@Gomez_Adams I think he installed php8 just fine but then when did apt upgrade it did upgraded PHP from non official repos with some latest changes which are not completely compatible with his distro or his setup and it broke it. when you do custom setups do so but make sure you know what you doing…

anyway non of us is helpful here, lets not spam the topic.

Thank you for this post. Definite improvement.

Now we’re getting somewhere…

The following PHP-FPM versions cannot be used : 8.0.10 (Apache module mod_fcgid is missing or not enabled)

Installed mod_fcgid - it was missing - don’t know why - if it comes with a virtualmin install, I certainly didn’t remove it so where did it go?

root@itweb1:/# apt install libapache2-mod-fcgid
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
libapache2-mod-fcgid
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 70.5 kB of archives.
After this operation, 272 kB of additional disk space will be used.
Get:1 Index of /ubuntu bionic/universe amd64 libapache2-mod-fcgid amd64 1:2.3.9-1 [70.5 kB]
Fetched 70.5 kB in 0s (155 kB/s)
Selecting previously unselected package libapache2-mod-fcgid.
(Reading database … 212098 files and directories currently installed.)
Preparing to unpack …/libapache2-mod-fcgid_1%3a2.3.9-1_amd64.deb …
Unpacking libapache2-mod-fcgid (1:2.3.9-1) …
Setting up libapache2-mod-fcgid (1:2.3.9-1) …
apache2_invoke: Enable module fcgid

After this install, config check still says

The following PHP-FPM versions cannot be used : 8.0.10 (Apache module mod_fcgid is missing or not enabled)

(other answers)

This is good to know. Updating the support page to explain that would be helpful to set expectations for users in need of support.

3 or 4 things were removed (sendmail, clamav, etc…) with apt -remove packagename and then apt purge packagename. I’m sure I could stand to learn better or more complete ways to get rid of un-needed packages and would welcome helpful professional input. This all occurs for every Linux server we deploy as part of our security hardening process and occurred about a year ago for this server. If we can find the root cause of the issue today, and it points to an email related package removal, then we will certainly update our process.

Have now. No change.

Good to know. I was under the opposite understanding. I thought I had seen the CGI method for decades and mod_php was the new one. I’m probably mixing up fast CGI or some other flavor of a CGI parser.

It is:
php-fpm is already the newest version (2:8.0+84+ubuntu18.04.1+deb.sury.org+1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I get where you’re coming from, but we built this server from the ground up last year with PHP 8 because of compliance requirements put upon us (we’ll grumble together about that later). It has been running PHP 8 for that entire time doing updates from the custom repo on an automated daily schedule. Today however, APT history only shows an update to Webmin and 3 days ago shows an update to libntfs. Could a PHP update have snuck in there and not get logged? You tell me, maybe so, but I can only go by what the logs are telling me at the moment.

The software we run on this machine is EXTREMELY basic. Not a reason in the world to have compatibility issues with PHP itself. Literally uses 3 basic PHP functions (fgetcsv, date, and echo). Seriously, VERY basic stuff from a PHP perspective (no external libraries, and about 50 lines of code total, most of which is white space). The power here and business reliance comes from the data that is parsed, not the complexity of the code. Essentially converting one text file format to another format because other software is lacking the ability to parse anything except one proprietary format.

If you can point me to somewhere I can verify this, that would be helpful. Apt history shows no updates to PHP, only an update to Webmin.

The mod_php issue may be a red herring, but just found that mod_fcgid was missing… don’t know if it was there before or how it would have been removed without any evidence in the apt logs, but doesn’t matter how or why at this moment.

Absolutely get your point. We rely on software like Virtualmin because we do not have any in-house Linux experts. Those seem to be hard to peel away from forums - I have offered six figures several times and have no luck finding someone.

This time (albeit a year ago now) we stepped outside the box a bit and updated PHP to 8 because of a compliance requirement outside of my control. After 60 days of testing last year, given the extremely basic PHP code we’re running, all was fine and was released into production.

A year’s worth of updates have not brought any issues to light, but maybe that is what occurred here? Completely open to that possibility, but so far I haven’t found any log saying that PHP was updated on this server this week. I do not know these systems well enough to know if I’m even looking in all the right spots however, so any guidance there would be helpful.

If there is anywhere other than the APT history to look at to know when an update has been installed for PHP, I’m all ears.

Fixed.

Try installing Virtualmin GPL version on local virtual machine and try removing, let’s say, postfix and this is what you will get:

root@ubuntu20-pro:/etc# apt-get remove postfix
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  awstats bind9 bind9-utils certbot clamav clamav-base clamav-daemon clamav-docs clamav-freshclam clamav-testfiles clamdscan db-util db5.3-util dovecot-core
  dovecot-imapd dovecot-pop3d etckeeper fonts-lato gcc gcc-9 geoip-database jailkit javascript-common libapache2-mod-fcgid libasan5 libatomic1
  libauthen-oath-perl libberkeleydb-perl libc-dev-bin libc6-dev libcc1-0 libcgi-fast-perl libcgi-pm-perl libclamav9 libcommon-sense-perl
  libconfig-inifiles-perl libcrypt-dev libcrypt-openssl-bignum-perl libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl libdbd-mysql-perl libdbi-perl
  libdigest-bubblebabble-perl libencode-locale-perl libevent-core-2.1-7 libevent-pthreads-2.1-7 libexporter-tiny-perl libexttextcat-2.0-0 libexttextcat-data
  libfcgi-bin libfcgi-dev libfcgi-perl libfcgi0ldbl libgcc-9-dev libgeoip1 libhiredis0.14 libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
  libhttp-date-perl libhttp-message-perl libimport-into-perl libio-html-perl libio-multiplex-perl libitm1 libjs-jquery libjson-perl libjson-xs-perl liblsan0
  liblua5.3-0 liblwp-mediatypes-perl libmail-authenticationresults-perl libmail-dkim-perl libmail-spf-perl libmecab2 libmemcachedutil2 libmoo-perl
  libmysqlclient21 libnet-cidr-perl libnet-dns-perl libnet-dns-sec-perl libnet-ip-perl libnet-rblclient-perl libnet-server-perl libnet-xwhois-perl
  libnetaddr-ip-perl libonig5 libparse-syslog-perl libperl4-corelibs-perl libquadmath0 libruby2.7 libspf2-2 libstrictures-perl libtfm1 libtsan0
  libtype-tiny-perl libtype-tiny-xs-perl libtypes-serialiser-perl libubsan1 linux-libc-dev manpages-dev mecab-ipadic mecab-ipadic-utf8 mecab-utils
  milter-greylist mysql-client mysql-client-8.0 mysql-client-core-8.0 mysql-common mysql-server mysql-server-8.0 mysql-server-core-8.0 ntpdate p7zip php-cgi
  php-fpm php-mbstring php-mysql php7.4-mbstring php7.4-mysql postgrey procmail procmail-wrapper proftpd-basic proftpd-doc python3-acme python3-certbot
  python3-configargparse python3-future python3-icu python3-josepy python3-mock python3-parsedatetime python3-pbr python3-ply python3-requests-toolbelt
  python3-rfc3339 python3-tz python3-zope.component python3-zope.event python3-zope.hookable rake re2c ri ruby ruby-minitest ruby-net-telnet
  ruby-power-assert ruby-test-unit ruby-xmlrpc ruby2.7 ruby2.7-doc rubygems-integration sa-compile sasl2-bin spamassassin spamc unrar webalizer
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  postfix postfix-pcre virtualmin-lamp-stack
0 upgraded, 0 newly installed, 3 to remove and 15 not upgraded.
After this operation, 4,708 kB disk space will be freed.
Do you want to continue? [Y/n] n

Even CGI is better than mod_php.

You should use our guide to install PHP 8. I did, it works for me just fine. However, there are hundreds of ways of breaking it. I don’t know what happened on your side exactly. If it is something obvious, we would fix it (prevent it from happening in the future for other people) but we need to know the steps to reproduce an issue.

There is no reason to put any extra efforts in building anything – just install the server with ./install.sh --minimal and later install your PHP 8.0, which works out of the box:

add-apt-repository ppa:ondrej/php && apt-get update
apt-get install php8.0-{cli,pdo,fpm,zip,gd,xml,mysql,cgi}
virtualmin check-config

If you later delete anything, for example, default version of PHP, you may get in trouble, unless you know exactly what you’re doing.

I get that, and I think I follow your point. However, given that MYSQL and Perl, and Python, etc… are all still installed and showing versions on the Virtualmin Software versions list, obviously the autoremove option was not run and all of those things were not removed.

Reviewing the exact things removed from this server now, postfix was not one of the things removed, it is still installed. ClamAV, ProFTPD, SpamAssassin are all gone, but those are the only things I see gone at the moment. I was not being precise there because I did not think it mattered given that all occurred last year. I can appreciate that certain removals may matter.

The re-check config error has gone away now. I haven’t changed anything else.

Apache is installed.
The following PHP versions are available : 8.0.10 (/usr/bin/php)
The following PHP-FPM versions are available on this system : 8.0.10 (php8.0-fpm)
The following PHP execution modes are available : cgi fcgid fpm

I now have 3 options for PHP execution mode. Tried FCGId, no change. Tried FPM, page loads!

Page isn’t loading fully, looks like it’s missing stylesheet info maybe… checking includes now…

1 Like

Essentially, we are on the same page and did just that. Although the --minimal switch is something I did not know about and will be very valuable to us in the future. Thank you for pointing that out!

Noted. In general, following guides from ‘big names’ hasn’t bitten us, but the PHP 8 guide we found did include the purge steps that we felt were required to satisfy our compliance requirement. We will have to review that with future testing to see if the purge steps are truly needed.

1 Like

The by-product of installing mod-fcgid is that it reset my PHP.INI settings. The only one that matters today is short_open_tags, which I know is deprecated and is on our list to address, but not today.

The php.ini that seems to apply however is eluding me. Usually in our virtualmin deployments the ini I care about is /home/domain/etc/php.ini that is a sym link to another file.

Now, that looks like the case, but the file link is different and I’m not able to write to this file at the moment even sudo as root.

lrwxrwxrwx 1 bsbks bsbks 14 Sep 3 16:08 php.ini → php8.0/php.ini

I am not currently finding the actual target file of that link…

I see, perhaps this will help then – edit /etc/webmin/phpini/config file and replace php_ini directive with the following:

php_ini=/etc/php*/apache2/php.ini,/etc/php/*/apache2/php.ini=Configuration for mod_php	/etc/php*/cgi/php.ini,/etc/php/*/cgi/php.ini=Configuration for scripts run via CGI	/etc/php*/cli/php.ini,/etc/php/*/cli/php.ini=Configuration for command-line scripts

Everything is working again. Thank you all for your help along the way. Not sure if there is one answer I can mark that will help anyone, but I’ll look.

Still do not know what actually happened today at 1100 central. I do have confirmation that it occurred at that time now as the system was used successfully at 1040 and we got our first call about it being broken at 1107 with confirmation at 1110.

The AAR at this point though is that I installed libapache2-mod-fcgid that may or may not have been there and disappeared somehow. Then set my sites to use fpm, that was installed, but never used before (I have always edited php.ini and restarted only apache to take effect, but now have to restart fpm which is brand new to me).

My sites must have been using the CGI wrapper execution method and whatever occurred at 1100 broke that method (and that method remains broken).

Our follow-up will be to build a new webserver using the --minimal option for virtualmin and we will submit a compliance exception request to avoid having to mess with PHP 8 again - because it seems likely that PHP 8 is somehow to blame for something here.

1 Like

I’ve seen nothing but problems from anybody that uses it.

not to get too off topic but, am I correct that when using the fpm variation, the INI file of /home/domain/etc/php.ini is NOT USED but instead the more general one of /etc/php.ini …

… and any custom domain specific changes have to be made in the domain specific file /etc/php-fpm.d/xxxxxxx.conf ?

Note there is a specific syntax required in those php-ftp.d files – not the syntax seen in /etc/php.ini

I welcome any corrections or expansion on the above !!