Operating System: Rocky Linux 9 (also tested on AlmaLinux)
Webmin Version: 2.202
Virtualmin Version: 7.20.2
Web Server: Apache 2.4.62
PHP Configuration: Multiple PHP versions installed via PHP-FPM (default and custom versions).
Issue Description
For over a year, I have been experiencing random PHP failures on my server, which manifest as 503 Service Unavailable errors. Here’s a summary of the problem:
Timing: Failures typically occurred every Sunday or Monday, irrespective of usage patterns.
Workaround: Restarting the affected PHP-FPM version or the server temporarily resolves the issue.
Scope: Only the manually installed PHP-FPM versions fail. The default system-installed PHP version works without any issues.
Reproduction: The issue recurs when:
A new virtual server is created.
Custom PHP versions are installed.
Despite multiple attempts at resolving this, the root cause remains elusive.
Steps Taken
I have extensively investigated and attempted fixes based on various suggestions:
Log Rotation Hypothesis:
It was suggested that log rotation might be causing PHP to stop responding.
Adjustments were made, but the issue persisted.
Backup Scheduling Hypothesis:
Suspected conflicts with scheduled backups were addressed by disabling backup jobs.
This had no impact on the problem.
Reinstallations and Distribution Testing:
Reinstalled the OS (both Rocky Linux and AlmaLinux).
Restored server backups.
Issues persisted on both distributions.
PHP-FPM Specifics:
Default system PHP version (e.g., 8.0) runs without issues.
Manually installed versions (e.g., 7.4, 8.1) fail consistently under the same conditions.
Switching to Debian:
Migrated to Debian-based distributions as a last resort.
The problem does not occur on Debian, suggesting the issue may be tied to Red Hat-based distributions or their PHP implementation.
Key Observations
The issue appears to affect Rocky Linux and AlmaLinux identically, potentially pointing to a shared upstream cause in Red Hat-based systems.
Custom-installed PHP versions consistently fail under similar conditions, while the default PHP version works reliably.
Creating a new virtual server or other resource-intensive operations seem to exacerbate the problem.
References
I have shared my experiences on the Virtualmin forums, where several others have encountered similar issues. Relevant discussions:
This is from Debian 11. You have anything different?
root@main:/etc/cron.weekly# ls -lath
total 24K
drwxr-xr-x 133 root root 12K Dec 6 10:21 ..
drwxr-xr-x 2 root root 4.0K Feb 6 2023 .
-rwxr-xr-x 1 root root 813 Feb 10 2019 man-db
-rw-r--r-- 1 root root 102 May 3 2015 .placeholder
And an obvious question. Is there some PHP software running maintenance once a week? I’d think most maintained software offers an upgrade path to 8 by now.
Then why is php7 on the server? A reluctance to upgrade from old versions of WP? Believe me, I know the pain of failed upgrades. I’ve walked out to way too many sunrises. But, that said, you seem to be chasing a problem you should be trying to eliminate by moving away from anyhow. Ultimately the fix for php7 problems is php8.
Your main problem here is what I see on a site I help moderate. There it is a little easier to say ‘upgrade to the latest version because there has been many bug fixes’ than it is with php7. Few take an interest in trying to solve a problem with old software.
I’d be very surprised if something is broken about Red Hat-based distros. It’s (Rocky, specifically) what I use for everything (including Virtualmin.com), and we haven’t seen similar issues.
I’m glad you got it sorted, but I wouldn’t want to encourage folks to do dramatic things like a complete OS switch without good reason. Seems like there has to be an explanation for what was going wrong on your system. But, it’s certainly true that third-party packages are always less well-tested than the distribution standard packages. I recommend using the OS standard packages when possible.
As you say the issue affects “custom” versions of PHP-FPM which are obviously distributed from a third-party repo. Perhaps these concerns should be directed to them for debugging, as if this is a bug, they’d be ones fixing it.
*** There is a reason why we “caution” the use of “third-party” repos and their software. They are not tested nor directly supported by the distro (ex. RedHat, Alma, Fedora, Debian, Ubuntu, etc). ***
Interesting read. I have two servers that used to run on CENTOS 7 but due its EOL had to change to AlmaLinux 8. With CENTOS 7: never had an issue but on both AlmaLinux instances I have near identical issues, very clearly related to PHP-FPM (running PHP8.3).
I’m not sure what to do yet. Don’t want to switch to Debian if I can help it.
I have been seeing similar issues, but only when mutliple versions of PHP are installed. I use the Remi Collet repo, which is has always been very reliable (especially as he is the maintainer of the Red Hat PHP releases).
One thing I have observed is that Virtualmin does not detect the PHP-FPM service as being ‘down’.
I am no expert in this situation. Only thing i do know is using the default apache install and used the documentation to add the versions of PHP. But if i can remember right there was menstions of a socket file that is not found when i go to that file using winscp the file is there 0B in size when i double click on it to open it gives me a simular error message that the file is not found yet its there and i was as root. And all of them are in one folder does not matter what version it was.
Thanks for confirming @crazyeyes198, it’s clearly got several ‘triggers’ reading through the other posts.
It’s an odd issue and I just encountered it again. This time it was due to very high load on the web server.
On this server we have 2 versions of PHP:
$ dnf repoquery --info --installed php*
....
Name : php-fpm
Version : 8.0.30
Release : 1.el9_2
Architecture : x86_64
Size : 8.2 M
Source : php-8.0.30-1.el9_2.src.rpm
Repository : @System
From repo : rhel-9-for-x86_64-appstream-rpms
and
Name : php83-php-fpm
Version : 8.3.14
Release : 1.el9.remi
Architecture : x86_64
Size : 8.2 M
Source : php83-php-8.3.14-1.el9.remi.src.rpm
Repository : @System
From repo : remi-safe
Apache returned a 503 error for the sites running PHP 8.3 and I could see the socket errors as above, however when I checked the status using php83-php-fpm everything was ‘apparently’ operating as expected.
systemctl status php83-php-fpm
● php83-php-fpm.service - The PHP FastCGI Process Manager
Loaded: loaded (/usr/lib/systemd/system/php83-php-fpm.service; enabled; preset: disabled)
Active: active (running)
The PHP 8.0 sites were not affected and php-fpm (the PHP 8.0 FPM process) did not need to be restarted.
I’ve noticed that if I manually install a PHP version—like PHP 7.4—and it fails under heavy load for example, all the other manually installed PHP versions seem to fail too. Is that normal?
But, that said, you seem to be chasing a problem you should be trying to eliminate by moving away from anyhow. Ultimately the fix for php7 problems is php8.