RedHat9 php-fpm restart of the base version deletes all .sock files for all php versions

Virtualmin
1
SYSTEM INFORMATION
OS type and version RedHat 9
Webmin version 2.402
Virtualmin version 7.30.8
Webserver version apache 2.4.62

am just starting to migrate some virtual servers from RedHat7 to RedHat9 – while keeping others on another server running RedHat8

and am seeing something that has taken me awhile to diagnose – if I have even found the actual issue :smiley:

I have used the REMI repos and have php versions 7.4, 8.0, 8.1, 8.2, 8.3 and 8.4 available — initially all is working well as I do a Virtualmin Backup, copy and then Restore to move sites from the RHEL7 server to RHEL9. Note on my new server the default php is 8.1.

then I started seeing random issues when just after a restore, other sites would stop processing php files/connections.

I forget offhand if Virtualmin lets me configure this, but it seems that all versions of php put their .sock files in /run/php-fpm/ – both on my older systems and the new RHEL9 server.

but when the php-fpm master process for 8.1 is restarted (again its the base/default version), such as during a restore of a RHEL7 server that uses 8.1, I think what happens is – EVERYTHING in /run/php-fpm/ gets deleted, not just any .sock files for 8.1 – in fact the directory /run/php-fpm/ also is deleted !! If I restart php74 or 80/82/83/84, those specific .sock files to get recreated.

it does appear that when stopping 8.4, only 8.4 .sock files are removed from /run/, leaving the others in place for the other php versions.

I have not looked yet for a configurable item, but did notice the configuration file for the default instance/stanza www.conf has its .sock file in a version specific location, such as /var/opt/remi/php84/run/php-fpm/www.sock

so I am torn between seeing how to use the version specific /run/ directory and trying to figure out what fundamental change occurred, perhaps in the systemd control file for php-fpm, between RHEL8 and RHEL9. I did find some google search references to what may be a new systemd service that is named php-fpm.socket, but have not found anything as to how to install it on my RHEL9 server, or if that would help at all.

I have cloned my production RHEL9 server to a test server so I can tinker with this all day long and not have an impact on my clients.

[ time passes ]

OMG – I have to make this quick edit before its too late for an edit update – maybe my older systems that started with much older releases of Virtualmin, have continued to use tcp ports and thus I never have had any experience with .sock files until RHEL9 and the newest Virtualmin ?? Perhaps this “issue” has been around for a long time but I never had the specific configuration to bring it to my attention !!

Please share your thoughts and advice, especially if my thinking and analysis is flawed !!

1 Like
2

Hi there,

There should be something else going on—it definitely isn’t us. Does this issue happen if you install a fresh RHEL 9? This is the recommended way, by the way!

I would highly advise against making such a migration, especially from RHEL 7, which has been over 10 years old. It will have a lot of loose ends and you’ll spend a lot more time with it rather than just spinning up a clean RHEL/Rocky/Alma 9 and migrating your website using Virtualmin’s backup and restore feature. It’s much easier:

3

Same difference?

4

Sorry my post was not clear – YES I have created a new RHEL9 server and am copying each virtual server over to it.

what I was trying to get at — the www.conf entry for REMI modules has its socket file in a REMI subdirectory …

should Virtualmin do the same – place each .sock file in that area instead of having all of them in /run/php-fpm/ ?

5

just got a chance to do a test on my old RHEL7 server.

changed two Virtual Servers to use sockets

virtualmin modify-web --domain example-A.com --php-fpm-socket
virtualmin modify-web --domain example-B.com --php-fpm-socket

this results in:

ls -1 /run/php-fpm/
14285922231368.sock
174851973941434.sock
php-fpm.pid

Note both virtual servers use php82 while the base/default is 7.3

doing

systemctl stop php-fpm

causes both .sock files to be deleted – in fact the /run/php-fpm/ directory is deleted.

then when I start php (version 73, the default), the directory is created and the .pid file is put in there – but nothing else shows up.

Only when I recycle php82 with:

systemctl restart php82-php-fpm

do the two expected .sock files reappear.

with RedHat7 being pretty old, I’m shocked no one else has experienced this issue on any RedHat/CentOS/Alma/Rocky system before :smiley:

observation — a few years ago there was a new config variable invented for Virtualmin – fpm_socket_dir – its referenced in /virtual-server/php-lib.pl in the subroutine get_php_fpm_socket_file but I don’t know how to set that thru the GUI.

6

Thank you very much for reporting and explain it!

You’re right! This is absolutely horrendous! :exploding_head: And, it not only happens when stopping the default php-fpm service but also when restarting it! It’s a massive bug, and I’m confused about how it went unnoticed for so long.

We will partially address it in Virtualmin 8 for all clean installs. Meanwhile you can easily fix it similarly by:

printf "[Service]\nRuntimeDirectoryPreserve=yes\n" > /etc/systemd/system/php-fpm.service.d/override.conf
systemctl daemon-reload
systemctl restart php-fpm

@Jamie, I think we should do the same in the Virtualmin post-install script, at least once for the upcoming 7.40.0. Or, rather better do it in config check?

observation — a few years ago there was a new config variable invented for Virtualmin – fpm_socket_dir – its referenced in /virtual-server/php-lib.pl in the subroutine get_php_fpm_socket_file but I don’t know how to set that thru the GUI.

You can set it directly by editing the /etc/webmin/virtual-server/config file, but I advise against it. On EL systems, the expected default socket directory for PHP-FPM is /run/php-fpm. Technically, we could use another path, but it won’t work with SELinux without labeling it correctly. Also, we want to stay as close to the default system configuration as possible.

What could help is to submit a bug to whoever supports the default PHP-FPM package and suggest they always add the RuntimeDirectoryPreserve=yes directive to their php-fpm.service. @Joe did it before for other EL packages, as far as I know.

However, this would only solve one part of the problem, because if the standard php-fpm.service is disabled or the default PHP package is uninstalled, then REMI PHP versions configured by Virtualmin to use the /run/php-fpm directory after reboot won’t be able to start. This is because REMI doesn’t configure their PHP-FPM systemd units to use a runtime directory at all. So, after a reboot, /run/php-fpm will not exist if the EL default PHP version is disabled or uninstalled. Ideally, REMI should address it and use default PHP-FPM directory. Yet, they instead choose to use a directory that always exists within their package, like /var/opt/remi/php84/run/php-fpm, which is understandable but not very compatible with system defaults.

@Jamie, for the overall solution, since we choose to stick to OS defaults, like using /run/php-fpm, we need to create not only a custom override for the default php-fpm package as I linked above, but also create an override for each REMI PHP-FPM version installed, in the config check to make sure an override for REMI creates a runtime directory. For example:

# /etc/systemd/system/php84-php-fpm.service.d/override.conf
[Service]
RuntimeDirectory=php-fpm
RuntimeDirectoryMode=0755
RuntimeDirectoryPreserve=yes

Arguably, we could just give up on all this entirely, and store all our sockets on all systems in a new /var/virtualmin/php-fpm directory that we can control, and those who work with SELinux or AppArmor will have to handle semanage and apparmor_parser manually.

In the end, I’d still suggest we stick to the system defaults. It won’t be hard to create the needed logic in the Virtualmin config check to make sure everything runs as expected and doesn’t get broken. Jamie, can I send you a PR to address this cleanly and make sure everything works, no matter what users do with their standard PHP package on EL systems?

7

thank you so much for looking at this so quickly … I was going nuts for a minute :smiley:

with all due respect, please don’t rush to fix my EL issue with REMI adjustments and cause issues with other distros … as I can’t remember if /run/php-fpm/ is universal or not.

and late last night, after my post, I too came across RuntimeDirectoryPreserve=yes and saw it fixed my problem !!

thanks again !!

8

Why wouldn’t me make our own runtime directory instead of fucking around with system packages? I thought that’s what we already did, actually…I feel like we’ve talked about using our own directory/path for something like this, but maybe it was some other service.

We’re not supposed to step on system packages.

9

To be clear, stuff that requires modifying system package files is not “system defaults”. We’re not doing the obvious thing by overriding system packaged unit files with modified versions. Sometimes it’s necessary, but it is, by definition, not “system default”.

10

There are tradeoffs in both cases. I covered it all in my previous comment.

Yes, we talked about it a lot here and decided to use distro-dependent socket directory, because this has always been our philosophy—to stick to OS defaults. And, back then we didn’t realize that the PHP-FPM service is configured the way it is on EL systems.

I’m not sure I fully understand what you mean, because creating a drop-in for a systemd unit doesn’t modify system packages. In fact, it’s the recommended way to adjust a unit since it survives updates without touching system package files.

I suggested adding a directive to make sure the runtime directory created earlier isn’t deleted on default php-fpm service stop or restart. We aren’t creating a modified version of the unit file; it’s just a drop-in.

11

I missed that conversation entirely. I’ll have to read and think on it.

12

unrelated? Persistent PHP-FPM Failure Issue on Rocky Linux and AlmaLinux

13

I don’t see how it could be? I mean, it’s not the same symptoms?

14

@Ilia is this related to the change from /var/run to /run for PHP socket files?

15

It cannot be related since /var/run is a symlink to /run.

16

The author in that thread states:

Scope: Only the manually installed PHP-FPM versions fail. The default system-installed PHP version works without any issues.

Since restarting the default php-fpm right now kills all other sites by removing all sockets for all REMI PHP-FPM, then yes, it’s very likely!

1 Like
17

Perhaps this topic might be worth considering in the overall scheme of php:

I suppose AppStream / OS php installs would not be affected?

18

We have to wrap this up! @Jamie, let me know one way or the other, so I can make my plans…

19

I would prefer this option for sure. If we have to modify the default systemd scripts in any way, it’s a sign we’ve done something wrong!

WOuld this just be undoing the changes we made in Fix to support distro dependent FPM socks directory · virtualmin/virtualmin-gpl@01a0256 · GitHub and Fix to use `/run` directly · virtualmin/virtualmin-gpl@103d3a5 · GitHub ?

20

I also think I prefer we have our own directory. We should also provide the policy for SELinux…I’m not sure how one does that for AppArmor, but I know you can just put the policy file in the package for SELinux, I assume it’s similar.

Oh, but why /var/virtualmin and not a run directory as the OS packages do it (e.g. /run/virtualmin/php-fpm or /var/run/virtualmin/php-fpm)? That seems non-standard, and I don’t like non-standard.