I opened Awstats last night to check a particular site’s traffic, and found that it hadn’t updated since June 26th. I tried to do a manual update, but got a (false) error that another update was in progress. After determining that there was no update in progress, I deleted the lock file, restarted Apache, and did some other poking around, but still got the same error.
I believe (but am not certain) that June 26th was also the day I switched all the sites over to PHP-FPM. I’d been holding off on it because I’d been getting a lot of 503’s for no discernible reason the first time I tried it.
I disabled the lock file in the Awstats configuration, but still was no able to update from the GUI. So I manually ran
/etc/webmin/virtualmin-awstats/awstats.pl example.tld , as well as tried to run Webalizer from within Virtualman to determine whether the problem may have been with the underlying access log rather than Awstats per se. After it became apparent that the processes were running but were going to take a long time, I went to bed.
This morning I awoke to 503’s on several sites, including the one that had the problem with Awstats not updating. Awstats was now up-to-date and all the missing days had been populated, but the site was 503 (along with several others). Nothing in the PHP-FPM logs, and nothing in the Apache error log except some complaints about my 404 directive in .htaccess using the full URL to the 404 page.
Restarting PHP-FPM didn’t help. Restarting Apache didn’t help. Even rebooting the server didn’t help other than bringing it to my attention that PHP-FPM wasn’t enabled as a start-up service, which I fixed.
Switching the PHP execution mode to FCGId fixed the problem on the site that had had the Awstats problem (which also is the busiest site hosted by the server). In fact, it fixed the problem and also fixed the 503’s on the other sites that were 503.
I switched the problem site back to PHP-FPM, and it 503’d, along with the same handful of other sites. So I switched the PHP execution back to FCGId and did some more digging.
What I found was that the problem site was the only one on the server that had the per-directory PHP option available. It was not selected for anything, but it was available; and the PHP 5 version was obsolete (5.3.something). I would have sworn I’d updated it after I installed Virtualmin, but apparently I didn’t.
Not surprisingly, the other sites that 503’d along with the problem site were also using the obsolete PHP 5. One of them needed PHP 5 because of some old MySQL code. The rest… I really don’t know why they were using it.
I did some quick updates to the one site with the old SQL and switched that and the other sites to PHP 7.2. I also updated PHP 5.3.something to 5.6.40. Then I re-enabled PHP-FPM on the problem site; and everything worked fine. Eight hours later, it still looks good.
Another odd thing is that the problem site no longer shows the per-directory PHP version option, which is fine because I don’t need it anyway. It just seems odd that it was available when PHP 5 was obsolete, but not after it’s been updated. Virtualmin picked up on the update when I re-ran the Configuration Check and gave it its blessing, however, so it knows it’s there.
In the end, I really have no idea why an obsolete PHP version would cause the problem site and the other sites to 503 when the problem site wasn’t even using PHP 5.
The problem site was one of the sites I migrated over from cPanel, so that may have had something to do with it. But most cPanel-related problems seem related to garbage code that cPanel puts in the .htaccess files, and there’s none of that in the problem site.
I also have no idea why the problem just manifested last night (the server has been online for more than a year), or whether it had anything to do with the original Awstats problem.
In any case, updating PHP 5, even though it was not being used by the problem site, seemed to fix the problem. Time will tell, of course.