Reboot server. 403 error. Manually restart apache2 and it starts to work fine.

Chris_C · November 28, 2012, 6:41pm

I’m running Virtualmin - latest version 3.96.GPL

2 virtual servers. Only use 1 for now.
Email and web are setup correctly - as far as I can see.
csf firewall is running and gives 36/37 score for security.

Here’s the problem.

When the server reboots, apache2 gives a 403 Forbidden error for any access to the web site.

I have to manually restart apache2 - with service apache2 restart - then the site works.

I want apache2 to work from the get-go - no 403 error - without having to manually restart it.

Help !

Eric · November 28, 2012, 8:28pm

Howdy,

That’s really unusual… I’m not sure what kind of problem you might be experiencing upon a reboot that causes Apache to not work initially, but it would work after restarting the service.

Are you by chance using a VPS? If so, what kind of VPS is it?

Also, which distro are you using? And do you see any errors in the system-wide Apache error logs?

-Eric

Chris_C · November 28, 2012, 8:48pm

Yes- it’s a VPS- SolusVM, OpenVZ container.

As for distro- my container is running Debian 6 (squeeze) x64.

System wide apache error log :
tail /var/log/apache2/error.log
Could not open input file: /home/mysitenamewhichis18characterslong/public_html/components/com_hwdvideoshare/xml/autogenerate.php
(repeated 10 times)
I checked that folder, and that file doesn’t exist. So it shouldn’t be a problem. I think it’s an optional file.

I looked around a bit in Virtualmin and noticed something weird:
First, my virtual server name is a bit long - it’s 18 characters long in fact.
The weird thing is, the “user” and “group” are missing from Webmin’s list of “users and groups”- must be a bug where Webmin chokes on the string being too long?
My OTHER virtual server name is 17 characters long, and THAT one has entries showing up in “users and groups”.
BUT, when I go to add the user and group for my 18-character virtual server inside Virtualmin, it says it already exists.
I verify by looking at the files in the public_html for this 18-character virtual server name, and they seem to all have their user and group ownership set to that 18-character name.
Maybe this is nothing or maybe it’s a clue to what’s happening.

Eric · November 28, 2012, 9:17pm

Hmm, could you paste in the contents of your /proc/user_beancounters file?

-Eric

Chris_C · November 28, 2012, 10:49pm

It’s a wide output… hope it gets formatted OK…

~# cat /proc/user_beancounters
Version: 2.5
uid resource 944: kmemsize lockedpages 0 privvmpages 424442 shmpages 1722 dummy 0 numproc 159 physpages 196435 vmguarpages 0 oomguarpages 196435 numtcpsock 96 numflock 65 numpty 1 numsiginfo 1 tcpsndbuf 1745408 tcprcvbuf 1767904 othersockbuf 326720 dgramrcvbuf 0 numothersock 219 dcachesize 1058253 numfile 4805 dummy 0 dummy 0 dummy 0 numiptent 151 ~#
held maxheld barrier limit failcnt
34365284 131456233 2147483646 2147483646 0
968 999999 999999 0
797854 786432 786432 98883
15185 524288 524288 0
0 0 0 0
323 999999 999999 0
264236 0 2147483647 0
0 524288 2147483647 0
264260 524288 2147483647 0
520 7999992 7999992 0
73 999999 999999 0
3 500000 500000 0
40 999999 999999 0
10674816 214748160 396774400 0
9393120 214748160 396774400 0
2024000 214748160 396774400 0
30176 214748160 396774400 0
466 7999992 7999992 0
1883634 2147483646 2147483646 0
9916 23999976 23999976 0
0 0 0 0
0 0 0 0
0 0 0 0
152 999999 999999 0

Eric · November 29, 2012, 12:34am

Howdy,

Every once in awhile, we run into a really strange, bizarre, issue… one that we think should never ever be able to happen.

And we troubleshoot for a bit, discovering that the issue is occurring on an OpenVZ-based VPS, with the user_beancounters showing RAM or network resource failures.

That may be what’s going on here as well, unfortunately

Your user_beancounters file is showing that 98,883 requests for RAM were made, but denied… and that sort of issue can cause some really strange behavior.

To solve that, you could ask your provider for more resources.

Or, you could try disabling some services on your system, so that less are being launched at boot time… it’s possible that would help.

You may also want to disable any Apache modules that you don’t need… that could help too. There’s some thoughts here on running Virtualmin on low-memory systems:

http://www.virtualmin.com/documentation/system/low-memory

The only thing that has me scratching my head is that usually these issues occur after some time of the server being online, it’s rare that they occur only during bootup.

So, just to make sure we’re not overlooking another kind of issue… what is that domain’s PHP Execution Mode set to? You can determine that in Server Configuration -> Website Options. If it’s set to FCGID, you may want to give CGI a try. Or if it’s CGI, try FCGID, and see if that makes a difference.

-Eric

Chris_C · November 29, 2012, 3:54am

I really appreciate the help.

I set it to CGI, saved, and rebooted the server.
Still the same bug - apache comes up with 403 Forbidden-
I even let it settle for 1 minute- still the 403 Forbidden error-
Until I manually do service apache2 restart- then apache starts to work immediately.

Could the user_beancounters privvmpages memory issue be related to php.ini memory_limit being set to 256M ?
I have 8 apache (prefork?) processes running as www-data + 1 running as root. I’m 99% sure that’s the default number of processes for apache prefork in virtualmin.
And my VPS is 1.5 GB + 1.5 GB Swap, total says 3 GB.
The fail counter is the same now as it was earlier today, so no new out of memory conditions.
But if 20 people surf the site at the same time it might run into memory issues and bump the fail counter?
I could reduce the 256M to 128M for a day, and see if that helps.
There’s no timestamped log from when those beancounters were failcnt’ed so it might’ve been all from a month ago when php.ini memory_limit was set in the 64M range.

Still stumped…!

Chris_C · November 29, 2012, 4:17am

Hold up, looks like I made some headway…


~# tail /var/log/virtualmin/mydomain.com_error_log

[Wed Nov 28 22:50:18 2012] [warn] RSA server certificate wildcard CommonName (CN) `*.mydomain.com’ does NOT match server name!?

[Wed Nov 28 22:50:20 2012] [crit] [client 72.14.199.71] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

[Wed Nov 28 22:50:36 2012] [crit] [client 173.44.37.242] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

~# ls -la /home/mydomain/.htaccess

ls: cannot access /home/mydomain/.htaccess: No such file or directory

~# touch /home/mydomain/.htaccess

~# ls -la /home/mydomain/.htaccess

-rw-r–r-- 1 root root 0 Nov 28 22:53 /home/mydomain/.htaccess

~# tail /var/log/virtualmin/mydomain.com_error_log

[Wed Nov 28 22:52:01 2012] [crit] [client 76.248.147.117] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

~# ls -la /home/mydomain/.htaccess

-rw-r–r-- 1 root root 0 Nov 28 22:53 /home/mydomain/.htaccess

~# chown mydomain:mydomain /home/mydomain/.htaccess
~# chown www-data:www-data /home/mydomain/.htaccess
~# tail /var/log/virtualmin/mydomain.com_error_log

[Wed Nov 28 22:54:48 2012] [crit] [client 180.76.5.90] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

~# service apache2 restart

Restarting web server: apache2 … waiting .

~# tail /var/log/virtualmin/mydomain.com_error_log

[Wed Nov 28 22:55:23 2012] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)

[Wed Nov 28 22:55:23 2012] [warn] RSA server certificate wildcard CommonName (CN) `*.mydomain.com’ does NOT match server name!?

~# ls -la /home/mydomain/.htaccess

-rw-r–r-- 1 www-data www-data 0 Nov 28 22:53 /home/mydomain/.htaccess

~#

First time I saw errors on a non-existent /home/mydomain/.htaccess

Although, putting the file there, and changing its ownership to the most likely user and group still didn’t fix the 403 error. Still had to restart apache2 manually for the site to work…

Eric · November 29, 2012, 2:29pm

Ah, you’re getting permission denied error when trying to read that .htaccess file.

If you go into System Settings and run the Re-Check Config, does it notice anything unusual?

Also, just to be sure there isn’t some sort of permission issue, you may want to go into Limits and Validation -> Validate Virtual Servers -> Fix Directory Permissions, you may want to try resetting the permissions for that particular domain.

However, it’s odd that any permission problem would be resolved by restarting Apache…

-Eric

Chris_C · November 29, 2012, 3:32pm

I do the System Settings, Re-Check Config. Nothing unusual- it says all is OK.

I do the Limits and Validation, Validate Virtual Servers, Fix Directory Permissions, reset permissions for that domain. It resets the permissions.

Reboot server. 403 error accessing any page. Tail the domain’s log. “Permission denied /home/mydomain/.htaccess pcfg_openfile unable to check htaccess file, ensure it is readable.”

service apache2 restart and the site comes right up.

The weird thing is… /home/mydomain isn’t even supposed to be accessible from the internet.
/home/mydomain should neither have nor need an .htaccess file in it.
The TOP folder for web access in my folder heirarchy should be /home/mydomain/public_html
Could THAT be the reason why apache is choking?
apache is told, through some configuration mistake, to serve the site from /home/mydomain but in fact apache’s blocked for security reasons since /home/mydomain is outside of the allowed correct web-accessible home folder of /home/mydomain/public_html ???

Eric · November 29, 2012, 4:12pm

Howdy,

Well, although you don’t serve content from there, you can still have a .htaccess file in /home/mydomain. Apache is checking there to see if you have one… but due to some sort of permission issue, it’s not able to read there (which it should have permission to see).

Until, very oddly, after you restart Apache. At which point it works fine.

You aren’t by chance using something like LDAP for users, are you?

-Eric

Chris_C · November 30, 2012, 4:13am

Excellent thinking. I’d attempted to get OpenLDAP for email users working- the idea was to have Zarafa store user info in LDAP instead of the operating system unix user accounts… but it never quite got working.

It’s probably messing up somewhere with the LDAP- maybe failing to authenticate, or invalid username-

EDIT:

Any suggestion on how to determine whether apache is using the “ldap users and groups” vs. standard unix users? And how to make apache successfully query the ownership/permissions of /home/mydomain/.htaccess , without breaking the current dovecot imap email login authentication (which is using standard unix users - I think ?!)?
FYI - I checked, and /home/mydomain has permissions 750 and ownership mydomain:mydomain - is this correct? Is mydomain:nobody more appropriate? The same/less/more secure?

Chris_C · November 30, 2012, 4:17am

I found this link - do you think it could help ?
http://linuxwindowsmaster.com/htaccess-pcfg_openfile-unable-to-check-htaccess-file-ensure-it-is-readable/

Eric · November 30, 2012, 2:48pm

Howdy,

LDAP is unfortunately not really my area of expertise… but what you could do is try stopping the LDAP service from running, and then run the command:

id USERNAME

And make sure that it returns user and group information for your username.

Then, try this command:

groups www-data

That groups shown there would need to include your username… that is, the www-data user needs to be added to the group of your user (and all the other Virtual Server owners on the system).

If that doesn’t work with LDAP stopped, that may mean some user data is being stored in LDAP.

The link you mentioned is indeed correct in that you’re seeing a permission problem… however, you shouldn’t be seeing one, and you even did the “Fix Directory Permissions”. So I suspect the issue is something, well, non-standard

The LDAP problem is definitely an intriguing possibility!

-Eric

Chris_C · November 30, 2012, 9:00pm

Eric, I believe you’ve found it.


~# id mydomain
uid=1002(mydomain) gid=1002(mydomain) groups=1002(mydomain)
~# service slapd stop

Stopping OpenLDAP: slapd.

~# id mydomain

id: mydomain: No such user
~# service slapd start

Starting OpenLDAP: slapd.

~# groups www-data

www-data : www-data list otherdomain mydomain
~# service slapd stop

Stopping OpenLDAP: slapd.

~# groups www-data

www-data : www-data list otherdomain
~# service slapd start

Starting OpenLDAP: slapd.

~#

The best guess I can make, at why it got into this state (mismatch between unix users and OpenLDAP users) is because I first installed virtualmin, and added the virtual server “otherdomain”.
Then only later I activated the LDAP Users and Groups virtualmin/webmin module , and set it to “authenticate against ldap not against local unix users” (or however it’s worded exactly), and added the virtual server “mydomain” (which is the one I’m having the trouble with - apache giving this 403 error until I restart apache2 service.)

My intention was to have apache2, dovecot, zarafa, everything, authenticate off openldap - and I assumed that’s what the LDAP Users and Groups module’s purpose is.

Could it possibly be a race condition between apache2 and OpenLDAP ?
Suppose apache2 completes starting up BEFORE OpenLDAP is fully up, and so apache2 thinks it should just use unix users, because OpenLDAP isn’t alive (yet) ??
Maybe apache2 sees a LOCAL UNIX group for “www-data” and thinks it’s OK to just go ahead and only look at the unix users/groups…? Except mydomain is NOT a member of www-data on the local unix groups. Neither does it exist as a user on local unix users.
And then on the apache2 restart, apache2 obviously now sees OpenLDAP up and running, and just authenticates off that, and everything works because in OpenLDAP, mydomain exists as a user, and is a member of www-data.

What do you think ?

Chris_C · December 5, 2012, 3:37am

Bump.

Just wondering whether you had a chance to take a look at this.

Is my hypothesis that it’s caused by apache2 starting before openldap, correct ? In your opinion ?

How to make it so that openldap starts first, then apache2 after ?