Even with suexec disabled, i still get the forbidden errors…
Do i need to change my permissions for suexec? They are currently: -rw-r–r-- 1
I think I have it working now, lets see…
Ok its "working now". PHP executes, etc…
But I’m trying to do the same ab command I did successfully with mod_php:
ab -n 10000 -c 100 http://www.example.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.example.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
apr_poll: The timeout specified has expired (70007)
Total of 7422 requests completed
And it times out… ?
Then I get a bunch of emails from LFD
Onderwerp: lfd on server.example.com: Excessive processes running under user example
Time: Â Â Â Â Â Â Â Â Â Wed Dec 17 05:08:48 2008 -0600
Account: Â Â Â Â Â Â example
Process Count: 61 (Not killed)
Process Information:
User:example PID:4515 Run Time:51(secs) Memory:181096(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4516 Run Time:51(secs) Memory:181096(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4517 Run Time:51(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4518 Run Time:51(secs) Memory:181092(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4523 Run Time:50(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4524 Run Time:50(secs) Memory:181096(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4525 Run Time:50(secs) Memory:181092(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4526 Run Time:50(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4527 Run Time:50(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4528 Run Time:50(secs) Memory:181088(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4540 Run Time:50(secs) Memory:181092(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
(and about 50 more such lines)
Also a ton of these emais:
Onderwerp: lfd on server.example.com: Excessive resource usage: example (8269)
Time: Wed Dec 17 05:09:58 2008 -0600
Account: example
Resource: Virtual Memory Size
Exceeded: 176 > 100 (MB)
Executable: /usr/bin/php-cgi
Command Line: /usr/bin/php-cgi
PID: 8269
Killed: No
So what’s going on here? Any ideas? I knew fastcgi would use more memory… but this?
Also, suexec is set up correctly, yet it still allows a file that I created as root "test.php"?
If I remove the PHP_FCGI_CHILDREN directive I no longer get these timeouts, but there is still something going on with this:
[root@server ~]# ab -n 10000 -c 100 http://www.example.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.example.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
Server Software: Apache
Server Hostname: www.example.com
Server Port: 80
Document Path: /
Document Length: 6724 bytes
Concurrency Level: 100
Time taken for tests: 51.277226 seconds
Complete requests: 10000
Failed requests: 20
(Connect: 0, Length: 20, Exceptions: 0)
Write errors: 0
Non-2xx responses: 20
Total transferred: 69465480 bytes
HTML transferred: 67116200 bytes
Requests per second: 195.02 [#/sec] (mean)
Time per request: 512.772 [ms] (mean)
Time per request: 5.128 [ms] (mean, across all concurrent requests)
Transfer rate: 1322.95 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 7
Processing: 0 489 4464.7 0 51032
Waiting: 0 489 4464.7 0 51032
Total: 0 489 4464.7 0 51032
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 17
80% 18
90% 32
95% 37
98% 67
99% 13000
100% 51032 (longest request)
See 99 and 100%, why are those taking so long in comparison to the rest?
I spoke too soon, i still get timeouts every few times. Perhaps because I added MaxRequestsPerProcess 500 to my fcgid.conf … (recommended for php)
Is it normal that this stuff is so high and so much during an ab test:
Real memory: 3.86 GB total / 1.70 GB free Swap space: 8 GB total / 7.98 GB free
18024 apache 866880 kB /usr/sbin/httpd.worker
17872 apache 801488 kB /usr/sbin/httpd.worker
17838 apache 801340 kB /usr/sbin/httpd.worker
2669 mysql 419436 kB /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-f …
18166 apache 409768 kB /usr/sbin/httpd.worker
18164 apache 409756 kB /usr/sbin/httpd.worker
3141 root 240748 kB /usr/bin/python -tt /usr/sbin/yum-updatesd
7528 example 223900 kB /usr/bin/php-cgi
7520 example 223068 kB /usr/bin/php-cgi
8203 example 215388 kB /usr/bin/php-cgi
8653 example 215388 kB /usr/bin/php-cgi
9550 example 215388 kB /usr/bin/php-cgi
10357 example 215388 kB /usr/bin/php-cgi
10446 example 215388 kB /usr/bin/php-cgi
10895 example 215388 kB /usr/bin/php-cgi
11078 example 215388 kB /usr/bin/php-cgi
11440 example 215388 kB /usr/bin/php-cgi
11702 example 215388 kB /usr/bin/php-cgi
11956 example 215388 kB /usr/bin/php-cgi
12743 example 215388 kB /usr/bin/php-cgi
13009 example 215388 kB /usr/bin/php-cgi
13532 example 215388 kB /usr/bin/php-cgi
13928 example 215388 kB /usr/bin/php-cgi
14436 example 215388 kB /usr/bin/php-cgi
8296 example 215384 kB /usr/bin/php-cgi
8566 example 215384 kB /usr/bin/php-cgi
8739 example 215384 kB /usr/bin/php-cgi
9730 example 215384 kB /usr/bin/php-cgi
10990 example 215384 kB /usr/bin/php-cgi
11746 example 215384 kB /usr/bin/php-cgi
11831 example 215384 kB /usr/bin/php-cgi
12219 example 215384 kB /usr/bin/php-cgi
13100 example 215384 kB /usr/bin/php-cgi
13187 example 215384 kB /usr/bin/php-cgi
13449 example 215384 kB /usr/bin/php-cgi
14095 example 215384 kB /usr/bin/php-cgi
15038 example 215384 kB /usr/bin/php-cgi
15380 example 215384 kB /usr/bin/php-cgi
7684 example 215380 kB /usr/bin/php-cgi
7686 example 215380 kB /usr/bin/php-cgi
7795 example 215380 kB /usr/bin/php-cgi
9188 example 215380 kB /usr/bin/php-cgi
9818 example 215380 kB /usr/bin/php-cgi
9993 example 215380 kB /usr/bin/php-cgi
10626 example 215380 kB /usr/bin/php-cgi
10709 example 215380 kB /usr/bin/php-cgi
11350 example 215380 kB /usr/bin/php-cgi
12306 example 215380 kB /usr/bin/php-cgi
12656 example 215380 kB /usr/bin/php-cgi
13204 example 215380 kB /usr/bin/php-cgi
13838 example 215380 kB /usr/bin/php-cgi
14011 example 215380 kB /usr/bin/php-cgi
14525 example 215380 kB /usr/bin/php-cgi
14859 example 215380 kB /usr/bin/php-cgi
15036 example 215380 kB /usr/bin/php-cgi
15296 example 215380 kB /usr/bin/php-cgi
12044 example 215376 kB /usr/bin/php-cgi
13360 example 215376 kB /usr/bin/php-cgi
7687 example 215264 kB /usr/bin/php-cgi
7690 example 215264 kB /usr/bin/php-cgi
8162 example 215264 kB /usr/bin/php-cgi
9006 example 215264 kB /usr/bin/php-cgi
9371 example 215264 kB /usr/bin/php-cgi
9462 example 215264 kB /usr/bin/php-cgi
10084 example 215264 kB /usr/bin/php-cgi
11612 example 215264 kB /usr/bin/php-cgi
12389 example 215264 kB /usr/bin/php-cgi
12831 example 215264 kB /usr/bin/php-cgi
12920 example 215264 kB /usr/bin/php-cgi
13667 example 215264 kB /usr/bin/php-cgi
14246 example 215264 kB /usr/bin/php-cgi
7691 example 215260 kB /usr/bin/php-cgi
7692 example 215260 kB /usr/bin/php-cgi
7694 example 215260 kB /usr/bin/php-cgi
8833 example 215260 kB /usr/bin/php-cgi
10267 example 215260 kB /usr/bin/php-cgi
11265 example 215260 kB /usr/bin/php-cgi
14260 example 215260 kB /usr/bin/php-cgi
14703 example 215260 kB /usr/bin/php-cgi
15129 example 215260 kB /usr/bin/php-cgi
7685 example 215256 kB /usr/bin/php-cgi
7693 example 215256 kB /usr/bin/php-cgi
7981 example 215256 kB /usr/bin/php-cgi
8070 example 215256 kB /usr/bin/php-cgi
8474 example 215256 kB /usr/bin/php-cgi
8920 example 215256 kB /usr/bin/php-cgi
9098 example 215256 kB /usr/bin/php-cgi
9277 example 215256 kB /usr/bin/php-cgi
10171 example 215256 kB /usr/bin/php-cgi
12484 example 215256 kB /usr/bin/php-cgi
12514 example 215256 kB /usr/bin/php-cgi
13755 example 215256 kB /usr/bin/php-cgi
14349 example 215256 kB /usr/bin/php-cgi
14611 example 215256 kB /usr/bin/php-cgi
14946 example 215256 kB /usr/bin/php-cgi
7688 example 215252 kB /usr/bin/php-cgi
7888 example 215252 kB /usr/bin/php-cgi
8386 example 215252 kB /usr/bin/php-cgi
9640 example 215252 kB /usr/bin/php-cgi
9907 example 215252 kB /usr/bin/php-cgi
10532 example 215252 kB /usr/bin/php-cgi
10806 example 215252 kB /usr/bin/php-cgi
11173 example 215252 kB /usr/bin/php-cgi
11526 example 215252 kB /usr/bin/php-cgi
12134 example 215252 kB /usr/bin/php-cgi
13547 example 215252 kB /usr/bin/php-cgi
15467 example 215252 kB /usr/bin/php-cgi
6842 apache 128268 kB /usr/sbin/httpd.worker
5778 root 128232 kB /usr/sbin/httpd.worker <-- is this normal ?
after checking the error log and using ab, these are the errors i found:
in /home/example/logs/error_log
[Wed Dec 17 10:29:57 2008] [warn] (104)Connection reset by peer: mod_fcgid: read data from fastcgi server error.
[Wed Dec 17 10:29:57 2008] [error] [client 69.65.x.x] Premature end of script headers: index.php
(multiple times)
in /var/log/httpd/error_log
[Wed Dec 17 10:30:01 2008] [notice] mod_fcgid: call /home/example/public_html/index.php with wrapper /home/example/fcgi-bin/php5.fcgi
[Wed Dec 17 10:30:10 2008] [notice] mod_fcgid: process /home/example/public_html/index.php(7693) exit(normal exit), terminated by calling exit(), return code: 0
(multiple times)
[Wed Dec 17 10:30:10 2008] [warn] mod_fcgid: cleanup zombie process 7693
I dunno. 
I’m, by no means, an expert on PHP and performance. I do know that response time under mod_fcgid is similar to mod_php, both in benchmarks in the wild, and my own testing. I kind of expect resource usage to be higher under extremely high load (as your testing indicates). There may be more useful tuning to be done, though, since I have seen benchmarks showing mod_fcgid performing better than mod_php.
A few things I can mentionerd that may be useful:
Memory usage in this circumstance is far lower than you think. Shared memory accounts for a large percentage of all used memory–these process are all nearly identical, and only the data space is "new" between each one, so probably 32k or 64k (depending on your memory limit setting in php.ini) and some extra overhead, is all that sets each of these processes apart.
5778 root 128232 kB /usr/sbin/httpd.worker <-- is this normal ?
Is what normal?
I don’t see anything abnormal about it, so I guess it’s normal.
Are you asking about the root owned process? Yes. That’s the parent, and it spawns children owned by apache (which then uses suexec to spawn children). Apache starts as root so it can bind to port 80.
One thing I will point out is that you seem to have mod_php loaded while testing mod_fcgid, which is giving mod_php a huge advantage. mod_php is part of your apache process, no matter what, if you’re loading it…so it’s ballooning your Apache up by a large amount, whether you use it or not. Our Apache here at Virtualmin.com, for example, is only 22.5MB, while yours is 128MB. We don’t run mod_php (all of our sites run under mod_fcgid and suexec…we eat our own dogfood). Even that could be made smaller. I believe a completely minimized Apache is about 8MB (though I’m not necessarily recommending going crazy with that–but if performance and resource usage is critical, it might be necessary to consider what you can take out).
Oh, I just noticed you’re using the worker MPM. This is bad news with PHP. Switch to prefork, per the PHP docs:
http://www.php.net/manual/en/install.unix.apache2.php
This may also contribute to the weirdness you’re seeing with failed requests and/or zombies. I dunno. I have no idea what the impact of using a threaded Apache with PHP is. I just know it’s a no no, according to the PHP people.
Actually, scratch that last one. Running in a FastCGI configuration means a threaded model for the parent Apache works fine.
Well, I don’t have mod_php loaded, infact I already trimmed the fat alot. Here’s the modules that are loaded:
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule mime_module modules/mod_mime.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule dir_module modules/mod_dir.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule version_module modules/mod_version.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule logio_module modules/mod_logio.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule actions_module modules/mod_actions.so
LoadModule env_module modules/mod_env.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule fcgid_module modules/mod_fcgid.so