Apache Error: (24)Too many open files: AH02179: apr_socket_accept

Virtualmin
1

Hi,

Error as title. Here’s a few facts:
Running:
CentOS 8
Apache/2.4.37
php-fpm 5.6, 7.3 and 7.4
Box has 32gb ram and Xeon E5
Virtualmin servers: 694 - all (very) low traffic.
Resource useage is low.

I’ve just run up about 400 sub domains under a vhost.
Apache refuses to start with little info at cli, logs show a lot of this error:
(24)Too many open files: AH02179: apr_socket_accept: (client socket)

I’m thinking its a problem with open file limits - either with apache or php-fpm.
Or mpm MaxClients/ServerLimit

For the moment I’ve just moved the vhost conf files out of conf.d and things are ok.

I’d be grateful for any input as to how to resolve, before I start making (too many random!) changes to configs.

I note from apachectl -M that mpm_events is loaded (whereas I thought virtualmin usually loaded mpm_prefork?)
apache loaded modules are shown below.

Regards

l.

[root@]# apachectl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 brotli_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 request_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 lua_module (shared)
 mpm_event_module (shared)
 proxy_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fcgi_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_hcheck_module (shared)
 proxy_scgi_module (shared)
 proxy_uwsgi_module (shared)
 proxy_wstunnel_module (shared)
 ssl_module (shared)
 systemd_module (shared)
 cgid_module (shared)
 fcgid_module (shared)
 http2_module (shared)
 security2_module (shared)
 proxy_http2_module (shared)
2

Wow - 700 virtual servers on a single 32GB Virtualmin system: good to know that it can be done.

The following works on a high-load CentOS 7 system and should work on your CentOS 8 system too but before you apply these settings please understand that doing so can (and will, on very high loads) make the system unstable.

  1. Edit /etc/sysctl.conf and append

    net.core.somaxconn=131072
    fs.file-max=131072

    Save file, then

    sudo sysctl -p

  2. Edit /usr/include/linux/limits.h and set

    NR_OPEN = 65535

  3. Edit /etc/security/limits.conf and append

    * soft nofile 65535
    * hard nofile 65535

That’s it. Let us know if this has increased the open file limit on your system and the error has disappeared.

It changes depending on the OS and version.

3

Hi @calport

thanks very much for your reply
your suggestions had some sucess:

[root@]#cat /proc/sys/fs/file-max
131072

[root@]# lsof | awk '{print $1}' | sort | uniq -c | sort -r -n | head
 220528 httpd
  49521 php-fpm

sadly no, the same errors are still showing
and temp doubling the numbers makes no difference

4

So just to wrap this up…
After a bit more research here’s what I’ve come up with as a workable solution and why.

This applies to a centos8 system where under systemd there are system wide file descriptor limits and per-process file descriptor limits
When apache starts its as the root user, then when its grabbed the ports it switches to run under the httpd user
And this is where my confusion arose and the suggestion by @calport didnt work.
I suspect (but havent confirmed) its because the limit of 1024 max open files that I kept seeing from
cat /proc/httpd_pid/limits
is wired into apache somewhere and is applied when it switches user at startup.
I assume its a in config file rather than hard wired altho I didnt find it yet.

So a solution:
create/edit /usr/lib/systemd/system/httpd.service.d/limits.conf

and add:
[Service]
LimitNOFILE=4096

then
[root]# systemctl daemon-reload
and
[root]# sytemctl restart httpd.service

I’ve undone the suggested changes to system wide limits.

I’ve lowered the httpd limit down from 65535 to 4096 at present and things are still good. I’ll prob halve it again in a day or two, try some load tests and see.

Hopefully setting a reasonable limit per process rather than a big system wide file limit avoids the risking the scenario calport warned of where the system runs out of control under some unexpected condition because system wide file limit is too high.

Regards

l.

1 Like
5

This topic was automatically closed 4 days after the last reply. New replies are no longer allowed.