| SYSTEM INFORMATION | |
|---|---|
| OS type and version | Redhat 9.4 |
| Virtualmin version | 7.20.2 |
Hi guys/girls,
So I was getting some very high amount of requests to Apache, which lead it to always reach its MaxWorkers settings, which in turn kept crashing it and bringing all the websites down. I couldn’t figure out which domain was the culprit (well I found a few - judging by the sizes of the access logs for them and last modified time), but the IPs were just too many to go through the logs.
I tried implementing a few rules with iptables in hopes to block the heavy traffic:
iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 3/s -j ACCEPT
iptables -I INPUT -p tcp --dport 443 -m state --state NEW -m limit --limit 3/s -j ACCEPT
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 20 -j DROP
iptables -A INPUT -p tcp -m tcp --dport 80 -m length --length 0 -j DROP
iptables -A INPUT -p tcp -m tcp --dport 443 -m length --length 0 -j DROP
but Apache still kept crashing, so I decided on ultimate measure to block all datacenter IPs.
However, this leads to new problem, which I am unable to track down.
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:12::d0d#53
Aug 25 04:11:57 host named[449335]: running
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:12::d0d#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:a8::e#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:a8::e#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fd::1#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:7fd::1#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:1::53#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:1::53#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2801:1b8:10::b#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2801:1b8:10::b#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:dc3::35#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:dc3::35#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:9f::42#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:9f::42#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fe::53#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:7fe::53#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2::c#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:2::c#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:503:ba3e::2:30#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:503:ba3e::2:30#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2f::f#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:2f::f#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:503:c27::2:30#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:503:c27::2:30#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2d::d#53
Aug 25 04:11:57 host named[449335]: network unreachable resolving './NS/IN': 2001:500:2d::d#53
Aug 25 04:11:57 host named[449335]: managed-keys-zone: Key 20326 for zone . is now trusted (acceptance timer complete)
Aug 25 04:11:57 host named[449335]: resolver priming query complete
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2::c#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2801:1b8:10::b#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:9f::42#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:12::d0d#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:1::53#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:503:c27::2:30#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2d::d#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:a8::e#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:503:ba3e::2:30#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2f::f#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fe::53#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:dc3::35#53
Aug 26 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fd::1#53
Aug 26 04:08:17 host named[449335]: managed-keys-zone: Key 20326 for zone . is now trusted (acceptance timer complete)
Aug 26 04:11:57 host named[449335]: _default: sending trust-anchor-telemetry query '_ta-4f66/NULL'
Aug 26 04:11:57 host named[449335]: network unreachable resolving '_ta-4f66/NULL/IN': 2001:500:1::53#53
Aug 26 04:11:57 host named[449335]: network unreachable resolving '_ta-4f66/NULL/IN': 2001:500:9f::42#53
Aug 26 04:11:57 host named[449335]: network unreachable resolving '_ta-4f66/NULL/IN': 2001:500:2d::d#53
Aug 26 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:9f::42#53
Aug 26 04:11:57 host named[449335]: managed-keys-zone: Key 20326 for zone . is now trusted (acceptance timer complete)
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2f::f#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2801:1b8:10::b#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:1::53#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fe::53#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fd::1#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:12::d0d#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2::c#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:503:ba3e::2:30#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:503:c27::2:30#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:9f::42#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:dc3::35#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:a8::e#53
Aug 27 04:08:17 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:500:2d::d#53
Aug 27 04:08:17 host named[449335]: managed-keys-zone: Key 20326 for zone . is now trusted (acceptance timer complete)
Aug 27 04:11:57 host named[449335]: _default: sending trust-anchor-telemetry query '_ta-4f66/NULL'
Aug 27 04:11:57 host named[449335]: network unreachable resolving '_ta-4f66/NULL/IN': 2001:7fe::53#53
Aug 27 04:11:57 host named[449335]: network unreachable resolving '_ta-4f66/NULL/IN': 2001:7fd::1#53
Aug 27 04:11:57 host named[449335]: network unreachable resolving './DNSKEY/IN': 2001:7fe::53#53
Aug 27 04:11:57 host named[449335]: managed-keys-zone: Key 20326 for zone . is now trusted (acceptance timer complete)
Aug 27 04:15:17 host systemd[1]: systemd-hostnamed.service: Deactivated successfully.
The above is a log from: “cat /var/log/messages | grep named” .
BIND is unable to connect to some IPv6 servers and is throwing errors. HOWEVER, the server itself doesn’t have IPv6 and if I remove the datacenter blocks, it will not print these messages.
The other issue connected to this is that I am unable to access any of the domains hosted on the server with my home network - if I connect through VPN or mobile internet however - it works just fine. Also from this network, I can connect to the server to the IP. I checked my IP range, it isn’t in the blocklist. 89.106.XX.XX
GitHub - jhassine/server-ip-addresses: Daily updated list of IP addresses / CIDR blocks used by data centers, cloud service providers, servers, etc. - this is the blocklist I am using.
I have all the CIDRs, blocked with ip route add blackhole subnet.
A dummy site on the server is: https://camerite.eu . IntoDNS.com also reports no faults and nslookup +trace fails on my network. I can’t pinpoint the reason why this could be failing on my network and working on the mobile network.
I am using as DNS 8.8.8.8 and 8.8.4.4 . All other internet works fine.