Easy mass block junk crawlers,open proxies,bots,harvesters,tor visitors,etc

General Discussion
1

Is there a way to block all junk requests to an entire server with multiple domains (not per domain basis)?
something very similar to http://www.blockscript.com/ solution (they are asking for ridiculous money, but they keep daily update to the database just like any antivirus company)

And what is the best way - at what level? (firewall would probably be best, no?)

I am talking about:
-Hosting provider IPs (most of them should not be visiting your website and many junk-crawlers,harvesters, DNS records robot-searchers are based on there):
-Bad bot IPs (webboar, botninja, etc)
-Black Search engine IPs (junk search engines’ robots not related to bluechip search engines like Google, Yahoo etc, similar to Bad Bots)
-Active open proxies (possibly Socks 4/5 as well)
-PlanetLab CoDeen proxies
-known VPN addresses
-Tor exit nodes (those visitors usually have something to hide)
Stuff like that.
Or some way to have all that with constant update and automate with some sort of a script?
System is Centos 6xx 64bit

2

Howdy,

It does sound like you’re looking for a managed list of IP’s to block.

A firewall is certainly a good place to do that – but there’s also tools that I imagine you could use, should you find a list of IP’s to block.

Apache itself could be told what IP’s to block globally, I believe. You could probably also do such a thing with mod_security.

The key would just be finding such a list. Hopefully some folks here have experience doing that who could offer some input on all that!

-Eric