[web-devel] Keep too busy crawlers away from your server

I know there are more sophisticated tools for that task, but I wrote a pretty simple program that works for me reliably for several months now. See the attached program. It scans the last 100 KB of a log-file you specify as command-line argument and checks whether certain clients excess allowedAccessesPerSecond. If so, their IPs are written to a blocked-hosts file and are removed from there only 5 days later. The blocked-hosts file in my example is used by arno-iptables-firewall to actually block the IP addresses. I added this program to my crontab: # m h dom mon dow command * * * * * process-log /var/log/http-access.log >>/var/log/http-block.log 2>>/var/log/http-block.err