I was checking apache access log and i noticed a lot of bots. I decided to add robots.txt and block all bots except google and bing. Turned out there are so called “bad bots” who ignore rules from robots.txt so i had to block via htaccess.
And you know what? today i have only one error.
I still have to find out how to block “PetalBot” as it somehow manages to get around block from htaccess.