Quote:
Originally Posted by NY Jester
Is there a robot txt to keep certain bots out? And does anyone have a list of "bad" bots?
|
I used to have quite a list of bots on my robots.txt but most of them ignored it making the file a waste of time.
Now i'm working on a bot trap but it's far from ready. When done it will trace access to the domain and if a bot hits blocked areas multiple times it will automatically add it to a ban list in htaccess. The beauty of this is that even if the bot changes names it will still get banned automatically due to its hits. The down side is that if the list grows too quickly the htaccess could slow your site down so it will need to be monitored.
As for a list of bots, there are a few good examples available via a quick search but always be sure to check them first to make sure you actually want all the bots included to be banned.