Greenguy's Board


Go Back   Greenguy's Board > Newbie Questions
Register FAQ Calendar Today's Posts

 
 
Thread Tools Search this Thread Rate Thread Display Modes
Prev Previous Post   Next Post Next
Old 2008-01-07, 04:22 AM   #5
bluebrit
Along for the ride and loving it.
 
Join Date: Aug 2005
Location: Canada
Posts: 1,873
Quote:
Originally Posted by NY Jester View Post
Is there a robot txt to keep certain bots out? And does anyone have a list of "bad" bots?
I used to have quite a list of bots on my robots.txt but most of them ignored it making the file a waste of time.

Now i'm working on a bot trap but it's far from ready. When done it will trace access to the domain and if a bot hits blocked areas multiple times it will automatically add it to a ban list in htaccess. The beauty of this is that even if the bot changes names it will still get banned automatically due to its hits. The down side is that if the list grows too quickly the htaccess could slow your site down so it will need to be monitored.

As for a list of bots, there are a few good examples available via a quick search but always be sure to check them first to make sure you actually want all the bots included to be banned.
bluebrit is offline   Reply With Quote
 


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 11:00 AM.


Mark Read
Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
© Greenguy Marketing Inc