|
2013-10-17, 01:39 PM | #1 |
You can now put whatever you want in this space :)
|
Blocking Unknown Robots/Spiders?
So you guys do care on what scans your domains?
if so what techniques do you guys use? I came across this blog post: http://michael.langley.id.au/blog/posts/28 Dunno if it is the better way to do this kind of thing... I'm interested in this because I check the stats on one of mine domains and got a lot of unknown robots...
__________________
|
2013-10-17, 01:48 PM | #2 |
Subversive filth of the hedonistic decadent West
Join Date: Mar 2003
Location: Southeast Florida
Posts: 27,936
|
Why would I care?
|
2013-10-17, 04:16 PM | #3 |
Shift Out / X-On
|
You can set the rules in robots.txt in your root of a domain but most both don't obey those.
|
2013-10-23, 05:29 AM | #4 |
You can now put whatever you want in this space :)
|
You can deny them in .htaccess (or robots.txt but as pc said, only the useful/nice bots are likely to comply) or use something like http://www.spambotsecurity.com/zbblock.php to stop them ...
|
|
|