Blocking Unknown Robots/Spiders?

mOrrI · 2013-10-17, 01:39 PM

So you guys do care on what scans your domains?

if so what techniques do you guys use?

I came across this blog post:

http://michael.langley.id.au/blog/posts/28

Dunno if it is the better way to do this kind of thing...

I'm interested in this because I check the stats on one of mine domains and got a lot of unknown robots...

Cleo · 2013-10-17, 01:48 PM

Why would I care?

pc · 2013-10-17, 04:16 PM

You can set the rules in robots.txt in your root of a domain but most both don't obey those.

hincapie · 2013-10-23, 05:29 AM

You can deny them in .htaccess (or robots.txt but as pc said, only the useful/nice bots are likely to comply) or use something like http://www.spambotsecurity.com/zbblock.php to stop them ...

2013-10-17, 01:39 PM	#1
mOrrI You can now put whatever you want in this space :) Join Date: Feb 2004 Location: Portugal Posts: 1,083	Blocking Unknown Robots/Spiders? So you guys do care on what scans your domains? if so what techniques do you guys use? I came across this blog post: http://michael.langley.id.au/blog/posts/28 Dunno if it is the better way to do this kind of thing... I'm interested in this because I check the stats on one of mine domains and got a lot of unknown robots... __________________

2013-10-17, 01:48 PM	#2
Cleo Subversive filth of the hedonistic decadent West Join Date: Mar 2003 Location: Southeast Florida Posts: 27,936	Why would I care? __________________ Free Rides on Uber and Lyft Uber Car: uberTzTerri Lyft Car: TZ896289

2013-10-23, 05:29 AM	#4
hincapie You can now put whatever you want in this space :) Join Date: Aug 2005 Location: Cyprus Posts: 963	You can deny them in .htaccess (or robots.txt but as pc said, only the useful/nice bots are likely to comply) or use something like http://www.spambotsecurity.com/zbblock.php to stop them ... __________________ Hinc Linklists/TGPs: Adult List Loasex.com

2013-10-17, 04:16 PM	#3
pc Shift Out / X-On Join Date: Jul 2007 Location: unknown Posts: 2,298	You can set the rules in robots.txt in your root of a domain but most both don't obey those.