Getting rid of spiders that keep coming back

Saturnin · 2008-04-24, 01:43 AM

I've banned (via robots.txt and httpaccess) the BaiDu spider from my sites, but it still keeps showing up. I don't want traffic from China (not in my audience demograhic), so is there a good way to ban this thing, or get myself banned by the Great Firewall of Aforementioned Place?

(Maybe put "Facts on Tibet" or something in the metatag? |goodidea)

RicRock67 · 2008-04-24, 09:55 AM

My guess is if it's ignoring your robots.txt it's spoofing the UserAgent.

I'd try to narrow down the ip block and stop it via a firewall

Way3 · 2008-04-25, 01:04 PM

Firewall will be the most effective, but .htaccess would work as well. Try blockacountry.com
Best of Luck!

HappySpanker · 2008-05-02, 04:25 PM

Odds are if you are able to identify the BaiDu spider in your logs, maybe an htaccess rewrite rule would deal with it. Blocking by IP is easy but very fallible, using rewrite rules is more fun. Like feed them a 301 to a creative place...

2008-04-24, 01:43 AM	#1
Saturnin A woman is like beer. They look good, they smell good, and you'd step over your own mother just to get one! Join Date: Jul 2006 Location: Edge of the Universe Posts: 53	Getting rid of spiders that keep coming back I've banned (via robots.txt and httpaccess) the BaiDu spider from my sites, but it still keeps showing up. I don't want traffic from China (not in my audience demograhic), so is there a good way to ban this thing, or get myself banned by the Great Firewall of Aforementioned Place? (Maybe put "Facts on Tibet" or something in the metatag? \|goodidea)

2008-04-24, 09:55 AM	#2
RicRock67 Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless Join Date: Apr 2008 Posts: 21	My guess is if it's ignoring your robots.txt it's spoofing the UserAgent. I'd try to narrow down the ip block and stop it via a firewall __________________ TrafficHolder.com - Buy/Sell Adult Traffic \| Link Trades - TBN

2008-04-25, 01:04 PM	#3
Way3 Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless Join Date: Jan 2007 Posts: 22	Firewall will be the most effective, but .htaccess would work as well. Try blockacountry.com Best of Luck! __________________ ICQ: 169-554-261 info[at]way3[dot]com

2008-05-02, 04:25 PM	#4
HappySpanker WHO IS FONZY!?! Don't they teach you anything at school? Join Date: May 2008 Posts: 47	Odds are if you are able to identify the BaiDu spider in your logs, maybe an htaccess rewrite rule would deal with it. Blocking by IP is easy but very fallible, using rewrite rules is more fun. Like feed them a 301 to a creative place...