 |
 |
|
 |
|
|
|
|
|
|||||||
 |
|
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
2008-04-24, 01:43 AM
|
#1 |
|
A woman is like beer. They look good, they smell good, and you'd step over your own mother just to get one!
Join Date: Jul 2006
Location: Edge of the Universe
Posts: 53
|
Getting rid of spiders that keep coming back
I've banned (via robots.txt and httpaccess) the BaiDu spider from my sites, but it still keeps showing up. I don't want traffic from China (not in my audience demograhic), so is there a good way to ban this thing, or get myself banned by the Great Firewall of Aforementioned Place?
(Maybe put "Facts on Tibet" or something in the metatag? |goodidea) |
|
|
|
2008-04-24, 09:55 AM
|
#2 |
|
Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless
Join Date: Apr 2008
Posts: 21
 |
My guess is if it's ignoring your robots.txt it's spoofing the UserAgent.
I'd try to narrow down the ip block and stop it via a firewall |
|
|
|
|
2008-04-25, 01:04 PM
|
#3 |
|
Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless
Join Date: Jan 2007
Posts: 22
|
Firewall will be the most effective, but .htaccess would work as well. Try blockacountry.com
Best of Luck!
__________________
 ICQ: 169-554-261 info[at]way3[dot]com |
|
|
|
|
2008-05-02, 04:25 PM
|
#4 |
|
WHO IS FONZY!?! Don't they teach you anything at school?
Join Date: May 2008
Posts: 47
|
Odds are if you are able to identify the BaiDu spider in your logs, maybe an htaccess rewrite rule would deal with it. Blocking by IP is easy but very fallible, using rewrite rules is more fun. Like feed them a 301 to a creative place...
|
|
|
|
|
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
Linear Mode
