robot.txt

Wazza · 2009-07-10, 11:09 PM

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...

Code:

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html

Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

2009-07-10, 11:09 PM	#8
Wazza I'm a jaded evil bastard, I wouldn't piss on myself if I was on fire... Join Date: Apr 2003 Location: Melbourne, Australia Posts: 808	Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you... Don't do this... Code: User-agent: * Disallow: /freesites/porn/freesite-mirror.html Disallow: /freesites/porn/freesite-mirror2.html Do do it for 2257 pages and cgi-bin etc SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs __________________ I sale Internet My sites have no traffic and no PR - let's trade - PM me