Quote:
Originally Posted by Wazza
Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...
Don't do this...
Code:
User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html
Do do it for 2257 pages and cgi-bin etc
SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs |
very good point I didnt even think about that... only the SEO values...