robot.txt

[rooty]

Originally Posted by Wazza

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html

Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

very good point I didnt even think about that... only the SEO values...