robots txt to prevent crawling of freesites

[Mr. Blue]

Excellent post Halfdeck, you've really explained the topic perfectly and now I do agree with your stance on it. Just a few things.

Originally Posted by Halfdeck

The way I'd go about free site mirrors now would be this:

/index.html
/main.html
/gallery1.html
/gallery2.html
/doorway1.html -> links to main.html
/doorway2.html -> link to main.html

That's the way I used to do my freesite linking for mirrored pages but it seems like a lot of linklist don't like that format for linking. I started to do the linking differently to meet with the requirements, but it annoyed me enough where I decided to go down to a small tight focused group of LinkLists instead.

P.S. Off topic, but if I ran a LL, I would think about tagging links to free sites with NOFOLLOW, as does Technorati tag pages, which are starting to rank very well on Google. You eliminate the reciprocal linking issue (turn all free site links into one way links), and possible negative trust brought on by linking to supplemental/duplicate content pages on untrusted domains.

Just two comments on it. When you refer to tagging links you mean the links coming in would have a rel="tag" type tagging? So, to use your above example domain if I were a LL owner...the link would be included as

<a href="http://www.nastyxvids.com/keyword/index.html" rel="tag">Keyword</a>

Something like that?

Also, since we're really talking about small link lists and their rules (as big linklist probably don't have to worry about the No Follow rule). Wouldn't it be wiser for smaller link lists to use recips more like TGPs use recips? Sorry, I'm more of a tgp guy so I'm going to explain this in those terms...but for TGPs we use a single recip with almost no hope of getting SE off that recip. What we use it for is more or less getting a percentage of traffic from other tgps that list the same gallery (hopefully getting listed with tgps of equal size or bigger than your own). So the focus of the recip is heavily on branding the domain name / name of the tgp or if it's a niche tgp highlighting the niche quality of your tgp that might draw people off of a general tgp to your site if they have a specific fetish.

Shouldn't smaller link lists move to that single recip that heavily brands instead of a category recip?