robots txt to prevent crawling of freesites

[Jel]

OK,

I have a guy submitting that on his root robots.txt has the following:

User-agent: *
Disallow: /gall/
Disallow: /gall1/
Disallow: /gall2/
Disallow: /gall3/
Disallow: /gall4/
Disallow: /gall5/
Disallow: /gall6/
Disallow: /gall7/
Disallow: /gall8/
Disallow: /gall9/
Disallow: /gall10/
Disallow: /gall11/
Disallow: /gall12/
Disallow: /gall13/
Disallow: /cgi-bin/
Disallow: /img/

domain: soccerwank.com (also sexcarrot.com with different directory names his freesites are in)

On the freesites themselves, in the head, is the meta:
meta name="robots" content="index, follow"

I was running a link checker and was getting flags with this message:
"The link was not checked due to robots exclusion rules. Check the link manually."
Hence me looking at the root robots file.

Seems very fishy to me, and this is titled 'possible cheaters' but I can't fathom whether this is an honest mistake, as obviously his freesites aren't going to get pickjed up by the SES, or just a way to glean traffic from LLs.

[Carrie]

Also looks like a way to turn recip links (A->B->A) into one-way links from the LLs to his domains. (One-way links being more valuable.)

[ponygirl]

Originally Posted by Jel

On the freesites themselves, in the head, is the meta:
meta name="robots" content="index, follow"

that is the thing that bugs me...why have that in the meta unless he forgot to take it out from a cut & paste off something else. Kinda tricky imo.

he's a member of the board, maybe we'll hear something.

[Preacher]

This isn't the only forum member doing this. Like Jel I'm not sure it's exactly cheating so I would like to hear more opinions on this.

[ponygirl]

Originally Posted by Preacher

This isn't the only forum member doing this. Like Jel I'm not sure it's exactly cheating so I would like to hear more opinions on this.

yes me too. I wouldn't consider it cheating, really - it's not breaking any rules, but it's misleading.

[virgohippy]

Originally Posted by ponygirl

yes me too. I wouldn't consider it cheating, really - it's not breaking any rules, but it's misleading.

If someone's going to take the extra time and energy to stuff their robots.txt file in an attempt to get more back then they're giving I would consider that following the rules, but still acting with malicious intent.

Malicious intent falls under the unspoken rule of, "I don't like your business practices, therefore, I don't want to do business with you."

In this situation, I probably wouldn't send a rejection email, or even ask what's up. I'd just silently make their sites dissappear with a quick click of the delete button.

Jel, thanks for bringing this issue up. As if I didn't already have enough to check for...

[Carrie]

Methinks folks don't look in this section often enough.
I think I'll send TT a note and tell him to look here; I like him and it surprises me that this kind of thing would be done on purpose. Maybe he's got a good explanation for it.
Either way, it certainly isn't in anyone's LL rules that it can't be done, so...? Weird situation.

[picXX]

Hi.

I am the owner of both soccerwank.com , sexcarrot.com and pornogata.com wich have the same robots.txt files and all the domains.

When i first started out building galleries and freesites, i was told to create a robots.txt file like that to prevent google from crawling thousands of duplicate galleryfiles and hundred of duplicated freesitefiles.

I am linking to the freesites on my mainsite, but offcourse that would not benefit all the LL i am submitting to.

I willl change the robots.txt files on all my domain asap, and prevent from crawling only the galleryfolders instead of the freesite directory folders.

I am sorry i have braught up the issue, cuz i was really not aware of it, i just followed some friends good advice.

[picXX]

And please dont see my as a possible cheater.. i have never even once tried to cheat fellow webmasters with intention.

[ponygirl]

Originally Posted by picXX

Hi.

I am the owner of both soccerwank.com , sexcarrot.com and pornogata.com wich have the same robots.txt files and all the domains.

When i first started out building galleries and freesites, i was told to create a robots.txt file like that to prevent google from crawling thousands of duplicate galleryfiles and hundred of duplicated freesitefiles.

I am linking to the freesites on my mainsite, but offcourse that would not benefit all the LL i am submitting to.

I willl change the robots.txt files on all my domain asap, and prevent from crawling only the galleryfolders instead of the freesite directory folders.

I am sorry i have braught up the issue, cuz i was really not aware of it, i just followed some friends good advice.

I rather thought it was something like that.

Originally Posted by picXX

And please dont see my as a possible cheater.. i have never even once tried to cheat fellow webmasters with intention.

That's why I wanted to see what this was all about...I've never seen anything in your sites suggesting you cheat in any way, and I don't think you are a cheater either picXX. Thanks for coming by and clearing it up

[picXX]

I have now removed the disallow to all directories at these 3 domains.

However, i do know most galleribuilders that have a HUB site are doing the same, but that is not me, so just wanted to tell that my directories are open for spiders now.

[picXX]

I don't think you are a cheater either picXX

Thank you... it gets my neckhair to raise and my body to freeze when i come over threads like this, even if this is the first thread i am accused for cheating, it is not wery comfortable to read that someone are considering you a cheat when all you do is to try to follow all the rules/tips/advice from experienced webmasters.

Thanks to Carrie allso, that braught this thread to my attention.

Virgohippy:

In this situation, I probably wouldn't send a rejection email, or even ask what's up. I'd just silently make their sites dissappear with a quick click of the delete button.

In you're case, i would never knew why i never got a listing, and therefore never could have fixed it... well, that's not wery Kosher either?

[Mr. Blue]

Unfortunately picxx you were a victim of bad advice. I'd suggest you no longer listen to the person that gave you this advice in the first place

There's a lot of bad information tossed around on boards, etc, and you really have to be careful on who you listen to. This board is a good place to post questions about LL as you won't get steered wrong as you'll be getting the info straight from the horses (owners) mouth.

[Jel]

Originally Posted by Mr. Blue

Unfortunately picxx you were a victim of bad advice. I'd suggest you no longer listen to the person that gave you this advice in the first place

There's a lot of bad information tossed around on boards, etc, and you really have to be careful on who you listen to. This board is a good place to post questions about LL as you won't get steered wrong as you'll be getting the info straight from the horses (owners) mouth.

Spot on Mr. Blue. picXX thanks for clearing that up, and removing that robots file

[Halfdeck]

I know LL owners like to get all the backlinks they can get their hands on (so would I), but you're forgetting one thing:

Duplicate content.

Getting linkbacks from supplemental pages is not going to do anyone any good.

EDIT:

Not to mention low quality backlinks from free sites aren't going to make or break your ranking on Google (though MSN probably eats them up). In a few years, who knows, Google may ignore them altogether.

One way to look at recips is advertising your LL via increasing brand awareness. Approach them as means of inflating your SE position -- and you're in violation of Google guidelines.

Preventing duplicate content is a legitimate reason for disallowing mirrors, however. A large percentage of supps under a domain may negatively impact the entire domain.

[ponygirl]

Originally Posted by Halfdeck

I know LL owners like to get all the backlinks they can get their hands on (so would I), but you're forgetting one thing:

Duplicate content.

Getting linkbacks from supplemental pages is not going to do anyone any good.

EDIT:

Not to mention low quality backlinks from free sites aren't going to make or break your ranking on Google (though MSN probably eats them up). In a few years, who knows, Google may ignore them altogether.

One way to look at recips is advertising your LL via increasing brand awareness. Approach them as means of inflating your SE position -- and you're in violation of Google guidelines.

Preventing duplicate content is a legitimate reason for disallowing mirrors, however. A large percentage of supps under a domain may negatively impact the entire domain.

I admit to knowing next to nothing about SEO, but I have heard lots about dup content...so I'll just say that I don't care if there is a nofollow for that reason. I'm not going to decline submits for that if someone really thinks it helps them with SEs - maybe it does what do I know

BUT make sure your meta tags don't say anything different, that's all.

now I know this was an honest mistake but it's a good opportunity for everyone to make sure they check the little details

[Mr. Blue]

Originally Posted by Halfdeck

I know LL owners like to get all the backlinks they can get their hands on (so would I), but you're forgetting one thing:

Duplicate content.

Getting linkbacks from supplemental pages is not going to do anyone any good.

EDIT:

Not to mention low quality backlinks from free sites aren't going to make or break your ranking on Google (though MSN probably eats them up). In a few years, who knows, Google may ignore them altogether.

One way to look at recips is advertising your LL via increasing brand awareness. Approach them as means of inflating your SE position -- and you're in violation of Google guidelines.

Preventing duplicate content is a legitimate reason for disallowing mirrors, however. A large percentage of supps under a domain may negatively impact the entire domain.

This is one of the reasons when I was submitting freesites regularly I'd only submit to around 12 so there wouldn't be duplicate page penalties. If however I wanted to submit to more than 12, I'd just change the page content enough to avoid the duplicate page penalty...you can keep the template the same, change the sales text, alt tags, title tags, etc, etc, etc. enough where you wouldn't get penalized for a duplicate page penalty. A little extra work, but feasibly 1 freesite can be optimized for a whole bunch of keywords depending on how many LL you're submitting to and how much you're changing the page.

[virgohippy]

Originally Posted by picXX

In you're case, i would never knew why i never got a listing, and therefore never could have fixed it... well, that's not wery Kosher either?

Yeah... you're probably right.

With my next reincarnation, I'll make it a point to include a "if you don't see your site listed within a couple weeks, and you don't recieve a declined reason, contact me here..."

But I still refuse to get my hands dirty!

[Preacher]

Originally Posted by Jel

...I was running a link checker and was getting flags with this message:
"The link was not checked due to robots exclusion rules. Check the link manually."
Hence me looking at the root robots file.

This is actually more of the point that I was interested in. There is a submitter out there who is disallowing any bot that hits his free-sites, I believe through htaccess.

Therefore whenever my bot ran across his domains all of his free-sites would get flagged and pulled as unavailable and I'd have to re-add them manually. This was incredibly annoying.

I actually did this for a while but finally got tired of it and just left his sites in an error/delisted status. He no longer submits to me, but I still see his name pop-up from time to time.

[virgohippy]

Originally Posted by Halfdeck

Preventing duplicate content is a legitimate reason for disallowing mirrors, however. A large percentage of supps under a domain may negatively impact the entire domain.

From an SEO POV for a LL, which would be worse:

Linking to a number of freesites on a domain with tons of dissallowed pages?

or

Linking to a number of freesites which may or may not be flagged for spam?

Seems to me most submitters aren't able to produce and submit more than a small handful of mirrors anyway.

Originally Posted by Mr. Blue

A little extra work, but feasibly 1 freesite can be optimized for a whole bunch of keywords depending on how many LL you're submitting to and how much you're changing the page.

As a submitter, I second your advice Mr. Blue.

Originally Posted by Preacher

I actually did this for a while but finally got tired of it and just left his sites in an error/delisted status...

Not only that, but the occasional bot check of existing listings must surely make regular check ups yet even more fun.

[Mr. Blue]

btw Halfdeck, this is a purely academic debate on my part and I respect your opinions...you may be completely right and I can be wrong on it. On the surface, at least to me, the robot.txt thing seems wrong, but I do understand your points on the topic and I'd be curious to see what others think on it as well.

[DJilla]

I've generally been of the mind that if its not in the rules you can pretty much do what you want and LL's when they encounter these odd situations (obviously only a few are doing this kind of thing) the LL has the responsibility to keep their listings up after confirming by hand or whatever. I admire Preacher for going that extra mile of rekeying his sites in when his script showed this links were disallowed by robots.txt. Also, there is the real politick that its probably not a good idea if LL's begin just ignoring your sites because you use this technique.

I guess what I became worried about was if picXX was aware of the other topics we were pointing out here and what they mean to him. He obviously is searching and taking advice on "best practices" by using the robots scheme but it seems he may have been frightened into taking it down immediately BEFORE he was able to get his pages in possible order to avoid the duplicate content penalty.

Does he clearly understand now that he should go back to his pages and change the text content, alt tags, and metatags on the mirrors etc AND maybe even re-submit some of them?. I hope so.

[Carrie]

It's gotten to the point where the "SE benefits" (incoming links, quicker spider access coming to your sites) far outweigh the traffic benefits when submitting to LLs. Most of the submitters I know *only* do it now for SE purposes, any sale the site might possibly make is just a bonus.

Used to be you could submit a freesite and make 3 sales the first week. Now you're lucky if you make 3 sales in the 6 months that it's up (and then some LL owners rip it down). The traffic ain't what it used to be... hence the LL owners trading with TGPs, increasing required number of pictures, making pictures bigger, etc.

If you take away the SE benefits then there really isn't much reason left to submit anymore, unless you like working your ass off for the possibility of an odd once-in-a-blue-moon signup. The time would be better spent doing other things.

[DJilla]

Originally Posted by Carrie

It's gotten to the point where the "SE benefits" (incoming links, quicker spider access coming to your sites) far outweigh the traffic benefits when submitting to LLs. Most of the submitters I know *only* do it now for SE purposes, any sale the site might possibly make is just a bonus.

That's an interesting take and I certainly felt the same way about the SE benefits when I got into it. I'm not sure if your referring specifically to just getting your site "spidered" more often in which case, ya you are probably right. But for PR benefit and getting specifc FS listed I think the marketplace has shown that both are pretty low liklihoods. Hence maybe the move towards submitting to LL's you know are high in traffic or aren't going to disappear next month (like mine). But, I don't think you can deny that the spiders will come as a result of your FS listing and is a good reason for expecting an LL to not use the nofollow tag. Perhaps the best and only place for a nofollow tag is strictly on sponsor galleries.

Originally Posted by Carrie

Used to be you could submit a freesite and make 3 sales the first week. Now you're lucky if you make 3 sales in the 6 months that it's up (and then some LL owners rip it down). The traffic ain't what it used to be... hence the LL owners trading with TGPs, increasing required number of pictures, making pictures bigger, etc.

My guess is that lower sales is due to more competition.

[Carrie]

Originally Posted by DJilla

My guess is that lower sales is due to more competition.

Probably a good part of it. I'd love to know if the # of submitters has grown, shrunk, or remained pretty steady over the past 4 years or so. That would be interesting.

I think another factor in less sales might be the sheer number of sites that the larger LLs have in their archives. The ones that have been around a few years have a few hundred thousand sites in there, most likely. I was never up for the "remove the listing after 3/6 months" (because sometimes the SEs don't even notice you for that long or they sandbox you for a while to see if you're gonna stick around), but I'm starting to think that dumping sites that are over a year old might help the submitters make more sales. There would still be a ton of sites to keep the bookmarkers around looking at the LLs ads, so maybe it wouldn't negatively affect the LL owners to do this. Would love your take on it D.