Robots.txt & freesites ?

[wolfie]

What's your opinion on this one?

I was checking broken links today and found out that site below have robots.txt file that disallow SE spiders to see these pages.
http://www.titfuckingxxx.com/free/hu...y-fucking/xxx/

Personally, I would like to see freesites submitted to my sites listed in SEs as well, so I deleted this one.

[Linkster]

I guess you just have to decide if you are running a Linklist or you are trying to use people to get rankings in search engines - since they are two different things

[RawAlex]

Some people do that to avoid duplicate page problems. People who build doorways are getting smart to the fact that google doesn't like them (making me feel better about asking to always be on an index.html page and not on index9898978.html). Obviously a robots.txt that blocks SE's from spidering the page limits your likely long term traffic, and that does change the balance between incoming and outgoing traffic potentials.

I can understand why they might do it, and I can understand you removing them from your list.

Alex

[ecchi]

Wolfie, what RawAlex says about why robots.txt excludes these gateways is true. However what he does not point out is that it is in your advantage if they do this, because if the site gets flagged as SE spam, and a link to you is on one of the pages as spam you are likely to loose PR, and if you are found on a lot of pages the SE's think is spam, you may even get blacklisted by the engines.

But WRT: "...so I deleted this one" - Did you email the submitter first and ask him why he did this? Because if you did and he did not reply, then fine, but if you didn't then you are a cheater, stealing bandwidth from your submitters, plain and simple.

Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking friend, and about as acceptable as fucking your kid sister. (In the UK it is an inprisonable offence, and I think the same applies to the US.)

[JK]

Hi wolfie, nice initials. As has already been pointed out in this thread, I disallowed those mirror pages to prevent Google, and other search engines, from indexing them.

Generally, I don't list mirror pages on my own link list and I wanted to see who else sent me any traffic, so that I may give them recips on my future free sites. You weren't one on the list.

I'll make sure not to submit my sites to porn-hawk in the future.

JK

[quest]

I have a question and Please pardon my ignorence...

why not just a noindex in the META tag on mirror pages?


Ben

[JK]

Why, when I can disallow the entire folder with one file?

I don't really see the difference, or the relevance.

[Magnoody]

Google won't even list duplicated sites if they are on the same domain on the same host so you're not loosing traffic |viking|

[Halfdeck]

I personally never thought of it before, but the robot.txt thing is a nice idea. Like Magnoody said, with duplicate content pages, you won't be getting SE traffic or PR even if there was no robots.txt there. If more LLs allowed people to link multiple index.htmls to one main.html, I wonder if this would ever be a problem.

[Jel]

Originally Posted by Halfdeck

If more LLs allowed people to link multiple index.htmls to one main.html, I wonder if this would ever be a problem.

I allow it

[wolfie]

Thanks to board search feature I found that people was replied to this.

Yeah, I see things different way now. Thanks for replying and been open-minded.

Link popularity was a thing I was thinking back then, but like Linkster said, it is not all. Content will keep surfers coming again and attract new surfers so I will list also mirror pages as always has done.

If I remember correct, I was looking that domain more closely because bot couldn't connect to it. To be honest, I had no idea that looking a robots.txt file is a crime, and I'm not that sure about it now either. If SE bots can read it, why not an human eye is not allowed from a site where link was submitted...
Anyway, this was only time I did it. Haven't got any reasons to do it again.

ecchi,
Thanks for a input also, but people decline freesites from a very different reasons. Including if they simply don't like the site. So would that make those people a cheater..? And I'm not only one who don't always send decline messages or questions. So answer to your question, I didn't sent email about this.

JUST A NOTE!
Porn-Hawk.com reviewers or I don't "fuck with your servers", don't cheat, don't steal nothing. Every company I have done business with have said only good things and being happy. I respect my business partners and propably that's why they like also me.

[ClickBuster]

Originally Posted by ecchi

Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking

ahahahaha...

[swedguy]

We need to get a new little smiley

A Roman Maroni on a big horse with a sword in his hand looking all big and mighty.

[wolfie]

I feel more like..
lol

[glowlite]

Some FYI - (from hard learned experience)

Any file on any server is Public Domain (US) That doesn't mean you or your server needs to actually exist physicaly in the US. It means if your files can be accessed from the US your files are 'free game' to US users. I have no idea about Canada, UK, Aus, etc, laws.

It's sorta like a newspaper article. If it's published in the public domain then it is truely public. Including any supporting documentation (files). That could mean .js or .css or .txt or whatever.

[ecchi]

Originally Posted by wolfie

ecchi,
Thanks for a input also, but people decline freesites from a very different reasons. Including if they simply don't like the site. So would that make those people a cheater..?

If it says in your rules that you will decline a site for a particular reason, then it is not cheating to decline the site. However if you make up a rule after receiving a post, then decline it for a reason that was not in your rules when it was posted, you are a cheater.

So either: It says in your rules that robots.txt files must not be used to stop indexing. Or you did list this site and were only spoofing us when you said "so I deleted this one". Or yes, you are a cheater.

Originally Posted by wolfie

Porn-Hawk.com reviewers or I don't "fuck with your servers"

If you are referring to my comment "Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)?" then please note that what glowlite says is very incorrect, thank God he is a webmaster and not a lawyer. You need a warrant from a judge before you can do this. Servers are private property unless you are allowed to look at files on them. Html, Shtml, etc are usually taken as files that you have permission to visit, but other files are not. Calling a file "robots.txt" basically means "no one but search engine spiders are invited to view this file". If whoever made that post to you is feeling petty minded they can use your first post to get you arrested.

[ecchi]

Originally Posted by glowlite

Some FYI - (from hard learned experience)

Any file on any server is Public Domain (US) That doesn't mean you or your server needs to actually exist physicaly in the US. It means if your files can be accessed from the US your files are 'free game' to US users. I have no idea about Canada, UK, Aus, etc, laws.

It's sorta like a newspaper article. If it's published in the public domain then it is truely public. Including any supporting documentation (files). That could mean .js or .css or .txt or whatever.

1. Saying "Any file on any server is Public Domain" is bollocks.

2. Even if it were not, saying "That doesn't mean you or your server needs to actually exist physically in the US" is also bollocks. If you do something that is illegal in the country I or my server exists in, I can sue you in that country. If I can convince the police to take an interest in that country, then they can apply for your extradition to stand trial in the country that I or my server exists in. In cases like this the US usually agree to extradition.

3. If you have an opinion on a legal matter like this, please always check your facts before making a post that someone might believe, and end up getting arrested for following. It is alright posting bollocks if it does not matter, but it is not alright if you are going to get someone even greener than you into trouble.

[RawAlex]

Ecchi, sorry, but robots.txt is an open and public file. No hack or trespass occurs in accessing the file.

There is a difference however between "public domain" and "free to be viewed by the public". People often confuse the terms. Any file not requiring a password on your server that can be accessed in a normal manner is open for publiv view.

However, it isn't public domain, which would suggest that others could use it for commercial purposes without permission, resell it, etc.

That isn't the case here.

Alex

[wolfie]

I see your point, but LL owners can't list all possible rules. Using common sence is mostly enaugh to get listed. And also reviewers do errors.

Besides I didn't think the way robots.txt can be used, before I read this thread. Now I see a clear reason why it was used. We don't all get all from moms milk!

Originally Posted by ecchi

However if you make up a rule after receiving a post, then decline it for a reason that was not in your rules when it was posted, you are a cheater.

Yes, I was referring to that. I saw the humour in that too - but "cheater" and "stealing" are strong words in serious business boards.

Originally Posted by ecchi

If you are referring to my comment "Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)?"

[DJilla]

Originally Posted by Halfdeck

I personally never thought of it before, but the robot.txt thing is a nice idea. Like Magnoody said, with duplicate content pages, you won't be getting SE traffic or PR even if there was no robots.txt there. If more LLs allowed people to link multiple index.htmls to one main.html, I wonder if this would ever be a problem.

I hope I'm not getting myself in trouble and if anyone has a problem with the practice I hope they'll post it in a response but:

After considering the robots.tx idea a ways back I kind of thought that people who weren't really up on its reason could get freaked out so I passed. BUT, I have begun using your second idea and haven't had any bad feedback. Everybody wants an index page recip so just give it to them. Obviously everybody can't be with penisbot or jays so I've arranged the folders to refer back to a "pages" folder all within the same particular free site sub folder. I'm very careful to change keywords, titles, content tages, and the warning page text on each index page so no duplicates exist.

This way I can watch traffic and see which recips really deliver the best and it gives me the chance to keep people together who like to be together. Maybe I'll get more listings and I understand that getting listings and PR in Google is important for every linklist too, not just direct traffic so I've done this so that I and they can benefit too.

IMHO as a new linklist owner and a long time self employed person I think it is just downright impolite (I can't say unprofessional because it is too common) not to notify a submitter of your successful relationship. I think unless you are one of the top 20 LL's it also helps to build goodwill with the submitters.

Re: Hacking. I'm not a lawyer of course but I'm pretty sure to qualify for hacking the owner has to make an effort to put some barrier or security on the file or at minium perhaps provide a notice and the user has to make an effort to circumvent this barrier (the barrier indicates intention of privacy). Something that is freely accessiable by typing in a url like www.domain/robots.txt probably wouldn't rise to that level. I may be wrong though.

[Greenguy]

Originally Posted by Linkster

I guess you just have to decide if you are running a Linklist or you are trying to use people to get rankings in search engines - since they are two different things

Exactly. I run a Link List, so I could care less if there's any SE stuff on the site at all.

[Simon]

...and about as acceptable as fucking your kid sister.

Okay, I guess I may as well out myself here.

My name is Simon ... and evidently I am a bad man.

No, I never had a kind sister, or an older sister for that matter, so it's hard to be sure what kind of a relationship we might have had.

But .. whenever I see something interesting on a website, I almost always hit the command keys to pop the source code to see how it was done. Often, if it was done by Javascript or CSS, I'll look for the link to the external JS or CSS file, and open that in my browser too.

I've looked at all kinds of readily available supporting files and page source codes this way. I've learned a lot by reading these Javascript, CSS, robots.txt, and other files. Mostly I've used the information to code better pages of my own.

So, you may as well mark me down on the list of people who "peep" behind the scenes of sites. Yes, I do it, I've done it for a long time, and I'll mostly likely keep on doing it.

Simon

--
a lawyer's job is to protect his client from other lawyers

[ecchi]

Originally Posted by wolfie

I see your point, but LL owners can't list all possible rules. Using common sence is mostly enaugh to get listed.

What you should have done is kept the link until after you got a reply to your post. But what you did do (according to your post) is delete the link then ask if you should delete the link. In other words you saw this as an opportunity to drop a link, then post a "mattinblack" style thread here. That is not "using your common sense", that is "cheating honest webmasters who have already given you a link back".

Originally Posted by Simon

My name is Simon ... and evidently I am a bad man.

You misunderstood what I said, I was not talking about looking at source code (which is how many people learn HTML). I was talking about poking around in someone's computer looking at files you have no legal access to. And despite what you have read here, a computer is private property, you cannot poke around in them without the owner's permission, that is illegal. RawAlex's comment on passwords is also wrong, passwords came about to stop people who were happy to break the law, not as a way of defining private property.

If someone goes out and leaves their front door open, that does not mean you are allowed to wander in. That is still trespassing. It is the same with computers, just because "the door is open" you cannot just wander in and poke around.

Personally I have no problems with anyone looking into files on an open server if they want to take the risk. But I do have a problem with people posting on any open board that this is OK to do this. Because some green newbie will read the post, believe it, do that, and end up in jail. It is simply not fair to post misinformation if that misinformation is likely to get someone else in trouble.

[Greenguy]

Originally Posted by ecchi

...If someone goes out and leaves their front door open, that does not mean you are allowed to wander in. That is still trespassing. It is the same with computers, just because "the door is open" you cannot just wander in and poke around...

Based on that statement & thinking, who gave you permission to access anything on this (or any other) domain?

Maybe you should tell Google:
http://www.google.com/search?sourcei...q=robots%2Etxt

[ecchi]

Originally Posted by Greenguy

Based on that statement & thinking, who gave you permission to access anything on this (or any other) domain?

Yes, technically even HTML pages need permission to view, it is just that it is accepted that they are for public view. But if a page is obviously not for the public (eg if a company puts it's financial report on a web page for the company directors only) they can take out criminal proceedings against anyone visiting it.

A few years ago Google were successfully sued for giving links to non passworded but non public HTML page. If someone as big as Google with the money for real legal muscle could not win, what hope for the rest of us?