Robots.txt & freesites ?

ecchi · 2005-10-06, 01:36 PM

Wolfie, what RawAlex says about why robots.txt excludes these gateways is true. However what he does not point out is that it is in your advantage if they do this, because if the site gets flagged as SE spam, and a link to you is on one of the pages as spam you are likely to loose PR, and if you are found on a lot of pages the SE's think is spam, you may even get blacklisted by the engines.

But WRT: "...so I deleted this one" - Did you email the submitter first and ask him why he did this? Because if you did and he did not reply, then fine, but if you didn't then you are a cheater, stealing bandwidth from your submitters, plain and simple.

Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking friend, and about as acceptable as fucking your kid sister. (In the UK it is an inprisonable offence, and I think the same applies to the US.)

JK · 2005-10-06, 05:24 PM

Hi wolfie, nice initials. As has already been pointed out in this thread, I disallowed those mirror pages to prevent Google, and other search engines, from indexing them.

Generally, I don't list mirror pages on my own link list and I wanted to see who else sent me any traffic, so that I may give them recips on my future free sites. You weren't one on the list.

I'll make sure not to submit my sites to porn-hawk in the future.

JK

quest · 2005-10-06, 05:58 PM

I have a question and Please pardon my ignorence...

why not just a noindex in the META tag on mirror pages?

Ben

JK · 2005-10-06, 06:37 PM

Why, when I can disallow the entire folder with one file?

I don't really see the difference, or the relevance.

ClickBuster · 2006-02-19, 03:24 PM

Quote:

Originally Posted by ecchi

Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking

ahahahaha...

2005-10-06, 05:24 PM	#2
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	Hi wolfie, nice initials. As has already been pointed out in this thread, I disallowed those mirror pages to prevent Google, and other search engines, from indexing them. Generally, I don't list mirror pages on my own link list and I wanted to see who else sent me any traffic, so that I may give them recips on my future free sites. You weren't one on the list. I'll make sure not to submit my sites to porn-hawk in the future. JK __________________ To alcohol! The cause of, and solution to, all of life’s problems

2005-10-06, 06:37 PM	#4
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	Why, when I can disallow the entire folder with one file? I don't really see the difference, or the relevance. __________________ To alcohol! The cause of, and solution to, all of life’s problems

2005-10-06, 01:36 PM	#1
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	Wolfie, what RawAlex says about why robots.txt excludes these gateways is true. However what he does not point out is that it is in your advantage if they do this, because if the site gets flagged as SE spam, and a link to you is on one of the pages as spam you are likely to loose PR, and if you are found on a lot of pages the SE's think is spam, you may even get blacklisted by the engines. But WRT: "...so I deleted this one" - Did you email the submitter first and ask him why he did this? Because if you did and he did not reply, then fine, but if you didn't then you are a cheater, stealing bandwidth from your submitters, plain and simple. Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking friend, and about as acceptable as fucking your kid sister. (In the UK it is an inprisonable offence, and I think the same applies to the US.)

2005-10-06, 05:58 PM	#3
quest Trying is the first step towards failure Join Date: Sep 2004 Location: North of the Motor City, MI. Posts: 122	I have a question and Please pardon my ignorence... why not just a noindex in the META tag on mirror pages? Ben