Robots.txt & freesites ?

ecchi · 2005-10-06, 01:36 PM

Wolfie, what RawAlex says about why robots.txt excludes these gateways is true. However what he does not point out is that it is in your advantage if they do this, because if the site gets flagged as SE spam, and a link to you is on one of the pages as spam you are likely to loose PR, and if you are found on a lot of pages the SE's think is spam, you may even get blacklisted by the engines.

But WRT: "...so I deleted this one" - Did you email the submitter first and ask him why he did this? Because if you did and he did not reply, then fine, but if you didn't then you are a cheater, stealing bandwidth from your submitters, plain and simple.

Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking friend, and about as acceptable as fucking your kid sister. (In the UK it is an inprisonable offence, and I think the same applies to the US.)

2005-10-06, 01:36 PM	#5
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	Wolfie, what RawAlex says about why robots.txt excludes these gateways is true. However what he does not point out is that it is in your advantage if they do this, because if the site gets flagged as SE spam, and a link to you is on one of the pages as spam you are likely to loose PR, and if you are found on a lot of pages the SE's think is spam, you may even get blacklisted by the engines. But WRT: "...so I deleted this one" - Did you email the submitter first and ask him why he did this? Because if you did and he did not reply, then fine, but if you didn't then you are a cheater, stealing bandwidth from your submitters, plain and simple. Also what the hell were you doing poking around in someone's server looking at non public files (like robots.txt)? That is hacking friend, and about as acceptable as fucking your kid sister. (In the UK it is an inprisonable offence, and I think the same applies to the US.)