Greenguy's Board - View Single Post - robots txt to prevent crawling of freesites

picXX · 2006-09-20, 07:28 AM

Hi.

I am the owner of both soccerwank.com , sexcarrot.com and pornogata.com wich have the same robots.txt files and all the domains.

When i first started out building galleries and freesites, i was told to create a robots.txt file like that to prevent google from crawling thousands of duplicate galleryfiles and hundred of duplicated freesitefiles.

I am linking to the freesites on my mainsite, but offcourse that would not benefit all the LL i am submitting to.

I willl change the robots.txt files on all my domain asap, and prevent from crawling only the galleryfolders instead of the freesite directory folders.

I am sorry i have braught up the issue, cuz i was really not aware of it, i just followed some friends good advice.

2006-09-20, 07:28 AM	#8
picXX They have the Internet on computers, now? Join Date: Feb 2006 Posts: 141	Hi. I am the owner of both soccerwank.com , sexcarrot.com and pornogata.com wich have the same robots.txt files and all the domains. When i first started out building galleries and freesites, i was told to create a robots.txt file like that to prevent google from crawling thousands of duplicate galleryfiles and hundred of duplicated freesitefiles. I am linking to the freesites on my mainsite, but offcourse that would not benefit all the LL i am submitting to. I willl change the robots.txt files on all my domain asap, and prevent from crawling only the galleryfolders instead of the freesite directory folders. I am sorry i have braught up the issue, cuz i was really not aware of it, i just followed some friends good advice.