Quote:
|
Originally Posted by raveo
Disallow: /private/ does this mean my members area ?
Would that fix the "broken" link that comes up when it tried to crawl the password protected area?
|
raveo
as DJilla said you need to change the example to
your own file/folder names and setup.
Your members area is /members/
so you would use
Disallow: /members/
The folders you don't have as in the example you need to remove from the robots.txt file
Also need to remove this part
End Format Example from the bottom.
Having a wacked robots.txt is worse then not having one at all. And it doesn't have anything to do with broken links. You just need to do a hunt and fix!