Robots.txt

[DJilla]

Originally Posted by raveo

Thanx sooooo much for the replies!

1) Disallow: /private/ does this mean my members area ?

2) Would that fix the "broken" link that comes up when it tried to crawl the password protected area?

and

3) Disallow: /images/

is this good or bad?
i thought that images were good to be crawled to get into the search engines- or is the alt text what is showing up there?

Do i just copy and paste this ?

It easier if I answer these in order.

1) Disallow: /images/ *this is the folder name you want to disallow*
the folders can be anything you want to name them. /private/ or /members/ or /whatever/. The ones in the example are commonly used. If you don't use those folder names replace them with the names of the ones you have and want to protect.

2) Broken links really have nothing to do with your robots.txt file. They are simply a typing or coding error on your part. You got find em and kill em!

3) Alt text is crawled on photos, etc. But usually you want to keep them out of your larger IMAGES folder where you might have EVERYTHING. You want the SE's to crawl the photos in your free site folders or main pages.... or maybe not.

Hope this helps!