Robots.txt

[raveo]

Originally Posted by DJilla

Broken internal and external links are the burden of an active webmaster. You always have to double check periodically for these and then hunt them down. This task however may only be tagentially related to a robots.txt file. Robots.txt usually serves to guide the SE and other bots into and out of you file system and perform other housecleaning services:

Begin Format Example For robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /private/
Disallow: /dbase/

User-agent: msnbot
Crawl-delay: 10

User-agent: Teoma
Crawl-delay: 10

User-agent: Slurp
Crawl-delay: 10

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

End Format Example

Notes:

The * asterisk is a wild card that means "All" crawlers/spiders/bots should stay out of that group of files or directories listed.

"Disallow: /" means they should stay out entirely.

"Crawl-delay:" slows the bot down so as to not overtax server resources. Crawl-delay should only be needed on very large sites with hundreds or thousands of pages.

Most larger search engines (good bots) will crawl your site whether or not you use a robots.txt file. However, some such as MSN seem to require it before they will begin crawling at all. All of the search engine bots will generally request the file on a regular basis to see if any changes have occurred.

I'm sure there's a good thread in here:
http://www.greenguysboard.com/board/...earchid=344302

For general use:
http://www.robotstxt.org/

Thanx sooooo much for the replies!

Disallow: /private/ does this mean my members area ?
Would that fix the "broken" link that comes up when it tried to crawl the password protected area?

and
Disallow: /images/

is this good or bad?
i thought that images were good to be crawled to get into the search engines- or is the alt text what is showing up there?

Do i just copy and paste this ?


Thanxxxxxxxxxx Raven!

[DJilla]

Originally Posted by raveo

Thanx sooooo much for the replies!

1) Disallow: /private/ does this mean my members area ?

2) Would that fix the "broken" link that comes up when it tried to crawl the password protected area?

and

3) Disallow: /images/

is this good or bad?
i thought that images were good to be crawled to get into the search engines- or is the alt text what is showing up there?

Do i just copy and paste this ?

It easier if I answer these in order.

1) Disallow: /images/ *this is the folder name you want to disallow*
the folders can be anything you want to name them. /private/ or /members/ or /whatever/. The ones in the example are commonly used. If you don't use those folder names replace them with the names of the ones you have and want to protect.

2) Broken links really have nothing to do with your robots.txt file. They are simply a typing or coding error on your part. You got find em and kill em!

3) Alt text is crawled on photos, etc. But usually you want to keep them out of your larger IMAGES folder where you might have EVERYTHING. You want the SE's to crawl the photos in your free site folders or main pages.... or maybe not.

Hope this helps!

[upallnite]

Originally Posted by raveo

Disallow: /private/ does this mean my members area ?
Would that fix the "broken" link that comes up when it tried to crawl the password protected area?

raveo
as DJilla said you need to change the example to your own file/folder names and setup.

Your members area is /members/
so you would use
Disallow: /members/

The folders you don't have as in the example you need to remove from the robots.txt file

Also need to remove this part End Format Example from the bottom.

Having a wacked robots.txt is worse then not having one at all. And it doesn't have anything to do with broken links. You just need to do a hunt and fix!