 |
|
|
|
|
|
|
|
|
2006-04-14, 09:12 AM
|
#1 |
|
Internet! Is that thing still around?
Join Date: Apr 2006
Posts: 5
|
Robots.txt
I have a ton of questions but #....
I ran a Site Analysis Report on my site and found 13 broken internal links 17 broken external hyperlinks internal seem to ALL be the ones leading to my members area http://www.terrorgasm.com/members/ external are http://click.richfetish.com:8080/ct?id=29314&bn=1532 (ect from this site) i didnt have a robots file so i uploaded this http://www.terrorgasm.com/robots.txt i am sure its inncorrect if anyone could help! (my revamping of the tour /free area, meta tage keywords are all a work in progress so i will be back with questions about that and perhaps reveiws and or suggestions ) Thanx so much in advance for your help! Raven ps is ticketsclub the kiss of death? Last edited by raveo; 2006-04-14 at 09:15 AM.. |
|
|
|
2006-04-15, 05:52 AM
|
#2 |
|
Kids are great, Appu. You can teach them to hate the things you hate and they practically raise themselves now-a-days, you know, with the internet and all
Join Date: Jan 2005
Location: Denmark
Posts: 199
   |
looks good regarding the robots.txt
|
|
|
|
|
2006-04-17, 11:08 AM
|
#3 |
|
You can now put whatever you want in this space :)
Join Date: Sep 2005
Posts: 525
|
Broken internal and external links are the burden of an active webmaster. You always have to double check periodically for these and then hunt them down. This task however may only be tagentially related to a robots.txt file. Robots.txt usually serves to guide the SE and other bots into and out of you file system and perform other housecleaning services:
Begin Format Example For robots.txt file: User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /private/ Disallow: /dbase/ User-agent: msnbot Crawl-delay: 10 User-agent: Teoma Crawl-delay: 10 User-agent: Slurp Crawl-delay: 10 User-agent: aipbot Disallow: / User-agent: BecomeBot Disallow: / User-agent: psbot Disallow: / End Format Example Notes: The * asterisk is a wild card that means "All" crawlers/spiders/bots should stay out of that group of files or directories listed. "Disallow: /" means they should stay out entirely. "Crawl-delay:" slows the bot down so as to not overtax server resources. Crawl-delay should only be needed on very large sites with hundreds or thousands of pages. Most larger search engines (good bots) will crawl your site whether or not you use a robots.txt file. However, some such as MSN seem to require it before they will begin crawling at all. All of the search engine bots will generally request the file on a regular basis to see if any changes have occurred. I'm sure there's a good thread in here: http://www.greenguysboard.com/board/...earchid=344302 For general use: http://www.robotstxt.org/ |
|
|
|
|
2006-04-17, 12:31 PM
|
#4 |
|
You can now put whatever you want in this space :)
Join Date: Oct 2004
Location: New Haven, CT
Posts: 985
|
Try checking your site with Xenu.
On your index, you have a link to file:///C:/Documents%20and%20Settings/Gateway%20User/Desktop/fix/2208.gif I'm still not sure what your robots.txt have to do with your broken links.
__________________
Success is going from failure to failure without a loss of enthusiasm. |
|
|
|
|
2006-04-18, 03:30 AM
|
#5 | |
|
Internet! Is that thing still around?
Join Date: Apr 2006
Posts: 5
|
Quote:
Yet another thing on my never ending to do list |
|
|
|
|
|
2006-04-18, 03:31 AM
|
#6 | |
|
Internet! Is that thing still around?
Join Date: Apr 2006
Posts: 5
|
so...........
Quote:
Disallow: /private/ does this mean my members area ? Would that fix the "broken" link that comes up when it tried to crawl the password protected area? and Disallow: /images/ is this good or bad? i thought that images were good to be crawled to get into the search engines- or is the alt text what is showing up there? Do i just copy and paste this ? Thanxxxxxxxxxx Raven! |
|
|
|
|
|
2006-04-19, 10:26 AM
|
#7 | |
|
You can now put whatever you want in this space :)
Join Date: Sep 2005
Posts: 525
|
Quote:
1) Disallow: /images/ *this is the folder name you want to disallow* the folders can be anything you want to name them. /private/ or /members/ or /whatever/. The ones in the example are commonly used. If you don't use those folder names replace them with the names of the ones you have and want to protect. 2) Broken links really have nothing to do with your robots.txt file. They are simply a typing or coding error on your part. You got find em and kill em! 3) Alt text is crawled on photos, etc. But usually you want to keep them out of your larger IMAGES folder where you might have EVERYTHING. You want the SE's to crawl the photos in your free site folders or main pages.... or maybe not. Hope this helps!  |
|
|
|
|
|
2006-04-20, 12:43 AM
|
#8 | |
|
Internet! Is that thing still around?
Join Date: Jan 2006
Location: FL
Posts: 9
|
Quote:
as DJilla said you need to change the example to your own file/folder names and setup. Your members area is /members/ so you would use Disallow: /members/ The folders you don't have as in the example you need to remove from the robots.txt file Also need to remove this part End Format Example from the bottom. Having a wacked robots.txt is worse then not having one at all. And it doesn't have anything to do with broken links. You just need to do a hunt and fix!
__________________
Zipped Adult Sites 4 fast easy FREE web sites and plug-ins! |
|
|
|
|
 |
|
|
Linear Mode
