Why wont Google crawl my damn site?

b1ng0 · 2005-06-15, 12:14 AM

Well, the exit links ( sponsor links ) go through this non-existent directory but an htaccess rule rewrites them to a script anyway so... I have it as exit but really it could be anything. I just chose exit so that those who know that they can look at the status bar to see where a link is going, realize that they will be leaving the site ( some of the links are a tad blind ). I'm also working under the assumption that search engines see my fake static pages and not the scripts they 301 or 302 to, or at least thats what they index.. otherwise whats the point. Basically I don't want them to try'n index pages that are nothing but redirects and flag my site as being shady or whatever. Do you mean to say that not having an exit directory defeats the purpose of having the robots.txt? I've never even attempted seo, and haven't done a whole lot of reading, so I'm kind of learning as I go along..

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode	Rate This Thread:

2005-06-15, 12:14 AM	#11
b1ng0 I'm going to the backseat of my car with the woman I love, and I won't be back for TEN MINUTES Join Date: Jan 2005 Posts: 85	Well, the exit links ( sponsor links ) go through this non-existent directory but an htaccess rule rewrites them to a script anyway so... I have it as exit but really it could be anything. I just chose exit so that those who know that they can look at the status bar to see where a link is going, realize that they will be leaving the site ( some of the links are a tad blind ). I'm also working under the assumption that search engines see my fake static pages and not the scripts they 301 or 302 to, or at least thats what they index.. otherwise whats the point. Basically I don't want them to try'n index pages that are nothing but redirects and flag my site as being shady or whatever. Do you mean to say that not having an exit directory defeats the purpose of having the robots.txt? I've never even attempted seo, and haven't done a whole lot of reading, so I'm kind of learning as I go along..