Why wont Google crawl my damn site?

b1ng0 · 2005-06-15, 01:14 AM

Well, the exit links ( sponsor links ) go through this non-existent directory but an htaccess rule rewrites them to a script anyway so... I have it as exit but really it could be anything. I just chose exit so that those who know that they can look at the status bar to see where a link is going, realize that they will be leaving the site ( some of the links are a tad blind ). I'm also working under the assumption that search engines see my fake static pages and not the scripts they 301 or 302 to, or at least thats what they index.. otherwise whats the point. Basically I don't want them to try'n index pages that are nothing but redirects and flag my site as being shady or whatever. Do you mean to say that not having an exit directory defeats the purpose of having the robots.txt? I've never even attempted seo, and haven't done a whole lot of reading, so I'm kind of learning as I go along..

Linkster · 2005-06-15, 07:40 AM

didnt mean to confuse you on the exit directory in the robots - I just thought it was unusual to see a virtual directory listed in there - but definitely if you have rewrites/virtual links being created that is the way to go

I noticed that your description tag in the head section of the site could probably use some work - make it something that will reach out and grab the surfers once you do get listed - since Google will normally use what is in the description tag if the dont find a good desc on the page - also Yahoo will use both the desc and keywords tag so make sure those are good keywords for you (although they seem to be whats on the page)

I guess you know that your 2257 link doesnt work - once it does you might want to have that in a directory that also doesnt get allowed in the robots.txt file

2005-06-15, 07:40 AM	#2
Linkster NO! Im not a female - but being a dragon, I do eat them. Join Date: Mar 2003 Location: Sex Delta Posts: 5,084	didnt mean to confuse you on the exit directory in the robots - I just thought it was unusual to see a virtual directory listed in there - but definitely if you have rewrites/virtual links being created that is the way to go I noticed that your description tag in the head section of the site could probably use some work - make it something that will reach out and grab the surfers once you do get listed - since Google will normally use what is in the description tag if the dont find a good desc on the page - also Yahoo will use both the desc and keywords tag so make sure those are good keywords for you (although they seem to be whats on the page) I guess you know that your 2257 link doesnt work - once it does you might want to have that in a directory that also doesnt get allowed in the robots.txt file __________________ Pussy Chompers Porn Links NSCash

2005-06-15, 01:14 AM	#1
b1ng0 I'm going to the backseat of my car with the woman I love, and I won't be back for TEN MINUTES Join Date: Jan 2005 Posts: 85	Well, the exit links ( sponsor links ) go through this non-existent directory but an htaccess rule rewrites them to a script anyway so... I have it as exit but really it could be anything. I just chose exit so that those who know that they can look at the status bar to see where a link is going, realize that they will be leaving the site ( some of the links are a tad blind ). I'm also working under the assumption that search engines see my fake static pages and not the scripts they 301 or 302 to, or at least thats what they index.. otherwise whats the point. Basically I don't want them to try'n index pages that are nothing but redirects and flag my site as being shady or whatever. Do you mean to say that not having an exit directory defeats the purpose of having the robots.txt? I've never even attempted seo, and haven't done a whole lot of reading, so I'm kind of learning as I go along..