Google Bot Info request

[mygem]

Hi,

Another one of life's silly questions.

My site has been active for around 3 weeks and it has backlinks that I created from signatures on forums and directories that I swapped links with.

Yet, there hasn't been the slightest sign of a Google Bot even sniffing the site, let alone indexing it and adding it to the SERPS.

Any suggestions? (http://www.porn-is-not-evil.com)

Thanks

[RawAlex]

mygem, I have a domain that is in the same sort of position. All sorts of things have been done to get googlebot to come to the party, and while I have had some good sniffs, nothing a single page has been cached.

What is the history of your domain? Was it up before? Did you buy it new or did you get it from someone?

Alex

[Jel]

Snap. Got a domain that hasn't been touched by googlebot, even with a good amont of backlinks from my own sites and others I traded with - been around 2 months now I bought it new, but have no idea if it was 'live' in a previous time, wayback machine shows no record whatsoever of it, although I hear that's not a definitive answer, whois shows no blacklist status, so I have no idea. I actaully sent an email to google asking whether I'd inadvertently bought a domain that they had blacklisted, and if there was anything that could be done if that was the case, but no reply.

On the (somewhat) brighter side, I hear of people that don't get indexed on some domains for up to 4 months and then google decides to visit and index.

[DJilla]

Originally Posted by Jel

Snap. Got a domain that hasn't been touched by googlebot, even with a good amont of backlinks from my own sites and others I traded with - been around 2 months now :

My site is well over a year old, is indexed with G and gets a nominal amount of SE traffic. In the last two months or so I've spent alot of time on upgrading the SEO of the pages. No google bot has visited either which I thought was really, really, odd. Especially, since MSN has been eating it up, 224 visits last month, 187 this month. So many I thought I was trapping their bot somewhere. Go figure?

[fatum]

Hi Mygem,

If i am not wrong you are running a blog kind of thing there. Are you using any script for that ? because the way i see, you have all your stuff on a huge index page and no subpages or directories..

If you are not already using a script, download and install wordpress to the site and publish your content with that.
With the ping feature of blog scripts (and google being a blog whore nowadays) your site will get spidered in 2-3 days...
just my 2 cents...

[Mattinblack]

for blogs pingomatic http://pingomatic.com/ping/ is useful if your software does not auto ping the updates service. I too had a site that stalled on google but a week after I submitted a sitemap for it they spidered me so maybe that would help?

[walrus]

I was reading some SE stuff earlier today and they said something about the algo dumping new sites in the "sandbox" and watching them for awhile before giving the the go. Anyone heard anything to confirm or dispell this?

edit: ya i type for shit...it works well for a guy who blogs.

[Agent]

Originally Posted by walrus

I was reading some SE stuff earlier today and they said something about the algo dumping new sites in the "sandbox" and watching them for awhile before giving the the go. Anyone heard anything to confirm or dispell this?

edit: ya i type for shit...it works well for a guy who blogs.

If that sandbox exists I don't think it'll prevent a domain from being added to the index. I can usually get a domain in the index within a day or two by linking it from a few of my older sites.

[fatum]

I think "sandbox" is the delay between your domains addition to the index and pagerank calculation... New sites have 0 pagerank and no matter how much inbound links you have it still takes 2-3 months to have a decent PR. At least thats the case for me...

[Mattinblack]

Yeah the buzz is that Google are fed up with the wannabe internet entrepeneurs who are on the get rich quick schemes, put up a site expecting the world to beat a path to their door, read the SE optimisation articles that are 12 months out of date and submit to loads of SE's then pull the site when they still only have 100 visitors a day and no income after 2 months work. By the time google indexes the site it no longer exists or has never been maintained. So now they wait to see if the site is gonna hang around and apply a 'maturation' formula during pagerank so they dont serve a top 20 page thats not there any more (this happens a lot with MSN).

[Halfdeck]

Fix your HTML: "http://validator.w3.org/" Validation may not help, but it won't hurt either.

[Allfetish]

I think "sandbox" is the delay between your domains addition to the index and pagerank calculation... New sites have 0 pagerank and no matter how much inbound links you have it still takes 2-3 months to have a decent PR. At least thats the case for me...

This is what I have heard and experienced as well. Some people say the sandbox is about 6-9 months however.

[Mattinblack]

Originally Posted by Allfetish

This is what I have heard and experienced as well. Some people say the sandbox is about 6-9 months however.

My Guru who gets paid megabucks (not by me!) for seo told me that for a new site OR new index page in a subdirectory or a new subdomain (Tip dont submit index pages in sub-directories to google! Have the index pages in case people nav to the directory by accident but never link to them from anywhere and dissalow them in Robots Txt.)

Within 1-2 Weeks - crawled complete site, indexed and cached top level only. You will be able to see site name, and description snippet if you search for the URL.

From here on site gets crawled every 7-14 days to see how often/whether its updated. Until:

Depending on demand/supply for site theme as assessed by googlebot.
6wks -12wks site fully indexed and incoming/outgoing links checked - note not ranked! Will appear in searches for minority keywords ("suckmyass" and "pornogirl" etc) when search engine gets desperate.
Certain subjects such as Medical, Computer Security and Scientific sites seem to be immune to this.

From here-on an aging/updating algorithm seems to be applied. Updating frequently seems to reduce the waiting time till the algorithm evaporates as does your site being repeatedly found by surfers
on the SE.

Time to full ranking 5-9 months UNLESS your page is in exceptionally high demand ie you ARE Brittany Spears or Harry Potter. OR possibly if your main keeywords are so obscure google only has a few sites on those searches.

Yeah it sucks. Plus I did a google search on one of my more critical keywords (ukescort) on google yesterday and found so many obvious spammers added since xmas that I reckon its not working! It was so bad I spent about a half hour going through the first six pages and emailed google with a long list of obviously auto generated spamsites.

For an Old Site then it will get re-indexed as often as Google has observed it changing. If you want to force a re-index then if you have an XML sitemap on Google Sitemaps resubmitting with the last changed date equal to the day of submission seems to get you re-indexed within 24 hours so long as you already have a sitemap with Google. Otherwise manually resubmitting your site works within one to two weeks.

[Allfetish]

VERY nice info! Thank you for sharing those details, I'm learning.

Originally Posted by Mattinblack

Time to full ranking 5-9 months UNLESS your page is in exceptionally high demand ie you ARE Brittany Spears or Harry Potter. OR possibly if your main keeywords are so obscure google only has a few sites on those searches.

Sounds like our sources mostly are in agreeance then on that.

Homage to the Google gods- order finally within the seeming neverending chaos!

[Halfdeck]

Originally Posted by Mattinblack

(Tip dont submit index pages in sub-directories to google! Have the index pages in case people nav to the directory by accident but never link to them from anywhere and dissalow them in Robots Txt.)

True, linking to index.htm/html can cause a havoc.

Note: Don't rely solely on robots.txt since Google likes to ignore it in some cases.

[DJilla]

Originally Posted by Mattinblack

new index page in a subdirectory or a new subdomain (Tip dont submit index pages in sub-directories to google! Have the index pages in case people nav to the directory by accident but never link to them from anywhere and dissalow them in Robots Txt.)

OOOOppps! Didn't know, never heard ths one! Any more info on how google sees index pages in subdirectories. Why they have a problem with this since it is a well know method to keep surfers where they belong?

[Allfetish]

Originally Posted by Halfdeck

True, linking to index.htm/html can cause a havoc.

Note: Don't rely solely on robots.txt since Google likes to ignore it in some cases.

What is the reason for this? Possibly the rewriting of the url in regards to duplication? e.g.

www.somedomain.com/dir/
www.somedomain.com/dir/index.html

This is interesting if I read it right as I know many linkslists prefer an index.html reciprical link, so as to help somewhat with discouraging doorway submissions.

[DJilla]

Originally Posted by Allfetish

This is interesting if I read it right as I know many linkslists prefer an index.html reciprical link, so as to help somewhat with discouraging doorway submissions.

Yeah! This was what I was thinking too!

[Mattinblack]

It still is fine to use index pages in subdirectories and subdomains its just that Google seems to see these as being new sites for the purposes of its aging algorithm. Just means they will now take longer to appear in Google search results. If they are OLD pages that have been there a while and Google knows about them then no issues... except see below.

A radical change in page content has the same effect. I had a PR5 non-adult adwords site which despite the traffic was not making me too much money. In November I gave the site a major makeover, kept the keys the same, the links the same, the page scanned the same for keyword density and relevance however the graphics and written content did totally change on each page.

Uploaded the site and three days later saw it go from PR6 and in the top 5 searches to unlisted! At first I thought I had been banned but typing in the URL brought up the site and description and a cached page. Gradually over the last few weeks I have seen a trickle of SE traffic from Google. This compares to quite a flood from MSN since I changed it.

This is what caused me to consult my Guru (see previous post). He said I should have changed my sub pages at the rate of one a week and my index page a little bit at a time over the same sort of period. He said that an index page missing on two consecutive googlebot runs could reset aging as well, but I have no experience of that.

[Halfdeck]

Originally Posted by Mattinblack

It still is fine to use index pages in subdirectories

This is a personal call, but keep in mind that linking to "www.domain.com/index.html" and "www.domain.com/" will usually result in Google indexing both urls, in some cases causing one or the other to go supplemental. This is similar to the www/non-www supplemental issue resulting in dupes. In the worst case scenario, the duplicates may lower your ranking in the SERPs; in the best case scenario they merely make a mess out of your Google listing by inflating (doubling) the number of pages listed in Google. As you know, supplementals can be a bitch to get rid of --- a good enough reason for me to take extra measures to make sure Google listings link to me using / and not /index.html.

One way to avoid the problem of course is if you link to "www.domain.com/" and never to "www.domain.com/index.html". You can also 404 /index.html, or use a base ahref= tag.

For free sites with multiple doorways, index.html, index.htm, warning.htm will probably be seen as unique pages (due to unique recip tables, provided you link to enough LLs), but say if you submit a teen-slut/index.html url and link from your hub to that page using /teen-slut/ or if your malicious competitor links to you using both urls, then you're going to end up with a supplemental listing in Google.

[DJilla]

Originally Posted by Halfdeck

As you know, supplementals can be a bitch to get rid of --- a good enough reason for me to take extra measures to make sure Google listings link to me using / and not /index.html.

One way to avoid the problem of course is if you link to "www.domain.com/" and never to "www.domain.com/index.html". You can also 404 /index.html, or use a base ahref= tag.

Ya this part just caught up with me a few months ago, now I'm really careful about that too!

BTW.. Lot of board chatter re: "BIG DADDY" new algo for google rankings messing stuff up again.

Basic info: http://www.mattcutts.com/blog/