Link List SEO / Seppuku

[Halfdeck]

Some good ideas in this thread.

One thing I did notice is you got ~897 pages in Google's main index and most of them are supplemental. That basically means you got more pages than your PageRank can support (or that you need more trusted backlinks). Tag cloud is a good idea, but you might try temporarily robots disallowing them (say, for 14 days) and see if a few pages pop into the main index.

[Useless]

Originally Posted by Halfdeck

Tag cloud is a good idea, but you might try temporarily robots disallowing them (say, for 14 days) and see if a few pages pop into the main index.

It is the oddest of coincidences that I just removed /pervetags from my robots.txt last night after looking at my scan stats in Google's webmaster tools. URLs restricted by robots.txt (2467) - and most them being tag pages. So I said, screw it, and allowed the hungry bot in.

I've been reading your blog, as well as those you link to, in order to get a better grip on Page Rank distribution and its relationship with the supplemental index. So I'm not completely oblivious of what you're saying - just damn close.

Here's my single issue: if I'm getting traffic from those supplemental pages, isn't that better than missing out on that keyword traffic completely? We can theorize that by blocking most of my superfluous pages (the tag and title pages) with robot.txt, that my Page Rank distribution would narrow down to my category pages and hopefully raise their value to Google and therefore [fingers crossed] gain me more traffic to those pages. But I've never had much success with Google on my other sites, which is why I've come to the point that I'm at now; creating a ton of pages with the hopes that a few of them will find some hits. Yes, I'm throwing shit at the wall and hoping that at least its odor will stick.

[Halfdeck]

I've been reading your blog

Oh so someone's reading my blog

isn't that better than missing out on that keyword traffic completely?

Sure. That's possible too; that's why I suggested a temporary test. Just having a couple extra pages in the main index probably won't mean a huge boost in traffic and if the keywords targeted by those pages are competitive, you may not rank high enough to see a noticeable difference. (Remember to write a unique set of robots disallows for Googlebot and MSNBot though, since MSNbot is much less demanding than Googlebot)

Still, I would at least think about disallowing "detail" pages like:

pervespace.com/xxx-links/executive-office-perks-490.html
pervespace.com/xxx-links/butt-fucking-movies-1699.html

As you get more backlinks, you can slowly remove the disallow and let the site expand while keeping 90% of the pages in the main index.

My personal experience is pages in the supplemental index really don't draw alot of traffic, first because if other pages in the main index are more relevant for a search term, they come up first. Second, Google doesn't index words/phrases in the supplemental and main index the same way (for the supplemental index, its more like a "compressed summary") so that a supplemental page may rank for fewer long tails than you want it to.

I had a site with 25K pages in the supplemental index and daily search traffic was just around 150/day. Yeah, its better than nothing, but that kind of traffic isn't enough to generate even 2 sales/month.

Here's an article on how John Scott saw an increase of 7k uniques/day after trimming down content (not a scientic test by any means, but something to think about).

creating a ton of pages with the hopes that a few of them will find some hits.

I hear ya, and its worth playing around with, but I think the best SEO is about creating a site you yourself will visit every day, with simple site navigation that doesn't make you click 10 x to get to where you wanna go.

Every single high-ranking site I can think of (Youtube, Wikipedia, del.icio.us) have a high number of daily visits. That's not just because Wikipedia ranks on the first page for everything under the sun; its because people are hooked on those sites and visit them on a daily basis. That kind of stickiness is the mother of organic links and high search engine rankings.

[Useless]

Originally Posted by Halfdeck

Sure. That's possible too; that's why I suggested a temporary test. Just having a couple extra pages in the main index probably won't mean a huge boost in traffic and if the keywords targeted by those pages are competitive, you may not rank high enough to see a noticeable difference.

Ok, you convinced me. I'll give it a shot. I just edited my robots.txt to disallow both the detail and tag pages. I have been seeing a trickle of traffic off of a few of those detail pages, and hate to lose a single daily hit, but I did say I was willing to experiment. Hopefully this proves helpful. If not, it's a lesson learned and it's not as if there is a heavy amount of traffic at risk.

[Useless]

Originally Posted by Useless Warrior

Ok, you convinced me. I'll give it a shot. I just edited my robots.txt to disallow both the detail and tag pages.

Well...apparently you don't have to keep the door open very long before the hungry vermin find their way in. Ever since I put the disallow back in place for my tag pages, Googlebot and Yahoo! Slurp have been munching almost exclusively on my tag pages. That disallow was missing less than 24 hours. Go figure.

I can't promise traffic to my submitters , but I can guarantee that I will direct the bots in your general direction (from many, many pages) .