To (Google) sitemap or not to sitemap, that is the question (old, probably outdated)

Warning: This page is pretty old and was imported from another website. Take any content here with a grain of salt.

There are lots of ways to get indexed by Google. Using Google Sitemaps is only one way - the way that seems to be a bit trendy at the moment. “In the beginning” (June / July 2005), when Google had first introduced Google Sitemaps, it was a sure-fire way to get indexed within hours. It really worked. I bet it not only worked for us, but for lots of spammer sites, so Google had to button it down a bit. It was a different situation just a month later in August 2005, and it will continue to change for as long as Google still has an engineer left :-).

What’s the best way to get indexed and listed on Google? We (Tobias and I) started an experiment - five sites, each set up with similar content but connected differently to Google (if you spot the URLs, please don’t publish them here, send me a mail if you want them confirmed, but we’d like to continue some tests with them). Take a guess - which sites were indexed first?

The factors we tested for indexing on Google

We assumed the following factors would have an influence on the time-to-index (TTI):

  1. Google Sitemaps (of course)
  2. Inbound links
  3. Users with their Google-Bar
  4. Adsense placement

The setup for our tests

Using these factors, we set up five test sites on completely new domains (no history), with no keywords in the domains. The content is all similar, randomly generated from English literature in public domain, each with 911 URLs, filled in with an occasional keyword from a medical textbook (to check the ranking, just for fun).

Our test-sites

  • Site A: Using Google Sitemaps, no inbound links, no visitors
  • Site B: Using Google Sitemaps + Adsense + a remote user with a Google-Bar (ok, not a real user, but rather a remote-controlled IE-user, clicking on a schedule - 6 times daily to random URLs within the site)
  • Site C: Same as Site B but without Google Sitemaps (just Adsense + remote user with Google-Bar)
  • Site D: Also without a Google Sitemap, but with inbound, deep links from outside URLs
  • Site E: With Google Sitemaps and inbound, deep links from outside URLs

Progress tracking

For each site we tracked the number of indexed URLs on Google, MSN and Yahoo as well as the ranking for our obscure medical term on an hourly basis. We bought the domains around mid-August 2005 and started the tests as soon as the domains were reachable. With the exception of a test-click every couple of weeks there were no other human visitors in our test-period (at least according to the logs).

Take a guess!

Which site made it first to the index? In which time frame? Did all sites get indexed? Which factors were most important? Which factors the least? Did all factors work towards getting the sites indexed?

Make your guesses and stay tuned for more :-) (again, please don’t reveal the URLs if you spot them).

(One thing to remember as we reveal the results - Googles algorithms change constantly; so whatever results came out of these test sites in August / September / October 2005 might not be valid any more by the time you read this.)

Warning: This page is pretty old and was imported from another website. Take any content here with a grain of salt.

Comments / questions

There's currently no commenting functionality here. If you'd like to comment, please use Twitter and @me there. Thanks!

Tweet about this - and/or - search for latest comments / top comments

Related pages