crawling

Painful crawl errors?

HELP! MY SITE HAS 939 CRAWL ERRORS!!1 I see this kind of question several times a week; you’re not alone - many websites have crawl errors. 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html (archive.org) In some cases, crawl errors may come from a legitimate structural issue within your website or CMS.

Watching crawler activity and indexed pages over time

Google and the other search engines are constantly changing their software and infrastructure. Google has apparently switched to a new infrastructure starting beginning 2006 and is currently working on optimizing the “settings”. How does all of this show in the test sites? How does it show in a normal site? Do the number of indexed pages for a “spammy” site go down? Does the activity of the crawlers change? What are the other engines doing?