Last year, Google announced Caffeine project to improve on web page indexing, accuracy and comprehensiveness of results and now, Caffeine is ready to be rolled out.
Our old index had several layers, some of which were refreshed at a faster rate than others; the main layer would update every couple of weeks. To refresh a layer of the old index, we would analyze the entire web, which meant there was a significant delay between when we found a page and made it available to you.
With Caffeine, we analyze the web in small portions and update our search index on a continuous basis, globally. As we find new pages, or new information on existing pages, we can add these straight to the index. That means you can find fresher information than ever before—no matter when or where it was published.
Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.
Google's new Caffeine search index will crawl the Web more frequently and in smaller bites in hopes of amassing Web content more quickly and comprehensively than before.
The new indexing system, dubbed "Caffeine", will provide the largest selection of web content ever offered on Google. It's faster - processing hundreds of thousands of pages in parallel every second - and takes up nearly 100 million gigabytes of storage in one database.
But, as happens with so many tweaks to Google's mysterious formula, some businesses and websites that rely heavily on Google referrals might see their traffic suffer as the changes could result in them falling further down the search rankings.
We moved the goalposts over here ... Google senior vice-president Alan Eustace describes the changes as "huge".
In a world exclusive interview before today's announcement of the update, Google fellow and 20-year search veteran Amit Singhal said that, while there might be some losers, the primary goal was to improve the user experience.
"As the web is moving forward, freshness is becoming critical and we have worked a lot on this. Starting tomorrow, our users will see 70 per cent fresher results in Google's search results," he said.
"In our world, users come first and, with Caffeine, we will observe our user search experience getting much better, much fresher."
The company dominates the online search business worldwide. Although Microsoft's Bing has made inroads in some markets, Google still has more than 90 per cent of the share of searches in Australia - 87.8 per cent through the google.com.au address and 5 per cent through google.com.
Bing has captured 3.2 per cent of Australian searches in its first year since launch, according to Hitwise's June 5 figures.
Singhal explained that the 70 per cent improvement figure was derived by measuring the "average age" of results across search queries and comparing it with that of the previous algorithm.
Word of Caffeine leaked last year when Google switched it on for a limited number of beta testers but from today it will be rolled out across all of its data centres.
Search engine optimisation (SEO) gurus have been working frantically to figure out the new system and how best to game it to their advantage.
Google fellow and 20-year search veteran Amit Singhal says search must stay fresh'
Matt Cutts Explains Caffeine Update
