Relevance in Aggregate: Dogpile

Dogpile came out with an update of its study about how there’s very little overlap among search top search results for the major engines. Here are the highlighted stats:

• Only 0.6% of 776,435 first-page search results were the same across the top four Web
  search engines

• Between 38% and 46% of all searches fail to elicit a click on a first-page search result,
  don’t meet users needs and drive users to try additional
  engines.

• Web searchers on
  average use three search engines a month
• Search result
  rankings differ significantly across major search engines; only 3.6 percent of
  the number-one ranked, non-sponsored search results were the same across all
  search engines in a given query

This is all interesting research, but I’m not sold on their analysis. Consider this quote from the release:

"This study
reinforces what we at InfoSpace have long known – often users do not find the
results they need with any single search engine.  Metasearch offers the most
robust and efficient search solution to meet their needs," said Rod Diefendorf, vice president of online
and local search at InfoSpace.  "With less than one percent overlap in first
page results, there is great value to using a metasearch engine like Dogpile.com
to quickly comb through multiple search engines at once for the most relevant
results."

What Dogpile has yet to convince me of is that it offers the most relevant results simply by aggregating everything. Let’s say Engine 1 is extremely
relevant, while Engines 2 and 3 are moderately relevant and Engine 4 is
practically irrelevant in terms of the quality of their results. Then Dogpile,
by culling results from all four engines, actually dilutes the relevance of
Engine 1. It’s probably about as good as Engines 2 and 3, and presumably far superior to Engine 4. If it only has those engines’ results to work with, it’s not going to be any better than Engine 1.

If Engine 1 is Google, then Dogpile would be a step down for most searchers given Google’s roughly 60% share of search queries. If Engine 1 is Ask.com, Dogpile would be a step up for the roughly 95% of searches conducted on other engines.

Here’s what I’d like to find out from Dogpile: If "between 38% and 46% of all searches fail to elicit a click on a first-page search result," do Dogpile searches perform any better?