Relevance in Aggregate: Dogpile

Dogpile came out with an update of its study about how there’s very little overlap among search top search results for the major engines. Here are the highlighted stats:

  • Only 0.6% of 776,435 first-page search results were the same across the top four Web
    search engines

  • Between 38% and 46% of all searches fail to elicit a click on a first-page search result,
    don’t meet users needs and drive users to try additional
    engines.

  • Web searchers on
    average use three search engines a month
  • Search result
    rankings differ significantly across major search engines; only 3.6 percent of
    the number-one ranked, non-sponsored search results were the same across all
    search engines in a given query

This is all interesting research, but I’m not sold on their analysis. Consider this quote from the release:

"This study
reinforces what we at InfoSpace have long known – often users do not find the
results they need with any single search engine.  Metasearch offers the most
robust and efficient search solution to meet their needs," said Rod Diefendorf, vice president of online
and local search at InfoSpace.  "With less than one percent overlap in first
page results, there is great value to using a metasearch engine like Dogpile.com
to quickly comb through multiple search engines at once for the most relevant
results
."

What Dogpile has yet to convince me of is that it offers the most relevant results simply by aggregating everything. Let’s say Engine 1 is extremely
relevant, while Engines 2 and 3 are moderately relevant and Engine 4 is
practically irrelevant in terms of the quality of their results. Then Dogpile,
by culling results from all four engines, actually dilutes the relevance of
Engine 1.
It’s probably about as good as Engines 2 and 3, and presumably far superior to Engine 4. If it only has those engines’ results to work with, it’s not going to be any better than Engine 1.

If Engine 1 is Google, then Dogpile would be a step down for most searchers given Google’s roughly 60% share of search queries. If Engine 1 is Ask.com, Dogpile would be a step up for the roughly 95% of searches conducted on other engines.

Here’s what I’d like to find out from Dogpile: If "between 38% and 46% of all searches fail to elicit a click on a first-page search result," do Dogpile searches perform any better?

2 thoughts on “Relevance in Aggregate: Dogpile

  1. For the record, I’m a columnist, not a full-time journalist (or a FT blogger, for that matter). I’d actually be doing someone else’s job by finding out. Sarcasm aside there, I will ask around and see if anyone has more info on this.

  2. So you are saying their first page click rate would need to be highest in the industry…The only one who would know that would be comscore. So as a reporter why not do your job and find out? Or maybe you can get that from INSP or the other engines…

Comments are closed.