Engagement DCG vs Subject Matter Expert DCG - Evaluating the Wisdom of the Crowd

Evaluating the relevance of a search engine result using Discounted Cumulative Gain (DCG) is a common way of quantifying query- document relevance precision. DCG may be computed using a customer engagement method or a subject matter expert (SME) evaluation method. It is a frequent but untested assumption that the results from these two methods of DCG computation are similar in size and correlate well. The difficulties involved with performing a comparison study have prevented rigorous testing of this assumption. Using an identical set of 375 frequent Natural Language queries, the same highly optimized search engine algorithm stack, and well-defined, rigorous test methodologies, both engagement and SME DCG results are computed and compared. Results show that engagement and SME DCG results are not similar in magnitude or trend correlation across the entire set of queries. Reasons for the discrepancies, including assumptions and various biases underlying the methods are discussed.

Doug Rosenoff

Doug is a Director / Principal Product Manager in the Lexis Nexis Global Platform group. He has been involved in legal publishing for more than 27 years, with both LexisNexis and Thomson Reuters. His areas of professional focus are search measurement and search algorithm development. He has US and international patents for various search algorithms and automated linking methods.