Search Quality: A Business-Friendly Perspective

Have you ever been asked to fix a bad search experience? Have you ever been asked to predict the outcome of a change to your search algorithm? Fixing search problems can feel like a game of whack-a-mole -- fixing one set of queries breaks others. It's a frustrating game of guesswork and trade-offs. But you're responsible for the overall performance of a search system, and the powers that be want assurances. Wouldn't it be great if you could report, with confidence, the overall quality of search? Search quality metrics such as MAP, MRR, and nDCG exist to help the search engineer, but mapping them to real-world business goals is challenging. And pointing to obscure metrics in the face of contrary perceptions is unsatisfying to everyone. Can we do better?

In this talk, we'll look at some of these evaluation metrics and how to use them. Then, we'll explore the gap between these metrics and the business perspective on search quality. Finally, we'll explore tactics you can use to tell an effective story about search quality that non-experts can understand.