How to start climbing the Relevance Mountain - and make sure you can keep climbing!
Anthony Groves • Location: Theater 5 • Back to Haystack 2020
Search relevance can be daunting to a newbie, especially when there is pressure from your stakeholders to improve it. The technical jargon and high entry barrier of relevance engineering can leave a newbie feeling overwhelmed with no idea where to start. DCG, nDCG, precision@k, MAP, MRR, ERR, LTR. What does all of this mean?! Why can’t you just fix the relevance algorithm and move on?
O’Reilly Media’s learning platform has had a dedicated search engine for many years, but it has suffered from a lack of relevance engineering. Adopting some of the most fundamental ways of measuring and improving relevance, the search team at O’Reilly has started to climb what OSC refers to as the “relevance mountain” and have a lot to share from the experience.
This talk will show how we used user-driven signal modeling to make impactful relevance improvements and a quick offline judgment process for building confidence in those improvements.
We will explore not only how to quickly get started on climbing the relevance mountain, but also how to garner more buy-in from stakeholders so that you can keep climbing. Making a quick impact noticeable by stakeholders has enabled us to invest much more time and focus on developing a lasting relevance engineering process for the company.
Watch the Video
Anthony Groves is the Technical Lead of the Search and Discovery team for O’Reilly Media’s online learning platform (formerly SafariBooks Online). He pushes the company to adopt modern search engineering practices, helping developers find their favorite animal programming book. Since joining O’Reilly in 2018, he has led the efforts on many “firsts” for the company in several years. This includes relevance improvements, Solr upgrades, and development of a custom Solr plugin. Prior to joining O’Reilly, Anthony worked for IBM as a software engineer on the Watson Health and IBM Open Platform teams.