Interleaving: from evaluation to self learning

John T. Kane • Back to Haystack 2018

Search is on the move. After decades of relying on counting terms, we observe a paradigm shift. As defining relevancy has become increasingly difficult, with the presence of a multitude of diverse and sometimes intertwined signals, new challenges for search engine practitioners call for new ways of thinking about search. Machine learning is such a new way of thinking, and it has become increasingly popular for solving search problems. Bloomberg released a learning to rank plugin for Apache Solr, while OSC did the same for Elasticsearch. In this talk, we introduce the next step in machine learning for search: online learning to rank.

In the talk, we discuss a method called *interleaving* in detail. We describe the history of interleaving, from its origin as evaluation method for search engines to being the algorithm that facilitates self-learning search engines. We show how interleaving can be implemented on top of any search engine and within any (web) interface. Finally, we dive into some real-world results of online learning to rank for e-commerce sites, including the impact on business metrics and an analysis of individual features.

View the Slides
John T. Kane

John T. Kane, Search Evangelist for 904Labs has 20+ years experience in the IT industry working for large corporations and small startups. He held Product Management roles at HP, LucidWorks, Staples and HPE. He was the PM for Lucidworks Fusion 1.0 from initial designs and features to launch. He was responsible for the global launch of a new Site Search engine for HPE.com. Search is his avocation as he follows all major IR conferences for 12+ years. 904Labs Search uses searcher-driven data and machine learning for real-time optimization of search.