Interleaving: from evaluation to self learning

Search is on the move. After decades of relying on counting terms, we observe a paradigm shift. As defining relevancy has become increasingly difficult, with the presence of a multitude of diverse and sometimes intertwined signals, new challenges for search engine practitioners call for new ways of thinking about search. Machine learning is such a new way of thinking, and it has become increasingly popular for solving search problems. Bloomberg released a learning to rank plugin for Apache Solr, while OSC did the same for Elasticsearch. In this talk, we introduce the next step in machine learning for search: online learning to rank.

In the talk, we discuss a method called *interleaving* in detail. We describe the history of interleaving, from its origin as evaluation method for search engines to being the algorithm that facilitates self-learning search engines. We show how interleaving can be implemented on top of any search engine and within any (web) interface. Finally, we dive into some real-world results of online learning to rank for e-commerce sites, including the impact on business metrics and an analysis of individual features.