From user actions to better rankings: Challenges of using search quality feedback for learning to rank

Agnes Van Belle • Back to Haystack Europe 2018

In this talk we’ll describe how we used different types of user feedback (both implicit and explicit) to improve search products for matching vacancies to CVs by Learning to Rank (LTR). We will focus on the pitfalls and surprising results we encountered when trying to leverage both types of feedback for LTR. Although there is a variety of literature about how to set up a system for explicit annotations, as well as much literature on how to model user click behaviour in search engines, the goal of using such feedback for training a reranker is often not targeted. We will aim to do this by describing two case studies, where for one we used explicit feedback (human-made ratings of relevance), and for the other we used implicit feedback (user click event data). Both types of feedback are theoretically known to have their own advantages. E.g. explicit feedback is seen as reliable yet costly to attain, while implicit feedback is assumed to be more noisy but the vast amount of it is supposed to cancel that out. We found some counterintuitive properties of both approaches - for example our explicit feedback mechanism relied on some specific implicit assumptions; and our implicit signals had strong structural biases when using a certain processing mechanism. We will describe the user feedback gathering procedure we conducted, the interpretation methods used on the feedback signals, and we will show some examples of patterns that the reranker(s) learnt to highlight some challenges and recommendations.

Download the Slides
Agnes Van Belle


Agnes is research engineer and team lead of Search R&D team at Texkernel, focusing on improving the retrieval and matching performance of their main search product that can be used for searching CVs and vacancies as well as automatically matching them. She joined Textkernel in 2014 and has since worked on developing, integrating and leveraging Learning to Rank and Neural IR techniques techniques in the product. Prior to this Agnes graduated from the University of Amsterdam in Artificial Intelligence, and worked on several data science projects for municipalities, enterprises and the government. In general her current interests are information retrieval, predictive modeling, data mining and reactive systems.