From user actions to better rankings: Challenges of using search quality feedback for learning to rank

In this talk we’ll describe how we used different types of user feedback (both implicit and explicit) to improve search products for matching vacancies to CVs by Learning to Rank (LTR). We will focus on the pitfalls and surprising results we encountered when trying to leverage both types of feedback for LTR. Although there is a variety of literature about how to set up a system for explicit annotations, as well as much literature on how to model user click behaviour in search engines, the goal of using such feedback for training a reranker is often not targeted. We will aim to do this by describing two case studies, where for one we used explicit feedback (human-made ratings of relevance), and for the other we used implicit feedback (user click event data). Both types of feedback are theoretically known to have their own advantages. E.g. explicit feedback is seen as reliable yet costly to attain, while implicit feedback is assumed to be more noisy but the vast amount of it is supposed to cancel that out. We found some counterintuitive properties of both approaches - for example our explicit feedback mechanism relied on some specific implicit assumptions; and our implicit signals had strong structural biases when using a certain processing mechanism. We will describe the user feedback gathering procedure we conducted, the interpretation methods used on the feedback signals, and we will show some examples of patterns that the reranker(s) learnt to highlight some challenges and recommendations.