Rated Ranking Evaluator: an Open Source Approach for Search Quality Evaluation

View the Slides View the Video

Every team working on Information Retrieval software struggles with the task of evaluating how well their system performs in terms of search quality(at a specific point in time and historically).

Evaluating search quality is important both to understand and size the improvement or regression of your search application across the development cycles, and to communicate such progress to relevant stakeholders.

To satisfy these requirements an helpful tool must be:

• flexible and highly configurable for a technical user
• immediate, visual and concise for an optimal business utilization

In the industry, and especially in the open source community, the landscape is quite fragmented: such requirements are often achieved using ad-hoc partial solutions that each time require a considerable amount of development and customization effort.

To provide a standard, unified and approachable technology, we developed the Rated Ranking Evaluator (RRE), an open source tool for evaluating and measuring the search quality of a given search infrastructure. RRE is modular, compatible with multiple search technologies and easy to extend. It is composed by a core library and a set of modules and plugins that give it the flexibility to be integrated in automated evaluation processes and in continuous integrations flows.

This talk will introduce RRE, it will describe its latest developments and demonstrate how it can be integrated in a project to measure and assess the search quality of your search application.

The focus of the presentation will be on a live demo showing an example project with a set of initial relevancy issues that we will solve iteration after iteration: using RRE output feedbacks to gradually drive the improvement process until we reach an optimal balance between quality evaluation measures.

Alessandro Benedetti

Sease

Alessandro Benedetti is a Search Consultant and R&D Software Engineer, founder of Sease Ltd. His focus is on information retrieval, information extraction, natural language processing, and machine learning. At Sease Alessandro is working on Search/Machine learning projects and consultancies.

When he isn't on clients projects, he is actively contributing to the open source community and presenting the applications of leading edge techniques in real world scenarios at meetups and conferences such as ECIR, the Lucene/Solr Revolution, ApacheCon, FOSDEM, Haystack Europe and Open Source Summit.