Measuring and Improving the R in RAG

Scott Stults • Location: Theater 7 • Back to Haystack 2024

The quality of text generated by a RAG system is limited by the quality of the text chunks it uses for context. The textbook way to measure the quality of these chunks is for a human expert to give each pair a judgement. Of course, this is expensive, time-consuming, and impossible in practice for any sizable corpus. An attractive alternative is to use an LLM to learn from the experts and generate these judgements as-needed. From there we can tune our retrieval and ultimately improve the quality of our generated responses.

Download the Slides Watch the Video

Scott Stults

Opensource Connections LLC

Scott Stults has led implementation teams on a variety of Department of Defense, UK Ministry of Defense, and commercial projects where implementing search has been constrained by sophisticated security requirements. He has implemented Multi-Level Security (MLS) in a Lucene-based environment and is working with the Solr community to develop best practices for securing Solr. As a co-founder of OpenSource Connections, which has been working with Solr since 2007, Mr. Stults leads the Solr Security Practice for the company.