Measuring and Improving the R in RAG

The quality of text generated by a RAG system is limited by the quality of the text chunks it uses for context. The textbook way to measure the quality of these chunks is for a human expert to give each pair a judgement. Of course, this is expensive, time-consuming, and impossible in practice for any sizable corpus. An attractive alternative is to use an LLM to learn from the experts and generate these judgements as-needed. From there we can tune our retrieval and ultimately improve the quality of our generated responses.

Scott Stults

Opensource Connections LLC

Scott Stults has led implementation teams on a variety of Department of Defense, UK Ministry of Defense, and commercial projects where implementing search has been constrained by sophisticated security requirements. He has implemented Multi-Level Security (MLS) in a Lucene-based environment and is working with the Solr community to develop best practices for securing Solr. As a co-founder of OpenSource Connections, which has been working with Solr since 2007, Mr. Stults leads the Solr Security Practice for the company.