Why RAG Projects Fail, and How to Make Yours Succeed

Chatbots and Question Answering systems built with RAG (Retrieval Augmented Generation) are the de facto standard for GenAI pilots across every industry. However, many of these projects are failing, due to causes related to both project organization and project execution. With the proper knowledge and vigilance, many of these causes can be avoided, or else recognized and mitigated. We will approach this problem from a technology product or services provider, review case studies from real projects, and explore drivers of failure as well as keys to success. On the project organization side, we will explore how clients set projects up to fail through choice of low-value use cases, user incentive misalignment, poor solution framing and expectation setting, and inadequate data readiness. On the project execution side, we will explore how providers fail to meet expectations through non-prescriptive user experience design, failing to define system capabilities, allowing poor data to impact solution quality perception, and just plain bad technology decisions. Though many of these risks are endemic to providing RAG systems in today’s technology climate, projects do overcome them and exceed client expectations! You will leave this session with the intuition to spot failure risks early, and proven strategies to manage them.

Colin Harman

Nesh

Colin has been implementing Search and Generative AI systems for 2.5 years as the CTO of Nesh. He's used these technologies to solve problems for many enterprises, and has been speaking and writing about them for a year. You can find his writing on his blog, Retrieve and Generate at https://colinharman.substack.com/ .