Relevance Proof in Yelp Search: LLM-Powered Annotations

In Yelp Search, we heavily rely on user reviews while choosing relevant search results, and we’ve also incorporated other relevant business information into our search system. Search result annotations are a key accompaniment to the results, as “review highlights” annotations can explain to the user why a business is relevant for their intent. We use LLM expansions to power these annotations use cases, while also leveraging our existing search index and highlighting functionality. In this talk, we’ll discuss the challenges we faced in building these annotations, including incorporating LLM outputs in our retrieval system, scaling up to 100% of traffic, and other difficulties in dealing with large amounts of textual data. Lastly, we’ll explain how we re-purposed our intelligent annotation system to create a new endpoint for internal RAG applications.

Pallavi Patil

Yelp

Pallavi Patil is a Tech Lead on the Search Quality team at Yelp, where she specializes in designing and building backend solutions that power UX features on Search Engine Results Pages (SERPs). Over the years she has led improvements on the search map view, powered new search carousels, and upgraded recall strategies. Most recently, Pallavi has been focusing on search result annotations, driving enhancements through the incorporation of LLM query expansions for review snippet selection. She also dedicates some of her time to promoting Sponsorship and Mentorship through the women’s employee resource group at Yelp.