Creating Representative Query Sets for Offline Evaluation
Karel Bergmann • Location: Theater 5 • Back to Haystack 2023
The scaling of AI/ML teams at Getty Images has resulted in an increased demand for experimentation. As an organization, we seek to better understand the implications of an experiment before proceeding to a customer-facing A/B test, and also to reduce the list of A/B test candidates to something more manageable. Offline testing is a tool that allows us to understand the impacts of sort algorithms on our guardrail metrics, and in some ways to estimate the impact to high-level customer metrics such as conversion and interaction. For offline testing to be indicative of online results, query sets need to be constructed that are representative of customer activity across a spectrum of query attributes. In this presentation, I will discuss a simple method to construct minimal, randomly sampled query sets that are representative across many attributes.
Download the Slides Watch the Video
Karel Bergmann
Getty ImagesKarel Bergmann Sr. Data Scientist at Getty Images As the data science lead on Getty Images' Ranking Track, Karel is involved in all things Ranking. That includes architectural design, algorithm development, tuning and measurement. His work ultimately seeks to improve search results and find best practices to integrate them into Getty Images’ ranking system. Currently, he is focused on bringing Natural Language Search to customers so they can find new ways to express themselves outside of search-speak. His educational background includes evolutionary search algorithms, multi-agent systems, cryptography, malware, SIGINT, compilers and computability theory. Before joining Getty Images, he held diverse industry roles including acting as an embedded data analyst within a professional cycling team. He works from a small hamlet in the foothills of the Rocky Mountains outside of Calgary, Canada. In his spare time, he likes to indulge in outdoor activities like ice climbing, cross country skiing, and lots of cycling.