Bayesian Optimization of Relevance at Shopify

Recently, Bayesian Optimization of a simple Elasticsearch query has been shown to deliver the best non-neural relevance on the MSMarco Ranking Task (https://www.elastic.co/blog/improving-search-relevance-with-data-driven-query-optimization). For these reasons, at Shopify, we’ve adopted bayesian optimization as a core part of our relevance experimentation workflow. Bayesian Optimization allows machine learning optimization without needing to deploy complex model infrastructure. It optimizes any component of a query or index by finding the ideal values for boosts and other parameters. We feel it’s an important first step before introducing a complete LTR workflow. At Shopify we want to share lessons learned from building our own search relevance bayesian optimizer from scratch. In this talk, we’ll share how it works, and how it’s used with every relevance experiment, and why it should be part of every relevance engineers toolset.

Doug Turnbull

In 2012, Doug saw search relevance would be central to user experiences. Sadly, search relevance was a topic clouded in esoteric mystery. Doug began to democratize this daunting field through blogging and speaking. In 2016, Doug’s book Relevant Search woke the industry up to the importance of search quality. Doug now works at Shopify, working to make great commerce search+discovery possible for everyone. Doug co-created Elasticsearch Learning to Rank with the Wikimedia Foundation and contributed to Trey Grainger’s upcoming AI Powered Search. You can find Doug at his website where he blogs about search and data.

Andy Toulis

Andy is working on search engines at Shopify. As a data scientist on the relevance team, his day-to-day work involves designing, implementing, and evaluating changes to search. Last year, he helped build search on the Shop App for Shopify. This year, he is improving search quality on Shopify storefronts.