Towards a Learning To Rank Ecosystem @ Snag - We've got LTR to work! Now what?

View the Slides View the Video

As the largest online marketplace for hourly jobs in the US, Snag strives to connect millions of job seekers with part/full time, hourly and on-demand employment opportunities. Snag started building its learning-to-rank (LTR)-based search system using the Elasticsearch learning-to-rank plugin in 2017 and has switched all of its user queries to LTR by mid-2018, generating significant lift to overall search quality. While fine-tuning and maintaining the LTR system over the past 12 months, our team has come to the realization that continued success of the LTR system requires not only a great ranking model, but also an ecosystem of intelligent metadata services and reliable data infrastructure.

This talk is a collection of examples about the growing pains and remedies of iterating LTR beyond v1.0 at Snag. To start, we will address a few nuances of LTR as a machine-learning problem, e.g. high sample complexity, potential biases from training data, limitations of BM25-based features, incorporation of user preferences, evaluation metrics to please both human users and SEO bots, etc. Then, we will present a few of our newest developments to supplement the current LTR system, including our posting deduplication services, job title normalization services, and architectural designs of our next-generation signal platform and posting enrichment pipeline.

Xun Wang

Snag

Xun Wang is a Principal Data Scientist from Snag’s Data team in its DC office. Since joining Snag in 2016, Xun has been the data science and machine learning engineering lead for Snag’s learning to rank search products, and has contributed to a variety of projects about NLP, forecasting, recommendation and data-driven decision making.

Prior to Snag, Xun was a Senior Data Scientist at Sentrana Inc, an early pioneer in large-scale predictive analytics with applications in sales and marketing. Xun holds a Ph.D. in Transportation Systems Engineering & Statistics from Cornell University. Before coming to the US in 2008, Xun spent most of the first 22 years of his life in his hometown Nanjing, China.