Information retrieval today is undergoing a paradigm shift, away from the prevailing techniques of the past few decades. Increasingly the focus is moving away from keyword and entity driven search and the inverted index that supports those approaches, in favor of more complex models supported by dense instead of sparse index structures. Neural IR models for retrieval and ranking are becoming increasingly popular but building and scaling these systems presents many challenges. In this talk we present an overview of the current state of Neural IR from the perspective of a large e-commerce company. Among the topics covered will be extracting signals from the clickstream, transformer models and ‘do you need one?’, augmenting the inverted index by predicting keywords, and hard negative mining. We will also cover an exciting new research area, framing semantic search as an Extreme Multi-Label Classification problem, and why the future of semantic search may lie in machine-learned indexes.