Session Description

Vector search inevitably involves representing the documents or items via embeddings at indexing time. In most cases, these embeddings remain frozen and contain strictly the information that the embedding model was able to deduct from the inputs.

In many cases, it would be useful to be able to update the embeddings when new information about the document or item arrives. Until now, this was only possible by fine-tuning the embedding model and re-embedding the entire collection of documents, a long, costly, and risky process.

We introduce per-document fine-tuning: using an embedding model with deterministic dimensions, we were able to fine-tune individual embeddings with new information while the model remained frozen. In the case of multimodal embeddings, this even enabled us to update an image embedding with additional information not found in the image. For example an e-commerce product whose embedding was generated from its picture (i. e. wide-legged diamond studded jeans) can have its embedding augmented with abstract characteristics such as the material (organic cotton), the esthetic (hippie look), and use cases (festival wear).

We will demonstrate the effect on precision and recall of enhancing multimodal e-commerce product embeddings with abstract characteristics using this method.