Embedding Lifecycle: Reindexing and Compression

Two models' vectors are not merely different, they are incomparable

The ANN lesson called re-embedding "the migration nobody plans for" and then moved on. This lesson is the plan, and the lever that pays for it.

Start with the fact that makes the migration unavoidable, stated precisely enough that it cannot be hand-waved. Two embedding models trained separately produce two different spaces. Not two views of one space with a rotation between them, and not one space where one model is noisier. Each model learned its own arrangement of meaning across its own axes, and the number that comes out of a cosine similarity between a vector from model A and a vector from model B is arithmetically well-formed and semantically meaningless. It is not a worse score. It is not a score.

That is why a model upgrade is a full corpus rebuild rather than a config change, and it is why the only thing that makes the rebuild survivable is that you planned the cutover before you needed it.

The migration is a blue-green deploy

You already have the pattern. Level 7's deployment-strategies lesson gave you blue-green: stand the new thing up beside the old one, move traffic when you have evidence, keep the old one warm long enough to go back. A re-embedding migration is that pattern applied to an index instead of a service, and treating it as a transfer rather than as new machinery is most of the answer.

state          reads served by   writes go to      rollback move
-----------    ---------------   ---------------   ----------------
1 build        index A           A                 nothing to undo
2 dual-write   index A           A and B           drop B
3 backfill     index A           A and B           drop B
4 validate     index A           A and B           drop B
5 flip alias   index B           A and B           point alias at A
6 retain       index B           A and B           point alias at A
7 retire       index B           B                 rebuild A from source

  the alias is the only thing the application knows about. it never
  names an index directly, which is what makes step 5 and its inverse
  a metadata change rather than a deploy.

  dual-write starts BEFORE the backfill, not after. a document that
  changes during a multi-day backfill has to land in both indexes, or
  B is quietly stale in exactly the documents that were most active.

The step teams skip is 4, and the step teams get wrong is the ordering of 2 and 3.

Validate against your corpus, not against a leaderboard

"The new model scores higher on the benchmark" is not evidence about your corpus. Benchmarks are averages over public datasets whose query distribution, document length and vocabulary are not yours, and the whole reason your retrieval system exists is that your corpus is not public data.

So step 4 is a labeled query set of your own: a few hundred to a few thousand queries with known-relevant documents, drawn from real traffic and judged by people who know the domain. Run it against A and against B at the same k, and compare recall and whatever ranking metric you gate on. Compare per-slice as well as overall, because a new model that is better on prose and worse on identifiers will look like a modest improvement in aggregate and like a regression to the half of your users who search for part numbers. The cutover criterion is written down before the run, not chosen after seeing it.

The lever that pays for all of this: you do not need float32

Now the half that changes the economics. The ANN lesson framed the index-family choice as a memory budget question. Two techniques move that budget by orders of magnitude, and they compose.

Matryoshka representation learning is a training objective that makes prefixes work. A model trained this way packs the coarsest information into the earliest dimensions, so the first 256 values of a 1024-dimension vector are themselves a usable embedding rather than a fragment of one. The paper's claim is that these nested prefixes are at least as accurate as independently trained low-dimensional representations, and it reports up to 14x smaller embeddings at the same accuracy on its benchmark, with no additional cost at inference. Truncation becomes a slice rather than a re-embed.

Quantization shrinks each dimension that remains. int8 stores each value in one byte. Binary quantization stores each value in one bit, keeping only its sign.

one vector, 1024 dimensions

  float32          1024 x 4 bytes                 = 4,096 bytes
  int8             1024 x 1 byte                  = 1,024 bytes    4x
  binary           1024 bits / 8                  =   128 bytes   32x
  MRL to 128 dims, then binary
                    128 bits / 8                  =    16 bytes  256x

an index of 100,000,000 vectors

  float32          100,000,000 x 4,096            = 409.6 GB
  int8             100,000,000 x 1,024            = 102.4 GB
  binary           100,000,000 x 128              =  12.8 GB
  MRL-128 + binary 100,000,000 x 16               =   1.6 GB

Those two levers are independent, which is why they multiply: 8x from the dimension cut times 32x from the bit width is the 256x on the last row. And the size of the number is the point. A corpus that needed a distributed HNSW cluster at 409.6 GB fits in one machine's RAM at 12.8 GB, which changes not just the bill but which index family is available to you.

The honest part: the size arithmetic above is exact, and the quality cost is not something anyone can quote for your corpus. MRL's degradation curve is measured per model and per dataset, so the dimension you truncate to is an experiment you run on your own labeled set, not a number you copy. That is the same discipline as the previous section, applied to a different knob.

Rescoring: get the quality back for almost nothing

Binary quantization keeps one bit per dimension, so it loses resolution and recall falls. The recovery is a two-pass search, and it is cheap because the second pass runs over a shortlist rather than the corpus.

top_k = 20, rescore_multiplier = 4

pass 1  search the BINARY index for 4 x 20 = 80 candidates
        distance is a Hamming-style comparison over 128-byte vectors,
        so this pass is both small in memory and fast

pass 2  take those 80 document vectors in full precision (or int8),
        score them against the FLOAT query vector, sort, keep 20

  the binary index is what lives in RAM and answers the search.
  the full-precision vectors only need to be reachable, which means
  they can sit on disk or in object storage: 80 reads per query,
  not 100,000,000.

The reported retention numbers are worth carrying: binary quantization alone preserves roughly 92.5 percent of retrieval performance, and with rescoring that rises to about 96 percent, while int8 with a rescore multiplier of 4 reaches around 99 percent. Measured speedups run about 3.7x for int8 and about 25x on average for binary. So the design is a memory decision with a latency bonus and a small, measurable quality cost that you buy back with a shortlist pass.

Table

The last row deliberately does not carry a published retention number. Truncation quality depends on the model and the corpus, so that cell is an experiment rather than a citation, and treating it as one is the difference between a plan and a hope.

Configuration	Bytes per vector at 1024 dims	Index of 100M vectors	Reported retrieval retention
float32	4,096	409.6 GB	baseline
int8	1,024	102.4 GB	about 99% with rescoring at 4x
binary	128	12.8 GB	about 92.5% alone
binary plus float rescoring	128 in RAM, full precision on disk	12.8 GB resident	about 96%
MRL to 128 dims plus binary	16	1.6 GB	measure on your own corpus

What is not a model change

Not every reason to rebuild is a new model, and confusing them wastes a migration.

Corpus drift. Your documents change over time: new products, new vocabulary, new document types. The model has not moved, and its output for a given input is fixed forever, so this is not model drift. It is your corpus moving away from the queries the model was good at, and it shows up as slice-level regressions rather than as an overall decline.

Query drift. Users start asking about things the corpus covers thinly. That is a content problem wearing a retrieval problem's clothes, and re-embedding will not fix it.

Index decay. The ANN lesson's tombstones: deletes mark nodes rather than stitching them out, the graph degrades, and recall drifts down with no code change. That is a compaction and rebuild schedule, not an embedding question.

The monitoring that separates them is the same labeled query set from step 4, re-run on a schedule and reported per slice, plus a distribution check on incoming documents and queries. A rebuild you scheduled because a number moved is cheap. A rebuild you scheduled because users complained is a quarter of firefighting.

Interview nuance: version the embedding model in the vector metadata, from day one. Every record carries the model id and the dimension it was written with, and every query path asserts on it. Without that field, a corpus that has taken writes from two model generations is unrecoverable except by a full rebuild, because nothing distinguishes the two populations: they have the same shape, they return the same kind of number, and the only symptom is that some results are inexplicably bad. With the field, the same situation is a filtered backfill you can run at your convenience. It costs four bytes per record and it is the difference between an incident and a chore.

Recap: vectors from two models are not comparable, so a model upgrade is a corpus rebuild, and the safe shape is the blue-green pattern you already know: build, dual-write before backfilling, validate on your own labeled query set rather than a leaderboard, flip an alias, retain, retire. Matryoshka training makes prefix truncation a slice instead of a re-embed, quantization takes 4x at int8 and 32x at binary, the two compose to 256x, and a rescoring pass over a shortlist buys most of the quality back while keeping only the small vectors resident. Drift, vocabulary shift and tombstone decay are separate diagnoses with separate fixes, and a model version on every record is what keeps a half-finished migration from becoming a rebuild.

Sources: Matryoshka representation learning · Binary and scalar embedding quantization · Vespa: Matryoshka with binary quantization · Operational advice for dense and sparse retrievers