Query Rewriting, Decomposition and HyDE

The query is not a good search key

The query path in the RAG architecture lesson starts at "embed query". That first box hides an assumption: that what the user typed is a usable search key. It usually is not, and the reason is structural rather than a matter of users being careless.

what the user typed:
  "why did checkout 500 after the migration"            7 tokens

the passage in the runbook that answers it:
  "Following the 2025-11 datastore cutover, the order service began
   returning HTTP 500 on POST /v1/orders whenever the idempotency
   key lookup timed out against the replica. Operators should ..."
                                                       about 45 tokens

words the two share:  500

  not "checkout"  (the passage says "order service", "POST /v1/orders")
  not "migration" (the passage says "cutover")
  not "why"       (the passage is a statement, the query is a question)

both become one vector in the same space. the embedding model is being
asked to bridge a length gap of six to one, a vocabulary gap, and a
register gap (interrogative against declarative) with no help at all.

Query understanding is the stage that closes those gaps before the index sees anything. It is entirely a design surface: every technique here is optional, each one costs something, and the interesting engineering is deciding which query gets which.

Conversational rewriting, the cheapest one

In a multi-turn assistant, a large share of turns are not standalone questions at all. They are fragments that refer to earlier turns, and an embedding of a fragment is an embedding of the wrong thing.

turn 1  user: "how do I rotate the signing key"
turn 2  user: "and what about the second one"

what goes to the index today:
  embed("and what about the second one")
  nearest neighbors: chunks about second attempts, second factors,
  a second-level cache. nothing about signing keys, and no amount of
  reranking fixes a candidate set that never contained the answer.

what goes to the index after a rewrite against the last three turns:
  embed("how do I rotate the second signing key")

One short model call, a standalone query out, and the whole downstream pipeline works on a well-formed key. This is the highest ratio of value to cost in the lesson and the one production teams most often skip, because the demo was single-turn.

HyDE: search with a hallucination

The most counterintuitive technique in retrieval is also the most instructive, because it takes the asymmetry above seriously. If queries are the wrong shape and documents are the right shape, then turn the query into a document before searching.

query:  "why did checkout 500 after the migration"

step 1  ask a model to answer it, with no retrieval at all
        "After the datastore migration, checkout requests returned
         HTTP 500 because the order service could not reach the
         idempotency store; the connection pool was sized for the
         old cluster and was exhausted by retry traffic."

        the model has never seen this codebase. this text may be
        wrong in every specific.

step 2  embed the generated text, not the question
        v = encode(pseudo_document)

step 3  search the real index with v, and throw the generated text away
        what comes back are real corpus documents, ranked by v

The pseudo-document is document-shaped: long, declarative, and full of the vocabulary a runbook uses. The encoder's lossy bottleneck is doing the work. It cannot preserve the invented specifics, so what the vector carries is the neighborhood, and the neighborhood is right even when the sentences are false. HyDE's authors describe exactly this: an unsupervised contrastive encoder filters out the incorrect details of the hypothetical document, and the resulting vector retrieves real documents by similarity. In the ARAGOG comparison of RAG techniques, HyDE was one of two changes that significantly improved retrieval precision.

What HyDE costs, and the shape of its failure

The cost is a generation on the critical path. Retrieval is not a stage users wait for on its own; it is the front half of a latency budget that ends in a streamed answer, and a 300ms-plus generation added before the index is even queried spends a large share of it before any evidence exists.

The failure shape is narrower and worth naming precisely. The vector lands wherever the pseudo-document points, so a confident hallucination about an unfamiliar domain drags the search into a coherent, plausible, wrong region, and unlike an under-specified query it does not come back with a weak candidate set that a confidence threshold could catch. It comes back with a strong one. That makes HyDE a technique to gate rather than default: run the cheap first-pass retrieval, and only spend the generation when the first pass came back thin.

Inverted HyDE: pay at ingestion instead

Now invert the idea, and the cost moves off the query path entirely. If the problem is that the space is populated with document-shaped text while queries are question-shaped, generate questions for each chunk at ingestion and embed those alongside it.

at ingestion, once per chunk:
  chunk: "Following the 2025-11 datastore cutover, the order service
          began returning HTTP 500 on POST /v1/orders whenever the
          idempotency key lookup timed out against the replica..."

  generate 3 questions this chunk answers:
    "why did the order service return 500 after the cutover"
    "what caused idempotency key lookups to time out"
    "which endpoint failed during the 2025-11 datastore migration"

  index each question vector pointing at the same chunk

at query time:
  embed("why did checkout 500 after the migration")
  this is now a question-to-question comparison, and the two sides
  finally have the same shape, length and register

One generation per chunk at ingestion, the same order of spend as contextual retrieval in the chunking lesson, and zero milliseconds added to any request. The trade is that ingestion-time questions are guesses about what will be asked, so they help most where the query distribution is stable and least where users ask things nobody anticipated.

Decomposition, and an honest negative result

Some questions are two questions. "Which of our regions missed the availability target last quarter, and what did we change in the one that missed it worst" cannot be answered by any single passage, because no passage contains both halves. Decomposition splits the question, retrieves per sub-question, and synthesizes.

It is not a free upgrade, and the measurements say so in two independent places. A 2026 study of agent-orchestrated adaptive RAG found query decomposition gave consistent gains in a structured domain (overall score +0.04, MRR +0.17 on a DevOps knowledge base) and degraded ranking precision on a multi-hop reasoning benchmark. In the ARAGOG comparison, multi-query approaches underperformed the naive baseline outright. Decomposition earns its place on structured domains and on genuinely compound questions; applied to every query it costs latency and can cost precision.

Routing is the answer, not any single technique

Stack all of the above on every request and the budget is gone before retrieval begins. The design that ships is a cheap classifier in front, and the arithmetic is what makes the case.

one pipeline, measured (p95, per stage)

  router classifier (a small encoder, no generation)    12 ms
  embed query                                           25 ms
  hybrid retrieve                                       70 ms
  rerank top 100 down to 8                             145 ms
  assemble context                                      20 ms
                                              baseline 260 ms

  plain turn                  12 + 260                        = 272 ms
  conversational rewrite      272 + 110                        = 382 ms
  HyDE                        272 + 340                        = 612 ms
  decompose into 3, rerank 100 per branch, branches in parallel
                              272 + 150 (split) + 30 (merge)   = 452 ms
  decompose into 3, rerank 40 per branch
                              rerank falls 145 -> 70, so
                              272 + 180 - 75                   = 377 ms

  budget: 400 ms p95

Read the last two lines together, because that is the lesson. Decomposition does not fit the budget at full rerank depth and does fit at reduced depth, which turns "should we decompose" into "what do we give back to afford it". HyDE does not fit on the synchronous path at all under this budget, which is exactly why the inverted, ingestion-time version of it exists.

Table

The first column is the technique everyone names in an interview. The second column is the one that decides whether it ships, and it is the same column for both rows that mention HyDE.

Technique	Where the model call happens	Added p95 on the query path	What it fixes
Conversational rewrite	query time, short	110 ms	fragments that refer to earlier turns
HyDE	query time, long	340 ms	the query and document shape mismatch
Inverted HyDE	ingestion, once per chunk	0 ms	the same mismatch, from the other side
Decomposition	query time, plus fan-out	105 to 180 ms	questions no single passage answers
Router	query time, no generation	12 ms	spending any of the above on queries that do not need it

The router itself needs a fallback, because a classifier is a model and models are wrong. The safe default is the plain path: a misrouted compound question returns a partial answer, which is recoverable, while a misrouted fragment sent through decomposition burns the budget and returns nothing. Route on the cheap side and let the abstention instruction catch the rest.

Interview nuance: name the router and its fallback, not the trick. "We would use HyDE" is a technique. "A small classifier tags each turn as standalone, follow-up or compound, follow-ups get a rewrite for 110ms, compound questions get a three-way decomposition paid for by dropping rerank depth to 40, everything else goes straight through, and an unconfident classification falls back to the plain path" is a design, and it is the only version of this answer that can be held to a latency number.

Recap: queries and documents live in one space with different shapes, lengths and registers. Conversational rewriting is the cheapest fix and the most commonly skipped. HyDE crosses the gap by generating a document and keeping only its vector, which works because the encoder discards the invented specifics, and it costs a generation before retrieval. Inverted HyDE buys the same effect at ingestion for nothing per request. Decomposition helps on structured and genuinely compound questions and is measured to hurt ranking precision on some multi-hop benchmarks. The shippable design is a router with a cheap fallback, defended with the latency arithmetic rather than with a preference.

Sources: HyDE · Survey of query optimization in LLMs · ARAGOG, comparing RAG techniques · Agent-orchestrated adaptive RAG