Document Parsing and Multimodal Retrieval

Everything upstream of the embedder

Trace the RAG pipeline backwards. The reranker orders chunks, the chunks came from a splitter, the splitter was handed text, and the text came from a parser. On a corpus of clean HTML that last step is invisible. On a corpus of PDFs it is the single largest source of error in the system, and it is the only stage with no downstream check on it: a parse error becomes a chunk, the chunk gets an embedding, the embedding gets retrieved or does not, and no later stage can tell that the words it is ranking were never in that order on the page.

The previous lesson optimized how a document becomes chunks. This one is about the stage before it, where the document becomes text at all.

Reading order

A PDF is not a document. It is a set of drawing instructions that place glyphs at coordinates. There is no paragraph, no column, no reading order. An extractor reconstructs those, and on a two-column page the naive reconstruction fails in a way that is easy to miss because the output is still fluent English.

one page of a 10-K, two columns, as a human reads it:

  +-------------------------------+-------------------------------+
  | Item 7. Management's          | Segment results. Logistics    |
  | Discussion and Analysis       | revenue rose 3% on higher     |
  |                               | freight volumes, while        |
  | Consolidated revenue for the  | Warehousing revenue fell 8%   |
  | year was $4.11B, an increase  | on the loss of two contracts. |
  | of 2% over the prior year.    |                               |
  +-------------------------------+-------------------------------+

an extractor that walks text runs top to bottom, left to right,
without a column model, emits them in this order:

  Item 7. Management's / Segment results. Logistics / Discussion and
  Analysis / revenue rose 3% on higher / freight volumes, while /
  Consolidated revenue for the / Warehousing revenue fell 8% / year
  was $4.11B, an increase / on the loss of two contracts. / of 2%
  over the prior year.

what the splitter hands the embedder:

  "Item 7. Management's Segment results. Logistics Discussion and
   Analysis revenue rose 3% on higher freight volumes, while
   Consolidated revenue for the Warehousing revenue fell 8% year was
   $4.11B, an increase on the loss of two contracts. of 2% over the
   prior year."

Read the last block as a retrieval engine would. "Consolidated revenue for the Warehousing revenue fell 8%" is a sentence, it embeds fine, and it is false. Nothing in the corpus says it; the page never said it; the parser wrote it. This is why ColPali's authors describe text extraction from visually rich documents as running "through lengthy and brittle processes": the brittleness is not the OCR character error rate, it is the layout reconstruction.

Tables lose their header binding

Reading order is the visible failure. Tables are the expensive one, because the output looks fine and the meaning is gone. A merged corner cell and a two-row header encode a relation: this number is Logistics, 2024, margin. Flattening to text drops the encoding and keeps the digits.

the table on the page:

  +-----------+---------------------+---------------------+
  |           |        2025         |        2024         |
  | Segment   +----------+----------+----------+----------+
  |           | Revenue  | Margin   | Revenue  | Margin   |
  +-----------+----------+----------+----------+----------+
  | Logistics |  2,940   |  11.2%   |  2,854   |  10.9%   |
  | Warehouse |  1,170   |   6.4%   |  1,272   |   7.1%   |
  +-----------+----------+----------+----------+----------+

serialization A, flatten to text (what a naive extractor emits):

  Segment 2025 2024 Revenue Margin Revenue Margin
  Logistics 2,940 11.2% 2,854 10.9%
  Warehouse 1,170 6.4% 1,272 7.1%

  ask "what was the Logistics margin in 2024" and all four numbers on
  the Logistics row are equally reachable. nothing in this text binds
  10.9% to 2024 rather than to 2025, so a model reading it has to guess
  from column order that is no longer present.

serialization B, one row per fact, header binding preserved:

  | Segment   | Year | Revenue | Margin |
  | Logistics | 2025 | 2,940   | 11.2%  |
  | Logistics | 2024 | 2,854   | 10.9%  |
  | Warehouse | 2025 | 1,170   | 6.4%   |
  | Warehouse | 2024 | 1,272   | 7.1%   |

  every cell now travels with the keys that identify it, so a single
  retrieved row is true on its own.

Serialization B is the same information written so that it survives being retrieved alone, which is the chunking lesson's standalone-claim rule applied one stage earlier. Element-based parsing, which annotates the structural elements of a document with a document-understanding model and chunks on those, is measured to improve RAG results on financial reports, and it reaches a workable chunk size without anyone tuning one.

The error cascade

Put the stages in a line and the property that makes this hard is visible.

Ingestion, from page to index

1. Render / OCRglyphs at coordinates become characters
2. Layoutcolumns, headings, tables, reading order
3. Serializeelements become text, with or without their keys
4. Chunksplit the text it was handed
5. Embedone vector per chunk, whatever the chunk says
6. Indexthe vector is now the corpus

That is the argument for treating parsing as an engineering surface with its own tests rather than as a library call. It is also the argument for the fork below, which removes the two highlighted stages entirely.

The visual fork: stop parsing

The 2024 answer to a bad parse was a better parser. There is a second answer, and it is a genuine architectural fork rather than an upgrade: do not extract text at all. Render each page to an image, embed the image directly with a vision-language model that emits one vector per image patch, and retrieve pages. When a page is retrieved, hand the page image to a vision model to read.

ColPali is the reference design: a vision-language model trained to produce multi-vector embeddings from images of document pages, matched with the scoring the late-interaction lesson covers. Its authors introduced the ViDoRe benchmark alongside it precisely because page-level retrieval over visually rich documents had no shared measurement, and report that the approach outperforms text-extraction pipelines while being simpler and end-to-end trainable.

What it removes is the entire left half of that pipeline. There is no OCR step to produce a character error, no layout detector to confuse a sidebar with an abstract, no serializer to drop a header binding. What it adds is a multi-vector index, which is the late-interaction lesson's subject, and a storage bill.

What the fork costs

corpus: 500,000 filings, 40 pages each = 20,000,000 pages

text path, one vector per chunk
  3 chunks per page x 20M pages       = 60,000,000 chunks
  1,024 dims x 4 bytes (float32)      = 4,096 bytes per vector
  60,000,000 x 4,096                  = 245.76 GB

page-image path, one vector per patch
  assume the encoder emits 1,024 patch vectors per page at 128 dims
  20,000,000 x 1,024                  = 20,480,000,000 vectors
  128 dims x 4 bytes                  = 512 bytes per vector
  20,480,000,000 x 512                = 10,485.76 GB, or 42.7x the text index

  the same vectors under 2-bit residual compression against a centroid,
  plus a 4-byte centroid id
  128 x 2 bits = 32 bytes, plus 4     = 36 bytes per vector
  20,480,000,000 x 36                 = 737.28 GB, or 3.0x the text index

Two readings of that block, and the second one is the useful one. Uncompressed, the visual index is out of the question at this corpus size for most budgets. Compressed with the standard residual scheme, it is three times the text index, which is an ordinary infrastructure conversation rather than an architectural veto. Compression is what moves this technique from a paper to a product, and it is why the cost question and the late-interaction question are the same question.

What teams actually ship: route, then measure

The hybrid is the answer, and it is not a compromise. Most corpora are mostly documents that parse cleanly. Route by document type: born-digital text PDFs and HTML take the parse path, scanned or dense-layout documents take the page-image path, and one field on each document records which path it took so the split is visible and reversible.

Routing needs a signal, and the signal is a parse-quality score computed at ingestion rather than a guess: does the page have an embedded text layer, what fraction of characters land inside detected columns, does the extracted text contain the words the page's own headings contain. A production parsing framework reports 96% or better on visual element detection and 93% on associating captions with their elements, which is the shape of number this stage should have. If you cannot say what yours is, you cannot route on it.

Interview nuance: the strong answer names a parsing evaluation, not a parser. Parser vendors and vision models both change quarterly, and an answer that names one is dated within a year. An answer that says "we hold out 200 pages sampled across document types, score reading order, table-cell recovery and caption association against a hand-labeled ground truth, and gate a parser change on it" is a system, and it is also the only way to tell a retrieval regression from a parser regression when both landed the same week.

Recap: parsing is the upstream dependency of every retrieval technique and the one stage no downstream stage can check. Reading order fails silently into fluent, false sentences, and flattened tables lose the header binding that made a number mean something. The architectural fork is to render pages and embed patches, which deletes OCR and layout from the pipeline and buys a multi-vector index whose bill is 43x uncompressed and about 3x with residual compression. Production systems route by document type on a measured parse-quality signal and hold a parsing evaluation the way they hold a retrieval evaluation.

Sources: ColPali · ViDoRe leaderboard · Production PDF element parsing · Element-based chunking