Retrieval architectures

Lateinteraction,up close

What becomes possible when every token — or every image patch — gets a voice.

Kasper Rømer GrøntvedColourbox~25 min

Adapted, with attribution, from Amélie Chatelain's Multi-vector Search: A Late Interaction Field Guide. Press r on any slide for its sources.

Open by saying plainly that this section is adapted from Amélie Chatelain's field guide, and that the original goes much deeper. Then get to the puzzle fast.
Orientation

Resetting the baseline

Single-vector dense search uses one fixed-size embedding for the query, and one for each indexed unit.

Encoded independently

Query and document never meet during encoding — so the whole document side is prepared offline, and query time is one similarity search.

But it makes one strong bet

That pool step preserved every clue a future query might care about — decided before anyone knew what would be asked.

Late interaction · a primer
Keep this tight. It is a shared-vocabulary slide, not an argument. The argument starts next.
The puzzle

Why does Claude Code use grep?

If we have strong language models and strong embedding models, why does a frontier coding agent still reach for lexical search?

$ grep -rn "refresh_token" auth/

"Grep is all you need" came back into fashion for a reason. It is worth understanding which reason.

Late interaction · a primer
Ask the room to guess before advancing. Most people answer "infra" — which is half right, and the other half is the interesting half.
The puzzle

Grep is hard to beat — and not only on infra

The boring infra reasons

  • Zero infrastructure — works on any machine
  • No model to deploy
  • No index to maintain
  • Privacy by default

But also: why it sticks

  • Lexical matching is extremely strong for code.
  • Granularity is non-negotiable — exact symbols, paths, functions and error strings all matter.
  • Code is highly formatted data, which grep can exploit.

So the bar for a learned retriever is high

It has to earn its keep by adding semantics without losing exact local clues — and justify deploying a model and maintaining an index on top.

Late interaction · a primer
The right-hand column is the part people underrate. Spend your time there.
The bottleneck

One vector has to carry every clue

A single query can hold several independent relevance constraints.

"What's the backoff for OAuth token refreshes after a 429?"

RetryPolicyHTTP 429 refresh_token()auth/session.py

↓ compressed into

[ single vector · dim d ]

→ diluted signal

The pooled vector must decide what to preserve before it knows which detail the query will care about.

Late interaction · a primer
This is not "a semantic question about authentication" — it is five constraints that all have to survive. This is the bottleneck to keep referring back to.
Capacity isn't the fix

Just increase the dimension?

The obvious instinct — and a clean result showing why it doesn't resolve the structural problem.

  • LIMIT builds a toy corpus from random attributes and names; documents hold combinations of them, and queries ask for those combinations.
  • It looks artificial — and it isn't. Product search has exactly this shape.
  • The question it poses: can one point encode every useful combination of evidence?
gaming monitor27"140 Hz

Three constraints that must all survive the encoding.

LIMIT construction: documents holding combinations of attributes and names
LIMIT relevance matrix: every query-document combination

Figures: Amélie Chatelain, Late Interaction Field Guide, from data in Weller et al., LIMIT

Late interaction · a primer
The paper also proves a mathematical capacity limit; skip that here and stay with the intuition.
Capacity isn't the fix

Bigger vectors plateau

More dimensions help — to a point. They do not change the structure of the problem.

Recall against embedding dimension: dense models climb then plateau, far below the lexical reference
Figure: Amélie Chatelain, Multi-vector Search: A Late Interaction Field Guide, from data in Weller et al., LIMIT

It improves — then flattens

More room buys real gains up to a point, then a visible ceiling. Capacity and structure are not the same thing.

And it gets expensive

Large d across a catalogue our size is an index cost we pay on every asset, forever.

More capacity ≠ preserving local evidence.

Late interaction · a primer
Land the one-line version: capacity is not structure.
Is grep all we need?

Exact match is a gift — but it's sparse

The wrong conclusion would be that BM25 is all we need.

BM25 performance collapsing on the synonym variant of the LIMIT corpus
Figure: Amélie Chatelain, Late Interaction Field Guide, from data in Weller et al., LIMIT

Vocabulary mismatch is alive and well

Lexical retrieval breaks the moment the user says auth and the code says credentials.

On LIMIT's synonym variant, BM25 collapses. Exact matching is only a gift when the vocabulary happens to line up.

So we want both

Exact-ish local evidence, which lexical search gives us free — and soft semantic matching, which dense retrieval gives us free.

At the same time. That is the whole ask.

Late interaction · a primer
Be explicit that this is not a pro-lexical slide. It sets up the "we want both" framing that late interaction answers.
Capacity isn't the fix

But wait — what's this?

One line on that chart doesn't behave like the ordinary dense models.

The same chart, with one line behaving unlike the other dense models
Figure: Amélie Chatelain, Multi-vector Search: A Late Interaction Field Guide
GTE-ModernColBERT

One line here does not behave like the others.

Is it just a huge pooled vector?

No. It uses a different retrieval paradigm.

So instead of squeezing more into one vector — keep multiple local vectors, and delay their interaction until the query arrives.

Late interaction · a primer
This is the hinge of the talk. Pause before turning the page.
Escaping the ceiling

Late interaction sits in the middle

It keeps the bi-encoder's offline document encoding, but lets the query interact with local document vectors during scoring.

Bi-encoder, late interaction and cross-encoder compared
Late interaction · a primer
The key words are "don't pool" and "delay". Everything deployable about this approach comes from documents still being encoded offline.
The operator

Reintroducing MaxSim

For every query token, find its best-matching document token — then sum those best matches.

Query-token by document-token similarity heatmap, with the best match per query token highlighted
Figure: Amélie Chatelain, Late Interaction Field Guide
Late interaction · a primer
Read the heatmap row by row: each row is a query token, each column a document token. For each row MaxSim keeps only the best cell, then sums. Under mean pooling, a rare high-signal term like OAuth contributes far less than it should.
Where this lands

The useful overlap

Multi-vector search sits where the two things we wanted actually meet.

Lexical search

Exact granularity

Local clues survive intact. Semantics do not.

Dense search

Soft semantic matching

Meaning survives pooling. Local clues may not.

Multi-vector search

Both, at a price

Late interaction tries to keep exact local evidence and soft semantic matching — for more storage and more scoring work.

Where the original lecture goes next

Code retrieval makes granularity impossible to ignore · multimodal document retrieval makes compression impossible to ignore · then the production objections: storage, latency, updates and the model ecosystem.

Late interaction · a primer
End by pointing people at the original for the field cases and production objections. This primer deliberately stops here.