We Re-Ran Our Own Benchmark. Here's What Broke, and What It Cost Us.
By James W. Kim
In April we published a benchmark result: our retrieval engine scored F1 0.577 on MuSiQue multi-hop QA against a LlamaIndex baseline at 0.429 and BM25 at 0.333, all under the same reader model.
That comparison had a problem, and it was ours. Our engine ran with a 20,000-word retrieval budget. The baselines ran at top-K = 5 — roughly 400 words. A 40–50× input-budget asymmetry, sitting in our own config files. Nobody caught us. We caught us, while preparing the evaluation-controls survey for peer-reviewed publication, and the moment we understood what it meant, the April comparison stopped being a result and became a confound.
So we went back and re-did everything. This post is the full accounting: what we fixed, what we predicted before running, where our predictions were wrong, and the part where the strongest number ever produced on our own public harness now belongs to the baseline.
The re-run
Run 2 matched what Run 1 didn’t:
- Input budgets matched. Baselines at top-K = 50 (≈4,500–4,900 words); our engine at a 5,000-word cutoff.
- Output budgets equalized. Every system capped at 4,000 tokens. (Run 1 quietly truncated our own answers under a tight cap — the April 0.577 was a floor for a bad reason)
- Reader reasoning declared. Minimal and high modes, explicitly as a factor and not an accident of defaults.
- Everything scored on the all-1000-answerable basis with the official MuSiQue scorer, same pinned corpus and questions as before.
Before the fair-grid comparison ran, we wrote down a prediction: matching budgets would raise the baselines and shrink our headline lift. It did. The budget-asymmetric +73.3% over BM25 became an honest +62.9%.
The result we didn’t want
At matched token budgets and matched (minimal) model reading, a flat LlamaIndex vector baseline beats our graph at both embedding models: 0.5301 vs our 0.456 at nomic-768d; 0.6359 vs our 0.583 at gemini-3072d.
We looked for an out. The obvious one: maybe the flat vector search advantage depends on the embedding. We computed the formal architecture × embedding interaction which is something, as far as our survey of the MuSiQue literature found, no published evaluation on this benchmark has ever done — and it’s not significant. The flat search advantage doesn’t care which embedding you pick. The embedding upgrade itself moved us +0.1185 and the flat baseline +0.106: statistically the same lift for both architectures. Better embeddings are a rising tide. They are nobody’s moat, including ours.
The tie that dissolved when we paid to test it
There was one comparison in the Run-2 grid that looked like good news: our best cell (gemini embedding, high reasoning, 0.625) versus the flat vector gemini baseline (0.6359) were not statistically different. A tie with the strongest arm on the board.
Except that comparison was confounded, and we couldn’t overlook it: our cell read with high reasoning; the baseline flat vector read with minimal reasoning. So before declaring “parity” anywhere, we were compelled to run the missing cell — the flat vector baseline at matched high reasoning. One knob. Byte-identical retrieved contexts, pinned by SHA-256. About $7 of API spend.
We pre-registered predictions first, in writing, before dispatch:
| Prediction | What we wrote down | What happened |
|---|---|---|
| Baseline-high F1 | 0.64–0.67, point estimate 0.655 | 0.6999 — above our ceiling |
| Most likely branch | 45%: baseline beats our best cell significantly | Correct — it did, +0.0748* |
| Reasoning-gain hypothesis | our compiled context would gain more from reasoning than flat text | Contradicted — flat gained +0.064 to our +0.043 |
To our chagrin, at matched reasoning modes the architecture gap is −0.0748* at high reasoning and −0.0534* at minimal — stable, significant, and carried by 2- and 3-hop questions (not significant at 4-hop, where everyone struggles). The truth hurts sometimes.
Which means the highest F1 ever recorded on our own public harness — 0.6999 — belongs to the flat vector search baseline. We built the arm, we predicted the direction, we missed the magnitude, and we are publishing it. If a skeptic with $7 of caramel macchiato money was ever going to find that number, it was going to be us first. Never forget we are scientists first and have a duty of truth.
One methodology finding fell out for free: a reasoning “level” setting and a numeric thinking budget are not interchangeable ways to provision the same behavior. A separate diagnostic experiment showed that a numeric thinking budget eliminates output truncation but suppresses productive thinking on the questions that weren’t truncating, netting worse overall. If your evaluation says “thinking enabled,” the operational definition is the measured token distribution, not the config flag. We suspect more than one published comparison is quietly confounded on exactly this confusing nomenclature.
Where the gap actually lives
Haters will read this Run 2 result and declare “graphs lose.” The decomposition says something more specific and meaningful to the community at large.
Both systems retrieve the gold evidence at saturation — 94.7–99.9% of questions, depending on embedding — at these budgets. The gap is not about finding the facts. Conditioning on the gold evidence being present in both contexts, the deficit persists and concentrates at 3-hop: it is a reading problem. Our engine hands the model context compiled from extracted atomic facts; the baseline hands it concatenated raw passages. For this reader, at every reasoning level we tested, the raw passages convert better — the flat arm actually thought less than our best cell and gained more.
We also now hold the first formal architecture × reasoning interaction estimate on this benchmark: not significant. More reasoning lifts both architectures about equally (+0.053 to +0.072), just as better embeddings do. Two commodity levers, both RAG architecture-independent. Neither one moves the gap. The gap largely lives in payload composition — how retrieved knowledge is assembled and presented for the model — and that is an engineering surface we own, not an intrinsic property of graphs we’re stuck with. Stay tuned to see whether we can close the gap without gaming the benchmark through architecture tweaks.
What survives, bluntly
Against the weaker lexical (BM25) baseline at matched budgets: a +62.9% lift. Gold evidence retrieval coverage at the strong embedding model: 99.1%. And a benchmark record that now demonstrates, from and on our own numbers, why we distrust most published RAG comparisons: we found four distinct confounds in our own program — input budgets, output provisioning, reasoning-policy semantics, reader-mode asymmetry — and every one of them had been quietly flattering somebody. The truth will set you free, but nobody said freedom was going to be pretty.
MuSiQue is a static, small-corpus benchmark: eleven thousand passages, everything fits, no time, no updates, no sessions. It is the home turf of flat retrieval, and we published our numbers on it anyway. What this engine is actually for — persistent memory, temporal reasoning, corpora where stuffing the context stops working — is a different regime with different physics, and that argument deserves its own post with its own numbers. This second run had a single job: show you the corrected record and the process that corrects it. Ouch.
Everything above is in the public repo — versioned runs, frozen Run-1 numbers with the asymmetry disclosure, the config delta, the dated corrections log, pinned inputs, and the scorer. The Run-1 baselines re-score on your machine today; the Run-2 verify-path extension is planned and listed. We invite you to check our work. That’s the point.