The Quality Wall: Why RAG Retrieval Degrades Silently and What the Research Actually Shows

In the last article, we described the two walls today’s enterprise face with RAG. The deployment mess of multiple databases, disparate query languages, many points of failure is the first wall. Then we have the quality wall — retrieval accuracy that degrades silently as the corpus grows, multi-hop reasoning that doesn’t work, hallucinations that persist despite retrieval.

What follows is not opinion. It is a synthesis of 31 peer-reviewed papers and industry studies published between 2024 and 2026, documenting twelve distinct failure modes in RAG quality. The point is to highlight the fact that this is a systemic issue where every link in the chain is a contributing factor. We do present here that our engine is able to address about five of these failure modes, but the more important point is that every user in the RAG space must be vigilant on mitigation measures to prevent silent quality degradation.

The quality ceiling is real and it’s measured

The assumption behind most RAG architectures is straightforward: retrieve relevant passages, feed them to a language model, get a better informed answer. The assumption holds for simple, one-point factual questions over a small body of knowledge. The moment any of those conditions change — the knowledge base grows, the question requires connecting information across multiple documents and sources, the retrieved context contains conflicts — answer quality degrades.

The worst part? It happens silently.

Getting worse as the corpus grows

Xiang et al. (“When to use Graphs in RAG,” ICLR 2026) ran the experiment that most RAG vendors rather you not see. They measured standard vector-based RAG accuracy on complex reasoning tasks across three corpus sizes: 56K tokens, 603K tokens, and 1.13M tokens. The result: accuracy dropped from 58.6% to 43.2% — a 26% relative decline as the corpus grew by a factor of 20. Graph-based retrieval (HippoRAG2) stayed flat at ~54% across all three sizes.

The fundamental cause of this degradation is retrieval noise accumulation. As the search space expands, embedding similarity captures more passages that are topically similar but factually irrelevant. The top-k list fills with well-scored distractors. The language model receives a bunch of context that looks relevant but isn’t, and then gets really confused. All the model can do at that point is guess.

The ICLR 2025 paper on long-context LLMs and RAG adds a counterintuitive finding: stronger retrieval models make this worse, not better. A stronger embedding model produces more “hard negatives” — distractors that are even more topically plausible and thus even more confusing to the reader LLM. Despite a retrieval model’s higher precision, the performance degradation as retrieval size increases is more pronounced than with BM25.

Multi-hop is not vector search friendly

MuSiQue, the benchmark we used to test our engine, was specifically designed so that no single retrieved passage contains the answer. You need passage A to identify entity X, then passage B that mentions entity X to find fact Y, then passage C that connects fact Y to the answer. That’s multi-hop reasoning, and it defeats similarity search because passage C may have zero lexical or semantic overlap with the original query.

The evidence is now extensive. Gupta et al. (EMNLP 2025) measured a 25–35 point accuracy drop when using RAG-retrieved chunks versus serving up the full document context directly. Retrieval can miss critical information, and even when it finds them, evidence may arrive partial or out of order. Their failure-mode analysis identifies missed final-hop integration and long-range drift as the dominant breakdown modes. The errors compound with hop count — 3-hop and 4-hop questions are where RAG systems fail most catastrophically.

Zarrinkia et al. (2026) pushed this further and found something remarkable: even when graph-based RAG achieves 77–91% context coverage — meaning the gold answer IS present somewhere in the retrieved text — accuracy is only 35–78%. Error decomposition shows that 73–84% of the remaining errors are reasoning failures, NOT retrieval failures. The retrieval worked. The model received the answer somewhere in the payload. It just couldn’t sort through it and find the answer it was looking for or chain it together from the ~10,000 tokens worth of entity descriptions, community summaries, and raw jumbled chunks it was given.

This is the pile problem. The retrieved context arrives as a flat pile of passages, ranked by some combination of similarity and relevance scores. The language model must somehow re-discover the multi-hop chain within that pile. When the pile is large, the model fails not because the information is missing, but because it’s buried.

The context itself makes things worse

Chroma Research (July 2025) tested 18 frontier models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3, among others) and found that every single one degrades as input context length increases. Not some models. All of them. Accuracy dropped past 30% for information positioned in the middle of the context window.

This now well-known problem is the “lost in the middle” phenomenon first documented by Liu et al. (TACL, 2024) and confirmed repeatedly since. Language models attend strongly to the beginning and end of their context but poorly to the middle. The attention pattern follows a U-shaped curve where models are better at using information at the very beginning and very end of their context but experience significant degradation in between those peaks. More context means a wider middle zone where information effectively disappears.

The Chroma study added a finding that challenges conventional RAG wisdom: coherent, well-structured input actually degrades attention more than shuffled input. Structure draws the model’s attention to structural signals like headings, section markers, formatting etc. instead of semantic content. The implication is unsettling: carefully organizing your retrieved passages might make the model worse at extracting the answer from them. It seems we just can’t win.

Paulsen (2025) found that effective context, i.e. the amount of context a model can actually use reliably, falls up to 99% below the advertised context window on complex tasks. A model advertising a 200K-token window may degrade significantly at 50K tokens. Bigger context windows are not a solution to the quality wall. They are, at best, a band-aid.

Hallucinations persist despite retrieval

RAG was supposed to solve hallucinations. Much to our dismay, the data says otherwise.

Magesh et al. (Journal of Empirical Legal Studies, 2025) conducted a preregistered, peer-reviewed evaluation of the two leading commercial legal AI research tools — both built on RAG architectures, both marketed to lawyers as reliable. Lexis+ AI hallucinated in 17% of responses. Westlaw AI-Assisted Research hallucinated in 33% of responses. For comparison, GPT-4 without any retrieval hallucinated at 43%. No question RAG helped, but it wasn’t the cure.

The mechanism was identified by ReDeEP (ICLR 2025 Spotlight): “Hallucinations occur when the Knowledge FFNs in LLMs overemphasize parametric knowledge in the residual stream, while Copying Heads fail to effectively retain or integrate external knowledge from retrieved content.” In plain English: the language model’s own training-time memories compete with the retrieved evidence, and sometimes the memories win. The model receives the correct answer in its context and generates a different one from its parameters.

This is a fundamental architectural property of transformers, not a retrieval problem. Better retrieval can deliver higher-quality evidence that gives the model’s attention mechanisms more to work with, but it cannot rewire the internal competition between parametric and contextual knowledge. Anyone who claims their RAG system eliminates hallucinations is either uninformed or lying. The Stanford data proves it. And we just have to deal with this reality for the time being.

Embeddings don’t see what you think they see

There’s a structural failure mode in embedding-based retrieval that most don’t recognize until it bites them. A detailed analysis in Towards Data Science (June 2026) states it precisely: “Embedding similarity measures topical proximity, not question-to-answer relationship. A page that talks about a topic will often score higher than a page that answers a question about the topic.” Ouch.

Definitions outscore values. Background sections outscore conclusions. Procedures outscore the concrete instances they describe. An enterprise knowledge base with multiple versions of a policy document will retrieve the version that discusses the policy most eloquently, not necessarily the version that is current and in force.

The adversarial fact problem (arxiv 2603.17781, 2026) makes this concrete: facts like “Erlotinib inhibits EGFR with IC50 = 2.3 nM” and “Erlotinib inhibits HER2 with IC50 = 45.1 nM” share the same drug, similar sentence structure, and substantial lexical overlap — embedding with cosine similarity above 0.95. They are virtually indistinguishable to the embedding model, yet they are about completely different targets and different binding affinities. The retriever cannot determine which relationship the user is asking about. In this particular evaluation, embedding retrieval achieved only 20% precision on adversarial fact sets.

Your chunks are wrong

The foundation of most RAG systems is fixed-size chunking pipeline: split every document at 512 tokens (or 256, or 1024) regardless of what the document is or where the boundaries fall. The evidence against this practice is now overwhelming.

NVIDIA’s benchmark (June 2025) tested multiple chunking strategies across diverse document types. Page-level chunking, which at least respects a natural document boundary, achieved the highest average accuracy at 0.648 with the lowest variance. Token-based chunking performed inconsistently, with accuracy swinging wildly depending on chunk size and document type.

A peer-reviewed clinical decision support study (Gomez-Cabello et al., MDPI Bioengineering, November 2025) compared adaptive chunking aligned to logical topic boundaries against a fixed-token baseline on 30 postoperative rhinoplasty questions. Adaptive chunking achieved 87% accuracy versus 50% for the fixed-size baseline (p=0.001, ANOVA with Tukey post hoc). That’s not a marginal improvement. The stark gap is consistent with the NVIDIA finding that respecting document structure matters.

Fixed-size chunking commits the same error regardless of input: it treats a legal contract the same as a clinical trial report the same as a conversation transcript the same as a codebase. A 512-token boundary will split a function definition mid-body, sever a legal clause from its qualifying conditions, or cut a clinical finding from its methodology. The resulting chunks are complete only by the tokenizer’s measure and incoherent by every other measure.

Compounding failures

No single failure listed above would constitute a “wall”. Quality walls are compound structures. Each failure amplifies the others:

Vector retrieval degrades at scale → more noise in retrieved context
More retrieved passages to compensate for noise → context rot kicks in → the model can’t use what it found
Multi-hop questions require chaining information across passages → each hop compounds the error probability
Embeddings miss non-obvious associations → the correct passage isn’t even in the top-k
Fixed-size chunking fragments coherent information → retrieval scores measure chunks, not answers
Even when retrieval succeeds, the model may hallucinate from its own parameters

And the ICLR finding that closes the loop: stronger retrievers don’t break the cycle, they produce harder distractors that make step 2 worse. Bigger context windows don’t break the cycle, they widen the dead zone in step 2. Better prompts don’t break the cycle, the failures are in retrieval and context delivery, upstream of the prompt.

This is the formidable quality wall. Not one failure, but a compound structure where each fix in one dimension still leaves the others to be fixed or makes them worse.

What we built differently

We didn’t set out to solve RAG quality problems. Honestly, before researching to write this article, I had no idea it was this bad. We set out to build a retrieval engine that actually worked on multi-hop questions for pharma and healthcare applications — domains where connecting information across documents isn’t optional, it’s the entire job. The quality wall was what we hit, and instead of banging our head against it, we started questioning what we could do about it in true and typical EluciDx fashion, and the architecture that emerged happens to address the quality wall at five key structural points.

1. Graph-based retrieval resists corpus-scale noise

Our Donna-Alfred engine stores knowledge in a weighted, typed graph rather than a vector index. Retrieval operates through associative activation — spreading energy through weighted connections from query anchor points — rather than embedding similarity search. The graph’s structural constraints (entity matching, typed relationships, confidence-gated edges) filter what pure vector similarity cannot: passages that are topically similar but informationally irrelevant.

The Xiang et al. data is directly relevant. Their finding that graph-based retrieval stays flat while vector retrieval degrades 26% at 20× corpus scale, describes the same architectural principle. Our MuSiQue results on the full 11,656-passage corpus are consistent: F1 0.577 with a compact embedding model versus LlamaIndex at 0.429, on the same corpus, same reader, same hardware. [Corrected 2026-07-17: restated on the all-1000-answerable basis used by the public evaluation harness.] The full methodology, results matrix, and honest limitations are in our first article.

2. Associative co-selection captures what similarity search misses

Zarrinkia et al. showed that 73–84% of multi-hop reasoning failures happen even when the answer is in the retrieved context. The problem is how the context arrives: as a flat pile requiring the language model to re-discover the chain.

Our engine’s spreading activation selects nodes that are informationally connected through graph pathways, even when they are semantically distant in embedding space. A node at hop 2 activates because the initial node had a weighted connection to it, not because both happened to embed near the query. The co-selected nodes are then grouped by source structure (source type, theme, topic, document order) with explicit metadata tags and energy annotations, so the language model receives organized context rather than a ranked pile. So instead of throwing everything at the LLM wall and seeing what sticks, we package and label everything neatly before delivering.

As a controlled experiment, we measured the interaction between retrieval quality and reader capability using a 2×2 matrix: two embedding models of different capabilities crossed with two reader reasoning modes. The combined improvement exceeded the sum of individual improvements by 58% — an unequivocal synergistic effect. Better retrieval doesn’t just give the model more relevant context; it creates reasoning opportunities that only a capable reader can exploit. That multiplicative interaction is consistent with a retrieval approach that selects connected information rather than similar information.

3. Context delivery designed for how language models actually attend

The Chroma and Liu findings of 30%+ accuracy drops for mid-context information, i.e. the Valley of Context Death describe a problem that most RAG systems ignore entirely. Standard practice is to concatenate retrieved passages and dump them into the context window in relevance-ranked order. Spray and pray.

Our context compiler does something different. For models that support it, it uses a bipartite positioning strategy: the highest-relevance content is placed at the beginning of the context window (the primacy zone) and remaining content at the end (the recency zone). The middle is filled with comparatively less important information. This is an explicit architectural response to the documented attention curve — the most important information goes where the model attends best.

In addition, every node in the context carries structured metadata: hierarchical level tags, activation energy scores, and optional temporal validity markers. A context legend teaches the reader LLM how to interpret these tags. The language model doesn’t just receive text from the engine, it receives an organized reference structure with an explicit decoder ring.

4. Source-aware parsing replaces arbitrary chunking

The engine doesn’t treat all documents the same. The preprocessing pipeline has format-specific converters for PDF (with geometric layout analysis), HTML (preserving markup hierarchy), plaintext, conversation (patent-pending) and code (AST-based parsing). Each format is parsed according to its own structural conventions.

The resulting content enters a multi-level node structure: source text at L0 (preserving the original document’s position ordering), extracted facts at L1, topic summaries at L2, and theme groups at L3 — with the critical property that a single node can belong to multiple themes simultaneously. Retrieval can activate at any granularity level depending on what the query needs. Source-type tracking differentiates conversations from documents from code, so the context compiler can group and present them appropriately.

Within text, boundary detection follows the content’s own information transitions rather than arbitrary fixed-size cuts. A legal clause stays with its qualifying conditions. A clinical finding stays with its methodology. A code function definition stays intact.

The adversarial fact problem and the “topical proximity ≠ answer relevance” failure are both embedding-only problems. The engine uses three retrieval signals: keyword-based activation (exact entity matches that embeddings smooth over), embedding-based similarity (semantic proximity for concepts), and graph-structure-based traversal (associative pathways through typed relationships). Research has already revealed no single signal is sufficient. The combination covers gaps that any individual method misses.

What we don’t (can’t) solve

The parametric override problem where the language model ignores retrieved evidence and generates from its own training-time parameters, is an LLM architecture problem fundamental to the transformer framework and not a retrieval problem. Our engine can deliver higher-quality, better-organized context that gives the model’s attention mechanisms more to work with. But we can’t rewire the internal competition between parametric and contextual knowledge.

The Stanford legal AI data (17% and 33% hallucination rates in purpose-built RAG systems) includes failure modes that no retrieval architecture can address. When the model starts fabricating a legal citation that sounds plausible, or mischaracterizes a real case by blending it with parametric memory, the failure is downstream of retrieval entirely.

We state this because the common practice of RAG solutions claiming “hallucination-free” or “99% accuracy” without disclosing what types of errors will actually persist for the end-user is exactly the quality wall’s camouflage. The wall is invisible when you only measure the metrics that look good.

What comes next

We are running additional benchmarks at conversational scale — 128K to 10M tokens — testing the failure modes documented in this article on production-representative data: temporal reasoning, knowledge updates, contradiction detection, multi-document synthesis. Those results, with the same transparency, will follow.

The full technical blog series: Blog 1: RAG Needs to Start Adding Real Value | Blog 2: The Deployment Mess | Blog 4: Multi-Source Synthesis in Healthcare AI

All claims in this article are fact-checked against DOC-010 (internal register) and cite published, peer-reviewed or pre-print sources. The engine described is proprietary to EluciDx Corp. (Vancouver, BC). Patent pending.