The Multi-Source Synthesis Problem in Healthcare AI: Is Multi-Hop Retrieval the Missing Key Piece?
By James W. Kim
Mirroring the pattern of enterprise AI, AI in healthcare is following a similar trajectory. Succeeding brilliantly in solving some defined problems - particularly when given narrow objectives - AI struggles immensely when complex information needs to be synthesized, which a fundamental problem considering human health is the sum of complex biological system interactions.
We trace this pattern across the spectrum of examples and analyze the core reason for early failures, then proceed to examine recent successes to see why multi-source synthesis is looking more and more like the key to unlocking the true power of AI in healthcare. AI still needs more development, but the wins tell a clear story on which direction should be taken.
Where AI Succeeds
The FDA has authorized 1,451 AI-enabled medical devices through December 2025, with 295 in 2025 alone - a single-year record (Innolitics 2025 Year in Review). The distribution is striking: 1,104 of those devices (76%) are radiology tools (FDA AI/ML-Enabled Medical Devices tracker; IntuitionLabs analysis, March 2026). Cardiology accounts for roughly 9%, neurology 5%. Everything else is scattered across dozens of specialties.
Part of this clustering can be attributed to unintentional regulatory bias - the FDA’s 510(k) pathway rewards narrow, well-bounded claims aimed at lower-risk indications for faster clearance, which structurally favors simpler diagnostic tools. But the regulatory incentive alone does not explain the pattern.
Radiology AI works, and the evidence is strong. Viz LVO, cleared in 2018 for large vessel occlusion detection, reduced stroke treatment times by 31 minutes in a 474-patient multicenter study (multicenter study presented at ISC 2025). That is a massive amount of brain tissue saved. Paige Prostate, cleared in 2021, cut false negatives by 70% and reduced false positives by 24% across 527 whole-slide images. GI Genius improved adenoma detection rates by 12% (RR 1.12) in a pooled meta-analysis of seven RCTs comprising 9,639 colonoscopy patients (Sattar et al., Cureus, 2025). Considering every 1% increase in adenoma detection is associated with a 3% decrease in interval colorectal cancer risk, this translates directly to lives saved at population scale.
These results are real. And we can see a common thread: one standardized input (an image), one discrete output (anomaly yes or no?) and unambiguous ground truth established through massive amounts of meticulously curated training data. The AI looks at ONE thing and tells you what it sees. When the input is simple, AI excels at what it does best - pattern recognition.
Where AI Fails
Now consider what happens when AI needs to look at multiple inputs simultaneously.
IBM’s Watson Health attempted several multi-source clinical AI projects. At MD Anderson Cancer Center, the Oncology Expert Advisor, designed to synthesize patient records, literature and treatment guidelines, consumed $62 million over four years before being cancelled without treating a single patient (UT System audit, 2017). Separately, Watson for Oncology, developed primarily with Memorial Sloan Kettering, was deployed to hundreds of hospitals internationally, but a STAT News investigation found that it recommended treatments described by physicians as unsafe, revealing its inability to reliably reason across multiple knowledge sources (Ross & Swetlitz, STAT News, 2017; Strickland, IEEE Spectrum, 2019). IBM ultimately sold Watson Health for roughly $1 billion in 2022 against over $4 billion in acquisitions. Different projects, same fundamental failure: multi-sourced complex clinical reasoning - a necessity for delivering true clinical utility - that AI simply could not handle.
The Epic Sepsis Model was designed to integrate vital signs, lab results, medications, and diagnostic codes to predict impending sepsis. External validation at Michigan Medicine found an AUC of 0.63, sensitivity of 33%, and a positive predictive value of 12%, which is roughly 8 false alerts for every true case (Wong et al., JAMA Internal Medicine, 2021). A subsequent evaluation across 145,885 emergency department encounters was even worse: sensitivity of 14.7%, PPV of 7.6% (Ostermayer et al., JAMIA Open, 2024). Clinicians, in their superior wisdom, learned to ignore it.
Clinical decision support (CDS) systems broadly follow the same trajectory. A meta-analysis covering 373,608 patients across 60 study arms found overall clinician uptake of just 34.2% (Kouri et al., Implementation Science, 2022). The CDS tools that actually get used tend to be simple single-parameter alerts, e.g. drug interaction warnings, dosing checks etc., which do not require multi-source clinical reasoning.
AI-driven drug discovery tells the same story. AI excels at the early, boundary-confined steps like predicting drug-target binding from molecular structure where Phase I success rates for AI-discovered compounds run 80-90% versus historic industry averages of roughly 40-65%. Seriously impressive. But Phase II success, which requires understanding how a molecule actually behaves in the complex, multi-variable reality of human biology, runs at approximately 40%, statistically indistinguishable from the industry baseline (Jayatunga et al., Drug Discovery Today, 2024). As of May 2026, no AI-discovered drug has received FDA NDA/BLA.
Recent Multi-Source Successes and What That Reveals
Our story has hope, however, and the recent successes are instructive.
The Johns Hopkins TREWS system (Targeted Real-time Early Warning System) integrates EHR data, vitals, labs, and medication history to detect sepsis, a genuinely multi-source problem. A prospective study monitoring 590,736 patients across five hospitals found that when a TREWS alert was confirmed by a provider within three hours, in-hospital sepsis mortality dropped by 3.3%, or an 18.7% relative reduction (Adams et al., Nature Medicine, 2022). The study design did spark some methodological debate, but the clinical signal is strong.
There is more direct evidence in drug repurposing. In January 2020, BenevolentAI used their proprietary biomedical knowledge graph to identify baricitinib - a JAK1/2 inhibitor approved for rheumatoid arthritis - as a potential COVID-19 treatment. The reasoning was explicitly multi-hop: the AI traversed from the drug through the JAK pathway to ACE2 receptor regulation to viral entry mechanisms, identifying that baricitinib could simultaneously inhibit viral entry and suppress the inflammatory cytokine storm (Smith et al., Frontiers in Pharmacology, 2021). That hypothesis, published in The Lancet within weeks (Richardson et al., The Lancet, February 2020), was subsequently validated in two major RCTs (ACTT-2 and CoV-BARRIER) which confirmed significant reductions in recovery time and mortality. The FDA issued an Emergency Use Authorization. From knowledge graph query to FDA authorization in under a year, this is what everybody is hoping the AI revolution would bring to pharma and healthcare.
This was not an isolated event. Researchers have since used similar AI-driven knowledge graph approaches to identify ketamine as a repurposing candidate for substance use disorders, validated retrospectively across millions of electronic health records showing significantly higher remission rates, and are now advancing toward clinical trials. In rare disease, SHEPHERD, a knowledge-graph-based few-shot learning system published in npj Digital Medicine (June 2025), demonstrates that graph traversal can identify diagnoses for conditions where the relevant evidence is scattered across case reports no individual clinician has encountered. A scoping review in Clinical Therapeutics (2024) found 47 peer-reviewed papers applying knowledge graphs to pharmacovigilance signal detection, though performance remains variable and comparisons to legacy methods are sparse.
A pattern within the pattern is emerging: multi-source AI succeeds when it is built on structured graph traversal of biomedical relationships. TREWS integrating clinical data streams. BenevolentAI traversing drug-pathway-disease connections. SHEPHERD linking phenotype descriptions to rare disease entities.
But each of these systems was architected for one specific problem. BenevolentAI’s knowledge graph is built around drug-target-pathway-disease relationships and it cannot easily, if at all, be dropped in to perform pharmacovigilance, rare disease diagnosis, or evidence synthesis at point of care. TREWS was trained on Johns Hopkins’ patient population and integrated into their EHR workflow - it cannot transfer to a different hospital without rebuilding from scratch, as Epic’s generalization failure demonstrates empirically. SHEPHERD maps phenotype clusters to rare disease entities - it has no drug nodes, no adverse event data, no clinical guideline content. These are not commercial limitations. They are architectural ones. Each system’s graph schema, node types, edge types, and traversal logic were purpose-built for one important but narrow class of problem.
The approach works. The question is whether the approach can be applied to work broadly.
Why Multi-Source AI Is Difficult - Five Contributing Factors
There are at least five well-documented factors that independently confound the solving of the multi-source problem, and each deserves honest weight.
Data fragmentation. Anyone who has worked in a hospital knows that clinical data lives across numerous unintegrated electronic systems, e.g. EHR, LIMS, pharmacy, imaging, scheduling, billing, etc. Industry analyses consistently report that large health systems operate hundreds of discrete software applications, with clinicians frequently switching between systems during a single shift. AI cannot reason across data it cannot access, and FHIR adoption remains incomplete.
Ground-truth labeling. Single-source outputs have crisp labels: the tumor is there or it is not. Multi-source outcomes involve counterfactual reasoning - did this treatment plan work? Would the patient have improved without intervention? Clean training labels for these immensely complex questions do not exist at scale and may never.
Regulatory pathway constraints. Over 94% of FDA AI device clearances go through the 510(k) pathway, which requires demonstrating substantial equivalence to a predicate device of a low- to mid-risk profile. This structurally favors narrow claims on less challenging indications. Of 691 devices analyzed through 2023, fewer than 1% were supported by RCT data, and fewer than 1% reported actual patient outcomes (Lin et al., JAMA Health Forum, 2025).
Alert fatigue. Even correct multi-source outputs face workflow integration challenges. Epic’s 12% PPV means 8 false alerts per true case. Clinicians were right to ignore it.
Liability. Multi-source clinical decision support implicitly claims a treatment recommendation, and no clinician will accept legal responsibility for following an AI recommendation they cannot explain to a malpractice attorney.
Each of these is real and important. But none of them explains why AI performs so poorly on multi-hop biomedical reasoning even when structured data is available and real-world confounds are removed.
The BioHopR benchmark (Kim, Abdulle & Wu, ACL Findings 2025), built on the PrimeKG biomedical knowledge graph, tested state-of-the-art LLMs on multi-hop biomedical questions where the knowledge graph structure was explicitly provided. The best model, OpenAI’s o3-mini, achieved 37.93% accuracy on 1-hop and 14.57% on 2-hop. GPT-4o and specialized biomedical models performed worse.
An important nuance: BioHopR tests the LLM’s reasoning over structured facts it has been provided, not its ability to find those facts. So we see the multi-hop biomedical problem has two layers: a retrieval failure (not finding and connecting the right evidence) and a reasoning failure (the LLM struggling to synthesize evidence it receives). Solving retrieval without solving reasoning will not be sufficient. But solving reasoning without solving retrieval is equally futile. After all, a model cannot reason over evidence it never has access to.
The Harvard Medical School KGARevion study (ICLR 2025) articulated the core challenge directly: LLMs with standard RAG (Retrieval Augmented Generation) “lack multi-source and grounded knowledge necessary for medical reasoning” and face challenges in “simultaneous consideration of dependencies across multiple medical concepts.” A study published in Nature Communications found that while current models exceed 85% accuracy on diagnostic tasks, treatment planning accuracy drops to approximately 30% (Qiu et al., Nature Communications, 2025). The models fail exactly at the stage where multi-source reasoning is required.
Retrieval architecture is not the only factor. But it is a rate-limiting step for the class of problems where knowledge exists and is not being connected.
The Scale of the Problem
This retrieval challenge operates against a backdrop that makes it dramatically harder: the sheer volume of biomedical knowledge being produced has permanently exceeded human synthesis capacity.
PubMed contains over 40 million citations. Approximately 1.57 million new citations were added in fiscal year 2023 alone. A study in the Journal of the Medical Library Association estimated that a primary care physician would need 29 hours per working day (not a typo) to stay current with the literature relevant to their specialty (Alper et al., 2004). That estimate was calculated when PubMed was roughly half its current size.
Physicians are very aware of this. A 2022 Doximity survey of 600 physicians found that 7 in 10 felt overwhelmed by new medical literature, even while 95% wanted to stay current. An analysis of clinical research utilization found that only 0.4% to 2.4% of published clinical research is ever cited in UpToDate - the dominant point-of-care reference - across 18 medical specialties (Abdalla et al., eLife, 2023). The other 97.6-99.6% of published clinical evidence is effectively invisible to the practicing clinician. An immense waste and loss for humanity.
The gold standard for synthesizing this body of evidence, i.e. the systematic review, takes an average of 67 weeks to produce (Borah et al., BMJ Open, 2017). Cochrane reviews show a median time from protocol to publication of 1.63 years. Basic research takes an average of 17 years to reach clinical guidelines (Morris et al., Journal of the Royal Society of Medicine, 2011). And a survival analysis of 100 systematic reviews found that the median time before new evidence triggered a signal for updating was just 5.5 years, with 23% showing such a signal within two years of publication (Shojania et al., Annals of Internal Medicine, 2007).
The knowledge exists, is growing and compounding. However, the capacity to synthesize it at the speed and scale required to make full use of it is significantly lagging behind.
Could Something Other Than Multi-Hop Retrieval Solve This?
Multi-hop graph retrieval is one proposed solution. There are alternative approaches to bridge this gap that are under active development, each with genuine promise.
Bigger context windows. If LLM context windows grow large enough and attention degradation is solved, you could theoretically stuff hundreds of papers into context and let the model find the connections internally. Context windows have grown dramatically and rapidly - Meta’s Llama 4 Scout advertises 10 million tokens, Gemini 1.5 Pro and Grok 4 support 2 million, Claude Sonnet 4 and GPT-4.1 support 1 million.
But context window size and effective reasoning capacity are not the same thing. The RULER benchmark (Hsieh et al., NVIDIA, COLM 2024) evaluated 17 long-context models and found that only half maintained satisfactory performance at just 32K tokens, far below the 128K-10M token lengths these models advertise. The “lost in the middle” phenomenon (Liu et al., TACL 2024) persists across every frontier model tested. BABILong (Kuratov et al., NeurIPS 2024) found that on multi-hop chains, LLMs effectively use only 10-20% of their input context. And a single 2-million-token inference on Gemini 2.5 Pro costs approximately $5 in input alone.
Agentic retrieval. LLM-driven research agents decompose complex questions into sub-queries and retrieve evidence iteratively. PRISM (arXiv 2510.14278), the strongest agentic system on MuSiQue, achieves F1 = 41.8 with GPT-4o - meaningful but well below graph-based systems. AgenticRAGTracer (You et al., 2026) found that even GPT-5 achieves only 22.6% exact match on 4-hop inference, with reasoning chains that either collapse prematurely or wander into over-extension. The HippoRAG paper (NeurIPS 2024) found that single-pass graph retrieval was 10-20× cheaper and 6-13× faster than iterative agentic approaches while achieving comparable or better accuracy.
The deeper issue: an agent can only follow chains its LLM thinks to search for. Graph traversal surfaces connections through associative pathways regardless of whether the question names them - the “unknown unknowns” discovery that agentic approaches cannot currently replicate.
Continuously fine-tuned domain models. Train the model on the entire biomedical literature so it “knows” the connections parametrically. Med-PaLM 2 achieves expert-level USMLE scores. OpenBioLLM-70B averages 86.06 across medical benchmarks.
But on genuine multi-hop biomedical reasoning (BioHopR), the results are striking: HuatuoGPT-o1-70B, a specialized biomedical reasoning model, achieves near-zero precision on 2-hop tasks (0.00%), while the general-purpose Llama-3.3-70B outperforms it substantially. Catastrophic forgetting remains unsolved at PubMed scale, parametric models cannot provide citation provenance, and a 2026 systematic review of 50 studies found that RAG significantly reduces hallucination rates compared to parametric-only approaches (Jubair, Authorea preprint, 2026).
Pre-computed synthesis. Run automated evidence synthesis offline. AI-assisted screening tools can substantially reduce literature screening time, and Cochrane’s Living Systematic Review initiative continuously updates reviews as new evidence arrives. But pre-computed synthesis can only answer questions that have been anticipated. Novel questions arising from individual patient circumstances cannot be pre-synthesized.
Retrieve broadly, reason deeply. Retrieve the top 500 passages via dense search, rerank to the top 50, feed those to a powerful reasoning model. MedRAG (Xiong et al., 2024) demonstrates up to 18% improvement on medical QA. MEGA-RAG achieves F1 = 0.79 on a public health QA task with a 40% reduction in hallucination rates.
But the probabilistic ceiling is difficult to dismiss. If single-hop recall is 80%, the probability of retrieving ALL four passages needed for a 4-hop chain is 0.80⁴ = 41%. At 70% recall, which is more realistic for biomedical text, 4-hop coverage drops to 24%. No amount of reranking can overcome a passage that was never retrieved.
The convergent pattern. The leading multi-hop retrieval systems published in 2024-2026, e.g. HippoRAG 2, PropRAG, StepChain GraphRAG, and GeAR (ACL Findings 2025), all combine graph structure with dense retrieval and selective LLM reasoning. Graph-based retrieval provides deterministic traversal, sub-second latency, full provenance, and the ability to surface unanticipated connections. The alternatives bring complementary strengths. But the research trajectory is telling: the leading systems in every category are adding graph structure, not removing it.
The Unmet Need, Defined
Healthcare does not have a single multi-hop problem. It has dozens. Evidence synthesis across 40 million papers. Rare disease diagnosis across scattered case reports. Pharmacovigilance signal detection across disparate global reporting systems. Regulatory intelligence across dozens of jurisdictions. Drug repurposing across biological pathways. Treatment planning across patient history, genomics, guidelines and literature.
Each of these is a distinct multi-hop challenge. Under the current paradigm, each requires its own purpose-built knowledge graph, its own curation pipeline, its own traversal logic, and its own multi-year engineering effort. BenevolentAI’s drug repurposing graph cannot do pharmacovigilance. TREWS cannot transfer to a different hospital. SHEPHERD cannot do evidence synthesis. The successes prove the approach works. But the architecture of each success prevents it from being applied to the next problem.
What healthcare needs is domain-agnostic multi-hop retrieval infrastructure — a system whose graph schema, ingestion pipeline, and traversal mechanics work across problems without being rebuilt from scratch for each one.
That is what we set out to build. The Donna-Alfred engine is a general-purpose multi-hop retrieval system that’s not tied to a single knowledge-category graph, institution, or disease ontology. It achieves F1 = 0.677 on MuSiQue, the hardest multi-hop QA benchmark in the field, with zero LLM calls at retrieval time and a 12.5 MB binary that runs on consumer hardware. The ingestion pipeline, the spreading activation traversal, and the context compilation make no assumptions about whether the source material is Wikipedia, regulatory filings, or peer-reviewed biomedical literature.
We have not yet proven that this performance holds on biomedical text. That validation is the immediate research priority, and the results will be published with the same methodology and transparency as everything in this series.
But the need is clear. Forty million papers. Dozens of unsolved multi-hop problems. Clinicians who cannot stay current. Patients waiting years for diagnoses hidden in literature no individual physician has read. The approach works and the successes prove it. What healthcare still lacks is the infrastructure to apply it broadly. That is what we are building.
James W. Kim is the founder of EluciDx Corp. and the architect of the Donna-Alfred retrieval engine. More writing at elucidx.ca/insights.