Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering
arXiv:2606.22419v1 [cs.CL] 21 Jun 2026
Madhulatha Mandarapu∗
Sandeep Kunkunuru†
Abstract A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benchmarks, and that retrieval can hurt strong models. We ask the natural follow-up: does structured knowledge-graph (KG) grounding change this, and when does grounding help at all? We contribute two results. First, a reproduction: the study’s headline HealthBench score (∼88) is the Consensus variant, not full HealthBench, where frontier models and ideal completions both score ∼46–47 under a physician-calibrated grader (agreement 82.5%); we reproduce GPT-5.2 Consensus = 90.9 and flag a score-deflating grader bug. Second, a knowledge-boundary result. Using a graph+vector engine (samyama-graph) over the public biomedical KG PrimeKG, neither naive triple retrieval nor an agentic natural-language-to-Cypher loop (82% successful queries) improves MedQA across a weak→strong model ladder (all |∆| ≤ 3.4). On a synthetic counterfactual KG, and on a hybrid benchmark mixing known and novel facts, the identical pipeline lifts out-of-training accuracy from chance to ∼100% (+68 to +79) while adding nothing on known facts (a no-LLM arm answers both). Across three regimes (no-knowledge, graph-aided, hybrid), grounding helps only insofar as the decisive fact lies outside the model’s training—public-KG facts are redundant, private and novel data are where it pays—matching the study’s institutional-data caveat.
1
Introduction
Vishwanath et al. [2026] report that three frontier LLMs outperform two deployed clinical retrievalaugmented (RAG) tools on medical-knowledge (MedQA), clinician-alignment (HealthBench), and real-clinical-query benchmarks, with the clinical tools no better than a search engine’s AI overview. They also note, consistent with prior work [Wu et al., 2024], that retrieval can degrade a strong base model. A natural question for the knowledge-graph community follows: does replacing flat document retrieval with structured graph grounding change the verdict, and—more fundamentally—under what conditions does grounding help a capable LLM at all? We study this with a graph+vector database (samyama-graph) over the public biomedical knowledge graph PrimeKG [Chandak et al., 2023], evaluating two grounding methods against an ungrounded baseline across a ladder of base models. Our findings are negative on public data and sharply positive on out-of-training data, and together they yield a simple, falsifiable account of when grounding helps. We deliberately foreground reproduction and honest negatives: tagging a high-profile clinical result requires that every claim survive scrutiny. ∗ †
[email protected] [email protected]
1
Contributions. 1. Reproduction & grader calibration (§4). We show the Vishwanath et al. [2026] HealthBench headline (∼88) corresponds to the Consensus variant, not full HealthBench (oss_eval), where frontier answers and the dataset’s own ideal completions both score ∼46–47 under a grader calibrated to physician labels (82.5% agreement). We reproduce GPT-5.2 Consensus = 90.9 and MedQA ≈ 90 (vs. reported 94.2; model-version drift), and report a grader-parser bug that deflates scores if uncorrected. 2. A knowledge-boundary law for grounding (§6). On PrimeKG, neither naive triple-RAG nor an agentic NLQ→Cypher loop improves MedQA at any model strength; on a synthetic counterfactual KG the same pipeline lifts chance to near-perfect; and a hybrid benchmark with both strata shows zero lift on known facts and a +68 to +78 point lift on novel facts within one experiment. Grounding value is gated by whether the decisive fact is out-of-training. 3. A three-regime framing (§3) aligning the experiments with the grounding-substrate view of Kunkunuru and Mandarapu [2026]: (i) no-knowledge / direct LLM, (ii) graph-aided (LLM queries the graph, optionally enriching it back as a chain of tasks), and (iii) hybrid; with a no-LLM deterministic arm that answers 100% on both strata, isolating the data layer from the LLM. 4. A reusable agentic GraphRAG artifact (§5) over samyama-graph (entity linking via vector search, self-correcting Cypher generation), plus two honest engine findings, and a pre-registration discipline that caught two spurious small-sample signals.
2
Background
Benchmarks. MedQA [Jin et al., 2021] is USMLE-style multiple choice. HealthBench [Arora et al., 2025] grades free-text answers against physician-written rubric items; it ships multiple subsets, including the full oss_eval set (many criteria per item, with negative-point penalties) and a Consensus subset (few, high-agreement, all-positive criteria). The scoring is a points fraction P P score = clip p / p , 0, 1 , which differs markedly between subsets. met i pi >0 i Grounding. GraphRAG retrieves subgraphs to ground generation; medical KG-RAG work typically reports MedQA, while HealthBench augmentation has used text RAG [Synduct, 2026]. We use PrimeKG [Chandak et al., 2023] (drug/disease/phenotype/gene), and the samyama-graph engine for vector search and Cypher. The agentic variant follows the knowledge-graph-as-data-layer thesis [Kunkunuru and Mandarapu, 2026]: the LLM queries a deterministic graph rather than reading retrieved text.
3
Three Regimes of Knowledge
We organize the study around three regimes of where the decisive knowledge lives (Figure 1), adapting the grounding-substrate view of Kunkunuru and Mandarapu [2026]: 1. No knowledge (direct LLM). The model answers from parametric memory alone (arm A0). 2. Graph-aided. The graph is queried before the LLM answers (NLQ→Cypher; arm A_agent) or computed without the LLM at all (deterministic handler; arm A_det), and—in the full substrate—missing facts can be written back and cached as a chain of tasks toward a goal (generation-augmented knowledge [Kunkunuru and Mandarapu, 2026]). 2
A0 — no knowledge. LLM alone. works iff fact ∈ training ∪ inferable (e.g. real drugs via INN nomenclature); else ≈chance A_text (flat RAG). source text in context. correct, but one LLM call per query; unstructured
Clinical question
A_agent (B). vector entity-link → LLM writes Cypher → execute. lift ≈0 intraining; +77 out-of-training (82% queries succeed) samyama-graph A_det (C). deterministic Cypher, no LLM. 100% whenever the fact is graphresident enrich (write) + query (read)
A_GAK (D). miss → agent extracts from a source → materialize (provenance) → cache. once per fact; novel 0 → 100
HNSW vector · OpenCypher · graph algorithms · enrich/cache
Answer
Figure 1: All grounding paths over samyama-graph in one view (arms = paper3 architectures A/B/C/D). No knowledge (A0) succeeds only when the fact is in training or inferable from surface structure (e.g. drug nomenclature); flat RAG (A_text) reads a source each query. The graph-aided paths — LLM-written Cypher (B), no-LLM deterministic (C), and generation-augmented knowledge (D: extract a novel fact from a source, tag provenance, cache, then serve via C/B) — make out-oftraining facts answerable and are redundant on in-training facts. Net: grounding helps iff the fact lies outside training ∪ inferable. 3. Hybrid. Some decisive facts are in the model’s training and in the graph (overlap, redundant) while others are only in the graph (novel, additive). This is the realistic deployment case. Our experiments instantiate all three: PrimeKG (§6) is the in-training overlap; the synthetic KG is the pure out-of-training case; and the hybrid benchmark mixes both in one experiment.
4
Reproduction and Grader Calibration
We reproduce the public components with deterministic decoding (temperature 0, seed 62) and the exact selection procedure from the released harness (random.Random(62).sample over the source order, single-user-turn filter), verified by matching the published filtered counts. MedQA. GPT-5.2 scores 90.0% on the matched 500-item subset (100% answer-parse rate), vs. the reported 94.2%. The gap is stable across two independent subsets and is unaffected by enabling web search; we attribute it to model-version drift (the original was accessed four months earlier). This is a faithful partial reproduction, not a discrepancy in method. HealthBench: which scale? On full oss_eval, GPT-5.2 scores 46.5 (macro) / 41.8 (the released pooled formula). Crucially, the dataset’s own ideal completions—the best possible answers—score only ∼47 under our grader. No model can score 88 when the ideal answer scores 47; the headline must use a different scale. The Consensus variant (few, all-positive criteria) does reach it: GPT-5.2 Consensus = 90.9 (95% CI [87.4, 94.4], N =150), bracketing the reported 88.0 (Table 1).
3
Table 1: Reproduction. The Nature headline HealthBench number is the Consensus scale, not full oss_eval. Ideal completions bound the full-set scale at ∼47. Setting HealthBench oss_eval (full) HealthBench Consensus MedQA (matched 500)
GPT-5.2
Ideal completions
Reported
46.5 / 41.8 90.9 [87.4, 94.4] 90.0
∼47 ∼82 —
— 88.0 94.2
Grader calibration. Our judge panel (gpt-4.1 + Claude Opus 4.6) agrees with physician gold labels on HealthBench’s meta-evaluation set at 82.5%, matching the original HealthBench grader’s reported agreement; thus the ∼47 ideal-completion score reflects the benchmark’s intended difficulty, not a harsh grader. We also document a parser failure mode—conflating “criteria not met” with a parse failure—that silently deflated scores until hardened (it returns true/false/none distinctly).
5
Method: grounding over samyama-graph
We load PrimeKG into samyama-graph (an OpenCypher graph+vector engine) and embed node names for HNSW vector search. We compare three arms with a fixed base model: • A0: base model, no retrieval. • A_kg (naive): vector-search the question, expand a one-hop clinical subgraph (degree-capped), format as triples. • A_agent (agentic): a fixed agent (gpt-4.1) does entity linking via vector search, writes multi-hop Cypher constrained to verbatim resolved entity names and specific relationship types, executes it on samyama-graph, and self-corrects for up to two rounds on error or empty result; the returned rows are the grounding context. • A_det (deterministic, no LLM): a pre-coded handler issues the Cypher lookup directly (no LLM in the loop) and matches the result to an option. This isolates the data layer from the LLM and mirrors Architecture C of Kunkunuru and Mandarapu [2026]. • A_GAK (generation-augmented knowledge): the realistic out-of-training loop (Architecture D of Kunkunuru and Mandarapu [2026]). The decisive fact lives in an unstructured source (not in training, not yet in the graph); on a miss, the agent extracts it and materializes provenance-tagged nodes/edges into the graph, which is then queried deterministically. Repeat questions about the same entity are cache hits (no LLM). Scope of engine capabilities used. A_agent/A_det exercise the engine’s vector search and Cypher; we do not use its graph-algorithm (PageRank, paths) or optimization (NSGA-II) primitives, because clinical multiple-choice is lookup-shaped. A different question class (criticality, interaction paths, treatment scheduling) would route to those Tier-2 primitives; we leave that to future work. Fixing the query-writer isolates whether the answering model benefits from grounding rather than its Cypher skill. The agent reaches an 82% non-empty query rate on MedQA after vector entity-linking and few-shot direction examples (from 0% when the LLM invented entity names). Engine findings (honest). Cypher expand lacks LIMIT-pushdown on high-degree nodes, so an unbounded one-hop expansion over hub entities materializes millions of rows; we work around it with a precomputed degree-capped adjacency. type(r) over multi-type or optional relationships errors 4
Table 2: Public KG (PrimeKG): MedQA accuracy. Naive and agentic grounding give no significant lift at any model strength (CIs ±4–7). Base model
A0
A_kg
A_agent
∆naive
∆agent
gpt-4.1-nano gpt-4o-mini gpt-4.1 gpt-5.2
71.3 82.7 89.3 92.7
69.3 79.3 90.0 91.3
72.7 82.0 90.0 92.0
−2.0 −3.4 +0.7 −1.4
+1.4 −0.7 +0.7 −0.7
Table 3: Out-of-training KG (synthetic counterfactual): the same agentic samyama-graph grounding turns chance into near-perfect at every model strength (N =150). Base model
A0 (alone)
A_agent (graph)
∆agent
gpt-4.1-nano gpt-4o-mini gpt-4.1 gpt-5.2
23.3 22.0 23.3 21.3
98.7 100.0 100.0 100.0
+75.4 +78.0 +76.7 +78.7
(“type() requires an edge”); and a comma-separated CREATE that declares nodes then references their variables for edges materializes nodes but not edges (so GAK issues node-creates then edge-creates via MATCH). All three are concrete, fixable engine gaps surfaced by this work.
6
Experiments
Setup. MedQA exact-match, N =150–200, base-model ladder gpt-4.1-nano → gpt-4o-mini → gpt-4.1 → gpt-5.2 (weak→strong). Per-question retrieval is computed once and reused across base models; intervals are 95% Wilson/normal. Public KG: no lift. Table 2 shows neither naive nor agentic grounding moves MedQA accuracy at any model strength; every lift is within ±3.4 points and inside the confidence intervals, despite the agent issuing correct queries 82% of the time (it does return correct facts, e.g. retrieving first-line antibiotics for a diphtheria vignette). A pilot at N =8 showed a clean monotone “crossover” (+25 → −12.5); it vanished at N =200—a reminder that small-N grounding signals are noise, which our pre-registration was designed to catch. Out-of-training KG: decisive. We build a synthetic counterfactual clinical KG (fictional drugs, diseases, phenotypes; seed 62) whose facts cannot be in any model’s training, and pose KG-answerable multiple-choice questions. Table 3: A0 sits at chance (∼22%, four options), while the identical agentic pipeline reaches ∼100% (+75 to +79, 100% query success), uniformly across the ladder—because all models equally lack the knowledge. Hybrid: the boundary within one benchmark. The cleanest test holds everything else fixed and varies only fact-novelty across items of one benchmark. We build a hybrid KG with a known stratum (24 canonical textbook disease→drug facts the models reliably know) and a novel stratum (80 synthetic counterfactual facts), and load both into one graph. Table 4: on the known stratum A0 is already 100% and grounding adds nothing (+0); on the novel stratum A0 is at chance (∼22–32%) and agentic grounding lifts it to 100% (+68 to +78). The no-LLM A_det handler answers 100% on 5
Table 4: Hybrid benchmark (one experiment, two strata). Lift is concentrated entirely on novel (out-of-training) facts; A_det (no LLM) is 100% on both. Known stratum (in-training)
Novel stratum (out-of-training)
Base model
A0
A_agent
A0
A_agent
gpt-4.1-nano gpt-4o-mini gpt-4.1 gpt-5.2
100 100 100 100
100 100 100 100
22.5 25.0 32.5 31.2
100 100 100 100
A_det (no LLM)
100
100
Figure 2: Across all settings: KG grounding (A_agent) matches A0 on in-training knowledge (lift ≈ 0) and is decisive on out-of-training knowledge (lift +72 to +77), averaged over the gpt-4.1nano→gpt-5.2 ladder. both strata—whenever the fact is graph-resident, the data layer alone suffices. Figure 2 summarizes all settings: grounding is flat in-training and decisive out-of-training. GAK: the mechanism that puts novel facts in the graph. The synthetic and hybrid experiments load novel facts into the graph by construction; the realistic loop must get them there. We test generation-augmented knowledge (A_GAK) on 40 novel facts (80 questions) whose answer lives only in an unstructured source monograph. Starting from an empty graph: A0 (no source) is at chance (20%); flat text-RAG (source in context) and A_GAK both reach 100% (Table 5). The difference is economics and form: text-RAG pays an LLM call per query (80), whereas A_GAK extracts each fact once (40 enrichment calls), materializes it provenance-tagged (all 160 nodes source:enriched), and serves the remaining 40 questions as deterministic cache hits (no LLM)— asymptotically zero marginal cost, queryable, and auditable. This closes the loop: novel knowledge enters the graph via the agent, then behaves like the in-graph facts of §6. Real out-of-training facts, and a refinement: “out-of-training” ̸= “unknown”. We test 13 real 2026 FDA novel-drug approvals (verified; post-dating most ladder models), asking 6
Table 5: A_GAK (Architecture D): facts live only in unstructured sources, absent from training and the graph. Accuracy parity with text-RAG, but the LLM is paid once per fact (then cached), with provenance. Arm A0 (no source) A_text (flat RAG) A_GAK (enrich+cache)
Accuracy
LLM calls (80 q)
Cache hits
Provenance-tagged nodes
20.0 100.0 100.0
80 80 40
— — 40
— — 160 (source:enriched)
“which drug is indicated for condition X?” with other recent drugs as distractors. Surprisingly, A0 = 84.6%—the models infer most indications from International Nonproprietary Name (INN) nomenclature (-drostat→aldosterone-synthase→hypertension; -penem→carbapenem→UTI; -trelvir→antiviral→COVID; pegzil-arginase→arginase deficiency; copper→Menkes). A postcutoff drug is therefore not automatically out of reach: its name carries class information. The clean signal is on the three approvals whose names do not encode the indication—Hunter syndrome (tividenofusp alfa), achondroplasia (navepegritide), blastic plasmacytoid dendritic cell neoplasm (pivekimab sunirine): there A0 = 0/3 while A_agent and A_det are 3/3. This refines the law: grounding helps to the extent the fact is neither in training nor reconstructable from surface structure (boundary = training ∪ inferable). The synthetic KG (random names, no stems) is the pure isolation; real nomenclature is itself latent knowledge, and grounding (deterministically, via A_det) still guarantees the uninferable residual. Robustness. Beyond the within-run 95% CIs, the out-of-training effect is stable across KGgeneration seeds: the synthetic lift is 76.6 ± 2.9 over 3 seeds × 4 models (range [72, 82]), and the hybrid novel-stratum lift is ∼77 ± 8. The public-KG null is stable across two independent full runs (all |∆| ≤ 5.5 both times). Generation is deterministic (temperature 0, seed 62); the headline conclusions do not hinge on a single run.
7
Discussion: a knowledge-boundary law
The two regimes differ only in the novelty of the grounding facts—same engine, same agentic pipeline, same models—and that alone flips the result from null to decisive. We state the account plainly: structured grounding raises accuracy only to the extent it supplies information the model’s parametric prior lacks. Public-KG facts (PrimeKG is in the training corpora) are redundant; even when the graph returns a correct fact, the model already knew it. Out-of-training facts are not, so grounding is decisive. This is consistent with the prior-versus-evidence view of retrieval [Wu et al., 2024] and explains why public-benchmark evaluations understate the value of grounding for the deployment that matters: private and institutional data. It operationalizes Vishwanath et al. [2026]’s own observation that the path forward lies with hospital-specific data, and the knowledge-graph-as-data-layer thesis of Kunkunuru and Mandarapu [2026].
8
Related Work
Medical KG-RAG systems report MedQA gains but rarely isolate the knowledge boundary; HealthBench augmentation has used text RAG [Synduct, 2026]; MIRAGE/MedRAG [Xiong et al., 2024] benchmarks medical RAG. Agentic and budgeted graph retrieval [He et al., 2024, Anonymous, 7
2025] provide the machinery we adopt rather than claim. ClashEval [Wu et al., 2024] frames the prior-versus-evidence tension we explain via novelty. Health-system-scale models [Jiang et al., 2023] and the source study [Vishwanath et al., 2026] motivate the institutional-data direction our synthetic result makes precise.
9
Limitations and Honest Negatives
The headline of our grounding study is a negative on public data; we report it because the agentic pipeline was built well (82% query success, verifiably correct facts) and still did not help. The synthetic positive uses fictional entities; it cleanly isolates the mechanism but is not a clinical benchmark—demonstrating the predicted lift on a real private clinical graph is future work. The crossover predicted by a prior-strength account is not visible here because both regimes are endpoints (prior fully covers, or is absent); an intermediate, graded-prior task would test it. Query generation uses a fixed capable agent (gpt-4.1); a weak model writing its own Cypher would conflate querying and answering skill. Grader and reproduction caveats are in §4. MedQA exact-match ignores reasoning quality; clinical tools in the source study lack APIs and are cited, not re-run.
10
Conclusion
Structured KG grounding does not beat frontier LLMs on public medical benchmarks—not because grounding is weak, but because public-KG facts are already in the models. The same agentic samyama-graph pipeline is decisive when the knowledge is out-of-training. The actionable claim for practitioners is to ground on data the model has not seen—private, recent, institutional—and to evaluate there, since public benchmarks understate grounding’s value. Along the way we corrected the scale of a widely-cited HealthBench result and surfaced two fixable engine gaps. Code and one-command reproduction are open. Reproducibility. All numbers regenerate from the released harness (https://github.com/ samyama-ai/clinical-llm-graphrag): deterministic decoding (seed 62), committed subset hashes, the synthetic KG generator (seed 62), and the agentic pipeline. Public data: MedQA, HealthBench (CC-BY-4.0), PrimeKG (MIT).
References Anonymous. Agrag: Advanced graph-based retrieval-augmented generation, 2025. Rahul K. Arora et al. Healthbench: Evaluating large language models towards improved human health, 2025. Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67, 2023. Xiaoxin He et al. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering, 2024. Lavender Y. Jiang et al. Health system-scale language models are all-purpose prediction engines. Nature, 619:357–362, 2023. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
8
Sandeep Kunkunuru and Madhulatha Mandarapu. Knowledge graphs as the missing data layer for llm-based industrial asset operations. arXiv preprint arXiv:2605.26874, 2026. Synduct. Dr.info: An agentic retrieval system for health question answering, 2026. Krithik Vishwanath, Anton Alyakin, Mrigayu Ghosh, others, Daniel A. Alber, Yindalon Aphinyanaphongs, and Eric Karl Oermann. General-purpose large language models outperform specialized clinical ai tools on medical benchmarks. Nature Medicine, 2026. doi: 10.1038/s41591-026-04431-5. Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence. In Advances in Neural Information Processing Systems, 2024. Guangzhi Xiong et al. Benchmarking retrieval-augmented generation for medicine (mirage/medrag), 2024.
9