Agentic clinical reasoning over longitudinal myeloma records:
arXiv:2604.24473v1 [cs.AI] 27 Apr 2026
a retrospective evaluation against expert consensus Johannes Moll∗,†1,2,3 , Jannik Lübberstedt†2 , Christoph Nuernbergk4 , Jacob Stroh4 , Luisa Mertens4 , Anna Purcarea4 , Christopher Zirn2 , Zeineb Benchaaben2 , Fabian Drexel1,2 , Hartmut Häntze2,5 , Anirudh Narayanan2,5 , Friedrich Puttkammer2,5 , Andrei Zhukov6 , Jacqueline Lammert7,8 , Sebastian Ziegelmayer2 , Markus Graf2 , Marion Högner4 , Marcus Makowski2 , Florian Bassermann4,9,10,11 , Lisa C. Adams2 , Jiazhen Pan1,12,13 , Daniel Rueckert1,13,14 , Krischan Braitsch‡4 , and Keno K. Bressem‡2,3 1 Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital,
Munich, Germany 2 Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, School of
Medicine and Health, Technical University of Munich, Munich, Germany 3 Department of Cardiovascular Radiology and Nuclear Medicine, German Heart Center, TUM University Hospital,
School of Medicine and Health, Technical University of Munich, Munich, Germany 4 Department of Medicine III, Klinikum rechts der Isar, TUM University Hospital, School of Medicine and Health,
Technical University of Munich, Munich, Germany 5 Department of Radiology, Charité – Universitätsmedizin Berlin, Berlin, Germany 6 Department of Gastroenterology, Infectious Diseases and Rheumatology, Charité – Universitätsmedizin Berlin, Berlin,
Germany 7 Chair of Medical Informatics, Institute of AI in Medicine and Healthcare, TUM School of Medicine and Health,
Technical University of Munich, Munich, Germany 8 Clinical Department of Gynecology, TUM University Hospital, TUM School of Medicine and Health, Munich,
Germany 9 TranslaTUM, Center for Translational Cancer Research, Technical University of Munich, Munich, Germany 10 Deutsches Konsortium für Translationale Krebsforschung, Heidelberg, Germany 11 Bavarian Cancer Research Center, Munich, Germany 12 Department of Engineering Science, University of Oxford, Oxford, UK 13 Munich Center for Machine Learning (MCML), Munich, Germany 14 Department of Computing, Imperial College London, London, UK
† These authors contributed equally to this work. ‡ These authors share senior authorship. ∗ Corresponding author: [email protected]
1
Abstract Background. Multiple myeloma is managed through sequential lines of therapy over years to decades, with each treatment decision depending on cumulative disease history distributed across dozens to hundreds of heterogeneous clinical documents. Whether large language model based systems can synthesise this evidence at a level approaching expert agreement has not been established. Methods. A retrospective evaluation was conducted on longitudinal clinical records of 811 patients with multiple myeloma treated at a tertiary medical centre between 2001 and 2026, covering 44,962 documents and 1,334,677 laboratory values, with external validation on MIMIC-IV. An agentic reasoning system was compared against single-pass retrieval-augmented generation (RAG), iterative RAG, and full-context input on 469 patient–question pairs derived from 48 templates stratified into three complexity levels. The reference standard was established by independent double annotation from four oncologists with adjudication by a senior haematologist. Findings. Iterative retrieval-augmented generation and full-context input converged on a shared performance ceiling (75·4% versus 75·8%, Bonferroni-corrected 𝑝 = 1·00). The agentic system reached 79·6% concordance (95% CI 76·4–82·8), significantly exceeding both baselines (+3·8 and +4·2 percentage points; 𝑝 = 0·006 and 0·007). Gains increased with question complexity, reaching +9·4 percentage points on criteria-based synthesis (𝑝 = 0·032), and with record length, reaching +13·5 percentage points in the top decile (exploratory, 𝑛 = 10). The system error rate (12·2%) was comparable to expert disagreement (13·6%), but severity distributions were inverted, with 57·8% of system errors classified as clinically significant against 18·8% of expert disagreements. Interpretation. Agentic reasoning was the only approach to exceed the shared performance ceiling, with gains concentrated on the most complex questions and longest records. The greater clinical consequence of residual system errors relative to expert disagreement indicates that prospective evaluation in routine care will be required before these findings translate into measurable patient benefit. Funding. Bayern Innovativ (Bavarian State Ministry of Economics), Grant Number: LSM-2403-0006.
Introduction Multiple myeloma is managed through sequential lines of therapy over years to decades, with each treatment decision depending on a cumulative record of prior exposures, documented responses, and evolving comorbidities that no single source can fully reflect.1 A typical patient accumulates dozens to hundreds of clinical documents over this trajectory, spanning laboratory data, pathology findings, imaging reports, and free-text notes, and the correct interpretation of any individual finding depends on temporal relationships that may span years.2 Determining whether a patient has progressed on a given regimen, whether prior toxicities preclude a planned therapy, or whether organ function meets eligibility criteria requires synthesis across document types and timepoints that no individual document can resolve. These demands fall on haematologists who already carry among the highest electronic health record burdens of any clinical specialty, with recent data showing more than 575 minutes per week in the record and over 219 minutes of after-hours documentation,3 a burden that scales with the complexity of the disease trajectory. Large language models (LLMs) have been applied to clinical record navigation across a range of tasks.4 Published approaches fall into three broad strategies: single-pass retrieval-augmented generation (RAG), iterative
2
RAG, and agentic systems that reason over specialised tools.5–7 Earlier work on guideline interpretation, domain-tuned question answering, and factual verification showed that retrieval can reduce hallucination on well-defined single-document tasks.8–11 More recently, Myers and colleagues tested longitudinal reasoning directly, comparing RAG against full-record long-context input on three tasks over hospitalised patients, finding equivalent performance at 128,000 tokens of context capacity and concluding that the absence of expert-adjudicated longitudinal datasets had prevented any test of whether this convergence holds for the harder multi-source synthesis tasks that dominate real clinical practice.12 Agentic approaches have been evaluated primarily in two settings: diagnostic dialogue agents operating on simulated patients,13, 14 and tool-using agents tested against structured databases where each task can be resolved from a small number of entries.15–17 Neither setting captures the demand of answering a treatment planning question from multiple instances of unstructured, partially contradictory longitudinal documentation. No evaluation of any such approach has been reported for multiple myeloma or any comparable disease trajectory, and whether clinically reliable performance is achievable against an independently annotated reference standard has not been established. Here, a retrospective evaluation of expert concordance was conducted on institutional records of 811 patients with multiple myeloma treated at a tertiary medical centre over 25 years. An agentic reasoning system was compared against single-pass RAG, iterative RAG, and full-context input, all using the same locally deployed open-weight LLM to keep data within institutional infrastructure. The reference standard was established through independent double annotation by four oncologists with adjudication by a senior haematologist, residual inter-rater variability was explicitly classified, and the clinical safety profile was characterised through structured error severity analysis. External validity was assessed on the MIMIC-IV critical care database.18, 19
Methods The study was approved by the Ethics Committee of the Technical University of Munich (approval 2024590-S-CB), and informed consent was waived given the retrospective design. Use of MIMIC-IV data was conducted under the PhysioNet credentialed data use agreement.20 All procedures followed the Declaration of Helsinki and applicable institutional guidelines. Study design and data sources
A retrospective evaluation of expert concordance was conducted on longitudinal multiple myeloma records from two independent institutions (Figure 1a). The primary dataset comprised all patients with a confirmed multiple myeloma diagnosis treated at TUM University Hospital between 1 January 2001 and 1 January 2026 (n=811). Textual reports were extracted from 44,962 German-language clinical documents, yielding a median of 55 documents per patient (IQR 20 to 76, Figure 1b) and a median follow-up of 6·3 years from diagnosis (IQR 1·7 to 9·7, Figure 1c), with details of the extraction and conversion pipeline provided
3
in Supplementary Methods A.1. Structured laboratory data (1,334,677 values) were extracted from the institutional laboratory information system and normalised to 731 canonical concepts, preprocessing details are provided in Supplementary Methods A.2. The external validation dataset comprised 716 patients identified by ICD code from the publicly available de-identified MIMIC-IV critical care database,19 yielding 26,767 clinical documents with a median of 26 documents per patient (IQR 11 to 53). This cohort was used without adaptation to assess transferability across institutions. Clinical question bank and evaluation cohorts
A bank of 48 clinical question templates covering core decision tasks in multiple myeloma management was developed in consultation with staff haematologists and reviewed for clinical representativeness. Templates were stratified into three complexity levels: single-record lookup (Level 1), temporal reasoning across multiple sources (Level 2), and criteria-based synthesis across document types and multiple timepoints (Level 3). The full template list with answer formats and scoring methods is provided in Supplementary Table B.2. Three non-overlapping patient sets were defined prior to system development (Figure 1d, Figure 2). A development set of ten patients was used exclusively for iterative system development and is excluded from all reported evaluations. For the primary analysis, 100 patients were sampled from the remaining TUM cohort using stratified sampling over number of reports and recency of the last available data point, with each patient assigned five questions (two Level 1, two Level 2, one Level 3), yielding 500 patient-question pairs. For external validation, 20 patients were selected from the MIMIC-IV cohort using the same procedure, yielding 100 pairs. Expert annotation and adjudication
All pairs were annotated by independent chart review prior to system evaluation, with raters blinded to each other and to all system outputs. Each TUM patient was reviewed by exactly two of four oncologists with multiple myeloma expertise (AP, CN, JS, LM). Pairs with direct agreement were included without adjudication, disagreements were reviewed by a senior haematologist (KB) and classified into one of five categories. Full protocol details and operational definitions are provided in Supplementary Methods. For the TUM cohort, direct agreement was reached on 65·2% of pairs, a further 28·6% were included after adjudication, and 6·2% were excluded, yielding 469 evaluable pairs (200 Level 1, 179 Level 2, 90 Level 3; Figure 1e). Pre-adjudication inter-rater agreement declined with complexity: 𝜅 = 0·69 at Level 1, 𝜅 = 0·60 at Level 2, and 𝜅 = 0·57 at Level 3 (Figure 1f). Among adjudicated cases, the most frequent categories were interchangeable or equivalent responses (39·2%) and clinically insignificant disagreement (36·4%), with clinically significant disagreement accounting for 8·4% (Figure 1g). For the MIMIC-IV cohort, 89 evaluable pairs were retained after the same procedure. Agentic system and comparators
The agentic system was designed to answer clinical questions requiring synthesis across temporally distributed, heterogeneous documentation. The system was distinguished from retrieval-based approaches by four
4
architectural components (Figure 3). Task-relevant modules were selected from an indexed clinical skill library encoding question-type-specific reasoning protocols. An ordered tool-use plan with explicit stopping conditions was constructed prior to retrieval. A structured memory state encoding the user query, retrieved evidence, missing information, and stopping conditions was updated iteratively after each step. Iterative execution was carried out against purpose-built tools, including report and laboratory value retrieval with type and date filters and deterministic clinical scoring calculators, with automatic re-query on insufficient evidence. Final answers followed a schema-defined format with inline source citations. Full implementation details are provided in Supplementary Methods. The same locally deployed 120-billion-parameter open-weight language model (gpt-oss-120b21 ) was used across all three comparator approaches and the same patient record database. In Simple RAG, single-pass dense retrieval was implemented without query rewriting or reranking.22 Iterative RAG was extended to include subquery rewriting, hybrid BM25 and dense retrieval fusion, cross-encoder reranking, and a multi-round sufficiency loop.5, 23 In the Full Context configuration, retrieval was bypassed entirely, all documents and laboratory values were concatenated in reverse chronological order until the context window was filled.12 Two additional comparators are reported in Supplementary Table B.3, and detailed specifications for all approaches are provided in Supplementary Methods. Error classification and citation sufficiency
All patient-question pairs for which the agentic system diverged from expert consensus were classified by a senior haematologist (KB) into one of six categories: clinically significant error, clinically insignificant error, partially correct, acceptable or ambiguous, annotation error, and pipeline failure. The error classification taxonomy was aligned with the adjudication categories applied during expert annotation, enabling direct comparison of system error rates with expert disagreement rates at equivalent severity levels. To assess the reproducibility of this classification, a blinded inter-rater reliability sub-study was conducted by an independent oncologist (JS) on a proportional stratified random sample of 46 of the 115 divergent patient-question pairs, with categories additionally collapsed into three severity strata for analysis. Citation sufficiency was assessed by two reviewers (KB, CN) on a stratified sample of 96 responses, with each of the 48 question templates represented at least once. Full classification criteria, sampling details, and inter-rater reliability protocol are provided in Supplementary Methods. Statistical analysis
The primary outcome was concordance with expert consensus, defined as the proportion of pairs for which system output matched the adjudicated reference on substantive content. Single-value categorical items were scored as binary, whereas list-type items were scored using entry-level F1 computed against the reference list, contributing a continuous value between zero and one. Secondary outcomes were the clinically significant error rate derived from structured error classification and citation sufficiency. Each system was evaluated across ten independent runs to control for stochastic variability, and per-question scores were averaged across runs before analysis. Individual concordance estimates and their 95% confidence intervals were computed by pair-level percentile bootstrap (𝑁boot = 10,000). Pairwise significance tests were computed 5
via cluster bootstrap (𝑁boot = 10,000, 𝛼 = 0·05), resampling whole patients with replacement to account for within-patient correlation. Bonferroni correction was applied within each stratum independently, and all reported 𝑝-values are Bonferroni-corrected unless explicitly stated otherwise. The relationship between patient record length and system performance was examined as a hypothesis-driven exploratory analysis, with patients stratified into four quantile-based bins prior to analysis (Supplementary Table B.9). Role of the funding source
The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.
Results Accuracy against expert consensus
The primary evaluation comprised 469 patient-question pairs across 100 patients from the TUM cohort, including 200 Level 1, 179 Level 2, and 90 Level 3 pairs retained after adjudication. Iterative RAG and Full Context performed equivalently, reaching 75·4% (95% CI [71·9–78·6]) and 75·8% ([72·3–79·3]) concordance with expert consensus respectively, with no statistically significant difference between them (−0·4 percentage points, Bonferroni-corrected 𝑝 = 1·00). Simple RAG trailed at 71·5% ([67·8–75·1]). The agentic system achieved an overall concordance of 79·6% ([76·4–82·8]) (Figure 4a, Supplementary Table B.3), significantly higher than both Full Context (+3·8 percentage points, 𝑝 = 0·006) and Iterative RAG (+4·2 percentage points, 𝑝 = 0·007). Across ten independent evaluation runs, the agentic system showed a standard deviation of 1·1 percentage points (Supplementary Table B.4). The skill library was identified by ablation analyses as the principal driver of performance. Its removal reduced overall concordance by 3·0 percentage points to 76·6%, whereas removal of type and date filters in retrieval tools, deterministic clinical scoring tools, structured memory state, or pre-planned tool use individually reduced concordance by at most 0·4 percentage points (Supplementary Table B.5). External validation on 89 evaluable patient-question pairs across 20 patients from the MIMIC-IV cohort preserved the system ranking (Figure 4e). The agentic system achieved an overall concordance of 84·9% ([77·8–91·2]), compared with 79·2% ([71·4–86·3]) for Simple RAG, 77·9% ([70·1–85·3]) for Iterative RAG, and 74·1% ([65·5–82·2]) for Full Context. The three non-agentic approaches clustered within overlapping confidence intervals, and the agentic system retained the highest concordance despite a change in documentation language (English versus German), institutional conventions, record structure, and the de-identification date-shifting applied to the source data. No system adaptation was performed between cohorts. Performance across question complexity and record length
Concordance decreased with question complexity across all approaches, but not uniformly (Figure 4a). For the agentic system, concordance fell from 86·1% ([81·9–90·1]) at Level 1 to 79·5% ([74·1–84·5]) at Level 2
6
and 65·1% ([56·8–73·3]) at Level 3. The advantage of the agentic system over Full Context increased monotonically with complexity, from +1·0 percentage point at Level 1 (𝑝 = 1·00) to +3·9 percentage points at Level 2 (𝑝 = 0·25) and +9·4 percentage points at Level 3 (𝑝 = 0·032) (Figure 4b). A similar gradient was observed relative to Iterative RAG, with a difference of +9·7 percentage points at Level 3 (𝑝 = 0·049) but no significant difference at Levels 1 or 2. At Level 3, the three non-agentic approaches ranged from 48·9% to 55·7%, whereas the agentic system reached 65·1%. Concordance was stratified by total patient record length in a hypothesis-driven exploratory analysis (Figure 4c). Patients were divided into four bins: three terciles of the lower 90th percentile (≤127k, 127 to 282k, and 282 to 541k characters) and the top decile (>541k characters, corresponding to approximately 245,000 tokens; 𝑛 = 10 patients, 47 evaluable pairs). In the lower three bins, no statistically significant differences were observed between the agentic system and either Full Context or Iterative RAG. In the top decile (𝑛 = 10 patients, 47 evaluable pairs; exploratory), the agentic system achieved 75·3% concordance compared with 63·9% for Iterative RAG (+11·4 percentage points) and 61·8% for Full Context (+13·5 percentage points), bootstrap confidence intervals for these comparisons were wide owing to the small patient count and these findings should be interpreted as hypothesis-generating. From the shortest to the longest bin, concordance fell by 7·1 percentage points for the agentic system, compared with 16·9 for Iterative RAG, 19·6 for Full Context, and 26·5 for Simple RAG. Clinical error profile and failure mechanisms
Error classification was performed on a single evaluation run with overall concordance of 77·8%. All 115 patient-question pairs for which the agentic system diverged from expert consensus on that run, including list-type responses with partial but incomplete overlap (F1 between zero and one), underwent structured classification by a senior haematologist into one of six categories (Table 1). Thirty-three divergences (28·7%) were classified as clinically significant errors, 24 (20·9%) as clinically insignificant errors, six (5·2%) as partially correct, 39 (33·9%) as acceptable or ambiguous responses in which both the system output and the reference annotation were defensible from the record, 11 (9·6%) as annotation errors in which the system response was correct and the reference annotation was not, and two (1·7%) as pipeline failures. The clinically significant error rate was 33 of 469 pairs (7·0%), with clinically significant errors distributed across complexity levels: 14 at Level 1, 12 at Level 2, and seven at Level 3. Blinded re-annotation of a proportional stratified random sample of 46 divergent pairs by an independent rater yielded a six-category 𝜅 of 0·667 (95 % CI [0·510, 0·833]) and a three-stratum 𝜅 of 0·802 ([0·687, 0·962]), indicating substantial to near-perfect reproducibility of the classification across severity levels. The system error rate, combining clinically significant and clinically insignificant categories, was 57 of 469 pairs (12·2%), comparable to the rate of expert disagreement on the same tasks (64 of 469 pairs, 13·6%, Figure 1g). Among system errors, 57·8% (33 of 57) were clinically significant, and among expert disagreements, 18·8% (12 of 64) were clinically significant. The severity distributions were therefore inverted despite comparable overall rates, with system errors carrying greater average clinical consequence than expert disagreements on the same tasks. Citation sufficiency was assessed on a stratified sample of 96 agentic system responses from the same run,
7
drawn evenly across complexity levels and concordance status, with each of the 48 question templates represented at least once (Figure 4d). All concordant responses were rated as fully supported by the retrieved source documents across all three complexity levels. Among discordant responses, the proportion rated as fully supported was 50% at Level 1, 62% at Level 2, and 50% at Level 3. Of the 22 responses with citation insufficiency, incomplete retrieval was the dominant failure mechanism, accounting for 83% of fully unsupported responses (𝑛 = 18), in the remaining 17%, a relevant document had been retrieved but was not incorporated into the final response.
Discussion Retrieval-augmented and full-context approaches converged on a shared performance ceiling on real longitudinal myeloma records, and agentic reasoning was the only approach to exceed it. The advantage was not uniform. It was negligible on simple single-record lookups, grew monotonically with question complexity, and was largest for patients with the longest and most complex disease trajectories, precisely those for whom the clinical stakes of accurate synthesis are highest. The convergence of Iterative RAG and Full Context at a shared ceiling indicates that the limiting factor is not the quantity of evidence presented to the model but the structure imposed on its integration. RAG delegates decomposition, evidence weighting, and clinical decision rules to a single generation step. Agentic reasoning externalises these operations as an explicit planning phase, iterative evidence gathering with re-query on insufficient results, and deterministic scoring calculators where clinical rules can be encoded without relying solely on language model interpretation. Ablation analyses indicate that the skill library contributes the largest individually separable gain, reducing concordance by 3·0 percentage points when removed, while the remaining components each contribute at most 0·4 percentage points in isolation, consistent with their role as interdependent scaffolding through which question-type-specific reasoning protocols are executed. The present findings extend the convergence ceiling previously observed on simpler clinical question answering12 to the harder longitudinal synthesis setting and show that structured reasoning is the first approach demonstrated to exceed it. The clinical profile of the agentic advantage identifies a coherent deployment target. The gap over the strongest non-agentic approach widened with both question complexity and total record length, concentrating on patients with the most treatment lines, relapse-remission cycles, and accumulating eligibility constraints. A decadelong myeloma trajectory with multiple sequential regimens and evolving comorbidities is simultaneously the case where manual chart review consumes the most physician time and the case where accurate longitudinal synthesis is most decision-relevant. Performance differences were negligible for patients in the lower 90th percentile of record length and largest in the top decile (exploratory, 𝑛 = 10 patients), and the agentic advantage over Full Context increased from 1·1 percentage points at Level 1 to 9·4 percentage points at Level 3. The system error rate (12·2%) was comparable to the rate of expert disagreement on the same tasks (13·6%), but the severity distributions were inverted: among system errors, 57·8% were clinically significant, against 18·8% of expert disagreements. The system therefore operated within the bounds of human annotator variability in aggregate, while its errors carried greater average clinical consequence. This distinction is not captured by concordance metrics alone and defines the gap between matching expert agreement in aggregate 8
and readiness for clinical deployment. One measurement asymmetry should be noted, expert disagreements were measured before adjudication, whereas system divergences were measured against the finalised reference standard. The dominant failure mechanism differed with task complexity. Errors on single-record lookup tasks arose predominantly from incomplete retrieval, as relevant documents were not returned by the search tools, a pattern directly addressable through retrieval engineering. Errors on multi-criterion synthesis tasks arose when the relevant evidence had been retrieved but was not correctly integrated during reasoning, a pattern consistent with the documented tendency of long-context language models to underweight information positioned in the middle of their input.24 The same mechanism likely explains part of the Full Context underperformance, where relevant passages were present in the context window but were not reliably attended to. Both limitations are properties of the underlying language model rather than of the agentic architecture. The model used here (gpt-oss-120b) was selected to run reproducibly on a single H200 GPU within institutional infrastructure, and larger or more recent models with stronger long-context reasoning may narrow the gap attributable to retrieval and integration failures. External validation on the MIMIC-IV critical care database preserved both the system ranking and the complexity gradient despite a change in documentation language, institutional conventions, and record structure, with no system adaptation between cohorts. The external cohort is small, and confirmation on larger prospectively defined datasets across further institutions remains needed. Limitations
The question bank was developed with haematologists at the same institution and finalised before system development, so implicit alignment between question design and system architecture cannot be excluded. The senior haematologist who drafted the question bank also served as sole adjudicator of annotation disagreements without independent second review, though error classification by the same rater showed substantial to near-perfect reproducibility on blinded re-annotation of a stratified random sample (𝜅 = 0·667 for six categories, 0·802 for three strata). Error classification was performed without formal blinding to system performance, introducing potential bias in clinical significance assignments. Expert disagreements were classified before adjudication whereas system divergences were measured against the finalised reference standard, an asymmetry that may favour the system. The agentic advantage was consistent across three of five backbone models but attenuated for the two with stronger baseline performance, suggesting its magnitude depends on the long-context integration capabilities of the underlying model (Supplementary Table B.6). Comparison with proprietary frontier models was precluded by institutional data privacy requirements. The evaluation is retrospective, and whether agentic assistance improves clinical decisions in prospective use has not been assessed. Conclusion
Agentic reasoning answered clinical questions on years of real longitudinal myeloma records at a level of concordance with expert consensus that retrieval-augmented and full-context approaches did not reach, and
9
its advantage increased with the complexity of cases. The total system error rate fell within the range of preadjudication inter-expert variability on the same tasks, although residual system errors carried greater average clinical consequence. Prospective evaluation in routine clinical care, with treating clinicians interacting with system outputs, will be required to determine whether these findings translate into measurable benefit at the point of care. Contributors
JM, LCA, KB, and KKB conceived and designed the study. JLu developed and coordinated the data acquisition and preprocessing pipeline, and contributed to system design and evaluation. JLa, SZ, MG, and MH contributed to data acquisition. CN, LM, AP, and JS contributed to the clinical question bank and annotated the data. KB drafted the clinical question bank, served as final adjudicator for all annotation disagreements, and contributed to the citation sufficiency and error classification reader studies. CN contributed to the error classification reader study. JM designed the agentic system, executed the experiments, performed result analysis and interpretation, and drafted the manuscript. CZ, ZB, FD, HH, AN, FP, and AZ provided iterative feedback on study design, execution, and interpretation, and revised the manuscript critically for important intellectual content. MM, FB, JP, DR, and KKB contributed to study design, supervised the study, and revised the manuscript. JM, JLu, KB, and KKB directly accessed and verified the underlying patient-level data reported in this study. All authors reviewed and approved the final version of the manuscript and accept responsibility for the decision to submit for publication. Declaration of interests
AP received travel support from Janssen-Cilag and Kite Gilead. FB received honoraria from Amgen, Johnson & Johnson, Bristol Myers Squibb (BMS), AbbVie, and GSK, travel support from Amgen, Johnson & Johnson, and BMS, and served on advisory boards for Amgen, Johnson & Johnson, BMS, AbbVie, and GSK. MH received honoraria from Johnson & Johnson, Sanofi, GSK, Oncopeptides, and Pfizer, payment for expert testimony from Johnson & Johnson and Oncopeptides, travel support from Johnson & Johnson, Oncopeptides, Pfizer, and Amgen, and served on advisory boards for Johnson & Johnson, Sanofi, Oncopeptides, and Pfizer. JLa received a Google Gemma Academic Program Award, received speaker honoraria from the Forum for Continuing Medical Education (Germany), AstraZeneca (Germany), and Novartis (Germany), and served on an advisory board for Novartis (Germany). All other authors declare no competing interests. Data sharing
The MIMIC-IV database is publicly available under the PhysioNet credentialled data use agreement at https://doi.org/10.13026/1n74-ne17. The in-house TUM dataset cannot be shared publicly owing to patient privacy regulations and institutional data governance requirements. Anonymised aggregate results supporting the findings of this study are available from the corresponding author ([email protected]) on reasonable request from the date of publication. Requests will be reviewed by the study team and a signed data access agreement will be required before release.
10
Acknowledgments
This study was financed by the public funder Bayern Innovativ (Bavarian State Ministry of Economics) Nuremberg, Grant Number: LSM-2403-0006. KKB and LCA are further grateful to be supported by the Else-Kröner-Fresenius-Foundation (2024_EKES.16, 2025_EKES.03). FB receives funding by the German Research Foundation (DFG) (TRR 387/1 - 514894665) and DFG BA 2851/7-1 (project ID: 537477296).
References [1] Rajkumar SV. Multiple myeloma: 2022 update on diagnosis, risk stratification, and management. American journal of hematology. 2022;97(8):1086-107. [2] Cui H, Unell A, Chen B, Fries JA, Alsentzer E, Koyejo S, et al. Timer: Temporal instruction modeling and evaluation for longitudinal clinical records. npj Digital Medicine. 2025;8(1):577. [3] Holmgren AJ, Apathy NC, Crews J, Shanafelt T. National trends in oncology specialists’ EHR inbox work, 2019–2022. JNCI: Journal of the National Cancer Institute. 2025;117(6):1253-9. [4] Du X, Zhou Z, Wang Y, Chuang YW, Li Y, Yang R, et al. Performance and improvement strategies for adapting generative large language models for electronic health record applications: a systematic review. International Journal of Medical Informatics. 2025:106091. [5] Liu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. Journal of the American Medical Informatics Association. 2025;32(4):605-15. [6] Qiu J, Lam K, Li G, Acharya A, Wong TY, Darzi A, et al. LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence. 2024;6(12):1418-20. [7] Truhn D, Azizi S, Zou J, Cerda-Alberich L, Mahmood F, Kather JN. Artificial intelligence agents in cancer research and oncology. Nature Reviews Cancer. 2026:1-14. [8] Ferber D, Wiest IC, Wölflein G, Ebert MP, Beutel G, Eckardt JN, et al. GPT-4 for information retrieval and comparison of medical oncology guidelines. Nejm Ai. 2024;1(6):AIcs2300235. [9] Lu KH, Mehdinia S, Man K, Wong CW, Mao A, Eftekhari Z. Enhancing Oncology-Specific Question Answering With Large Language Models Through Fine-Tuned Embeddings With Synthetic Data. JCO Clinical Cancer Informatics. 2025;9:e2500011. [10] Liu X, Zhang L, Munir S, Gu Y, Wang L. Verifact: Enhancing long-form factuality evaluation with refined fact extraction and reference facts. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing; 2025. p. 17919-36. [11] Chung P, Swaminathan A, Goodell AJ, Kim Y, Momsen Reincke S, Han L, et al. Verifying Facts in Patient Care Documents Generated by Large Language Models Using Electronic Health Records. NEJM AI. 2025;3(1):AIdbp2500418. 11
[12] Myers S, Dligach D, Miller TA, Barr S, Gao Y, Churpek M, et al. Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRsf. arXiv [preprint] arXiv:250814817. 2025. [13] Tu T, Schaekermann M, Palepu A, Saab K, Freyberg J, Tanno R, et al. Towards conversational diagnostic artificial intelligence. Nature. 2025;642(8067):442-50. [14] Schmidgall S, Ziaei R, Harris C, Reis E, Jopling J, Moor M. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv [preprint] arXiv:240507960. 2024. [15] Shi W, Xu R, Zhuang Y, Yu Y, Zhang J, Wu H, et al. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing; 2024. p. 22315-39. [16] Jiang Y, Black KC, Geng G, Park D, Zou J, Ng AY, et al. MedAgentBench: a virtual EHR environment to benchmark medical LLM agents. Nejm Ai. 2025;2(9):AIdbp2500144. [17] Lee G, Bach E, Yang E, Pollard T, Johnson A, Choi E, et al. Fhir-agentbench: Benchmarking llm agents for realistic interoperable ehr question answering. arXiv [preprint] arXiv:250919319. 2025. [18] Johnson A, Pollard T, Horng S, Celi LA, Mark R. MIMIC-IV-Note: Deidentified free-text clinical notes. PhysioNet. 2023 Jan. Version 2.2. Available from: https://doi.org/10.13026/1n74-ne17. [19] Johnson AE, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific data. 2023;10(1):1. [20] Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. circulation. 2000;101(23):e215-20. [21] Agarwal S, Ahmad L, Ai J, Altman S, Applebaum A, Arbus E, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv [preprint] arXiv:250810925. 2025. [22] Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems. 2020;33:9459-74. [23] Doan NN, Härmä A, Celebi R, Gottardo V. A hybrid retrieval approach for advancing retrieval-augmented generation systems. In: Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024); 2024. p. 397-409. [24] Liu NF, Lin K, Hewitt J, Paranjape A, Bevilacqua M, Petroni F, et al. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics. 2024;12:157-73.
12
Figures Figure 1 a Document Conversion
Radiology Report
Extraction & Structuring
Metadata Indexing
Discharge Summary
Quality Control and Validation
Expert Adjudication
QA Annotation by 4 Oncologists
71,729 semi-structured reports
Patient ID
Medical History
1,527 patients across 2 institutions
Report Type
Diagnosis Treatment
{ json }
4,374,481 laboratory results
Report Date
.MD
Laboratory Results
Document ID
b
558 expert-annotated QA pairs
d Number of Documents per Patient TUM (n=811) MIMIC (n=716) TUM median MIMIC median
Patients (%)
25 20 15 10
Phase I: Question Curation
Phase II: Cohort Sampling
Phase III: Expert Annotation
Curate questions from active MM clinical practice
Dev set: 10 patients (pre2020 diagnosis)
No expert annotation for development
Excluded from evaluations
5 0
c
9 9 9 9 9 9 9 9 9 0– 10–1 20–3 40–5 60–7 80–9 0–14 0–19 0–39 10 15 20 Documents per patient
Follow-up Span per Patient
35
TUM (n=811) MIMIC (n=716) TUM median MIMIC median
Patients (%)
30 25 20
Standardize and stratify by complexity Harmonize phrasing, assign complexity levels (L1/L2/L3)
Sample 100 patients for in-house evaluation Stratified by overall report count & last report date
Double annotation for in-house and external Independent annotation by two domain experts
Design structured answer schemas Define expected fields and formats per question-type
External validation set: 20 patients (MIMIC-IV) Independent cohort from separate institution
Senior expert adjudication Fifth annotator resolves all annotation conflicts
48 templates across five clinical tasks and three complexity levels
10/100/20 patients across development, evaluation, and external validation
558 gold-standard QA pairs: 469 in-house (TUM) + 89 external (MIMIC-IV)
15 10 5 0
1
0–
e
2 1–
5 5 0 4 6 8 2– 4– 6– 8–1 10–1 15–2 Follow-up span (years)
f
Annotation results
g
Inter-rater reliability
100
TUM (n = 143) TUM L1 (n=192)
Proportion of annotated pairs (%)
Adjudication results
28.6% 75
0.69
83.3%
22.0%
39.2%
TUM L2 (n=168)
0.60
74.6% 36.4%
TUM L3 (n=83)
0.57
61.4%
50
MIMIC (n = 22) MIMIC L1 (n=40) 65.2%
0.34
75.0%
13.6% 27.3%
67.0%
25
MIMIC L2 (n=32)
0.74
MIMIC L3 (n=14)
0.73
84.4% 40.9%
0
TUM Direct agreement Adjudicated
MIMIC Excluded
0
57.1%
0.2 0.4 0.6 0.8 Cohen's κ / Concordance score (%) Cohen's κ
Concordance
1.0
Interchangeable / equivalent Clinically insignificant disagreement Clinically significant disagreement Ambiguous question / evidence Single rater abstained Different granularity
Figure 1: Construction of longitudinal cohorts and expert-annotated evaluation dataset enabling clinically grounded assessment of longitudinal reasoning. (a) Overview of data sources and preprocessing pipeline across two institutions, including document extraction, structuring, metadata indexing, and quality control applied to heterogeneous clinical records. (b) Distribution of document counts per patient, demonstrating substantial variability in record density and reflecting the complexity of real-world longitudinal documentation. (c) Distribution of follow-up duration, highlighting long-term disease trajectories in the TUM cohort compared with shorter observation windows in MIMIC-IV. (d) Study design and cohort construction, including development, in-house evaluation, and external validation sets. (e) Annotation outcomes showing proportions of direct agreement, adjudicated cases, and exclusions. (f) Inter-rater reliability across predefined complexity levels, reported as Cohen’s 𝜅 and observed agreement, illustrating moderate agreement for clinically complex tasks. The low 𝜅 at MIMIC Level 1 reflects high prevalence of negative responses inflating the chance-agreement baseline. (g) Distribution of adjudication categories, indicating that a substantial proportion of disagreements reflects clinically insignificant or interchangeable interpretations rather than true errors.
13
Figure 2
TUM University Hospital In-house database
MIMIC-IV Note dataset P=145,915 patients
P=811 patients With multiple myeloma diagnosis
P=35 patients Excluded with insuficient documentation
P=716 patients With multiple myeloma diagnosis
P=776 patients Matching documentation criteria
Development set P=10 patients (no expert annotation)
P=662 patients Matching documentation criteria
Evaluation set P=100 patients Stratified sampling
n=500 question-answer pairs Two level 1, two level 2, and one level 3
P=45 patients Excluded with insufficient documentation
External validation set P=20 patients Stratified sampling
n=31 question-answer pairs Excluded postadjudication
n=100 question-answer pairs Two level 1, two level2, and one level 3
n=469 evaluable question-answer pairs Primary analysis
n=11 question-answer pairs Excluded postadjudication
n=89 evaluable question-answer pairs External validation
Figure 2: Cohort selection yields representative evaluation sets for longitudinal clinical reasoning tasks. Flow diagram of patient inclusion from institutional (TUM) and external (MIMIC-IV) datasets, including filtering based on diagnosis and documentation criteria. The primary evaluation cohort comprised 100 patients with 500 annotated question–answer pairs, of which 469 were retained after adjudication. The external validation cohort included 20 patients with 100 annotated pairs, of which 89 were retained. Stratified sampling ensured coverage across varying documentation density and temporal extent, supporting evaluation of reasoning under heterogeneous real-world conditions.
14
Figure 3 a
User workflow
b
c
Agent workflow I. Receive patient and question
Patient
Structured model context
d
Tools and Skills
Task context Required evidence
Is this patient eligible for CAR-T therapy?
Stop conditions Retrieved information
II. Interpret question and select suitable agent skills
Skill library
Retrieved skills
Task context
III. Draft ordered tooluse plan
Retrieved skills
Tool library
Ordered tool-use plan {}
IV. Execute tools to retrieve information
{}
{}
{}
{}
{}
{}
{}
{}
{} +
Task context -
Database
Retrieved information +
Patient Is this patient eligible for CAR-T therapy? Answer: Yes Reasoning: The patient has received therapies with IMiD [1], an anti-CD38 antibody [2], and …
[1] 2025-10-09 discharge_note
V. Sort and prioritize retrieved information
Ordered tool-use plan {}
{}
{}
{}
[2] 2023-05-29 tumor_board
Task context
[3] 2020-12-04 rad_report [4] 2026-02-14 Creatinine
{}
Structured retrieval Report Type tool Report Date
Retrieved information
VI. Formulate answer with sources
Figure 3: Agentic system enables structured, traceable clinical reasoning across longitudinal patient records. (a) User-facing workflow illustrating query input and generation of citation-backed answers grounded in patient records. (b) Internal agent workflow, including question interpretation, retrieval of task-specific clinical skills, generation of an ordered tool-use plan, iterative evidence retrieval, and synthesis. (c) Structured model context integrating task requirements, retrieved domain knowledge, intermediate evidence, and stopping criteria for reasoning completion. (d) Tool and skill library supporting structured access to clinical reports, laboratory trajectories, and deterministic scoring systems. Together, these components enable multi-step reasoning in which each intermediate step is explicitly linked to retrieved evidence, ensuring that final answers are verifiable and traceable to source documents.
15
Figure 4 a
b
QA Concordance on the evaluation set (N = 469)
100%
79.1%
Concordance (%)
80%
77.7% 75.4%
79.5% 79.6% 65.1%
75.7% 75.8% 55.7%
55.4% 48.9%
60%
Level 1 – Simple Level 2 – Medium Level 3 – Complex
90%
86.1%
85.1%
82.4% 74.4% 71.5%
Concordance by level across systems
100%
Level 3 – Complex Overall
40%
+1.0 pp
+2.7 pp
+3.3 pp 80% Concordance (%)
Level 1 – Simple Level 2 – Medium
20%
+3.8 pp*
-2.0 pp
+3.3 pp
70% 60%
+9.4 pp** +0.3 pp
50%
+6.5 pp**
40% 30%
0%
Simple RAG System
Iterative RAG System
Full Context
c
Agentic System
Simple RAG System
Iterative RAG System
Full Context
Agentic System
Concordance by difficulty level vs. patient record length Level 1 (easy)
70%
75%
Simple RAG Iterative RAG Full Context Agent
65%
≤127k
78% 68% 58%
Level 1
100%
≤127k
284-540k >541k 128-284k Patient record length (characters)
e
Level 2
100%
Level 3
80%
31%
60%
Accuracy (%)
50% 19% 100%
100%
100% 40%
62%
50%
20%
Correct
Incorrect
Correct
Incorrect
40%
50%
284-540k >541k 128-284k Patient record length (characters)
Level 1 – Simple Level 3 – Compl Level 2 – Medium Overall 96.6% 92.5% 94.7% 87.2% 82.8% 77.8% 79.2% 75.2% 72.8% 74.1% 84.9% 77.9% 64.3%
60%
52.9%
47.9%
47.1%
40% 20% 0%
Correct
≤127k
QA Concordance on the external validation set (MIMIC-IV, N = 89)
Not supported
31%
80% Cases (%)
Partially supported
50%
38%
Citation sufficiency (N = 96) Fully supported
60%
30%
48%
284-540k >541k 128-284k Patient record length (characters)
d
Concordance (%)
85%
0%
Level 3 (hard)
88% Concordance (%)
Concordance (%)
95%
55%
Level 2 (medium)
98%
Incorrect
Simple RAG System
Iterative RAG System
Full Context
Agentic System
Figure 4: Agentic reasoning improves accuracy, with largest gains in clinically complex tasks requiring longitudinal synthesis. (a) Overall and stratified concordance with expert consensus across system configurations on the primary evaluation cohort, showing superior performance of the agentic system. (b) Performance differences by complexity level, demonstrating that the advantage of the agentic approach increases with task difficulty. (c) Concordance stratified by patient record length and complexity level, showing convergence of all systems for shorter records and the largest agentic advantage among patients in the top decile of record length (>541k characters; exploratory, 𝑛 = 10 patients), where non-agentic configurations decline sharply. (d) Citation sufficiency analysis showing the proportion of responses fully, partially, or not supported by retrieved source documents across complexity levels and concordance status. (e) External validation on the MIMIC-IV cohort, confirming preservation of system ranking and robustness across institutions and documentation structures.
16
Table 1 Table 1: Error classification of agentic system divergences from expert consensus. All 115 patient-question pairs for which the agentic system output diverged from the adjudicated reference annotation on a single evaluation run (overall concordance 77·8%) were classified by a senior haematologist. The clinically significant error rate was 33 of 469 evaluable pairs (7·0%). Classification Clinically significant error Clinically insignificant error Partially correct Acceptable / ambiguous Annotation error Pipeline failure Total
Overall 33 (28·7%) 24 (20·9%) 6 (5·2%) 39 (33·9%) 11 (9·6%) 2 (1·7%) 115
17
Level 1 14 5 1 9 3 1 33
Level 2 12 11 5 12 6 0 46
Level 3 7 8 0 18 2 1 36
A
Supplementary Methods
A.1
Document preprocessing and extraction pipeline
Discharge summaries, radiology reports, pathology reports, tumour board proceedings, and ancillary document types (cardiology, cytology, flow cytometry, and genomic diagnostics reports) were exported as PDF files from the institutional SAP electronic health record system at TUM University Hospital and converted to structured Markdown using a rule-based module that applied font, size, positioning, and layout metadata to exclude non-clinical content and to identify section boundaries, which were preserved as hierarchical headers in the output. A post-processing step merged sections shorter than 50 words into the nearest adjacent section and split sections exceeding 350 words into overlapping chunks with a 50-word overlap. Laboratory reports stored as structured data were handled separately and excluded from text segmentation. Document-level metadata (patient identifier, report type, report date, and document identifier) were captured at export and stored alongside each section as structured fields, enabling retrieval queries to filter by type, date, or patient without parsing free text. Report type labels were mapped to a controlled vocabulary of nine canonical categories used as filter arguments by the agentic system’s retrieval tool. Sections were indexed using FTS5 full-text search with Porter stemming, metadata fields were stored as unindexed columns to support type- and date-based filtering without affecting relevance scoring. A.2
Laboratory data normalisation and concept mapping
Laboratory values were extracted from the institutional SAP system as timestamped observations comprising a test name, a numeric or categorical result, the reporting unit, and the laboratory-defined reference range. Because test names in the source system reflect free-text entry conventions that evolved over more than two decades of clinical use, a two-stage normalisation pipeline was developed to enable reliable programmatic retrieval. In the first stage, a catalogue of canonical laboratory concepts was constructed from all test names observed across the myeloma cohort. Each unique institutional code was assigned a stable identifier derived from a deterministic hash of its normalised form. Text normalisation comprised case folding, transliteration of German umlauts (ä → ae, ü → ue, ö → oe, ß → ss), removal of punctuation and special characters, and whitespace collapsing, yielding 731 unique canonical codes. In the second stage, an alias index was built to resolve synonymous test names to canonical codes. A total of 1,054 observed name variants were mapped to 736 canonical target codes, producing 1,404 alias entries. Of the canonical codes, 228 (31%) were associated with two or more synonymous input names, reflecting institutional naming changes over time, the coexistence of abbreviated and full-length designations (e.g. crp (c-reakt. pr) and crp (c-reakt. protein)), and specimen-type suffixes (e.g. albumin, albumin sm, albumin l). A set of search terms was pre-computed per canonical code from tokenised variants to support fuzzy matching at query time. The resulting catalogue spans myeloma-specific markers (serum and urine immunoglobulin free light chains 𝜅 and 𝜆, free light chain ratio, serum protein electrophoresis, immunofixation, immunoglobulins G, A, M, and 18
D, 𝛽2 -microglobulin), prognostic parameters (lactate dehydrogenase, C-reactive protein, calcium, albumin, haemoglobin, creatinine), and the broader set of routine haematological, biochemical, and coagulation tests documented over each patient’s care. No unit conversion was applied, original test name, reporting unit, and institutional reference range were returned alongside each numeric result. Laboratory values were indexed by their original timestamps with no temporal windowing imposed. A.3
Clinical question bank: development and instantiation
The clinical question bank was developed by a senior haematologist (KB) in consultation with the annotating oncologists (AP, CN, JS, LM) prior to any system development or evaluation. Templates were drafted to cover core clinical decision tasks in the longitudinal management of multiple myeloma, including current and prior treatment status, treatment response assessment, toxicity and dose modification documentation, diagnostic workup completion, disease staging, comorbidity scoring, and therapy eligibility determination. The initial template set was revised through iterative consensus discussion until coverage of the intended task space was judged representative. All revisions were completed before the evaluation cohort was defined and before any system outputs were generated. Each template was assigned by consensus to one of three complexity levels based on the cognitive operations required to produce a correct answer. Level 1 (single-record lookup) comprised questions answerable from a single document or structured data entry, such as whether the patient was currently receiving a specific medication. Level 2 (temporal reasoning) required integration of information across multiple documents or timepoints, such as identifying all documented treatment intervals for a given agent or determining the best documented response under a specified regimen. Level 3 (criteria-based synthesis) required structured synthesis across document types, laboratory values, and clinical timepoints against a multi-criterion decision rule, such as evaluating eligibility for BCMA-directed CAR-T cell therapy. The final bank comprised 48 templates (20 at Level 1, 18 at Level 2, and 10 at Level 3) spanning five clinical task categories (single choice, treatment intervals, first occurrence, staging, and eligibility). The full template list is provided in Supplementary Table B.2. The answer schema for each template determined the scoring method. Templates requiring a single categorical response (a yes/no/not documented answer, a single date, or a single staging score) were assigned binary scoring. Templates requiring enumeration of multiple entries (cycle start dates with doses, treatment intervals) were assigned list-type F1 scoring, in which precision, recall, and their harmonic mean were computed over the set of response entries against the reference list. For each patient in the evaluation cohort, five questions were instantiated from the template bank (two at Level 1, two at Level 2, and one at Level 3) by random draw without replacement from a shuffled deck at each complexity level (seed = 42). When the deck was exhausted it was reshuffled, ensuring balanced reuse across patients. Date placeholders were replaced with the date of the last available clinical report for that patient. For the MIMIC-IV cohort, de-identified dates were used as recorded in the dataset, which applies a consistent per-patient shift that preserves temporal ordering and intervals. Whether a given question was answerable from a particular patient’s record was determined only during expert annotation. The same instantiation procedure was applied to both cohorts.
19
A.4
Annotation protocol
Prior to annotation, all four oncologists (AP, CN, JS, LM) received structured training from the senior haematologist (KB) on the question bank, the answer schema conventions, the complexity-level definitions, and the criteria for abstention. Each annotator was assigned 50 patients from the TUM evaluation cohort by randomised balanced allocation such that each patient was reviewed by exactly two annotators and no annotator pair was systematically overrepresented (seed = 42). For the MIMIC-IV cohort, 20 patients were distributed between two of the same annotators using the same allocation procedure. Annotation was performed using a purpose-built Streamlit web application that presented each question with schema-aware input widgets (single-choice dropdowns, multi-interval entry forms, criteria tables with per-criterion status fields, date entry, and free-text fields). Annotators were instructed to use the abstain checkbox when an answer was not clearly supported by available documentation up to the cutoff date, and could optionally select one or more evidence documents and provide free-text comments. All responses were persisted to CSV logs with timestamps. For the TUM cohort, annotators reviewed patient records using the institutional SAP interface, which they use routinely in clinical practice. Each question specified a cutoff date, annotators were instructed to disregard documentation dated after the cutoff and, for conflicting statements, to prefer the most recent documentation before that date. For the MIMIC-IV cohort, annotators accessed an equivalent data browser providing report viewing and structured laboratory retrieval over the de-identified database. Annotators were blinded to each other’s responses throughout, the application enforced single-user login and stored responses in separate log files. No model-generated content was displayed at any point during annotation. A.5
Adjudication rules and disagreement taxonomy
All patient-question pairs for which two independent annotators did not reach direct agreement were referred to the senior haematologist (KB) for adjudication. Three case types entered adjudication: categorical disagreements (both annotators provided a response but on differing substantive content), single-rater abstentions (one annotator provided a response and the other abstained), and dual abstentions (both annotators abstained). Dual abstentions were excluded from the evaluable set without adjudication. The adjudicator reviewed each disagreement using the same record interface and cutoff-date convention as the original annotators. Both annotators’ responses were presented side by side, including evidence selections and comments, and the adjudicator provided a final answer using the same schema-aware interface. The adjudicator could adopt either annotator’s response or provide an independent answer, in single-rater abstention cases, the non-abstaining rater’s response was independently verified rather than accepted by default. Pairs for which the adjudicator also abstained were excluded from the evaluable set. Each adjudicated disagreement was classified into one of five categories. Interchangeable or equivalent responses covered cases in which both answers were clinically synonymous or differed only in phrasing, abbreviation, or formatting. Clinically insignificant disagreement covered cases in which one annotator made an error or omission that would not alter a clinical decision. Clinically significant disagreement covered cases in which the two annotators reached substantively different clinical conclusions with different management
20
implications. Ambiguous question or evidence covered cases in which the record did not contain sufficient or unambiguous information, or the instantiated question admitted more than one defensible interpretation. Difference in response granularity covered cases in which both annotators were correct but reported at different levels of detail. Single-rater abstentions were adjudicated but not assigned a disagreement category. The adjudicated answer served as the reference standard for all concordance analyses, for pairs with direct agreement, the shared answer was used without adjudication. A.6
Expert annotation and adjudication outcomes
For the TUM cohort, direct agreement was reached on 326 of 500 pairs (65·2%). A further 143 (28·6%) were included after adjudication and 31 (6·2%) were excluded (29 dual abstentions, 2 adjudicator abstentions), yielding 469 evaluable pairs in the primary analysis (200 Level 1, 179 Level 2, 90 Level 3; Figure 1e). Pre-adjudication inter-rater agreement was 𝜅 = 0·69 at Level 1 (observed agreement 83·3%, 𝑛 = 192), 𝜅 = 0·60 at Level 2 (observed agreement 74·6%, 𝑛 = 168), and 𝜅 = 0·57 at Level 3 (observed agreement 61·4%, 𝑛 = 83) (Figure 1f). Among adjudicated cases, the most frequent disagreement categories were interchangeable or equivalent responses (39·2%) and clinically insignificant disagreement (36·4%), with the remainder attributable to clinically significant disagreement (8·4%), ambiguous question or evidence (7·0%), differences in response granularity (3·5%), and single-rater abstentions (5·6%) (Figure 1g). For the MIMIC-IV cohort, direct agreement was reached on 67·0% of pairs. A further 22·0% were included after adjudication and 11·0% were excluded, yielding 89 evaluable pairs. Pre-adjudication inter-rater agreement was 𝜅 = 0·74 at Level 2 (observed agreement 81·2%, 𝑛 = 32) and 𝜅 = 0·73 at Level 3 (observed agreement 78·6%, 𝑛 = 14). The lower Level 1 kappa (𝜅 = 0·34, observed agreement 75·0%, 𝑛 = 40) reflects the high prevalence of negative responses in the MIMIC-IV records (72·5% of Level 1 answers were “False”), which inflates the chance-agreement baseline and suppresses 𝜅 even when observed agreement is high. A.7
Agentic system: implementation details
The agentic system was implemented as a four-phase reasoning pipeline. Each phase corresponded to a structured prompt (termed a signature) whose output was parsed as JSON and validated before progression to the next phase. The four phases were question assessment and skill selection, tool-use plan construction, iterative tool execution, and final answer synthesis. A schematic overview is provided in Figure 2. Phase I: Question assessment and skill selection.
Upon receipt of a clinical question, the system derived a short medical analysis identifying the clinical intent and key hypotheses, a list of required and currently missing information points, and a preliminary complexity assessment. In parallel, a subset of skill modules was selected from the indexed clinical skill library based on question content and available skill summaries. Heuristic keyword matching served as a fallback to ensure that question-type-specific style skills were activated even when the language model omitted them, for example, questions containing temporal keywords (aktuell, derzeit) triggered the current-status style skill, and questions referencing eligibility criteria triggered the eligibility style skill. A base structured-annotation style skill was
21
always included to enforce the two-line output format. Clinical skill library.
The skill library comprised modular instruction packages in four categories. Workflow skills encoded task-specific decomposition strategies and evidence priorities for clinical tasks such as therapy reconstruction, eligibility assessment, staging score calculation, and laboratory trend interpretation. Parsing skills provided normalisation rules for institutional abbreviations, German-language date formats, and entity synonymy (e.g., mapping trade names to generic drug names). Style skills defined answer structure templates for each question type, including required fields and formatting conventions. Policy skills encoded deterministic precedence rules for resolving contradictory evidence, including temporal authority (most recent document takes precedence), plan-versus-administered therapy disambiguation (administered therapy overrides documented plans), and contradiction resolution heuristics. Skills were indexed by identifier and category. The language model selected relevant skill identifiers from a summary prompt listing all available skills. Selected skills were rendered into two context blocks: a workflow context block (workflow, parsing, and knowledge skills) injected during tool execution, and a style context block (style skills) injected during final answer synthesis. Policy skills were attached deterministically based on the selected style and workflow skills rather than by model selection, ensuring consistent application of temporal authority and contradiction resolution rules regardless of model selection behaviour. Across the evaluation cohort, an average of 6.7 skill modules were selected per question, corresponding to approximately 8,000 tokens of skill context, compared with approximately 49,000 tokens required to load all 41 modules unconditionally. The contribution of the selection mechanism to accuracy and efficiency is reported in the ablation analysis (Table B.5). Fixed baseline prompt.
All system configurations, including ablation conditions in which the skill library was disabled, shared a fixed baseline prompt of approximately 500 tokens. This prompt provided task framing (instructing the model to answer clinical questions from retrieved context only), the two-line output format specification, all answer schema definitions with worked examples covering single-choice, list, multi-interval, cyclestart-dose-list, and criteria-table formats, citation formatting rules using bracketed context identifiers, the distinction between “Nicht dokumentiert” (information absent from the record) and “Nein” (documented negative finding), and drug name normalisation mappings covering drug class to substance mappings (CD38 antibodies, immunomodulatory drugs, proteasome inhibitors, CAR-T, and bispecific T-cell engagers), common abbreviations (Dara, Btz, Len), and standard regimen names (VRd, KRd, Dara-Rd, DVRd). Clinical reasoning workflows, temporal reasoning policies, disease-specific reference knowledge (staging criteria, response categories), and structured retrieval strategies were provided exclusively by the skill library, ensuring that ablation comparisons isolated the contribution of clinical reasoning protocols from formatting, schema, and normalisation knowledge.
22
Phase II: Tool-use plan construction.
A structured tool-use plan was drafted as a JSON object containing an ordered array of steps and a set of global stopping conditions. Each step specified a step number, a natural-language objective, the tool to be invoked, the tool arguments as a JSON object, a list of evidence requirements, and a conditional stopping rule. Up to three plan construction attempts were permitted, if the language model returned an invalid or empty plan, a repair prompt was appended and the plan was regenerated. If all three attempts failed, the run was terminated as a pipeline failure. In ablation conditions where planning was disabled, tools were selected reactively at each execution step without a pre-drafted plan. Phase III: Iterative tool execution.
Across execution rounds, the system maintained a structured memory state as a JSON object encoding the original query, accumulated evidence from prior tool calls, outstanding missing information, and the global stopping conditions defined in the tool-use plan. This state was updated after each round and passed forward to inform tool selection, re-query decisions, and termination. The plan was executed step by step, with up to eight tool-use rounds permitted per question. At each round, the language model received the current plan step, accumulated evidence from prior rounds, the list of still-missing information, and the set of allowed tools, then selected one of three actions. It could invoke a specified tool, advance to the next plan step without a tool call, or terminate execution and proceed to answer synthesis. If the model chose to terminate before executing at least one tool call for the current step and further planned steps remained, a repair prompt required it to either execute the planned tool or explicitly skip the step. Duplicate queries (identical tool name and query string) were blocked and the model was notified. A negative query cache tracked report retrieval queries that returned no results, preventing re-execution of identical unsuccessful queries. Tool results were cached within each run to avoid redundant calls with identical arguments. An automatic adjustment mechanism progressively broadened retrieval parameters after repeated empty results: after one failure, the number of returned results was doubled (up to a cap of 30), after two failures, the query was reduced to its keyword subset, after three failures, temporal scope restrictions were removed. A context budget of 120,000 tokens was enforced, if the estimated token count of accumulated context nodes exceeded this limit, retrieval was terminated early and the system proceeded to answer synthesis. Context nodes were deduplicated by section identifier before being passed to the final answer phase. Tool specifications.
Five tool classes were available to the agentic system. The report retrieval tool performed keyword search over the document database using BM25 ranking (BM25Okapi from the rank_bm25 library). The tool accepted a free-text query string, a report type filter selecting from nine canonical document categories, a temporal scope (all records, the most recent document, a specific date, or a date range), and corresponding date parameters. A report type synonym mapping normalised common German-language and abbreviated report type names to canonical values, and query strings were sanitised to remove answer-schema tokens that the language model
23
occasionally included in retrieval queries. The laboratory query tool fetched structured laboratory values by canonical marker key directly from the laboratory database table, accepting a list of canonical lab keys with the same temporal scope options as the report retrieval tool. The set of available canonical keys was determined per patient at runtime and provided to the language model in the prompt to prevent queries for non-existent markers. A default limit of five results per marker was applied. Four deterministic clinical scoring calculators were provided, each implemented as a stateless function with validated inputs and JSON output. The ISS (International Staging System) assigned Stage I when serum 𝛽2 microglobulin was below 3·5 mg/L and serum albumin was at least 3·5 g/dL, Stage III when 𝛽2 -microglobulin was 5·5 mg/L or above, and Stage II otherwise. The R-ISS (Revised ISS) assigned Stage I when ISS Stage I criteria were met together with the absence of high-risk cytogenetic abnormalities (del(17p), t(4;14), or t(14;16)) and normal LDH, Stage III when ISS Stage III criteria were met together with either high-risk cytogenetics or elevated LDH, and Stage II otherwise. The R2-ISS (Second Revision ISS) incorporated additional risk factors including the LDH/ULN ratio and 1q gain/amplification status, yielding four stages (I–IV). The HCT-CI (Hematopoietic Cell Transplantation Comorbidity Index) was computed from 17 boolean comorbidity flags with weights of 1 (arrhythmia, cardiac disease, inflammatory bowel disease, diabetes requiring medication, cerebrovascular disease, psychiatric disturbance, mild hepatic abnormality, obesity with BMI above 35, and persistent infection), 2 (rheumatologic disease, peptic ulcer, moderate-to-severe renal impairment, and moderate pulmonary disease), or 3 (prior solid tumour, heart valve disease, severe pulmonary disease, and moderate-to-severe hepatic disease), with risk groups defined as low (score 0), intermediate (score 1–2), and high (score ≥ 3). Phase IV: Final answer synthesis.
After tool execution, accumulated context nodes were deduplicated and assigned sequential citation identifiers. The language model received the original question, patient context, a plan execution summary, the formatted context snippets with citation identifiers, the style context from selected skills, the response requirements derived from the question assessment, and the output of the policy engine where applicable. The policy engine was a deterministic module that ranked retrieved evidence items by temporal authority and recency, resolved contradictions between planned and administered therapies, and produced a structured resolution (select, abstain, or conflict) passed to the final answer signature. The final answer was required to conform to a two-line format comprising an “Answer:” line containing the schema-aligned value and a “Reasoning:” line containing one to two sentences with inline citation references. If the initial output did not conform to this format, a repair prompt was appended and the answer was regenerated (up to two attempts). If no valid answer was produced after all attempts, the run was recorded as a pipeline failure. Citation identifiers appearing in the answer were validated against the set of available context nodes, and citations referencing non-existent sources were flagged as potentially hallucinated.
24
Decoding and inference settings.
All language models were deployed locally on a single NVIDIA H200 GPU using a vLLM inference server with LiteLLM as the routing layer, providing an OpenAI-compatible API endpoint. The primary model (gpt-oss-120b) was served in mxfp4 precision. Decoding parameters were set to the values recommended by the respective model developers and held constant within each model’s evaluation, no parameter optimisation was performed on the evaluation cohort. Run-to-run variability attributable to stochastic decoding was characterised through the ten-run stability analysis (Supplementary Table 4). Pipeline failure definition.
A pipeline failure was recorded when the system did not return a parseable, schema-conformant response within the permitted retry budget (up to three attempts for tool-use plan construction and up to two for final answer synthesis). In the classified error run, two of 469 pairs (0·4%) were classified as pipeline failures. A.8
Comparator configurations
Simple RAG, Iterative RAG, and Full Context are described in the main text. All comparators used the same locally deployed 120-billion-parameter language model (gpt-oss-120b), the same patient record database, and the same structured answer prompt requiring the two-line format. Temperature was set to 0·2 for all comparator configurations. Maximum output tokens were set to 512 for answer generation and 256 for query generation where applicable. The embedding model used for dense retrieval was distiluse-base-multilingual-cased-v2, a multilingual sentence transformer producing normalised embeddings for cosine similarity computation. The LLM Baseline configuration received only the patient identifier, the question text, a reference date, and the answer schema, with no record access or retrieval context of any kind. The system prompt instructed the model to answer based solely on its parametric knowledge, establishing a floor for performance without record access. Advanced RAG extended Simple RAG with query rewriting into up to eight focused retrieval queries, hybrid fusion of BM25 and dense scores (BM25 built over character trigrams with 𝑘 1 = 1 · 2 and 𝑏 = 0 · 75, fusion weight 𝛼 = 0 · 5), and cross-encoder reranking using bge-reranker-v2-m3. The top 20 documents after reranking were packed into the context window subject to the same 120,000-token budget. This configuration is reported in Supplementary Table B.3 but is not included in the main comparator set. A.9
Scoring and concordance computation
The concordance metric is defined in the main text. For single-value categorical items, scoring was binary: a score of 1 was assigned if the system output matched the adjudicated reference in substantive content and 0 otherwise. Matching was performed after normalisation of both the system output and the reference annotation, including extraction of the answer value from the two-line output format, case-insensitive comparison, whitespace trimming, and removal of formatting artefacts. Format deviations that did not alter substantive content were not penalised. Semantic equivalences were applied where clinically appropriate, for
25
therapy-related questions, “nie verabreicht” (never administered) and “nicht dokumentiert” (not documented) were treated as equivalent when the patient record contained no evidence the therapy had ever been given. For list-type items, F1 was computed over the set of response entries against the reference list, with each entry treated as an atomic unit. Precision was the fraction of system entries matching a reference entry, recall was the fraction of reference entries matched by a system entry, F1 was their harmonic mean. For list-type items preceded by a status field, the status was evaluated separately: an incorrect status (e.g., “Nie verabreicht” when reference entries existed) resulted in a score of 0 for the entire item regardless of entry-level matching. Overall concordance was computed as the mean score across all evaluable patient-question pairs, where each pair contributed either a binary score (0 or 1) or a continuous F1 score (0 to 1) depending on the answer schema type. Per-question scores were first averaged across ten independent runs, and overall concordance was computed from these per-question means. Confidence intervals and pairwise significance tests were computed using a cluster bootstrap procedure. Whole patients were resampled with replacement, retaining all questions per sampled patient to account for within-patient correlation; the concordance statistic was recomputed on each resample over 𝑁boot = 10,000 iterations. The 2·5th and 97·5th percentiles of the bootstrap distribution defined the 95% confidence interval (two-sided). The two-sided 𝑝-value for pairwise differences was derived as the proportion of bootstrap resamples in which the absolute deviation of the resampled difference from its mean equalled or exceeded the absolute observed difference (shift-to-null method). Bonferroni correction was applied within each stratum using 𝑚 = 6 headline pairwise comparisons, capped at 1·0. Results of all pairwise comparisons are reported in Supplementary Table B.3. A.10
Error classification protocol
Error classification was performed by the senior haematologist (KB) on the evaluation run with the lowest overall concordance among the ten independent runs (77·8%, compared with a 10-run mean of 79·6%), selected to provide the most conservative basis for error rate estimation. All 115 patient-question pairs for which the system output diverged from the adjudicated expert consensus, including list-type responses with partial but incomplete overlap, were submitted for classification. Classification was performed using a purpose-built review application that presented each case with the question text, answer schema, the adjudicated reference annotation, and the full agent response comprising the extracted answer, the reasoning chain with inline citation identifiers, and all retrieved source documents. Retrieved documents were separated into those cited in the agent’s reasoning and those retrieved but not cited, each could be expanded to display the full report text with the cited passage highlighted. The reviewer additionally had access to the institutional SAP interface to verify claims against the primary clinical record when needed. Each divergence was classified into exactly one of six categories. Annotation error (category A) covered cases in which the system response was correct and the adjudicated reference was wrong or incomplete. Acceptable or ambiguous (category B) covered cases in which both the system response and the reference were clinically defensible given the available evidence. Partially correct (category C) covered cases in which the system captured the correct clinical concept but with incomplete or imprecise details. Clinically insignificant error
26
(category D) covered cases in which the system response was incorrect but the error would not alter a clinical decision. Clinically significant error (category E) covered cases in which the system response was incorrect in a way that could alter a clinical decision. Pipeline failure (category F) covered cases in which the system did not produce a valid, schema-conformant response within the permitted retry budget. The threshold separating clinically insignificant from clinically significant errors was whether a treating clinician relying on the system response rather than the correct answer would be expected to make a different management decision. The clinically significant error rate reported in the main text (7.0%, 33 of 469 evaluable pairs) was defined as the proportion of category E errors over all evaluable pairs. Because the classified run had below-average concordance, this rate represents a conservative upper bound on the expected clinically significant error rate across runs. Error classification was performed by a single reviewer without independent second review, a limitation acknowledged in the main text. To assess the reproducibility of the primary error classification, a blinded re-annotation sub-study was conducted on a proportional stratified random sample of the 115 divergent patient–question pairs. The pairs were collapsed into three severity strata before sampling, comprising cases classified as annotation error or acceptable and ambiguous (categories A and B), cases classified as partially correct or clinically insignificant error (categories C and D), and cases classified as clinically significant error or pipeline failure (categories E and F). A 40 % proportional stratified random sample was drawn from each stratum (seed = 42), with any rounding remainder assigned to the E+F stratum, yielding 𝑛 = 46 cases. Cases were presented in order of patient and question identifier to an independent rater (JS) blinded to the original classification, who applied the same six-category taxonomy and clinical significance threshold used in the primary error classification. Agreement is quantified using Cohen’s 𝜅 for both the collapsed three-stratum scheme and the full six-category scheme, with 95 % confidence intervals computed by bootstrap (𝑁boot = 10,000). A.11
Citation sufficiency assessment
Citation sufficiency was assessed on the same evaluation run used for error classification (overall concordance 77·8%). A stratified sample of 96 agentic system responses was drawn, comprising 16 responses from each of six cells defined by the cross-classification of complexity level (Level 1, Level 2, Level 3) and concordance status (concordant, discordant), with the constraint that each of the 48 question templates contributed at least one response. Within each cell, responses were drawn randomly from the available pool after satisfying the template coverage constraint. Assessment was performed by two reviewers (KB, CN), who divided the 96 cases between them. For each sampled response, the reviewer was presented with the question text, the adjudicated reference annotation, and the full agent output with inline citation identifiers. Each cited document could be expanded to display the full report text with the cited passage highlighted, documents retrieved but not cited were displayed separately. The reviewer additionally had access to the institutional SAP interface. For each response, the reviewer assessed whether the cited sources fully supported the answer on a three-point scale: fully supported (cited evidence clearly and completely justified the answer), partially supported (cited evidence covered some aspects but was incomplete or insufficient), or not supported (cited evidence did not justify the answer). Support was assessed independently of answer correctness. For responses rated as partially
27
or not supported, the reviewer identified whether a relevant document appeared in the retrieved-but-not-cited set, and classified the dominant failure mechanism as one of three types: reasoning failure (the relevant document was retrieved but not used or was misinterpreted during synthesis), retrieval failure (the relevant document existed in the record but was not returned by the retrieval tools), or true knowledge gap (the information required to answer the question was not present in the patient record).
28
B
Supplementary Tables
B.1
Supplementary Table 1: Baseline characteristics of evaluation cohorts
Demographic and disease characteristics of patients in the primary evaluation (TUM, 𝑛 = 100) and external validation (MIMIC-IV, 𝑛 = 20) cohorts. Continuous variables are reported as median (IQR), categorical variables as 𝑛 (%). Characteristic Age at diagnosis, years Sex (male), 𝑛 (%) Documents per patient Laboratory values per patient Follow-up span (record), years
B.2
TUM (𝑛 = 100) 65·1 (57·3–74·0) 63 (63%) 50·5 (31·0–83·0) 1414 (751–2278) 5·5 (2·7 – 11·0)
MIMIC-IV (𝑛 = 20) 66·5 (58·8-77·5) 13 (65%) 29·0 (14·8–61·8) 3411 (1989–7411) 3·3 (1·2 – 4·6)
Supplementary Table 2: Clinical question bank
All 48 question templates used for evaluation, grouped by clinical task category and complexity level. Each template includes the question identifier, complexity level, clinical task category, template text with placeholder variables, expected answer format, and scoring method (binary or list-type F1). [date] denotes the date of the last available report. ND = not documented.
29
ID
L
Category
Template text
Answer format
Scoring
Is the patient receiving lenalidomide on
Yes / No / ND / Unclear
Binary
Yes / No / ND / Unclear
Binary
Yes / No / ND / Unclear
Binary
Level 1 – Simple Q01
1
Single choice
[date]? Q02
1
Single choice
Is the patient receiving bortezomib on [date]?
Q03
1
Single choice
Is the patient receiving daratumumab on [date]?
Q04
1
Single choice
Is there documented evidence that a whole- Yes / No / Unclear
Binary
body MRI has been performed? Q05
1
Single choice
Is there documented evidence that a PET- Yes / No / Unclear
Binary
CT has been performed? Q06
1
Single choice
30
Yes / No / Unclear
Binary
Has the patient received at least one CD38- Yes / No / Unclear
Binary
Is there documented evidence that FISH has been performed?
Q07
1
Single choice
antibody therapy? Q08
1
Single choice
Has the patient received at least one IMiD
Yes / No / Unclear
Binary
Has the patient received at least one protea- Yes / No / Unclear
Binary
therapy? Q09
1
Single choice
some inhibitor? Q10
1
Single choice
Has the patient received an autologous
Yes / No / Unclear
Binary
SCT? Q11
1
Single choice
Has the patient received an allogeneic SCT?
Yes / No / Unclear
Binary
Q12
1
Single choice
Has the patient received CAR-T therapy?
Yes / No / Unclear
Binary
Q13
1
Single choice
Has the patient received BiTE therapy?
Yes / No / Unclear
Binary
Q14
1
Single choice
Has the patient received dialysis?
Yes / No / Unclear
Binary
Q15
1
Single choice
Has the patient been diagnosed with sepsis
Yes / No / Unclear
Binary
or septic shock?
ID
L
Category
Template text
Answer format
Q16
1
Single choice
Has the patient been mechanically venti- Yes / No / Unclear
Scoring Binary
lated (invasive or non-invasive)? Q17
1
Single choice
Is renal failure / AKI documented?
Yes / No / ND / Unclear
Binary
Q18
1
Single choice
Is clinically relevant anaemia present on
Yes / No / ND / Unclear
Binary
[date] (documented or lab-confirmed)? Q19
1
Single choice
Has the patient received red blood cell trans- Yes / No / Unclear
Binary
fusions? Q20
1
Single choice
Has the patient been diagnosed with pneu- Yes / No / Unclear
Binary
monia? Level 2 – Medium Q21
2
Treatment in- Which cycle start dates (C*D1) and doses tervals
31
Q22
2
Q23
2
for bortezomib are documented?
Treatment in- Which cycle start dates (C*D1) and doses tervals
≤12×(date; dose; unit)
List-type F1
≤12×(date; dose; unit)
for lenalidomide are documented?
List-type F1
Treatment in- Which melphalan exposures are docu- ≤3×(type; date; dose; unit)
List-type
tervals
F1
mented (type, approximate date, dose per episode)?
Q24
2
Treatment in- Which doxorubicin exposures are docu- ≤3×(form; date; dose; unit)
List-type
tervals
F1
mented (form, approximate date, cumulative dose per episode)?
Q25 Q26
2 2
Treatment in- In which documented intervals was carfil- ≤3×(start–end / ongoing)
List-type
tervals
F1
zomib administered (start–end or ongoing)?
Treatment in- In which documented intervals was poma- ≤3×(start–end / ongoing)
List-type
tervals
F1
lidomide administered (start–end or ongoing)?
ID
L
Category
Q27
2
Treatment in- In which documented intervals was tervals
Template text
Answer format
Scoring
≤3×(start–end / ongoing)
List-type
meropenem administered (start–end or on-
F1
going)? Q28
2
Single choice
Was toxicity, dose reduction, or discontin- Yes / No / Unclear + free uation documented under carfilzomib? If
Binary
text
yes, reason? Q29
2
Single choice
Was toxicity, dose reduction, or discontinu- Yes / No / Unclear + free ation documented under pomalidomide? If
Binary
text
yes, reason? Q30
2
Single choice
Best documented response under isatux- CR/VGPR/PR/SD/PD imab + Pom-Dex?
Q31
2
Single choice
Best
documented
32
KRD
/
Binary
/
Binary
/
Binary
/
Binary
/
Binary
/
Binary
Never / ND / Unclear response
under
CR/VGPR/PR/SD/PD
(carfilzomib–lenalidomide– Never / ND / Unclear
dexamethasone)? Q32 Q33 Q34
2 2 2
Single choice Single choice Single choice
Best documented response under the docu- CR/VGPR/PR/SD/PD mented first-line therapy?
Never / ND / Unclear
Best documented response after high-dose
CR/VGPR/PR/SD/PD
melphalan?
Never / ND / Unclear
Best documented response under BiTE?
CR/VGPR/PR/SD/PD Never / ND / Unclear
Q35
2
Single choice
Best documented response under CAR-T?
CR/VGPR/PR/SD/PD Never / ND / Unclear
Q36
2
First rence
Q37
2
First rence
occur- First documented CT report describing new
Date / ND / Unclear
Binary
Date / ND / Unclear
Binary
osteolytic lesions? occur- First documented episode of renal failure or dialysis-requiring AKI?
ID
L
Category
Template text
Answer format
Scoring
Q38
2
Staging
ISS stage on [date] calculated from albu- Score (I/II/III) + date + Binary min and 𝛽2 M (last labs ≤90 days prior)?
source
ECOG score on [date] (last documented
Score (0–4) + date + source
Binary
Score + date + source
Binary
Level 3 – Complex Q39
3
Staging
≤180 days, else inferred from explicit functional description)? Q40
3
Staging
HCT-CI score on [date] (last documented ≤365 days, else derived from comorbidities)?
Q41
3
Staging
R-ISS stage on [date] (labs ≤90 days; cyto- Stage (I/II/III) + date + Binary genetics from last available report)?
Q42
3
Staging
source
R2-ISS stage on [date] (labs ≤90 days; cy- Stage (I/II/III/IV) + date + Binary
33
togenetics incl. 1q status from last available
source
report)? Q43 Q44 Q45
3 3 3
Single choice Single choice Single choice
Is the patient triple-class refractory on
Yes / No / Unclear + free
[date]? (with documented justification)
text
Is the patient quadruple-class refractory on
Yes / No / Unclear + free
[date]? (with documented justification)
text
Is the patient eligible for ASCT on [date]
Yes / No / Unclear + free
based on age, comorbidities, performance
text
Binary Binary Binary
status, and organ function? Q46
Q47
3
3
Single choice
Single choice
Which therapy achieved the best docu- Free
text
(therapy)
+ Binary
mented response up to [date] (per IMWG
CR/VGPR/PR/SD/PD / ND
criteria)?
/ Unclear
Which risks currently dominate: disease
Progression / Toxicity / Both
progression or therapy toxicity?
/ Unclear + free text
Binary
ID
L
Category
Template text
Answer format
Q48
3
Eligibility
Which BCMA-CAR-T eligibility criteria
Criteria
are met / not met / missing on [date]? Is
met/missing) + Yes / No /
the patient eligible overall?
Unclear
table
Scoring (met/not
Binary
34
B.3
Supplementary Table 3: Pairwise cluster bootstrap significance tests for concordance differences
All pairwise comparisons between system configurations, overall and stratified by complexity level. Differences are reported as System A minus System B in percentage points; a negative value indicates that System B outperformed System A. Confidence intervals and 𝑝-values were obtained by cluster bootstrap with 𝑁boot = 10,000 resamples, resampling whole patients with replacement and retaining all questions per sampled patient to account for within-patient correlation. The N column reports the number of patients contributing to each stratum. Bonferroni-corrected 𝑝-values were computed by multiplying raw 𝑝-values by the number of pairwise comparisons within each stratum (6 headline comparisons), capped at 1·0. Significance codes reflect Bonferroni-corrected 𝑝-values: ∗∗∗ 𝑝 < 0 · 001; ∗∗ 𝑝 < 0 · 01; ∗ 𝑝 < 0 · 05; ns 𝑝 ≥ 0 · 05. Subset Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Overall Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1 Level 1
Comparison (A vs B) Baseline vs Simple RAG Baseline vs Advanced RAG Baseline vs Iterative RAG Baseline vs Full Context Baseline vs Agentic System Simple RAG vs Advanced RAG Simple RAG vs Iterative RAG Simple RAG vs Full Context Simple RAG vs Agentic System Advanced RAG vs Iterative RAG Advanced RAG vs Full Context Advanced RAG vs Agentic Iterative RAG vs Full Context Iterative RAG vs Agentic Full Context vs Agentic Baseline vs Simple RAG Baseline vs Advanced RAG Baseline vs Iterative RAG Baseline vs Full Context Baseline vs Agentic System Simple RAG vs Advanced RAG Simple RAG vs Iterative RAG Simple RAG vs Full Context Simple RAG vs Agentic Advanced RAG vs Iterative RAG Advanced RAG vs Full Context Advanced RAG vs Agentic Iterative RAG vs Full Context Iterative RAG vs Agentic Full Context vs Agentic
N patients
Diff A−B (pp)
95% CI
Raw 𝑝
Bonf. 𝑝
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
−70·22 −74·81 −74·14 −74·55 −78·31 −4·58 −3·92 −4·33 −8·09 +0·67 +0·26 −3·51 −0·41 −4·17 −3·76 −78·90 −84·50 −82·25 −84·90 −86·00 −5·60 −3·35 −6·00 −7·10 +2·25 −0·40 −1·50 −2·65 −3·75 −1·10
[−74·00, −66·44] [−77·93, −71·52] [−77·21, −70·96] [−77·81, −71·17] [−81·37, −75·20] [−6·99, −2·22] [−6·46, −1·38] [−6·80, −1·95] [−11·25, −4·93] [−0·64, +1·94] [−1·72, +2·18] [−5·80, −1·26] [−2·50, +1·64] [−6·68, −1·78] [−6·13, −1·47] [−83·55, −74·05] [−88·65, −80·00] [−86·65, −77·45] [−89·00, −80·60] [−90·15, −81·60] [−9·15, −2·10] [−7·30, +0·45] [−9·60, −2·75] [−11·65, −2·50] [−0·10, +4·60] [−3·35, +2·50] [−4·95, +1·90] [−5·95, +0·60] [−7·80, +0·40] [−4·35, +2·25]
<0·001 <0·001 <0·001 <0·001 <0·001 <0·001 0·0024 0·0003 <0·001 0·3083 0·7907 0·0025 0·6938 0·0012 0·0010 <0·001 <0·001 <0·001 <0·001 <0·001 0·0025 0·0911 0·0012 0·0020 0·0615 0·7878 0·3907 0·1134 0·0733 0·5078
<0·001 <0·001 <0·001 <0·001 <0·001 0·0006 0·0144 0·0018 <0·001 1·000 1·000 0·0150 1·000 0·0072 0·0060 <0·001 <0·001 <0·001 <0·001 <0·001 0·0150 0·5466 0·0072 0·0120 0·3690 1·000 1·000 0·6804 0·4398 1·000
35
∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗ ∗∗∗ ∗∗∗
ns ns ∗∗
ns ∗∗ ∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗
ns ∗∗ ∗∗
ns ns ns ns ns ns
Supplementary Table 3 (continued): Pairwise cluster bootstrap significance tests for concordance differences Subset Comparison (A vs B) N patients Diff A−B (pp) 95% CI Raw 𝑝 Bonf. 𝑝 Level 2 Baseline vs Simple RAG 100 −72·65 [−78·74, −66·40] <0·001 <0·001 Level 2 Baseline vs Advanced RAG 100 −75·89 [−81·22, −70·41] <0·001 <0·001 Level 2 Baseline vs Iterative RAG 100 −75·87 [−80·90, −70·57] <0·001 <0·001 Level 2 Baseline vs Full Context 100 −73·88 [−79·84, −67·74] <0·001 <0·001 Level 2 Baseline vs Agentic 100 −77·76 [−82·79, −72·40] <0·001 <0·001 Level 2 Simple RAG vs Advanced RAG 100 −3·24 [−7·28, +0·47] 0·0981 0·5886 Level 2 Simple RAG vs Iterative RAG 100 −3·22 [−7·11, +0·55] 0·1019 0·6114 Level 2 Simple RAG vs Full Context 100 −1·23 [−5·15, +2·63] 0·5401 1·000 Level 2 Simple RAG vs Agentic 100 −5·11 [−8·76, −1·56] 0·0058 0·0348 Level 2 Advanced RAG vs Iterative RAG 100 +0·01 [−1·37, +1·36] 0·9859 1·000 Level 2 Advanced RAG vs Full Context 100 +2·01 [−1·32, +5·58] 0·2474 1·000 Level 2 Advanced RAG vs Agentic 100 −1·87 [−4·57, +0·81] 0·1645 0·9870 Level 2 Iterative RAG vs Full Context 100 +1·99 [−1·27, +5·34] 0·2354 1·000 Level 2 Iterative RAG vs Agentic 100 −1·89 [−4·52, +0·69] 0·1515 0·9090 Level 2 Full Context vs Agentic 100 −3·88 [−7·78, −0·32] 0·0418 0·2508 Level 3 Level 3 Baseline vs Simple RAG 90 −46·11 [−54·78, −37·67] <0·001 <0·001 Level 3 Baseline vs Advanced RAG 90 −51·11 [−59·11, −43·11] <0·001 <0·001 Level 3 Baseline vs Iterative RAG 90 −52·67 [−60·78, −44·44] <0·001 <0·001 Level 3 Baseline vs Full Context 90 −52·89 [−61·67, −44·22] <0·001 <0·001 Level 3 Baseline vs Agentic 90 −62·33 [−71·00, −53·11] <0·001 <0·001 Level 3 Simple RAG vs Advanced RAG 90 −5·00 [−9·56, −0·55] 0·0295 0·1770 Level 3 Simple RAG vs Iterative RAG 90 −6·56 [−11·44, −1·89] 0·0078 0·0468 Level 3 Simple RAG vs Full Context 90 −6·78 [−13·22, −0·22] 0·0420 0·2520 Level 3 Simple RAG vs Agentic 90 −16·22 [−25·11, −7·11] 0·0004 0·0024 Level 3 Advanced RAG vs Iterative RAG 90 −1·56 [−4·56, +1·22] 0·2847 1·000 Level 3 Advanced RAG vs Full Context 90 −1·78 [−6·44, +3·00] 0·4521 1·000 Level 3 Advanced RAG vs Agentic 90 −11·22 [−18·44, −4·00] 0·0020 0·0120 Level 3 Iterative RAG vs Full Context 90 −0·22 [−5·33, +5·11] 0·9310 1·000 Level 3 Iterative RAG vs Agentic 90 −9·67 [−16·67, −2·67] 0·0082 0·0492 Level 3 Full Context vs Agentic 90 −9·44 [−16·22, −2·67] 0·0054 0·0324
36
∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗
ns ns ns ∗∗
ns ns ns ns ns ns ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗
ns ∗∗
ns ∗∗∗
ns ns ∗∗
ns ∗∗ ∗∗
B.4
Supplementary Table 4: Run-to-run stability across system configurations
Concordance estimates across ten independent evaluation runs for each system configuration on the primary evaluation cohort (TUM, 𝑛 = 469). Results are reported as mean ± standard deviation across runs, overall and stratified by complexity level. Individual run results are listed to characterise the distribution. System Baseline Simple RAG System Advanced RAG System Iterative RAG System Full Context Agentic System
B.5
Overall 1·3 ± 0·5% 71·5 ± 0·9% 76·1 ± 1·7% 75·4 ± 2·2% 75·8 ± 0·5% 79·6 ± 1·1%
Level 1 0·1 ± 0·2% 79·0 ± 1·7% 84·7 ± 1·5% 82·4 ± 2·0% 85·0 ± 1·0% 86·2 ± 1·1%
Level 2 1·8 ± 1·0% 74·4 ± 0·9% 77·7 ± 1·3% 77·6 ± 3·0% 75·7 ± 1·1% 79·5 ± 2·4%
Level 3 2·8 ± 1·1% 48·9 ± 2·7% 53·9 ± 4·5% 55·4 ± 3·0% 55·7 ± 1·8% 65·1 ± 2·2%
Supplementary Table 5: System ablations
Concordance across ablated system configurations on the primary evaluation cohort (TUM, 𝑛 = 469). Values are mean concordance (%) across the number of independent runs indicated, overall and stratified by complexity level. Each row removes or replaces one or more components of the full agentic system. The unfiltered skill library condition loads all 41 skill modules unconditionally rather than selecting by query content. Configurations without a skill library use the same fixed baseline prompt as the comparator systems, without the agent-specific reasoning skills. Configuration Standard configuration No deterministic clinical scoring tools No type and date filters in retrieval tools Full skill library, unfiltered No structured memory state Reactive tool selection (no pre-planned use) No skill library
B.6
Overall 79·6% 79·6% 79·5% 79·3% 79·2% 79·2% 76·6%
Level 1 86·2% 85·1% 87·7% 87·7% 85·9% 86·6% 86·0%
Level 2 79·5% 80·0% 78·1% 78·7% 79·0% 78·7% 77·9%
Level 3 65·1% 66·7% 63·8% 62·0% 64·9% 63·8% 53·3%
Supplementary Table 6: System performance across language model backbones
All four system configurations were evaluated under each backbone model on the primary evaluation cohort (469 patient-question pairs, TUM). Each model was deployed locally via vLLM using the inference settings recommended by the respective developer. Weight quantization reflects the precision of the loaded model checkpoint (MXFP4 or FP8), Qwen3-Next-80B-A3B-Instruct additionally used an FP8 KV cache. Overall concordance (%) is reported as mean across ten runs. The model used in the primary analysis is indicated (∗ ).
37
Model gpt-oss-120b∗ GLM-4.5-Air Qwen3-Next-80B-A3B-Instruct gemma-4-31B-it gpt-oss-20b
B.7
Params 120B 110B 81B 33B 21B
Quantization MXFP4 FP8 FP8 FP8 MXFP4
Simple RAG 71·5% 64·9% 75·0% 74·7% 67·9%
Iterative RAG 75·4% 68·2% 76·5% 82·4% 72·1%
Full Context 75·8% 72·0% 76·1% 81·3% 68·4%
Agentic 79·6% 77·9% 76·8% 81·6% 77·4%
Supplementary Table 7: Concordance by clinical task category and system configuration
Concordance stratified by clinical task category (single choice, treatment intervals, first occurrence, staging, eligibility) for each system configuration on the primary evaluation cohort (TUM, 𝑛 = 469). Values are mean concordance across ten independent runs with 95% bootstrap confidence intervals (cluster bootstrap, 10,000 resamples, patient-level resampling). Category sample sizes: single choice 𝑛 = 326, treatment intervals 𝑛 = 70, first occurrence 𝑛 = 20, staging 𝑛 = 45, eligibility 𝑛 = 8. Results for the eligibility category are based on a small sample and should be considered hypothesis-generating only. System Baseline Simple RAG Advanced RAG Iterative RAG Full Context Agentic System
B.8
Single choice 0·1 [0·0–0·2] 76·9 [76·3–77·5] 81·8 [81·0–82·8] 80·9 [79·1–82·1] 83·5 [82·9–84·0] 86·4 [85·7–87·0]
Treatment intervals 2·6 [1·7–3·6] 74·2 [73·0–75·2] 74·6 [73·5–75·6] 74·6 [73·0–76·0] 70·9 [70·1–71·7] 76·1 [74·9–77·3]
First occurrence 0·0 [0·0–0·0] 38·0 [35·5–40·5] 52·5 [47·5–57·0] 52·0 [48·5–55·5] 40·0 [37·5–42·5] 50·5 [46·0–55·0]
Staging 8·4 [6·2–10·9] 55·6 [54·7–56·4] 56·4 [55·3–57·6] 55·3 [53·3–57·1] 53·6 [52·0–55·3] 52·4 [50·4–54·7]
Eligibility 0·0 [0·0–0·0] 1·2 [0·0–3·8] 25·0 [16·2–33·8] 32·5 [26·2–40·0] 21·2 [13·8–28·7] 58·8 [53·8–63·7]
Supplementary Table 8: Execution characteristics by complexity level and system configuration
Wall-clock time per question on the primary evaluation cohort (TUM, 𝑛 = 469 patient-question pairs from 100 patients), stratified by complexity level and system configuration. Unless otherwise indicated, all values are median (IQR) across all patient-question pairs from a single-concurrency timing run on a single NVIDIA H200 GPU. Skills selected, tool calls, and document retrieval metrics are reported for the agentic system only.
38
Metric Agentic system Skills selected per question Tool calls per question Documents retrieved Documents cited of which ≥3 documents (%) Wall-clock time per question, s Iterative RAG Wall-clock time per question, s Full Context Wall-clock time per question, s Simple RAG Wall-clock time per question, s
B.9
Level 1 (𝑛 = 200)
Level 2 (𝑛 = 179)
Level 3 (𝑛 = 90)
Overall (𝑛 = 469)
6·0 (6·0–7·0) 2·0 (1·0–3·0) 15·0 (7·5–26·0) 2·0 (2·0–2·0) 5·9 18·5 (16·7–20·6)
6·0 (6·0–7·0) 3·0 (2·0–4·0) 27·0 (14·0–49·0) 2·0 (2·0–2·0) 9·5 21·3 (19·4–23·8)
8·0 (7·0–9·0) 4·0 (3·0–4·0) 27·0 (16·0–46·0) 2·0 (2·0–4·0) 47·4 28·7 (26·3–34·2)
7·0 (6·0–7·0) 3·0 (2·0–4·0) 20·0 (10·0–39·0) 2·0 (2·0–2·0) 15·2 20·9 (18·2–25·2)
6·0 (5·0–7·3)
6·4 (5·1–8·6)
12·3 (7·2–23·4)
6·6 (5·3–9·5)
4·7 (3·8–5·6)
5·0 (4·0–6·0)
6·0 (5·1–6·9)
5·0 (4·0–6·0)
2·2 (1·8–2·8)
2·2 (1·8–2·8)
2·7 (2·2–3·1)
2·3 (1·8–2·9)
Supplementary Table 9: Patient stratification by record length
Patients were stratified by total clinical record length (character count of the full concatenated record) into four bins for the context-length sensitivity analysis. Three bins span the lower 90th percentile of the distribution (Q1–Q3), defined at the 33rd and 67th percentiles, the fourth bin (Q4) comprises the top decile. Evaluable pairs are summed across all complexity levels. Bin Q1 (≤p33) Q2 (p33–p67) Q3 (p67–p90) Q4 (>p90) Total
Character range ≤127k 127k–282k 282k–541k >541k ≤1,076k
Patients (𝑛) 33 34 23 10 100
39
Evaluable pairs (𝑛) 155 160 107 47 469
C
Supplementary Figures
C.1
Supplementary Figure 1: Accuracy by patient sex, vital status, and age group Accuracy by patient sex (bootstrap 95% CI)
Level 1
Level 2
Level 3 100
80
80
80
80
60
40
20
60
40
60
40
20
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
Accuracy (%)
100
Accuracy (%)
100
Accuracy (%)
Accuracy (%)
All levels 100
60
40
20
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
20
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent Male
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
Female
Accuracy by patient vital status (bootstrap 95% CI)
Level 1
Level 2
Level 3 100
80
80
80
80
60
40
20
60
40
20
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
Accuracy (%)
100
Accuracy (%)
100
Accuracy (%)
Accuracy (%)
All levels 100
60
40
60
40
20
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
20
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
Alive
0 AG AG AG em le R ced R tive R ic Syst p Sim Advan Itera Agent
Deceased
Accuracy by patient age group (quartiles, ref. 06 Jan 2026) — bootstrap 95% CI Level 1
Level 2
Level 3 100
80
80
80
80
60 40 20
0 tem RAG RAG RAG ple anced rative tic Sys im n S Ite Adv Age
60 40 20
60 40 20
0 tem RAG RAG RAG ple anced rative tic Sys im n S Ite Adv Age ≤64 yr
0 tem RAG RAG RAG ple anced rative tic Sys im n S Ite Adv Age Age group 64–74 yr 74–83 yr >83 yr
Accuracy (%)
100
Accuracy (%)
100
Accuracy (%)
Accuracy (%)
All levels 100
60 40 20
0 tem RAG RAG RAG ple anced rative tic Sys im n S Ite Adv Age
Figure S1: Concordance with expert consensus stratified by patient sex (male, female), vital status (alive, deceased), and age group at the reference date (quartiles: ≤64, 64–74, 74–83, >83 years), shown overall and by complexity level for all four system configurations. Bootstrap 95% confidence intervals were computed by cluster bootstrap resampling at the patient level (𝑁boot = 10,000).
40
C.2
Supplementary Figure 2: Accuracy by question template across different systems Per-question accuracy by system Level 1 – Simple
Level 2 – Medium
Level 3 – Complex
Average accuracy
1.0 0.8 0.6 0.4 0.2
0 Q1 1 Q1 2 Q1 3 Q1 4 Q1 5 Q1 6 Q1 7 Q1 8 Q1 9 Q2 0 Q2 1 Q2 2 Q2 3 Q2 4 Q2 5 Q2 6 Q2 7 Q2 8 Q2 9 Q3 0 Q3 1 Q3 2 Q3 3 Q3 4 Q3 5 Q3 6 Q3 7 Q3 8 Q3 9 Q4 0 Q4 1 Q4 2 Q4 3 Q4 4 Q4 5 Q4 6 Q4 7 Q4 8
9
8
Q1
Q0
7
Q0
6
5
Q0
Q0
4
Q0
3
Q0
2
Q0
Q0
Q0
1
0
Question ID RAG System Single choice
Iterative RAG
Treatment intervals
Full Context
Agentic System
First occurrence
Staging
Eligibility
Figure S2: Per-question concordance averaged across ten independent evaluation runs for each of the 48 clinical question templates, stratified by system configuration and clinical task category (single choice, treatment intervals, first occurrence, staging, eligibility). Templates are ordered by complexity level (Level 1, Level 2, Level 3) and grouped by task category within each level.
41