NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

arXiv:2611.08369v1 [cs.AI] 5 Nov 2026

Pengkai Wang1,2∗ Wei-Wei Zhang3,4∗ Yan Li3,4∗ Min Tang5 Zhitian Hou1,2 Zeyu Liu2 Guanghao Zhu1,2 Yuanyi Wang1,2 Yanggan Gu1,2 Wenjun Wang1,2 Minheng Ni1,2 Congkai Xie2 Zhijie Sang2 Jianmin Wu1,2 Ying Sun3,4† Hongxia Yang1,2† 1 The Hong Kong Polytechnic University 2 infix.ai 3 Sun Yat-sen University 4 Sun Yat-sen University Cancer Center 5 Monash University

§ GitHub Code

Hugging Face Data

Abstract Current medical benchmarks typically break down clinical competence into isolated factual questions, single-image tasks, or static case vignettes. These tests score individual answers, but reveal less about whether a model can integrate subspecialty knowledge with multimodal evidence and track a patient’s condition throughout a course of care. To close this gap, we introduce NPCBench, a clinical apprenticeship benchmark for nasopharyngeal carcinoma (NPC). NPCBench mirrors specialist training, from guideline and foundational knowledge to case planning, longitudinal review, CarePath integration, and open-ended consultation. It contains 4,777 component and consultation items and a 100-patient CarePath cohort with 417 scored diagnostic and care tasks. The episode-level evaluation tests whether diagnostic and care answers are jointly correct within a care pathway. By linking stepwise tasks to patient-level care trajectories, NPCBench evaluates whether models ground decisions in visual evidence, interpret changes in patient state, and meet clinical requirements in consultation. Across 30 evaluated models, we observe a composition gap: strong performance on individual tasks does not ensure jointly correct diagnostic and care answers for the same patient. Among 21 multimodal models, strict CarePath success reaches at most 23%. Paired controls and physician calibration further examine evidence grounding, longitudinal revision, and consultation judging. Overall, NPCBench broadens the evaluation of medical artificial intelligence beyond static question-answering paradigms to an apprenticeship-oriented assessment of clinical workflows, thereby offering a testbed for evidence-based and temporally coherent reasoning in oncology care.

1

Introduction

Medical large language models (LLMs) and multimodal LLMs (MLLMs) now achieve strong results in medical examinations, visual question answering, and broad benchmark suites [1–6]. However, exam-style accuracy offers limited insight into subspecialty care, where clinicians must synthesize disease-specific guidelines, multimodal evidence, and evolving patient states over time. During clinical apprenticeship, physicians do not learn these skills in isolation. Recalling a clinical guideline is only the first step; specialists must also ground staging in diagnostic images, tailor treatment plans to patient constraints, and revise decisions as new responses or toxicities emerge. Evaluating medical AI therefore requires following this same progression from foundational knowledge to longitudinal ∗ Equal contribution. † Corresponding authors.

40th Conference on Neural Information Processing Systems (NeurIPS 2026). Track on Evaluations and Datasets.

(a) NPC Clinical Scenario Care pathways

Baseline Clinical Evidence Symptoms & Signs

Labs

EBV DNA

Clinical Notes

Standard

Management

State change

Follow-up

State change

Diagnosis & Stage

Structure ID Invasion

Personalized plan

Patient

Factors

Case

Evidence update

Care pathway

Case-level Planning Real-World

O1

Treatment

New evidence

MRI Sequence ID

Classification

L6

Initial visit

Multimodal Evidence Grounding Pathology

Objective Evidence • Labs • EBV-DNA • Exams

Guideline decision

Rules

Case

Orders

Decision Tree

Atomic Guideline Application

Treatment revision Condition change

Branch switch

Evaluation Metrics

Longitudinal Revision

Initial case

Follow-up

Longitudinal Evidence Stream Guideline Knowledge

CSCO NCN

L5

Management

(c) Pathway-Sensitive Evaluation

Guideline Memory NCCN

L4

CSCO

Guidelines

Progression

(b) Clinical Apprenticeship Dimensions

L3

NCCN Guidelines

NCCN

Exams

MRI

L2

Treatment

Guideline grounding

Pathology

L1

Diagnosis & Stage

Workup

Treatment revision

Condition change

Decision Acc.

All-care

Joint Success

Correctness at each care decision

All scored care answers correct

Both image labels and all care answers correct

CarePath: 100 patients, 417 tasks Consult Rubrics

Open query Response

High step accuracy successful pathway

LLM Judge & Expert Judge

Clinician rubrics overlay for open-ended responses

!

Figure 1: Overview of NPCBench. (a) NPC care combines pathology, MRI, biomarkers, and clinical records under guideline constraints. (b) The apprenticeship framework spans component skills, longitudinal revision, and CarePath evaluation; O1 assesses consultations. (c) Changing patient states drive reporting of local and joint correctness. CarePath evaluates each decision using the provided clinical context; separate serial controls examine feedback from prior answers. management. While isolated tasks can pinpoint weaknesses in individual skills, only linked patient episodes can reveal whether these skills succeed together for the same patient [7–9]. Evaluating the progression from individual competencies to integrated clinical care requires a disease setting in which multimodal evidence and sequential decisions are closely connected. Nasopharyngeal carcinoma (NPC), prevalent in Southern China and Southeast Asia [10–12], is managed according to staging systems and clinical guidelines [13–15]. Diagnosis, staging, and risk assessment require integrating histopathology, multi-sequence MRI, and plasma EBV DNA [11, 16]. Clinical evidence and management objectives vary across stages of care, with subsequent decisions shaped by both the current patient state and preceding clinical events. During treatment and follow-up, tumor response, treatment-related toxicities, and recurrence or progression may influence management strategies [12, 14, 15]. NPC therefore provides a suitable setting for evaluating both individual specialist competencies and their integration across successive stages of care. To systematically assess these capabilities, we introduce NPCBench, a clinical apprenticeship benchmark for NPC modeled on the progressive training of specialist physicians. As shown in Fig. 1, it organizes the evaluation into six progressive levels and an open-ended consultation suite. L1 and L2 test guideline recall and application. L3 tests pathology and MRI evidence grounding. L4 tests case planning from a single patient state, L5 longitudinal decision revision, and L6 linked CarePath episodes. O1 evaluates rubric-based specialist consultation. By pairing component tasks with patient trajectories, NPCBench asks a central measurement question: Are diagnostic and care answers jointly correct for the same patient when component scores are high? Models with similar average accuracy can exhibit different patient-level error patterns, or even select the right treatment 2

while misinterpreting the underlying MRI evidence. NPCBench therefore pairs task-level scores with strict patient-level correctness, while holding patients and questions fixed in matched controls to examine how visual evidence, clinical chronology, and answer formats shape model decisions. Using NPCBench, we investigate four research questions (RQ1–RQ4) across 30 medical and generalpurpose models. RQ1 examines how well models translate guideline knowledge into case-level planning: guideline recall exceeds 92%, whereas the highest case-planning accuracy is 71.9%. RQ2 probes whether staging predictions are grounded in accurate visual interpretation: complete invasion correctness reaches at most 5.0% on the 100-patient anatomical panel, showing that correct stage labels often lack complete anatomical support. RQ3 examines decision revision and patient-level success: history can correct answers and introduce new errors. In CarePath, Gemini-3.1 achieves 89.9% Care accuracy but 11.0% Strict success; Claude-4.6 has the highest Strict success at 23.0%. RQ4 assesses consultation safety and automated judge reliability: GPT-5.1, the highest-scoring consultation model, triggers major-error criteria in 14.6% of responses under the primary judge, while rubric judges show sensitivity–specificity imbalances against physician annotations. We summarize the main contributions of this work as follows. ① A clinical apprenticeship evaluation framework. We shift the evaluation focus from static answer correctness to pathway correctness. NPCBench assesses whether models make consistent clinical decisions across patient-state transitions, integrating guideline constraints, multimodal evidence grounding, and longitudinal disease evolution. ② A specialist-curated multimodal and longitudinal dataset. We release 4,777 component and consultation items alongside 100 CarePath episodes with 417 scored diagnostic and care tasks, unifying guideline rules, pathology slides, multi-sequence MRI scans, longitudinal EHR trajectories, and consultation rubrics within a single oncology pathway. ③ Diagnostic dissection of the clinical composition gap. Across 30 models, we combine capability profiling with controlled interventions and physician-calibrated judging to identify where visual grounding separates from treatment decisions, how patient history changes decisions, and where high local scores coexist with patient-level errors.

2

Related Work

From exam QA to clinically structured benchmarks. Early evaluations of medical LLMs relied on static, examination-style question–answering benchmarks, including MedQA [17], MedMCQA [18], PubMedQA [19], the medical subsets of MMLU [20], MultiMedQA [2], MedBench [21], CMB [22], and CMExam [23]. These benchmarks capture medical knowledge and local reasoning, but they do not test whether a model can accumulate patient evidence, review decisions, and preserve coherence across the care pathway. Recent broad suites such as MedHELM [8], MedXpertQA [24], and AnesSuite’s AnesBench [25] move the evaluation toward clinically meaningful capabilities and (a) Pediatric

L1 Guidelines

Staging

(e)

L2 Application

Toxicity

met.

Adult M0

RT

L5 Revision 305 questions Plan/ exec.

Toxicity Regimen switch Progress.

(d)

N stage

Metastatic

(f)

L3 MRI 3,700 items

L3 Pathology 69 cases

NPC−

Initial T stage

EBV

L4 Planning

(c) Sequence

Recur.

165 questions

Initial tx

185 questions

Rec./

Workup

Follow-up

(b)

250 questions

Toxicity

(g) Diagnosis

Metast./ evidence

L6 CarePath

Treatment

O1 Consultation 103 consultations

Workup

follow-up Workup

(h)

217 care decisions

Response/

Phase

NPC+

Structure cards

Imaging/ staging Tox./ supp.

Diagnosis Treatment Special pop.

Adverse events

Figure 2: Task composition of NPCBench. Evaluation items are distributed across six progressive levels (L1–L6) and the open-ended consultation suite (O1). Hatched slices each account for less than 5% of their subset; L6 shows its 217 care decisions. More clinical category names are defined in Appendix A. 3

specialty-aware reasoning. NPCBench follows this direction and makes the clinical apprenticeship ladder explicit: it measures component skills alongside longitudinal decisions and episode-level care correctness. Multimodal, longitudinal, and rubric-based clinical evaluation. Medical MLLM benchmarks have expanded from image QA datasets such as VQA-RAD [26], SLAKE [27], and PathVQA [28] to broader evaluations of multimodality, trustworthiness, endoscopy, clinical reasoning, and sequence grounding [29–33]. In parallel, HealthBench [34], LiveMedBench [35], LLMEval-Med [36], MedDialogRubrics [37], and ClinConsensus [38] use expert-validated rubrics or temporally grounded cases to evaluate open-ended clinical responses. Workflow-oriented resources such as MedJourney [39], MedAgentBench [40], MedAgentBoard [41], and MTBBench [42] further shift evaluation from static snapshots toward state transitions. NPCBench brings these directions into a single disease setting: pathology and MRI evidence grounding, patient-level longitudinal revision, strict CarePath episodes, and criterion-level consultation rubrics. Appendix B gives a fuller comparison with recent medical benchmark families.

3

NPCBench: A Clinical Apprenticeship Benchmark

3.1

Overview

NPCBench organizes the clinical apprenticeship framework into a progressive capability ladder (L1–L6) and cross-level consultation rubrics (O1). The ladder spans guideline memory, atomic application, multimodal evidence grounding, case-level planning, longitudinal decision revision, and linked care episodes. Lower levels (L1–L4) assess individual capability components; L5–L6 examine longitudinal decisions and joint diagnostic–care correctness within patients. O1 complements these tasks by assessing free-text consultation quality. These tasks assess whether specialist decisions follow guidelines, use multimodal evidence, and remain consistent with a patient’s clinical course. Four design principles guide task construction and clinical review: (i) Competence isolation requires each level in the progressive capability ladder (L1–L6) to measure a distinct, clinically meaningful component of specialist competence. (ii) Evidence-boundary control fixes the available evidence at the intended clinical decision point, removing future treatment, outcomes, and other answer-leaking cues. (iii) Clinical determinacy requires reference decisions supported by guidelines or expert consensus, given the provided clinical evidence. (iv)Auditable open-ended scoring uses case-specific consultation rubrics (O1) to assess clinical justification, evidence grounding, conditional reasoning, and safety. All patient materials are de-identified, and NPC oncologists verified component validity, decision determinacy, and the fidelity of patient-state transitions. Figure 2 shows the composition of each subset, and Fig. 3 illustrates the pipeline for constructing L1–L5 items. 3.2

Construction and Validation

Expert-curated clinical materials are converted into standardized evaluation items and reference answers. L1–L2 were derived from CSCO and NCCN guidelines; L3 from pathology and multisequence MRI images; and L4–L6 and O1 from de-identified real-world patient records, covering caselevel planning, longitudinal decision-making, full CarePath evaluation, and open-ended consultations with scoring rubrics. Multidisciplinary reviewers in oncology, pathology, and radiology assessed clinical validity, evidence sufficiency, and reference standards. For L1–L2, experts verified guideline consistency and answer determinacy; for L3, they validated pathology labels, MRI sequence and anatomical structure identification, invasion status, and T/N staging against imaging evidence. For L4, they assessed case sufficiency and appropriateness of reference decisions given patient-specific clinical states; for L5–L6, they verified evidence from decision-points, reference actions, and transitions from clinical state; and for O1, they reviewed core clinical requirements, supporting evidence, conditional recommendations, and major/minor error criteria to ensure clinically valid and auditable scoring. 3.3

Evaluation Levels and Scoring Units

L1.G-Mem: Guideline memory. The 250 expert-reviewed questions assess NPC guideline knowledge: 153 have both CSCO and NCCN support, 80 CSCO support only, and 17 NCCN support only. 4

(a) Guidelines Structuring

L1 Guideline Memory Rule QA

NCCN

CSCO

250 questions • 153 dual-source • 80 CSCO

MCQ Construction

•

L2 Atomic Guideline Application

Knowledge Decision Tree

Decision Family

Expert Validation

17 NCCN

185 questions • 84 initial • 30 recurrent • 44 metastatic • 27 toxicity

Standard Case Construction

(b)

题⺫：根据 AJCC/UICC 第9版与 CSCO 2025，M1b 指？ A. 仅⾻转移且⾻灶 >3 个时才属 M1b B. 仅肺转移 C. ≥1 个器官/部位存在 >3 个转移病灶 D. 任何肝转移均属 M1b

Task 1 Sequence Type

L3

Task 2 Grounding

T1, T2, T1+C

MRI tasks

Pathology tasks

(c) Real-Case

Structuring

Raw EHR Records

Real State Snapshot

MCQ

clinical notes · labs · exams · pathology ·orders

Patient State Extraction

Decision Timeline Visit states New evidence · transitions

questions • 116 treatment • 6 toxicity • 43 progression

Construction

L5 Longitudinal Revision

Expert Validation

305

questions Real Timeline

Prefix

Decision-Turn Extraction

...

T1–T4 (400) N0–N3 (400)

165

L4 Case-level Planning

T/N

• 100 patients • 1–5 turns each • state-update decisions

Structure ID cavernous sinus? invasion?

and more

T / N stage

Answer: C

病例：患者男，63岁，ECOG PS 0。鼻咽非角化性癌（未分化型），cT1N3M0，III期。 左颈 II–IV 区多发融合淋巴结，最大短径 >3 cm，包膜外侵犯明显；全身检查未见远处转 移。EBV-DNA 0 copy/mL，血常规及肝肾功能正常，尚未治疗。 问题：最优治疗策略是？ A. 单纯根治性 IMRT B. TPF 诱导化疗序贯同期放化疗 C. GP 诱导化疗 2–3 周期序贯同期放化疗 Answer: C D. 同期放化疗 + 辅助化疗

18 NPC structures

病例（节选）：男，42岁，鼻咽癌 cT3N3M0。 2024-09 确诊，完成根治性放化疗后⼝服卡培他滨。 2025-06 出现肝及⾻转移；ECOG PS 1，肝肾功能基本正常。 问题：最合理的⼀线系统治疗？ A. 吉⻄他滨 + 顺铂（GP） B. 特瑞普利单抗 + 吉⻄他滨 + 顺铂 C. 吉⻄他滨单药 D. 多⻄他赛 + 顺铂 + 5-FU …

Initial

Diagnosis Staging Treatment

New evidence

Answer: B

Follow-up

Current turn: Next best action ？ 已完成 2 程 GP 诱导及 暂停顺铂，护肾及⽀持治疗，按指征抗感染。 2 程顺铂同期放化疗。 暂约1个⽉后复诊，评估以下治疗（选项节选）： ⼝腔黏膜炎3度；肌酐 A. ⼝服替吉奥 B. 静脉紫杉醇 153.5 μmol/L，WBC C. 静脉吉⻄他滨 D. ⼝服卡培他滨 … 2.71×10^9/L，Hb 99 g/L。 Answer: D

Figure 3: Construction of the L1–L5 component tasks. Guideline questions, image tasks, and clinical checkpoints undergo specialist review of the available evidence and reference answers. They probe treatment indications, staging implications, recommendation boundaries, surveillance, and contraindications. Item accuracy is the primary endpoint. L2.A-App: Atomic guideline application. The 185 short-vignette questions cover initial definitive management, recurrence, metastatic/systemic management, and toxicity or contraindications. The main score averages accuracy over these four capability groups, giving each group equal weight. L3.MM-EG: Multimodal evidence grounding. Pathology contains 69 overview cases (39 NPCpositive and 30 negative). MRI comprises 200 sequence-recognition questions, 400 T-stage cases, and 400 N-stage cases, with patient-disjoint T and N cohorts. Within the 400 T-stage cases, a 100-patient anatomical panel provides 2,700 structure cards covering 18 base structures, evaluating both structure identification and tumor invasion assessment. Overall, we report pathology, MRI sequence, structure-identification, T-stage, and N-stage accuracy separately, with additional metrics for invasion accuracy, joint structure–invasion correctness, and evidence-grounded T-stage success. L4.C-Plan: Case-level planning. The 165 questions cover 84 patient groups and provide clinical snapshots of stage, treatment history, response, organ function, and other relevant constraints. Plausible alternative plans test whether a model identifies the next action appropriate to that specific state. Patient-macro accuracy weights patients equally. L5.L-Rev: Longitudinal revision. The 305 questions cover 100 trajectories with one to five scored checkpoints at treatment, reassessment, toxicity, recurrence, or follow-up. Each checkpoint requires interpreting new findings in the context of earlier treatment and response. We report turn-micro accuracy and all-turn patient success. The primary task supplies full history in one prompt; matchedhistory and serial-interaction controls are evaluated separately. L6.CP-Epi: CarePath episodes. The 100-patient CarePath cohort comprises 417 tasks: 100 pathology assessments, 100 MRI T-stage assessments, and 217 care decisions. Clinical records remain patient-linked; care-decision selection uses model-informed difficulty enrichment to prioritize clinical decisions that challenge models (Appendix C.3.5). Strict requires correct pathology, MRI T-stage, and every care answer; structure predictions provide auxiliary diagnostics. Each care question is answered independently with clinical context, incorporating the model’s pathology/T predictions and reference N/M/overall stages where required. O1.C-Rub: ConsultRubrics. The 103 consultation items span 42 patients and 1,726 rubric criteria: 1,069 positive requirements and 657 negative criteria. Satisfied positive requirements add weighted 5

credit; triggered major and minor errors deduct two and one points, respectively. The sum is floored at zero after item-level aggregation, normalized to 0–100, and averaged over all 103 item scores. The primary judge and physician calibration are described below.

4

Experimental Evaluation

4.1

Experimental Setup

Evaluation Models. We evaluated 30 medical and general-purpose models, including HuatuoGPTo1 [43], Lingshu [4], Qwen, GPT-OSS [44, 45], and models from the GPT and Gemini families [46, 47]. While all 30 models complete the text-based tasks (L1, L2, L4, L5 and O1), the 21 multimodal models also complete the image-grounded tasks L3 and L6 (full model inventory in Appendix C.1). Evaluation protocol. All models follow standardized task templates and scoring rules, with modelspecific image processing and generation settings detailed in Appendix C and full prompts in Appendix E. For O1 calibration, automated judges and physicians evaluate identical response sets within each panel. All locally hosted generators and judges run on four NVIDIA A800-SXM4-80GB GPUs. Patient-level Metrics. On CarePath episodes, we report both decision-level accuracy (Care) and strict patient-level success (Strict), which requires correct pathology, MRI T stage, and all selected care answers for the same patient. For a given model, let p = 1, . . . , P index patients and Sp denote PP the nonempty set of scored care decisions for patient p, with np = |Sp | and N = p=1 np . Let cpj , dp and tp ∈ {0, 1} indicate whether the care response j, the pathological label, and the T-stage MRI label are correct, respectively. With P patients and N total decisions, the percentage endpoints are P

Care =

P

100 X X cpj , N p=1

Strict =

j∈Sp

Y 100 X dp tp cpj . P p=1

(1)

j∈Sp

Appendix C.2.1 defines the remaining task metrics and 95% confidence intervals, and Appendix C.5.1 details the consultation scoring. 4.2

Guideline Knowledge and Case Planning

Table 1 summarizes model performance across all benchmark tasks. Addressing RQ1, we examine how foundational guideline competence translates into case-level planning. Although Gemini3.1-Pro-Preview leads the guideline recall with 94.0% and the atomic application with 92.5%, its precision drops to 71.9% in L4, followed by Claude-Sonnet-4.6 and Gemini-2.5-Pro. In addition, models with nearly identical L1 and L2 scores—such as GPT-5.1 and DeepSeek-V3.1—diverge sharply on L4 planning (63.5 vs. 51.8%). Although these subsets differ in item composition and weighting, this divergence shows that multi-factor case planning must be evaluated directly rather than inferred from guideline accuracy. Furthermore, a paired comparison on 71 physician-audited L1 items reveals that even high guideline scores partly rely on multiple-choice cues: removing answer options while holding the target knowledge fixed lowers accuracy by 7.0 percentage points (pp) for Gemini-3.1-Pro-Preview, 12.7 pp for GPT-4.1, and 15.5 pp for GPT-5.1 shown as in Figure. 7i. 4.3

Evidence Grounding and Longitudinal Revision

Stage accuracy and anatomical grounding reveal different weaknesses. For RQ2, we first assess whether models can assign report-derived T and N categories across separate 400-patient cohorts. The highest T-stage accuracy is 43.5% across 400 patients, compared to a baseline of 37.5% in the majority-class. N-stage accuracy reaches 42.8% across a separate 400 patients, whose N1 majority baseline is 40.0%. However, overall accuracy masks differences in performance across different stages: Qwen3.6-27B achieves the highest overall N accuracy of 42.8% but only 31.9% balanced accuracy, whereas Gemini-3.1 maintains consistent performance across stages. Thus the total stage accuracy alone can therefore obscure class-imbalance shortcuts, a pattern that persists across T-staging image formats (Table A11) and on an external 53-patient cohort from a second center (Table A12). A 100-case anatomical panel tests whether stage predictions rest on valid visual evidence. Direct T-stage accuracy reaches 43.0%, yet complete 18-structure invasion and evidence-grounded T-stage 6

Table 1: Performance across NPCBench tasks. Accuracy and success rates are percentages; C-Rub. is a 0–100 score under GPT-OSS-120B (103 consultations). A-App./C-Plan use group/patient macro averages. T-Stg./N-Stg. each cover 400 cases. L-Rev. reports Turn accuracy and All-turn success; CP-Epi. reports Care accuracy and Strict joint pathology/T/care success (100 patients, 217 care decisions). Bold/underline mark the two highest distinct displayed values. †: models used in CarePath construction. –: unavailable interface. MM-EG Model

L-Rev.

CP-Epi.

G-Mem. A-App. Path. T-Stg. N-Stg. Seq. Struct. C-Plan Turn All-turn Care Strict C-Rub. M EDICAL -S PECIFIC LLM S

HuatuoGPT-o1-72B HuatuoGPT-o1-7B Baichuan-M2-32B Fleming-R1-32B Fleming-R1-7B

88.4 77.2 22.8 85.6 84.4

73.9 52.9 2.4 37.6 60.6

– – – – –

– – – – –

HealthGPT-Pro-4B HealthGPT-Pro-8B† Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B

80.0 81.2 78.0 82.4 86.0 78.4 85.6

59.0 61.2 56.9 56.8 74.5 60.3 61.0

56.5 56.5 56.5 56.5 60.9 56.5 59.4

Qwen3-30B-A3B GPT-OSS-20B GPT-OSS-120B DeepSeek-V3.1

83.2 74.4 86.0 90.0

66.2 58.3 75.4 81.0

– – – –

Qwen3-VL-8B-Instruct† Qwen3-VL-30B-A3B-Instruct† Qwen3.5-9B Qwen3.5-27B† Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B

83.6 86.0 85.2 89.2 86.4 87.6 85.2 88.0

61.5 67.3 67.8 75.8 75.2 71.4 67.9 69.7

56.5 56.5 72.5 71.0 66.7 71.0 66.7 59.4

GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6

90.4 90.0 94.0 93.2 88.4 93.2

80.9 77.6 92.5 85.8 71.8 86.1

62.3 71.0 63.8 60.9 65.2 56.5

– – – – –

– – – – –

– – – – –

46.9 43.2 5.3 30.3 37.2

63.9 55.1 66.2 75.7 59.3

26.0 20.0 31.0 40.0 23.0

– – – – –

– – – – –

10.1 5.3 38.8 28.6 24.8

9.4 8.6 5.6 7.5 21.8 9.3 11.4

50.8 48.7 53.1 48.3 53.4 42.6 53.3

26.2 38.7 59.3 47.2 70.2 49.2 70.5

4.0 9.0 26.0 16.0 37.0 14.0 37.0

19.8 0.0 20.3 0.0 39.6 0.0 39.6 0.0 47.5 10.0 44.2 9.0 55.3 2.0

14.7 18.8 8.7 10.7 7.6 3.2 9.0

– – – –

51.3 32.1 49.4 51.8

63.3 48.9 60.3 74.4

27.0 13.0 23.0 42.0

– – – –

– – – –

26.8 28.0 44.9 28.2

4.7 36.2 38.9 47.6 50.3 41.4 44.8 3.7

52.3 52.3 58.0 61.0 54.5 50.3 46.8 52.0

69.8 68.5 60.7 76.4 72.8 79.7 76.1 58.4

38.0 36.0 24.0 46.0 41.0 53.0 49.0 21.0

52.5 47.5 53.0 62.7 63.1 65.9 61.8 47.5

2.0 6.0 1.0 1.0 1.0 5.0 5.0 0.0

25.6 29.1 34.7 46.8 41.4 47.2 42.4 9.5

63.5 54.4 71.9 67.0 59.8 69.2

75.7 78.7 87.9 82.0 69.5 85.2

45.0 45.0 71.0 56.0 42.0 63.0

71.0 2.0 61.8 2.0 89.9 11.0 80.6 15.0 65.0 6.0 80.6 23.0

58.4 33.3 47.8 38.5 34.4 47.5

M EDICAL -S PECIFIC MLLM S 38.5 39.5 34.5 12.5 29.0 37.5 37.5

20.0 20.3 19.8 20.5 20.0 3.8 20.0

54.0 50.0 30.0 58.0 75.5 47.0 75.0

O PEN -S OURCE LLM S – – – –

– – – –

– – – –

O PEN -S OURCE MLLM S 37.5 37.5 26.3 37.0 35.5 37.5 43.5 36.8

20.0 21.8 20.3 27.3 21.8 42.8 20.3 20.0

29.5 40.0 65.5 67.0 65.0 75.5 61.5 45.5

P ROPRIETARY (M)LLM S 27.3 39.8 31.0 37.5 38.0 33.5

32.8 22.0 38.5 26.0 25.8 27.3

47.0 48.5 77.0 52.0 59.0 57.0

26.3 28.3 72.0 48.6 24.5 27.9

success (requiring correct labels and final stage) both fall below 5.0% (Table A15; Fig. 7a). Correct stage labels thus frequently coexist with flawed anatomical assessments. To isolate this breakdown, we test a fixed staging rule across three evidence conditions (Fig. 7f): reference findings (Ref.), predicted invasion with known anatomy (Known), and fully predicted evidence (Pred.). With reference findings, both a deterministic rule and a fixed GPT-OSS-120B stager reach 99.0% accuracy (Tables A17 and A18), confirming that downstream staging logic is sound. However, the accuracy of Gemini-3.1 drops to 49.0% (Known) and 36.0% (Pred.). Even with correct anatomical localization, flawed invasion recognition remains the primary bottleneck. A separate 25-case static vs. cine MRI comparison yields an inconclusive T-stage difference (section C.4). Longitudinal decision accuracy does not ensure patient-level success. For RQ3, we first examine L5 performance across 305 decisions from 100 patients. Gemini-3.1-Pro-Preview achieves the highest turn accuracy (87.9%) and all-turn patient success (71.0%), followed by Claude-Sonnet-4.6 (85.2% and 63.0%, respectively; Fig. 4b). While turn accuracy measures individual decisions in isolation, all-turn success is substantially lower across models because it depends on how errors distribute across trajectories—any single misstep invalidates an entire patient episode. 7

(a) L3 staging

(b) L5 revision

(c) L6 care

(d) L6 diagnosis

(e) Joint success

100

100

100

100

50

50

50

50

50

Score (%)

100

0

0

T

N

rn Tu

Gemini-3.1

0

l-

n tur

Al

GPT-5.1

0

e ar

C

Claude-4.6

Al

e ar

l-c

Qwen3.6-27B

0

th. Pa

T

HealthGPT-8B

Lingshu-32B

t Pa

T

h.+

Al

e ar

l-c

Str

ict

Figure 4: Staging, longitudinal decisions, and patient-level correctness. (a), L3 T/N accuracy on separate 400-patient cohorts. (b), L5 decision accuracy (Turn) and all-turn success: 305 decisions, 100 patients. (c), L6 Care accuracy and All-care success: 217 decisions, 100 patients. (d), Pathology (Path.) and MRI T accuracy; majority-class baselines are 99% and 58%, respectively. (e), Joint pathology/T correctness (Path.+T), All-care, and their intersection (Strict). All-turn and All-care require every decision correct. Panels (c–e) use the same 100 patients; gray curves show other models. In paired controlled runs across the same 305 questions and 100 patients, we compare full chronological history against a snapshot that strips time markers and regroups the same records by content (Table A25). Full history yields higher turn-accuracy point estimates for 14 of 20 models, yet the patient-bootstrap 95% confidence intervals span zero for 19 models; DeepSeek-V3.1 gains 4.9 pp and Gemini-3.1 gains 3.3 pp. Net gains can hide offsetting errors: GPT-5.1 is correct on 22 questions only with full history and 19 only with the snapshot. Thus a 1.0 pp turn-accuracy gain coexists with a 1.0 pp decline in all-turn success. Three of 20 models show opposite directions for these endpoints (Fig. 7b,g). Correcting one turn may leave other errors, whereas a new error can invalidate an otherwise correct episode. To separate record order from temporal information, an order-only control shuffles complete lines while retaining all time markers. Nineteen of 20 models have confidence intervals spanning zero, with Gemini-3.1 as the sole exception (+3.0 pp; 95% CI: 0.6–5.7 pp). Tables A25 and A26 report the paired results. A separate conversation-retention comparison involving 87 patients also yields an interval spanning zero. 4.4

Consultation Quality and Physician Calibration

For RQ4, we assess whether consultations meet case requirements and avoid clinical errors. GPT5.1 has the highest ConsultRubrics score with 58.4, although the primary GPT-OSS-120B judge identifies major and minor errors in 14.6% and 40.8% of its responses. Because sound clinical advice and unsafe recommendations frequently coexist within the same free-text answer, aggregate rubric scores must be interpreted alongside explicit error rates and topic breakdowns (Fig. 5a,c,d). Because the 103 consultations span 42 underlying cases, weighting the 42 cases equally rather than individual questions shifts model scores by only −2.0 to +0.1 points and leaves GPT-5.1 highest, as shown in Fig. 5b.

Table 2: Physician–judge agreement. Agreement (%) and Cohen’s κ are measured against physician annotations. Targeted Audit (n = 548) and Expanded Audit (n = 1,013) denote the two calibration sets; n counts criterion-level labels. Targeted Audit Expanded Audit (n = 548) (n = 1,013) Judge GPT-4.1 GPT-OSS-120B Qwen3-30B Qwen3.6-27B

Agree. 77.7 81.4 73.2 79.4

κ Agree.

0.553 0.610 0.472 0.578

77.6 70.5 76.9 77.8

κ

0.547 0.387 0.538 0.544

To assess judge reliability and sensitivity to calibration-sample composition, we conduct a Targeted Audit (548 criterion labels) and an Expanded Audit (1,013 labels from 54 responses across nine models on six cases; Table 2). GPT-OSS-120B achieves the highest agreement in the Targeted Audit (81.4%, κ = 0.610), whereas Qwen3.6-27B leads the Expanded Audit (77.8%, κ = 0.544), indicating differences in judge performance across calibration samples. In the Expanded Audit, the primary GPT-OSS-120B judge agrees with physicians on 70.5% of criteria (case-resampled 95% CI: 66.2–77.7%), with 51.8% sensitivity and 85.8% specificity (Fig. A7). A positive label means that a criterion holds, including the presence of an error. Overlooking a satisfied 8

(a) Errors vs score

100

20

100

(c) Clinical topics

(d) Criterion rates Core answer

80

Case mean

C-Rub. score

r=-0.96 40

(b) Score weighting

Tag score

60

60 40

Evidence 50

Conditions No major error No minor error

20 0 20

40

60

Major-error rate (%)

0

0

50

Item mean

Gemini-3.1

100

0

GPT-5.1

AE Dx Eld Ped Tx Wk

Claude-4.6

0

50

Criteria (%)

100

Qwen3.6-27B

Figure 5: Consultation quality under the GPT-OSS-120B judge. (a), Major-error rate and C-Rub. score; major errors contribute to score deductions. (b), Item-mean and case-mean scores for the same 103 consultations from 42 cases. (c), Clinical tags: adverse events (AE), diagnosis (Dx), elderly (Eld), pediatric (Ped), treatment (Tx), and workup (Wk), with two to 43 questions per tag. (d), Criterion-level rates: core answers, supporting evidence, and conditional requirements satisfied; major and minor error criteria not triggered. These error-avoidance rates differ from the response-level major-error rate in (a). requirement withholds credit, whereas overlooking an error avoids a deduction. Figure 7e shows recognition of physician-positive labels (sensitivity) and negative labels (specificity); panel j compares agreement by criterion type. Integrating Evidence and Decisions within Patients Q For RQ3, all-care correctness is Cp = j∈Sp cpj and joint pathology–MRI T-stage correctness is Fp = dp tp (Eq. (1)). In the CarePath cohort, Gemini-3.1 achieves the highest Care accuracy (195/217 decisions, 89.9%) and All-care success (83.0%), yet its Strict success is only 11.0%. In contrast, Claude-4.6 achieves lower Care (80.6%) and All-care (64.0%) but the highest Strict success (23.0%; exact 95% CI: 15.2–32.5%; Fig. 4c–e). Thus care accuracy alone does not identify which models most often answer all diagnostic and care questions correctly for a patient.

Strict success (%)

4.5

Joint correctness identifies the patients behind this gap. For Gemini-3.1, 72 of 100 patients have all care decisions correct despite at least one incorrect pathology or MRI T-stage label; the corresponding number is 41 for Claude-4.6. Conversely, Hulu-Med32B has 38 patients with both diagnostic labels correct but at least one incorrect care decision (Fig. 6; Table A20). Care answers and diagnostic labels can therefore disagree in correctness.

(a) Care and joint success Claude-4.6

30 20

Gemini-3.1

Qwen3.6-27B

10

GPT-5.1

0 0

25

50

75

100

All-care correct (%) Other models (17)

Construction (4)

(b) Diagnostic–care agreement

Gemini-3.1 11 2 GPT-5.1 Claude-4.6 Qwen3.6-27B 5 0

72 44

7

23

41 28 25

5 12 47 15

5

21

62 50

75

100

Patients (%) Both

Care only

Image only

Neither

Figure 6: Diagnostic and care correctness in To further examine whether diagnostic evidence af- 100 patients. (a), All-care versus Strict for 21 fects care decisions, we compare predicted and ref- models; diamonds mark the four construction erence pathology, MRI T-stage, and invasion find- models. (b), Joint outcomes for four models. ings in 25 CarePath patients. Fourteen local mod- Care means all care answers are correct; Image els answer the same 146 care questions under both requires correct pathology and T labels; Both conditions, with other clinical context fixed. Patient- equals Strict. averaged care accuracy, excluding diagnostic scores, changes by −3.2 to +4.1 pp, with 12 of 14 intervals including zero (Fig. 7d; Table A23). These mixed changes show no consistent care benefit from reference evidence under the supplied clinical context. Care and All-care outcomes also differ across episode groups. Nine of 21 models have lower decision-weighted Care but higher patient-weighted All-care in the 75 assigned-pathology episodes 9

than in the 25 patient-matched episodes. Claude-4.6 scores 76.5% versus 84.3% for Care, but 68.0% versus 52.0% for All-care (Table A21). The groups contribute 102 and 115 decisions, respectively. Figure 7c,h further examines performance by decision count: 1–2/3/4–5 for L5 and 1/2/3–6 for CarePath, as patient-level success requires every scored decision to be correct.

50

0

ld Go

.

n d ow Pre Kn

10 0 −5 −10 0 5 Turn change (pp)

Evidence input Gemini-3.1

GPT-5.1

Rate (%)

40

0

(i) Answer format

1 2 3–6 n=52 n=23 n=25 Decisions per patient Claude-4.6

P O T-4 S . Q S-1 1 we 20 Q we n3- B n3 30B .6 -2 7B

0 50 100 Baseline care (%)

100

25

Sp

70

−15

(h) L6 by depth

50

Se

G

1–2 3 4–5 n=32 n=31 n=37 Decisions per patient

(g) Turn vs patient

All-turn change (pp)

T accuracy (%)

(f) Evidence to T

100

0

Agreement (%)

0 30 60 Corrected decisions

0

100

85 70 MCQ Short Answer format

Qwen3.6-27B

(j) Agreement

100

Req. Maj. Min.

70 40 P O T-4 S . Q S-1 1 we 20 Q we n3- B n3 30B .6 -2 7B

0

50

(e) Judge Se/Sp

15

G

15 30 45 T accuracy (%)

30

(d) Evidence effect

Care change (pp)

0

All-turn success (%)

2

(c) L5 by depth

100

Strict success (%)

4

(b) History effect

60

Accuracy (%)

(a) Grounded T

6

Regressed decisions

Grounded success (%)

Class imbalance also matters: constant T3 predictions score 58% in CarePath, above the best model’s 52%. The highest balanced T accuracy is 30.5%, versus 25% for constant predictions (Table A19; Fig. A4). L3 therefore complements CarePath’s joint endpoint with stage-discrimination and grounding tests.

Other models

Figure 7: Evidence, decision changes, and evaluation conditions. (a),(f), Evidence-grounded T success (Grounded T) and fixed-rule staging on 100 patients. (b),(g), Corrections, regressions, and gains from snapshot to full history. (c),(h), L5 all-turn and L6 Strict success by decision count (n: patients). (d), Care change with reference pathology, MRI T stage, and invasion findings. (i), MCQ versus short-answer accuracy on 71 matched questions; gray denotes GPT-4.1. (e),(j), Judge sensitivity/specificity (Se/Sp) and agreement on requirements (Req.), major errors (Maj.), and minor errors (Min.). Points represent models or judges; OSS-120B is GPT-OSS-120B. Changes are percentage points (pp); bars show 95% patient-cluster (d) or response-cluster (e,j) intervals.

5

Conclusion

NPCBench links foundational guideline knowledge and multimodal image interpretation to longitudinal decision-making, joint patient-level correctness, and specialist consultation in NPC care. Evaluating 30 models reveals that local accuracy rarely guarantees coherent patient management: stage predictions frequently lack complete anatomical support, longitudinal history corrects prior mistakes while introducing new errors, and high-scoring consultations still trigger critical safety violations. On CarePath episodes, accurate treatment choices routinely coexist with flawed image interpretations, capping strict patient-level success at just 23.0% despite substantially higher decisionlevel accuracy. These findings establish evidence–decision consistency as a central benchmark target: meaningful clinical progress requires models to ground interpretations in valid evidence and sustain accurate decisions across the entire course of care for the same patient. Limitations. Our evaluation is constrained by primarily single-center records, assigned pathology in 75 model-selected CarePath episodes, score sensitivity to class balance and episode length, and the six-case Expanded Audit. Future more multi-center cohorts with fully patient-matched modalities and prospective studies are therefore needed to establish whether benchmark gains improve clinical care. 10

Acknowledgments This work was supported by the National Natural Science Foundation of China under Grants 82441027 and 82441026; the Fundamental Research Funds for Sun Yat-sen University under Grant PTQN202603; the Hong Kong RGC (TRS: T41-517/25-N; GRF: 15228325); ITC RAISe+ (RAI/24/1/086A); and the Research Institute for Generative Artificial Intelligence at PolyU. We also thank Deepexi Technology Co., Ltd. for its support. Competing interests. The authors declare no competing interests.

References [1] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. [2] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950, 2025. [3] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541–28564, 2023. [4] Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044, 2025. [5] Zeyu Liu, Zhitian Hou, Guanghao Zhu, Zhijie Sang, Congkai Xie, and Hongxia Yang. Infimed: Low-resource medical mllms with advancing understanding and reasoning. arXiv preprint arXiv:2505.23867, 2025. [6] Pengkai Wang, Pengwei Liu, Qi Zuo, Zhijie Sang, Congkai Xie, and Hongxia Yang. Infimedorbit: Aligning llms on open-ended complex tasks via rubric-based incremental training. arXiv preprint arXiv:2510.15859, 2025. [7] Ahmed Alaa, Thomas Hartvigsen, Niloufar Golchini, Shiladitya Dutta, Frances Dean, Inioluwa Deborah Raji, and Travis Zack. Position: Medical large language model benchmarks should prioritize construct validity. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. [8] Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, et al. Holistic evaluation of large language models for medical tasks with medhelm. Nature Medicine, pages 1–9, 2026. [9] Wei-Wei Zhang, Pengkai Wang, Yan Li, Zhijie Sang, Guan-Qun Zhou, Jun-Yan Li, Shu-Han Zhou, Pei-Wen Liu, Ting-Xi Tang, Min Tang, et al. Patient-specific rubric-based evaluation of large language models across longitudinal cancer care. arXiv preprint arXiv:2610.35847, 2026. [10] Freddie Bray, Mathieu Laversanne, Hyuna Sung, Jacques Ferlay, Rebecca L Siegel, Isabelle Soerjomataram, and Ahmedin Jemal. Global cancer statistics 2022: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians, 74(3):229–263, 2024. [11] Yu-Pei Chen, Anthony TC Chan, Quynh-Thu Le, Pierre Blanchard, Ying Sun, and Jun Ma. Nasopharyngeal carcinoma. The Lancet, 394(10192):64–80, 2019. [12] Kenneth CW Wong, Edwin P Hui, Kwok-Wai Lo, Wai Kei Jacky Lam, David Johnson, Lili Li, Qian Tao, Kwan Chee Allen Chan, Ka-Fai To, Ann D King, et al. Nasopharyngeal carcinoma: an evolving paradigm. Nature reviews Clinical oncology, 18(11):679–695, 2021. 11

[13] Jian-Ji Pan, Hai-Qiang Mai, Wai Tong Ng, Chao-Su Hu, Jin-Gao Li, Xiao-Zhong Chen, James CH Chow, Edwin Wong, Victor Lee, Ling-Yu Ma, et al. Ninth version of the ajcc and uicc nasopharyngeal cancer tnm staging classification. JAMA oncology, 10(12):1627–1635, 2024. [14] NCCN Clinical Practice Guidelines in Oncology: Head and Neck Cancers. National Comprehensive Cancer Network, 2025. URL https://www.nccn.org/guidelines. [15] Chinese Society of Clinical Oncology. CSCO Guidelines for Diagnosis and Treatment of Nasopharyngeal Carcinoma. People’s Medical Publishing House, Beijing, 2024. [16] KC Allen Chan, John KS Woo, Ann King, Benny CY Zee, WK Jacky Lam, Stephen L Chan, Sam WI Chu, Constance Mak, Irene OL Tse, Samantha YM Leung, et al. Analysis of plasma epstein–barr virus dna to screen for nasopharyngeal cancer. New England Journal of Medicine, 377(6):513–522, 2017. [17] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. [18] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022. [19] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2567–2577, 2019. [20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [21] Yan Cai, Linlin Wang, Ye Wang, Gerard De Melo, Ya Zhang, Yanfeng Wang, and Liang He. Medbench: A large-scale chinese benchmark for evaluating medical large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17709–17717, 2024. [22] Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, et al. Cmb: A comprehensive medical benchmark in chinese. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6184–6205, 2024. [23] Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36:52430–52452, 2023. [24] Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025. [25] Xiang Feng, Wentao Jiang, Zengmao Wang, Yong Luo, Pingbo Xu, Baosheng Yu, Hua Jin, and Jing Zhang. Anessuite: A comprehensive benchmark and dataset suite for anesthesiology reasoning in llms. arXiv preprint arXiv:2504.02404, 2025. [26] Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1): 180251, 2018. [27] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semanticallylabeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021. 12

[28] Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020. [29] Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024. [30] Shengyuan Liu, Boyun Zheng, Wenting Chen, Zhihao Peng, Zhenfei Yin, Jing Shao, Jiancong Hu, and Yixuan Yuan. EndoBench: A comprehensive evaluation of multi-modal large language models for endoscopy analysis. arXiv preprint arXiv:2505.23601, 2025. URL https://arxiv. org/abs/2505.23601. [31] Jingkun Yue, Siqi Zhang, Zinan Jia, Huihuan Xu, Zongbo Han, Xiaohong Liu, and Guangyu Wang. Medsg-bench: A benchmark for medical image sequences grounding. arXiv preprint arXiv:2505.11852, 2025. [32] Tianhong Zhou, Yin Xu, Yingtao Zhu, Chuxi Xiao, Haiyang Bian, Lei Wei, and Xuegong Zhang. Drvd-bench: Do vision-language models reason like human doctors in medical image diagnosis? arXiv preprint arXiv:2505.24173, 2025. [33] Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. Advances in Neural Information Processing Systems, 37: 140334–140365, 2024. [34] Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin QuiñoneroCandela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. [35] Zhiling Yan, Dingjie Song, Zhe Fang, Yisheng Ji, Xiang Li, Quanzheng Li, and Lichao Sun. Livemedbench: A contamination-free medical benchmark for llms with automated rubric evaluation. arXiv preprint arXiv:2602.10367, 2026. [36] Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, et al. Llmeval-med: a real-world clinical benchmark for medical llms with physician validation. arXiv preprint arXiv:2506.04078, 2025. [37] Lecheng Gong, Weimin Fang, Ting Yang, Dongjie Tao, Chunxiao Guo, Peng Wei, Bo Xie, Jinqun Guan, Zixiao Chen, Fang Shi, et al. Meddialogrubrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models. arXiv preprint arXiv:2601.03023, 2026. [38] Xiang Zheng, Han Li, Wenjie Luo, Weiqi Zhai, Yiyuan Li, Chuanmiao Yan, Tianyi Tang, Yubo Ma, Kexin Yang, Dayiheng Liu, et al. Clinconsensus: A consensus-based benchmark for evaluating chinese medical llms across difficulty levels. arXiv preprint arXiv:2603.02097v1, 2026. URL https://arxiv.org/abs/2603.02097v1. [39] Xian Wu, Yutian Zhao, Yunyan Zhang, Jiageng Wu, Zhihong Zhu, Yingying Zhang, Yi Ouyang, Ziheng Zhang, Huimin Wang, Zhenxi Lin, et al. Medjourney: Benchmark and evaluation of large language models over patient clinical journey. Advances in Neural Information Processing Systems, 37:87621–87646, 2024. [40] Yixing Jiang, Kameron C Black, Gloria Geng, Danny Park, James Zou, Andrew Y Ng, and Jonathan H Chen. MedAgentBench: a virtual EHR environment to benchmark medical LLM agents. Nejm Ai, 2(9):AIdbp2500144, 2025. [41] Yinghao Zhu, Ziyi He, Haoran Hu, Xiaochen Zheng, Xichen Zhang, Zixiang Wang, Junyi Gao, Liantao Ma, and Lequan Yu. Medagentboard: Benchmarking multi-agent collaboration with conventional methods for diverse medical tasks. arXiv preprint arXiv:2505.12371, 2025. 13

[42] Kiril Vasilev, Alexandre Misrahi, Eeshaan Jain, Phil F Cheng, Petros Liakopoulos, Olivier Michielin, Michael Moor, and Charlotte Bunne. Mtbbench: A multimodal sequential clinical decision-making benchmark in oncology. arXiv preprint arXiv:2511.20490, 2025. [43] Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925, 2024. [44] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [45] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. [46] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [47] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [48] Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37:94327–94427, 2024. [49] Yang Zhou, Tan L Faith, Yanyu Xu, Sicong Leng, Xinxing Xu, Yong Liu, and Rick S Goh. Benchx: A unified benchmark framework for medical vision-language pretraining on chest x-rays. Advances in Neural Information Processing Systems, 37:6625–6647, 2024. [50] Yuli Wang, Jian Peng, Yuwei Dai, Craig Jones, Haris Sair, Jinglai Shen, Nicolas Loizou, Jing Wu, Wen-Chi Hsu, Maliha Imami, et al. Enhancing vision-language models for medical imaging: bridging the 3d gap with innovative slice selection. Advances in Neural Information Processing Systems, 37:99947–99964, 2024. [51] Jing Hao, Yuxuan Fan, Yanpeng Sun, Kaixin Guo, Lizhuo Lin, Jinrong Yang, Qi Yong H Ai, Lun M Wong, Hao Tang, and Kuo Feng Hung. Towards better dental ai: A multimodal benchmark and instruction dataset for panoramic x-ray analysis. arXiv preprint arXiv:2509.09254, 2025. [52] Juntao Jiang, Jiangning Zhang, Yali Bi, Jinsheng Bai, Weixuan Liu, Weiwei Jin, Zhucun Xue, Yong Liu, Xiaobin Hu, and Shuicheng Yan. M3cotbench: Benchmark chain-of-thought of mllms in medical image understanding. arXiv preprint arXiv:2601.08758, 2026. [53] Zonghai Yao, Benlu Wang, Yifan Zhang, Junda Wang, Iris Xia, Zhipeng Tang, Shuo Han, Feiyun Ouyang, Zhichao Yang, Arman Cohan, et al. Medical thinking with multiple images. arXiv preprint arXiv:2604.16506, 2026. [54] Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 599–621, 2024. [55] Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae W Park. Mdagents: An adaptive collaboration of llms for medical decision-making. Advances in Neural Information Processing Systems, 37:79410–79452, 2024. 14

[56] Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, et al. Medagentsbench: Benchmarking thinking models and agent frameworks for complex medical reasoning. arXiv preprint arXiv:2503.07459, 2025. [57] Dyke Ferber, Omar SM El Nahhas, Georg Wölflein, Isabella C Wiest, Jan Clusmann, MarieElisabeth Leßmann, Sebastian Foersch, Jacqueline Lammert, Maximilian Tschochohei, Dirk Jäger, et al. Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology. Nature cancer, 6(8):1337–1349, 2025. [58] Samuel Schmidgall, Rojin Ziaei, Carl Harris, Ji Woong Kim, Eduardo Pontes Reis, Jeffrey Jopling, and Michael Moor. AgentClinic: a multimodal benchmark for tool-using clinical AI agents. npj Digital Medicine, 2026. [59] Luyang Luo, Sung Eun Kim, Xiaoman Zhang, Julius M Kernbach, Roshan Kenia, Julian N Acosta, Larry A Nathanson, Adrian D Haimovich, Adam Rodman, Ethan Goh, et al. A clinical environment simulator for dynamic ai evaluation. Nature medicine, pages 1–8, 2026.

15

Appendix Contents A

Data Curation and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.1

Task scope and composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

A.2

MRI cohorts and patient coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

A.3

CarePath episodes and consultation criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

B

Detailed Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

C

Evaluation Protocol and Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 C.1

Evaluation models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

C.2

Task definitions and scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

C.3

Extended results and diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.3.1

Longitudinal decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

C.3.2

MRI T and N staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

C.3.3

Anatomical evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

C.3.4

Staging ablations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

C.3.5

CarePath patient outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

C.4

Controlled input comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

C.5

Consultation scoring and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

D

Clinical Review and Judge Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

E

Evaluation Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

F

Illustrated Task Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

A

Data Curation and Statistics

The following sections describe the clinical sources, question construction, patient coverage, and reference annotations. Clinical data and ethics. The retrospective use of clinical data was approved under IRB protocol B2026-465-01. Patient materials follow the hospital’s de-identification procedure. Direct identifiers and unnecessary location details are removed, and dates are expressed relative to the patient’s clinical course where appropriate. L4 retains clinical decision dates needed to identify the applicable guideline edition. Age and sex are retained when needed for reasoning. NPC oncology specialists review clinical validity, reference answers, supporting evidence, and transitions between care decisions. Task construction. L1 and L2 translate guideline knowledge and decision thresholds into questions with adjudicated answers. L4 combines diagnosis, stage, treatment history, response, toxicity, and patient-specific constraints into fixed-state planning cases. L5 organizes longitudinal records around clinical decision checkpoints, retaining the cumulative evidence available at each point and removing subsequent orders or outcomes that reveal the target decision. L6 links pathology, MRI, and selected care decisions within patient episodes. O1 uses consultation scenarios with separate criteria for core conclusions, supporting evidence, conditional reasoning, and clinically important errors. L1 retains item-level guideline references and supporting evidence. All 350 L2/L4 items include their construction sources and reviewed answer rationales. Source records distinguish the guideline edition stated in a question from references supporting its answer: the former is available to the model, whereas the latter explains the reference decision. Recorded edition constraints occur in two L2 and four L4 questions and are included in the model inputs; edition and section references are available for a subset of items.

16

Clinical review and answer format. The evaluated sets contain 250 L1, 185 L2, and 165 L4 questions. Reviewers checked the evidence available at the decision point, the applicable guideline, and whether the options supported a determinate answer. L2 includes one explicit multiple-select item and L4 includes two; the other items are single choice. Multiple-select answers must match the complete reference set. The question specifies this requirement, and selecting only part of the set receives no credit. Reviewed rationales accompany all 350 L2/L4 items. Construction examples. Figure 3 shows shortened examples of the released task formats. The examples retain the answer labels and decision-relevant information; full inputs and reviewed rationales accompany the dataset. Data and code availability. De-identified benchmark inputs, multimodal assets, reference annotations, and supplementary control datasets are publicly available on Hugging Face; evaluation code and inference scripts are available on GitHub. The released tasks include the CarePath cohort with 100 patients and 217 scored care decisions. Author-owned code is licensed under MIT and original data and annotations under CC BY 4.0; third-party materials retain their own terms. A.1

Task Scope and Composition Table A1: Benchmark scope, data handling, and intended use.

Item

Description

Measurement target

Guideline application, image interpretation, selected care decisions, and consultation quality in NPC; component scores and joint patient correctness are reported separately. Research benchmarking, model comparison, clinical error analysis, and evaluation-method development. Direct diagnosis, treatment recommendation, triage, or patient-specific clinical decision-making without physician supervision. Guideline-derived questions, expert-authored vignettes, de-identified patient records, pathology/MRI evidence, longitudinal checkpoints, selected CarePath decisions, and consult rubrics. Names, contact details, addresses, and institution-specific identifiers are removed. Dates use relative clinical time where appropriate; L4 retains dates needed to determine the applicable guideline edition. Clinically relevant age, sex, and time intervals are preserved. Shared prompts, adjudicated reference answers, final-answer extraction, O1 judge comparisons, and specialist review of answer clarity. Disease-specific retrospective data, 2D MRI cards, assigned pathology specimens in 75 CarePath episodes, variable scoring depth, and dependence on the automated consultation judge.

Intended use Out-of-scope use Data boundary

Privacy handling

Evaluation controls Known limitations

Evaluation cohorts. The L6 cohort contains 100 patients and 417 scored tasks: 100 pathology assessments, 100 MRI T-stage assessments, and 217 care decisions. Image-evidence replacement and serial-interaction analyses use 146 care questions for 25 patients within this cohort, as described in section C.4. These analyses use a fixed 146-question set, while the main benchmark scores 217 selected care decisions across all 100 patients. The L1 guideline-memory set covers CSCO-only, NCCN-only, and dual-source questions. Among the 250 L1 items, 153 are supported by both CSCO and NCCN sources, 80 are CSCO-only, and 17 are NCCN-only. The L2 atomic-application set is organized into four capability families: initial definitive management (84 items), post-treatment recurrence (30), metastatic/systemic management (44), and toxicity or contraindication management (27). Task categories. Figure 2 uses the four L2 scoring groups. O1 contains 43 treatment, 35 adverseevent, 11 diagnosis, 12 special-population, and two workup questions; the special-population group comprises four elderly and eight pediatric items. The figure and score breakdown therefore refer to the same item tags. The figure abbreviates recurrent/metastatic disease as Rec./met., recurrence as Recur., radiotherapy as RT, treatment as tx, progression as Progress., planning/execution as Plan/exec.,

17

metastatic disease as Metast., toxicity/supportive care as Tox./supp., and special populations as Special pop. EBV denotes Epstein–Barr virus; M0 denotes no distant metastasis. The L4 case-level planning set covers initial treatment (16 items), disease progression (43), regimen switching (100), and toxicity management (6), with patient-level aggregation over 84 patients. Table A2: Task scale, source materials, and reference annotation. Subset

Evaluation unit

L1

Source material

Reference annotation

Guideline-memory MCQ 250 questions

NPC guidelines

L2

Short-vignette MCQ

185 questions

Clinical vignettes

L3

Pathology/MRI multimodal tasks

Pathology and MRI images

L4

Case-level MCQ

L5

Sequential patient-turn MCQ Selected CarePath decisions

69 pathology overview questions; 2,700 evaluated MRI cards; 400 T-stage cases; 400 N-stage cases; 200 MRI sequence questions 165 questions over 84 patients 100 real patients; 305 questions 100 patients; 417 scored tasks (100 pathology, 100 MRI T, 217 care) 103 questions across 42 cases

Expert-validated guideline knowledge Specialist-reviewed answers Expert-validated pathology, imaging, and staging labels

L6

O1

A.2

Open-ended consult rubric item

Scale

Clinical vignettes

Specialist-reviewed plans

De-identified longitudinal Physician-adjudicated records decision turns Clinical episodes Adjudicated care decisions; 75 pathology proxies Clinical consultation scenarios

Structured scoring anchors

MRI Cohorts and Patient Coverage

The T-stage cohort contains 400 patients, with T1/T2/T3/T4 counts of 50/50/150/150. The patientdisjoint N-stage cohort contains 400 patients, with N0/N1/N2/N3 counts of 80/160/80/80. Both use eighth-edition report-derived staging labels. The T cohort includes 374 M0, 14 M1, and 12 patients with unspecified M status; all 400 N-stage patients are M0. The 100-patient anatomical panel is contained within the T-stage cohort. The 200 sequence-recognition patients overlap the T and N cohorts in 67 and 59 patients, respectively, giving 874 unique patients across the main L3 MRI tasks. Pathology is counted separately as 69 specimens. Patient coverage is calculated from the union of the linked MRI records. T-stage examinations span March 2021 to June 2024, as do N-stage examinations. The external MRI examinations were acquired at Nanfang Hospital from January 2023 to December 2024. Fifty-three examinations have a definite report-derived T label for scoring; two without a definitive reference are excluded. The external class counts are 4/8/31/10 for T1/T2/T3/T4. L2 contains 47 clinical source cases and 138 constructed scenarios. L4, L5, L6, and O1 contain 84, 100, 100, and 42 task-level patient records, respectively. These task-level counts include cross-task overlap and therefore do not define a unique benchmark-wide patient total. Clinical records used in the EHR tasks span 2008–2025; these are source-record dates, not a common follow-up interval. A.3

CarePath Episodes and Consultation Criteria

Table A3 separates the two sources of CarePath cohort episodes. The corresponding clinical review is described in Appendix D; Table A4 reports the number of decisions per patient.

18

Table A3: Membership and evidence linkage within the CarePath cohort. All 100 patients have matched MRI and clinical records; the image-label comparison uses 146 fixed questions for the 25 patients with matched pathology. Episode source

Patients Care nodes Pathology evidence

Treatment-selection coverage

Matched pathology

25

115 Patient-matched specimens

Assigned pathology

75

102 Assigned specimen proxies

CarePath cohort

100

217 25 matched; 75 proxy

Includes selected initial-treatment decisions No scored initial-treatment-selection question Six treatment items are excluded from scoring

Table A4: Scoring depth and the weight of each patient group. Depth is the number of scored decisions for one patient. Patient-level success weights patients equally; turn/node accuracy weights the scored decisions equally. L5 and the CarePath cohort are separate evaluation cohorts. L5

CarePath cohort

Depth

Patients

Turns

Patients

Care nodes

Node share (%)

1 2 3 4 5 6

13 19 31 24 13 0

13 38 93 96 65 0

52 23 3 7 8 7

52 46 9 28 40 42

24.0 21.2 4.1 12.9 18.4 19.4

Total

100

305

100

217

100.0

Table A5: Consultation questions and rubric criteria by clinical group. Positive requirements comprise core answers, supporting evidence, and conditional recommendations. The 1,069 positive and 657 error criteria sum to 1,726. Clinical group

Questions

Core

Evidence

Conditional

Major

Minor

Treatment Adverse-event management Diagnosis Special populations Workup

43 35 11 12 2

182 127 31 53 8

171 105 51 49 10

152 69 22 36 3

140 104 52 36 5

148 92 37 34 9

Total

103

401

386

282

337

320

B

Detailed Related Work

Static medical QA and broad medical suites. Examination-style benchmarks such as MedQA, MedMCQA, PubMedQA, the medical subsets of MMLU, CMB, and CMExam have provided a practical basis for assessing medical knowledge and localized reasoning skills [17–20, 22, 23]. MultiMedQA and MedBench extended this approach by combining multiple medical QA datasets and testing more general medical reasoning across them [2, 21]. More recent benchmarks—including MedHELM, MedXpertQA, and AnesSuite’s AnesBench—emphasize construct validity, specialtyspecific reasoning, and clinical abilities alongside exam-style performance [8, 24, 25]. NPCBench focuses on NPC and reports component performance and joint patient correctness separately. Multimodal medical evidence grounding. Medical MLLM evaluation has progressed beyond image-question datasets like VQA-RAD, SLAKE, and PathVQA toward benchmarks with broader modality coverage and greater clinical specialization [26–28]. Frameworks such as OmniMedVQA, GMAI-MMBench, BenchX, CARES, and BrainMD assess dimensions including the breadth of

19

medical imaging modalities, model reliability, and cross-modality generalization [29, 33, 48–50]. More domain-specific benchmarks—EndoBench, DrVD-Bench, MedSG-Bench, MMOral, M3CoTBench, and MedThinkVQA—target fine-grained image understanding, temporal or sequence-level grounding, and reasoning across multiple images [30–32, 51–53]. NPCBench examines these components in NPC, where pathology underpins diagnosis, MRI anatomy delineates local spread, sequence recognition shapes image interpretation, and assessment of local invasion guides staging and treatment intensity. Rubric-based and real-world clinical response evaluation. Open-ended clinical evaluation is increasingly grounded in explicit scoring rubrics and physician-validated criteria. HealthBench assesses multi-turn health-related dialogues using item-specific rubrics; LiveMedBench focuses on temporally separated clinical cases; LLMEval-Med incorporates physician adjudication combined with checklist-guided evaluation; MedDialogRubrics and ClinConsensus extend rubric-based assessment frameworks to consultation and longitudinal care scenarios [34–38]. O1 ConsultRubrics uses the same criterion-level approach for NPC consultation, in which positive requirements capture key clinical conclusions, supporting patient evidence, and conditions that change the recommendation, whereas negative criteria differentiate between major and minor safety-critical errors. Workflow, longitudinal, and agentic clinical benchmarks. Workflow evaluations test information gathering, interaction, and decisions across multiple steps. MedJourney structures evaluation around the patient journey, whereas MedAgents, MDAgents, MedAgentsBench, and MedAgentBoard focus on collaborative problem-solving, tool integration, and multi-step clinical task execution [39, 41, 54– 56]. In oncology, MTBBench emulates multidisciplinary tumor-board decision-making, and recent work on autonomous oncology agents investigates specialized workflows for treatment decision support [42, 57]. AgentClinic evaluates tool use and information gathering in simulated clinical encounters, while MedAgentBench tests tasks in a FHIR-compatible electronic health record [40, 58]. Clinical-environment simulators additionally expose models to explicit state transitions [59]. NPCBench evaluates related questions within NPC: L5 tests decisions under different representations of longitudinal records, and L6 checks image labels and selected care answers for the same patient. Its primary care evaluation supplies clinical context without feeding previous care answers back, whereas separate serial controls test conversation retention. This distinction separates joint patient correctness from autonomous interaction with a clinical environment.

C

Evaluation Protocol and Additional Results

The protocol specifies each task’s inputs, model interface, and scoring rule. L1/L2/L4/L5 use closedchoice answers; L3 uses diagnostic labels and structured image findings; L6 combines pathology and T-stage labels with care decisions; and O1 uses criterion-level judgments of free-text consultations. Table A6 summarizes the inputs and scoring populations for the linked and matched analyses. Table A6: Inputs and endpoints in the linked and matched evaluations. The CarePath cohort contains 417 scored tasks, including 217 care decisions. Image-evidence replacement and serialinteraction analyses evaluate 146 care questions for 25 patients within this cohort. Evaluation

Scored units

Evidence and history

Endpoint

L3 direct staging

400 T-stage and 400 N-stage patients 100 patients; 217 care decisions

Structure cards or same-plane sequence groups; AJCC/UICC edition 8 AJCC edition 9 MRI; predicted pathology and T-stage labels and supplied clinical context; no prior care answers Predicted versus reference image findings; other clinical context fixed Reference image labels; persistent versus stateless interaction Full history versus a snapshot with historical time markers removed and records regrouped

T and N accuracy; anatomical diagnostics on 100 T cases

CarePath cohort

Image-evidence control Serial control L5 history control

25 patients; 146 care decisions 25 patients; 146 care decisions 100 patients; 305 turns

20

Decision-weighted Care; patient-level Strict

Paired change in patient-macro care accuracy Paired change in all-care correctness Turn-accuracy change and all-turn correctness

Computing resources and generation settings. Local experiments used institutional NVIDIA A800-SXM4-80GB servers with up to eight GPUs; the multimodal experiments also used a four-GPU server. The local environment used CUDA 12.8, PyTorch 2.10.0, vLLM 0.19.1, Transformers 5.6.2, and Accelerate 1.13.0. Models were run through vLLM or model-specific multimodal interfaces, with parallelism selected according to model size. Task-specific settings specify the model interface, image processing, sampling parameters, and output limits. CarePath evaluation uses maximum output lengths of 1,024 tokens for local models and 8,192 for API models. Missing or incomplete final answers receive zero credit. For L5 primary and paired history evaluations, GPT-OSS-20B allows 16,384 output tokens and HealthGPT-Pro-4B 32,768. Fleming-R1-7B allows up to 16,384 within its 32,768-token context; each matched history/snapshot/shuffled triplet uses the same output allowance. O1 uses output limits of 2,048 tokens for the 23 local generators and 16,384 tokens for the seven API generators. Its primary judge uses temperature 0 and top-p 1, with a 2,048-token output limit and an 8,192-token retry for incomplete criterion judgments. All 30 models have valid judgments for all 1,726 criteria. The released evaluation code specifies the per-model settings. Answer extraction. Closed-answer tasks use a common final-answer rule. An explicit final boxed or labelled answer takes precedence over options discussed in intermediate reasoning; a standalone option is accepted when unambiguous. Missing or conflicting final answers are scored incorrect. Multimodal tasks extract the requested diagnostic and anatomical fields. O1 averages all 103 itemlevel rubric scores for every model. MRI inputs and generation. T staging uses two image presentations: 100 patients have 27 structure-centred cards per patient, and 300 have ordered groups of physically matched MRI planes. N staging uses the latter presentation for all 400 patients. Within an axial group, T1, T2, and enhanced T1 images are stacked vertically; a coronal group pairs enhanced fat-suppressed T1 with precontrast non-fat-suppressed T1. Each slice retains a 512 × 512 display tile. Axial and coronal groups are 512 × 1644 and 512 × 1096 lossless PNGs, respectively. A request contains one patient’s ordered groups, with 44–79 groups per case. The T range extends from the suprasellar cistern to inferior C3; the N range extends from the skull base to the inferior sternoclavicular joint. Clinicians reviewed the selected anatomical coverage and report labels. Qwen, Lingshu, HealthGPT, and Hulu-Med-4B use a 128,000-token context and an 8,192-token output budget for the plane-group tasks. Hulu-Med-4B uses spatial merge 2 with a 98,304 visualtoken budget; Hulu-Med-7B/32B use merge 4, a 22,000 visual-token budget, and their 32,768-token context. Hulu images enter the processor as height–width–channel arrays. InternVL3.5 uses its 40,960-token context, dynamic cropping with at most four patches per group, and 33–35 uniformly sampled groups. Sampling allocates images proportionally between axial and coronal groups and retains both endpoints of each group. API inputs preserve the same plane-group pixels: GPT-5.1 uses lossless WebP and the other services use lossless PNG. GPT-5.1 uses snapshot gpt-5.1-2025-11-13 with reasoning effort set to none. GPT-4.1 receives at most 50 groups, sampled proportionally within each anatomical group with both endpoints retained; the other API models receive all groups. Output limits are 4,096 tokens for GPT-4.1 and Claude, 8,192 for GPT-5.1 and Gemini Flash, and 16,384 for Gemini Pro. Empty Claude answers are retried at most three times; incomplete structured answers use 8,192 output tokens. Answers outside the requested categories are scored incorrect. Table A11 separates T accuracy by presentation, so the combined score can be interpreted alongside its two constituent cohorts. CarePath cohort uses its own ten-page MRI input and ninth-edition prompt. C.1

Evaluation Models

Table A7 lists all 30 evaluated models. The nine text-only models cover the five text-based subsets; the 21 image-capable models additionally complete pathology, MRI, and CarePath evaluation. The inventory spans medical and general-purpose models, with local inference for open-weight models and API access for hosted services. The four models used in the CarePath cohort construction are marked throughout the results.

21

Table A7: Evaluated models and task interfaces. Text models cover L1/L2/L4/L5/O1; imagecapable models additionally cover L3 and L6. † marks the four models used during CarePath cohort construction. Model

Input interface

Inference

Medical-Specific LLMs HuatuoGPT-o1-72B HuatuoGPT-o1-7B Baichuan-M2-32B Fleming-R1-32B Fleming-R1-7B

Text Text Text Text Text

Local Local Local Local Local

Medical-Specific MLLMs HealthGPT-Pro-4B HealthGPT-Pro-8B† Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B

Text and image Text and image Text and image Text and image Text and image Text and image Text and image

Local Local Local Local Local Local Local

Open-Source LLMs Qwen3-30B-A3B GPT-OSS-20B GPT-OSS-120B DeepSeek-V3.1

Text Text Text Text

Local Local Local API

Open-Source MLLMs Qwen3-VL-8B-Instruct† Qwen3-VL-30B-A3B-Instruct† Qwen3.5-9B Qwen3.5-27B† Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B

Text and image Text and image Text and image Text and image Text and image Text and image Text and image Text and image

Local Local Local Local Local Local Local Local

Proprietary (M)LLMs GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6

Text and image Text and image Text and image Text and image Text and image Text and image

API API API API API API

C.2

Task Definitions and Scoring

C.2.1

Notation and Aggregation

All definitions apply to one evaluated model on one stated cohort; the model index is suppressed. An indicator 1{E} equals one if statement E is true and zero otherwise. A sum counts contributions, a product of binary indicators requires every factor to be one, and |A| denotes the number of elements in a finite set A.

22

Table A8: Task interfaces and scoring units used in the main NPCBench table. Formal definitions follow in sections C.2.1 and C.5.1. Subset

Input

Response

Scoring unit and endpoint

L1 L2 L3

Guideline MCQ Compact vignette and MCQ Pathology overview; MRI panels or structure cards Single-case history and MCQ Longitudinal history through the current checkpoint Images, clinical context, and care questions Case and consultation question

Option letter Option letter(s) Diagnostic labels and structured findings Option letter(s) Option letter

Item accuracy over 250 questions Macro accuracy over four capability groups Pathology: case; T/N stage: patient; sequence: item; structure: card accuracy Patient-macro accuracy over 84 patients Turn accuracy and all-turn patient success; 305 turns, 100 patients Care-node accuracy and strict patient success; 217 nodes, 100 patients Mean normalized rubric score over 103 questions

L4 L5 L6 O1

Diagnostic labels and option letters Free-text consultation

Table A9: Notation for closed-answer and patient-level evaluation. Each task uses the same finalized cohort and scoring set for every evaluated model. Symbol

Meaning and domain

I, n = |I| yi , ŷi , ai P, p Sp , np , N cpj dp , tp

Scored item set and its size; i ∈ I indexes one question, image item, or turn. Reference label, parsed model label, and correctness ai = 1{ŷi = yi } ∈ {0, 1}. Number of evaluated patients and patient index p ∈ {1, . . . , P }. P Scored L6 care-node set for patient p, its size np = |Sp | ≥ 1, and total scored nodes N = p np . Correctness of L6 care node j ∈ Sp for patient p; a binary indicator. Binary correctness indicators for pathology and MRI T stage; these symbols are not the predicted class labels. Q All-care correctness Cp = j∈Sp cpj and joint image-label correctness Fp = dp tp ; both are binary.

Cp , Fp

Missing or conflicting extracted answers have correctness zero and remain in the scored denominator. The six L6 treatment questions excluded for insufficient decision evidence are omitted from Sp for every model. The CarePath cohort has P = 100 and N = 217. The image-label analysis uses 146 care questions for 25 patients within this cohort. For closed-answer tasks, the item-micro accuracy and group-macro accuracy are Accmicro =

100 X ai , n

G

Accgroup =

i∈I

100 X 1 X ai . G g=1 |Ig |

(A1)

i∈Ig

Here the G nonempty, disjoint sets Ig partition I. Micro accuracy weights each scored item equally; group macro weights each group equally. L2 uses G = 4 clinical capability groups. For patient aggregation, let Ip contain the scored items belonging to patient p in the task being evaluated: P P P 100 X i∈Ip ai 100 X Y Accpatient = , AllCorrect = ai . (A2) P p=1 |Ip | P p=1 i∈Ip

Each Ip is nonempty. L4 reports patient-macro accuracy over 84 patients; L5 reports turn-micro accuracy and AllCorrect as all-turn success over 100 patients. For L6, Ip = Sp and ai = cpj under the corresponding node indexing; AllCorrect is called AllCare. Its primary Care and Strict endpoints follow Eq. (1); AllCare omits the pathology and T-stage requirements.

23

Patient intersections. Let C = {p : Cp = 1} and F = {p : Fp = 1} be patient sets, and let superscript c denote a complement within {1, . . . , P }. Table A20 reports set cardinalities. Equivalently, the four counts are nab =

P X

1{Cp = a, Fp = b},

a, b ∈ {0, 1},

p=1

1 X 1 X

nab = P.

(A3)

a=0 b=0

Thus n11 = |C ∩ F|, n10 = |C ∩ F c |, n01 = |C c ∩ F|, and n00 = |C c ∩ F c |. Strict is 100n11 /P ; AllCare is 100(n11 + n10 )/P . All four groups refer to the same patients. C.2.2

Classification Diagnostics

For a target class k, TPk counts correctly predicted members of that class, FNk counts missed members, and FPk counts other examples predicted as class k. Class recall Rk , precision Qk , and secondary F 1k are proportions: Rk =

TPk , TPk + FNk

Qk =

TPk , TPk + FPk

F 1k =

2TPk . 2TPk + FPk + FNk

(A4)

These ratios require nonzero denominators; undefined diagnostic ratios are unavailable. For K PK evaluated classes with reference support, balanced accuracy is 100K −1 k=1 Rk . T staging uses K = 4 (T1–T4), so a majority-class predictor has 25% balanced accuracy. Per-class recalls shown as percentages equal 100Rk . L3 structure-card accuracy and these class-balanced measures have different aggregation units; the main structure column uses card-micro accuracy. C.2.3

Paired Effects and Uncertainty (0)

(1)

For the binary endpoint in Eq. (A6), index the M matched units by i. Let ai and ai be correctness P (0) (1) under baseline and intervention, respectively. The counts are u = i 1{ai = 0, ai = 1} and P (0) (1) v = = 1, ai = 0}. The remaining M − u − v units retain the same correctness. i 1{ai Consequently, ∆ ∈ [−100, 100] is a percentage-point difference and Φ ∈ [0, 100] is a percentage of correctness flips; changes between two incorrect answers do not contribute to either count. The image-label comparison instead compares within-patient care accuracies. For condition k ∈ P (k) (k) {0, 1}, define the proportion Ap = n−1 p j∈Sp cpj . With k = 0 for predicted image labels and k = 1 for reference image labels, its reported effect is P

 100 X  (1) ∆patient = Ap − A(0) . p P p=1

(A5)

P

p np = 146. The better, worse, and same columns in Table A23 count patients (1) (0) according to the sign of Ap − Ap . The patient-macro effect also incorporates each patient’s change

Here P = 25 and in accuracy.

Confidence intervals. Care accuracy and paired patient comparisons use a patient-cluster bootstrap. Each draw resamples patients with replacement, retaining all of a sampled patient’s scored units and both conditions of a pair. Each draw recomputes the relevant estimator: a ratio of turn totals for micro accuracy or a mean of patient accuracies for patient macro. The 2.5th and 97.5th percentiles form the reported 95% interval. L1 controls resample paired items; physician-calibration intervals resample consultation-response clusters. CarePath cohort strict-success intervals are exact binomial (Clopper–Pearson) intervals over 100 patients. Multi-model intervals are descriptive and unadjusted. C.3

Extended Results and Diagnostic Analyses

The family profiles in Fig. A1 show that high guideline scores coexist with low strict patient-level success across model groups. Figure A2 retains the differences between individual models. We then examine longitudinal decisions, anatomical evidence, staging, and patient-level outcomes before comparing controlled changes to the inputs.

24

(a) Family median profile

(b) Best score per task

Med.-Specific LLMs 84 (n=5)

53

Med.-Specific MLLMs 81 (n=7)

60

35

51

49

16

40

0

9

86

74

41

53

70

37

55

10

19

Open-Source LLMs 85 (n=4)

71

--

50

62

25

--

--

28

90

81

--

52

74

42

--

--

45

Open-Source MLLMs 86 (n=8)

69

46

52

71

40

57

2

38

89

76

54

61

80

53

66

6

47

Proprietary (M)LLMs 92 (n=6)

83

42

65

80

50

76

8

43

94

93

56

72

88

71

90

23

58

M

G-

--

37

64

26

--

--

25

88

74

--

47

76

40

--

--

39

100

60 40

t em - App -EG -Plan turn l-turn care stric -Rub M C C A L5 5 al L6 L6 M L

Score

80

20 0

t em - App -EG -Plan turn l-turn care stric -Rub M C C A L5 5 al L6 L6 M L

M

G-

Figure A1: Family-level capability profile. Family medians (a) and task-wise best scores within each family (b); the latter may come from different models. MM-EG averages five L3 endpoints; L6 uses the CarePath cohort. Per-model results appear in Table 1 and Fig. A2. C.3.1

L5 Longitudinal Revision Results

L5 reports turn accuracy and all-turn patient success because the two measures answer different questions. Turn accuracy counts correct next actions, whereas all-turn success requires every scored answer for one patient to be correct. All models use the same boxed single-letter answer interface. The 100 patients have 13/19/31/24/13 trajectories with 1/2/3/4/5 scored turns, respectively; the all-turn requirement therefore ranges from one answer to five. Table A10 reports both endpoints so that differences in individual decisions can be distinguished from differences in complete episodes. Table A10: Longitudinal decision performance on 100 patients and 305 questions (%). Turn accuracy weights decisions equally; patient macro weights patients equally; all-turn success requires every scored answer for a patient to be correct. Model API models DeepSeek-V3.1 GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6 Local models HuatuoGPT-o1-72B HuatuoGPT-o1-7B Baichuan-M2-32B Fleming-R1-32B Fleming-R1-7B HealthGPT-Pro-4B HealthGPT-Pro-8B Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-30B-A3B GPT-OSS-20B GPT-OSS-120B

Turn

Patient macro

All-turn success

74.4 75.7 78.7 87.9 82.0 69.5 85.2

74.2 73.7 77.5 88.6 81.6 71.7 85.6

42.0 45.0 45.0 71.0 56.0 42.0 63.0

63.9 55.1 66.2 75.7 59.3 26.2 38.7 59.3 47.2 70.2 49.2 70.5 63.3 48.9 60.3

62.1 53.4 65.5 74.1 59.3 24.9 37.3 59.1 47.1 70.4 49.4 70.1 63.5 48.0 58.0

26.0 20.0 31.0 40.0 23.0 4.0 9.0 26.0 16.0 37.0 14.0 37.0 27.0 13.0 23.0

25

Longitudinal decision performance (continued) Model

Turn

Patient macro

All-turn success

Qwen3-VL-8B-Instruct Qwen3-VL-30B-A3B-Instruct Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B

69.8 68.5 60.7 76.4 72.8 79.7 76.1 58.4

70.0 67.6 60.7 74.5 72.9 79.6 74.6 58.4

38.0 36.0 24.0 46.0 41.0 53.0 49.0 21.0

Model performance by clinical capability HuatuoGPT-o1-72B

88.4

73.9

--

--

--

--

--

46.9

63.9

26.0

--

--

HuatuoGPT-o1-7B

77.2

52.9

--

--

--

--

--

43.2

55.1

20.0

--

--

5.3

Baichuan-M2-32B

22.8

2.4

--

--

--

--

--

5.3

66.2

31.0

--

--

38.8

Fleming-R1-32B

85.6

37.6

--

--

--

--

--

30.3

75.7

40.0

--

--

28.6

Fleming-R1-7B

84.4

60.6

--

--

--

--

--

37.2

59.3

23.0

--

--

24.8

HealthGPT-4B

80.0

59.0

56.5

38.5

20.0

54.0

9.4

50.8

26.2

4.0

19.8

0.0

14.7

HealthGPT-8B

81.2

61.2

56.5

39.5

20.3

50.0

8.6

48.7

38.7

9.0

20.3

0.0

18.8

Hulu-Med-4B

78.0

56.9

56.5

34.5

19.8

30.0

5.6

53.1

59.3

26.0

39.6

0.0

8.7

Hulu-Med-7B

82.4

56.8

56.5

12.5

20.5

58.0

7.5

48.3

47.2

16.0

39.6

0.0

10.7

Hulu-Med-32B

86.0

74.5

60.9

29.0

20.0

75.5

21.8

53.4

70.2

37.0

47.5

10.0

7.6

Lingshu-7B

78.4

60.3

56.5

37.5

3.8

47.0

9.3

42.6

49.2

14.0

44.2

9.0

3.2

Lingshu-32B

85.6

61.0

59.4

37.5

20.0

75.0

11.4

53.3

70.5

37.0

55.3

2.0

9.0

Qwen3-30B

83.2

66.2

--

--

--

--

--

51.3

63.3

27.0

--

--

26.8

10.1

GPT-OSS-20B

74.4

58.3

--

--

--

--

--

32.1

48.9

13.0

--

--

28.0

GPT-OSS-120B

86.0

75.4

--

--

--

--

--

49.4

60.3

23.0

--

--

44.9

DeepSeek-V3.1

90.0

81.0

--

--

--

--

--

51.8

74.4

42.0

--

--

28.2

Qwen3-VL-8B

83.6

61.5

56.5

37.5

20.0

29.5

4.7

52.3

69.8

38.0

52.5

2.0

25.6

Qwen3-VL-30B

86.0

67.3

56.5

37.5

21.8

40.0

36.2

52.3

68.5

36.0

47.5

6.0

29.1

Qwen3.5-9B

85.2

67.8

72.5

26.3

20.3

65.5

38.9

58.0

60.7

24.0

53.0

1.0

34.7

Qwen3.5-27B

89.2

75.8

71.0

37.0

27.3

67.0

47.6

61.0

76.4

46.0

62.7

1.0

46.8

Qwen3.5-35B

86.4

75.2

66.7

35.5

21.8

65.0

50.3

54.5

72.8

41.0

63.1

1.0

41.4

Qwen3.6-27B

87.6

71.4

71.0

37.5

42.8

75.5

41.4

50.3

79.7

53.0

65.9

5.0

47.2

Qwen3.6-35B

85.2

67.9

66.7

43.5

20.3

61.5

44.8

46.8

76.1

49.0

61.8

5.0

42.4

InternVL3.5-38B

88.0

69.7

59.4

36.8

20.0

45.5

3.7

52.0

58.4

21.0

47.5

0.0

9.5

GPT-5.1

90.4

80.9

62.3

27.3

32.8

47.0

26.3

63.5

75.7

45.0

71.0

2.0

58.4 33.3

GPT-4.1

90.0

77.6

71.0

39.8

22.0

48.5

28.3

54.4

78.7

45.0

61.8

2.0

Gemini-3.1

94.0

92.5

63.8

31.0

38.5

77.0

72.0

71.9

87.9

71.0

89.9

11.0

47.8

Gemini-2.5-Pro

93.2

85.8

60.9

37.5

26.0

52.0

48.6

67.0

82.0

56.0

80.6

15.0

38.5

Gemini-2.5-Flash

88.4

71.8

65.2

38.0

25.8

59.0

24.5

59.8

69.5

42.0

65.0

6.0

34.4

Claude-4.6

93.2

86.1

56.5

33.5

27.3

57.0

27.9

69.2

85.2

63.0

80.6

23.0

47.5

.

p.

h.

g.

g.

q.

ct.

an

em -M

G

A

p

-A

t Pa

T-S

t

-S

N

t

Se

0

ru St

Pl

C-

25

L5 rn Tu

L5 n ur ll-t

A 50

L6 re Ca

L6 t ric St

75

.

b Ru

C-

100

Score (%)

Figure A2: Model performance by clinical capability. Scores follow Table 1; gray cells denote unavailable interfaces. L6 uses the CarePath cohort, and L3 uses its static-image cohorts.

26

C.3.2

MRI T and N Staging

Table A11 separates card-based and plane-group T results because patient composition and image representation differ. The combined T400 score weights all 400 patients equally; N-stage balanced accuracy weights N0–N3 equally. Predictions outside the requested categories count as incorrect. Table A11: T-stage accuracy on 400 patients and N-stage accuracy on 400 different patients (%). T-stage results are also shown separately for 100 patients with structure cards and 300 with fivesequence plane groups. N-stage balanced accuracy averages recall over N0–N3. Majority-class baselines are 37.5% for T400 and 40.0% for N400. Model

T400 T100 cards T300 groups N400 N bal.

Qwen3-VL-8B Lingshu-7B Qwen3.5-9B Qwen3-VL-30B Lingshu-32B Qwen3.5-27B Qwen3.5-35B Qwen3.6-27B Qwen3.6-35B Hulu-Med-4B HealthGPT-4B HealthGPT-8B Hulu-Med-7B Hulu-Med-32B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-2.5-Flash Gemini-2.5-Pro Gemini-3.1 Claude-4.6

37.5 37.5 26.3 37.5 37.5 37.0 35.5 37.5 43.5 34.5 38.5 39.5 12.5 29.0 36.8 27.3 39.8 38.0 37.5 31.0 33.5

35.0 35.0 26.0 35.0 35.0 34.0 43.0 35.0 35.0 35.0 39.0 33.0 15.0 16.0 33.0 39.0 30.0 34.0 33.0 34.0 33.0

38.3 38.3 26.3 38.3 38.3 38.0 33.0 38.3 46.3 34.3 38.3 41.7 11.7 33.3 38.0 23.3 43.0 39.3 39.0 30.0 33.7

20.0 3.8 20.3 21.8 20.0 27.3 21.8 42.8 20.3 19.8 20.0 20.3 20.5 20.0 20.0 32.8 22.0 25.8 26.0 38.5 27.3

25.0 4.7 25.3 27.2 25.0 33.3 27.0 31.9 25.2 24.5 25.0 25.3 25.3 25.0 25.0 30.6 27.5 29.5 28.6 38.4 28.1

Prediction counts distinguish single-category predictions from answers outside the allowed categories. On the 300 plane-group T cases, Qwen3-VL-8B/30B-A3B, Lingshu-32B, and HealthGPT-Pro-4B always predict T4 (115/300 correct), whereas Hulu-Med-7B always predicts T1 (35/300). All five have 25.0% balanced accuracy. On N400, only 46/400 Lingshu-7B answers use a valid category (11.5%), all N3; 15 are correct, and balanced accuracy is 4.7%. Second-centre MRI assessment. The Nanfang Hospital panel uses clinical T-stage references associated with 53 MRI examinations. Each examination provides ten four-row panels, sampling 30 positions per acquired sequence across the panels. All ten images enter one request with an eighth-edition T-only prompt. Table A12 reports accuracy, balanced accuracy, and per-stage recall. T3 accounts for 31 of 53 cases, giving a majority-class accuracy of 58.5%. Hulu-Med-4B reaches 60.4% accuracy but 28.1% balanced accuracy, whereas Qwen3.5-35B reaches 50.9% and 39.8%, respectively. The class-dependent ordering shows that higher overall accuracy can coincide with poorer recognition of less frequent stages. This panel evaluates MRI staging at a second centre. C.3.3

L3 Multimodal Evidence-Grounding Diagnostics

The L3 diagnostics examine three linked questions: whether the marked anatomy is identified, whether its invasion status is correct, and whether the resulting evidence supports the T category. They use the 100-case anatomical panel, with 18 invasion labels and one T category per patient. Structure identification is scored over 2,700 cards covering 18 base structures; nasopharynx cards are excluded from this primary MRI scoring set. The text-stager experiment uses structured findings as input; the CarePath and cine inputs are described in sections C.3.5 and C.4. Patient-level measures then distinguish partial evidence from complete evidence. Pt. Inv. averages invasion correctness over all 18 structures within each patient and then across patients. Complete Inv. 27

Table A12: T-stage results at Nanfang Hospital on 53 patients with definite reference labels (T1/T2/T3/T4: 4/8/31/10). Two other evaluated examinations lacked a definite reference T stage and are not scored. Accuracy, balanced accuracy and per-stage recall are percentages; the majority-class baseline is 58.5%. Model HealthGPT-4B HealthGPT-8B Qwen3-VL-8B Lingshu-7B Qwen3.5-9B Qwen3-VL-30B Lingshu-32B Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Qwen3.5-27B Qwen3.5-35B Qwen3.6-27B Qwen3.6-35B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-2.5-Flash Gemini-2.5-Pro Gemini-3.1 Claude-4.6

Accuracy Balanced 17.0 22.6 18.9 39.6 35.8 18.9 37.7 60.4 18.9 17.0 18.9 50.9 17.0 24.5 28.3 13.2 18.9 41.5 35.8 34.0 41.5

T1

T2

T3

T4

22.5 0.0 0.0 0.0 90.0 24.9 0.0 0.0 9.7 90.0 25.0 0.0 0.0 0.0 100.0 22.0 0.0 0.0 58.1 30.0 26.1 0.0 12.5 41.9 50.0 25.0 0.0 0.0 0.0 100.0 28.0 0.0 0.0 41.9 70.0 28.1 0.0 12.5 100.0 0.0 22.0 0.0 75.0 12.9 0.0 17.4 0.0 0.0 9.7 60.0 33.1 50.0 12.5 0.0 70.0 39.8 75.0 0.0 74.2 10.0 38.1 100.0 12.5 0.0 40.0 24.0 0.0 0.0 16.1 80.0 33.7 0.0 25.0 9.7 100.0 28.7 75.0 0.0 0.0 40.0 25.0 0.0 0.0 0.0 100.0 26.2 0.0 0.0 54.8 50.0 30.6 0.0 0.0 32.3 90.0 28.7 0.0 12.5 32.3 70.0 20.1 0.0 12.5 67.7 0.0

requires all 18 labels to be correct; EG T additionally requires the correct final T stage. A model can therefore obtain a correct direct T label while failing complete evidence or EG T. Table A15 reports these endpoints together, and the ablations below examine which errors remain when anatomical identity or invasion labels are supplied by the reference. Table A13: Anatomical diagnostic panel (100 patients). Interpretation of the L3 MRI staging metrics and ablations. These metrics are reported together because each isolates a different part of the evidence-to-stage chain. Metric

Scoring target

Direct T

The final t_stage field generated by the evaluated model in the patient-level MRI prompt

Complete Inv. EG T

Oracle-slot T Model-slot T

LLM-slot T

Gold Rule / Gold LLM-slot

Interpretation

Measures whether the model can output the correct stage label, but does not require the supporting structure-level evidence to be correct Whether all 18 base-structure invasion labels Measures whether every scored invasion are correct for one patient finding is correct Complete Inv. plus correct Direct T for the Strict evidence-grounded staging success; a same patient model must both extract the evidence and produce the right stage Deterministic staging after placing predicted Isolates invasion recognition while removing invasion labels into the gold structure slots structure-localization errors Deterministic staging after placing predicted Measures the full invasion labels into the model-predicted structure-localization-plus-invasion evidence structure slots chain Fixed GPT-OSS-120B staging from Tests whether a stronger text stager can recover model-generated structure-card summaries T stage from the generated evidence representation Deterministic or GPT-OSS-120B staging from Estimates the ceiling imposed by the gold 18-structure evidence 18-structure evidence abstraction

The patient-level MRI comparison uses the same structured response fields for local and API models. 28

40

Gemini-3.1

35

Qwen3.5-35B

30

GPT-5.1

25 20 15

20

25

30

35

40

Joint anatomy + invasion (%)

Balanced T recall (%)

(a) Overall and balanced T accuracy

(b) Anatomy and invasion accuracy

Qwen3.5-27B Qwen3.6-27B

20

0

45

20

Direct T accuracy (%)

Score (%)

Direct T

LLM-slot T

40

60

Structure ID accuracy (%)

(c) Evidence-based staging 100

Gemini-3.1

40

(d) Recall for each T category Gemini-3.1

60

27

17

43

Lingshu-7B

73

0

0

69

Qwen3.5-9B

27

87

20

6

Qwen3.5-35B

7

0

57

63

Balanced

75 50 25

Hulu-Med-4B

27

13

9

74

HealthGPT-Pro-4B

7

7

11

94

Qwen3.5-27B

0

33

31

51

GPT-5.1

0

0

20

91

T1

T2

T3

T4

0 e .1 .6 7B 7B 3.1 nc T-5 e-4 nire 5-2 6-2 ud mi GP 3. 3. a e n n l e e G C w w Q Q

100 75 50 25 0

fe Re

Figure A3: L3 evidence-grounding diagnostics. Panels (a,b,d) use the 100-case anatomical panel; panel (c) uses the GPT-OSS-120B text-stager experiment. (a) Direct T-stage accuracy can diverge from class-balanced T recall. (b) Base-structure localization does not guarantee correct structure-plusinvasion evidence. (c) When gold 18-structure evidence is supplied, a fixed GPT-OSS-120B stager nearly reaches the rule ceiling; when model-generated evidence is supplied, LLM-slot T-stage scores remain much lower. Reference shows gold-evidence LLM-slot accuracy and balanced recall, with no direct-image prediction. (d) Per-stage recall shows uneven recognition of the four T categories. Table A14: Local L3 component results. Structure and joint card accuracies weight all 2,700 cards equally. Inv. is conditioned on correct structure identification. Pt. Inv. averages correctness over the 18 invasion labels per patient. EG T requires all 18 labels and the T category to be correct. Pathology and sequence match Table 1; all values are percentages. Model

Path.

Struct.

Inv.

Joint

Pt. Inv.

EG T

Seq.

HealthGPT-Pro-4B HealthGPT-Pro-8B Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-VL-8B-Instruct Qwen3-VL-30B-A3B-Instruct Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B

56.5 56.5 56.5 56.5 60.9 56.5 59.4 56.5 56.5 72.5 71.0 66.7 71.0 66.7 59.4

9.4 8.6 5.6 7.5 21.8 9.3 11.4 4.7 36.2 38.9 47.6 50.3 41.4 44.8 3.7

30.4 41.6 30.9 48.0 46.6 6.8 50.0 33.1 20.6 36.6 48.9 30.3 46.2 31.4 45.0

2.9 3.6 1.7 3.6 10.1 0.6 5.7 1.6 7.4 14.3 23.3 15.3 19.1 14.1 1.7

43.3 48.4 47.7 62.4 59.8 57.9 37.6 41.3 37.6 66.0 53.9 66.4 45.0 61.7 54.7

0.0 0.0 0.0 5.0 3.0 4.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0

54.0 50.0 30.0 58.0 75.5 47.0 75.0 29.5 40.0 65.5 67.0 65.0 75.5 61.5 45.5

29

Table A15: MRI diagnostics on the anatomical panel for 100 patients. Complete Inv. requires all 18 invasion labels to be correct; Direct T requires the correct final T-category field; EG T requires both. Missing or invalid labels remain incorrect in their respective endpoints. Model

Complete Inv.

Direct T

EG T

0.0 0.0 0.0 5.0 3.0 4.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 1.0 0.0 1.0

39.0 33.0 35.0 15.0 16.0 35.0 35.0 35.0 35.0 26.0 34.0 43.0 35.0 35.0 33.0 39.0 30.0 34.0 33.0 34.0 33.0

0.0 0.0 0.0 5.0 3.0 4.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 1.0 0.0 1.0

HealthGPT-Pro-4B HealthGPT-Pro-8B Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-VL-8B-Instruct Qwen3-VL-30B-A3B-Instruct Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6

C.3.4

L3 MRI Staging Ablations

The staging comparisons examine two possible sources of error: extracting anatomical findings and assigning a stage from those findings. The deterministic rule and GPT-OSS-120B stager assign T categories from structured anatomical findings. When gold 18-structure evidence is supplied, the stager approaches the gold ceiling; when model-generated evidence is supplied, performance remains much lower. Under this fixed structured-evidence interface, the contrast locates a major performance gap in structure and invasion extraction. The class recalls in Table A16 show that several models recognize advanced T categories more often than early categories. For example, GPT-5.1 reaches 39.0 Direct T but 27.9 Balanced T because most correct cases are T4, whereas Gemini-3.1-Pro-Preview has lower micro Direct T but the strongest class-balanced recall among the listed patient-staging runs. The deterministic structure-only rule assigns T4 when cavernous sinus invasion is present; otherwise T3 when skull-base, pterygoid-base, upper-cervical, or paranasal-sinus structures are invaded; otherwise T2 when the parapharyngeal-space, medial/lateral pterygoid, or longus-capitis labels are positive; otherwise T1. Levator veli palatini and carotid sheath are not used for upstaging in this 18-structure rule. The 99.0 Gold Rule ceiling indicates that one gold T4 case depends on staging evidence outside the scored 18-base-structure set.

30

Table A16: T-category diagnostics on the anatomical panel. The 100 patients comprise 15 T1, 15 T2, 35 T3, and 35 T4 cases. Balanced T averages the four class recalls. T-bal60 uses a fixed subset of 15 cases per stage, with the same patient membership for all models. Model HealthGPT-Pro-4B HealthGPT-Pro-8B Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-VL-8B-Instruct Qwen3-VL-30B-A3B-Instruct Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6

T1

T2

T3

T4

Direct T

Bal. T

T-bal60

6.7 0.0 26.7 100.0 80.0 73.3 0.0 0.0 0.0 26.7 0.0 6.7 0.0 0.0 0.0 0.0 0.0 60.0 0.0 0.0 6.7

6.7 0.0 13.3 0.0 0.0 0.0 0.0 0.0 0.0 86.7 33.3 0.0 13.3 0.0 0.0 0.0 0.0 26.7 6.7 6.7 6.7

11.4 5.7 8.6 0.0 2.9 0.0 0.0 0.0 0.0 20.0 31.4 57.1 0.0 25.7 5.7 20.0 20.0 17.1 14.3 11.4 65.7

94.3 88.6 74.3 0.0 8.6 68.6 100.0 100.0 100.0 5.7 51.4 62.9 94.3 74.3 88.6 91.4 65.7 42.9 77.1 82.9 22.9

39.0 33.0 35.0 15.0 16.0 35.0 35.0 35.0 35.0 26.0 34.0 43.0 35.0 35.0 33.0 39.0 30.0 34.0 33.0 34.0 33.0

29.8 23.6 30.7 25.0 22.9 35.5 25.0 25.0 25.0 34.8 29.0 31.7 26.9 25.0 23.6 27.9 21.4 36.7 24.5 25.2 25.5

31.7 25.0 26.7 25.0 20.0 36.7 25.0 25.0 25.0 40.0 28.3 36.7 25.0 16.7 20.0 35.0 23.3 33.3 26.7 26.7 25.0

Table A17: Deterministic evidence-to-stage analysis on the 100-patient anatomical panel. Inv. is conditional on correct anatomy; Joint requires both anatomy and invasion. Oracle-slot places predicted invasion in the reference anatomy slot; Model-slot uses predicted anatomy. Unfilled slots take the fixed zero default, whereas complete-evidence correctness requires every label. Gold Rule measures the ceiling of the 18-structure mapping. Model HealthGPT-Pro-4B HealthGPT-Pro-8B Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-VL-8B-Instruct Qwen3-VL-30B-A3B-Instruct Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6

Struct.

Inv.

Joint

Gold Rule

Oracle-slot

Model-slot

Direct T

9.4 8.6 5.6 7.5 21.8 9.3 11.4 4.7 36.2 38.9 47.6 50.3 41.4 44.8 3.7 26.3 28.3 72.0 48.6 24.5 27.9

30.4 41.6 30.9 48.0 46.6 6.8 50.0 33.1 20.6 36.6 48.9 30.3 46.2 31.4 45.0 27.0 22.5 65.0 47.1 52.3 39.8

2.9 3.6 1.7 3.6 10.1 0.6 5.7 1.6 7.4 14.3 23.3 15.3 19.1 14.1 1.7 7.1 6.4 46.8 22.9 12.8 11.1

99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0 99.0

35.0 35.0 35.0 30.0 52.0 35.0 35.0 35.0 35.0 36.0 47.0 37.0 50.0 36.0 35.0 34.0 35.0 49.0 39.0 45.0 32.0

26.0 23.0 15.0 28.0 35.0 33.0 35.0 35.0 33.0 39.0 29.0 30.0 32.0 34.0 15.0 36.0 27.0 36.0 35.0 32.0 37.0

39.0 33.0 35.0 15.0 16.0 35.0 35.0 35.0 35.0 26.0 34.0 43.0 35.0 35.0 33.0 39.0 30.0 34.0 33.0 34.0 33.0

31

Direct T is the patient-level T-stage answer generated by the evaluated model from the L3 patient prompt. Rule-slot T applies the deterministic structure rule described above to the evidence used in each experiment. The LLM-slot experiment uses a separate set of model-generated evidence summaries as input to GPT-OSS-120B. Its Rule-slot column applies the deterministic rule to those same summaries; Table A17 uses the primary L3 responses. The gap between the 99.0 gold-evidence ceiling and model-evidence LLM-slot scores shows how much the same downstream stager depends on the quality of its structured input. Table A18: LLM-stager ablation for structure-evidence-based MRI T staging. All 21 models contribute evidence summaries for the same 100 patients. A fixed GPT-OSS-120B stager infers T stage from each model’s anatomical labels, invasion predictions, and aggregated votes. Gold 18-structure evidence yields 99.0 LLM-slot T and 99.3 balanced T from these anatomical findings. Model

C.3.5

Direct T

Rule-slot T

LLM-slot T

Bal. LLM T

Lingshu-7B Lingshu-32B InternVL3.5-38B Qwen3-VL-8B-Instruct Qwen3-VL-30B-A3B-Instruct HealthGPT-Pro-4B HealthGPT-Pro-8B Hulu-Med-4B Hulu-Med-7B Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.6-27B Qwen3.6-35B-A3B Hulu-Med-32B

35.0 35.0 33.0 35.0 35.0 39.0 33.0 35.0 15.0 26.0 34.0 43.0 35.0 35.0 16.0

33.0 35.0 15.0 35.0 33.0 26.0 23.0 15.0 28.0 39.0 29.0 30.0 32.0 34.0 35.0

32.0 35.0 15.0 35.0 33.0 26.0 23.0 15.0 29.0 39.0 29.0 30.0 32.0 34.0 34.0

23.8 25.0 25.0 25.0 23.5 25.2 25.0 25.0 29.3 27.9 20.7 21.4 22.9 24.3 24.3

GPT-5.1 GPT-4.1 Gemini-3.1-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4.6

39.0 30.0 34.0 33.0 34.0 33.0

36.0 27.0 36.0 35.0 32.0 37.0

36.0 27.0 36.0 35.0 29.0 36.0

25.7 19.3 25.7 25.0 20.7 27.6

CarePath Evaluation and Cohort Composition

The CarePath cohort contains 100 patients, 811 clinical context entries, and 217 scored care decisions. Each model provides 100 pathology predictions, 100 MRI predictions, and 217 care answers. Table A19 reports component performance and patient-level outcomes; Table A20 gives their intersections. Figure A5 shows how requiring pathology, T stage, and care answers to be correct together narrows the set of successful patients. Cohort selection and modality linkage. HealthGPT-Pro-8B, Qwen3-VL-8B, Qwen3-VL-30BA3B, and Qwen3.5-27B participated in cohort and decision selection. They are marked in the results tables, and the other 17 models were not used in cohort construction. The 25 episodes with matched pathology contribute 115 decisions; the 75 with assigned specimens contribute 102. MRI and clinical records are patient-linked throughout. All assigned specimens are labelled NPC-positive; the corresponding episode records identify their specimen source. Selection used the four construction models’ care answers with gold pathology and MRI evidence and reference history. Twenty-five patients with matched multimodal data were included. Among the other 86 reviewed patients, those with at least one construction-model error were ranked first by total errors across models and checkpoints, then by the number of checkpoints missed by at least two models, checkpoints missed by any model, maximum errors per checkpoint, and mean errors per checkpoint; a fixed source identifier broke remaining ties. The first 75 patients completed the CarePath cohort. Within these 100 patients, checkpoints answered correctly by all four models were

32

removed from scoring. This procedure yields 217 scored decisions, including a 93-decision subset missed by at least two construction models. Selection therefore targets clinically relevant decisions that these models found difficult. Question review and scoring depth. Six treatment questions are excluded from scoring because the source records do not establish a unique treatment choice at the specified checkpoint. The identified issues include combining ordered and administered medicines and incomplete evidence of radiotherapy. Their exclusion reasons accompany the data, and all 100 patients remain in the scored cohort. The scored set contains 52/23/3/7/8/7 patients with 1/2/3/4/5/6 care decisions, respectively. The 75 assigned-specimen episodes contain no scored initial-treatment-selection question; those questions occur in the matched-pathology group. Clinical inputs and endpoints. Care prompts incorporate predicted pathology and T-stage labels where specified: 207 reference predicted pathology, and 133 retain supplied N/M/overall-stage information. Prior care answers are not supplied in the primary protocol. Strict success requires correct pathology, correct MRI T stage, and every selected care answer for a patient. Care accuracy weights the 217 decisions equally; all-care correctness and strict success weight the 100 patients equally. MRI structure predictions and the 811 clinical context entries support the evaluation but are not additional strict-success criteria. (a) MRI T-stage labels

Patients

100

(b) Pathology labels 99

100

75

(c) Decision counts 100

75

75

50

50

25

25

58 50 23

25 7 0

T1

12 T2

T3

1

0

T4

Negative

Always T3: 58% accuracy

|

Positive

0

52 23

1

2

3

7

8

7

3

4

5

6

Scored decisions per patient

Always positive: 99% accuracy

Figure A4: Class balance and scoring depth in the CarePath cohort. T1–T4 counts are 7/12/58/23; pathology labels are 99 positive and one negative. For 1/2/3/4/5/6 scored care decisions, patient counts are 52/23/3/7/8/7. These distributions belong to the selected L6 cohort; L3 has different populations and a different MRI staging edition. (a) Care vs. strict

(b) Joint correctness

Score (%)

80 60 40 20 0 Care-turn

(c) Strict success

100

Patients correct (%)

100

Strict episode

Gemini-3.1

60

GPT-5.1

40

Claude-4.6

20 0 Path.

11

Gemini-3.1

80

2 23 5

Qwen3.6-27B

+ MRI T

+ care

0

20

40

Strict success (%)

GPT-5.1

Claude-4.6

Qwen3.6-27B

Figure A5: Local and joint correctness in the CarePath cohort. (a) Care accuracy uses 217 decisions; strict success uses 100 patients. (b) Nested intersections count the same patients: correct pathology, correct pathology and T stage, then all selected care answers as well. Each successive requirement narrows the set of successful patients. (c) Exact binomial 95% intervals for the four representative models. Panels (a,b) include all 17 models not used in selection; the remaining 13 are gray. All 21 model scores appear in Table A19.

33

Table A19: CarePath cohort outcomes for all 21 models. T and balanced T use 100 patients, care uses 217 scored care decisions, and all-care/strict use 100 patients. Brackets are 95% intervals: patient-cluster bootstrap for care, exact binomial for strict. †: construction models. Model HealthGPT-4B HealthGPT-8B† Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-VL-8B† Qwen3-VL-30B† Qwen3.5-9B Qwen3.5-27B† Qwen3.5-35B Qwen3.6-27B Qwen3.6-35B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-3.1 Gemini-2.5-Pro Gemini-2.5-Flash Claude-4.6

T

Bal. T

Care [95% CI]

All-care

Strict [95% CI]

10.0 4.0 8.0 4.0 52.0 45.0 19.0 21.0 23.0 33.0 15.0 11.0 18.0 48.0 37.0 39.0 22.0 19.0 26.0 22.0 40.0

9.3 14.3 8.4 14.3 26.3 26.3 27.5 26.0 25.0 30.1 25.4 27.4 24.8 22.0 30.5 28.7 21.9 22.4 19.1 22.6 24.1

19.8 [14.4, 25.8] 20.3 [15.3, 25.5] 39.6 [32.5, 46.7] 39.6 [33.3, 46.2] 47.5 [41.3, 53.5] 44.2 [37.3, 50.8] 55.3 [48.8, 61.4] 52.5 [46.2, 58.6] 47.5 [40.5, 53.7] 53.0 [46.0, 59.3] 62.7 [55.0, 69.3] 63.1 [56.0, 69.4] 65.9 [58.5, 72.3] 61.8 [53.9, 68.6] 47.5 [41.5, 53.5] 71.0 [64.4, 77.0] 61.8 [54.7, 68.0] 89.9 [85.4, 93.9] 80.6 [75.0, 86.2] 65.0 [58.4, 71.8] 80.6 [75.3, 85.6]

8.0 5.0 15.0 19.0 19.0 16.0 24.0 19.0 14.0 17.0 30.0 27.0 33.0 26.0 24.0 46.0 29.0 83.0 71.0 47.0 64.0

0.0 [0.0, 3.6] 0.0 [0.0, 3.6] 0.0 [0.0, 3.6] 0.0 [0.0, 3.6] 10.0 [4.9, 17.6] 9.0 [4.2, 16.4] 2.0 [0.2, 7.0] 2.0 [0.2, 7.0] 6.0 [2.2, 12.6] 1.0 [0.0, 5.4] 1.0 [0.0, 5.4] 1.0 [0.0, 5.4] 5.0 [1.6, 11.3] 5.0 [1.6, 11.3] 0.0 [0.0, 3.6] 2.0 [0.2, 7.0] 2.0 [0.2, 7.0] 11.0 [5.6, 18.8] 15.0 [8.6, 23.5] 6.0 [2.2, 12.6] 23.0 [15.2, 32.5]

Table A20: CarePath cohort patient-level joint states for all 21 models. C: patients with all selected care answers correct; F: patients with both image labels correct. Entries are set cardinalities; complements are within the same 100 patients. The four intersection counts sum to 100. † denotes construction models. Model

|C|

|F|

|C ∩ F|

|C ∩ F c |

|C c ∩ F|

|C c ∩ F c |

HealthGPT-4B HealthGPT-8B† Hulu-Med-4B Hulu-Med-7B Hulu-Med-32B Lingshu-7B Lingshu-32B Qwen3-VL-8B† Qwen3-VL-30B-A3B† Qwen3.5-9B Qwen3.5-27B† Qwen3.5-35B Qwen3.6-27B Qwen3.6-35B-A3B InternVL3.5-38B GPT-5.1 GPT-4.1 Gemini-3.1 Gemini-2.5 Pro Gemini-2.5 Flash Claude-4.6

8 5 15 19 19 16 24 19 14 17 30 27 33 26 24 46 29 83 71 47 64

9 4 8 1 48 45 15 14 22 5 8 5 10 16 7 9 7 16 23 20 38

0 0 0 0 10 9 2 2 6 1 1 1 5 5 0 2 2 11 15 6 23

8 5 15 19 9 7 22 17 8 16 29 26 28 21 24 44 27 72 56 41 41

9 4 8 1 38 36 13 12 16 4 7 4 5 11 7 7 5 5 8 14 15

83 91 77 80 43 48 63 69 70 79 63 69 62 63 69 47 66 12 21 39 21

34

Table A21: CarePath cohort outcomes by pathology source. Matched specimens: 25 patients and 115 care decisions; assigned specimens: 75 patients and 102 decisions. Within each group, columns give care accuracy, all-care correctness, both image labels correct, and strict success (%). † marks construction models. Matched specimens (n = 25)

C.4

Assigned specimens (n = 75)

Model

Care All-care Image Strict Care All-care Image Strict

Claude-4.6 GPT-4.1 GPT-5.1 Gemini-2.5-Flash Gemini-2.5-Pro Gemini-3.1 HealthGPT-4B HealthGPT-8B† Hulu-Med-32B Hulu-Med-4B Hulu-Med-7B InternVL3.5-38B Lingshu-32B Lingshu-7B Qwen3-VL-30B† Qwen3-VL-8B† Qwen3.5-27B† Qwen3.5-35B Qwen3.5-9B Qwen3.6-27B Qwen3.6-35B

84.3 75.7 77.4 60.0 73.9 83.5 10.4 16.5 50.4 40.9 35.7 46.1 62.6 48.7 55.7 60.9 78.3 78.3 66.1 81.7 79.1

52.0 32.0 32.0 8.0 28.0 44.0 0.0 0.0 4.0 8.0 4.0 4.0 12.0 4.0 4.0 8.0 40.0 28.0 16.0 44.0 40.0

40.0 4.0 16.0 28.0 28.0 20.0 36.0 4.0 24.0 12.0 4.0 28.0 20.0 8.0 28.0 8.0 4.0 8.0 8.0 0.0 8.0

12.0 0.0 4.0 0.0 8.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 8.0

76.5 46.1 63.7 70.6 88.2 97.1 30.4 24.5 44.1 38.2 44.1 49.0 47.1 39.2 38.2 43.1 45.1 46.1 38.2 48.0 42.2

68.0 28.0 50.7 60.0 85.3 96.0 10.7 6.7 24.0 17.3 24.0 30.7 28.0 20.0 17.3 22.7 26.7 26.7 17.3 29.3 21.3

37.3 8.0 6.7 17.3 21.3 14.7 0.0 4.0 56.0 6.7 0.0 0.0 13.3 57.3 20.0 16.0 9.3 4.0 4.0 13.3 18.7

26.7 2.7 1.3 8.0 17.3 14.7 0.0 0.0 13.3 0.0 0.0 0.0 1.3 12.0 8.0 2.7 1.3 0.0 1.3 6.7 4.0

Controlled Input Comparisons

Table A22: Design of diagnostic comparisons. The O1 comparison applies different judges to the same Expanded Audit response set. Control

Held fixed

L6 image evidence (25 pa- Patients, care decisions, tients) supplied clinical context L5 history (100 patients) Baseline/current evidence, question, options, reference answer L1 options (71 pairs) Physician-audited target knowledge MRI interface (25 patients) Model, patients, matched image frames O1 Expanded Audit (1,013 Responses, criteria, labels) physician labels

Changed

Reported endpoint

Pathology, MRI T stage, and Patient-macro care invaded structures difference Historical time markers and Turn-accuracy difference record ordering MCQ versus option-free answer Static panel versus cine presentation Automated judge

Paired item-accuracy difference T and structure differences Agreement and polarity-specific agreement

Reference image-evidence comparison. This experiment asks whether incorrect image-derived findings account for errors in care answers. Fourteen local models answer the same 146 care questions for 25 patients within the 100-patient CarePath cohort, using predicted or reference pathology, MRI T stage, and invaded structures. Supplied N/M and overall-stage fields remain fixed. These patients have patient-matched pathology, MRI, and clinical records and contribute 115 decisions to the main benchmark; the paired analysis uses its fixed 146-question set. Additional conditions replace pathology or MRI findings separately or vary the supplied history. Table A23 reports joint evidence replacement with clinical context held fixed, using mean within-patient care accuracy and 10,000 paired patient-bootstrap draws. The endpoint scores care answers alone, so replacing an incorrect image label raises the score only if a care answer also improves. 35

Table A23: L6 image-evidence replacement: scores and patient changes. Predicted and reference pathology, MRI T stage, and invasion findings are compared on the same 25 selected patients and 146 care decisions. Scores are patient-macro care accuracy (%); differences and paired patient-bootstrap 95% intervals are in pp. B/W/S counts patients with better/worse/unchanged care accuracy under reference inputs. Figure 7d plots baseline accuracy against the paired difference. Row 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Model HealthGPT-8B Lingshu-7B Hulu-Med-4B HealthGPT-4B Qwen3.6-27B Qwen3.6-35B-A3B Qwen3.5-27B Lingshu-32B Qwen3.5-9B Qwen3.5-35B Hulu-Med-7B Hulu-Med-32B Qwen3-VL-30B-A3B Qwen3-VL-8B

Predicted

Reference

Difference [95% CI]

B/W/S

21.1 54.4 48.0 23.9 85.5 84.2 82.6 68.4 71.0 79.0 36.9 56.8 64.6 67.8

17.9 53.0 46.8 22.9 84.6 84.5 83.1 69.9 73.5 81.9 40.1 60.2 68.5 71.8

-3.2 [-10.5, 3.8] -1.4 [-5.0, 1.8] -1.3 [-3.9, 1.3] -1.0 [-9.2, 8.0] -0.9 [-4.4, 2.4] +0.2 [-2.5, 2.9] +0.5 [-2.7, 3.8] +1.5 [-1.7, 4.9] +2.5 [-0.5, 5.7] +2.8 [-1.0, 6.6] +3.2 [-0.6, 7.0] +3.4 [0.8, 6.4] +3.8 [-1.0, 8.6] +4.1 [0.7, 7.8]

8/10/7 2/3/20 1/3/21 7/9/9 2/3/20 2/2/21 3/3/19 4/2/19 4/1/20 5/1/19 6/2/17 5/0/20 6/1/18 6/1/18

History representation and record order. The history comparisons separate two changes that might affect a clinical decision: removing temporal markers and changing the order in which records are presented. The snapshot removes relative-day and admission markers, then groups the historical records by content. Baseline and current findings, the question, options, and reference answer remain unchanged. Although the records match once temporal markers are ignored, their clinical timing is no longer explicit; this contrast combines loss of timing with reorganization. The order-only condition instead shuffles complete lines within their original sections, retaining every time marker. Twenty models each answer the same 305 questions for 100 patients in all three conditions. These controls use condition-matched generations, with a full-history baseline evaluated alongside each altered representation. Full-history minus snapshot accuracy is positive for 14 models; only DeepSeekV3.1 has a patient-bootstrap interval wholly above zero (Table A25). In the order-only comparison, 19 intervals include zero, while Gemini-3.1 improves by 3.0 pp (95% CI: 0.6–5.7; Table A26). Both analyses use 10,000 paired patient-bootstrap draws, with intervals unadjusted across models. Figure A6 places the full-history comparison alongside the image-evidence comparison, showing both the direction and uncertainty of each paired difference. The history-dependent subset contains 118 turns from 69 patients and compares full history with current-state information alone (Table A24). A separate Gemini-3.1 experiment evaluates packedhistory, persistent, and stateless interaction on 292 turns from 87 patients. Persistent and stateless turn accuracy are 82.5% and 83.6%, a difference of −1.0 pp (95% CI: −3.8 to 1.7). The interval includes zero. Table A24: History-dependent subset: 118 turns from 69 patients. Full history is compared with current-state information alone. Turn accuracy is in percent; differences and paired patient-bootstrap 95% intervals are in pp. Model HealthGPT-8B Lingshu-7B Qwen3-VL-8B Qwen3.5-9B

History Current state Difference [95% CI] 24.6 62.7 47.5 46.6

28.0 63.6 50.8 44.1

-3.4 [-9.7, 2.8] -0.8 [-8.4, 6.4] -3.4 [-12.4, 5.5] +2.5 [-6.3, 11.7]

Paired decision changes. Across M matched turns, let u count incorrect-to-correct changes and v correct-to-incorrect changes. Net accuracy change ∆ in percentage points and correctness-flip rate Φ

36

in percent are

u−v u+v , Φ = 100 . (A6) M M For the full-history versus snapshot L5 comparison, M = 305 and each condition covers the same 100 patients. All-turn gain is the difference between the two conditions’ proportions of patients with every scored turn correct; it is not derived from u and v alone. ∆ = 100

For L1, MRI T staging, and L5, a changed unit switches between correct and incorrect across the paired conditions. For the 25-case L6 control, a changed patient has different within-patient care accuracy. These counts capture offsetting improvements and regressions. GPT-5.1 changes correctness on 41 of 305 L5 turns (22 corrected, 19 regressed), with a net gain of 1.0 pp and an all-turn success change of −1.0 pp. Under gold pathology and MRI evidence, HealthGPT-8B improves care accuracy for eight patients and worsens it for ten; the remaining seven are unchanged. The L6 net difference averages the magnitudes of the within-patient accuracy changes. Table A25: L5 full history versus the time-stripped snapshot. The snapshot removes historical time markers and regroups records (305 turns, 100 patients). Accuracy is in percent and differences in pp, with paired patient-bootstrap 95% intervals. +/− count turns corrected/harmed by full history. Model DeepSeek-V3.1 Fleming-R1-7B GPT-4.1 GPT-5.1 GPT-OSS-120B GPT-OSS-20B Gemini-3.1 HealthGPT-4B HealthGPT-8B HuatuoGPT-o1-7B Hulu-Med-32B Hulu-Med-4B Hulu-Med-7B Lingshu-7B Qwen3-30B Qwen3-VL-8B Qwen3.5-27B Qwen3.5-35B Qwen3.5-9B Qwen3.6-35B-A3B

History

Snapshot

Difference [95% CI]

+/−

71.5 59.7 75.1 71.5 48.9 52.8 89.5 36.4 12.5 45.6 69.5 58.0 43.9 53.4 60.0 68.2 78.4 73.8 67.9 75.4

66.6 61.3 75.7 70.5 45.6 48.2 86.2 40.0 11.8 46.2 68.5 56.1 41.6 51.1 58.4 66.9 78.0 71.1 68.9 77.7

+4.9 [1.4, 8.5] -1.6 [-6.5, 3.2] -0.7 [-4.1, 2.9] +1.0 [-3.2, 5.2] +3.3 [-1.6, 8.1] +4.6 [-1.4, 10.7] +3.3 [0.0, 6.4] -3.6 [-8.9, 1.6] +0.7 [-2.5, 3.9] -0.7 [-4.5, 3.1] +1.0 [-1.6, 3.7] +2.0 [-1.6, 5.5] +2.3 [-1.0, 5.4] +2.3 [-0.3, 5.0] +1.6 [-1.3, 4.4] +1.3 [-1.4, 4.1] +0.3 [-2.0, 2.6] +2.6 [-0.3, 5.6] -1.0 [-4.1, 2.0] -2.3 [-5.1, 0.3]

22/7 21/26 14/16 22/19 29/19 46/32 18/8 29/40 13/11 14/16 9/6 17/11 15/8 12/5 14/9 11/7 8/7 14/6 10/13 6/13

A three-condition input example. The accompanying example provides one complete L5 prompt triplet and its record correspondence. Each prompt has the same 46 historical records; 44 carry relative-day or admission markers in the full-history version. Table A27 shows two translated treatment-record excerpts. Drug names, doses, cycle number, and treatment intent are preserved, whereas the snapshot omits the admission/day prefix. The shuffled condition keeps each complete source line and changes its position. Baseline/current findings and all answer options remain identical. Removing the time markers changes the temporal information available for the decision. Success at different scoring depths. Figure 7c groups L5 patients by 1–2, 3, and 4–5 scored turns (n = 32, 31, 37); panel h groups CarePath cohort patients by 1, 2, and 3–6 scored care decisions (n = 52, 23, 25). For each model and group, success is the number of patients meeting the endpoint divided by the group’s patient count: all-turn correctness for L5 and strict success for the CarePath cohort. The groups differ in both clinical composition and the number of decisions required for complete success. Serial CarePath interaction. The 25-case serial control uses gold pathology and MRI evidence. For GPT-4.1, GPT-5.1, and Gemini-3.1, persistent-minus-stateless all-care correctness differences 37

Table A26: L5 order-only control on 305 turns from 100 patients. Full lines are shuffled within their sections while retaining time markers. Accuracy is in percent; full-history minus shuffled differences and patient-bootstrap 95% intervals are in pp. +/− count corrected/regressed turns. Model

History Shuffled Difference [95% CI] +/−

Gemini-3.1 Qwen3.5-27B Qwen3.6-35B GPT-4.1 Qwen3.5-35B DeepSeek-V3.1 GPT-5.1 Hulu-Med-32B Qwen3-VL-8B Qwen3.5-9B Qwen3-30B Fleming-R1-7B Hulu-Med-4B Lingshu-7B GPT-OSS-20B GPT-OSS-120B HuatuoGPT-o1-7B Hulu-Med-7B HealthGPT-4B HealthGPT-8B

89.5 78.4 75.4 75.1 73.8 71.5 71.5 69.5 68.2 67.9 60.0 59.7 58.0 53.4 52.8 48.9 45.6 43.9 36.4 12.5

86.6 79.0 76.7 77.4 74.4 68.9 71.8 68.5 69.2 69.8 61.0 61.3 59.0 51.8 51.8 45.9 43.9 44.6 35.7 12.1

+3.0 [0.6, 5.7] -0.7 [-2.3, 1.0] -1.3 [-3.2, 0.3] -2.3 [-5.4, 0.7] -0.7 [-3.6, 2.2] +2.6 [-0.3, 5.8] -0.3 [-3.6, 2.9] +1.0 [-0.7, 2.7] -1.0 [-3.4, 1.4] -2.0 [-4.6, 0.3] -1.0 [-3.7, 1.9] -1.6 [-6.3, 3.0] -1.0 [-3.2, 1.0] +1.6 [-0.6, 3.9] +1.0 [-4.9, 6.9] +3.0 [-1.7, 7.7] +1.6 [-1.7, 5.2] -0.7 [-3.7, 2.3] +0.7 [-4.1, 5.7] +0.3 [-2.6, 3.4]

(a) Image evidence

(b) History format DeepSeek-V3.1 Fleming-R1-7B GPT-4.1 GPT-5.1 GPT-OSS-120B GPT-OSS-20B Gemini-3.1 HealthGPT-4B HealthGPT-8B HuatuoGPT-o1-7B Hulu-Med-32B Hulu-Med-4B Hulu-Med-7B Lingshu-7B Qwen3-30B Qwen3-VL-8B Qwen3.5-27B Qwen3.5-35B Qwen3.5-9B Qwen3.6-35B

HealthGPT-4B HealthGPT-8B Hulu-Med-32B Hulu-Med-4B Hulu-Med-7B Lingshu-32B Lingshu-7B Qwen3-VL-30B Qwen3-VL-8B Qwen3.5-27B Qwen3.5-35B Qwen3.5-9B Qwen3.6-27B Qwen3.6-35B −20

12/3 2/4 2/6 8/15 8/10 16/8 16/17 5/2 6/9 5/11 7/10 22/27 5/8 9/4 38/35 32/23 18/13 10/12 39/37 11/10

−10

0

10

−20

Paired difference (pp)

−10

0

10

Paired difference (pp)

Figure A6: Paired controls for evidence and history. (a) Gold versus predicted pathology/MRI inputs: difference in patient-macro care accuracy across 14 local models on the 25 selected patients/146 care decisions. (b) Full history versus the time-stripped, regrouped snapshot: difference in turn accuracy across 20 models on 100 patients/305 turns. Dots show paired differences and bars patient-cluster 95% bootstrap intervals; all models in each control are retained. are 16, 20, and 8 pp, with paired 95% intervals of [−4,36], [−4,44], and [0,20], respectively. These outcomes assess downstream interaction under corrected pathology and MRI evidence on these 25 selected cases.

38

Table A27: What the L5 input conditions preserve and change. Excerpts from one actual prompt triplet; English translations condense repeated source-field names. Record positions refer to the 46 historical lines, excluding baseline/current findings. Complete original-language prompts and the line correspondence are included in the accompanying example. Condition

Cisplatin record

Docetaxel record

Timing and position

Full history

Admission 2, day 26; induction, cycle 2; cisplatin 53.0 mg Induction, cycle 2; cisplatin 53.0 mg Admission 2, day 26; induction, cycle 2; cisplatin 53.0 mg

Admission 2, day 26; induction, cycle 2; docetaxel 105.0 mg Induction, cycle 2; docetaxel 105.0 mg Admission 2, day 26; induction, cycle 2; docetaxel 105.0 mg

Prefixes retained; records 23 and 24

Snapshot Shuffled order

Prefixes removed; records 33 and 42 Full lines retained; records 19 and 29

Answer-option sensitivity. Physician review identifies 71 MCQ/very-short-answer pairs with the same target knowledge. Table A28 reports the number correct under each format and the paired difference. This control measures sensitivity to answer options while holding the knowledge target fixed. Table A28: Physician-audited same-target L1 pairs. MCQ minus option-free accuracy, with paired 95% bootstrap intervals in percentage points. This is an option-cue sensitivity analysis. Model Gemini-3.1 GPT-4.1 GPT-5.1

MCQ correct

VSAQ correct

Difference [95% CI]

67/71 61/71 65/71

62/71 52/71 54/71

7.0 [0.0, 14.1] 12.7 [2.8, 22.5] 15.5 [5.6, 25.4]

MRI presentation. The matched comparison evaluates Qwen3-VL-8B on 25 patients using a static panel, cine composed of the same frames, and an exploratory full-sequence cine condition. T accuracy is 32% in each condition. Same-frame cine minus static accuracy is 0.0 pp (95% CI: −16.0 to 16.0), comprising two wrong-to-right and two right-to-wrong changes. Structure-label micro accuracy changes by 1.3 pp (−9.6 to 11.8). Full-sequence cine changes T accuracy by 0.0 pp and structure accuracy by 2.2 pp (−7.3 to 11.1). The wide intervals leave substantial uncertainty about interface effects. A separate 11-model cine panel evaluates 2,700 card-matched tasks, 200 sequence tasks, and 100 patient-level tasks per model. The supplementary cine dataset provides the videos and their frame correspondence. Primary scoring reports card, sequence, and T-stage accuracy, while auxiliary scoring reports structure and invasion measures. This panel describes cine performance across additional models; the paired static/cine inference rests on the 25-patient comparison above. C.5

Consultation Scoring and Results

O1 evaluates 103 consultation responses from each model against 1,726 clinician-defined criteria. Positive requirements measure what an answer should contain; error criteria identify claims or recommendations that warrant deductions. C.5.1

Scoring Protocol

Table A29 reports the primary O1 ConsultRubrics results. Each model first generates one free-text consultation response for each of the 103 O1 items. Each response is assessed against its item-specific criteria, yielding 1,726 binary criterion judgments per model. For item i, let Ai , Eimaj , and Eimin be the disjoint sets of positive requirements, major-error criteria, and minor-error criteria. Criterion k has a binary judgment zik ∈ {0, 1}, corresponding to criteria_met. A positive-criterion judgment of one means the requirement is satisfied; a negative-criterion judgment of one means the error is present. Each positive anchor has clinician-assigned weight wik ≥ 0 in points. The available positive P score is Bi = k∈Ai wik = 10 points for every item.

39

Each triggered major error deducts two points, and each minor error deducts one point. The raw score ri , normalized score Ri , and primary model score are    X  X X ri = max 0, wik zik − 2 zik − zik ,   k∈Ai k∈Eimin k∈Eimaj (A7) ri 1 X Ri = 100 , O1 = Ri . Bi |V| i∈V

Here V contains items with valid judgments for every criterion. Thus ri ∈ [0, 10] is measured in points, while Ri and O1 lie in [0, 100]. Deductions accumulate across criteria before the item score is floored at zero. The primary mean weights items equally; a clinical-tag score restricts V to items with that tag. All 30 generators have valid judgments for all 103 items, so |V| = 103. s Error P rates have a different unit from score deductions. For severity s ∈ {maj, min}, let Hi = z count triggered criteria. The item-level error rate is s ik k∈E i

ErrorRates =

100 X 1{His > 0}. |V|

(A8)

i∈V

An item with several errors of the same severity contributes once to this percentage, although each error contributes its deduction in Eq. (A7). Figure 5d instead uses criteria as the unit: positive-section rates are the percentage of criteria satisfied, and error-avoidance rates are the percentage of error criteria not triggered. Each criterion counts once in that panel; clinician-assigned weights enter the consultation score in Eq. (A7). In panel (a), r denotes the Pearson correlation across models between major-error rate and O1 score. The primary judge is GPT-OSS-120B with greedy decoding and medium reasoning effort. Its criterion-specific prompt is shown in section E. All 30 generators use this same judge and scoring rule. Physician calibration separately assesses four judges on shared response–criterion pairs, as described in Appendix D. C.5.2

Consultation Scores and Clinical Groups

Table A29 reports the score and the fraction of responses triggering each error severity. GPT-5.1 has the highest mean score, 58.4, but still triggers a major-error criterion in 14.6% of responses. Thus a higher score indicates better coverage and fewer deductions on average, rather than uniformly reliable clinical advice. The 103 questions belong to 42 cases. The primary mean weights questions equally. Averaging first within each case and then across cases changes model scores by −2.0 to +0.1 points and leaves GPT-5.1 highest, as shown in Figure 5b. This comparison shows how the unequal number of questions per case affects the aggregate score. Table A29 separates adverse-event management, diagnosis, treatment selection, special populations, and workup. Elderly and pediatric cases are reported separately here and combined in Figure 2h. Group sizes range from two workup questions to 43 treatment questions; these scores describe the included clinical questions rather than equally precise estimates for each area. Table A29: Consultation scores and error rates under the GPT-OSS-120B judge. Every model uses 103 questions and 1,726 criteria. Major/Minor are percentages of answers with at least one corresponding error. Clinical-group scores cover adverse events (AE; 35 questions), diagnosis (Dx; 11), elderly patients (Eld.; 4), pediatric patients (Ped.; 8), treatment (Tx; 43), and workup (2). All scores lie in 0–100. Model

Score Major Minor

AE

Dx Eld. Ped.

Tx Workup

GPT-5.1 Gemini-3.1 Claude-4.6 Qwen3.6-27B

58.4 47.8 47.5 47.2

69.2 58.4 52.1 61.2

42.6 50.1 41.5 26.5

51.2 39.1 41.5 39.6

14.6 12.6 15.5 24.3

40.8 59.2 58.3 52.4

40

85.4 60.5 76.7 67.5

57.4 35.5 49.8 39.9

60.1 60.1 63.3 65.1

Consultation scores and error rates (continued)

D

Model

Score Major Minor

AE

Dx Eld. Ped.

Tx Workup

Qwen3.5-27B GPT-OSS-120B Qwen3.6-35B Qwen3.5-35B Baichuan-M2-32B Gemini-2.5-Pro Qwen3.5-9B Gemini-2.5-Flash GPT-4.1 Qwen3-VL-30B Fleming-R1-32B DeepSeek-V3.1 GPT-OSS-20B Qwen3-30B Qwen3-VL-8B Fleming-R1-7B HealthGPT-8B HealthGPT-4B Hulu-Med-7B HuatuoGPT-o1-72B InternVL3.5-38B Lingshu-32B Hulu-Med-4B Hulu-Med-32B HuatuoGPT-o1-7B Lingshu-7B

46.8 44.9 42.4 41.4 38.8 38.5 34.7 34.4 33.3 29.1 28.6 28.2 28.0 26.8 25.6 24.8 18.8 14.7 10.7 10.1 9.5 9.0 8.7 7.6 5.3 3.2

59.5 61.8 55.3 55.4 53.5 52.3 43.4 47.9 49.1 44.8 47.4 40.8 50.7 39.4 37.3 39.3 32.0 27.8 21.7 20.1 11.7 17.7 13.7 15.4 10.6 5.9

31.7 29.5 26.9 34.7 25.5 31.8 20.0 28.4 24.5 12.4 23.8 30.3 16.5 17.9 22.9 15.0 24.2 18.2 10.2 17.2 16.2 13.0 13.5 14.7 14.1 9.7

42.2 36.6 31.9 30.5 27.3 26.9 28.0 24.1 22.4 19.4 15.0 17.0 15.6 18.0 15.1 13.6 8.0 3.8 1.5 2.0 6.6 2.6 2.1 1.3 0.5 0.2

19.4 20.4 24.3 21.4 17.5 21.4 28.2 26.2 25.2 32.0 28.2 32.0 29.1 35.0 35.0 38.8 36.9 41.7 49.5 43.7 48.5 54.4 48.5 54.4 50.5 59.2

50.5 49.5 56.3 61.2 58.3 67.0 65.0 72.8 64.1 68.9 68.0 67.0 72.8 72.8 78.6 74.8 82.5 88.3 91.3 86.4 88.3 89.3 90.3 93.2 97.1 92.2

61.3 51.7 88.8 60.9 80.0 70.0 58.8 63.0 42.9 54.6 45.0 46.7 26.3 34.2 41.7 69.6 25.0 14.2 21.3 7.5 0.0 10.0 31.2 0.0 0.0 0.0

27.4 29.3 36.7 33.5 35.3 30.5 33.7 23.2 23.2 20.5 11.5 14.8 13.9 23.1 17.1 12.9 6.4 2.3 9.4 0.5 0.6 0.0 5.4 1.3 0.0 0.0

56.7 59.2 60.0 63.3 33.3 50.9 62.5 39.2 63.3 40.0 53.4 52.6 17.5 44.2 61.7 21.7 28.3 50.9 5.0 11.7 51.7 5.9 5.0 10.0 0.0 5.0

Clinical Review and Judge Calibration

NPC specialists reviewed each item’s clinical question, available evidence, reference answer, and scoring criteria. They checked staging, prior treatment, toxicity, organ function, and guideline assumptions, and standardized drug names, doses, units, and answer options. Future orders and outcomes were removed from inputs when they disclosed the target decision. Question-review outcomes and the six unscored CarePath cohort treatment questions appear in sections A and C.3.5. D.1

Clinical Annotation and Adjudication

In routine care, pathology and radiology examinations were initially interpreted by a physician with 1–3 years of experience and finalized by a senior subspecialist with more than five years of experience. Pathology reference labels were taken from finalized reports. For benchmark MRI annotation, another physician independently reviewed the examinations, reports, and staging criteria to assign structure-invasion and T-category labels. Three senior NPC radiation oncologists resolved disagreements by consensus. The independent benchmark review followed routine clinical reporting and preceded consensus adjudication. Review of clinical decisions. For L2, reviewers checked whether the vignette supplied the facts needed for one guideline decision and whether the answer options supported a determinate choice. Missing variables, leading wording, and inconsistent guideline editions prompted revision. For L4, they assessed whether diagnosis, stage, treatment history, response, and contraindications supported the proposed management plan. The evaluated questions distinguish single-choice and explicit multiple-select items (section A). For L5, review preserved the order of clinical events after date normalization and restricted each question to evidence available at that checkpoint. Distractors differed in evidence use, treatment timing, toxicity management, or follow-up requirements. For O1, specialists separated essential

41

conclusions, supporting case evidence, and conditional recommendations, then assigned major and minor error penalties according to clinical severity. D.2

Physician Calibration Audits

Targeted Audit. For the automated O1 rubric judge, we constructed the Targeted Audit at the same criterion level as the scorer. The packet contained 811 criterion-level labels from 48 modelgenerated responses and was stratified to cover high judge disagreement, large score gaps, clinically safety-sensitive tags, and control cases with expected high or low rubric satisfaction. The physician worksheet marked responses whose completeness required review. Seventeen marked Gemini responses, comprising 263 criterion labels, were excluded because severe truncation prevented adjudication. Marked GPT-5.1 and GPT-OSS-120B responses remained eligible after physician confirmation that their individual criteria could still be judged. This left 548 criterion-level labels for the Targeted Audit agreement analysis. The retained set covers GPT-5.1, Qwen3.6-27B-Instruct, GPT-OSS-120B, and Gemini-3.1-Pro-Preview responses, allowing calibration across both API and local response generators. The 548 labels include 286 from marked responses: 184 GPT-5.1 and 102 GPT-OSS-120B labels. Excluding these responses leaves 262 labels for a sensitivity analysis, which gives the same judge ranking. The Expanded Audit broadens response-generator coverage, as described below. Qwen3.6-27B criterion judging uses non-thinking mode, temperature 0, top-p 1, and a 768-token output limit. Outputs missing required fields are retried up to four times; three remaining Targeted Audit judgments use JSON-schema-constrained decoding with the same prompts and model. Expanded Audit. Physicians annotated 1,017 criteria from 54 responses generated by nine models on six cases: 859 labels were collected with generator identity hidden and 158 came from the Targeted Audit. Comparisons use the 1,013 criteria judged by all four models; four criteria without a Qwen3.6-27B judgment are excluded from every judge’s comparison. Agreement is 77.6% for GPT-4.1, 70.5% for GPT-OSS-120B, 76.9% for Qwen3-30B, and 77.8% for Qwen3.6-27B, with Cohen’s κ of 0.547, 0.387, 0.538, and 0.544. The Targeted Audit (548 labels) and Expanded Audit (1,013 labels) are analysed separately because their response compositions differ. The primary O1 judge uses a criterion-specific template with four calibration examples and explicit interpretation of requirements and errors. D.3

Agreement Measures and Sensitivity Analyses

Within either calibration set, let hℓ , ĥℓ ∈ {0, 1} be the physician and judge labels for criterion ℓ, and let n be the number of valid paired labels. Define TP, FP, TN, and FN as the counts of (hℓ , ĥℓ ) = (1, 1), (0, 1), (0, 0), (1, 0), respectively; they sum to n. A physician-positive label means the criterion is true. For a negative error criterion, this means an error is present, not that the response is clinically correct. Observed agreement po , physician-positive rate qH , judge-positive rate qJ , chance agreement pe , and Cohen’s κ are TP + TN TP + FN TP + FP po = , qH = , qJ = , n n n (A9) po − p e pe = qH qJ + (1 − qH )(1 − qJ ), κ= . 1 − pe Sensitivity is TP/(TP + FN), specificity is TN/(TN + FP), and balanced agreement is their arithmetic mean. The first conditions on physician-positive labels and the second on physiciannegative labels. These rates and po are proportions; percentage agreement is 100po , as in Table 2. The 262-label sensitivity table reports these measures as proportions, with True rate equal to qJ . Cohen’s κ is dimensionless. Each audit is evaluated separately. Agreement by criterion type. Figure 7e,j compares four judges with the 1,013 physician labels in the Expanded Audit, from 54 responses: nine generators each answer six cases. Sensitivity and specificity condition on physician-positive (n = 456) and physician-negative (n = 557) labels. Type-specific agreement counts matching binary judgments among 599 positive requirements, 225

42

Table A30: Coverage of the Targeted Audit for O1. Flagged labels retained are included in the final column and are removed only in the 262-label sensitivity analysis. Response model

Sampled resp.

Excluded resp.

Flagged labels retained

Included labels

15 8 7 18

0 0 0 17

184 0 102 0

264 136 132 16

GPT-5.1 Qwen3.6-27B-Instruct GPT-OSS-120B Gemini-3.1-Pro-Preview

Table A31: Targeted Audit sensitivity after excluding all flagged responses. Agreement statistics are proportions; κ is dimensionless. Acc. is overall agreement, Bal. acc. averages sensitivity and specificity, and True rate is the fraction of labels marked true by the judge. Judge model GPT-4.1 GPT-OSS-120B Qwen3-30B-A3B-Instruct Qwen3.6-27B

n

Acc.

Bal. acc.

κ

Sens.

Spec.

True rate

262 262 262 262

0.767 0.840 0.752 0.782

0.772 0.826 0.760 0.781

0.533 0.665 0.506 0.558

0.802 0.739 0.811 0.775

0.742 0.914 0.709 0.788

0.489 0.363 0.511 0.450

major-error criteria, or 189 minor-error criteria. The positive requirements comprise core answers, key evidence, and conditional modifiers. These measures describe criterion recognition within consultations. The figure’s intervals resample 54 complete response clusters, retaining all criteria within each response. We use the same 10,000 draws for every judge and endpoint (seed 20261029), with the 2.5th and 97.5th percentiles as interval bounds. These responses share six case inputs, so a separate case-level analysis below retains all nine generators when resampling a case. The two analyses quantify variation between responses and between cases. (a) Physician agreement

(b) Sensitivity / specificity

GPT-4.1 GPT-OSS-120B Qwen3-30B Qwen3.6-27B 60

70

80

90

Agreement (%)

50

60

70

80

90

Conditional agreement (%) Sensitivity

Specificity

Figure A7: Physician–judge agreement in the Expanded Audit. All panels use the same 1,013 criterion labels from 54 consultation responses to six cases. (a) Overall agreement and 95% responsecluster bootstrap intervals. (b) Agreement conditional on physician-positive and physician-negative labels (sensitivity and specificity); connecting lines pair the two measures for each judge. Table A32 reports the case-level resampling analysis and a leave-one-case-out check. The former retains all nine generators and every criterion for each sampled case; the latter examines whether agreement depends strongly on one case. These checks quantify sensitivity to the six cases represented in the Expanded Audit.

43

Table A32: Expanded Audit: physician–judge agreement on 1,013 shared criteria, 54 responses and six cases. Case-cluster 95% intervals resample six complete cases 10,000 times. The last column gives agreement after omitting each case in turn. Values are percentages. Judge

Agreement Case-cluster 95% CI Leave-one-case-out range

GPT-OSS-120B Qwen3-30B Qwen3.6-27B GPT-4.1

E

70.5 76.9 77.8 77.6

[66.2, 77.7] [72.3, 81.1] [74.1, 81.6] [72.7, 82.5]

[68.5, 71.8] [75.1, 78.4] [76.8, 79.1] [75.9, 79.2]

Evaluation Prompts

These templates show the task instructions and response formats. Chinese-language instructions are rendered in English. Bracketed fields and ellipses stand for the full case text, options, or anatomical lists supplied in each request; images accompany the multimodal tasks. The O1 judge instructions and four calibration examples are reproduced below. The L2 and L4 cards show the single-choice format. One L2 item and two L4 items explicitly request multiple selections in the question; these require the complete reference option set, with no credit for missing or extra selections. E.1

Guideline and Clinical Choice Tasks

Prompt A1. L1: Guideline memory

Prompt A2. L2: Rule application

Question [guideline question] Options A. [option A] B. [option B] ... Answer with the option letter from the given choices directly.

Context [clinical vignette] Question [application question] Options A. [option A] B. [option B] ... Answer with the option letter from the given choices directly.

Prompt A3. L4: Case planning

Prompt A4. L5: Longitudinal revision

Context [case at the current decision state] Question [management question] Options A. [option A] B. [option B] ... Answer with the option letter from the given choices directly.

Context [patient history up to this turn] Question At this decision point, which next action best matches guideline-concordant NPC specialist management? Options A. [action A] B. [action B] ... Output the final answer directly. It must contain exactly one option letter inside \boxed{}, for example \boxed{A}.

44

E.2

Multimodal Evidence Grounding

Prompt A5. L3: Pathology overview Given the provided whole-slide pathology overview image, determine whether the case is nasopharyngeal carcinoma. Output only 0 or 1. 0 indicates normal tissue, and 1 indicates nasopharyngeal carcinoma.

Prompt A6. L3: MRI sequence and plane You will see one NPC MRI sequence-recognition image composed of a 3×3 panel. All nine images are slices from the same sequence of the same patient. Sequence names and explicit textual cues have been removed. Choose the matching joint sequence-plane class. Options A. T1_Ax; B. T2_Ax; C. T1+C_Ax; D. T1_COR; E. T1_SAG; F. T1+C_SAG; G. T1+C_COR_FS. Output only one option letter inside \boxed{}, for example \boxed{A}.

Prompt A7. L3: MRI structure card You will see one NPC MRI structure card. The card is a fixed 4×3 image. Each row corresponds to one MRI sequence, and each column contains one of three consecutive slices selected from the same sequence by maximizing the annotated structure-box area. The yellow rectangle marks the anatomical structure under evaluation. Row sequences [four sequence names in row order] Tasks Identify the base anatomical structure marked by the yellow box and output structure_id. Determine whether the structure is invaded and output invasion. Select the structure from [18 base-structure choices]. Left and right structures are collapsed for primary scoring. invasion is 0 (not invaded) or 1 (invaded). Response Output only one JSON object, without explanations, markdown, or code fences: {"structure_id":"nasal_passage", "invasion":0}

Prompt A8. L3: T staging from structure cards (100 patients) You will see 27 MRI structure cards from the same NPC patient. Each card has a fixed 4×3 layout; the yellow rectangle marks the structure under evaluation. Tasks Determine invasion status for all 18 base anatomical structures. Integrate this evidence using the eighth edition AJCC/UICC TNM criteria for NPC and output the patient-level T stage. structure_invasion must cover all 18 structures with a value of 0 or 1. t_stage must be T1, T2, T3, or T4. Response Output only one JSON object, without explanations, markdown, or code fences: {"structure_invasion":{"nasal_passage":0, "oropharynx":0, "...":0}, "t_stage":"T1"}

45

Prompt A9. L3: T staging from aligned sequence groups (300 patients) You will see MRI image groups from one NPC patient. Axial groups contain T1, T2, and enhanced T1 at matched anatomical planes. Coronal groups contain enhanced fat-suppressed T1 and precontrast non-fat-suppressed T1 at matched planes. Tasks Determine invasion status for all 18 base anatomical structures. Integrate this evidence using the eighth edition AJCC/UICC TNM criteria for NPC and output the patient-level T stage. The structure list and JSON response fields are the same as in the structure-card T-stage task above. All selected image groups enter one request.

Prompt A10. L3: N staging from aligned sequence groups (400 patients) You will see axial and coronal MRI image groups from one NPC patient, with the same sequence arrangement described above. The axial coverage extends from the skull base to the inferior sternoclavicular joint. Task Determine the N stage using the eighth edition AJCC/UICC TNM criteria for NPC. Response Output only one JSON object containing n_stage and the supporting imaging evidence. n_stage must be N0, N1, N2, or N3.

E.3

CarePath episodes

The pathology and MRI prompts produce diagnostic and staging predictions. Each care question receives the specified image-derived findings and clinical context. Earlier care answers are included only in the separate history-feedback experiments. The care prompt supplies the available clinical context, including N/M/overall-stage information in 133 questions. The clinical context and imageprediction fields jointly define the evidence available for the decision. Prompt A11. L6: Pathology classification Based on the provided pathology image, determine whether the case supports a diagnosis of nasopharyngeal carcinoma. Output only 0 or 1. 0 means the image does not support NPC; 1 means it supports NPC.

Prompt A12. L6: MRI staging You will see MRI panels from the same NPC patient. Each panel combines sequences and planes from the same examination. Tasks Assign 0 or 1 to the invasion status of each candidate structure. Infer the AJCC ninth-edition T stage and briefly state the imaging evidence. The nasopharynx is recorded as the background primary site and excluded from structure-invasion scoring. Evaluate [18 candidate structures]. Response Output only JSON with the following fields: {"structure_invasion":{[structure]:0, ...}, "involved_structures":[...], "T_stage":"T1", "evidence":"[brief imaging evidence]"} T_stage must be T1, T2, T3, or T4.

46

Prompt A13. L6: Care decision Information is provided in clinical order. Answer using only the information available at the current decision point; do not assume findings absent from the question. Prior diagnostic findings [predicted pathology; predicted MRI T stage and invaded structures; supplied N/M and overall-stage fields, where specified] Clinical information [available history and current checkpoint] Question [care decision] Options A. [action A]; B. [action B]; . . . ; [last option and action] Output only one option letter from [valid option letters].

E.4

Consultation Generation and Judging

The response model sees the case and question. The judge additionally sees the generated answer and one signed criterion. A true judgment earns credit for a positive requirement and incurs a deduction for a negative error criterion. Prompt A14. O1: Consultation response You are assisting with a nasopharyngeal carcinoma specialist consultation. Read the de-identified case information and answer the clinical question. Your answer should be clinically concise, evidence-grounded, and should state important uncertainty or missing information when relevant. Case [case_input] Question [question]

Prompt A15. O1: Criterion judge You are evaluating an answer in a nasopharyngeal carcinoma clinical consultation benchmark. Your job is to look at the case, the clinical question, the assistant response, and one rubric item. Judge whether the assistant response matches the rubric item according to the criterion type shown inside the rubric item. Case: [case] Question: [question] Assistant Response: [assistant_response] Rubric Item: [rubric_item] Instructions Return a JSON object with exactly the following fields: "explanation" and "criteria_met". • "criteria_met" must be a boolean. • "explanation" must be a brief string explaining why the assistant response does or does not satisfy the rubric item. Refer to specific content from the assistant response when possible. • Judge only the assistant response. Do not give credit for information that appears only in the case or question unless the assistant response also uses or states it. • First identify the criterion type in the rubric item. • For a POSITIVE REQUIREMENT, return true only when all required elements are substantially satisfied by the assistant response. • For a NEGATIVE ERROR CRITERION, return true only if the assistant response actually makes the described mistake, unsafe recommendation, contradiction, or omission. Return false if the assistant avoids the error, gives the correct answer, or merely discusses the issue as a caveat without committing the error.

47

Prompt A15. O1: Criterion judge (continued) • For a NEGATIVE ERROR CRITERION, a true judgment must be supported by specific offending content in the assistant response. If you cannot identify such content, return false. • If a negative criterion is about omission, such as "fails to mention" or "未说明", return true only when the required information is absent from the assistant response. • If the rubric item uses phrases such as "such as", "for example", "including", "包括", "例如", or "如", the assistant response does not need to mention every example, but it must satisfy the core clinical concept. • Accept clinically equivalent wording, synonyms, abbreviations, and reasonable Chinese-English variants. • Do not require exact wording from the rubric item. • Do not give extra credit for content outside the rubric item. • For a POSITIVE REQUIREMENT, if the response is vague, generic, or only partially satisfies the item, return false. • If the assistant response contradicts the rubric item, return false. • If the answer states uncertainty appropriately and the rubric item allows uncertainty or missing information, count that as satisfying the item.

Calibration example 1

Calibration example 2

If the rubric item is "states that the adverse event is pruritus" and the assistant response says "the patient currently has skin itching", return: { "explanation": "The response identifies the current adverse event as skin itching, which is clinically equivalent to pruritus.", "criteria_met": true }

If the rubric item is "states that the severity cannot be determined from the available information" and the assistant response directly assigns CTCAE grade 2 without explaining missing grading information, return: { "explanation": "The response gives a specific CTCAE grade but does not state that severity cannot be determined from the available information.", "criteria_met": false }

Calibration example 3

Calibration example 4

Negative error criterion present. If the criterion type is NEGATIVE ERROR CRITERION, the rubric item is "incorrectly treats anemia as the main current adverse event", and the assistant response says "the main current adverse event is anemia", return: { "explanation": "The response identifies anemia as the main current adverse event, which matches the mistake described in the rubric item.", "criteria_met": true }

Negative error criterion absent. If the criterion type is NEGATIVE ERROR CRITERION, the rubric item is "misclassifies cT3N0M0 as T4 or N-positive disease", and the assistant response correctly says "the clinical stage is cT3N0M0 because MRI supports T3 and there is no nodal or distant metastasis", return: { "explanation": "The response correctly states cT3N0M0 and does not misclassify the case as T4 or N-positive disease.", "criteria_met": false }

Final instruction. Return only the JSON object. Do not include markdown fences or any other text.

48

F

Illustrated Task Inputs

These examples show the three L3 image interfaces and their reference labels. The English summaries identify the requested outputs; complete task prompts appear in section E. Table A27 separately illustrates how longitudinal text inputs change across the three history conditions. Pathology overview

NPC-positive: 1

Negative control: 0

Task. Determine whether the provided whole-slide overview shows NPC; return 0 (normal tissue) or 1 (NPC). MRI inputs Structure and invasion

Sequence and plane

Identify the structure marked in yellow and determine whether it is invaded. Reference: nasal_passage; invasion=1.

Identify the joint sequence–plane category from a nine-slice panel with sequence-name cues removed. Reference: E, T1_SAG (sagittal T1).

Patient-level staging. T staging uses structure cards for 100 patients and aligned sequence groups for 300 patients. Both formats request all 18 base-structure invasion labels and a T1–T4 category under the eighth-edition AJCC/UICC criteria. N staging uses sequence groups for a disjoint cohort of 400 patients, with inferior coverage extending to the sternoclavicular joint. All images selected for a patient enter one request; a single card does not determine the final stage.

F.1

Reading Model Responses

The following responses illustrate why the scoring targets distinguish anatomical identity from invasion and distinguish an error criterion from a positive requirement.

49

Example 1: Anatomical identity and invasion Evaluation. GPT-5.1, MRI structure-identification experiment. Structure card. Atlas (C1 vertebra). Gold structure and invasion. atlas (C1 vertebra); invasion=0 Prediction. longus_capitis; invasion=0. Interpretation. The numeric invasion value matches, but refers to the wrong anatomical structure. The response fails the joint structure–invasion target; checking the binary value alone would miss this localization error. Example 2: An error criterion that is not triggered Context. In the physician-review packet, a 60-year-old man with recurrent NPC after definitive chemoradiotherapy had received GP plus camrelizumab and was assessed as T3N1. The question asked for the current treatment strategy and guideline basis. Response. Gemini-3.1-Pro-Preview proposed “camrelizumab monotherapy maintenance (preferred systemic strategy)”, with additional workup and review of hepatitis B and prior radiation dose. Error criterion. Choosing local radical treatment, such as re-irradiation or surgery, as the main strategy at this stage instead of systemic treatment. Interpretation. The response centres systemic treatment, so this criterion is false and incurs no deduction. This judgment concerns one criterion; the remaining requirements and errors must still be assessed before scoring the consultation.

50

Related documents

• Uncertainty Estimation in Pathology Foundation Models via Deep Mutual Learning#1187466
• From Research Frontier to Laboratory Bench: Design of a Four-Tier Experimental Teaching System for Multimodal Medical Image Intelligent Diagnosis#1037473
• From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature#1142587
• AdaptMIL: Domain Adaptation for Multiple Instance Learning in Digital Pathology#1411982
• Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity#1117589