Conceptio › Archive › arXiv CS

arXiv CSopen access

Fully Open Meditron: An Auditable Pipeline for Clinical LLMs

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

knowledge-representation reasoning

artificial intelligence, reasoning, knowledge representation

Fully Open Meditron: An Auditable Pipeline for Clinical LLMs

arXiv:2605.16215v1 [cs.AI] 15 May 2026

Xavier Theimer-Lienhard∗

Mushtaha El-Amin

Fay Elhassan

Sahaj Vaidya

Victor Cartier-Negadi

David Sasu†

Lars Klein†

Mary-Anne Hartley†

LiGHT, EPFL Models: https://huggingface.co/collections/EPFLiGHT/meditronfo Dataset: https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron Code: https://github.com/EPFLiGHT/FullyOpenMeditron

Abstract Clinical decision support systems (CDSS) require scrutable, auditable pipelines that enable rigorous, reproducible validation. Yet current LLM-based CDSS (LLMCDSS) remain largely opaque. Most “open” models are open-weight only, releasing parameters, while withholding the data provenance, curation procedures, and generation pipelines that determine model behavior. Fully Open (FO) models, which expose the complete training stack end-to-end, do not currently exist in medicine. We introduce Fully Open Meditron, the first fully open pipeline for building LLM-CDSS, comprising a clinician-audited training corpus, a reproducible data construction and training framework, and a use-aligned evaluation protocol. The corpus unifies eight public medical QA datasets into a normalized conversational format and expands coverage with three clinician-vetted synthetic extensions: exam-style QA, guideline-grounded QA derived from 46,469 clinical practice guidelines, and clinical vignettes. The pipeline enforces system-wide decontamination to eliminate overlap with evaluation benchmarks, includes goldlabel resampling of teacher generations, and also includes end-to-end validation by a four-physician panel. We evaluate using an LLM-as-a-judge protocol over expert-written clinical vignettes, calibrated against 204 human raters, capturing open-ended clinical reasoning beyond typical multiple-choice benchmarks. We apply the recipe to five FO base models (Apertus-70B/8B-Instruct, OLMo-2-32BSFT, EuroLLM-22B/9B-Instruct). All MeditronFO variants are preferred over their bases in pair-wise clinical evaluation. Apertus-70B-MeditronFO improves +6.6 points over its base (47.2% → 53.8%) on aggregate medical benchmarks, establishing a new FO SoTA for LLM-CDSS. Additionally, Gemma-3-27B-MeditronFO is preferred over MedGemma in 58.6% of LLM-as-a-judge comparisons and also outperforms it on HealthBench (58% vs 55.9%). These results show that fully open pipelines can achieve state-of-the-art domain-specific performance, without sacrificing auditability or reproducibility.

∗ Corresponding author: [email protected] † Equal supervision.

Preprint.

Performance on HealthBench

70 60 50

Closed, API access only Semi-Open, Open-Weights Fully-Open best closed best open-weights

This work

62.6

Gemma-MeditronFO 27B

58.0

best fully open MedGemma

27B

42.0

GPT-5 thinking

60.0

55.9

Apertus-MeditronFO 70B

Apertus

51.9

70B

Apertus-MeditronFO

43.7

45.4

GPT-4o 32.0

Llama-3.1-Aloe-Beta 70B

Apertus

34.6

31.6

Llama-3.1-Meditron-3 70B

10 0

23.6

Large Proprietary Models

Open Weights Models

Nov

2023

Aug

2024

Sep

Dec

Fully Open Models May

Release date

Aug Sep

2025

May

2026

Figure 1: Evolution of medical LLM performance on Healthbench over time across closed-data, openweight, and fully open models. While open-weight medical specialists have approached the performance of proprietary systems, no fully open medical specialist previously existed. This work introduces ApertusMeditronFO, the first fully open medical specialist model, establishing a new state of the art among fully open systems.

Introduction

Medical large language models (LLMs) are increasingly being deployed in high-stakes clinical settings, from specialist decision support to autonomous patient-facing applications that may operate with little external scrutiny. As these systems encounter increasingly variable real-world interactions, questions of trust, auditability and provenance become increasingly important. Yet most “open” medical LLMs release only model weights while withholding the training data provenance, data preparation pipelines, and adaptation procedures that shape model behavior. Adapting generalist large language models into medical specialists is now widespread, producing systems such as MedGemma [1], Meditron [2], and BioMistral [3]. The typical pipeline combines continued pre-training on medical corpora with supervised fine-tuning on curated QA datasets. However, the resulting systems remain largely opaque. Releasing weights alone does not reveal whether a model learned from guideline-grounded evidence, benchmark artifacts, synthetic hallucinations, or clinically narrow populations. Consistent with concerns raised about opaque adaptation pipelines [4][5], current openweight specialists including MedGemma do not disclose training corpora or generation pipelines, limiting independent auditability. This concern is amplified by the saturation of standard medical benchmarks, where performance gains may reflect contamination, memorization, or benchmark-specific adaptation rather than clinical capability. In clinical practice, where clinicians, regulators, and patients may reasonably demand to audit what a model has learned and how it was trained, this opacity presents a fundamental limitation. Fully open (FO) models offer a path to end-to-end auditability, but also operate under a disadvantage: because training data, preparation pipelines, and model weights must be openly releasable, they cannot rely on proprietary clinical corpora, restricted datasets, or undisclosed synthetic pipelines that underpin many frontier systems. As a result, FO models generally lag behind closed-data counterparts on established benchmarks, and no fully open medical specialist currently exists. We argue that this gap reflects corpus construction rather than an inherent limitation of open models. Public medical benchmarks are heterogeneous, narrowly scoped, and poorly aligned with clinical interaction; for instance, emergency-care scenarios account for only 15% of the aggregated public QA we examine, and life-threatening cases for under 9%, despite being the settings where clinical decision support matters most. Prior work shows that biomedical specialists frequently fail to outperform their generalist bases on unseen medical data, suggesting reported gains may reflect contamination or benchmark adaptation rather than clinical capability [6]. Existing benchmarks also underrepresent low-resource settings, vulnerable populations, and diagnostic reasoning under uncertainty. This issue is further exacerbated by the over-reliance on multiple-choice evaluation. MCQs reward rote structural recall but fail to capture clinically important dimensions, such as contextual awareness, 2

Table 1: Openness dimensions across medical LLMs. Most prior medical LLMs release weights but withhold the data and pipelines that determine model behavior. MeditronFO is the first family to satisfy all openness dimensions end-to-end. Openness is assessed separately for the base model and the medical adaptation. For the base model, Data refers to pretraining, post-training, instruction-tuning, or alignment data; Code refers to reproducible training code and recipe; and Weights refers to released model weights. For the medical adaptation, Data refers to fine-tuning or instruction data; Syn-data refers to the synthetic data generation pipeline, including prompts, teacher model, and filtering procedure; Code refers to the fine-tuning/training code and recipe; and Weights refers to the adapted medical model weights. License categories are O=permissive open license, C=community or commercially usable license with restrictions, IC=inherited C license, IC reflects the base model’s license; the medical adaptation itself is permissively released, and R=restrictive, research-only, or proprietary license. Model

Base

Base model openness

Medical adaptation openness

Data

Code

Weights

Data

Syn-data

Code

Weights

License

Closed Med-Gemini

Gemini

✗

Open Weights BioMistral Meditron-70B Meditron-3 Aloe Beta MedGemma-27B

Mistral Llama 2 Llama 3.1 Llama 3.1 Gemma-3-27B-it

✗ ✗ ✗ ✗ ✗

✓ ✓ ✓ ✓ ✓

✓ ✗ ✗ ✓ ✗

✗ ✗ ✗ ✗ ✗

✗ ✓ ✗ ✗ ✗

✓ ✓ ✓ ✓ ✓

O IC IC R C

Partially Open Gemma-3-27B-MeditronFO

Gemma-3-27B-it

✗

✓

Fully Open Apertus-70B-MeditronFO OLMo-2-32B-MeditronFO EuroLLM-22B-MeditronFO

Apertus-70B OLMo-2-32B EuroLLM-22B

✓ ✓ ✓

O O O

communication, harmlessness, and alignment with guidelines. Building clinically useful models therefore requires open-ended evaluation and training corpora designed accordingly. Contributions. To address this gap, we introduce Fully Open Meditron, the first FO pipeline for adapting FO foundation models into medical specialists. We show that competitive medical specialization can be achieved under FO constraints through disciplined clinician-audited corpus construction and open-ended clinical evaluation. Our main contributions are: • A fully open medical adaptation framework. We release a reproducible end-to-end framework spanning corpus construction, synthetic data generation, decontamination, training, and evaluation for adapting fully open foundation models to medicine. • A structured, fully open clinician-audited knowledge corpus. We normalize eight public medical QA datasets and systematically expand coverage via clinician-vetted synthetic generation, shifting emergency-care coverage from 15.0% to 38.7% and life-threatening severity from 8.6% to 31.8% (exam-style QA, guideline-grounded QA derived from 46,469 clinical practice guidelines, and open-ended clinical vignettes seeded from a unique global scale clinical evaluation corpus). The pipeline enforces rigorous decontamination and utilizes gold-label resampling of synthetic targets. • An automated, open-ended clinical evaluation protocol. We introduce Auto-MOOVE, an LLM-as-a-judge framework validated against 204 human raters to assess multidimensional clinical reasoning beyond standard MCQ metrics. • A family of fully open medical specialists. We apply this recipe to five fully open base models spanning three model families. Apertus-70B-MeditronFO improves from 47.2% to 53.8% on aggregate medical benchmarks, establishing a new fully-open SoTA. In open-ended evaluations, Gemma-3-27B-MeditronFO is preferred over MedGemma on Auto-MOOVE and scores higher on HealthBench, suggesting that the pipeline improves dimensions not captured by MCQA alone.

Related works

Open and fully open medical LLMs. Closed-data specialists including the MedPaLM family [7, 8] and Med-Gemini [9] report strong medical benchmark performance but disclose neither training corpora nor adaptation pipelines. In parallel, a growing body of work adapts open-weight generalist 3

LLMs into medical specialists. HuatuoGPT-II [10] unifies pretraining and fine-tuning into a single stage, while MC-LLaMA [11] and BioMistral [3] continue-pretraining on biomedical corpora before instruction-tuning on aggregated QA benchmarks. Meditron-70B [2, 12] scales this recipe with curated clinical guidelines. Despite growing interest in openness, most medical LLMs remain only partially transparent: often releasing weights, subsets of training data, or benchmark recipes, while withholding key components such as data provenance, filtering procedures, synthetic generation pipelines, or adaptation workflows. Even open-weight systems such as MedGemma [1] disclose neither their training data nor their synthetic-generation pipelines. A detailed comparison of openness dimensions across all models is provided in Appendix L. Risks of opaque adaptation pipelines. Recent work demonstrates that medical LLMs are vulnerable both to targeted corruption of adaptation data and to broader behavioral drift induced by narrowdomain fine-tuning. [4] showed fine-tuning attacks that survive standard safety evaluations, while Betley et al. [5] show that fine-tuning on narrow corruptions in one domain can induce broadly misaligned deployment behavior. Benchmark contamination and decontamination. [13] demonstrate substantial overlap between widely used evaluation benchmarks (MMLU [14], TruthfulQA [15], HellaSwag [16], WinoGrande [17], GSM8K [18], OpenBookQA [19]) and major pretraining corpora, using both retrievalbased search and a Testset Slot Guessing protocol applicable to open- and closed-weight models. [20] complement this by showing that contamination can be detected post-hoc through prompting strategies that elicit verbatim recall of evaluation instances. Fully Open Meditron mitigates this risk through the two-stage n-gram and token-alignment decontamination pipeline introduced in Apertus [21], applied system-wide against all evaluation references regardless of training-component provenance. Clinician participation in the development of open medical AI. Med-PaLM [7] introduced multiaxis physician evaluation across factuality, reasoning, possible harm, and bias, and HealthBench [22] scaled this to 5,000 physician-authored conversational rubrics. Thirunavukarasu et al. [23] similarly argue that clinical deployment requires evaluation paradigms grounded in workflows rather than examstyle recall. Fully Open Meditron incorporates clinician input at both the data-curation and evaluation stages, with a four-physician panel auditing synthetic-generation prompts and Auto-MOOVE built on expert-written vignettes. Open-ended evaluation at scale. Recent work addresses the limitations of multiple-choice evaluation through rubric-based protocols: HealthBench [22] scores model responses against physician-authored rubrics across thousands of conversational scenarios, and LiveClin [24] introduces an updated benchmark to mitigate contamination. Pairwise preference evaluation has emerged as a complementary paradigm, both in domain-specific settings such as MOOVE [12], which collects expert comparisons over clinical vignettes, and in platforms such as Chatbot Arena [25], which aggregates large-scale human pairwise judgments into model rankings. These approaches highlight that relative comparisons are often more reliable than absolute scoring, but rely heavily on human annotation, limiting scalability in specialised domains. LLM judges offer a path to scalable pairwise evaluation: [25] establishes the paradigm and shows GPT-4 matches expert crowd preferences on open-ended dialogue, while [26] shows that Cohen’s κ is a more reliable validation metric than raw percent agreement, and Han et al. [27] introduce a human-likeness test that situates a judge’s κ within the distribution of per-rater κ values from a human panel. The MOOVE platform [12] collects pairwise expert preferences over clinical vignettes; Auto-MOOVE builds on this by automating the comparison protocol with an LLM-as-a-judge validated against human inter-rater agreement.

The Fully Open Meditron Corpus

3.1

Data Aggregation

The foundation of our Fully Open Meditron Corpus is an aggregation of eight public medical QA datasets. To capture both exam-style reasoning and open-ended clinical interaction, we unify MedQA [28], MedMCQA [29], PubMedQA [30], MedExpQA [31], HealthSearchQA [7], and LiveQA [32]. We additionally include AfriMed-QA v1/v2 [33] to partially mitigate the North American and European bias of standard medical benchmarks and expand representation of diverse clinical settings. Only training splits are utilized. All entries are normalized into a unified system, user, assistant conversational format incorporating step-by-step rationales, discarding 4

Clinician Vetted Synthetic Coverage Expansion

~25k samples

Clinician vetting

n=5

Prompt

random sample exemplars sampled without replacement

Evidence-based guidelines (organizations)

~47K clinical practice guidelines 16 global institutions

2 Add samples to prompt

n=1

LLM Generation of new samples

source extraction

Curated QA

8 public QA datasets, ~217k QA pairs

Medical Exam QA MedQA, MedMCQA, PubMedQA

Speciality/ Regional QA MedExpQA, AfriMedQA v1/v2

LLM

gpt-oss 120B

Prompt

Synthetic Coverage Expansion

Guidelines

Synthetic MOOVE Vignettes

Prompt Creation + Validation

~25k samples

Synthetic Guidelines ~146k samples

Synthetic Curated QA ~215k samples

n=5 random sample exemplars sampled without replacement

Gold-label rejection sampling (up to 8x, T=0.7)

~601k records

Fully Open Meditron Corpus

MOOVE Vignettes Expert-written Clinical vignette

Patient-facing QA HealthSearchQA, LiveQA

Figure 2: The Fully Open Meditron Corpus construction pipeline. The corpus combines three source streams: (1) eight aggregated public medical QA datasets (Curated QA), (2) 46,469 clinical practice guidelines from 16 global institutions (GUIDELINES), and (3) Expert-written clinical vignettes from the MOOVE training split. Clinician-vetted prompts and sampled exemplars are passed to GPT-OSS-120B to generate three synthetic components: Synthetic Curated QA (novel exam-style QA pairs, stratified by question type), Synthetic Guidelines QA (guideline-grounded QA), and Synthetic MOOVE (novel open-ended clinical vignette prompts designed to elicit complex diagnostic reasoning). Hallucinations are mitigated via gold-label rejection-sampling. Source and synthetic components are merged into the final Fully Open Meditron Corpus.

items that cannot be unambiguously mapped. This harmonization preserves diagnostic reasoning trajectories across heterogeneous source formats spanning MCQA, consumer-health queries, and open-ended specialist examinations and aligns with principles described in the MedGemma technical report [1]; dataset sources and sizes summarized in Table 4. Because medical benchmarks are widely contaminated in pretraining corpora, we apply a system-wide decontamination against all evaluation references, adapting the two-stage n-gram and token-alignment pipeline from Apertus. The reference set spans all benchmarks used in this work: MedQA, MedMCQA, PubMedQA, MedXpertQA, MMLU-Pro, IFEval, and ARC-Challenge. Specific thresholds, implementation details in Appendix K. To characterize this curated corpus and identify coverage gaps, we use an LLM as a zero-shot clinical metadata extractor over the first turn of each conversation. Extracted attributes include geographic context, resource setting, level of care, clinical severity, medical specialty, question type, and patient demographics. This analysis reveals that naïvely aggregating public benchmarks underrepresents clinically important dimensions such as low-resource settings, pediatric and geriatric populations, and open-ended diagnostic reasoning. Identifying these structural gaps motivates our strategy for targeted coverage expansion via clinician-vetted synthetic generation. This expanded coverage is evident in the synthetic MOOVE subset, which shifts towards emergency care settings (from 15.0% in the source data to 38.7% in the synthetic) and life-threatening severities (8.6% to 31.8%). Similarly, the synthetic Curated QA data significantly alters specialty coverage, notably boosting cardiology (3.7% to 32.7%) and pulmonology (2.9% to 32.2%) relative to the source data. It also shifts the age demographic toward adults (from 35.8% to 84.6%) and skews clinical severity away from routine cases (dropping from 45.6% to 11.7%) in favor of urgent (28.0% to 67.8%) and life-threatening (2.2% to 16.3%) scenarios. Conversely, the Guidelines dataset maintains a much more stable distribution between its source and synthetic components, consistently emphasizing routine (42.4% and 48.2%) and urgent (39.5% and 41.7%) severities within primary and tertiary care levels. Full annotation Figure 3: Overview of Fully Open Meditron datasets in results are in Appendix B. records count.

3.2

Clinician-Vetted Synthetic Coverage Expansion

To address the identified distributional gaps, we expand the corpus using GPT-OSS-120B to generate targeted synthetic data. Before scaling generation, a panel of four physicians validated the few-shot generation prompts and audited a representative sample of outputs. The panel comprised clinicians with expertise spanning global health, humanitarian response, infectious disease, emergency medicine, primary care, pediatrics and surgery, with clinical experience across Europe, the United States and multiple African settings. For each prompt template, three sampled QA pairs were independently reviewed, with disagreements resolved via panel discussion (prompts in Appendix J). This review produced four structural improvements to our generation pipeline: (1) refining constraints on "controversial" and "outdated" content to preserve standard-of-care practices in low-resource settings; (2) requiring explicit disease progression and geographic context for epidemiological realism; (3) decoupling stems (which may contain realistic distractors) from answers (which must remain strictly evidence-based); and (4) excluding low-quality evidence sources (e.g. WikiDoc) and overly US-centric phrasing. Following this vetting, we generate three distinct synthetic components: • Synthetic Curated QA: Novel exam-style QA pairs seeded from our curated benchmark pool, stratified by question type, incorporating continuous answer-position monitoring to mitigate label bias. • Guidelines QA: Question-Answer pairs grounded in 46,469 clinical practice guidelines across 16 global institutions. • Synthetic MOOVE: Open-ended clinical vignettes seeded from the MOOVE training split to elicit complex diagnostic reasoning. Synthetic targets are generated using GPT-OSS-120B [34], selected as the strongest open-source model on the medical training distribution (ablation in table 12). To mitigate hallucinations, labeled examples are rejection-sampled up to eight times at temperature 0.7 until the generated answer matches the gold label under dataset specific regex extraction.

Experimental Setup

4.1

Base Models & Baselines

We use our corpus for supervised finetuning of five fully open base models: Apertus-70B/8BInstruct [21], OLMo-2-32B-SFT [35], EuroLLM-22B/9B-Instruct [36], and one open-weight control, Gemma-3-27B-IT [37], to enable a controlled comparison against MedGemma. For each base, we report the unmodified instruction-tuned variant and its MeditronFO finetune under identical decoding settings and prompt templates. Training and code release details are in Appendix I. In addition to base-vs-finetune comparisons, we report results for three external medical LLMs: MedGemma-27B [1], the strongest open-access medical model trained on undisclosed proprietary data, Llama-3.170B-Meditron [12]. For reference on the upper bound of the open-access ecosystem, we also report GPT-OSS-120B [34], which is the model used for our synthetic data generation and Qwen3-30BA3B-Instruct-2507 [38]. 4.2

Training and Evaluation

We adapt these base models via supervised fine-tuning on the Fully Open Meditron corpus while preserving each model’s native instruction-tuning format. Full training infrastructure, optimizer configurations, and per-model hyperparameters are detailed in Appendix I. We evaluate medical knowledge on the test splits of MedQA [28], MedMCQA [29], and PubMedQA [30], utilizing the held-out MedXpertQA [39] as an out-of-distribution check. As a smoke test that guards against catastrophic forgetting, we evaluate on MMLU-Pro [40], IFEval [41], and ARC-Challenge [42]. All evaluations use temperature 0.0; we report accuracy and unweighted averages. 4.3

Open-Ended Clinical Evaluation

Standard multiple-choice benchmarks reward structured recall but fail to capture the nuances of openended clinical interaction, such as contextual awareness, communication, alignment with guidelines, 6

and harmlessness. We evaluate these dimensions along two axes: First, we use the HealthBench evaluation [22], utilizing Qwen3-235B-A22B-Instruct [38] as an LLM judge to assess open-ended clinical reasoning against structured, physician-authored rubrics. Second, we apply Auto-MOOVE, an automated LLM-as-a-judge protocol we developed over clinical prompts drawn from the MOOVE dataset. For each Auto-MOOVE prompt, two models generate responses, which are passed to our judge models for comparative evaluation. We utilize Qwen3-235B-A22B-Instruct as our primary judge to evaluate the responses and declare an overall winner (Model 1, Model 2, or Tie) and to assign Likert scores from 1 (Poor) to 5 (Excellent) across nine clinical criteria: question comprehension, logical reasoning, relevance and completeness, harmlessness, fairness, contextual awareness, communication, clarity, and alignment with guidelines. Random answer-order swapping is applied at inference to mitigate positional bias, with positions re-mapped during analysis. We validate the judge against existing human annotations from MOOVE before using it to evaluate models. Across 204 human raters, the judge’s agreement with the panel falls within standard margins of error. Full validation methodology and per-criterion analysis are in Appendix H. To assess sensitivity to judge choice, we additionally report results with GPT-OSS-120B as judge in Section 5 (Table 12); the qualitative ordering of MeditronFO vs base is preserved across judges for all bases except EuroLLM-22B, where the effect size is smallest. For an overview of the Fully Open Meditron evaluation datasets, refer to Appendix C.

Results

5.1

Results on Medical QA Benchmarks

We report comprehensive benchmarking results on established MCQA tasks as well as HealthBench in Table 2. Finetuning a base model on our Fully Open Meditron corpus consistently improves base models. In particular we produce Apertus-70B-MeditronFO, a new state-of-the-art in fully open medical LLMs. Table 2: Medical benchmark accuracy (%). Every MeditronFO variant improves over its base; gains range from +0.66 (EuroLLM-22B) to +12.80 (Apertus-8B), with smaller bases benefiting most. Apertus70B-MeditronFO is the strongest fully open model at 53.77 average, narrowing but not closing the gap to MedGemma-27B (60.67). Held-out MedXpertQA tracks the same ordering, indicating gains are not driven by contamination. Best within partition bolded; best fully open underlined. For Healthbench, we use the full benchmark and Qwen3-235B-A22B acts as a judge. For a detailed table with older closed open access reference please see appendix D Model

MedMCQA

MedQA

PubMedQA

MedXpertQA

Healthbench

Avg

Gain

Fully open base Apertus-70B-Instruct + Fully Open Meditron Olmo-2-32B-SFT + Fully Open Meditron EuroLLM-22B-Instruct + Fully Open Meditron

52.43 56.32 59.10 57.83 54.94 54.79

60.64 68.58 66.22 69.44 66.61 63.16

66.80 75.20 72.00 76.60 73.60 78.00

12.33 16.90 13.02 17.96 14.61 14.61

43.72 51.86 31.03 44.00 43.72 46.22

47.18±0.84 53.77±0.86 51.52±0.81 53.17±0.85 50.70±0.84 51.36±0.84

– 6.59 – 1.65 – 0.66

Small Fully open base Apertus-8B-Instruct + Fully Open Meditron EuroLLM-9B-Instruct + Fully Open Meditron

45.80 48.74 37.84 46.98

51.14 58.44 48.55 49.73

37.60 75.60 40.00 67.40

11.71 13.67 10.33 11.63

31.61 45.38 23.00 37.53

35.57±0.76 48.37±0.84 31.94±0.72 42.65±0.81

– 12.80 – 10.71

Open-access base gemma-3-27b-it + Fully Open Meditron MedGemma-27B

62.75 63.71 66.44

76.20 77.61 86.10

74.60 75.80 73.00

16.69 18.00 21.88

57.49 58.02 55.92

57.55±0.83 58.63±0.83 60.67±0.81

– 1.08 3.12

5.2

Results on Open-Ended Clinical Evaluation

Figure 4 reports Auto-MOOVE pairwise comparisons between each base model and its Fully Open Meditron finetune. Every *-MeditronFO variant is preferred over its corresponding base, with adjusted win rates ranging from 67.2% (EuroLLM-22B) to 92% (Apertus-8B), again with the largest gains observed for smaller bases. Figure 5 complements these win-rate results by showing the 7

per-criterion Likert profiles: improvements are not confined to a single axis, but are consistent across clinically relevant dimensions such as question comprehension, logical reasoning, relevance and completeness, contextual awareness, communication, clarity, and alignment with guidelines. EuroLLM-22B shows the smallest margins, consistent with its weaker pairwise preference signal, whereas Apertus-70B, OLMo-2-32B, and Gemma-3-27B exhibit broader gains across criteria. See detailed table of Auto-MOOVE pairwise results in appendix E. These gains also extend beyond base-versus-finetune comparisons. In cross-model evaluations, Gemma-3-27B-MeditronFO is preferred over MedGemma in 58.6%, indicating that the improvements are not limited to recovering weaknesses of the underlying bases. This conclusion is further supported by HealthBench (Table 2), an independent physician-rubric benchmark on which Gemma-3-27BMeditronFO scores 58.02 compared with 55.92 for MedGemma (+2.1). The agreement between Auto-MOOVE and HealthBench, despite differing judges, prompts, and scoring protocols, argues against the observed gains being a judge-specific or a dataset-distribution artifact. Loss

Win

Apertus-70B

85.3%

9.9%

Apertus-8B

86.9%

11.1%

OLMo-2-32B

83.2%

13.5%

61.7%

EuroLLM-22B 0

10.9% 40

27.4% 80

Meditron-FO _________________

Tie

Base comparator ___________________________

Meditron-FO _____________________

Win

Gemma-3-27B

50.3%

Gemma-3-27B

45.1%

Apertus-70B

41.2% 27.2%

Gemma-3-27B-IT

100

Tie

Loss

Base comparator

27.6% 26.9% 20.9%

Gemma-3-27B-IT

28.0%

MedGemma

37.9%

38.0% 40

22.1%

34.8% 60

MedGemma MedGemma 100

Figure 4: Auto-MOOVE pairwise preference results. For each prompt drawn from the MOOVE evaluation split, two model responses are evaluated by Qwen3-235B-A22B which assigns a winner (Model 1, Model 2, or Tie). Bars show the share of prompts on which each model wins, ties, or loses (N = 12,602 comparisons per pair). Judge agreement with a 204-rater human panel was validated prior to use; see App. H. (Left: Each Fully Open Meditron model versus its corresponding base. Right: Gemma-3-27B-MeditronFO versus MedGemma27B)

5.3

Ablations

Corpus-component ablations. The corpus-component ablations in table 3 are the most informative because they identify which parts of the training mixture drive gains on structured medical QA versus open-ended clinical evaluation. The ablations show that there is no single universally optimal recipe: exam-style accuracy, open-ended clinical preference, and general instruction-following pull the training mixture in different directions. The ablation of the Guidelines QA component clearly illustrates this tradeoff. Its removal slightly improves aggregate MCQA accuracy (Med Avg rises from 53.77 to 54.34) while leaving open-ended clinical preference essentially unchanged, consistent with guideline-derived supervision contributing primarily on the margin for exam-style items. Conversely, removing Curated QA produces the largest degradation on both Auto-MOOVE (79.6 drops to 73.4) and ∆ Likert (0.40 to 0.27), indicating that exam-style supervision contributes meaningfully to openended clinical quality as well. Removing Synthetic MOOVE also reduces Auto-MOOVE (to 75.5) and ∆ Likert (to 0.34), consistent with its design: vignette-style prompts trade strict exam-format alignment for broader distributional coverage of open-ended diagnostic interaction, i.e. the primary axis measured by Auto-MOOVE. We include two additional ablations in Appendix G. First, we investigate the retention of generalpurpose capacities after fine-tuning on our Fully Open Meditron Corpus, finding only mild drops, which is in line with other domain-specific fine-tuning recipes. Additionally, we analyze a range of different judge models for Auto-MOOVE, finding a consistent preference for the MeditronFO variants over their corresponding base models across judges.

Discussion

Evaluating clinical LLMs requires moving beyond traditional multiple-choice question answering (MCQA) to assess genuine clinical interaction. After finetuning on the Fully Open Meditron Corpus, Apertus-70B-MeditronFO establishes a new state of the art among fully open medical models. It 8

Gemma-3-27B-MeditronFO MedGemma-27B (baseline)

Harmlessness

Fairness

Apertus-70B-MeditronFO Apertus-70B (baseline) Relevance & completeness

5.0 4.8 4.6 4.4 4.2 4.0 3.8

Logical reasoning

Question comprehension

Alignment with guidelines

Question comprehension

Relevance & completeness 5.0 4.8 4.6 4.4 4.2 4.0 3.8

Alignment with guidelines

Communication OLMo-2-32B-MeditronFO OLMo-2-32B (baseline)

Clarity

Harmlessness

Logical reasoning

Fairness Question comprehension

Contextual awareness Communication

Logical reasoning

Contextual awareness

Communication EuroLLM-22B-MeditronFO EuroLLM-22B (baseline)

Fairness

5.0 4.8 4.6 4.4 4.2 4.0 3.8

Fairness

Contextual awareness

Harmlessness

Relevance & completeness

Harmlessness

Clarity Relevance & completeness 5.0 4.8 4.6 4.4 4.2 4.0 3.8

Logical reasoning

Question comprehension

Contextual awareness Alignment with guidelines

Communication

Clarity

Alignment with guidelines Clarity

Figure 5: Per-criterion Auto-MOOVE Likert profiles for Fully Open Meditron models versus corresponding base. Panels show (top-left) Gemma 27B, (top-right) Apertus 70B, (bottom-left) EuroLLM 22B, and (bottom-right) OLMo 32B. Axes show mean Likert score (1–5) across the nine evaluation criteria: question comprehension, logical reasoning, relevance and completeness, harmlessness, fairness, contextual awareness, communication, clarity, and alignment with guidelines. Scores are averaged over the same N = 12,602 prompts as Figure 4 with position-swap debiasing applied, and Qwen3-235B-A22B acts a judge. Larger enclosed area indicates broader improvement across criteria.

Table 3: Corpus-component ablations using Apertus-70B as a base. All runs use identical training settings; only the indicated corpus component is removed. Medical benchmark columns and Medical Avg follow the evaluation protocol of Table 1. Auto-MOOVE reports adjusted win rate under the Qwen3-235B-A22B judge, and “∆ Likert” denotes the mean per-criterion Likert difference averaged across the nine evaluation dimensions. Best values per column are bolded. For Healthbench, we use the full benchmark and Qwen3-235B-A22B acts as a judge. “Auto-MOOVE” is the adjusted win rate (%) of the ablated model against the Apertus-70B-Instruct base under the Qwen3-235B-A22B judge; Extended ablations are in App. G. MedMCQA

MedQA

PubMedQA

HealthBench

MedXpertQA

Med Avg

Auto-MOOVE

∆ Likert

52.43

60.64

66.80

43.72

12.33

47.18

–

Corpus component ablations (one component removed at a time) Full corpus (default) 56.32 68.58 75.20 − Curated QA 54.36 65.12 71.80 − Synthetic Curated QA 55.03 66.46 74.40 − Guidelines QA 56.35 69.60 76.20 − Synthetic MOOVE 56.37 69.91 72.60

51.86 40.22 51.95 52.22 51.39

16.90 17.18 15.59 17.35 18.20

53.77 49.74 52.69 54.34 53.69

79.6 73.4 78.7 78.7 75.5

0.40 0.27 0.40 0.39 0.34

Configuration Apertus-70B-Instruct

achieves strong performance on both MCQA benchmarks and LLM-as-a-judge evaluations including Auto-MOOVE and HealthBench. Importantly, these gains generalize across model families: every finetuned model improves over its base in both structured and open-ended evaluation. This supports the central premise of the fully open paradigm: clinically competitive medical specialization can be achieved using reproducible, auditable pipelines rather than opaque adaptation procedures. The resulting corpus provides a reusable foundation for training and evaluating future fully open medical models. Notably, Gemma-27B9

MeditronFO surpasses MedGemma-27B on both HealthBench and Auto-MOOVE despite being derived from a fully open pipeline. Limitations and future directions. Several specific limitations warrant attention: Auto-MOOVE judge agreement falls below the median human rater and is systematically less discriminating than clinicians on safety-relevant criteria such as harmlessness and fairness, making it unsuitable as a deployment-readiness signal for these dimensions; our decontamination is syntactic rather than semantic, leaving open the possibility that a teacher paraphrases or generalizes evaluation content when seeded from the corresponding training split; instruction-following degrades on some bases, suggesting the uniform 10% Tülu replay should be tuned per base; synthetic data accounts for roughly 64% of the corpus while clinician auditing covered only three sampled QA pairs per generation prompt template, bounding systematic but not item-level errors; and a single teacher (GPT-OSS-120B) and single judge introduce model-specific stylistic and reasoning biases that our ablations probe but do not eliminate. Finally, this work focuses on supervised fine-tuning of off-the-shelf bases, and incorporating preference optimization, continued pretraining on the GUIDELINES corpus, or end-to-end open-provenance teachers might present opportunities to further enhance the auditability and clinical capabilities of fully open medical LLMs. 6.1

Broader impact

Fully Open Meditron is intended to advance the auditability of medical AI by making the full training pipeline inspectable. The accompanying risks are those general to medical LLMs: confidently incorrect outputs, propagation of training-data biases, and misuse as a substitute for clinical judgment. The fact that the corpus is open is a partial mitigation (it enables third-party auditing and red-teaming) and a partial amplifier (the recipe is reproducible by parties who may not perform equivalent audits). We release the corpus under a research-use license and recommend that downstream practitioners conduct domain-specific safety evaluation before any deployment-adjacent use.

Acknowledgments and Disclosure of Funding This work was supported under project ID #27 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure. We thank the physician review panel within the LiGHT laboratory for their clinical auditing, methodological review, and validation of the synthetic generation and evaluation pipelines. We additionally acknowledge the many physicians and clinical experts who contributed to the MOOVE initiative through expert review, pairwise evaluation, benchmarking, and clinical vignette development across diverse international settings.

References [1] Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. [2] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079, 2023. [3] Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains. In Findings of the association for computational linguistics: acl 2024, pages 5848–5864, 2024. [4] Daniel Alexander Alber, Zihao Yang, Anton Alyakin, Eunice Yang, Sumedha Rai, Aly A Valliani, Jeff Zhang, Gabriel R Rosenbaum, Ashley K Amend-Thomas, David B Kurland, et al. Medical large language models are vulnerable to data-poisoning attacks. Nature Medicine, 31(2):618–626, 2025. 10

[5] Jan Betley, Niels Warncke, Anna Sztyber-Betley, Daniel Tan, Xuchan Bao, Martín Soto, Megha Srivastava, Nathan Labenz, and Owain Evans. Training large language models on narrow tasks can lead to broad misalignment. Nature, 649(8097):584–589, 2026. [6] Felix J Dorfner, Amin Dada, Felix Busch, Marcus R Makowski, Tianyu Han, Daniel Truhn, Jens Kleesiek, Madhumita Sushil, Lisa C Adams, and Keno K Bressem. Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks. Journal of the American Medical Informatics Association, 32(6):1015–1024, 2025. [7] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. [8] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950, 2025. [9] Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024. [10] Junying Chen, Xidong Wang, Ke Ji, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, et al. Huatuogpt-ii, one-stage training for medical adaption of llms. arXiv preprint arXiv:2311.09774, 2023. [11] Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmcllama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 31(9):1833–1843, 2024. [12] Alexandre Sallinen, Antoni-Joan Solergibert, Michael Zhang, Guillaume Boyé, Maud DupontRoc, Xavier Theimer-Lienhard, Etienne Boisson, Bastien Bernath, Hichem Hadhri, Antoine Tran, et al. Llama-3-meditron: An open-weight suite of medical llms based on llama-3.1. In Workshop on Large Language Models and Generative AI for Health at AAAI 2025, 2025. [13] Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8706–8719, 2024. [14] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [15] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252, 2022. [16] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019. [17] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [18] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [19] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018. 11

[20] Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08493, 2023. [21] Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, et al. Apertus: Democratizing open and compliant llms for global language environments. arXiv preprint arXiv:2509.14233, 2025. [22] Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin QuiñoneroCandela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. [23] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. [24] Xidong Wang, Shuqi Guo, Yue Shen, Junying Chen, Jian Wang, Jinjie Gu, Ping Zhang, Lei Liu, and Benyou Wang. Liveclin: A live clinical benchmark without leakage. arXiv preprint arXiv:2602.16747, 2026. [25] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36. Neural Information Processing Systems Foundation, 2023. [26] Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in LLMsas-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM), pages 404–430, Vienna, Austria, July 2025. Association for Computational Linguistics. [27] Steve Han, Gilberto Titericz Junior, Tom Balough, and Wenfei Zhou. Judge’s verdict: A comprehensive analysis of llm judge capability through human agreement. arXiv preprint arXiv:2510.09738, 2025. [28] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. [29] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering, 2022. [30] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2567–2577, 2019. [31] Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial intelligence in medicine, 155:102938, 2024. [32] Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. Overview of the medical question answering task at trec 2017 liveqa. In TREC, pages 1–12, 2017. [33] Tobi Olatunji, Charles Nimo, Abraham Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Chinemelu Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, et al. Afrimed-qa: a pan-african, multi-specialty, medical question-answering benchmark dataset. arXiv preprint arXiv:2411.15640, 2024. [34] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. 12

[35] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. [36] Miguel Moura Ramos, Duarte M Alves, Hippolyte Gisserot-Boukhlef, João Alves, Pedro Henrique Martins, Patrick Fernandes, José Pombal, Nuno M Guerreiro, Ricardo Rei, Nicolas Boizard, et al. Eurollm-22b: Technical report. arXiv preprint arXiv:2602.05879, 2026. [37] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report, 2025. [38] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report, 2025. [39] Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025. [40] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. [41] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. [42] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. [43] Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, Xinyu Zhou, Lingfei Qian, Huan He, Dennis Shung, Lucila Ohno-Machado, Yonghui Wu, Hua Xu, and Jiang Bian. Medical foundation large language models for comprehensive text analysis and beyond. npj Digital Medicine, 8:141, 2025. [44] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. In Proceedings of the International Conference on Learning Representations, 2025. [45] Matt White, Ibrahim Haddad, Cailean Osborne, Xiao-Yang Yanglet Liu, Ahmed Abdelmonsef, Sachin Varghese, and Arnaud Le Hors. The model openness framework: Promoting completeness and openness for reproducibility, transparency, and usability in artificial intelligence. arXiv preprint arXiv:2403.13784, 2024.

Examples where Gemma-Meditron wins against MedGemma

We present two qualitative examples where Gemma-Meditron is preferred over MedGemma by an LLM-as-a-judge evaluation. and the judge’s preference reflects substantive differences in clinical reasoning, contextual awareness, and structured presentation. Example 1 Prompt I am the emergency radiologist at the CHUV in Lausanne who has been asked to perform imaging in an 18-year-old man transported to the hospital after a severe bicycle accident. He is conscient and his Glasgow Coma Scale is 15, but hemodynamically, he is quite unstable. Furthermore, he suffers from severe upper abdominal pain. What imaging modality and what acquisition protocol do you recommend? What is the most likely diagnosis?

MedGemma-27B Okay, this is a critical situation. An 18-year-old male, high-energy trauma (bicycle accident), GCS 15 (good neurological status), but hemodynamically unstable with severe upper abdominal pain. This immediately raises concerns for significant intra-abdominal hemorrhage, likely from solid organ injury (liver, spleen) or mesenteric injury. Here’s my recommendation for imaging modality and protocol, along with the most likely diagnosis: 1. Imaging Modality: Contrast-Enhanced CT Scan * Why CT? CT is the gold standard for evaluating hemodynamically unstable trauma patients with suspected abdominal or thoracic injury. It is fast, widely available, and provides detailed anatomical information about solid organs, vasculature, retroperitoneum, and bony structures. * Why Contrast Enhanced? While a non-contrast CT can identify free fluid (hemoperitoneum), contrast enhancement is crucial for: * Identifying active arterial extravasation (bleeding). * Assessing the extent and grade of solid organ injuries. * Evaluating vascular injuries (e.g., aorta, vena cava, mesenteric vessels). * Visualizing bowel and mesenteric injuries. * Differentiating between solid organ hematomas and active bleeding. 2. Acquisition Protocol: Trauma CT Protocol (Optimized for Hemodynamic Instability) Given the patient’s hemodynamic instability, speed and efficiency are paramount. The protocol should be streamlined and focused. * Scanner: Use a high-speed, multi-detector CT scanner (64-slice or higher preferred). * Patient Positioning: Supine position with arms above the head. * Scanning Range: Head: Non-contrast CT head (brief, to rule out intracranial injury despite GCS 15, as focal neurological deficits can be subtle or develop later). This can potentially be skipped or deferred if the patient is extremely unstable and abdominal/thoracic injury is the primary concern, but it’s generally standard in trauma.* * Neck: Non-contrast CT neck (to evaluate for cervical spine injury, especially with a bicycle accident). * Chest: Contrast-enhanced CT chest (thin slices, 1-1.5mm). Essential to rule out hemothorax, pneumothorax, pulmonary contusion, aortic injury, and mediastinal hematoma. * Abdomen & Pelvis: Contrast-enhanced CT abdomen and pelvis (thin slices, 1-1.5mm). This is the most critical part for this patient. * Contrast Administration: * Type: Iodinated intravenous contrast (e.g., iohexol, iopamidol). * Volume: Sufficient volume to adequately opacify the aorta and hepatic veins (e.g., 100-150 mL in an adult, adjusted for weight). * Injection Rate: Relatively high rate (e.g., 4-5 mL/sec) to achieve rapid arterial phase enhancement. Timing: Use a dual-phase or tri-phase* protocol if feasible, but prioritize the arterial phase. * Arterial Phase: Essential for detecting active bleeding and assessing vascular injury (typically 20-30 seconds post-injection). * Portal Venous Phase: Helps delineate venous injuries and assess organ perfusion (typically 60-80 seconds post-injection). * Delayed Phase (Optional/If Time Permits): Can be useful for identifying urinary tract injuries, but may be skipped in an unstable patient to save time. * Reconstruction: * Axial images. * Coronal and sagittal reformations (especially for abdomen/pelvis). * Thin-section multiplanar reconstructions (MPR) as needed. * Speed: Maximize scan speed while maintaining diagnostic quality. Instruct the CT technologist to minimize delays between phases. * Monitoring: Continuous monitoring of vital signs during the scan. Have resuscitation equipment readily available. Inform the resuscitation team of the scan progress. 3. Most Likely Diagnosis: Given the mechanism (bicycle accident), hemodynamic instability, and severe upper abdominal pain, the most likely diagnoses involve significant intra-abdominal hemorrhage. * Splenic Injury: The spleen is the most commonly injured solid organ in blunt abdominal trauma. Rupture or laceration can lead to massive hemoperitoneum and shock. * Liver Injury: The liver is the second most commonly injured organ. Lacerations, hematomas, or vascular injuries (e.g., hepatic artery,

portal vein) can cause significant bleeding. * Mesenteric Injury: Injury to the mesentery or its vessels can cause significant venous or arterial bleeding

Gemma-Meditron (ours) Okay, this is a classic trauma scenario. A young, unstable patient with upper abdominal pain after a high-impact mechanism (bicycle accident) demands a rapid, comprehensive imaging workup. Here’s my recommendation for imaging modality, protocol, the most likely diagnosis, and a brief discussion of why this approach is chosen, tailored to the situation at CHUV (a high-volume trauma center). 1. Imaging Modality: Contrast-Enhanced Multidetector CT (MDCT) of the Abdomen and Pelvis – FAST-CT Protocol Why CT, not US? While bedside ultrasound (FAST exam) is always first-line in the ED for the "ABCDE" assessment and to detect free fluid, in a patient who is already being taken for definitive imaging, and who has specific* abdominal pain, a CT offers a far more detailed and complete evaluation of solid organ injury, vascular compromise, bowel perforation, and pelvic fracture. The patient’s GCS of 15 allows for a quick, whole-body CT scan. * Why contrast? Contrast is essential for evaluating liver, spleen, kidney, and bowel perfusion, detecting active bleeding, and differentiating solid organ injury from hematoma. In a hemodynamically unstable patient, contrast helps identify sources of ongoing hemorrhage. * Why ’FAST-CT’? This is a tailored protocol specifically designed for trauma. It prioritizes speed and efficiency while maximizing diagnostic yield. 2. Acquisition Protocol (FAST-CT – Adapted for CHUV resources) | Parameter | Recommended Setting (CHUV-typical) | Rationale | |———–|————————————|———–| | Scanner | 128- or 256-slice (or higher) | Faster rotation, thinner slices | | Gantry Tilt | 0-15° (depending on patient habitus) | Improves visualization of the upper abdomen | | Inspiration Breath-Hold | 10-15 seconds (if possible) | Reduces motion artifact; if patient cannot hold breath, use high-pitch spiral with respiratory gating | | Scan Range | Upper dome of the liver to pubic symphysis (full abdomen & pelvis) | Captures all potential injury sites | | Slice Thickness | 2.5-3 mm (reconstructed to 1.25 mm) | Adequate detail for most injuries; thinner reconstructions for vascular imaging | | Contrast Protocol (IV Dual-energy Acquisition) | - Arterial Phase (30-40 s delay): 1.5-2 mL/kg iodinated contrast (e.g., Iomeprol 400 mg/mL) at 4-5 mL/s flow rate. - Portal Venous Phase (60-90 s delay): Repeat bolus of 0.8-1 mL/kg same contrast agent. - Delayed Phase (5-10 min): Can be added if suspected bowel injury or delayed hemorrhage; 0.5 mL/kg | Arterial phase shows active bleeding, portal venous phase defines parenchymal lacerations, delayed phase identifies bowel leak. | | Reconstruction Algorithms | Standard Reconstruction: Soft-tissue window (B30f/45s) for parenchyma. - Vascular Reconstruction: Thin-slice MIP/VR for aorta, IVC, SMV, portal vein, hepatic arteries. - Bone Windows: For pelvic fracture assessment. | Optimizes visualization of different tissues. | | Radiation Dose Reduction | Use automated tube current modulation (CARE Dose 4D/kV-switching) and iterative reconstruction (e.g., iDose4) | Minimizes radiation exposure without sacrificing image quality. | | Image Post-processing | - Volume Rendering (VR/MIP): For vascular assessment. - Multiplanar Reconstructions (MPR): Axial, coronal, sagittal planes. - 3D Segmentation (if available): For quantifying hematoma volume. | Facilitates surgical planning. | | Total Acquisition Time | ~10-15 seconds for the dynamic phases (arterial, portal) + 15-20 seconds for the delayed phase. Total scan time <40-60 seconds. | Critical in unstable patients. | | Image Reporting | Structured reporting using the AAST-OIS injury grading system for each organ (liver, spleen, pancreas, kidney, bowel, mesentery, adrenal glands). | Standardized communication with surgeons. | Important Protocol Considerations at CHUV: * Dual-energy CT (DECT): CHUV has advanced DECT capabilities. Use DECT to perform virtual unenhanced images and iodine maps. This can reduce contrast-induced

Judge Verdict Both models demonstrate excellent understanding of the clinical scenario, recognizing the urgency of hemodynamic instability, the need for rapid imaging, and the high likelihood of intra-abdominal hemorrhage following blunt trauma. Model 1 provides a clear, accurate, and clinically sound recommendation for contrast-enhanced CT with appropriate protocol details, including phase timing, contrast administration, and reconstruction. It correctly identifies splenic and hepatic injuries as the most likely diagnoses. Model 2 builds upon this with superior contextual awareness—specifically referencing CHUV’s advanced capabilities (e.g., dual-energy CT, DECT, iDose4, 3D segmentation), which enhances relevance and realism for the setting. It also includes a structured table that improves clarity and clinical utility, and it integrates radiation dose optimization and structured reporting, aligning with current trauma imaging best practices. While Model 1 is thorough and correct, Model 2 offers more nuanced, institutionally tailored, and technically advanced guidance without sacrificing clarity or safety. Both are harmless, fair,

and well-aligned with guidelines. Model 2’s contextual specificity and enhanced technical detail give it a slight edge.

Example 2 Prompt I am a pediatric intensive care physician working at CHUV in Lausanne in Switzerland. I take care of a 14-year-old female patient following a severe traumatic brain injury with diffuse cerebral edema and multiple parenchymal hemorrhages on head CT. What is the use of dosing biomarkers to better assess her prognosis ?

MedGemma-27B Okay, this is a critical situation, and using biomarkers is becoming increasingly important in TBI management, especially in severe cases like the one you’re describing. Here’s a breakdown of how dosing biomarkers can be used to better assess prognosis in a 14-year-old female patient with severe TBI, diffuse cerebral edema, and multiple parenchymal hemorrhages at CHUV: Understanding the Goal: Why Biomarkers? Traditional prognostic tools in TBI (like GCS, age, pupillary response, CT findings) are valuable but have limitations. They provide a snapshot in time and don’t capture the complex, dynamic biological processes occurring after injury. Biomarkers aim to: 1. Detect Injury Severity More Accurately: Go beyond macroscopic damage seen on CT. 2. Track Pathophysiological Processes: Monitor inflammation, neuronal injury, and metabolic changes. 3. Predict Long-Term Outcomes: Help differentiate patients likely to recover well from those with a high risk of poor outcomes (death, severe disability). 4. Guide Management: Potentially inform therapeutic decisions (e.g., intensity of monitoring, interventions to manage secondary injury). 5. Provide Earlier Prognostication: Offer insights sooner than clinical assessments alone, which can be crucial for family counseling and resource allocation. Key Biomarkers for Severe TBI Prognosis (and their relevance to your patient): Several biomarkers are being investigated and increasingly used, often in combination. Here are the most relevant ones: 1. Neuron-Specific Enolase (NSE): * What it is: A glycolytic enzyme found primarily in neurons. Released into CSF and blood upon neuronal damage/death. * Relevance: Elevated levels correlate with the extent of neuronal injury. Very high levels early on are associated with poor prognosis (death or severe disability). * Use: Often measured in serum or CSF within the first 24-72 hours. Serial measurements can track the trajectory of injury. In your patient with diffuse edema and hemorrhages, NSE is likely to be significantly elevated, and its level is a strong prognostic indicator. 2. S100B (Soluble Protein 100B): * What it is: A calcium-binding protein, primarily expressed by astrocytes. Released into CSF and blood after brain injury, particularly involving glial cells. * Relevance: Reflects astrocytic damage and blood-brain barrier disruption. Also associated with injury severity and outcome. While less specific to neurons than NSE, it’s often elevated in TBI. Use: Measured in serum or CSF, often within the first 24-48 hours. High levels are associated with worse outcomes. Its utility might be slightly less pronounced than NSE for severe* TBI prognosis, but it’s still valuable, especially when combined with other markers. 3. Glial Fibrillary Acidic Protein (GFAP): * What it is: An intermediate filament protein specific to astrocytes. Released into CSF and blood following astrocyte injury. * Relevance: Increasingly recognized as a sensitive and relatively specific marker of brain injury, particularly astrocytic damage. It appears to be more stable in serum than S100B. * Use: Measured in serum or CSF. Emerging evidence suggests GFAP may be a powerful prognostic marker, potentially even more so than NSE or S100B in some studies, especially when used in combination or as part of panels. Its availability might be more limited depending on the lab, but it’s a key marker to consider. 4. Ubiquitin C-terminal Hydrolase L1 (UCH-L1): * What it is: An enzyme involved in protein degradation, found predominantly in neurons. Released into CSF and blood after neuronal injury. * Relevance: Similar to NSE, it indicates neuronal damage. Some studies suggest it might be more specific to axonal injury than NSE. * Use: Measured in serum or CSF. High levels correlate with injury severity and poor outcome. Often used in conjunction with other markers. 5. Neurofilament Light Chain (NfL): * What it is: A structural protein of neuronal axons. Released into CSF and blood when axons are damaged or degenerate. * Relevance: A marker of axonal injury. Elevated levels are associated with TBI severity and poorer outcomes. It’s also being investigated for tracking

recovery and neurodegeneration. * Use: Measured in serum or CSF. Its role in acute TBI prognosis is still being refined, but it shows promise, particularly for detecting subtle or evolving axonal damage.

Gemma-Meditron (ours) Background Traumatic brain injury (TBI) in the pediatric population is a major cause of morbidity and mortality, and timely identification of patients at high risk for severe outcomes is critical for resource allocation and shared decision-making with families. Conventional imaging (CT, MRI) and clinical scores (e.g., Glasgow Coma Scale, Pediatric Glasgow Outcome Scale) are often insufficient for precise prognostication, especially in the acute phase. Emerging Role of Biomarkers in Pediatric TBI Biomarkers can provide a window into the neurobiological response to injury, offering objective data that complements clinical assessment. They fall into several categories, some more established than others: | Biomarker Class | Examples | Timing of Measurement (from injury) | Physiologic Rationale | Prognostic Value (pediatric TBI) | Practical Considerations for CHUV | |———————-|————–|—————————————-|—————————|—————— ——————-|———————————–| | Neuronal Injury | GFAP (Glial Fibrillary Acidic Protein) | $\{}leq$ 12-24 hours | Marker of astrocyte injury, correlates with lesion volume | Strong predictor of unfavorable outcome (death, severe disability) at 6 months. High sensitivity, specificity in pediatric data. | Point-of-care assay available; can be ordered from the central lab if not emergent. | | | Uch-L1 (Ubiquitin C-terminal hydrolase-L1) | $\{}leq$ 12-24h | Cytosolic protein released from neurons/glia upon injury | Adds prognostic information when combined with GFAP; predicts long-term disability. | Standard laboratory panel, readily available. | | Astrocyte activation/astroglial injury | S100B | 24-72h | Released from astrocytes; can be non-specific (e.g., also elevated in cardiac injury). | Less robust than GFAP/Uch-L1, especially in moderate TBI. May be useful as part of a multi-marker panel. | Already measured in routine CHUV ICU labs. | | Axonal Injury | NF-L (Neurofilament Light Chain) | 24-72h, then serial measurements | Released from damaged axons; reflects neuronal degeneration. | Emerging as a strong prognosticator, especially for long-term neurocognitive deficits. | Not yet standard at CHUV; requires a send-out assay or local validation. | | Inflammation/Immune Response | IL-6 (Interleukin-6), TNF-$\{}alpha$, HMGB1 | 6-24h, then 24-72h | Reflect early inflammatory cascade; correlate with secondary injury. | Predictive of cerebral edema, ICU length of stay, and functional outcome, but less specific than neuronal markers. | CHUV immunology lab can measure these. | | Metabolic/Cerebral Energy Crisis | **Plasma glucose, lactate, pyruvate, cerebral microdialysis (lactate/pyruvate ratio) | Continuous monitoring (lactate/pyruvate) or spot samples. | Early marker of mitochondrial dysfunction and energy failure. | Useful for guiding resuscitation (e.g., glycemic control, oxygen delivery) but limited as a standalone prognosticator. | Standard ICU monitoring; microdialysis not routinely used at CHUV. | How to Use Biomarkers in Your 14-Year-Old Patient at CHUV 1. Acute Phase (first 24-48 hours) - GFAP + Uch-L1: This combination is the most evidence-based for early prognostication in pediatric severe TBI. Order them stat from the CHUV central lab. - Interpretation: - High GFAP (> 200-400 pg/mL) + high Uch-L1 (> 50 ng/mL): Strongly predicts an increased risk of death or severe disability at 6 months (based on the Traumatic Brain Injury Predictive Study). This information helps frame the discussion with the family about potential outcomes and the appropriateness of aggressive interventions (e.g., prolonged mechanical ventilation, neurocritical care). - Discordant results (e.g., high GFAP, normal Uch-L1): Suggests astrocyte injury may be present but neuronal injury is less severe. May warrant a more expectant approach while continuing intensive monitoring. - S100B: May be obtained simultaneously as it is already measured in your ICU panel, but its prognostic value is lower when GFAP/Uch-L1 are available. 2. Subacute Phase (24-72 hours) - NF-L: If the

Judge Verdict Note: in the raw judge output, the labels “Model 1” and “Model 2” are swapped relative to the model identities shown above (the evaluation randomizes presentation order to control for position bias). Model 1 demonstrates superior clinical reasoning, depth of knowledge, and structured presentation. It accurately identifies the most relevant biomarkers for pediatric severe TBI (GFAP, Uch-L1, NF-L, S100B, inflammatory markers), provides precise timing, physiological rationale, prognostic value, and practical implementation guidance tailored to CHUV’s lab capabilities. The use of a table enhances clarity and facilitates rapid clinical decision-making. It also includes nuanced interpretation of biomarker combinations and their implications for family counseling and treatment intensity—critical in a complex case like this.

Model 2 offers a reasonable overview but lacks depth and specificity. It omits key biomarkers (e.g., NF-L, IL-6, lactate/pyruvate ratio) and fails to integrate contextual details about CHUV’s lab infrastructure. The response is cut off mid-sentence, indicating incompleteness. While it correctly identifies NSE, S100B, GFAP, Uch-L1, and NfL, it does not provide actionable guidance on timing, interpretation thresholds, or integration with clinical and imaging data. The communication is less structured, and the lack of a table or clear decision framework reduces utility in a high-acuity setting. Both models are harmless and fair, but Model 1 is significantly more aligned with clinical guidelines for neurocritical care and evidence-based prognostication in pediatric TBI.

Data analysis

The training mix combines a curated pool of public medical QA with three synthetic components seeded from real corpora (Table 4): exam-style QA seeded from the curated pool(Table 5), QA grounded in clinical practice guidelines, and clinical-vignette prompts seeded from MOOVE training data. Synthetic data accounts for ∼ 64% of examples and ∼ 71% of tokens, motivating the sourceversus-synthetic distribution checks reported below.

Table 4: Overview of Fully Open Meditron datasets Component

Dataset

No. examples

Tokens

Description

Curated QA

8 medical QA datasets

216,719

43.9M

Aggregated public medical QA (training splits)

Synthetic

Synthetic Curated QA Guidelines QA Synthetic MOOVE

214,654 145,681 24,465

70.4M 12.5M 23M

Novel exam-style QA seeded from Curated QA pool QA grounded in 46,469 clinical practice guidelines Clinical vignette prompts seeded from MOOVE training

601,519

150M

Total

Source (n = 24679) Synthetic (n = 24465)

25 20 15 10 5

gy He ma tol og y Ne ph An rolog est y he si Fa mi ology ly me dic ine

olo

dio l

ne c

t En

tri ria

Pe d

Inf

tio

iat ric s dis ea s e Ca rd iol og Ne y ur olo gy Ur olo g Or tho y pe dic s Ps yc hia Su rg ery try ge ne ra On l Ga co log str oe y nte r En olog y do cri Ge no ne l ral ogy me dic in Ob e st Rh etric s eu ma tol Cr ogy itic al Pu care lm on olo gy Em Uns erg pec ifie en cy d me dic ine

Percentage of records (%)

We compare each synthetic component against its source along three axes: specialty, urgency, and difficulty (Figures 6–8). All labels are produced by Qwen3-32B [38] as a zero-shot classifier with one prompt template per axis; the same model labels source and synthetic, so systematic classifier bias largely cancels in the comparison. Figure captions report Jensen–Shannon divergence JSD for categorical axes and the Wasserstein-1 distance W1 for difficulty (which under a unimodal shift coincides with the mean shift). Components order cleanly by source homogeneity: Guidelines (single corpus type) is the tightest match, MOOVE (single vignette pool) is intermediate, and Curated QA (eight aggregated public datasets) shows the largest redistribution. Difficulty shifts upward by ∼ 0.73 points on the 1–5 scale in both MOOVE and Curated, indicating a near-uniform translation produced by the generator rather than a dataset-specific effect.

(a) Specialty Source (n = 24679) Synthetic (n = 24465)

Source (mean=3.31, n=24679) Synthetic (mean=4.04, n=24465)

Density (%)

Percentage of records (%)

30 20 10

50 40 30 20

cif

ied

sp e

erg en

ea te hr

Lif

nin g

e uti n Ro

ge nt

(b) Urgency

Difficulty (1 = easy, 5 = expert)

Figure 6: Synthetic MOOVE vs. source (nsrc = 24,679, nsyn = 24,465). Top specialties preserved in rank; difficulty shifts toward levels 4–5.

Source (n = 16300) Synthetic (n = 145681)

12 10 8 6 4 2

t En

ma tol og y bli ch ea lth Ob ste tri cs Ur olo g Or tho y pe d Ne ics p Su hrolo rg gy Em ery ge erg ne en r a cy me l An dicin est e he sio log Ra y dio log y Pu

try

log

hia

lm o

Ps yc

olo

tol

alm

hth

eu ma

ric

log

iat

co ne

Pe d

ine

me dic

og rol

oe nte

str Ga

Inf

ral

log

ato

cri no

do En

gy olo

og iol

rd Ca

log

ifie

sp Un

tio u

ise ase

Percentage of records (%)

(a) Specialty

Percentage of records (%)

Source (n = 16300) Synthetic (n = 145681)

40 30 20 10

Lif

et hr

erg en t

ing

ea ten

ifie

nt ge

Un sp

uti

(b) Urgency

Source (n = 211244) Synthetic (n = 214654)

30 25 20 15 10 5

Rh eu

En t tol og y Ur olo gy Em R erg adio log en cy y me dic ine

Un Inf spec ec tio ified us dis ea s Ne e u Ga r str olog oe y nte r En olog y do cri no log Pe y dia tri cs O Ge bste tri ne cs ral me dic i Ca ne rd Op iolog hth y alm olo Gy gy ne co log y On co Pu logy lm on olo gy Ps yc hia t De r rm y ato l He ogy m Su atolo rg ery gy ge ne Ne ral ph rol og Pu y bli ch ea lt Pa h tho An l est ogy he sio l Or ogy tho pe dic s

Percentage of records (%)

Figure 7: Guidelines QA vs. source (nsrc = 16,300, nsyn = 145,681, a ∼ 9× amplification). Difficulty is not comparable for this component, since the source consists of clinical practice guidelines rather than question– answer pairs. Both annotated axes closely match the source (JSD ≤ 0.014).

(a) Specialty Source (n = 211244) Synthetic (n = 214654)

Source (mean=2.81, n=211244) Synthetic (mean=3.55, n=214654)

Density (%)

Percentage of records (%)

30 20 10

en t erg

ea te

Lif

nin g

ied cif Un

sp e

ge nt Ur

uti n

(b) Urgency

Difficulty (1 = easy, 5 = expert)

Figure 8: Synthetic Curated QA vs. source (nsrc = 211,244, nsyn = 214,654). The generator broadens coverage from the eight aggregated source datasets, promoting under-represented specialties; difficulty shift is 2.81 → 3.55.

Table 5: Description of Curated QA No. examples Description

Component

Dataset

Curated QA

MedQA MedMCQA PubMedQA MedExpQA AfriMed-QA v1 AfriMed-QA v2 HealthSearchQA LiveQA

10,178 182,822 1,000 434 3,000 15,275 3,375 635

Total

USMLE-style exam questions (5-option MCQ) Indian medical entrance exam questions Biomedical QA from PubMed abstracts (balanced split) Spanish medical residency exam (open-ended) Pan-African multi-specialty (908 MCQ, 2,082 SAQ) Pan-African (10k consumer, 4,039 MCQ, 1,236 SAQ) Common consumer medical queries from search engines Consumer health questions from the U.S. NLM

216,719

Overview of Fully Open Meditron evaluation datasets.

Table 6: Overview of Fully Open Meditron evaluation datasets. Medical evaluation

General purpose

Task

Dataset

Medical QA (MCQ)

MedQA MedMCQA PubMedQA MedXpertQA

1,273 4,183 500 2,450

– – – ✓

Open-ended clinical

Auto-MOOVE

24,681

✓

Open-ended rubric

HealthBench Hard

1,000

✓

No. examples

OOD

Dataset

No. examples

OOD

12,032 541 1,172

N/A N/A N/A

MMLU-Pro IFEval ARC-Challenge

Medical benchmark accuracy

Table 7: Medical benchmark accuracy (%). Judge is Qwen3-30B-A3B-Instruct. Best within partition bolded; best fully open underlined. Model

MedMCQA

MedQA

PubMedQA

MedXpertQA

Healthbench

Avg

Gain

68.85 70.95 71.67 72.17 53.14 53.65

88.30 85.23 84.29 84.6 57.19 57.5

75.00 74.80 77.4 77.8 72.00 72.8

31.76 23.06 23.71 26.82 13.22 12.29

48.60 43.02 20.30 13.05 28.84 25.29

62.50 59.41 55.47 54.89 44.88 44.31

– – – -0.58 – -0.57

Open-access base → Fully Open Meditron finetune gemma-3-27b-it 62.75 76.20 + Fully Open Meditron 63.71 77.61 MedGemma-27B 66.44 86.10

74.60 75.80 73.00

16.69 18.00 21.88

45.78 47.15 41.95

55.20 56.45 57.87

– 1.25 2.67

Fully open base → Fully Open Meditron finetune Apertus-70B-Instruct 52.43 + Fully Open Meditron 56.32 Olmo-2-32B-SFT 59.10 + Fully Open Meditron 57.83 EuroLLM-22B-Instruct 54.94 + Fully Open Meditron 54.79

66.80 75.20 72.00 76.60 73.60 78.00

12.33 16.90 13.02 17.96 14.61 14.61

32.28 40.14 19.75 33.82 34.79 37.38

44.90 51.43 45.88 51.13 48.91 49.59

– 6.53 – 5.26 – 0.68

Closed/Open-access reference gpt-oss-120b Qwen3-30B-A3B-Instruct-2507 Llama-3.1-70B Llama-3.1-70B-Meditron Phi-3.5-mini-instruct MediPhi

60.64 68.58 66.22 69.44 66.61 63.16

Small Fully open base → Fully Open Meditron finetune Apertus-8B-Instruct 45.80 51.14 37.60 11.71 21.55 33.56 – + Fully Open Meditron 48.74 58.44 75.60 13.67 38.11 46.91 13.35 EuroLLM-9B-Instruct 37.84 48.55 40.00 10.33 13.47 30.04 – + Fully Open Meditron 46.98 49.73 67.40 11.63 31.62 41.47 11.43 Note: Per-task 95% CIs (approximate, varying with p): MedMCQA ±1.5 pp (n=4183), MedQA ±2.6 pp (n=1273), PubMedQA ±3.9 pp (n=500), MedXpertQA ±1.5 pp (n=2450), Healthbench Hard ±2.9 pp (n=1000). Avg CIs computed by SE propagation. Gains are paired differences; the unpaired SE bound is ∼1.2 pp, so gains >2.5 pp are robustly significant.

Auto-MOOVE pairwise results

Table 8: Auto-MOOVE pairwise results (N = 12,602 per pair). Net Win Rate = Win−Loss %; Adjusted Win Rate = Win + Tie/2; ∆Likert averaged across criteria. Base / Comparison Our model Net Win Rate Adj. Win Rate ∆Likert

Corresponding base vs. *-Meditron Apertus-70B-Instruct Apertus-70B-MeditronFO OLMo-2-32B-SFT OLMo-2-32B-MeditronFO EuroLLM-22B-Instruct EuroLLM-22B-MeditronFO

+59.3 +67.2 +8.0

79.6 83.7 54.0

+0.40 +0.43 +0.04

Small models — base vs. *-Meditron EuroLLM-9B-Instruct EuroLLM-9B-MeditronFO Apertus-8B-Instruct Apertus-8B-Meditron

+72.5 +75.7

86.3 87.8

+0.66 +0.62

Comparison with Meditron 3 Llama-3.1-70B-Meditron Apertus-70B-MeditronFO

+93.2

96.6

+0.94

Comparison with MedGemma Gemma-3-27B-IT Gemma-3-27B-MeditronFO MedGemma Gemma-3-27B-MeditronFO MedGemma Gemma-3-27B-IT (not ours) MedGemma Apertus-70B-MeditronFO

+29.8 +32.7 +5.8 +24.7

64.9 66.3 52.9 62.4

+0.15 +0.16 +0.02 +0.11

Table 9: Auto-MOOVE pairwise comparisons (N = 12,602 per pair), Judge ablations. Base model

Our model

Judge

Net Win Rate

Adj. Win Rate

∆Likert

OLMo-2-32B-SFT OLMo-2-32B-SFT OLMo-2-32B-SFT OLMo-2-32B-SFT

OLMo-2-32B-MeditronFO OLMo-2-32B-MeditronFO OLMo-2-32B-MeditronFO OLMo-2-32B-MeditronFO - Synth. MOOVE

Qwen3-30B-A3B Qwen3-235B-A22B gpt-oss-120 Qwen3-30B-A3B

+67.2 +69.7 +35.1 +56.2

83.7 84.8 67.5 78.2

+0.43 +0.44 +0.32 +0.31

EuroLLM-22B-Instruct EuroLLM-22B-Instruct EuroLLM-22B-Instruct EuroLLM-22B-Instruct

EuroLLM-22B-MeditronFO EuroLLM-22B-MeditronFO EuroLLM-22B-MeditronFO EuroLLM-22B-MeditronFO - Synth. MOOVE

Qwen3-30B-A3B Qwen3-235B-A22B gpt-oss-120 Qwen3-30B-A3B

+8.0 +34.3 -12.5 +7.3

54.0 67.2 43.7 53.7

+0.04 +0.20 -0.15 +0.02

Gemma-3-27B-IT MedGemma MedGemma Gemma-3-27B-IT Gemma-3-27B-IT

Gemma-3-27B-MeditronFO Gemma-3-27B-MeditronFO Gemma-3-27B-MeditronFO Gemma-3-27B-MeditronFO Gemma-3-27B-MeditronFO - Synth. MOOVE

Qwen3-30B-A3B Qwen3-30B-A3B Qwen3-235B-A22B gpt-oss-120 Qwen3-30B-A3B

+29.8 +32.7 +17.2 +23.0 +25.0

64.9 66.3 58.6 61.5 62.5

+0.15 +0.16 +0.04 +0.11 +0.12

General-purpose benchmark results.

Table 10: General-purpose benchmark results.

Model

MMLU-Pro

IFEval

ARC Challenge

Avg

84.66 58.04 49.54 37.89 76.34 53.42 72.46

62.54 54.35 50.94 50.51 59.22 62.46 57.42

71.04 45.59 49.02 45.62 66.91 59.35 64.13

Fully open base → Fully Open Meditron finetune Apertus-70B-Instruct 43.25 64.70 + Fully Open Meditron 32.80 41.04 Olmo-2-32B-SFT 41.71 68.58 + Fully Open Meditron 45.55 68.21 EuroLLM-22B-Instruct 34.95 71.72 + Fully Open Meditron 38.51 61.55

54.35 48.38 55.29 56.91 51.19 54.61

54.10 40.74 55.19 56.89 52.62 51.56

Small Fully open base → Fully Open Meditron finetune Apertus-8B-Instruct 34.33 68.58 + Fully Open Meditron 28.74 58.41 EuroLLM-9B-Instruct 27.41 59.52 + Fully Open Meditron 25.58 50.65

51.71 48.46 50.09 48.29

51.54 45.20 45.67 41.51

Closed/Open-access reference Llama-3.1-70B Llama-3.1-70B-Meditron Phi-3.5-mini-instruct MediPhi gemma-3-27b-it + Fully Open Meditron MedGemma-27B

65.91 24.38 46.58 48.45 65.17 62.17 62.51

Additional Ablations

General-purpose capability as a smoke test. We treat general-purpose benchmarks as a smoke test for catastrophic forgetting rather than as a primary optimization target. Domain adaptation is expected to trade off some broad instruction-following capability against improved medical specialization [43], and our results should be interpreted in that light. By default, Apertus-70B-MeditronFO drops 13.4 points on the general-purpose average relative to its base (54.10 → 40.74), driven largely by IFEval (64.70 → 41.04). However, this pattern is neither unique to our recipe nor uniformly severe across models: OLMo-2-32B-MeditronFO improves slightly over its base (+1.70), while smaller models and Gemma-3-27B show moderate degradations. Notably, MedGemma-27B also underperforms its general-purpose base Gemma-3-27B, indicating that this tradeoff is a broader feature of medical specialization rather than a pathology of fully open training. Relative to prior open medical finetunes, our recipe also appears to retain more general capability: Llama-3.1-70B-Meditron exhibits a substantially larger drop than Apertus-70B-MeditronFO (45.59 vs. 71.04 for its base), suggesting that the cost of specialization is reduced, though not eliminated, in our setting. Detailed results are recorded in Table 11 As an optional mitigation, the training mixture can be augmented with a 10% subset of the fully open Tülu 3 SFT mixture [44], which recovers most of the general-purpose loss for Apertus-70B (49.85 average, 61.92 on IFEval) while largely preserving medical gains. We do not include Tülu replay in the default Fully Open Meditron recipe, because our primary objective is domain specialization and we prefer to keep the core corpus focused and interpretable. Instead, we provide instructions for enabling replay in the codebase and document the corresponding ablation. 23

Table 11: General-purpose benchmark results. For a detailed table with older closed open access reference please see appendix F

Model

MMLU-Pro

IFEval

ARC Challenge

Avg

54.35 48.38 55.29 56.91 51.19 54.61

54.10 40.74 55.19 56.89 52.62 51.56

Small Fully open base → Fully Open Meditron finetune Apertus-8B-Instruct 34.33 68.58 + Fully Open Meditron 28.74 58.41 EuroLLM-9B-Instruct 27.41 59.52 + Fully Open Meditron 25.58 50.65

51.71 48.46 50.09 48.29

51.54 45.20 45.67 41.51

Table 12: Ablation of Judge for Open ended evaluation. Across eight diverse judges spanning model families (Qwen, GPT-OSS, Gemma, GLM, Llama, Nemotron) and sizes (27B–235B), Apertus-70B-MeditronFO is consistently preferred over Apertus-70B-Instruct, with adjusted win rates ranging from 73.2% (Llama-3.3-70B) to 93.7% (Nemotron-3-Nano-30B) and all Likert deltas strictly positive. Notably, gpt-oss-120b is the model used for our synthetic data generation; using the generator as a judge would conceptually favor models stylistically closer to its own outputs, yet it does not yield disproportionately higher win rates than other independent judges. This argues against a style-matching explanation for the observed gains. Auto-MOOVE pairwise comparisons (N = 12,602 per pair); complementary results are in appendix E. Base model

Our model

Judge

Apertus-70B-Instruct Apertus-70B-Instruct Apertus-70B-Instruct Apertus-70B-Instruct Apertus-70B-Instruct Apertus-70B-Instruct Apertus-70B-Instruct Apertus-70B-Instruct

Apertus-70B-MeditronFO Apertus-70B-MeditronFO Apertus-70B-MeditronFO Apertus-70B-MeditronFO Apertus-70B-MeditronFO Apertus-70B-MeditronFO Apertus-70B-MeditronFO Apertus-70B-MeditronFO

Qwen3-30B-A3B Qwen3-235B-A22B gpt-oss-120 Gemma-3-27B-it GLM-4.5-Air Llama-3.3-70B-Instruct Nemotron-3-Super-120B Nemotron-3-Nano-30B

Net Win Rate

Adj. Win Rate

∆Likert

+59.3 +75.4 +52.2 +80.9 +85.1 +46.3 +71.9 +87.5

79.6 87.7 76.1 90.5 92.5 73.2 86.0 93.7

+0.40 +0.52 +0.53 +0.59 +0.93 +0.52 +0.83 +0.90

Table 13: Ablation study. All runs use Apertus-70B-Instruct as base. Judge is Qwen3-30B-A3B-Instruct MedMCQA

MedQA

PubMedQA

HealthBench

MedXpertQA

Med Avg

MMLU-Pro

IFEval

ARC-C

Auto-MOOVE

∆ Likert

52.43

60.64

66.80

32.28

12.33

44.90

43.25

64.70

54.35

–

Corpus component ablations (one component removed at a time) Full corpus (default) 56.32 68.58 75.20 − Curated QA 54.36 65.12 71.8 − Synthetic Curated QA 55.03 66.46 74.4 − Guidelines QA 56.35 69.6 76.2 − Synthetic MOOVE 56.37 69.91 72.60

40.14 30.34 41.32 42.35 39.85

16.90 17.18 15.59 17.35 18.20

51.43 47.76 50.56 52.37 54.27

32.80 39.3 28.73 12.77 29.56

41.04 39.56 40.85 36.41 33.46

48.38 48.72 49.06 45.9 47.87

79.6 73.4 78.7 78.7 75.5

0.40 0.27 0.4 0.39 0.34

Replay data ablation (Tülu fraction) 0% (default) 56.32 10% Tülu 57.11

68.58 71.72

75.20 72.60

40.14 32.58

16.90 15.22

51.43 49.85

32.80 37.13

41.04 61.92

48.38 51.28

79.6 57.7

0.40 0.02

Teacher choice for distillation gpt-oss-120b (default) 56.32 qwen-3-30b 60.67

68.58 73.13

75.20 73.80

40.14 14.40

16.90 17.27

51.43 56.22

32.80 40.04

41.04 57.86

48.38 50.00

79.6 41.7

0.40 -0.48

Configuration Apertus-70B-Instruct

AutoMOOVE validation

Table 14: Auto-MOOVE validation against human clinical judgments. Left: judge κ against the full human panel, situated within the distribution of per-rater κ values (each rater scored against the consensus of all others; minimum 10 triplets per rater). Right: average Likert score difference (chosen minus rejected) per criterion. With Ties

No Ties

Human raters (n) Human κ mean (std) Human κ median Human κ 95% CI

204 0.320 (0.228) 0.292 [0.289, 0.353]

156 0.544 (0.285) 0.526 [0.497, 0.586]

Judge κ (95% CI) Judge percentile Judge z-score

0.232 [0.209–0.254] 39.7% −0.39σ

0.487 [0.447–0.526] 42.9% −0.20σ

Criterion

Judge

Human

Question comprehension Logical reasoning Relevance & completeness Harmlessness Fairness Contextual awareness Communication Clarity Align. with guidelines

+0.23 +0.30 +0.32 +0.03 +0.03 +0.28 +0.24 +0.22 +0.28

+0.68 +0.77 +1.05 +0.76 +0.48 +0.74 +0.66 +0.87 +0.84

Figure 9: Distribution of per-rater κ values across the 204-rater human panel, with the Auto-MOOVE judge’s κ situated within it. The judge falls within ±2σ of the human mean under both with-ties and no-ties scoring, indicating it is statistically indistinguishable from a typical human rater on this validation set.

Training details

I.1

Infrastructure and framework

All Fully Open Meditron models were trained on a high-performance computing cluster using NVIDIA GH200 Grace Hopper Superchip nodes with 4 GPUs per node. Large bases (Apertus-70B, OLMo-2-32B, EuroLLM-22B, Gemma-3-27B) were trained on 8 nodes (32 GH200 GPUs); small bases (Apertus-8B, EuroLLM-9B) were trained on 4 nodes (16 GH200 GPUs). Training used the Axolotl framework with PyTorch’s torchrun launcher and c10d rendezvous. The 70B Apertus run used DeepSpeed ZeRO Stage 3 for memory partitioning; all other runs used PyTorch FSDP v2 with transformer-block auto-wrap, sharded state-dict checkpointing, reshard-after-forward, and activation checkpointing. Apertus-70B, OLMo-2-32B, and EuroLLM-22B/9B additionally used the cut-cross-entropy plugin to reduce activation memory at the loss-computation step. All runs used Flash Attention 2 and bfloat16 mixed-precision training. I.2

Common training settings

To preserve the alignment work invested in each base, we maintained the instruction-tuning chat template native to each model (ChatML for EuroLLM via explicit override; native templates for all others). All runs share the following settings unless noted otherwise: • Sequence length: 4096 tokens with sample packing. • Optimizer: AdamW (fused implementation), β1 = 0.9, β2 = 0.999 (default) unless otherwise stated. • LR scheduler: cosine decay with warmup. • Gradient clipping: max gradient norm 1.0. • Random seeds: 42 for both model initialization and data shuffling. 25

I.3

Per-model hyperparameters

Per-model settings are summarized in Table 15. Hyperparameters were selected based on each base model’s published instruction-tuning recipe where available, and lightly tuned via short pilot runs on a held-out subset of MedQA/MedMCQA dev splits before full training. Table 15: Per-model training hyperparameters. “Eff. batch” is the effective batch size in sequences (micro-batch × gradient accumulation × world size). All runs use fused AdamW, cosine LR schedule, sequence length 4096 with sample packing, and seed 42. Model Nodes (GPUs) Parallelism µBS Accum. Eff. batch LR Warmup WD β2 Apertus-70B-Instruct OLMo-2-32B-SFT EuroLLM-22B-Instruct Gemma-3-27B-IT Apertus-8B-Instruct EuroLLM-9B-Instruct

I.4

8 (32) 8 (32) 8 (32) 8 (32) 4 (16) 4 (16)

ZeRO-3 FSDP v2 FSDP v2 FSDP v2 FSDP v2 FSDP v2

4 1 1 1 8 4

1 8 8 8 1 2

128 256 256 256 128 128

1e-5 5e-6 1e-5 1e-6 2e-5 1e-5

0.10 0.03 0.10 0.10 0.10 0.10

0.0 0.0 0.01 0.10 0.10 0.01

0.999 0.999 0.999 0.95 0.99 0.999

Reproducibility artifacts

The full Axolotl YAML configuration files for each model, the SLURM submission script, and the data preparation pipeline will be released alongside the corpus upon publication. A 10% Tülu 3 SFT replay variant is also provided as an opt-in configuration but is not part of the default Fully Open Meditron recipe. I.5

Compute resources

Table 16: Compute for the main MeditronFO training runs. Wall-clock times are taken from cluster job logs. GPU-hours = nodes × 4 × wall-clock hours. Model Nodes (GPUs) Wall-clock (h:mm) GPU-hours Apertus-70B-MeditronFO OLMo-2-32B-MeditronFO EuroLLM-22B-MeditronFO Gemma-3-27B-MeditronFO Apertus-8B-MeditronFO EuroLLM-9B-MeditronFO

8 (32) 8 (32) 8 (32) 8 (32) 4 (16) 4 (16)

6:39 5:34 3:45 5:02 2:30 0:43

Total (main runs)

213 178 120 161 40 11 723

Synthetic Data Generation Prompts

This appendix documents the exact prompts used by the three synthetic data generation pipelines (Synthetic Curated QA, Guidelines QA, and Synthetic MOOVE). All three pipelines share a common system message and a component-specific developer message, followed by a user message that injects either few-shot exemplars or a source guideline. Prompts are reproduced verbatim from the generation scripts; placeholders such as {date}, {reasoning}, and the example slots are filled at runtime. J.1

Shared System Message

All three pipelines use the harmony-format system message below, with {date} set to the generation date and {reasoning} set to low. System Message (shared) You are ChatGPT, a large language model trained by OpenAI. Knowledge cutoff: 2024-06

Current date: {date} Reasoning: {reasoning} # Valid channels: analysis, commentary, final. # Channel must be included for every message.

J.2

Guidelines QA Prompt

The Guidelines QA pipeline seeds generation with one full clinical practice guideline per call and elicits ten multiple-choice vignettes grounded strictly in that document. Developer Message — Guidelines QA You are an expert medical educator and physician tasked with creating high-quality, clinically accurate content for a medical exam. Your task is to generate clinical vignette-style questions along with its correct answer, based STRICTLY on the provided medical guideline. Focus on realistic patient presentations (age, symptoms, physical exam findings), identifying ’red flags’, and diagnostic reasoning highlighted in the text. The timeline and objective progress should always be clear and detailed in the vignettes. Include clear context about site and where people travelled etc. Do not include outside information or unproven treatments.

User Message — Guidelines QA Here is the medical guideline: === GUIDELINE START === {guideline_text} === GUIDELINE END === Based ONLY on the guideline above, generate exactly 10 unique MULTIPLE-CHOICE clinical vignette questions and their answers. Each question should present a realistic patient scenario that tests the diagnostic or management principles in the text. For each vignette, provide 4-5 plausible multiple-choice options (A-E). Ensure distractors represent common diagnostic pitfalls or ’next best steps’ that are incorrect based strictly on the provided guideline. You MUST format EACH of the 10 items exactly as follows, using these specific XML tags: <qa> <question> Patient scenario and the specific question here. A) [Option 1] B) [Option 2] C) [Option 3] D) [Option 4] </question> <answer>The rationale explaining your chain of thought without mentioning the guideline and then Answer: correct answer</answer> </qa>

J.3

Synthetic Curated QA Prompt

The Synthetic Curated QA pipeline samples five exemplars without replacement from the curated benchmark pool and produces a single new QA pair per call. The pool is partitioned into labeled (multiple-choice, carrying a label_letter) and unlabeled (open-ended) buckets, and the user message is specialized accordingly. Developer Message — Synthetic Curated QA You are an expert medical educator and physician tasked with creating high-quality, clinically accurate content for a medical exam. Your task is to generate a new, unique, clinical vignette-style question along with its evidence-based correct answer. The timeline and progress should always be clear and detailed in the vignettes. Include clear context about site and where people travelled etc. The content must reflect realistic clinical scenarios, standard-of-care protocols, and well established medical consensus. Avoid scientifically controversial treatments. You will be provided with 5 examples. Use them strictly to understand the desired format, diagnostic difficulty, and clinical depth. DO NOT copy them. Generate a completely new, scientifically rigorous question that would be unconditionally approved by a medical review board. User Message — Synthetic Curated QA Here are example questions and answers to model your format on: --- Example 1 --<question>{question_1} <answer>{answer_1} --- Example 2 --<question>{question_2} <answer>{answer_2} ... (5 examples total) ... # If sampled from the labeled bucket: Now generate a brand new, unique, and clinically accurate MULTIPLE-CHOICE medical question and its detailed answer. Ensure the answer matches the formatting tags above. # If sampled from the unlabeled bucket: Now generate a brand new, unique, and clinically accurate OPEN-ENDED medical question and its detailed answer. Ensure the answer matches the formatting tags above.

J.4

Synthetic MOOVE Prompt

The Synthetic MOOVE pipeline samples five exemplar prompts without replacement from the MOOVE training split and generates a single new open-ended clinical scenario per call. Only the question stem is generated; assistant responses are produced downstream. Developer Message — Synthetic MOOVE You are an expert medical educator and physician tasked with creating high-quality, clinically accurate content. Your task is to generate a

new, unique, and realistic medical scenario or question prompt. The content must reflect realistic clinical presentations, inquiries from colleagues, or patient encounters. The timeline and objective progress should always be clear and detailed. Include clear context about site and where people travelled etc. You will be provided with 5 examples. Use them strictly to understand the desired format, diagnostic difficulty, and clinical depth. DO NOT copy them. Generate a completely new question that would be unconditionally approved by a medical review board. User Message — Synthetic MOOVE Here are example prompts to model your format and clinical depth on: --- Example 1 --<question> {prompt_1} </question> --- Example 2 --<question> {prompt_2} </question> ... (5 examples total) ... Now, acting as an expert medical educator, generate a brand new, unique, and clinically accurate medical scenario or question. Wrap your generated scenario strictly within <question> and </question> tags.

Decontamination details

We apply a two-stage n-gram and token-alignment decontamination pipeline adapted from Apertus [21]2 to the full curated corpus. The reference set aggregates the prompts of all evaluation benchmarks used in this work: MedQA, MedMCQA, PubMedQA, MedXpertQA, MMLU-Pro, IFEval, and ARC-Challenge. Training samples are tokenized with alehc/swissai-tokenizer. In the first stage, samples sharing any 8-gram with a reference prompt are flagged as candidates. In the second stage, each candidate is token-aligned against the matched reference and removed if the normalized alignment difference is at most τ = 0.5. This filters incidental n-gram overlaps while still catching lightly paraphrased contaminations. For each dataset, the pipeline outputs a decontaminated corpus and a report logging removed samples and their matched references.

Full openness comparison across medical LLMs

We assess all models discussed in this work along four openness dimensions defined by the Model Openness Framework (MOF) [45]: released weights, publicly available training data, a reproducible training recipe, and medical specialisation. As argued in Section 2, releasing weights alone does not constitute full openness: the pretraining data and training recipe of the base model determine what clinical knowledge and biases the model has absorbed, yet these dimensions are undisclosed for every major open-weight base used in prior medical LLM work (Llama 2, Llama 3.1, Mistral, Qwen2.5, Gemma 3). Apertus is the first base model at this scale to satisfy all MOF dimensions simultaneously, and Apertus-MeditronFO inherits this property while adding medical specialisation. 2 https://github.com/swiss-ai/posttraining-data/tree/main/04-decontamination

Table 17: Openness dimensions across medical LLMs and their base models, following the Model Openness Framework. YES = fully satisfied; ~ = partial (e.g. some data disclosed, recipe absent); NO = not satisfied. “Medical specialist” denotes a model adapted for clinical tasks via continued pretraining or supervised fine-tuning on medical data. Model

Base model

Base openness

Closed specialists MedPaLM MedPaLM-2 Med-Gemini

Open weights

Open data

Open training

Medical specialist

PaLM (closed) PaLM 2 (closed) Gemini (closed)

Closed Closed Closed

NO NO NO

YES YES YES

Open-weight medical specialists PMC-LLaMA Llama 2 (open-wt) BioMistral Mistral (open-wt) Meditron-70B Llama 2 (open-wt) HuatuoGPT-o1 Qwen2.5 (open-wt) ReasonMed Qwen2.5 (open-wt) Aloe Beta Llama 3.1 / Qwen2.5 Hippocrates Llama 2 / Mistral MediPhi Phi-3.5 (open-wt) Llama-3-Meditron Llama 3.1 (open-wt) MedGemma-27B Gemma 3 (open-wt)

Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight

YES YES YES YES YES YES YES YES YES YES

NO NO ~ NO YES ~ YES NO ~ NO

NO NO NO NO NO NO ~ NO ~ NO

YES YES YES YES YES YES YES YES YES YES

Fully open specialists (this work) Apertus-70B-MeditronFO Apertus-70B Apertus-8B-MeditronFO Apertus-8B OLMo-2-32B-MeditronFO OLMo-2-32B EuroLLM-22B-MeditronFO EuroLLM-22B EuroLLM-9B-MeditronFO EuroLLM-9B

Fully open Fully open Fully open Fully open Fully open

YES YES YES YES YES

Tier 1 — Pretraining bases (open-weight only)

Llama 2

Mistral-7B

Llama 3.1

Qwen2.5

Weights only

15T tokens, opaque

18T tokens, opaque ← opaque wall

Tier 2 — Medical specialist models (ﬁne-tuning stage open)

PMC-LLaMA

Hippocrates

BioMistral

HuatuoGPT / Aloe

Weights + data

Closest prior eﬀort

Weights + data

Weights only

MOF gap: pretraining corpus, training recipe, intermediate checkpoints — not disclosed by any base model above

closes the gap Tier 3 — Fully open foundation

Apertus (base model) Weights · pretraining data + compliance docs · training recipe intermediate checkpoints · permissive licence

Tier 4 — Fully open medical model

medical specialisation

Apertus-Meditron 8B

Apertus-Meditron 70B

All four MOF dimensions satisﬁed

Figure 10: Medical LLM Openness Tiers

Licenses of existing assets

Table 18 lists each existing asset used in this work, with its originating reference, version where applicable, public URL, and license. All assets are used in accordance with their respective licenses and restricted to research purposes consistent with the originating works. Table 18: Licenses of existing assets used in this work. Datasets and models are listed with their originating reference, version, public URL, and license. † License terms verified at time of writing; downstream users should re-verify upstream terms before redistribution. Asset

Reference

URL

License

Jin et al. [2021] Pal et al. [2022] Jin et al. [2019] Alonso et al. [2024] Singhal et al. [2023] Abacha et al. [2017] Olatunji et al. [2024] Chen et al. [2023b] Sallinen et al. [2025]

https://github.com/jind11/MedQA https://medmcqa.github.io https://pubmedqa.github.io https://huggingface.co/datasets/HiTZ/MedExpQA via Med-PaLM release https://github.com/abachaa/LiveQA_MedicalTask_TREC2017 https://huggingface.co/datasets/intronhealth/afrimedqa_v2 via Meditron release via Llama-3-Meditron release

MIT MIT MIT CC BY-NC-SA 4.0 CC BY 4.0 Open / research use† CC BY 4.0† Per-source (mixed); research use† Research use†

Zuo et al. [2025] Wang et al. [2024] Zhou et al. [2023] Clark et al. [2018] Arora et al. [2025]

https://huggingface.co/datasets/TsinghuaC3I/MedXpertQA https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro https://github.com/google-research/google-research/tree/master/instruction_following_eval https://allenai.org/data/arc https://github.com/openai/simple-evals

MIT† MIT Apache 2.0 CC BY-SA 4.0 MIT†

Hernández-Cano et al. [2025] Hernández-Cano et al. [2025] OLMo et al. [2024] Ramos et al. [2026] Ramos et al. [2026] Team et al. [2025]

https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509 https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509 https://huggingface.co/allenai/OLMo-2-0325-32B-SFT https://huggingface.co/utter-project/EuroLLM-22B-Instruct https://huggingface.co/utter-project/EuroLLM-9B-Instruct https://huggingface.co/google/gemma-3-27b-it

Apache 2.0 Apache 2.0 Apache 2.0 Apache 2.0† Apache 2.0 Gemma Terms of Use

Sellergren et al. [2025] Sallinen et al. [2025] Corbeil et al. [2025] Yang et al. [2025] Yang et al. [2025] Agarwal et al. [2025]

https://huggingface.co/google/medgemma-27b-text-it https://huggingface.co/OpenMeditron/Meditron3-70B https://huggingface.co/microsoft/MediPhi https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507 https://huggingface.co/Qwen/Qwen3-235B-A22B https://huggingface.co/openai/gpt-oss-120b

Health AI Developer Foundations TOS Llama 3.1 Community License MIT† Apache 2.0 Apache 2.0 Apache 2.0

— — — —

https://github.com/axolotl-ai-cloud/axolotl https://pytorch.org https://github.com/deepspeedai/DeepSpeed https://github.com/Dao-AILab/flash-attention

Apache 2.0 BSD 3-Clause Apache 2.0 BSD 3-Clause

Source QA datasets (training) MedQA MedMCQA PubMedQA MedExpQA HealthSearchQA LiveQA-Med AfriMed-QA v1/v2 GUIDELINES corpus MOOVE (training split) Evaluation benchmarks MedXpertQA MMLU-Pro IFEval ARC-Challenge HealthBench Base models (fine-tuned) Apertus-70B-Instruct Apertus-8B-Instruct OLMo-2-32B-SFT EuroLLM-22B-Instruct EuroLLM-9B-Instruct Gemma-3-27B-IT

Reference / baseline models (evaluation only) MedGemma-27B Llama-3.1-70B-Meditron MediPhi Qwen3-30B-A3B-Instruct-2507 Qwen3-235B-A22B gpt-oss-120b Frameworks and infrastructure Axolotl PyTorch (FSDP v2) DeepSpeed (ZeRO-3) FlashAttention 2

Record · ID 192384 · SHA-256 bf95a7b8ed353112
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.