1
A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency
arXiv:2604.22348v1 [cs.LG] 24 Apr 2026
Nanae Aratake1 , Taisei Tosaki1 , Yuji Okamoto1 , Eiichiro Uchino1 , Masaki Nakamura2 , Nobutomo Matsui3 , Akiko Hatakama4 , and Yasushi Okuno1
Abstract—Clinical risk prediction using longitudinal medical data supports individualized care. Self-supervised foundation models have emerged as a promising approach for leveraging large-scale unlabeled healthcare records. In natural language processing, scaling laws suggest that larger models achieve predictably lower pretraining losses, supporting the foundation model paradigm. However, for structured medical data— characterized by a limited vocabulary and sparse observations— whether increasing model size consistently improves downstream predictions is unclear, as most studies evaluate only a single model scale. In this study, we evaluated the relationship between model scale and downstream task performance for structured medical foundation models. Using a random sample (2.3 million patients, 32 hospitals) from a nationwide 519-hospital Japanese claims database, we pretrained encoder-only Transformers at five scales (2.2M–101M parameters) for disease incidence and medication prediction. Downstream performance saturated at taskdependent thresholds: disease prediction benefited from larger models (32M–101M), whereas medication prediction saturated at 11M, reducing pretraining time by 178 h. Across all tasks, the best-performing model consistently outperformed a Light Gradient Boosting Machine baseline in the area under the precision–recall curve. These findings indicate that, unlike the monotonically decreasing pretraining loss, the optimal model size varied depending on task characteristics. This task-dependent saturation provides practical guidance for balancing predictive performance and computational cost in structured medical foundation models. Index Terms—Computational efficiency, foundation models, self-supervised learning, structured medical data, transformer models
I. I NTRODUCTION LINICAL risk prediction, which estimates the likelihood of disease onset or therapeutic intervention, supports individualized care and resource allocation [1], [2]. Longitudinal medical data that capture evolving patient states have attracted growing interest for this purpose [3]. Most existing clinical risk prediction models rely on supervised learning [4]. Yet a major bottleneck in supervised approaches is the curation of task-specific labels, which is costly [5]. Acquiring sufficient labels is often infeasible for rare diseases or novel clinical endpoints. Self-supervised pretraining followed by fine-tuning with limited labeled data has
C
1 Graduate School of Medicine, Kyoto University, Kyoto, Japan. 2 Medical Data Vision Co., Ltd., Tokyo, Japan. 3 IQVIA Solutions Japan G.K., Tokyo, Japan. 4 DeSC Healthcare, Inc., Tokyo, Japan.
Corresponding author: Yasushi [email protected]).
Okuno
(e-mail:
emerged as a promising paradigm. This approach enables models to leverage large volumes of unlabeled clinical records before adapting to specific tasks. In natural language processing (NLP), scaling laws demonstrate that pretraining loss decreases predictably with model size, data, and computation [6]. This predictability supports the foundation model paradigm [7]. However, the optimal model size that is most useful for downstream fine-tuning depends on the task characteristics and labeled data availability [8]. In the medical domain, where labeled data tend to be scarce, scaling up pretrained models may not be equally beneficial for all downstream tasks. This concern is particularly relevant to structured medical data, which, unlike free text, possess a relatively limited vocabulary and sparse, heterogeneous observations. Such properties may render representation learning prone to early saturation, suggesting that scaling gains may not translate uniformly into downstream improvements. Existing structured medical foundation models [9]–[12] typically evaluate downstream performance using only one model, which is often the largest variant. As a result, the performance– cost tradeoff across model sizes remains unexamined. Furthermore, clinical tasks differ fundamentally in nature. Disease onset prediction may require longer-range contextual modeling of partially unobserved biological processes, whereas medication initiation may depend more heavily on recent information governed by clinical guidelines and prescription patterns. Thus, the model size most useful for downstream fine-tuning is likely to be task dependent. Beyond model size selection, the choice of modeling approach itself requires justification. Transformers [13] have become the predominant architecture for medical foundation models, owing to their capacity for long-range sequence modeling via self-attention [9], [10]. By contrast, gradientboosted decision trees (GBDTs) remain strong baselines for structured data [14], and the practical value of pretrained Transformers must be judged by whether their gains justify the added computation. Therefore, a systematic evaluation that jointly considers the model size, task characteristics, and computational constraints is required. In this study, we systematically evaluated the relationship between the model size and downstream performance of structured medical foundation models. We used a nationwide hospital-based claims/Diagnosis Procedure Combination (DPC) database from Medical Data Vision Co., Ltd. (MDV) [15], [16]. This database was selected because its
2
comprehensive 519-hospital coverage provides the clinical diversity essential for foundation models. From a random sample of approximately 2.3 million patients across 32 hospitals, we constructed token sequences integrating diagnosis and medication codes, age in days, and sex. Using these sequences, we pretrained the encoder-only Transformer models [13], [17] at five scales spanning 2.2M– 101M parameters. This specific range was chosen to systematically evaluate the trade-offs between predictive performance and computational efficiency. We then fine-tuned these models to two fundamentally different task types: one-year incident disease prediction and one-year new medication initiation prediction. The performance was compared with that of a Light Gradient Boosting Machine (LGBM) [18] baseline to assess the tradeoff among predictive performance, model capacity, and actual pretraining time. The main contributions of this study are as follows: • Systematic evaluation across five scales (2.2M–101M parameters) revealed a task-dependent capacity ceiling. Disease prediction benefited from larger models, whereas medication prediction saturated at 11M, achieving a 76% reduction in pretraining time (53.9 h vs. 232.2 h) without compromising predictive performance. • For each task, the task-optimal pretrained model consistently surpassed the LGBM baseline in area under the precision–recall curve (AUPRC), demonstrating that appropriately sized—not uniformly massive— foundation models justify their pretraining cost. Evaluating task-specific capacity ceilings is therefore essential for resource-efficient model development. II. R ELATED W ORK A. Scaling Trends in Medical Data Scaling research on medical foundation models has advanced primarily in clinical texts. GatorTron [19], NYUTron [20] and Med-PaLM [21] show monotonic downstream improvements with model size, but these findings rely on the vast vocabulary and complex context of free text. Whether the same holds true for structured data with a limited vocabulary and few observations remains unclear. This study addresses this gap by examining the relationship between model size and downstream task performance, specifically for structured medical data. B. Structured Medical Foundation Models and Task Suitability BEHRT [9] and Med-BERT [10] demonstrated the value of structured medical foundation models, but evaluated only a single model size, leaving the saturation point of downstream performance relative to computing unexplored. TransformEHR [22] proposes an encoder–decoder generative framework for disease outcome prediction. GenHPF [23] introduced a multitask multisource predictive framework. However, both studies focused on architectural design rather than the relationship between model scale and downstream performance. EHRSHOT [24] examined scaling with respect to data volume and prediction horizons in generative settings.
Recently, Waxler et al. [25] compared multiple model sizes for generative medical event predictions in zero-shot settings. Although recent studies have explored adapting pretrained language models for tabular prediction [26], a systematic analysis of model size versus downstream performance for encoder-only architectures with fine-tuning of medical data remains lacking. The present study fills this gap by comparing disease prediction tasks, which require complex contextual understanding, with medication prediction tasks, which exhibit stronger regularity. This comparison provides insights into task-appropriate model size selection. C. Deep Learning versus GBDTs for Structured Data GBDTs frequently match or exceed the performance of deep learning in tabular data [14], [27]. For clinical deployment, the adoption of a large Transformer is justified only if the performance gain offsets the computational overhead. This study identified the model size threshold at which Transformers surpass GBDTs. These thresholds offer practical guidance for model size selection in clinical settings. III. M ETHODS We pretrained encoder-only Transformers on patient-level sequences derived from nationwide claims/DPC data and finetuned them for incident disease prediction and new medication initiation prediction (Fig. 1). Diagnoses were encoded using the International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10), the standard diagnostic classification maintained by the World Health Organization. Medications were encoded using the YJ code, a 12-character individual drug code administered within the Japanese National Health Insurance drug pricing system by the Ministry of Health, Labour and Welfare (MHLW). To consider the effect of label imbalance on predictive performance, we selected targets with different positive rates for both disease and medication prediction: primary hypertension (ICD-10: I10, 13.8%), chronic kidney disease (ICD-10: N189, 2.63%), amlodipine (YJ: 2171022, 14.9%), and pregabalin (YJ: 1190017, 6.93%). The performance was evaluated using the area under the receiver operating characteristic curve (AUROC) and AUPRC. These metrics were specifically chosen because AUROC serves as a standard measure of overall discriminative ability, whereas AUPRC is particularly crucial for accurately reflecting classification performance in clinical tasks characterized by low positive event rates. A. Dataset 1) Preprocessing We used anonymized outpatient and inpatient records from the MDV claims/DPC database. As introduced, we utilized a random sample of 32 facilities from the 519-facility parent database. This dataset includes diagnosis codes, medication codes, sex, and age in days (Table I). To avoid potential training instability arising from extremely short sequences, we restricted the analysis to patients with at least two diagnoses
3
TABLE I: Dataset Overview Data source Number of hospitals Observation period Patients (total) Patients (codes ≥ 2) Sex ratio ICD-10 code coverage YJ medication code coverage Age and sex coverage
Medical Data Vision Co., Ltd. (MDV) 32 (sampled from 519 in Japan) April 2014 – November 2024 2,445,160 2,294,687 Male: 47.6%, Female: 52.4% 99.5% 85.4% 100%
or medication codes (n = 2,294,687). To improve the training stability and prioritize the learning of frequently occurring codes, we excluded diagnosis and medication codes occurring fewer than 10,000 times across the entire observation period. For YJ (medication) codes, only the first seven digits were used to collapse product-level differences to approximate pharmacological action levels. After these filtering steps, the vocabulary comprised 936 diagnoses and 994 medication codes. 2) Ethical Statement This was a retrospective observational study using anonymously processed data provided by MDV under a data use agreement. The Japanese Act on the Protection of Personal Information (enacted in 2003 and amended in 2022) defines anonymously processed data as information that have been irreversibly processed, rendering it technically and legally impossible to identify specific individuals. The restoration of such data to their original state is strictly prohibited by law. The acquired data were managed on an access-controlled university server restricted to authorized researchers. The Ethical Guidelines for Life Sciences and Medical Research Involving Human Subjects in Japan (issued in 2021 and partially amended in 2023) exclude studies that solely use anonymously processed data from the scope of ethical review. In accordance with these national guidelines, the Ethics Committee of Kyoto University Graduate School and Faculty of Medicine, Kyoto University Hospital (http://www.ec.med.kyoto-u.ac.jp) maintains an institutional policy stating that studies relying exclusively on preexisting anonymously processed data do not require an application for ethical review, barring exceptional circumstances. Consequently, ethical approval for this study was formally waived based on this institutional policy, and the requirement to obtain informed consent from the study participants was not applicable. B. Token Sequence Construction Following previous work [9], [10], each patient’s clinical history was represented by a single chronologically ordered sequence (Fig. 2). For patient p, the sequence vp begins with a classification control token [CLS], followed by a sex attribute token ([M] or [F]) and chronologically ordered clinical tokens from position three onward. Each clinical token at position i (i ≥ 3) is defined as a pair vpi = (zpi , aip ) comprising a categorical clinical code zpi and numerical age in days aip at the time of recording. Denoting the sequence length for patient p as n, the token sequence is vp = ([CLS], [M] or [F], (zp3 , a3p ), . . . , (zpn , anp )).
(1)
The overall sequence statistics are summarized in the Supplementary Material. C. Input Embedding Construction The token sequence vp defined in (1) is mapped to a d-dimensional vector sequence Ep = (E1p , E2p , . . . , Enp ) ∈ Rn×d . At each position i, the input embedding Eip comprises a categorical embedding corresponding to the discrete token identity and an embedding of age in days, which is a continuously valued attribute associated with each clinical event. a) Categorical Embeddings We used separate embedding matrices for the special tokens, diagnosis codes, and medication codes. The token type at position i is denoted cip ∈ {0, 1, 2}, where the type of vector for the entire sequence is cp = (c1p , . . . , cnp ) ∈ {0, 1, 2}n . Here, cip = 0 denotes special tokens, cip = 1 denotes diagnoses, and cip = 2 denotes medications. By letting Vl , Vicd , and Vyj denote the respective vocabulary sizes, the corresponding embedding matrices are Wl ∈ RVl ×d ,
Wicd ∈ RVicd ×d ,
Wyj ∈ RVyj ×d .
(2)
b) Age-in-Days Embedding Age (in days) was used as the continuous temporal attribute for each clinical event. This scalar must be mapped into the same latent space as the categorical embeddings; hence, we encoded it using Piecewise Linear Encoding (PLE) [28], which partitions the observed range into intervals and linearly interpolates within each interval. This encoding can effectively capture the distributional properties of numerical features. Let a ∈ R denote a pre-standardized continuous scalar. To define PLE : R → Rd , we partitioned the value range into d nonoverlapping intervals Bt = [bt−1 , bt ) for t = 1, . . . , d, where b0 = min(a) and bd = max(a). The interval endpoints {bt } were determined from the training data using the quantiles of the empirical distribution: bt = Qt/d ({a(r) }),
t = 1, . . . , d − 1,
(3)
where Qt/d is the t/d-quantile of the training data. Using these quantile endpoints, scalar a was mapped to a d-dimensional embedding that reflected its statistical distribution. The kth component of PLE(a) was 0, k>j k < j (k = 1, . . . , d), (4) uk (a) ≜ 1, a − bj−1 , k=j bj − bj−1 where j satisfies bj−1 ≤ a < bj . When combining PLE with other embeddings through summation, an affine transformation is recommended to align the representation spaces. c) Final Input Embedding By applying PLE to aip with an affine transformation (Wage ∈ Rd×d , b ∈ Rd ), the embedding at position i is Eip = 1(cip = 0) Wl [zpi ] + 1(cip = 1) Wicd [zpi ] + 1(cip = 2) Wyj [zpi ] + 1(cip ≥ 1) (Wage PLE(aip ) + b), (5)
4
Pretraining (MLM Task)
4.7M
2.2M
5 Pretrained Model Sizes 32M 11M
101M
Large-Scale Unlabeled Data Independent Fine-tuning (4 Tasks: 1-Year Prediction of Diagnosis & Prescriptions)
Task 1.
Task 2.
Task 3.
Task 4.
ICD : I10 Primary Hypertension Positive Rate:13.8%(High)
ICD : N189 Chronic kidney disease Positive Rate:2.63%(Low)
YJ : 2171022 Amlodipine Positive Rate:14.9%(High)
YJ : 1190017 Pregabalin Positive Rate:6.93 %(Low)
Fig. 1: Overview of pretraining and fine-tuning. Masked language modeling (MLM) pretraining was performed on token sequences pairing clinical codes with age in days from approximately 2.3 million patients. Five encoder-only Transformers (2.2M–101M parameters) were independently fine-tuned on two disease prediction tasks (primary hypertension: 13.8%; chronic kidney disease: 2.63%) and two medication prediction tasks (amlodipine: 14.9%; pregabalin: 6.93%). Percentages in parentheses indicate the positive rate for each target event.
where W[·] denotes the embedding lookup and 1(·) is the indicator function. The Transformer fθ receives Ep = (E1p , . . . , Enp ) and produces output Hp . D. Training Procedure a) Pretraining We employed masked language modeling (MLM) with a maximum sequence length of 8,000. Sequences longer than this limit were truncated to retain the earliest 8,000 chronologically ordered tokens. Importantly, 99.97% of the patient sequences in our dataset fell within this limit, ensuring minimal loss of clinical information. Fifteen percent of the tokens were randomly masked, and the model was trained to recover the true values. As each token vpi = (zpi , aip ) comprises a categorical numerical pair, the loss was constructed using distribution-appropriate components. The final loss is the unweighted sum of three independent terms: cross-entropy losses for diagnosis codes and medication codes (each with a vocabulary-specific output head and softmax) and mean squared error (MSE) for age in days. The model acquires general-purpose representations of each patient’s medical context by jointly learning the categorical clinical codes and numerical ages. b) Fine-tuning For each task, only the sequence preceding the target event date was used as the input. A binary classification was performed to predict whether the target diagnosis or medication would occur within one year. Operationally, the hidden state of the [CLS] token was extracted from the output of the pretrained model and passed through a single-layer
output head with a sigmoid activation function to compute the event probability. Binary cross-entropy loss was used with fullparameter fine-tuning. To simulate limited-label conditions representative of real clinical settings, we varied the number of labeled patients between 100, 500, and 1,000. E. LGBM Baseline As a strong tabular baseline, LGBM was trained on countbased aggregated history features derived from the same input window as that of the transformer. Diagnosis and medication codes observed in the chronologically ordered history were converted into per-code counts, and sex and age in days at the index date were appended, yielding 1,932 features. This baseline preserves aggregate historical information but does not explicitly model the token order. IV. E XPERIMENTS A. Dataset Splitting The dataset was randomly split into pretraining (70%), validation (10%), test (10%), and fine-tuning (10%) subsets (seed 42), with no patient overlap. The pretraining partition comprised 1,606,268 patients, with 229,565 validation, 229,293 test, and 229,561 fine-tuning patients; each downstream subset exceeded 229,000 patients, providing sufficient statistical power for evaluation. From the fine-tuning cohort (229,561), patients satisfying the following criteria were extracted per task: (i) observation period ≥1 year, (ii) positive cases with a one-year window before the first target event, and (iii) negative cases with
5
Medical Code Embeddings
[CLS]
Positional Emb. (Age in Days)
-
YJ : 3969010
ICD10 : N40
YJ : 2590008
[M]/[F]
Unspecified Diabetes Mellitus
Sitagliptin
Hyperplasia of the prostate
Tamsulosin
-
23100
23100
35000
35000
EM_(N-1)
EM_N
Sex
ICD10 : E14
…
ENCODER-ONLY TRANSFORMER EM_1
EM_3
EM_2
4 Task Fine-tuning Linear, Sigmoid
EM_4
…
MLM prediction
Categorical (ICD, YJ, Sex): Cross Entropy | Numerical (Age in Days): Mean Squared Error
Fig. 2: Input-sequence construction and training objectives. Each clinical event token pairs a code (ICD-10 or YJ) with age in days. The embedding is the element-wise sum of the code embedding and the age-in-days embedding. The sequence begins with [CLS] (for classification) and a sex token ([M]/[F]), followed by chronologically ordered event tokens. For pretraining, we employ masked language modeling (MLM), applying cross-entropy loss to categorical features (medical codes and sex) and mean squared error (MSE) to the numerical feature (age in days).
no target event and a secured one-year window. Balanced samples (1:1) of 100, 500, and 1,000 labeled patients were drawn. To examine the robustness, we generated three patientlevel dataset realizations (seeds 42, 123, and 456). In each realization, class balancing (1:1 positive: negative) was applied only to the limited-label training subset of sizes 100, 500, or 1,000 fine-tuning dataset. B. Computational Cost Evaluation We computed the theoretical floating-point operations (FLOPs) for each model and compared them against the test loss to evaluate the relationship between the model size and computational cost. Computations were performed on the Miyabi system at the University of Tokyo, utilizing 16 NVIDIA GH200 nodes for pretraining and a single node for fine-tuning. We also measured the actual pretraining wallclock time (cumulative active training and validation time to reach the minimum validation loss, excluding queuing delays). C. Downstream Task Evaluation For each model size, we compared the downstream task performance of the pretrained model with that of a randomly initialized (from-scratch) model with identical architecture, both fine-tuned on the four downstream tasks. This comparison isolates the contribution of pretraining from the representational capacity of the architecture itself. Three independent patient-level datasets (seeds 42, 123, and 456) were used per label count condition; from-scratch models additionally used three initialization seeds (1001, 1002, and 1003) to reduce initialization dependence, with the mean performance reported. The metrics AUROC and AUPRC were compared across all five pretrained models and LGBM for each label count condition.
D. Hyperparameter Settings a) Transformer Pretraining The hidden dimensions varied across 128, 256, 512, 1024, and 2048, with layers fixed at four, attention heads at 16, batch size at 512, and the maximum sequence length at 8,000. Default learning rates of 1e-4 and 3e-4 were initially applied to all models. While smaller models (128, 256, 512) trained stably with these defaults, the 1024 and 2048 models exhibited early overfitting; therefore, a grid search over {5e-5, 3e-5, 1e5, 5e-6} was performed for these two larger models. b) Transformer Fine-tuning Batch size: 8, learning rate: 1e-5, maximum epochs: 20 (for 100 patients) or 40 (for 500/1,000). Early stopping was triggered when the validation AUROC improvement <0.001 for two consecutive epochs. c) LGBM Five-fold cross-validated grid search over learning rate: {0.01, 0.05, 0.1}, maximum depth: {3, 4, 6}, leaves: {7, 15, 31}, subsampling: {0.8, 1.0}, column subsampling: {0.8, 1.0}, and trees: {50, 100, 200}. V. R ESULTS A. Pretraining Loss across Model Sizes Fig. 3 and Table II show the relationship between the computational cost measured in giga floating-point operations (GFLOPs) and the test loss for the five transformer models pretrained at their respective optimal learning rates. GFLOPs represent the theoretical operations required for one forward/backward pass expressed in units of 109 FLOPs. The computational cost increased exponentially with model size, whereas the test loss decreased substantially from the 11M model onward and continued to improve in the 101M model. Alongside the exponential increase in theoretical GFLOPs, the actual pretraining time increased substantially from 14.4 h for the 2.2M model to 232.2 h for the 101M model (Table II).
6
TABLE II: Model scales and computational costs for pretraining Hidden Dimension d Parameters (M )a GFLOPs Optimal Learning Rate 128 2.16 68.90 3e-4 256 4.65 142.20 1e-4 512 11.40 302.20 1e-4 1024 31.97 675.70 3e-5 2048 101.43 1636.52 1e-5 a M = 106 . b Wall-clock time required to reach the minimum validation loss.
Test Loss
4.7M
7.5 7.0 6.5 11M
32M
6.0
101M
D. Model Size Comparison with LGBM
103
102
Computational Cost (GFLOPs)
AUPRC
Fig. 3: Pretraining loss versus computational cost. The test loss decreases with increasing GFLOPs as the model size grows. Each plot corresponds to a model size where M = 106 indicates the number of parameters. GFLOPs denotes 109 floating-point operations. 0.450 0.425 0.400 0.375 0.350 0.325 0.300 0.275 0.250
2. Chronic Kidney Disease
1. Primary Hypertension 0.14 0.12 0.10 0.08 0.06
0.8 0.7 0.6 0.5 0.4 0.3 0.2 102
Fig. 5 compares AUPRC across all five pretrained models and LGBM for each dataset split. Some variations in performance were observed across the fine-tuned dataset splits, even for the same prediction task. For disease prediction (primary hypertension and chronic kidney disease), the 32M or 101M models were optimal, and for medication prediction (amlodipine and pregabalin), the 11M model was optimal. In each case, the task-optimal model consistently outperformed LGBM in AUPRC. The pretraining time for the 11M model, which was optimal for the medication prediction tasks, was 53.9 h (Table II). Scaling up to the 101M model increased the pretraining time to 232.2 h; however, this additional computational cost did not yield further AUPRC improvements for these specific tasks.
4. Pregabalin
3. Amlodipine
0.75 0.70 0.65 0.60 0.55 0.50 0.45 0.40
Test Loss 8.22 7.88 6.10 5.93 5.74
outperformed their from-scratch counterparts. Across all tasks and model sizes, the performance improved consistently with increasing labeled data. Medication prediction tasks tended to achieve a higher overall performance than disease prediction tasks.
2.2M 8.0
Pretraining Time (hours)b 14.4 25.0 53.9 107.3 232.2
103
VI. D ISCUSSION
Pretrained Model From Scratch Model
102
Number of Patients
103
Fig. 4: Pretrained vs. from-scratch AUPRC at the 11M scale across four tasks, with labeled patient counts of 100, 500, and 1,000. Three dataset splits shown per condition. The pretrained model (green) generally outperforms the from-scratch model (black). The x-axis is shared across all panels. B. Fine-tuning Data Statistics Table III summarizes the input-sequence statistics for each task. Medication prediction tasks had longer sequences, more time points, and longer observation periods than disease prediction tasks. C. Pretrained versus From-scratch Models Fig. 4 compares the pretrained and from-scratch AUPRC at the representative 11M scale; the results for all five sizes are shown in Supplementary Fig. S2. Although the benefits of pretraining were occasionally insufficient for certain model sizes, label counts, or task types, the pretrained models generally
This study systematically evaluated the interplay among model size, task characteristics, and computational constraints for structured medical foundation models. The pretraining loss decreased consistently with model size, confirming a scaling trend consistent with the NLP domain. However, the downstream performance exhibited task-dependent saturation, with the optimal model size diverging across tasks. For disease prediction, large models (32M–101M) achieved superior performance, suggesting that these tasks benefit from a greater model capacity. For medication prediction, performance was saturated in the mid-scale model (11M). As our timing analysis revealed, scaling from 11M to 101M increased the pretraining time from 53.9 to 232.2 h; however, this over four-fold increase in computational cost provided no additional predictive benefit. Medication initiation is largely governed by clinical practice guidelines, which may impose a stronger regularity on prediction targets. This suggests that for such regular tasks, the model capacity exceeds the information density of structured data at a relatively early point owing to the limited vocabulary and sparsity compared with free text. These findings offer a critical counter-perspective to the prevailing “bigger is always better” paradigm observed in recent medical foundation models. For instance, prior studies exploring generative decoder-only models on longitudinal event sequences (e.g., Curiosity [25]) have reported continuous
7
TABLE III: Input sequence statistics for four prediction tasks (Median [InterQuartile Range]) Statistic Sequence length (tokens) Time points (days) Observation period (days)
Primary Hypertension 49 [17–142] 15 [6–37] 1431 [690–2539]
2. Chronic Kidney Disease
1. Primary Hypertension
0.45 0.40 0.35 0.30 0.25 0.20
102
103
AUPRC
0.40 0.35 0.30 0.25 102
0.4
103
0.40 0.35 0.30 0.25 103
103
102
Seed 123
0.8
0.6 0.5 0.4 103
102
0.3
102
Seed 456
0.6 0.5 0.4 102
103
0.3
102
Seed 123
103
103
Seed 456
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0.7
103
102
103
0.8
0.200 0.175 0.150 0.125 0.100 0.075 0.050 0.025
102 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0.7
Seed 456
0.45
102
0.3
0.200 0.175 0.150 0.125 0.100 0.075 0.050 0.025
Seed 456
0.50
0.20
0.5
Seed 123
0.45
0.20
0.6
103
Seed 42
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0.7
102
Pregabalin 116 [37–364] 31 [12–75] 1883 [989–3017] 4. Pregabalin
Seed 42
0.8
0.200 0.175 0.150 0.125 0.100 0.075 0.050 0.025
Seed 123
0.50
Amlodipine 92 [31–287] 26 [10–63] 1766 [872–2916]
3. Amlodipine
Seed 42
Seed 42
0.50
Chronic Kidney Disease 81 [25–277] 22 [9–60] 1737 [889–2891]
102
103
Number of Patients 2.2M
4.7M
11M
32M
101M
LGBM
Fig. 5: AUPRC of five pretrained Transformers and LGBM across four tasks. Disease prediction benefits from the 32M/101M model; medication prediction saturates at 11M. The task-optimal model consistently outperforms LGBM.
performance improvements with increasing model scale for autoregressive simulation tasks. In contrast, our results using encoder-only models highlight a distinct performance plateau for specific downstream discriminative tasks. This divergence suggests that while generative modeling of entire patient timelines may benefit from massive parameter spaces, targeted classification tasks—particularly highly regularized clinical interventions like medication initiation—reach a representation capacity ceiling at much smaller scales. Thus, our study indicates that architectural design (generative vs. discriminative) and specific task characteristics inherently dictate the optimal model capacity, challenging the uniform necessity of massive models in structured healthcare data. These insights have important implications for the clinical deployment of constrained computational resources. Rather than uniformly adopting the largest model, biomedical practitioners should align both the model architecture and capacity with the nature of the target task. Tasks involving natural disease progression may benefit from larger models, whereas rule-based clinical interventions (e.g., medication, laboratory
testing, or surgery) may saturate at smaller scales. For regular tasks such as medication prediction, fine-tuning from a midscale model can surpass the LGBM baseline while drastically reducing the pretraining time. These findings indicate that for structured medical foundation models, task-appropriate model size selection is the key to balancing prediction quality and computational cost. Limitations include: (1) reliance on claims/DPC data without laboratory values, vital signs, or clinical notes, which are available in other databases [29] and could further enrich foundation model representations [30]; (2) hospital-based data without guaranteed cross-facility longitudinal linkage [31]; (3) evaluation limited to disease and medication prediction; different trends may emerge for other tasks such as readmission prediction; and (4) single-dataset evaluation without external or prospective validation [32]. VII. C ONCLUSION We constructed structured medical foundation models at five scales (2.2M–101M parameters) and compared their down-
8
stream performances across disease onset and medication initiation predictions. The models were pretrained using a random sample (2.3 million patients, 32 hospitals) from a nationwide 519-hospital Japanese claims/DPC database and fine-tuned under limited-label conditions. Disease prediction benefited from large models (32M–101M), whereas medication prediction was saturated in the midscale model (11M), revealing that the benefit of model scaling is task-dependent. For each task, the best-performing model consistently outperformed the LGBM baseline. These results demonstrate that task-appropriate model size selection can prevent the excessive computational hours required to pretrain unnecessarily large models, offering practical guidance for deploying structured medical foundation models under computational constraints. ACKNOWLEDGMENT This study is supported by JST Moonshot R&D (JPMJMS2021), JST Research and Development Program for Next-Generation Edge AI Semiconductors (JPMJES2511), JSPS KAKENHI (JP25K00148, JP25H02626, JP26K14994), and a project (JPNP14004) commissioned by the New Energy and Industrial Technology Development Organization (NEDO). The authors also gratefully acknowledge the University of Tokyo for the use of the Miyabi system for primary analyses and the Institute of Science Tokyo for the use of the TSUBAME supercomputer for preliminary work. C ONFLICT OF I NTEREST M. Nakamura, N. Matsui, and A. Hatakama are employees of Medical Data Vision Co., Ltd. (MDV); IQVIA Solutions Japan G.K.; and DeSC Healthcare, Inc., respectively. MDV provided the medical claims data used in this study. These coauthors contributed specialized expertise regarding the collection, structure, and extraction logic of medical claims data, and supported the construction of the analytical dataset. Although the study received support in the form of data provision, it was conducted independently by the authors. None of the affiliated companies had any role in the study design, data analysis, interpretation of the results, or the decision to publish. No other commercial or financial relationships are construed as potential conflicts of interest. R EFERENCES [1] US Preventive Services Task Force, “Screening for peripheral artery disease and cardiovascular disease risk assessment with the ankle-brachial index: US preventive services task force recommendation statement,” JAMA, vol. 320, no. 2, pp. 177–183, Jul. 2018. [2] H. Freisling et al., “Lifestyle factors and risk of multimorbidity of cancer and cardiometabolic diseases: A multinational cohort study,” BMC Med., vol. 18, p. 5, Jan. 2020. [3] A. Rajkomar et al., “Scalable and accurate deep learning with electronic health records,” npj Digit. Med., vol. 1, p. 18, May 2018. [4] R. Miotto, L. Li, B. A. Kidd, and J. T. Dudley, “Deep patient: An unsupervised representation to predict the future of patients from the electronic health records,” Sci. Rep., vol. 6, p. 26094, May 2016. [5] B. K. Beaulieu-Jones and C. S. Greene, “Semi-supervised learning of the electronic health record for phenotype stratification,” J. Biomed. Inform., vol. 64, pp. 168–178, Dec. 2016. [6] J. Kaplan et al., “Scaling laws for neural language models,” arXiv:2001.08361, Jan. 2020.
[7] R. Bommasani et al., “On the opportunities and risks of foundation models,” arXiv:2108.07258, Aug. 2021. [8] J. Hoffmann et al., “Training compute-optimal large language models,” arXiv:2203.15556, Mar. 2022. [9] Y. Li et al., “BEHRT: Transformer for electronic health records,” Sci. Rep., vol. 10, p. 7155, Apr. 2020. [10] L. Rasmy, Y. Xiang, Z. Xie, C. Tao, and D. Zhi, “Med-BERT: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction,” npj Digit. Med., vol. 4, p. 86, May 2021. [11] C. Pang et al., “CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks,” in Proc. Mach. Learn. Health (ML4H), 2021, pp. 239–260. [12] Y. Li et al., “Hi-BEHRT: Hierarchical transformer-based model for accurate prediction of clinical events using multimodal electronic health records,” IEEE J. Biomed. Health Inform., vol. 26, no. 2, pp. 709–719, 2022. [13] A. Vaswani et al., “Attention is all you need,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 5998–6008. [14] L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?” arXiv:2207.08815, Jul. 2022. [15] T. Laurent et al., “Context and considerations for use of two Japanese real-world databases in Japan: Medical Data Vision and Japanese Medical Data Center,” Drugs—Real World Outcomes, vol. 9, no. 2, pp. 175–187, Jun. 2022. [16] N. Stamas et al., “Use of healthcare claims data to generate real-world evidence on patients with drug-resistant epilepsy: Practical considerations for research,” J. Health Econ. Outcomes Res., vol. 11, no. 1, pp. 57–66, Feb. 2024. [17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,” arXiv:1810.04805, Oct. 2018. [18] G. Ke et al., “LightGBM: A highly efficient gradient boosting decision tree,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 3146– 3154. [19] X. Yang et al., “A large language model for electronic health records,” npj Digit. Med., vol. 5, no. 1, p. 194, 2022. [20] L. R. Jiang et al., “Health system-scale language models are all-purpose prediction engines,” Nature, vol. 619, pp. 357–362, 2023. [21] K. Singhal et al., “Large language models encode clinical knowledge,” Nature, vol. 620, pp. 172–180, 2023. [22] Z. Yang, A. Mitra, W. Liu, D. Berlowitz, and H. Yu, “TransformEHR: Transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records,” Nature Commun., vol. 14, p. 7857, 2023. [23] K. Hur et al., “GenHPF: General healthcare predictive framework for multi-task multi-source learning,” IEEE J. Biomed. Health Inform., vol. 28, no. 10, pp. 6098–6108, 2024. [24] M. Wornow, Y. Rahul, E. Steinberg, S. Fleming, N. H. Shah, and J. A. Fries, “EHRSHOT: An EHR benchmark for few-shot evaluation of foundation models,” in Proc. 37th Conf. Neural Inf. Process. Syst. (NeurIPS) Datasets Benchmarks Track, 2023. [25] S. Waxler et al., “Generative medical event models improve with scale,” arXiv:2508.12104, 2025. [26] J. Yan et al., “Making pre-trained language models great on tabular prediction,” in Proc. Int. Conf. Learn. Representations (ICLR), 2024. [27] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko, “Revisiting deep learning models for tabular data,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2021. [28] Y. Gorishniy, I. Rubachev, and A. Babenko, “On embeddings for numerical features in tabular deep learning,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2022. [29] A. E. W. Johnson et al., “MIMIC-IV, a freely accessible electronic health record dataset,” Sci. Data, vol. 10, p. 1, 2023. [30] M. Moor et al., “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, pp. 259–265, 2023. [31] Y. Jin, J. G. Weberpals, S. V. Wang, R. J. Desai, D. Merola, and K. J. Lin, “The impact of longitudinal data-completeness of electronic health record data on the prediction performance of clinical risk scores,” Clin. Pharmacol. Therapeutics, vol. 113, no. 6, pp. 1359–1367, Jun. 2023. [32] A. Datta et al., “A multi-center study on the adaptability of a shared foundation model for electronic health records,” npj Digit. Med., vol. 7, p. 191, 2024.
9
Supplementary Materials
1e5 6.0
Count
5.0
A. Summary Statistics of the Full Analysis Cohort (N = 2,294,687) Table S1 summarizes the full analysis cohort (N = 2,294,687). Fig. S1a–S1d show distributions of sequence length, unique timestamps, observation duration, and age. Note that the sequence length reported here counts only the clinical event tokens (ICD-10 and YJ codes), excluding the initial control tokens ([CLS] and sex). Sequence length and unique timestamps exhibited heavy right tails, while age peaked at 27,000–30,000 days (74–82 years), consistent with an older adult population.
4.0 3.0 2.0 1.0 0.0
0
500
1000 1500 2000 2500 3000 3500 4000
Observation Duration [days] Fig. S1c. Distribution of observation duration (days). The yaxis indicates patient count.
TABLE S1: Input-sequence statistics for the full analysis cohort (N = 2,294,687 patients)
Sequence length (tokens) Number of unique timestamps (days) Observation duration (days) Age at first record (days)
22 [7–104] 7 [2–25] 249 [23–1,645] 26,335 [21,634–29,563]
0.6
0.2 0.0
2.0
Count
0.8
0.4
2.5
0
5000
10000 15000 20000 25000 30000 35000 40000
Age [days]
1.5
Fig. S1d. Distribution of age in days. The y-axis indicates patient count.
1.0 0.5 0.0
101
102
103
104
105
Sequence Length [tokens] Fig. S1a. Distribution of sequence length (tokens; x-axis on a logarithmic scale). The y-axis indicates patient count. 1e5 5.0 4.0
Count
1e6
1.0
1e5
3.0 2.0 1.0 0.0
1.2
Median [Interquartile Range]
Count
Statistic
100
101
102
103
Number of Unique Timestamps [days]
Fig. S1b. Distribution of the number of unique timestamps (days; x-axis on a logarithmic scale). The y-axis indicates patient count.
B. Pretrained Versus From-Scratch Models Across All Five Model Sizes (AUROC/AUPRC) Fig. S2 extends the comparison between pretrained and fromscratch models, shown for the 11M model as a representative example in Fig. 4 of the main text, to all five model sizes (2.2M, 4.7M, 11M, 32M, and 101M). For each model size, test AUROC (left column) and test AUPRC (right column) are plotted against the number of labeled patients used for finetuning (100, 500, and 1,000). Pretrained models are shown in model-specific colors (2.2M = blue, 4.7M = orange, 11M = green, 32M = red, and 101M = purple), whereas from-scratch models are shown as gray-to-black dashed lines. Across all model sizes, the performance gain associated with pretraining was more evident for AUPRC than for AUROC. In the fine-tuning datasets used in this study, the prevalence of positive labels ranged from 2.63% to 14.9%, indicating substantial class imbalance toward the negative class. The clearer improvement observed in AUPRC, a metric that more appropriately reflects performance under class imbalance, therefore provides practically important evidence for the value of pretraining in clinically relevant prediction tasks.
1. Primary Hypertension
1. Primary Hypertension
2. Chronic Kidney Disease
2. Chronic Kidney Disease
3. Amlodipine
AUPRC
AUROC
10
3. Amlodipine
4. Pregabalin
4. Pregabalin
Number of Patients
Number of Patients
2.2M Pretrained Model Seed 42 Seed 123 Seed 456
From Scratch Model Seed 42 Seed 123 Seed 456
1. Primary Hypertension
1. Primary Hypertension
2. Chronic Kidney Disease
2. Chronic Kidney Disease
3. Amlodipine
AUPRC
AUROC
11
3. Amlodipine
4. Pregabalin
4. Pregabalin
Number of Patients
Number of Patients
4.7M Pretrained Model Seed 42 Seed 123 Seed 456
From Scratch Model Seed 42 Seed 123 Seed 456
1. Primary Hypertension
1. Primary Hypertension
2. Chronic Kidney Disease
2. Chronic Kidney Disease
3. Amlodipine
AUPRC
AUROC
12
3. Amlodipine
4. Pregabalin
4. Pregabalin
Number of Patients
Number of Patients
11M Pretrained Model Seed 42 Seed 123 Seed 456
From Scratch Model Seed 42 Seed 123 Seed 456
1. Primary Hypertension
1. Primary Hypertension
2. Chronic Kidney Disease
2. Chronic Kidney Disease
3. Amlodipine
AUPRC
AUROC
13
3. Amlodipine
4. Pregabalin
4. Pregabalin
Number of Patients
Number of Patients
32M Pretrained Model Seed 42 Seed 123 Seed 456
From Scratch Model Seed 42 Seed 123 Seed 456
1. Primary Hypertension
1. Primary Hypertension
2. Chronic Kidney Disease
2. Chronic Kidney Disease
3. Amlodipine
AUPRC
AUROC
14
3. Amlodipine
4. Pregabalin
4. Pregabalin
Number of Patients
Number of Patients
101M Pretrained Model Seed 42 Seed 123 Seed 456
From Scratch Model Seed 42 Seed 123 Seed 456
Fig. S2. Comparison of test AUROC (left column) and test AUPRC (right column) between pretrained and from-scratch models across all five model sizes (2.2M = blue, 4.7M = orange, 11M = green, 32M = red, and 101M = purple). The x-axis indicates the number of patients with fine-tuning labels. Solid colored lines denote pretrained models, whereas gray-to-black dashed lines denote from-scratch models. Each line corresponds to one of three data splits (seed = 42, 123, or 456).