Conceptio › Archive › arXiv CS

arXiv CSopen access

MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling

arXiv:2605.13711v1 [cs.LG] 13 May 2026

Hsing-Huan Chung1 Shijun Li1 Yoav Wald2 Xing Han3 Suchi Saria3 Joydeep Ghosh1 1 University of Texas at Austin 2 Technion-IIT 3 Johns Hopkins University

Abstract Multimodal irregular time series (MITS) consist of asynchronous and irregularly sampled observations from heterogeneous numerical and textual channels. In healthcare, for example, patients’ electronic health records (EHR) include irregular lab measurements and clinical notes. The irregular timing and channel patterns of observations carry predictive signal alongside the numerical values and textual content. LLMs are natural candidates for processing such heterogeneous data, given their extensive pretrained knowledge spanning textual and numerical domains. Existing MITS models focus on specialized fusion mechanisms without leveraging LLMs for unified MITS processing, while LLM-based approaches for irregular time series focus on purely numerical observations, leaving MITS an underexplored setting for LLMs. Moreover, no prior work has examined whether LLMs can learn to use informative sampling patterns inherent to MITS. We introduce MILM (Multimodal Irregular time series Language Model), which represents MITS as time-ordered triplets in Extensible Markup Language (XML) format and fine-tunes an LLM through a two-stage strategy for MITS classification. The first stage trains on value-redacted MITS to predict from sampling patterns alone, and the second stage trains on full MITS to jointly model sampling patterns and observed values. Our two-stage model (MILM-2S) and its single-stage counterpart (MILM-Direct) achieve the best and second-best average performance on multiple EHR datasets. Further value redaction evaluations confirm that sampling patterns carry predictive signal and that MILM-2S learns to exploit them. In the value pending evaluation we introduce, where some values are unavailable at prediction time, MILM-2S outperforms MILM-Direct by a larger margin compared to standard evaluation. For MILM-2S, preserving the time and channel of value-pending observations as additional sampling information further improves in-hospital mortality prediction.

Introduction

Multimodal irregular time series (MITS) [1] consist of asynchronous and irregularly sampled observations from heterogeneous channels spanning numerical and textual modalities. MITS are prevalent in healthcare [2], finance [3], and system monitoring [4] among others. For example, in electronic health records (EHR) [5–8], a patient’s stay comprises lab measurements and clinical notes irregularly ordered and authored over time due to their on-demand nature. Unlike regularly sampled time series, which arise from periodic measurement of data processes, MITS are jointly governed by data processes and sampling processes, where the latter determine when and which channels are observed. This dual-process nature introduces irregularity in the timing and channel patterns of observations, making them informative signals beyond the observed values. Combined with the heterogeneity of modalities, this makes MITS a richer yet more challenging source of predictive information. In this work, we focus on MITS classification motivated by clinical prediction tasks [2, 9, 10]. A canonical example is in-hospital mortality prediction, where a model processes a patient’s lab measurements and clinical notes to predict whether the patient will survive their hospital admission. Preprint.

Accurate prediction requires interpreting numerical lab values alongside clinical narratives, while also exploiting when and which lab tests are ordered and clinical notes are authored as informative signals about the patient’s evolving condition. LLMs are natural candidates for processing such heterogeneous observations, given their extensive pretrained knowledge spanning both textual and numerical domains. Existing MITS models [2, 9, 10] focus on specialized multimodal fusion mechanisms, leaving the use of LLMs for unified processing of the temporal structure, numerical values, and textual content underexplored. Meanwhile, LLM-based approaches for irregular time series [11, 12] focus on purely numerical observations without the textual modalities that make MITS a particularly natural fit for LLMs. Moreover, no prior work has examined whether and how LLMs can leverage sampling patterns in MITS as predictive signal in their own right. To this end, we develop an LLM-based framework for classifying MITS with informative sampling. We introduce MILM (Multimodal Irregular time series Language Model), a framework that serializes MITS into time-ordered triplets in Extensible Markup Language (XML) format and adapts an LLM via a two-stage fine-tuning strategy. Time-ordered serialization preserves the causal structure of the data and sampling processes, while the XML format makes the structured nature of MITS explicit. MILM first performs value redaction fine-tuning, teaching the model to predict from sampling patterns alone, and then continues fine-tuning on the full data to jointly utilize observed values and the learned sampling patterns. Empirically, the two-stage variant, MILM-2S, achieves the best average rank across all datasets and metrics, while the variant directly fine-tuned on the full data, MILM-Direct, ranks second overall. We further evaluate the models under value redaction and show that sampling patterns carry predictive signal, that Stage 1 effectively captures this signal, and that Stage 2 retains most of this capability while still outperforming both the off-the-shelf LLM and MILM-Direct. We also introduce a realistic value pending evaluation, where some values are unavailable at prediction time. MILM-2S consistently outperforms MILM-Direct under both countermeasures of dropping value-pending observations and revealing their time and channel. For MILM-2S, preserving the time and channel of value-pending observations as additional sampling information further improves in-hospital mortality prediction on MIMIC-IV. In summary, our contributions are: • We present the first study of LLMs for MITS classification and the first investigation of whether and how LLMs can learn to exploit informative sampling patterns in this setting. • We propose MILM (Multimodal Irregular time series Language Model), which represents MITS as time-ordered triplets in XML format and applies two-stage fine-tuning to first learn from sampling patterns and then jointly model sampling patterns and values. • We present extensive experiments on EHR-derived MITS datasets, showing that MILM achieves strong performance, while additional value redaction and value pending evaluations demonstrate that it can effectively learn and use informative sampling patterns.

Related Work

2.1

Irregular Time Series

Irregular time series [13] arise when observations arrive at non-uniform time intervals and different variables are recorded asynchronously. One common approach is to represent each sample with aligned value, mask, and time-gap matrices, allowing sequence models to exploit observed values and missingness patterns [14]. Other methods such as IP-Net [15], mTAND [16], and ATENet [17] learn interpolation representations at reference times. Another line of work explicitly models continuous-time latent dynamics with ODEs [18, 19], CDEs [20], and SDEs [21, 22]. Several recent approaches adapt attention [23, 24] or recast irregular time series as sets [25], graphs [26–28], or images [29] to address asynchronous measurements and cross-variable dependencies. The above approaches do not consider the multimodal setting where channels may also include free text. 2.2

Multimodal Irregular Time Series

Multimodal irregular time series (MITS) [1] extend the irregular time series setting beyond purely numerical observations to include additional modalities such as text. Khadanga et al. [9] use timestamped clinical notes with numerical time series for ICU prediction, but handle the irregular note stream with simple temporal weighting. UTDE [2] uses mTAND [16] to model both irregular numerical observations and timestamped clinical note embeddings. The resulting representations of 2

the text and numerical channels are then fused with self- and cross-attention blocks for prediction. FuseMoE [10] further extends multimodal fusion to fleximodal settings with additional modalities and mixture-of-experts routing. Time-IMM [1] is a recent benchmark for forecasting on irregular multimodal time series. In contrast, we study time-series-wise MITS classification where each full trajectory is itself a labeled sample, in line with prior clinically motivated multimodal prediction settings [2, 9, 10]. To the best of our knowledge, prior work has not studied the use of LLMs for jointly processing the temporal structure, numerical values, and textual content under this setting. 2.3

LLM for Time Series

Prior LLM-for-time-series methods have largely focused on regularly sampled numerical series, either by serializing values into text for direct prompting [30, 31] or by aligning numerical sequences to the LLM representation space [32–39]. There is also a line of work on multimodal settings [40–46] where time series are combined with text used as context or summaries rather than as observations within the temporal sequence. Adjacent work on time-series foundation models [47–50] also relies on large transformer architectures, but is pretrained on time-series corpora rather than natural language. Aside from regularly sampled time series, LLMs have also been applied to continuous-time event streams [51–53]. These works focus on temporal point process problems such as next-event time/type prediction and intensity estimation, whereas we study time-series-wise prediction from timestamped channel-value observations whose values can be numerical or textual. On the other hand, LLMs for purely numerical irregular time series [11, 12] have also been explored. VITAL [11] extends Time-LLM [34], a representation alignment approach for regularly sampled time series, to handle missingness. ISTS-PLM [12] uses time, variable, and value embedders to encode irregular numerical observations before processing them with a pretrained language model. In contrast, we study LLMs for multimodal irregular time series, where observations arrive asynchronously across channels and their values may be either numerical measurements or unstructured text. 2.4

Informative Presence/Missingness/Sampling

Informative presence or missingness [54, 55] refers to the setting where the presence or absence of data is itself informative because data are often missing not at random [56]. Informative sampling [57, 58] extends this idea to time: the fact that measurements are taken at particular times can also carry signal, e.g., increasingly frequent laboratory tests may indicate worsening health status. Prior work has examined informative sampling as a source of bias in longitudinal regression [57, 59], treatment effect forecasting [58], and time-to-event modeling under changing sampling policies [60]. Related ideas also appear in irregular time series models that treat sampling patterns as part of the predictive signal, for example through measurement intensity [15, 18] or elapsed time and missingness indicators [14]. More recently, Kobayashi et al. [61] study the impact of informative missingness on LLM predictions from static data. In contrast, we study whether LLMs can learn and use informative sampling patterns in the temporal setting of multimodal irregular time series.

Problem Formulation

3.1

Multimodal Irregular Time Series

We consider multimodal irregular time series (MITS) with observations at arbitrary time points within a time range IT = [Tstart , Tend ]. We denote the set of channel types by C. In the EHR example (Fig. 1a), C could contain numerical laboratory test channels and textual clinical note channels. For each channel c ∈ C, we consider two interacting processes. Data Process. Let Xc (t) : IT → Xc denote the value of channel c at time t, where Xc is the channel value space (e.g., R for numerical measurements, a string space for clinical notes). This process is only observed when channel c is sampled. Sampling Process. Let Nc (t) : IT → N be a counting process recording the cumulative number of observations of channel c up to time t. It increments by 1 whenever channel c is observed: dNc (t) = 1 if an observation occurs at time t, and dNc (t) = 0 otherwise, where dNc (t) = Nc (t) − lims→t− Nc (s). The sampling processes may depend on the data processes. 3

Creatinine

0.7

Glucose

107 141 114 91

0.6 125

118 200

136

Sodium Radiology Note

136

Time Since ICU Admission (Hr)

Clinical Note Count (First 24 Hr)

100

PORTABLE CHEST ___ COMPARISON: ___...

Lab Measurement Count (First 24 Hr)

(a)

Survival (b)

Mortality

Survival

Mortality

(c)

Figure 1: 1a: A patient’s lab measurement and clinical note trajectory from the first 24 hours of an ICU stay, drawn from the MIMIC-IV-Demo database [62]. 1b and 1c: Distributions of numbers of lab measurements and clinical notes within 24 hours grouped by outcome. Although counts are rough summaries of sampling patterns, mortality cases already exhibit more frequent observations. The full sampling patterns carry finer-grained signals that may be even more predictive of the outcome. (c)

(c)

For each channel c, let t1 < · · · < tnc denote the times at which dNc (t) = 1, i.e. the nc = Nc (Tend ) (c) (c) c observation times of channel c. The per-channel observation sequence is sc = ((tk , xk ))nk=1 , (c) (c) where xk = Xc (tk ) is the value or content. The full MITS is the collection {sc |c ∈ C}. 3.2

Multimodal Irregular Time Series Classification

Let Y ∈ {1, . . . , K} denote the label of a MITS sample. We observe a dataset D = {Si }M i=1 consisting of M i.i.d. realizations of the random observed data object S = ({sc |c ∈ C}, Y ). The goal is to learn a MITS classifier f that minimizes the expected classification loss ES [l(f ({sc |c ∈ C}), Y )], where l could be the 0/1 loss or cross-entropy loss. Each sc encodes two sources of information: (c) (c) (i) the observed values xk that directly reflect the data process and (ii) the sampling patterns tk and nc , which are also informative about Y because sampling may depend on both the observed and unobserved parts of the data process. For example, a patient with a critically abnormal lab result or a worsening clinical state not yet documented in the system may be assigned more follow-up tests. Indeed, in the MIMIC-IV EHR database [5, 6, 63], patient stays resulting in in-hospital mortality exhibit more lab measurements and clinical notes than surviving stays (Fig. 1b and 1c), though the total count is merely a coarse summary. The full sampling pattern that encompasses timings, channel identities, and orderings encodes even richer information. An effective MITS classifier should make use of both observed values and sampling patterns for prediction.

Method

We introduce MILM: Multimodal Irregular time series Language Model (Fig. 2), a framework that serializes MITS into time-ordered triplets in XML format and employs a two-stage fine-tuning strategy to adapt an LLM to exploit both sampling patterns and values for MITS classification. 4.1

MITS Representation

We design a MITS representation that enables LLMs to process numerical and textual observations. Prior work on LLMs for irregular time series [11, 12] views time series observations as numerical objects and transforms them into a language model’s token embedding space, using representations foreign to the LLM’s pretraining. This limits the model’s ability to draw on pretrained knowledge. We instead represent MITS directly in the LLM’s native token vocabulary, enabling joint processing of numerical and text modalities without representational alignment. We explain the process below: Time-Ordered Serialization. As shown in Fig. 2, a MITS {sc |c ∈ C} is a sparse 2-D object indexed by channel and time. We apply time-ordered serialization, augmenting each observation (c) (c) (tk , xk ) ∈ sc with its channel label c and globally sorting by time, breaking ties by predefined n channel order, to obtain a flattened sequence ((tP i , ci , xi ))i=1 , where ti , ci , and xi are the time, channel, and value of the i’th observation, n = c∈C nc , and ti ≤ tj for all i < j. Preserving 4

) ) /uL q/L q/L) te (K E o lls e (m mE gy N ( e t p C o a od rbon n Ga diol o l a R d B Bica Anio Re

0.05

2.85

2.30

3.04

Stage 1: Value-Redaction Fine-Tuning <answer>

Task

Data Format

Value-Redacted MITS Final Instruction

<time> 0.05 hour </time> <channel> Anion Gap (mEq/L) </channel> <time> 0.05 hour </time> <channel> Red Blood Cells (K/uL) </channel> <time> 2.30 hour </time> <channel> Red Blood Cells (K/uL) </channel> <time> 3.27 hour </time> <channel> Radiology Note </channel> ... <time> 19.10 hour </time> <channel> Red Blood Cells (K/uL) </channel>

20.00 13.00

Stage 2: Full Data Fine-Tuning & Inference

...

... 17.93

</answer>

LLM

11.00

3.27

4.43

A B

2.73

A B

</answer>

LLM 19.10

2.85 19.00 14.00

Task

Data Format

Time (hr)

Time-Ordered Serialization & XML Representation

Full MITS

Final Instruction

Text

<time> 0.05 hour </time> <channel> Anion Gap (mEq/L) </channel> <value> 11.00 </value> <time> 0.05 hour </time> <channel> Red Blood Cells (K/uL) </channel> <value> 2.85 </value> <time> 2.30 hour </time> <channel> Red Blood Cells (K/uL) </channel> <value> 3.04 </value> <time> 3.27 hour </time> <channel> Radiology Note </channel> <value> ... </value> ... <time> 19.10 hour </time> <channel> Red Blood Cells (K/uL) </channel> <value> 2.85 </value>

Figure 2: An illustration of MILM. MILM serializes the MITS to a time-ordered XML representation and fine-tunes the LLM in two stages. Value-redaction fine-tuning first teaches the model to predict from sampling patterns alone, while full-data fine-tuning then incorporates value information on top of the developed sampling-pattern knowledge. During inference, MILM uses the Stage 2 prompt format and extracts the answer-token logits as prediction scores. global temporal order ensures that the causal structure of the sampling and data processes is directly reflected in the sequence and accessible to the model. XML-Formatted Triplet Representation. Each observation (ti , ci , xi ) in the serialized sequence has three semantically distinct components: when it occurred, which channel was observed, and what value was recorded. We represent each observation as an XML-formatted triplet with tags <time>, <channel>, and <value>, explicitly encoding this three-part structure so that the boundary between observations and the role of each component remains unambiguous to the model. Consecutive observations are separated by a newline. For example, the following snippet specifies a MITS with 3 observations from the glucose, anion gap, and radiology note channels: <time> 0.88 hours </time> <channel> Glucose (mg/dL) </channel> <value> 170.00 </value> <time> 3.80 hours </time> <channel> Anion Gap (mEq/L) </channel> <value> 11.00 </value> <time> 4.00 hours </time> <channel> Radiology Note </channel> <value> Portable chest... </value>

This representation has two advantages. First, expressing observations directly in the LLM’s native vocabulary allows pretrained token embeddings to apply without representation alignment. Preserving timings and channel identities as natural language additionally grounds each value in semantic context, enabling the model to draw on pretrained knowledge when interpreting it. Second, the XML format signals that the input is a data object indexed by time and channel rather than free-form text, preserving the structured nature of MITS within a text-based representation. 4.2

LLM For MITS Classification

We describe how we use LLMs for MITS classification, covering prompt and target construction for both inference and supervised fine-tuning (SFT), and prediction score extraction at inference time. Prompt and Target Construction. To condition the LLM for the prediction task, the prompt consists of three specification blocks followed by the serialized MITS and a final instruction. The task specification describes the data domain and the prediction target, e.g., “timestamped observation from an ICU stay” and “predict whether the patient will survive their hospital admission”. The data 5

format specification explains the triplet XML format and the semantics of each tag (e.g., the <time> tag records hours since the start of the ICU stay). The output format specification maps each class label to an alphabet letter and instructs the model to place its prediction in an <answer> tag (e.g., <answer> B </answer>). See Appendix A for the full prompt templates. For SFT, the training target is the ground-truth class letter and the enclosing <answer> tags. The model is trained via next-token prediction loss on the full target sequence. Prediction Score Extraction. Following Robinson and Wingate [64], at inference we extract prediction scores from the logits at the position immediately following the <answer> open tag and apply softmax over the class letter tokens to obtain a probability distribution. We prefer logitbased extraction over verbalizing class probabilities because logits faithfully reflect the model’s class assignment belief, whereas verbalizing interposes an extra decoding step. Also, verbalizing probability scores requires numerical confidence targets during SFT, which are typically unavailable. 4.3

Two-Stage Fine-Tuning

An effective MITS classifier should exploit signals from both values and sampling patterns. However, the signal in sampling patterns is subtler than that in values, particularly when text channels are present and value tokens constitute a large portion of the sequence. To avoid bypassing sampling patterns in favor of the more immediately apparent value signal, we propose a two-stage fine-tuning strategy that forces sampling-pattern representations to develop before value information is introduced. Stage 1: Value Redaction Fine-Tuning. In the first stage, we fine-tune the LLM on MITS with all values redacted. Each observation is presented as a time-channel pair, as in the following example: <time> 0.88 hours </time> <channel> Glucose (mg/dL) </channel> <time> 3.80 hours </time> <channel> Anion Gap (mEq/L) </channel> <time> 4.00 hours </time> <channel> Radiology Note </channel>

The data format specification in the prompt is adjusted to reflect the redacted format. An additional note clarifies that values were recorded but withheld and instructs the model to rely on timing and channel patterns for prediction. Stage 1 trains the model to develop predictive representations of the sampling processes {Nc (t)|c ∈ C} through the timings and channel identities of observations. Stage 2: Full Data Fine-Tuning. In the second stage, we continue fine-tuning from the Stage 1 checkpoint on the full MITS data, using the same representation and prompt construction described in Sections 4.1 and 4.2. Building on the sampling-pattern representations developed in Stage 1, Stage 2 trains the model to jointly exploit both the data processes {Xc (t)|c ∈ C} and the sampling processes {Nc (t)|c ∈ C}. The Stage 2 model is used for the final prediction. This two-stage scheme is enabled by the text-based representation: removing the <value> tokens preserves the presence of each observation while unambiguously signaling that its value is withheld, allowing the same model architecture to process both redacted and full MITS without modification.

Experiments

5.1

Setup

Data. We conduct experiments on datasets extracted from two major public EHR databases: MIMIC-IV [5, 6, 63] and eICU Collaborative Research Database [7, 8]. Similar to Jeanselme et al. [60], we extract the laboratory tests as the numerical channels due to their inherent irregular sampling nature. In addition, we extract the clinical notes as the text channel. On each database, we develop two datasets corresponding to the in-hospital mortality (IHM) and length-of-stay (LOS) prediction tasks [65, 66] given the first 24 hours of laboratory tests and clinical notes of the ICU stay. We describe the datasets in detail in Appendix B. Baselines. We select 3 classes of irregular time series model baselines: (i) Non-LLM-based: GRUD [14], mTAND [16], and VITAL-stats [11] (ii) LLM-based: VITAL-LLM [11] and ISTS-PLM [12], and (iii) Multimodal: UTDE [2] and FuseMoE [10]. Since the datasets belong to the biomedical domain, we use BioBERT [67] as the note embedding model. The unimodal baselines do not use the 6

Table 1: Results on the in-hospital mortality (IHM) and length-of-stay (LOS) prediction datasets extracted from MIMIC-IV and eICU. Best results are boldfaced and second-best results are underlined. MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

Avg. Rank

Method

AU-ROC

GRU-D

75.91±1.03 74.52±1.71 77.38±0.90 77.90±1.16 75.23±3.10 76.21±1.45 76.99±0.66 77.80±0.68 66.43±2.23 67.02±2.34 80.49±0.79 80.07±0.52 69.82±0.00

54.08±1.02 52.28±4.13 56.52±1.55 58.57±0.68 56.14±3.72 56.27±0.78 54.06±2.06 54.32±1.67 39.61±2.45 39.22±3.20 60.10±1.14 59.06±0.62 48.36±0.00

70.08±0.83 70.93±2.54 70.98±0.96 74.30±0.82 75.68±2.78 76.13±0.24 73.62±0.96 73.99±0.98 67.60±1.29 67.97±0.99 73.92±0.87 73.68±0.61 67.98±0.00

46.89±1.17 47.84±2.45 48.99±1.68 54.26±1.76 56.50±3.53 57.03±1.62 52.02±2.89 53.83±3.61 45.94±2.75 47.49±1.52 54.04±1.77 52.09±1.13 44.82±0.00

66.04±0.99 66.85±1.21 67.13±1.35 66.99±1.33 72.98±0.92 70.57±0.75 70.51±1.48 69.77±1.05 61.92±2.72 61.19±2.51 68.93±0.72 69.02±0.65 64.53±0.00

34.98±3.05 35.97±1.18 36.42±3.11 37.11±2.52 41.58±1.16 40.43±1.27 39.36±2.68 38.54±2.01 28.68±2.44 28.09±2.53 36.32±1.18 36.12±0.32 29.43±0.00

60.48±2.14 57.40±3.51 58.10±1.77 64.79±1.03 63.96±3.05 65.37±2.15 60.01±2.42 59.83±1.07 60.14±2.04 59.91±2.13 62.66±1.33 62.51±1.39 55.71±0.00

51.13±3.09 45.84±4.02 50.36±1.29 52.20±0.64 50.35±4.23 52.48±3.29 48.12±1.56 48.67±1.42 50.14±2.51 49.63±1.64 52.79±1.44 52.82±1.38 45.58±0.00

10.5 12.0 8.9 5.9 5.6 4.6 8.8 8.0 13.1 13.4 5.8 6.4 13.8

MILM-Direct 80.82±0.63 63.69±1.29 78.24±0.60 57.43±2.01 71.08±0.69 41.77±1.72 64.97±1.42 55.76±1.08 MILM-2S 81.30±0.70 63.86±1.47 78.82±0.17 57.45±1.14 72.47±1.49 42.27±2.10 66.05±1.79 56.22±1.20

2.3 1.1

+ Note

mTAND + Note

UTDE FuseMoE VITAL-stats + Note

VITAL-LLM + Note

ISTS-PLM + Note

Qwen3-4B

text channel by default. For those models, we implement a “+ Note” version that uses the timestampto-text fusion (TTF) module proposed by Chang et al. [1] to learn the text channel representation to be concatenated with the numerical channel representation before the prediction head. MILM Setup. We fine-tune Qwen3-4B-Instruct-2507 [68] and experiment with two variants of MILM: MILM-Direct, which directly does full data fine-tuning, and MILM-2S, which applies the two-stage fine-tuning strategy described in Section 4. Both variants use QLoRA [69] with rank r = 16, α = 16, and dropout 0.05 applied to the query, key, value, and output projection matrices of the LLM. We use a learning rate of 10−4 with a cosine schedule and warmup ratio 0.05. Stage 1 of MILM-2S trains for 8 epochs. Stage 2 initializes from the best Stage 1 checkpoint, and the number of training epochs is dataset-dependent. At inference, we use SGLang [70] to serve the fine-tuned model in bfloat16 precision with logit-based prediction score extraction. We also evaluate the off-the-shelf Qwen3-4B-Instruct-2507 (hereafter Qwen3-4B) as a zero-shot baseline. See Appendix C for further details regarding implementation and hyper-parameters. Evaluation. Each dataset is split into 70%/15%/15% for train/validation/test. We use Area Under the Receiver Operating Characteristic Curve (AU-ROC) and Average Precision (AP) to evaluate the quality of the classification scores and report the average and standard deviation over 5 runs. 5.2

Main Results

Table 1 summarizes performance across all four datasets. MILM-2S achieves the best average rank of 1.1 across all eight metric-dataset combinations, with MILM-Direct a close second at 2.3. Among the baselines, the native multimodal models FuseMoE (avg. rank 4.6) and UTDE (avg. rank 5.6) outperform unimodal ones. Their advantage highlights the importance of well-designed multimodal fusion strategies. In contrast, simply augmenting unimodal models with a separately computed text representation via concatenation (“+ Note”) often fails to help and sometimes hurts performance, suggesting that the integration strategy matters as much as having the text channel at all. MILM takes a complementary approach. By interleaving numerical and text observations in a time-ordered serialization with causal structure preserved, the LLM jointly attends to both modalities, achieving the best overall performance. Comparing the two MILM variants against the zero-shot Qwen3-4B baseline illuminates the contribution of each training stage. Qwen3-4B achieves above-random performance (e.g., AU-ROC of 69.82 on MIMIC-IV-IHM), confirming that our constructed prompt conditions the LLM to effectively leverage its pretrained knowledge for the task even without fine-tuning. Yet it ranks last overall (avg. rank 13.8), indicating that domain-specific MITS patterns require further adaptation. MILM-Direct substantially closes this gap through direct fine-tuning on full MITS data, and MILM-2S improves further, demonstrating that first learning from sampling patterns alone provides a better foundation. 7

AU-ROC (%)

MIMIC-IV-IHM 60 40 20 0

AP (%)

MIMIC-IV-LOS 60 40 20 0

MIMIC-IV-IHM

MIMIC-IV-LOS 40

20 0

eICU-IHM

eICU-LOS 40

eICU-LOS

Qwen3-4B MILM-Direct MILM-2S (Stage 1) MILM-2S (Stage 2)

Figure 3: Value redaction evaluation across datasets. All models are tested with observed values removed and only time and channel information retained. MILM-2S (Stage 1) substantially outperforms both Qwen3-4B and MILM-Direct, while MILM-2S (Stage 2) retains most of the performance.

Among the LLM-based irregular time series baselines, VITAL-LLM performs the weakest overall (avg. rank 13.1), even trailing Qwen3-4B on several datasets. Both VITAL-LLM and ISTS-PLM require aligning numerical observations to the LLM’s token embedding space via text prototype reprogramming in VITAL-LLM [11], and via learnable time, variable, and value embedders in ISTS-PLM [12]. VITAL-LLM’s reprogramming matrix spans the entire vocabulary, making it difficult to train. MILM sidesteps this alignment problem entirely by serializing MITS directly as XML-formatted text, preserving the LLM’s pretrained token embeddings and the knowledge encoded within them, and outperforms both baselines across all four datasets. Note that the benefit of the text channel varies noticeably between the two databases. MIMIC-IV contains rich free-text clinical notes authored by physicians, whereas eICU stores structured care plan entries generated through the Philips eCareManager telehealth system [7], which are comparatively more templated and less semantically varied. This difference is reflected in a simple count: adding the text channel (“+ Note”) improves AU-ROC or AP over the numerical-only version in 13 cases among unimodal baselines on MIMIC-IV, but in only 7 on eICU. Nevertheless, MILM-2S remains the top-ranked method on eICU as well, demonstrating that it can outperform baselines even when the text channel carries weaker predictive signal. Interestingly, the performance gap between MILM-2S and MILM-Direct is larger on eICU (e.g., 1.39 AU-ROC on eICU-IHM) than on MIMIC-IV (0.48 on MIMIC-IV-IHM), suggesting that when the text channel carries weaker predictive signal, Stage 1’s emphasis on sampling patterns may provide a more important initialization advantage. 5.3

Value Redaction Evaluation

We conduct value redaction evaluation to understand how well each model exploits sampling patterns. We test off-the-shelf Qwen3-4B, MILM-Direct, and the Stage 1 and Stage 2 checkpoints of MILM-2S with all observed values removed, keeping only the time and channel information. The results are shown in Fig. 3. MILM-Direct does not improve over Qwen3-4B on MIMIC-IV, but does on eICU, where Qwen3-4B performs worse than random. MILM-2S (Stage 1) shows substantial improvement over both Qwen3-4B and MILM-Direct, demonstrating that sampling patterns are informative and can be learned through value redaction fine-tuning. MILM-2S (Stage 2) typically experiences a small degree of forgetting (e.g., a maximum decrease of 2.72 AU-ROC on MIMIC-IV-LOS), with the exception of MIMIC-IV-IHM, where performance slightly increases. Nevertheless, MILM-2S (Stage 2) still predicts more accurately from sampling patterns alone than both Qwen3-4B and MILM-Direct, showing that it retains most sampling-pattern knowledge after full-data fine-tuning. 5.4

Value Pending Evaluation

We study a practical evaluation setup we call value pending evaluation. In practice, some lab measurements or notes are taken before the prediction time, but their results are not yet available in the database when the model is queried for a prediction [71]. For these observations, the time and channel information are known, but the values are not. We consider two countermeasures: dropping these observations entirely, or showing their presence (i.e., the fact that a channel was sampled at a particular time) without revealing the values. For example, in the following snippet: 8

Table 2: Value pending evaluation results under two countermeasures: drop observation and show presence. Best results are boldfaced and second-best results are underlined. MIMIC-IV-IHM

MIMIC-IV-LOS

Countermeasure

Method

AU-ROC

Drop Observation

MILM-Direct MILM-2S

79.04±0.91 79.49±0.76

59.23±1.63 60.02±1.57

76.86±0.52 77.64±0.33

56.44±1.61 57.17±1.78

Show Presence

MILM-Direct MILM-2S

79.21±0.65 80.34±0.64

58.14±1.37 61.23±1.62

76.30±0.64 77.08±0.38

55.01±2.07 56.46±1.50

<time> 19.10 hours </time> <channel> Anion Gap (mEq/L) </channel> <value> 14.00 </value> <time> 19.10 hours </time> <channel> Bicarbonate (mEq/L) </channel> <time> 19.10 hours </time> <channel> Red Blood Cells (K/uL) </channel> <value> 2.85 </value>

the Bicarbonate value measured at 19.10 hours is not yet available, while those of Anion Gap and Red Blood Cells are. Showing presence conveys extra sampling information but risks exposing the model to a mixed format of pairs and triplets never seen before, challenging its generalization capability. In MIMIC-IV, each lab event and clinical note has two associated timestamps: charttime and storetime. charttime is the clinically relevant event time used for MITS construction, e.g., time of specimen acquisition or note authoring, and storetime is when the result became available in the database. For the value pending evaluation, we withhold the values of observations whose storetime exceeds the prediction time of 24 hours since ICU admission. eICU does not have an equivalent pair of time columns, so we focus this evaluation on MIMIC-IV. The results are shown in Table 2. See Appendix E for dataset statistics and full results including baselines under drop observation. Across both datasets and countermeasures, MILM-2S outperforms MILM-Direct by a larger margin compared to standard evaluation. With drop observation, MILM-2S better utilizes the remaining observations for prediction. With show presence, MILM-2S handles the mixed pair-triplet format more effectively than MILM-Direct, as its two-stage training has exposed it to time-channel pairs in Stage 1 and full triplets in Stage 2, despite never training on the two formats mixed together. On MIMIC-IV-LOS, show presence underperforms drop observation for MILM-2S, likely reflecting the slight forgetting of sampling-pattern knowledge during Stage 2 noted in Section 5.3. On MIMIC-IVIHM, however, show presence achieves the best overall result, suggesting that MILM-2S can utilize the extra sampling information for better in-hospital mortality prediction. Value-pending observations tend to fall at the end of the timeline and are often critical for predicting future outcomes [72]. Their unavailability can cause a nontrivial performance drop. We introduce value pending evaluation as a novel and practical evaluation setting, and encourage follow-up work in irregular time series modeling to consider this setting.

Conclusion

We introduce MILM, the first LLM-based framework for classifying multimodal irregular time series (MITS) with informative sampling. MILM represents observations as time-ordered triplets in XML format and applies a two-stage fine-tuning strategy. The first stage teaches the LLM to predict solely from sampling patterns, i.e., when and which channels are observed, while the second stage incorporates value information on top of the learned sampling patterns. The two-stage model achieves the strongest average performance across multiple EHR datasets. Value redaction evaluation further confirms that sampling patterns are predictive and that two-stage fine-tuning effectively learns and retains this predictive capability. We also introduce a practical value pending evaluation setup, where the values of some observations are unavailable at prediction time. In this setting, the two-stage model outperforms direct full data fine-tuning and leverages the time and channel information of valuepending observations to improve in-hospital mortality prediction. Future work includes extending MILM to additional modalities and studying robustness under sampling distribution shift, such as deployment across hospitals with different sampling policies. 9

References [1] Ching Chang, Jeehyun Hwang, Yidan Shi, Haixin Wang, Wei Wang, Wen-Chih Peng, and TienFu Chen. Time-IMM: A dataset and benchmark for irregular multimodal multivariate time series. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=yeqrrn51TL. [2] Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International conference on machine learning, pages 41300–41313. PMLR, 2023. [3] Ross Koval, Nicholas Andrews, and Xifeng Yan. Multimodal language models for financial forecasting from interleaved sequences of text and time series. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 987–1001, 2025. [4] Lingzhe Zhang, Tong Jia, Mengxi Jia, Yifan Wu, Aiwei Liu, Yong Yang, Zhonghai Wu, Xuming Hu, Philip S Yu, and Ying Li. A survey of aiops for failure management in the era of large language models. arXiv preprint arXiv:2406.11213, 2024. [5] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023. [6] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Brian Gow, Benjamin Moody, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV. PhysioNet, October 2024. doi: 10.13026/ kpb9-mt58. URL https://doi.org/10.13026/kpb9-mt58. Version 3.1. [7] Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):180178, 2018. [8] Tom Pollard, Alistair Johnson, Jesse Raffa, Leo Anthony Celi, Omar Badawi, and Roger Mark. eICU Collaborative Research Database. PhysioNet, April 2019. doi: 10.13026/C2WM1R. URL https://doi.org/10.13026/C2WM1R. Version 2.0. [9] Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. Using clinical notes with time series data for icu management. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6432–6437, 2019. [10] Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. Advances in Neural Information Processing Systems, 37: 67850–67900, 2024. [11] Jeong Eul Kwon, Joo Heung Yoon, and Hyo Kyung Lee. Mind the missing: Variable-aware representation learning for irregular ehr time series using large language models. arXiv preprint arXiv:2509.22121, 2025. [12] Weijia Zhang, Chenlong Yin, Hao Liu, and Hui Xiong. Unleashing the power of pre-trained language models for irregularly sampled time series. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 3831–3842, 2025. [13] Satya Narayan Shukla and Benjamin M Marlin. A survey on principles, models and methods for learning from irregularly sampled time series. arXiv preprint arXiv:2012.00168, 2020. [14] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018. [15] Satya Narayan Shukla and Benjamin Marlin. Interpolation-prediction networks for irregularly sampled time series. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=r1efr3C9Ym. 10

[16] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=4c0J6lwQ4_. [17] Sangho Lee, Kyeongseo Min, Youngdoo Son, and Hyungrok Do. Adaptive time encoding for irregular multivariate time-series classification. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [18] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019. [19] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32, 2019. [20] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in neural information processing systems, 33: 6696–6707, 2020. [21] Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International conference on machine learning, pages 19388–19405. PMLR, 2022. [22] Abdul Fatir Ansari, Alvin Heng, Andre Lim, and Harold Soh. Neural continuous-discrete state space models for irregularly-sampled time series. In International Conference on Machine Learning, pages 926–951. PMLR, 2023. [23] Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Contiformer: Continuous-time transformer for irregular time series modeling. Advances in Neural Information Processing Systems, 36:47143–47175, 2023. [24] Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. Warpformer: A multi-scale modeling approach for irregular clinical time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3273–3285, 2023. [25] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference on Machine Learning, pages 4353–4363. PMLR, 2020. [26] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Kwm8I7dU-l5. [27] Weijia Zhang, Chenlong Yin, Hao Liu, Xiaofang Zhou, and Hui Xiong. Irregular multivariate time series forecasting: A transformable patching graph neural networks approach. In Forty-first International Conference on Machine Learning, 2024. [28] Vijaya Krishna Yalavarthi, Kiran Madhusudhanan, Randolf Scholz, Nourhan Ahmed, Johannes Burchert, Shayan Jawed, Stefan Born, and Lars Schmidt-Thieme. Grafiti: Graphs for forecasting irregularly sampled time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16255–16263, 2024. [29] Zekun Li, Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series. Advances in Neural Information Processing Systems, 36:49187–49204, 2023. [30] Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 36(11):6851–6864, 2023. [31] Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in neural information processing systems, 36: 19622–19635, 2023. 11

[32] Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948, 2023. [33] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems, 36:43322–43355, 2023. [34] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=Unb5CVPtae. [35] Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. Test: Text prototype aligned embedding to activate llm’s ability for time series. arXiv preprint arXiv:2308.08241, 2023. [36] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes: Autoregressive time series forecasters via large language models. Advances in Neural Information Processing Systems, 37:122154–122184, 2024. [37] Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. S 2 ip-llm: Semantic space informed prompt learning with llm for time series forecasting. In Forty-first International Conference on Machine Learning, 2024. [38] Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tao Xia. Calf: Aligning llms for time series forecasting via cross-modal fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18915–18923, 2025. [39] Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards llm-empowered multivariate time series forecasting via cross-modality alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18780–18788, 2025. [40] Anastasiya Belyaeva, Justin Cosentino, Farhad Hormozdiari, Krish Eswaran, Shravya Shetty, Greg Corrado, Andrew Carroll, Cory Y McLean, and Nicholas A Furlotte. Multimodal llms for health grounded in individual-specific data. In Workshop on Machine Learning for Multimodal Healthcare Data, pages 86–102. Springer, 2023. [41] Nimeesha Chan, Felix Parker, William Bennett, Tianyi Wu, Mung Yao Jia, James Fackler, and Kimia Ghobadi. Medtsllm: Leveraging llms for multimodal medical time series analysis. arXiv preprint arXiv:2408.07773, 2024. [42] Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. Gpt4mts: Prompt-based large language model for multimodal time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 23343–23351, 2024. [43] Mingyue Cheng, Yiheng Chen, Qi Liu, Zhiding Liu, Yucong Luo, and Enhong Chen. Instructime: Advancing time series classification with multimodal language modeling. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pages 792–800, 2025. [44] Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12694–12702, 2025. [45] Geon Lee, Wenchao Yu, Kijung Shin, Wei Cheng, and Haifeng Chen. Timecap: Learning to contextualize, augment, and predict time series events with large language model agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18082–18090, 2025. 12

[46] Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A Xu, Winnie Chow, Martin Maritsch, Robert Jakob, Ning Wang, Juncheng Liu, Aradhana Verma, et al. Opentslm: Time-series language models for reasoning over multivariate medical text-and time-series data. arXiv preprint arXiv:2510.02410, 2025. [47] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688, 2023. [48] Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. arXiv preprint arXiv:2409.16040, 2024. [49] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815, 2024. [50] Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, et al. Chronos-2: From univariate to universal forecasting. arXiv preprint arXiv:2510.15821, 2025. [51] Zefang Liu and Yinzhu Quan. Tpp-llm: Modeling temporal point processes by efficiently fine-tuning large language models. arXiv preprint arXiv:2410.02062, 2024. [52] Shubham Gupta, Thibaut Durand, Graham Taylor, et al. Last stop for modeling asynchronous time series. arXiv preprint arXiv:2502.01922, 2025. [53] Quyu Kong, Yixuan Zhang, Yang Liu, Panrong Tong, Enqi Liu, and Feng Zhou. Byte-token enhanced language models for temporal point processes analysis. In Proceedings of the ACM Web Conference 2026, pages 7013–7023, 2026. [54] Rose Sisk, Lijing Lin, Matthew Sperrin, Jessica K Barrett, Brian Tom, Karla Diaz-Ordaz, Niels Peek, and Glen P Martin. Informative presence and observation in routine health data: a review of methodology for clinical risk prediction. Journal of the American Medical Informatics Association, 28(1):155–166, 2021. [55] Amelia LM Tan, Emily J Getzen, Meghan R Hutch, Zachary H Strasser, Alba GutiérrezSacristán, Trang T Le, Arianna Dagliati, Michele Morris, David A Hanauer, Bertrand Moal, et al. Informative missingness: What can we learn from patterns in missing laboratory data in the electronic health record? Journal of biomedical informatics, 139:104306, 2023. [56] Roderick JA Little and Donald B Rubin. Statistical analysis with missing data. John Wiley & Sons, 2019. [57] Haiqun Lin, Daniel O Scharfstein, and Robert A Rosenheck. Analysis of longitudinal data with irregular, outcome-dependent follow-up. Journal of the Royal Statistical Society Series B: Statistical Methodology, 66(3):791–813, 2004. [58] Toon Vanderschueren, Alicia Curth, Wouter Verbeke, and Mihaela Van Der Schaar. Accounting for informative sampling when learning to forecast treatment outcomes over time. In International Conference on Machine Learning, pages 34855–34874. PMLR, 2023. [59] Alessandro Gasparini, Keith R Abrams, Jessica K Barrett, Rupert W Major, Michael J Sweeting, Nigel J Brunskill, and Michael J Crowther. Mixed-effects models for health care longitudinal data with an informative visiting process: A monte carlo simulation study. Statistica Neerlandica, 74(1):5–23, 2020. [60] Vincent Jeanselme, Glen Martin, Matthew Sperrin, Niels Peek, Brian Tom, and Jessica Barrett. Prediction of survival outcomes under clinical presence shift: A joint neural network architecture. arXiv preprint arXiv:2508.05472, 2025. [61] Yuta Kobayashi, Vincent Jeanselme, and Shalmali Joshi. Mind the data gap: Missingness still shapes large language model prognoses. arXiv preprint arXiv:2512.00479, 2025. 13

[62] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV Clinical Database Demo. PhysioNet, January 2023. doi: 10.13026/dp1f-ex47. URL https://doi.org/10.13026/dp1f-ex47. Version 2.2. [63] Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IVNote: Deidentified free-text clinical notes. PhysioNet, January 2023. doi: 10.13026/1n74-ne17. URL https://doi.org/10.13026/1n74-ne17. Version 2.2. [64] Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=yKbprarjc5B. [65] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019. [66] Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benchmarking machine learning models on multi-centre eicu critical care dataset. Plos one, 15(7):e0235424, 2020. [67] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020. [68] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [69] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088– 10115, 2023. [70] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37: 62557–62583, 2024. [71] Erkin Otles, Jeeheh Oh, Benjamin Li, Michelle Bochinski, Hyeon Joo, Justin Ortwine, Erica Shenoy, Laraine Washer, Vincent B Young, Krishna Rao, et al. Mind the performance gap: examining dataset shift during prospective validation. In Machine Learning for Healthcare Conference, pages 506–534. PMLR, 2021. [72] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems, 29, 2016. [73] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7. [74] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. URL https://arxiv.org/ abs/1412.6980. [75] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [76] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019. [77] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 14

[78] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321, 2019. [79] Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. [80] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6.

Prompt Templates

We present the prompt templates for the in-hospital mortality (IHM) and length-of-stay (LOS) prediction tasks. The templates are organized into three groups based on the MITS representation used. The first group uses the full MITS representation, corresponding to Stage 2 of MILM and inference. The second group uses the value-redacted MITS representation, corresponding to Stage 1 of MILM. The third group corresponds to value pending evaluation with show presence, where some observations have been measured, but their values are unavailable at prediction time and are shown as time-channel pairs. A.1

Prompt Templates for Full MITS

Full MITS Prompt Template: In-Hospital Mortality Prediction System You are a critical care physician predicting outcomes from ICU monitoring data including laboratory measurements and radiology notes.

User TASK: Analyze timestamped observations from an ICU stay and predict whether the patient will survive their hospital admission. DATA FORMAT: Each observation is a triplet: <time> hours_since_start </time> <channel> channel_name </channel> <value> value </value> OUTPUT FORMAT (FOLLOW EXACTLY): Provide your answer in <answer> </answer> tags. Write ONLY a single letter:

A or B

(A) SURVIVAL (B) MORTALITY Do NOT include any explanatory text outside the tags. Example 1 (SURVIVAL): <answer> A </answer> Example 2 (MORTALITY): <answer> B </answer> [OBSERVATIONS] ... [/OBSERVATIONS]

Start directly with <answer>.

Prompt Template (continued) Based on the observations above, predict whether this patient will experience in-hospital mortality. Follow the output format exactly.

Start directly with <answer>.

Full MITS Prompt Template: Length-of-Stay Prediction System You are a critical care physician predicting outcomes from ICU monitoring data including laboratory measurements and radiology notes.

A or B

(A) LONG_STAY (B) SHORT_STAY Do NOT include any explanatory text outside the tags.

Start directly with <answer>.

Example 1 (LONG_STAY): <answer> A </answer> Example 2 (SHORT_STAY): <answer> B </answer> [OBSERVATIONS] ... [/OBSERVATIONS] Based on the observations above, predict whether this patient will have a short ICU stay and survive. Follow the output format exactly.

A.2

Start directly with <answer>.

Prompt Templates for Value-Redacted MITS

Value-Redacted MITS Prompt Template: In-Hospital Mortality Prediction System You are a critical care physician predicting outcomes from ICU monitoring data including laboratory measurements and radiology notes.

User TASK: Analyze timestamped observations from an ICU stay and predict whether the patient will survive their hospital admission.

Prompt Template (continued) DATA FORMAT: Each observation is a pair: <time> hours_since_start </time> <channel> channel_name </channel> IMPORTANT: - For each observation, the value/content was measured and recorded, but is not provided here. - When making the prediction, use the information conveyed by the timings and channel types themselves — what types of laboratory measurements were ordered and what notes were recorded, when, in what sequence, and how frequently — and what these patterns might signal about the patient’s clinical trajectory and mortality risk. OUTPUT FORMAT (FOLLOW EXACTLY): Provide your answer in <answer> </answer> tags. Write ONLY a single letter:

A or B

(A) SURVIVAL (B) MORTALITY Do NOT include any explanatory text outside the tags.

Start directly with <answer>.

Example 1 (SURVIVAL): <answer> A </answer> Example 2 (MORTALITY): <answer> B </answer> [OBSERVATIONS] ... [/OBSERVATIONS] Based on the observations above, predict whether this patient will experience in-hospital mortality. Follow the output format exactly.

Start directly with <answer>.

Value-Redacted MITS Prompt Template: Length-of-Stay Prediction System You are a critical care physician predicting outcomes from ICU monitoring data including laboratory measurements and radiology notes.

User TASK: Given timestamped observations from the first 24-hour window of an ICU stay, predict whether the patient will ultimately have a short ICU stay (<96 hours) and survive. DATA FORMAT: Each observation is a pair: <time> hours_since_start </time> <channel> channel_name </channel> CLASS DEFINITION: - SHORT_STAY: ICU stay < 96 hours and survived. - LONG_STAY: ICU stay >= 96 hours or death. IMPORTANT: - For each observation, the value/content was measured and recorded, but is not provided here. - When making the prediction, use the information conveyed by the timings and channel types themselves - what types of laboratory measurements were ordered and what notes were recorded, when, in what sequence, and how frequently - and what these patterns might signal about whether the patient’s ICU course is likely to stabilize quickly (short stay and survive) versus become prolonged or deteriorate (ICU stay >= 96 hours or death).

Prompt Template (continued) OUTPUT FORMAT (FOLLOW EXACTLY): Provide your answer in <answer> </answer> tags. Write ONLY a single letter:

A or B

(A) LONG_STAY (B) SHORT_STAY Do NOT include any explanatory text outside the tags.

Start directly with <answer>.

A.3

Start directly with <answer>.

Prompt Templates for Value Pending Evaluation with Show Presence

Value Pending Evaluation with Show Presence Prompt Template: In-Hospital Mortality Prediction System You are a critical care physician predicting outcomes from ICU monitoring data including laboratory measurements and radiology notes.

User TASK: Analyze timestamped observations from an ICU stay and predict whether the patient will survive their hospital admission. DATA FORMAT: Each observation is one of the following: 1. A triplet: <time> hours_since_start </time> <channel> channel_name </channel> <value> value </value> 2. A pair: <time> hours_since_start </time> <channel> channel_name </channel> IMPORTANT: - For some observations, the measurement was made but the value/content is not available at the time of prediction.

These observations are shown as time-channel pairs.

- When making the prediction, use both the available values/content and the information conveyed by the timings and channel types themselves — what types of laboratory measurements were ordered and what notes were recorded, when, in what sequence, and how frequently — and what these patterns might signal about the patient’s clinical trajectory and mortality risk. OUTPUT FORMAT (FOLLOW EXACTLY): Provide your answer in <answer> </answer> tags. Write ONLY a single letter:

A or B

(A) SURVIVAL (B) MORTALITY Do NOT include any explanatory text outside the tags. Example 1 (SURVIVAL): <answer> A </answer>

Start directly with <answer>.

Prompt Template (continued) Example 2 (MORTALITY): <answer> B </answer> [OBSERVATIONS] ... [/OBSERVATIONS] Based on the observations above, predict whether this patient will experience in-hospital mortality. Follow the output format exactly.

Start directly with <answer>.

Value Pending with Show Presence Evaluation Prompt Template: Length-of-Stay Prediction System You are a critical care physician predicting outcomes from ICU monitoring data including laboratory measurements and radiology notes.

A triplet:

<time> hours_since_start </time> <channel> channel_name </channel> <value> value </value> 2.

A pair:

<time> hours_since_start </time> <channel> channel_name </channel> CLASS DEFINITION: - SHORT_STAY: ICU stay < 96 hours and survived. - LONG_STAY: ICU stay >= 96 hours or death. IMPORTANT: - For some observations, the measurement was made but the value/content is not available at the time of prediction.

These observations are shown as time-channel pairs.

- When making the prediction, use both the available values/content and the information conveyed by the timings and channel types themselves - what types of laboratory measurements were ordered and what notes were recorded, when, in what sequence, and how frequently - and what these patterns might signal about whether the patient’s ICU course is likely to stabilize quickly (short stay and survive) versus become prolonged or deteriorate (ICU stay >= 96 hours or death). OUTPUT FORMAT (FOLLOW EXACTLY): Provide your answer in <answer> </answer> tags. Write ONLY a single letter:

A or B

(A) LONG_STAY (B) SHORT_STAY Do NOT include any explanatory text outside the tags. Example 1 (LONG_STAY): <answer> A </answer> Example 2 (SHORT_STAY): <answer> B </answer> [OBSERVATIONS] ...

Start directly with <answer>.

Prompt Template (continued) [/OBSERVATIONS] Based on the observations above, predict whether this patient will have a short ICU stay and survive. Follow the output format exactly.

Dataset Details

B.1

EHR Database Overview

Start directly with <answer>.

MIMIC-IV. The Medical Information Mart for Intensive Care IV (MIMIC-IV) [5, 6, 63] is a publicly available EHR dataset sourced from the Beth Israel Deaconess Medical Center (BIDMC) in Boston, MA, covering hospital and ICU admissions from 2008 to 2022. MIMIC-IV is organized into three modules. The hosp module contains hospital-wide data, including laboratory measurements, microbiology cultures, medication orders, billing, and patient tracking. The icu module contains ICU bedside data, including charted observations, infusions, and patient outputs, recorded with both a clinically relevant timestamp (charttime) and a database availability timestamp (storetime). The note module contains deidentified free-text clinical notes, including discharge summaries and radiology reports authored by physicians. Data were deidentified in accordance with the HIPAA Safe Harbor provision and are available via PhysioNet. eICU. The eICU Collaborative Research Database (eICU-CRD) v2.0 [7, 8] is a publicly available multi-center critical care database built from data collected through the Philips eICU Program, a telehealth ICU monitoring system in which remote care teams monitor patients at participating hospitals via the eCareManager platform. The database spans admissions from 2014 to 2015 across hundreds of ICU units at hospitals throughout the United States. Available data include vital signs, laboratory measurements, medications, APACHE severity scores, admission diagnoses, and care plan documentation. The care plan is documented using structured multiple-choice lists covering care provider types, prognosis, treatment status, goals of care, and end-of-life planning. Unlike the rich physician-authored free-text notes in MIMIC-IV, eICU care plan entries are comparatively templated and less linguistically varied. The database is deidentified per HIPAA Safe Harbor and available via PhysioNet. B.2

Dataset Extraction

We describe the extraction pipeline for each database separately, covering cohort definition, channel selection, filtering criteria, and label construction. Both databases use a 24-hour observation window from ICU admission and a 70%/15%/15% random train/validation/test split at the ICU stay level. B.2.1

MIMIC-IV

Numerical Channels. We extract 21 laboratory tests from the labevents table in the MIMIC-IV hosp module: Glucose, Sodium, Chloride, Creatinine, Urea Nitrogen, Bicarbonate, Anion Gap, Hemoglobin, Hematocrit, Magnesium, Platelet Count, Phosphate, White Blood Cells, Calcium (Total), MCH, Red Blood Cells, MCHC, MCV, RDW, Neutrophils, and Vancomycin. Each lab measurement is timestamped by its charttime, which records the clinically relevant time of specimen collection. The storetime, which records when the result became available in the database, is also retained for the value pending evaluation in Section 5.4. Text Channel. We extract radiology notes from MIMIC-IV-Note [63], which contains deidentified free-text radiology reports authored by physicians. Each note carries a charttime and a storetime. Discharge summaries are excluded as they may contain explicit outcome information. Radiology reports do not have this concern. Cohort and Filtering. For both tasks, we restrict observations to the first 24 hours of the ICU stay. We require each stay to have at least 64 lab measurements and between 2 and 6 notes within this window, and retain only stays satisfying both modality requirements. The lower bound on note count 20

ensures that notes form a temporal series rather than an isolated observation, while the upper bound removes ICU stays with an anomalously high number of notes to prevent context length explosion. IHM Task. The cohort consists of ICU stays with a length of stay (LOS) of at least 24 hours. The label is in-hospital mortality, derived from the hospital_expire_flag field in the admissions table. LOS Task. The cohort consists of ICU stays with an LOS of at least 48 hours. Following Han et al. [10], the binary label is short stay, defined as an ICU stay shorter than 96 hours with the patient surviving, and 0 otherwise. B.2.2

eICU

Numerical Channels. We extract 24 laboratory tests from the lab table: glucose, potassium, sodium, chloride, creatinine, BUN, bicarbonate, anion gap, Hgb, Hct, magnesium, platelets, phosphate, WBC, calcium, MCH, RBC, MCHC, MCV, RDW, neutrophils, and Vancomycin (random, trough, and peak). These correspond to the same clinical measurements selected for MIMIC-IV. Timestamps are derived from labresultoffset, which records minutes elapsed since admission and is converted to hours. Text Channel. We extract care plan notes from the note table, which stores clinical documentation entered through the eCareManager telehealth platform. Each row in the table represents a single form field selection, with notepath encoding a full hierarchy of section, field label, and field value (e.g., notes/Progress Notes/Interventions/Major/Sepsis - evaluation and management), and notevalue storing the authoritative leaf value. We define a note event as all rows sharing the same patient stay, timestamp (noteoffset), and note type (e.g., Brief Progress, Admission, Intubation), and exclude CPR-type notes, as their presence may constitute label leakage for the mortality prediction task. Because the eCareManager system interleaves clinical documentation with UI control fields (e.g., print settings, view mode toggles, sign-off confirmations), many rows in the table are structural artifacts rather than clinical observations. We filter out these noise rows, specifically those whose section corresponds to system UI paths (e.g., View and Save), whose field label corresponds to scaffolding controls (e.g., View Options, Save Options, Sign As), or whose field value is a generic modal selection (e.g., System View, Performed, Not Significant). For the remaining informative rows, we format each field differently depending on its section. In the Interventions and Assessment and Plan sections, the field value itself is the full clinical problem or intervention name (e.g., Respiratory failure - evaluation and management), so we emit only the value. In all other sections, the field label names a clinical attribute and the value records the finding, so we emit a field_label: field_value token (e.g., Smoking Status: denies smoking). All informative tokens from a note event are then deduplicated and concatenated into a single note string prefixed with its note type. For example, a Brief Progress note event might produce: [Brief Progress] Communication with other healthcare providers and/or family Respiratory failure - evaluation and management

and an Admission note event might produce: [Admission] Smoking Status: denies smoking Ethanol Use: rare

Note events whose reconstructed text is empty after noise removal are discarded. Cohort and Filtering. We restrict observations to the first 24 hours of the ICU stay. We require each stay to have at least 16 lab measurements and between 3 and 10 care plan note events within this window, and retain only stays satisfying both modality requirements. IHM Task. The cohort consists of ICU stays with a LOS of at least 24 hours. The binary label is in-hospital mortality, defined as hospitaldischargestatus == "Expired". LOS Task. The cohort consists of ICU stays with a LOS of at least 48 hours. The binary label is short stay, defined as an ICU stay shorter than 96 hours with the patient surviving at unit discharge (unitdischargestatus == "Alive"). 21

Median Mean

Count

2000

eICU-IHM

MIMIC-IV-LOS

MIMIC-IV-IHM 2000

3000

0 0

5000 10000

Note length (chars)

2000

1000

0 0

5000 10000

Note length (chars)

eICU-LOS

0 0

500

0 0

1000

Note length (chars)

500

1000

Note length (chars)

Figure 4: Distribution of note character lengths across all four datasets. Dashed red and solid orange vertical lines indicate the median and mean, respectively. B.3

Dataset Statistics and Analyses Table 3: Dataset statistics. # MITS Samples Positive Rate (%) # Labs Per MITS # Notes Per MITS

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

3,404 25.79 81.9±17.7 3.12±1.17

2,896 27.83 82.2±17.8 3.15±1.17

3,667 22.42 36.4±18.8 3.86±1.23

2,712 37.13 36.8±19.0 3.87±1.24

We present the basic statistics of the 4 datasets in Table 3, including the number of MITS samples, positive rate, number of lab measurements per MITS, and number of clinical notes per MITS. We further report the number of lab measurements per sample for each channel in Table 4. We also report the character count per note for each dataset to highlight differences in note properties between MIMIC-IV and eICU. The note length distributions are shown in Fig. 4. Note lengths in both MIMICIV and eICU exhibit a long-tailed distribution. Notes in MIMIC-IV-IHM and MIMIC-IV-LOS can reach up to 5,000 characters, whereas those in eICU-IHM and eICU-LOS reach only about 1,000 characters, reflecting the richer free-text content of MIMIC-IV compared to eICU.

Model and Hyper-Parameter Configurations

We present the model and hyper-parameter configurations of each method below. Hyper-parameters are selected based on validation AU-ROC. MILM. We fine-tune with QLoRA [69] (rank r = 16, α = 16, dropout 0.05) on the query, key, value, and output projection matrices. We use AdamW [73] with a learning rate of 10−4 , cosine schedule, warmup ratio 0.05, and effective batch size 8. Stage 1 (value-redaction fine-tuning) trains for 8 epochs across all datasets. The number of Stage 2 epochs is dataset-dependent, as shown below:

# Stage 2 Epochs

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

GRU-D. GRU-D [14] extends the standard Gated Recurrent Unit with two trainable temporal decay mechanisms for irregularly sampled time series with missing values. Specifically, an input decay γx decays each unobserved feature toward its empirical mean as a function of the elapsed time since its last observation, while a hidden state decay γh similarly decays the GRU hidden state. The binary missingness mask is additionally fed into the GRU gates as an informative signal. We use a hidden size of 32 followed by an MLP with a single hidden layer of size 64. Dropout and recurrent dropout are both set to 0.1. The model is trained with the Adam optimizer [74] at a learning rate of 10−3 , with L2 weight decay of 10−4 on the MLP hidden layer weights and gradient clipping at norm 1.0. The batch size is 32. The number of training epochs is dataset-dependent, as shown below:

Table 4: Lab measurement counts per sample per channel. For eICU, Vancomycin (∗ ) aggregates the random, trough, and peak sub-types. MIMIC-IV Channel Anion Gap Bicarbonate Blood Urea Nitrogen (BUN) Calcium, Total Chloride Creatinine Glucose Hematocrit (Hct) Hemoglobin (Hgb) Magnesium MCH MCHC MCV Neutrophils (-polys) Phosphate Platelet Count Potassium Red Blood Cells (RBC) RDW Sodium Vancomycin∗ White Blood Cells (WBC)

# Epochs

eICU

IHM

LOS

IHM

LOS

4.18±1.39 4.21±1.38 4.22±1.35 3.78±1.74 4.33±1.36 4.22±1.36 3.43±4.35 5.28±1.70 1.91±2.59 3.98±1.55 4.73±1.49 4.74±1.49 4.74±1.49 0.77±0.89 3.80±1.72 4.96±1.70 — 4.74±1.49 4.73±1.49 4.27±1.47 0.13±0.38 4.74±1.49

4.17±1.39 4.20±1.37 4.21±1.35 3.76±1.73 4.32±1.36 4.21±1.35 3.57±4.46 5.31±1.74 2.00±2.68 3.97±1.54 4.75±1.51 4.75±1.51 4.75±1.50 0.79±0.90 3.78±1.72 4.98±1.73 — 4.75±1.50 4.75±1.51 4.27±1.47 0.13±0.38 4.76±1.51

1.73±1.37 1.99±1.28 1.98±1.27 1.93±1.27 2.01±1.29 1.98±1.26 2.03±1.36 1.90±1.30 1.94±1.35 1.29±1.17 1.64±1.01 1.65±1.01 1.65±1.01 0.69±0.83 0.97±1.13 1.67±1.03 2.26±1.49 1.65±1.00 1.50±1.04 2.17±1.52 0.07±0.28 1.65±1.00

1.75±1.39 2.02±1.30 2.00±1.27 1.95±1.28 2.04±1.30 2.00±1.26 2.06±1.39 1.93±1.30 1.95±1.32 1.33±1.19 1.66±1.00 1.66±1.00 1.66±1.01 0.70±0.85 1.00±1.14 1.68±1.02 2.28±1.49 1.67±1.00 1.53±1.05 2.19±1.51 0.07±0.28 1.67±0.99

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

200

150

mTAND. mTAND [16] models irregularly sampled time series by learning a continuous-time embedding of observation timestamps and using a multi-head attention mechanism to interpolate observed values onto a fixed set of reference time points, producing a fixed-length representation without requiring imputation or temporal discretization. For classification, we use the encoder-only variant (mTAND-Enc), where the mTAND module maps irregular observations to K reference-point embeddings that are then aggregated by a GRU, followed by a two-layer residual projection head. We use an attention embedding dimension of 64, a time embedding dimension of 32, 8 attention heads, K = 24 reference points (matching the 24-hour observation window), and a GRU hidden size of 128. Dropout is set to 0.3. The model is trained with the Adam optimizer with gradient clipping at norm 1.0 and a batch size of 32 for 250 epochs across all datasets. The learning rate is dataset-dependent, as shown below: MIMIC-IV-IHM Learning Rate

2 × 10

−4

MIMIC-IV-LOS 1 × 10

−4

eICU-IHM

eICU-LOS

−4

1 × 10−4

2 × 10

UTDE. UTDE [2] is a multimodal framework for irregular EHR time series that unifies two complementary embedding strategies via a learned gate: an imputation-based embedding (forwardfill followed by a 1-D convolution) and an mTAND-based interpolation embedding, yielding a unified time series representation. Clinical notes are cast as multivariate irregular time series and encoded through a separate mTAND module. The two modalities are then fused through J identical layers of interleaved self-attention and cross-attention. We use the self_cross fusion variant with Adam optimizer, a learning rate of 2 × 10−4 , batch size 32, gradient clipping at 1.0, dropout 0.3, dtime = 32 time embedding dimensions, 8 attention heads, 2 self-attention layers, and 2 cross-attention layers. Dataset-specific hyper-parameters are summarized below:

Hyper-Parameter Embedding Dim. (d) Max. # Notes Used # Epochs

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

64 5 150

32 9 250

64 9 250

FuseMoE. FuseMoE [10] shares the per-modality encoding backbone with UTDE, using mTAND for irregularly sampled time series and a projection layer for clinical note embeddings, but replaces the cross-attention fusion with a sparse Mixture-of-Experts (MoE) fusion layer. In each layer, a joint router receives the concatenated embeddings of all modalities and dispatches each sample to the top-K of E expert MLPs. We use Adam with a learning rate of 2 × 10−4 , batch size 32, gradient clipping at 1.0, dropout 0.3, d = 64 embedding dimensions, dtime = 32 time embedding dimensions, 8 attention heads, 2 self-attention layers, 2 cross-encoder layers, 3 experts, top-K=2 routing, a joint router, MoE hidden size 512, and a load-balance auxiliary loss with coefficient 0.01. Dataset-specific hyper-parameters are summarized below: Hyper-Parameter

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

Max. # Notes Used # Epochs

5 100

9 150

VITAL-stats & VITAL-LLM. VITAL [11] is a variable-aware framework that differentiates between two types of clinical variables: frequently measured vital signs processed through LLM reprogramming, and infrequently measured lab tests embedded via per-channel summary statistics or a learnable [Not measured] token. Since the numerical channels of our datasets contain only laboratory measurements, we evaluate both branches separately: VITAL-stats applies the lab branch, representing each channel by a four-dimensional summary vector (mean, median, min, max) followed by feature mixing. VITAL-LLM applies the vital sign reprogramming branch, which requires regulargrid input and is therefore applied to an hourly-discretized version of the lab channels. For fairness of comparison, VITAL-LLM uses the same backbone LLM as MILM (Qwen3-4B-Instruct-2507). Both variants are trained with Adam, learning rate 1 × 10−3 , dropout 0.1, and gradient clipping at 1.0. VITAL-stats uses batch size 128 and dff = 32 uniformly. VITAL-LLM uses batch size 8 with 16 gradient accumulation steps (effective batch size 128), 1,000 text prototypes, 6 reprogramming attention heads, and bfloat16 precision. Dataset-specific hyper-parameters are summarized in Tables 5 and 6. Table 5: Dataset-specific hyper-parameters for VITAL-stats. Hyper-Parameter # Epochs

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

100

Table 6: Dataset-specific hyper-parameters for VITAL-LLM. Hyper-Parameter dff

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

ISTS-PLM. ISTS-PLM [12] adopts a series-based representation of irregularly sampled time series, processing each variable’s observations as a univariate sequence through two successive pretrained language model (PLM) stages. The first stage is a time-aware PLM that replaces the absolute positional embeddings with learnable continuous-time embeddings, enabling the model to reason about intra-series temporal dynamics. The second stage is a variable-aware PLM that replaces positional embeddings with learnable variable embeddings, capturing inter-series correlations across asynchronous channels. We use GPT-2 [75] as the time-aware PLM and BERT [76] as the variableaware PLM, following the original paper. Applying Qwen3 is not feasible because Qwen3 uses rotary position embeddings (RoPE) [77], which are fused into the attention computation and cannot be straightforwardly replaced by continuous-time embeddings. We use the first 6 layers of each PLM 24

Table 7: Total wall-clock training time (minutes) for baselines, run on an NVIDIA RTX 3090. MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

15.00 17.12 3.82 6.05 6.20 5.03 0.50 0.78 428.78 698.17 8.67 25.70

12.88 14.70 3.22 5.33 5.55 5.13 0.85 1.40 372.51 359.30 23.06 21.63

8.02 7.22 2.98 5.90 11.87 11.86 0.35 0.72 352.32 465.97 8.42 18.97

4.43 5.43 2.23 4.25 11.98 11.82 0.50 1.00 121.15 167.53 9.62 18.60

GRU-D + Note

mTAND + Note

UTDE FuseMoE VITAL-stats + Note

VITAL-LLM + Note

ISTS-PLM + Note

Table 8: Total wall-clock training time (minutes) for MILM, run on an NVIDIA H200. MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

254.38 162.00

207.07 168.83

306.73 311.45

206.18 212.18

MILM (Stage 1) MILM (Stage 2)

(dmodel = 768), follow the original paper to freeze all PLM parameters except layer normalization, use dropout 0.1, Adam optimizer, batch size 6, and gradient clipping at 1.0 for 40 epochs. Dataset-specific hyper-parameters are summarized below:

Hyper-Parameter Learning rate

MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

2 × 10−3

1 × 10−3

5 × 10−4

Timestamp-to-Text Fusion (TTF) Module. The Timestamp-to-Text Fusion (TTF) module, introduced in Time-IMM [1], addresses the challenge of integrating asynchronously timestamped clinical notes with numerical time series by producing a temporally-aware and fixed-size text representation. We adopt the T2V-XAttn variant. Each note embedding is concatenated with a Time2Vec encoding [78] of its timestamp, projected to a common dimension, and then attended to by a single learned query vector via single-head cross-attention, yielding a pooled text representation that is conditioned on both the semantic content and the temporal position of each note. This representation is concatenated to the representation of numerical observations before the classification head. We apply TTF to augment GRU-D, mTAND, VITAL-stats, VITAL-LLM, and ISTS-PLM, producing the corresponding “+ Note” baseline variants reported in the main table. All TTF encoders use 768-dimensional BioBERT [67] note embeddings. We use the 6 most recent notes per patient on MIMIC-IV and the 10 most recent notes per patient on eICU.

Compute Resources

All baseline experiments are run on an NVIDIA GeForce RTX 3090 (24 GB). MILM requires more GPU memory due to LLM fine-tuning, and is therefore run on an NVIDIA H200 GPU (96 GB) on a GH200 node. This overhead reflects the per-layer LoRA fine-tuning that genuinely adapts the LLM’s weights to domain-specific temporal knowledge and gives MILM the capacity to predict under either setting: from sampling patterns alone or from both patterns and values. In contrast, the LLM-based irregular time series baselines keep the LLM weights frozen by design and train surrounding modules. We report total wall-clock training time per run in Tables 7 and 8, and peak GPU memory usage in Table 9. 25

Table 9: Peak GPU memory usage (GB). MIMIC-IV-IHM

MIMIC-IV-LOS

eICU-IHM

eICU-LOS

0.03 0.11 0.38 0.34 0.59 0.85 0.03 0.12 11.16 11.27 4.87 4.89

0.03 0.11 0.58 0.63 0.62 0.85 0.03 0.12 11.18 11.29 2.86 3.75

0.02 0.10 0.20 0.31 0.95 1.17 0.03 0.12 11.22 11.35 3.13 3.31

0.02 0.10 0.12 0.23 0.79 1.19 0.03 0.12 11.24 11.37 2.81 2.46

48.96 64.20

44.99 71.20

26.48 39.93

34.68 52.43

GRU-D + Note

mTAND + Note

UTDE FuseMoE VITAL-stats + Note

VITAL-LLM + Note

ISTS-PLM + Note

MILM (Stage 1) MILM (Stage 2)

Table 10: Average number (left) and rate (right) of pending observations per ICU stay. Lab Note

MIMIC-IV-IHM

MIMIC-IV-LOS

2.74 0.43

2.71 0.45

MIMIC-IV-IHM

MIMIC-IV-LOS

3.33% 13.53%

3.30% 14.02%

Lab Note

Table 11: Average and median storetime − charttime delay (hours) per channel across all ICU stays. MIMIC-IV-IHM

MIMIC-IV-LOS

Channel

Avg.

Median

Avg.

Median

Anion Gap Bicarbonate Calcium, Total Chloride Creatinine Glucose Hematocrit Hemoglobin MCH MCHC MCV Magnesium Neutrophils Phosphate Platelet Count RDW Red Blood Cells Sodium Urea Nitrogen Vancomycin White Blood Cells Radiology Note

1.26 1.24 1.20 1.22 1.22 0.09 0.61 0.10 0.62 0.62 0.62 1.21 1.27 1.21 0.66 0.62 0.62 1.23 1.19 2.31 0.64 5.73

1.13 1.12 1.08 1.10 1.08 0.05 0.52 0.07 0.53 0.53 0.53 1.08 0.87 1.10 0.55 0.53 0.53 1.10 1.08 1.58 0.53 3.17

1.25 1.23 1.20 1.22 1.21 0.09 0.61 0.10 0.61 0.61 0.61 1.20 1.26 1.21 0.66 0.61 0.61 1.22 1.19 2.31 0.64 5.74

1.12 1.10 1.08 1.08 1.08 0.05 0.52 0.07 0.52 0.52 0.52 1.08 0.87 1.08 0.53 0.52 0.52 1.10 1.08 1.59 0.53 3.23

Value Pending Evaluation Details

E.1

Dataset Statistics and Analyses

For the value pending evaluation in Section 5.4, an observation’s value is considered pending if its storetime, the timestamp at which the result was available in the database, exceeds the 24-hour prediction time. When storetime is absent, we conservatively fall back to charttime, treating the observation as available on time. The same rule is applied to radiology notes. Table 10 reports the average number and rate of pending observations per ICU stay. On average, each stay has approximately 2.74 (IHM) and 2.71 (LOS) pending lab measurements, representing about 26

MIMIC-IV-LOS

ICU stays (%)

MIMIC-IV-IHM 40 20 0 0

Pending event count

40 30 20 10 0 0

MIMIC-IV-IHM

Pending event count

MIMIC-IV-LOS

0 0

Pending event rate (%)

0 0

Pending event rate (%)

Figure 5: Distribution of pending event count (left two) and pending event rate (right two) per ICU stay at the 24-hour prediction time.

MIMIC-IV-IHM Note

MIMIC-IV-IHM Lab

80 60 40 20 0 0

% within group

80 Pending False 60 True 40 20 0 0 10

10 5 20

Charttime (hours)

0 0

Charttime (hours)

MIMIC-IV-LOS Lab

MIMIC-IV-LOS Note

10 5 10

Charttime (hours)

0 0

Charttime (hours)

Figure 6: Distribution of charttime for value-pending and non-pending observations.

3.3% of all lab observations. Notes are pending at a higher rate: 13.5% (IHM) and 14.0% (LOS). Table 11 shows the average and median storetime − charttime delay per channel. Lab delays cluster around 0.6 to 1.3 hours depending on the panel, while note delays are markedly longer (median ≈ 3.2 hours), consistent with the higher note pending rate. Figure 5 shows how these counts and rates vary across individual ICU stays. Both are right-skewed: the majority of stays have few or no pending observations, but a nontrivial tail of stays can have 10 to 30% of their observations pending. Figure 6 shows p(charttime | value pending) and p(charttime | value not pending) separately for labs and notes. The pending distributions are visibly right-shifted relative to the non-pending distributions, confirming that value-pending observations typically fall near the end of the 24-hour observation window, where recent measurements tend to be most informative for outcome prediction.

E.2

Full Results

We present the full value pending evaluation results in Table 12, extending Table 2 to include baselines with the drop observation countermeasure. Note that it is not straightforward to make the baselines show the presence of value-pending observations without showing the values, as these models cannot process NaN values. Qwen3-4B and MILM are able to adopt the show presence countermeasure due to the flexibility of text-based representations. Value pending evaluation has a larger impact on natively multimodal methods such as UTDE, Qwen34B, and MILM since the note channel has a higher pending rate than lab channels (Table 10). However, the two-stage training of MILM-2S effectively mitigates this challenge. By training the model to predict solely from sampling patterns, Stage 1 develops two complementary capabilities. First, by learning to extract predictive signals without value tokens, MILM-2S makes better use of the remaining observations under drop observation when some note content is missing. Second, by learning to treat time-channel patterns as signals in their own right, MILM-2S makes better use of the additional sampling information conveyed under show presence. Both capabilities translate directly to stronger performance than MILM-Direct under value pending evaluation. Despite operating under a more significant train-test discrepancy than unimodal baselines, MILM-2S with show presence and MILM-2S with drop observation remain the best-performing methods on MIMIC-IV-IHM and MIMIC-IV-LOS, respectively. 27

Table 12: Full value pending evaluation results. Best results are boldfaced and second-best results are underlined. MIMIC-IV-IHM Countermeasure

Method

AU-ROC

Drop Observation

GRU-D

Qwen3-4B MILM-Direct MILM-2S

75.07±1.03 73.60±1.79 76.56±1.14 77.46±1.55 74.22±2.93 75.49±1.47 76.44±0.84 77.10±0.63 67.96±1.61 67.98±2.77 79.95±0.59 79.52±0.53 68.33±0.00 79.04±0.91 79.49±0.76

52.61±1.07 50.21±4.12 54.63±1.70 56.54±1.05 52.54±3.02 55.00±0.92 53.14±2.13 52.86±1.99 40.66±2.00 40.42±3.07 59.31±0.77 58.22±0.64 44.06±0.00 59.23±1.63 60.02±1.57

69.14±1.20 70.12±2.56 71.19±0.83 73.02±1.02 73.89±3.17 74.56±0.46 73.70±0.95 74.24±1.25 68.77±1.59 68.80±1.26 73.93±0.73 73.63±0.51 67.39±0.00 76.86±0.52 77.64±0.33

46.37±1.43 47.51±2.46 49.83±1.41 52.86±2.83 52.80±3.16 55.10±1.75 52.58±2.72 54.47±4.03 47.70±3.36 48.37±1.83 54.27±1.59 52.46±0.92 44.37±0.00 56.44±1.61 57.17±1.78

Qwen3-4B MILM-Direct MILM-2S

67.40±0.00 79.21±0.65 80.34±0.64

39.80±0.00 58.14±1.37 61.23±1.62

66.54±0.00 76.30±0.64 77.08±0.38

43.50±0.00 55.01±2.07 56.46±1.50

+ Note

mTAND + Note

UTDE FuseMoE VITAL-stats + Note

VITAL-LLM + Note

ISTS-PLM + Note

Show Presence

MIMIC-IV-LOS

MedGemma vs. Qwen

Due to the domain of interest, it is natural for us to consider using large language models pretrained for the biomedical and healthcare domains, such as MedGemma [79]. MedGemma is a collection of medical vision-language foundation models built on the Gemma 3 architecture, designed to interpret and reason about medical images and text across modalities including radiology, dermatology, histopathology, and ophthalmology. In our preliminary experiments, we compare the zero-shot performance of the 4B instruction-tuned variants of both MedGemma and Qwen3. We report the results below: MIMIC-IV-IHM Model MedGemma-4B Qwen3-4B

MIMIC-IV-LOS

Unparseable (%)

AU-ROC

Unparseable (%)

AU-ROC

62.70 0.00

57.81 69.82

36.27 48.36

89.66 0.00

52.66 67.98

33.23 44.82

eICU-IHM Model MedGemma-4B Qwen3-4B

eICU-LOS

Unparseable (%)

AU-ROC

Unparseable (%)

AU-ROC

65.70 0.00

52.47 64.53

22.40 29.43

80.64 0.00

53.38 55.71

43.33 45.58

A large portion of the responses generated by MedGemma-4B are unparseable. For example, instead of generating <answer> A </answer>, it may generate A, violating the output format specified in the prompt. In contrast, the responses from Qwen3-4B are all parseable, demonstrating its stronger instruction-following capability and stability. We evaluate AU-ROC and AP by assigning a neutral score of 0.5 to unparseable responses to maintain a consistent evaluation cohort. Despite being pretrained on biomedical and healthcare domain data, MedGemma-4B underperforms Qwen3-4B on all 8 metric-dataset combinations. We suspect this is because MedGemma’s domain-specific training is primarily oriented toward medical image understanding, with text supervision derived from relatively small medical QA datasets, whereas Qwen3 is pretrained on a substantially larger and more diverse corpus of 36 trillion tokens across 119 languages, which may better support the text representation inputs used in our task. Given the stronger instruction-following capability and better overall performance of Qwen3-4B, we select it as the base LLM for this work. 28

Limitations

MILM provides a framework for using LLMs to make predictions from MITS with informative sampling. Although the method can be extended to image channels when the LLM includes a vision encoder and image content is serialized as image tokens, we do not consider image modalities in this work due to the substantially greater computational cost compared with processing numerical and text channels using text-only LLMs. We leave the extension to additional modalities for future work.

Broader Impacts

MILM is a research framework for applying LLMs to multimodal irregular time series classification in clinical settings. On the positive side, better usage of informative sampling patterns in ICU data could support the development of improved clinical prediction systems. On the negative side, the models in this work are trained and evaluated on US-based EHR datasets (MIMIC-IV and eICU), and may not generalize equally across hospital systems, countries, or patient populations. We emphasize that MILM is not validated for direct clinical deployment, and any downstream application would require rigorous prospective validation.

Asset Licenses

Datasets. MIMIC-IV [5, 6], MIMIC-IV-Note [63], and the eICU Collaborative Research Database [7, 8] are available via PhysioNet under the PhysioNet Credentialed Health Data License 1.5.0. All datasets are used in compliance with their respective data use agreements. Models. Qwen3-4B-Instruct-2507 [68], BioBERT [67], and BERT [76] are released under the Apache License 2.0. GPT-2 [75] is released under the Modified MIT License. MedGemma [79] is released under the Health AI Developer Foundations Terms of Use. All pretrained models are accessed via the Hugging Face Transformers library [80], which is also released under the Apache License 2.0.

Record · ID 180666 · SHA-256 2df7cc07916fcdd0
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.