ITGPT: Generative Pretraining on Irregular Timeseries Antoine Honoré⋆ , Ming Xiao⋆
arXiv:2605.16069v1 [cs.LG] 15 May 2026
⋆
Division of Information Science and Engineering, KTH Royal Institute of Technology, Stockholm, Sweden
Abstract—Timeseries regression models often struggle to leverage large volumes of labeled multimodal data, particularly when the data are irregularly sampled or contain missing values. This is common in domains like healthcare and predictive maintenance, where data are collected from unreliable sources, and labeling requires expert knowledge or costly equipments. Transformerbased large language models have proven effective on structured data such as text through self-supervised learning (SSL) and generative pretraining (GPT) frameworks. However, such models lack the flexibility to efficiently process irregularly sampled multimodal timeseries data. In this paper, we introduce ITGPT, an attention-based architecture designed for handling multimodal, irregularly sampled timeseries by allowing training with both SSL losses and GPT-like objectives. We evaluate its performance on a healthcare task with the TIHM dataset, and a predictive maintenance task with the CompX dataset. Our results demonstrate that ITGPT achieves state-of-the-art performance without requiring resampling, feature fusion or explicit data imputation. Furthermore, when labels are scarce, ITGPT effectively leverages unlabeled data through SSL and GPT training, outperforming the purely supervised approach. This represents an important step towards efficiently using large and unstructured timeseries datasets for practical inference tasks. The code is available on GitHub: https://github.com/antoinehonore/itgpt. Index Terms—Multimodal data, Irregular Sampling, Attention Mechanism
I. I NTRODUCTION Multimodality arises when multiple sources of information are used to characterize a system [1]. In many practical cases, sources might be de-synchronized, with different sampling frequencies and missing data. For instance, in predictive maintenance (PdM) the quality of a mechanical piece must often be established from sensors with various sampling frequencies, e.g. noise and vibration sensors [2]. Similarly in the medical field, a patient’s health status must be established using information from fast or slow sources, for instance electrocardiograms (up 500 samples per second) combined with blood gas analysis (one sample per day) [3]. Additionally, in these contexts, labels about the health status of a patient or a mechanical piece might require expert knowledge and/or expensive sensors which limits their availability for supervised learning schemes. Current models handle irregularly sampled multimodal data with computationally expensive or overly simplistic resampling, imputation and feature fusion This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre. Correspondence: [email protected].
approaches. These approaches lack the ability to predict future input data and thus, only allow supervised learning frameworks. Current approach for processing multimodal data often consider use cases related to image, text or speech processing. Attention-based models such as Transformers are established as strong models that allow training with few labels, using self-supervised learning (SSL) or generative pretraining (GPT) frameworks. These frameworks are key towards prediction models utilizing large amount of unlabeled data in PdM or healthcare contexts [4], [5]. SSL is a machine learning paradigm that leverages unlabeled data to learn representations of input data by generating supervisory signals from the data itself [6]. Unlike traditional supervised learning, SSL does not require manual annotations, making it highly scalable and particularly valuable for domains where labeled data is scarce or expensive to obtain. A common approach involves predicting parts of the data from other parts, e.g. predicting the future of a timeseries from its past. SSL has been key for advancing domains such as computer vision and natural language processing, often serving as a pretraining step before fine-tuning on downstream tasks. The success of SSL reflects its ability to uncover structure and semantics in raw data without external supervision. GPT is a strategy in which models are initially trained in an unsupervised manner on large datasets, in order to learn the distribution of the input data, typically using SSL objectives. This method became prominent with the introduction of models where the transformer architecture was pretrained to predict the next token in a sequence, capturing a wide range of linguistic patterns and world knowledge [7]. After pretraining, the model can be fine-tuned on specific supervised tasks with relatively little additional data, leveraging the rich representations learned during the generative phase. This two-step process has set new benchmarks and forms the backbone of many state-of-the-art natural language processing systems such as ChatGPT, Mixtral and DeepSeek. In the context of irregularly sampled multimodal data, there are currently no model to efficiently leverage unlabeled data for learning in SSL or GPT frameworks. A. Related works The problem of handling irregularly sampled multimodal data was addressed with Kalman filters in the context of linear state space models [8]. These approaches provide efficient closed form inference models, but rely on strong knowledge
of the linear processes and measurement systems. Recurrent neural networks with input space augmentation was shown promising for handling unimodal irregularly sampled signals [9]. This nonetheless might lead to a too large input space dimension. Other RNNs with neural ODEs for modeling continuous time were also shown relevant, but computationally expensive. Other kernel-based Gaussian processes were also investigated, but scale quadratically with the number of training samples [10]. ITNet [11] can be used to produce a multivariate data timeseries with arbitrary time sampling instants. This output time series can then be used as an estimator for some quantity of interest in a supervised learning framework. However, this often requires large quantities of accurately labeled data thus ignoring possibly large datasets of unlabeled data. While this is straight to do on regularly sampled timelines, there is currently no standard method for multimodal and irregularly sampled timeseries data. B. Contributions Our contributions are as follows: 1) We propose a framework called ITGPT which extends on ITNet to allow GPT-like objectives with irregularly sampled multimodal input data. 2) We evaluate our model on two recent datasets for a healthcare and predictive maintenance tasks. 3) We study our model when varying key hyperparameters: the model depth and the mixing layers. 4) We show experimentally that our model benefits from SSL or GPT training frameworks when a few labeled samples are available for training. II. M ETHODS In this section we start by introducing the timeseries regression problem and the multimodal input data in II-A. For completeness we introduce the ITNet model in II-B. The ITGPT architecture is finally described in II-C. We use the notation ∀N ∈ N, [N ] = {1, . . . , N }. A. Problem formulation We introduce the problem of timeseries regression based on multimodal and irregular timeseries. Let M ∈ N the number of observed modalities. Suppose that we have access to N ∈ N independent realizations of the modalities of a system, in addition to a target timeseries. For an observation i ∈ [N ], the data of modality m ∈ [M ] are denoted: (i)
(i) X̃m = [x̃m,1 , . . . , x̃
Importantly, all timestamps across modalities have the same time reference, i.e. identical timestamps encountered in different modalities refer to the same time instant. Let (i) (i) L(i) m ×(dm +1) , Xm = [X̃m , τ (i) (3) m ]∈R denote the matrix composed of the concatenation of the (i) data and the timestamps. Let X (i) = {Xm }M m=1 denote the collection of(i) data from all modalities for observation i. Let Ỹ (i) ∈ RLy ×dc denote the target sequence for obser(i) vation i, with (Ly , dc ) ∈ N2 respectively its length and dimension. The target variable comes with its own timeline (i) (i) (i) τ y ∈ RLy (similar to Eq. (2)). We write Y (i) = [Ỹ (i) , τ y ] ∈ (i) RLy ×(dc +1) . Finally, a dataset with N observations is denoted N D = {(X (i) , Y (i) )}i=1 . For each realization i, the aim is to perform regression of (i) Ỹ (i) at time instants in τ y , using data X (i) . Importantly, the regression must be causal. That is, the target variable at time t must be estimated only using data prior to t. B. ITNet For completeness, this section introduces ITNet similarly to our previous work [11]. Let t ∈ τ y denote a time instant at which the target variable is sampled. Let ∀m ∈ [M ], Xm,t denote the sub-matrix of rows of Xm with the corresponding sampling times strictly prior to t; similarly, τ m,t denote the sub-vector of τ m with time instants strictly prior to t. The regression task consists in estimating the target sequence at all time t ∈ τ y , using all data prior to t across all modalities. ITNet is formulated as follows with all available data up to time t: dc ITNet ({Xm,t }M m=1 ) = W [g1 (t, X1,t ), . . . , gM (t, XM,t )] ∈ R , (4) where ∀m ∈ [M ], gm (., .) are causal cross-attention mechanisms with output embedding dimension do ∈ N, and a linear mixing layer W ∈ Rdc ×M do . 1) Causal cross-attention: In this section, we give a complete description of the modality-specific causal cross-attention mechanisms as proposed in [12], see also Fig. 2. The output of the cross-attention for a timestamp t is specified as a function of t and of the modality data up to t, i.e. gm (t, Xm,t ). In vector form, the m-th causal cross-attention mechanisms is expressed as follows:
gm (t, Xm,t ) =
d ∑ αt,t′ vt′ ∈ R o , (m)
t′ ∈τ
(5)
m,t
(i)
(i) (i) m,Lm
]⊺ ∈ RLm ×dm ,
(1)
where (Lm , dm ) ∈ N2 denote respectively the number of (i) samples and the data dimension, and x̃m,t ∈ Rdm . In addition, we assume that we have access to a sorted sequence of sampling timestamps for each modality and observation: (i)
(i) (i) ⊺ L(i) m . τ (i) m = [tm,1 , . . . , tm,L(i) ] ∈ R m
(2)
where vt′ ∈ Rdo denotes the row of fV (Xm,t ) corresponding to time instant t′ . (m) The coefficients αt,t′ ∈ [0, 1] are calculated with a similarity function, and so that no future information is used in the computation of the output at time t: ⎧ Sim(qt ,kt′ ) ⎪ ⎪ (m) αt,t′ = ⎨ ∑s∈τ m,t Sim(qt ,ks ) ⎪ ⎪ 0 ⎩
if t′ < t, otherwise,
(6)
X
Anchor Timeline Modality 1
Linear
Anchor xN
V Modality 2
V
K V
Anchor data
W
K V
Q Causal cross attention Q Causal cross attention
Modality M Data Timeline
Encoder Anchor Linear
X hat
X
Y hat
Q Causal cross attention
K V
K V
Encoder
Q Causal cross attention Decoder
...
...
Z
Q Causal cross attention
K
Data Timeline
Decoder
Linear
K
Data Timeline
Encoder
ITGPT
Q Causal cross attention
Z
(a) Multilayer ITGPT (b) Single layer ITGPT. The encoder and decoder are based on ITNet. The causal cross attention with prediction heads. and the computation of Q, K and V are further detailed in Fig. 2
Fig. 1: ITGPT architecture description. The encoder and decoders are two ITNet models. X and Z are multimodal data and the following linear layers are specific to each modality. Anchor denotes a timeseries of anchor data with a predefined timeline.
Linear
Modality m
Timeline
Anchor Timeline
PE
PE
+
K
Linear
PE
+
Q Causal cross attention
Timeline
Data
projects scalar timestamps to a cube: [−1, 1]dk . This is done by sampling sines and cosines of different wavelengths. The scalar product following the position encoding in equation (7) leads to timestamp differences projected in a similar way (See appendix). The keys and values are computed from the data and with additive position encoding of the corresponding timestamps t′ ∈ τ m :
Z
V
kt′ = fK (xm,t′ ) = x̃m,t′ W K + p(t′ ) ∈ Rdk ,
Fig. 2: Causal cross-attention in ITNet. We use additive positional encoding for the computation of the key and value tensors from the data timeline. The positional encoding of the anchor timeline is directly used in the computation of the query tensor. PE: position encoding. where the case kt′ ∈ Rdk denotes the row of fK (Xm,t ) corresponding to time instant t′ . Similarly, let qt ∈ Rdk denotes the row of fQ (τ y ) corresponding to time instant t. In vanilla attention, the similarity function is: qTt kt′ Sim(qt , kt′ ) = exp ( √ ) ∈ R+ , dk
(7)
where dk ∈ N is the dimension of the keys and queries, i.e. the output dimension of fQ and fK . a) Queries, keys and values: To compute queries, fQ encodes t in dimension dk (assumed an even number) using the position encoding (PE) function p(t) as described in [12]: T
qt = fQ (t) = p(t) = [. . . , sin (ωi t) , cos (ωi t) , . . . ] ∈ Rdk , (8) − d2i dk k where ∀i ∈ [ 2 ], ωi = Λ and Λ ∈ R+ should be larger than the largest timestamp. This position encoding scheme
(9)
where W K ∈ Rdm ×dk is trainable. Similarly: vt′ = fV (xm,t′ ) = x̃m,t′ W V + p(t′ ) ∈ Rdo ,
(10)
where W V ∈ Rdm ×do is trainable. C. ITGPT ITNet cannot be used in a GPT framework because the output cannot directly be compared to its input. In ITGPT, we chain together pairs of ITNets: ● a first ITNet (the encoder) is used with multimodal data as input and outputs a single multivariate timeseries with a predefined timeline (the ”anchor” data). ● the second ITNet (the decoder) uses the anchor data as input and uses no output mixing layer, i.e. outputs multimodal data. The anchor data are similar to the target sequence of ITNet but serve a difference purpose: in ITGPT they are used to anchor, or synchronize all modalities on the same time sampling instants. In the decoder, the initial data timelines are used for query computation (as opposed to the anchor timeline in the encoder), and the anchor data is used as input to the keys and values computation (as opposed to the input multimodal data). This forms a model with multimodal data both in the input and in the output. An arbitrary number L of
M
where {Em }M m=1 = {Xm }m=1 , and a residual connection is introduced in (12) with Z (0) = 0 ∈ RL×da and φ a componentwise non-linearity, e.g. ReLU. The model is depicted in Fig. 1: with additional prediction and embedding transforms in Fig. 1a and the details of an encoder/decoder pair in Fig. 1b. Equation (13) can be viewed as an ITNet layer without output mixing layer. (0)
III. E XPERIMENTS We describe the datasets used to evaluate our ITGPT architecture in SectionIII-A. Next, in III-B we present the training strategies and the loss functions used to test ITGPT abilities with a few labeled samples and SSL-like objectives. Section III-C describes our architectures with the hyperparameters we fixed and the ones we varied. A. Datasets 1) Technology Integrated Health Management (TIHM): a) Description: We evaluated our model using the TIHM dataset, an open dataset designed for remote healthcare monitoring in dementia [13]. The dataset consists in longitudinal, in-home observation of 56 people living with dementia (PLWD). Data were collected by multiple connected sensors and medical devices. b) Modalities: In-home activity was monitored with passive infrared sensors, door sensors and under-mattress sleep mats, which provided minute level data on sleep states heart rate and respiration. Physiological were collected daily using commercially available devices measuring blood pressure, heart rate, body temperature, body weight, muscle mass and hydration. We provide an overview of the data modalities and the different collection methods in Table A.2. c) Labeling: The dataset includes clinically validated health event labels. Alerts were generated through predefined thresholds and analytical models, and then verified by a monitoring team. Events include agitation episodes, abnormal blood pressure, body temperature, body water levels, heart rate, and weight changes. In addition, demographic information (age group, sex, dementia diagnosis) is provided for all participants but were not used in our model since the authors reported that they do not impact performances. d) Performance metrics: We report the binary sensitivity (Recall) and the specificity scores, similarly to the original paper. In addition, we report the AUROC to get a unique comparison metric based on specificity and sensitivity. The metrics are reported in a timeseries split cross-validation. Roughly speaking, this scheme consists in training on the
beginning of all patient timeseries and validating on the end (See [13] for details). 2) SCANIA Component X: We used the CompX dataset, a publicly available real-world multivariate time series dataset designed for predictive maintenance research in heavy-duty trucks [14]. a) Description: The dataset comprises sensor-derived operational data, maintenance records, and truck specification information collected from a fleet of 28596 SCANIA vehicles. The underlying system ”CompX”, remains anonymized to protect proprietary information. ● The operational data, recorded via onboard sensors and extracted from vehicle control units, are formatted as a multivariate time series, capturing temporal dynamics through unevenly sampled readouts. These include both numerical counters and histogram-encoded variables, the latter offering compact summaries of feature distributions across bins defined by domain-informed thresholds. The different variables are depicted in Fig A.1. ● Vehicle specifications, such as engine type and configuration, are included as categorical variables and free from missing values. Time progression is represented by usage intervals for CompX. b) Modalities: The training set contains 107 anonymized features, available at irregular time steps for each vehicle. The features include: six histogram-based variables (with up to 36 bins); eight univariate numerical counters; categorical vehicle specifications encoded as one-hot vectors leading to binary vectors of dimension 94. All these variables were used as individual modalities, which leads to a total of M = 15 modalities with varying sampling timelines and dimensions, a summary is available in Table A.1. A challenge in the dataset is the irregularity in time deltas between consecutive time points. We illustrate this on one modality in Fig 3.
104 Count (log-scale)
such encoding/decoding pairs can then be chained together to obtain a deep model with increased capacity. The encoding/decoding operation at layer l > 0 is formulated as follows: (l−1) L×da ⎧ A(l) = ITNet ({[Em , τ m ]}M , (11) ⎪ m=1 ) ∈ R ⎪ ⎪ ⎪ ⎪ (l) (l−1) (l) L×d ⎨Z =Z + φ (A ) ∈ R a , (12) ⎪ ⎪ ⎪ ⎪ (l) L ×d (l) ⎪ ⎩ Em = gm (τ m , Z ) ∈ R m m , ∀m ∈ [M ], (13)
103 102 101 100
0
20
40 60 80 100 Average time delta in 171_0
120
Fig. 3: Histogram of average time deltas between samples of modality ”171 0” across all vehicles in the training dataset. c) Labeling: Maintenance records were compiled from official SCANIA workshop logs, including repair invoices and service orders. These records identify whether CompX was repaired or replaced during the observation period for each
vehicle. For modeling purposes, the component is considered failed at time t if any repair or replacement of CompX was documented at time t; otherwise, it is considered healthy. Each time step in a vehicle recording is labeled with one of six classes, Class 1: No failure within 48h; Class 2: Failure within 24h to 48h; Class 3: Failure within 12h to 24h; Class 4: Failure within 6h to 12h; Class 5: Failure within 6h; Class 6: Potential failure within 48h (censored data). d) Performance metrics: For the CompX dataset, our performance metric for the validation sets is the area under the precision recall characteristic (AUPRC). This metric is the most adequate in imbalanced multiclass classification problems, over, for instance, raw accuracy or AUROC, which tend to neglect disproportionate amounts of false positives over true positives. Additionally, the metric is threshold independent, which makes its calculation independent from potential poor calibrations across several folds. AUPRC is extended to multiclass by averaging one-vs-rest binary AUPRC scores corresponding to each class. The metrics are computed in a 5-fold cross-validation framework to evaluate the ability of the models to generalize. B. Model training Our models are trained with backpropagation using the ADAM optimizer. We use different training schemes to study the performances of ITGPT in settings where very few labels are available. First, we evaluate a pure SSL scheme, denoted “CE+SSL”, where all the training dataset is used to compute a SSL loss, and only a few labels are used to compute a supervised learning loss “CE”. Second, we evaluate a GPT scheme, denoted “GPT→CE”, where the model is first trained only on the GPT loss, and then is finetuned on the supervised learning loss. 1) Supervised learning loss: Our supervised training loss is the standard cross-entropy loss between true and estimated labels: lCE (y, ŷ) = ∑ yk ln ŷk (14) k
2) SSL and GPT loss: ITGPT allows to use a selfsupervised learning loss in input data space, in this paper we experiment with the MSE loss of a one-step ahead prediction of the input data: M
1 ˆ t,m ∣∣22 , ∑ ∣∣x̃t,m − x̃ L m=1 m t∈τ m
lMSE (X , Xˆ ) = ∑
(15)
ˆ t,m is the output of where ∀m ∈ ×[M ], ∀t ∈ {2, . . . , Tm }, x̃ ITGPT computed from data strictly prior to t. For the training scheme denoted ”CE+SSL”, the loss for a training sample i is the sum of the MSE loss on in input data space and the CE loss only if the label is made available : lCE+SSL (X (i) , Xˆ (i) ) = lMSE (X (i) , Xˆ (i) ) + 1(i ∈ L).lCE (y(i) , ŷ(i) ),
(16)
where L is the set with the indices of training samples with available labels, and 1 is the indicator function.
For the training scheme denoted ”GPT→CE”, the model is first pretrained with the MSE loss on 2 epochs, and then trained only on the CE loss with the data with the available labels for 5 epochs, i.e. for epoch n ∈ [7], the loss function is formulated: ⎧ lMSE (X (i) , Xˆ (i) ) if n ≤ 2, ⎪ ⎪ ⎪ ⎪ lGPT→CE (X , X ) = ⎨ lCE (y, ŷ) if n > 2, i ∈ L, ⎪ ⎪ ⎪ ⎪ if n > 2, i ∉ L. ⎩0 (17) 3) Censored data in CompX: Class 6 in the original CompX dataset accounts for censored data (As described in III-A2c). This class is ambiguous in the sense that samples from classes 1 to 5 might be labeled as 6. Training with a CE loss for the samples in this class will thus make the dataset harder to fit, since all samples could be classified as 6. This is illustrated in Fig. 4d where many more samples from class 6 are misclassified into the other classes equally We thus ignore samples with class 6 when computing a CE loss. The input data are still used without labels in the SSL and GPT training schemes where the MSE loss does not dependently on the label. (i)
ˆ (i)
C. Hyperparameters A list of the hyper-parameters is available in Table I. The results of various architectural choices are reported on the CompX dataset because it is the dataset with the most complexity in terms of number of samples and . 1) Model architecture: We study the impact of varying the architecture of two important components of the model: the depth and the mixing layers. The mixing layer corresponds to the layer which outputs the anchor data from the encoding of the data from individual modalities (See Fig. 1b). We experiment with linear mixing layers, as well as multilayer perceptron layers with 1 and 2 hidden layers. The depth corresponds to the number of chains of pairs of ITNet models, with a depth of 1 corresponding to an ITNet encoder alone. We experiment with depth from 1 to 7 pairs of encoder/decoder, and the models of increasing depth are expected to overfit data. We also add experiments varying the percentage of dropout during training, in an attempt to control the overfitting. To reduce the number of possible combinations of hyperparameters, we found reasonable values for other parameters. IV. R ESULTS We start with the results on the TIHM dataset in Section IV-A. The results of experiments with various architecture choices on the CompX dataset are presented in Section IV-B1. We then show and discuss the results of our experiments with fewer labeled samples in Section IV-B2. The results are reported in box plots depicting the median and first and third quartiles. The experiments are seeded so that the folds are identical across hyperparameter choices.
Dropout
0.48
0%
10%
20%
30%
0.46
0.44
0.44
0.42
0.42
AUPRC
0.46
AUPRC
Mixing layer
0.48
0.4
0.38
Linear
MLP/1
MLP/2
4
5
6
0.4
0.38
0.36
0.36
0.34
0.34
0.32
0.32
0.3
1
2
3
4
5
6
7
1
2
3
Model depth
Loading [MathJax]/extensions/MathMenu.js
Loss
GPT → CE
CE
7
Model depth
(a) Comparing levels of dropout for various model depth with linear mixing layers. 0.32
0.3
(b) Comparing linear, MLP/1 and MLP/2 mixing layers for various model depth, with dropout=10%. 5719 vehicles, 225019 samples >48 48-24 24-12 12-6 <6 U
CE+SSL
23623 1945 407 734 0 >48 152811 (0.85) (0.13) (0.01) (0.00) (0.00) (0.00)
0.28
122 1742 610 54 11 0 48-24 (0.05) (0.69) (0.24) (0.02) (0.00) (0.00)
AUPRC
0.3
True Labels
0.26 0.24 0.22
16 240 697 307 53 0 24-12 (0.01) (0.18) (0.53) (0.23) (0.04) (0.00) 10 6 157 351 149 0 12-6 (0.01) (0.01) (0.23) (0.52) (0.22) (0.00) 5 9 39 186 599 0 <6 (0.01) (0.01) (0.05) (0.22) (0.71) (0.00)
0.2
1976 12459 11816 7004 6881 0 U (0.05) (0.31) (0.29) (0.17) (0.17) (0.00)
28 57 85 114 142 171 200 228 257 285 (0.1%) (0.2%) (0.3%) (0.4%) (0.5%) (0.6%) (0.7%) (0.8%) (0.9%) (1.0%)
Predicted Labels
N (%) labeled training vehicles
(c) Performances in a small labeled dataset setting. CE: Training only with cross-entropy; CE+SSL: Training with both cross entropy and a self-supervised loss.; GPT→CE: Pretraining with the self-supervised loss, fine tuning with cross-entropy loss.
(d) Confusion matrix on a validation fold for a model trained to predict 5 classes. Class 6 attracts samples from all other classes and thus, should not be used for training.
Fig. 4: Results in a 5-fold cross-validation framework on the CompX dataset.
Fixed
Varied
Short Description key/queries dimensions key/query/value maps Activation function Anchor dimension Batch size Optimizer Learning rate Number of epochs Chain depth Dropout probability Mixing layer % used labels
Symbol dk fQ/K/V φ da B / λ Ne L p W pl
Value 32 Linear ReLU 64 64 ADAM 5 × 10−4 20 {1, 2, 3, 4, 5, 6, 7} {0, 0.1, 0.2, 0.3} {Linear, MLP/1, MLP/2} {0.1%, 0.2%, . . . , 1%}
TABLE I: List of fixed and varied hyperparameters for the CompX dataset.
interpolation. The GPT+CE training method does not improve performances over the classical CE loss function. However, we see that the CE+SSL framework improve the mean recall performance marginally from 0.63 to 0.73. The other metrics, specificity, AUROC, and F1score remain similar. Recall
Spec
AUROC
F1score
0.68 (0.14) 0.73 (0.17) 0.6 (0.22)
0.77 (0.08) 0.76 (0.09) 0.78 (0.1)
0.78 (0.05) 0.79 (0.06) 0.76 (0.06)
0.25 (0.11) 0.26 (0.11) 0.23 (0.12)
Loss CE CE+SSL GPT → CE
TABLE II: Numerical results for several binary classification metric. A. TIHM dataset We start with the results obtained on TIHM with several losses and all the available labeled data. The results are shown in Table II for several losses. We also mention that the original paper [13] reports specificity and sensitivity of around 0.8 with a linear model and linear interpolation. Table II shows that our model performs similarly, although slightly under the performance reported by the original authors. We mention this is done entirely from the raw data and without the need for
Next we discuss the results when the training scheme and the number of available samples vary. Overall, as expected, the recall performance decrease and the specificity increases as we train with an increasing number of labels. We see only marginal differences between the training schemes, particularly for the specificity. Interestingly, the recall performance of the “CE+SSL” scheme remains stable after 25 labeled samples, while both “CE” and “GPT→CE” performances keep
decreasing. This is coherent with the mean performances in the table reported previously. Loss
GPT → CE
CE+SSL
25 (50%)
35 (70%)
CE
1
Recall
0.8 0.6 0.4 0.2
10 (20%)
15 (30%)
20 (40%)
30 (60%)
40 (80%)
45 (90%)
50 (100%)
N (%) labeled patients
(a) Recall/Sensitivity performance Loss
GPT → CE
CE+SSL
2) Varying the amount of available labels: Both our GPT→CE and CE+SSL training schemes improve over the simple CE loss scheme (Fig. 4c). We note that the performances of ”CE+SSL” increase faster with the amount of available data compared to ”GPT→CE”, with the highest score achieved when 85 samples are available. The lowest performance our models obtain is around 0.2 AUPRC when only 28 samples are labeled. When the amount of labels increases, the performances of the models trained with the CE loss only increases as expected. The performances of the models trained with both GPT→CE and CE+SSL also increases but only until 171 available labels. With more available samples, the performances of GPT→CE decreases slightly, and the performances of CE+SSL saturates a little under 0.28 AUPRC.
CE
V. C ONCLUSION
0.8
Spec
0.6 0.4 0.2 0 10 (20%)
15 (30%)
20 (40%)
25 (50%)
30 (60%)
35 (70%)
40 (80%)
45 (90%)
50 (100%)
N (%) labeled patients
(b) Specificity performance
Fig. 5: Recall (Sensitivity) and Specificity for an increasing amount of available labels. B. SCANIA component X (CompX) dataset 1) Varying ITGPT architectures: We start by presenting the results obtained on the full CompX dataset with a CE loss while varying the dropout probability, the chain depth and the type of mixing layer. Fig. 4a shows that increasing the probability of dropout has a detrimental effect at depth 1 and a beneficial effect at depth larger than 2. This shows that overfitting must be controlled at large depth, in turn confirming that our architecture does not suffer from gradient collapse. This is important to be able to scale the capacity of the model for complex datasets. A similar conclusion can be drawn for MLP/1 and MLP/2 mixing layers on Fig. A.2 and Fig. A.3. An additional observation for these mixing layers is that dropout probability > 30% seem to decrease performances compared to 10% and 20%. This is a different behavior from the linear mixing layer case. In our setup, several choices of architectures achieve similar performance around 0.44 AURPC score: MLP/2 mixing layers, dropout probability 10% and depth 3 or 6; as well as the MLP/1 mixing layers, dropout probability 20% and depth of 6 or 7. The results for linear mixing layers are slightly lower with the best-performing model at 0.42 AUPRC with a depth of 4 and a dropout probability of 20%. Overall, this experiment confirms the sanity of our ITGPT architecture and stresses the importance of controlling overfitting when the model capacity increases.
In this work, we introduced ITGPT, a transformer-based architecture specifically designed to handle multimodal, irregularly sampled timeseries data without requiring resampling, feature fusion, or imputation. ITGPT leverages both selfsupervised learning (SSL) and generative pretraining (GPTstyle) objectives, making it especially effective in low-label regimes. Our experiments demonstrate that ITGPT achieves state-of-the-art performance on a predictive maintenance task using the CompX dataset, particularly when labels are scarce. We show that SSL and GPT pretraining significantly boost performance over purely supervised learning, with the CE+SSL scheme benefiting the most from increasing amounts of labeled data. We also explored architectural variants, revealing that dropout plays a critical role in controlling overfitting at deeper depths, and that MLP-based mixing layers outperform linear ones under optimal settings. Our best configurations reach AUPRC scores of 0.44, validating the capacity and robustness of the ITGPT architecture. Finally, we identify several promising directions for future research, including: Improving interpretability through unitary mixing layers that track individual modality contributions; ● Reducing computational complexity, especially via more efficient attention mechanisms and modality-parallel CUDA kernels; ● Exploring sparse dot-product computations to optimize batching and reduce unnecessary inter-sample attention. ●
Overall, ITGPT offers a strong and flexible foundation for tackling real-world timeseries prediction tasks, and opens possibilities to leverage large and unstructured datasets in industrial or health care applications. VI. ACKNOWLEDGEMENTS The authors wish to thank Baptiste Cavarec for valuable discussions on experimental design and insightful feedback during results interpretation. We used ChatGPT for suggesting rephrasings of the abstract and conclusion.
R EFERENCES [1] Y. Zong, O. M. Aodha, and T. M. Hospedales, “Self-Supervised Multimodal Learning: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, pp. 5299–5318, July 2025. [2] T. Zonta, C. A. da Costa, R. da Rosa Righi, M. J. de Lima, E. S. da Trindade, and G. P. Li, “Predictive maintenance in the Industry 4.0: A systematic literature review,” Computers & Industrial Engineering, vol. 150, p. 106889, Dec. 2020. [3] D. Ravı̀, C. Wong, F. Deligianni, M. Berthelot, J. Andreu-Perez, B. Lo, and G.-Z. Yang, “Deep Learning for Health Informatics,” IEEE Journal of Biomedical and Health Informatics, vol. 21, pp. 4–21, Jan. 2017. [4] Y. Wang, C. M. Albrecht, N. A. A. Braham, L. Mou, and X. X. Zhu, “Self-Supervised Learning in Remote Sensing: A review,” IEEE Geoscience and Remote Sensing Magazine, vol. 10, pp. 213–247, Dec. 2022. [5] R. Krishnan, P. Rajpurkar, and E. J. Topol, “Self-supervised learning in medicine and healthcare,” Nature Biomedical Engineering, vol. 6, pp. 1346–1352, Dec. 2022. [6] J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao, “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, pp. 9052–9071, Dec. 2024. [7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine Learning, pp. 8748–8763, PMLR, July 2021. [8] Q. Gan and C. Harris, “Comparison of two measurement fusion methods for Kalman-filter-based multisensor data fusion,” IEEE Transactions on Aerospace and Electronic Systems, vol. 37, pp. 273–279, Jan. 2001. [9] Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu, “Recurrent Neural Networks for Multivariate Time Series with Missing Values,” Scientific Reports, vol. 8, p. 6085, Apr. 2018. [10] J. Futoma, S. Hariharan, and K. Heller, “Learning to detect sepsis with a multitask Gaussian process RNN classifier,” in Proceedings of the 34th International Conference on Machine Learning (D. Precup and Y. W. Teh, eds.), vol. 70 of Proceedings of Machine Learning Research, pp. 1174–1182, PMLR, Aug. 2017. [11] A. Honoré, P. Appelquist, and M. Xiao, “ITNet: Irregular Timeseries Data Fusion with Attention Mechanisms,” in 2025 28th International Conference on Information Fusion (FUSION), pp. 1–7, July 2025. [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Dec. 2017. [13] F. Palermo, Y. Chen, A. Capstick, N. Fletcher-Loyd, C. Walsh, S. Kouchaki, J. True, O. Balazikova, E. Soreq, G. Scott, H. Rostill, R. Nilforooshan, and P. Barnaghi, “TIHM: An open dataset for remote healthcare monitoring in dementia,” Scientific Data, vol. 10, p. 606, Sept. 2023. [14] Z. Kharazian, T. Lindgren, S. Magnússon, O. Steinert, and O. Andersson Reyna, “SCANIA Component X dataset: A real-world multivariate time series dataset for predictive maintenance,” Scientific Data, vol. 12, p. 493, Mar. 2025.
for a wide range of machine learning tasks, including classification, regression, anomaly detection, and survival analysis. The perturbation applied to the dataset leads to features in the range [0, 109 ] across all vehicles. Scaling with large factors might lead to underflow since the range varies quite a lot per vehicle. Thus, we chose to apply a logarithmic normalization: x ↦ ln (1 + x). 2) Labeling: To ensure label quality, only vehicles with a complete service history within the SCANIA workshop network were included. This restriction avoids ambiguities arising from missing third-party repair data and helps reduce label noise. However, it does introduce selection bias toward vehicles that consistently use SCANIA-authorized maintenance services. Modality Dimension Total number name of samples
% vehicles with complete modality
specs 397 459 291 158 272 167 370 0 835 0 309 0 837 0 427 0 666 0 171 0 100 0
1.00 0.80 0.68 0.68 0.80 0.81 0.81 0.74 0.82 0.82 0.82 0.74 0.82 0.82 0.74
Aspect
Modality
In-home activity
Kitchen Bedroom Hallway Bathroom Fridge Door Front Door Lounge Back Door Body Temperature Systolic blood pressure Diastolic blood pressure Heart rate Skin Temperature Sleep HR Sleep RR Body weight Total body water O/E - muscle mass
Physiology
The samples can be irregularly spaced across observations or modalities. Let (i)
(i)
δ (i) m = [0, tm,2 − tm,1 , . . . , t
(i)
L(i)
(i)
]⊺ ∈ R+ m , (18)
Sleep mat
denote the vector of time difference between two consecutive timestamps.
Smart scale
(i)
m,Lm
−t
(i)
m,Lm −1
B. Additional CompX Data Information 1) Normalization: Perturbation strategies were used to ensure data privacy and commercial confidentiality. The original timestamps and the true variable names were removed from the data, and feature values were scaled with an undisclosed factor. According to the authors, the dataset retains its utility
28.60K 1.32M 1.31M 1.31M 1.32M 1.32M 1.32M 1.31M 1.32M 1.32M 1.32M 1.31M 1.32M 1.32M 1.31M
TABLE A.1: Description of the modalities in the training dataset of 28596 vehicles. Modalities are considered complete for a vehicle if the sampling and label timelines coincide.
A PPENDIX A. Position encoding
94 36 20 11 10 10 10 1 1 1 1 1 1 1 1
Dimension
# samples
1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2
2196 2162 2148 2124 2113 2099 1944 1813 1580 353 353 353 121 757 757 275 113 113
TABLE A.2: Modality description
4
10000
2
5000
0.0
0
20
time
40
60
370_0
0.04 0.00 0.02 0.04 0
20
time
40
459
60000
0
20
time
40
0
60
0
20
time
40
60
837_0
0 1e7
20
time
20
time
40
0.0
60
100_0
40
60
0.6 0.4 0.2 0.0
0
20
time
40
60
40
60
158
0 1e7
20
time
427_0
6
2 0 1e7
20
time
40
0
60
2.0
397
1.0
0.5
0.5
0.0
0.0
0
20
time
0 1e7
20
40
60
1.0
309_0
0.8
time
40
60
167
1.5
1.0
600 500 400 300 200 100 0
60
4
1.5
272
0.8
0
40
0.5
1
0
time
1.0
2
20000
20
1.5
3
40000
0 1e7
2.0
666_0
50000 40000 30000 20000 10000 0 60 0 1e6 4
0.02
835_0
6
15000
0.5
1.0
291
20000
1.0
10000 8000 6000 4000 2000 0
1e7
25000
171_0
0
5
10 time
15
specs
time
0.0
60
Dropout
0%
10%
20%
0.3
0.050 0.025 0.000 0.025 0.050 time
4
5
30%
AUPRC
0.42 0.4
0.38 0.36 0.34 0.32 2
3
0.4
0.32
0.44
1
0.42
0.34
0.46
0.3
30%
0.36
Fig. A.1: Example input multimodal data for a single vehicle. One plot corresponds to one modality. The sampling instants are highlighted with dots.
0.48
20%
0.38
0.2 40
10%
0.44
0.4
20
0%
0.46
0.6
0
Dropout
0.48
AUPRC
1e6 1.5
6
7
Model depth
Fig. A.2: Comparing levels of dropout for various model depth and MLP/1 mixing layers
1
2
3
4
5
6
7
Model depth
Fig. A.3: Comparing levels of dropout for various model depth and MLP/2 mixing layers