Distribution-Aware Diffusion-LLM for Robust Ultra-Long-Term Time Series Forecasting Falguni Ghosh1,3[0000−0002−2786−2044] , Vahid Hashemi3[0000−0002−9167−7417] , and Bernhard Kainz1,2[0000−0002−7813−5023]
arXiv:2606.23391v1 [cs.LG] 22 Jun 2026
1
Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, 91052, Germany {falguni.ghosh}@fau.de 2 Imperial College London, London SW7 2RH, United Kingdom 3 AUDI AG, Auto-Union-Straße 1, 85057 Ingolstadt, Germany
Abstract. Time series forecasting is a fundamental machine learning task. Recent work has explored Large Language Models (LLMs) for this purpose due to their strong generalization, pattern recognition, and zeroshot or few-shot capabilities. Despite their suitability for long-context learning, LLMs face challenges in multimodal settings: they lack calibrated probabilistic modeling for non-text data and struggle to align heterogeneous representations. To address these issues, we propose a new framework Diffusion-LLM that integrates a conditional diffusion model into an LLM-based forecasting pipeline. This joint design enables learning the conditional distribution of future data while improving semantic alignment in a shared latent space. We evaluate Diffusion-LLM on six long-term forecasting benchmarks, including ETT, Weather, and ECL. Our method consistently outperforms existing LLM-based baseline, achieving notable gains in ultra-long-term and few-shot forecasting and demonstrating the value of distribution-aware regularization for enhancing robustness and generalization in time series LLMs. Keywords: Time Series Forecasting · Multimodal Alignment · Large Language Model · Diffusion Models.
1
Introduction
Time series forecasting is essential in domains such as energy systems [29,3], healthcare [16], climate science [11], and supply chain management [18]. Many applications, including energy demand planning, climate modeling, and battery lifetime prediction [13,30] require ultra-long-term forecasts extending thousands or more steps ahead, often from limited historical data. LLMs have recently emerged as promising forecasters due to their strong generalization, pattern recognition, and zero-/few-shot abilities [6]. However, applying pretrained LLMs to time series remains challenging. Their representations are tuned for semantic structure in language, not temporal dynamics, making cross-modal alignment difficult and leading to degraded performance and potential multimodal hallucinations [24]. Moreover, MSE-trained LLM forecasters tend to regress toward the mean and fail to capture the full distribution
2
F. Ghosh et al.
of possible futures, especially for irregular or noisy series [27]. As generation progresses, attention increasingly concentrates on recent predictions, reducing global context awareness and amplifying uncertainty underestimation [23,8]. To address these issues, we incorporate a Denoising Diffusion Probabilistic Model (DDPM) [7] into an LLM-based forecasting pipeline. Using the reprogramming strategy of TimeLLM [10], both inputs and targets are embedded into a shared token space. The DDPM is jointly trained to estimate the conditional distribution of forecast embeddings given the lookback window, providing a distribution-aware signal that regularizes the LLM and strengthens multimodal alignment. This results in refinement of the shared embedding space, improved robustness, and long-horizon forecasting. Our key contributions are: – We introduce DDPMs as implicit regularizers for multimodal LLMs, enabling joint alignment and distribution modeling in a unified embedding space. – We propose Diffusion-LLM, a framework that models the distribution of reprogrammed time series patches to enhance temporal reasoning. – We show that our method significantly improves ultra-long-term and few-shot forecasting performance across multiple benchmarks.
2
Related Work
2.1
LLM in Time Series Forecasting:
Recent research adapts LLMs to time series using several strategies. Promptingbased methods treat time series as raw text [36,6], but lose temporal semantics due to modality mismatch. Quantization approaches discretize sequences via VQVAE or clustering [26,39], often requiring two-stage training. Vision-as-bridge methods encode series as images interpreted by vision-language models [32], but rely on paired visual data and lack generality. Tool-augmented approaches let LLMs generate code or API calls [19] though they introduce complexity and are not end-to-end forecasters. Alignment-based approaches instead learn time-series encodings compatible with LLM semantic spaces [42]. These methods fall into two categories: – Contrastive alignment: ETP [14], TEST [25], and TENT [47] use contrastive objectives to align sensor or physiological signals with text descriptions, effective when multimodal pairs exist. – LLM-backbone alignment: GPT4TS [46], LLM4TS [2], and TimeLLM [10] feed reprogrammed time-series patches into frozen or partially frozen LLMs. GPT4TS freezes self-attention layers to preserve pretrained knowledge; LLM4TS uses autoregressive alignment followed by parameter-efficient tuning; TimeLLM reprograms series into token sequences resembling natural language. Time-VLM [43] extends this paradigm to include image modalities. Our work builds on this alignment-based direction but introduces diffusionbased regularization to strengthen distributional modeling, an aspect overlooked in classical models [1,38,9,31] which focus on deterministic or multiscale decomposition rather than probabilistic uncertainty or multimodal alignment.
Diffusion-LLM for Ultra-Long-Term Forecasting
Forecasting Loss from LLM component
C
Denoising Diffusion Probabilistic Model (Fully Connected network with residual connections)
ZT
Diffusion Process
Negative Log Likelihood Loss from DDPM component
Word Embedding Space
Conditioning
Frozen
Output Projection
Trainable
Embedded Prediction Window (Z)
Target Time Series Segment (Forecasting Window)
Patch Encoder (Instance Norm + Patching + Reprogramming)
Embedded Input Window
Input Time Series Segment (Lookback Window)
Pretrained LLM (Body)
Output Embedding
Pretrained LLM (Embedder)
B Concatenation
Prompt (Task Instruction + Dataset Description)
Prompt Embedding
A
3
Fig. 1: Training architecture of Diffusion-LLM. (A) The prompt, input, and target time series are reprogrammed into a shared token embedding space using a frozen LLM encoder and a trainable patch encoder. (B) The encoded input is used for direct forecasting via a frozen LLM body + trainable output projection module. (C) A conditional DDPM is trained to model the distribution of the encoded target, conditioned on the input, by predicting the added noise. The final loss combines forecasting and diffusion-based regularization. 2.2
DDPM in Time Series Forecasting:
DDPM-based forecasters generally pair diffusion models with autoregressive backbones. TimeGrad [20] corrupts future values with noise and denoises them conditioned on RNN-encoded lookback windows. ScoreGrad [37] follows a similar feature extraction pipeline but employs conditional SDE-based score matching. Unlike these models, we do not use DDPMs as stand-alone generative forecasters. Instead, we leverage DDPMs as auxiliary learners that regularize LLM-based predictors, improving robustness and uncertainty modeling without replacing the LLM’s forecasting role.
3
Methodology
Our proposed framework, Diffusion-LLM, enhances LLM-based time series forecasting by integrating a conditional DDPM as a regularizer. The model estimates the conditional distribution of the forecast window given the lookback window within a shared embedding space of text prototypes produced through time-series reprogramming, improving both probabilistic modeling and multimodal alignment. An overview of the training architecture is shown in Figure 1. The framework consists of three main components: A. Time Series Encoder (Reprogramming and Embedding): Following the TimeLLM reprogramming strategy [10], raw time series is tokenized into patches and mapped to text-like prototypes via an attentionbased mechanism (Figure 2b). During training, both the lookback window x
4
F. Ghosh et al.
87
27
Top prototypes serial numbers
47
Top text prototypes composed of time Series-relevant words
Reprogrammed patch Embeddings
(a) Prompt.
n
ral
aly
tio
po Tem
om
rre la
An
Co
ast
rity na tio
Fo rec
Sta
y
n La g
tio
Au
toc
orr ela
lity na aso Se
Vo lat
Tre n
ilit
d
34
Multi-Head Attention
Patch Embedder
Top text prototypes composed of generic words 25
0.008
Top prototypes serial numbers
Text prototypes
0.006
80
Time Series Patches
Linear
0.002 0.000
23
Instance Norm
0.004
85
Patching
(b) Patch Encoding.
0.002
mo nd mu seu m wa ter fall
dia
t
ch en kit
ke t
lle wa
isla
roc
r ita
low pil
gu
nd
2
0.004
ple
Pre-trained Word Embeddings (LLM Vocabulary)
ap
Weather is recorded every 10 minutes for the 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. Predict the next 2048 steps given the previous 512 steps information attached. The input has a minimum of < min >, a maximum of < max >, and a median of < med > The overall trend is < up/downward > The top five lags are < lags >.
69
Linear
(c) Prototype composition.
Fig. 2: (a) Sample prompt used for the Weather dataset. (b) Patch encoding mechanism, where a time series window is normalized, patched, and matched to text prototypes learned and derived from pretrained LLM vocabulary. Attention between patches and prototypes enables the model to associate temporal patterns with language-like cues (‘steady up’, ‘periodic pattern’ etc.). (c) Prototype composition visualization, showing the top-matched prototypes for a set of time-series-related and generic words after training, illustrating how the learned prototype space can form prototypes with slightly higher affinity for time seriesrelated vocabulary over generic words in initial linear layer already. This distinction becomes more pronounced after the attention layers (Figure 5). and forecast window y are encoded using the shared encoder ϕllmenc , producing latent representations zx and zy . We maintain the three-part prompt design from the baseline - dataset details, task instruction, and statistical information (Figure 2a). For simplicity, we omit the frozen prompt embedder in notation and denote the patch encoder directly as ϕllmenc : zx = ϕllmenc (x),
zy = ϕllmenc (y).
(1)
The encoded time series becomes a semantic representation (prototype) describing its patterns and statistics in the LLM’s embedding space (Figure 2c), enabling the pretrained model to process it using its native architecture. B. Forecasting via LLM: The encoded input zx is passed to an output module ϕllmout , consisting of the pretrained frozen LLM body and a small trainable output projection layer, to produce the forecast ŷ: ŷ = ϕllmout (zx ),
Lforecast = ∥y − ŷ∥2 .
(2)
This component exploits the pretrained reasoning and pattern-recognition capabilities of LLMs without full finetuning, retaining efficiency and generalization. As shown in [10,5], reprogramming frozen LLMs can be more efficient than parameter-efficient finetuning approaches such as QLoRA [4].
Diffusion-LLM for Ultra-Long-Term Forecasting
5
Algorithm 1 Diffusion-LLM Training Require: Time series dataset D = {(x, y)}, LLM encoder module ϕllmenc , LLM output module ϕllmout , DDPM model θddpm , regularization weight λ. Let αt denote the noise Q schedule coefficients at time t and ᾱt = ts=1 αs . Initialize parameters of ϕllmenc , ϕllmout , θddpm . for each training iteration do Sample a batch B = {(xi , yi )} from D for each (x, y) in B do 1. Encode input and target windows (a) zx ← ϕllmenc (x), (b) zy ← ϕllmenc (y) 2. Forecasting prediction and loss (a) ŷ ← ϕllmout (zx ), (b) Lforecast ← ∥y − ŷ∥2 3. DDPM loss (a) Sample noise ϵ ∼ N (0, t ∼ Uniform(1, T ) √ √I) and timestep (b) Noised sample: z̃y = ᾱt zy + 1 − ᾱt · ϵ (c) Predict noise: ϵ̂ ← θddpm (z̃y , t, zx ) (d) Lddpm ← ∥ϵ − ϵ̂∥2 4. Combine losses (a) Ljoint ← Lforecast + λ · Lddpm end for Update ϕllmenc , trainable projection part of ϕllmout , θddpm using gradients of Ljoint end for
C. Distribution Regularization via DDPM: To strengthen the model’s ability to capture the token distribution of time series representations, we incorporate a conditional DDPM that learns the conditional distribution p(zy | zx ) through a denoising process. During training, noise is added to zy to obtain z̃y , and the DDPM predicts the noise ϵ: z̃y ∼ q(z̃y | zy , t),
ϵ̂ = ϵθ (z̃y , t, zx ),
Lddpm = ∥ϵ − ϵ̂∥2 .
(3)
Here, ϵθ denotes the DDPM denoising network parameterized by θddpm , t is the diffusion timestep, and ϵ ∼ N (0, I) is the sampled noise in the forward noising process q(z̃y | zy , t). As shown in [7], this objective is equivalent to maximizing the conditional likelihood, Lddpm = − log pθ (zy | zx ).
(4)
The DDPM thus serves as both a probabilistic constraint and an auxiliary learner that enriches the shared embedding space through semantic alignment. The overall model is jointly optimized for forecasting and distribution estimation via Ljoint = Lforecast + λ · Lddpm .
(5)
This regularization is alignment-agnostic and can be integrated into existing LLM-based forecasting methods with minimal modification, as demonstrated in our enhancement of TimeLLM [10]. We jointly optimize all the learnable parameters. The full training procedure is provided in Algorithm 1. During inference, only the LLM modules are used to generated forecasts (Figure 3). The inference steps are formally defined in Algorithm 2.
6
F. Ghosh et al.
Algorithm 2 Diffusion-LLM Inference Require: Input time series x, trained encoder ϕllmenc , trained output module ϕllmout 1. Encode the input window zx ← ϕllmenc (x) 2. Generate forecast ŷ ← ϕllmout (zx ) return ŷ as the predicted forecast window
4
Experiments and Results
Forecasting Loss from LLM component
Output Projection
Pretrained LLM (Body)
Output Embedding
Patch Encoder (Instance Norm + Patching + Reprogramming)
Embedded Input Window
Input Time Series Segment (Lookback Window)
Pretrained LLM (Embedder)
Concatenation
Prompt (Task Instruction + Dataset Description)
Prompt Embedding
A. Model Architecture: We use the 7B variant of LLaMA [28] as the backbone LLM. The diffusion module is a lightweight DDPM implemented as a stack of fully connected layers with skip connections (details in A.3). All experiments are run on NVIDIA A100 and H100 GPUs. B. Long-Term Forecasting: We evaluate Diffusion-LLM on six standard long-term forecasting benchmarks: ETTh1, ETTh2, ETTm1, ETTm2 [44], Weather, and ECL [34] (details in A.1). As our method serves as an add-on to existing LLM-based approaches, we report competitive results with established bench- Fig. 3: Inference pipeline of Diffusion-LLM. marks (Table 3) and provide Only the LLM modules are used to generate a comprehensive comparison forecasts for new inputs. against the TimeLLM baseline, including mean and standard deviation for MSE and MAE metrics (Table 1, Table 2). For long-term forecasting, Diffusion-LLM achieves performance comparable to TimeLLM. C. Ultra-Long-Term Forecasting: Ultra-long-term forecasting remains highly challenging due to increased uncertainty and weaker dependence on recent history. We evaluate {1024, 2048} forecast lengths on the same datasets, and our Diffusion-LLM consistently outperforms TimeLLM in this regime (Table 1). Improvements are especially pronounced on smaller datasets such as ETTh1 and ETTh2, with MSE reductions of 19.26% and 11.38%, respectively, highlighting the benefit of modeling the full conditional distribution of the target window. D. Few-Shot Forecasting (10% and 5%): To assess few-shot generalization, we train both models using only 10% of the data. Diffusion-LLM consistently surpasses TimeLLM across all long- and ultra-long-term settings (Table 2); on ETTh1, it achieves a 20.62% improvement even for long-term horizons. In the more extreme 5% setting, the advantage becomes even clearer, with Diffusion-LLM improving ETTh1 performance by 25.79%. These results show that diffusion-based regularization substantially enhances generalization in lowdata regimes without modifying the frozen LLM backbone.
Diffusion-LLM for Ultra-Long-Term Forecasting
7
Table 1: Comparison of TimeLLM [10] and Diffusion-LLM on long-term and ultra-long-term forecasting across standard benchmarks. Long-term results are averaged over horizons H ∈ {96, 192, 336, 720} with a 512-step input; ultralong-term averages over H ∈ {1024, 2048}. Each cell reports MSE, MAE, and standard deviations across runs (lower is better; best in bold). Diffusion-LLM outperforms TimeLLM on 4/6 datasets in the ultra-long-term setting, with substantial gains on more challenging smaller datasets such as ETTh1 and ETTh2. Dataset
Long-term TimeLLM
Ultra-long-term
Diffusion-LLM (Ours)
MSE
MAE
MSE
ETTh1
0.449±0.025
0.457±0.015
ETTh2 ETTm1 ETTm2
TimeLLM
Diffusion-LLM (Ours)
MSE
MAE
0.427±0.004 0.446±0.010
0.758±0.018
0.600±0.011
0.612±0.011 0.558±0.004
0.373±0.009 0.409±0.006
0.387±0.003
0.425±0.002
0.589±0.013
0.543±0.007
0.522±0.009 0.512±0.004
0.381±0.008
0.376±0.004 0.399±0.002
0.484±0.009
0.472±0.012
0.465±0.001 0.452±0.001
0.271±0.003 0.330±0.003
0.334±0.003
0.369±0.001
0.410±0.020 0.425±0.014
0.422±0.008
Weather
0.259±0.019 0.288±0.017
0.304±0.001
0.329±0.001
0.424±0.008
0.407±0.001 0.394±0.001
ECL
0.171±0.002 0.277±0.003
0.200±0.004
0.303±0.002
0.272±0.001 0.356±0.000
0.406±0.006
MAE
0.401±0.004
MSE
0.297±0.005
MAE
0.436±0.004 0.376±0.004
Table 2: Comparison of TimeLLM [10] and Diffusion-LLM in few-shot long and ultra-long-term forecasting across standard benchmarks. Few-shot (10% or 5%) indicate training with only 10% or 5% of the available data to assess generalization under scarcity. Other experimental details follow the protocol of Table 1. ‘–’ denotes insufficient data for a meaningful training set. Diffusion-LLM consistently outperforms TimeLLM in few-shot ultra-long-term settings. Dataset
Few-shot (10%) long-term TimeLLM MSE
MAE
ETTh1
0.834±0.073
0.614±0.022
ETTh2
0.422±0.009
Few-shot (10%) ultra-long-term
Diffusion-LLM (Ours) MSE
MAE
TimeLLM
Few-shot (5%) long-term
Diffusion-LLM (Ours)
TimeLLM
MSE
MAE
MSE
MAE
MSE
MAE
0.662±0.004 0.564±0.001
-
-
-
-
0.988±0.066
0.662±0.021
0.443±0.005
0.398±0.003 0.432±0.002
-
-
-
-
0.415±0.014
ETTm1
0.504±0.001 0.462±0.003
0.502±0.029 0.464±0.014
1.056±0.101 0.691±0.036
0.660±0.062 0.550±0.026
0.600±0.011
ETTm2
0.327±0.017 0.361±0.008
0.336±0.003
0.370±0.003
0.582±0.022 0.506±0.003
0.442±0.000 0.447±0.000
Weather
0.256±0.000 0.291±0.002
0.319±0.008
0.340±0.004
0.480±0.005 0.430±0.002
ECL
0.190±0.000 0.288±0.001
0.197±0.000
0.294±0.000
0.292±0.000 0.367±0.003
Few-shot (5%) ultra-long-term
Diffusion-LLM (Ours) MSE
MAE
TimeLLM
Diffusion-LLM (Ours)
MSE
MAE
MSE
MAE
0.728±0.029 0.582±0.013
-
-
-
-
0.435±0.008
0.392±0.003 0.428±0.003
-
-
-
-
0.515±0.006
0.528±0.014 0.480±0.005
0.924±0.032 0.666±0.011
0.628±0.003 0.536±0.001
0.330±0.005 0.367±0.003
0.346±0.001
0.381±0.003
0.522±0.018 0.480±0.003
0.450±0.004 0.444±0.006
0.428±0.003 0.406±0.000
0.304±0.006 0.326±0.003
0.329±0.005
0.347±0.003
0.477±0.007 0.434±0.004
0.424±0.006 0.406±0.004
0.281±0.001 0.358±0.004
0.192±0.000 0.289±0.001
0.201±0.003
0.298±0.000
-
-
-
-
Our results indicate that while multimodal LLMs can capture short-term structure, very long horizons with higher uncertainty require explicit distribution modeling. Incorporating the DDPM helps the encoder learn richer representations in the shared embedding space. As diffusion models estimate full probability distributions, the regularization they provide is most beneficial under high uncertainty; this can introduce a mild trade-off, slightly reducing point accuracy for short horizons. Empirically (Tables 1, 2), the gains are largest on challenging smaller datasets (e.g., ETTh1, ETTm1). Even for other datasets, benefits emerge as forecast horizons lengthen or training data becomes scarce. Figure 4 further shows that under severe data scarcity, Diffusion-LLM degrades substantially less than TimeLLM, underscoring its robustness in high-uncertainty settings.
5
Model Analysis
We present ablation studies highlighting design choices, with empirical results in Table 5 (Appendix A.3).
8
F. Ghosh et al. 2
Input Ground Truth Prediction Forecast Start
Sensitivity to Training Data Availability (ETTh1)
1
Long-Term Forecasting MSE (↓)
Value
0 1 2 3 4 0
500
1000
Timestep
1500
2000
2500
TimeLLM 2
Input Ground Truth Prediction Forecast Start
1
Value
0 1 2 3 4 0
500
1000
Timestep
1500
2000
2500
1.0
0.8 0.7 0.6 0.5 0.4
Diffusion-LLM
TimeLLM DiffusionLLM
0.9
100%
50%
10%
5%
Amount of Training Data Used
(a) Forecasting instance
(b) Performance under data-scarcity
Fig. 4: (a) Ultra-long-term forecasting on an ETTh1 sample (512 lookback, 2048 forecast). TimeLLM deviates faster than Diffusion-LLM in the later regions. (b) Long-term forecasting comparison under data-scarcity, showing slower degradation and greater robustness of Diffusion-LLM. Protocol follows Table 1. Table 3: Long-term forecasting results. Each cell reports (MSE, MAE) averaged over forecasting horizons H ∈ {96, 192, 336, 720}. Lower values (indicated by darker shading) are better. While our method is primarily designed to improve ultra-long-term forecasting and performance under data scarcity relative to LLM-only baselines, it is competitive with general long forecasting methods. Method
ETTh1 MSE MAE
ETTh2 MSE MAE
ETTm1 MSE MAE
ETTm2 MSE MAE
Weather MSE MAE
ECL MSE MAE
Diffusion-LLM (Ours) LDM4TS [22] GPT4TS [46] DLinear [40] PatchTST [17] TimesNet [34] FEDformer [45] Autoformer [35] Stationary [15] ETSformer [33] LightTS [41] Informer [44] Reformer [12]
0.427 0.443 0.465 0.422 0.413 0.458 0.440 0.496 0.570 0.542 0.491 1.040 1.029
0.387 0.387 0.381 0.431 0.330 0.414 0.437 0.450 0.526 0.439 0.602 4.431 6.736
0.376 0.352 0.388 0.357 0.351 0.400 0.448 0.588 0.481 0.429 0.435 0.961 0.799
0.334 0.333 0.284 0.267 0.255 0.291 0.305 0.327 0.306 0.293 0.409 1.410 1.479
0.304 0.245 0.237 0.248 0.225 0.259 0.309 0.338 0.288 0.271 0.261 0.634 0.803
0.200 0.199 0.167 0.166 0.161 0.192 0.214 0.227 0.193 0.208 0.229 0.311 0.338
0.446 0.454 0.455 0.437 0.430 0.450 0.460 0.487 0.537 0.510 0.479 0.795 0.805
0.425 0.427 0.412 0.446 0.379 0.427 0.449 0.459 0.516 0.452 0.543 1.729 2.191
0.399 0.387 0.403 0.378 0.380 0.406 0.452 0.517 0.456 0.425 0.437 0.734 0.671
0.369 0.380 0.339 0.333 0.315 0.333 0.349 0.371 0.347 0.342 0.436 0.810 0.915
0.329 0.283 0.270 0.300 0.264 0.287 0.360 0.382 0.314 0.334 0.312 0.548 0.656
0.303 0.299 0.263 0.263 0.252 0.295 0.327 0.338 0.296 0.323 0.329 0.397 0.422
Architectural Variants and Conditioning Strategies: We compared a 1D U-Net [21] with a fully connected DDPM. Despite U-Net’s capacity, the simpler architecture performed similarly or better, indicating overparameterization is unnecessary. DDPM conditions on concatenated prompt and time-series embeddings. We also tested concatenation versus attention-based conditioning and found simple concatenation most robust (A.1., A.3. in Table 5).
Diffusion-LLM for Ultra-Long-Term Forecasting Attention Over Batches during Diffusion-LLM 2048 steps inference (Ratio of attention to TS prototypes vs Generic prototypes: 1.2422x, p=0.0000)
Attention Over Batches during TimeLLM 2048 steps inference (Ratio of attention to TS prototypes vs Generic prototypes: 1.0077x, p=0.1451)
0.36
0.355
Attention Sum to Top Prototypes
Attention Sum to Top Prototypes
0.360
0.350 0.345 0.340 0.335 0.330 0.325
9
Top prototypes composed of 40 time series-related words Top prototypes composed of 40 generic words 0
10
20 30 40 ETTh1 testset batch index
(a) TimeLLM
50
60
0.35 0.34 0.33
Top prototypes composed of 40 time series-related words Top prototypes composed of 40 generic words
0.32 0.31 0.30 0.29 0
10
20 30 40 ETTh1 testset batch index
50
60
(b) Diffusion-LLM
Fig. 5: We derive the top 10 prototypes for time-series-related words and for generic words (like figure 2c) for 2048 forecast inference on ETTh1. Then we plot the attention allocated over these two types of prototypes. Diffusion-LLM shows significantly stronger attention toward time-series prototypes, indicating improved alignment between time series patches and semantic prototypes. Channel Independence: Adding feature-ID conditioning slightly degraded performance (A.1., A.2. in Table 5), suggesting DDPM benefits from shared latent representations rather than explicit channel separation. Encoder Sharing and DDPM Contribution: DDPM with separate encoders for lookback and forecast improved ultra-long-term ETTh1 forecasting by 10.81%; shared encoder added another 12.48% gain (A.1., B.1., B.2. in Table 5). Figure 6 shows the effect of λ with best performance at λ = 1, balancing LLM and DDPM contributions. Multimodal Alignment Analysis: Figure 5 shows Diffusion-LLM places noticeably stronger attention on time-series-related prototypes than TimeLLM, indicating more stable temporal–semantic alignment under extreme horizons. Efficiency Analysis: Compared to TimeLLM, Diffusion-LLM adds minimal overhead in the largest setting (2048 steps): +1.82% GPU memory, +11.54% parameters, and only 0.39% slower training (Table 8). Inference uses only LLM (Figure 3), so speed remains unchanged.
6
Conclusion
In this work, we introduced Diffusion-LLM, a low-overhead but powerful extension to LLM-based time series forecasting frameworks that integrates a conditional diffusion model for distributional regularization. Our method improves performance in ultra-long-term forecasting and few-shot learning scenarios, where uncertainty and data scarcity pose major challenges. By modeling the conditional distribution of future representations in the shared embedding space, Diffusion-LLM enhances the LLM’s ability to reason over long horizons and generalize from limited data. Promising future directions include more adaptive reprogramming strategies, applying diffusion-based regularization to other embedding spaces, and exploring diffusion for direct generative forecasting to enable uncertainty-aware multi-predictions. Extending the framework for addi-
10
F. Ghosh et al.
tional modalities or for LLM uncertainty estimation also remains an exciting direction. Diffusion-LLM offers a principled and effective enhancement to time series LLMs, combining the strengths of probabilistic modeling and pretrained language models in a unified framework without loss of existing efficiency. Acknowledgements. This research was made possible through an industry collaboration with the Audi PhD Program. We also acknowledge HPC resources from NHR@FAU (projects b143dc, b180dc), funded by federal and Bavarian state authorities and Gerhard Wellein’s and his team’s HPC approach. NHR@FAU hardware is partially funded by DFG 440719683. Additional support was received from ERC projects MIA-NORMAL 101083647, DFG 513220538 and 512819079, and the state of Bavaria (HTA). We used coding agents and LLMs from Anthropic, OpenAI, Google, and Mistral AI, for text polishing, coding, experiment orchestration, and cluster monitoring.
References 1. Benidis, K., Rangapuram, S.S., Flunkert, V., Wang, Y., Maddix, D., Turkmen, C., Gasthaus, J., Bohlke-Schneider, M., Salinas, D., Stella, L., et al.: Deep learning for time series forecasting: Tutorial and literature survey. ACM Computing Surveys 55(6), 1–36 (2022) 2. Chang, C., Wang, W.Y., Peng, W.C., Chen, T.F.: Llm4ts: Aligning pre-trained llms as data-efficient time-series forecasters. ACM Trans. Intell. Syst. Technol. 16(3) (Apr 2025) 3. Chou, J.S., Tran, D.S.: Forecasting energy consumption time series using machine learning techniques based on usage patterns of residential householders. Energy 165, 709–726 (2018) 4. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLoRA: Efficient finetuning of quantized LLMs. In: Thirty-seventh Conference on Neural Information Processing Systems (2023) 5. Dombrowski, M., Reynaud, H., Müller, J.P., Baugh, M., Kainz, B.: Trade-offs in fine-tuned diffusion models between accuracy and interpretability. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 21037–21045 (2024) 6. Gruver, N., Finzi, M., Qiu, S., Wilson, A.G.: Large Language Models Are Zero-Shot Time Series Forecasters (2023), version Number: 3 7. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020) 8. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. 43(2) (Jan 2025) 9. Hyndman, R., Athanasopoulos, G.: Forecasting: Principles and Practice. OTexts, Australia, 2nd edn. (2018) 10. Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J.Y., Shi, X., Chen, P.Y., Liang, Y., Li, Y.F., Pan, S., Wen, Q.: Time-LLM: Time series forecasting by reprogramming large language models. In: The Twelfth International Conference on Learning Representations (2024)
Diffusion-LLM for Ultra-Long-Term Forecasting
11
11. Karevan, Z., Suykens, J.A.: Transductive lstm for time-series prediction: An application to weather forecasting. Neural Networks 125, 1–9 (2020) 12. Kitaev, N., Kaiser, L., Levskaya, A.: Reformer: The efficient transformer (2020) 13. Li, T., Zhou, Z., Thelen, A., Howey, D.A., Hu, C.: Predicting battery lifetime under varying usage conditions from early aging data. Cell Reports Physical Science 5(4), 101891 (2024) 14. Liu, C., Wan, Z., Cheng, S., Zhang, M., Arcucci, R.: Etp: Learning transferable ecg representations via ecg-text pre-training. In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8230–8234 (2024) 15. Liu, Y., Wu, H., Wang, J., Long, M.: Non-stationary transformers: Exploring the stationarity in time series forecasting. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022) 16. Morid, M.A., Sheng, O.R.L., Dunbar, J.: Time series prediction using deep learning methods in healthcare. ACM Trans. Manage. Inf. Syst. 14(1) (Jan 2023) 17. Nie, Y., Nguyen, N.H., Sinthong, P., Kalagnanam, J.: A time series is worth 64 words: Long-term forecasting with transformers. In: The Eleventh International Conference on Learning Representations (2023) 18. Pacella, M., Papadia, G.: Evaluation of deep learning with long short-term memory networks for time series forecasting in supply chain management. Procedia CIRP 99, 604–609 (2021), 14th CIRP Conference on Intelligent Computation in Manufacturing Engineering, 15-17 July 2020 19. Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., dahai li, Liu, Z., Sun, M.: ToolLLM: Facilitating large language models to master 16000+ realworld APIs. In: The Twelfth International Conference on Learning Representations (2024) 20. Rasul, K., Seward, C., Schuster, I., Vollgraf, R.: Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 8857–8868. PMLR (18–24 Jul 2021) 21. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. pp. 234–241. Springer International Publishing, Cham (2015) 22. Ruan, W., Zhong, S., Wen, H., Liang, Y.: Vision-enhanced time series forecasting via latent diffusion models (2025) 23. Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., tau Yih, S.W.: Trusting your evidence: Hallucinate less with context-aware decoding (2023) 24. Shukor, M., Cord, M.: Implicit multimodal alignment: On the generalization of frozen LLMs to multimodal inputs. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024) 25. Sun, C., Li, H., Li, Y., Hong, S.: TEST: Text prototype aligned embedding to activate LLM’s ability for time series. In: The Twelfth International Conference on Learning Representations (2024) 26. Talukder, S.J., Yue, Y., Gkioxari, G.: TOTEM: TOkenized time series EMbeddings for general time series analysis. Transactions on Machine Learning Research (2024) 27. Tang, H., Zhang, C., Jin, M., Yu, Q., Wang, Z., Jin, X., Zhang, Y., Du, M.: Time series forecasting with LLMs: Understanding and enhancing model capabilities 26(2), 109–118 (2025)
12
F. Ghosh et al.
28. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: Llama: Open and efficient foundation language models (2023) 29. Uremović, N., Bizjak, M., Sukič, P., Štumberger, G., Žalik, B., Lukač, N.: A new framework for multivariate time series forecasting in energy management system. IEEE Transactions on Smart Grid 14(4), 2934–2947 (2023) 30. Wang, X., Kang, Y., Hyndman, R.J., Li, F.: Distributed arima models for ultralong time series. International Journal of Forecasting 39(3), 1163–1184 (2023) 31. Wen, R., Torkkola, K., Narayanaswamy, B., Madeka, D.: A multi-horizon quantile recurrent forecaster (2018) 32. Wimmer, C., Rekabsaz, N.: Leveraging vision-language models for granular market change prediction. In: Proceeding of the Workshop On Multimodal AI For Financial Forecasting at Association for the Advancement of Artificial Intelligence (Muffin@AAAI) (2023) 33. Woo, G., Liu, C., Sahoo, D., Kumar, A., Hoi, S.: ETSformer: Exponential smoothing transformers for time-series forecasting (2023) 34. Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., Long, M.: Timesnet: Temporal 2dvariation modeling for general time series analysis. In: The Eleventh International Conference on Learning Representations (2023) 35. Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 22419–22430. Curran Associates, Inc. (2021) 36. Xue, H., Salim, F.D.: Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering pp. 1–14 (2023) 37. Yan, T., Zhang, H., Zhou, T., Zhan, Y., Xia, Y.: Scoregrad: Multivariate probabilistic time series forecasting with continuous energy-based generative models (2021) 38. Yang, J., Zheng, Z., Li, C., Li, Y., Song, J., Sh, Y.: Miel: Enhancing long-and ultra-long-term time series forecasting with multi-scale input and ensemble linear networks. IEEE Internet of Things Journal (2025) 39. Yu, X., Chen, Z., Ling, Y., Dong, S., Liu, Z., Lu, Y.: Temporal data meets llm – explainable financial time series forecasting (2023) 40. Zeng, A., Chen, M., Zhang, L., Xu, Q.: Are Transformers Effective for Time Series Forecasting? Proceedings of the AAAI Conference on Artificial Intelligence 37(9), 11121–11128 (Jun 2023) 41. Zhang, T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng, S., Li, J.: Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures (2022) 42. Zhang, X., Chowdhury, R.R., Gupta, R.K., Shang, J.: Large language models for time series: A survey (2024) 43. Zhong, S., Ruan, W., Jin, M., Li, H., Wen, Q., Liang, Y.: Time-VLM: Exploring multimodal vision-language models for augmented time series forecasting. In: Forty-second International Conference on Machine Learning (2025) 44. Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: Beyond efficient transformer for long sequence time-series forecasting. In: The ThirtyFifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference. vol. 35, pp. 11106–11115. AAAI Press (2021)
Diffusion-LLM for Ultra-Long-Term Forecasting
13
45. Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 27268–27286. PMLR (17–23 Jul 2022) 46. Zhou, T., Niu, P., Wang, X., Sun, L., Jin, R.: One fits all: Power general time series analysis by pretrained LM. In: Thirty-seventh Conference on Neural Information Processing Systems (2023) 47. Zhou, Y., Yang, J., Zou, H., Xie, L.: Tent: Connect language models with iot sensors for zero-shot activity recognition (2023)
Diffusion-LLM for Ultra-Long-Term Forecasting
A
Supplementary Material
A.1
Dataset Details
i
We evaluate Diffusion-LLM on six widely-used benchmark datasets for longterm time series forecasting. These datasets span multiple domains, including energy, weather, and offer a diverse testbed for assessing the performance and generalization of our method. The ILI dataset [34] was considered but its shorter standard forecast window of H ∈ {24, 36, 48, 60} and the unavailability of enough data for ultra-long forecasting make it unsuitable for our evaluation. – ETTm1 and ETTm2: These datasets are derived from the Electricity Transformer Temperature (ETT) dataset. ETTm1 and ETTm2 contain measurements sampled every 15 minutes, with seven features including oil temperature and load. – ETTh1 and ETTh2: These datasets also come from the ETT collection but are sampled at an hourly resolution. Like ETTm1 and ETTm2, it includes seven variables, capturing environmental and operational characteristics of electric transformers. – Weather: The Weather dataset is sourced from the UCI Machine Learning Repository and contains meteorological data collected from a local weather station. It includes 21 continuous variables (e.g., temperature, humidity, pressure) recorded every 10 minutes. – ECL (Electricity Consumption Load): This dataset consists of hourly electricity consumption data from 321 clients in Europe. Dataset
Dim.
Dataset Size (Train, Val, Test)
Frequency
Domain
Task
ETTm1 ETTm2 ETTh1 ETTh2 Weather Electricity
7 7 7 7 21 321
(34465, 11521, 11521) (34465, 11521, 11521) (8545, 2881, 2881) (8545, 2881, 2881) (36792, 5271, 10540) (18317, 2633, 5261)
15 min 15 min 1 hour 1 hour 10 min 1 hour
Temperature Temperature Temperature Temperature Weather Electricity
Long-term Forecasting Long-term Forecasting Long-term Forecasting Long-term Forecasting Long-term Forecasting Long-term Forecasting
Table 4: Overview of datasets used in Diffusion-LLM. Each dataset varies in dimensionality, sampling frequency, and domain. Forecasting horizons are standardized across all datasets. For all datasets, we follow the standard data preprocessing and splitting protocols used in prior work such as PatchTST and Time-LLM (Available from the library in https://github.com/thuml/Time-Series-Library/tree/main). Specifics of the dataset are added in table 4.
A.2
Evaluation Metrics
To evaluate model performance on time series forecasting, we adopt two standard regression metrics:
ii
F. Ghosh et al.
– Mean Squared Error (MSE): This metric computes the average of the squared differences between the predicted values and the ground truth: N
MSE =
1 X (yi − ŷi )2 N i=1
A lower MSE indicates better performance and penalizes larger errors more heavily due to the squared term. – Mean Absolute Error (MAE): MAE measures the average absolute difference between predictions and actual values: N
MAE =
1 X |yi − ŷi | N i=1
MAE is more robust to outliers compared to MSE and provides an intuitive measure of forecast accuracy. A.3
Experiment Details
Model Architecture: Our model adopts a denoising diffusion probabilistic modeling (DDPM) framework for time series forecasting. The underlying structure is a lightweight residual multilayer perceptron (MLP). The model consists entirely of fully connected layers and skip connections. Let x ∈ RB×L×D denote a batch of input time series, where B is the batch size, L is the sequence length, and D is the input dimensionality. The model maps a noisy input xt to a denoised prediction x̂0 through the following components: Input and Context Projection: The input sequence is projected from D to a hidden dimension H via a linear layer. A conditioning signal (e.g., a context window or past data), also of dimension D, is mean-pooled over the temporal axis, broadcast to match the sequence length, and projected into the same hidden space. The two are summed along with a time embedding to produce the initial hidden state: h = Linearin (x) + Linearcond (repeat(mean(c))) + TimeEmbedding(t) Time Embedding: To encode the diffusion timestep t, we use a sinusoidal embedding of dimension H, similar to positional embeddings in transformers. This embedding is passed through a linear layer and ReLU activation: temb = ReLU(Lineartime (Sinusoidal(t))) This time embedding is broadcast across the temporal dimension and added to the hidden state. Class Conditioning (Optional):
Diffusion-LLM for Ultra-Long-Term Forecasting Variant
ETTh1-2048 MSE
A.1. DiffusionLLM A.2. DiffusionLLM with Class Conditioning (A2) A.3. DiffusionLLM with Complex U-Net & Attention Conditioning B.1. DDPM with Separate Lookback and Forecast Encoders B.2. Without DDPM (TimeLLM-style baseline)
0.729 0.746 0.732 0.833 0.934
iii
Table 5: Ablations on ETTh1 in predicting 2048 steps ahead (MSE reported). Best result highlighted in bold.
Task-Dataset Text Prototype Backbone Layers Input Length T LTF - ETTh1 LTF - ETTh2 LTF - ETTm1 LTF - ETTm2 LTF - Weather LTF - ECL
1000 1000 1000 1000 1000 1000
32 32 32 32 32 32
512 512 512 512 512 512
Patch Dim. dm Heads K FF Dim. df f 16 16 16 16 16 16
8 8 8 8 8 8
128 128 128 128 128 32
LR∗ Loss Batch Size Epochs 10−3 10−3 10−3 10−3 10−2 10−2
MSE MSE MSE MSE MSE MSE
16 16 16 16 64 128
50 50 100 100 100 100
Table 6: LLM hyperparameters used for each dataset in Diffusion-LLM. All models use the same LLaMA-7B backbone with frozen weights. The different features in the dataset are used as different classes for the conditional DDPM. Each class is added to the hidden representation at every timestep. Residual Blocks: The hidden representation is passed through two residual blocks, each consisting of a linear layer followed by a GELU activation and residual skip connection: h ← h + GELU(Linear(h)) Output Projection: Finally, a linear output layer maps the hidden representation back to the original input dimension: x̂0 = Linearout (h) Noise Schedule: We experiment with two types of noise schedules for the diffusion process: – Linear Schedule. A simple linear beta schedule is defined as: 1000 1000 βt = linspace · 10−4 , · 0.02, T T T where T is the total number of diffusion steps. – Cosine Schedule. We define the cosine schedule over T steps as: (t/T + s) π ᾱt+1 2 ᾱt = cos · , βt = 1 − 1+s 2 ᾱt where s is a small constant (e.g., 0.008), and βt is clipped to the range [0, 0.999] for numerical stability.
iv
F. Ghosh et al.
Hyperparameter
Value / Description
input_dim hidden_dim time_emb_dim num_classes residual_blocks activation output_proj timesteps beta_schedule sampling_timesteps objective loss_function self_conditioning parameter_count
4096 (Dimensionality of input time series patches) 512 (Hidden layer size used throughout the DDPM model) 512 (Dimensionality of sinusoidal time embedding) 0 (No class conditioning used in final version) 2 (Number of residual blocks in the DDPM architecture) GELU (Activation function used in residual blocks) Linear (Final layer to project hidden state back to input dimension) 1000 (Total number of diffusion steps) cosine (Noise schedule used for diffusion process) 1000 (Number of steps used during sampling) pred_noise (Training objective: predict added noise) MSE (Loss computed between predicted and target noise) False (Optional technique to improve sample quality; not used) ∼7M (Approximate number of parameters added by DDPM)
Table 7: DDPM hyperparameters used in Diffusion-LLM. These settings are shared across all datasets.
Model Diffusion-LLM TimeLLM
Training Time (GPU-h)
Max GPU Mem Usage (MiB)
Trainable Params (M)
Speed (s/iter)
6.437 6.461
33188 32592
6.461 6.437
0.397 0.395
Table 8: Efficiency analysis for ETTh1 forecasting 2048 steps ahead. Training time and resource usage are reported for Diffusion-LLM and TimeLLM. To avoid underestimating our baseline, for the LLM part, we use the same hyperparameters as [10] apart from Weather and Electricity dataset where we use larger batch size of 64 and 128 to accommodate computing time. The hyperparameters are listed in the table 6. For our DDPM architecture, we use same hyperparameters for all datasets. It is a residual MLP-based backbone with a hidden dimension of 512 throughout. The input and conditioning sequences, each with dimensionality 4096, are projected to the hidden space using separate linear layers. The model includes two residual blocks, each with a single linear layer followed by GELU activation and skip connection. A sinusoidal time embedding of size 512 is used, followed by a linear projection to match the hidden dimension. The output is projected back to the original input dimension via a final linear layer. Overall, the model contains six main linear layers, all operating at the hidden size of 512. The DDPM model adds only approximately 7 M parameters. Further, adding the condition into the DDPM model in different scenarios for different datasets always yielded similar results with 1-2 percent deviation only in either direction, hence in the final version, we have not used the class conditioning. The DDPM hyperparameters are listed in the table 7.
Diffusion-LLM for Ultra-Long-Term Forecasting
v
Effect of on MSE (ETTh1, Forecast Horizon = 2048)
=0.00 MSE=0.934
Interpolated MSE (piecewise linear) Observed MSE
Mean Squared Error (MSE)
0.90
0.85
0.80 =0.60 MSE=0.748 =0.75 MSE=0.738
0.75 0.0
0.2
0.4
0.6
0.8
=1.50 MSE=0.744
=1.00 MSE=0.729
Regularization Weight
1.0
1.2
1.4
Fig. 6: Impact of regularization weight (λ) on forecasting performance (MSE) for ETTh1 dataset with a 2048-step horizon. The plot shows that λ = 1 achieves the best performance (MSE = 0.729), indicating that an equal contribution from the forecasting loss and the diffusion-based regularization provides optimal balance. Smaller λ values (e.g., 0 for TImeLLM or 0.6) under-regularize the embedding space, limiting the benefit of distribution-aware alignment, while larger λ values (e.g., 1.5) overemphasize the diffusion objective, causing over-regularization and slight performance degradation. This demonstrates the importance of tuning λ to balance deterministic forecasting and probabilistic embedding refinement.