Extreme Adaptive Transformer for Time Series Forecasting
E XTREME A DAPTIVE T RANSFORMER FOR T IME S ERIES FORECASTING Sanjeev Shrestha Hui Liu Yifan Zhang Department of Computer Science, Missouri State University {ss472s, HuiLiu, YifanZhang}@MissouriState.edu
arXiv:2607.02437v1 [cs.LG] 2 Jul 2026
A BSTRACT Time series forecasting remains challenging when the underlying data contain rare but critical extreme events. This issue is particularly important in hydrologic forecasting, where streamflow distributions are often highly skewed and extreme peaks can have substantial impacts on flood monitoring, water resource management, and early warning systems. Although Transformer-based forecasting models have achieved strong performance by modeling long-range temporal dependencies, they typically treat all time points uniformly and may therefore underrepresent rare extreme patterns. In this paper, we propose the Extreme-Adaptive Transformer (Exformer), a forecasting framework designed to explicitly model temporal dependencies involving both normal and extreme events. Exformer introduces an extreme-adaptive attention mechanism composed of three sparse components: Local, Stride, and Extreme. The Local and Stride components capture short-term and periodic temporal dependencies, respectively, while the Extreme component selectively models event-aware dependencies between normal and extreme streamflow patterns. Experiments on four real-world hydrologic streamflow datasets show that Exformer achieves superior 3-day forecasting performance compared with state-of-the-art baselines. Our findings demonstrate that explicitly incorporating extreme-aware attention improves the forecasting capacity of Transformer models on imbalanced time series with rare but consequential events. Code is available at https://github.com/sanzexstha/Exformer.
1
I NTRODUCTION
Multivariate time series (MTS) forecasting plays a crucial role in many real-world domains, including hydrology (Li & Anastasiu, 2025), transportation (Lv et al., 2015), finance (Mohan et al., 2019), and environmental monitoring (Hewage et al., 2021). In long-term time series forecasting, extreme events are infrequent but particularly important. For example, sudden streamflow peaks caused by heavy rainfall occur rarely but can substantially affect forecasting performance and real-world decision-making. Consequently, accurately predicting streamflow under extreme conditions is critical for water resource management, flood warning, drought monitoring, and public safety. However, forecasting extreme events remains challenging because extreme conditions provide substantially fewer observations than normal conditions. Traditional statistical methods, such as autoregressive models, moving average models, exponential smoothing, and ARIMA-based variants, have been widely used for time series forecasting (Box & Pierce, 1970). Although these methods can be effective for relatively stable series, they often struggle to model nonlinear temporal dependencies, high variance, and extreme values. More recently, deep learning models, including recurrent neural networks (Lai et al., 2018), hybrid architectures (Oreshkin et al., 2020), and graph neural networks (Wu et al., 2020), have been developed to improve forecasting performance by learning complex temporal representations from data. Among them, Transformerbased models have been widely studied for time series forecasting because of their ability to capture long-range dependencies (Wen et al., 2023). Nevertheless, general forecasting models can still perform poorly on highly skewed data, particularly when the target series contains rare extreme events that are underrepresented relative to normal observations. 1
Extreme Adaptive Transformer for Time Series Forecasting
A key issue in extreme-event forecasting is the imbalance between normal and extreme observations. Since most time points correspond to normal values, a forecasting model can obtain low average error while still failing to capture the most important peaks. This limitation is especially problematic in hydrologic forecasting, where extreme streamflow values may correspond to flood-related events. Previous extreme-adaptive approaches have attempted to address this issue by designing models that place greater emphasis on extreme values. DAN (Li et al., 2024) introduced a polar representation learning framework that separately models far and near representations and uses a distance-weighted multi-loss objective to improve robustness to extreme events. PFformer (Li & Anastasiu, 2025) further improved multivariate streamflow forecasting by using position-free embedding strategies, including Enhanced Feature-based Embedding and Auto-Encoder-based Embedding, to better capture dependencies between streamflow and auxiliary variables such as rainfall. Despite these advances, existing attention mechanisms are generally token-agnostic with respect to event severity and therefore do not condition query-key selection on whether a token corresponds to a normal or extreme event. Standard full attention computes interactions over all query-key pairs, resulting in quadratic complexity and potentially introducing many low-relevance interactions that dilute attention to rare extreme-event patterns. Sparse attention mechanisms, such as local or stride attention (Zhang et al., 2024c), reduce computational cost by restricting the attention pattern, but they do not explicitly preserve or model dependencies among extreme-event tokens. To address the aforementioned limitations, we propose Exformer, an Extreme-Adaptive Transformer for multivariate time series forecasting. Exformer introduces an extreme-adaptive attention mechanism that dynamically adjusts the candidate key set according to whether a query corresponds to a normal or extreme time step. For normal time steps, Exformer employs Local and Stride attention components (Zhang et al., 2024c) to capture short-range temporal patterns and periodic dependencies among normal observations. For extreme time steps, Exformer incorporates an Extreme Attention component that enables each extreme query to selectively attend to other extreme keys. By integrating these components, Exformer reduces redundant query-key computations while emphasizing the temporal patterns most relevant to extreme-event forecasting. Our main contributions are summarized as follows: • We introduce an Extreme-Adaptive Attention mechanism composed of Local, Stride, and Extreme components. Unlike static sparse attention mechanisms, the proposed mechanism adaptively selects query-key interactions according to whether each query corresponds to a normal or extreme input token, enabling the model to capture both regular temporal dependencies and rare extreme-event patterns while reducing the computational cost. • We propose Exformer, an encoder-only Transformer framework for long-term time series forecasting. By incorporating Extreme-Adaptive Attention, Exformer provides a lightweight yet effective architecture for modeling imbalanced time series data. • We conduct experiments on four real-world hydrologic streamflow datasets and compare Exformer with recent state-of-the-art baselines. The results demonstrate that Exformer achieves the best results in most RMSE and MAPE comparisons while reducing attention computation compared with full-attention baselines. In addition, ablation studies verify the effectiveness of the proposed Extreme-Adaptive Attention mechanism.
2
R ELATED W ORK
2.1
T IME S ERIES F ORECASTING
Time series forecasting has been extensively studied using both statistical and deep learning methods. Classical approaches, including autoregressive models, moving average models, exponential smoothing, VAR, and ARIMA-based variants, provide simple and interpretable forecasting frameworks. However, these methods typically rely on linear assumptions and often have limited capacity to model nonlinear temporal dependencies, complex multivariate interactions, and highly skewed distributions. In hydrologic forecasting, hybrid methods that combine decomposition techniques with ARIMA-based models have also been explored for long-term streamflow prediction, but they are not specifically designed to capture rare yet critical extreme values. 2
Extreme Adaptive Transformer for Time Series Forecasting
Deep learning methods have become increasingly popular for time series forecasting because of their ability to learn nonlinear temporal representations from data. Recurrent neural networks have been used to model sequential dependencies, while convolutional and graph-based models have been applied to capture local temporal patterns and cross-variable relationships. More recently, Transformer-based models have been widely adopted for time series forecasting due to their ability to capture long-range dependencies through self-attention. Despite their promising performance, general forecasting models are often optimized for overall prediction accuracy and may underrepresent sparse but important peaks in datasets with rare and severe extreme events. This limitation motivates the development of forecasting models that explicitly account for extreme-event distributions and event-aware temporal dependencies.
2.2
E XTREME -A DAPTIVE T IME S ERIES F ORECASTING
Extreme-event forecasting is challenging because extreme values are rare, imbalanced, and often exhibit patterns that differ from those of normal observations. In hydrologic time series, this problem is amplified by high skewness and kurtosis, with most values near normal flow levels, but a small number of values corresponding to severe streamflow peaks. Several studies have explored extremeaware learning to improve forecasting in such conditions. eGRU (Zhang et al., 2024a) extends the vanilla GRU by using separate hidden states for normal and extreme events, allowing distinct temporal patterns to be captured. NEC+ (Li et al., 2023) models the distributions of normal and extreme events and trains multiple predictors to improve robustness to extreme hydrologic conditions. DAN (Li et al., 2024) further extends extreme hydrologic forecasting by learning polar representations that separate far and near hidden spaces, preserving information about extreme and normal values while using distance-weighted losses, gate control vectors, and Kruskal-Wallis sampling to handle data imbalance. PFformer (Li & Anastasiu, 2025) extends extreme-aware forecasting with a Transformer-based architecture. It introduces enhanced feature-based and auto-encoder-based embeddings to better model dependencies between streamflow and auxiliary variables such as rainfall. It also uses clustering-based oversampling and a multi-objective loss to improve performance during severe events. The Exformer differs from DAN and PFformer by that it directly focuses on the attention mechanism. Instead of relying mainly on polar representation learning or position-free embeddings, Exformer modifies the self-attention pattern based on whether the query token is normal or extreme. This allows the model to preserve extreme-to-extreme dependencies while still efficiently modeling local and periodic dependencies among normal tokens.
2.3
S PARSE ATTENTION M ECHANISMS
Transformers have achieved remarkable performance in several domains, including natural language processing (Vaswani et al., 2017), computer vision (Dosovitskiy et al., 2021), and time series analysis (Wen et al., 2023). However, the standard self-attention mechanism computes dot products between all query-key pairs, resulting in quadratic computational complexity in the input sequence length. This limits its scalability for long sequences and introduces redundant computation. To address this issue, several sparse attention mechanisms have been proposed. Longformer (Beltagy et al., 2020) restricts attention to local or dilated windows while allowing selected global tokens to attend across the sequence. BigBird (Zaheer et al., 2020) combines local, random, and global attention patterns to reduce the number of query-key interactions. Informer (Zhou et al., 2021) introduces ProbSparse attention to focus on the most informative queries. Autoformer (Wu et al., 2021) introduces a decomposition-based architecture and an Auto-Correlation mechanism to capture period-based temporal dependencies for long-term forecasting. FEDformer (Zhou et al., 2022) models time series in the frequency domain using Fourier- and Wavelet-based components, allowing it to capture global temporal patterns efficiently. Other time-series Transformer models improve efficiency through alternative mechanisms, such as patch-based representations in PatchTST (Nie et al., 2023), cross-variable modeling in Crossformer (Zhang & Yan, 2023), and variate-token representations in iTransformer (Liu et al., 2024). 3
Extreme Adaptive Transformer for Time Series Forecasting
Dozer self-attention (Zhang et al., 2024c) is a sparse attention mechanism that captures temporal dependencies through local and stride-based attention patterns. Local attention focuses on nearby time steps, while stride attention captures periodic or seasonal dependencies. However, these sparse patterns do not explicitly distinguish between normal and extreme time steps. In datasets with rare extreme events, this can be limiting because extreme observations may contain important information that should be preserved even when they are temporally distant. Extreme-Adaptive Attention builds on sparse temporal attention by introducing content-aware components. Its Local and Stride components capture regular temporal patterns, while its Extreme component allows queries corresponding to extreme tokens to selectively attend to keys associated with other extreme tokens. Unlike other static sparse attention mechanisms, the resulting attention structure can adapt to each input sequence, enabling the model to preserve rare but informative extreme-to-extreme dependencies while also reducing redundant computation.
3
M ETHOD
To address the aforementioned limitations, we propose Exformer, a forecasting framework equipped with an extreme-adaptive attention mechanism. In this section, we first introduce the encoder-only architecture of Exformer and present a step-by-step illustration of its input-to-output forecasting process. We then describe the proposed extreme-adaptive attention mechanism, which constructs sparse query-key interactions by preserving local, periodic, and extreme-event dependencies while filtering less informative attention pairs to improve forecasting accuracy and computational efficiency.
Transformer Encoder
1x1 Conv
0 0 0 1 1 0 0 1
Decompose
Inputs:
Linear
F RAMEWORK
DI Embed
3.1
Outputs:
Linear
Figure 1: The architecture of our proposed Exformer framework. Given historical multivariate observations (X ∈ RI×D ) with I input time steps and D variables, the MTS forecasting task aims to predict future values (Xpred ∈ RO×D ) over the next O time steps. Figure 1 illustrates the overall framework of Exformer. Following previous decompositionbased forecasting methods (Wang et al., 2023; Wu et al., 2021; Zhou et al., 2022), Exformer first decomposes the input sequence into seasonal and trend components, denoted as (Xs ∈ RI×D ) and (Xt ∈ RI×D ), respectively. The seasonal component is modeled by the Exformer encoder, while the trend component is forecast using a linear prediction layer. The dimension-invariant embedding layer (Zhang et al., 2024b) transforms the input MTS sequence into multi-channel feature maps while preserving the temporal and variable dimensions. It then partitions the sequence along the temporal dimension into non-overlapping patches, producing patched embeddings Xenc ∈ Rc×Nenc ×p×D , where c denotes the number of embedded feature-map channels, Nenc = ⌈I/p⌉ is the number of encoder patches, and p is the patch size. Since the proposed extreme-adaptive attention operates at the patch-token level, we assign a binary normal/extreme label to each patch. Specifically, an anomaly detection method first computes an outlier score st for each time step. A threshold τ is then used to obtain a time-step-level label ℓt ∈ 0, 1, where (ℓt = 0) denotes a normal time step and ℓt = 1 denotes an extreme time step. For each temporal patch Pm , the 4
Extreme Adaptive Transformer for Time Series Forecasting
patch-level label em is obtained by aggregating the time-step labels within the patch. The resulting patch labels are used to construct an extreme-aware mask that guides the attention mechanism to distinguish normal and extreme patch tokens. The patched embeddings Xenc ∈ Rc×Nenc ×p×D , together with their patch-level normal/extreme labels, are fed into the Exformer encoder. The patch labels are used to construct the extreme-aware attention mask, enabling the encoder to model patch-level temporal dependencies with the proposed extreme-adaptive attention mechanism instead of canonical full attention. The encoder outputs from all patches are concatenated along the temporal dimension to reconstruct the full sequence representation, which is then projected from the input length to the prediction length using a linear layer. A 1×1 convolutional layer is applied to the learned latent representations to generate predictions for the seasonal component. In parallel, the trend component is predicted by a linear layer that maps the historical trend to future time steps. The seasonal and trend predictions are then summed to obtain the final forecast Xpred ∈ RO×D . 3.2
E XTREME A DAPTIVE ATTENTION
The standard scaled dot-product attention is defined as d Q, K, V = Linear(Xenc ), (1) QK ⊤ Attention(Q, K, V ) = Softmax √ V, (2) dk where Q, K, and V denote the queries, keys, and values obtained from the embedded input sequence d of the d-th series, denoted by Xenc ∈ Rc×Nenc ×p . As in DozerAttention, we flatten the feature-map d and patch-size dimensions so that Xenc ∈ RNenc ×(c×p) , where each token corresponds to the latent representation of a patch of length p. The scaling factor dk denotes the dimensionality of the query and key vectors.
Although DozerAttention (Zhang et al., 2024c) reduces redundant computation by restricting attention to sparse temporal patterns, it does not explicitly distinguish between normal and extreme input tokens. In highly skewed hydrologic time series data, this limitation is important because extreme observations are rare but often contain more informative patterns than normal observations. To address this issue, we propose Extreme-Adaptive Attention, which consists of three sparse components: Local, Stride, and Extreme, as illustrated in Figure 2. The Local and Stride components are applied only when the query corresponds to a normal token, whereas the Extreme component is applied when the query corresponds to an extreme token. Local The Local component captures short-range temporal dependencies by allowing a query corresponding to a normal token to attend only to nearby normal keys within a predefined temporal window. Let ei , ej ∈ 0, 1 denote the patch-level label of the temporal patch at index i and j, where 0 indicates that the patch represents a normal event and 1 indicates an extreme event. Equation (3) defines the Local component as follows: Alocal i,j =
qi ∗ kj , 0,
if j ∈ {|i − j| ≤ ⌊w/2⌋}, ei = ej = 0 otherwise
(3)
Where A denotes the attention matrix, whose entries represent the production between queries and keys, and w denotes the local window size. The subscripts i and j denote the temporal index of the query and key vectors, respectively. Stride Time series data often exhibit recurring seasonal patterns. To capture these periodic dependencies, we define the Stride component, in which each normal token query attends only to normal keys at fixed temporal intervals. The Stride component is defined as: q ∗ kj , if j ∈ {|i − j| mod s = 0}, ei = ej = 0 stride Ai,j = i (4) 0, otherwise 5
Extreme Adaptive Transformer for Time Series Forecasting
1 0 0 0 0 1 0 0 0 0 0 1 0 0
1 0 0 0 0 1 0 0 0 0 0 1 0 0