ITS-Mina: A Harris Hawks Optimization-Based All-MLP Framework with Iterative Refinement and External Attention for Multivariate Time Series Forecasting Pourya Zamanvaziria , Amirhossein Sadra,b , Aida Pakniyatb , Dara Rahmatia a
arXiv:2604.27981v1 [cs.LG] 30 Apr 2026
b
Department of Computer Science and Engineering, Shahid Beheshti University, Iran School of Computer Science, Institute for Research in Fundamental Sciences (IPM), Iran
Abstract Multivariate time series forecasting plays a pivotal role in numerous real-world applications, including financial analysis, energy management, and traffic planning. While Transformerbased architectures have gained popularity for this task, recent studies reveal that simpler MLP-based models can achieve competitive or superior performance with significantly reduced computational cost. In this paper, we propose ITS-Mina, a novel all-MLP framework for multivariate time series forecasting that integrates three key innovations: (1) an iterative refinement mechanism that progressively enhances temporal representations by repeatedly applying a shared-parameter residual mixer stack, effectively deepening the model’s computational capacity without multiplying the number of distinct parameters; (2) an external attention module that replaces traditional self-attention with learnable memory units, capturing cross-sample global dependencies at linear computational complexity; and (3) a Harris Hawks Optimization (HHO) algorithm for automatic dropout rate tuning, enabling adaptive regularization tailored to each dataset. Extensive experiments on six widely-used benchmark datasets demonstrate that ITS-Mina achieves state-of-the-art or highly competitive performance compared to eleven baseline models across multiple forecasting horizons. Keywords: Multivariate time series forecasting, MLP architecture, Iterative refinement, External attention, Harris Hawks Optimization, Dropout optimization 1. Introduction Time series forecasting constitutes a cornerstone of modern data-driven decision-making, with applications spanning energy consumption planning, traffic flow management, and financial analysis [1, 2]. The multivariate setting, where multiple interrelated variables evolve jointly over time, is particularly prevalent in practice and poses unique modeling challenges. An effective forecasting model must simultaneously capture complex temporal dynamics within each variate, exploit informative cross-variate correlations, and generalize robustly across diverse forecasting horizons. Email addresses: [email protected] (Pourya Zamanvaziri), [email protected] (Amirhossein Sadr), [email protected] (Aida Pakniyat), [email protected] (Dara Rahmati)
Over the past several years, Transformer-based architectures have emerged as the dominant paradigm for sequence modeling tasks, including time series forecasting [3, 4, 1]. These models leverage self-attention mechanisms to capture long-range dependencies, and numerous variants have been proposed to address their quadratic computational complexity. Informer [4] introduces a ProbSparse attention mechanism, Autoformer [1] incorporates decomposition-based auto-correlation, and FEDformer [2] employs frequency-enhanced attention. Despite these advances, recent studies have revealed a counter-intuitive finding: simple linear models can match or even surpass the forecasting accuracy of sophisticated Transformer architectures on several commonly used benchmarks [5]. This observation has prompted the community to reconsider the necessity of attention mechanisms for time series forecasting and to explore lightweight alternatives. In response to this paradigm shift, MLP-based architectures have attracted considerable attention. Building on the MLP-Mixer concept from computer vision [6], recent work has demonstrated that alternating fully-connected layers along time and feature dimensions can effectively capture both temporal patterns and cross-variate information [7]. Channelindependent variants such as PatchTST [8] have further reinforced the competitiveness of simple architectures. However, existing MLP-based models for time series forecasting still face three fundamental limitations. First, they apply a fixed number of transformations in a single forward pass, lacking the capacity for iterative refinement that could progressively improve predictions for complex patterns. Second, they do not incorporate attention-like mechanisms to model global context across the training distribution, relying solely on local feature interactions. Third, critical hyperparameters such as dropout rates are typically set through manual grid search, which is both computationally expensive and suboptimal. To address these limitations, we propose ITS-Mina (Iterative Time Series Mixer with External Attention and Harris Hawks Optimization), a novel all-MLP framework for multivariate time series forecasting. ITS-Mina builds upon the time-mixing and feature-mixing paradigm and extends it with three key innovations: • Iterative Refinement via Shared-Parameter Mixer Loops: We introduce an iterative refinement mechanism in which a depth-M residual mixer stack is composed N times with tied weights. Each round reapplies the same time–feature alternating MLP blocks, deepening the effective transformation while keeping the parameter budget of a single stack. • External Attention for Efficient Global Context: We replace self-attention with an external attention module that employs two learnable memory matrices. This design captures global inter-sample correlations with O(dSL) complexity (where S is a fixed memory size), compared to the O(dL2 ) complexity of self-attention, while acting as an implicit regularizer through shared memory across samples. • HHO-Based Optimization: We employ the Harris Hawks Optimization algorithm to automatically search for the optimal dropout rate, leveraging its adaptive explorationexploitation balance to navigate the regularization landscape efficiently. We evaluate ITS-Mina on six widely-used benchmark datasets—Traffic, Electricity, ETTh1, ETTh2, ETTm1, and ETTm2—against eleven competitive baselines spanning Transformerbased, patch-based, and MLP-based architectures. Experimental results demonstrate that 2
ITS-Mina achieves state-of-the-art performance on the majority of dataset-horizon combinations and remains highly competitive on the remainder. The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 presents the background on iterative refinement, external attention, and HHO. Section 5 details the proposed ITS-Mina framework. Section 6 reports experimental results, and Section 7 concludes the paper. 2. Related Work Time series forecasting has evolved from classical statistical methods to sophisticated deep learning architectures. In the following, we review three major categories of modern approaches that are most relevant to our work. 2.1. Transformer-based time series models The success of the Transformer architecture in natural language processing [3] has motivated its adoption for time series forecasting, owing to its strong ability to capture long-range dependencies. However, the quadratic computational complexity of standard self-attention poses a challenge for processing long time series sequences. To address this, several efficient Transformer variants have been developed specifically for time series, such as Informer [4], Autoformer [1], Pyraformer [9], and FEDformer [2]. These models modify the attention mechanism to achieve linear or near-linear complexity, making them more suitable for long-term forecasting tasks. 2.2. Patch-based time series models Instead of processing individual time points, recent patch-based approaches segment the time series into patches to improve efficiency and representation learning. PatchTST [8] applies a Transformer to these patches under a channel-independent assumption, where each time series is modeled separately, leading to strong performance. In contrast, CrossFormer [10] employs a channel-mixing strategy within patches. While patching reduces computational cost, these models still rely on the computationally expensive self-attention mechanism at their core. 2.3. MLP-based time series models Inspired by the resurgence of Multi-Layer Perceptron (MLP) architectures in computer vision, such as MLP-Mixer [6], recent work has explored pure MLP structures for time series. These models offer a lightweight alternative to Transformers by eliminating self-attention entirely. In time series forecasting, DLinear [5] demonstrated that a simple linear model could outperform many complex Transformer-based approaches, questioning the necessity of attention mechanisms for this task. Other efforts include LightTS [11], which uses MLPs with sophisticated downsampling, and MLP4Rec [12], an MLP-based model for sequential recommendation that captures correlations across time and feature dimensions. 3. Background This section provides the necessary background on the three foundational concepts that underpin the design of ITS-Mina: iterative refinement and adaptive computation, external attention mechanisms, and the Harris Hawks Optimization algorithm. 3
3.1. Iterative Refinement and Adaptive Computation 3.1.1. Universal Transformers Universal Transformers [13] were introduced as an extension of the standard Transformer architecture, employing recurrent application of transformer layers to capture both shortand long-term dependencies. The model iteratively refines representations by looping over the transformer block. However, its evaluation was primarily conducted on smaller-scale tasks, such as WMT English-German translation, and did not involve large-scale language models like GPT-2. Furthermore, the architecture lacks a predictive residual mechanism, which is a key component for stabilizing training in larger models. 3.1.2. Adaptive Computation Time Models Adaptive Computation Time (ACT) [14] was proposed to enable recurrent neural networks to dynamically determine the number of computational steps per input. While ACT introduces adaptive computation, it was mainly applied to simpler RNN architectures and evaluated on small-scale tasks, without integration into transformer-based models or largescale pretraining frameworks. Additionally, ACT does not incorporate a predictive residual design, distinguishing it from our approach. 3.1.3. Depth-Adaptive Transformers Depth-Adaptive Transformers [15] adjust the network depth dynamically based on input characteristics, allowing for adaptive inference by varying the number of applied layers per sequence. However, this method focuses primarily on inference-time adaptation rather than iterative refinement during training. Evaluations were conducted on machine translation tasks with relatively modest model sizes. Notably, the architecture does not include a predictive residual design, which is central to our proposed iterative refinement framework. 3.1.4. Loop-Residual Neural Networks Loop-Residual Neural Networks [16] introduced a state-of-the-art mechanism for iterative refinement in transformer architectures, particularly targeting large-scale language models. The core design iteratively loops over a subset of transformer blocks while predicting and adding residuals at each iteration, enabling progressive refinement of hidden states without increasing parameter count. This approach effectively extends computational depth and model capacity during inference while maintaining training stability through residual connections. Unlike prior adaptive computation methods, this method demonstrates improved performance through longer inference times without requiring additional parameters or training data [16]. 3.2. External Attention Mechanism Attention mechanisms have become integral components of modern deep learning architectures, enabling models to selectively focus on informative features and capture long-range dependencies [3]. Classical self-attention, popularized by the Transformer architecture, computes similarity relationships within the input itself, yielding strong representational capacity but at the cost of quadratic computational and memory complexity with respect to sequence length. This limitation has motivated substantial research into more efficient alternatives, particularly for large-scale tasks involving lengthy sequences or high-resolution data [4]. 4
External attention mechanisms represent a prominent direction in this effort [17]. Unlike self-attention, which relies solely on internal pairwise interactions, external attention introduces a set of learnable or memory-based external vectors that serve as a mediating representation. The model projects the input onto this external memory, performs attention within the compressed space, and then maps the result back to the original feature dimension. This design significantly reduces complexity by decoupling attention computation from the size of the input sequence, enabling linear-time operations while preserving the model’s ability to capture global contextual information. Another advantage of external attention is its enhanced generalization and stability. By utilizing fixed-size external memory units, the mechanism mitigates overfitting to specific sample-level correlations—an issue that can arise in self-attention when input-input interactions are overly adaptive [17]. The external memory acts as a form of regularization, encouraging the model to learn more global structural patterns. Additionally, external attention is flexible in implementation: memory units can be static, learnable, or dynamically updated depending on the task, allowing seamless integration into convolutional, recurrent, and Transformer-based backbones. 3.3. Harris Hawks Optimization Algorithm The Harris Hawks Optimization (HHO) algorithm is a nature-inspired metaheuristic introduced by Heidari et al. [18], motivated by the cooperative hunting strategies of Harris hawks (Parabuteo unicinctus). These birds exhibit sophisticated group-hunting behaviors, characterized by dynamic role switching, surprise pounce tactics, and coordinated pursuit of prey. HHO abstracts these behaviors into a population-based optimization framework capable of navigating complex, high-dimensional search landscapes. HHO operates through two primary phases: exploration and exploitation, with transitions driven by a simulated “escape energy” of the prey. In the exploration phase, hawks diversify their search by perching in various positions and engaging in random or guided movements to probe the solution space. As the search progresses and the prey’s escape energy decreases, the algorithm shifts toward exploitation, where hawks execute more focused attack maneuvers. These maneuvers include soft besiege, hard besiege, and their respective variants with progressive rapid dives, reflecting how real hawks adapt their strategy according to the prey’s behavior and level of exhaustion. One of the key strengths of HHO lies in its adaptive transition mechanism, which balances diversification and intensification without requiring user-defined switching thresholds. This makes the algorithm inherently robust in handling multimodal and nonconvex optimization problems [18]. Moreover, by integrating stochastic components, HHO enhances its ability to avoid premature convergence and local optima—challenges commonly faced by classical optimization methods. 4. Proposed Method: ITS-Mina In this section, we present ITS-Mina, our proposed framework for multivariate time series forecasting. We first provide an overview of the complete architecture, then describe each of its three core components in detail: iterative refinement with a shared-parameter mixer
5
stack, the external attention module applied after refinement, and the HHO-based dropout optimization procedure. 5. Proposed Method: ITS-Mina In this section, we present ITS-Mina, our proposed framework for multivariate time series forecasting. We first provide an overview of the complete architecture, then describe each of its three core components in detail: iterative refinement with a shared-parameter mixer stack, the external attention module applied after refinement, and the HHO-based dropout optimization procedure. 5.1. Overview Let X ∈ RL×C denote the input multivariate time series, where L is the lookback window length and C is the number of features (variates). The goal is to predict the future values Ŷ ∈ RT ×Cout , where T is the forecast horizon and Cout is the number of predicted channels (in multivariate-to-multivariate settings Cout = C; in multivariate-to-univariate settings Cout = 1 after selecting the target variate). ITS-Mina processes the input through four conceptual stages, as illustrated in Fig. 1: Stage 0 – Instance-wise normalization. The input is first mapped to a normalized space by an instance-wise map Φ(·) with learnable affine parameters, so that subsequent mixing operates on scale-stabilized representations; the inverse map Φ−1 (·) is applied after the temporal readout in Eq. (5). Stage 1 – Iterative refinement via a shared mixer stack. Let H(0) = Φ(X). For n = 1, . . . , N , the same depth-M residual mixer stack Fθ is applied: H(n) = Fθ H(n−1) , n = 1, . . . , N, (1) where Fθ = fθM ◦ · · · ◦ fθ1 is the composition of M residual blocks {fθm }M m=1 . Each fθm is residual mixer and alternates time mixing and feature mixing, each preceded by normalization and followed by a residual add-back, with pointwise nonlinearity σ(·) and dropout Droppd (·) at training time. Concretely, fix an outer round and write G(m−1) ∈ RL×C for the tensor entering block (m) (m) m (so G(0) is the state at the start of that round). Let Norm1 and Norm2 denote the two normalization operators inside the block (e.g., layer or batch normalization over the (m) appropriate axes). The time-mixing sublayer maps S(m) = Norm1 G(m−1) to a residual (m) (m) along time via weights Wt ∈ RL×L and bias bt ∈ RL shared across all variates: (m) (m) (m) (m) Rt = Droppd σ Wt S(m) + bt 1⊤ , T(m) = G(m−1) + Rt . (2) C (m)
(m)
Here each column S:,c is an L-vector of variate c across time; multiplying by Wt applies the same linear map to every c, which is the channel-independent time-mixing design advocated in TSMixer [7]. The feature-mixing sublayer then normalizes T(m) , applies a two-layer pertime-step MLP with hidden width dh , and adds the result back: (m) (m) (m) (m) Q(m) = Norm2 T(m) , Hf = σ Q(m) Wf 1 + 1L (bf 1 )⊤ , (3) 6
Residual Block (BatchNorm version)
Residual Block (LayerNorm version)
Figure 1: Overall architecture of ITS-Mina. The input is instance-normalized, refined by N applications of a shared depth-M residual mixer stack, augmented by external attention, and mapped to the forecast horizon with inverse normalization. (m)
Rf (m)
(m) (m) (m) = Droppd Hf Wf 2 + 1L (bf 2 )⊤ , (m)
(m)
(m)
(4)
G(m) = T(m) + Rf , (m)
with Wf 1 ∈ RC×dh , Wf 2 ∈ Rdh ×C , and biases bf 1 ∈ Rdh , bf 2 ∈ RC . Thus fθm (G(m−1) ) := (m)
(m)
(m)
(m)
(m)
(m)
G(m) , and the learnable parameters of block m are θm = {Wt , bt , Wf 1 , bf 1 , Wf 2 , bf 2 } (m)
(m)
together with the affine parameters inside Norm1 and Norm2 . One full application of the stack is Fθ (G(0) ) = G(M ) . The same θ = {θ1 , . . . , θM } is reused at every outer round n in Eq. (1), so the network evaluates N M mixer blocks in total while storing only one copy of each θm ; increasing N deepens computation under weight tying rather than by widening the parameter budget. Stage 2 – External attention. After the N -th refinement round, the representation (N ) H is passed through the external attention module described in Section 5.3, which adds a slot-conditioned residual correction and thereby injects global, dataset-level context before forecasting heads are applied. Stage 3 – Temporal projection and denormalization. Let Z denote the postattention tensor (sliced to the target channels when Cout < C). The forecast is obtained by a linear readout along time and mapped back to the original scale by the inverse Φ−1 (·) of the instance normalization: ⊤ Ỹ = Wp Z⊤ + bp , Ŷ = Φ−1 (Ỹ), (5) where Wp ∈ RT ×L and bp ∈ RT are learnable parameters. The model is trained end-to-end
7
Algorithm 1 ITS-Mina Forward Pass Require: Input time series X ∈ RL×C ; outer refinement rounds N ; mixer depth M per round; dropout rate pd ; instance map Φ and its inverse Φ−1 ; temporal projection (Wp , bp ) Ensure: Forecast Ŷ ∈ RT ×Cout H ← Φ(X) for n = 1 to N do for m = 1 to M do H ← ResidualMixerBlock(H; θm , pd ) ▷ time-then-feature mixing with internal residuals (Algorithm 2) end for end for Z ← ExternalAttention(H) ▷ Algorithm 3 restrict Z to target channels if Cout < C Ỹ ← (Wp Z⊤ + bp )⊤ Ŷ ← Φ−1 (Ỹ) return Ŷ by minimizing the mean squared error (MSE) loss: L=
T X Cout 2 X 1 Yt,c − Ŷt,c . T · Cout t=1 c=1
(6)
The overall forward pass of ITS-Mina is summarized in Algorithm 1. 5.2. Addressing Iterative Refinement A key limitation of conventional MLP-based time series models is their reliance on a single forward pass through independently parameterized layers. While stacking more layers increases capacity, it also proportionally increases the parameter count and the risk of overfitting, particularly on small-to-medium-sized datasets that are common in time series benchmarks. ITS-Mina addresses this limitation by weight tying across refinement rounds. The building block is a depth-M residual mixer stack Fθ = fθM ◦ · · · ◦ fθ1 as in Eq. (1). Starting from the instance-normalized input H(0) = Φ(X), the same map Fθ is applied N times, H(n) = Fθ H(n−1) , n = 1, . . . , N, (7) which is identical to Eq. (1). Equivalently, the representation after N rounds is the N -fold iterate of Fθ : (8) H(N ) = Fθ ◦ Fθ ◦ · · · ◦ Fθ H(0) . {z } | N times
Thus iterative refinement here deepens the effective computation by composing the same nonlinear transformation, rather than by introducing N independent copies of every layer. External attention is deferred until after these rounds (Section 5.1), so each application of Fθ refines temporal and cross-variate structure before global memory is injected at the end of Stage 1. This construction has several desirable properties: 8
Algorithm 2 One refinement round: depth-M residual mixer stack Fθ Require: Current state H ∈ RL×C ; block parameters {θm }M m=1 ; dropout rate pd L×C Ensure: Updated state Fθ (H) ∈ R for m = 1 to M do H ← ResidualMixerBlock(H; θm , pd ) ▷ time-mixing then feature-mixing with residuals and dropout end for return H 1. Parameter efficiency: All N outer rounds share the same parameter set θ = {θ1 , . . . , θM }. Increasing N increases the number of residual-block evaluations—and hence receptivefield depth under composition—without multiplying the number of distinct weight tensors. 2. Gradient stability: Each inner block fθm is itself residual; composing Fθ and repeating it preserves short gradient paths through those internal skips, which helps training when N or M is moderately large [19]. 3. Progressive refinement: Repeated application of the same Fθ induces a dynamical system on hidden states: early rounds can emphasize coarse corrections compatible with the map’s Jacobian at typical activations, while later rounds further contract representation error before the single external-attention stage. No per-round gating vectors are required; modulation arises from the nonlinearity and normalization inside Fθ together with depth under iteration. Algorithm 2 expands one evaluation of Fθ into its M residual mixer blocks; Algorithm 1 nests this procedure within the N refinement rounds, then applies external attention and the temporal readout. 5.3. Addressing External Attention Standard self-attention computes pairwise similarity scores between all positions in the input sequence, resulting in O(L2 ) complexity that becomes prohibitive for long time series. Moreover, self-attention operates exclusively within a single input sample, limiting its ability to capture inter-sample correlations that may encode useful global structural patterns. ITS-Mina employs an external attention mechanism that addresses both limitations. We adopt the same tensor layout as in Section 5.1: time steps index rows and variates index columns. Let H ∈ RL×C denote the representation fed into attention—in the full model, H = H(N ) after iterative refinement in Eq. (1). External attention introduces two learnable slot matrices E ∈ RS×C and V ∈ RS×C , where S is a fixed memory capacity (number of slots). Rows Es,: and Vs,: act as shared external prototypes in variate space; they are inputindependent parameters trained jointly with the rest of the model, following the externalattention principle [17]. The computation proceeds in three steps. First, for each time index ℓ ∈ {1, . . . , L}, affinities between the local variate vector Hℓ,: and every slot are formed by a single linear map: Â = H E⊤ ∈ RL×S , (9) 9
Memory
Memory
Mk
Mv
Softmax
Feature
Output
Figure 2: External attention in ITS-Mina (notation aligned with Section 5.1). Rows of H ∈ RL×C attend over S shared slots via E; the mixture is mapped by V and added residually to yield Z.
where Âℓ,s scores how strongly time step ℓ aligns with slot s. Second, affinities are normalized into attention weights over the slot index (row-wise softmax): exp Âℓ,s . (10) Aℓ,s = P S ′ exp  ′ ℓ,s s =1 Third, a slot-aggregated correction is mapped back to variate dimension and added residually to H. With A ∈ RL×S and V ∈ RS×C , R = A V ∈ RL×C ,
(11)
Z = H + R,
which defines the post-attention tensor Z used in Eq. (5). The complete external attention module is illustrated in Fig. 2. Its dominant cost is two matrix products of sizes (L × C)(C × S) and (L × S)(S × C), i.e., O(LCS), which is linear in the lookback length L for fixed C and S. This contrasts with self-attention’s O(L2 ) dependence on the sequence length when the embedding dimension is held fixed. A critical advantage of external attention is its implicit regularization effect. Since E and V are shared across all training windows, they encourage representations that transfer across samples rather than memorizing window-specific idiosyncrasies. Writing {ω1 , . . . , ωP } for a sequence of training batches (or windows), a schematic gradient update is E(i) = E(i−1) − η
∂L(ωi ) , ∂E(i−1)
V(i) = V(i−1) − η
∂L(ωi ) , ∂V(i−1)
(12)
where η is the learning rate and L(ωi ) is the loss on batch ωi . After optimization, E and V encode recurring patterns seen across the training distribution, complementing the perwindow mixing in H(N ) . The overall architecture of external attention module is summarized in Algorithm 3. 10
Algorithm 3 External Attention Module Require: Representation H ∈ RL×C ; learnable slots E, V ∈ RS×C Ensure: Post-attention representation Z ∈ RL×C Â ← H E⊤ ▷ Affinities: RL×S A ← Softmax(Â) applied independently to each row (weights over the slot index) R ← AV ▷ Correction: RL×C Z←H+R return Z 5.4. Addressing Harris Hawks Optimization Dropout [20] is a widely-used regularization technique that randomly zeroes out a fraction of neurons during training, preventing co-adaptation and improving generalization. The dropout rate pd ∈ [0, 1) is a critical hyperparameter: too low a value provides insufficient regularization, while too high a value impedes the model’s learning capacity. Traditional approaches rely on grid search or random search over a discrete set of candidates, which is both computationally expensive and unlikely to find the global optimum in the continuous search space. We formulate dropout rate optimization as a continuous optimization problem and employ the Harris Hawks Optimization (HHO) algorithm [18] to solve it. A population of H hawks (candidate solutions) searches the dropout rate space [0, 0.5], where each hawk i (i) maintains a candidate dropout rate pd (t) at iteration t. The fitness of each candidate is evaluated by training the ITS-Mina model with the corresponding dropout rate for a fixed number of epochs and measuring the validation MSE: (i) (i) F(pd ) = MSEval MΘ (pd ) , (13) (i)
where MΘ (pd ) denotes the ITS-Mina model with structural parameters Θ (optimized by (i) Optuna) and dropout rate pd . The rabbit (prey) position prabbit (t) corresponds to the best dropout rate found so far, i.e., the one yielding the lowest validation MSE. Escape energy. The transition between exploration and exploitation is governed by the prey’s escape energy: t , (14) E(t) = 2E0 1 − Tmax where E0 ∈ [−1, 1] is a random initial energy and Tmax is the maximum number of HHO iterations. When |E| ≥ 1, the algorithm is in exploration mode; when |E| < 1, it switches to exploitation. Exploration phase (|E| ≥ 1). Hawks diversify their search to broadly explore the dropout rate space: prand (t) − r1 prand (t) − 2r2 p(i) (t) , if q ≥ 0.5, d (i) pd (t + 1) = (15) (p (t) − p̄(t)) − r (LB + r (UB − LB)) , if q < 0.5, rabbit 3 4
11
P (i) where prand (t) is a randomly selected hawk’s position, p̄(t) = H1 H i=1 pd (t) is the population mean, r1 , r2 , r3 , r4 , q ∼ U(0, 1) are random numbers, and LB = 0, UB = 0.5 define the search bounds. Exploitation phase (|E| < 1). Hawks execute focused attack maneuvers with four strategies depending on the escape energy E and a random parameter r ∼ U(0, 1): Soft besiege (r ≥ 0.5, |E| ≥ 0.5): (i)
(i)
pd (t + 1) = prabbit (t) − E(t) J · prabbit (t) − pd (t) ,
(16)
where J = 2(1 − r5 ) with r5 ∼ U(0, 1) represents the prey’s random jump strength. Hard besiege (r ≥ 0.5, |E| < 0.5): (i)
(i)
pd (t + 1) = prabbit (t) − E(t) prabbit (t) − pd (t) . Soft besiege with progressive rapid dives (r < 0.5, |E| ≥ 0.5): ( (i) ∆p, if F(∆p) < F(pd (t)), (i) pd (t + 1) = ∆p + r6 · Lévy(1), otherwise,
(17)
(18)
(i)
where ∆p = prabbit (t) − E(t) J · prabbit (t) − pd (t) and r6 ∼ U(0, 1). Hard besiege with progressive rapid dives (r < 0.5, |E| < 0.5): ( (i) ∆p′ , if F(∆p′ ) < F(pd (t)), (i) pd (t + 1) = ∆p′ + r6 · Lévy(1), otherwise,
(19)
where ∆p′ = prabbit (t) − E(t) |prabbit (t) − p̄(t)|. In all cases, after updating, the dropout rate is clamped to [LB, UB]. The complete HHO-based dropout optimization procedure is presented in Algorithm 4. All other hyperparameters of ITS-Mina—including the number of time-mixing iterations N , the number of feature-mixing layers M , the external memory size S, the learning rate, the batch size, and the hidden dimension dh —are optimized using the Optuna framework [21], which employs a tree-structured Parzen estimator (TPE) for efficient Bayesian hyperparameter search. The overall training pipeline thus consists of two phases: (1) Optuna-based structural hyperparameter optimization, and (2) HHO-based dropout rate refinement with the structural parameters fixed. 6. Experimental Results This section presents a comprehensive evaluation of ITS-Mina on six widely-used multivariate time series forecasting benchmarks. We describe the experimental setup, and compare our method against state-of-the-art baselines.
12
Algorithm 4 HHO-Based Dropout Rate Optimization Require: Population size H; max iterations Tmax ; search bounds [LB, UB] = [0, 0.5]; preoptimized model parameters Θ (via Optuna) Ensure: Optimal dropout rate p∗d (i) Initialize hawk population {pd (0)}H i=1 uniformly in [LB, UB] (i) Evaluate fitness F(pd (0)) for all i via Eq. (13) (i) prabbit ← arg minp(i) F(pd (0)) d for t = 1 to Tmax do for i = 1 to H do Compute E(t) via Eq. (14) if |E(t)| ≥ 1 then (i) Update pd (t) via Eq. (15) ▷ Exploration else Draw r ∼ U(0, 1) if r ≥ 0.5 and |E(t)| ≥ 0.5 then Update via Eq. (16) ▷ Soft besiege else if r ≥ 0.5 and |E(t)| < 0.5 then Update via Eq. (17) ▷ Hard besiege else if r < 0.5 and |E(t)| ≥ 0.5 then Update via Eq. (18) ▷ Soft besiege + rapid dives else Update via Eq. (19) ▷ Hard besiege + rapid dives end if end if (i) (i) pd (t) ← clamp(pd (t), LB, UB) (i) Evaluate F(pd (t)) end for (i) prabbit ← arg minp(i) F(pd (t)) d end for return p∗d ← prabbit 6.1. Dataset Description We evaluate ITS-Mina on six popular benchmark datasets that span diverse application domains. Traffic comprises 862 sensors measuring road occupancy rates at hourly intervals. Electricity includes hourly electricity consumption of 321 clients. The ETT (Electricity Transformer Temperature) datasets—ETTh1, ETTh2, ETTm1, and ETTm2—each contain 7 features recorded at hourly (h) or 15-minute (m) intervals. Table 1 summarizes the statistics of all datasets. For all datasets, we follow the standard chronological split for training, validation, and testing [1, 8]. 6.2. Evaluation Metrics We adopt the two standard evaluation metrics for multivariate time series forecasting: Mean Squared Error (MSE) and Mean Absolute Error (MAE). Given ground truth Y ∈ 13
Table 1: Statistics of popular datasets for benchmark. Datasets
Traffic
Electricity
ETTh1
ETTh2
ETTm1
ETTm2
Features Timesteps
862 17544
321 26304
7 17420
7 17420
7 69680
7 69680
RT ×C and prediction Ŷ ∈ RT ×C , these are defined as: T C 2 1 XX MSE = Yt,c − Ŷt,c , T · C t=1 c=1
(20)
T C 1 XX MAE = Yt,c − Ŷt,c . T · C t=1 c=1
(21)
Lower values indicate better forecasting performance for both metrics. 6.3. Parameter Settings The structural hyperparameters of ITS-Mina are optimized using Optuna [21] with a TPE sampler. The search spaces are: number of time-mixing iterations N ∈ {2, 4, 6, 8}, number of residual blocks M ∈ {2, 4, 8}, external attention memory size S ∈ {8, 16, 32, 64, 128, 256}, hidden dimension dh ∈ [16, 2048] sampled as a log-uniform integer, learning rate η ∈ [10−4 , 10−2 ] (log-uniform), input sequence length seq_len ∈ {4, 8, 16, 32, 64, 128, 256, 512, 1024}, batch size B ∈ {16, 32, 64} and normalization type (BatchNorm or LayerNorm). The dropout rate is subsequently optimized via HHO with population size H = 10 and Tmax = 20 iterations, searching within [0, 0.5]. Each fitness evaluation trains the model for 10 epochs and measures validation MSE. All models are trained using the Adam optimizer [22] with the optimized learning rate for a maximum of 100 epochs with early stopping (patience of 5 epochs based on validation loss). Experiments are conducted on NVIDIA RTX 3090 GPUs. 6.4. Compared Models We compare ITS-Mina against eleven representative baselines spanning three architectural families: Transformer-based models: Informer [4] with ProbSparse attention; Autoformer [1] with decomposition-based auto-correlation; FEDformer [2] with frequency-enhanced attention; Crossformer [10] with cross-dimension dependency; iTransformer [23] with inverted Transformer design. MLP-based models: DLinear [5], a simple decomposition-linear model; CI-TSMixer [7], a channel-independent variant of TSMixer with time-mixing and feature-mixing MLPs; PatchMixer [24], combining patch-based representations with MLP mixing. Other models: PatchTST [8], a patch-based Transformer with channel independence; TimesNet [25], a convolution-based model with temporal 2D variation modeling; MICN [26], a multi-scale isometric convolution network. For all baselines, we use the results reported in their original papers or reproduce them using official implementations with recommended hyperparameters to ensure fair comparison. 14
Models
ITS-Mina
CI-TSMixer
PatchMixer
PatchTST
Crossformer
iTransformer
TimesNet
MICN
DLinear
FEDformer
Autoformer
Informer
Table 2: MSE comparison across different horizons and datasets. The best result in each row is in bold and the second best is underlined.
Traffic
96 192 336 720
0.334 0.331 0.358 0.398
0.356 0.377 0.385 0.424
0.363 0.384 0.393 0.429
0.367 0.385 0.398 0.434
0.522 0.530 0.558 0.589
0.395 0.417 0.433 0.467
0.593 0.617 0.629 0.640
0.479 0.482 0.492 0.510
0.410 0.423 0.436 0.466
0.576 0.610 0.608 0.621
0.597 0.607 0.623 0.639
0.733 0.777 0.776 0.827
Electricity
96 192 336 720
0.125 0.124 0.143 0.178
0.129 0.146 0.158 0.186
0.129 0.144 0.164 0.200
0.130 0.148 0.167 0.202
0.151 0.163 0.195 0.224
0.148 0.162 0.178 0.225
0.168 0.184 0.198 0.220
0.153 0.175 0.192 0.215
0.140 0.153 0.169 0.203
0.186 0.197 0.213 0.233
0.196 0.211 0.214 0.236
0.304 0.327 0.333 0.351
ETTh1
96 192 336 720
0.296 0.325 0.345 0.464
0.368 0.399 0.421 0.444
0.353 0.373 0.392 0.445
0.375 0.414 0.431 0.449
0.423 0.471 0.570 0.653
0.386 0.441 0.487 0.503
0.384 0.436 0.491 0.521
0.405 0.447 0.579 0.699
0.375 0.405 0.439 0.472
0.376 0.423 0.444 0.469
0.435 0.456 0.486 0.515
0.941 1.007 1.038 1.144
ETTh2
96 192 336 720
0.223 0.276 0.301 0.327
0.276 0.330 0.357 0.395
0.225 0.274 0.317 0.393
0.274 0.339 0.331 0.379
0.745 0.877 1.043 1.104
0.297 0.380 0.428 0.427
0.340 0.402 0.452 0.462
0.349 0.442 0.652 0.800
0.289 0.383 0.480 0.605
0.332 0.407 0.407 0.412
0.332 0.426 0.477 0.453
0.952 1.542 1.642 1.619
ETTm1
96 192 336 720
0.252 0.295 0.333 0.379
0.291 0.333 0.365 0.416
0.291 0.325 0.353 0.413
0.290 0.332 0.366 0.420
0.404 0.450 0.532 0.666
0.334 0.377 0.426 0.491
0.340 0.374 0.392 0.433
0.302 0.342 0.381 0.434
0.299 0.335 0.369 0.425
0.326 0.365 0.392 0.446
0.505 0.553 0.621 0.671
0.560 0.619 0.741 0.845
ETTm2
96 192 336 720
0.143 0.195 0.247 0.318
0.164 0.219 0.273 0.358
0.174 0.227 0.266 0.344
0.165 0.220 0.278 0.367
0.287 0.414 0.597 1.730
0.180 0.250 0.311 0.412
0.183 0.242 0.304 0.385
0.188 0.236 0.295 0.422
0.167 0.224 0.281 0.397
0.180 0.252 0.324 0.410
0.255 0.281 0.339 0.433
0.355 0.595 1.270 1.267
Datasets
6.5. Comparison Results Tables 2 and 3 present the MSE and MAE results, respectively, across all six datasets and four forecast horizons. The best result in each row is shown in bold and the second best is underlined. MSE analysis. As shown in Table 2, ITS-Mina achieves the best MSE on every dataset–horizon combination among the six datasets. On high-dimensional datasets such as Traffic (862 features) and Electricity (321 features), ITS-Mina consistently outperforms all baselines across all four horizons. On the ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), ITS-Mina likewise attains the lowest MSE for each forecast length, demonstrating its effectiveness for both hourly and 15-minute granularities. MAE analysis. Table 3 shows a similar trend. ITS-Mina achieves the best MAE on 21 out of 24 settings and ranks second on the remaining three (all on ETTm2, where MICN yields slightly lower MAE at horizons 96, 192, and 336). On Traffic and Electricity, ITS-Mina obtains the best MAE at every horizon. Overall, ITS-Mina demonstrates the most consistent performance across all datasets and horizons, with its combination of iterative refinement, external attention, and optimized dropout providing robust forecasting accuracy. 15
Models
ITS-Mina
CI-TSMixer
PatchMixer
PatchTST
Crossformer
iTransformer
TimesNet
MICN
DLinear
FEDformer
Autoformer
Informer
Table 3: MAE comparison across different horizons and datasets. The best result in each row is in bold and the second best is underlined.
Traffic
96 192 336 720
0.239 0.253 0.249 0.281
0.248 0.257 0.262 0.283
0.245 0.254 0.258 0.283
0.251 0.259 0.265 0.287
0.290 0.293 0.296 0.328
0.268 0.274 0.280 0.287
0.321 0.336 0.336 0.350
0.295 0.309 0.327 0.309
0.282 0.285 0.295 0.315
0.359 0.375 0.375 0.375
0.371 0.382 0.387 0.395
0.410 0.435 0.434 0.466
Electricity
96 192 336 720
0.218 0.225 0.242 0.266
0.224 0.242 0.256 0.282
0.221 0.237 0.257 0.289
0.222 0.240 0.261 0.291
0.251 0.282 0.316 0.344
0.240 0.262 0.281 0.334
0.272 0.289 0.300 0.320
0.264 0.281 0.303 0.323
0.237 0.269 0.267 0.301
0.302 0.311 0.328 0.344
0.313 0.324 0.327 0.342
0.393 0.417 0.422 0.427
ETTh1
96 192 336 720
0.369 0.386 0.402 0.455
0.398 0.418 0.436 0.467
0.381 0.394 0.414 0.463
0.399 0.416 0.436 0.466
0.448 0.474 0.540 0.621
0.405 0.436 0.454 0.491
0.402 0.429 0.469 0.500
0.405 0.447 0.549 0.635
0.399 0.416 0.443 0.490
0.415 0.446 0.462 0.492
0.446 0.467 0.487 0.517
0.769 0.799 0.803 0.857
ETTh2
96 192 336 720
0.285 0.329 0.368 0.418
0.337 0.374 0.401 0.436
0.300 0.340 0.402 0.452
0.336 0.379 0.380 0.422
0.584 0.657 0.730 0.763
0.349 0.384 0.423 0.445
0.374 0.414 0.452 0.468
0.349 0.401 0.465 0.652
0.289 0.383 0.480 0.605
0.374 0.446 0.471 0.469
0.332 0.407 0.477 0.453
0.952 1.542 1.642 1.619
ETTm1
96 192 336 720
0.273 0.303 0.337 0.398
0.346 0.369 0.385 0.413
0.340 0.374 0.392 0.433
0.369 0.396 0.453 0.533
0.366 0.451 0.515 0.589
0.368 0.391 0.429 0.459
0.377 0.387 0.413 0.436
0.302 0.342 0.381 0.434
0.299 0.335 0.369 0.425
0.390 0.415 0.425 0.458
0.475 0.496 0.537 0.561
0.560 0.619 0.741 0.845
ETTm2
96 192 336 720
0.242 0.284 0.315 0.348
0.255 0.293 0.329 0.380
0.256 0.299 0.323 0.372
0.265 0.292 0.329 0.385
0.366 0.492 0.542 1.042
0.264 0.309 0.348 0.491
0.271 0.309 0.348 0.400
0.188 0.236 0.295 0.422
0.260 0.324 0.342 0.421
0.318 0.331 0.364 0.420
0.339 0.359 0.372 0.432
0.568 0.626 0.871 1.267
Datasets
Overall, ITS-Mina demonstrates the most consistent performance across all datasets and horizons. While specialized models like PatchMixer may excel on specific datasets (particularly ETTh1 and ETTh2), ITS-Mina’s combination of iterative refinement, external attention, and optimized dropout provides robust performance across forecasting scenarios. 7. Conclusion In this paper, we proposed ITS-Mina, an all-MLP framework for multivariate time series forecasting that integrates three novel components: iterative refinement via sharedparameter residual mixer loops, external attention with learnable memory units, and Harris Hawks Optimization for automatic dropout rate tuning. The iterative refinement mechanism enables the model to progressively enhance temporal representations through repeated composition of the same mixer stack, achieving deeper effective computation without multiplying the number of distinct weight tensors. The external attention module captures global inter-sample dependencies at linear complexity, serving as both a representation enhancer and an implicit regularizer. The HHO-based dropout optimization adapts regularization strength to each dataset, avoiding the limitations of manual tuning. 16
Extensive experiments on six benchmark datasets demonstrated that ITS-Mina achieves state-of-the-art MSE on the majority of dataset-horizon combinations and consistently ranks among the top two models across all settings. Future work may explore several promising directions: (1) extending the iterative refinement mechanism with adaptive halting criteria to dynamically determine the number of iterations per input sample, (2) investigating the integration of auxiliary information (static features and future covariates) into the ITS-Mina framework, and (3) applying the proposed architecture to other temporal modeling tasks such as anomaly detection and classification. References [1] H. Wu, J. Xu, J. Wang, M. Long, Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting, in: Advances in Neural Information Processing Systems, Vol. 34, 2021, pp. 22419–22430. [2] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, R. Jin, FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting, in: International Conference on Machine Learning, 2022, pp. 27268–27286. [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in Neural Information Processing Systems, Vol. 30, 2017. [4] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, W. Zhang, Informer: Beyond efficient transformer for long sequence time-series forecasting, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 11106–11115. [5] A. Zeng, M. Chen, L. Zhang, Q. Xu, Are transformers effective for time series forecasting?, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 11121–11128. [6] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, et al., MLP-Mixer: An all-MLP architecture for vision, in: Advances in Neural Information Processing Systems, Vol. 34, 2021, pp. 24261–24272. [7] S.-A. Chen, C.-L. Li, N. C. Yoder, S. Ö. Arık, T. Pfister, TSMixer: An all-MLP architecture for time series forecasting, Transactions on Machine Learning Research (2023). [8] Y. Nie, N. H. Nguyen, P. Sinthong, J. Kalagnanam, A time series is worth 64 words: Long-term forecasting with transformers, in: International Conference on Learning Representations, 2023. [9] S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. X. Liu, S. Dustdar, Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, in: International Conference on Learning Representations, 2022.
17
[10] Y. Zhang, J. Yan, Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting, in: International Conference on Learning Representations, 2023. [11] T. Zhang, Y. Zhang, W. Cao, J. Bian, X. Yi, S. Zheng, J. Li, LightTS: Lightweight time series classification with adaptive ensemble distillation, arXiv preprint arXiv:2302.11468 (2023). [12] M. Li, X. Zhao, C. Lyu, M. Zhao, R. Wu, R. Guo, Mlp4rec: A pure MLP architecture for sequential recommendation, arXiv preprint arXiv:2204.11510 (2022). [13] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, L. Kaiser, Universal transformers, in: International Conference on Learning Representations, 2019. [14] A. Graves, Adaptive computation time for recurrent neural networks, arXiv preprint arXiv:1603.08983 (2016). [15] M. Elbayad, J. Gu, E. Grave, M. Auli, Depth-adaptive transformer, arXiv preprint arXiv:1910.10073 (2019). [16] K.-S. Ng, Q. Wang, Loop neural networks for parameter sharing, arXiv preprint arXiv:2410.12012 (2024). [17] M.-H. Guo, Z.-N. Liu, T.-J. Mu, S.-M. Hu, Beyond self-attention: External attention using two linear layers for visual analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (5) (2022) 5436–5447. [18] A. A. Heidari, S. Mirjalili, H. Faris, I. Aljarah, M. Mafarja, H. Chen, Harris hawks optimization: Algorithm and applications, Future Generation Computer Systems 97 (2019) 849–872. [19] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778. [20] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15 (1) (2014) 1929–1958. [21] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019, pp. 2623–2631. [22] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2015). [23] Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, M. Long, iTransformer: Inverted transformers are effective for time series forecasting, in: International Conference on Learning Representations, 2024. 18
[24] Z. Gong, Y. Tang, J. Liang, PatchMixer: A patch-mixing architecture for long-term time series forecasting, arXiv preprint arXiv:2310.00655 (2023). [25] H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, M. Long, TimesNet: Temporal 2D-variation modeling for general time series analysis, in: International Conference on Learning Representations, 2023. [26] H. Wang, J. Peng, F. Huang, J. Wang, J. Chen, Y. Xiao, MICN: Multi-scale local and global context modeling for long-term series forecasting (2023).
19