arXiv CSopen access

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

arXiv:2605.14938v1 [cs.LG] 14 May 2026

Yuehao Liu1 Shanyan Guan2 Weijia Zhang1 Xuanming Shang1 Yanhao Ge2 Wei Li2 Chao Ma1 * 1 MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2 vivo Mobile Communication Co., Ltd. {yuehao.liu, weijia.zhang, sxm2021, chaoma}@sjtu.edu.cn {guanshanyan, halege, liwei.yxgh}@vivo.com Project page: https://fxmangd26.github.io/Octopus/

,PDJH1HW5

Abstract



Continual learning in multimodal large language models (MLLMs) aims to sequentially acquire knowledge while mitigating catastrophic forgetting, yet existing methods face inherent limitations: architecture-based approaches incur additional computational overhead and often generalize poorly to new tasks, rehearsal-based methods rely on storing historical data, raising privacy and storage concerns, and conventional regularization-based strategies alone are insufficient to fully prevent parameter interference. We propose Octopus, a two-stage continual learning framework based on History-Free Gradient Orthogonalization (HiFGO), which enforces gradient-level orthogonality without historical task data. Our proposed two-stage finetuning strategy decouples task adaptation from regularization, achieving a principled balance between plasticity and stability. Experiments on UCIT [20] show that Octopus establishes state-of-the-art performance, surpassing prior SOTA by 2.14% and 6.82% in terms of Avg and Last.

1. Introduction Continual learning [5, 49, 53, 58, 62, 65, 73], sequentially learning across multiple task without forgetting, allows multimodal large language models (MLLMs) [39, 68] to incrementally integrate knowledge across tasks, thereby exhibiting human-like adaptability when encountering novel scenarios. Catastrophic forgetting [1, 13, 15, 25, 46, 61, 66] constitutes the fundamental challenge in continual learning, referring to the degradation of previously acquired knowledge as a model adapts to new tasks. Current continual learning approaches for MLLMs can * Corresponding author.

)OLFNUN

 







  

&/(950DWK

 

  

 

 







$U;LY4$

 9L]:L]

  ,FRQ4$

)LQH7XQH /Z)

(:& /3

2/R5$ 0R(/R5$

+L'H//D9$ 2FWRSXVRXUV

Figure 1. Performance comparison between Octopus (ours) and existing approaches on UCIT [20] in terms of Last. Results demonstrate that Octopus establishes a new SOTA performance and outperforms all competing methods by a substantial margin.

be broadly classified into three categories: architecturebased, rehearsal-based and regularization-based methods. Architecture-based methods [8, 20, 26] typically assign several LoRA modules to store task-specific information, which tends to deteriorate the model’s generalization to unseen tasks and sacrificing computational efficiency during inference. Rehearsal-based methods [7, 57, 57] maintain memory modules that store historical task information, such as past data or intermediate layer activations. However, in real-world applications, access to historical task data may be impractical or raise data privacy concerns, while maintaining replay buffers incurs additional storage overhead. In contrast, regularization-based approaches [63, 74] are not

subject to the aforementioned limitations. These methods seek to constrain parameter updates within subspaces that minimally affect previously tasks to mitigate interference. The primary objective of regularization is to mitigate parameter interference, wherein the acquisition of parameters for new tasks does not compromise the performance of previously learned tasks, thereby alleviating catastrophic forgetting. primarily focus on enforcing parameter orthogonality. [63, 74] primarily focus on enforcing parameter orthogonality. However, studies in model merging [59] suggest that parameter orthogonality is insufficient to fully prevent parameter interference. We theoretically demonstrate that, beyond parameter orthogonality, it is of greater significance that gradient orthogonality must be enforced, which is consistent with prior works [14, 67]; however, existing gradient orthogonality-based methods [14, 67] remain fundamentally constrained by their reliance on historical data. To address this limitation, we propose Octopus, a two-stage continual learning framework based on historyfree gradient orthogonalization. Specifically, we propose History-Free Gradient Orthogonalization (HiFGO) characterizing sensitivity of previous task parameters within current data distribution, leveraging only past weights (instead of past data) and current task data to enforce orthogonal constraints in gradient space. Moreover, we observed in experiments that the regularization constraints tend to compete with the objectives of original task, which is consistent with studies in multi-task learning [6, 10, 69, 70]. To alleviate this competition, we propose a two-stage finetuning strategy maintaining parameters in proximity to the optimal solution under incorporation of regularization constraints. Extensive experiments on UCIT [20] benchmark demonstrate that our framework achieves state-of-the-art (SOTA), surpassing the previous SOTA [20] by 2.14% and 6.82% in terms of Avg and Last, respectively. It indicates that our proposed HiFGO effectively preserves previously learned knowledge while learning new tasks. Moreover, our twostage finetuning strategy substantially enhances the performance ceiling of regularization-based methods, allowing them to approach even exceed the performance of multitask training while preserving the efficacy of regularization, achieving effective balance between plasticity and stability.

2. Related Work Parameter-Efficient Model Adaptation. MLLMs have achieved remarkable success across diverse tasks [3, 17, 19, 43, 51, 71]; however, maintaining a separately fine-tuned model for each new task incurs prohibitive computation and storage costs. Parameter-efficient fine-tuning (PEFT) [9, 23, 28, 32, 34, 52] have been proposed to address this issue. Approaches such as Adapters [23] and AdaptFormer [9] introduce trainable modules into pretrained networks, yet they increase model complexity and fail to ensure task isolation.

Prompt-tuning [32] and Prefix-tuning [34] introduce learnable representations into into Transformer layers, but their expressiveness is limited to input-level manipulation. Besides, LoRA and its variants decompose weight updates into low-rank subspace, enabling efficient and scalable adaptation, and thus have become mainstream in MLLM finetuning, and have been certified to exhibit a lower degree of forgetting compared to full-parameter fine-tuning. Conventional Continual Learning. The central objective of continual learning is to alleviate catastrophic forgetting in sequential task learning. Classical approaches such as EWC [31] and LwF [35] address this by introducing regularization terms that penalize updates of parameters critical to previous tasks. Methods such as DER [4], DGR [54] and GEM [40] implicitly preserve knowledge from previous tasks in order to mitigate forgetting by reinforcing historical knowledge during training of new tasks. Another line of research such as Piggyback [44] tackles catastrophic forgetting through architectural adaptation, expanding base model with task-specific modules or adaptive parameters to enable sequential learning without forgetting. Continual Learning for Multi-Modal Language Models. Prompt-based methods such as L2P [65], DualPrompt [64], and CODA-Prompt [56] enhance knowledge retention by tuning soft prompt vectors without altering model weights. MoE-based approaches, including HiDe-LLaVA [20] and MoILE [29], assign task-specific experts via routing mechanisms. While effective, both incur additional inference or storage costs. Orthogonal gradient strategies like OGD [14] mitigate task interference by constraining updates to directions orthogonal to previous gradients, yet require access to past task data. Recent extensions—OLoRA [63], BiLoRA [74], and InfLoRA [36]—replace stored gradients with parameter vectors but still face limited efficacy and the inherent plasticity–stability trade-off. Octopus alleviates these limitations, achieving inference-efficient, rehearsalfree, and highly effective solution to continual learning.

3. Preliminaries 3.1. Low-Rank Adaptation Low-Rank Adaptation (LoRA) [24] introduces a parameterefficient finetuning paradigm for large-scale pre-trained models. Formally, consider a pre-trained parameter matrix W0 ∈ Rd×k . Instead of optimizing W0 , LoRA parameterizes the weight update as product of two low-rank matrices: \Delta W = B A,

(1)

where A ∈ Rr×k and B ∈ Rd×r with r ≪ min(d, k). The adapted weight is then defined as: W = W_0 + \Delta W = W_0 + B A.

(2)

+

B

Transformer Block

Task N

Question: Is the dotted line a line of symmetry? Answer the question using a single word or phrase. Answer: no.

History-Free Gradient Orthogonalization

Ai

+

Bi

Ai-1

+

Data 1 𝜕𝑳𝑫𝒊 (𝜽j ) 𝜕𝜽j

Constraint

Pretrained Param.

Pretrained Param.

… Sequential Learning

A

NO Need

Bi-1

+…+

i−1 j=1

A1

GPWC

MLLM

Task 1

STAGE 2

Prev. Data

Question: What is happening in the image? Generate a brief caption for the image. Answer: Two young guys with shaggy hair look at their hands while hanging out in the yard.

B1

STAGE 1

Transformer Block

Figure 2. Overall pipeline for continual learning in MLLMs (left) and architecture of our proposed Octopus (right). In the context of continual learning, MLLMs are required to sequentially learn multiple tasks while overcoming the challenge of catastrophic forgetting caused by non-stationary data distributions. To address this, we propose Octopus, which adopts a two-stage fine-tuning paradigm. In the first stage, MLLM learns task-specific knowledge without constraints, enabling full adaptation to current task. In the second stage, we apply History-Free Gradient Orthogonalization (HiFGO) to mitigate parameter interference, while simultaneously constraining the parameter updates within an optimal solution space, thereby maintaining a effective balanced trade-off between plasticity and stability.

tive likelihood across the entire task sequence:



6JO)7

6HT)7

6HT)7Z+L)*2

0HWULF9DOXH



\theta ^* = \arg \max _\theta \frac {1}{N} \sum _{i=1}^N \frac {1}{|\mathcal {D}_i|} \sum _{(x_k, y_k) \in \mathcal {D}_i} \log p_\theta (y_k \mid x_k), (3)

   

5 &

$ &

$ )

9 ,

Figure 3. Effectiveness of HiFGO. We present the performance of single-task finetuning, sequential fine-tuning of two tasks, and fine-tuning with HiFGO constraints added after sequential finetuning (higher is better). The abbreviations in the table represent dataset names, with details provided in Sec. 5.3.

where pθ (yk | xk ) represents conditional probability of observing label yk given input xk under the model fθ . This suggests that the goal of continual learning is to learn a unified parameter set that is generalized across all datasets, inherently balancing plasticity and stability during training process. However, in practice, it is fundamentally constrained by catastrophic forgetting (CF) [15, 46], a phenomenon in which sequential optimization on new tasks induces rapid deterioration of previously acquired knowledge.

4. Methodology During fine-tuning, only the low-rank factors A and B are updated while original weights W0 remain frozen. Owing to its efficiency and scalability, LoRA has become a popular technique for PEFT in various domains and and has been widely applied in continual learning tasks.

3.2. Continual Learning Continual Learning (CL) considers the problem of incrementally acquiring knowledge from a sequence of tasks T1 , T2 , . . . , TN while preserving robust performance on previous tasks. Let fθ denote a model parameterized by θ, and Di = (xk , yk ) denote the dataset corresponding to task Ti . From a probabilistic perspective, the objective of CL can be formulated as maximizing the expected predic-

In this work, we propose Octopus, a two-stage continual learning framework based on history-free gradient orthogonalization for MLLMs. We first provide a theoretical justification in Sec. I.1 for why the gradient directions of previous tasks intrinsically capture model’s sensitivity subspace, thereby suggesting that beyond parameter orthogonality, it is of greater significance that gradient orthogonality must be enforced to mitigate parameter interference. Building upon this insight, we introduce a history-free gradient orthogonalization in Sec. 4.2 to mitigate parameter interference, which operates without reliance on historical task data. Our method offers a effective and data-efficient solution to mitigating catastrophic forgetting in continual learning. Moreover, to mitigate the degradation in fine-tuning performance induced by parameter regularization, we intro-

duce a two-stage finetuning framework in Sec. 4.3 that allows LoRA to satisfy the imposed constraints while remaining close to optimal manifold. This design effectively balances plasticity and stability, thereby further enhancing the model’s capability for continual learning. The overall architecture of our proposed method is illustrated in Fig. 2.

4.1. Analysis on Parameter Interference We denote W0 as the pretrained parameters of MLLM, θi as the LoRA parameters after fine-tuning on Task i, Di as the corresponding training dataset for Task i, and LDi (θ) represents the expectation of loss function evaluated on Di given model parameters θ. For clarity, we first consider a sequential learning scenario involving two tasks, which can be naturally generalized to the multi-task case. Let W0 denotes the pretrained weights, θ1 and θ2 as the optimized LoRA weights of Task 1 and Task 2, respectively. Our objective is to ensure that the training on Task 2 does not degrade the performance achieved on Task 1. Formally, by defining θ1′ = W0 + θ1 , we require the following condition to hold, which we define as the lossless condition: L_{\mathcal {D}_1}{(\theta _1')} = L_{\mathcal {D}_1}{(\theta _1' + \theta _2)}.

(4)

We perform Taylor expansion around θ1′ as:

i=1

1: Initialize θ1,1 following vanilla LoRA 2: for i in 1 : N do 3: Select subset Di1 and Di2 from Di ▽ Stage-1 Finetuning 4: Finetune on Di1 using L1 (Eq. 9) and update θi,1 ▽ Stage-2 Finetuning 5: Compute all GPWC (4.2) for Task i 6: Finetune on Di2 using L2 (Eq. 10) and update θi,2 N P 7: Return W0 + θi,2 as the merged LoRA weight i=1

and highly oscillatory, thereby limiting the effectiveness of gradient-based regularization. To address this issue, we propose a history-free gradient orthogonalization paradigm that enables effective parameter disentanglement across tasks.

4.2. History-Free Gradient Orthogonalization

L_{\mathcal {D}_1}(\theta _1' + \theta _2) = L_{\mathcal {D}_1}(\theta _1') + \langle \frac {\partial L_{\mathcal {D}_1}(\theta _1')}{\partial \theta _1'}, \theta _2 \rangle + \mathcal {O}(||\theta _2||^2), \label {eq:taylor} (5) where O(||θ2 ||2 ) denotes the second- and higher-order residual terms. Since LoRA weights are typically much smaller in magnitude than pretrained weights during finetuning, O(||θ2 ||2 ) can be safely neglected. Therefore, satisfying the lossless condition reduces to: \langle \frac {\partial L_{\mathcal {D}_1}(\theta _1')}{\partial \theta _1'}, \theta _2 \rangle = 0. \label {eq:orth}

Algorithm 1 Octopus Input: Pretrained weights W0 ; Number of tasks N ; Data of different tasks {Di }N i=1 N P Output: Merged LoRA weights W0 + θi,2

(6)

This observation suggests that the orthogonality between parameters of Task 2 and gradients of Task 1 on D1 provides a principled guarantee to effectively mitigate parameter interference. In contrast, parameter orthogonality—i.e., OLoRA [63]—fails to guarantee lossless disentanglement, ∂LD1 (θ1′ ) as ∂θ and θ1 may correspond to distinct directional ′ 1 semantics. Specifically, derived through fine-tuning from pretrained weights, θ1 represents the trajectory from pretrained weights toward local optimum, rather than the instantaneous gradient direction at θ1′ . Furthermore, the update direction of θ1 may vary throughout the optimization process, implying that θ1 does not necessarily align with a consistent gradient orientation on fixed parameters. However, in practical scenarios, historical data are often difficult to obtain or restricted due to data privacy concerns. Moreover, as the parameters of previous tasks typically converge to local optima, the gradient at θ1′ tends to be weak

To achieve balance between data privacy preservation and model efficacy, we introduce a novel approach termed History-Free Gradient Orthogonalization (HiFGO). The central objective of this method is to accurately characterize the mutual influence between current and previous tasks without requiring access to any historical task data. Motivated by SD [72], which demonstrates that the representation space of each task can be decomposed into a stability-related subspace (task-shared subspace) and a plasticity-related subspace (task-specific subspace), we propose to quantify inter-task interference via Gradients of Previous parameters Within Current data distribution (GPWC). Intuitively, GPWC reflects beneficial update directions that encapsulate the reusable knowledge embedded in earlier tasks, effectively capturing the shared representational subspace across tasks. Moreover, since previous parameters have converged to local optima within their respective domain, direction of their gradients on current data distribution also reveals optimization conflicts between tasks, indicating the potential for parameter updates in the current task to degrade performance on earlier ones. To mitigate such interference, we introduce a gradient orthogonality constraint that enforces orthogonality between current parameters and GPWC. This regularization effectively disentangles parameter updates across tasks, promoting stability while maintaining adaptability. Concretely, during fine-tuning of Task i, we incorporate the following

Avg Last

Method Zero-shot Multi-task Sequential Finetune Vanilla Rehearsal

Replay !

ImageNet-R 17.20 89.53 49.31 88.71

L2P [65] MoELoRA [8] HiDe-LLaVA [20]

% % %

41.52 64.49 85.70

LwF [35] EWC [31] O-LoRA [63] Octopus (ours)† Octopus (ours) Sequential Finetune Vanilla Rehearsal

% % % % % !

55.60 54.23 75.26 89.69 88.41 37.63 88.43

L2P [65] MoELoRA [8] HiDe-LLaVA [20]

% % %

32.73 49.87 80.50

LwF [35] EWC [31] O-LoRA [63] Octopus (ours)† Octopus (ours)

% % % % %

40.27 39.05 69.36 88.80 88.20

-

ArXivQA VizWiz IconQA 52.16 41.07 18.70 92.20 62.70 61.86 78.40 50.48 53.44 86.13 54.15 61.76 ▽ Architecture-based 82.32 51.98 52.21 82.42 49.54 56.87 92.70 54.10 66.87 ▽ Regularization-based 79.86 53.23 54.87 80.13 53.14 55.06 86.73 55.86 58.47 91.34 61.04 60.18 93.30 61.65 59.91 72.33 43.47 41.70 85.36 51.50 60.10 ▽ Architecture-based 80.41 43.72 42.16 77.63 43.65 46.40 89.83 48.78 62.90 ▽ Regularization-based 75.93 42.76 44.38 77.88 43.24 45.33 82.42 48.64 53.66 89.83 56.44 59.90 93.03 58.46 60.50

CLEVR-Math 18.63 70.40 55.53 70.53

Flickr30k 46.99 58.49 57.95 58.13

Average 32.46 72.53 57.52 69.90

43.16 56.35 59.12

52.77 58.34 55.15

53.99 61.33 68.94

56.51 57.52 57.38 68.09 66.32 35.63 67.13

56.34 55.94 53.52 57.61 56.87 57.95 58.13

59.40 59.34 64.54 71.33 71.08 48.12 68.44

39.25 36.47 47.97

52.77 58.34 55.15

48.51 52.06 64.19

37.43 39.72 42.53 70.60 69.00

56.34 55.94 53.52 57.11 56.87

49.52 50.20 58.36 70.45 71.01

Table 1. Comparison with various methods on UCIT [20] in terms of Avg and Last. The best and second methods are labeled with bold and underline styles. Zero-shot evaluates pretrained model without finetuning. Multi-task jointly finetunes model across all datasets, whereas Sequential Finetune adapts only one LoRA module sequentially to all tasks. These settings provide an empirical characterization of the lower bound, upper bound, and baseline for continual learning methods. † denotes the use of Historical task proxy approximation in Eq. 8

orthogonality loss term into the optimization objective: L_{orth}(\theta _i) = \sum \limits _{j = 1}^{i - 1}\left (\frac {\partial L_{\mathcal {D}_i}(\theta _j')}{\partial \theta _j'}\right )^T\theta _i, \label {orth-loss}

where θj′ = W0 +

j P

(7)

θm is denoted as the merged LoRA

causes the training cost to grow linearly with the number of tasks. To address this issue, we introduce a lightweight approximation based on a valid proxy of historical tasks parameters. Instead of computing the constraint with respect ′ to all historical parameters, we use the parameter θi−1 from the most recent task as a proxy representing the entire task history, as:

m=1

weights of Task j, while LDi (θj′ ) is denoted as the loss function of parameter θj′ on Di We evaluate the efficacy of HiFGO through a controlled sequential fine-tuning protocol. We sequentially finetune an MLLM on two tasks (named Task 1 and Task 2),which would induce substantial degradation on Task 1 in comparison to single-task finetuning. Then we further finetune MLLM on Task 2 for a few steps with HiFGO constraint. We evaluate the performance of Task 1, as shown in Fig. 3. The results show that sequential finetuning markedly degrades prior-task performance, whereas introducing HiFGO nearly restores it to the level of single-task finetuning, which demonstrate that HiFGO effectively suppresses parameter interference and preserves previously acquired knowledge. Historical task proxy approximation. In Eq. 7, the parameters of the current task are required to compute the inner product with the GPWC of each historical task, which

L_{orth}'(\theta _i) = \left (\frac {\partial L_{\mathcal {D}_i}(\theta _{i-1}')}{\partial \theta _{i-1}'}\right )^T\theta _i \label {new_orth}

(8)

This approximation is motivated by the observation that a well-trained continual learning model preserves the performance of earlier tasks after each training stage, thereby implicitly encoding historical knowledge in the latest parameters. As a result, the computational cost of the orthogonal loss is reduced from O(t) to O(1).

4.3. Two-stage Finetuning Strategy Although the magnitudes of LoRA weights are typically much smaller than those of the pretrained parameters during fine-tuning, the vanilla fine-tuning paradigm can still incur non-negligible errors due to the influence of higher-order terms. To alleviate this issue, it is natural to introduce additional regularization terms that constrain the scale of LoRA

Settings Prev. Param. GPWC Prev. Param. & GPWC

Curr. Task (GPWC) Prev. Task

Imd. Last Imd. Last Imd. Last Imd. Last Imd. Last

ImageNet-R ArXivQA VizWiz IconQA CLEVR-Math ▽ Constrain current parameters orthogonal to what 88.47 91.73 60.74 56.73 61.33 86.67 92.27 50.62 57.00 57.40 88.47 93.13 61.04 60.47 63.63 88.20 93.03 58.46 60.50 69.00 88.47 93.20 60.50 60.77 63.60 88.27 92.90 59.01 60.27 68.83 ▽ Compute Gradients on what 88.47 93.13 61.04 60.47 63.63 88.20 93.03 58.46 60.50 69.00 88.47 93.53 61.53 61.67 65.70 86.50 73.43 51.47 58.57 46.40

Flickr30k

Average

56.29 56.29 56.87 56.87 56.99 56.99

69.22 66.71 70.60 71.01 70.59 71.04

56.87 56.87 56.31 57.15

70.60 71.01 71.20 62.25

BWT -2.51 +0.41 +0.45

+0.41 -8.95

Table 2. Effectiveness of History-Free Gradient Orthogonalization. We compare the performance of three orthogonalization strategies (orthogonal to param., GPWC or both) and that of two type of gradients (GPWC and gradients of prev. param. on prev. task). Our method is emphasized in bold for clarity. We report the performance on each dataset under different settings in terms of Imd. and Last.

\mathcal {L}_1 = \frac {1}{|\mathcal {D}_i|} \sum _{(x_k, y_k) \in \mathcal {D}_i} L_{ce}(f_{\theta _{i, 1}'}(x_k), y_k). \label {loss_1}

(9)

′ Here, θi,1 = W0 +θi,1 , where θi,1 is initialized from θi−1,1 . For the first task, θ1,1 is initialized from scratch. During the second training stage, we simultaneously activate both fine-tuning objective and regularization to ensure parameter updates not interfering with or degrading the performance of previously learned tasks. The loss function for Task i in this stage is defined as follows:

\begin {aligned} \mathcal {L}_2 = \frac {1}{|\mathcal {D}_i|} \sum _{(x_k, y_k) \in \mathcal {D}_i} (&L_{ce}(f_{\theta _{i, 2}'}(x_k), y_k) + \lambda _1 L_{orth}(\theta _{i, 2}) \\ &+ \lambda _2 L_{norm}(\theta _{i, 2})), \end {aligned} \label {loss_2}

(10)

  %:7

weights, thereby mitigating the impact of such higher-order effects. However, our experimental observations indicate that imposing such constraint considerably degrades the fine-tuning performance and, consequently, compromises the model’s continual learning capability. We attribute this phenomenon to two primary factors: (1) the additional constraints substantially reduce the effective parameter search space, and (2) the interference among multiple loss objectives increases the risk of the optimization process being trapped in suboptimal local minima. As a result, the optimized solution can diverge significantly from that obtained through standard fine-tuning. To address this challenge, we draw inspiration from annealing schedule [16] and propose a two-stage finetuning strategy, which enables model to first explore optimal solution in current data without constrains, and subsequently perform constrained refinement around local optimal. Specifically, during the first stage, both regularization terms are deactivated, allowing the model to freely adapt and approach a local optima region for current task. The loss function for Task i in this stage is defined as follow:

     

 

      





3DUDP *3:&

 

3DUDP *3:& 

  1XPEHURI7DVNV







Figure 4. Backward transfer (BTW) of model after fine-tuning on each task with different orthogonalization method. ′ where θi,2 = W0 +

i P

θm,2 and θi,2 is initialized by θi,1 .

m=1

Lnorm (θi ) is the L2-regularization term, and λ1 , λ2 are denoted as the weight of regularization terms. This design enforces stability across tasks by explicitly regularizing the optimization trajectory within a subspace that preserves prior knowledge. Experimental results demonstrate that the proposed two-stage finetuning paradigm effectively safeguards model’s performance, achieving a favorable trade-off between retaining past knowledge and optimizing for the current task. Our complete algorithmic procedure is summarized in Algo. 1.

5. Experiments 5.1. Experimental Setup Benchmark. To evaluate the effectiveness of Octopus, we conduct experiments on UCIT [20], which is specifically designed to assess the continual learning capability of MLLMs under realistic, instruction-driven scenarios. It consists of a sequence of multimodal instruction datasets covering diverse visual domains and linguistic tasks, including visual question answering, caption generation, and mathematical reasoning. Each task introduces novel visual

Method w/ two-stage finetuning w/o two-stage finetuning

Imd. Last Imd. Last

ImageNet-R 88.47 88.20 87.47 86.90

ArXivQA 93.13 93.03 92.53 92.20

VizWiz 61.04 58.46 48.61 45.96

IconQA 60.47 60.50 40.70 38.90

CLEVR-Math 63.63 69.00 49.90 47.53

Flickr30k 56.87 56.87 55.62 55.62

Average 70.60 71.01 62.47 61.18

BWT +0.41 -1.29

Table 3. Effectiveness of two-stage finetuning strategy. We report the performance on each dataset under both w/ and w/o two-stage fine-tuning settings in terms of Imd. and Last.

and textual distributions, thereby inducing significant domain shifts and catastrophic forgetting challenges.

Implementation details. Following UCIT [20], we use LLaVA-v1.5-7b [39] as the base multimodal model and embed LoRA modules in all linear layers of language model. In practice, we set Di1 = Di and select a small subset of Di as Di2 to enforce constraints. The number of training epochs for all tasks is set to 1. We set batch size to 16 for all methods and run experiments on 4 × H20/P800 GPUs.

5.2. Main Results We conduct comprehensive evaluations of Octopus on UCIT [20], comparing it against a diverse set of existing methods, as summarized in Tab. 1 and Fig. 1. For the sake of fairness, we compare Octopus with only architecturebased and regularization-based approaches, while including the vanilla rehearsal method as a reference. The quantitative results presented in Tab. 1 demonstrate our proposed Octopus achieves state-of-the-art (SOTA) performance and outperforms the best previous methods by 2.14% and 6.82% in Avg and Last metrics, respectively. Specifically, on one hand, Octopus demonstrates substantial improvements over existing regularization-based methods such as EWC [31], LwF [35] and OLoRA [63]. Octopus achieves 6.54% and 12.65% performance improvement over OLoRA in terms of Avg and Last, which indicates

   

    1XPEHURI7DVNV (a)



Z,PG Z/DVW ZR,PG ZR/DVW

 0HWULF9DOXH

0HWULF9DOXH

Metrics. Following UCIT [20], we adopt Last and Avg metrics to evaluate the continual learning performance of MLLMs. Last denotes the average accuracy over all tasks after sequentially learning the entire task stream, while Avg, on the other hand, represents the mean accuracy across all tasks throughout the training process. In addition, we adopt Imd. in several experiments to denote performance of a given task immediately after fine-tuning in the sequence learning process, which to some extent reflects the upper bound of the performance of that task during continual learning. We also report Backward Transfer (BWT) [40] in several experiments, which measures the average performance degradation on previous tasks and thereby reflects the degree of forgetting exhibited by continual learning method. For VQA tasks, we employ accuracy as the evaluation criterion, while for captioning tasks, we report the average score over multiple metrics, including BLEU1-4 [48], METEOR [12], ROUGE-L [37], and CIDEr [60].

5$9,&) $,5)&9 ,)5&$9



   

    1XPEHURI7DVNV



(b)

Figure 5. Average performance after fine-tuning on each task under different orderings (a) or w/ and w/o two-stage finetuning (b).

that our HiFGO exhibits more effective weight disentanglement compared to orthogonalization based on weight space. On the other hand, Octopus significantly outperforms other MoE-based approaches, such as MoELoRA [8] and HiDeLLaVA [20], demonstrating that our method achieves superior mitigation of inter-task parameter interference while maintaining high inference efficiency. Moreover, it is particularly noteworthy that Octopus even surpasses the vanilla rehearsal baseline, which explicitly constructs a replay buffer to store and reuse data samples from previous tasks. It underscores the ability of our method to achieve a more favorable plasticity–stability trade-off without relying on additional memory buffers. The explicit disentanglement of task representations also facilitates positive inter-task transfer, as evidenced by the ArXivQA Last metric exceeding that of Multi-task, while rehearsal-based strategies often suffer from task interference and lead to suboptimal performance.

5.3. Model Analysis Effectiveness of History-Free Gradient Orthogonalization. Tab. 2 presents the comparison results obtained under different orthogonalization methods (orthogonal to previous parameters, GPWC or both) and settings (compute gradient of previous parameters on current / previous dataset), demonstrating that our proposed HiFGO exhibits a substantial performance advantage over existing parameter-space or history-based gradient-space orthogonalization. Fig. 4 illustrates the average forgetting observed during sequential learning process, quantified using BWT. The results reveal that HiFGO exhibits substantially stronger resistance to forgetting compared to parameter orthogonalization (+0.41 v.s. -2.51 BWT), and even yields positive

Method w/o norm-regularization

λ = 2e − 3 λ = 5e − 3

w/ norm-regularization λ = 1e − 2 λ = 3e − 2

Imd. Last Imd. Last Imd. Last Imd. Last Imd. Last

ImageNet-R ArXivQA VizWiz IconQA CLEVR-Math Flickr30k Average BWT 89.03 93.37 63.31 65.03 73.07 56.33 73.36 -8.59 81.50 81.43 47.65 57.57 63.27 57.25 64.77 88.73 92.03 61.90 62.30 70.30 56.78 72.01 -3.36 85.07 86.47 54.93 61.63 67.07 56.78 68.65 88.23 93.87 62.34 63.37 68.27 57.13 72.20 -1.48 87.40 91.23 58.20 62.13 68.23 57.13 70.72 88.47 93.13 61.04 60.47 63.63 56.87 70.60 +0.41 88.20 93.03 58.46 60.50 69.00 56.87 71.01 87.93 89.60 54.92 43.67 51.57 57.01 64.12 +2.91 88.07 92.93 49.46 54.57 60.17 57.01 67.03

Table 4. Effectiveness of norm regularization. We report performance under both w/ and w/o settings in terms of Imd. and Last. Order Last Avg

R-A-V-I-C-F 71.01 71.08

A-I-R-F-C-V 69.04 70.04

I-F-R-C-A-V 69.16 70.00

Table 5. Results of different task orders on UCIT benchmark. We adopt an abbreviation scheme to simplify the representation of task sequence notation, as explained in Sec. 5.3

backward transfer (+0.41 BWT), which suggests that, the acquisition of a new task not only mitigates the degradation of performance on previous tasks but can further improve performance through the incorporation of newly acquired knowledge. within sequential learning process. Furthermore, jointly enforcing both forms of orthogonal constraints yields no additional gains, underscoring that HiFGO constitutes a more principled and inherently effective constraint mechanism compared to parameter orthogonalization. Comparison in Tab. 2 between HiFGO and history-based gradient orthogonalization reveals that the latter exhibits substantial forgetting (–8.95 BWT), which may stem from the fact that previous tasks have already converged to local optima, where gradient directions become less informative and highly oscillatory. In contrast, the GPWC employed in HiFGO provides a more faithful characterization of intertask interference, thereby playing a substantially more effective role in mitigating catastrophic forgetting. Effectiveness of the two-stage finetuning strategy. Tab. 3 reports effectiveness of our proposed two-stage finetuning strategy, indicating that our two-stage finetuning strategy yields a substantial influence on the Imd. and Last metrics, while exerting a relatively minor effect on BWT. It suggests that our two-stage finetuning strategy effectively bolsters model’s efficacy during fine-tuning phase of individual task, elevates the performance upper bound in sequential learning, and thereby enhances overall performance. Fig. 5b provides additional support by illustrating average Imd. and Last across varying task counts. Specifically, our twostage finetuning strategy exerts a significant impact on the respective performance of Imd. and Last, but does not significantly affect the gap between them. Effectiveness of norm regularization. As mentioned in

Sec. 4.3, norm regularization would effectively mitigates perturbations induced by high-order terms in Eq. 5, yet it moderately degrades model’s finetuning efficacy. This observation is substantiated by Tab. 4: stronger norm regularization yields superior performance in alleviating catastrophic forgetting and even facilitates more pronounced positive backward transfer, but induces a marginal drop in the Imd. metric. Consequently, a trade-off must be struck between these two objectives—an optimal balance between plasticity and stability is achieved via an appropriate norm regularization strength (we choose λ = 1e − 2 in practice). Influence of task ordering. For clarity, we represent each dataset by a letter; for instance, I-F-R-C-A-V denotes the learning sequence IconQA → Flickr30k → ImageNet-R → CLEVR-Math → ArXivQA → VizWiz. Tab. 5 presents the performance of of sequential learning under varying task orderings, in terms of Last and Avg, and Fig. 5a illustrates accumulative average post–sequential-learning performance across varying numbers of tasks, where task count i corresponds to average performance over the first i tasks of R-AV-I-C-F. Results show that Octopus demonstrates remarkable insensitivity to task order, with performance remaining highly consistent across different permutations or number of tasks. These observations indicate that Octopus exhibits strong robustness and ensures stable, order-invariant performance throughout the continual learning process.

6. Conclusion We propose Octopus, a two-stage continual learning framework based on history-free gradient orthogonalization. Specifically, our proposed HiFGO effectively mitigates catastrophic forgetting without respect to historical data, while our two-stage finetuning strategy achieves an effective trade-off between plasticity and stability. Experiments on UCIT corroborate that Octopus attains state-of-the-art (SOTA) performance, outperforming the prior SOTA by 2.14% and 6.82% in terms of the Avg and Last, respectively. Due to space constraints, our limitations: upper bound on the number of tasks and performance degradation on highly analogous tasks are detailed in Supplementary Material.

Acknowledgments. This work was supported in part by NSFC (62322113, 62376156), Shanghai Municipal Science and Technology Major Project (2025SHZDZX025G15, 2021SHZDZX0102), and the Fundamental Research Funds for the Central Universities. We thank Kunlunxin for their technical support in training and evaluation on P800.

References [1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In ICCV, 2021. 1 [2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In ECCV, 2016. 1, 2 [3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2 [4] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. NeurIPS, 2020. 2 [5] Meng Cao, Yuyang Liu, Yingfei Liu, Tiancai Wang, Jiahua Dong, Henghui Ding, Xiangyu Zhang, Ian Reid, and Xiaodan Liang. Continual llava: Continual instruction tuning in large vision-language models. arXiv preprint arXiv:2411.02564, 2024. 1 [6] Rich Caruana. Multitask learning. Machine learning, 1997. 2 [7] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, P Dokania, P Torr, and M Ranzato. Continual learning with tiny episodic memories. In Workshop on Multi-Task and Lifelong Reinforcement Learning, 2019. 1 [8] Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao. Coin: A benchmark of continual instruction tuning for multimodel large language models. NeurIPS, 2024. 1, 5, 7, 2 [9] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS, 2022. 2 [10] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020. 2 [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 1 [12] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, 2014. 7, 1 [13] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020. 1

[14] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics, 2020. 2 [15] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999. 1, 3 [16] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. arXiv preprint arXiv:1903.10145, 2019. 6 [17] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 2 [18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 1 [19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 2 [20] Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, DaHan Wang, Xu-Yao Zhang, and Cheng-Lin Liu. Hidellava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. arXiv preprint arXiv:2503.12941, 2025. 1, 2, 5, 6, 7, 4 [21] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. 1 [22] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021. 1 [23] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019. 2 [24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022. 2 [25] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in classincremental learning. In CVPR, 2021. 1 [26] Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multimodal large language model with dual momentum mixtureof-experts for continual visual question answering. In CVPR, 2025. 1 [27] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 1

[28] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022. 2 [29] Ziqi Jia, Anmin Wang, Xiaoyang Qu, Xiaowen Yang, and Jianzong Wang. Hierarchical-task-aware multi-modal mixture of incremental lora experts for embodied continual learning. arXiv preprint arXiv:2506.04595, 2025. 2 [30] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 1 [31] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka GrabskaBarwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017. 2, 5, 7 [32] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. 2 [33] Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.00231, 2024. 1 [34] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021. 2 [35] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 2017. 2, 5, 7 [36] Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In CVPR, 2024. 2 [37] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. 7, 1 [38] Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022. 1 [39] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 1, 7 [40] David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. NeurIPS, 2017. 2, 7 [41] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. 1 [42] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022. 1 [43] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel

Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. 2 [44] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, 2018. 2 [45] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 1 [46] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. 1989. 1, 3 [47] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. 1 [48] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002. 7, 1 [49] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 2019. 1 [50] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015. 1 [51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019. 2 [52] Wu Ran, Weijia Zhang, ShuYang Pang, Qi Zhu, Jinfan Liu, JingSheng Liu, Xin Cao, Qiang Li, Yichao Yan, and Chao Ma. Correlated low-rank adaptation for convnets. In NeurIPS, 2025. 2 [53] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurIPS, 2019. 1 [54] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. NeurIPS, 2017. 2 [55] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. 1 [56] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, 2023. 2 [57] James Seale Smith, Lazar Valkov, Shaunak Halbe, Vyshnavi Gutta, Rogerio Feris, Zsolt Kira, and Leonid Karlinsky. Adaptive memory replay for continual learning. In CVPR, 2024. 1 [58] Shikhar Srivastava, Md Yousuf Harun, Robik Shrestha, and Christopher Kanan. Improving multimodal large lan-

guage models using continual learning. arXiv preprint arXiv:2410.19925, 2024. 1 [59] George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. arXiv preprint arXiv:2410.19735, 2024. 2 [60] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pages 4566–4575, 2015. 7, 1 [61] Lei Wang, Jun Liu, Liang Zheng, Tom Gedeon, and Piotr Koniusz. Meet jeanie: a similarity measure for 3d skeleton sequences via temporal-viewpoint alignment. IJCV, 2024. 1 [62] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. TPAMI, 2024. 1 [63] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP, 2023. 1, 2, 4, 5, 7 [64] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022. 2 [65] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022. 1, 2, 5 [66] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019. 1 [67] Zeyuan Yang, Zonghan Yang, Yichen Liu, Peng Li, and Yang Liu. Restricted orthogonal gradient projection for continual learning. AI Open, 2023. 2 [68] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 2024. 1 [69] Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 2018. 2 [70] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 2021. 2 [71] Xuanle Zhao, Xuexin Liu, Haoyue Yang, Xianzhen Luo, Fanhu Zeng, Jianling Li, Qi Shi, and Chi Chen. Chartedit: How far are mllms from automating chart analysis? evaluating mllms’ capability via chart editing. arXiv preprint arXiv:2505.11935, 2025. 2 [72] Zhen Zhao, Zhizhong Zhang, Xin Tan, Jun Liu, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Rethinking gradient projection continual learning: Stability/plasticity feature space decoupling. In CVPR, 2023. 4 [73] Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, DeChuan Zhan, and Ziwei Liu. Class-incremental learning: A survey. TPAMI, 2024. 1 [74] Hao Zhu, Yifei Zhang, Junhao Dong, and Piotr Koniusz. Bilora: Almost-orthogonal parameter spaces for continual learning. In CVPR, 2025. 1, 2

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models Supplementary Material This material provides supplementary information on the proposed Octopus framework. It presents more details on our experimental settings (Sec. G), further experimental results (Sec. H), and more analysis, visualizations, and discussions (Sec. I) to complement the main manuscript. Finally, we discuss the limitations of Octopus.

G. Further Experimental Details

BLEU BLEU was originally introduced in [48] as a metric for measuring the n-gram similarity between a predicted sentence and its reference. It provides multiple variants depending on the n-gram order, with BLEU-1, BLEU-2, BLEU-3, and BLEU-4 being the most commonly used. The computation is formulated as follows: p_n = \frac {\sum \limits _{C\in \{Candidates\}}\sum \limits _{n\mbox {-}gram\in C}{Count_{clip}(n\mbox {-}gram)}}{\sum \limits _{C'\in \{Candidates\}}\sum \limits _{n\mbox {-}gram'\in C'}{Count(n\mbox {-}gram')}}.

G.1. Benchmarks In the main manuscript, we evaluate our method on UCIT [20], while in the supplementary material we further report its performance on CoIN [8]. Below, we provide detailed descriptions of both evaluations. UCIT [20] UCIT [20] is proposed to rigorously evaluate multimodal large language models in settings where instruction-following abilities must be incrementally acquired across heterogeneous visual–linguistic domains. This benchmark is constructed by unifying six widely used multimodal instruction datasets, including ImageNetR [22], ArXivQA [33], VizWiz-Caption [21], IconQA [41], CLEVR-Math [38] and Flickr30k [50], and reorganizing them into a sequential task stream. These datasets collectively cover natural-image captioning, open-ended visual question answering and mathematical reasoning. CoIN [8] CoIN [8] is introduced as a rigorous benchmark for evaluating multimodal large language models (MLLMs) under a sequential instruction-tuning paradigm. By integrating eight distinct task categories covering multiple datasets including VQAv2 [18], VizWiz [21], ScienceQA [42], TextVQA [55], GQA [27], OCR-VQA [47], ImageNet [11] and RECCOCO [30, 45], CoIN [8] captures a wide spectrum of multimodal challenges, from visual question answering and grounding to image classification and OCR-based VQA, thereby exposing models to large shifts in both visual domain and task semantics.

G.2. Metrics For standard VQA tasks, we adopt accuracy as the evaluation metric. For image captioning, we follow the protocol of UCIT and use the average score across multiple metrics, including BLEU-1-4 [48], METEOR [12], ROUGE-L [37], CIDEr [60] and SPICE [2], as our evaluation criterion.

(11) METEOR METEOR was proposed in [12] as a text evaluation metric that integrates lemmatization, synonym matching, and a weighted precision–recall formulation to more finely assess the semantic correspondence between generated outputs and reference texts. The computation is formulated as follows: Score = Fmean\cdot (1 - Penalty),

(12)

Fmean = \frac {10PR}{R + 9P},

(13)

where:

Penalty = 0.5\cdot \left (\frac {\#chunks}{\#unigrams\_matched}\right )^3.

(14)

ROUGE-L ROUGE-L was proposed in [37] as an automatic text evaluation metric based on longest-commonsubsequence matching, which quantifies the structural overlap between a generated sequence and a reference sequence to assess their content similarity. CIDEr CIDEr was proposed in [60] as a TF–IDF–weighted consensus-based evaluation metric that measures the content consistency and informational relevance between generated descriptions and reference descriptions by emphasizing semantically distinctive n-grams. The computation is formulated as follows: CIDER(c_i, S_i) = \sum \limits _{n=1}^Nw_nCIDER_n(c_i, S_i),

(15)

where: CIDER_N(c_i, S_i) = \frac {1}{m}\sum \limits _j{\frac {g^n(c_i)\cdot g^n(s_{ij})}{||g^n(c_i)||\cdot ||g^n(s_{ij})||}} (16)

Avg Last

Method Zero-shot Multi-task Sequential Finetune

Replay -

SciQA 69.79 82.36 64.22

L2P [65] MoELoRA [8] HiDe-LLaVA [20]

% % %

70.52 68.38 74.92

LwF [35] EWC [31] O-LoRA [63] Octopus (ours) Sequential Finetune

% % % %

-

65.20 65.11 73.32 81.73 57.43

L2P [65] MoELoRA [8] HiDe-LLaVA [20]

% % %

70.21 62.02 73.20

LwF [35] EWC [31] O-LoRA [63] Octopus (ours)

% % % %

60.71 59.75 72.56 79.72

Image Viz REC Text 9.93 45.50 58.47 57.75 89.63 52.51 65.83 61.27 40.13 43.87 38.32 55.04 ▽ Architecture-based 26.89 45.53 45.21 56.84 48.50 44.22 40.23 55.62 76.72 51.24 61.84 57.13 ▽ Regularization-based 40.63 43.22 40.05 56.23 40.89 44.09 39.67 54.92 68.37 50.26 61.12 57.75 84.73 52.30 62.83 57.77 28.90 41.88 30.05 51.39 ▽ Architecture-based 23.31 44.21 43.76 56.25 37.21 43.32 33.22 52.05 69.28 50.76 59.18 56.92 ▽ Regularization-based 30.58 41.49 36.01 52.80 31.88 42.26 34.96 51.06 62.84 48.43 58.97 57.66 82.18 51.49 62.17 53.60

GQA 60.77 59.93 55.89

VQA 66.50 65.67 60.61

OCR 64.93 62.03 64.78

Average 54.21 67.40 52.86

59.03 57.04 62.83

63.52 62.14 68.15

64.11 65.75 64.76

53.96 55.24 64.70

54.67 56.03 60.96 62.08 50.76

60.64 61.12 65.71 67.36 53.28

65.12 64.55 63.31 64.72 64.78

53.22 53.30 62.60 66.69 47.31

58.46 53.12 61.33

62.32 57.92 67.12

64.11 65.75 64.76

52.83 50.58 62.81

47.07 51.84 59.14 60.50

53.43 55.30 63.21 66.38

65.12 64.55 63.31 64.72

48.40 48.95 60.77 65.10

Table 6. Comparison with various methods on CoIN [8] in terms of Avg and Last. The best and second methods are labeled with bold and underline styles. Zero-shot evaluates the pretrained model without task-specific finetuning. Multi-task jointly finetunes the model across all datasets, whereas Sequential Finetune adapts only one LoRA module sequentially to individual tasks. These settings provide an empirical characterization of the lower bound, upper bound, and baseline for continual learning methods.

STAGE 1

STAGE 2

STAGE 1

(a)

(b)

STAGE 2

STAGE 1

STAGE 2

(c)

Figure 6. Finetuning curve comparison between w/ two-stage finetuning and w/o two-stage finetuning on VizWiz-Caption (a), IconQA (b) and CLEVR-Math (c).

SPICE SPICE was proposed in [2] as a scenegraph–based semantic evaluation metric that assesses the degree of semantic alignment between generated and reference descriptions by comparing their structured representations of objects, attributes, and relations. The computation is formulated as follows: SPICE(c, S) = F_1(c, S) = \frac {2P(c, S)R(c, S)}{P(c, S) + R(c, S)}

(17)

H. Further Experimental Results We conduct extensive comparisons against a variety of methods on the CoIN [8] benchmark, and the results are presented in Tab. 6. Compared with the current state-of-theart (SOTA), our approach achieves improvements of 1.99% and 2.29% on Avg and Last, respectively. Consistent with

the main manuscript, the competing methods are grouped into two categories: regularization-based and architecturebased. On one hand, our method significantly outperforms other regularization-based approaches, surpassing OLoRA [63] by 4.09% and 4.33% on Avg and Last, respectively. This further demonstrates that HiFGO exhibits more effective weight disentanglement compared to orthogonalization in the weight space, and that it is applicable to diverse data and task formats with strong robustness. On the other hand, our method also outperforms architecture-based approaches, indicating that it successfully mitigates parameter interference across tasks. In contrast, MoE tends to suffer from inaccurate task-origin prediction when processing test inputs, which limits its performance.

I. Further Analysis I.1. Theoretical Analysis of GPWC ∗ For T ask A and T ask B in the task sequence, we denote θA ∗ as the parameters after fine-tuning on T ask A, and l(x; θA ) ∗ as the loss function under input x and parameter θA . Then, GPWC can be expressed as:

g_{gpwc} = \mathbb {E}_{x\sim D_B}[\nabla _{\theta }\mathcal {l}(x;\theta _A^*)], where DB is the data distribution of T ask B. Let \mathcal {L}_A(\theta ) = \mathbb {E}_{x\in D_A}[\nabla _{\theta }\mathcal {l}(x;\theta )],

(18)

Seq. FT

Octopus

T Figure 7. ∆θs→t HS ∆θs→t Comparison for Seq. FT and Octopus

(19) 𝑤2

\mathcal {L}_B(\theta ) = \mathbb {E}_{x\in D_B}[\nabla _{\theta }\mathcal {l}(x;\theta )],

(20)

we define a family of tasks:

⊥ 𝑔𝐺𝑃𝑊𝐶

𝜃B∗ 𝜃∗

\mathcal {L}_\lambda (\theta ) = \mathcal {L}_A(\theta ) + \lambda (\mathcal {L}_B(\theta ) - \mathcal {L}_A(\theta )),

(21) 𝑂0

(22)

which further implies \nabla _\theta \mathcal {L}_A(\theta ^*(\lambda )) + \lambda (\nabla _\theta \mathcal {L}_B(\theta ^*(\lambda )) - \nabla _\theta \mathcal {L}_A(\theta ^*(\lambda ))) = 0. (23) Differentiating the above equation with respect to λ, we obtain \begin {aligned} &\frac {\partial }{\partial \lambda }\nabla _\theta \mathcal {L}_A(\theta ^*(\lambda )) + \nabla _\theta \mathcal {L}_B(\theta ^*(\lambda )) - \nabla _\theta \mathcal {L}_A(\theta ^*(\lambda )) \\ + &\lambda \frac {\partial }{\partial \lambda }(\nabla _\theta \mathcal {L}_B(\theta ^*(\lambda )) - \nabla _\theta \mathcal {L}_A(\theta ^*(\lambda ))) = 0. \end {aligned}

(24) ∗ and By substituting λ = 0 (at which point θ∗ (λ)|λ=0 = θA ∗ ) = 0), we obtain ∇θ LA (θ∗ (λ))|λ=0 = ∇θ LA (θA \frac {\partial \theta ^*}{\partial \lambda }\frac {\partial }{\partial \theta ^*} \mathbb {E}_{x\in D_A}[\nabla _{\theta }\mathcal {l}(x;\theta _A^*)] + \mathbb {E}_{x\in D_B}[\nabla _{\theta }\mathcal {l}(x;\theta _A^*)] = 0, (25) which implies gGP W C = HA v, where HA denotes the ∗ Hessian matrix of T ask A evaluated at θA , and v = ∗

− ∂θ∂λ(λ)

∗ θ ∗ (λ)=θA

corresponds to the tangent direction in

parameter space induced by the data manifold of T ask B, ∗ which can be interpreted as the adaptation direction of θA toward T ask B. Let {ui } denote the P eigenvectors of HA , and v can be decomposed as v = γi ui , and i

g_{GPWC} = H_Av = \sum \limits _{i}\lambda _i\gamma _iu_i,

𝜃𝐵 ∇𝐿orth 𝑔𝐺𝑃𝑊𝐶

where λ = 0/λ = 1 represents T ask A/B. Under Lλ (θ), the optimal parameters θ∗ (λ) satisfy: \left .\frac {\partial }{\partial \theta }\mathcal {L}_\lambda (\theta )\right |_{\theta =\theta ^*(\lambda )} = 0,

∇𝐿𝑐𝑒

(26)

𝜃𝐴∗

𝑤1

Figure 8. A toy example for GPWC.

which is a weighted combination with the eigenvalues {λi } of HA as weights. This indicates that GPWC primarily captures the projection of v onto the high-curvature directions of HA . These directions correspond to those along which ∗ updating θA under the data distribution of T ask B is more likely to degrade the performance on T ask A. Therefore, constraining the parameter update direction to be orthogonal to GPWC can effectively alleviate catastrophic forgetting. Further, we provide an empirical analysis in Fig. 7: T we compare the quadratic ∆θs→t HS ∆θs→t for Seq. FT and Octopus, where ∆θs→t represent parameter increment of “source → target”, and HS is Hessian of source task. GPWC gains a lower quadratic value, proving it can significantly mitigate the performance impact on prior tasks. A toy example We provide a toy example to intuitively illustrate the role of GPWC. We illustrate the intuition using a simple linear regression model y = w1 x1 + w2 x2 . We ∗ ∗ show the optimal parameters θA and θB of two linear regression tasks TA and TB in Fig. 8, and the blue and orange dashed curves represent the loss contours of TA and TB , respectively. After the model is trained on TA , the parameters ∗ converge to θA . If the model is subsequently trained on TB without any constraint, the parameter update will proceed ∗ ∗ approximately along the line connecting θA and θB due to ∗ linearity, eventually converging to θB . As a result, the per-

VizWiz-Caption

OLoRA

Octopus (ours)

Q: What is happening in the image? Generate a brief caption for the image. I: A computer screen with a blue background and white text L: A computer screen is displaying a message about a certificate I: A computer screen with a blue background and white text L: A computer screen is displaying a blue screen with white text

Q: What is happening in the image? Generate a brief caption for the image. I: A piece of paper with a questionnaire on it L: A person is filling out a form

Figure 9. Sensitivity analysis of λ1 and λ2

formance on TA deteriorates significantly. Now consider the case where the GPWC constraint is introduced. When the model parameters move toward a point θB , the GPWC ∗ ∗ direction is defined as the direction from θA to θB . Notably, this direction also corresponds to the direction along which the loss on TA increases during optimization for TB . Therefore, the gradient direction of Lorth becomes opposite to that of the gradient of the task loss Lce . Consequently, when a balance between Lce and Lorth is achieved during training, the resulting parameter θ∗ attains strong performance on TB while effectively preserving the performance on TA .

I.2. Example Analysis We provide a comparative analysis in Fig. 10 of our Octopus against OLoRA [63]. Octopus demonstrates markedly superior retention of previously acquired task capabilities after sequential learning. In the image captioning task, Octopus preserves salient visual details more faithfully, while in multimodal mathematical reasoning tasks, Octopus not only reproduces the correct answers obtained at finetuning time but, in some cases, autonomously corrects errors.

I.3. Comparison of FineTuning Dynamics for TwoStage Strategy To further investigate the effectiveness of our two-stage finetuning strategy, we visualize the finetuning dynamics on several datasets in Fig. 6, comparing the cases with and without the proposed strategy. As shown in the figure, during the first stage of unconstrained finetuning, the twostage strategy converges faster and achieves a better final convergence result. In the second stage of constrained finetuning, the introduction of constraints does not substantially compromise the performance gains achieved during the first stage; after a certain number of steps, the model is still able to reach the performance level of the unconstrained setting. In contrast, directly applying constrained finetuning from the beginning leads to a noticeable degradation in finetuning performance, limiting the model’s plasticity. Our two-stage finetuning strategy strikes an effective balance between plasticity and stability, retaining adaptation capacity while avoiding excessive performance loss.

I: A piece of paper with a questionnaire on it L: A piece of paper with a questionnaire on it

CLEVR-Math

OLoRA

Octopus (ours)

Q: Subtract all cylinders. How many objects are left? Answer the question using a single word or phrase. I: 7 L: 6

LABEL: 7

I: 6 L: 7

LABEL: 7

Q: Subtract 1 purple cylinders. How many objects are left? Answer the question using a single word or phrase.

I: 5 L: 4

LABEL: 5

I: 5 L: 5

LABEL: 5

Figure 10. Instance-wise comparison between Octopus (ours) and OLoRA on the UCIT benchmark. Here, I denotes Imd. and L denotes Last. We illustrates the output comparison of the two methods after fine-tuning on a specific dataset and upon the completion of all training procedures.

I.4. Sensitive analysis of λ1 and λ2 . λ1 and λ2 are set to balance the loss magnitudes. In the experiments, we default to setting λ1 = 2 × 10−2 and λ2 = 1 × 10−2 , and a comprehensive sensitivity analysis of the Last metric on the UCIT dataset with respect to λ1 and λ2 is presented in Fig. 9. Results shows that the the performance remains consistently high and stable when λ1 ∈ [1 × 10−2 , 6 × 10−2 ] and λ2 ∈ [6 × 10−3 , 1.6 × 10−2 ]. The wide effective ranges demonstrate that our method is not overly sensitive to these hyperparameters and exhibits strong robustness against their variations, which validates substantial practical applicability of Octopus .

I.5. Extra Number of Parameters for Inference In Fig. 11, we compare the additional parameters introduced during inference between Octopus and the current SOTA method HiDe-LLaVA [20] (We apply LoRA to all linear layers with rank = 48 and alpha = 96 for each method). MoE-based approaches, such as HiDeLLaVA [20], require the loading of multiple expert modules—i.e., multiple LoRAs—during inference, and further necessitate an additional network component for task-ID assignment. In contrast, Octopus introduces only a single LoRA during inference, identical to standard sequential fine-tuning with LoRA. As shown in Fig. 11, compared with HiDe-LLaVA [20],

1XPEHURI3DUDPHWHUV0



6HTXHQWLDO)LQHWXQH +L'H//D9$ 2FWRSXV

   



  1XPEHURI7DVNV





Figure 11. Comparison of the additional number of parameters required during inference.

Octopus incurs only a negligible number of additional parameters during inference, and more importantly, this overhead does not scale with the number of tasks. This design substantially reduces storage costs and improves inference efficiency. Limitations Despite substantial improvements achieved, several limitations remain that warrant further investigation. First, similar to other LoRA-based regularization methods, our framework is constrained by the number of tasks; second, our framework would impair the performance of individual tasks for those that are highly similar in domain but differ in problem formulation. These limitations highlight the need for more sophisticated designs capable of overcoming task capacity constraints while effectively handling similar tasks. We hope our work provides valuable insights for future continual learning research for MLLMs.