arXiv CSopen access

Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Tao Hu1,2 Da-Wei Zhou1,2(B) School of Artificial Intelligence, Nanjing University 2 State Key Laboratory for Novel Software Technology, Nanjing University

arXiv:2605.10765v1 [cs.CV] 11 May 2026

{hut, zhoudw}@lamda.nju.edu.cn

Abstract Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world deployment often requires continual capability expansion across sequential tasks. In such scenarios, Multimodal Continual Instruction Tuning (MCIT) aims to acquire new capabilities while limiting catastrophic forgetting. Existing methods mainly follow a module-composition paradigm: they maintain task-level prompts or LoRA experts and dynamically route or aggregate a subset of them at inference. However, samples within the same task can still differ substantially in visual scenes, question intents, and reasoning demands. This motivates instance-level adaptation to individual query-image pairs rather than only selecting or combining task-level modules. To this end, we propose D RAPE (Dynamic Cross-Modal Prompt Generation), a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT. Instead of selecting prompts from a fixed pool, D RAPE derives prompt queries from the textual instruction and cross-attends to visual patch features, producing query-image conditioned prompts that are prepended to the frozen LLM. To mitigate forgetting during sequential updates, D RAPE applies null-space gradient projection to the shared projector and uses CLIP-based prototype routing for task-label-free generator selection at inference. Extensive experiments on MCIT benchmarks show that D RAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.

Introduction

Recent Multimodal Large Language Models (MLLMs) [25, 4] achieve strong performance through large-scale multimodal instruction tuning [52, 37], enabling them to handle diverse vision-language tasks [29, 9]. Yet practical deployment rarely stops at a fixed task mixture: new domains, instruction styles, and capabilities arrive over time, and the model must absorb them without sacrificing what it has already learned. This setting gives rise to Multimodal Continual Instruction Tuning (MCIT) [2], where the central challenge is to extend multimodal capability under sequential updates while limiting catastrophic forgetting [7, 50]. Existing MCIT methods mainly follow a module-composition paradigm. They freeze the pre-trained LLM and vision backbone, maintain lightweight task-specific modules such as prompts or LoRA experts, and dynamically route or combine these modules at inference [6, 1, 2, 47, 10, 8, 46]. Although such routing or retrieval mechanisms make the effective configuration input-dependent, the adaptation remains restricted to selecting or recombining discrete modules learned from previous tasks. In practice, however, samples within the same MCIT task can still differ substantially in visual scenes, question intents, reasoning demands, and instruction–image interactions. These differences require the model to dynamically adapt to individual query–image pairs rather than only selecting Preprint.

ScienceQA Input Samples Dynamic Prompts Static Prompt

OCRVQA

TextVQA

+1.96

70 60

+3.18

+1.54

50 40

VQAv2

-0.84

+4.53

+0.67

+2.09

Grounding VizWiz Dynamic Prompt (Ours)

ImageNet

+1.79

GQA

Static Prompt

Figure 1: Why instance-specific prompts are needed. Left: t-SNE visualization on 200 samples from the VQAv2 test set. For each sample, we average-pool the input embeddings and the corresponding instance-specific soft prompt to obtain visualization features. For the static baseline, a single shared task prompt is average-pooled and plotted for comparison. The instance-specific soft prompts follow the sample distribution more closely, suggesting that they better reflect instance-level variation than static task prompts. Right: task-wise performance comparison between instance-specific and static task prompts on CoIN. Instance-specific prompts achieve stronger overall performance, suggesting the benefit of modeling within-task variation rather than using one fixed task-level prompt. or recombining task-level components. This motivates an alternative approach that synthesizes continuous prompts directly from the current multimodal input. This instance-level variation is further illustrated in Figure 1. In the left panel, the instance-specific soft prompts generated for each sample by our cross-modal prompt generator follow the sample distribution more closely than static task prompts, suggesting that useful adaptation should track instancelevel structure rather than collapse all samples in a task into the same prompt. In the right panel, instance-specific prompts achieve stronger overall performance than static task prompts on CoIN, further suggesting that within-task multimodal diversity cannot be fully captured by one fixed task-level representation. These observations motivate a continual-learning framework that remains lightweight like prior parameter-efficient methods while allowing the adaptation itself to vary across samples. To address this issue, we propose D RAPE (Dynamic Cross-Modal Prompt Generation), a promptlearning framework that generates continuous instance-specific soft prompts for MCIT. Although instance-dependent prompting has been explored in prior language-only or vision-only settings [45, 20, 44, 26], these methods do not directly model query–image conditioned adaptation under sequential multimodal instruction tuning. In contrast, D RAPE synthesizes prompts from both textual instructions and visual patch features: a lightweight cross-modal prompt generator summarizes instruction tokens into prompt queries, cross-attends to visual patch features, and produces prompts that are prepended to the frozen LLM. Rather than selecting or recombining discrete task-level components, the model synthesizes prompts that adapt to the fine-grained multimodal structure of each sample. To reduce forgetting during sequential updates, we combine this prompt generator with two complementary mechanisms. First, we apply null-space gradient projection to the shared projector so that updates are biased toward directions that minimally interfere with previously observed feature subspaces. Second, we use a CLIP-based prototype router to select the most relevant task-specific generator without task labels at inference. Together, these components form an input-conditioned cross-modal prompt-generation framework for rehearsal-free MCIT, improving adaptation to diverse query–image pairs while limiting catastrophic forgetting throughout training over sequential tasks.

Related Work

Multimodal Large Language Models. Multimodal Large Language Models (MLLMs) extend large language models to vision-language tasks by connecting visual encoders with language models through cross-modal alignment modules [33, 25, 4]. Typically, visual features are projected into the language embedding space and combined with textual instructions, enabling the model to generate responses conditioned on both modalities [25, 4]. With large-scale multimodal instruction tuning [52, 42, 27], MLLMs have achieved strong performance on visual reasoning [17, 49], visual question answering [9], instruction following [53], and multimodal generation [4]. Despite these 2

advances, most existing MLLMs are trained on a fixed mixture of tasks and datasets [51, 2]. As a result, adapting them to newly emerging domains, instruction styles, or capabilities often requires additional fine-tuning, which may interfere with previously learned multimodal knowledge [34, 50]. This static-training assumption motivates studying how MLLMs can be adapted after deployment. Continual Instruction Tuning. Continual Instruction Tuning extends standard instruction tuning by enabling foundation models to learn a sequence of tasks while remaining aligned with human instructions [11]. As MLLMs are increasingly deployed in open-world settings, mitigating catastrophic forgetting [50] in Multimodal Continual Instruction Tuning (MCIT) has become an important problem [43, 51]. Existing methods generally follow several directions. Early approaches often adapt continual learning techniques from natural language processing [40, 38, 39], using regularization or replay to preserve past knowledge. Recent parameter-efficient methods, such as MoELoRA [6], ProgLoRA [47], and HiDe-LLaVA [10], keep the MLLM backbone frozen and update only lightweight task-specific modules. At inference time, these methods typically route, select, or assemble stored task-level components according to the current input. While these MCIT methods mainly focus on task-level module routing or assembly, instance-aware prompting has also been studied in several related settings, including instance-dependent prompt generation for NLP [44], visual instance-aware or input-adaptive prompt tuning [45, 20], compact capsule prompt representations [26], and image-conditioned prompt generation for continual learning [18].

Preliminaries

3.1

Multimodal Continual Instruction Tuning.

We consider a typical MLLM [24] that processes an image v and a textual instruction q into a concatenated multimodal sequence z = [w; u] ∈ R(m+s)×d . Here, w = π(ϕ(v)) denotes the projected visual features from a vision encoder ϕ(·) and a projector π(·), while u = ψ(q) represents the text embeddings. In the continual instruction tuning setting, the model encounters a sequence t of tasks {D1 , D2 , . . . , DT } arriving sequentially, where each task Dt = {(vi , qi , yi )}N i=1 consists of image–instruction–answer triplets, and yi denotes the target answer token sequence. Under the strict rehearsal-free constraint (i.e., past data is inaccessible), the goal is to optimize a set of tunable parameters Θ to generalize across all tasks seen so far:   L X Θ∗t = arg min E(v,q,y) ∼ D≤t − log pΘ (yj | z, y<j ) , (1) Θ

j=1

where D≤t denotes the joint distribution over all observed tasks, L is the length of the target answer sequence y, and yj denotes its j-th token. In practice, since training is performed exclusively on Dt , the learning procedure must inherently mitigate catastrophic forgetting. 3.2

Baselines in MCIT

To overcome forgetting while maintaining the pre-trained multimodal knowledge, existing rehearsalfree MCIT methods typically freeze the LLM backbone and rely on two dominant paradigms: Prompt Tuning and Mixture-of-Experts (MoE) with LoRA. Prompt Tuning: Prompt-based methods [36, 54, 48] adapt the frozen model by prepending learnable continuous vectors (soft prompts) to the input sequence. To handle a continuous sequence of tasks, they typically maintain a dynamic prompt pool P. For a given input, specific prompts P ∈ RLp ×d are retrieved or matched based on instance features. The augmented input becomes z′ = [P; z], and the retrieved prompts, or task-specific prompts when task identities are used, are updated during training. MoE with LoRA Experts: An alternative paradigm [2, 6, 1] focuses on parameter-efficient finetuning via Low-Rank Adaptation (LoRA). To prevent forgetting, these methods allocate task-specific LoRA experts and freeze them after learning each task. During inference, a routing mechanism dynamically blends these historical experts. For a given input, the dynamically assembled parameter Θadd is computed as: t X Θadd = ω k Θk , (2) k=1

Training stage at task t

Cross-modal prompt generator (G in left)

Projection

Prompt

∆𝑤 𝜫(𝒕)

MLP (Eq. 8)

𝑈

𝛴

𝒄𝒕 (Eq. 15) 𝒆𝒊 SVD Task prototype t (Eq. 14)

CLIP

(Eq. 5)

𝜫(𝒕)

𝑉

Multi-Head Attention

𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 + ∆𝑾𝜫𝒕

𝑴𝒕

(Eq. 9) Vision encoder

Update

(Eq. 7)

Multi-Head Attention

(Eq. 4)

𝑮

Down-projection

Prompt

Projector

LLM

Inference stage at task t CLIP

Tokenizer

(Eq. 14)

𝒆 (Eq. 17)

Original input embeddings

𝒄𝟏 𝒄𝟐

𝒄𝒕

𝑮𝒕 ∗ Prompt

Figure 2: Illustration of D RAPE. Left: Training on task t. A task-specific generator synthesizes soft prompts, while the shared visual projector is regularized by projection onto the complement of the retained principal subspace. Feature statistics M(t) are decomposed via SVD to obtain a projection matrix Π(t) for the next task. After training, a task prototype ct is registered in a frozen CLIP embedding space. Top-Right: Cross-modal prompt generator. It summarizes instruction tokens into prompt queries and cross-attends to visual features to synthesize instance-specific soft prompts. Bottom-Right: Inference without task labels. The router compares the current input against stored prototypes and selects the corresponding generator Gt∗ . where Θk represents the LoRA parameters optimized for task k, and ωk is the routing weight predicted by a gating network based on current input features. Discussion. Both paradigms provide practical solutions for MCIT, but they also expose trade-offs. For prompt tuning, the reliance on a discrete prompt pool P can limit how flexibly the adaptation responds to within-task variation. For MoE with LoRA experts, the model gains stronger task-level specialization, but its behavior is still governed by routing and recombination among stored taskspecific components. These approaches can be effective, yet they may be less suitable when subtle multimodal differences call for more input-specific adaptation. This motivates exploring whether MCIT can benefit from dynamically synthesizing prompts for each sample while still preserving historical knowledge through lightweight constraints.

D RAPE: Dynamic Cross-Modal Prompt Generation

We propose D RAPE, a prompt learning framework for MCIT that moves from static component reuse toward input-conditioned prompt synthesis. As illustrated in Figure 2, D RAPE contains three key components. First, a cross-modal prompt generator produces instance-specific soft prompts conditioned on both the visual and textual features of each input. Second, null-space gradient projection constrains updates to the shared visual projector so as to reduce interference with previously observed feature subspaces. Third, a CLIP-guided prototype router selects the most relevant taskspecific generator at inference. In the main experiments, the router stores one learned prototype per task, built from fused CLIP text-image features. 4.1

Cross-Modal Prompt Generation

The core of D RAPE is a cross-modal prompt generator Gt that, given an input’s projected visual features w ∈ Rm×d and textual instruction embeddings u ∈ Rs×d , synthesizes Lp instance-specific soft prompt vectors Pi = Gt (wi , ui ) ∈ RLp ×d . The subscript t indicates that a separate task-specific generator is maintained for each task. During training, the generator masks out image placeholders, padding tokens, and answer positions so that prompt synthesis depends on the non-target instruction context rather than the supervised response. In our default configuration, the generation process uses two stages: instruction-aware query initialization followed by vision-guided prompt synthesis. 4

Instruction-Aware Query Initialization. The textual instruction encodes the task intent (e.g., “describe this image” vs. “what is the OCR text”) and should guide how visual information is extracted. We first project the masked instruction embeddings into a latent space via a linear mapping fu : Rd → RH : hu = fu (u) ∈ Rs×H , (3) where H is the generator’s hidden dimension. In the default ‘segment’ pooling mode, the instruction sequence is partitioned into Lp roughly equal segments {S1 , . . . , SLp }, and a masked average pooling is computed within each segment: P u j∈Sp hj · ⊮[j is valid] , p = 1, . . . , Lp , h̄p = (4) max |{j ∈ Sp : j is valid}|, 1 yielding an initial pooled query matrix H̄ ∈ RLp ×H . These pooled summaries are then used as queries in a multi-head attention layer over the full instruction features: Q = LN MHA(H̄, hu , hu ) ∈ RLp ×H , (5) where MHA denotes multi-head attention with a key padding mask that ignores invalid positions, and LN denotes layer normalization. The resulting Q contains Lp instruction-aware query vectors that are passed to the visual cross-attention stage. Vision-Guided Prompt Synthesis. With the instruction-derived queries in hand, we extract taskrelevant visual information through cross-modal attention. The projected visual features w are first mapped to the latent space: hv = fv (w) ∈ Rm×H , (6) d H where fv : R → R is a linear projection. The instruction queries Q then attend to the visual features via cross-modal attention: R = LN(Q + MHA(Q, hv , hv )) ∈ RLp ×H .

(7)

A dropout layer is applied to R for regularization. Finally, a two-layer MLP projects the fused representations back to the LLM’s embedding space, producing the instance-specific soft prompts: Pi = fhead (R) ∈ RLp ×d ,

(8)

where fhead maps RH → R2H → Rd with a GELU activation in between. The instance-specific soft prompts are prepended to the multimodal input sequence, yielding z′ = [Pi ; z], and the LLM generates the response via pθ (y | z′ ). Discussion. This design choice reflects an asymmetry between modalities: the instruction indicates which information is needed, whereas the image provides the content that must be adapted to. By initializing prompt queries from the instruction and refining them through visual cross-attention, D RAPE performs input-conditioned prompt adaptation instead of selecting from a fixed pool of historical components. The implementation also supports mean-pooling, static-prompt, and learnablequery variants for additional analysis beyond the default setting. 4.2

Null-Space Gradient Projection for Projector Preservation

While each task maintains its own generator, the visual projector π(·) is shared across all tasks and continuously updated during sequential training. Updates from later tasks may therefore alter the visual-to-language mappings used by earlier tasks and contribute to catastrophic forgetting. To mitigate this effect, we constrain the gradient updates of π using an approximate null-space projection derived from the empirical feature subspaces observed in previous tasks. Concretely, during training on task Dt , forward hooks attached to each linear layer of π collect the input features ṽ passing through that layer. The statistics and projections below are maintained per linear layer, with layer indices omitted for clarity. After training completes, these features are merged with the statistics accumulated from previous tasks to form a cumulative second-moment matrix: P N (t−1) M(t−1) + i ṽi ṽi⊤ M(t) = , N (t) = N (t−1) + Nt , (9) N (t) where Nt is the number of feature vectors collected from Dt for the corresponding layer and M(0) = 0. We then decompose M(t) via singular value decomposition: M(t) = U Σ V ⊤ ,

Σ = diag(σ1 ≥ · · · ≥ σdv ). 5

(10)

The leading singular directions provide an empirical estimate of the dominant feature subspace used by the tasks observed so far. We identify the effective rank r as the smallest index satisfying: Pr k=1 σk ≥ ϵ, (11) Pdv k=1 σk where ϵ is the energy-retention threshold. To preserve at least one update direction, we enforce r < dv . The directions outside the retained subspace form the complementary basis V⊥ = V[:, r+1:dv ] , and we construct the projection matrix: Π(t) = V⊥ V⊥⊤ ∈ Rdv ×dv .

(12)

During training on task Dt+1 , Π(t) is applied to the gradients of each weight matrix in π via registered gradient hooks. For a weight matrix W ∈ Rd×dv , the projected gradient is ∇W ′ = ∇W Π(t) = ∇W V⊥ V⊥⊤ .

(13)

This operation restricts the projector update to the complementary subspace of the dominant feature directions accumulated from previous tasks. To see why this helps preserve earlier-task mappings, consider an earlier-task input feature ṽold to the same linear layer. If this feature is well captured by the retained principal subspace, then its component in the complementary subspace is small, i.e., V⊥⊤ ṽold ≈ 0. After a gradient step with learning rate η, let ∆oold denote the first-order change in the layer output for this old feature. Using the projected gradient in Eq. (13), we have ∆oold = −η∇W ′ ṽold = −η∇W V⊥ V⊥⊤ ṽold ≈ 0. (14) Thus, when previous-task features mainly lie in the retained principal subspace, the projected update has limited effect on their projected representations. In this way, the projection helps reduce forgetting in the shared projector while still allowing updates along complementary directions for new tasks. Discussion. The energy threshold ϵ determines how many dominant feature directions from previous tasks are retained. A larger ϵ preserves more historical feature directions, making subsequent projector updates less likely to overwrite visual-to-language mappings used by earlier tasks. In contrast to replay-based methods, this strategy stores only compact second-moment statistics for the shared projector. Thus, it provides a lightweight complement to the task-specific generators for reducing forgetting without retaining past training samples. 4.3

CLIP-Guided Prototype Routing

Since D RAPE maintains a separate generator Gt for each task, a routing mechanism is required at inference to select the appropriate generator without task labels. We use frozen CLIP embeddings as a shared semantic space for routing. In particular, the visual features are directly reused from the CLIP-based vision tower of LLaVA, so the router does not require loading an additional CLIP encoder, introducing only marginal extra cost for text-side encoding. Task Prototype Computation. After learning each task, we register one compact prototype in the CLIP space. The routing feature is constructed by concatenating the normalized CLIP text embedding of the instruction and the normalized CLIP image embedding: ẽi = norm([ξ(qi ); γ(vi )]) .

(15)

During training, fused embeddings from the current task are cached temporarily and used to register t the task prototype once training on Dt is finished. Let {ẽi }N i=1 denote the cached embeddings of task t. We initialize the prototype ct by the normalized mean of these embeddings. For t > 1, we further refine ct with a classification objective that pulls current-task embeddings closer to ct than to t−1 previously stored prototypes. For a minibatch {ẽi }B i=1 and earlier prototypes {cs }s=1 , the loss is: B

Lcls = −

1 X exp(cos(ẽi , ct )/τ ) log , Pt−1 B i=1 exp(cos(ẽi , ct )/τ ) + s=1 exp(cos(ẽi , cs )/τ )

(16)

where τ is the temperature parameter and cos(·, ·) denotes cosine similarity. For the first task, no previous prototype is available, so Eq. 16 is not applied and c1 is simply registered as the normalized 6

Table 1: Main results on the CoIN benchmark with LLaVA-v1.5-7B as the backbone (higher is better). The best and second-best values are marked in bold and underline, respectively. Methods ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAv2 OCR-VQA Average Finetune 26.00 25.38 28.51 33.07 26.52 0.10 40.00 52.92 29.06 CODA-Prompt [36] 58.15 50.16 24.04 54.33 48.94 17.83 55.86 54.42 45.46 DualPrompt [41] 56.40 47.12 34.96 42.03 44.14 12.01 54.43 53.36 43.05 L2P [54] 54.42 46.04 30.36 57.09 42.19 9.38 50.45 54.03 42.99 MoELoRA [2] 47.34 32.91 38.73 37.15 42.48 0.97 42.77 57.50 37.48 Continual LLaVA [1] 58.67 49.99 57.66 62.53 42.32 16.25 64.33 74.91 53.33 ModalPrompt [48] 68.42 56.40 41.13 61.11 50.13 36.69 66.90 59.68 55.06 31.19 62.71 64.44 59.09 ProgLoRA [47] 74.84 51.83 83.90 49.93 53.87 D RAPE (Ours) 70.67 59.61 94.16 61.37 54.43 67.92 66.53 65.11 67.48

mean prototype. After registration, the cached instance-level embeddings are discarded. As a result, only one compact prototype is retained for each learned task. Inference Routing. Given a test sample, we compute its routing feature ẽ in the same CLIP space and score each task by cosine similarity: st = cos(ẽ, ct ). The router selects

t∗ = arg

max

t∈{1,...,T }

(17) st ,

(18)

and activates the corresponding generator Gt∗ to produce the prompt. Discussion. The router is lightweight: it stores only one prototype per task and does not rely on replay samples or historical feature banks. Moreover, by reusing the CLIP-based vision tower already available in LLaVA, it avoids introducing an additional CLIP model. Using fused text-image embeddings makes routing depend on both instruction semantics and visual content, while the prototype update keeps each task representation compact yet discriminative. 4.4

Summary of D RAPE

D RAPE addresses multimodal continual instruction tuning through three complementary mechanisms: a cross-modal prompt generator for input-conditioned prompt synthesis, a projection-based constraint that reduces interference in the shared visual projector, and a CLIP-guided prototype router for practical task-specific generator selection without task labels. Training. When learning task Dt , only the current generator Gt and the shared visual projector π are updated. The vision encoder ϕ, the LLM backbone θ, and all historical generators {G1 , . . . , Gt−1 } remain frozen. The training objective is the standard autoregressive language modeling loss optimized over the tunable parameters Θt = {Gt , π}: Lt = −

L X

log pθ (yj | [Pi ; z], y<j ) ,

(19)

j=1

where Pi = Gt (wi , ui ) is the synthesized prompt. For t > 1, the null-space projection in Eq. (13) is applied to the gradients of π via backward hooks. After completing Dt , three post-training steps are executed: (1) the current generator Gt is frozen; (2) the cumulative second-moment matrix M(t) is updated and the new projection matrix Π(t) is computed for the next task; and (3) the task prototype ct is registered in the router from the current task’s temporary fused CLIP embeddings, after which those per-instance embeddings are discarded. Inference. Given a test input (v, q) without task labels, the router computes the fused CLIP feature ẽ, selects t∗ by prototype similarity, and activates Gt∗ to synthesize Pi = Gt∗ (wi , ui ). The frozen LLM then generates the response from [Pi ; z].

Experiments

5.1

Implementation Details

Datasets. We conduct experiments on two MCIT benchmarks. The first is CoIN [2], which consists of eight sequential VQA tasks: ScienceQA [29], TextVQA [35], ImageNet [5], GQA [16], VizWiz [12], 7

Table 2: Main results on the UCIT benchmark with LLaVA-v1.5-7B as the backbone (higher is better). The best and second-best values are marked in bold and underline, respectively. Methods Zero-shot LoRA-FT [14] O-LoRA [39] MoELoRA [2] ModalPrompt [48] CL-MoE [15] HiDe [10] SEFE [3] D RAPE (Ours)

ImgNet-R 16.27 58.03 77.50 70.07 51.07 66.33 84.03 80.83 85.07

ArxivQA 53.73 77.63 78.07 77.70 87.27 77.00 90.73 78.00 91.60

VizWiz 38.39 44.39 44.50 44.69 48.11 44.78 44.43 47.01 55.68

IconQA 19.20 67.40 63.13 50.03 39.23 51.87 58.93 69.63 62.53

CLEVR 20.63 61.77 64.73 54.03 46.57 53.53 41.37 65.83 65.77

Flickr30k 41.88 58.22 58.16 57.34 42.93 57.42 54.25 57.92 55.82

Average 61.24 64.35 58.98 52.53 58.49 62.29 66.54 69.41

Table 3: Ablation study on the CoIN benchmark. ∆ indicates the average accuracy change relative to the full model. The best results are shown in bold. Variants ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAv2 OCR-VQA Average ∆ D RAPE (Full) 70.67 59.61 94.16 61.37 54.43 67.92 66.53 65.11 67.48 w/o Cross-Modal Attn. 69.39 59.44 94.14 61.11 47.77 67.87 66.55 64.22 66.31 -1.17 w/o Null-Space Proj. 69.94 55.78 83.62 61.48 49.83 65.43 67.43 65.38 64.86 -2.62

Grounding (RefCOCO) [19, 30], VQAv2 [9], and OCR-VQA [31]. The second is UCIT [10], which contains six sequential tasks: ArxivQA [21], CLEVR-Math [22], IconQA [28], ImageNet-R [13], VizWiz-caption [12], and Flickr30k [32]. Together, these two benchmarks let us evaluate our method in both a widely used MCIT setting and a cleaner setting with reduced data-overlap concerns. Comparison Methods. We compare D RAPE with classic prompt-based continual learning approaches, including CODA-Prompt [36], DualPrompt [41], and L2P [54], as well as recent MCIT baselines, including MoELoRA [2], Continual LLaVA [1], ModalPrompt [48], and ProgLoRA [47]. For UCIT, we additionally include LoRA-FT [14], O-LoRA [39], CL-MoE [15], HiDe [10], and SEFE [3]. We also report standard sequential fine-tuning as a forgetting-heavy lower bound. Training Setup. We adopt LLaVA-v1.5-7B [23] as the backbone MLLM, with CLIP-ViT-L/14336 [33] as the vision encoder and a two-layer MLP with GELU activation as the visual projector. For the proposed generator, we set the hidden dimension to H = 512 and the prompt length to Lp = 10. The null-space threshold is ϵ = 0.99. Each task is trained for one epoch with per-device batch size 4 and gradient accumulation steps 2. We use cosine learning-rate decay with warmup ratio 0.03, setting the peak learning rates to 2 × 10−4 for the generator and 2 × 10−5 for the projector. All experiments are conducted on 4 NVIDIA RTX 4090 GPUs with DeepSpeed ZeRO-2 and bf16 precision. Evaluation Metrics. Following [2], we denote by As,t the performance on task s after training up to task t, with T total tasks. The primary metric is the final average accuracy, defined as PT Ā = T1 s=1 As,T . We also report Backward Transfer (B) and Mean Accuracy (M), with their detailed formulations provided in the appendix. 5.2

Benchmark Comparison

Table 1 reports the benchmark comparison on CoIN, where D RAPE achieves the highest average accuracy among all compared methods (67.48), outperforming the strongest baseline ProgLoRA (59.09) by 8.39 points and obtaining the best results on four tasks: TextVQA, ImageNet, VizWiz, and Grounding. Table 2 further shows that this advantage generalizes to UCIT, where D RAPE again achieves the best average performance (69.41), surpassing the strongest baseline SEFE (66.54) by 2.87 points and ranking first on ImgNet-R, ArxivQA, and VizWiz. These results show that the gains of D RAPE are robust across both the standard CoIN benchmark and the cleaner UCIT setting. Stage-wise forgetting analysis using Backward Transfer and Mean Accuracy is provided in the appendix. 5.3

Further Analysis

Ablation Study. To evaluate the contribution of each core component in D RAPE, we conduct controlled ablations in Table 3. Removing the vision cross-attention stage (“w/o Cross-Modal Attn.”) reduces the average accuracy from 67.48 to 66.31, a drop of 1.17 points, with the clearest degradation appearing on visually demanding tasks such as VizWiz and OCR-VQA. Disabling null-space gradient projection (“w/o Null-Space Proj.”) causes a larger drop to 64.86, i.e., 2.62 points below the full model, with especially visible declines on ImageNet, TextVQA, VizWiz, and Grounding. This pattern is consistent with the intended role of the projector constraint: it does not eliminate forgetting entirely, but it helps reduce cross-task interference during sequential updates. 8

100

Avg. Oracle: 68.17% Avg. Predicted: 67.47% Avg. No Routing: 57.65%

94.46 94.16

Oracle Routing Predicted Routing

No Routing

H=768 H=1024

90 73.76 70.67

72.06

68.03 67.92

59.91 59.61

67.30 66.53 67.35 63.27

62.04 61.37 61.69

65.16 65.11 65.16

57.92 54.69 54.43

Accuracy (%)

H=256 H=512

100

80 70 60

50 43.51

50 30.24

40 ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

VQAv2

OCRVQA

Task ID

Task

Figure 3: Routing ablation and generator hidden-dimension sensitivity on the CoIN benchmark. Left: oracle routing, the learned prototype router, and a no-routing variant that always uses the last-task generator. Right: per-task final accuracy under different generator hidden dimensions H. Q: Which year's calendar is this? A: 2016

Original

Baseline

Q: What type of book is this? A: Calendars

Ours

Original

Baseline

Ours

Figure 4: Prompt-to-image attention visualizations on OCR-VQA examples. For each example, Column 1 shows the original image, Column 2 shows the baseline attention map, and Column 3 shows the attention map produced by D RAPE. By generating prompts conditioned on both textual instructions and visual features, D RAPE attends more selectively to query-relevant textual evidence, such as the printed year in the left example and the calendar-related text in the right example. Routing Behavior and Generator Hidden Dimension. Figure 3 presents two additional analyses. The left panel shows our learned prototype router achieves an average accuracy of 67.48, closely approaching the oracle upper bound (68.17) and vastly outperforming the no-routing baseline (57.65). This advantage is most prominent on ScienceQA and ImageNet, where the router effectively prevents the severe task mismatches seen in the baseline. To further illustrate this, we provide case studies in the appendix, showing that the advantage of D RAPE is especially clear on samples requiring fine-grained instance-level adaptation. The right panel evaluates the generator’s hidden dimension (H). While H = 256 underperforms (especially on VizWiz), increasing H from 512 to 768 or 1024 yields only marginal, plateauing improvements. Overall, the prototype router proves highly effective, and a moderate capacity suffices; thus, we adopt H = 512 as our default setting. Visualization. Figure 4 visualizes prompt-to-image attention maps on OCR-VQA examples, where Baseline denotes the static task-level prompt variant. By generating prompts conditioned on both textual instructions and visual features, D RAPE attends more selectively to query-relevant textual regions. For the year-recognition example, D RAPE focuses on the lower-left text containing “2016 Calendar”; for the book-type example, it attends to “Engagement Calendar 2016”, which supports the answer “Calendars”. Compared with the more diffuse baseline attention, these results suggest that cross-modal prompt generation helps D RAPE capture visual evidence required by the current instruction. Additional qualitative case studies are provided in the appendix.

Conclusion

In this paper, we study multimodal continual instruction tuning with a particular focus on how models adapt to diverse samples encountered over a task stream. We find that continual adaptation should operate not only across tasks, but also across distinct samples within the same task, since different query-image pairs may require substantially different grounding, reasoning, or text-understanding behaviors. To address this challenge, we propose D RAPE, a framework that replaces static task-level adaptation with instance-specific prompt generation conditioned on the current multimodal input. Extensive experiments on the CoIN and UCIT benchmarks demonstrate that D RAPE achieves strong performance across diverse multimodal tasks and maintains competitive continual learning behavior. Limitations. While D RAPE is developed and validated in the prompt-tuning setting considered in this paper, we have not yet extended the same instance-specific adaptation idea to other lightweight adaptation paradigms such as LoRA. Exploring how instance-specific generation can be integrated with LoRA-style updates is an interesting direction for future work. 9

References [1] Meng Cao, Yuyang Liu, Yingfei Liu, Tiancai Wang, Jiahua Dong, Henghui Ding, Xiangyu Zhang, Ian Reid, and Xiaodan Liang. Continual llava: Continual instruction tuning in large vision-language models. arXiv preprint arXiv:2411.02564, 2024. [2] Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao. Coin: A benchmark of continual instruction tuning for multimodel large language models. Advances in neural information processing systems, 37:57817–57840, 2024. [3] Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, and Sam Kwong. Sefe: Superficial and essential forgetting eliminator for multimodal continual instruction tuning. arXiv preprint arXiv:2505.02486, 2025. [4] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250–49267, 2023. [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [6] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv preprint arXiv:2312.09979, 4(7), 2023. [7] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4): 128–135, 1999. [8] Chendi Ge, Xin Wang, Zeyang Zhang, Hong Chen, Jiapei Fan, Longtao Huang, Hui Xue, and Wenwu Zhu. Dynamic mixture of curriculum lora experts for continual multimodal instruction tuning. arXiv preprint arXiv:2506.11672, 2025. [9] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. [10] Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13572–13586, 2025. [11] Haiyang Guo, Fanhu Zeng, Fei Zhu, Jiayi Wang, Xukai Wang, Jingang Zhou, Hongbo Zhao, Wenzhuo Liu, Shijie Ma, Da-Han Wang, et al. A comprehensive survey on continual learning in generative models. arXiv preprint arXiv:2506.13045, 2025. [12] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018. [13] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340–8349, 2021. [14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. [15] Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering. In Proceedings of the computer vision and pattern recognition conference, pages 19608–19617, 2025. [16] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.

[17] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017. [18] Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11847–11857, 2023. [19] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. [20] Minh Le, Anh Nguyen, Huy Nguyen, Chau Nguyen, Anh Tuan Tran, and Nhat Ho. Revisit visual prompt tuning: The expressiveness of prompt experts. In The Fourteenth International Conference on Learning Representations. [21] Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14369–14387, 2024. [22] Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022. [23] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. [24] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. [25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. [26] Yiyang Liu, James C Liang, Heng Fan, Wenhao Yang, Yiming Cui, Xiaotian Han, Lifu Huang, Dongfang Liu, Qifan Wang, and Cheng Han. All you need is one: Capsule prompt tuning with a single vector. arXiv preprint arXiv:2510.16670, 2025. [27] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International conference on machine learning, pages 22631–22648. PMLR, 2023. [28] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. [29] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. [30] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. [31] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019. [32] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015. [33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.

[34] Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. Advances in neural information processing systems, 34:6747–6761, 2021. [35] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. [36] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attentionbased prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909–11919, 2023. [37] Shengbang Tong, David Fan, Jiachen Li, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17001–17012, 2025. [38] Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, and Hinrich Schütze. Rehearsal-free modular and compositional continual learning for language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 469–480, 2024. [39] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10658–10671, 2023. [40] Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Inscl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 663–677, 2024. [41] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631–648. Springer, 2022. [42] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. [43] Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01364, 2024. [44] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vinod Vydiswaran, and Hao Ma. Idpg: An instance-dependent prompt generation method. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5507–5521, 2022. [45] Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-aware prompt tuning. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 2880–2889, 2025. [46] Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye, De-Chuan Zhan, and Da-Wei Zhou. Same: Stabilized mixture-of-experts for multimodal continual instruction tuning. arXiv preprint arXiv:2602.01990, 2026. [47] Yahan Yu, Duzhen Zhang, Yong Ren, Xuanle Zhao, Xiuyi Chen, and Chenhui Chu. Progressive lora for multimodal continual instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2779–2796, 2025. [48] Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. Modalprompt: Towards efficient multimodal continual instruction tuning with dual-modality guided prompt. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12137–12152, 2025. [49] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. Advances in neural information processing systems, 35:29776–29788, 2022.

[50] Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. [51] Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024. [52] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Guoyin Wang, et al. Instruction tuning for large language models: A survey. ACM Computing Surveys, 58(7):1–36, 2026. [53] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. [54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.

Appendix In this appendix, we provide additional details and supplementary analyses for D RAPE , including evaluation metrics, null-space projection analysis, additional benchmark results, sensitivity analysis, efficiency comparison, routing diagnostics, qualitative case studies, descriptions of compared methods, detailed result matrices, and broader-impact discussion. Section A provides the definitions of the continual evaluation metrics used in our experiments, including backward transfer and mean accuracy. Section B analyzes why null-space gradient projection can reduce interference with previous tasks. Section C reports stage-wise continual performance on CoIN and UCIT, analyzing forgetting behavior throughout sequential training. Section D analyzes the sensitivity of different methods to the number of prompts or LoRA experts. Section E compares the training time, active trainable parameters, and per-task storage overhead of D RAPE with representative baselines. Section F presents routing diagnostics, including the row-normalized routing confusion matrix. Section G provides qualitative case studies on GQA, VQAv2, and OCR-VQA to illustrate instancespecific prompt behavior. Section H summarizes the compared baseline methods. Section I summarizes the training and inference procedures of D RAPE . Section J reports the detailed continual-learning result matrices on CoIN and UCIT.

Details of Evaluation Metrics

We provide the detailed definitions of the continual evaluation metrics used in our experiments. Let As,t denote the performance on task s after training up to task t, where T is the total number of tasks. Backward Transfer (BWT). BWT measures the performance drop on previously learned tasks and Pt−1 1 quantifies catastrophic forgetting at incremental stage t, defined as Bt = t−1 s=1 (As,s − As,t ) for t = 2, . . . , T . A lower BWT indicates better resistance to catastrophic forgetting. Mean Accuracy (MA). MAP evaluates the average performance over all learned tasks at incremental t 1 stage t, defined as Mt = t s=1 As,t for t = 1, . . . , T . A higher MA indicates better continual learning ability throughout the task sequence. For stage-wise continual evaluation, we report both metrics across incremental stages and further average them over t = 2, . . . , T .

Analysis of Null-Space Gradient Projection

We provide a first-order analysis to illustrate why the null-space gradient projection can reduce interference on previously learned tasks. We follow the notation in Section 4.2 and make the layer (t) (t) index explicit: Mℓ , Vℓ,⊥ , and Πℓ denote the second-moment matrix, complementary basis, and projection matrix for projector layer ℓ, respectively. The same argument applies independently to each linear layer of the shared projector π(·). Let ṽ(ℓ) ∈ Rdℓ denote the input feature to layer ℓ. After observing tasks up to Dt , the empirical second-moment matrix maintained for this layer is h i (t) Mℓ = Ê≤t ṽ(ℓ) ṽ(ℓ)⊤ , (20) where Ê≤t denotes the empirical average over features collected from tasks up to t. Its singular value decomposition is (t)

Mℓ = Uℓ Σℓ Vℓ⊤ ,

Σℓ = diag(σℓ,1 , . . . , σℓ,dℓ ), 14

(21)

with σℓ,1 ≥ · · · ≥ σℓ,dℓ ≥ 0. Following the energy criterion in Section 4.2, we split the right singular vectors into the retained principal subspace and its complement: Vℓ = [Vℓ,∥ , Vℓ,⊥ ],

(22)

where Vℓ,∥ contains the top rℓ directions and Vℓ,⊥ contains the remaining directions. Since rℓ is selected to retain at least an ϵ fraction of the empirical energy, we have Prℓ k=1 σℓ,k ≥ ϵ, (23) Pdℓ k=1 σℓ,k which implies dℓ X

σℓ,k ≤ (1 − ϵ)

k=rℓ +1

dℓ X

σℓ,k .

(24)

k=1

Therefore, the empirical energy of previous-task features in the complementary subspace is bounded by dℓ h i X (t) (t) ⊤ (ℓ) 2 ⊤ Ê≤t ∥Vℓ,⊥ ṽ ∥2 = tr Vℓ,⊥ Mℓ Vℓ,⊥ = σℓ,k ≤ (1 − ϵ)tr(Mℓ ). (25) k=rℓ +1

In practice, we set ϵ = 0.99, which retains most of the empirical feature energy from previous tasks. As a result, previous-task features are expected to have only small components in the complementary (ℓ) subspace. Intuitively, for an earlier-task feature ṽold , this means (ℓ)

⊤ Vℓ,⊥ ṽold ≈ 0.

(26)

Now consider the weight matrix Wℓ of layer ℓ. During training on the next task, the gradient is projected as (t) ⊤ ∇Wℓ′ = ∇Wℓ Πℓ = ∇Wℓ Vℓ,⊥ Vℓ,⊥ . (27) (ℓ)

For an earlier-task feature ṽold , the first-order change in the layer output induced by this projected update is (ℓ) ⊤ (ℓ) ∆oold = −η∇Wℓ Vℓ,⊥ Vℓ,⊥ ṽold , (28) where η is the learning rate. Since the projected update acts only through the complementary component of the old feature, we can rewrite it as (ℓ) ⊤ (ℓ) ∆oold = −η∇Wℓ Vℓ,⊥ Vℓ,⊥ ṽold ≈ 0. (29) Equivalently, its magnitude can be bounded as (ℓ)

(ℓ)

⊤ ∥∆oold ∥2 ≤ η∥∇Wℓ ∥2 Vℓ,⊥ ṽold

. 2

(30)

Thus, when earlier-task features are well captured by the retained principal subspace, their components in Vℓ,⊥ are small, and the projected update has a limited first-order effect on their layer outputs. (ℓ)

In the ideal case where ṽold ∈ span(Vℓ,∥ ), we have (ℓ)

⊤ Vℓ,⊥ ṽold = 0,

(31)

and therefore the projected update does not change the layer output for that feature to first order. Overall, this analysis shows why null-space projection helps reduce first-order interference on earlier-task representations while preserving update directions for new-task adaptation.

Continuous Continual Performance

We report stage-wise continual metrics on CoIN and UCIT to further analyze forgetting behavior during sequential training. Table 4 shows the results on CoIN. Compared with previous methods, D RAPE achieves the lowest average backward-transfer value and the highest average mean accuracy among the compared methods, with B = 1.50 and M = 68.92. Relative to the strongest baseline 15

ModalPrompt, this corresponds to 2.22 points lower backward transfer and 12.82 points higher mean accuracy. These results indicate that D RAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. Table 5 further reports the stage-wise continual metrics on UCIT. D RAPE again achieves the lowest average backward transfer, with B = 0.04, substantially lower than the other baselines. Its Bt values remain close to zero across incremental stages and even become negative at early stages, indicating limited forgetting and slight positive backward transfer. Meanwhile, D RAPE obtains an average mean accuracy of 76.31, which is comparable to the best-performing baseline SEFE (76.35) and higher than the other compared methods. These results further show that D RAPE preserves previously acquired knowledge effectively while retaining competitive adaptation to newly introduced tasks. Method

TextVQA B2 ↓ M 2 ↑

ImageNet B3 ↓ M 3 ↑

GQA B4 ↓ M4 ↑

VizWiz B5 ↓ M5 ↑

Grounding B6 ↓ M6 ↑

OCR-VQA B8 ↓ M 8 ↑

Average B↓ M↑

Finetune CODA-Prompt MoELoRA ModalPrompt ProgLoRA Ours

44.30 11.54 41.31 6.55 17.21 3.37

65.53 27.70 52.47 4.40 6.78 2.48

64.42 14.38 32.76 3.16 17.60 1.32

51.98 12.78 33.81 4.51 12.27 1.07

67.08 5.71 43.89 23.65 35.61 29.06 12.72 39.28 14.05 39.42 7.27 45.46 41.41 25.59 30.80 34.34 26.12 37.48 3.98 50.96 2.02 54.07 1.41 55.06 18.31 50.60 9.56 57.66 7.53 59.09 0.83 68.08 0.78 67.80 0.66 67.48

53.26 26.20 14.34 43.18 36.95 36.29 3.72 56.10 12.75 60.42 1.50 68.92

44.14 57.88 43.13 64.50 59.92 65.11

32.52 34.05 34.08 56.34 73.60 74.37

22.75 43.44 41.71 57.63 60.71 71.51

25.55 42.76 37.71 54.15 61.35 68.07

VQAv2 B7 ↓ M7 ↑

Table 4: Continual performance metrics at each incremental stage on the CoIN benchmark. Bt and Mt stand for Backward Transfer and Mean Accuracy at incremental stage t. Method

ArxivQA B2 ↓ M2 ↑

VizWiz B3 ↓ M3 ↑

IconQA B4 ↓ M4 ↑

CLEVR B5 ↓ M5 ↑

Flickr30k B6 ↓ M6 ↑

Average B↓ M↑

LoRA-FT O-LoRA MoELoRA ModalPrompt CL-MoE HiDe SEFE Drape

1.37 1.33 1.16 0.07 3.90 2.04 0.57 -0.14

8.81 6.47 5.85 0.00 8.40 1.53 4.35 -0.02

13.55 11.18 11.78 0.04 13.04 4.80 8.51 0.02

15.28 13.20 10.31 0.04 11.39 6.44 9.30 0.17

18.78 13.99 14.63 0.15 15.56 9.20 11.33 0.16

11.56 9.23 8.74 0.06 10.46 4.80 6.81 0.04

90.83 92.40 91.44 69.20 89.84 90.55 90.73 88.58

75.77 78.26 77.52 62.20 75.65 78.18 78.19 77.59

71.04 72.70 65.66 56.45 64.74 74.08 74.17 73.82

68.40 69.02 65.68 54.57 65.15 67.94 72.15 72.16

61.24 64.35 58.98 52.53 58.49 62.29 66.54 69.41

73.46 75.35 71.85 58.99 70.77 74.61 76.35 76.31

Table 5: Continual performance metrics at each incremental stage on the UCIT benchmark. Bt and Mt denote the backward transfer and mean accuracy at incremental stage t.

Prompt/LoRA Number Sensitivity

We further analyze how the number of prompts or LoRA experts affects continual performance on the CoIN benchmark. Figure 5 illustrates the final average accuracy and the last-task accuracy when varying the number of prompts for each task and the number of MoE LoRA experts. Specifically, the x-axis values {2, 5, 10} denote the number of prompts used for each task in prompt-based methods and the number of LoRA experts in MoELoRA. Generally, increasing the number of prompts or LoRA experts brings modest improvements, and the gains gradually saturate once the number becomes moderately large. For baselines like MoELoRA, CODA-Prompt, and ModalPrompt, increasing the corresponding number brings steady enhancements, whereas D RAPE remains relatively stable across the tested settings. Considering the trade-off between effectiveness and efficiency, we set the default number of prompts to Lp = 10, as it delivers strong performance without further increasing the prompt budget.

Efficiency Comparison

Table 6 compares training time and active trainable parameters against representative prompt-based and LoRA-based baselines. Here, “Trainable Param” refers to the parameters updated during a single task update, i.e., the current task generator plus the shared projector. Under this active-update metric, D RAPE updates 32M parameters per task, which is substantially smaller than MoELoRA and CODA-Prompt, while also requiring less training time in our setup. In terms of storage, each new task adds only one frozen task-specific generator, occupying about 21.04 MB in our setting. 16

MoELoRA CODA-Prompt

ModalPrompt Ours

Average Accuracy (%)

Last Accuracy (%)

70 60 50 40 30 20 10

MoELoRA CODA-Prompt

ModalPrompt Ours

70 60 50 40 30

Number of Prompts/LoRA

Figure 5: Impact of prompt and LoRA expert numbers on the CoIN benchmark. We vary the number of prompts for each task in prompt-based methods and the number of MoE LoRA experts in MoELoRA. Left: last-task accuracy. Right: final average accuracy. This per-task growth is modest: roughly 50 task-specific generators would amount to about 1 GB of additional storage, which is still about one tenth of the full LLaVA backbone whose weights alone require more than 10 GB. These results indicate that the proposed adaptation mechanism remains lightweight both during training and in long-term per-task expansion. Method

Training (Hour)

Trainable Param

MoELoRA 7.27 360M CODA-Prompt 3.43 165M ModalPrompt 2.54 20M Ours 1.59 32M Table 6: Efficiency comparison in terms of training time and active trainable parameters on the CoIN benchmark. We average the training time for one epoch across datasets.

Routing Diagnostics

Figure 6 complements the routing ablation by showing where the learned router departs from oracle task selection. The dominant diagonal entries indicate that the router usually activates the correct task generator, which helps explain why learned routing remains close to the oracle-routing upper bound and far outperforms the no-routing variant in the main results. The largest off-diagonal errors reveal two main patterns. First, GQA and VQAv2 are strongly confused with each other, with GQA samples often routed to VQAv2 and VQAv2 samples often routed to GQA. Second, ScienceQA has a more dispersed routing pattern, with a noticeable fraction of samples routed to OCR-VQA and VQAv2. These errors do not necessarily lead to proportional drops in final task performance, since the misrouted generators often correspond to semantically related visual-question-answering or textreading behaviors. This helps explain why the learned router remains close to oracle routing despite imperfect task-level routing accuracy. Because task sizes differ substantially, the row-normalized view is more informative than raw confusion counts for cross-task comparison.

Case Study

To further examine the qualitative behavior of D RAPE beyond aggregate accuracy, we present representative case studies on GQA, VQAv2, and OCR-VQA. In all figures, Baseline denotes the static task-level prompt variant, while Drape denotes our instance-specific prompt variant. Each example reports the prediction of the two variants together with the ground-truth answer. The left example in each figure corresponds to a relatively simple case where both methods succeed, while 17

100% 66.7%

5.7%

1.6%

1.4%

0.7%

1.7%

8.7%

13.6%

(2,830)

(240)

(67)

(60)

(28)

(71)

(367)

(578)

0.1%

0.2%

0.1%

1.7%

(6)

(10)

(7)

(84)

–

97.9%

TextVQA (n=5,000)

–

ImageNet (n=5,050)

–

GQA (n=12,578)

–

0.9% (37)

Grounding (n=30,969)

–

VQAv2 (n=214,354)

–

VizWiz (n=4,319)

OCRVQA (n=99,926)

ien Sc

–

0.9% (109)

1.7%

0.1%

<0.1%

96.3%

(73)

(5)

(1)

(4,158)

–

<0.1%

0.4%

27.2%

1.8%

(20)

(828)

(58,297)

(3,952)

–

A GQ

A VQ

Ne ge

–

(1)

(7,817)

(369)

–

<0.1%

62.1%

0.4%

(5,049)

(18)

(131)

x Te

99.9%

0.1%

Q ce

–

(4,893)

100% (30,969)

–

(37)

iz zW Vi

0.5%

(22)

(23)

–

70.5%

<0.1%

(151,188)

(69)

<0.1%

99.4%

(15)

(99,374)

v2 QA

in nd

–

(4,634)

–

<0.1%

60%

36.8%

–

80%

Routing Ratio

Source Category

ScienceQA (n=4,241)

40%

20%

A VQ CR

Routed Category

Figure 6: Row-normalized routing confusion matrix on the final task. Each row corresponds to a ground-truth task and sums to 100%. Diagonal entries indicate routing accuracy, while off-diagonal entries reveal the dominant misrouting patterns. The largest off-diagonal entries occur between GQA and VQAv2, and from ScienceQA to OCR-VQA and VQAv2.

the middle and right examples illustrate more challenging cases in which the static prompt fails but D RAPE produces the correct answer. These examples provide qualitative evidence that a single shared task-level prompt can be sufficient for standard inputs, but may become less reliable when the required adaptation depends on fine-grained query–image interactions. GQA. Figure 7 shows three examples from GQA. In the left example, both variants correctly answer the query “What device is sitting next to the mouse pad?” with “Keyboard,” suggesting that both prompt strategies can handle a direct object-grounding case. The middle example requires more precise spatial grounding: for the query “Where is the skinny person standing?”, the static prompt predicts the coarse spatial answer “Left,” whereas D RAPE correctly identifies the supporting object, “Table.” In the right example, the query “Does the calf have brown color and large size?” requires compositional attribute verification. The static variant incorrectly predicts “Yes,” while D RAPE correctly answers “No.” These cases suggest that instance-specific prompt generation can better accommodate shifts from simple grounding to finer spatial reasoning and compositional attribute judgment within the same task. VQAv2. Figure 8 presents examples from VQAv2. In the left example, both variants correctly answer “What does the truck on the left sell?” with “ice cream,” indicating that a static task-level prompt can be adequate for a straightforward open-domain VQA query. The middle example asks “Is this cat clawing the chair?”, which depends on fine-grained local action recognition. The static prompt predicts “no,” whereas D RAPE correctly predicts “yes.” In the right example, the query “What color are the towels?” requires local attribute recognition; the static prompt answers “green,” while D RAPE correctly answers “blue.” These examples indicate that the benefit of instance-specific prompting is not limited to relational reasoning, but also extends to subtle local visual evidence in general VQA. OCR-VQA: same image, different queries. Figure 9 fixes the image and varies the query on the same book cover. This setting isolates the effect of query-dependent adaptation, since the visual 18

Query: What device is sitting next to the mouse pad? Answer the question using a single word or phrase.

Query: Where is the skinny person standing? Answer the question using a single word or phrase.

Query: Does the calf have brown color and large size? Answer the question using a single word or phrase.

Baseline:

Keyboard

Baseline:

Left

Baseline:

Yes

Drape:

Keyboard

Drape:

Table

Drape:

Ground truth:

Keyboard

Ground truth:

Table

Ground truth:

Figure 7: Case studies on GQA. The left example is a relatively simple case where both variants are correct. The middle and right examples require finer spatial grounding and compositional attribute verification, respectively. In these more challenging cases, the static task-level prompt fails, whereas the instance-specific prompt generated by D RAPE produces the correct answer. Query: What does the truck on the left sell? Answer the question using a single word or phrase.

Query: Is this cat clawing the chair? Answer the question using a single word or phrase.

Query: What color are the towels? Answer the question using a single word or phrase.

Baseline:

ice cream

Baseline:

green

Drape:

ice cream

Drape:

yes

Drape:

blue

Ground truth:

ice cream

Ground truth:

yes

Ground truth:

blue

Figure 8: Case studies on VQAv2. The left example is a relatively simple case where both variants succeed. The middle and right examples require fine-grained local action recognition and attribute recognition, respectively. In these cases, the static task-level prompt produces incorrect predictions, whereas D RAPE remains aligned with the visual evidence relevant to the current query.

input remains unchanged while the required reasoning behavior changes. In the left example, both variants correctly identify the author as “Edward De Bono.” In the middle example, the query shifts to title extraction. The static prompt predicts “Unanswerable,” whereas D RAPE recovers the full title “Creativity Workout: 62 Exercises to Unlock Your Most Creative Ideas.” In the right example, the query asks for the book genre. The static prompt predicts the incorrect category “Arts & Photography,” while D RAPE correctly answers “Health, Fitness & Dieting.” These results suggest that even for a fixed image, different textual queries may require different prompt behaviors, and that instance-specific prompt generation can better adapt to such query-dependent semantic shifts. OCR-VQA: same query, different images. Figure 10 provides the complementary analysis by fixing the query type and varying the image. All examples ask for the book title. In the left example, both variants correctly recover the title from a relatively simple cover. In the middle example, the static prompt predicts the incomplete title “2016 Cat Naps Daily Calendar,” while D RAPE correctly recovers “Cat Naps 2016 Boxed/Daily Calendar.” In the right example, the static prompt predicts “Unanswerable,” whereas D RAPE correctly identifies “OMG Facts 2016 Boxed/Daily Calendar.” This comparison suggests that even when the instruction form is fixed, the appropriate adaptation still depends on the specific visual layout and textual content of each image. Overall observation. Across GQA, VQAv2, and OCR-VQA, we observe a consistent qualitative pattern. Static task-level prompts are often sufficient for relatively simple or canonical samples,

Query: Who wrote this book? When the provided information is insufficient, respond with 'Unanswerable’. Answer each question using a single word or phrase. Baseline:

Edward De Bono

Drape:

Edward De Bono

Ground truth:

Edward De Bono

Query: What is the title of this book? When the provided information is insufficient, respond with 'Unanswerable’. Answer each question using a single word or phrase. Baseline:

Unanswerable

Drape:

Creativity Workout: 62 Exercises to Unlock Your Most Creative Ideas

Ground truth: Creativity Workout: 62 Exercises to Unlock Your Most Creative Ideas

Query: What is the genre of this book? When the provided information is insufficient, respond with 'Unanswerable’. Answer each question using a single word or phrase. Baseline:

Arts & Photography

Drape:

Health, Fitness & Dieting

Ground truth:

Health, Fitness & Dieting

Figure 9: OCR-VQA case study with a fixed image and different queries. Although the visual input is unchanged, the required behavior varies across queries, ranging from author recognition to title extraction and genre identification. The static task-level prompt handles the easier query but becomes less reliable on the more demanding ones, whereas D RAPE adapts more effectively to the query-specific requirements. 19

Query: What is the title of this book? When the provided information is insufficient, respond with 'Unanswerable’. Answer each question using a single word or phrase. Baseline:

The Complete Runner’s Day-by-Day Log 2016 Calendar

Baseline:

2016 Cat Naps Daily Calendar

Baseline:

Unanswerable

Drape:

The Complete Runner’s Day-by-Day Log 2016 Calendar

Drape:

Cat Naps 2016 Boxed/Daily Calendar

Drape:

OMG Facts 2016 Boxed/Daily Calendar

Ground truth: The Complete Runner’s Day-by-Day Log 2016 Calendar

Ground truth: Cat Naps 2016 Boxed/Daily Calendar

Ground truth: OMG Facts 2016 Boxed/Daily Calendar

Figure 10: OCR-VQA case study with a fixed query type and different images. Both variants correctly recover the title in a relatively simple example, while the more challenging covers require finer OCR-style extraction. In these harder cases, the static task-level prompt produces an incomplete answer or falls back to “Unanswerable,” whereas D RAPE recovers the correct full title.

but they can become brittle when the task requires fine-grained grounding, local visual recognition, compositional attribute verification, or OCR-style text extraction. By generating prompts conditioned on the current query–image pair, D RAPE provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. These case studies complement the quantitative results by illustrating how instance-specific prompt generation can improve within-task adaptation in multimodal continual instruction tuning.

Comparing Methods

In this section, we provide brief descriptions of the methods compared in the paper. Finetune sequentially fine-tunes the model on each incoming task without an explicit mechanism to preserve previous knowledge. MoELoRA maintains multiple task-related LoRA experts together with a gating function, so that only part of the additional parameters are activated for each input. In this way, it aims to preserve previously acquired knowledge while adapting to the current sample through expert selection. L2P maintains a prompt pool in memory space and retrieves relevant prompts for each input, thereby balancing task-invariant and task-specific knowledge through prompt selection. DualPrompt introduces two types of prompts, namely general prompts and expert prompts, to model task-invariant and task-specific knowledge, respectively. These prompts are attached to different transformer layers to facilitate continual adaptation and knowledge retention. CODA-Prompt learns a set of input-conditioned prompts for rehearsal-free continual learning and dynamically composes them during inference. ModalPrompt extends prompt-based continual learning to the multimodal setting by introducing learnable prompts for vision-language adaptation while keeping most backbone parameters frozen. ProgLoRA incrementally expands task-specific LoRA parameters across tasks and preserves previous knowledge by freezing historical modules. We follow the standard configuration in its released implementation and use the default LoRA hyperparameters. LoRA-FT performs sequential low-rank adaptation without an explicit forgetting-mitigation mechanism. O-LoRA reduces interference across tasks by imposing orthogonality constraints on LoRA updates. It aims to preserve previously learned knowledge while maintaining adaptation ability on new tasks. CL-MoE is an expert-based continual tuning method that improves adaptation through taskspecialized modules and expert routing. It dynamically selects or combines experts according to the current input. HiDe is motivated by CKA similarity analysis, which suggests that top and lower transformer layers exhibit different representation patterns in continual learning. It hierarchically decouples adaptation by using task-specific LoRA expansion with dual-modality anchor matching in higher layers, while fusing LoRAs in lower layers to preserve shared knowledge without router training. 20

Algorithm 1 Training of D RAPE for Multimodal Continual Instruction Tuning Require: Task stream {Dt }Tt=1 , frozen vision encoder ϕ, frozen LLM θ, shared projector π, generator set {Gt }Tt=1 , energy threshold ϵ Ensure: Frozen task-specific generators {Gt }Tt=1 , shared projector π, projection matrices {Π(t) }Tt=1 , task prototypes {ct }Tt=1 1: Initialize cumulative feature statistics M(0) ← 0 2: for t = 1 to T do 3: Activate current generator Gt ; freeze historical generators {G1 , . . . , Gt−1 } 4: Attach forward hooks to collect projector-layer input features ṽ 5: Temporarily cache fused CLIP routing features ẽi for samples in Dt 6: for each mini-batch B ⊂ Dt do 7: Compute visual features w = π(ϕ(v)) and text embeddings u = ψ(q) 8: Generate instance-specific prompts Pi = Gt (wi , ui ) 9: Compute autoregressive loss Lt with input [Pi ; z] 10: if t > 1 then 11: Apply null-space projection Π(t−1) to gradients of π 12: end if 13: Update Gt and π 14: end for 15: Freeze Gt 16: Update cumulative second-moment statistics M(t) 17: Compute Π(t) from the complementary subspace of M(t) 18: Initialize ct as the normalized mean of cached CLIP features 19: if t > 1 then 20: Refine ct with the prototype classification loss in Eq. (16) 21: end if 22: Register ct and discard cached instance-level CLIP features 23: end for

Algorithm 2 Inference of D RAPE without Task Labels Require: Test sample (v, q), frozen generators {Gt }Tt=1 , task prototypes {ct }Tt=1 , shared projector π Ensure: Predicted answer ŷ 1: Compute routing feature ẽ = norm([ξ(q); γ(v)]) 2: Compute prototype scores st = cos(ẽ, ct ) for all tasks t 3: Select generator index t∗ = arg maxt∈{1,...,T } st 4: Compute visual features w = π(ϕ(v)) and text embeddings u = ψ(q) 5: Generate prompt Pi = Gt∗ (wi , ui ) 6: Output prediction ŷ from the frozen LLM conditioned on [Pi ; z]

SEFE addresses forgetting through answer-style diversification and RegLoRA. RegLoRA regularizes the top-M % elements of LoRA update matrices to preserve critical historical knowledge during continual adaptation.

Pseudocode of D RAPE

We summarize the training and inference procedures of D RAPE in Algorithms 1 and 2. During training, D RAPE learns one task-specific generator at a time, updates the shared visual projector with null-space gradient projection, and registers a compact CLIP-space prototype after each task. During inference, the prototype router selects the most relevant generator without requiring task labels. 21

Table 7: Detailed continual instruction tuning results matrices for different methods on the CoIN benchmark. Finetune

ScienceQA

TextVQA

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

82.45 38.15 0.96 13.91 8.46 0.00 9.10 26.00

50.14 0.58 15.78 25.17 0.00 27.58 25.38

96.03 5.67 4.60 0.00 6.62 28.51

55.65 38.12 0.27 43.92 33.07

51.42 0.00 19.10 26.52

34.00 0.03 0.10

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

72.83 68.07 32.05 47.53 65.94 5.74 32.57 54.42

57.16 26.73 46.02 37.68 42.96 48.65 46.04

39.43 18.03 1.72 33.92 7.41 30.36

60.47 56.29 39.44 47.32 57.09

CODA-Prompt

ScienceQA

TextVQA

ImageNet

GQA

70.26 58.72 36.96 50.78 55.37 33.56 48.34 58.15

57.05 34.95 53.52 47.21 47.67 49.54 50.16

30.26 10.12 6.78 32.07 20.72 24.04

59.35 56.43 44.62 47.72 54.33

48.01 43.39 35.73 48.94

34.42 13.03 17.83

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

77.05 70.50 68.57 68.82 67.48 66.58 68.12 68.42

58.50 58.18 56.08 55.05 55.68 56.43 56.40

42.26 43.43 37.60 35.92 40.22 41.13

L2P

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

ModalPrompt ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

ImageNet

GQA

62.17 61.81 61.95 60.92 61.11

VizWiz

Grounding

OCRVQA

MoELoRA

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

59.17 40.00

52.92

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

75.78 34.47 22.61 32.37 45.32 58.76 33.01 47.34

51.80 0.04 34.04 38.13 9.08 48.42 32.91

79.60 42.48 2.63 5.64 10.61 38.73

57.95 43.80 31.87 49.78 37.15

58.70 11.45 32.23 42.48

36.77 1.75 0.97

64.58 42.77

57.50

VQAV2

OCRVQA

Dualprompt

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

VQAV2

OCRVQA

67.16 52.20 28.49 49.70 57.88 18.27 36.77 56.40

53.12 24.77 47.94 51.17 39.64 49.85 47.12

46.40 12.06 21.34 29.77 22.48 34.96

55.10 48.03 44.06 27.96 42.03

51.62 35.97 41.08 44.14

30.82 13.51 12.01

61.27 54.43

53.36

VQAV2

OCRVQA

VQAV2

OCRVQA

1.87 8.17 9.38

59.40 50.45

54.03

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

Grounding

VQAV2

OCRVQA

ProgLoRA

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

60.87 55.86

54.42

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

76.27 59.06 70.92 51.56 65.62 40.08 76.90 74.84

60.78 52.57 50.33 50.79 47.53 56.79 51.83

97.32 79.68 81.17 77.75 73.90 83.90

61.27 48.99 50.61 53.64 49.93

60.16 48.30 40.63 53.87

39.35 35.96 31.19

65.83 62.71

64.44

VQAV2

OCRVQA

D RAPE

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Grounding

VQAV2

OCRVQA

73.66 70.29 69.21 69.87 69.79 70.01 70.57 70.67

59.93 59.43 59.71 59.54 59.51 59.49 59.61

94.46 94.50 94.53 94.46 94.32 94.16

61.96 61.88 61.93 61.19 61.37

54.60 54.55 54.32 54.43

68.03 68.07 67.92

66.65 66.53

65.11

47.90 39.64 34.52 42.19 VizWiz

VQAV2

48.81 48.74 51.19 50.13

36.88 36.63 36.69

64.99 66.90

ScienceQA TextVQA ImageNet GQA VizWiz Grounding VQAV2 OCRVQA

59.68

Detailed Result Matrices

In Tables 7–8, we report the final accuracy matrices for all methods on CoIN and UCIT. Results may vary across hardware and software environments. Table 8: Detailed continual instruction tuning results matrices for different methods on the UCIT benchmark. Flickr30k

O-LoRA

ImgNet-R

ArxivQA

VizWiz

IconQA

CLEVR

78.33 61.77

58.22

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

91.40 90.07 81.50 82.40 81.07 77.50

94.73 91.70 77.73 77.87 78.07

61.59 54.06 50.62 44.50

76.63 62.00 63.13

73.53 64.73

58.16

CLEVR

Flickr30k

ModalPrompt

ImgNet-R

ArxivQA

VizWiz

IconQA

CLEVR

Flickr30k

51.10 51.03 51.10 51.20 51.17 51.07

87.37 87.37 87.27 87.23 87.27

48.14 48.03 47.96 48.11

39.30 39.40 39.23

47.07 46.57

42.93

ImgNet-R

ArxivQA

VizWiz

IconQA

CLEVR

Flickr30k

57.42

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

90.87 88.83 88.17 87.00 86.23 84.03

92.27 91.90 89.70 90.07 90.73

54.46 46.51 44.28 44.43

73.13 64.37 58.93

54.77 41.37

54.25

Flickr30k

D RAPE

ImgNet-R

ArxivQA

VizWiz

IconQA

CLEVR

Flickr30k

57.92

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

85.13 85.27 85.20 85.27 85.23 85.07

91.90 91.87 91.63 91.57 91.60

55.70 55.77 55.62 55.68

62.63 62.27 62.53

66.10 65.77

55.82

LoRA-FT

ImgNet-R

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

92.00 90.63 74.00 72.73 68.97 58.03

91.03 91.40 78.53 77.60 77.63

61.90 53.03 49.43 44.39

79.87 67.67 67.40

MoELoRA

ImgNet-R

ArxivQA

VizWiz

IconQA

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

91.23 90.07 80.30 80.53 79.37 70.07

92.80 92.03 77.20 77.40 77.70

60.22 51.19 48.68 44.69

53.73 51.30 50.03

71.67 54.03

57.34

CL-MoE

ImgNet-R

ArxivQA

VizWiz

IconQA

CLEVR

Flickr30k

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

91.30 87.40 76.23 77.33 75.73 66.33

92.27 90.53 76.20 75.70 77.00

60.18 51.09 48.27 44.78

54.33 52.80 51.87

73.23 53.53

ImgNet-R

ArxivQA

VizWiz

IconQA

CLEVR

90.80 90.23 83.60 84.17 82.90 80.83

91.23 89.73 77.20 77.13 78.00

61.24 56.36 53.54 47.01

SEFE ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

ArxivQA

VizWiz

IconQA

78.93 71.43 69.63

CLEVR

75.73 65.83

ImgNet-R ArxivQA VizWiz IconQA CLEVR Flickr30k

HiDe

Flickr30k

Record · ID 175281 · SHA-256 343120a24f15a73b
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.