PRISM: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning Jun-Tao Tang2,* , Yu-Cheng Shi2,* , Zhen-Hao Xie1,2 , Da-Wei Zhou1,2,† 1 School of Artificial Intelligence, Nanjing University, China 2 National Key Laboratory for Novel Software Technology, Nanjing University, China * Equal contribution † Correspondence: [email protected]
arXiv:2605.26110v1 [cs.LG] 25 May 2026
Abstract
instruction-following framework (Lee et al., 2024), thereby achieving unprecedented versatility. However, real-world deployment operates in dynamic environments where data arrives as a continuous stream (Krempl et al., 2014). To maintain longterm utility, MLLMs must continuously absorb new knowledge and adapt to emerging instruction formats via continual instruction tuning. Conventional fine-tuning methods, when applied sequentially to such evolving data streams, tend to overwrite previously learned representations, resulting in catastrophic forgetting of prior capabilities (McCloskey and Cohen, 1989; Zhou et al., 2024). To address this fundamental challenge, Multimodal Continual Instruction Tuning (MCIT) (Chen et al., 2024; Xie et al., 2026) has emerged as a critical research paradigm, focusing on equipping MLLMs with the capacity to learn incrementally while rigorously preserving established knowledge. Current MCIT research faces significant engineering challenges. Most existing methods are implemented by directly modifying the base MLLM training codebase. Given the architectural complexity of modern MLLMs, such modifications lead to highly divergent code structures and training logic across approaches. In existing toolkits (Chen et al., 2024; Guo et al., 2025c), each method maintains a full copy of the MLLM codebase, tightly coupling algorithmic logic with core training infrastructure. Consequently, these frameworks lack a highly integrated and decoupled architecture. This structural fragmentation obscures core implementation details, making code reuse and subsequent development significantly more challenging. Furthermore, many traditional continual learning techniques do not support essential large-scale training infrastructure, such as gradient checkpointing (Chen et al., 2016) and DeepSpeed (Rasley et al., 2020). This incompatibility severely restricts their scalability to MLLMs and hinders fair comparisons with continual learning baselines.
Multimodal Large Language Models (MLLMs) achieve versatility by reformulating diverse tasks into a unified instruction-following framework via instruction tuning. However, realworld deployment requires continuous adaptation to emerging tasks, motivating Multimodal Continual Instruction Tuning (MCIT). Despite its growing importance, current MCIT research is hindered by severe engineering bottlenecks. Existing methods are typically implemented by directly modifying the base MLLM codebase, which imposes substantial implementation overhead and yields methodspecific architectures that severely limit code reuse and fair comparison. To address this, we introduce PRISM, a plug-in reproducible codebase specifically designed for scalable MCIT research. It separates algorithmic development from the backbone implementation via a lightweight plugin registration mechanism, enabling new strategies to be integrated as independent plugins without modifying the underlying MLLM codebase, thereby eliminating structural fragmentation and accelerating method development. PRISM natively supports widely used large-scale training pipeline, thereby enabling reproducible and scalable MCIT experimentation. Code is available at https://github.com/LAMDA-CL/Prism.
1
Introduction
Recently, multimodal Large Language Models (MLLMs) (Bai et al., 2023; Zhu et al., 2023) have demonstrated remarkable potential across diverse domains, largely driven by their ability to interpret and execute tasks through natural language instructions. Through instruction tuning (Zhang et al., 2023a; Tong et al., 2025), MLLMs reformulate both unimodal vision tasks (e.g., image classification and visual grounding (Deng et al., 2021)) and vision-language tasks (e.g., visual question answering (Goyal et al., 2017)) into a unified 1
Table 1: Comparison of PRISM to existing representative MCIT toolkits.
bone, enabling new methods to be integrated with minimal code changes.
Feature CoIN MCITlib PRISM Implemented Algorithms 4 8 9 Supported Benchmarks 1 3 3 Unified Backbone Design ✗ ✗ ✓ Large-scale Experiment Support ✗ ✗ ✓
Tab. 1 systematically compares representative MCIT toolkits, revealing critical limitations in both quantitative coverage and engineering infrastructure. CoIN (Chen et al., 2024) exhibits a narrow scope, offering only 4 continual learning algorithms and relying on a single benchmark. MCITlib (Guo et al., 2025c) further expands this landscape with 8 mainstream algorithms and 3 evaluated datasets; however, both frameworks fundamentally lack a unified backbone and automated support for large-scale experiments. Consequently, they often necessitate fragmented configurations, manual intervention, and inconsistent training protocols, which hinder fair cross-method comparison and impede scalable, reproducible research. To bridge these gaps, we introduce PRISM, a plugin-driven reproducible infrastructure specifically designed for scalable MCIT research. It decomposes complex workflows into reusable components for methods, benchmarks, backbones, and evaluation modules, establishing a unified foundation for systematic development. This architecture supports broad algorithmic coverage that encompasses both traditional continual learning baselines and specialized MCIT approaches. By strictly decoupling algorithmic logic from infrastructure maintenance, PRISM transforms conventional research pipelines. New methods and benchmarks are integrated as standalone plugins through a lightweight registration mechanism, which isolates implementation details from the underlying MLLM codebase and eliminates structural redundancy. The modular design consolidates training logic into focused wrappers, enabling researchers to inspect and extend algorithms without navigating fragmented repositories. Furthermore, standardized training workflows combined with native support for distributed optimization techniques such as DeepSpeed ensure reproducible experimentation and enable efficient large-scale model training. Our main contributions are:
• A unified benchmarking suite with centralized configuration management, streamlining large-scale experiments and establishing a shared standard for fair method comparison.
2
Usage of PRISM
Dependencies. PRISM is built on a modular infrastructure stack for MCIT. The core neural architectures are implemented using PyTorch (Paszke et al., 2019) and DeepSpeed (Rasley et al., 2020) for memory-efficient distributed training, HuggingFace Transformers (Wolf et al., 2020) and PEFT (Mangrulkar et al., 2022) for backbone model management and parameter-efficient fine-tuning, and libraries such as NumPy (Harris et al., 2020), SciPy (Virtanen et al., 2020), tqdm (da Costa-Luis, 2019), and einops (Rogozhnikov, 2022) for numerical operations, monitoring, and tensor manipulation. Notably, our framework is highly extensible and seamlessly supports the integration of multiple custom multimodal backbones such as LLaVA (Liu et al., 2023), which comprises a CLIP (Radford et al., 2021) vision encoder, a large language model and a visual projector. The project relies solely on widely adopted open-source libraries. Supported Benchmarks. We consider 3 benchmarks with diverse domain gaps and task formats, following (Guo et al., 2025a; Xie et al., 2026): • CoIN (Chen et al., 2024): 8 sequential VQA and image understanding tasks: ScienceQA (Lu et al., 2022), TextVQA (Singh et al., 2019), ImageNet (Deng et al., 2009), GQA (Hudson and Manning, 2019), VizWiz (Gurari et al., 2018), Grounding (Kazemzadeh et al., 2014; Mao et al., 2016), VQAv2 (Goyal et al., 2017), and OCRVQA (Mishra et al., 2019). • UCIT (Guo et al., 2025a): 6 diverse tasks spanning visual reasoning and captioning: ImageNet-R (Hendrycks et al., 2021), ArxivQA (Li et al., 2024), Vizcap (Gurari et al., 2018), IconQA (Lu et al., 2021), CLEVR (Lindström and Abraham, 2022), and Flickr30k (Plummer et al., 2015). • TriGap (Xie et al., 2026): A long-horizon task sequence consisting of 10 tasks covering document understanding, medical imaging,
• A lightweight plugin design that decouples algorithm development from the MLLM back2
Supported Benchmarks TriGap ChemVQA PMCVQA
ChartQA
Roadside
DocVQA
FloodNetVQA
ArxivQA
InfographicVQA
CoIN ScienceQA TextVQA
VizWiz Rec
ImageNet
VQAv2
GQA
OCRVQA
UCIT Vizcap
CLEVR
ArxivQA CLEVR
Infra
LLaVA
DeepSpeed
InternVL
Gradient Checkpointing
Custom
PEFT Injection
method2
InternVL
Backbone
shared
… LLaVA (Whole MLLM codebase) eval train Language model Model Multimodal encoder
Flickr30k
IconQA
tuners
utils task1.sh Scripts task2.sh
IconQA
ImageNet-R
Backbone
method1 Peft
LLaVA
Prism
Existing toolbox
Multimodal projector llava_arch
Core PEFT Config
data eval
train CL_Model_Loader CL_Model_saver Method1_LoRA.py Method2_LoRA.py benchmark method backbone
Run.py (unified entrance) Method1.py
Peft Scripts
Method
LLaVA (Whole MLLM codebase)
Method2.py Method3.py
methodx
Implemented Methods CL-MoE CVPR 25
Structure-based DISCO ICCV 25
SAME ICML 26
HiDe-LLaVA ACL 25
Baselines
Replay-based Prompt-based Replay-LoRA
ModalPrompt EMNLP 25
Zero-shot
FT-LoRA
MoE-LoRA NeurIPS 24
Figure 1: Overview of the PRISM toolkit. Its plugin-based design decouples algorithmic development from infrastructure maintenance: new methods, backbones, and benchmarks integrate via lightweight registration, enabling reproducible and extensible MCIT research.
and domain-specific VQA: PMCVQA (Zhang et al., 2023b), DocVQA (Mathew et al., 2020), ChartQA (Masry et al., 2022), IconQA (Lu et al., 2021), InfographicVQA (Mathew et al., 2022), ArxivQA (Li et al., 2024), Roadside (Guan et al., 2026), ChemVQA (Sabando et al., 2020), FloodNetVQA (Sarkar et al., 2023; Rahnemoonfar et al., 2021), and CLEVR (Lindström and Abraham, 2022).
based methods, i.e., ModalPrompt (Zeng et al., 2025). All methods share a unified PEFT injection interface; new methods are seamlessly added via method/<name>/integration.py and registered with @CLMethodFactory.register(). Evaluation Metrics. Following standard continual learning evaluation protocols (Zhou et al., 2024; Guo et al., 2025a), we denote At as the model’s accuracy after the t-th incremental stage. PRISM employs the following primary metrics:
Task Organization. Following the protocols in continual instruction tuning (Chen et al., 2024; Guo et al., 2025a), PRISM organizes tasks sequentially. Each benchmark defines a fixed task order, where the model incrementally learns each task. Implemented Methods. PRISM implements a total of 9 representative continual learning methods and baselines for multimodal LLMs. These are systematically categorized into: (1) Baselines, which establish performance boundaries for evaluation, including Zero-shot (Zero-shot LLaVA without any fine-tuning), FT-LoRA (sequential full LoRA fine-tuning representing catastrophic forgetting), and MoE-LoRA (Chen et al., 2024); (2) Structure-based methods, which mitigate forgetting via explicit parameter isolation or routing, covering HiDe-LLaVA (Guo et al., 2025a), DISCO (Guo et al., 2025b), CL-MoE (Huai et al., 2025), and SAME (Xie et al., 2026); (3) Replay-based methods, i.e., Replay-LoRA (LoRA with taskpartitioned experience replay); and (4) Prompt-
• Last Accuracy AB : performance after the final task. P • Average Accuracy Ā = T1 Tt=1 At : mean accuracy across all incremental stages. • Forgetting Measure: FT is utilized to measure the average performance drop of each task from its best-achieved accuracy to the final stage, i.e., FT = 1 PT −1 t=1 maxt≤l≤T −1 (Al,t − AT,t ) T −1 For VQA tasks (e.g., VQAv2, TextVQA, GQA, VizWiz, ScienceQA), accuracy is computed via string-matching with normalization following the standard VQA evaluation protocol (Antol et al., 2015). For captioning tasks (e.g., Flickr30k, Vizcap), standard COCO metrics, including CIDEr (Vedantam et al., 2015), BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin, 2004), SPICE (Anderson et al., 2016)) are employed. For classification-style tasks 3
Table 2: Average performance of different methods on the UCIT benchmark. The best and second-best results are highlighted in bold and underline, respectively. Methods Zero-shot FT-LoRA Replay-LoRA MoE-LoRA (Chen et al., 2024) HiDe-LLaVA (Guo et al., 2025a) ModalPrompt (Zeng et al., 2025) CL-MoE (Huai et al., 2025) DISCO (Guo et al., 2025b) SAME (Xie et al., 2026)
ImageNet-R 18.88 29.33 76.93 58.43 87.62 80.50 64.12 88.88 89.91
ArxivQA 52.62 55.30 87.07 77.57 91.12 90.62 78.38 94.25 91.40
Vizcap 38.75 45.51 54.31 44.83 42.68 60.13 44.83 47.52 55.33
IconQA 21.25 26.13 56.43 68.90 57.62 63.50 62.00 69.50 77.51
CLEVER 21.12 13.07 36.40 56.73 31.00 55.75 50.75 60.75 68.85
Flickr30k 41.44 58.07 55.94 58.27 50.41 57.09 58.06 56.32 55.43
Average 32.34 37.90 61.18 60.79 60.08 67.93 59.69 69.54 73.07
Table 3: Average performance of different methods on TriGap benchmark. The best and second-best results are highlighted in bold and underline, respectively. Methods Zero-shot FT-LoRA Replay-LoRA MoE-LoRA (Chen et al., 2024) HiDe-LLaVA (Guo et al., 2025a) ModalPrompt (Zeng et al., 2025) CL-MoE (Huai et al., 2025) DISCO (Guo et al., 2025b) SAME (Xie et al., 2026)
PMCVQA 35.40 34.20 33.70 39.03 37.00 38.23 40.53 42.03 41.60
DocVQA 12.68 23.32 33.95 37.49 33.20 38.23 36.79 43.50 43.87
ChartQA 9.36 9.84 14.00 12.44 10.52 11.92 13.72 18.01 17.56
IconQA 19.27 37.07 46.67 43.43 41.97 44.73 52.70 63.13 64.03
InfographicVQA 5.06 23.53 28.97 35.17 24.09 37.37 32.27 38.23 39.57
(e.g., ImageNet-R, ArxivQA, IconQA, CLEVR), exact-match accuracy is used. Basic Usage. PRISM centralizes all experimental parameters (benchmarks, methods, training protocols) in human-readable Python configuration files, eliminating the need to modify underlying code. Users can simply adjust parameters within the configuration files and run standardized commands as:
Roadside 7.40 7.00 9.40 7.93 7.73 10.13 7.77 11.02 10.83
ChemVQA 5.30 12.70 15.90 20.70 11.17 12.43 18.33 22.13 21.77
FloodNetVQA 47.41 80.31 73.81 90.41 57.39 71.52 80.09 80.25 81.09
CLEVR 20.37 60.27 58.80 67.00 23.00 52.50 65.90 55.87 54.50
Average 21.60 37.21 39.08 44.45 32.53 40.15 44.11 46.54 46.53
Among MCIT strategies, structure-based methods demonstrate the strongest performance through parameter isolation and expert routing. The replaybased approach ensures memory retention via historical data rehearsal. Furthermore, while promptbased methods minimize trainable parameters, they require significantly more training epochs to converge, resulting in prolonged training time. Beyond these category-specific trends, we observe substantial performance fluctuations across benchmarks. Notably, on a highly challenging benchmark such as TriGap, the amount of parameters allocated per task significantly impacts final accuracy.
python run.py {train|infer} <task_ids> \ --benchmark <benchmark> --method <method> where <benchmark> is one of the supported benchmarks; <method> corresponds to one of the implemented methods; and <task_ids> specifies the sequential task indices to run. Configuration. All experimental settings and parameters are centralized in a modular configuration system. For a detailed breakdown of the configuration files and directory structure (covering methods, benchmarks, backbones, and DeepSpeed settings), please refer to Appendix D.
3
ArxivQA 53.77 83.83 75.57 90.90 79.20 84.47 93.00 91.27 90.46
4
Conclusion
In this paper, we introduce PRISM, a pluginextensible toolbox that lowers the engineering barrier in multimodal continual instruction tuning. By decoupling algorithm development from infrastructure via lightweight registration, PRISM enables researchers to implement and reproduce methods by modifying a minimal amount of code. PRISM establishes a shared infrastructure for reproducible, extensible, and scalable MCIT research. Limitations. PRISM does not currently cover all MCIT methods and MLLM backbones. However, its plugin-centric architecture inherently streamlines the integration of new algorithms. Extending this coverage to a broader range of methods and MLLM families remains future work.
Experiment
We evaluate all methods on UCIT and TriGap using the LLaVA-v1.5-7B backbone, trained on 4 NVIDIA RTX 5090 GPUs. Comprehensive results are summarized in Tab. 2 and 3, with detailed implementation settings provided in Appendix C. Overall, the baselines establish clear performance boundaries: Zero-shot serves as a reference for initial capability, while FT-LoRA and MoE-LoRA illustrate typical catastrophic forgetting patterns. 4
References
Haiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu, DaHan Wang, Jian Xu, Xu-Yao Zhang, and Cheng-Lin Liu. 2025b. Federated continual instruction tuning. In ICCV, pages 1325–1335.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In ECCV, pages 382–398. Springer.
Haiyang Guo, Fei Zhu, Hongbo Zhao, Fanhu Zeng, Wenzhuo Liu, Shijie Ma, Da-Han Wang, and XuYao Zhang. 2025c. Mcitlib: Multimodal continual instruction tuning library and benchmark. arXiv preprint arXiv:2508.07307.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV, pages 2425–2433.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, and 1 others. 2020. Array programming with numpy. nature, 585(7825):357–362.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL, pages 65–72. Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao. 2024. Coin: A benchmark of continual instruction tuning for multimodel large language models. NeurIPS, 37:57817–57840.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, and 1 others. 2021. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340–8349.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174.
Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. 2025. Clmoe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering. In CVPR, pages 19608–19617.
Casper O da Costa-Luis. 2019. tqdm: A fast, extensible progress meter for python and cli. Journal of Open Source Software, 4(37):1277. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248– 255. Ieee.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. 2021. Transvg: Endto-end visual grounding with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1769–1779.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pages 787–798.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.
Georg Krempl, Indre Žliobaite, Dariusz Brzeziński, Eyke Hüllermeier, Mark Last, Vincent Lemaire, Tino Noack, Ammar Shaker, Sonja Sievi, Myra Spiliopoulou, and 1 others. 2014. Open challenges for data stream mining research. ACM SIGKDD explorations newsletter, 16(1):1–10.
Runwei Guan, Rongsheng Hu, Shangshu Chen, Ningyuan Xiao, Xue Xia, Jiayang Liu, Beibei Chen, Ziren Tang, Ningwei Ouyang, Shaofeng Liang, and 1 others. 2026. Roadscenevqa: Benchmarking visual question answering in roadside perception systems for intelligent transportation system. In AAAI, volume 40, pages 4366–4375.
Jusung Lee, Sungguk Cha, Younghyun Lee, and Cheoljong Yang. 2024. Visual question answering instruction: Unlocking multimodal large language model to domain-specific visual multitasks. arXiv preprint arXiv:2402.08360.
Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. 2025a. Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In ACL, pages 13572–13586.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. In ACL, pages 14369–14387.
5
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32.
Adam Dahlgren Lindström and Savitha Sam Abraham. 2022. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer imageto-sentence models. In ICCV, pages 2641–2649.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, KaiWei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214.
Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Roberson Murphy. 2021. Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access, 9:89644–89654.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In CVPR, pages 11–20.
Alex Rogozhnikov. 2022. Einops: Clear and reliable tensor manipulations with einstein-like notation. In ICLR.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263– 2279, Dublin, Ireland. Association for Computational Linguistics.
María Virginia Sabando, Pavol Ulbrich, Matías Selzer, Jan Byška, Jan Mičan, Ignacio Ponzoni, Axel J Soto, María Luján Ganuza, and Barbora Kozlíková. 2020. Chemva: interactive visual analysis of chemical compound similarity in virtual screening. IEEE Transactions on Visualization and Computer Graphics, 27(2):891–901.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In WACV, pages 1697–1706.
Argho Sarkar, Tashnim Chowdhury, Robin Roberson Murphy, Aryya Gangopadhyay, and Maryam Rahnemoonfar. 2023. Sam-vqa: Supervised attentionbased visual question answering model for postdisaster damage assessment on remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 61:1–16.
Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. 2020. Docvqa: A dataset for vqa on document images. corr abs/2007.00398 (2020). arXiv preprint arXiv:2007.00398. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947–952. IEEE.
Shengbang Tong, David Fan, Jiachen Li, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. 2025. Metamorph: Multimodal understanding and generation via instruction tuning. In ICCV, pages 17001– 17012.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318.
6
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR, pages 4566–4575. Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, and 1 others. 2020. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In EMNLP, pages 38– 45. Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, HanJia Ye, De-Chuan Zhan, and Da-Wei Zhou. 2026. Same: Stabilized mixture-of-experts for multimodal continual instruction tuning. arXiv preprint arXiv:2602.01990. Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. 2025. Modalprompt: Towards efficient multimodal continual instruction tuning with dual-modality guided prompt. In EMNLP, pages 12137–12152. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Guoyin Wang, and 1 others. 2023a. Instruction tuning for large language models: A survey. ACM Computing Surveys. Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023b. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415. Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. 2024. Classincremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9851–9873. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
7
A
Brief Introduction of Reproduced Methods
method that performs online SVD of the covariance matrix of LoRA parameters within a sliding window. The principal singular vectors are retained as task-anchoring directions, and a curvature-aware importance score guides parameter consolidation across tasks.
Zero-shot. A baseline that evaluates the frozen pre-trained LLaVA model on all tasks without any fine-tuning, measuring the inherent zero-shot generalization of the multimodal backbone. FT-LoRA. A sequential LoRA fine-tuning baseline that injects trainable low-rank adapters into the LLM backbone. Each task is trained sequentially, with only the LoRA parameters updated while the base model remains frozen. Replay-LoRA. A replay-assisted LoRA method that maintains a task-partitioned memory buffer of training examples from previous tasks. During each training step, stored examples are sampled and replayed alongside the current task data to reinforce prior knowledge. MoE-LoRA (Chen et al., 2024). A mixture-ofexperts LoRA variant that introduces multiple expert LoRA groups per layer with a learned soft router. The router produces a weighted combination of expert outputs, enabling the model to dynamically allocate capacity across tasks. HiDe-LLaVA (Guo et al., 2025a). A HiDe-style mixture-of-experts LoRA approach that maintains per-layer task-specific expert LoRA groups. During training, only the expert corresponding to the current task is activated; during inference, task identity is inferred via CLIP-based image and text anchor matching to route to the appropriate expert. CL-MoE (Huai et al., 2025). A continual learning mixture-of-experts method using input-dependent per-layer per-token routing, eliminating the need for explicit task-ID gating. Combined with memory replay, it provides a strong task-agnostic baseline for continual instruction tuning. DISCO (Guo et al., 2025b). A diagonal mask routing MoE-LoRA approach that learns per-task CLIP-based image and text prototypes. During inference, cosine similarity between the input features and stored prototypes produces diagonal mask weights for expert aggregation, enabling taskidentity-aware routing without explicit task IDs. ModalPrompt (Zeng et al., 2025). A promptbased method that learns per-task soft prompts prepended to the input embedding sequence. At inference, dual-modal guidance is used to select the top-K most relevant prompts, with a tunable balance parameter λ controlling the image-text mixing weight. SAME (Xie et al., 2026). A spectral anchor-based
B
Brief Introduction of Selected Benchmarks
Tables 4 and 5 summarize the dataset compositions of the UCIT (Guo et al., 2025a) and TriGap (Xie et al., 2026) benchmarks, respectively. Both benchmarks strictly enforce an unseen-data protocol: all samples are rigorously filtered to ensure zero overlap with the pre-training or supervised fine-tuning (SFT) corpora of the underlying MLLMs, thereby eliminating potential information leakage and guaranteeing fair evaluation of continual learning capabilities. UCIT comprises six tasks with training sets ranging from 24k to 40k samples and a uniform test split of 3k per task, offering a lightweight and standardized protocol for efficient method validation. In contrast, TriGap expands the scope to ten highly heterogeneous domains, with training sizes varying from 10k to 40k to reflect real-world data availability across specialized fields (e.g., medical imaging, autonomous driving, chemical analysis). By maximizing both the task sequence length and inter-domain distribution shifts, TriGap serves as a comprehensive, high-difficulty benchmark designed for stress-testing long-term knowledge retention. Together, these two benchmarks form a complementary evaluation suite: UCIT provides a controlled baseline, while TriGap offers a rigorous, large-scale setting for assessing model robustness and anti-forgetting capabilities under extreme distribution shifts.
C
Implementation Details
All methods are built upon the LLaVA-1.5 architecture, which consists of a Vicuna-7B LLM backbone and a CLIP-ViT-L/14 vision encoder. Unless otherwise noted, all methods share the following training configuration: AdamW optimizer with learning rate 2 × 10−4 , cosine schedule with 3% warmup, weight decay 0.0, bf16 mixed precision, model max length 2048, gradient checkpointing enabled, and 1 training epoch. All adapter modules are injected exclusively into the LLM backbone, with LoRA target modules and rank configurations for select methods adopted directly from their official imple8
Table 4: Details of datasets used in UCIT benchmark.
Dataset ImageNet-R ArxivQA VizWiz IconQA CLEVR-Math Flickr30k
Train 24000 40000 40000 30000 40000 40000
Test 3000 3000 3000 3000 3000 3000
Domain Description Object recognition with artistic renditions Academic paper figure understanding Visual assistance for visually impaired Icon comprehension Mathematical reasoning on synthetic scenes Image captioning for real-world photos
Table 5: Details of datasets used in TriGap benchmark.
Dataset PMCVQA DocVQA ChartQA IconQA InfographicVQA ArxivQA Roadside ChemVQA FloodNetVQA CLEVR
Train 40000 30000 25000 10000 20000 10000 40000 40000 10000 10000
Test 3000 3000 3000 3000 3000 3000 3000 3000 3000 3000
Domain Description Medical image analysis and diagnosis Document understanding and text extraction Chart and graph reasoning Icon comprehension Infographic information extraction Academic paper figure analysis Autonomous driving scene understanding Molecular structure analysis Disaster scene assessment Mathematical reasoning on synthetic scenes Hyperparameters. We set the LoRA rank and scaling factor as r = 96, α = 192 for UCIT; and r = 80, α = 160 for TriGap. LoRA dropout is fixed at 0.05. Training runs for 1 epoch with a learning rate of 2×10−4 (cosine schedule, warmup ratio 0.03) and a projector learning rate of 2×10−5 . Per-device batch size is 12 for all tasks on CoIN and UCIT.
mentations. C.1
Zero-shot
Zero-shot serves as a parameter-free baseline that bypasses continual instruction tuning entirely. Insertion. No PEFT modules or task-specific adapters are injected. The model operates directly on the frozen pretrained MLLM weights without any parameter updates or checkpoint loading throughout the continual learning sequence. Hyperparameters. As an inference-only baseline, Zero-shot is excluded from the training pipeline. Evaluation adopts the standard decoding configuration (e.g., conversation template and temperature) shared across all methods. C.2
C.3
Replay-LoRA
Replay-LoRA extends FT-LoRA by incorporating a task-partitioned experience replay buffer to mitigate catastrophic forgetting. Insertion. The adapter insertion follows FT-LoRA (LoRA on attention and FFN layers of the LLM trunk). Additionally, a task-partitioned replay buffer stores samples from previous tasks. During training on task t, historical samples are merged into the current dataloader via a replay-sidecar JSON configuration. Hyperparameters. LoRA configurations match FT-LoRA (r = 96, α = 192 for UCIT; r = 80, α = 160 for TriGap; dropout 0.05; 1 epoch; LR 2 × 10−4 ; projector LR 2 × 10−5 ). Replay-specific
FT-LoRA
FT-LoRA is a sequential fine-tuning baseline that applies standard LoRA adapters without continual learning mechanisms. Insertion. Standard LoRA adapters are injected into the attention and FFN linear layers (qproj , kproj , vproj , oproj , gateproj , upproj , downproj ) of the LLM trunk. The vision tower and multimodal projector remain frozen. 9
settings include a total buffer capacity of 180 samples (evenly distributed across the first N − 1 tasks) and a per-example sampling probability of 0.7. Perdevice batch sizes are 12. C.4
modules. The LoRA rank is adjusted to be divisible by the number of tasks, and α is set to 2 × adjusted_r. Hyperparameters. We use r = 96, α = 192 for UCIT; and r = 80, α = 160 for TriGap. LoRA dropout is 0.05. The routing temperature τ is set to 0.05, and the CLIP feature dimension is 768 (matching CLIP-ViT-L/14). Training uses a perdevice batch size of 4 for all benchmarks.
HiDe-LLaVA
HiDe-LLaVA introduces a hierarchical decoupling mechanism with task-specific LoRA experts and dual-modal prototype routing. Insertion. The attention and FFN layers are replaced with HiDeMOELoraLinear modules. Each layer hosts N task-specific LoRA experts (where N is the total number of tasks) alongside a lightweight per-layer router. Adapters are applied exclusively to the LLM, while frozen CLIP-derived image and text anchors are stored per task for inference-time routing. Hyperparameters. LoRA settings are r = 96, α = 192 (UCIT), and r = 80, α = 160 (TriGap), with dropout 0.05. Training uses 1 epoch, LR 2 × 10−4 , and projector LR 2 × 10−5 . The CLIP feature dimension is 768 (CLIP-ViT-L/14). Per-device batch sizes are 12. Routing. During training, the active expert is selected via the current task ID. At inference, dualmodal prototype matching assigns the task: image and text features are compared to per-task anchors using cosine similarity, combined as 0.5 · simimage + 0.5 · simtext , with the argmax index yielding the predicted_task_id. On the final transformer block, only the predicted expert is activated; on preceding blocks, LoRA deltas from all experts are fused via summation. Text-only inputs default to text-anchor matching. C.5
C.7
ModalPrompt does not use LoRA adapters. Instead, it introduces per-task learnable soft prompt tokens and prompt transformation MLPs. Insertion. Soft prompts are prepended to the input sequence at the embedding level. Each task is assigned a learnable prompt of prefix_len = 10 continuous tokens and a dedicated prompt transform MLP that maps the prompt into a feature space aligned with CLIP representations. The transformation MLP is trained via a cosine similarity loss against the corresponding CLIP features. Hyperparameters. The number of top-K prompts selected per inference step is transfer_num = 1. The dual-modal guidance coefficient λ is set to 0.5, balancing image and text prototype similarities as λ · simimage + (1 − λ) · simtext . The prototype momentum for EMA updates is 0.9. ModalPrompt is trained for 4 epochs. Per-device batch sizes are 4 for all benchmarks. C.8
MoE-LoRA
Insertion. MoE-LoRA replaces FFN linear layers (gateproj , upproj , downproj ) with MoELoRALinear modules. The total LoRA rank r must be divisible by the number of experts N , with each expert receiving rank r/N . Hyperparameters. We use rank r = 96, α = 192 for UCIT; and r = 80, α = 160 for TriGap. LoRA dropout is 0.05. Per-device batch sizes are 4 for all benchmarks.
CL-MoE
Insertion. CL-MoE replaces all FFN linear layers (gateproj , upproj , downproj ) with CLMoELinear modules. Each layer contains N independent LoRA expert branches, where N is the total number of tasks. The total LoRA rank is evenly split across experts (per-expert rank = r/N ). Hyperparameters. We use r = 96, α = 192 for UCIT; and r = 80, α = 160 for TriGap. LoRA dropout is set to 0.05. The task embedding dimension is 64. Training uses a per-device batch size of 4 across all benchmarks. C.6
ModalPrompt
C.9
SAME
Insertion. SAME replaces only FFN linear layers (gateproj , upproj , downproj ) with SAMELinear modules. Each layer maintains per-task LoRA expert weights along with task-wise covariance matrices for spectral analysis and parameter sharing. Hyperparameters. We use r = 96, α = 192 for UCIT; and r = 80, α = 160 for TriGap. LoRA dropout is 0.05. SAME-specific hyperparameters include: the curvature/saliency threshold τscore =
DiSCO
Insertion. DiSCO replaces the same set of FFN linear layers as CL-MoE with DiscoMOELoraLinear 10
0.1, curvature EMA momentum µ = 0.9, curvature estimation window size 3, maximum number of principal components 64, and cumulative energy ratio 0.9 for SVD-based truncation. Per-device batch sizes are 4 for all benchmarks.
D
Configuration Details
All experimental settings and parameters in our framework are centralized and can be configured in the following files and directories: • config/run_config.py: Defines global CLI arguments for training/inference, covering benchmark, method, and GPU allocation. • config/methods/: Method-specific hyperparameter configurations. • config/benchmarks/: Benchmark-specific configurations, including task definitions, dataset paths, and evaluation hooks. • config/backbone/: Backbone identifier and default conversation template. • config/paths/: Filesystem paths for model weights, datasets and checkpoints. • config/deepspeed/: DeepSpeed ZeRO configuration files (stage 2, 3, and 3 offload).
11