Hyperbolic Multimodal Continual Learning

Hyperbolic Multimodal Continual Learning

Jiahong Liu 1 Ming Shen 1 Xiaohao Liu 2 Rex Ying 3 Menglin Yang 4 Tat-Seng Chua 2 Irwin King 1 Continual Learning of Multimodal Hyperbolic Representations

arXiv:2608.09572v1 [cs.LG] 10 Aug 2026

Abstract

Task 1

Hyperbolic geometry has recently emerged as a powerful representation space for multimodal learning, as it naturally captures hierarchical semantic structure across modalities. Despite this progress, how such representations behave under continual learning poses fundamentally different challenges that remain underexplored. This work provides a geometric perspective on this problem and establishes a theoretical foundation for representation preservation in hyperbolic space, showing that preventing forgetting requires crossmodal invariance under a shared hyperbolic isometry. We further show that forgetting in hyperbolic continual learning involves both semantic relation drift and hierarchy-related distortion, motivating preservation of both cross-modal relational structure and hierarchical geometry. Guided by these insights, a principled continual learning framework is derived that preserves essential geometric structure while allowing effective adaptation to new tasks. Experiments on continual multimodal benchmarks corroborate the effectiveness of the proposed approach. The code is available in the project repository.

Task 2

Task 3

……

Task N

✗. Structure collapse

of previous tasks

Catastrophic Forgetting!

Ours ↓ 74% Less Forgetting!

Figure 1. Continual learning in hyperbolic representation space. Multimodal representations are embedded in hyperbolic space to capture semantic hierarchy across tasks. Under naive continual updates, distortions in the hyperbolic geometry lead to catastrophic forgetting and collapse of previously learned structures, whereas structure-aware hyperbolic updates significantly reduce forgetting.

modal data and contrastive objectives, such models exhibit robust zero-shot transfer capabilities within a fixed representation space (Radford et al., 2021; Han et al., 2023; Chen et al., 2023; Li et al., 2025). Most existing approaches embed multimodal representations in high-dimensional Euclidean spaces, implicitly assuming a flat geometric structure for semantic concepts (Radford et al., 2021; Han et al., 2023; Chen et al., 2023).

1. Introduction Multimodal Contrastive Learning (MCL) (Radford et al., 2021; Dufumier et al., 2024; Liu et al., 2025c;b) has become a dominant paradigm for learning joint representations across modalities, enabling effective alignment between heterogeneous data sources. By leveraging large-scale multi-

However, multimodal semantic concepts often exhibit inherent hierarchical structure (Desai et al., 2023; Pal et al., 2025; Kim et al., 2024). Across modalities, abstract and high-level concepts typically subsume a large number of more specific instances, inducing asymmetric and partially ordered semantic structures rather than flat metric clusters. Such differences in semantic specificity and abstraction are difficult to faithfully capture in Euclidean spaces, where distances and neighborhoods are uniformly structured (Krioukov et al., 2010; Nickel & Kiela, 2017; 2018; Ganea et al., 2018a).

1 Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China 2 School of Computing, National University of Singapore, Singapore 3 Department of Computer Science, Yale University, New Haven, CT, United States 4 AI Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China. Correspondence to: Jiahong Liu <jiahong.liu21@gmail.com>, Irwin King <king@cse.cuhk.edu.hk>.

Hyperbolic Multimodal Learning. Hyperbolic geometry provides a principled representation space for hierarchical semantics (Krioukov et al., 2010; Nickel & Kiela, 2017; 2018; Ganea et al., 2018b;a). Recent advances in hyperbolic

rd

Proceedings of the 43 International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).

1

Hyperbolic Multimodal Continual Learning

2. Preliminary

multimodal representation learning, such as MERU and related hyperbolic vision–language models (Desai et al., 2023; Mandica et al., 2024; Kim et al., 2024; Pal et al., 2025; Poppi et al., 2025), show that leveraging hyperbolic geometry leads to higher-quality multimodal representations. Specifically, hyperbolic space encodes semantic information along two complementary geometric dimensions: (i) geodesic distances capture semantic similarity, ensuring that concept proximity reflects relational closeness; (ii) entailment cones encode semantic hierarchy, organizing concepts according to their level of abstraction, with generic concepts typically located more centrally and more specific concepts extending outward along entailment cones (Ganea et al., 2018a; Desai et al., 2023; Pal et al., 2025).

2.1. Hyperbolic Geometry Hyperbolic space is a non-Euclidean geometry with constant negative curvature. The Lorentz model of hyperbolic geometry (hyperboloid model) is widely used due to its numerical stability for optimization (Nickel & Kiela, 2018; Mishne et al., 2023). We provide a detailed introduction in Appendix B.2. Lorentz Model of Hyperbolic Space. The d-dimensional hyperbolic space with sectional curvature −K (K > 0) can be realized as the upper sheet of a hyperboloid embedded in (d + 1)-dimensional Minkowski space. Let G = diag(−1, 1, . . . , 1) ∈ R(d+1)×(d+1) denote the Minkowski metric tensor. The hyperbolic manifold is defined as:

Despite these advantages, existing hyperbolic multimodal models are almost exclusively developed under static learning assumptions, where the full data distribution is available upfront (Desai et al., 2023; Mandica et al., 2024; Kim et al., 2024; Pal et al., 2025; Poppi et al., 2025). In realistic multimodal systems, however, data distributions evolve over time as new concepts, domains, and tasks emerge (Kirkpatrick et al., 2017; Yu et al., 2024a; Ni et al., 2023; Liu et al., 2025a), leaving continual learning in hyperbolic representation spaces largely unexplored.

HdK = {x ∈ Rd+1 | x⊤ Gx = −1/K, x0 > 0}.

(1)

Each point x = [x0 ; x[1:d] ]⊤ ∈ HdK consists of a time-like coordinate x0 and d space-like coordinates x[1:d] , reflecting the Lorentzian signature of the metric. Lorentzian Inner Product and Distance. The Lorentzian inner product is defined as:

Continual Learning as a Geometric Problem. Because hyperbolic semantics are encoded by geometric quantities, continual parameter updates that violate their invariants distort semantic similarity and hierarchy, leading to catastrophic forgetting (Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017; Ni et al., 2023). This perspective highlights the fundamental challenge that need to be addressed: what geometric conditions must be preserved to prevent forgetting in hyperbolic multimodal representations.

⟨x, y⟩L = x⊤ Gy = −x0 y0 +

d X

xi yi ,

(2)

−1/2 dK arcosh (−K⟨x, y⟩L ) L (x, y) = K
−1/2 =K arcosh −Kx⊤ Gy .

(3)

i=1

which induces the geodesic distance:

Lorentz Transformation Layer. Let z ∈ Rd+1 be a Lorentz embedding, where z0 > 0 denotes the time-like coordinate and zs ∈ Rd denotes the space-like coordinates. A Lorentz transformation layer (Chen et al., 2022) is defined as follows p  K −1 + ∥Wz∥22 , (4) f (W; z) = Wz

Contributions. This work explores continual learning for hyperbolic multimodal representations and addresses the above fundamental questions. To the best of our knowledge, it provides the first geometric study of this setting. In particular, we make the following contributions: • First, we establish a theoretical foundation for representation preservation in hyperbolic space, showing that it requires cross-modal invariance under a shared hyperbolic isometry (e.g., the same rotation), yielding necessary and sufficient geometric conditions.

where W ∈ Rd×(d+1) , K > 0 is the magnitude of the negative curvature, so the manifold has sectional curvature matrix is decomposed as W = −K. The transformationd×1 w0 Ws , w0 ∈ R , Ws ∈ Rd×d , where w0 and Ws act on the time-like and space-like components of the input embedding, respectively.

• Second, guided by the above theoretical conditions and analysis, we derive a principled continual learning framework for hyperbolic multimodal representations, which preserves essential geometric structure while allowing effective adaptation to new tasks.

2.2. Hyperbolic Multimodal Learning ′

′

m N Given a paired dataset X m,m = {(xm i , xi )i=1 } ⊂ m m′ X × X , where N is the number of paired samples, ′ X m and X m represent the data spaces of modality m

Across diverse continual multimodal benchmarks, HMCL consistently reduces catastrophic forgetting while adapting to new tasks. 2

Hyperbolic Multimodal Continual Learning ′

m and modality m′ , respectively. Each pair (xm i , xi ) consists of semantically corresponding samples from different modalities. The model produces embedding matrices ′ m′ d Zm , Zm ∈ RN ×(d+1) , where each row zm i , zi ∈ H represents the hyperbolic embedding for the i-th sample pair.

3. Problem Statement 3.1. Hyperbolic Multimodal Continual Framework We consider multimodal continual learning under hyperbolic representations, in which multiple modalities are aligned into a common hyperbolic space and learned sequentially over time. The goal is to ensure that the learned embeddings preserve critical geometric relationships, both within and across modalities, while adapting to new tasks. We adopt the established continual learning framework for multi-modal contrastive learning (Liu et al., 2025b), where M modalities are processed through pre-trained encoders Em : Xm → Rdm followed by learnable trans(t) formation layers Tm : Rdm → Rd for each modality m ∈ {1, 2, . . . , M } at task t. The complete forward pass consists of:

Contrastive Similarity. The hyperbolic contrastive similarity matrix is computed as: ′

′

m m Sm→m = −dK L (Z , Z )

  ′ = −K −1/2 arcosh −KZm G(Zm )⊤ ,

(5)

′

where Sm→m ∈ RN ×N denotes the similarity matrix, G = diag(−1, 1, . . . , 1) ∈ R(d+1)×(d+1) is the Minkowski metric tensor, and arcosh is applied element-wise. Each ele′ m m′ ment Sm→m = −dK ij L (zi , zj ) represents the negative hyperbolic distance between the i-th embedding from modality m and the j-th embedding from modality m′ .

Features before continual learning: The frozen encoder extracts features from task t − 1 data:

The unidirectional contrastive loss is formulated as: N 1 X

′ exp(Sm→m /τ ) ii log PN Lm→m′ = − ′ m→m N i=1 /τ ) j=1 exp(Sij

m N ×dm Zm,∗ t−1 = Em (Xt−1 ) ∈ R

where Em denotes the pre-trained encoder for modality m.

(6)

Transformation layer: To effectively accumulate knowledge through incremental updates, a learnable transformation layer adapts the features:
m,∗ (t) m,t N ×(d+1) Zm,t , Zm,∗ t−1 = Tm (Zt−1 ) = f W t−1 ∈ R (12) (t) where Tm denotes the transformation layer for modality m at task t, Wm,t is a learnable weight matrix of the linear transformation stage t, and f (·, ·) is a function ensuring that the outputs still lie on the Lorentz manifold Hd .

where τ is the temperature parameter. The symmetric contrastive loss is defined as:
1 Lcontrast = Lm→m′ + Lm′ →m (7) 2 Entailment cone. For hierarchical relationships, each emd bedding zm i ∈ H defines an entailment cone with aperture: ! 2κ −1 m √ aper(zi ) = sin , (8) K ∥(zm i )[1:d] ∥2

Similarly, for the current task t data: (t) Zm,t = Tm (Em (Xtm )) ∈ RN ×(d+1) t

where a constant κ = 0.1 is used for setting boundary conditions near the origin. The aperture naturally encodes semantic hierarchy: embeddings closer to the origin (i.e., smaller ∥(zm i )[1:d] ∥2 ) correspond to higher-level, more general concepts and have larger apertures aper(zm i ), while embeddings farther from the origin represent more specific concepts with ′ m′ smaller apertures. Writing ℓm,m = ⟨zm i , zi ⟩L , the extei rior angle is:   ′ m,m′ m m (zi )0 + K(zi )0 ℓi m′ −1  . q ext(zm i , zi ) = cos m,m′ 2 ∥(zm ) ∥ (Kℓ ) − 1 i [1:d] 2 i (9)

(13)

Notation Justification. In our notation system, Zm,s repret sents the embedding of task t data for modality m processed at stage s. For example, Zm,∗ t−1 – task t − 1 data processed by frozen pre-trained encoder (∗ indicates pre-trained weights); Zm,t t−1 – task t−1 data processed by learnable transformation at stage t. We list the notation table in Appendix A. 3.2. Key Challenges in Hyperbolic Geometry Stability. In hyperbolic representation learning, the preservation of historical knowledge fundamentally depends on maintaining the geometric structure of the embedding space. This requires maintaining three geometric properties of the learned representations1 . ∀m, m′ ∈ {1, 2, . . . , M }, m ̸= m′ :

The entailment loss (Ganea et al., 2018b) enforces partial order constraints to ensure proper hierarchical relationships between cross-modal concepts (e.g., textual representations at higher semantic levels than their visual counterparts): ′

(11)

1 Given that arccosh(·) is a strictly monotonic function in Equation (19), equality of similarity values S directly corresponds to equality of the underlying Lorentzian inner products in Definition 2. We thus simplify our notation accordingly.

′

m m m m Lentail (zm i , zi ) = max(0, ext(zi , zi ) − aper(zi )). (10)

3

Hyperbolic Multimodal Continual Learning

transformation of the form   1 0⊤ R= e , 0 R

(P1) Intra-modal preservation: m,t−1 ⊤ m,t−1 m,t ⊤ Zm,t t−1 G(Zt−1 ) = Zt−1 G(Zt−1 ) ;

(P2) Inter-modal preservation: ′

e ∈ SO(d), R

such that, for all modalities m ∈ {1, . . . , M },

′

m ,t ⊤ m,t−1 m ,t−1 ⊤ Zm,t ) ; t−1 G(Zt−1 ) = Zt−1 G(Zt−1

m,t−1 ⊤ Zm,t t−1 = Zt−1 R .

(P3) Hierarchical preservation: m,t−1 ∥(zm,t t−1 )[1:d] ∥ = ∥(zt−1 )[1:d] ∥.

Proof sketch. Conditions (P1) and (P2) preserve all required Lorentzian inner products for task t−1 between stages t−1 and t. By Theorem 2 (and Corollary 3) in Appendix E.1, this yields a unique shared Lorentz transformation L ∈ SO+ (1, d) acting identically across modalities. Condition (P3) enforces spatial-norm invariance; Together, these give Theorem 1; the detailed proof is shown in Appendix E.1.

Intra-modal preservation ensures that domain-specific knowledge within each modality remains intact, while intermodal preservation maintains the learned cross-modal associations that enable effective multimodal understanding. Hierarchical preservation protects the conceptual hierarchies captured by the hyperbolic cone structure, which are crucial for representing complex semantic relationships between concepts (Detailed explanations are shown in Appendix D).

Role of non-degeneracy. The non-degeneracy condition is used to obtain the sharpest identifiable characterization of the preservation map. When old-task embeddings span the relevant Lorentzian subspace, the preserved Gram structure determines a shared isometry uniquely, making the preservation requirement maximally constrained. If some representation directions collapse or become redundant, the same preservation conditions still apply, but the compatible isometry may no longer be unique because the old-task geometry is less informative along those collapsed directions. Thus, non-degeneracy should be interpreted as an identifiability condition for the strongest theoretical statement, rather than as a restriction required by the HMCL algorithm.

Plasticity. The model maintains the capability to learn from new datasets and adapt to evolving multimodal distributions. For the current task t, the model effectively acquires new knowledge by learning appropriate represen(1,2,...,M ) (1,2,...,M ) M tations: {Zm,t , where pt rept }m=1 ∼ pt resents the joint distribution of the t-th task dataset across all M modalities. This ensures the model can effectively capture task-specific patterns and cross-modal relationships while maintaining its representational capacity. Challenges. The above formulations clarify what stability and plasticity demand in hyperbolic multimodal continual learning. The challenge that must be addressed to make these objectives achievable in practice is: the stability conditions (P1)–(P3) are principled but cumbersome to enforce or analyze directly, motivating a simpler equivalent characterization (Theorem 1) .

4.2. Hierarchy Preservation at First Order In hyperbolic space, the time-like coordinate is functionally constrained by the spatial components through the geometry of the Lorentz manifold. Leveraging this structural dependency, this section characterizes the first-order conditions under which hierarchical structure is preserved, forming the theoretical basis for the update rules derivation. Corollary 1 (Updates under hierarchical preservation). Conm,t−1 sider an update ∆z = zm,t applied to the old-task t−1 −zt−1 m,t−1 embedding zt−1 . If the time-like coordinate is required to be first-order invariant, i.e. (P3), then the induced spatial update must satisfy

4. Principles of Geometric Updates In this section, we derive geometric principles that determine the legality of parameter updates through preservation of geometric invariants in hyperbolic representations.

z⊤ [1:d] ∆z[1:d] = 0.

4.1. Geometric Characterization This section provides a characterization of the conditions under which old-task representations are preserved across continual learning stages.

(14)

Key takeaway. Corollary 1 indicates that preserving the hierarchy of old-task embeddings enforces that the spatial update is orthogonal to the current embedding’s spatial direction. By Equation (30), admissible updates are restricted within the tangent space so as to suppress first-order drift along the time-like dimension.

Theorem 1 (Geometric Characterization of Preservation). Assume a non-degeneracy condition for task t − 1 representations at stage t−1. Then the preservation conditions (P1), (P2), and (P3) hold for task t−1 representations between stages t−1 and t if and only if there exists a shared 4

Hyperbolic Multimodal Continual Learning Semantic Hierarchy Partial order be maintained

Semantic Similarity

Theorem 1 & Corollary 1

Still be farther

𝑅 Image 𝑅

∗ 𝑅

Exhausted doggo

Still be closer

Top-Down View

Exhausted doggo

∆𝐳

Text ∗ 𝑅

∆𝐳 [ROOT]

So cute

So cute

[ROOT]

Task 𝑡 − 1, Stage 𝑡 − 1

Task 𝑡 − 1, Stage 𝑡

Figure 2. Geometric interpretation of hyperbolic multimodal continual learning. Multimodal representations are embedded in hyperbolic space, where distance encodes semantic similarity and distance to the origin encodes semantic hierarchy. Preserving old-task knowledge across tasks requires maintaining these geometric relations under a shared hyperbolic isometry. The top-down view illustrates that first-order hierarchy preservation restricts updates to lie within the tangent space.

Moreover, HMCL uses the block-wise constraint ∆w0m = 0. The time-like output is reconstructed from the spatial output and is therefore preserved by the first-order spatial rotation.

Geometric interpretation. Figure 2 illustrates hyperbolic multimodal continual learning from a geometric perspective. In hyperbolic space, distances encode semantic similarity, while distance to the origin captures semantic hierarchy. As established in Theorem 1, preserving old-task knowledge across tasks requires updates to act as a shared hyperbolic isometry, maintaining both properties.

Corollary 2 (Block-wise Canonical Update). Under the admissibility condition of Proposition 1, the canonical admissible update is characterized by 2 m Zm,∗ t−1,s ∆Ws

The top-down view highlights the consequence of Corollary 1: at first order, hierarchy preservation restricts updates to changes that keep representations’ distance to the origin unchanged. Conventional continual learning methods lack such constraints, leading to violations of the stability conditions and degradation on previously learned tasks.

⊤

= 0,

∆w0m = 0,

(16)

and is given in closed form by
∆Wsm = δsm I − Pt−1 ,

(17)

 † m,∗ m,∗ ⊤ ⊤ where Pt−1 = (Zm,∗ ) Z (Z ) Zm,∗ t−1,s t−1,s t−1,s t−1,s .

5. Hyperbolic Multimodal Continual Learning

The detailed proofs of Proposition 1 and Corollary 2 are provided in Appendix F.1 and Appendix F.2, respectively.

Based on Theorem 1 and Corollary 1, we further derive a principled methodology for identifying admissible parameter updates that satisfy the geometric characterization established in this section.

This section establishes a principled link between geometric preservation and parameter updates in hyperbolic multimodal continual learning.

5.1. Admissible Parameter Updates Key takeaway. We show that preserving intramodal, inter-modal, and hierarchical structure in turn induces a strict first-order constraint on admissible parameter updates (Proposition 1). Under this constraint, the optimal update is given by an orthogonal projection, providing a closed-form and geometry-consistent parameter update rule (Corollary 2).

Proposition 1 (Admissible Parameter Updates). Consider the Lorentz transformation layer in Section 2.1, with parameters Wm,t = [ w0m,t Wsm,t ]. Assume Theorem 1 holds at stage t − 1. For an infinitesimal update Wm,t = Wm,t−1 − η ∆Wm with η → 0+ , ∆Wsm is admissible if and only if there exists Ω⊤ = −Ω such that

m Zm,∗ t−1 [1 : d] ∆Ws

⊤

⊤ = Zm,t−1 t−1 [1 : d]Ω .

2

(15) 5

m,∗ To simplify notation, Zm,∗ t−1,s represents Zt−1 [1 : d].

Hyperbolic Multimodal Continual Learning

• RQ2: Does HMCL reduce radial, angular, and crossmodal old-task drift? (Section 6.3)

Algorithm 1: HMCL with Canonical Updates Input: Old-task spatial representations N ×d Zm,∗ ; t−1,s ∈ R Current parameters Wm,t−1 = [w0m,t−1 , Wsm,t−1 ]; Unconstrained update direction δsm ; Step size η. Output: Updated parameters Wm,t . Compute a low-rank orthonormal basis V of Zm,∗ t−1,s via SVD (or PCA) (Lopez-Paz & Ranzato, 2017); Project the unconstrained update:
∆Wsm = δsm I − VV⊤ ;

• RQ3: Does HMCL preserve semantic hierarchies after continual training? (Section 6.4) Supplementary ablations, robustness checks, and additional traversals are referenced below and reported in Appendix G.2, Appendix G.3, and Appendix G.4, respectively. 6.1. Implementation Details Tasks & datasets. We evaluate under a continual multimodal learning setting with both classification and retrieval tasks. The task stream consists of up to 15 datasets, including image–text classification benchmarks (e.g., CIFAR-10/100 (Krizhevsky, 2009), Caltech-101 (FeiFei et al., 2004), Food-101 (Bossard et al., 2014)) and crossmodal retrieval benchmarks (COCO (Lin et al., 2014) and Flickr30k (Young et al., 2014)). Tasks are presented sequentially without revisiting previous data, using a fixed task order across all runs. Shuffled-stream results in Table 6 show that HMCL retains consistent gains under alternative task orders. Dataset statistics are provided in Appendix G.1.

Set the time-like update to zero ∆w0m = 0; Update parameters: Wsm,t = Wsm,t−1 −η ∆Wsm ,

w0m,t = w0m,t−1 .

return Wm,t .

5.2. Training with Canonical Updates (HMCL) We now describe how the canonical update in Corollary 2 is incorporated into training. At each stage, the update restricts parameter changes to directions orthogonal to the subspace induced by old-task representations, while remaining compatible with standard gradient-based optimization. The procedure is summarized in Algorithm 1, which applies the canonical projection to the unconstrained update and enforces stability of hyperbolic old-task representations.

Implementation details. We adopt pretrained multimodal encoders as frozen feature extractors and optimize lightweight transformation layers throughout continual training. All representations are modeled in the Lorentz (hyperboloid) space with fixed curvature K = 0.1. The transformation module consists of a single linear layer without activation or dropout.

Remark 1. The canonical update introduces no additional learnable parameters. The projection basis V depends only on old-task representations and can be computed once per stage. In implementation, we maintain a fixed-size d × d covariance summary of the retained old-task spatial subspace, rather than storing separate bases or historical representations for every previous task; hence the memory overhead is constant with respect to the number of tasks. As a result, the update incurs negligible overhead and can be seamlessly integrated into existing training pipelines.

Models are optimized using AdamW optimization (Loshchilov & Hutter, 2017). For classification tasks, we train for 10 epochs with a learning rate 5 × 10−4 and batch size 1024. For retrieval tasks, we train for 15 epochs with a learning rate of 5 × 10−5 and a batch size of 1024. The entailment loss weight is set to 0.2 for all tasks, and weight decay is set to 0. All experiments are implemented in PyTorch (Paszke et al., 2019) and conducted on a single NVIDIA RTX A6000 GPU. The retained-variance ratios of the memory projector are 0.9, 0.8, and 0.9 for MERU-L, MERU-B, and HyCoCLIP-B, respectively. To stabilize finite-step optimization, at the start of task t we snapshot the shared head matrix Wt,0 and, after every projected AdamW optimizer step (one step per epoch because gradients are accumulated over the epoch), apply the step pullback W ← (1 − β)W + βWt,0 . We use β = 0.10 for MERU-L/MERU-B and β = 0.20 for HyCoCLIP-B. Pullback is applied only to the shared bias-free head matrix; the temperature and all frozen backbone parameters are unchanged. Table 5 further tests AdamW with nonzero weight decay and confirms that HMCL remains effective under this optimizer mismatch.

We discuss finite-step AdamW optimization and its residual approximation error in Appendix F.3.

6. Experiments The experiments are designed to systematically evaluate continual learning for hyperbolic multimodal representations. We organize the evaluation around the following research questions. • RQ1: Does HMCL improve performance and reduce forgetting across backbones? (Section 6.2) 6

Hyperbolic Multimodal Continual Learning Table 1. Average performance (mean ± std.) on classification and retrieval tasks. Bold indicates statistical significance (p < 0.05). The dedicated retrieval columns use image-to-text R@K, with K = 5 for MERU and K = 1 for HyCoCLIP; the Overall columns retain text-to-image R@5 for retrieval tasks. HMCL uses the reported step-pullback configuration. Classification Backbone

MERU-L

MERU-B

HyCoCLIP-B

Retrieval

Overall

Method

Acc ↑

BWTA ↑

R@K ↑

BWTRK ↑

Performance ↑

BWT ↑

Vanilla

40.59 ± 0.21

−6.46 ± 0.23

31.97 ± 0.31

−6.46 ± 0.23

38.22 ± 0.18

−3.49 ± 0.24

EWC GEM C-FLAT HMCL (ours) ∆ vs. Vanilla

40.52 ± 0.15 40.55 ± 0.16 39.28 ± 0.30 43.34 ± 0.12 +6.8%

−6.43 ± 0.15 −6.40 ± 0.16 −6.23 ± 0.27 −0.74 ± 0.08 +88.5%

32.96 ± 0.08 33.00 ± 0.07 31.42 ± 0.17 40.26 ± 0.27 +25.9%

−3.53 ± 0.25 −3.50 ± 0.24 −2.56 ± 0.30 −1.28 ± 0.21 +80.2%

38.22 ± 0.13 38.25 ± 0.14 37.16 ± 0.27 41.36 ± 0.10 +8.2%

−6.04 ± 0.13 −6.02 ± 0.13 −5.75 ± 0.23 −1.07 ± 0.09 +69.3%

Vanilla

44.34 ± 0.07

−7.26 ± 0.14

33.66 ± 0.19

−5.43 ± 0.19

41.82 ± 0.07

−5.43 ± 0.19

EWC GEM C-FLAT HMCL (ours) ∆ vs. Vanilla

44.48 ± 0.28 44.91 ± 0.76 44.88 ± 0.54 46.15 ± 0.03 +4.1%

−6.63 ± 0.28 −7.00 ± 0.72 −7.58 ± 0.61 −2.46 ± 0.05 +66.1%

31.87 ± 0.45 29.06 ± 3.00 28.20 ± 3.16 39.57 ± 0.31 +17.6%

−4.36 ± 0.39 −7.15 ± 3.11 −7.91 ± 3.55 −2.20 ± 0.28 +59.5%

41.94 ± 0.22 42.09 ± 0.55 42.04 ± 0.23 44.21 ± 0.03 +5.7%

−6.34 ± 0.25 −6.81 ± 0.73 −7.34 ± 0.70 −2.36 ± 0.03 +56.5%

Vanilla

40.70 ± 0.20

−5.46 ± 0.14

45.34 ± 0.35

−5.46 ± 0.14

42.33 ± 0.20

−5.42 ± 0.63

EWC GEM C-FLAT HMCL (ours) ∆ vs. Vanilla

40.54 ± 0.24 40.58 ± 0.27 40.67 ± 0.18 43.18 ± 0.07 +6.1%

−5.51 ± 0.14 −5.48 ± 0.16 −5.49 ± 0.13 −0.28 ± 0.10 +94.9%

45.62 ± 0.35 45.67 ± 0.41 45.31 ± 0.28 57.55 ± 0.31 +26.9%

−4.52 ± 0.31 −4.46 ± 0.36 −4.52 ± 0.24 −0.70 ± 0.32 +87.2%

42.25 ± 0.22 42.29 ± 0.24 42.32 ± 0.17 46.50 ± 0.07 +9.9%

−5.53 ± 0.12 −5.51 ± 0.14 −5.50 ± 0.12 −0.43 ± 0.09 +92.1%

Evaluation metrics. For retrieval tasks, the released implementation stores image-to-text and text-to-image Recall@k for k ∈ {1, 5, 10}. The dedicated retrieval column reports image-to-text R@K, using K = 5 for the two MERU backbones and K = 1 for HyCoCLIP-B to follow the operating point used by each backbone’s evaluation protocol. The Overall Performance/BWT aggregation remains fixed to text-to-image R@5 for every backbone. For classification tasks, we report average accuracy. To measure stability, we use signed backward transfer, BWT = Performancefinal − Performanceright after task , so a negative value indicates forgetting. All results are obtained by averaging over five independent runs with different random seeds.

include representative regularization- and constraint-based methods, namely Elastic Weight Consolidation (EWC) and Gradient Episodic Memory (GEM), which are commonly adopted baselines in multimodal continual learning (Ni et al., 2023). Both methods are applied directly to the hyperbolic multimodal training objective. (3) C-FLAT. C-FLAT introduces contrastive regularization to encourage consistency between current and historical representations across tasks in continual vision–language learning (Bian et al., 2024). We adopt the original formulation and apply it directly in the hyperbolic setting. For all baselines, we replace the original Euclidean training objective with the hyperbolic contrastive loss to better match the underlying geometry. All other continual learning strategies remain unchanged, and we adopt their original formulations and apply them directly in the hyperbolic setting. This protocol compares the continual-learning mechanisms under the same hyperbolic objective and backbone. However, these baselines remain geometry-agnostic in their update rules: they do not explicitly preserve shared cross-modal isometries or hierarchy-related constraints, which is precisely the gap targeted by HMCL. All baselines share the same backbone, task stream, optimizer, and learning-rate schedule as HMCL. This controlled setup isolates the update mechanism, directly testing whether preserving Lorentz-space invariants improves old-task stability.

Baselines. To the best of our knowledge, there is currently no prior work that specifically studies continual learning for hyperbolic multimodal representations. We therefore adapt representative continual learning strategies originally developed in Euclidean space for comparison. (1) Vanilla. The Vanilla baseline sequentially fine-tunes the hyperbolic multimodal model on each task using the standard contrastive objective, without any explicit mechanism to mitigate forgetting. Naive continual training in vision–language models is known to suffer from severe cross-modal misalignment and spatial disorder (Ni et al., 2023). (2) EWC and GEM. We 7

Hyperbolic Multimodal Continual Learning

Figure 3. Dataset-wise old-task representation drift from the task-end checkpoint to the final checkpoint. Gray circles denote Vanilla and blue diamonds denote HMCL; shaded rows indicate retrieval tasks. We measure four complementary changes: radial drift for hierarchy-related radius changes, angular drift for within-modality directional distortion, cross-modal drift for image–text relational structure, and paired-distance drift for matched image–text geodesic distances. Lower values indicate better preservation.

6.2. Main Results (RQ1)

but fails to consistently preserve classification and overall performance. These results suggest that methods developed under Euclidean assumptions are insufficient when continual updates must preserve hyperbolic invariants governing both cross-modal relations and hierarchy.

Experimental Results. HMCL consistently improves overall performance across backbones. Table 1 reports average performance on continual multimodal classification and retrieval across three hyperbolic backbones (MERU-L, MERU-B, and HyCoCLIP-B). Across all settings, HMCL achieves the highest classification accuracy, retrieval performance (R@K), and overall score. For example, on MERUL, HMCL improves the Overall metric from 38.22 to 41.36, corresponding to a relative gain of +8.2% over the Vanilla baseline. On HyCoCLIP-B, HMCL increases image-to-text R@1 by +26.9%. These consistent gains indicate that the effectiveness of HMCL generalizes across model architectures and capacities.

Geometry-aware constraints are key to stable multimodal continual learning. The consistent advantages of HMCL align with the theoretical analysis: by restricting continual updates to shared hyperbolic isometries, HMCL preserves both cross-modal similarity and hierarchical structure. This geometric consistency enables simultaneous improvements in classification and retrieval, rather than trading off performance across tasks. Figure 5 complements this result by showing that removing the block-wise constraint weakens stability across backbones.

HMCL substantially mitigates catastrophic forgetting. In terms of backward transfer (BWT), HMCL consistently yields significantly less negative values than competing approaches. On MERU-L, classification BWT improves from −6.46 to −0.74, corresponding to an 88.5% relative reduction in forgetting. Similar improvements are observed for retrieval BWT and across all three backbones. In contrast, EWC and GEM provide only marginal relief, while C-FLAT alleviates forgetting in some cases but often at the expense of overall performance.

6.3. Quantitative Drift Analysis (RQ2) Protocol. To directly connect forgetting with geometric representation change, we measure old-task drift after subsequent continual updates. The four metrics below are organized by the preservation conditions in Section 3. • Radial drift measures changes in hyperbolic radius, i.e., the spatial-norm/time-like signal tied to semantic hierarchy. It diagnoses violations of hierarchical preservation in (P3). • Angular drift removes magnitude effects and measures within-modality directional distortion. It reflects whether intra-modal neighborhoods remain stable as required

Conventional continual learning methods struggle in hyperbolic multimodal settings. Regularization-based methods such as EWC and constraint-based approaches like GEM show limited effectiveness and, in some cases, high variance. C-FLAT partially improves retrieval stability 8

Hyperbolic Multimodal Continual Learning

HMCL(ours)

Vanilla

HMCL(ours)

Vanilla

HMCL(ours)

Vanilla

hat

volleyball

relaxation

relaxation

defeat

workout

fashion

interest

vacation

aloof

workout

pilates

style

fashion

peace

sunken

peace

race

[ROOT]

[ROOT]

[ROOT]

[ROOT]

[ROOT]

[ROOT]

Figure 4. Case study: image traversals and hierarchical concept retrieval. For three Flickr30K images, we traverse each embedding toward [ROOT] and retrieve concepts from specific to generic. HMCL yields coherent hierarchies (e.g., hat → fashion → style), whereas Vanilla drifts toward irrelevant or inconsistent concepts after continual training.

Lorentz hyperboloid using the exponential map. At each discrete step, we perform nearest-neighbor retrieval from the candidate set of text embeddings.

by (P1). • Cross-modal drift measures changes in the image– text relational structure. It corresponds to inter-modal preservation in (P2). • Paired-distance drift tracks geodesic-distance changes of matched image–text pairs. It directly tests crossmodal semantic alignment under (P2), while remaining sensitive to hierarchy-related radial changes under (P3).

Results and Analysis. In experiments focusing solely on nouns and adjectives, Figure 4 illustrates the comparison between HMCL and the Vanilla baseline model after continuous learning. For example, in the left image, the most salient concrete concept is a hat: HMCL follows a coherent path from “hat” to broader categories such as “fashion” and “style” as it approaches [ROOT]. Vanilla instead jumps to the unrelated concept “volleyball” before drifting through “interest” and “fashion”, indicating a disrupted local-to-generic hierarchy. The traversal paths thus provide a qualitative counterpart to the drift analysis: HMCL keeps image-specific concepts connected to stable generic anchors as embeddings move toward [ROOT]. This suggests that HMCL preserves not only nearest-neighbor retrieval quality, but also the abstraction ordering encoded by the hyperbolic radial geometry, consistent with Appendix G.4.

Results. As shown in Figure 3, HMCL reduces all four drift types for most datasets, including both classification and retrieval tasks. Averaged over old tasks, the reductions are 61.0%, 57.8%, 73.8%, and 59.5%, respectively, with aggregate statistics and significance in Table 7. Takeaway: forgetting involves both hierarchy-related distortion and relation-level drift. 6.4. Case Study (RQ3) Data Preparation. To explore the efficacy of our method in preserving hierarchical relationships within hyperbolic space during continual learning, we conduct a detailed case study using image traversals on the Flickr30K dataset. We construct a hierarchical text set by extracting nouns and adjectives from the original captions to represent varying levels of semantic abstraction, following the dataset details in Appendix G.1.

7. Conclusion This work presents the first systematic study of continual learning for hyperbolic multimodal representations. We establish a theoretical foundation showing that preserving previously learned knowledge requires cross-modal invariance under a shared hyperbolic isometry, and that forgetting is primarily driven by distortions along the radial dimension that encodes semantic hierarchy. We derive HMCL, a principled framework that restricts parameter updates to geometry-preserving directions. Experiments across multiple hyperbolic backbones demonstrate that HMCL consistently improves both performance and stability, substantially reducing catastrophic forgetting compared to existing continual learning methods.

Image Traversals. We utilize the image traversal technique to qualitatively evaluate the structural integrity of the learned representation space. This method involves performing a shortest-path traversal along the geodesic connecting a specific image embedding to the manifold’s origin, denoted as [ROOT], which represents the most generic semantic concept. The traversal is executed by interpolating 20 equally spaced steps in the tangent space at the origin. These interpolated vectors are subsequently projected back onto the 9

Hyperbolic Multimodal Continual Learning

Acknowledgments. Jiahong Liu and Irwin King acknowledge Hong Kong RGC support (CUHK 2300246, C1043-24G, 14203425, 2151317); Menglin Yang acknowledges Guangdong Natural Science Foundation support (2026A1515012118).

Vedaldi, A. Describing textures in the wild, 2013. URL https://arxiv.org/abs/1311.3618. Desai, K., Nickel, M., Rajpurohit, T., Johnson, J., and Vedantam, S. R. Hyperbolic image-text representations. In International Conference on Machine Learning, pp. 7694–7731. PMLR, 2023. Doan, T., Behpour, S., Li, X., He, W., Gou, L., and Ren, L. A streamlined approach to multimodal few-shot class incremental learning for fine-grained datasets. arXiv preprint arXiv:2403.06295, 2024. Dufumier, B., Castillo-Navarro, J., Tuia, D., and Thiran, J.-P. What to align in multimodal contrastive learning? arXiv preprint arXiv:2409.07402, 2024. Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pp. 178–178, 2004. Fu, Y., Li, J., Liu, J., Xing, Q., Wang, Q., and King, I. Hc-glad: Dual hyperbolic contrastive learning for unsupervised graph-level anomaly detection, 2024. URL https://arxiv.org/abs/2407.02057. Ganea, O., Becigneul, G., and Hofmann, T. Hyperbolic entailment cones for learning hierarchical embeddings. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1646–1655. PMLR, 10–15 Jul 2018a. URL https://proceedings.mlr.press/v80/ ganea18a.html. Ganea, O., Bécigneul, G., and Hofmann, T. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018b. Han, J., Zhang, R., Shao, W., Gao, P., Xu, P., Xiao, H., Zhang, K., Liu, C., Wen, S., Guo, Z., et al. Imagebindllm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905, 2023. He, N., Liu, J., Zhang, B., Bui, N., Maatouk, A., Yang, M., King, I., Weber, M., and Ying, R. Position: Beyond euclidean–foundation models should embrace noneuclidean geometries. arXiv preprint arXiv:2504.08896, 2025. Helber, P., Bischke, B., Dengel, A., and Borth, D. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification, 2019. URL https: //arxiv.org/abs/1709.00029. Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. URL https://arxiv.org/abs/ 1612.06890. Kim, W., Chun, S., Kim, T., Han, D., and Yun, S. Hype: Hyperbolic entailment filtering for underspecified images

Limitation and Future directions. This study follows a hyperbolic geometry-preserving route: HMCL constrains admissible updates to preserve relational and hierarchical structure in Lorentz space, rather than using replay. Combining this route with replay-style mechanisms, such as exemplar/generative replay or hyperbolic memory selection, and extending it to full-backbone adaptation remain promising directions.

Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. The main limitations and future directions of the technical contribution are discussed in the conclusion.

References Ayoughi, M., Atigh, M. G., Derakhshani, M. M., Snoek, C. G., Mettes, P., and Groth, P. Continual hyperbolic learning of instances and classes. arXiv preprint arXiv:2506.10710, 2025. Bian, A., Li, W., Yuan, H., Wang, M., Zhao, Z., Lu, A., Ji, P., Feng, T., et al. Make continual learning stronger via c-flat. Advances in Neural Information Processing Systems, 37:7608–7630, 2024. Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision, pp. 446– 461. Springer, 2014. Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930, 2020. Chami, I., Ying, Z., Ré, C., and Leskovec, J. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32, 2019. Chen, S., Li, H., Wang, Q., Zhao, Z., Sun, M., Zhu, X., and Liu, J. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. NeurIPS, 36:72842–72866, 2023. Chen, W., Han, X., Lin, Y., Zhao, H., Liu, Z., Li, P., Sun, M., and Zhou, J. Fully hyperbolic neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5672–5686, 2022. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and 10

Hyperbolic Multimodal Continual Learning

and texts. In European Conference on Computer Vision, pp. 247–265. Springer, 2024. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. Krioukov, D., Papadopoulos, F., Kitsak, M., Vahdat, A., and Boguná, M. Hyperbolic geometry of complex networks. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 82(3):036106, 2010. Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Lam, T.-Y. Introduction to Quadratic Forms over Fields, volume 67 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2005. ISBN 978-1-4704-8017-2. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. doi:10.1109/5.726791. Li, J., Li, Y., Fu, Y., Liu, J., Liu, Y., Yang, M., and King, I. Clip-powered domain generalization and domain adaptation: A comprehensive survey. arXiv preprint arXiv:2504.14280, 2025. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740–755. Springer, 2014. Liu, J., Yang, M., Zhou, M., Feng, S., and Fournier-Viger, P. Enhancing hyperbolic graph embeddings via contrastive learning, 2022. URL https://arxiv.org/abs/ 2201.08554. Liu, J., Fu, X., Yang, M., Zhang, W., Ying, R., and King, I. Flatland: Personalized federated learning via tailored lorentz space. In FedKDD: International Joint Workshop on Federated Learning for Data Mining and Graph Analytics, 2024. URL https://openreview.net/ forum?id=vVikOgFrpW. Liu, W., Zhu, F., Wei, L., and Tian, Q. C-clip: Multimodal continual learning for vision-language model. In The Thirteenth International Conference on Learning Representations, 2025a. Liu, X., Xia, X., Ng, S.-K., and Chua, T.-S. Continual multimodal contrastive learning. arXiv preprint arXiv:2503.14963, 2025b. Liu, X., Xia, X., Ng, S.-K., and Chua, T.-S. Principled multimodal representation learning. arXiv preprint arXiv:2507.17343, 2025c. Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017. Loshchilov, I. and Hutter, F. Decoupled weight decay regu-

larization. arXiv preprint arXiv:1711.05101, 2017. Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft, 2013. URL https://arxiv.org/abs/1306.5151. Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765–7773, 2018. Mallya, A., Davis, D., and Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European conference on computer vision (ECCV), pp. 67–82, 2018. Mandica, P., Franco, L., Kallidromitis, K., Petryk, S., and Galasso, F. Hyperbolic learning with multimodal large language models. In European Conference on Computer Vision, pp. 382–398. Springer, 2024. Mishne, G., Wan, Z., Wang, Y., and Yang, S. The numerical stability of hyperbolic representation learning. In International Conference on Machine Learning, pp. 24925–24949. PMLR, 2023. Moretti, V. The interplay of the polar decomposition theorem and the lorentz group. arXiv preprint mathph/0211047, 2002. Ni, Z., Wei, L., Tang, S., Zhuang, Y., and Tian, Q. Continual vision-language representation learning with off-diagonal information. In International Conference on Machine Learning, pp. 26129–26149. PMLR, 2023. Nickel, M. and Kiela, D. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017. Nickel, M. and Kiela, D. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International conference on machine learning, pp. 3779–3788. PMLR, 2018. Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 722–729, 2008. doi:10.1109/ICVGIP.2008.47. Pal, A., van Spengler, M., di Melendugno, G. M. D., Flaborea, A., Galasso, F., and Mettes, P. Compositional entailment learning for hyperbolic vision-language models. In The Thirteenth International Conference on Learning Representations, 2025. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019. Poppi, T., Kasarla, T., Mettes, P., Baraldi, L., and Cucchiara, R. Hyperbolic safety-aware vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4222–4232, 2025. Qiu, Z., Liu, J., Chen, Y., and King, I. Hihpq: Hierarchical 11

Hyperbolic Multimodal Continual Learning

hyperbolic product quantization for unsupervised image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4614–4622, 2024. doi:10.1609/aaai.v38i5.28261. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748–8763, 2021. Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International symposium on graph drawing, pp. 355–366. Springer, 2011. Shimizu, R., Mukuta, Y., and Harada, T. Hyperbolic neural networks++. arXiv preprint arXiv:2006.08210, 2020. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642. Association for Computational Linguistics, 2013. URL https://aclanthology.org/D13-1170/. Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant CNNs for digital pathology, 2018. URL https://arxiv.org/abs/1806. 03962. Xiong, C., Liu, J., Chen, H., Zheng, H., Wu, X., Zheng, Y., Sung, J. J., and King, I. Enhancing multimodal survival prediction with pathology reports in hyperbolic space, 2024. URL https://openreview.net/forum? id=PbC786k7qc. Yang, M., Li, Z., Zhou, M., Liu, J., and King, I. Hicf: Hyperbolic informative collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2212–2221, 2022a. doi:10.1145/3534678.3539475. Yang, M., Zhou, M., Liu, J., Lian, D., and King, I. Hrcf: Enhancing collaborative filtering via hyperbolic geometric regularization. In Proceedings of the ACM web conference 2022, pp. 2462–2471, 2022b. Yang, M., Liu, J., King, I., and Ying, R. UHCone: Uni-

12

versal hyperbolic cone for implicit hierarchical learning. In ICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling, 2024a. URL https://openreview.net/forum? id=BBgop7vYvX. Yang, M., Verma, H., Zhang, D. C., Liu, J., King, I., and Ying, R. Hypformer: Exploring efficient transformer fully in hyperbolic space. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3770–3781, 2024b. Yang, M., B B, R. S., Feng, A., Xiong, B., Liu, J., King, I., and Ying, R. Hyperbolic fine-tuning for large language models. In Advances in Neural Information Processing Systems, 2025a. Yang, M., Zhou, M., Zhang, T., Liu, J., Li, Z., Pan, L., Xiong, H., and King, I. Hyperbolic graph neural networks: A review of methods and applications, 2025b. URL https://arxiv.org/abs/2202.13852. Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67–78, 2014. Yu, D., Zhang, X., Chen, Y., Liu, A., Zhang, Y., Yu, P. S., and King, I. Recent advances of multimodal continual learning: A comprehensive survey. arXiv preprint arXiv:2410.05352, 2024a. Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., and He, Y. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219–23230, 2024b. Zhang, Y., Zhu, H., Yang, M., Liu, J., Ying, R., King, I., and Koniusz, P. Understanding and mitigating hyperbolic dimensional collapse in graph contrastive learning. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1984–1995, 2025. doi:10.1145/3690624.3709249. Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., and You, Y. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 19125–19136, 2023.

Appendix Contents A Notation Table

14

B Preliminaries

15

B.1 Overview of Hyperbolic Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

B.2 Lorentz Manifold: Formal Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

B.3 Lorentz Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

C Related Work

17

C.1 Hyperbolic Multimodal Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 C.2 Continual Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D Justifications of Problem Statement

18

E Supplementary for Theoretical Characterization

19

E.1 Proof of Geometric Characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

E.2 Proof of Hierarchy Preservation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

F Deviations for Methodology

23

F.1

Proof of Proposition 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

F.2

Proof for Corollary 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

F.3

Finite-step optimization with AdamW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

G Supplementary for Implementation Details

25

G.1 Dataset Details and Task Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

G.2 Supplementary Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

G.3 Supplementary Robustness and Drift Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

G.4 Supplementary Materials for Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

13

Hyperbolic Multimodal Continual Learning

A. Notation Table For reference, we provide a comprehensive summary of the notation used throughout this paper. Table 2. Summary of notation used throughout the paper. Symbol

Description

Hyperbolic geometry and the Lorentz model d K HdK LdK G ⟨·, ·⟩L dK L (x, y) o x0 x[1:d] SO+ (1, d) SO(d)

Dimension of hyperbolic space Positive curvature magnitude (K > 0); the sectional curvature is −K d-dimensional hyperbolic space with curvature −K Lorentz manifold (HdK , G) with metric structure Minkowski metric tensor, G = diag(−1, 1, . . . , 1) ∈ R(d+1)×(d+1) P Lorentzian inner product: ⟨x, y⟩L = x⊤ Gy = −x0 y0 + di=1 xi yi −1/2 Geodesic distance: dK arcosh (−K⟨x, y⟩L ) L (x, y) = K

Origin of the Lorentz model: o = (K −1/2 , 0, . . . , 0)⊤ ∈ HdK Time-like coordinate (first component of x) Space-like coordinates: (x1 , . . . , xd )⊤ ∈ Rd Proper orthochronous Lorentz group Special orthogonal group (rotation group) on Rd

Multimodal learning M Number of modalities m, m′ Modality indices, m, m′ ∈ {1, 2, . . . , M } N Number of samples (batch size) Xm Data space of modality m ′ m′ N X m,m Paired dataset: {(xm i , xi )}i=1 m xi The i-th input sample from modality m d Hyperbolic embedding of the i-th sample from modality m, where zm zm i ∈ HK i Continual learning framework t Task/stage index (t) Transformation layer for modality m at task t Tm Wm,t Learnable weight matrix for modality m at stage t Xtm Input data for task t, modality m Zm,∗ Task t−1 embeddings from the frozen pre-trained encoder (∗ indicates pre-trained weights) t−1 Zm,t−1 Task t−1 embeddings processed by the transformation layer at stage t−1 t−1 Zm,t Task t−1 embeddings processed by the transformation layer at stage t t−1 Zm,t Task t embeddings processed by the transformation layer at stage t t f (W, ·) Transformation function ensuring outputs lie on the Lorentz manifold HdK Similarity measures and loss functions ′ ′ Sm→m Similarity matrix from modality m to m′ : Sm→m ∈ RN ×N ′ m m′ Sm→m Pairwise similarity (negative distance): −dK ij L (zi , zj ) τ Temperature parameter for contrastive loss Lm→m′ Unidirectional contrastive loss from modality m to m′ Lcontrast Symmetric contrastive loss: 12 (Lm→m′ + Lm′ →m ) aper(z) Entailment cone aperture (half-angle) at embedding z ext(z, z′ ) Exterior angle between embeddings z and z′ κ Boundary constant for entailment cones (typically κ = 0.1) Lentail Entailment loss for enforcing hierarchical relationships Theoretical analysis and parameter updates + L General Lorentz transformation,  L ∈ ⊤SO  (1, d) 1 0 e R Spatial rotation matrix: R = e with R ∈ SO(d) 0 R e R Rotation matrix in SO(d) (spatial block of R) δm Unconstrained update direction (e.g., gradient) for modality m η Learning rate / stepsize

14

Hyperbolic Multimodal Continual Learning

B. Preliminaries B.1. Overview of Hyperbolic Space Hyperbolic space is a non-Euclidean geometry characterized by constant negative curvature. Unlike Euclidean space, which has zero curvature (flat), or spherical geometry, which has positive curvature, hyperbolic space curves "away from itself" at every point. This fundamental property gives hyperbolic space several distinctive characteristics: Exponential growth of volume: n-dimensional hyperbolic space Hn with curvature −1, the volume of a ball of R r In n−1 radius r grows as VHn (r) ∝ 0 sinh (t) dt, which behaves asymptotically as e(n−1)r for large r. In contrast, the volume of a Euclidean ball grows polynomially as rn . For instance, in the hyperbolic plane (n = 2), the area of a disk is A(r) = 2π(cosh(r) − 1), compared to πr2 in the Euclidean plane. Natural hierarchy representation: The exponential growth property makes hyperbolic space particularly well-suited for embedding hierarchical structures such as trees, taxonomies, and knowledge graphs with low distortion (Nickel & Kiela, 2017; Sarkar, 2011). B.2. Lorentz Manifold: Formal Definitions The Lorentz model realizes d-dimensional hyperbolic space as a hyperboloid embedded in (d + 1)-dimensional Minkowski space. We now provide the formal definitions. Definition 1 (Minkowski Space and Metric Tensor). The (d + 1)-dimensional Minkowski space is the vector space Rd+1 equipped with the indefinite metric tensor G = diag(−1, 1, . . . , 1) ∈ R(d+1)×(d+1) , where K > 0 denotes the magnitude of the negative curvature. The metric tensor G has signature (−, +, . . . , +), with one negative eigenvalue corresponding to the time dimension and d positive eigenvalues corresponding to the space dimensions. Definition 2 (Lorentzian Inner Product). The Lorentzian inner product (also called the Minkowski inner product) for vectors x, y ∈ Rd+1 is defined as: d X ⟨x, y⟩L = x⊤ Gy = −x0 y0 + xi yi . i=1

Definition 3 (Lorentz Manifold). The d-dimensional Lorentz manifold (or hyperboloid model of hyperbolic space) with sectional curvature −K (K > 0) is defined as: HdK = {x ∈ Rd+1 | ⟨x, x⟩L = −1/K, x0 > 0} = {x ∈ Rd+1 | x⊤ Gx = −1/K, x0 > 0}. We denote the Lorentz manifold with its metric structure as LdK = (HdK , G). Remark 2. The constraint x⊤ Gx = −1/K expands to −x20 + x21 + x22 + · · · + x2d = −1/K, or equivalently x20 − (x21 + x22 + · · · + x2d ) = 1/K. This is a two-sheeted hyperboloid in Rd+1 . The condition x0 > 0 selects the connected upper sheet. For any point x = (x0 , x1 , . . . , xd )⊤ ∈ HdK , q x0 = K −1 + ∥x[1:d] ∥2 ≥ K −1/2 , where x[1:d] = (x1 , . . . , xd )⊤ denotes the space dimensions. Definition 4 (Induced Norm to the Origin (Desai et al., 2023)). Given the origin of the Lorentz manifold o = (K −1/2 , 0, . . . , 0) ∈ LdK and x ∈ HdK , the induced distance to the origin is q  √ −1/2 −1/2 2 . dK (x, o) = K arcosh( K x ) = K arcosh 1 + K∥x ∥ 0 [1:d] L 15

(18)

Hyperbolic Multimodal Continual Learning

A fundamental concept in any geometric space is the notion of distance. In hyperbolic space, the geodesic distance (the length of the shortest path between two points) is given by the following formula. Definition 5 (Lorentzian Distance). The geodesic distance between two points x, y ∈ LdK is:
−1/2 dK arcosh (−K⟨x, y⟩L ) = K −1/2 arcosh −Kx⊤ Gy . L (x, y) = K

(19)

Remark 3. To verify that this formula is well-defined, observe that for any x, y ∈ HdK : • The Lorentzian inner product satisfies ⟨x, y⟩L ≤ −1/K (with equality if and only if x = y), by the reverse Cauchy– Schwarz inequality for future-directed time-like vectors. • Therefore, −K⟨x, y⟩L ≥ 1, so the argument of arcosh lies in [1, ∞). −1/2 • When x = y, −K⟨x, x⟩L = 1, giving dK arcosh(1) = 0. L (x, x) = K

Remark 4 (Comparison with Euclidean Distance). The hyperbolic distance grows much more slowly than the Euclidean distance in the ambient space. For points far from the origin, the hyperbolic distance grows approximately logarithmically with the Euclidean distance. This property allows hyperbolic space to embed exponentially growing structures (like trees) with bounded distortion. The symmetries of the Lorentz model of hyperbolic geometry are described by the Lorentz group, which consists of all linear transformations that preserve the Lorentzian inner product. Definition 6 (Proper Orthochronous Lorentz Group). The proper orthochronous Lorentz group is defined as: n o SO+ (1, d) := L ∈ R(d+1)×(d+1) : L⊤ GL = G, det(L) = 1, L00 ≥ 1 , where G = diag(−1, 1, . . . , 1) is the Lorentz metric tensor. Remark 5. The three conditions in the definition have the following interpretations: • L⊤ GL = G: The transformation preserves the Lorentzian inner product, i.e., ⟨Lx, Ly⟩L = ⟨x, y⟩L for all x, y. • det(L) = 1: The transformation is proper (orientation-preserving), excluding reflections. • L00 ≥ 1: The transformation is orthochronous (time-orientation preserving), ensuring that the upper sheet of the hyperboloid is mapped to itself. The Lorentz group acts transitively on HdK : for any two points x, y ∈ HdK , there exists L ∈ SO+ (1, d) such that Lx = y. B.3. Lorentz Neural Networks Definition 7 (Lorentz Transformation Layer (Chen et al., 2022)). Let z ∈ Rd+1 be a Lorentz embedding, where z0 > 0 denotes the time-like coordinate and zs ∈ Rd denotes the space-like coordinates. A Lorentz transformation layer is defined as p  K −1 + ∥Wz∥22 f (W; z) = , W ∈ Rd×(d+1) , (20) Wz where K > 0 is the magnitude of the negative curvature, so the sectional curvature is −K. Remark 6. The time-like coordinate in Equation (20) is reconstructed deterministically from the spatial component to satisfy the Lorentz constraint. We further decompose the transformation matrix as   W = w0 Ws ,

w0 ∈ Rd×1 , Ws ∈ Rd×d ,

where w0 and Ws act on the time-like and space-like components of the input embedding, respectively.

16

(21)

Hyperbolic Multimodal Continual Learning

C. Related Work C.1. Hyperbolic Multimodal Learning Hyperbolic geometry is a powerful framework for representation learning on hierarchical and scale-free data (Krioukov et al., 2010; Nickel & Kiela, 2017; 2018; Ganea et al., 2018b; Shimizu et al., 2020; Chami et al., 2019; Yang et al., 2025b; He et al., 2025). Recent work has broadened hyperbolic modeling beyond graph embeddings to recommendation, retrieval, fully hyperbolic transformers, foundation-model fine-tuning, and federated adaptation (Yang et al., 2022b;a; Qiu et al., 2024; Yang et al., 2024b; 2025a; Liu et al., 2024). In parallel, hyperbolic contrastive learning has been studied for graph embeddings and anomaly detection, including analyses of dimensional collapse in curved contrastive spaces (Liu et al., 2022; Zhang et al., 2025; Fu et al., 2024). Hyperbolic geometry has recently been introduced to multimodal representation learning to capture the inherent hierarchical structure between visual and linguistic concepts. Desai et al. (Desai et al., 2023) proposed MERU, the first large-scale hyperbolic vision-language model that projects CLIP-style embeddings onto the Lorentz hyperboloid. Subsequent works have extended hyperbolic multimodal learning along several directions. Pal et al. (Pal et al., 2025) introduced compositional entailment learning to capture intra-modal hierarchies, modeling how complex scenes compose from simpler visual primitives and how compound phrases relate to atomic concepts. Yang et al. (Yang et al., 2024a) further investigated hyperbolic cones for implicit hierarchical learning without requiring explicit parent-child supervision. For scaling to larger architectures, Mandica et al. (Mandica et al., 2024) investigated hyperbolic embeddings in billion-parameter models like BLIP-2. Kim et al. (Kim et al., 2024) leveraged hyperbolic entailment cones for data filtering, using image specificity metrics derived from cone apertures to remove underspecified samples from pretraining datasets. More recently, Poppi et al. (Poppi et al., 2025) proposed HySAC, which exploits hyperbolic geometry to model safety hierarchies in vision-language models. Beyond image-text pretraining, hyperbolic multimodal modeling has also been explored for hierarchical relations between pathology reports and whole-slide images (Xiong et al., 2024). Despite these advances in static settings, how hyperbolic multimodal representations behave under continual learning, where the geometric structure encoding semantic hierarchy must be preserved across sequential tasks, remains unexplored. C.2. Continual Learning Continual learning aims to enable models to sequentially acquire new knowledge while retaining previously learned information (Yu et al., 2024a). The central challenge is mitigating catastrophic forgetting, where learning new tasks degrades performance on earlier ones. Existing approaches can be broadly categorized into three types: (i) replay-based methods, which store and rehearse representative samples from previous tasks, such as Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) and Dark Experience Replay (DER) (Buzzega et al., 2020); (ii) regularization-based methods, which constrain parameter updates to preserve important weights, exemplified by Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017); and (iii) parameter isolation methods, which allocate distinct model components to different tasks to prevent interference, such as Progressive Neural Networks (Rusu et al., 2016), PackNet (Mallya & Lazebnik, 2018), and Piggyback (Mallya et al., 2018). Recent work has extended continual learning to vision-language models, where preserving cross-modal alignment poses additional challenges beyond single-modality forgetting (Ni et al., 2023). Ni et al. (Ni et al., 2023) identified Spatial Disorder, which comprises intra-modal rotation and inter-modal deviation, as a key failure mode in continual CLIP training. Subsequent methods have explored knowledge distillation to retain zero-shot capabilities (Zheng et al., 2023), mixture-ofexperts adapters for parameter-efficient adaptation (Yu et al., 2024b), and contrastive regularization such as C-FLAT (Bian et al., 2024) to align feature representations across tasks. While recent work has explored hyperbolic geometry for continual classification (Ayoughi et al., 2025; Doan et al., 2024), these methods focus on single-modal image classification and do not address cross-modal geometric preservation. In contrast, our work targets hyperbolic multimodal representations, where semantic hierarchy encoded through radial coordinates and entailment cones requires geometry-aware continual learning strategies.

17

Hyperbolic Multimodal Continual Learning

D. Justifications of Problem Statement The motivation for this investigation stems from the fundamental challenge in continual learning scenarios where new knowledge must be integrated without catastrophic forgetting of historical information. In hyperbolic spaces, semantic knowledge is encoded through geometric relationships, making the preservation of these geometric structures paramount for maintaining learned representations. That is to maintain the semantic similarity and the semantic specificity (hierarchy) (Desai et al., 2023). Recall the three preservation goals: m,t ⊤ m,t−1 m,t−1 ⊤ (P1) Intra-modal preservation: Zm,t t−1 G(Zt−1 ) = Zt−1 G(Zt−1 ) , ∀m ∈ {1, 2, . . . , M }; ′

′

m ,t ⊤ m,t−1 m ,t−1 ⊤ (P2) Inter-modal preservation: Zm,t ) , ∀m, m′ ∈ {1, 2, . . . , M }, m ̸= m′ ; t−1 G(Zt−1 ) = Zt−1 G(Zt−1 m,t−1 (P3) Hierarchical preservation: ∥(zm,t t−1 )[1:d] ∥ = ∥(zt−1 )[1:d] ∥, , ∀m ∈ {1, 2, . . . , M }.

Table 3. Correspondence between semantic preservation goals and geometric invariants in hyperbolic space.

Critical Semantic Properties

Geometric Foundation

Preservation Conditions

− Primary Goals Semantic Similarity Semantic Specificity (Hierarchy)

dL (x, y) ∝ ⟨x, y⟩L dL (x, o) ∝ ∥x[1:d] ∥

(P1), (P2) (P3)

− Consequence (Proposition 2) Partial Order Structure

aper(x), ext(x, y)

Automatically ensured

As shown in Table 3, the preservation goals directly target the geometric foundations of hyperbolic semantic encoding. • Semantic Similarity: (P1) and (P2) preserve semantic similarity through inner product invariance. Since Lorentzian distances (Equation (19)) depend monotonically on inner products ⟨x, y⟩L , preserving these inner products ensures that semantic similarity remain invariant. • Semantic Hierarchy: (P3) preserves semantic specificity through spatial norm invariance. The induced norm (Definition 4) indicates that distance to origin depends uniquely on spatial norm ∥x[1:d] ∥. Larger spatial norms position concepts farther from the generic origin, encoding greater specificity in the hierarchical structure. ′

m Proposition 2 (Concepts Partial Order Invariance). Let zm ∈ HdK be hyperbolic embeddings. If both spatial i , zi m m′ m′ norms ∥(zi )[1:d] ∥ and ∥(zi )[1:d] ∥, together with the Lorentzian inner product ⟨zm i , zi ⟩L , are preserved, then the m′ entailment relationship zm i ⊒ zi remains invariant.

′

m Proof. The entailment relationship is determined by the condition Lentail (zm i , zi ) = 0, which occurs when: ′

m m ext(zm i , zi ) ≤ aper(zi )

From the given definitions: ! 2κ √ K ∥(zm i )[1:d] ∥   ′ ′ m m m m (zi )0 + K(zi )0 ⟨zi , zi ⟩L  m′ −1  q ext(zm i , zi ) = cos
2 m′ ∥(zm ) ∥ K⟨zm −1 i [1:d] i , zi ⟩L −1 aper(zm i ) = sin

(22)

(23)

p Since the time-like coordinates are determined by the corresponding spatial norms through z0 = K −1 + ∥z[1:d] ∥2 , the aperture and exterior angle depend only on the two preserved spatial norms and the preserved Lorentzian inner product. Hence the entailment condition, and therefore the partial-order relation, remains unchanged. 18

Hyperbolic Multimodal Continual Learning

E. Supplementary for Theoretical Characterization Definition 8 (Joint Embedding Matrix). Given N paired samples from task t ∈ T processed at stage s ∈ S with modalities m,s′ m1 , m2 , and individual embedding matrices Zm,s ∈ RN ×(d+1) for each modality m ∈ {m1 , m2 }. Let zm,s ∈ Hd t t,i , zt,i for all i = 1, . . . , N and m ∈ {m1 , m2 }. the joint embedding matrix is defined as: Zst,all :=

 m1 ,s  Zt ∈ R2N ×(d+1) , 2 ,s Zm t

where the vertical concatenation preserves the correspondence between paired samples across modalities. Definition 9 (Extended Lorentz Gram Matrix). Given the joint embedding matrix Zst,all ∈ R2N ×(d+1) and the Lorentz metric tensor G := diag(−1, 1, . . . , 1) ∈ R(d+1)×(d+1) , we define the extended Lorentz Gram matrix as: Gst,ext := Zst,all G(Zst,all )⊤  m1 ,s 1 ,s ⊤ Zt G(Zm ) t = m2 ,s m1 ,s ⊤ Zt G(Zt )

(24)  1 ,s Zm G(Ztm2 ,s )⊤ t ∈ R2N ×2N . 2 ,s Zm G(Ztm2 ,s )⊤ t

(25)

The diagonal blocks encode intra-modal Lorentzian inner products and the off-diagonal blocks capture cross-modal geometric relationships in the hyperbolic space. Definition 10 (Lorentz Boost). A Lorentz boost represents relative motion at constant velocity without any rotation of the spatial coordinate axes. Given a velocity vector v ∈ Rn (in units of the speed of light), where ∥v∥ < 1 and γ = √ 1 2 , 1−∥v∥

the Lorentz boost matrix is expressed as: "

γ B= −γv

−γv⊤ γ2 I + 1+γ vv⊤

#

Definition 11 (Lorentz Rotation). A Lorentz rotation refers to the rotation of the spatial coordinates. The corresponding Lorentz rotation matrices are given by:  ⊤ 1 R= ⊤ 0

 0⊤ , R̃

where R̃⊤ R̃ = I and det(R̃) = 1,

i.e., R̃ ∈ SO(n) is a special orthogonal matrix. E.1. Proof of Geometric Characterization Lemma 1 (Witt’s Extension Theorem (Lam, 2005)). Let (V, Q) be a non-degenerate quadratic space, where V is a vector space and Q : V × V → R is a non-degenerate quadratic form. If ϕ : U → V ′ is an isometry between subspaces U, V ′ ⊂ V, then ϕ extends to an isometry of the entire space V. Remark 7. The matrix setting of Lemma 1 can be rewritten as: given matrices Z1 , Z2 ∈ Rm×n and a non-degenerate ⊤ symmetric matrix Q ∈ Rn×n satisfying Z1 QZ⊤ 1 = Z2 QZ2 , there exists an orthogonal transformation T ∈ O(Q) such ⊤ n×n ⊤ that Z2 = Z1 T , where O(Q) := {T ∈ R : T QT = Q}. Lemma 2 (Hyperboloid Isometries are Proper Orthochronous). Let L ∈ O(1, d) be an isometry that maps the hyperbolic space Hd := {x ∈ Rd+1 : x⊤ Gx = −1, x0 > 0} to itself. Then L ∈ SO+ (1, d). Proof. Since L maps Hd to itself, for any z = (z0 , z1 , . . . , zd )⊤ ∈ Hd with z0 > 0 and z⊤ Gz = −1, we have Lz ∈ Hd , Pd which requires (Lz)0 = L00 z0 + i=1 L0i zi > 0 and (Lz)⊤ G(Lz) = −1 Pd First, consider any z ∈ Hd . Since z⊤ Gz = −1, we have z02 = 1 + i=1 zi2 ≥ 1, so z0 ≥ 1 (using z0 > 0). For the constraint (Lz)0 > 0 to hold for all such z, consider the limiting case where spatial components approach zero. Taking z = (1, 0, . . . , 0)⊤ ∈ Hd , we get (Lz)0 = L00 > 0. 19

Hyperbolic Multimodal Continual Learning

From L ∈ O(1, d), we have L⊤ GL = G. Comparing the (0,0) entries: (L⊤ GL)00 = −L200 +

d X

L2i0 = −1

(26)

i=1

which rearranges to L200 = 1 +

Pd

2 i=1 Li0 ≥ 1. Since L00 > 0, we get L00 ≥ 1.

Next, L maps the connected manifold Hd to itself and preserves its orientation as a hypersurface in Rd+1 . Therefore, we have det(L) = +1. Finally, combining L ∈ O(1, d), det(L) = +1, and L00 ≥ 1, we conclude L ∈ SO+ (1, d). ′

Theorem 2 (Lorentzian Isometry Uniqueness). Let Zst,all , Zst′ ,all ∈ R2N ×(d+1) be two joint embedding matrices. If ′ the following conditions hold: (1) Gram matrix equivalence: Gst,ext = Gst′ ,ext , where Gst,ext = Zst,all G(Zst,all )⊤ ; ′ (2) Non-degeneracy: rank(Zst,all ) = rank(Zst′ ,all ) = d + 1. Then there exists a unique L ∈ SO+ (1, d) such that: ′ Zst′ ,all = Zst,all L⊤ , where SO+ (1, d) := {L ∈ R(d+1)×(d+1) : L⊤ GL = G, det(L) = 1, L00 ≥ 1} is the proper orthochronous Lorentz group. Proof. The proof established follows: (i) the existence of an isometry between the row vectors, (ii) its extension to the whole space (i.e., a global isometry), (iii) restriction of the resulting transformation to the orthochronous Lorentz group SO+ (1, d), and (iv) uniqueness. ′

By condition (2), both Zst,all and Zst′ ,all have full rank (d + 1). Hence, their rows span the entire Lorentzian space R1,d . Define a correspondence between row vectors m,s ϕ : zm,s t,i 7→ zt′ ,i

′

(1 ≤ i ≤ N, m ∈ {m1 , m2 }).

Condition (1) guarantees that ϕ preserves the Lorentzian inner product: ′

′

′

′

m ,s m,s m ,s ⟨zm,s t,i , zt,j ⟩L = ⟨zt′ ,i , zt′ ,j ⟩L ,

∀i, j, m, m′ .

Thus ϕ is an isometry between two generating sets of R1,d . Since the space Rd+1 is non-degenerate, Lemma 1 ensures that ϕ extends uniquely to a global isometry of the ambient quadratic space (i.e., any isometry between subspaces of a non-degenerate quadratic space can be extended to an isometry of the entire space). Consequently, there exists L ∈ O(1, d) such that ′

Zst′ ,all = Zst,all L⊤ . It remains to identify the proper subgroup. The orthogonal group O(1, d) consists of four connected components, classified by determinant sign and preservation or reversal of time orientation. ′

Next, according to the definition of joint embedding matrices (Definition 8), zm,s , zm,s ∈ Hd for all i = 1, . . . , N and i i d m ∈ {m1 , m2 }. This ensures that all embeddings lie on the forward sheet H (Definition 3) and L transform Hd to itself, so L ∈ SO+ (1, d) (Lemma 2). ′

Finally, uniqueness follows from full rank. Suppose both L1 , L2 ∈ SO+ (1, d) satisfy the required relation, i.e., Zst′ ,all = s ⊤ Zst,all L⊤ 1 = Zt,all L2 . Then ⊤ Zst,all (L⊤ 1 − L2 ) = 0. Since Zst,all has full rank (d + 1), it follows that L1 = L2 . This proves the uniqueness claim. Combining the existence, subgroup restriction, and uniqueness arguments, we obtain the desired result and complete the proof. Corollary 3. Under the conditions of Theorem 2, the unique Lorentz transformation L ∈ SO+ (1, d) acts identically on both modalities: ′ ′ 1 ,s 1 ,s ⊤ 2 ,s 2 ,s ⊤ Zm = Zm L and Zm = Zm L . t t t′ t′ ′

i ,s i ,s ⊤ In particular, if L1 , L2 ∈ SO+ (1, d) are transformations such that Zm = Zm Li for i ∈ {1, 2}, then L1 = L2 . t t′

20

Hyperbolic Multimodal Continual Learning

Proof. We prove by contradiction. Suppose there exist distinct transformations L1 , L2 ∈ SO+ (1, d) with L1 ̸= L2 such that: ′

1 ,s Zm = Ztm1 ,s L⊤ 1 t′

(27)

′ 2 ,s Zm = Ztm2 ,s L⊤ 2 t′

(28)

Then the joint embedding transformation becomes: ′

′ Zst′ ,all =

1 ,s Zm t′

!

′

2 ,s Zm t′

 m1 ,s ⊤  Zt L1 = 2 ,s ⊤ Zm L2 t

However, this contradicts Theorem 2, which guarantees the existence of a unique L ∈ SO+ (1, d) such that:  m1 ,s ⊤  Zt L s′ s ⊤ Zt′ ,all = Zt,all L = Ztm2 ,s L⊤ This implies that the transformation matrices for the two modalities must be the same to maintain consistency across the entire embedding. Specifically, the assumption that L1 ̸= L2 leads to a contradiction, as it violates the invariance of the cross-modal blocks. These cross-modal blocks, represented by the off-diagonal blocks of the extended Gram matrix, would ′ ′ 1 ,s ⊤ 2 ,s 1 ,s 2 ,s no longer remain consistent (i.e., Zm L1 GL2 Zm ̸= Zm GZm ), breaking the requirement that the inter-modal t t t′ t′ relationships are preserved. Hence, any valid transformation matrices for the individual modalities must be identical: L1 = L2 .

Remark 8. Based on Definition 9, the preservation conditions (P1) and (P2) can be equivalently expressed as Gtt−1,ext = t Gt−1 t−1,ext , where Gt−1,ext represents the extended Lorentz Gram matrix for task t − 1 data processed at the stage t. Theorem 2 and Corollary 3 indicate that this preservation uniquely determines a single transformation L ∈ SO+ (1, d) applying uniformly to both modalities. Lemma 3 (Generation of SO+ (1, d) (Moretti, 2002)). Every element of SO+ (1, d) can be expressed as a finite composition of Lorentz rotations and boosts. Remark 9. Both the Lorentz boost and the Lorentz rotation are linear transformations defined directly within the Lorentz manifold. Specifically, for any x ∈ Hd , the transformations Bx and Rx both result in vectors in Hd . m,t−1 Theorem 3 (Boost Elimination, Restatement). Let L ∈ SO+ (1, d) satisfy zm,t for embeddings from task t−1 = Lzt−1 t − 1. If condition (P3) holds: m,t−1 ∥(zm,t t−1 )[1:d] ∥ = ∥(zt−1 )[1:d] ∥

for all embeddings, then L must be a pure spatial rotation:   1 0⊤ L= , 0 R̃

R̃ ∈ SO(d).

Proof. By Lemma 3, write L = RB where R is a spatial rotation and B is a boost. For any embedding z = [z0 , z[1:d] ]⊤ , according to Definition 10 we have:     γ2 (Lz)[1:d] = (RBz)[1:d] = R̃(Bz)[1:d] = R̃ −γvz0 + I + vv⊤ z[1:d] . 1+γ Since R̃ preserves norms, condition (P3) requires:   γ2 −γvz0 + I + vv⊤ z[1:d] = ∥z[1:d] ∥. 1+γ 21

Hyperbolic Multimodal Continual Learning

Case 1: If v = 0, then γ = 1. Hence, B = I and the condition trivially holds. Case 2: If v ̸= 0, consider the specific embedding z = [1, 0]⊤ (i.e., origin o). Then:

  γ2 (Bz)[1:d] = −γv · 1 + I + vv⊤ · 0 1+γ = −γv + 0 = −γv.

(29)

The spatial norm after transformation is ∥ − γv∥ = γ∥v∥ > 0, but the original spatial norm is ∥0∥ = 0. Therefore, ∥(Bz)[1:d] ∥ = γ∥v∥ ̸= 0 = ∥z[1:d] ∥, which violates condition (P3). Therefore, we must have v = 0, implying B = I and L = R. Since this argument applies to any embedding satisfying condition (P3), we conclude that L must be a pure spatial rotation of the form given in the theorem statement.

E.2. Proof of Hierarchy Preservation This subsection provides technical proofs supporting the analysis in Appendix 4.2. In particular, the following lemma establishes a first-order relationship between spatial and time-like variations of Lorentz embeddings, which is subsequently used to derive the update constraint in Corollary 1. Lemma 4 (First-order variation of the time-like coordinate). Let z ∈ Rd+1 be a Lorentz embedding satisfying 1 −z2[0] + ∥z[1:d] ∥22 = − K with z[0] > 0. Then, for a first-order perturbation ∆z, the variation of the time-like coordinate satisfies ! z⊤ [1:d] ∆z[1:d] . (30) ∆z[0] = z[0]

Proof. Consider the Lorentz-hyperboloid constraint ϕ(z) := −z2[0] + ∥z[1:d] ∥22 +

1 = 0. K

(31)

For a perturbation e z = z + ∆z, a first-order Taylor expansion of ϕ at z yields ϕ(e z) = ϕ(z) + ∇ϕ(z)⊤ ∆z + O(∥∆z∥22 ) = ∇ϕ(z)⊤ ∆z + O(∥∆z∥22 ). Since e z remains on the manifold up to first order, i.e., ϕ(e z) = 0, the linear term must vanish, i.e., ∇ϕ(z)⊤ ∆z = 0.

(32)

  −2z[0] The gradient of ϕ is given by ∇ϕ(z) = . Substituting into Equation (32) yields −2z[0] ∆z[0] + 2z⊤ [1:d] ∆z[1:d] = 0. 2z[1:d] Dividing by 2 and using z[0] > 0 gives ! z⊤ [1:d] ∆z[0] = ∆z[1:d] , z[0] which proves Equation (30).

22

Hyperbolic Multimodal Continual Learning m,t−1 Corollary 4 (Updates under hierarchical preservation, Restatement). Consider an update ∆z = zm,t applied t−1 − zt−1 m,t−1 to the old-task embedding zt−1 . If the time-like coordinate is required to be first-order invariant, i.e. (P3), then the induced spatial update must satisfy z⊤ (33) [1:d] ∆z[1:d] = 0.

Proof. Let z ∈ Rd+1 denote an old-task embedding and ∆z the induced update. Condition (P3) enforces first-order invariance of the time-like  coordinate, i.e., ∆z[0] = 0. By Lemma 4, any first-order perturbation on the Lorentz hyperboloid  satisfies ∆z[0] =

z⊤ [1:d] z[0]

∆z[1:d] . Combining the two relations yields z⊤ [1:d] ∆z[1:d] = 0, since z[0] > 0.

F. Deviations for Methodology F.1. Proof of Proposition 1 Proposition 3 (Admissible Parameter Updates, Restatement of Proposition 1). Consider the Lorentz transformation m,t m,∗ layer in Appendix 2.1, with parameter decomposition Wm,t = [ w0m,t Wsm,t ]. Let zm,t ; zt−1 ) denote the t−1 = f (W representation of task t−1 data at stage t. Assume Theorem 1 holds at stage t−1, so that the induced isometry preserves the time-like coordinate and acts as a spatial rotation. For an infinitesimal update Wm,t = Wm,t−1 − η ∆Wm , η → 0+ . For the block-wise HMCL update, ∆w0m = 0. Ignoring second-order terms, ∆Wsm is admissible if and only if there exists Ω⊤ = −Ω such that m Zm,∗ t−1 [1 : d] ∆Ws

⊤

⊤ = Zm,t−1 t−1 [1 : d]Ω .

(34)

Because the time-like output is reconstructed from the spatial output, the spatial rotation also preserves it to first order.

Proof. Consider the old-task representations evaluated at stage t: m,t Zm,t , Zm,∗ t−1 [1 : d] = f[1:d] W t−1

= f[1:d] Wm,t−1 − η ∆Wm , Zm,∗ t−1

  m,t−1 m , Zm,∗ = f[1:d] Wm,t−1 , Zm,∗ + O(η 2 ) t−1 − η DW f[1:d] W t−1 ∆W
m,∗ m ⊤ = Zm,t−1 + O(η 2 ). t−1 [1 : d] − η Zt−1 [1 : d] ∆Ws

(35) (36) (37)

On the other hand, by Theorem 1, preservation of old-task representations implies the existence of a spatial isometry R ∈ SO(d) such that m,t−1 ⊤ Zm,t t−1 [1 : d] = Zt−1 [1 : d]R ,

R⊤ = I − η Ω⊤ + O(η 2 ),

Ω⊤ = −Ω

m,t−1 ⊤ 2 = Zm,t−1 t−1 [1 : d] − η Zt−1 [1 : d]Ω + O(η ).

(38) (39)

Comparing the O(η) terms in Equation (37) and Equation (39) yields m Zm,∗ t−1 [1 : d] ∆Ws

⊤

⊤ = Zm,t−1 t−1 [1 : d]Ω ,

Ω⊤ = −Ω.

(40)

p Finally, for y0 = K −1 + ∥ys ∥22 and ∆ys = ys Ω⊤ , ∆y0 = ys⊤ Ω⊤ ys /y0 = 0 because Ω is skew-symmetric. Thus the reconstructed time-like coordinate is preserved to first order. This completes the proof. 23

Hyperbolic Multimodal Continual Learning

F.2. Proof for Corollary 2 Corollary 5 (Canonical Admissible Update, Restatement of Corollary 2). Under the admissibility condition of Proposition 1, consider the admissible update associated with a given unconstrained update direction δsm , while penalizing the induced spatial rotation. Then, in the minimal-rotation regime, the admissible solution satisfies Ω⋆ = 0, and admissibility reduces to
m ⊤ Zm,∗ = 0, ∆w0m = 0. (41) t−1,s ∆Ws The resulting canonical admissible update is given by the null-space projection
∆Wsm = δsm I − Pt−1 ,

 † m,∗ ⊤ ⊤ Pt−1 = (Zm,∗ Zm,∗ Zm,∗ t−1,s ) t−1,s (Zt−1,s ) t−1,s .

Proof. We start from the general admissibility condition in Proposition 1,
m ⊤ ⊤ Zm,∗ = Zm,t−1 t−1,s ∆Ws t−1 [1 : d]Ω ,

Ω⊤ = −Ω.

(42)

(43)

Let δsm denote a given unconstrained update direction for the spatial block. We consider the rotation-regularized admissible update defined by λ 1 2 ∆Wsm − δsm F + ∥Ω∥2F s.t. (43). (44) min m ∆Ws , Ω 2 2 Introducing a Lagrange multiplier Λ, the Lagrangian is L=

1 λ 2 ∆Wsm − δsm F + ∥Ω∥2F 2 2 m ⊤ ⊤ + Λ, Zm,∗ (∆W ) − Zm,t−1 . s t−1,s t−1 [1 : d]Ω

(45) (46)

Taking first-order optimality conditions yields ∆Wsm = δsm − Λ⊤ Zm,∗ t−1,s ,
1 ⊤ Ω = − Skew (Zm,t−1 t−1 [1 : d]) Λ , λ

(47) (48)

where Skew(A) = (A − A⊤ )/2. Substituting Equation (47) into the admissibility constraint Equation (43) gives m,∗ ⊤ m,∗ m ⊤ −1 Zm,∗ ). t−1,s (Zt−1,s ) Λ = Zt−1,s (δs ) + O(λ

In the minimal-rotation regime λ → ∞, the second term vanishes, yielding m,∗ ⊤
† m,∗ Λ = Zm,∗ Zt−1,s (δsm )⊤ . t−1,s (Zt−1,s )

(49)

(50)

Substituting into Equation (47) gives m,∗ ⊤
† m,∗
⊤ ∆Wsm = δsm I − (Zm,∗ Zm,∗ Zt−1,s , t−1,s ) t−1,s (Zt−1,s )

(51)

⋆

while Equation (48) implies Ω = 0. Together with Proposition 1, this completes the proof. F.3. Finite-step optimization with AdamW The derivation in the main text characterizes local admissibility, while practical training applies the projected update repeatedly with a finite learning rate. In our implementation, the projection matrix is fixed within each task stage and every raw spatial update is first mapped to δsm (I − VV⊤ ), so the protected old-task subspace is removed before the optimizer step. For Adam-style momentum, this means the first-moment estimate remains in the projected subspace by induction as long as the stage projector is fixed. Adaptive coordinate-wise preconditioning and finite step sizes can introduce higher-order approximation error, so we do not claim exact global preservation for arbitrary trajectories; instead, HMCL suppresses the dominant first-order leakage into old-task directions, leaving only optimizer-induced and higher-order residuals. Empirically, the robustness analysis in Appendix G.3 shows that the method remains effective under AdamW with nonzero weight decay. 24

Hyperbolic Multimodal Continual Learning

G. Supplementary for Implementation Details G.1. Dataset Details and Task Stream Our continual multimodal learning setting consisting of 15 datasets, covering both image–text classification and cross-modal retrieval tasks. Classification datasets. The classification benchmarks include CIFAR-10 and CIFAR-100 (Krizhevsky, 2009), Caltech101 (Fei-Fei et al., 2004), Food-101 (Bossard et al., 2014), Flowers-102 (Nilsback & Zisserman, 2008), EuroSAT (Helber et al., 2019), DTD (Cimpoi et al., 2013), CLEVR (Johnson et al., 2016), Aircraft (Maji et al., 2013), MNIST (LeCun et al., 1998), PCAM (Veeling et al., 2018), SST-2 (Socher et al., 2013), and Country211 (Radford et al., 2021). These datasets span diverse visual domains and label granularities, ranging from coarse-grained object recognition to fine-grained and scene-level classification. Each dataset is treated as an independent classification task and evaluated using standard image–text classification protocols. Retrieval datasets. Cross-modal retrieval is evaluated on COCO (Lin et al., 2014) and Flickr30k (Young et al., 2014), which consist of paired image–text data and are widely used in multimodal retrieval benchmarks. Both datasets support bidirectional retrieval, including image-to-text and text-to-image retrieval. Task stream construction. All datasets are arranged into a fixed task stream and presented sequentially. At each step, training is performed on the current dataset only, without access to data from previous tasks. Classification and retrieval tasks are interleaved in the task stream to reflect realistic continual multimodal learning scenarios. The same task order is used across all experiments. Dataset statistics. Key statistics of all datasets, including the number of samples, the number of classes for classification tasks, and the number of image–text pairs for retrieval tasks, are summarized in Table 4. Table 4. Dataset statistics and task types used in the continual multimodal stream. Dataset

Task

Classes

Train

Test

Evaluation

FGVC Aircraft (Maji et al., 2013) Caltech-101 (Fei-Fei et al., 2004) CIFAR-10 (Krizhevsky, 2009) CIFAR-100 (Krizhevsky, 2009) CLEVR (Johnson et al., 2016) Country211 (Radford et al., 2021) DTD (Cimpoi et al., 2013) EuroSAT (Helber et al., 2019) Flowers-102 (Nilsback & Zisserman, 2008) Food-101 (Bossard et al., 2014) MNIST (LeCun et al., 1998) PCAM (Veeling et al., 2018) SST-2 (Socher et al., 2013)

Classification Classification Classification Classification Classification Classification Classification Classification Classification Classification Classification Classification Classification

100 102 10 100 8 211 47 10 102 101 10 2 2

3,334 2,448 45,000 45,000 4,500 31,650 1,880 5,000 1,020 68,175 48,000 262,144 6,920

3,333 6,084 10,000 10,000 5,000 21,100 1,880 5,000 6,149 25,250 10,000 32,768 1,821

Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy

COCO (Lin et al., 2014) Flickr30K (Young et al., 2014)

Retrieval Retrieval

– –

118,287 29,000

5,000 1,000

Recall@K Recall@K

G.2. Supplementary Ablation Study Figure 5 compares the full HMCL model with a variant that removes the block-wise constraint (w/o block) across three hyperbolic backbones. While the ablated variant still outperforms the Vanilla baseline, it consistently underperforms the full model in terms of backward transfer (BWT) for both classification and retrieval. This gap is especially pronounced on MERU-L and MERU-B, indicating that block-wise constraints play a central role. These ablation results confirm that explicitly enforcing geometry-aware constraints at the block level is essential for preserving hyperbolic structure during continual learning. Without such constraints, continual updates remain vulnerable to geometric drift, even when trained with hyperbolic losses. We provide additional optimizer robustness and shuffled task-order analyses in Appendix G.3. 25

Hyperbolic Multimodal Continual Learning Vanilla

Ours

w/o block

Retrieval

Overall

40

Classification

MERU-B Retrieval

Overall

40

50

Classification

HyCoCLIP-B Retrieval

Overall

40

30

30

20

20

10

10

10

0

0

0

30 20

BW T

Pe rf

BW T-R 5

R@ 5

BW T-A

Ac c

BW T

Pe rf

BW T-R 5

R@ 5

BW T-A

Ac c

BW T

Pe rf

BW T-R 5

R@ 5

BW T-A

10

Ac c

Metric value (mean ± std)

Classification

MERU-L

Figure 5. Ablation of block-wise admissible updates. Results comparing Vanilla, HMCL (w/ block), and a variant without block-wise decomposition (w/o block). Removing the separation between time-like and space-like updates leads to consistently worse backward transfer, highlighting the importance of block-wise constraints for stability. Table 5. Robustness under AdamW with nonzero weight decay (10−2 ). Method

Acc ↑

BWTA ↑

R@5 ↑

BWTR5 ↑

Performance ↑

BWT ↑

Vanilla HMCL (ours) ∆ vs. Vanilla

41.22 45.25 +9.8%

-11.41 -3.31 +71.0%

24.67 35.50 +43.9%

-15.41 -5.66 +63.3%

39.01 43.95 +12.7%

-11.94 -3.62 +69.7%

G.3. Supplementary Robustness and Drift Analyses We provide the additional analyses requested during rebuttal to clarify the implementation–theory connection and to test whether the observed gains are robust beyond the default optimizer and task order. These experiments are intended as stress tests rather than new assumptions: the AdamW experiment checks optimizer mismatch under nonzero weight decay, the task-order experiment checks sensitivity to shuffled streams, and the drift analysis directly measures whether HMCL reduces both hierarchy-related and relation-related representation changes. G.4. Supplementary Materials for Case Study To further examine the structural advantages of hyperbolic representations in preserving visual–semantic hierarchies, we provide additional geodesic-interpolation case studies on Flickr30K and COCO val2017. For each query image, we interpolate from the image embedding to the [ROOT] node and retrieve the nearest textual item or concept along the geodesic path. This traversal allows us to inspect whether the learned representation moves smoothly from image-specific semantics toward more abstract concepts near the origin. We compare our method, HMCL, with a Vanilla continual fine-tuning baseline. Flickr30K case study. For Flickr30K, we construct a hierarchical text set by extracting nouns and adjectives from the original captions, so that the retrieved items reflect different levels of semantic abstraction. As shown in Figure 6 and Figure 7, cases (1)–(3) perform retrieval over a mixed pool of nouns, adjectives, and captions, while cases (4)–(6) restrict the retrieval pool to captions only. The blue rows correspond to the model state immediately after training on Flickr30K, and the green rows correspond to the model state after subsequent continual-learning tasks. HMCL exhibits fine-grained transitions from image-specific descriptions to more generic concepts and preserves this structure after continual learning, whereas Vanilla is more prone to semantic drift and hierarchical collapse. COCO case study. We further conduct a case study on COCO val2017 to more directly evaluate whether the learned visual–semantic hierarchy remains stable after continual learning. For each COCO image, we first retrieve the nearest caption anchor from the full COCO validation caption pool. We then trace the geodesic path from the image embedding to the origin and retrieve the nearest concepts from a curated three-level vocabulary, where upper rows correspond to more 26

Hyperbolic Multimodal Continual Learning Table 6. Stability under shuffled task orders. These are fresh seed-42 runs from the released implementation, using step pullback with β = 0.10 for MERU and β = 0.20 for HyCoCLIP. The retrieval column uses image-to-text R@K (K = 5 for MERU and K = 1 for HyCoCLIP), while Overall retains text-to-image R@5. Classification

Retrieval

Overall

Stream

Backbone

Method

Acc ↑

BWTA ↑

R@K ↑

BWTRK ↑

Performance ↑

BWT ↑

seq2 seq2 seq2 seq2 seq2 seq2 seq2 seq2 seq2

MERU-L MERU-L MERU-L MERU-B MERU-B MERU-B HyCoCLIP-B HyCoCLIP-B HyCoCLIP-B

Vanilla HMCL (ours) ∆ vs. Vanilla Vanilla HMCL (ours) ∆ vs. Vanilla Vanilla HMCL (ours) ∆ vs. Vanilla

41.00 43.16 +5.3% 43.46 45.97 +5.8% 42.49 42.99 +1.2%

-11.00 -1.45 +86.8% -10.49 -3.13 +70.2% -8.32 -0.41 +95.1%

29.67 39.38 +32.7% 25.29 37.71 +49.1% 32.28 55.94 +73.3%

-10.94 -5.73 +47.6% -17.09 -6.80 +60.2% -15.13 -2.95 +80.5%

38.40 41.22 +7.4% 40.03 43.93 +9.7% 43.59 46.08 +5.7%

-11.00 -2.14 +80.6% -10.49 -3.51 +66.5% -8.32 -1.02 +87.8%

seq3 seq3 seq3 seq3 seq3 seq3 seq3 seq3 seq3

MERU-L MERU-L MERU-L MERU-B MERU-B MERU-B HyCoCLIP-B HyCoCLIP-B HyCoCLIP-B

Vanilla HMCL (ours) ∆ vs. Vanilla Vanilla HMCL (ours) ∆ vs. Vanilla Vanilla HMCL (ours) ∆ vs. Vanilla

43.83 43.20 -1.4% 43.23 45.87 +6.1% 43.32 42.34 -2.3%

-7.84 -1.15 +85.3% -9.29 -2.76 +70.3% -6.38 -0.49 +92.3%

28.43 39.33 +38.3% 22.09 38.14 +72.7% 29.99 55.38 +84.7%

-13.47 -5.88 +56.3% -20.00 -5.93 +70.3% -17.53 -4.54 +74.1%

40.73 41.16 +1.0% 39.53 43.85 +10.9% 44.23 45.45 +2.8%

-7.84 -1.94 +75.3% -9.29 -3.14 +66.2% -6.38 -1.14 +82.2%

Table 7. Summary of quantitative representation-drift analysis. Reductions are averaged over old tasks and compare HMCL against Vanilla. Drift metric

What it measures

Reduction

Radial drift Angular drift Cross-modal drift Paired-distance drift

Changes in hyperbolic radius / time-like hierarchy of old-task embeddings. Within-modality directional distortion after removing magnitude effects. Changes in image–text relational structure for old-task representations. Increase in geodesic distance between matched image–text pairs.

61.0% 57.8% 73.8% 59.5%

The reductions are statistically significant across old tasks (p ≤ 3.1 × 10−4 ). The analysis supports the revised interpretation that forgetting includes both hierarchy-related distortion and relation-level drift, rather than being caused solely by radial changes.

image-specific concepts and lower rows approach more abstract concepts near [ROOT]. We compare HMCL and Vanilla at two checkpoints: T = 6, immediately after COCO is learned, and T = 15, after nine subsequent tasks. The selected cases follow a hard filtering protocol: both models must correctly retrieve the caption and hierarchy anchor at T = 6, while at T = 15, HMCL must still preserve both, and Vanilla must fail in at least one of them. This protocol isolates cases where both models initially solve the task, but only HMCL maintains the visual–semantic hierarchy after continual learning. The vertical paths in Figure 8–Figure 11 show that HMCL maintains stable semantic anchors and smooth abstraction toward the origin, whereas Vanilla is more likely to exhibit caption drift or hierarchy-anchor collapse after subsequent tasks.

27

Hyperbolic Multimodal Continual Learning

HMCL(ours)

Vanilla

A dog runs on the green grass near a wooden fence . A black and white dog is running in a grassy garden surrounded by a white fence .

(1)

dog

handsome

sincerity

fondness

[ROOT] HMCL(ours)

[ROOT] Vanilla

A dog runs on the green grass near a wooden fence . A black and white dog is running in a grassy garden surrounded by a white fence . dog

aloof

sincerity

pleasure

[ROOT]

[ROOT]

HMCL(ours)

Vanilla

The man with pierced ears is wearing glasses and an The man with pierced ears is wearing glasses and an orange hat . orange hat .

(2)

hat

fashion

fashion

style

[ROOT] HMCL(ours)

[ROOT] Vanilla

The man with pierced ears is wearing glasses and an The man with pierced ears is wearing glasses and an orange hat . orange hat . fashion

(3)

kind

cousin

new

[ROOT]

[ROOT]

HMCL(ours)

Vanilla

A girl in a pink shirt slides down an inflatable fun slide .

Young girl sliding down an inflated slide .

summer

summer

conventional

quality

[ROOT] HMCL(ours)

[ROOT] Vanilla

A girl in a pink shirt slides down an inflatable fun slide .

Young girl sliding down an inflated slide .

summer

summer

quick

material

[ROOT]

[ROOT]

Figure 6. Flickr30K geodesic-interpolation case studies, page 1. Cases (1)–(3) retrieve over a mixed pool of captions, nouns, and adjectives. Blue rows show the checkpoint immediately after learning Flickr30K, and green rows show the final checkpoint after subsequent continual-learning tasks.

28

Hyperbolic Multimodal Continual Learning HMCL(ours)

Vanilla

Numerous Asian people working behind a counter .

Numerous Asian people working behind a counter .

An Asian food market where a woman is picking out An Asian food market where a woman is picking out food . food . (4)

Employees at a sushi restaurant prepare for dinner time rush .

A cyclist wearing a black helmet is riding by some black vans .

[ROOT] HMCL(ours)

[ROOT] Vanilla

Numerous Asian people working behind a counter .

Employees at a sushi restaurant prepare for dinner time rush .

An Asian food market where a woman is picking out food .

A cyclist wearing a black helmet is riding by some black vans .

A cyclist wearing a black helmet is riding by some black vans .

Two wet dogs run into the surf at sunset .

[ROOT]

[ROOT]

HMCL(ours)

(5)

Vanilla

2 boys in the foreground in a karate competition and A young boy demonstrates karate in a gymnasium . coaches in background looking on with another coach sitting at table .

(5)

A young boy demonstrates karate in a gymnasium .

A little girl dressed in yellow splashes in a shallow pool .

Man standing by a poster of religious beliefs .

A child plays at a playground .

[ROOT] HMCL(ours)

[ROOT] Vanilla

2 boys in the foreground in a karate competition and Two people are sitting outdoors on a blanket near a coaches in background looking on with another coach tree . sitting at table . A young boy demonstrates karate in a gymnasium .

A surfer jumps a wave .

Man standing by a poster of religious beliefs .

A couple enjoying a glass of white wine .

[ROOT]

[ROOT]

HMCL(ours)

Vanilla

A young boy is sitting on the floor trying to take off A woman wearing glasses and an orange dress is his boots . standing with another woman wearing glasses and a green dress . (6)

A young boy puts on his boots at the top of a flight of stairs .

Three young men wearing hooded sweatshirts , standing at a bench .

A young boy demonstrates karate in a gymnasium .

A couple enjoying a glass of white wine .

[ROOT] HMCL(ours)

[ROOT] Vanilla

A young boy is sitting on the floor trying to take off his boots .

An Indian girl playing a guitar .

A young boy puts on his boots at the top of a flight of stairs .

A couple enjoying a glass of white wine .

couple enjoying a glass of white wine .

Two wet dogs run into the surf at sunset .

[ROOT]

[ROOT]

Figure 7. Flickr30K geodesic-interpolation case studies, page 2. Cases (4)–(6) restrict retrieval to the caption pool. Blue rows show the checkpoint immediately after learning Flickr30K, and green rows show the final checkpoint after subsequent continual-learning tasks.

29

Hyperbolic Multimodal Continual Learning

(1)

(2)

(3)

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A windsurfer watches people kite surfing at the beach

A windsurfer watches people kite surfing at the beach

A windsurfer watches people kite surfing at the beach

Some one is flying a kite high on the beach.

beach cloudy sky ↓ ↓ object ↓ ↓ ↓

beach ↓ sky ↓ ↓ object ↓ ↓ ↓

beach ↓ ↓ sky ↓ ↓ object ↓ ↓

cloudy ↓ ↓ ↓ sky object ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

Two giraffe standing next to each other in a field.

Two giraffe standing next to each other on a dirt road next to a grass field.

Two giraffe standing next to each other on a dirt road next to a grass field.

A giraffe next to a stone fence staring off into the distance.

giraffe ↓ ↓ field area image object ↓ ↓

giraffe ↓ area ↓ image ↓ object ↓ ↓

giraffe ↓ ↓ dry forest ↓ image object ↓

dry ↓ ↓ area ↓ beautiful object ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A bowl of tomato soup and two pieces of grilled cheese.

A bowl of tomato soup is next to grilled cheese.

A bowl of tomato soup and two pieces of grilled cheese.

Two cooked hot dogs on a plate with parsley.

sandwich ↓ ↓ dinner object ↓ ↓ ↓ ↓

sandwich ↓ white dinner ↓ ↓ object ↓ ↓

sandwich ↓ ↓ ↓ white object ↓ ↓ ↓

empty ↓ white ↓ object ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A herd of cattle sitting in front of a church with a steeple.

Four cows stand in the parking lot of a church.

A herd of cattle sitting in front of a church with a steeple.

A horse drawn buggy sits in front of pyramid shaped mosques.

horse area ↓ structure ↓ image ↓ object ↓

horse ↓ structure ↓ beautiful ↓ image object ↓

horse area ↓ ↓ ↓ structure image ↓ object

church ↓ structure old beautiful ↓ ↓ object ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

(4)

(5)

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A large Siamese cat sitting on top of a laptop computer.

A large Siamese cat sitting on top of a laptop computer.

A large Siamese cat sitting on top of a laptop computer.

A cat is relaxing on top of an old television.

cat ↓ ↓ ↓ object ↓ ↓ ↓ ↓

cat ↓ ↓ white object ↓ ↓ ↓ ↓

cat ↓ ↓ ↓ cat image object ↓ ↓

window white ↓ ↓ object ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

Figure 8. COCO geodesic-interpolation case studies, page 1.

30

Hyperbolic Multimodal Continual Learning

(6)

(7)

(8)

(9)

(10)

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A freight train with a green engine coming down the tracks.

A freight train with a green engine coming down the tracks.

A green train is going down the tracks in a rural setting.

A jet is towed down the tarmac on a dreary gray day.

locomotive ↓ train ↓ bright object ↓ ↓ ↓

locomotive ↓ train ↓ old object ↓ ↓ ↓

locomotive ↓ train large object ↓ ↓ ↓ ↓

train blue bright object ↓ ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

Two green stop lights in front of a skyscraper

Two green stop lights in front of a skyscraper

Two green street lights in front of a building with a lot of windows.

Two green street lights in front of a building with a lot of windows.

building ↓ light bright object ↓ ↓ ↓ ↓

building ↓ light view object ↓ ↓ ↓ ↓

building ↓ ↓ bright large object ↓ ↓ ↓

light ↓ bright object ↓ ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A couple of storm troopers standing next to each other.

A couple of storm troopers standing next to each other.

A couple of storm troopers standing next to each other.

A couple of storm troopers standing next to each other.

group photo ↓ ↓ ↓ object ↓ ↓ ↓

group photo ↓ ↓ image object ↓ ↓ ↓

group ↓ ↓ photo ↓ image object ↓ ↓

empty photo ↓ ↓ object ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A grilled chicken sandwich sits beside french fries made with real potatoes.

A grilled chicken sandwich sits beside french fries made with real potatoes.

A grilled chicken sandwich sits beside french fries made with real potatoes.

A grilled chicken sandwich sits beside french fries made with real potatoes.

sandwich ↓ food ↓ object ↓ ↓ ↓ ↓

sandwich ↓ ↓ dinner ↓ object ↓ ↓ ↓

sandwich ↓ ↓ large object ↓ ↓ ↓ ↓

banana sandwich ↓ object ↓ ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A cat curled up in a box with a Pirates hat on.

A black cat in a box wearing a pirate hat

A cat curled up in a box with a Pirates hat on.

A cat curled up in a box with a Pirates hat on.

cat ↓ ↓ object ↓ ↓ ↓ ↓ ↓

cat black ↓ object ↓ ↓ ↓ ↓ ↓

cat ↓ ↓ ↓ ↓ object ↓ ↓ ↓

black ↓ ↓ object ↓ ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

Figure 9. COCO geodesic-interpolation case studies, page 2.

31

Hyperbolic Multimodal Continual Learning HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A computer monitor sitting above a computer keyboard.

A computer monitor sitting above a computer keyboard.

A computer monitor sitting above a computer keyboard.

A black and white photo of a computer keyboard, flash drive, and mouse on a marble tabletop.

laptop ↓ ↓ photo object ↓ ↓ ↓ ↓

laptop ↓ white image ↓ object ↓ ↓ ↓

laptop ↓ dark ↓ image ↓ object ↓ ↓

mirror modern white ↓ object ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

Gray cat lying on seat cushion on outdoor wooden chair.

Gray cat lying on seat cushion on outdoor wooden chair.

Gray cat lying on seat cushion on outdoor wooden chair.

Gray cat lying on seat cushion on outdoor wooden chair.

cat ↓ ↓ object ↓ ↓ ↓ ↓ ↓

cat ↓ ↓ object ↓ ↓ ↓ ↓ ↓

cat ↓ ↓ ↓ object ↓ ↓ ↓ ↓

water ↓ small ↓ object ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

(11)

(12)

(13)

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A stack of random books are placed in front of a computer.

A stack of books sit on a desk next to a bottle of beer.

A stack of books sitting on a desk with a bottle of beer.

A stack of books sit on a desk next to a bottle of beer.

book ↓ ↓ object ↓ ↓ ↓ ↓ ↓

book ↓ ↓ object ↓ ↓ ↓ ↓ ↓

book ↓ ↓ ↓ image object ↓ ↓ ↓

banana ↓ object ↓ ↓ ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A white bowl of broccoli cheese soup looks delicious.

A white bowl of broccoli cheese soup looks delicious.

A white bowl of broccoli cheese soup looks delicious.

A white bowl of broccoli cheese soup looks delicious.

broccoli ↓ dish ↓ ↓ object ↓ ↓ ↓

broccoli dish ↓ ↓ ↓ ↓ ↓ object ↓

broccoli ↓ dish ↓ ↓ object ↓ ↓ ↓

dish ↓ ↓ ↓ ↓ object ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

(14)

(15)

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

The red, double decker bus is driving past other buses.

The red, double decker bus is driving past other buses.

The red, double decker bus is driving past other buses.

The red, double decker bus is driving past other buses.

bus ↓ red ↓ photo object ↓ ↓ ↓

bus ↓ ↓ vintage image object ↓ ↓ ↓

bus ↓ ↓ red photo ↓ object ↓ ↓

sidewalk red ↓ ↓ old object ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

Figure 10. COCO geodesic-interpolation case studies, page 3.

32

Hyperbolic Multimodal Continual Learning

(16)

(17)

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

A cardboard box with a teddy bear in the street.

A cardboard box with a teddy bear in the street.

A cardboard box with a teddy bear in the street.

A cardboard box with a teddy bear in the street.

bear ↓ ↓ animal object ↓ ↓ ↓ ↓

bear ↓ ↓ object ↓ ↓ ↓ ↓ ↓

bear ↓ ↓ animal ↓ object ↓ ↓ ↓

dog ↓ ↓ object ↓ ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

HMCL T =6

HMCL T =15

Vanilla T =6

Vanilla T =15

The toilet paper containers in this public restroom are empty and need to be refilled.

Two empty toilet paper roll holders hang in the stall while the empties lay around and a nice roll sets on the bar.

The toilet paper containers in this public restroom are empty and need to be refilled.

Two empty toilet paper roll holders hang in the stall while the empties lay around and a nice roll sets on the bar.

empty ↓ small ↓ object ↓ ↓ ↓ ↓

empty small ↓ thing object ↓ ↓ ↓ ↓

empty ↓ ↓ small image ↓ object ↓ ↓

metal ↓ ↓ small object ↓ ↓ ↓ ↓

[ROOT]

[ROOT]

[ROOT]

[ROOT]

Figure 11. COCO geodesic-interpolation case studies, page 4.

33

Related documents

• Hardware-Efficient Neuro-Symbolic Networks with the Exp-Minus-Log Operator#14038
• Multimodal learning#31047
• Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models#1049692
• Taming Modality Entanglement in Continual Audio-Visual Segmentation#819260
• Taming Modality Entanglement in Continual Audio-Visual Segmentation#809268