Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models Yishun Lu and Wes Armour
arXiv:2605.16165v1 [cs.CV] 15 May 2026
University of Oxford, Oxford, United Kingdom [email protected], [email protected]
Abstract. Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality competition that destabilizes optimization and limits large-batch scaling. We show that first-order optimizers such as AdamW are vulnerable to cross-modality gradient heterogeneity, while secondorder preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment. Building on this insight, we propose ML-FOPSOAP, a second-order optimization framework with Multi-Level Variance Correction. Our Fisher-Orthogonal Projection suppresses varianceinduced modality conflicts, reducing the trade-off between visual generation and textual understanding. To make this practical under large gradient accumulation, we introduce a hierarchical folding strategy that captures fine-grained variance with low micro-step overhead. Experiments on Janus and Emu3 show consistent gains across both modalities and stable training at batch size 8192. Compared with AdamW, our method improves sample efficiency by up to 1.4× and accelerates wall-clock training by up to 1.5×, offering a robust optimizer for scaling multimodal foundation models. Keywords: Multimodal Alignment · Modality Competition · SecondOrder Optimization · Variance Control · Large-Batch Training
1
Introduction
In recent years, unifying image generation and text understanding under the autoregressive next-token prediction paradigm has become a major trend in the development of vision-language foundation models. However, this deep multi-modal integration introduces a severe modality competition problem during pretraining, which often makes joint training highly unstable and convergence extremely slow. To address this challenge, most existing approaches in the literature primarily focus on model architecture. For example, researchers have attempted to physically disentangle modality-specific representations by designing decoupled visual encoders, introducing multi-way attention mechanisms, or constructing dedicated alignment modules [2, 9, 21, 23, 25]. Beyond architectural modifications, existing efforts to alleviate modality conflict, such as dynamic loss weighting, curriculum scheduling, and gradient balancing [3,4,7,19], remain largely external to the optimizer. In practice, parameter
2
Yishun Lu and Wes Armour
updates are still governed by first-order methods like AdamW [13], which rely on diagonal moment estimates. These methods fail to model the underlying local curvature of the loss surface or the geometric coupling between modalities. This reliance on first-order optimization becomes a critical bottleneck because multi-modal training induces gradient estimators with vastly different noise patterns and curvature sensitivities [8]. Specifically, visual objectives often involve high-dimensional and spatially diffuse prediction errors, whereas textual objectives are typically concentrated on a more compact semantic manifold [26]. This geometric mismatch creates a complex cross-modal covariance structure. Consequently, under standard diagonal preconditioning, modalities associated with larger gradient variance (vision) exert a disproportionate influence on the update trajectory, suppressing lower-variance but semantically dense signals (text) [18, 24]. To resolve this variance conflict at the optimization level, the recently proposed Fisher-Orthogonal Projection (FOP) [14] accelerates convergence by projecting gradients in Fisher geometry. However, its original reliance on K-FAC [15] is memory-prohibitive for the massive vocabularies and wide hidden layers of modern foundation models. To address this, we leverage structured two-dimensional (2D) tensor preconditioners. Interestingly, our empirical analysis reveals that while standard methods like Shampoo [20] are easily destabilized by early-stage visual outliers, the SOAP optimizer [1, 22] provides a highly robust, memoryefficient surrogate for local optimization metrics. Yet, even with SOAP, we observe a lingering trade-off where textual understanding is inadvertently sacrificed to accommodate high-entropy visual noise. Thus, integrating FOP with SOAP becomes essential to fully neutralize these cross-modality conflicts. Furthermore, scaling these unified models to ultra-large global batch sizes (e.g., 8192) requires massive gradient accumulation across numerous micro-steps. Applying variance correction only at the coarse boundaries of a long accumulation window fails to preserve the fine-grained gradient discrepancies arising within the window. Inspired by the telescoping coarse-to-fine viewpoint of the Multi-Level Monte Carlo (MLMC) paradigm [5], we propose a novel Multilevel Hierarchical Gradient Folding mechanism via Fisher-Orthogonal Projection (ML-FOP) . Instead of classical scalar expectation estimation, our approach recursively fuses computationally lightweight cumulative snapshots to preserve intra-step multi-scale variance to achieve multi-level variance correction. Through this methodology, we present a new optimization framework that natively mitigates modality competition and enables the training of unified autoregressive models with ultra-large global data batches, dramatically improving hardware utilization on modern GPU clusters. Specifically, our contributions are as follows: – Second-Order Optimization for Modality Competition. We systematically investigate the optimization dynamics of unified multi-modal autoregressive models. We reveal that while first-order methods (AdamW [13]) and standard uncentered second-order methods (Shampoo [20]) struggle with extreme visual gradient outliers, the SOAP optimizer [22] inherently miti-
Abbreviated paper title
3
gates this cross-modality heterogeneity, establishing a robust foundation for multi-modal alignment. Compared to AdamW, our ML-FOP-SOAP framework delivers a 1.4× gain in sample efficiency and a 1.5× acceleration in end-to-end wall-clock time, uniquely preserving robust performance across both modalities. – Variance Control via Fisher-Orthogonal Projection (FOP). Building upon the SOAP baseline, we introduce a geometry-aware FOP mechanism. We demonstrate that projecting gradient differences orthogonally to the primary descent trajectory actively neutralizes modality competition. This effectively eliminates the conventional trade-off between textual understanding and visual generation, yielding strict Pareto improvements across both tasks. – Efficient Multi-Level FOP for Massive Accumulation. To make FOP tractable under massive gradient accumulation, we propose an efficient MultiLevel hierarchical folding mechanism (ML-FOP). This dyadic telescoping strategy captures fine-grained, cross-modality variance across accumulation segments without the prohibitive computational wall-clock overhead of applying projections at every micro-step. – Ultra-Large-Batch multi-modal Alignment Benchmark. We validate our framework on representative unified architectures (Janus and Emu3) scaled on massive datasets (LLaVA-3M and 12M [11, 12]). We successfully scale the global batch size up to 8192, a regime where existing optimizers suffer catastrophic degradation.
2
Related Work
2.1
Native Unified Vision-Language Models
The dominant paradigm in vision-language foundation models has recently shifted towards unifying image and text modalities within a single autoregressive nexttoken prediction framework [21,23,25]. Unlike earlier compositional architectures that connect a pre-trained vision encoder (e.g., CLIP) to a Large Language Model via projection layers [2, 9], native autoregressive models map all continuous visual signals into discrete tokens, treating both modalities equally. While this architectural convergence enables seamless generation and understanding, it inevitably exacerbates modality competition during joint pre-training. Our work does not alter these advanced model architectures; rather, we provide the optimization infrastructure necessary to stabilize their training and scale them to unprecedented batch sizes. 2.2
Modality Competition and Multitask Optimization
The phenomenon of modality conflict is closely related to gradient interference in multitask learning. Traditional approaches to alleviate this include dynamic loss weighting [3], gradient projection techniques, and curriculum-based data
4
Yishun Lu and Wes Armour
scheduling [4, 7, 19]. However, these methods operate largely external to the optimizer itself. Parameter updates in foundation models are still universally governed by first-order optimizers like AdamW [13]. AdamW relies on diagonal preconditioning, which independently scales coordinates but fundamentally fails to capture the complex, non-diagonal cross-modal covariance structure and the distinct local geometry induced by high-variance visual objectives and dense textual signals [8]. 2.3
Efficient Second-Order Preconditioning
To address the limitations of first-order methods, second-order optimization explicitly models the local curvature of the loss surface. The gold standard for this is Natural Gradient Descent (NGD), which preconditions the Euclidean gradient g using the inverse of the Fisher Information Matrix (FIM), F , yielding the optimal update direction g̃ = F −1 g. However, computing and inverting the exact FIM is fundamentally intractable for deep neural networks. Previous methods often employ explicit approximations, such as K-FAC [15], to make the inversion feasible. The recently proposed Fisher-Orthogonal Projection (FOP) [14] successfully leverages K-FAC to accelerate multi-modal convergence. Unfortunately, explicitly modeling curvature via K-FAC still incurs catastrophic memory overhead when applied to modern VLMs with massive vocabularies and wide hidden layers. To scale geometry-aware optimization, we turn to lightweight tensor preconditioners like Shampoo [20] and SOAP [1, 22], which construct preconditioners based on empirical second-order gradient statistics. Crucially, while standard Shampoo utilizes uncentered second moments that are highly vulnerable to early-stage visual gradient outliers, SOAP projects gradients into a skewed eigenbasis and applies coordinate-wise adaptive normalization. This provides a highly robust, memory-efficient surrogate for the natural gradient: P(g) ≈ F −1 g. By replacing explicit Fisher inversions with SOAP, our framework amortizes the prohibitive memory overhead while capturing the local geometry necessary for multi-modal training.
3
Methodology
3.1
Mathematical Formulation of Modality Competition
Our goal is to formalize modality competition in unified autoregressive multimodal models, such as Janus [25] and Emu3 [23], as an optimization imbalance induced by heterogeneous stochastic gradients. Consider a model parameterized by θ, trained on both image and text data. The joint objective can be written as \mathcal {L}(\theta ) = \mathbb {E}_{x \sim \mathcal {D}_{\mathrm {img}}}\left [L_{\mathrm {img}}(\theta ; x)\right ] + \mathbb {E}_{y \sim \mathcal {D}_{\mathrm {text}}}\left [L_{\mathrm {text}}(\theta ; y)\right ] \label {eq:joint_multimodal_objective} (1) In stochastic optimization, the full gradient ∇L(θ) is approximated using minibatches Bimg ⊂ Dimg and Btext ⊂ Dtext . The practical update is therefore based on the stochastic gradient:
Abbreviated paper title
5
g = g_{\mathrm {img}} + g_{\mathrm {text}} = \frac {1}{|\mathcal {B}_{\mathrm {img}}|} \sum _{x \in \mathcal {B}_{\mathrm {img}}} \nabla _{\theta } L_{\mathrm {img}}(\theta ; x) + \frac {1}{|\mathcal {B}_{\mathrm {text}}|} \sum _{y \in \mathcal {B}_{\mathrm {text}}} \nabla _{\theta } L_{\mathrm {text}}(\theta ; y) \label {eq:stochastic_joint_gradient} (2) which serves as a Monte Carlo estimator of the full gradient ∇L(θ). The key difficulty is that the image and text components generally induce gradient estimators with substantially different noise scales and covariance structures. Let Σimg = Cov(gimg ) , Σtext = Cov(gtext ). In many multi-modal training scenarios, it can be observed that a strong imbalance between these covariance scales, often reflected by Tr(Σimg ) ≫ Tr(Σtext ). Examples can be seen in the supplementary material. This imbalance does not merely affect the magnitude of the stochastic gradient, but also its effective optimization geometry in the shared parameter space. Under first-order adaptive optimizers such as AdamW [13], the update is constructed from per-parameter first and second moment estimates. Abstracting t away bias correction for clarity, the update takes the form dt = √m vt +ϵ , where mt and vt denote the exponential moving averages of the gradient and squared gradient. Since this update rule relies on diagonal preconditioning, it can only rescale coordinates independently and does not explicitly model cross-parameter coupling or curvature interactions across modalities [1]. As a result, when one modality contributes disproportionately high gradient variance, its noisy directions may dominate the optimization trajectory, while weaker but semantically informative signals from another modality are underutilized. 3.2
Fisher Information for Curvature Awareness
To resolve modality conflict induced by Euclidean first-order updates, we seek an optimization geometry that reflects the intrinsic structure of the model distribution rather than raw coordinate-wise gradient magnitudes. Natural gradient descent [17] provides such a perspective by measuring parameter perturbations through the change they induce in the predictive distribution. Specifically, for a small parameter step ∆θ, the Kullback–Leibler divergence between the model before and after the update admits the local approximation D_{\mathrm {KL}}\!\left (p_{\theta } \,\|\, p_{\theta +\Delta \theta }\right ) \approx \frac {1}{2}\Delta \theta ^{\top } F \Delta \theta \label {eq:kl_local_fisher_method}
(3)
where, F is the Fisher information matrix, defined as F = \mathbb {E}_{z \sim p(\cdot \mid \theta )} \left [ \nabla \log p(z \mid \theta )\nabla \log p(z \mid \theta )^{\top } \right ] \label {eq:fisher_definition_method}
(4)
Equation (3) shows that the Fisher information matrix defines the local metric of the statistical manifold. Unlike diagonal first-order preconditioners in AdamW, F captures structured curvature information and parameter coupling, thereby providing a geometry-aware notion of distance in parameter space. This is especially important in multi-modal training. High-variance visual gradients may correspond to directions that are statistically noisy or locally
6
Yishun Lu and Wes Armour
flat, while lower-variance textual gradients may align with sharper and more informative directions for joint generalization. Preconditioning the gradient by F −1 yields the natural gradient update d_{\mathrm {NGD}} = F^{-1} g \label {eq:natural_gradient_update_method}
(5)
Under this transformation, updates are no longer determined solely by Euclidean scale, but are normalized with respect to the local geometry of the model distribution. Consequently, directions associated with large stochastic variance can be damped when they are not supported by informative curvature, whereas lower-variance but high-curvature directions can be preserved or amplified. This provides the mathematical motivation for introducing Fisher-aware optimization in order to balance multi-modal training dynamics. The details of proof are shown in Supplementary section A. 3.3
Tractable Curvature via Tensor Preconditioners
To make curvature-aware optimization tractable for large multi-modal autoregressive models, we must address the prohibitive computational and memory costs of the exact Fisher Information Matrix F . For a neural network with N parameters, the Fisher matrix F ∈ RN ×N requires O(N 3 ) for explicit inversion, which is fundamentally infeasible for models with billions of parameters. Structured tensor preconditioners, such as Shampoo [20] and SOAP [22], provide a scalable approximation to the geometry induced by F . Consider a parameter tensor whose gradient is represented as a matrix G ∈ Rm×n , and let g = vec(G) be its vectorized form. These methods approximate the local curvature by assuming a Kronecker-factored structure of the second-moment matrix: \hat {F} \approx L \otimes R \label {eq:fisher_kronecker_approx} (6) where the left and right factors L ∈ Rm×m and R ∈ Rn×n capture the correlations along the respective tensor dimensions: L \approx \mathbb {E}[G G^{\top }], \qquad R \approx \mathbb {E}[G^{\top } G] \label {eq:shampoo_factors}
(7)
Crucially, unlike Natural Gradient Descent (NGD) which typically applies the inverse Fisher F −1 , tensor preconditioners like SOAP aim to approximate the inverse square-root of the Fisher matrix, F −1/2 , to achieve adaptive coordinatewise scaling in the spectral domain. Under the property (L ⊗ R)−1/2 = L−1/2 ⊗ R−1/2 , the preconditioned update can be computed without materializing the full N × N matrix. In practice, to maintain symmetric preconditioning for the matrix G, the update is applied as: \mathcal {P}(g) = \mathrm {vec}\left ( L^{-1/4} G R^{-1/4} \right ) \label {eq:inverse_fisher_application}
(8)
where L−1/4 and R−1/4 are computed via eigendecomposition (as in SOAP) or iterative methods (as in Shampoo). Equation (8) is mathematically equivalent
Abbreviated paper title
7
to applying (R1/4 ⊗ L1/4 )−1 g, providing a memory-efficient surrogate for the geometry-aware update. Despite this efficiency, applying such preconditioning directly to the averaged batch gradient ḡ still suffers from severe modality competition. The fundamental limitation lies in the sequential nature of the standard update: \Delta \theta _{\mathrm {SOAP}} \propto \mathcal {P}(\bar {g}), \quad \text {where} \quad \bar {g} = \frac {1}{K}\sum _{i=1}^{K} g_i \label {eq:soap_update_avg}
(9)
During the averaging step, fine-grained gradient discrepancies between microbatches are smoothed out. In multi-modal training, where image and text samples produce gradients with vastly different noise scales and directions, the averaged vector ḡ becomes biased towards the modality with higher variance (typically vision). Consequently, the preconditioner P(·) only adjusts the coarse, batch-level geometry of an already biased signal; it cannot recover the modalityspecific informative directions that were neutralized during the initial averaging process. This motivates the need for variance-aware correction before the preconditioning stage. 3.4
Variance Control via Fisher-Orthogonal Projection
To recover the critical intra-batch variance information that is prematurely lost when simply averaging gradients, we introduce Fisher-Orthogonal Projection (FOP) on gradient differences. During gradient accumulation, let g1 and g2 denote two stochastic gradients computed from distinct micro-batches. We define their average and difference as: g_{\mathrm {avg}} = \frac {1}{2}(g_1 + g_2), \qquad g_{\mathrm {diff}} = g_1 - g_2 \label {eq:avg_and_diff}
(10)
Here, gdiff inherently captures the micro-level variance and heterogeneity within the accumulation window. In the context of multi-modal alignment, it explicitly encodes the optimization conflict between processing an image-centric microbatch and a text-centric micro-batch. Instead of explicitly forming the prohibitive Fisher matrix F to perform geometric projections, we construct a lightweight forward metric proxy M(v) ≈ F v directly from the internal states of the tensor preconditioners (e.g., Shampoo or SOAP). Using this computationally efficient proxy, we extract a Fisher⊥ orthogonal residual gdiff by projecting gdiff away from the mean descent direction gavg in the Riemannian manifold: g_{\mathrm {diff}}^{\perp } = g_{\mathrm {diff}} - \left ( \frac {\langle g_{\mathrm {diff}}, \mathcal {M}(g_{\mathrm {avg}}) \rangle }{\langle g_{\mathrm {avg}}, \mathcal {M}(g_{\mathrm {avg}}) \rangle + \epsilon } \right ) g_{\mathrm {avg}} \label {eq:fisher_orthogonal_residual_main}
(11)
where ϵ is a small numerical stabilizer. By construction, this residual satisfies ⊥ ⊥ ⟨gdiff , M(gavg )⟩ ≈ ⟨gdiff , F gavg ⟩ = 0. It perfectly isolates the pure cross-modality variance without interfering with the primary optimization trajectory.
8
Yishun Lu and Wes Armour
The final combined gradient direction is then constructed by augmenting the average gradient with this variance-aware correction: g_{\mathrm {comb}} = g_{\mathrm {avg}} + \beta g_{\mathrm {diff}}^{\perp } \label {eq:combined_direction}
(12)
where β is an adaptive mixing coefficient. This combined direction is subsequently mapped back to the parameter space using the preconditioner P(·) to obtain the final update step: d_{\mathrm {FOP}} = F^{-1} g_{\mathrm {comb}} \label {eq:fop_update}
(13)
This formulation elegantly injects variance control into the preconditioned subspace without breaking the tractable O(m3 + n3 ) complexity bound. The detailed step-by-step mathematical derivation, including the computation of the optimal adaptive coefficient β, is provided in Supplementary material Section B. Standard SOAP or Shampoo updates are inherently restricted to the onedimensional subspace defined by the batch mean, yielding an update dSOAP ≈ F −1 gavg . By contrast, FOP expands the search space by injecting the Fisher⊥ orthogonal variance residual gdiff . As we show in Supplementary material Section C, under a local quadratic surrogate objective J(d) governing the natural gradient descent, the FOP update satisfies an inequality: J_{\mathrm {FOP}}(\beta ) < J_{\mathrm {SOAP}}, \quad \forall \beta \neq 0 \label {eq:fop_superiority} 3.5
(14)
Multi-Scale Variance Correction via MLMC-Inspired Hierarchical Folding
To scale multi-modal autoregressive models to ultra-large batch sizes (e.g., 8192), training must rely on extensive gradient accumulation across K micro-steps. Applying standard FOP only once, using coarse summaries from the start and end of a long accumulation window, would fail to explicitly preserve intermediate fine-scale gradient discrepancies arising within the window. To address this, we borrow the telescoping coarse-to-fine viewpoint of the Multi-Level Monte Carlo (MLMC) paradigm [5]. In classical computational statistics, MLMC achieves optimal variance reduction by telescoping an expectation across a hierarchy of discretization resolutions, effectively combining coarse global estimates with fine local corrections. Unlike classical MLMC, our objective here is not to estimate a scalar expectation via unbiased multi-level sampling, but rather to preserve the rich, withinstep variance structure of gradients during massive accumulation. We propose a Multi-level Hierarchical Gradient Folding mechanism. We discretize the accumulation window into a dyadic hierarchy of L levels, where the snapshot intervals Lj ∈ {1, 2, 4, 8, . . . , K}. Instead of discarding intermediate gradients, we record computationally lightweight cumulative average snapshots: \bar {g}_{1:k} = \frac {1}{k}\sum _{i=1}^k g_i \label {eq:cumulative_snapshots}
(15)
Abbreviated paper title
9
By simple algebra on cumulative averages, we reconstruct the missing segment averages (the segment-level averages), bridging adjacent dyadic levels without explicitly storing every micro-batch: \bar {g}_{L_{j-1}+1 : L_j} = \frac {L_j \bar {g}_{1:L_j} - L_{j-1} \bar {g}_{1:L_{j-1}}}{L_j - L_{j-1}} \label {eq:segment_average}
(16)
By treating these recovered segment averages as multi-scale summaries whose discrepancies encode variance across the accumulation hierarchy, we perform a recursive, bottom-up folding process using our FOP operator: z_0 = \bar {g}_{1:1}, \qquad z_j = \mathrm {FOP}_{\mathrm {accum}}(z_{j-1}, \bar {g}_{L_{j-1}+1:L_j}) \label {eq:mlmc_fop_recursion}
(17)
where FOPaccum (·, ·) combines the previous folded state with the recovered segment average through Fisher-aware residual filtering.
4
Experimental Setup
4.1
Model and dataset
To systematically evaluate the efficacy of our hierarchical folding optimization, we conduct experiments on two representative unified multi-modal autoregressive architectures: Janus [25] and Emu3 [23]. While both models utilize an autoregressive framework for unified multi-modal alignment, they adopt fundamentally distinct visual encoding paradigms. Emu3 relies on a purely discrete tokenizer, fully converting continuous visual signals into a discrete space to share Fa single, unified vocabulary with text. In contrast, Janus employs a decoupled visual encoding strategy: it extracts continuous semantic features (via SigLIP [27]) for multi-modal understanding, while utilizing a VQ tokenizer [16] to map images into discrete IDs specifically for visual generation. To accommodate their respective architectural designs and spatial resolution requirements, we configure the maximum training sequence length to 1024 for the Janus variants and 5120 for the Emu3 variant. Furthermore, rather than directly adopting the massive parameter scales from their original papers (which typically range from 7B to over 8B parameters ), we construct three scaled-down, intermediate-capacity variants: Janus-400M, and Emu3-600M. The primary motivation for this downscaling is computational tractability and ablation rigor. Investigating the intricate optimization dynamics of modality competition under ultra-large batch sizes (e.g., 8192) requires extensive hyperparameter tuning and hundreds of gradient accumulation steps. Evaluating fullscale 8B models under these extreme accumulation settings is computationally prohibitive. By systematically scaling down the network capacity while maintaining the core architectural proportions, these intermediate variants perfectly preserve the severe cross-modality gradient heterogeneity observed in full-scale models. This allows us to rigorously and efficiently ablate our multi-scale variance control mechanisms. The exact scaling configurations, layer depths, and
10
Yishun Lu and Wes Armour
parameter counts for each variant are detailed in Supplementary material section D. To thoroughly evaluate the robustness and efficacy of our optimization methods under realistic, large-scale conditions, we construct a massive multi-modal pre-training corpus by combining the widely adopted LLaVA-3M and LLaVA12M datasets [10–12]. By jointly training our scaled-down autoregressive models on this combined dataset, we expose the optimization trajectory to highly diverse and extensive image-text pairs. This comprehensive data scale is specifically designed to induce severe cross-modality gradient heterogeneity, providing a rigorous testbed for validating our multi-scale variance control under massive gradient accumulation settings. 4.2
Implementation Details
To ensure balanced multi-modal alignment, both architectures are jointly trained on Image-to-Text (I2T) and Text-to-Image (T2I) tasks with a 50%/50% probability split. However, the specific mixing strategies are tailored to their respective architectural paradigms. For Janus, we utilize step-level synchronous task routing across distributed ranks. In each training step, a specific task (understanding or generation) is randomly sampled and synchronized across all ranks to maintain consistent gradient directions for the decoupled encoders. In contrast, Emu3 leverages its unified next-token prediction framework to implement sample-level mixing within each batch. To further enhance bidirectional alignment and prevent positional bias, we randomly permute the relative order of image tokens and text captions within each micro-batch. All experiments are executed on a high-performance cluster equipped with AMD MI300X GPUs, each providing 192GB of VRAM. To handle the memory demands of large-scale pre-training, especially for Emu3’s extended sequence length of 5120 tokens, we employ gradient checkpointing across all model variants. The micro-batch size per GPU is configured as 64 for Janus-400M and 8 for Emu3-600M. We utilize gradient accumulation to reach the target global batch size, and to maintain stable optimization dynamics, we p strictly adhere to the square-root learning rate scaling rule (ηscaled = ηbase × Btotal /1024). For hyperparameter tuning, we conduct a systematic grid search to identify the optimal peak learning rate for each optimizer and model variant. Regarding the learning rate schedules, we use a 10% warmup ratio by default. We uses cosine decay with a minimum LR floor of 1e-6. Crucially, Janus defaults to freezing all vision-related modules, tightly focusing the optimization on the autoregressive backbone and naturally accounting for the gap between its trainable and total parameter counts.
5
Experiments
The overall training efficiency and convergence behaviors of the proposed optimization strategies are first evaluated under a baseline batch size of 1024. In
Abbreviated paper title Janus-400M Shampoo
11
9
8
8
7
7
6
6
5
5
4
4
3
3 0
1
2
3 Train Tokens (MB) Janus-400M Shampoo
11
4
5
6 1e3
0
9
3 Train Tokens (MB) Janus-400M SOAP
4
5
7
Loss
8
6 1e3
AdamW SOAP FOP-SOAP ML-FOP-SOAP MUON
9
7
6
5
5
4
4
3
2
10
8 6
1
11
AdamW SHAMPOO FOP-SHAMPOO ML-FOP-SHAMPOO MUON
10
Loss
AdamW SOAP FOP-SOAP ML-FOP-SOAP MUON
10
Loss
Loss
9
Janus-400M SOAP
11
AdamW SHAMPOO FOP-SHAMPOO ML-FOP-SHAMPOO MUON
10
11
3 0
5
10
15 20 25 Wallclock Time (s)
30
35
40 1e3
0
5
10
15 20 25 Wallclock Time (s)
30
35
40 1e3
Fig. 1: 2x2 train-loss comparison for pretraining Janus-400M. Left column: SHAMPOO family; right column: SOAP family. Top row: loss vs trained tokens; bottom row: loss vs wallclock time.
Figure 1, a comparative analysis of the training loss for the Janus-400M architecture is presented across the SHAMPOO (left column) and SOAP (right column) optimizer families. To rigorously define the optimization plateau, the dashed horizontal lines indicate the convergence thresholds established by extending the AdamW baseline training for 20× the standard token volume according to a “Chinchilla-optimal” scaling law [6]. Under this regime, the SHAMPOO family underperforms the AdamW baseline. Early-stage vision-text gradient outliers heavily skew SHAMPOO’s uncentered second-moment matrices. Conversely, SOAP mitigates this by projecting gradients into the skewed eigenbasis and applying coordinate-wise adaptive normalization, effectively insulating the network from divergence. Consequently, standard SOAP reaches the convergence threshold at merely 3.2B tokens—a ∼ 1.4× sample efficiency improvement over AdamW (4.5B tokens), while MUON entirely fails to approach it. Crucially, applying the Fisher-Orthogonal Projection (FOP) to the SOAP optimizer amplifies stability. By neutralizing variance-induced cross-modality conflicts, FOP-SOAP and ML-FOP-SOAP break through vanilla SOAP’s plateau to achieve lower global losses. Furthermore, the Multi-Level (ML) mechanism successfully amortizes FOP’s projection latency. As a result, ML-FOP-SOAP hits the target threshold in around 3.1B tokens (a ∼ 1.4× efficiency gain over AdamW) and requires around 5.5 hours of wall-clock time. This yields an endto-end actual training acceleration of ∼ 1.5× compared to AdamW’s 8.3 hours.
12
Yishun Lu and Wes Armour
Table 1: Cross-architecture throughput and memory comparison between Janus-400M and Emu3-600M under global batch sizes (BS) of 1024 and 8192. For each architecture, the results are split into two sub-rows: the first row reports the average time per global training step (seconds), and the second row reports the peak VRAM consumption (GB). Batch Size
BS = 1024
BS = 8192
Arch
Metric AdamW MUON Sham. F-Sh. ML-F-Sh. SOAP F-So. ML-F-So. AdamW MUON Sham. F-Sh. ML-F-Sh. SOAP
Janus
Time 7.76 7.85 12.93 15.64 VRAM 121.86 121.48 128.49 126.92
13.87 126.93
8.40 12.22 129.39 127.28
8.62 131.65
60.94 60.90 59.91 74.48 121.86 121.48 128.49 126.92
114.32 126.93
65.53 78.93 129.39 127.28
Emu3
Time 236.91 252.51 N/A VRAM 144.38 139.91 N/A
N/A N/A
250.63 273.56 145.61 146.77
259.67 151.08
1859.13 1949.13 N/A 144.38 139.91 N/A
N/A N/A
1945.83 2135.59 1971.47 145.61 146.77 151.08
N/A N/A
N/A N/A
F-So. ML-F-So. 67.43 131.65
Janus-400M I2T Loss
4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5
AdamW SOAP ML-FOP-SOAP MUON
Loss
Loss
These findings substantiate that robust second-order preconditioners establish a critical multi-modal foundation, while FOP-based variance control unlocks deeper convergence. Given SHAMPOO’s vulnerability to early-stage outliers, subsequent analyses will focus exclusively on the SOAP family.
Train Tokens (MB) Janus-400M T2I Loss
8.0
6.5
8 7
5.5
6 0.0
0.5
1.0
1.5 2.0 Train Tokens (MB)
2.5
3.0
3.5 1e3
AdamW SOAP ML-FOP-SOAP MUON
9
6.0
5.0
Train Tokens (MB) Emu3-600M T2I Loss
10
Loss
Loss
7.0
AdamW SOAP ML-FOP-SOAP MUON
11
AdamW SOAP ML-FOP-SOAP MUON
7.5
Emu3-600M I2T Loss
4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5
5
0
1
2
3 4 5 Train Tokens (MB)
6
7
8 1e3
Fig. 2: Decoupled training loss curves for image-to-text (I2T) and text-to-image (T2I) tasks across Janus-400M (left) and Emu3-600M (right) architectures.
To explicitly verify how the proposed strategies resolve modality competition, the overall objective is decoupled into Image-to-Text (I2T) and Text-toImage (T2I) losses in Figure 2. In the Janus-400M architecture (left column), the global optimization trajectory is heavily dominated by the high-entropy T2I generation task, which consistently exhibits a much larger numerical magnitude than the I2T understanding task. Under this severe gradient heterogeneity, first-order methods like AdamW are overwhelmed by visual noise, resulting in sub-optimal convergence. The second-order SOAP optimizer establishes
Abbreviated paper title Janus-400M SOAP
10
8
8 Loss
Loss
AdamW SOAP FOP-SOAP ML-FOP-SOAP MUON
9
7 6 5
7 6
4
5
3 0
1
2
10
3 Train Tokens (MB) Janus-400M SOAP
4
5
6 1e3
0
8
2
4 6 Train Tokens (MB) Emu3-600M SOAP
10
AdamW SOAP FOP-SOAP ML-FOP-SOAP MUON
9
8
10 1e3 AdamW SOAP FOP-SOAP ML-FOP-SOAP MUON
9 8 Loss
7 Loss
Emu3-600M SOAP
10
AdamW SOAP FOP-SOAP ML-FOP-SOAP MUON
9
13
6 5
7 6
4
5
3 0
5
10
15 20 25 Wallclock Time (s)
30
35
40 1e3
0
50
100
150 200 Wallclock Time (s)
250
300
350 1e3
Fig. 3: Training loss convergence across Janus-400M (left) and Emu3-600M (right) under a highly scaled batch size setting (Batch size = 8192). The top row tracks loss against processed Train Tokens, while the bottom row tracks it against Wallclock Time.
a stronger foundation by accelerating the dominant T2I task significantly faster than AdamW. However, while SOAP improves T2I generation, its corresponding I2T loss rapidly plateaus, indicating that standard second-order preconditioning still inadvertently sacrifices textual understanding to accommodate visual noise. In stark contrast, ML-FOP-SOAP successfully neutralizes this conflict; by leveraging geometry-aware variance control, it shields the fragile I2T learning process and continuously drives both losses downward simultaneously. Furthermore, this optimization superiority is even more pronounced in the Emu3-600M architecture (right column), which operates under a pure unified next-token prediction paradigm. In this setting, the limitations of standard optimizers are drastically exposed: AdamW struggles profoundly, particularly on the I2T task, converging to a remarkably higher loss plateau than all other methods. While standard SOAP marginally improves upon AdamW, ML-FOP-SOAP achieves a strict Pareto improvement, yielding significantly deeper and faster convergence on both the I2T and T2I objectives simultaneously. These decoupled results directly substantiate the two central claims of this work: while secondorder preconditioners effectively manage baseline multi-modal heterogeneity better than first-order methods, the addition of the FOP mechanism is absolutely critical for controlling variance and fully resolving modality competition across diverse architectures. To demonstrate scalability, training dynamics are evaluated at a global batch size of 8192 (Figure 3). In Janus-400M and Emu3-600M, AdamW suffers from se-
14
Yishun Lu and Wes Armour
vere large-batch degradation, prematurely plateauing above baseline thresholds. While standard SOAP initially outperforms AdamW, it eventually stagnates. Only ML-FOP-SOAP maintains a continuous downward trajectory, achieving the lowest global loss. Wallclock evaluations confirm that our Multi-Level (ML) hierarchical folding strategy mitigates the computational overhead of micro-step projections, enabling robust, efficient scalability to ultra-large batches without catastrophic latency. Additional examples are provided in the supplementary materials.
6
Conclusion
In this work, we systematically addressed the optimization bottleneck of modality competition in natively unified multi-modal autoregressive models. We demonstrated that while the second-order SOAP optimizer establishes a robust foundation against early-stage outliers and multi-modal heterogeneity, it still falls prey to the conventional trade-off between textual understanding and visual generation. To resolve this, we introduced ML-FOP-SOAP. By flexibly applying a Fisher-Orthogonal Projection, our method actively neutralizes cross-modality variance, yielding strict Pareto improvements that drive both task objectives downward simultaneously. Furthermore, our Multi-Level hierarchical folding strategy successfully amortizes projection latency during massive gradient accumulation. This enables flawless scaling to an ultra-large batch size of 8192, breaking the late-stage convergence plateaus that plague standard optimizers. Delivering a 1.4× improvement in sample efficiency and 1.5× faster wall-clock convergence over AdamW, our framework provides a robust and scalable foundation for accelerating large-scale multi-modal pretraining.
References 1. Abreu, N., Vyas, N., Kakade, S., Morwani, D.: The potential of second-order optimization for llms: A study with full gauss-newton. arXiv preprint arXiv:2510.09378 (2025) 2. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond (2023), https://arxiv.org/abs/2308.12966 3. Chen, Z., Badrinarayanan, V., Lee, C.Y., Rabinovich, A.: Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In: International conference on machine learning. pp. 794–803. PMLR (2018) 4. Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, 49250–49267 (2023) 5. Giles, M.B.: Multilevel monte carlo methods. Acta numerica 24, 259–328 (2015) 6. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L.A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J.W., Vinyals, O., Sifre, L.: Training compute-optimal large language models (2022), https://arxiv.org/abs/2203.15556
Abbreviated paper title
15
7. Kontras, K., Chatzichristos, C., Blaschko, M., De Vos, M.: Improving multimodal learning with multi-loss gradient modulation. arXiv preprint arXiv:2405.07930 (2024) 8. Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems 35, 17612–17625 (2022) 9. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 10. Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., Zhang, L., Gao, J., Li, C.: Llava-plus: Learning to use tools for creating multimodal agents (2023), https://arxiv.org/abs/2311.05437 11. lmms-lab: Llava-recap-cc12m (2024), https://huggingface.co/datasets/lmmslab/LLaVA-ReCap-CC12M, hugging Face dataset, accessed 2026-03-05 12. lmms-lab: Llava-recap-cc3m (2024), https://huggingface.co/datasets/lmmslab/LLaVA-ReCap-CC3M, hugging Face dataset, accessed 2026-03-05 13. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 14. Lu, Y., Armour, W.: Beyond the mean: Fisher-orthogonal projection for natural gradient descent in large batch training. arXiv preprint arXiv:2508.13898 (2025) 15. Martens, J., Grosse, R.: Optimizing neural networks with kronecker-factored approximate curvature. In: International conference on machine learning. pp. 2408– 2417. PMLR (2015) 16. van den Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning (2018), https://arxiv.org/abs/1711.00937 17. Pascanu, R., Bengio, Y.: Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584 (2013) 18. Peng, X., Wei, Y., Deng, A., Wang, D., Hu, D.: Balanced multimodal learning via on-the-fly gradient modulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8238–8247 (2022) 19. Qian, C., Han, K., Liu, J., Yuan, Z., Zhu, Z., Wang, J., Lyu, C., Chen, J., Liu, Z.: Dyncim: Dynamic curriculum for imbalanced multimodal learning. arXiv preprint arXiv:2503.06456 (2025) 20. Shi, H.J.M., Lee, T.H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., Rabbat, M.: A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497 (2023) 21. Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024) 22. Vyas, N., Morwani, D., Zhao, R., Kwun, M., Shapira, I., Brandfonbrener, D., Janson, L., Kakade, S.: Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321 (2024) 23. Wang, X., Cui, Y., Wang, J., Zhang, F., Wang, Y., Zhang, X., Luo, Z., Sun, Q., Li, Z., Wang, Y., et al.: Multimodal learning with next-token prediction for large multimodal models. Nature pp. 1–7 (2026) 24. Wilson, A.C., Roelofs, R., Stern, M., Srebro, N., Recht, B.: The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems 30 (2017) 25. Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al.: Janus: Decoupling visual encoding for unified multimodal understanding and generation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 12966–12977 (2025)
16
Yishun Lu and Wes Armour
26. Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., Finn, C.: Gradient surgery for multi-task learning. Advances in neural information processing systems 33, 5824–5836 (2020) 27. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training (2023), https://arxiv.org/abs/2303.15343
Abbreviated paper title
17
Supplementary Material for Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models A
Theoretical Analysis of Fisher Information for Modality Balancing
In this appendix, we provide a mathematical analysis of why first-order optimization can lead to modality starvation when jointly training visual and textual modalities with heterogeneous statistical structure, and how the Fisher information matrix introduced in natural gradient descent (NGD) helps mitigate this issue through a metric transformation on the underlying Riemannian manifold. A.1
First-Order Dynamics of Modality Competition
Consider the parameters of a unified autoregressive multimodal model, denoted by θ ∈ Rd . In multimodal next-token prediction, the training objective is the mixed expectation of the negative log-likelihoods over image data (I) and text data (T ): \mathcal {L}(\theta ) = \mathbb {E}_{x \sim \mathcal {D}_I}\bigl [-\log p(x \mid \theta )\bigr ] + \mathbb {E}_{y \sim \mathcal {D}_T}\bigl [-\log p(y \mid \theta )\bigr ] \triangleq \mathbb {E}[L_I(\theta )] + \mathbb {E}[L_T(\theta )]. \label {eq:joint_objective} (18) In practical training, we compute the stochastic gradient using minibatches: g = \nabla L_I(\theta ) + \nabla L_T(\theta ). \label {eq:joint_gradient}
(19)
Because visual tokens often carry substantially higher information density and more fine-grained local variation, whereas textual tokens lie on a more abstract semantic manifold, the corresponding empirical gradient covariance matrices are \Sigma _I = \mathrm {Cov}(\nabla L_I(\theta )), \qquad \Sigma _T = \mathrm {Cov}(\nabla L_T(\theta )). \label {eq:modality_covariances} (20) In many settings, one empirically observes \mathrm {Tr}(\Sigma _I) \gg \mathrm {Tr}(\Sigma _T). \label {eq:cov_trace_imbalance}
(21)
When using first-order optimizers such as AdamW, the parameter update ∆θ1st is driven by exponential moving averages of the stochastic gradients. In Euclidean space, the covariance of the total stochastic gradient is \mathrm {Cov}(g) = \Sigma _I + \Sigma _T + 2\mathrm {Cov}(\nabla L_I(\theta ), \nabla L_T(\theta )). \label {eq:joint_gradient_covariance}
(22)
The cross-covariance term 2Cov(∇LI (θ), ∇LT (θ)) explicitly captures the correlation and potential interference between the modalities. If ΣI dominates the variance scale, and the cross-covariance indicates misalignment (i.e., gradient conflict), the parameter trajectory can become disproportionately influenced by
18
Yishun Lu and Wes Armour
the high-variance directions induced by the visual modality. Meanwhile, the effective update signal from the text modality may be heavily suppressed or distorted by the cross-modality noise. This severe imbalance, exacerbated by standard Euclidean optimization, leads to slower progress or even stagnation for one modality, which we refer to as modality starvation.
A.2
KL Divergence and the Fisher Information Matrix
To mitigate this variance-dominated behavior under the Euclidean metric, one may instead define optimization steps on the Riemannian manifold induced by the model distribution. Let the change in model predictions induced by a parameter perturbation ∆θ be measured by the Kullback–Leibler (KL) divergence D_{\mathrm {KL}}(p_\theta \,\|\, p_{\theta +\Delta \theta }). \label {eq:kl_definition_symbolic}
(23)
Applying a second-order Taylor expansion of the log-likelihood around θ, we obtain \log p(z \mid \theta +\Delta \theta ) \approx \log p(z \mid \theta ) + \nabla \log p(z \mid \theta )^\top \Delta \theta + \frac {1}{2}\Delta \theta ^\top \nabla ^2 \log p(z \mid \theta )\Delta \theta . \label {eq:taylor_loglikelihood} (24) Substituting this expansion into the definition of the KL divergence yields D_{\mathrm {KL}}(p_\theta \,\|\, p_{\theta +\Delta \theta }) = \mathbb {E}_{p_\theta } \left [ \log p(z \mid \theta ) - \log p(z \mid \theta +\Delta \theta ) \right ], \label {eq:kl_definition}
(25)
and therefore D_{\mathrm {KL}}(p_\theta \,\|\, p_{\theta +\Delta \theta }) \approx -\mathbb {E}_{p_\theta }\!\left [\nabla \log p(z \mid \theta )^\top \Delta \theta \right ] -\frac {1}{2} \mathbb {E}_{p_\theta }\!\left [\Delta \theta ^\top \nabla ^2 \log p(z \mid \theta )\Delta \theta \right ]. \label {eq:kl_taylor_expansion} (26) Since the score function has zero expectation, \mathbb {E}_{p_\theta }\bigl [\nabla \log p(z \mid \theta )\bigr ] = 0, \label {eq:score_zero_mean}
(27)
the first term vanishes. The remaining second-order term defines the Fisher information matrix: F = -\mathbb {E}_{p_\theta }\!\left [\nabla ^2 \log p(z \mid \theta )\right ]. \label {eq:fisher_hessian_form}
(28)
Hence, the KL divergence admits the following local quadratic approximation: D_{\mathrm {KL}}(p_\theta \,\|\, p_{\theta +\Delta \theta }) \approx \frac {1}{2}\Delta \theta ^\top F \Delta \theta = \frac {1}{2}\|F^{1/2}\Delta \theta \|_2^2. \label {eq:kl_local_quadratic}
(29)
This shows that, locally, the Fisher information matrix defines the intrinsic metric of the statistical manifold.
Abbreviated paper title
A.3
19
Why Can F −1 Mitigate Modality Competition?
The key observation follows from the information matrix equality (also known as Bartlett’s identity). For log-likelihood objectives, F = -\mathbb {E}_{p_\theta }\!\left [\nabla ^2 \log p(z \mid \theta )\right ] = \mathbb {E}_{p_\theta }\!\left [ \nabla \log p(z \mid \theta )\nabla \log p(z \mid \theta )^\top \right ]. \label {eq:bartlett_identity}
(30)
Thus, the Fisher matrix simultaneously encodes both the local curvature of the loss landscape and the uncentered second moment of the score function. Under natural gradient descent, the update direction becomes d_{\mathrm {NGD}} = F^{-1} g = F^{-1}(\nabla L_I + \nabla L_T). \label {eq:ngd_update}
(31)
To understand how this preconditioned update addresses modality imbalance, we analyze the magnitude of the update in Fisher geometry. Rather than relying on the centered covariance matrix (which strictly equals the Fisher matrix only when the expected gradient is zero at a local optimum), we examine the expected squared Fisher norm of the natural gradient directly: \mathbb {E}\bigl [\|d_{\mathrm {NGD}}\|_F^2\bigr ] = \mathbb {E}\bigl [d_{\mathrm {NGD}}^\top F d_{\mathrm {NGD}}\bigr ] = \mathbb {E}\bigl [g^\top F^{-1} F F^{-1} g\bigr ] = \mathbb {E}\bigl [g^\top F^{-1} g\bigr ]. \label {eq:fisher_norm_ngd}
(32)
Using the cyclic property of the trace, this can be rewritten in terms of the uncentered second moment matrix E[gg ⊤ ]: \mathbb {E}\bigl [g^\top F^{-1} g\bigr ] = \mathrm {Tr}\!\left (F^{-1}\mathbb {E}[gg^\top ]\right ). \label {eq:trace_identity}
(33)
In practice, the empirical Fisher information matrix is approximated precisely by this uncentered second moment of the mini-batch gradients, i.e., F ≈ E[gg ⊤ ]. Under this standard approximation, we obtain: \mathbb {E}\bigl [\|d_{\mathrm {NGD}}\|_F^2\bigr ] \approx \mathrm {Tr}(F^{-1}F) = \mathrm {Tr}(I) = d, \label {eq:fisher_norm_dimension}
(34)
where d is the dimension of the parameter space. The derivation above demonstrates a powerful property: after preconditioning by F −1 , the overall update magnitude is effectively whitened and bounded by the dimension d under the intrinsic geometry of the model. Regardless of how severe the variance imbalance (ΣI ≫ ΣT ) or the cross-modality interference (2Cov(∇LI , ∇LT )) may be in the raw Euclidean gradient g, the natural gradient normalizes these updates. Directions associated with different modalities are no longer dominated by pure gradient scale; instead, they are structurally scaled by a metric that inherently accounts for both uncertainty and curvature. This provides a rigorous explanation for why Fisher-aware optimization fundamentally alleviates modality starvation.
B
Formulation of Fisher-Orthogonal Projection
In this section, we provide the complete mathematical construction of the FisherOrthogonal Projection (FOP) briefly introduced in Section 3.4.
20
Yishun Lu and Wes Armour
B.1
Fisher-Orthogonal Projection (FOP)
Given two stochastic gradients g1 and g2 within a mini-batch or gradient accumulation window, we define their average and difference as: g_{\mathrm {avg}} = \frac {1}{2}(g_1 + g_2), \qquad g_{\mathrm {diff}} = g_1 - g_2 \label {eq:app_avg_diff_pair}
(35)
To project gdiff onto the subspace orthogonal to gavg under the Fisher metric, we need to compute inner products of the form ⟨u, F v⟩. To avoid the intractable O(N 3 ) cost of explicitly forming F , we utilize the Kronecker factors maintained by the SOAP/Shampoo preconditioners to construct a forward metric proxy M(·): \mathcal {M}(v) \approx Fv \label {eq:app_forward_metric_proxy} (36) Using this efficient proxy, the Fisher-space projection coefficient is calculated as: s_{\mathrm {proj}} = \frac {\langle g_{\mathrm {diff}}, \mathcal {M}(g_{\mathrm {avg}}) \rangle }{\langle g_{\mathrm {avg}}, \mathcal {M}(g_{\mathrm {avg}}) \rangle + \epsilon } \label {eq:app_fisher_projection_coeff}
(37)
where ϵ is a small numerical stabilizer. We then construct the Fisher-orthogonal residual by subtracting the parallel component: g_{\mathrm {diff}}^{\perp } = g_{\mathrm {diff}} - s_{\mathrm {proj}} g_{\mathrm {avg}} \label {eq:app_fisher_orthogonal_residual} (38) By construction, this residual strictly satisfies the Fisher orthogonality condition ⊥ , F gavg ⟩ = 0, ensuring that our variance correction captures pure hetero⟨gdiff geneity without interfering with the primary descent trajectory. Finally, the combined direction gcomb is formed and preconditioned to obtain the actual parameter update dFOP :
C
g_{\mathrm {comb}} = g_{\mathrm {avg}} + \beta g_{\mathrm {diff}}^{\perp } \label {eq:app_combined_direction}
(39)
d_{\mathrm {FOP}} = \mathcal {P}(g_{\mathrm {comb}}) \approx F^{-1} g_{\mathrm {comb}} \label {eq:app_fop_update}
(40)
Quadratic Surrogate Analysis of Fisher-Orthogonal Projection
In multimodal optimization, analyzing a single averaged surrogate objective often masks the underlying modality competition. To rigorously demonstrate why Fisher-Orthogonal Projection (FOP) is necessary, we model the local loss reduction using second-order Taylor approximations for the individual modalities (e.g., Vision and Text), denoted as J1 (d) and J2 (d). Let the stochastic gradients of the two modalities be g1 and g2 . We define the batch average and difference as: g_{\mathrm {avg}} = \frac {1}{2}(g_1 + g_2), \qquad g_{\mathrm {diff}} = g_1 - g_2
(41)
Abbreviated paper title
21
Consequently, individual gradients can be expressed as g1 = gavg + 12 gdiff and g2 = gavg − 12 gdiff . The local quadratic surrogate for modality i under the true Fisher information matrix F is: J_i(d) = -g_i^{\top } d + \frac {1}{2} d^{\top } F d \label {eq:app_individual_surrogate} C.1
(42)
Objective Value for Pure SOAP
A standard preconditioned optimizer like SOAP targets the average gradient, applying the update dSOAP = F −1 gavg . Substituting this into the surrogate for modality 2: J_2(d_{\mathrm {SOAP}}) &= -g_2^{\top } (F^{-1} g_{\mathrm {avg}}) + \frac {1}{2} (F^{-1} g_{\mathrm {avg}})^{\top } F (F^{-1} g_{\mathrm {avg}}) \nonumber \\ &= -\left (g_{\mathrm {avg}} - \frac {1}{2} g_{\mathrm {diff}}\right )^{\top } F^{-1} g_{\mathrm {avg}} + \frac {1}{2} g_{\mathrm {avg}}^{\top } F^{-1} g_{\mathrm {avg}} \nonumber \\ &= -\frac {1}{2} \|g_{\mathrm {avg}}\|^2_{F^{-1}} + \frac {1}{2} g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {avg}} \label {eq:app_j2_soap}
(43) ⊤ This equation perfectly illustrates modality starvation. The term 12 gdiff F −1 gavg acts as a gradient conflict penalty. If this term is positive and large, modality 2 suffers from severe optimization degradation under the shared SOAP update.
C.2
Objective Value for FOP
The FOP update introduces an orthogonal variance correction to break this rigid ⊥ average: dFOP = F −1 (gavg + βgdiff ). Substituting this into the surrogate for the starving modality 2: J_2(d_{\mathrm {FOP}}) &= -g_2^{\top } F^{-1} (g_{\mathrm {avg}} + \beta g_{\mathrm {diff}}^{\perp }) + \frac {1}{2} (g_{\mathrm {avg}} + \beta g_{\mathrm {diff}}^{\perp })^{\top } F^{-1} (g_{\mathrm {avg}} + \beta g_{\mathrm {diff}}^{\perp }) \nonumber \\ &= -\left (g_{\mathrm {avg}} - \frac {1}{2} g_{\mathrm {diff}}\right )^{\top } F^{-1} (g_{\mathrm {avg}} + \beta g_{\mathrm {diff}}^{\perp }) \nonumber \\ & \qquad \qquad + \frac {1}{2} \left ( \|g_{\mathrm {avg}}\|^2_{F^{-1}} + 2\beta g_{\mathrm {avg}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp } + \beta ^2 \|g_{\mathrm {diff}}^{\perp }\|^2_{F^{-1}} \right ) \label {eq:app_j2_fop_expand}
(44) Expanding the linear term yields: -\|g_{\mathrm {avg}}\|^2_{F^{-1}} - \beta g_{\mathrm {avg}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp } + \frac {1}{2} g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {avg}} + \frac {1}{2} \beta g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp } ⊤ ⊥ Notice the crucial algebraic cancellation of the cross-terms: the −βgavg F −1 gdiff ⊤ −1 ⊥ from the linear part perfectly cancels the +βgavg F gdiff from the quadratic part. Recombining the remaining terms gives:
J_2(d_{\mathrm {FOP}}) &= \left (-\frac {1}{2} \|g_{\mathrm {avg}}\|^2_{F^{-1}} + \frac {1}{2} g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {avg}}\right ) + \frac {1}{2} \beta g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp } + \frac {1}{2} \beta ^2 \|g_{\mathrm {diff}}^{\perp }\|^2_{F^{-1}} \nonumber \\ &= J_2(d_{\mathrm {SOAP}}) + \frac {1}{2} \beta g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp } + \frac {1}{2} \beta ^2 \|g_{\mathrm {diff}}^{\perp }\|^2_{F^{-1}} \label {eq:app_j2_fop_final}
(45)
22
Yishun Lu and Wes Armour
C.3
Strict Loss Reduction for the Starving Modality
To find the optimal adjustment that assists the starving modality, we minimize J2 (dFOP ) with respect to β. Taking the derivative and setting it to zero: \frac {\partial J_2(d_{\mathrm {FOP}})}{\partial \beta } = \frac {1}{2} g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp } + \beta \|g_{\mathrm {diff}}^{\perp }\|^2_{F^{-1}} = 0 \label {eq:app_beta_derivative}
(46)
Solving for the optimal coefficient β ∗ :
\beta ^* = - \frac {1}{2} \frac {g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp }}{\|g_{\mathrm {diff}}^{\perp }\|^2_{F^{-1}}} \label {eq:app_optimal_beta}
(47)
Substituting β ∗ back into Equation (45) yields the final loss reduction:
J_2(d_{\mathrm {FOP}}(\beta ^*)) &= J_2(d_{\mathrm {SOAP}}) - \frac {1}{8} \frac {\left (g_{\mathrm {diff}}^{\top } F^{-1} g_{\mathrm {diff}}^{\perp }\right )^2}{\|g_{\mathrm {diff}}^{\perp }\|^2_{F^{-1}}} \label {eq:app_strict_loss_reduction}
(48)
⊥ Since the Fisher metric F −1 is positive definite, the norm ∥gdiff ∥2F −1 is strictly positive. As long as the projected variance difference correlates with the true gradient conflict, the subtracted term is strictly positive, yielding:
J_2(d_{\mathrm {FOP}}(\beta ^*)) < J_2(d_{\mathrm {SOAP}})
C.4
(49)
Physical Interpretation
This rigorous derivation completely reframes the utility of FOP. Mathematically, a pure SOAP step based on gavg is the strict global minimum of the average quadratic surrogate. However, optimizing only for the average inherently punishes individual modalities with conflicting curvatures. Equation (48) algebraically proves that by intelligently injecting Fisher-orthogonal variance, FOP escapes the rigid minimum of the average surrogate to secure a strictly guaranteed loss reduction for the disadvantaged (starving) modality. FOP achieves true modality balancing not by contradicting the Newton direction, but by dynamically correcting the modality-specific penalty induced by gradient averaging.
D
Detailed Model Configurations and Scaling
In Section 4, we introduced the scaled-down variants of the Janus and Emu3 architectures used in our experiments. Here, we provide the exact architectural hyperparameters and parameter counts to ensure full reproducibility.
Abbreviated paper title
D.1
23
Janus Scaled Variants
For the Janus family, we base our architectural blueprint on the official Janus-Pro-1B skeleton but apply structured scaling rules to the number of transformer layers, hidden dimensions, and attention heads. The multimodal projection layers scale proportionally with the language model’s hidden size. – Janus-400M : Configured with 14 hidden layers, a hidden dimension of 1024, an intermediate FFN size of 2816, and 8 attention heads. This configuration yields approximately 408 million trainable parameters (with a total of 799 million parameters when including frozen vision embeddings). – Janus-1B: Configured with 20 hidden layers, a hidden dimension of 1536, an intermediate FFN size of 4096, and 12 attention heads. This configuration yields approximately 911 million trainable parameters (with a total of 1.30 billion parameters). D.2
Emu3 Scaled Variants
For the Emu3 family, we utilize explicit configuration presets to shrink the model scale while strictly maintaining the core Emu3 architectural properties and vocabulary size (184,622). – Emu3-600M: Configured with 32 hidden layers, a hidden dimension of 1024, an intermediate FFN size of 2816, and 16 attention heads (with 16 KV heads). This configuration yields an active parameter count of approximately 600 million. By utilizing these precise configurations, we maintain the fundamental multimodal optimization challenges of the original architectures while making largebatch gradient accumulation studies computationally feasible. D.3
Hyperparameter Search and Reproducibility
To guarantee the exact reproducibility of our optimization trajectories, all experiments—including baselines and our proposed methods—are conducted using a fixed random seed of 0. Given the sensitivity of second-order and preconditioned optimizers to the learning rate, we refrain from using heuristic defaults. Instead, we perform a rigorous half-decade logarithmic grid search to determine the optimal peak learning rate for each configuration. The search space is defined by the set: \eta \in \{0.1, 0.0316, 0.01, 0.00316, 0.001, 0.000316\} \label {eq:lr_grid}
(50)
which perfectly corresponds to steps of 10−0.5 in the logarithmic scale. For the tensor preconditioners (Shampoo and SOAP) imported from the distributed_shampoo repository, we retain their official default structural hyperparameters (e.g., block sizes and preconditioner update frequencies) to ensure that the baselines operate at their intended optimal algorithmic capacity.