SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment Tianyu Dong1 , Yangyang Liu2 , Jiang Zhou1 , Xinwei Wu1 , Xiaohu Zhao2 , Hao Wang2 , Heng Liu2 , Linlong Xu2 , Longyue Wang2 , Weihua Luo2 , Shaolin Zhu1 * , Deyi Xiong1 1
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China 2 Alibaba Group, China {tydong, zhushaolin, dyxiong}@tju.edu.cn Abstract
arXiv:2606.25821v1 [cs.CL] 24 Jun 2026
Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages—which suffer from a scarcity of highquality training data— often have their tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This crosslingual routing divergence consequently hinders their efficacy in multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages as anchors to low-resource languages. SARA explicitly aligns the routing distribution of multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across languages. We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning (e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU benchmark). Further analyses show that SARA effectively addresses performance bottlenecks in low-resource languages, providing a scalable pathway to enhance multilingual capabilities in sparse architectures. Our code is available at https://github.com/iMoriton/sara.
1
Introduction
The paradigm of LLMs has shifted toward sparse MoE architectures, as evidenced by recent openweight milestones such as Mixtral (Jiang et al., * Corresponding author.
2024), DeepSeek-V3 (Liu et al., 2024), DeepSeekR1 (Guo et al., 2025a) and Qwen3 (Yang et al., 2025). By decoupling model capacity from computational cost, MoE models achieve remarkable scalability and allow distinct subsets of parameters or experts to specialize in specific domains (Dai et al., 2024). However, the efficacy of MoE primarily relies on routers correctly dispatching tokens to the most competent experts. While these models show exceptional capability in their dominant training languages (e.g., English), extending their specialized prowess to a broader spectrum of lowresource languages remains a formidable challenge (Imani et al., 2023; Etxaniz et al., 2024; Zhu et al., 2024b). Empirical analyses of the internal mechanisms of MoE reveal a fundamental bottleneck: crosslingual routing divergence where semantically equivalent inputs in different languages trigger disparate expert activation pathways (Bandarkar et al., 2025). Recent studies have identified the existence of super experts (Su et al., 2025), defined as a sparse subset of parameters responsible for encoding sophisticated domain knowledge such as mathematical reasoning. However, these components are predominantly optimized for the datarich patterns of high-resource languages. When the model processes semantically equivalent inputs in low-resource languages, the routing network often struggles to generalize. Due to surface-level lexical variations, the router fails to dispatch tokens to these high-competence experts and instead directs them toward generalist or irrelevant pathways (Chi et al., 2022). As a result, the model may possess the necessary parametric knowledge, but lack the routing logic required to activate it for low-resource language inputs. Such routing misalignment not only degrades reasoning performance in low-resource languages, but also disrupts the consistency of internal representations across languages. As observed by Bandarkar et al. (2025),
this inconsistency leads to performance limitations where the hidden states of low-resource inputs diverge from their high-resource counterparts due to disparate expert composition. Recent evaluations highlight that existing LLMs still encounter significant performance plateaus across diverse and nuanced multilingual scenarios (Zhang et al., 2026; Chen et al., 2025). Prior efforts to enhance multilingualism in LLMs have largely focused on continual pre-training or instruction tuning (Li et al., 2025c; Zhu et al., 2024a; Li et al., 2025a). While recent efforts have explored expert pruning (Zhang et al., 2025b) or load balancing optimization (Guo et al., 2025b) to improve efficiency, they primarily optimize the routing mechanism for computational throughput rather than cross-lingual semantic consistency. Recent works (Zhou et al., 2025; Dong et al., 2025; Zhu et al., 2025) regulate updates via routing priors or parameter detection, revealing that unconstrained training conversely leads to catastrophic forgetting. Consequently, the core challenge remains unaddressed: how to encourage semantically equivalent inputs to trigger similar expert activation pathways across languages? To bridge this gap, we propose Semantically Anchored Routing Alignment (SARA), a novel and statistics-driven framework designed to align expert activation patterns across languages. Our method leverages the robust routing distributions of high-resource languages as semantic anchors to rectify the routing behaviors observed in low-resource settings. Unlike traditional distillation targeting output logits, SARA explicitly minimizes the distributional discrepancy in the routing probability space. This encourages the routing logic to remain invariant to the input language, effectively transferring the model’s capabilities to low-resource languages. We implement this via a multi-stage pipeline: (1) constructing a semantically aligned parallel instruction corpus; (2) extracting reliable routing priors from dominant languages; (3) finetuning via a Jensen-Shannon (JS) divergence constraint that penalizes routing deviations. Our contributions are threefold: (i) We propose the SARA framework, which moves beyond traditional token-level distillation by treating the routing probability distribution of high-resource languages as semantic anchors. (ii) By applying a symmetric JS divergence constraint on intermediate layers, SARA effectively leverages the model’s existing high-resource knowl-
edge to make low-resource languages attain highresource inference capabilities by rectifying their expert activation pathways. (iii) We conduct a comprehensive evaluation across 2 LLMs and 3 challenging benchmarks (e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5MoE-instruct on Global-MMLU benchmark). Our results outperform standard instruction tuning, offering a scalable pathway for enhancing multilingual capabilities in sparse architectures.
2
Related Work
Multilingual Alignment. Research on crosslingual alignment in LLMs has predominantly focused on instruction fine-tuning and the strategic curation of training data. Recent studies demonstrate that the specific composition of supervised fine-tuning (SFT) data and the integration of parallel instruction corpora (Penedo et al., 2025) are pivotal for enhancing functional consistency across languages. Frameworks such as Lingualift (Zhang et al., 2024) introduce effective two-stage instruction tuning strategies specifically tailored to bolster performance in low-resource tasks. Aya (Üstün et al., 2024) and JetMoE (Shen et al., 2024) further emphasize that distilling low-resource language knowledge from teacher models can bridge the performance gap of student models in lowresource languages. Additionally, adapting models to severely underrepresented languages highlights the critical necessity of large-scale, languagespecific data curation combined with continual pretraining (Pan et al., 2025). While these methods demonstrate substantial improvements, they are fundamentally data-centric and mainly rely on the availability of high-quality parallel instructions and expert-curated multilingual corpora. Efficiency-Oriented MoE Architectures. Recent advances in MoE have focused on improving the quality of specialized parameter subsets through diverse initialization and hybrid designs. SCoMoE (Zeng and Xiong, 2023) optimizes MoE training efficiency by restructuring communication patterns to alleviate expensive all-to-all operations. Jamba (Lieber et al., 2024) and ERNIE 4.5 (BaiduERNIE-Team, 2025) explore architectural fusion and heterogeneous scaling to optimize inference efficiency. Meanwhile, research into dynamic routing has sought to refine expert selection density based on task difficulty (Huang et al., 2024; Yang et al., 2024) and mitigate computational redun-
Stage1: Semantically Anchored Data Construction
EN Query {Q: “What is the capital of France”}
MoE LLM
EN QA Dataset (Q&A)
EN Answer
GPT-5 mini
{A: “Paris”}
Translate (EN→BN/SW/TE)
W1 Forward
Wn
E1 E2 ···
Propagation
W2 ···
MoE LLM
Routing Network
Multilingual QA Dataset (BN/SW/TE Q&A)
Probability
Stage2: Capturing Target Routing Priors EN QA Dataset (Q&A)
{BN Q:"!াে$র রাজধানী কী?", A: "প0ািরস”} {SW Q:"Mji mkuu wa Ufaransa ni upi?", A: "Paris"} {TE Q:"!ా#$% &ాజ()* ఏ,ట.?", A: "/ా&01”}
En
Stage3: Sequence-Level Routing Alignment
Expert ID EN Routing Distributions Target Probability per Expert (eg., W1:0.1, W2:0.3, ···, Wn: 0.2)
Loss Calculation
Cross-Entropy Loss(LCE)
Logits
MoE LLM (Trainable)
Probability
LCE=-∑log 𝑃(𝑦|𝑥) Multilingual QA Dataset
JS Divergence Loss(LJS)
Total Loss L=LCE+𝜆LBLLB +𝜆JSLJS
LJS=1/2[KL(P||M)+KL(Q||M)] Expert ID
Multilingual Routing Distributions
Load Balancing Loss(LLB)
LLB=N ∑fi ·Pi
Figure 1: Illustration of the proposed SARA framework. The framework consists of three stages: (1) Generate semantically aligned parallel data via GPT-5 mini translation; (2) Perform forward propagation on high-resource inputs to extract dense routing probability distributions as target priors; (3) Fine-tune the model with a composite objective. This minimizes the Jensen-Shannon divergence between the multilingual and anchor routing distributions, along with cross-entropy and load balancing losses, to encourage cross-lingual expert consistency.
dancy via expert pruning (Lu et al., 2024). However, these optimizations are primarily designed for general-purpose efficiency and lack specialized mechanisms for multilingual tasks. Routing Distribution Adjustments. A bottleneck in the multilingual capability of MoE is crosslingual routing divergence (Bandarkar et al., 2025). Semantically equivalent inputs in different languages are often routed to distinct experts. This phenomenon undermines representational consistency and generalization across languages. Although existing routing-centric frameworks such as HyperMoE (Zhao et al., 2024) and RoMA (Li et al., 2025b) have explored adjustments to routing distribution, their objectives are largely confined to task-specific knowledge transfer or intra-domain generalization rather than cross-lingual semantic alignment. Consequently, they do not resolve the routing mismatch that occurs when moving from high-resource to low-resource linguistic contexts. Our SARA implements internal routing distillation specifically for multilingual consistency. By utilizing the model’s own routing distributions on high-resource languages, SARA encourages mech-
anistic consistency across the routing distribution.
3
Proposed Framework
To address the bottlenecks arising from routing misalignment in MoE, we propose the SARA framework. The key idea is to treat the expert activation patterns of high-resource languages, as the ground-truth signal for semantic processing. As illustrated in Figure 1, SARA first constructs strictly semantic-aligned parallel data to ensure content consistency. Then we perform offline forward propagation to extract the routing distributions of high-resource languages as priors. Finally, a distribution alignment objective is applied to pull the routing distributions of multilingual inputs toward those of their anchor texts. This encourages semantically equivalent inputs to induce similar expert activation distributions at the routing level, bridging the cross-lingual gap. 3.1 Semantically Anchored Data Construction The goal of SARA is to align the routing logic of low-resource languages with that of high-resource anchors. To establish a reliable semantic foundation for this anchoring process, we construct a par-
allel instruction dataset that captures the model’s intrinsic inference logic in high-resource contexts: • Correctness-based Filtering: We first perform inference on high-resource training sets (e.g., English or Chinese). As detailed in Appendix A.7 (referencing A.6), we require the model to enclose the final answer or option within a \boxed{}. To objectively verify the results, we utilize the regular expression r"\boxed{(.*?)}" to extract all formatted contents from the model’s output. We designate the final element in the resulting list as the predicted answer. Only samples where the model generates a correct answer are retained. These verified samples serve as the standard for extracting internal routing patterns. • Parallel Corpus Synthesis: For each verified sample, we translate the entire interaction (including the prompt and the logical reasoning steps) into target low-resource languages. This ensures that the low-resource samples are semantically identical to the high-resource anchor samples. By maintaining strict content consistency, the discrepancy in routing can be primarily attributed to linguistic variation, providing a clean supervision signal for alignment. 3.2
Capturing Target Routing Priors
We extract routing distributions from high-resource data to obtain the optimal routing strategy for crosslingual alignment. We perform forward propagation on the concatenated sequence of the prompt and its verified response to capture the complete inference process. We adopt the standard sparse MoE architecture. Given a token representation h ∈ Rd (where d is the hidden dimension), a gating network parameterized by Wg ∈ Rd×N (with N denoting the number of experts) computes the routing scores u and the probability distribution P: u = h · Wg ,
(1)
P = Softmax(u).
(2)
MoE models typically use a top-k selection strategy to improve efficiency. Let T denote the set of indices for the top-k probabilities, and Ej represent the j-th expert network. The sparse output G(h) and the layer output y are defined as: ( Pj , if j ∈ T G(h)j = , (3) 0, otherwise
y=
X
G(h)j · Ej (h).
(4)
j∈T
To obtain stable semantic routing priors, we utilize the dense routing probability distributions produced by the gating network, rather than the discrete top-k expert selections. The rationale is that for low-resource languages, the discrete expert selection during inference often diverges from highresource anchors. (i) For each anchor interaction i, let Sanchor = [xi ; yi ] denote the full sequence, where xi represents the input prompt and yi represents the verified response. We perform forward propagation on this concatenated sequence to capture the routing logic of the complete inference process. We record the softmax-normalized routing distribution Rl at each (i) layer l for every token t ∈ Sanchor : (i) (l) Rl (Sanchor,t ) = Softmax hanchor,t · Wg ∈ RN . (5) To obtain a stable routing distribution, we aggregate token-level routing probabilities into a sequence(l,i) level prior P̄anchor using validity masks mt : 1 X (i) mt · Rl (Sanchor,t ). m t t t
(l,i)
P̄anchor = P
(6)
The mask mt filters out [PAD] tokens to prevent non-semantic routing noise from biasing the aggregated distribution. 3.3
Sequence-Level Routing Alignment
In this stage, we fine-tune the model on multilingual inputs to align their expert activation patterns with the extracted high-resource priors. For a multilingual interaction i, we similarly concatenate (i) (i) the translated prompt xlang and response ylang to (i)
(i)
(i)
form Slang = [xlang ; ylang ]. We then compute its (l,i)
sequence-level routing profile Q̄lang : (l,i)
Q̄lang = P
1 X (i) mt · Rl (Slang,t ). m t t t
(7)
Routing Alignment Loss (LJS ). We align (l,i) the multilingual profile Q̄lang with the high(l,i)
resource anchor prior P̄anchor by minimizing the Jensen–Shannon (JS) divergence: LJS =
1 |Ltarget |
Lend X l=Lstart
(l,i) (l,i) JS P̄anchor ∥ Q̄lang , (8)
where |Ltarget | is the number of selected layers, and Lstart and Lend denote the start and end indices of the intermediate layers (e.g., layers 7 to 34 for Qwen3). The JS divergence is defined via the Kullback-Leibler (KL) divergence as: 1 1 JS(P∥Q) = KL(P∥M) + KL(Q∥M), (9) 2 2 with M = 12 (P+Q) being the average distribution. The choice of JS divergence over the standard KL divergence is critical due to the following 2 factors: • Symmetric Semantic Regularization: Unlike KL divergence, JS divergence is symmetric, providing a balanced metric for distributional similarity. Since high-resource anchor inputs and their multilingual counterparts represent different linguistic realizations of the same semantic intent, JS divergence facilitates a more stable alignment toward a shared routing distribution. • Numerical Stability in Sparse Routing: In MoE gating, the softmax function assigns nonzero routing probabilities to all experts, but in practice, the probabilities for unselected experts are often small. KL divergence is hypersensitive to low-probability tails, leading to disproportionately large or vanishing gradients. Task Loss (LCE ). To maintain the model’s fundamental generative and reasoning capabilities, we apply the standard cross-entropy loss for next-token prediction to the model output in the training data: LCE = −
T X
log P (yt |x<t ).
(10)
t=1
Load Balancing Loss (LLB ). We retain the load balancing loss from Switch Transformers (Fedus et al., 2022) to prevent expert collapse and ensure efficient parameter utilization, formulated as LLB = N
N X
fi · P̄i ,
(11)
i=1
where N is the number of experts, fi is the fraction of tokens dispatched to expert i, and P̄i denotes the average routing probability across the batch. The final training process uses a composite objective that balances task performance, expert utilization and cross-lingual routing consistency: Ltotal = LCE + λLB LLB + λJS LJS ,
(12)
where λJS is the hyperparameter controlling the strength of the semantically anchored routing alignment.
4
Experiments
We extensively evaluated the effectiveness of the proposed SARA framework. We also conducted ablation studies and in-depth analysis to verify the specific contribution of routing alignment in lowresource languages. 4.1
Datasets
We constructed semantically anchored instructiontuning datasets for 5 low-resource languages(hi, ne, bn, te, sw) by applying the pipeline described in Section 3.1 to the high-resource splits of MMLUProX (Xuan et al., 2025) and GSM8K (Cobbe et al., 2021; Yu et al.). * We established 2 independent experimental tracks using English and Chinese as high-resource anchors, respectively, to evaluate the robustness of SARA across different semantic pivots. We utilized Qwen3-30B-A3B for initial inference and capture routing distributions from samples verified for correctness. For each anchor, the filtered interactions were translated into 5 target languages using GPT-5 mini, as detailed in Appendix A.1. To ensure a rigorous zero-shot evaluation, we removed any training samples overlapping with the test sets of our evaluation benchmarks. Following this process, the final corpus for each independent anchor track includes 7,000 parallel samples derived from MMLU-ProX and 7,000 samples from GSM8K per target language. We evaluate performance on 3 diverse multilingual benchmarks: Global-MMLU (Singh et al., 2025), BELEBELE (Bandarkar et al., 2024) and MGSM (Shi et al.). 4.2
Baselines
We compared SARA against the following baselines:(1) Vanilla LM: The original instructiontuned sparse MoE model (Qwen3-30B-A3B) without any further cross-lingual alignment. (2) FFT: Standard supervised full fine-tuning on the same translated dataset, but without the routing alignment objective (LJS ). (3) AES (Guo et al., 2025b): *We note that there is ongoing discussion in the community regarding whether hi(Hindi) should be categorized as a lowresource language (Holtermann et al., 2024; Hangya et al., 2022; Singh et al., 2023; Dubossarsky and Dairkee, 2024; Sharma and Bhattacharyya, 2025; Anoop et al., 2021).
It introduces an orthogonality loss to reduce representational overlap among experts and a variance loss to encourage discriminative routing decisions. (4) ShifCon (Zhang et al., 2025a): It enhances multilingual capabilities by aligning the internal representations of non-dominant languages with the dominant language subspace. 4.3
Settings
We implemented SARA using the PyTorch framework and fine-tune Qwen3-30B-A3B for 2 epochs with a global batch size of 256. All models were trained on 16 × NVIDIA H100 80GB GPUs, and each training session took approximately 15 hours. The learning rate was set to 2e-5 with a cosine decay scheduler. Regarding loss coefficients, the load balancing weight λLB follows the model’s default configuration, while the routing alignment weight λJS is fixed at 1.5. We selectively applied LJS to the intermediate layers (l ∈ [7, 34]) to focus the supervision signal on the model’s languageagnostic semantic core (see Appendix A.2 for details). The English-anchored and Chinese-anchored models were trained as independent instances to verify the framework’s robustness across different high-resource origins. A detailed sensitivity analysis of λJS and the rationale for layer selection are provided in Appendix A.3 and A.2. During evaluation, we reported the average performance over 3 independent runs (Top-p = 1, temperature = 0.1). 4.4
Main Results
The performance of SARA on Global-MMLU, BELEBELE and MGSM is summarized in Table 1. Qwen3-30B-A3B shows limited cross-lingual generalization with a pronounced performance drop on low-resource languages such as Swahili (sw) and Telugu (te). FFT substantially improves multilingual performance and serves as a strong baseline. However, FFT does not explicitly address cross-lingual routing divergence. We perform onetailed paired t-tests comparing SARA and FFT; the detailed statistical protocol and results are provided in Appendix A.4. We further compare SARA with AES and ShifCon under 2 anchor settings. AES relies on expert specialization and regularization, but often underperforms FFT. ShifCon applies representation-level alignment and is generally competitive with FFT. However, neither method consistently matches SARA’s performance on lowresource languages. This suggests that indirect regularization or hidden-state alignment is insufficient
to correct cross-lingual expert routing. We also find that results with a Chinese anchor are weaker than those with an English anchor on all methods. This indicates less stable routing behavior when Chinese is used as the semantic pivot. The gap largely reflects the model’s lower baseline performance in Chinese. By directly aligning routing probability distributions between high-resource anchors and their multilingual counterparts, SARA achieves the best average performance under the English anchor. We further validate the effectiveness of SARA on Phi-3.5-MoE-instruct (Abdin et al., 2024) in Appendix A.5, demonstrating that explicit expert routing alignment is an effective mechanism for transferring semantic capabilities across languages. 4.5
Ablation Study
To analyze the effectiveness of SARA, we conducted a comprehensive ablation study on Qwen330B-A3B in Table 2. Respectively, -q and -g denote routing priors extracted from the model’s own internal inference traces and from externally generated data (via GPT-5 mini). The suffixes -en and -sw indicate the choice of high-resource (English) and low-resource (Swahili) anchor languages. For layer selection, -s restricts routing alignment to the selected intermediate layers, while -a applies alignment across all layers and -r aligns a random subset of layers (see Appendix A.6 for details). The comparison between -q-en-s and -g-en-s highlights the decisive advantage of self-anchoring over external distillation. While external teachers provide high-quality text, they lack mechanistic consistency with the model’s internal expert pathways. This is most evident in the MGSM benchmark, where the -g-en-s variant suffers a substantial performance degradation. We observe that the average MGSM score drops to 53.00% compared to 87.40% for our proposed method. It confirms that rectifying routing logic via the model’s own highresource inference traces is more effective than injecting external knowledge. Layer selection strategy also proves critical for maintaining linguistic flexibility. On the MGSM benchmark, aligning a random subset of layers (-qen-r) and all layers (-q-en-a) both lead to a decrease in average accuracy. This supports our analysis that forcing alignment on shallow and deep layers disrupts the model’s ability to process languagespecific surface variations. The -q-sw-s variant demonstrates that anchor quality is bounded by
Benchmark
Anchor & Method Vanilla LM
English Anchor FFT AES Global-MMLU ShifCon SARA (Ours) Chinese Anchor FFT AES ShifCon SARA (Ours) Vanilla LM
BELEBELE
English Anchor FFT AES ShifCon SARA (Ours) Chinese Anchor FFT AES ShifCon SARA (Ours)
hi
bn
te
sw
en/zh
Avg.
73.97 ±0.40 71.91 ±0.17 71.72 ±0.27 65.60 ±0.70 59.39 ±0.36 81.80 ±0.26 70.73 71.53 ±0.69 69.52 ±0.45 70.04 ±0.71 63.58 ±0.56 56.09 ±0.26 81.03 ±0.21 68.63 74.32 ±0.61 72.25 ±0.78 72.18 ±0.56 65.68 ±0.40 60.15 ±0.77 82.12 ±0.41 71.12 74.66 ±0.15 72.60 ±0.59 72.70 ±0.26 65.76 ±0.68 61.00 ±0.43 82.45 ±0.44 71.53 72.69 ±0.47 70.38 ±0.11 70.23 ±0.62 64.28 ±0.18 57.88 ±0.53 77.20 ±0.62 68.78 69.85 ±0.72 68.24 ±0.67 68.57 ±0.38 62.13 ±0.74 55.09 ±0.43 76.54 ±0.64 66.74 72.55 ±0.24 70.64 ±0.25 70.53 ±0.68 64.48 ±0.61 58.65 ±0.35 77.24 ±0.69 69.02 72.62 ±0.59 70.90 ±0.68 70.84 ±0.58 64.68 ±0.27 59.46 ±0.48 77.27 ±0.67 69.30 82.78 ±0.77 79.67 ±0.14 84.89 ±0.78 73.11 ±0.73 56.22 ±0.27 94.67/91.78 78.32 82.22 ±0.69 82.00 ±0.37 84.11 ±0.22 76.56 ±0.72 77.78 ±0.71 95.00 ±0.11 82.95 79.53 ±0.26 78.85 ±0.61 81.54 ±0.77 74.58 ±0.49 73.52 ±0.60 93.84 ±0.54 80.31 82.85 ±0.12 80.75 ±0.70 83.58 ±0.49 77.02 ±0.37 77.84 ±0.41 95.12 ±0.31 82.86 83.44 ±0.64 80.89 ±0.22 83.67 ±0.17 77.44 ±0.35 77.89 ±0.28 95.22 ±0.49 83.09 80.89 ±0.43 81.44 ±0.18 84.44 ±0.23 76.78 ±0.66 75.67 ±0.77 92.00 ±0.44 81.87 78.56 ±0.67 78.03 ±0.46 80.58 ±0.20 73.54 ±0.24 71.09 ±0.39 90.55 ±0.21 78.73 82.05 ±0.35 79.45 ±0.61 83.35 ±0.30 76.58 ±0.20 76.55 ±0.69 91.82 ±0.67 81.63 83.22 ±0.55 79.56 ±0.66 83.44 ±0.25 76.67 ±0.42 77.44 ±0.27 91.89 ±0.31 82.04
Vanilla LM
MGSM
ne
68.15 ±0.67 63.20 ±0.26 64.86 ±0.63 57.46 ±0.71 23.66 ±0.69 79.65/76.97 59.28
English Anchor FFT AES ShifCon SARA (Ours) Chinese Anchor FFT AES ShifCon SARA (Ours)
–
–
84.80 ±0.24 77.60 ±0.70 48.40 ±0.14 96.00/89.20 79.20
– – – –
– – – –
88.40 ±0.79 84.00 ±0.45 80.40 ±0.50 96.00 ±0.22 87.20 85.20 ±0.37 78.40 ±0.65 76.00 ±0.55 94.40 ±0.76 83.50 87.60 ±0.18 80.40 ±0.65 82.40 ±0.62 97.20 ±0.38 86.90 88.00 ±0.51 80.80 ±0.50 83.20 ±0.77 97.60 ±0.60 87.40
– – – –
– – – –
86.00 ±0.58 83.60 ±0.38 76.80 ±0.25 94.80 ±0.26 85.30 82.40 ±0.64 77.60 ±0.30 74.00 ±0.69 84.80 ±0.52 79.70 85.20 ±0.66 80.40 ±0.58 78.40 ±0.73 86.80 ±0.68 82.70 85.60 ±0.62 80.80 ±0.79 79.20 ±0.70 87.20 ±0.33 83.20
Table 1: Performance comparison of Qwen3-30B-A3B on Global-MMLU, BELEBELE, and MGSM. Note that the evaluation language for each benchmark is aligned with the respective semantic anchor: en Anchor models are evaluated on English test sets, while zh Anchor models are evaluated on Chinese test sets. Results for hi and ne are omitted for MGSM as they are not supported by the benchmark. Benchmark
Method
hi
ne
bn
te
sw
en
Avg.
Global-MMLU
-g-en-s -q-en-a -q-en-r -q-sw-s -q-en-s (Ours)
73.74 69.56 70.03 70.47 74.66
72.03 68.09 67.69 68.20 72.60
72.52 67.78 68.11 68.05 72.70
69.67 61.81 61.84 63.01 65.76
58.32 55.40 55.46 55.59 61.00
81.47 81.78 80.64 82.13 82.45
71.29 67.40 67.30 67.91 71.53
BELEBELE
-g-en-s -q-en-a -q-en-r -q-sw-s -q-en-s (Ours)
78.22 79.22 80.00 79.11 83.44
77.11 77.78 76.11 79.11 80.89
80.00 80.89 82.00 81.56 83.67
73.89 72.56 74.11 73.78 77.44
72.33 74.56 76.00 76.44 77.89
93.56 94.33 92.67 95.00 95.22
79.19 79.89 80.15 80.83 83.09
MGSM
-g-en-s -q-en-a -q-en-r -q-sw-s -q-en-s (Ours)
-
-
52.80 86.80 84.00 85.60 88.00
47.60 80.40 82.40 79.60 80.80
48.80 76.80 78.80 77.20 83.20
62.80 95.20 94.40 95.20 97.60
53.00 84.80 84.90 84.40 87.40
Table 2: Ablation study on Qwen3-30B-A3B investigating the impact of routing prior sources, anchor languages and layer selection strategies. Results for hi and ne are omitted for MGSM as they are not supported by the benchmark.
the model’s native reasoning stability in the pivot language. Utilizing the low-resource Swahili anchor results in an average Global-MMLU score of 67.91% lower than the 71.53% achieved with the English anchor. This result shows that stable rout-
ing patterns from high-resource anchors provide more reliable priors which can help guide expert selection for low-resource inputs.
(a) JS divergence without fine-tuning
(b) JS divergence after FFT
(c) JS divergence after SARA
Figure 2: Comparison of layer-wise routing divergence across different fine-tuning strategies. SARA demonstrates superior alignment in intermediate layers compared to Vanilla LM and FFT.
5.2
5
Analysis
5.1
Routing Consistency
We calculated the JS divergence of the routing distributions for 5 languages relative to the English anchor on Global-MMLU, as shown in Figure 2a. We further analyzed the impact of fine-tuning on this routing behavior. As shown in Figure 2b, FFT leads to a reduction in JS divergence compared to the base model. However, this reduction is incomplete; significant divergence remains perceptible in the intermediate layers. While FFT induces some degree of alignment as a byproduct of task supervision, it is insufficient to correct the internal expert selection logic. In contrast, SARA explicitly minimizes this discrepancy. As demonstrated in Figure 2c, SARA effectively suppresses the remaining divergence, flattening the curves to near-zero levels across layers 7 to 34. This confirms that SARA effectively aligns routing distributions, encouraging semantically equivalent inputs to activate shared expert subsets (see Appendix A.2 for details).
As shown in Figure 3, we analyzed the training dynamics on Global-MMLU to understand the specific impact of data quantity and training iterations. In the initial phase, where the model processes only 1,000 samples per language, FFT outperforms SARA. This suggests that satisfying the routing alignment constraint in addition to learning the multilingual task is more challenging than the single objective of standard fine-tuning. However, this trend reverses as the training progresses. By the end of the first epoch, SARA effectively closes the performance gap with FFT. Furthermore, as training progresses into the second epoch, the performance improvement from FFT becomes insignificant. In contrast, SARA leverages these additional training iterations to consolidate mechanistic consistency, ultimately unlocking a higher performance ceiling through internal routing. By the third epoch, both methods enter a saturation regime with only marginal gains, yet SARA consistently converges to a higher plateau than FFT, indicating a superior asymptotic performance enabled by routing consistency.