arXiv CSopen access

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

2026-4-29

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi, Yichao Du, Feihu Jiang, Longyue Wang* , Weihua Luo Alibaba International Digital Commerce

arXiv:2604.25578v1 [cs.CL] 28 Apr 2026

* Corresponding Author: [email protected]

We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. MarcoMoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-Instruct variants, which surpass the performance of competing models possessing 3–14× more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights. GitHub: Marco-LLM Marco-MoE Base: Marco-Nano-Base

Marco-Mini-Base

Marco-MoE Instruct: Marco-Nano-Instruct

Marco-Mini-Global-Base

Marco-Mini-Instruct

1. Introduction The rapid advancement of Large Language Models (LLMs) has transformed natural language understanding, yet achieving a balance between extensive linguistic coverage and high per-language proficiency remains a critical hurdle, often referred to as the "curse of multilinguality" [Conneau et al., 2020]. This challenge stems from the fact that expanding a model’s language coverage within a fixed parameter budget often degrades performance on individual languages due to capacity bottlenecks and cross-lingual interference. While leading models such as Qwen3 [Yang et al., 2025] and Gemma3 [Team et al., 2025] attempt to bypass these constraints through massive-scale pre-training on up to 36T tokens or sophisticated strong-to-weak distillation [Comanici et al., 2025], such strategies remain computationally prohibitive for most pre-training scenarios. Meanwhile, existing small-scale multilingual models, such as Tiny-Aya [Salamanca et al., 2026], typically use dense architectures that struggle to reconcile broad linguistic breadth with deep task proficiency, as they lack the inherent architectural flexibility to scale effectively across diverse language families without compromising performance. Mixture-of-Experts (MoE) architectures [Shazeer et al., 2017] offer a promising solution to this capacity bottleneck by leveraging conditional computation, which enhances model capacity without increasing the number of activated parameters. However, the computational overhead of training large-scale MoE models from scratch has spurred the emergence of MoE Upcycling [Komatsuzaki et al., 2023], a paradigm that initializes sparse models using pre-trained dense checkpoints. Yet current approaches predominantly rely on coarse-grained expert replication, in which entire feed-forward © 2026 Alibaba International Digital Commerce. All rights reserved

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

English 70

Nano-class

Multilingual Cultural

Average across 10 benchmarks

Average across 11 benchmarks 65

62.8

62.5 59.2

60 55

Multilingual General

Average across 7 benchmarks

53.7

52.9

38.9

40 31.7

32.5

31.7

30.6

50.3

51.0 45.4

44.4

59.1

22.7

41.3

39.5

English

Multilingual General

Average across 7 benchmarks

Mini-class

Average across 11 benchmarks 75

80 75

Multilingual Cultural

Average across 10 benchmarks

75.5 73.3 70.9

48.6

47.9

64.8

66.7

36.8

Ministral3-3B-Instruct LFM2-8B-A1B

Granite4-Tiny Marco-Nano-Instruct

71.0

69.1

67.7 63.1

60 36.9

Qwen3-1.7B-Instruct Qwen3-VL-2B-Instruct

70 65

44.5

45 65.8

50.8

Qwen3-4B-Instruct Ministral3-8B-Instruct

54.5 49.9

Gemma3-12B-Instruct Granite4-Small

LFM2-24B-A2B Marco-Mini-Instruct

Figure 1 | Performance comparison of Marco-MoE Instruct models against open-weight Instruct models across English, General Multilingual, and Cultural & Regional Multilingual benchmarks. Models are grouped into Nano-class (top, ≤ 3B activated parameters) and Mini-class (bottom, ≤ 12B activated parameters). Despite activating only 0.6B and 0.86B parameters respectively, MarcoNano-Instruct and Marco-Mini-Instruct consistently match or outperform models with 3–14× more activated parameters. networks (FFNs) of a dense transformer are replicated. This monolithic duplication deviates from existing wisdom that uses fine-grained experts [Dai et al., 2024], thereby impeding the emergence of specialized representations necessary for nuanced multilingual reasoning. To bridge this gap, we introduce Marco-MoE, a family of compact, highly sparse multilingual MoE models designed to break these traditional trade-offs in multilinguality. Our work distinguishes itself from previous efforts in three fundamental ways: 1. First Sparse Multilingual Upcycling: To the best of our knowledge, this is the first work to leverage the MoE upcycling paradigm specifically to optimize multilingual performance in compact model sizes. By repurposing pre-trained dense representations into a fine-grained MoE framework, we significantly reduce computational overhead while enhancing model capacity. 2. Fine-Grained Expert Specialization: Unlike conventional coarse-grained upcycling methods that replicate entire FFN blocks, we employ a sub-matrix splitting technique to initialize a large number of fine-grained experts. This approach, combined with Drop-Upcycling [Nakamura et al., 2025], catalyzes specialized expert convergence and minimizes the redundancy bottlenecks typical of monolithic replication. 3. Full Transparency and Openness: In contrast to many models that keep their pre-training data and recipes proprietary, Marco-MoE is fully transparent. We disclose the entirety of our pre-training datasets, our data synthesis methodologies, and our rigorous four-stage pretraining curriculum spanning 5.1T tokens. By open-sourcing our models, data, and training recipes, Marco-MoE sets a new standard for accessible, high-performance multilingual LLM development. Our empirical results demonstrate that Marco-Mini-Base (0.86B activated parameters) and Marco2

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Marco-Mini (17B-A0.8B)

Qwen3-4B

55 50

Marco-Nano (8B-A0.6B)

Gemma3-4B Tiny-Aya-3.35B Granite4-Tiny (7B-A1.5B) Trinity-Mini (26B-A3B)

Multilingual Performance

Average Multilingual Performance

Qwen3-1.7B

SmolLM3-3B

Llama3.2-3B Trinity-Nano (6B-A1B)

0.1

Open-weights Open-weights, data, training recipes

0.2 0.5 Estimated Train Compute (FLOPs)

Qwen3-4B

55 50

Tiny-Aya-3.35B

Marco-Nano (8B-A0.6B)

Gemma3-4B Granite4-Tiny

Qwen3-1.7B

Trinity-Mini SmolLM3-3B Llama3.2-3B

1.0 1e24

Marco-Mini (17B-A0.8B)

Open-weights Open-weights, data, training recipes

Trinity-Nano

English Performance

Qwen3-1.7B Trinity-Nano (6B-A1B) Granite-Tiny (7BA1.5B) Marco-Nano (8B-A0.6B)

55 45 35 25 Europe

Asia-Pacific

West-Asia

South-Asia

Average Multilingual Performance

Figure 2 | Comparison of performance and efficiency. Our Marco-MoE base models demonstrate a superior performance-to-compute ratio in (a) and set the state-of-the-art for simultaneous proficiency in both English and multilingual capabilities in (b). Llama3.2-3B SmolLM3-3B Gemma3-4B Tiny-Aya-3.35B

Qwen3-4B Trinity-Mini (26B-A3B) Marco-Mini (17B-A0.8B)

45 35 25 Europe

Asia-Pacific

West-Asia

South-Asia

Figure 3 | Multilingual benchmark performance of base models across geographic regions. Nano-Base (0.6B activated parameters) achieve a superior performance-to-compute ratio (Figure 2). Notably, our models set a new state-of-the-art for simultaneous proficiency in both English and multilingual capabilities, particularly excelling in long-tail and low-resource languages where the performance gap in specific geographic regions (e.g., West and South Asia) over dense counterparts widens significantly (Figure 3). We further adapt the Marco-MoE base models through a two-stage post-training pipeline to develop the Instruct variants. The pipeline starts with Supervised Fine-Tuning (SFT) on high-quality instruction-response pairs, followed by On-Policy Distillation (OPD) [Lu and Lab, 2025, Yang et al., 2025] to efficiently transfer knowledge from high-capacity teacher models. As shown in Figure 1, Marco-Nano-Instruct establishes a highly efficient frontier within the compact model class, surpassing LFM2-8B-A1B-Instruct in English proficiency despite activating 2.5× fewer parameters. Furthermore, it consistently outperforms larger dense and sparse baselines across general and regional multilingual benchmarks. In the larger Mini-class, Marco-Mini-Instruct achieves the highest overall scores in both general and cultural multilingual tasks with only 0.86B activated parameters, representing a fraction of the computational footprint required by competing models in the 4B to 32B parameter range.

2. Model Architecture 2.1. Architecture Overview Marco-MoE LLM leverages a decoder-only Transformer architecture [Vaswani et al., 2023], substituting conventional Feed-Forward Network (FFN) layers with sparse Mixture-of-Experts (MoE) layers [Shazeer et al., 2017]. This substitution facilitates enhanced model capacity and accuracy while

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Model

Marco-Nano-Base

Marco-Mini-Base

Num Layers Model Dimension FFN Intermediate Dimension

28 1024 3072

Q-heads KV-heads Head Dimension

16 8 128

Expert Dimension Total Experts Number of Activated Experts

384 232 8

768 256 8

Tie Embedding

True

Table 1 | Summary of the Marco-MoE LLM architectures. significantly reducing the number of activated parameters. To optimize performance, we implement a granular MoE architecture following Dai et al. [2024]. Other architectural refinements, including Grouped-Query Attention (GQA) [Ainslie et al., 2023], RMSNorm [Zhang and Sennrich, 2019], SwiGLU activation [Dauphin et al., 2017], and Rotary Positional Embeddings (RoPE) [Su et al., 2023], are aligned with the Qwen framework [Yang et al., 2025]. Detailed configurations are summarized in Table 1. Specifically, the Marco-Nano-Base variant comprises 8B total parameters with 0.6B activated (7.5% active ratio), while the Marco-Mini-Base Base contains 17.3B total parameters with 0.86B activated (5% active ratio). Both of them are upcycled from the Qwen3-0.6B-Base model [Yang et al., 2025] using the method detailed in §2.2. 2.2. Upcycle From Dense Model Upcycling [Komatsuzaki et al., 2023] serves as an efficient paradigm for initializing MoE models by repurposing pre-trained dense models. By leveraging the rich representations already encoded within a dense model, this approach substantially reduces computational overhead while achieving performance comparable to that of MoE models trained from scratch. However, naive upcycling, the practice of simply replicating the FFN layers of a dense model to initialize experts, inherently discourages expert diversification. Because identical initializations provide no initial gradient variance between experts, they struggle to specialize during the subsequent training phase. This lack of heterogeneity prevents the MoE architecture from reaching its full capacity, often leading to suboptimal convergence rates over extended training periods. To mitigate these issues and promote expert specialization, recent methods introduce random noise into the FFN replicas [Yang et al., 2024], ensuring each expert begins from a distinct point in the weight space. In this work, we adopt Drop-Upcycling [Nakamura et al., 2025], a strategy centered on the selective re-initialization of expert parameters. Specifically, indices are randomly sampled along the intermediate dimensions of the dense model’s FFNs, and the weights along the corresponding column or row axes are dropped. These parameters are then re-initialized using a Gaussian distribution, with the mean and variance ( 𝜇 and 𝜎2 ) derived directly from the statistics of those dropped weights. In this work, we adopt this upcycling strategy, adapting it to fit within the fine-grained MoE architecture, as shown in Figure 4. Given the transition to a fine-grained MoE architecture, a structural discrepancy arises between the expert dimension 𝐻𝑒𝑥 𝑝𝑒𝑟𝑡 and the intermediate FFN dimension 𝐻𝑑𝑒𝑛𝑠𝑒 of the ancestral dense model. To facilitate seamless parameter inheritance, we first reconfigure the dense model into a pseudo-MoE framework where all experts remain active. In a standard dense Transformer FFN, the computation is defined as: 𝑌 = (Act( 𝑋𝑊𝑔𝑎𝑡𝑒 ) 𝑋𝑊𝑢𝑝 )𝑊𝑑𝑜𝑤𝑛 , where the weight matrices 𝑊𝑢𝑝 ∈ ℝ 𝐷 × 𝐻𝑑𝑒𝑛𝑠𝑒 , 𝑊𝑔𝑎𝑡𝑒 ∈ ℝ 𝐷 × 𝐻𝑑𝑒𝑛𝑠𝑒 , 4

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

ANCESTRAL DENSE Pre-trained Model

Pre-trained Dense Layer (e.g., Qwen3-0.6B)

input

(𝑖)

𝑊𝑢𝑝 × 𝑁1/3 Pseudo-MoE w/ Weight Partition (𝑁 modular slices)

(𝑖)

𝑊𝑔𝑎𝑡𝑒 × 𝑁1/3

𝑊𝑑𝑜𝑤𝑛 × 𝑁1/3

𝑀×

Fine-Grained Expert Replication (𝑀 copies)

(𝑖) 𝑊𝑢𝑝

Output

Fine-grained Expert enables Local Specialization

𝑀×

𝑀× (𝑖)

(𝑖) 𝑊𝑔𝑎𝑡𝑒

𝑊𝑑𝑜𝑤𝑛

…

𝑁 × 𝑀 Experts

Sparse MoE Ensembling

Drop-Upcycling Adding Noise

Router

Figure 4 | The overview of our method on upcycling dense models to MoE models with fine-grained experts. and 𝑊𝑑𝑜𝑤𝑛 ∈ ℝ 𝐻𝑑𝑒𝑛𝑠𝑒 × 𝐷 represent the up-projection, gate-projection, and down-projection layers, respectively. To initialize the MoE layers, we partition these weights into 𝑁 = 𝐻𝑑𝑒𝑛𝑠𝑒 / 𝐻𝑒𝑥 𝑝𝑒𝑟𝑡 slices along the intermediate dimension: (1)

(2)

(𝑁 )

𝑊𝑢𝑝 → [𝑊𝑢𝑝 , 𝑊𝑢𝑝 , ..., 𝑊𝑢𝑝 ]

(1)

(2)

(𝑁 )

𝑊𝑔𝑎𝑡𝑒 → [𝑊𝑔𝑎𝑡𝑒 , 𝑊𝑔𝑎𝑡𝑒 , ..., 𝑊𝑔𝑎𝑡𝑒 ]

(1)

(2)

(𝑁 )

𝑊𝑑𝑜𝑤𝑛 → [𝑊𝑑𝑜𝑤𝑛 , 𝑊𝑑𝑜𝑤𝑛 , ..., 𝑊𝑑𝑜𝑤𝑛 ]

Following this partitioning, the 𝑖-th expert 𝐸 𝑖 is initialized using the corresponding weight pair (𝑖) (𝑖) (𝑖) (𝑊𝑢𝑝 , 𝑊𝑔𝑎𝑡𝑒 , 𝑊𝑑𝑜𝑤𝑛 ). However, a direct weight initialization for each expert via naive slicing introduces a significant scaling disparity. In the dense model, the FFN computes an aggregate output by summing all 𝑁 Í constituent slices: 𝑌 = 𝑖𝑁=1 𝐸 𝑖 ( 𝑥 ). Conversely, a MoE model typically employs a softmax-based gating Í mechanism where the routing probabilities 𝑝𝑖 are constrained such that 𝑝𝑖 = 1. The resulting Í𝑁 output is a weighted combination: 𝑌𝑀𝑜𝐸 = 𝑖=1 𝑝𝑖 𝐸 𝑖 ( 𝑥 ). This creates a functional mismatch. While the dense model implicitly assigns a unit coefficient to each slice, the MoE model distributes weight according to 𝑝𝑖 . Under a uniform routing assumption where 𝑝𝑖 ≈ 𝑁1 , the MoE output magnitude is effectively reduced by a factor of 𝑁 relative to the original dense signal [He et al., 2025a]. To ensure the Pseudo-MoE is mathematically identical to the source dense model, the MoE output must be scaled by the total number of experts ( 𝑁 ): 𝑌𝑀𝑜𝐸 = 𝑁 ×

𝑁 ∑︁

𝑝𝑖 × 𝐸 𝑖 ( 𝑥 )

𝑖=1

In practice, rather than altering the model’s forward-pass logic, we apply a scaling factor 𝜆 = 𝑁 1/3 directly to the sliced weight matrices. This approach preserves the numerical integrity of the original dense model while maintaining a consistent architectural implementation. Figure 5 demonstrates that this weight scaling makes training more stable by reducing loss spikes. Finally, we apply upcycling to 5

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

2.0

Loss

1.9 1.8 1.7 1.6

Upcycling No-Scaling From-Scratch 20B

40B

Tokens

60B

80B

100B

Figure 5 | Ablation on weight scaling and upcycling. We train MoE models with 0.8B active, 17B total parameters on 100B tokens. Upcycling significantly speeds up pre-training, leading to faster convergence. Weight scaling on expert parameters helps reduce loss spikes while achieving similar final training loss. the pseudo-MoE to expand it to a real MoE model with zero-initialized router weights, together with the Drop-Upcycling technique to promote expert diversification and specialization.

3. Pre-Training 3.1. Pre-training Data Marco-Nano-Base and Marco-Mini-Base are pre-trained on a large corpus of high-quality curated and synthetically-generated data derived from both open-sourced and our curated datasets spanning 29 languages: English, Chinese, Arabic, German, Spanish, French, Korean, Japanese, Portuguese, Turkish, Indonesian, Italian, Dutch, Polish, Russian, Vietnamese, Thai, Bengali, Czech, Hebrew, Ukrainian, Malay, Urdu, Kazakh, Greek, Romanian, Hungarian, Nepali, Azerbaijani. 3.1.1. High-Quality English Data To enhance the general language understanding of Marco-MoE, we leverage Nemotron-CC-v2 [NVIDIA et al., 2025a] as the primary source of high-quality English data, specifically utilizing the High and High-Synthetic partitions. This is supplemented by an internal web-crawled English corpus, curated by applying the Fineweb-EDU [Lozhkov et al., 2024] classifier and retaining only the top 10% of documents. Furthermore, to bolster targeted capabilities, we incorporate the Diverse questionanswer (QA) pairs from Nemotron-CC-v2. These pairs encompass various formats, including yes/no, open-ended, and multiple-choice questions, centered on factual information within the text. 3.1.2. Reasoning and Instruction Data Our dataset incorporates a diverse array of open-source sources spanning the STEM, Coding, and instruction-following domains. Specifically, we leverage Nemotron-Pretraining-SFT-v1 [NVIDIA et al., 2025b], Nemotron-Pretraining-Specialized-v1 [NVIDIA, 2025], Nemotron-CC-Math-v1 [Mahabadi et al., 2025], FineMath [Allal et al., 2025], MegaMath [Zhou et al., 2025], OpenThoughts31.2M [Guha et al., 2025], and FLAN [Wei et al., 2022a, OLMo et al., 2025]. Collectively, these provide a robust foundation of reasoning-intensive synthetic problems, high-quality domain-specific 6

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Model

Avg. nld_Latn ron_Latn vie_Latn ell_Grek zsm_Latn tha_Thai urd_Arab ben_Beng kaz_Cyrl

Qwen3-4B-Multilingual 82.2 + Multilingual Diverse QA 83.5

83.7 85.3

83.2 85.6

80.6 83.6

82.6 84.6

81.2 83.0

78.9 80.9

75.6 78.8

74.2 77.9

72.9 77.1

Table 2 | The effects of including multilingual diverse QA data into the pre-training mixture. Performance of selected languages on the BELEBELE benchmark is presented. documents, and diverse instruction data. 3.1.3. High-Quality Multilingual Data Marco-MoE is pre-trained on a large amount of high-quality multilingual data sourced from the web and synthesized by using various strategies. 3.1.3.1 Web-Crawled Data We aggregate multilingual web-crawled data for 28 target languages, sourcing from open-source repositories. Our primary resource is Fineweb-2 [Penedo et al., 2025], which provides corpora for over a thousand languages. To ensure data quality, we prioritize the FineWeb2-HQ variant [Messmer et al., 2025], which utilizes a model-based classifier to filter out low-quality documents for 20 languages. We use this dataset when a high-quality version is available for a language. For the remaining languages that lack high-quality subsets, we employ a rephrasing strategy to mitigate the inherent noise in web-crawled content. Specifically, we employ Qwen3-30B-A3B-Instruct [Yang et al., 2025] to generate synthetic data using four prompting templates derived from Su et al. [2025] and Maini et al. [2024]: 1. Distillation: Reformulating the text into a concise and clear passage. 2. Knowledge Extraction: Isolating informative facts while discarding uninformative content. 3. Structured Listing: Organizing key information into a systematic list. 4. Encyclopedic Paraphrasing: Generating high-quality, Wikipedia-style prose. We specifically curate the training data for Chinese by sourcing from the Fineweb-Edu-Chinese-V2.1 dataset [Yu et al., 2025b]. To ensure data integrity, we restrict our selection to the high-quality partition, exclusively utilizing samples with a quality score exceeding 3. 3.1.3.2 Synthetic Data Multilingual QA Data. NVIDIA et al. [2025b] established that incorporating high-quality multilingual question-answering (QA) data significantly enhances model performance across a variety of downstream tasks. As part of the Nemotron-CC-v2 release, translated versions of the English Diverse question-answer (QA) dataset are provided for 13 languages. To extend coverage to the remaining languages, we leverage the Qwen3-30B-A3B-Instruct model to translate the English Diverse QA data. We employ a curated translation prompt (see Appendix Prompt D) designed to preserve the integrity of non-translatable elements, such as LaTeX syntax and code snippets. To maximize translation fidelity, we implement a granular processing pipeline wherein documents are segmented into individual lines for discrete translation, followed by a reconstruction phase to restore document coherence. Furthermore, language identification tools are utilized to exclude offtarget translations. For low-resource languages where Qwen3-30B-A3B-Instruct exhibits performance degradation, such as Urdu and Kazakh, we apply heuristic filtering mechanisms to remove outputs characterized by repetitive 10-grams. To evaluate the efficacy of the curated dataset, we perform an ablation study employing an annealing phase of 100B tokens, following the methodology of OLMo et al. [2025]. Specifically, 7

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

GlobalMMLU Model

Avg. zho_Hans deu_Latn rus_Cyrl nld_Latn ces_Latn ukr_Cyrl tur_Latn heb_Hebr ben_Beng

Qwen3-4B-Multilingual 64.2

65.9

65.6

65.8

66.0

63.7

62.9

60.9

57.5

55.7

+ Multilingual STEM 65.6

66.9

67.1

67.0

67.3

65.5

64.8

63.3

60.3

59.1

MMMLU Model

Avg. arb_Arab ben_Beng deu_Latn ind_Latn ita_Latn jpn_Jpan kor_Hang por_Latn zho_Hans

Qwen3-4B-Multilingual 60.7 + Multilingual STEM 62.2

57.4 60.5

50.8 54.9

62.7 65.4

64.1 64.5

63.5 62.4

58.7 60.6

57.2 60.6

64.1 65.0

63.2 64.2

MGSM Model

Avg. eng_Latn spa_Latn fra_Latn deu_Latn rus_Cyrl zho_Hans jpn_Jpan tha_Thai ben_Beng

Qwen3-4B-Multilingual 73.7 + Multilingual STEM 81.6

82.0 88.4

80.8 88.0

72.4 83.2

74.0 84.0

84.4 87.6

71.2 79.2

60.0 76.0

76.0 80.4

62.4 67.6

Table 3 | The effects of including multilingual STEM data into the pre-training mixture. Performance of selected languages on the GlobalMMLU, MMMLU, and MGSM benchmarks is presented. we continuously pre-train a Qwen3-4B base model checkpoint for an additional 100B tokens. In the experimental group, the data mixture consists of 50% translated diverse QA data and 50% standard web-crawled data. To ensure a fair comparison, we evaluate this against a baseline variant pre-trained for the same duration solely on the default web mixture. Performance is measured via the BELEBELE benchmark [Bandarkar et al., 2024], with results detailed in Table 2. The integration of synthesized multilingual QA pairs yields consistent improvements, with an average gain of 1.4% across 29 languages. Notably, these performance gains are most pronounced in low-resource languages, including Urdu, Bengali, and Kazakh. Multilingual STEM Data. STEM corpora are characterized by high information density and have been shown to enhance both the general and task-specific capabilities of LLMs. Nevertheless, such resources remain scarce across most non-English languages. To address this disparity, we employ a translationbased data synthesis pipeline to project English STEM knowledge into various target languages. Specifically, for general-domain STEM content, we leverage the Nemotron-Pretraining-SFT-v1 dataset as our primary source, applying our synthesis framework to generate high-fidelity multilingual data across 28 languages. Furthermore, we curate a specialized multilingual mathematical dataset by translating the OpenMathInstruct-2 corpus [Toshniwal et al., 2024]. To assess the efficacy of the curated multilingual STEM dataset, we conduct an ablation study utilizing a 100B-token annealing phase with a 50% mixture of curated data. Performance is measured using the GlobalMMLU [Singh et al., 2025], MMMLU, and MGSM [Shi et al., 2023] benchmarks. As demonstrated in Table 3, the integration of multilingual STEM synthetic data significantly enhances performance across relevant benchmarks, yielding an average absolute gain of up to 8%. Similarly, these benefits become increasingly pronounced as the resource level of the target language decreases. Cultural and Regional Data. An effective multilingual LLM must transcend basic cross-lingual processing; it must interpret the nuanced socio-cultural contexts and conceptual frameworks inherent to each language. However, such culturally salient knowledge is often fragmented across vast web corpora, and its volume is disproportionately small compared to language-agnostic or general-domain data. To mitigate this data scarcity, we employ two distinct strategies to curate high-quality datasets rich in regional and cultural features. We first identify web-crawled documents (e.g., Fineweb-2) characterized by high cultural density. Specifically, we utilize an LLM-based evaluator (see Appendix Prompt D) to assess multilingual documents across three dimensions: quality, geographic origin, and domain. We retain only those 8

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Model Qwen3-4B-Multilingual + Cultural & Regional Data

INCLUDE

Indo-MMLU

Turkish-MMLU

62.3 63.5

60.1 61.2

60.1 64.0

Table 4 | The effects of including cultural and regional data into the pre-training mixture. Performance of selected languages on the INCLUDE, Indo-MMLU, and Turkish-MMLU benchmarks is presented. Dataset Nemotron-CC-v2/High-Quality Nemotron-CC-v2/Diverse-QA Nemotron-CC-v2/High-Quality-Synthetic Nemotron-Pretraining-SFT-v1/Nemotron-SFT-General Nemotron-Pretraining-Specialized-v1/Nemotron-Pretraining-STEM-SFT Nemotron-Pretraining-Specialized-v1/Nemotron-Pretraining-RQA Nemotron-Pretraining-Specialized-v1/Nemotron-Pretraining-Wiki-Rewrite Nemotron-Pretraining-Specialized-v1/Nemotron-Pretraining-InfiniByte-Reasoning Nemotron-Pretraining-Specialized-v1/Nemotron-Pretraining-Math-Textbooks Nemotron-CC-v2.1/High-Quality-DQA peS2o Nemotron-Pretraining-SFT-v1/Nemotron-SFT-MATH Nemotron-CC-Math-v1/4plus OpenMathInstruct-2 Finemath/finemath-4plus Finemath/infiwebmath-4plus Megamath/megamath-web-pro Megamath/megamath-qa/qwen2.5 Nemotron-Pretraining-SFT-v1/Nemotron-SFT-Code Megamath/megamath-text-code-block Nemotron-Pretraining-Specialized-v1/Nemotron-Pretraining-Scientific-Coding Stackexchange OpenThoughts3-1.2M FLAN Fineweb-Edu-Chinese-V2.1 Fineweb-2 & Fineweb-2-HQ Nemotron-CC-v2/Translated-Diverse-QA Dolma3-dolmino-mix-100B-1125 Fineweb2-Culture Translated-Nemotron-SFT-General Translated-OpenMathInstruct-2 Synthetic-Regional-MCQs

Phase1-Mix

Phase2-Mix

Phase3-Mix

Phase4-Mix

20.54% 18.5% 0.94% 6.0% 2.61% 2.44% 0.27% 1.0% 0.87% 0.27% 0.41% 6.93% 1.30% 0.50% 0.33% 0.30% 0.45% 0.11% 1.90% 1.43% 0.04% 0.04% 1.80% 1.0% 6.69% 13.99% 9.33% -

6.5% 18.33% 5.17% 2.02% 7.70% 10.37% 0.17% 0.64% 0.70% 0.17% 0.26% 4.45% 0.69% 2.25% 1.22% 0.92% 0.026% 0.028% 2.69% 0.64% 14.74% 9.15% 6.11% 5.06% -

14.13% 1.18% 4.50% 6.06% 0.1% 0.37% 0.41% 0.1% 0.15% 2.60% 0.40% 1.31% 0.71% 0.54% 0.015% 0.016% 1.57% 0.37% 10.0% 30.0% 22.5% 2.96% -

6.5% 0.30% 1.15% 1.56% 0.026% 0.096% 0.10% 0.026% 0.039% 0.67% 0.10% 0.37% 0.18% 0.14% 0.004% 0.004% 0.40% 0.096% 0.89% 14.3% 0.79% 25.0% 25.0% 12.5% 9.82%

Table 5 | Pre-training datasets and mixture ratios for the four pre-training stages. documents that achieve a quality score exceeding 7 and whose predicted country tag aligns with the primary language. To maximize token utility, these curated regional documents are further augmented through rephrasing and QA generation strategies [Team et al., 2026] for specific lowresource languages. The final dataset is denoted as Fineweb2-Culture. Complementing our web-mining efforts, we develop a pipeline for generating diverse synthetic data focused on culture-specific topics. This process begins with the gathering of coarse-grained cultural themes. We then leverage Gemini-2.5-Pro [Comanici et al., 2025] to perform hierarchical expansion, decomposing these themes into fine-grained sub-topics. Finally, these sub-topics serve as the basis for generating a vast array of diverse multiple-choice questions (MCQs), utilizing a prompt adapted from Bercovich et al. [2025] and Nathawani et al. [2025] (see Appendix Prompt D). We denote the final dataset as Synthetic-Regional-MCQs. To evaluate the efficacy of the curated data, we conduct an ablation study utilizing an annealing phase with a 10B-token budget. Performance was assessed across three cultural benchmarks: INCLUDE [Romanou et al., 2025], Indo-MMLU [Koto et al., 2023], and Turkish-MMLU [Yüksel et al., 2024]. As illustrated in Table 4, the integration of our synthetic cultural data consistently enhances model performance on tasks requiring localized cultural knowledge. 9

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Instruction 1.0% Multilingual 30.01%

Academic 0.41% Math 10.79% Wiki 0.27% STEM 12.32%

Multilingual Breakdown

Synthetic 1.8%

7.0%

zho_Hans 2.7% rus_Cyrl 2.4% spa_Latn 2.0% fra_Latn 1.8% pol_Latn 1.7% deu_Latn 1.6% jpn_Jpan 1.6% nld_Latn 1.5% por_Latn 1.4% ita_Latn 1.4% kor_Hang 1.2% arb_Arab 1.0% vie_Latn 0.9% tha_Thai 0.7% Nemotron-CC High Synthetic tur_Latn 19.5% 0.5% ukr_Cyrl 0.4% ind_Latn zsm_Latn 0.1% 0.1% heb_Hebr Code 3.42% Nemotron-CC High 20.54%

Share of Training Data (%)

(a) Data mixture of Phase 1. Instruction 0.67% Multilingual 30.0%

Academic 0.27% Synthetic 12.93% Wiki Math0.18% 6.07%

Multilingual Breakdown

Nemotron-CC High 3.33%

14.7%

zho_Hans 1.8% rus_Cyrl 1.6% spa_Latn 1.5% ind_Latn 1.3% Nemotron-CC High Synthetic fra_Latn 23.33% 1.2% deu_Latn 1.2% pol_Latn 1.1% nld_Latn 1.0% por_Latn 1.0% kor_Hang 1.0% jpn_Jpan 0.9% ita_Latn 0.8% arb_Arab Code tha_Thai 0.6% 2.28% vie_Latn 0.6% ukr_Cyrl 0.4% tur_Latn 0.4% zsm_Latn 0.1% STEM heb_Hebr 0.1% 20.94% 0

Share of Training Data (%)

(b) Data mixture of Phase 2. Multilingual Breakdown

Instruction 0.36%

Multilingual 62.5%

Nemotron-CC High Synthetic zho_Hans pol_Latn 14.3% jpn_Jpan nld_Latn tha_Thai rus_Cyrl kor_Hang Code arb_Arab 1.22% por_Latn Math deu_Latn 3.25% ita_Latn spa_Latn fra_Latn ukr_Cyrl ind_Latn heb_Hebr STEM npi_Deva 11.22% ben_Beng ron_Latn ces_Latn ell_Grek Wiki zsm_Latn 0.1% azj_Latn hun_Latn Synthetic urd_Arab 6.93% kaz_Cyrl Academic tur_Latn 0.15% vie_Latn

1.7% 1.7% 1.7% 1.7% 1.7% 1.5% 1.5% 1.5% 1.5% 1.5% 1.5% 1.5% 1.5% 1.5% 1.5% 1.3% 1.2% 1.2% 1.2% 0.6% 0

11.7%

3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 2.8%

Share of Training Data (%)

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Instruction 0.12% Nemotron-CC High Synthetic 4.77%

Multilingual 87.5%

Code 0.41% Math 1.08% STEM 3.74% Wiki 0.03% Synthetic 2.31% Academic 0.05%

Multilingual Breakdown zho_Hans fra_Latn ell_Grek tha_Thai ben_Beng spa_Latn zsm_Latn vie_Latn kaz_Cyrl pol_Latn ron_Latn ind_Latn nld_Latn deu_Latn jpn_Jpan ces_Latn hun_Latn urd_Arab por_Latn ukr_Cyrl rus_Cyrl tur_Latn heb_Hebr ita_Latn arb_Arab azj_Latn npi_Deva kor_Hang

1.6% 0

4.0%

3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 3.2% 2.9% 2.7% 2.6% 3

Share of Training Data (%)

(d) Data mixture of Phase 4.

Figure 6 | Data mixtures for each pre-training phase. 3.2. Pre-training Recipe 3.2.1. Four-Stage Data Mixture We adopt a four-stage data mixture strategy to pre-train Marco-MoE models, processing a total of 5.1T tokens. The data mixtures used in each stage are shown in Figure 6 and Table 5. 1. Stage 1 (0 – 2.4T tokens): The foundational training stage comprises a high-quality blend of English, reasoning, and instruction corpora, alongside multilingual web and QA data. The multilingual component covers 19 languages (Chinese, Arabic, German, Spanish, French, Korean, Japanese, Portuguese, Turkish, Indonesian, Italian, Dutch, Polish, Russian, Vietnamese, Thai, Hebrew, Ukrainian, and Malay). 2. Stage 2 (2.4T – 4.1T tokens): To optimize the curriculum, we adjust the data mixture by upsampling reasoning corpora and downsampling English web data. We simultaneously upsample Chinese data to improve targeted linguistic performance. 3. Stage 3 (4.1T – 4.6T tokens): The sampling weights for English, reasoning, and instruction domains are reduced to allocate token bandwidth for nine newly introduced languages: Bengali, Czech, Urdu, Kazakh, Greek, Romanian, Hungarian, Nepali, and Azerbaijani. We concurrently upsample medium-resource languages to ensure balanced cross-lingual efficacy. 4. Stage 4 (4.6T – 5.1T tokens): We further decay the sampling weights of the English, reasoning, and instruction corpora, integrating a diverse array of curated multilingual synthetic data to explicitly target and boost specific reasoning and linguistic capabilities across all languages. 3.2.2. Hyperparameters We pre-train the Marco-MoE models on a total of 5.1T tokens using the Megatron-LM framework [Shoeybi et al., 2020], employing the Warmup-Stable-Decay learning rate (LR) schedule [Hu et al., 2024]. Specifically, the LR is linearly warmed up over 8.4B tokens to its peak value, which is subsequently maintained throughout Stage 1. Peak learning rates are determined based on our training FLOPs, guided by the scaling laws proposed by Tian et al. [2025], yielding 4.9505 × 10−4 for Marco-Nano-Base and 4.6854 × 10−4 for Marco-Mini-Base. During Stage 2, the LR undergoes a linear decay to 1.0 × 10−5 over 1.7T tokens. In Stages 3 and 4, the LR is further linearly decayed to zero over an additional 1T tokens. Initial pre-training utilizes a sequence length of 8,192 and 11

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

2.0

Phase 1

Phase 2

Phase 3

Phase 4

1.8

Loss

1.6 1.4 1.2 1.0 0.8

Tokens

Figure 7 | The training loss graph for Marco-Mini-Base. We indicate the boundaries between training phases where data mixtures are switched. Marco-Nano-Base Learning Rate Schedule LR Warmup steps Peak LR Final LR Batch size (# instances) Sequence length Total training tokens Marco-Mini-Base Learning Rate Schedule LR Warmup steps Peak LR Final LR Batch size (# instances) Sequence length Total training tokens

Stage-1

Stage-2

Stage-3

Stage-4

Constant 890 4.9505 × 10−4 4.9505 × 10−4 1,152 8,192 2.4T

Linear Decay 0 4.9505 × 10−4 1 × 105 1,584 8,192 1.7T

Linear Decay 0 1 × 10−5 6 × 10−6 2,304 8,192 500B

Linear Decay 0 6 × 10−6 0 2,304 8,192 500B

Stage-1

Stage-2

Stage-3

Stage-4

Constant 890 4.6854 × 10−4 4.6854 × 10−4 1,152 8,192 2.4T

Linear Decay 0 4.6854 × 10−4 1 × 105 1,584 8,192 1.7T

Linear Decay 0 1 × 10−5 6 × 10−6 2,304 8,192 500B

Linear Decay 0 6 × 10−6 0 2,304 8,192 500B

Table 6 | Pre-training hyperparameters for each stage of Marco-Nano-Base and Marco-Mini-Base. a batch size of 1,152, resulting in approximately 9.4M tokens per batch. This batch size is later expanded to 1,584 (roughly 13M tokens per batch) to accommodate a hardware scaling from 128 to 176 GPUs in Stage 2. In Stages 3 and 4, we further increase the batch size to 2,304 (roughly 19M tokens per batch). Following Muennighoff et al. [2025], we stabilize MoE training by incorporating a Load Balancing Loss [Shazeer et al., 2017] with a coefficient of 0.01 and a Router Z-loss [Zoph et al., 2022] with a coefficient of 0.001. Optimization is performed using AdamW [Loshchilov and Hutter, 2019] with a weight decay of 0.1, 𝛽1 = 0.9, and 𝛽2 = 0.95. Table 6 presents the detailed hyperparameters across the four training stages. Figure 7 illustrates the detailed pre-training loss trajectory of Marco-Mini-Base. 3.3. Base Model Evaluation 3.3.1. Experimental Settings Evaluation Benchmarks. Unless otherwise specified, all model evaluations are conducted using the Light-Eval framework [Habib et al., 2023] to ensure a standardized and fair comparison. The evaluation suite is divided into two primary dimensions: 12

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

1. English Proficiency: We assess general language understanding and reasoning via a broad array of benchmarks, including MMLU [Hendrycks et al., 2021a], MMLU-Redux [Gema et al., 2025], MMLU-Pro [Wang et al., 2024], BBH [Suzgun et al., 2023], AGIEval [Zhong et al., 2024], ARCEasy/Challenge [Clark et al., 2018], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2019], and several commonsense reasoning tasks such as BoolQ [Clark et al., 2019], CommonsenseQA [Talmor et al., 2019], OpenBookQA [Banerjee et al., 2019], PIQA [Bisk et al., 2019], SIQA [Sap et al., 2019], and GSM8K [Cobbe et al., 2021]. 2. Multilingual Capability: To measure performance across diverse languages, we evaluate the model on both general multilingual benchmarks and datasets featuring cultural and regional knowledge assessment. General datasets include GlobalMMLU [Singh et al., 2025], MMLUProX-Lite [Xuan et al., 2025], BELEBELE [Bandarkar et al., 2024], multilingual HellaSwag & ARC [Lai et al., 2023], MMMLU, MGSM [Shi et al., 2023], and translation tasks with FLORES200 [Team et al., 2022] and WMT24++ [Freitag et al., 2024]. For cultural and regional benchmarks with native-sourced content, we evaluate the models on INCLUDE [Romanou et al., 2025], Global-PIQA [Chang et al., 2025], CMMLU [Li et al., 2024a], C-Eval [Huang et al., 2023], ArabicMMLU [Koto et al., 2024a], TurkishMMLU [Yüksel et al., 2024], GreekMMLU [Zhang et al., 2026], KazakhMMLU [Togmanov et al., 2025], IndoMMLU [Koto et al., 2023], IndoCareer [Koto, 2025], and IndoCulture [Koto et al., 2024b]. Baselines. We compare Marco-Nano-Base and Marco-Mini-Base against a diverse set of open-source baselines, including both open-weight and open-weight & open-data models, selected for their explicit multilingual focus as well as their comparable model scale and training compute. The comparison set includes Qwen3 1.7B and 4B [Yang et al., 2025], Granite4-Tiny [IBM Research, 2025], Llama3.23B [Grattafiori et al., 2024], SmolLM3-3B [Bakouch et al., 2025], Gemma3-4B [Team et al., 2025], Tiny-Aya-3.35B [Salamanca et al., 2026], and Trinity Nano and Mini [Singh et al., 2026]. 3.3.2. Main Results We evaluate the proposed Marco-Nano-Base and Marco-Mini-Base across three axes: English capability, multilingual generalization, and multilingual cultural/regional knowledge. The results in Table 7 show a clear pattern: the Marco-MoE family is strongest where broad cross-lingual transfer matters most, and does so with an unusually favorable compute–performance tradeoff. Marco-Nano-Base establishes a highly competitive efficiency frontier for compact multilingual models, while Marco-Mini-Base emerges as the strongest overall model in the comparison. English results. On English benchmarks, Marco-Mini-Base achieves the best average performance overall (63.7), surpassing all baselines, including Qwen3-4B Base (63.3). This is a particularly strong result because it is obtained with substantially lower training compute than Qwen3-4B (1.56 × 1023 v.s. 8.64 × 1023 FLOPs), and way less activated and total parameters than Trinity-Mini (0.86B v.s. 3.85B). The gains are not driven by a single benchmark: Marco-Mini-Base is consistently strong across knowledge, commonsense, and reasoning tasks, achieving the top result on ARC-Challenge (56.3) and CommonsenseQA (61.5), while remaining near the top on knowledge-intensive and math-oriented benchmarks including MMLU, MMLU-Redux, MMLU-Pro, and GSM8K, where in particular, Qwen3-4B remains stronger due to its excessive exposure to knowledge and reasoning-heavy pre-training data. These results establish Marco-Mini-Base as a top-tier general-purpose English model, not merely a multilingual model with acceptable English transfer.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Benchmark (Metric)

# Shots

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Nano Tiny Nano 3B 3B 4B 3.35B 4B Mini Mini Base

Base

# Activated Params # Total Params # Train FLOPs (×1023 )

1.7B 1.7B 3.67

1.09B 6B 0.654

1.47B 7B 2.03

0.6B 8B 1.40

3B 3B 1.62

3B 3B 1.98

4B 4B 0.96

3.35B 3.35B 1.21

4B 4B 8.64

3.85B 26.5B 2.31

0.86B 17.3B 1.56

English MMLU (Acc) MMLU-Redux (Acc) MMLU-Pro (Acc) AGIEval (Acc) BBH (EM) ARC-Easy (Acc) ARC-Challenge (Acc) HellaSwag (Acc) WinoGrande (Acc) BoolQ (Acc) CommonsenseQA (Acc) OpenBookQA (Acc) PIQA (Acc) SIQA (Acc) GSM8K (EM) Average

5-shot 0-shot 5-shot 0-shot 3-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 5-shot -

65.1 61.2 33.2 35.9 54.5 69.3 42.8 66.6 57.1 74.6 49.5 36.4 75.5 47.8 69.1 55.9

64.7 60.1 32.0 31.4 49.3 77.9 53.5 77.4 57.1 71.5 54.1 42.0 69.6 52.7 57.8 56.7

69.1 65.8 32.1 36.1 59.9 78.5 52.3 77.9 58.6 63.5 55.9 43.6 80.6 53.0 70.7 59.8

64.7 62.9 35.9 38.4 53.5 75.3 49.4 69.2 53.4 71.2 55.7 39.4 76.5 46.0 69.7 57.5

57.6 56.9 26.0 31.2 47.1 71.8 46.0 75.6 58.6 75.2 60.4 42.2 78.2 51.0 27.3 53.7

62.6 58.4 35.1 34.5 60.0 78.5 52.6 76.1 58.9 79.3 55.4 40.4 79.1 49.8 67.4 59.2

61.1 57.7 28.8 32.6 52.2 82.6 54.1 76.7 61.4 76.6 61.1 42.6 80.3 50.4 39.3 57.2

58.6 51.7 26.9 29.0 46.8 76.5 47.4 71.0 56.6 74.6 60.4 40.4 76.9 49.9 58.0 55.5

75.2 71.3 45.9 44.0 72.3 75.0 49.9 74.4 59.6 74.2 52.9 42.6 77.4 53.0 81.7 63.3

71.4 68.2 41.3 39.7 57.6 80.6 57.8 82.8 60.8 72.5 57.7 44.8 71.7 52.5 57.5 61.1

72.8 68.8 45.3 41.9 65.1 82.4 56.3 77.4 57.7 74.2 61.5 44.6 81.1 49.4 76.4 63.7

Multilingual – General GlobalMMLU (Acc) MMMLU (Acc) MMLU-ProX-Lite (Acc) BELEBELE (Acc) mHellaSwag (Acc_norm) mARC-Challenge (Acc_norm) FLORES-200 (En-Xx) (BLEU) FLORES-200 (Xx-En) (BLEU) WMT24++ (En-Xx) (BLEU) WMT24++ (Xx-En) (BLEU) MGSM (EM) Average

5-shot 0-shot 5-shot 0-shot 0-shot 0-shot 5-shot 5-shot 5-shot 5-shot 8-shot -

49.6 48.6 27.2 67.5 43.9 34.7 18.6 31.5 18.3 28.3 58.8 38.8

43.6 41.2 20.3 54.5 42.5 30.9 15.1 31.1 15.0 28.0 40.6 33.0

54.8 52.3 30.1 61.2 53.2 39.9 25.4 36.7 21.9 30.7 56.7 42.1

52.2 52.6 28.9 73.8 48.8 36.9 24.7 33.6 20.7 28.1 65.3 42.3

43.2 44.0 22.4 60.1 49.0 34.2 23.5 34.6 16.4 28.9 22.4 34.4

46.7 47.3 28.3 54.3 49.6 36.1 19.7 30.3 17.8 27.4 50.8 37.1

50.8 47.4 24.3 65.7 55.2 41.5 32.1 39.7 27.7 34.0 36.6 41.4

50.0 44.5 24.3 65.4 53.5 37.2 30.2 37.3 26.1 32.7 38.4 39.9

61.6 59.3 38.5 81.5 53.2 42.5 25.4 36.8 23.9 32.9 76.0 48.3

52.6 50.9 32.2 67.6 51.5 37.5 13.7 24.1 7.5 10.6 57.2 36.9

64.2 62.0 39.2 79.8 58.6 45.4 32.3 40.1 28.1 34.4 75.6 50.9

Multilingual – Cultural & Regional INCLUDE (Acc) 5-shot Global-PIQA (Acc_norm) 0-shot CMMLU (Acc) 5-shot C-Eval (Acc) 5-shot ArabicMMLU (Acc) 3-shot TurkishMMLU (Acc) 5-shot GreekMMLU (Acc) 5-shot KazakhMMLU (Acc) 5-shot IndoMMLU (Acc) 0-shot IndoCareer (Acc) 3-shot IndoCulture (Acc) 0-shot Average -

51.2 60.3 66.1 65.1 57.6 47.9 58.1 52.1 51.0 53.9 51.6 55.9

43.9 52.3 49.6 47.6 44.0 29.6 52.2 43.1 41.5 46.7 49.8 45.5

52.1 64.0 53.5 50.9 60.5 41.8 62.3 52.6 49.0 53.0 51.3 53.7

53.2 64.3 55.5 56.0 55.8 48.9 64.1 53.1 51.0 52.1 57.4 55.6

45.5 62.2 44.1 43.1 48.9 36.7 56.4 44.7 47.0 48.6 50.1 47.9

46.2 60.9 50.1 47.9 60.6 28.4 64.0 47.4 43.7 47.7 44.5 49.2

52.6 69.4 50.2 48.5 61.6 43.7 63.4 52.1 48.5 53.4 59.1 54.8

53.9 67.9 58.8 57.6 63.2 45.2 66.3 47.1 52.0 56.6 58.5 57.0

61.4 65.4 76.2 76.6 67.0 60.6 69.4 62.3 60.1 61.5 61.1 65.6

51.9 57.2 58.6 57.1 57.1 43.0 59.7 49.6 51.0 55.2 57.6 54.4

61.7 72.3 68.0 66.0 67.1 62.7 70.3 62.6 59.9 61.5 62.3 65.0

Table 7 | Comparison among Marco-Nano-Base & Marco-Mini-Base and other open-source base models of comparable sizes and training costs. The best and second best results are marked in bold and underline. Marco-Nano-Base is also strong relative to its active size. With only 0.6B activated parameters, it achieves an English average of 57.5, outperforming several larger baselines, including Llama3.2-3B, Gemma3-4B, and Tiny-Aya-3.35B. It is especially competitive on AGIEval, CommonsenseQA, and MMLU-Pro, indicating that the proposed design preserves substantial reasoning and commonsense capacity even in a highly compact regime. This is a meaningful result: Marco-Nano-Base is efficient without collapsing on core English capability. Multilingual general results. The multilingual general benchmarks show the clearest advantage of the Marco-MoE family. Marco-Mini-Base achieves the best average score (50.9), outperforming all baselines, including Qwen3-4B-Base (48.3). It ranks first on a broad set of tasks covering multilingual

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

knowledge, reasoning, and translation, including GlobalMMLU (64.2), MMMLU (62.0), mHellaSwag (58.6), mARC-Challenge (45.4), FLORES-200 and WMT24++ in both directions. On MGSM, it achieves 75.6, effectively matching the strongest baseline. This consistent performance indicates that Marco-Mini-Base is a well-balanced multilingual model rather than one optimized for a narrow subset of tasks. Marco-Nano-Base is also highly competitive, achieving an average of 43.7. Notably, it obtains the best scores on BELEBELLE and MGSM among all compared models, while remaining strong on the rest. Given its small activated footprint, these results highlight particularly strong cross-lingual reasoning efficiency. Multilingual cultural and regional results. The cultural and regional benchmarks provide the most demanding test of whether multilingual performance extends beyond broad language transfer into localized knowledge and culturally grounded reasoning. On this suite, Marco-Mini-Base achieves an average score of 65.0, ranking second overall and very close to Qwen3-4B-Base (65.7), while outperforming all remaining baselines by a clear margin. The model is particularly strong on GlobalPIQA (72.3), ArabicMMLU (67.1), TurkishMMLU (62.7), GreekMMLU (70.3), KazakhMMLU (62.6), and IndoCulture (62.3), where it achieves the best score or is effectively tied for best. This broad coverage across multiple linguistic and cultural contexts suggests that Marco-Mini-Base captures more than generic multilingual transfer; it also learns substantial region-specific knowledge and localized reasoning ability. Marco-Nano-Base also demonstrates strong performance for its size, achieving an average of 55.6, which is competitive with Qwen3-1.7B-Base (55.9) and stronger than several larger baselines, including Llama3.2-3B-Base (47.9), SmolLM3-3B-Base (49.2), and Gemma3-4B-Base (54.8). It performs well on INCLUDE (53.2), Global-PIQA (64.3), TurkishMMLU (48.9), GreekMMLU (64.1), KazakhMMLU (53.1), and IndoCulture (57.4). These results indicate that Marco-Nano-Base maintains robust regional and cultural generalization despite its compact active architecture, making it a strong option for efficient multilingual deployment across diverse language communities. There are, however, some clear weaknesses: CMMLU and C-Eval remain two difficult benchmarks for the Marco models, where Qwen3-4B-Base holds a substantial advantage. This suggests that highly localized, exam-style Chinese knowledge is one of the few areas where the strongest competing baseline remains ahead, indicating that Marco-Mini-Base may still underperform specialized competitors on certain culturally concentrated or exam-style evaluations. Nevertheless, the broader pattern remains favorable: Marco-Mini-Base is one of the strongest culturally grounded multilingual models in the comparison, and Marco-Nano-Base is unusually capable for its size. Summary. Marco-Mini-Base is the strongest overall model in the comparison. It achieves the best average on English and multilingual general evaluation, and near-best performance on multilingual cultural/regional benchmarks. This establishes it as a highly capable multilingual foundation model rather than a narrowly optimized system. Besides, Marco-Nano-Base defines a compelling compact regime. With only 0.6B activated parameters, it remains competitive on English, strong on multilingual benchmarks, and particularly effective on multilingual reasoning. This places it on a favorable efficiency frontier for multilingual deployment. More importantly, the results show that the Marco design scales in the right direction: increasing model capacity from Nano to Mini yields large and consistent gains, especially in multilingual transfer, without sacrificing compute efficiency. The remaining gaps are concentrated in a small number of specialized benchmarks rather than reflecting a broader weakness. Overall, the experiments demonstrate that the Marco-MoE family delivers state-of-the-art multilingual performance with strong English capability and excellent compute efficiency.

Qwen3-1.7B Trinity-Nano (6B-A1B) Granite-Tiny (7BA1.5B) Marco-Nano (8B-A0.6B)

55 45 35 25 High-Resource

Medium-Resource

Average Multilingual Performance

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Llama3.2-3B SmolLM3-3B Gemma3-4B Tiny-Aya-3.35B

Qwen3-4B Trinity-Mini (26B-A3B) Marco-Mini (17B-A0.8B)

45 35 25

Low-Resource

High-Resource

Medium-Resource

Low-Resource

Figure 8 | Comparison between Marco-MoE models and baselines across different language resource levels. English General Multilingual Cultural & Regional

Stage-1

Stage-2

Stage-3

Stage-4

59.4 40.6 54.0

63.5 46.3 59.8

63.7 50.3 63.2

63.7 52.1 65.0

Table 8 | Average performance on different benchmark categories after each pre-training stage. Perbenchmark results are shown in Appendix Table 13. 3.3.3. Analysis Marco-MoE models excel in long-tail languages. Figure 8 demonstrates the superior multilingual capabilities of the Marco-MoE models, particularly in long-tail languages. In the approximately 1B-parameter category, Marco-Nano-Base consistently outperforms counterparts such as Granite4Tiny and Qwen3-1.7B, which have more activated parameters, and the performance gap widens significantly as language resource availability decreases. A similar trend is observed at the 3B-4B scale, where Marco-Mini-Base achieves the highest overall performance across all resource tiers, with a substantial lead in low-resource languages over established models. Multi-stage pre-training gradually improves downstream performance. As shown in Table 8, the four-stage pre-training curriculum yields consistent improvements across all evaluation dimensions. From Stage-1 to Stage-2, performance increases substantially on every benchmark category, highlighting the importance of the learning-rate decay phase for enhancing overall model capability. From Stage-3 onward, both general multilingual and cultural & regional multilingual performance continue to improve, while English performance largely saturates, suggesting that the data-mixture switch introduced in Stage-3 effectively strengthens multilingual capability without degrading English performance. Overall, these results confirm the effectiveness of the multi-stage curriculum: each stage provides complementary gains, and the later stages are particularly important for improving multilingual coverage and culturally grounded knowledge. MoE expert activation patterns resemble language similarity. To investigate the degree to which MoE routing dynamics align with established linguistic taxonomies, we formalize the concept of a Language-Expert Signature. For each language 𝐿, we randomly sample 100 documents with 𝑁𝐿 tokens from the FineWeb-2 corpus and perform an inference pass to extract the routing statistics. We define the specialization score for an expert 𝐸 𝑖 at layer 𝑙 as the proportion of tokens from 𝐿 that get routed to a particular expert 𝐸 𝑖 : Specialization( 𝐸 𝑖,𝑙 , 𝐿) =

𝑁 𝐸𝑖,𝑙 ,𝐿 , 𝑁𝐿

where 𝑁𝐸𝑖,𝑙 ,𝐿 denotes the frequency with which expert 𝐸 𝑖 at layer 𝑙 is activated for language 𝐿. This procedure yields a language-specific activation matrix M𝐿 ∈ ℝ 𝐷 × 𝐸 , where 𝐷 and 𝐸 represent the total number of layers and experts per layer, respectively. To quantify the functional similarity between 16

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

1.0 Pearson Correlation

0.8 0.6 0.4 0.2 0.0

English Chinese Korean Japanese German Dutch Spanish French Portuguese Italian Turkish Kazakh Azerbaijani Hungarian Polish Romanian Czech Russian Ukrainian Indonesian Malay Vietnamese Thai Arabic Hebrew Urdu Greek Bengali Nepali

Figure 9 | Pearson correlation heatmap between language-specific expert activation patterns. Structured experts are shared among related languages (e.g., Romance, East Asian, and Austronesian), while low-correlation pairs highlight specialized expert utilization. any two languages 𝐿𝑖 and 𝐿 𝑗 , we flatten them into vectors to represent the expert activation patterns for each language across all layers and expert indices. We then compute the Pearson Correlation Coefficient between these vectors to measure their alignment. Figure 9 visualizes the pairwise correlation between languages based on their expert activation signatures. There exists a high-correlation cluster for Romance languages (Spanish, French, Portuguese, and Italian), suggesting that the router leverages shared morphological and syntactic structures by assigning them to common expert pools. Similar clustering is observed in the Slavic (Russian and Ukrainian), Austronesian (Indonesian and Malay), and Indic (Bengali and Nepali) groups, underscoring the model’s ability to exploit cross-lingual transfer. Conversely, languages written in unique scripts or isolated grammars, such as Thai, Vietnamese, Arabic, and Hebrew, exhibit low correlations with other languages. This indicates that the router has effectively partitioned specialized expert subsets to handle unique linguistic features, thereby minimizing language interference. Interestingly, English remains relatively isolated, suggesting that the model may dedicate a distinct, high-capacity expert pool to its primary training language to maintain representational precision. In Appendix Figure 10, we demonstrate that this pairwise correlation can be used for language clustering, with the results largely mirroring established linguistic family structures.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

# Shots

Gemma3 4B Base

Tiny-Aya 3.35B Base

Qwen3 4B Base

Marco Mini Global Base

# Activated Params # Total Params # Train FLOPs (×1023 )

4B 4B 0.96

3.35B 3.35B 1.21

4B 4B 8.64

0.86B 17.3B 1.584

5-shot 0-shot 5-shot 0-shot 3-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 0-shot 5-shot -

61.1 57.7 28.8 32.6 52.2 82.6 54.1 76.7 61.4 76.6 61.1 42.6 80.3 50.4 39.3 57.2

58.6 51.7 26.9 29.0 46.8 76.5 47.4 71.0 56.6 74.6 60.4 40.4 76.9 49.9 58.0 55.5

75.2 71.3 45.9 44.0 72.3 75.0 49.9 74.4 59.6 74.2 52.9 42.6 77.4 53.0 81.7 63.3

72.9 68.9 44.5 41.0 65.0 82.4 57.0 77.2 58.3 75.6 61.2 45.0 80.7 48.4 76.4 63.6

5-shot 0-shot 5-shot 0-shot 0-shot 0-shot 5-shot 5-shot 5-shot 5-shot 8-shot -

49.1 45.0 23.3 62.3 51.9 39.3 27.9 39.2 26.0 34.4 35.7 39.5

48.4 42.8 23.5 62.5 50.3 35.7 25.6 37.2 24.4 32.9 36.6 37.3

57.8 54.8 35.6 74.0 48.5 39.3 25.8 33.4 19.6 31.2 69.1 44.5

60.9 58.2 36.2 76.0 54.4 41.2 29.5 40.2 26.0 34.5 71.7 48.1

52.3 67.8 50.2 48.5 61.6 43.7 63.4 52.1 48.5 53.4 59.1 54.6

53.5 66.7 58.8 57.6 63.2 45.2 66.3 47.1 52.0 56.6 58.5 56.9

60.0 61.8 76.2 76.6 67.0 60.6 69.4 62.3 60.1 61.5 61.1 65.1

61.1 70.2 67.9 66.2 66.6 63.1 70.4 61.8 59.5 61.8 62.5 64.7

Benchmark (Metric)

Table 9 | Comparing Marco-Mini-Global-Base on different benchmark categories with baselines on 64 languages. 3.3.4. Scaling up to 64 Languages To evaluate the scalability of the Marco-MoE framework, we expand the model’s linguistic coverage from 29 core languages to 64 languages with 35 newly introduced languages: Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Serbian, Bulgarian, Persian, Maltese, Hindi, Marathi, Gujarati, Punjabi, Tamil, Telugu, Tagalog, Javanese, Khmer, Lao, Burmese, Amharic, Swahili, Yoruba, Igbo, Zulu. Following the multi-stage curriculum, we branch from the Stage-2 checkpoint and recalibrate the data mixtures in Stages 3 and 4 to integrate pre-training corpora for the 35 newly introduced languages. This variant, designated as Marco-Mini-Global-Base, was trained on an additional 1.4T tokens and consistently 18

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

demonstrates superior performance over comparable baselines as shown in Table 9. Notably, MarcoMini-Global-Base preserves robust English proficiency, achieving 63.6 compared to the 63.7 average of the original Marco-Mini-Base, while simultaneously increasing the performance advantage in multilingual tasks from 2.6% to 3.6% relative to Qwen3-4B-Base.

4. Post-Training We employ a two-stage post-training pipeline, comprising supervised fine-tuning (SFT) and on-policy distillation [Lu and Lab, 2025] from high-capacity teacher models, to develop general Instruct models. This pipeline produces models for low-latency responses to standard user queries by bypassing explicit, computationally intensive reasoning processes. 4.1. Supervised Fine-Tuning 4.1.1. Data General Instruction Data. Our SFT process leverages the Dolci-Instruct dataset [Olmo et al., 2025], originally employed to train the Olmo3-Instruct models. To further enhance the instruction-following capabilities of Marco-MoE, we augment this corpus with additional data sourced from Yang et al. [2026]. Knowledge-Intensive Data. We observe that scientific data is underrepresented within the DolciInstruct dataset. To address this gap, we augment the corpus by sourcing scientific prompts from Yang et al. [2026] and distilling the corresponding responses from Gemini3-Flash. Translation Data. For translation data, our data curation pipeline begins by sourcing parallel corpora from the web-mined NLLB translation dataset [Team et al., 2022]. To ensure high data fidelity, we implement a multi-stage filtering strategy. First, we apply heuristic text filters adapted from Gopher [Rae et al., 2022] and FineWeb [Penedo et al., 2024] to the English segments to exclude low-quality pairs. To further enhance semantic alignment, we leverage the Qwen3-Embedding-8B model [Zhang et al., 2025] to compute relevance scores for each translation pair. Ultimately, we retain only the top 10,000 highest-scoring pairs per language. Multilingual & Cultural Data. To generate culturally-grounded data, we leverage Wikidata [Vrandečić and Krötzsch, 2014] as our primary knowledge source within a multi-stage generation pipeline. Initially, we identify a taxonomy of topics pertinent to regional and cultural contexts. For each target language, we then filter entities within these topics, selecting only those whose "country" property aligns with the corresponding linguistic region. Utilizing these retrieved entities and their associated metadata, we employ Gemini3-Flash to synthesize coherent textual descriptions. Building on these descriptions, we adapt a back-translation approach [Li et al., 2024b] to prompt the model to derive corresponding instructions. Recognizing that descriptions derived solely from structured properties can be overly formulaic or information-sparse, we further task Gemini3-Flash with augmenting these instructions by recalling relevant background knowledge, thereby producing more natural and comprehensive responses. This process yields a high-quality, diverse dataset of instruction-response pairs tailored to the cultural nuances of each language. In addition, we augment these pairs by transforming them into multiple-choice question-answering (QA) formats with concise step-by-step chain-of-thoughts to further enhance training data diversity.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Category

Datasets

Mixture Ratio

Instruction Following

Nemotron-RL-instruction-following Nemotron-RL-instruction-following-structured-outputs

25%

Konwledge & Reasoning

Nemotron-RL-ReasoningGym-v1 Nemotron-RL-knowledge-mcqa

25%

Nemotron-Cascade-RL-RLHF

10%

DAPO-Math-17k Skywork-OR1-RL-Data

10%

Translation Multilingual & Cultural Data Nemotron-SFT-Multilingual-v1

30%

Alignment Math

Multilingual

Table 10 | Datasets and mixture ratios for the on-policy distillation stage. 4.1.2. Implementation Details We fine-tune Marco-MoE utilizing the SLIME framework [Zhu et al., 2025]. The learning rate is initialized at 1 × 10−5 and follows a cosine decay schedule, reaching a minimum of 1 × 10−6 . The model is trained for a single epoch with a global batch size of 512, totaling approximately 4,000 training steps. We set the maximum context length to 8,192 tokens. Optimization is performed using AdamW with a weight decay of 0.1, 𝛽1 = 0.9, and 𝛽2 = 0.95. The whole SFT process takes around 24 hours using 64 GPUs. 4.2. On-Policy Distillation from Stronger Models On-policy learning involves training a model on samples generated by its own current policy, fundamentally enabling it to correct its own errors. While this approach is a cornerstone of reinforcement learning (RL), it is often constrained by sparse feedback; RL typically yields a limited amount of information per training episode regardless of sequence length [Schulman and Lab, 2025], resulting in significant sample inefficiency and high computational overhead. In contrast, off-policy learning seeks to distill the behavior of a superior teacher model. By leveraging supervision for every token produced by the teacher, off-policy methods provide substantially denser learning signals. However, this paradigm is susceptible to distribution shift: the student model is trained on trajectories native to the teacher rather than those it will encounter during its own inference. This can lead to compounding errors, where early deviations from the teacher’s distribution cause the model to diverge into unfamiliar states. To bridge these two paradigms, On-Policy Distillation (OPD) [Lu and Lab, 2025, Yang et al., 2025] utilizes trajectories sampled from the student model while incorporating the dense per-token supervision characteristic of off-policy distillation. Consequently, OPD allows the model to learn from its own generations while maintaining the high signal density required for efficient optimization. 4.2.1. Data We employ a diverse set of data sources to post-train the SFT checkpoint using OPD, with the exact data mixtures shown in Table 10. Instruction Following. We sample instruction-following prompts from the Nemotron-RL-instructionfollowing dataset. To further enhance the ability of Marco-MoE in following specific output formats and generating structured data, we further augment this set with prompts from Nemotron-RL-instructionfollowing-structured-outputs. Knowledge & Reasoning. We gather a diverse collection of datasets focused on reasoning and 20

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

knowledge grounding. Specifically, we sample reasoning tasks from Nemotron-RL-ReasoningGym-v1, which encompasses a wide array of domains, including algebra, geometry, and graph theory. Furthermore, we incorporate Nemotron-RL-knowledge-mcqa, a dataset comprising knowledge-intensive multiple-choice question-answering (MCQA) samples across various scientific disciplines. Alignment. To facilitate human preference alignment and enhance conversational proficiency, we utilize alignment prompts from the Nemotron-Cascade-RL-RLHF dataset to enhance Marco-MoE. Math. While Marco-MoE is not explicitly optimized for mathematical reasoning, we incorporate the DAPO-Math-17k [Yu et al., 2025a] and Skywork-OR1-RL-Data [He et al., 2025b] datasets to enhance the model’s general problem-solving proficiency. Multilingual. The multilingual corpus incorporates a subsample of our curated translation and cultural data. We further augment this with the Nemotron-SFT-Multilingual-v1 dataset, which provides high-quality translations of mathematical and scientific problems. 4.2.2. Implementation Details We implement OPD within the SLIME framework and adopt a cascaded distillation strategy that progressively transfers knowledge from increasingly stronger teacher models. Specifically, for both Marco-Nano-Instruct and Marco-Mini-Instruct models, we first use Qwen3-30B-A3B-Instruct-2507 as the initial teacher. After the student model converges, we switch to the stronger Qwen3-Next-80BA3B-Instruct and continue distillation. Training uses a constant learning rate of 1 × 10−6 and a global batch size of 512. To improve training efficiency, we sample only two responses per prompt, resulting in 1,024 sequences per training step. The maximum sequence lengths are set to 2,048 tokens for prompts and 8,192 tokens for responses. The complete OPD procedure for Marco-Nano-Instruct involves approximately 1,900 steps with Qwen3-30B-A3B-Instruct-2507 followed by 1,000 steps with Qwen3-Next-80B-A3B-Instruct; Marco-Mini-Instruct undergoes 1,900 steps for each phase. The total training process requires roughly 110 hours on a cluster of 64 GPUs. 4.3. Instruct Model Evaluation 4.3.1. Experimental Settings Evaluation Benchmarks. Following base mode evaluation settings, all models are conducted using the Light-Eval framework [Habib et al., 2023] to ensure fair comparisons. The evaluation suite is divided into two primary dimensions: 1. English Proficiency: We assess general language understanding and reasoning via diverse benchmarks, including MMLU [Hendrycks et al., 2021a], MMLU-Redux [Gema et al., 2025], MMLU-Pro [Wang et al., 2024], AGIEval [Zhong et al., 2024], GPQA-Diamond [Rein et al., 2023], GSM8K [Cobbe et al., 2021], and MATH [Hendrycks et al., 2021b]. 2. Multilingual Capability: To measure performance across diverse languages, we evaluate the model on both general multilingual benchmarks and datasets assessing cultural and regional knowledge. General datasets include GlobalMMLU [Singh et al., 2025], MMLU-ProX-Lite [Xuan et al., 2025], MMMLU, MGPQA [Huang et al., 2025], MGSM [Shi et al., 2023], PolyMath [Wang et al., 2025] and translation tasks with FLORES-200 [Team et al., 2022] and WMT24++ [Freitag et al., 2024]. For cultural and regional benchmarks, we evaluate the models on INCLUDE [Romanou et al., 2025], Global-PIQA [Chang et al., 2025], CMMLU [Li et al., 2024a], C-Eval [Huang et al., 2023], ArabicMMLU [Koto et al., 2024a], TurkishMMLU [Yüksel et al., 2024], GreekMMLU [Zhang et al., 2026], KazakhMMLU [Togmanov et al., 2025], IndoMMLU [Koto et al., 2023], IndoCareer [Koto, 2025], and IndoCulture [Koto et al., 2024b]. 21

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Benchmark (Metric)

Qwen3 Qwen3-VL Ministral3 LFM2 Granite4 Marco Qwen3 Ministral3 Gemma3 Granite4 LFM2 Marco 1.7B 2B 3B 8B-A1B Tiny Nano 4B 8B 12B Small 24B-A2B Mini Instruct

Instruct

Instruct Instruct

Instruct

# Activated Params # Total Params

1.7B 1.7B

2B 2B

3.84B 3.84B

1.5B 8.3B

1.47B 7B

0.6B 8B

4B 4B

8.8B 8.8B

12B 12B

9B 32B

2B 24B

0.86B 17.3B

English MMLU (Acc) MMLU-Redux (Acc) MMLU-Pro (Acc) AGIEval (Acc) GPQA-Diamond (Acc) GSM8K (EM) MATH (EM) Average

62.4 62.4 35.2 39.6 27.5 77.9 70.6 53.7

62.1 62.2 38.3 33.0 21.0 79.7 73.7 52.9

69.8 69.6 49.5 44.7 31.6 79.0 70.2 59.2

72.1 71.9 49.5 45.2 31.9 84.6 82.6 62.5

50.8 51.2 25.3 30.7 28.3 71.1 53.4 44.4

73.2 73.3 54.5 49.8 22.2 86.7 79.6 62.8

80.8 80.9 66.9 51.7 50.8 88.6 93.4 73.3

79.8 79.9 63.9 52.4 44.8 89.5 86.2 70.9

76.2 76.2 55.8 43.6 35.2 89.7 83.8 65.8

76.7 76.7 57.1 44.7 38.6 83.9 75.7 64.8

74.9 74.9 57.6 49.0 39.7 87.2 83.9 66.7

83.4 83.5 70.7 55.4 50.3 93.1 91.8 75.5

Multilingual – General GlobalMMLU (Acc) MMMLU (Acc) MMLU-ProX-Lite (Acc) MGPQA (Acc) FLORES-200 (En-Xx) (BLEU) FLORES-200 (Xx-En) (BLEU) WMT24++ (En-Xx) (BLEU) WMT24++ (Xx-En) (BLEU) MGSM (EM) PolyMath (EM) Average

46.3 49.0 28.6 25.3 12.7 28.2 13.2 26.4 63.6 23.4 31.7

45.9 49.0 30.3 22.3 15.3 28.6 14.6 26.2 67.6 25.5 32.5

38.4 39.4 26.7 18.8 8.3 18.9 4.4 8.3 47.0 16.3 22.7

49.0 56.5 33.8 27.2 14.9 20.1 14.6 17.9 56.5 26.5 31.7

43.0 44.1 22.1 25.9 22.5 30.4 18.9 25.1 55.3 18.7 30.6

58.7 59.9 43.2 21.6 22.3 31.1 18.7 27.3 76.5 29.6 38.9

70.2 71.3 58.3 41.0 22.1 33.5 20.9 29.9 84.4 47.2 47.9

55.4 56.4 43.3 30.5 17.5 31.0 14.4 24.2 68.7 26.4 36.8

69.2 69.4 51.3 32.8 35.6 40.3 32.1 35.5 84.0 35.5 48.6

67.4 68.1 51.6 35.0 31.9 32.2 26.6 27.5 75.7 28.9 44.5

57.0 62.3 43.3 32.7 19.2 22.7 16.0 18.8 67.8 29.3 36.9

73.3 73.7 61.2 41.8 30.6 36.8 26.8 31.3 87.4 44.7 50.8

44.4 65.8 63.3 63.2 46.9 39.6 48.0 47.1 49.3 45.7 47.7 51.0

35.4 50.6 48.9 50.6 22.7 38.6 38.4 41.4 35.2 36.0 37.2 39.5

43.5 60.8 52.7 50.8 56.5 26.3 40.0 39.6 41.1 41.7 45.9 45.4

38.6 63.3 39.2 39.4 43.4 31.6 44.8 39.6 37.2 34.7 42.8 41.3

54.3 70.7 60.0 60.8 56.5 59.9 61.6 56.3 56.3 54.9 59.1 59.1

63.8 79.6 78.6 80.4 66.0 71.6 68.6 66.6 64.4 62.2 58.7 69.1

50.7 61.3 67.4 68.0 41.4 48.2 49.5 59.1 52.4 53.4 47.8 54.5

65.0 82.2 60.8 59.7 70.1 64.4 77.7 66.8 65.3 63.2 69.6 67.7

60.3 80.2 59.6 59.4 66.3 57.9 71.7 63.5 59.6 56.3 59.3 63.1

49.1 69.0 56.7 56.7 61.3 33.4 44.7 47.6 42.7 43.7 44.2 49.9

65.6 84.2 75.3 75.4 67.8 74.7 72.5 68.8 65.7 64.4 67.1 71.0

Multilingual – Cultural & Regional INCLUDE (Acc) 44.9 62.0 Global-PIQA (Acc) CMMLU (Acc) 60.4 C-Eval (Acc) 58.7 ArabicMMLU (Acc) 48.8 TurkishMMLU (Acc) 42.7 GreekMMLU (Acc) 48.7 KazakhMMLU (Acc) 46.0 IndoMMLU (Acc) 48.8 IndoCareer (Acc) 46.1 45.8 IndoCulture (Acc) Average 50.3

Table 11 | Comparison among Marco-Nano-Instruct & Marco-Mini-Instruct and other open-source Instruct models of comparable sizes. The best and second best results are marked in bold and underline. Avg@8 accuracies are reported, except for GlobalMMLU and MMMLU where Acc@1 is reported. Baselines. We compare Marco-Nano-Instruct and Marco-Mini-Instruct against various open-source Instruct models. The comparison set includes dense models: Qwen3-1.7B-Instruct (i.e., nonthinking), Qwen3-4B-Instruct-2507 [Yang et al., 2025], Qwen3-VL-2B-Instruct [Bai et al., 2025], Ministral-3-3B-Instruct-2512 & Ministral-3-8B-Instruct-2512 [Liu et al., 2026], and Gemma3-12BInstruct [Team et al., 2025]; MoE models: Granite4-Tiny & Granite4-Small [IBM Research, 2025], and LFM2-8B-A1B & LFM2-24B-A2B [Amini et al., 2025]. All models are evaluated using a sampling temperature of 1.0 and a top-p value of 1.0. For most benchmarks, we generate eight responses per test instance with a maximum response length of 8,192 and report the mean accuracy. However, for GlobalMMLU and MMMLU, we only sample a single response per query to optimize computational efficiency, given the substantially larger scale of these datasets. 4.3.2. Main Results We demonstrate the results of Instruct models in Table 11. English Results. On English benchmarks, Marco-Mini-Instruct achieves an average score of 75.5, effectively surpassing Qwen3-4B-Instruct (73.3) and significantly outperforming Ministral3-8B-Instruct

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Marco-Nano-Instruct

English General Multilingual Cultural & Regional

Marco-Mini-Instruct

Stage-1

Stage-2

Stage-1

Stage-2

60.5 38.0 58.6

62.8 38.9 59.1

+2.3 +0.9 +0.5

73.7 49.3 69.9

75.5 50.8 71.0

+1.8 +1.5 +1.1

Table 12 | Performance progression across the two-stage cascaded OPD. The Δ indicates the absolute improvement from Stage-1 (utilizing the 30B-A3B teacher) to Stage-2 (transitioning to the 80B-A3B teacher) across three benchmark categories. Per-benchmark results are shown in Appendix Table 14. (70.9) and Gemma3-12B-Instruct (65.8), despite utilizing a fraction of the activated parameters. This efficiency is even more pronounced when compared to other sparse models; Marco-Mini-Instruct significantly outperforms LFM2-24B-A2B (66.7) and Granite-4-Small (64.8), even though these baselines activate 2B and 9B parameters, respectively. Similarly, Marco-Nano-Instruct defines a highly efficient frontier, achieving an average score of 62.8. It notably surpasses LFM2-8B-A1B (62.5) while requiring only 40% of its activated parameter budget (0.6B v.s. 1.5B). Multilingual General Results. The advantages of the Marco-MoE models are most pronounced in general multilingual capabilities, where Marco-Mini-Instruct ranks first among all evaluated models with an average score of 50.8. It not only surpasses Qwen3-4B-Instruct but also demonstrates nearly double the efficiency of comparable MoE competitors. For instance, Marco-Mini-Instruct outperforms LFM2-24B-A2B by an absolute point of 12.4, despite having 38% fewer total parameters and 57% fewer activated parameters. Marco-Nano-Instruct also shows remarkable cross-lingual transfer, achieving 38.9 and outperforming not only baselines with comparable size but also the significantly larger LFM2-24B-A2B variant. Multilingual Cultural and Regional Results. Regarding localized knowledge and culturally grounded reasoning, Marco-Mini-Instruct emerges as the top-performing model with an average of 71.0, outperforming Qwen3-4B-Instruct (69.1) and the substantially larger Gemma3-12B-Instruct (67.7). The performance gap between Marco-MoE and other MoE baselines widens substantially in this category, with Marco-Mini-Instruct outperforming LFM2-24B-A2B-Instruct (49.9) by 21.1 points. Even the compact Marco-Nano-Instruct (59.1) demonstrates superior cultural proficiency relative to both LFM2-8B-A1B and LFM2-24B-A2B models. Nevertheless, Marco-MoE models still lag behind Qwen3 models in highly localized Chinese knowledge reasoning, as evidenced by lower relative performance on the CMMLU and C-Eval benchmarks. 4.3.3. The Effects of Cascaded Distillation Table 12 demonstrate the efficacy of our cascaded distillation strategy, which progressively transfers knowledge from increasingly capable teacher models. Both Marco-Nano-Instruct and Marco-MiniInstruct models achieve consistent gains across all evaluation axes upon transitioning from the 30B-A3B to the 80B-A3B teacher. This progressive improvement validates that the student models have not reached their capacity limits after the first stage. Instead, OPD from a stronger teacher provides higher-quality reward signals that allow the model to correct more sophisticated errors.

5. Related Work MoE Model Upcycling. The computational overhead of training large-scale Mixture-of-Experts (MoE) models from scratch has led to the emergence of MoE Upcycling [Komatsuzaki et al., 2023], a paradigm that initializes sparse models from pre-trained dense checkpoints. Prominent methods,

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

such as Sparse Upcycling (e.g., Mixtral 8x7B [Jiang et al., 2024]), typically utilize coarse-grained expert construction. In these frameworks, the feed-forward networks (FFNs) of a dense transformer are replicated in their entirety to form the expert pool, followed by continued pre-training. Because all experts are identical at initialization, this approach may impede the emergence of specialized representations. While recent strategies attempt to break this symmetry by injecting random noise during replication [Yang et al., 2024, Nakamura et al., 2025], the fundamental units of the model remain monolithic. While computationally convenient, coarse-grained upcycling stands in tension with recent architectural trends suggesting that a large number of fine-grained experts is more effective than a few coarse-grained experts under an equivalent parameter budget [Dai et al., 2024]. Our method shifts toward fine-grained expert upcycling. Rather than duplicating entire FFN blocks, we decompose dense weights into modular sub-components via sub-matrix splitting before expanding them into an expert ensemble. Our method mitigates the redundancy bottlenecks inherent in monolithic block replication and facilitates expert specialization. Multilingual LLMs The development of multilingual Large Language Models (LLMs) is fundamentally constrained by the "curse of multilinguality" [Conneau et al., 2020], wherein increasing language coverage within a fixed-parameter budget leads to performance degradation due to capacity bottlenecks and cross-lingual interference. While cross-lingual transfer can benefit low-resource languages [Joshi et al., 2020, Nigatu et al., 2024], it is most effective between typologically related languages [He et al., 2025c], making it difficult for small-scale models to maintain high performance across diverse families [Üstün et al., 2024]. Recent state-of-the-art compact models have adopted varying strategies to mitigate these constraints: Qwen3-4B [Yang et al., 2025] and Gemma3-4B [Team et al., 2025] utilize massive pretraining (up to 36T tokens) and sophisticated strong-to-weak distillation and claim to support 119 and 140+ languages, respectively. In contrast, models such as SmolLM3 [Bakouch et al., 2025] prioritize per-language depth over breadth, focusing exclusively on a narrow subset of high-resource languages. Other efforts, including SEA-LION [Ong and Limkonchotiwat, 2023], Sailor2 [Dou et al., 2025], EuroLLM [Martins et al., 2024, Ramos et al., 2026], and Franken-Adapter [Jiang et al., 2025], address capacity limitations through regional specialization. While extreme cases like Apertus [Apertus et al., 2025] expand coverage to over 1,000 languages, they often encounter substantial performance trade-offs on standard benchmarks. Despite these advances, most existing small-scale multilingual models rely on dense architectures, which inherently struggle to balance breadth and proficiency. Our work introduces the first compact, highly-sparse Multilingual MoE models with a fully open pre-training recipe. By leveraging the conditional computation of MoE, our method effectively breaks the curse of multilinguality, scaling support to 64 languages without the performance degradation typical of dense counterparts. This architectural shift facilitates specialized expert activation patterns that preserve the integrity of individual languages while maximizing cross-lingual transfer across related languages. Post-Training and On-Policy Distillation. Standard post-training typically commences with Supervised Fine-Tuning (SFT) utilizing high-quality instruction-response pairs to establish foundational instruction-following capabilities [Wei et al., 2022b, Chung et al., 2022]. However, SFT frequently suffers from exposure bias and significant distribution shift [Gudibande et al., 2023]. Because the model is trained exclusively on trajectories native to the teacher rather than those encountered during its own inference, it often lacks the robustness required to recover from compounding errors during auto-regressive generation. To address these distribution mismatches, on-policy learning, the basic element of reinforcement

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

learning (RL), has emerged as an effective paradigm. In this framework, the model is trained on samples generated by its own current policy, providing a principled method for interactive alignment by exposing the student to its actual state distribution. This allows the model to learn directly from its own sampled trajectories rather than relying on external, static data. While RL techniques like PPO [Schulman et al., 2017] or GRPO [Shao et al., 2024] offer powerful optimization, they are often constrained by sparse reward signals and significant computational overhead. In contrast, On-Policy Distillation (OPD) [Lu and Lab, 2025, Yang et al., 2025] provides substantially higher signal density that enables efficient optimization comparable to RL performance at a reduced compute cost. In our work, we implement an OPD pipeline, adopting a cascaded distillation strategy. By progressively transferring knowledge from increasingly capable teachers, we ensure the iterative refinement of multilingual and reasoning capabilities while maintaining superior sample efficiency.

6. Conclusion and Discussion In this work, we introduced Marco-MoE, a family of fully open-source, multilingual, and highly sparse MoE models that effectively mitigate the capacity bottlenecks inherent in dense architectures. By employing a fine-grained upcycling strategy, we successfully converted pre-trained dense models into highly sparse MoE layers that activate only around 5% of parameters per token. Our base models, Marco-Nano-Base and Marco-Mini-Base, demonstrate state-of-the-art performance across English and multilingual benchmarks, establishing a superior performance-to-compute ratio over similarly sized dense and MoE baselines. Leveraging these foundations, we developed Marco-MoEInstruct variants through a lightweight two-stage post-training pipeline. Empirical results indicate that Marco-Nano-Instruct and Marco-Mini-Instruct consistently outperform competing architectures possessing 3–14× more activated parameters. Furthermore, our analysis elucidates that Marco-MoE naturally learns structured expert activation signatures that mirror established linguistic family structures, facilitating the scalable expansion to 64 languages while minimizing the cross-lingual interference typical of dense counterparts. By disclosing our complete four-stage training curriculum, comprehensive datasets, and model weights, we aim to provide a transparent and high-performance foundation for future multilingual research. While we have demonstrated the efficient scaling of Marco-MoE to 64 languages, this coverage encompasses only a small fraction of global linguistic diversity, particularly compared to massively multilingual models that target over 1,000 languages. Our reliance on translation-based and synthetic data pipelines to boost performance in low-resource settings, while empirically validated, underscores the persistent scarcity of authentic, high-quality multilingual reasoning data in naturalistic environments. Future research will focus on broadening Marco-MoE ’s linguistic scope while investigating more sophisticated routing mechanisms to enhance expert specialization for extremely long-tail languages. Furthermore, because our current framework necessitates retraining the full model to integrate new languages, developing methods for modular, incremental expansion remains a compelling direction for efficient language scaling.

References J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298/. 25

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafioti, H. Kydlíček, A. P. Lajarín, V. Srivastav, J. Lochner, C. Fahlgren, X.-S. Nguyen, C. Fourrier, B. Burtenshaw, H. Larcher, H. Zhao, C. Zakka, M. Morlon, C. Raffel, L. von Werra, and T. Wolf. Smollm2: When smol goes big – data-centric training of a small language model, 2025. URL https://arxiv.org/abs/2502.02737. A. Amini, A. Banaszak, H. Benoit, A. Böök, T. Dakhran, S. Duong, A. Eng, F. Fernandes, M. Härkönen, A. Harrington, R. Hasani, S. Karwa, Y. Khrustalev, M. Labonne, M. Lechner, V. Lechner, S. Lee, Z. Li, N. Loo, J. Marks, E. Mosca, S. J. Paech, P. Pak, R. N. Parnichkun, A. Quach, R. Rogers, D. Rus, N. Saxena, B. Schlager, T. Seyde, J. T. H. Smith, A. Tadimeti, and N. Tumma. Lfm2 technical report, 2025. URL https://arxiv.org/abs/2511.23404. P. Apertus, A. Hernández-Cano, A. Hägele, A. H. Huang, A. Romanou, A.-J. Solergibert, B. Pasztor, B. Messmer, D. Garbaya, E. F. Ďurech, I. Hakimi, J. G. Giraldo, M. Ismayilzada, N. Foroutan, S. Moalla, T. Chen, V. Sabolčec, Y. Xu, M. Aerni, B. AlKhamissi, I. A. Mariñas, M. H. Amani, M. Ansaripour, I. Badanin, H. Benoit, E. Boros, N. Browning, F. Bösch, M. Böther, N. Canova, C. Challier, C. Charmillot, J. Coles, J. Deriu, A. Devos, L. Drescher, D. Dzenhaliou, M. Ehrmann, D. Fan, S. Fan, S. Gao, M. Gila, M. Grandury, D. Hashemi, A. Hoyle, J. Jiang, M. Klein, A. Kucharavy, A. Kucherenko, F. Lübeck, R. Machacek, T. Manitaras, A. Marfurt, K. Matoba, S. Matrenok, H. Mendonça, F. R. Mohamed, S. Montariol, L. Mouchel, S. Najem-Meyer, J. Ni, G. Oliva, M. Pagliardini, E. Palme, A. Panferov, L. Paoletti, M. Passerini, I. Pavlov, A. Poiroux, K. Ponkshe, N. Ranchin, J. Rando, M. Sauser, J. Saydaliev, M. A. Sayfiddinov, M. Schneider, S. Schuppli, M. Scialanga, A. Semenov, K. Shridhar, R. Singhal, A. Sotnikova, A. Sternfeld, A. K. Tarun, P. Teiletche, J. Vamvas, X. Yao, H. Zhao, A. Ilic, A. Klimovic, A. Krause, C. Gulcehre, D. Rosenthal, E. Ash, F. Tramèr, J. VandeVondele, L. Veraldi, M. Rajman, T. Schulthess, T. Hoefler, A. Bosselut, M. Jaggi, and I. Schlag. Apertus: Democratizing open and compliant llms for global language environments, 2025. URL https://arxiv.org/abs/2509.14233. S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu. Qwen3-vl technical report, 2025. URL https://arxiv.org/abs/2511.21631. E. Bakouch, L. Ben Allal, A. Lozhkov, N. Tazi, L. Tunstall, C. M. Patiño, E. Beeching, A. Roucher, A. J. Reedi, Q. Gallouédec, K. Rasul, N. Habib, C. Fourrier, H. Kydlicek, G. Penedo, H. Larcher, M. Morlon, V. Srivastav, J. Lochner, X.-S. Nguyen, C. Raffel, L. von Werra, and T. Wolf. SmolLM3: smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3, 2025. L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. N. Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.44. URL https://aclanthology.org/2024.acl-long.44/. P. Banerjee, K. K. Pal, A. Mitra, and C. Baral. Careful selection of knowledge to solve open book question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6120–6129, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1615. URL https://aclanthology.org/P19-1615. A. Bercovich et al. Llama-nemotron: Efficient reasoning models, 2025. URL https://arxiv.org/abs/ 2505.00949. 26

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641. T. A. Chang et al. Global piqa: Evaluating physical commonsense reasoning across 100+ languages and cultures, 2025. URL https://arxiv.org/abs/2510.24081. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction-finetuned language models, 2022. URL https://arxiv.org/abs/2210.11416. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https:// aclanthology.org/N19-1300. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/ 1803.05457. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. G. Comanici et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.747. URL https://aclanthology.org/2020.acl-main.747. D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.70. URL https://aclanthology.org/2024.acl-long.70/. Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks, 2017. URL https://arxiv.org/abs/1612.08083. L. Dou, Q. Liu, F. Zhou, C. Chen, Z. Wang, Z. Jin, Z. Liu, T. Zhu, C. Du, P. Yang, H. Wang, J. Liu, Y. Zhao, X. Feng, X. Mao, M. T. Yeung, K. Pipatanakul, F. Koto, M. S. Thu, H. Kydlíček, Z. Liu, Q. Lin, S. Sripaisarnmongkol, K. Sae-Khow, N. Thongchim, T. Konkaew, N. Borijindargoon, A. Dao, M. Maneegard, P. Artkaew, Z.-X. Yong, Q. Nguyen, W. Phatthiyaphaibun, H. H. Tran, M. Zhang, S. Chen, T. Pang, C. Du, X. Wan, W. Lu, and M. Lin. Sailor2: Sailing in south-east asia with inclusive multilingual llms, 2025. URL https://arxiv.org/abs/2502.12982.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

M. Freitag, N. Mathur, D. Deutsch, C.-K. Lo, E. Avramidis, R. Rei, B. Thompson, F. Blain, T. Kocmi, J. Wang, D. I. Adelani, M. Buchicchio, C. Zerva, and A. Lavie. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In B. Haddow, T. Kocmi, P. Koehn, and C. Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.wmt-1.2. URL https://aclanthology.org/2024.wmt-1.2/. A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao, X. Du, M. R. Ghasemi Madani, C. Barale, R. McHardy, J. Harris, J. Kaddour, E. Van Krieken, and P. Minervini. Are we done with MMLU? In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5069–5096, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.262. URL https://aclanthology.org/2025.naacl-long.262/. A. Grattafiori et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. A. Gudibande, E. Wallace, C. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine, and D. Song. The false promise of imitating proprietary llms, 2023. URL https://arxiv.org/abs/2305.15717. E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, A. Suvarna, B. Feuer, L. Chen, Z. Khan, E. Frankel, S. Grover, C. Choi, N. Muennighoff, S. Su, W. Zhao, J. Yang, S. Pimpalgaonkar, K. Sharma, C. C.-J. Ji, Y. Deng, S. Pratt, V. Ramanujan, J. Saad-Falcon, J. Li, A. Dave, A. Albalak, K. Arora, B. Wulfe, C. Hegde, G. Durrett, S. Oh, M. Bansal, S. Gabriel, A. Grover, K.-W. Chang, V. Shankar, A. Gokaslan, M. A. Merrill, T. Hashimoto, Y. Choi, J. Jitsev, R. Heckel, M. Sathiamoorthy, A. G. Dimakis, and L. Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https://arxiv.org/abs/2506.04178. N. Habib, C. Fourrier, H. Kydlíček, T. Wolf, and L. Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/lighteval. E. He, A. Khattar, R. Prenger, V. Korthikanti, Z. Yan, T. Liu, S. Fan, A. Aithal, M. Shoeybi, and B. Catanzaro. Upcycling large language models into mixture of experts, 2025a. URL https: //arxiv.org/abs/2410.07524. J. He, J. Liu, C. Y. Liu, R. Yan, C. Wang, P. Cheng, X. Zhang, F. Zhang, J. Xu, W. Shen, S. Li, L. Zeng, T. Wei, C. Cheng, B. An, Y. Liu, and Y. Zhou. Skywork open reasoner 1 technical report, 2025b. URL https://arxiv.org/abs/2505.22312. Y. He, A. Benhaim, B. Patra, P. Vaddamanu, S. Ahuja, P. Chopra, V. Chaudhary, H. Zhao, and X. Song. Scaling laws for multilingual language models. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 4257–4273, Vienna, Austria, July 2025c. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.221. URL https://aclanthology.org/2025.findings-acl.221/. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxiv.org/abs/2103. 03874.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

S. Hu, Y. Tu, X. Han, G. Cui, C. He, W. Zhao, X. Long, Z. Zheng, Y. Fang, Y. Huang, X. Zhang, Z. L. Thai, C. Wang, Y. Yao, C. Zhao, J. Zhou, J. Cai, Z. Zhai, N. Ding, C. Jia, G. Zeng, dahai li, Z. Liu, and M. Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=3X2L2TFr0f. X. Huang, W. Zhu, H. Hu, C. He, L. Li, S. Huang, and F. Yuan. Benchmax: A comprehensive multilingual evaluation suite for large language models. arXiv preprint arXiv:2502.07346, 2025. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, jiayi lei, Y. Fu, M. Sun, and J. He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=fOrm2rGX2r. IBM Research. Granite 4.0 language models. 0-language-models, 2025. Accessed: 2025-10-01.

https://github.com/ibm-granite/granite-4.

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024. URL https://arxiv.org/abs/2401.04088. F. Jiang, H. Yu, G. Chung, and T. Cohn. Franken-adapter: Cross-lingual adaptation of llms by embedding surgery, 2025. URL https://arxiv.org/abs/2502.08037. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.560. URL https://aclanthology.org/2020.acl-main. 560. A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/ forum?id=T5nUQDrM4u. F. Koto. Cracking the code: Multi-domain LLM evaluation on real-world professional exams in Indonesia. In W. Chen, Y. Yang, M. Kachuee, and X.-Y. Fu, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 938–948, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8-89176-194-0. doi: 10.18653/v1/2025.naacl-industry.69. URL https://aclanthology.org/2025.naacl-industry.69/. F. Koto, N. Aisyah, H. Li, and T. Baldwin. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023.emnlp-main.760. URL https://aclanthology.org/2023.emnlp-main.760/. F. Koto, H. Li, S. Shatnawi, J. Doughman, A. Sadallah, A. Alraeesi, K. Almubarak, Z. Alyafeai, N. Sengupta, S. Shehata, N. Habash, P. Nakov, and T. Baldwin. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand, Aug. 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.334. URL https://aclanthology.org/2024.findings-acl.334/. 29

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

F. Koto, R. Mahendra, N. Aisyah, and T. Baldwin. IndoCulture: Exploring geographically influenced cultural commonsense reasoning across eleven Indonesian provinces. Transactions of the Association for Computational Linguistics, 12:1703–1719, 2024b. doi: 10.1162/tacl_a_00726. URL https: //aclanthology.org/2024.tacl-1.92/. V. Lai, C. Nguyen, N. Ngo, T. Nguyen, F. Dernoncourt, R. Rossi, and T. Nguyen. Okapi: Instructiontuned large language models in multiple languages with reinforcement learning from human feedback. In Y. Feng and E. Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-demo.28. URL https://aclanthology.org/2023.emnlp-demo.28/. H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285, Bangkok, Thailand, Aug. 2024a. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.findings-acl.671. URL https://aclanthology.org/2024.findings-acl.671/. X. Li, P. Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer, J. E. Weston, and M. Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=1oijHJBRsT. A. H. Liu, K. Khandelwal, S. Subramanian, V. Jouault, A. Rastogi, A. Sadé, A. Jeffares, A. Jiang, A. Cahill, A. Gavaudan, A. Sablayrolles, A. Héliou, A. You, A. Ehrenberg, A. Lo, A. Eliseev, A. Calvi, A. Sooriyarachchi, B. Bout, B. Rozière, B. D. Monicault, C. Lanfranchi, C. Barreau, C. Courtot, D. Grattarola, D. Dabert, D. de las Casas, E. Chane-Sane, F. Ahmed, G. Berrada, G. Ecrepont, G. Guinet, G. Novikov, G. Kunsch, G. Lample, G. Martin, G. Gupta, J. Ludziejewski, J. Rute, J. Studnia, J. Amar, J. Delas, J. S. Roberts, K. Yadav, K. Chandu, K. Jain, L. Aitchison, L. Fainsin, L. Blier, L. Zhao, L. Martin, L. Saulnier, L. Gao, M. Buyl, M. Jennings, M. Pellat, M. Prins, M. Poirée, M. Guillaumin, M. Dinot, M. Futeral, M. Darrin, M. Augustin, M. Chiquier, M. Schimpf, N. Grinsztajn, N. Gupta, N. Raghuraman, O. Bousquet, O. Duchenne, P. Wang, P. von Platen, P. Jacob, P. Wambergue, P. Kurylowicz, P. R. Muddireddy, P. Chagniot, P. Stock, P. Agrawal, Q. Torroba, R. Sauvestre, R. Soletskyi, R. Menneer, S. Vaze, S. Barry, S. Gandhi, S. Waghjale, S. Gandhi, S. Ghosh, S. Mishra, S. Aithal, S. Antoniak, T. L. Scao, T. Cachet, T. S. Sorg, T. Lavril, T. N. Saada, T. Chabal, T. Foubert, T. Robert, T. Wang, T. Lawson, T. Bewley, T. Bewley, T. Edwards, U. Jamil, U. Tomasini, V. Nemychnikova, V. Phung, V. Maladière, V. Richard, W. Bouaziz, W.-D. Li, W. Marshall, X. Li, X. Yang, Y. E. Ouahidi, Y. Wang, Y. Tang, and Z. Ramzi. Ministral 3, 2026. URL https://arxiv.org/abs/2601.08584. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu. K. Lu and T. M. Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. R. K. Mahabadi, S. Satheesh, S. Prabhumoye, M. Patwary, M. Shoeybi, and B. Catanzaro. Nemotroncc-math: A 133 billion-token-scale high quality math pretraining dataset, 2025. URL https: //arxiv.org/abs/2508.15096. P. Maini, S. Seto, R. Bai, D. Grangier, Y. Zhang, and N. Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings 30

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14044–14072, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.757. URL https://aclanthology.org/2024.acl-long.757/. P. H. Martins, P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski, P. Colombo, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins. Eurollm: Multilingual language models for europe, 2024. URL https://arxiv.org/abs/2409.16235. B. Messmer, V. Sabolčec, and M. Jaggi. Enhancing multilingual llm pretraining with model-based data selection. arXiv, 2025. URL https://arxiv.org/abs/2502.10361. N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, E. P. Walsh, O. Tafjord, N. Lambert, Y. Gu, S. Arora, A. Bhagia, D. Schwenk, D. Wadden, A. Wettig, B. Hui, T. Dettmers, D. Kiela, A. Farhadi, N. A. Smith, P. W. Koh, A. Singh, and H. Hajishirzi. OLMoe: Open mixture-ofexperts language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xXTkbTBmqq. T. Nakamura, T. Akiba, K. Fujii, Y. Oda, R. Yokota, and J. Suzuki. Drop-upcycling: Training sparse mixture of experts with partial re-initialization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=gx1wHnf5Vp. D. Nathawani, I. Gitman, S. Majumdar, E. Bakhturina, A. Sunil Mahabaleshwarkar, , J. Zhang, and J. Polak Scowcroft. Nemotron-Post-Training-Dataset-v1, July 2025. URL https://huggingface.co/ datasets/nvidia/Nemotron-Post-Training-Dataset-v1. H. H. Nigatu, A. L. Tonja, B. Rosman, T. Solorio, and M. Choudhury. The Zeno’s paradox of ‘lowresource’ languages. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17753–17774, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. emnlp-main.983. URL https://aclanthology.org/2024.emnlp-main.983/. NVIDIA. Nemotron 3 Nano: Open, efficient mixture-of-experts hybrid Mamba-Transformer model for Agentic reasoning, 2025. URL https://research.nvidia.com/labs/nemotron/files/ NVIDIA-Nemotron-3-Nano-Technical-Report.pdf. Technical report. NVIDIA et al. Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models, 2025a. URL https://arxiv.org/abs/2504.03624. NVIDIA et al. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025b. URL https://arxiv.org/abs/2508.14444. T. Olmo, :, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi. Olmo 3, 2025. URL https://arxiv.org/abs/2512.13961. T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi,

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

N. Dziri, A. Ettinger, M. Guerquin, D. Heineman, H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, J. Morrison, T. Murray, C. Nam, J. Poznanski, V. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg, D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi. 2 olmo 2 furious, 2025. URL https://arxiv.org/abs/2501.00656. D. Ong and P. Limkonchotiwat. SEA-LION (Southeast Asian languages in one network): A family of Southeast Asian language models. In L. Tan, D. Milajevs, G. Chauhan, J. Gwinnup, and E. Rippeth, editors, Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 245–245, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.nlposs-1.26. URL https://aclanthology.org/2023.nlposs-1.26/. G. Penedo, H. Kydlíček, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V. Werra, and T. Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https: //arxiv.org/abs/2406.17557. G. Penedo, H. Kydlíček, V. Sabolčec, B. Messmer, N. Foroutan, A. H. Kargaran, C. Raffel, M. Jaggi, L. V. Werra, and T. Wolf. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=jnRBe6zatP. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, S. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J.-B. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. Hechtman, L. Weidinger, I. Gabriel, W. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher, 2022. URL https: //arxiv.org/abs/2112.11446. M. M. Ramos, D. M. Alves, H. Gisserot-Boukhlef, J. Alves, P. H. Martins, P. Fernandes, J. Pombal, N. M. Guerreiro, R. Rei, N. Boizard, A. Farajian, M. Klimaszewski, J. G. C. de Souza, B. Haddow, F. Yvon, P. Colombo, A. Birch, and A. F. T. Martins. Eurollm-22b: Technical report, 2026. URL https://arxiv.org/abs/2602.05879. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022. A. Romanou, N. Foroutan, A. Sotnikova, S. H. Nelaturu, S. Singh, R. Maheshwary, M. Altomare, Z. Chen, M. A. Haggag, S. A, A. Amayuelas, A. H. Amirudin, D. Boiko, M. Chang, J. Chim, G. Cohen, A. K. Dalmia, A. Diress, S. Duwal, D. Dzenhaliou, D. F. E. Florez, F. Farestam, J. M. Imperial, S. B. Islam, P. Isotalo, M. Jabbarishiviari, B. F. Karlsson, E. Khalilov, C. Klamm, F. Koto, D. Krzemiński, G. A. de Melo, S. Montariol, Y. Nan, J. Niklaus, J. Novikova, J. S. O. Ceron, D. Paul, E. Ploeger, J. Purbey, S. Rajwal, S. S. Ravi, S. Rydell, R. Santhosh, D. Sharma, M. P. Skenduli, A. S. Moakhar, B. soltani moakhar, A. K. Tarun, A. T. Wasi, T. O. Weerasinghe, S. Yilmaz, M. Zhang, I. Schlag, M. Fadaee, S. Hooker, and A. Bosselut. INCLUDE: Evaluating multilingual language understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=k3gCieTXeY.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019. A. R. Salamanca, D. Abagyan, D. D’souza, A. Khairi, D. Mora, S. Dash, V. Aryabumi, S. Rajaee, M. Mofakhami, A. Sahu, T. Euyang, B. Prince, M. Smith, H. Lin, A. Locatelli, S. Hooker, T. Kocmi, A. Gomez, I. Zhang, P. Blunsom, N. Frosst, J. Pineau, B. Ermis, A. Üstün, J. Kreutzer, and M. Fadaee. Tiny aya: Bridging scale and multilingual depth, 2026. URL https://arxiv.org/abs/2603.11510. M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 4463–4473, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. URL https://aclanthology.org/D19-1454. J. Schulman and T. M. Lab. Lora without regret. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250929. https://thinkingmachines.ai/blog/lora/. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/ 1701.06538. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=f R3wGCk-IXp. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/ abs/1909.08053. S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, S. Ruder, W.-Y. Ko, A. Bosselut, A. Oh, A. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.919. URL https://aclanthology.org/2025.acl-long.919/. V. Singh, L. Krauss, S. Jaghouar, M. Sirovatka, C. Goddard, F. Obied, J. M. Ong, J. Straube, Fern, A. Harley, C. Stewart, C. Kealty, M. Panahi, S. Kirsten, A. Deshpande, A. Vij, A. Bresnu, P. Veldurthi, R. Ravishankar, H. Bishnoi, D. Team, A. A. Team, P. I. Team, M. McQuade, J. Hagemann, and L. Atkins. Arcee trinity large technical report, 2026. URL https://arxiv.org/abs/2602.17004. D. Su, K. Kong, Y. Lin, J. Jennings, B. Norick, M. Kliegl, M. Patwary, M. Shoeybi, and B. Catanzaro. Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset. In

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2459–2475, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.123. URL https://aclanthology.org/2025.acl-long.123/. J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl. 824. URL https://aclanthology.org/2023.findings-acl.824. A. Talmor, J. Herzig, N. Lourie, and J. Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421. G. Team et al. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786. K. Team et al. Kimi k2: Open agentic intelligence, 2026. URL https://arxiv.org/abs/2507.20534. N. Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, and J. Wang. No language left behind: Scaling human-centered machine translation, 2022. URL https://arxiv.org/abs/2207.04672. C. Tian, K. Chen, J. Liu, Z. Liu, Z. Zhang, and J. Zhou. Towards greater leverage: Scaling laws for efficient mixture-of-experts language models, 2025. URL https://arxiv.org/abs/2507.17702. M. Togmanov, N. Mukhituly, D. Turmakhan, J. Mansurov, M. Goloburda, A. Sakip, Z. Xie, Y. Wang, B. Syzdykov, N. Laiyk, A. F. Aji, E. Kochmar, P. Nakov, and F. Koto. KazMMLU: Evaluating language models on Kazakh, Russian, and regional knowledge of Kazakhstan. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14403–14416, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.701. URL https://aclanthology.org/2025.acl-long.701/. S. Toshniwal, W. Du, I. Moshkov, B. Kisacanin, A. Ayrapetyan, and I. Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024. A. Üstün, V. Aryabumi, Z. Yong, W.-Y. Ko, D. D’souza, G. Onilude, N. Bhandari, S. Singh, H.-L. Ooi, A. Kayid, F. Vargus, P. Blunsom, S. Longpre, N. Muennighoff, M. Fadaee, J. Kreutzer, and S. Hooker. Aya model: An instruction finetuned open-access multilingual language model. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.845. URL https://aclanthology.org/2024.acl-long.845/. 34

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762. D. Vrandečić and M. Krötzsch. Wikidata: a free collaborative knowledgebase. Commun. ACM, 57 (10):78–85, Sept. 2014. ISSN 0001-0782. doi: 10.1145/2629489. URL https://doi.org/10.1145/ 2629489. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id= y10DM6R2r3. Y. Wang, P. Zhang, J. Tang, H.-R. Wei, B. Yang, R. Wang, C. Sun, F. Sun, J. Zhang, J. Wu, Q. Cang, Y. Zhang, F. Huang, J. Lin, F. Huang, and J. Zhou. Polymath: Evaluating mathematical reasoning in multilingual contexts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=B1vCImy6yI. J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum?id=gEZrGCozdqR. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners, 2022b. URL https://arxiv.org/abs/2109.01652. W. Xuan, R. Yang, H. Qi, Q. Zeng, Y. Xiao, A. Feng, D. Liu, Y. Xing, J. Wang, F. Gao, J. Lu, Y. Jiang, H. Li, X. Li, K. Yu, R. Dong, S. Gu, Y. Li, X. Xie, F. Juefei-Xu, F. Khomh, O. Yoshie, Q. Chen, D. Teodoro, N. Liu, R. Goebel, L. Ma, E. Marrese-Taylor, S. Lu, Y. Iwasawa, Y. Matsuo, and I. Li. MMLU-ProX: A multilingual benchmark for advanced large language model evaluation. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1513–1532, Suzhou, China, Nov. 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.79. URL https://aclanthology.org/2025.emnlp-main.79/. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan. Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Z. Yang, Z. Liu, Y. Chen, W. Dai, B. Wang, S.-C. Lin, C. Lee, Y. Chen, D. Jiang, J. He, R. Pi, G. Lam, N. Lee, A. Bukharin, M. Shoeybi, B. Catanzaro, and W. Ping. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation, 2026. URL https://arxiv.org/abs/2603.19220.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W.-Y. Ma, Y.-Q. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025a. URL https://arxiv.org/abs/2503.14476. Y. Yu, Z. Dai, Z. Wang, W. Wang, R. Chen, and J. Pei. Opencsg chinese corpus: A series of high-quality chinese datasets for llm training, 2025b. URL https://arxiv.org/abs/2501.08197. A. Yüksel, A. Köksal, L. K. Senel, A. Korhonen, and H. Schuetze. TurkishMMLU: Measuring massive multitask language understanding in Turkish. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7035–7055, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-emnlp.413. URL https://aclanthology.org/2024.findings-emnlp.413/. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472. B. Zhang and R. Sennrich. Root mean square layer normalization, 2019. URL https://arxiv.org/abs/ 1910.07467. Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https://arxiv.org/abs/2506.05176. Y. Zhang, M. Konomi, C. Xypolopoulos, K. Divriotis, K. Skianis, G. Nikolentzos, G. Stamou, G. Shang, and M. Vazirgiannis. Greekmmlu: A native-sourced multitask benchmark for evaluating language models in greek, 2026. URL https://arxiv.org/abs/2602.05150. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In K. Duh, H. Gomez, and S. Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.findings-naacl.149. URL https://aclanthology.org/2024.findings-naacl.149/. F. Zhou, Z. Wang, N. Ranjan, Z. Cheng, L. Tang, G. He, Z. Liu, and E. P. Xing. Megamath: Pushing the limits of open math corpora. arXiv preprint arXiv:2504.02807, 2025. Preprint. Z. Zhu, C. Xie, X. Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.org/abs/2202.08906.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

A. Per-Benchmark Results across Pre-training Phases Benchmark (Metric)

Stage-1

Stage-2

Stage-3

Stage-4

66.1 63.0 35.6 37.5 55.6 80.1 53.1 73.0 55.5 74.5 59.1 44.2 79.2 47.9 66.3 59.4

72.8 69.4 42.7 40.6 63.7 82.5 57.4 77.6 59.0 74.9 61.3 44.6 79.9 50.6 75.0 63.5

73.5 68.9 44.8 41.9 62.0 82.7 57.4 77.5 58.0 76.1 61.5 44.8 80.7 49.0 75.9 63.7

72.8 68.8 45.3 41.9 65.1 82.4 56.3 77.4 57.7 74.2 61.5 44.6 81.1 49.4 76.4 63.7

52.4 51.5 29.2 63.4 51.4 39.1 22.3 30.2 19.8 26.5 61.0 40.6

59.5 59.3 34.2 70.5 55.2 42.3 25.6 34.5 23.8 30.9 73.1 46.3

62.6 60.5 38.2 80.1 58.2 44.8 31.7 40.0 27.7 34.4 75.1 50.3

64.2 62.0 39.2 79.8 58.6 45.4 32.3 40.1 28.1 34.4 75.6 52.1

Multilingual – Cultural & Regional INCLUDE (Acc) 51.5 64.9 Global-PIQA (Acc_norm) CMMLU (Acc) 58.6 C-Eval (Acc) 57.1 ArabicMMLU (Acc) 60.3 TurkishMMLU (Acc) 50.1 GreekMMLU (Acc) 41.8 KazakhMMLU (Acc) 47.9 IndoMMLU (Acc) 51.1 IndoCareer (Acc) 54.8 IndoCulture (Acc) 55.6 Average 54.0

56.7 66.6 63.8 62.9 65.8 60.1 51.6 54.3 56.3 60.1 60.0 59.8

61.0 70.6 64.2 63.5 66.6 61.2 68.1 60.5 59.1 60.3 60.5 63.2

61.7 72.3 68.0 66.0 67.1 62.7 70.3 62.6 59.9 61.5 62.3 65.0

Table 13 | Per-benchmark model performance across the four pre-training stages.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

B. Per-Benchmark Results across the Two-Stage Cascaded OPD Marco-Nano-Instruct

Marco-Mini-Instruct

Stage-1

Stage-2

Stage-1

Stage-2

English MMLU (Acc) MMLU-Redux (Acc) MMLU-Pro (Acc) AGIEval (Acc) GPQA-Diamond (Acc) GSM8K (EM) MATH (EM) Average

71.5 71.7 51.4 48.4 19.4 85.4 75.7 60.5

73.2 73.3 54.5 49.8 22.2 86.7 79.6 62.8

+1.7 +1.6 +3.1 +1.4 +2.8 +1.3 +3.9 +2.3

82.1 82.0 67.6 54.6 47.2 92.1 90.2 73.7

83.4 83.5 70.7 55.4 50.3 93.1 91.8 75.5

+1.3 +1.5 +3.1 +0.8 +3.1 +1.0 +1.6 +1.8

58.6 59.6 41.9 20.5 21.5 31.5 17.8 27.2 74.4 27.3 38.0

58.7 59.9 43.2 21.6 22.3 31.1 18.7 27.3 76.5 29.6 38.9

+0.1 +0.3 +1.3 +1.1 +0.8 −0.4 +0.9 +0.1 +2.1 +2.3 +0.9

71.8 72.4 57.2 39.8 29.9 36.9 26.3 31.9 85.8 41.4 49.3

73.3 73.7 61.2 41.8 30.6 36.8 26.8 31.3 87.4 44.7 50.8

+1.5 +1.3 +4.0 +2.0 +0.7 −0.1 +0.5 −0.6 +1.6 +3.3 +1.5

Multilingual – Cultural & Regional INCLUDE (Acc) 54.1 Global-PIQA (Acc) 69.3 CMMLU (Acc) 60.1 C-Eval (Acc) 60.2 ArabicMMLU (Acc) 56.8 TurkishMMLU (Acc) 58.7 GreekMMLU (Acc) 61.6 KazakhMMLU (Acc) 56.0 IndoMMLU (Acc) 56.4 IndoCareer (Acc) 53.8 IndoCulture (Acc) 57.8 58.6 Average

54.3 70.7 60.0 60.8 56.5 59.9 61.6 56.3 56.3 54.9 59.1 59.1

+0.2 +1.4 −0.1 +0.6 −0.3 +1.2 0.0 +0.3 −0.1 +1.1 +1.3 +0.5

64.5 83.7 74.4 74.1 66.7 73.3 71.0 67.4 64.7 63.3 66.0 69.9

65.6 84.2 75.3 75.4 67.8 74.7 72.5 68.8 65.7 64.4 67.1 71.0

+1.1 +0.5 +0.9 +1.3 +1.1 +1.4 +1.5 +1.4 +1.0 +1.1 +1.1 +1.1

Table 14 | Per-benchmark performance progression across the two-stage cascaded OPD. The Δ indicates the absolute improvement from Stage-1 (utilizing the 30B-A3B teacher) to Stage-2 (transitioning to the 80B-A3B teacher) across each benchmark.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

C. Language Clustering through Expert Activation Patterns To further corroborate that expert routing in Marco-MoE encodes linguistically meaningful structure, we perform agglomerative hierarchical clustering based on the Language-Expert Signatures shown in Figure 9. Specifically, we treat each language’s flattened activation vector M𝐿 as a point in expert-activation space, adopt 1 − 𝜌 (M𝐿𝑖 , M𝐿 𝑗 ) (with 𝜌 the Pearson correlation coefficient) as the pairwise distance between language 𝐿𝑖 and 𝐿 𝑗 , and apply average-linkage clustering to construct the dendrogram shown in Figure 10. The resulting hierarchy tightly mirrors established linguistic taxonomies: Romance, Slavic, Germanic, and East Asian languages each coalesce into compact subtrees at small merge distances, while at deeper levels phylogenetically related branches further converge into broader super-clusters that echo the Indo-European macro-family. Languages with isolated typology or unique scripts, in contrast, attach to the tree only at large distances, reflecting the router’s allocation of dedicated expert subsets to accommodate their distinctive morphology. This unsupervised rediscovery of language families offers mechanistic evidence that Marco-MoE has learned to exploit cross-lingual regularities in a manner that mirrors human-curated linguistic phylogenies, promoting positive transfer among related languages while isolating typologically distant ones from mutual interference.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Hierarchical Clustering of Languages via Expert Activation 1.0 Pearson Correlation

0.8 0.6 0.4 0.2 0.0 Romanian Polish Czech Russian Ukrainian German Dutch Greek Bengali Nepali Korean Japanese Chinese Thai Turkish Azerbaijani Kazakh Hungarian French Italian Spanish Portuguese Indonesian Malay English Vietnamese Hebrew Arabic

Ro ma

nia n Po lish Cz ec Ru h s Uk sian rai ni Ge an rm an Du tch Gr ee Be k ng al Ne i pa Ko li re Jap an an e Ch se ine se

Th a Tur i Az kish erb aij an Ka i Hu zakh ng ari an Fre nc h Ita lia Sp n Po anish rtu g Ind uese on esi an Ma lay E Vie nglis tna h me s He e bre w Ara bic Urd u

Urdu

Figure 10 | Hierarchical clustering of expert activation patterns, demonstrating that the model’s neural routing trajectories tightly mirror established linguistic family structures. The emergence of distinct blocks for Romance, Slavic, Germanic, and East Asian groups illustrates how the MoE architecture naturally discovers and exploits cross-lingual commonalities.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

D. Synthetic Data Generation Prompts Translation Prompt Translate the following text into [Language], ensuring that all LaTeX symbols, code snippets, and any related formatting structures remain unchanged and intact. Only translate the non-code and non-symbol portions of the text, preserving all technical formatting exactly as presented. Pay careful attention to maintaining the original layout and ensuring the translated text integrates seamlessly with non-translated elements. [Document]

Regional Web Document Annotation Prompt System Prompt: You are given a [Document] from the web. Your goal is to: [Document]: {} Task 1: Training Value Score (1–10) Analyze the content and assign a score based on quality and trustworthiness for LLM training: 1. 10 (Pristine): Factually impeccable, encyclopedic (e.g., peer-reviewed papers). 2. 9 (Excellent): Expert-level, well-researched (e.g., reputable news). 3. 8 (Very Good): Reliable, solid blog posts or documentation. 4. 7 (Good): Standard "how-to" guides, clear information. 5. 6 (Acceptable): Average, coherent but superficial (e.g., forum posts). 6. 5 (Borderline): Mixed quality, requires heavy cleaning. 7. 4 (Low Quality): Problematic, high fluff, questionable facts. 8. 3 (Very Low Quality): SEO spam, keyword stuffing, machine-generated. 9. 2 (Harmful/Junk): Hate speech, dangerous theories, malicious code. 10. 1 (Toxic): Unintelligible, entirely devoid of value. Task 2: Subject Classification Subject Classification: Assign a subject/category to the document based on its content. Choose from the following categories: - Accounting - Agriculture - Anthropology - Architecture and Design - Arts & Humanities - Biology Business administration - Business ethics - Business - Chemistry - Computer Science - Culturology Earth science - Economics - Education - Engineering - Environmental studies and forestry - Family and consumer science - Finance - Geography - Health - History - Human physical performance and recreation - Industrial and labor relations - International trade - Journalism - media studies - and communication Language - Law - Library and museum studies - Literature - Logic - Management - Marketing - Math Medicine - Military Sciences - Multiple exams - Performing arts - Philosophy - Physics - Political sciences - Psychology - Public Administration - Public Policy - Qualimetry - Religious studies - Risk management and insurance - Social work - Sociology - STEM - Transportation - Visual Arts - Driving License - Marine License - Medical License - Professional Certifications - Others Task 3: Country/Region Classification Decide if the content is culturally specific (e.g., Japan) or "No specific country." Output Format: 1. Text Quality Score (1-10): 2. Subject Category: 3. Country/Region: Only generate the final result without additional descriptions.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Regional MCQs Generation Prompt Write a question on [country], more specifically on [sub_topic], with up to four choices. Desired difficulty level: [difficulty]. The required language: [language] - For questions marked "intermediate", the question should require applying basic concepts independently or in simple combinations. The relationships between concepts should be straightforward and directly tied to the problem. Solutions should be accessible with a foundational understanding and methodical effort. - For questions marked "complex", the question should require identifying and combining multiple clear but non-obvious connections between concepts. The relationships may span different areas within the topic, requiring structured reasoning and careful application of techniques. Focus on recognizing patterns and synthesizing information rather than straightforward computation. - For questions marked "hard", the question should involve intricate or abstract relationships between concepts. Solving them should demand advanced reasoning skills, mastery of topic-specific techniques, and the ability to deal with ambiguities or edge cases. Solutions often require navigating multiple layers of logic or constraints. - For questions marked "brutal", the question should push problem-solving to the limit, requiring mastery of nuanced relationships across diverse concepts. It should demand precise reasoning, significant abstraction, and fluency in advanced techniques. Problems may include nested or multi-step dependencies, where overlooking a detail can derail the solution. - For questions marked "borderline unsolvable", the question should challenge even experts, involving exceptionally intricate or unconventional problem structures. It should combine deep abstraction, subtle constraints, and highly interconnected concepts. While solutions exist, they often require extensive expertise and innovative approaches. Avoid speculative or unsolvable topics (e.g., "Does God exist?" or "How can time travel be achieved?"). Ensure that one of the four choices is correct. Your question must include a valid answer within the given choices; you should not specify which one is correct. Your response should not contain any headers, explanations, or your answer to the question. Question: [Question] (A): [Option A] (B): [Option B] (C): [Option C] (D): [Option D]

Regional MCQ Example Question: 在《素问·痹论》“痛者，寒气多也，有寒故痛也”的理论基础上，若患者表现为关节剧烈疼痛、如锥刺刀割，且痛处固定不移，遇寒则痛甚，得热则稍减，局部皮色不红、触之不温，甚至出现明显的“筋脉拘急”导致关节屈伸不利。从寒邪“收引凝滞”的病理特性及其对“营卫气血运行受阻”的深度逻辑推导，下列关于“痛痹（寒痹）”发病机制的描述最准确且全面的是： A: 寒邪凝滞导致血流减慢，其深层逻辑在于寒邪易于诱发“血瘀化热”的病理转型。因此，痛痹的治疗重点应在于清热化瘀，以缓解因热扰筋脉所致的局部红肿与活动受限。 B: 寒为阴邪，极易损伤人体阳气，导致卫外之气涣散，使得风邪乘虚深袭经络。其病机关键在于风寒相搏引起的“气行游走”，导致关节局部出现一过性的拘挛与酸楚感。 C: 寒邪性主收引，入经则使经脉绌急，入络则使血行凝滞；其剧烈疼痛的根源不仅在于“凝则不通”，更在于寒邪遏阻阳气，使营卫之气在受阻部位产生激烈的“郁遏交争”与脉络极度挛缩。 D: 寒邪侵袭导致阳气受损，其病理重点在于肾阳虚衰而不能温煦经脉，导致水饮内停并流注于骨节。临床辨证核心应聚焦于“水饮溢于经络”所致的关节肿胀与沉重感。 Answer: 根据《素问·痹论》的理论及中医病理逻辑，对“痛痹（寒痹）”的发病机制进行如下推理： 1. 寒邪的特性分析： (a) 收引 (Constriction)：寒邪侵袭，导致经络、筋脉缩急。正如文中所述“筋脉拘急”，这是因为寒性收缩，使脉道、筋膜挛缩，从而出现关节屈伸不利。

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

(b) 凝滞 (Stagnation)：寒则气血凝涩不通，“不通则痛”。寒邪入于脉络，使血流减慢甚至瘀阻，形成固定不移的剧烈疼痛。 2. 对营卫气血的影响： (a) 营卫受阻：营行脉中，卫行脉外。寒邪侵入后，卫阳被遏，营血凝滞。 (b) 郁遏交争：剧烈疼痛（如锥刺刀割）的深层逻辑在于，寒邪作为阴邪遏制了局部阳气的运行，阳气试图冲破寒邪的阻碍，在受阻部位产生剧烈的搏击、交争（即“郁遏交争”），加之脉络极度挛缩，导致疼痛感远重于其他类型的痹症。 3. 选项辨析： (a) A项：误导在于“血瘀化热”和“清热化瘀”。虽然长期瘀滞可能化热，但题目明确描述为“皮色不红、触之不温、遇寒痛甚”，属于典型的寒证，治疗应温经散寒，而非清热。 (b) B项：误导在于“风邪深袭”和“气行游走”。“游走”是风痹（行痹）的特点，而本题强调“痛处固定不移”，是寒邪凝滞的表现。 (c) C项：精准地抓住了寒邪“收引”导致“经脉绌急”、“凝滞”导致“血行不畅”的要点。同时，它深入解释了剧烈疼痛的根源在于寒邪遏阻阳气，使营卫在局部产生激烈的郁遏交争，符合“痛者，寒气多也”的深度病理逻辑。 (d) D项：误导在于“水饮内停”和“关节肿胀沉重”。这描述的是湿痹（着痹）的特征，而非痛痹。 C

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

E. Base Model Results by Language Language GlobalMMLU MMMLU MMLU-ProX-Lite BELEBELE mHellaSwag mARC-Challenge FLORES-200 WMT24++ MGSM INCLUDE Global-PIQA eng_Latn zho_Hans por_Latn fra_Latn rus_Cyrl deu_Latn ita_Latn spa_Latn jpn_Jpan kor_Hang arb_Arab ind_Latn vie_Latn ron_Latn tha_Thai zsm_Latn ces_Latn nld_Latn ukr_Cyrl pol_Latn tur_Latn ell_Grek hun_Latn heb_Hebr ben_Beng npi_Deva urd_Arab kaz_Cyrl azj_Latn

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗

✗ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✓ ✓ ✗ ✗ ✗

✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗

✓ ✓ ✗ ✓ ✓ ✓ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗

✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Table 15 | Languages included in each of the evaluated benchmark. Language arb_Arab ben_Beng ces_Latn deu_Latn eng_Latn fra_Latn heb_Hebr ind_Latn ita_Latn jpn_Jpan kor_Hang nld_Latn npi_Deva pol_Latn por_Latn ron_Latn rus_Cyrl spa_Latn tur_Latn ukr_Cyrl vie_Latn zho_Hans zsm_Latn Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 45.5 39.4 48.2 52.6 61.6 54.0 41.2 51.8 53.9 49.7 47.5 50.7 37.5 48.5 55.0 50.1 50.8 55.0 46.0 47.1 49.9 56.0 48.1 49.6

36.3 33.0 42.2 47.5 61.7 51.0 35.5 44.3 48.2 43.3 40.0 46.3 32.4 42.5 50.1 45.4 44.5 50.5 38.1 40.5 39.8 49.4 41.4 43.6

51.6 47.2 56.5 58.1 65.2 59.6 48.4 55.8 59.0 53.7 52.2 58.0 47.8 53.7 59.7 56.4 55.9 60.1 49.9 53.7 49.7 54.8 53.1 54.8

49.0 43.1 51.6 54.7 62.9 55.6 47.2 53.3 55.6 51.4 49.9 54.3 43.6 51.8 56.3 53.4 53.3 56.3 48.3 50.5 51.7 54.1 52.7 52.2

39.3 34.0 42.9 45.8 54.6 47.1 35.7 44.9 47.0 41.1 40.2 45.6 34.5 42.9 48.0 44.7 43.5 48.2 41.6 41.8 43.7 44.1 42.6 43.2

49.5 31.6 41.8 53.3 60.1 54.4 33.5 45.8 54.8 48.0 46.9 46.3 35.9 42.3 54.9 45.1 51.4 55.3 38.8 43.1 46.8 50.2 43.3 46.7

48.1 46.1 51.3 52.8 58.7 53.3 46.7 52.0 54.1 48.6 48.1 52.6 45.2 50.7 53.6 52.4 51.4 54.3 49.1 49.9 49.8 50.8 49.8 50.8

48.5 44.9 50.2 51.6 56.6 51.8 47.6 51.1 52.5 49.0 49.6 50.4 45.5 48.8 52.8 50.4 49.7 53.0 48.2 48.0 48.9 50.8 49.2 50.0

58.2 50.6 61.8 65.3 71.8 66.3 51.9 63.9 66.0 60.7 59.3 63.9 49.7 60.9 66.2 63.5 64.1 66.9 57.7 60.7 61.1 66.0 60.6 61.6

44.0 38.1 51.7 57.6 68.5 60.6 43.1 52.8 57.7 53.2 49.9 54.6 39.9 51.3 60.3 53.8 55.3 60.7 46.9 50.0 49.5 58.6 50.6 52.6

62.9 55.6 64.4 64.9 71.4 66.1 60.8 65.6 66.8 63.4 61.8 66.4 54.7 65.0 66.8 65.3 66.4 67.5 62.5 64.7 63.9 65.1 64.6 64.2

Table 16 | Per-language model performance comparison on GlobalMMLU.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Language arb_Arab ben_Beng deu_Latn fra_Latn ind_Latn ita_Latn jpn_Jpan kor_Hang por_Latn spa_Latn zho_Hans Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 42.9 37.3 50.5 51.1 49.2 51.5 47.4 45.8 52.5 52.1 54.3 48.6

32.7 30.0 43.2 46.2 41.4 44.3 39.2 38.1 46.2 47.6 44.0 41.2

47.9 44.0 54.0 56.0 52.0 55.5 50.4 49.9 56.2 56.2 53.2 52.3

48.6 44.6 54.0 54.6 53.9 55.9 51.2 49.9 56.0 55.9 54.2 52.6

39.7 34.6 46.8 47.7 45.1 46.4 41.6 41.2 47.7 48.3 44.7 44.0

47.1 32.2 50.4 51.8 43.9 51.2 46.3 44.9 51.9 52.8 47.9 47.3

43.7 39.9 49.7 50.4 49.1 50.5 42.7 45.7 51.1 52.0 47.2 47.5

42.8 37.9 46.1 45.8 45.2 46.0 44.0 43.6 46.5 46.1 45.7 44.5

53.7 46.3 62.0 62.8 60.3 63.2 58.3 55.3 61.9 63.9 64.0 59.2

41.1 36.8 53.1 57.6 50.8 54.7 49.0 46.3 56.9 57.3 56.1 50.9

60.3 55.1 63.9 63.3 62.4 64.1 61.2 60.0 63.8 65.1 62.7 62.0

Table 17 | Per-language model performance comparison on MMMLU. Language arb_Arab azj_Latn ben_Beng ces_Latn deu_Latn ell_Grek eng_Latn fra_Latn heb_Hebr hun_Latn ind_Latn ita_Latn jpn_Jpan kaz_Cyrl kor_Hang nld_Latn npi_Deva pol_Latn por_Latn ron_Latn rus_Cyrl spa_Latn tha_Thai tur_Latn ukr_Cyrl urd_Arab vie_Latn zho_Hans zsm_Latn Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 69.1 55.7 52.7 68.9 73.1 66.6 81.4 76.3 62.0 63.3 71.9 73.7 69.6 50.9 71.4 69.3 48.1 68.4 75.1 69.1 73.9 74.6 66.0 64.7 67.9 53.7 71.6 79.9 70.1 67.6

52.2 37.3 38.0 55.2 64.8 43.2 76.9 66.9 44.7 48.1 59.7 63.6 55.7 34.2 56.9 60.3 37.4 53.1 68.0 60.0 65.4 67.8 44.3 50.1 52.0 39.4 55.0 71.1 58.7 54.5

63.4 47.9 54.4 62.6 66.1 61.3 75.1 68.8 57.7 56.1 60.6 66.8 59.7 45.0 65.4 64.1 51.6 56.1 69.4 63.3 68.9 68.4 50.6 56.4 62.7 55.1 64.7 72.7 61.0 61.2

73.2 65.3 63.8 78.7 80.3 75.0 81.9 79.3 71.2 71.6 75.1 78.0 69.8 61.6 74.4 77.0 60.2 74.4 77.9 79.2 79.7 77.2 70.1 72.0 75.2 65.6 75.9 79.2 76.8 73.8

60.6 45.1 46.8 61.0 64.6 62.0 74.8 68.2 49.2 59.2 62.2 67.2 58.1 42.6 63.9 64.1 43.6 59.1 68.4 60.6 67.2 68.8 57.9 60.6 61.3 46.3 65.2 70.9 63.7 60.1

62.9 35.6 33.2 52.8 64.3 63.2 71.0 63.8 44.4 42.4 57.2 62.7 55.4 32.1 61.4 54.6 36.4 51.9 66.6 55.1 65.3 66.3 58.4 44.9 55.4 39.3 57.8 63.7 55.9 54.3

66.3 55.4 58.0 66.6 69.1 68.6 74.8 70.6 64.3 64.6 67.0 68.0 63.2 56.1 68.4 68.6 56.7 64.9 71.2 68.7 70.4 68.6 62.0 65.7 69.3 56.1 66.0 68.4 68.0 65.7

63.8 44.2 60.7 67.7 71.0 69.7 82.2 72.1 65.3 61.7 68.2 65.7 65.6 33.7 69.1 69.7 58.9 63.0 70.4 67.0 71.0 68.6 65.3 64.7 67.4 56.1 68.8 76.6 67.4 65.4

81.8 69.6 70.1 83.2 87.0 80.2 89.2 87.2 74.3 79.9 83.1 85.3 79.0 67.0 83.6 84.0 67.1 83.3 84.0 84.0 85.7 85.2 78.2 77.3 83.9 72.3 83.8 86.6 81.8 80.6

69.2 48.3 53.0 69.4 76.0 68.1 84.3 77.3 62.6 61.2 70.1 73.7 73.3 39.9 74.9 71.6 50.4 67.1 75.2 74.1 75.7 75.8 60.8 62.1 68.3 56.1 71.6 80.3 70.1 67.6

82.9 73.7 71.7 82.4 83.9 82.2 87.1 84.2 78.2 78.2 79.2 80.6 76.1 71.7 80.1 81.7 70.6 80.6 84.1 83.4 82.8 82.9 76.1 76.8 82.4 74.0 80.9 82.6 82.2 79.8

Table 18 | Per-language model performance comparison on BELEBELE. Language ben_Beng deu_Latn eng_Latn fra_Latn jpn_Jpan rus_Cyrl spa_Latn tha_Thai zho_Hans Avg

18.0 45.6 58.4 52.0 23.6 46.8 57.6 20.0 43.2 40.6

37.2 60.0 70.4 63.2 46.0 64.0 66.0 42.8 60.8 56.7

51.2 66.4 78.4 69.6 49.6 74.0 71.2 62.0 65.6 65.3

10.4 24.0 34.0 27.2 14.0 22.8 27.6 20.4 21.6 22.4

12.0 60.0 74.0 61.2 34.8 59.2 64.4 49.2 42.8 50.8

31.2 35.6 46.0 42.0 25.6 36.8 42.4 36.0 34.0 36.6

28.8 38.8 52.4 48.8 23.6 44.0 44.4 30.0 35.2 38.4

66.0 76.0 85.2 75.6 64.0 84.0 83.2 74.0 75.6 76.0

36.8 63.6 60.4 63.2 46.4 63.6 67.6 45.2 67.6 57.2

64.8 79.2 83.2 77.6 68.8 83.2 76.4 70.4 76.4 75.6

Table 19 | Per-language model performance comparison on MGSM. 45

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Language arb_Arab ben_Beng deu_Latn fra_Latn hun_Latn ind_Latn ita_Latn nld_Latn npi_Deva por_Latn ron_Latn rus_Cyrl spa_Latn ukr_Cyrl vie_Latn zho_Hans Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 39.7 30.4 46.1 51.3 35.8 45.9 48.9 44.1 30.2 50.8 41.2 46.8 52.4 40.7 45.5 53.0 43.9

34.8 29.7 45.3 53.8 33.1 43.1 47.6 43.0 29.4 51.4 39.9 45.7 52.9 38.4 39.9 52.6 42.5

51.1 36.8 59.7 64.5 42.1 52.1 60.9 59.0 34.8 62.9 51.3 54.6 65.4 51.6 48.0 56.7 53.2

44.1 33.4 51.5 56.2 41.5 51.9 54.6 51.7 31.9 55.7 50.3 50.5 56.7 46.8 48.0 52.6 48.6

41.6 32.3 53.3 57.5 42.9 51.7 55.0 53.8 32.1 56.8 48.2 51.0 59.3 46.3 49.4 51.9 48.9

50.7 29.0 59.7 63.9 32.2 45.2 62.3 45.0 31.2 63.3 40.4 56.4 65.8 42.5 49.0 56.3 49.6

50.3 36.7 59.0 63.8 49.4 58.3 61.4 60.8 34.7 63.5 58.2 57.0 64.9 53.8 54.7 57.1 55.2

49.7 36.4 57.4 62.3 47.4 57.8 59.8 56.5 34.5 61.7 55.7 54.4 62.4 51.6 53.6 54.4 53.5

46.7 34.7 56.4 61.1 44.0 53.9 58.6 54.0 32.6 60.0 51.3 54.4 62.1 49.4 52.2 59.8 52.0

43.1 32.6 56.5 64.3 39.6 53.3 58.5 54.1 31.9 61.5 49.7 56.0 64.3 48.8 49.3 60.7 51.5

55.2 38.2 63.5 67.3 49.9 61.9 65.3 64.3 35.5 67.1 60.3 62.0 68.3 58.1 57.5 62.4 58.5

Table 20 | Per-language model performance comparison on mHellaSwag. Language arb_Arab ben_Beng deu_Latn fra_Latn hun_Latn ind_Latn ita_Latn nld_Latn npi_Deva por_Latn ron_Latn rus_Cyrl spa_Latn ukr_Cyrl vie_Latn zho_Hans Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 32.5 27.2 36.2 37.4 29.7 37.8 38.8 33.4 23.9 40.8 35.0 36.9 37.3 33.8 34.9 39.6 34.7

25.4 22.7 32.3 37.4 25.7 30.9 35.9 31.2 23.6 37.0 29.6 31.8 36.2 30.4 27.7 36.6 30.9

37.6 31.1 44.4 44.5 33.8 41.4 45.3 41.9 26.4 47.4 37.2 41.6 46.8 38.9 36.7 43.7 39.9

34.6 28.7 38.1 41.2 32.9 38.3 41.4 39.3 25.7 41.0 39.3 39.6 39.4 36.9 34.4 40.1 36.9

29.6 27.1 35.5 38.3 32.4 35.5 36.8 33.8 23.8 38.8 35.1 35.6 39.0 35.3 33.0 37.4 34.2

36.4 24.3 42.3 44.1 25.5 33.8 45.3 31.7 24.6 45.8 30.8 40.4 46.8 30.3 35.2 40.7 36.1

39.0 29.7 45.5 45.7 39.0 46.9 44.1 43.5 26.9 47.6 41.4 40.4 49.3 40.8 40.6 44.4 41.5

35.6 29.1 38.7 41.4 35.5 41.1 40.5 35.8 26.3 42.6 38.8 36.9 42.6 34.6 36.2 39.0 37.2

38.9 31.1 44.9 45.9 38.7 45.1 45.3 41.2 28.1 45.9 41.0 43.6 46.0 39.5 42.2 48.2 41.6

30.8 25.1 41.5 44.1 31.5 39.0 42.7 37.8 25.5 45.8 38.4 40.7 43.4 36.0 33.5 44.8 37.5

43.4 33.8 48.8 51.2 41.2 47.8 52.6 48.4 30.0 50.8 45.2 48.7 51.3 44.5 42.4 46.6 45.4

Table 21 | Per-language model performance comparison on mARC-Challenge. Language arb_Arab ben_Beng ces_Latn deu_Latn eng_Latn fra_Latn hun_Latn ind_Latn ita_Latn jpn_Jpan kor_Hang por_Latn rus_Cyrl spa_Latn tha_Thai ukr_Cyrl urd_Arab vie_Latn zho_Hans Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 24.1 21.6 28.2 26.9 33.7 29.4 26.4 28.6 29.9 26.5 26.7 27.7 26.7 29.3 25.9 26.5 21.9 29.3 27.4 27.2

16.3 13.3 21.4 22.1 32.3 23.6 16.2 21.9 22.3 19.4 18.5 22.4 21.3 24.7 14.1 16.5 17.5 20.1 21.4 20.3

26.2 25.3 31.8 30.8 35.9 31.8 26.7 29.4 32.8 30.1 29.8 30.3 31.1 32.8 26.9 31.8 28.6 28.2 31.3 30.1

24.3 23.6 29.6 30.6 34.7 31.6 27.0 27.9 32.0 27.7 27.6 30.6 32.1 31.1 24.7 28.9 25.0 29.6 30.4 28.9

23.3 19.9 20.9 22.3 27.2 23.3 22.3 22.4 23.8 20.2 21.6 22.6 24.1 23.5 21.3 20.9 20.9 23.1 21.8 22.4

29.4 19.2 25.7 30.3 36.7 32.5 24.3 26.4 31.0 28.9 27.0 31.3 28.9 31.6 27.2 25.5 24.7 28.2 27.9 28.2

21.3 20.7 23.3 24.7 28.4 25.3 24.0 26.0 27.4 23.8 22.3 25.7 25.5 25.0 21.9 24.5 22.8 24.7 24.7 24.3

22.8 23.8 23.1 24.3 27.0 24.5 22.1 25.5 24.0 24.3 23.3 27.0 25.9 24.7 22.1 24.1 22.6 26.5 24.1 24.3

34.2 31.8 39.8 38.8 44.6 40.3 36.6 40.8 40.8 37.6 37.6 40.6 40.8 41.0 33.3 38.3 34.4 40.3 39.3 38.5

27.7 15.1 32.0 35.7 42.3 38.8 27.2 31.8 36.1 33.3 30.8 39.5 35.9 38.9 21.8 30.3 25.9 32.8 35.7 32.2

39.5 34.2 40.8 40.5 45.4 41.5 35.9 41.2 41.2 37.1 35.7 40.3 40.0 42.0 37.1 38.9 35.9 40.3 38.1 39.2

Table 22 | Per-language model performance comparison on MMLU-ProX-Lite. 46

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Language arb_Arab azj_Latn ben_Beng deu_Latn ell_Grek fra_Latn heb_Hebr hun_Latn ind_Latn ita_Latn jpn_Jpan kaz_Cyrl kor_Hang nld_Latn npi_Deva pol_Latn por_Latn rus_Cyrl spa_Latn tur_Latn ukr_Cyrl urd_Arab vie_Latn zho_Hans zsm_Latn Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 49.1 47.8 40.5 42.4 44.9 54.4 56.9 41.6 59.5 65.1 63.1 40.0 58.0 52.8 37.0 46.7 54.1 57.6 59.8 46.4 57.1 36.6 52.9 65.5 49.5 51.2

38.6 38.3 34.9 43.2 38.0 48.7 53.3 32.2 53.8 56.2 52.5 33.8 43.0 48.6 32.6 40.1 49.9 47.1 53.3 40.9 48.7 38.9 38.5 49.9 42.5 43.9

51.1 45.1 49.5 47.5 45.5 63.0 57.8 43.5 52.9 68.1 65.3 41.0 56.4 61.0 47.4 48.7 53.9 51.1 64.5 51.3 54.9 37.5 42.7 51.6 50.5 52.1

50.4 47.1 44.0 46.0 48.2 55.1 63.3 46.4 57.5 63.7 61.5 48.6 56.4 58.1 49.4 47.6 52.3 53.8 60.5 53.6 58.9 37.2 57.8 56.0 55.5 53.2

43.5 38.5 34.3 36.0 38.2 54.9 48.4 38.5 53.5 58.9 47.1 39.2 46.6 53.7 37.6 41.8 48.1 48.4 55.8 47.4 51.3 36.6 44.7 49.4 45.3 45.5

52.0 33.0 32.1 39.6 44.4 61.6 47.8 34.5 51.5 63.3 55.3 35.0 52.0 47.0 36.8 41.2 53.2 51.4 58.9 40.0 51.5 32.7 45.3 54.7 40.3 46.2

57.2 46.2 48.4 41.0 45.1 55.8 59.1 42.5 58.2 64.2 58.7 41.4 55.2 56.4 48.6 50.7 47.4 52.4 59.8 57.5 62.5 44.6 54.4 52.8 54.7 52.6

56.9 41.6 45.6 41.7 49.3 58.7 66.0 49.8 61.3 64.4 61.7 33.2 60.8 58.6 51.6 51.1 51.5 53.1 61.1 53.3 63.6 39.8 55.5 59.6 56.5 53.9

61.8 51.8 53.3 56.1 55.6 65.9 65.3 52.7 65.5 73.0 78.2 45.0 63.8 66.4 49.0 60.4 66.2 63.0 69.3 58.6 66.5 44.6 64.4 76.1 62.3 61.4

49.3 43.6 42.0 48.9 40.8 63.0 61.8 38.5 56.5 65.7 67.9 35.6 54.6 59.2 42.4 51.3 56.1 53.8 62.4 45.6 54.2 43.8 52.9 56.7 50.7 51.9

62.1 56.6 50.4 51.8 60.0 68.0 69.1 50.2 68.5 72.1 77.4 54.6 64.2 71.0 54.2 58.9 61.9 60.7 70.9 58.8 63.6 42.3 65.6 64.0 65.3 61.7

Table 23 | Per-language model performance comparison on INCLUDE.

Language arb_Arab azj_Latn ben_Beng ces_Latn deu_Latn ell_Grek eng_Latn fra_Latn heb_Hebr hun_Latn ind_Latn ita_Latn jpn_Jpan kaz_Cyrl kor_Hang nld_Latn npi_Deva pol_Latn por_Latn ron_Latn rus_Cyrl spa_Latn tha_Thai tur_Latn ukr_Cyrl urd_Arab vie_Latn zho_Hans zsm_Latn Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 53.0 52.0 45.0 54.0 66.0 64.0 74.0 65.5 48.0 50.0 70.0 62.0 72.0 65.0 53.0 57.0 54.0 53.0 56.5 63.0 71.0 73.3 60.0 46.0 70.0 57.0 65.0 56.0 49.0 59.5

47.0 45.0 36.0 40.0 63.0 58.0 71.0 59.5 44.0 41.0 53.0 55.0 63.0 45.0 47.0 47.0 44.0 45.0 51.0 63.0 59.0 61.7 47.0 42.0 54.0 52.0 53.0 54.0 49.0 51.4

58.0 59.0 59.0 63.0 64.0 64.0 79.0 73.5 60.0 52.0 61.0 69.0 76.0 63.0 56.0 69.0 59.0 59.0 60.5 69.0 71.0 73.7 53.0 44.0 76.0 70.0 66.0 56.5 55.0 63.4

53.0 48.0 55.0 62.0 61.0 58.0 73.0 69.0 53.0 58.0 72.0 71.0 73.0 75.0 56.0 65.0 63.0 65.0 58.0 69.0 73.0 76.3 62.0 58.0 74.0 71.0 67.0 57.5 53.0 63.8

55.0 52.0 49.0 60.0 71.0 62.0 72.0 69.5 51.0 48.0 68.0 70.0 68.0 68.0 52.0 64.0 54.0 56.0 58.5 63.0 77.0 76.3 58.0 56.0 70.0 63.0 65.0 53.0 51.0 61.4

50.0 53.0 38.0 54.0 68.0 57.0 79.0 73.5 50.0 48.0 52.0 80.0 71.0 57.0 50.0 54.0 47.0 51.0 64.0 65.0 82.0 77.7 65.0 40.0 67.0 55.0 63.0 55.5 57.0 59.4

62.0 54.0 60.0 71.0 73.0 61.0 78.0 72.5 56.0 61.0 75.0 79.0 81.0 78.0 51.0 75.0 66.0 71.0 68.0 71.0 81.0 80.7 61.0 68.0 83.0 76.0 79.0 54.0 57.0 69.1

65.0 53.0 63.0 70.0 70.0 60.0 75.0 73.5 58.0 54.0 79.0 80.0 79.0 55.0 58.0 70.0 67.0 70.0 65.0 70.0 77.0 80.3 66.0 56.0 76.0 77.0 75.0 52.0 63.0 67.5

60.0 57.0 44.0 61.0 65.0 63.0 76.0 70.0 51.0 53.0 70.0 78.0 75.0 64.0 57.0 61.0 58.0 64.0 69.5 68.0 79.0 78.3 64.0 53.0 78.0 72.0 72.0 59.5 47.0 64.4

48.0 50.0 41.0 45.0 70.0 58.0 72.0 66.0 46.0 42.0 59.0 56.0 68.0 49.0 53.0 51.0 50.0 50.0 61.0 68.0 64.0 70.7 48.0 47.0 61.0 58.0 52.0 56.0 60.0 55.9

59.0 63.0 63.0 69.0 66.0 58.0 82.0 77.0 60.0 64.0 80.0 83.0 77.0 80.0 64.0 83.0 69.0 78.0 68.0 76.0 85.0 82.3 68.0 70.0 87.0 81.0 76.0 56.0 67.0 72.1

Table 24 | Per-language model performance comparison on Global-PIQA.

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Language arb_Arab azj_Latn ben_Beng ces_Latn deu_Latn ell_Grek fra_Latn heb_Hebr hun_Latn ind_Latn ita_Latn jpn_Jpan kaz_Cyrl kor_Hang nld_Latn npi_Deva pol_Latn por_Latn ron_Latn rus_Cyrl spa_Latn tha_Thai tur_Latn ukr_Cyrl urd_Arab vie_Latn zho_Hans zsm_Latn Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 16.6 1.1 4.6 16.4 30.5 10.0 39.8 7.9 9.2 32.1 25.9 17.1 1.6 9.4 20.8 2.0 15.9 40.5 26.9 27.1 26.4 22.6 12.3 15.0 1.7 32.5 32.0 22.9 18.6

7.2 0.3 2.3 9.2 32.5 5.0 44.0 6.5 3.4 24.1 23.8 14.0 0.3 6.5 15.8 1.7 12.8 40.2 29.9 28.1 26.1 9.7 4.3 12.1 1.9 16.9 28.3 16.7 15.1

31.5 1.9 21.3 34.4 39.7 17.9 48.3 25.9 13.2 33.7 31.7 23.7 2.1 16.1 29.0 13.5 21.2 47.5 37.5 33.4 29.4 19.1 12.2 27.9 12.6 28.4 30.8 27.1 25.4

22.6 10.7 14.7 27.8 32.5 21.7 43.0 20.5 18.9 37.8 27.9 19.4 8.2 13.2 27.1 9.4 22.2 42.6 36.3 27.5 27.2 29.3 18.7 24.8 11.4 33.8 27.6 34.4 24.7

18.8 6.0 12.6 27.6 34.6 22.2 43.5 19.6 20.6 35.3 29.4 18.4 4.6 12.6 27.4 6.2 22.8 44.3 31.9 30.3 29.3 20.2 16.8 25.8 8.1 32.5 27.1 29.0 23.5

30.5 0.3 0.6 8.7 39.0 28.3 46.9 2.1 1.8 23.4 32.1 20.7 0.3 14.8 17.1 2.0 10.0 46.5 12.3 32.8 30.2 27.0 4.8 13.7 0.6 30.0 29.7 17.5 18.7

34.0 11.0 26.0 34.3 41.1 31.9 49.3 32.9 28.6 45.0 32.9 26.0 15.1 21.7 31.4 19.2 28.7 49.9 42.7 36.0 31.3 35.4 30.1 33.4 16.4 40.9 33.1 39.9 32.1

34.8 1.0 21.3 35.8 39.3 33.0 48.4 34.6 25.3 46.2 33.9 22.8 0.3 22.8 29.6 23.7 26.6 49.6 40.6 33.6 31.2 27.3 26.1 31.0 15.5 41.1 30.9 40.6 30.2

24.4 3.5 13.3 25.7 37.0 19.9 46.1 16.3 19.0 39.2 30.8 23.8 6.2 16.7 26.5 9.1 20.9 47.5 34.5 31.9 29.9 30.3 21.4 25.0 7.3 37.8 36.7 30.5 25.4

6.6 1.3 2.9 13.3 25.9 8.3 33.4 8.7 5.2 23.5 19.3 8.1 1.4 6.3 16.8 3.8 12.3 32.3 26.7 21.7 20.3 12.8 7.4 12.3 3.6 16.9 18.2 14.6 13.7

33.2 16.3 23.7 33.3 41.3 28.1 50.3 30.4 25.5 45.6 34.1 27.0 18.2 23.4 32.1 17.9 29.3 50.7 41.0 36.6 31.1 36.5 29.1 33.6 19.7 40.8 36.1 39.8 32.3

Table 25 | Per-language model performance comparison on FLORES En-Xx translation direction. Language arb_Arab azj_Latn ben_Beng ces_Latn deu_Latn ell_Grek fra_Latn heb_Hebr hun_Latn ind_Latn ita_Latn jpn_Jpan kaz_Cyrl kor_Hang nld_Latn npi_Deva pol_Latn por_Latn ron_Latn rus_Cyrl spa_Latn tha_Thai tur_Latn ukr_Cyrl urd_Arab vie_Latn zho_Hans zsm_Latn Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 33.3 15.4 22.7 36.8 42.6 28.9 44.4 30.8 29.8 40.6 34.7 25.2 19.6 25.1 32.1 22.4 30.0 46.1 40.4 33.9 32.2 26.9 29.2 35.1 21.4 34.8 30.0 39.1 31.6

30.3 10.6 19.6 37.0 45.4 28.6 48.5 32.4 28.4 40.9 37.3 26.9 7.3 25.1 33.7 19.6 30.1 50.4 42.7 36.2 33.5 20.9 27.9 36.0 17.9 32.0 30.8 39.8 31.1

40.3 20.7 32.4 42.0 46.9 35.4 49.5 40.5 31.5 44.0 38.9 29.4 22.2 30.3 36.5 33.9 33.7 52.5 46.6 38.4 34.6 27.6 34.6 40.5 31.7 36.7 32.5 44.2 36.7

35.1 20.5 25.6 38.9 43.2 33.4 45.6 37.4 29.1 41.7 35.4 25.3 25.5 25.1 33.9 26.8 30.6 48.6 42.4 34.7 33.5 28.8 29.8 38.0 25.7 34.5 28.9 41.7 33.6

35.7 21.1 25.5 39.8 45.1 36.0 46.3 38.3 34.4 42.2 38.3 25.4 22.9 26.3 34.4 23.2 31.7 50.1 44.2 36.3 35.0 29.7 33.7 39.3 26.1 36.2 30.0 42.4 34.6

40.6 8.9 5.8 32.8 46.3 38.2 48.3 21.4 21.4 39.2 38.6 27.5 5.6 27.2 31.2 15.9 27.9 50.6 39.0 38.0 35.7 31.7 23.3 35.2 10.8 36.3 30.6 39.1 30.3

42.4 25.8 34.3 43.3 48.4 40.0 49.2 46.0 38.5 47.1 40.0 31.6 32.3 32.9 36.8 38.1 34.6 53.0 47.5 39.5 37.2 34.6 40.2 43.1 34.7 40.0 32.9 47.4 39.7

41.6 17.0 31.7 42.3 46.6 39.8 48.0 44.5 37.3 46.3 39.1 29.7 8.0 31.0 35.6 36.7 34.1 51.8 45.8 37.3 36.5 33.9 37.8 41.5 33.0 38.9 31.9 46.8 37.3

40.1 22.2 31.0 41.2 45.9 35.5 47.8 38.8 36.4 45.4 38.1 30.4 27.8 31.4 34.8 31.4 31.7 51.9 43.5 36.8 35.1 32.3 37.1 40.0 29.5 38.3 34.6 44.4 36.9

22.7 8.3 13.7 26.6 36.7 20.9 38.8 22.7 21.2 32.4 28.8 20.3 8.0 20.4 26.4 18.8 23.6 38.4 32.1 28.0 25.1 18.1 19.6 28.0 17.6 24.0 22.9 31.1 24.1

42.9 27.1 33.4 43.6 49.2 40.0 50.5 46.2 38.1 47.1 40.8 31.8 33.8 32.8 36.9 36.7 35.8 53.7 48.2 41.5 36.7 36.0 39.6 44.5 34.3 40.1 34.9 48.1 40.2

Table 26 | Per-language model performance comparison on FLORES Xx-En translation direction. 48

Marco-MoE : Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Language arb_Arab ben_Beng ces_Latn deu_Latn ell_Grek fra_Latn heb_Hebr hun_Latn ind_Latn ita_Latn jpn_Jpan kor_Hang nld_Latn pol_Latn por_Latn ron_Latn rus_Cyrl spa_Latn tha_Thai tur_Latn ukr_Cyrl urd_Arab vie_Latn zho_Hans Avg

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 9.1 6.4 14.4 23.1 11.1 32.4 8.0 10.1 23.3 29.5 12.0 9.4 22.2 14.6 29.1 22.0 20.5 33.2 17.3 10.8 13.4 3.0 26.6 20.5 17.6

4.8 2.8 9.4 24.5 7.5 28.5 7.7 3.9 17.7 25.6 8.2 6.5 18.9 11.7 29.2 22.1 20.1 34.7 7.1 4.1 10.1 2.5 16.2 17.3 14.2

13.9 16.0 23.7 28.8 17.2 34.3 18.5 9.6 23.0 34.7 14.6 13.6 29.4 18.6 33.7 28.6 24.1 34.5 12.3 8.0 21.0 12.4 22.4 18.6 21.3

11.4 12.7 18.8 24.5 18.7 31.8 15.9 15.1 28.0 30.3 10.7 10.7 25.6 17.4 29.9 30.0 19.6 33.5 20.2 12.8 19.4 12.0 26.1 16.5 20.5

7.1 9.5 13.9 20.2 16.3 29.0 10.9 12.0 18.5 23.8 6.6 8.1 20.5 13.1 25.3 20.8 17.9 33.3 11.6 8.6 15.3 8.3 19.7 13.4 16.0

13.1 1.0 7.0 28.9 30.0 38.0 3.4 2.3 15.8 31.4 12.8 11.7 16.6 8.0 32.5 9.3 24.3 38.5 19.7 3.5 9.2 1.5 21.4 17.5 16.6

14.9 18.4 27.2 30.6 35.4 40.6 24.4 22.3 33.1 37.0 15.4 16.9 31.6 22.6 35.2 36.7 26.5 41.2 27.0 25.5 29.0 22.3 31.3 20.2 27.7

15.7 16.6 28.6 29.0 36.2 38.7 25.5 19.7 33.6 34.2 12.6 17.8 29.8 21.1 33.1 33.7 24.5 39.8 20.6 19.4 26.2 18.0 32.9 18.5 26.1

11.5 12.6 21.8 28.9 22.0 38.3 14.3 16.7 29.9 34.3 15.5 15.8 27.3 18.9 33.8 31.2 24.2 39.7 23.6 18.5 21.0 10.2 30.4 22.9 23.5

3.2 3.0 6.0 12.9 4.4 12.4 4.7 4.0 7.5 12.1 4.6 5.1 10.1 5.9 12.0 9.4 8.3 14.6 5.1 4.9 6.7 2.5 8.1 7.6 7.3

15.0 17.8 26.0 32.0 30.7 41.6 25.8 20.5 33.7 38.5 18.1 19.1 34.2 24.4 36.2 36.6 25.5 42.0 26.1 24.2 28.5 20.2 33.4 21.9 28.0

Table 27 | Per-language model performance comparison on WMT24++ En-Xx translation direction.

Qwen3 Trinity Granite4 Marco Llama3.2 SmolLM3 Gemma3 Tiny-Aya Qwen3 Trinity Marco 1.7B Base Nano Base Tiny Base Nano Base 3B Base 3B Base 4B Base 3.35B Base 4B Base Mini Base Mini Base 17.1 20.2 30.4 32.2 31.9 37.4 26.4 24.8 30.3 37.9 18.9 22.0 33.3 25.6 35.6 34.0 24.8 39.4 22.7 24.1 29.6 25.0 27.2 25.2 28.2

15.2 14.0 30.3 32.9 29.4 40.1 27.5 22.3 30.7 40.1 19.5 19.9 33.6 26.2 38.2 37.4 27.1 41.1 16.5 21.1 27.8 21.3 25.8 26.5 27.7

18.4 24.1 34.2 35.4 35.1 39.3 31.9 27.3 32.9 41.0 18.9 23.1 35.3 29.8 39.7 38.4 27.2 41.2 21.0 25.8 33.1 29.3 27.7 26.8 30.7

15.1 21.6 30.0 31.3 33.0 36.9 30.2 24.3 30.8 37.2 17.4 18.7 33.2 27.0 35.5 36.3 24.8 38.1 22.2 23.5 29.4 27.5 27.7 23.1 28.1

18.4 16.2 31.7 31.6 36.0 38.0 27.8 26.1 32.4 38.1 18.4 20.6 33.0 27.4 36.5 36.8 25.3 41.7 23.1 24.6 28.3 27.7 27.3 25.1 28.8

19.7 4.6 25.3 34.6 39.8 40.0 21.4 16.5 29.7 42.0 19.5 22.3 31.6 23.1 36.3 32.4 26.9 42.5 26.2 16.5 28.0 15.2 26.2 26.0 26.9

23.2 30.7 35.4 36.0 42.4 41.0 35.2 29.1 34.6 43.5 23.7 26.9 35.6 32.1 39.3 43.3 29.7 45.3 29.2 32.7 35.9 36.7 31.7 28.0 34.2

22.8 27.7 34.9 34.2 40.5 40.1 34.6 28.2 34.7 42.1 21.5 25.5 37.1 30.4 39.3 40.5 27.7 43.7 27.6 30.4 32.8 34.1 31.3 26.2 32.8

21.3 27.1 35.6 36.1 38.9 40.5 32.0 29.1 34.0 42.3 23.2 25.8 37.2 30.3 39.6 40.6 28.5 44.1 27.8 30.2 33.6 32.0 31.5 29.9 33.0

6.3 7.3 11.4 12.0 10.1 15.1 9.5 9.4 12.8 14.6 6.2 6.6 12.5 10.5 14.2 12.4 10.7 16.5 7.6 8.5 10.2 8.1 9.7 8.9 10.5

21.6 28.1 36.9 36.3 41.1 42.1 36.3 30.5 35.8 43.3 23.1 26.4 38.8 33.9 41.2 42.7 29.9 45.6 29.1 32.8 36.9 34.6 32.8 30.1 34.6

Table 28 | Per-language model performance comparison on WMT24++ Xx-En translation direction.

Record · ID 141549 · SHA-256 619ca5763e993e13
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.