arXiv CSopen access

MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models

· arxiv_cs

arXiv CS · Papers · License: Open Access

machine learning, deep learning, neural networks

M O EMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models Xuanming Cui2,∗ , Shlok Kumar Mishra1 , Wentao Bao1 , Aashu Singh1 , Zihao Wang1 , Xiangjun Fan1 , Jun Xiao1 , Ser-Nam Lim2,† , Jianpeng Cheng1,† 1

AI at Meta, 2 University of Central Florida Work done at Meta, † Equal advising

Universal multimodal embedding (UME) increasingly demands encoder’s capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size, retrieval effort, or scales the encoder into a heavy multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via reasoning tokens. However, embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose M O EMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoEbased UME, M O EMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only ∼3B active parameters, M O EMB surpasses TTE-based methods with > 4× active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based embedding models towards large-scale retrieval and recommendation systems. Date: September 7, 2026

Introduction

Embedding models map heterogeneous inputs into a shared space for retrieval. While earlier embedding tasks focus primarily on simple symmetric retrieval, recent universal multimodal embedding (UME) tasks now require a single encoder to handle diverse modalities, with diverse tasks spanning classification, question answering, retrieval, grounding, MoEMB-A3B (ours) Embed-RL Qwen3.5-2B and document search (Meng et al., 2025). As the embedding tasks grow more varied and complex, performance depends more on the encoder’s capacity for visual reasoning and instruction following. However, most work on scaling embeddings still targets the stored representation instead of that capacity (Fig. 2). Matryoshka Representation Learning adjusts the embedding dimensionality (Kusupati et al., 2022), while multi-vector retrieval methods (Khattab and Zaharia, 2020; Faysse et al., 2025; Xiao et al., 2025) keep multiple vectors per input for finegrained retrieval. These methods scale storage and retrieval budget, but they leave the encoder itself untouched, and so cannot supply the understanding and reasoning that harder UME tasks demand.

UME-R1 PLUME

+ Adaptive computing MoEMB-A10B (ours)

VLM2Vec-V2

32 experts 50

Queries per second ( ↑ )

arXiv:2609.08663v1 [cs.LG] 8 Sep 2026

∗

2B, no reasoning

2B, no reasoning 256 experts

5 2 1

2B + reasoning

0.5

2B + reasoning 0.2 0.1 56

MMEB-V2 accuracy (%) ( ↑ )

Figure 1 Accuracy vs. QPS. Scaling via MoE (M O EMB) raises accuracy by > 10 points at near-constant throughput. Details can be found in C.2.

(a) Dimensions

(b) Multi-vector

(d) Mixture-of-Experts x

max sim ...

MRL

...

router

...

Colbert / ColPali

e2 e3

e4 ... en

∑

TTE / UME-R1

MoEMB (ours)

Figure 2 Four ways to scale multimodal embeddings. Dimension (a) and multi-vector (b) methods scale the output representation; TTE-based methods (c) scale via intermediate reasoning tokens. M O EMB (d) instead scales the encoder along the expert axis through sparse MoE. 72

MMEB accuracy (%)

To scale the encoder itself, recent works turn multimodal large language models (MLLMs) into embedders, and these now lead UME benchmarks (Jiang et al., 2025; Lin et al., 2025; Li et al., 2026). However, even at larger scales than CLIP-based (Radford et al., 2021) lightweight models, these MLLM-based embedders still struggle to cover the diverse tasks of UME. We empirically find (Fig. 3) that one embedder trained jointly over all tasks is outperformed by the same model trained per task; Qwen3-VL-Embedding likewise reports degraded MTEB (Muennighoff et al., 2023) performance against text-only Qwen3-Embedding (Zhang et al., 2025b). Two explanations are possible: destructive gradient interference between tasks, or a model simply not capacious enough to serve them all at once. These issues motivate us to explore the design space and training recipe when scaling up embedder.

specialist (one group) generalist (all tasks) 70.5

−1.92 68.6

68 66.9

66.9

65.8

−1.13

−3.02 63.9

62 Qwen3.5-2B dense, 2B act.

Qwen3.5-9B dense, 9B act.

Qwen3.5-35B-A3B MoE, 3.1B act.

Figure 3 Performance degradation for switching from generalist to specialist shrinks with model’s total capacity. Each model trained on the full MMEB mixture (generalist) vs. one task alone (specialist). Details in App. D.2.

A recent line of think-then-embed (TTE) methods scale for reasoning-heavy retrieval by generating intermediate reasoning tokens before producing the final embedding (Cui et al., 2025; Lan et al., 2025; Cheng et al., 2026). However, these methods improve performance with a significant cost of computation. For instance, on the text partition of MMEB-V3 (Huang et al., 2026), it takes 1.7 days for a regular MLLM-based 2B encoder (Meng et al., 2025) to complete evaluation, while UME-R1-2B (Lan et al., 2025), a TTE-based model, needs roughly 7.8 H200-days to encode queries, and 431 H200-days for candidates, which is prohibitively expensive for real-world deployment. In general, scaling a UME encoder is difficult: (1) search and recommendation systems needs embed astronomically many queries and items, under low latency; (2) contrastive training encodes and backpropagates through both queries and targets, so the activation memory is doubled; (3) added parameters trade against batch size, a key component of contrastive quality; and (4) UME tasks vary widely in complexity: for simpler tasks the extra LLM backbone may be wasteful, since the MLLM’s own contrastively pretrained vision encoder (Radford et al., 2021) could already suffice. To address challenges (1–3), we propose M O EMB, the first Mixture-of-Experts (MoE) UME model. Instead of scaling the embedding dimension, dense parameters, or reasoning tokens, we scale the encoder along the expert axis. Sparse MoE is well studied for language modeling (Jiang et al., 2024; Team et al., 2026) but remains underexplored for embedding models, for which we fill the gap. We study three approaches to obtain an MoE-based embedder: (1) adapting a pretrained native sparse MoE, (2) dense-to-MoE upcycling, and (3) modality-level MoE; to our knowledge, M O EMB is the first to adapt and scale pretrained sparse multimodal MoE backbones for UME. Expert scaling gains over 10 points at negligible additional compute (Fig. 1): with only ∼3.1B active parameters, M O EMB-A3B reaches a new SOTA of 70.4 on MMEB-V2, surpassing all UME models trained on the MMEB V2 dataset, including TTE-based encoders that require significantly more computes. Next, to tackle issue (4), heterogeneous complexity across UME tasks, we conduct the first comprehensive study of adaptive computation for MoE-based UME embedders. Prior adaptive computation primarily focus on text generation (Huang et al., 2025a; Chen et al., 2024; Bai et al., 2025) and traditional vision tasks like classification (Yin et al., 2022), with little work on embeddings. Moreover, these works typically study one type of adaptive computation action at a time. In this work, we aim at providing a systematic exploration for adaptive computation in MoE-based UME models, where we study a wide range of actions including token pruning, layer skipping, early exit, expert skipping, adaptive top-k, and expert pruning – under both training-based and inference-only settings. Empirically, we find token

pruning and adaptive top-k give the strongest accuracy–compute trade-off. Applying them saves 50% compute with < 1% performance degradation. Our contributions are twofold. 1. Expert scaling for UME. (a) We introduce expert capacity as a scaling axis for UME and conduct the first comprehensive study of its model designs and training recipes. (b) Without any special data mining or sophisticated multi-stage training, M O EMB sets a new SOTA on two large-scale benchmarks, MMEB-V2 and MRMR (Zhang et al., 2025a), among models trained only on public MMEB-family data, surpassing TTE-based encoders that consume significantly more compute. 2. Adaptive computation for MoE-based UME. (a) We conduct the first systematic exploration for adaptive computation for MoE-based UME embedders, across six actions and both training-based and inference-only methods. (b) By leveraging adaptive computation methods, we achieve 50% compute reduction for < 1% performance degradation. (c) We are the first to explore token pruning based on Gated Delta Networks’ β as a strong alternative to attention scores. (d) We provide corresponding inference implementation for realizing saved computes into wall-clock saving.

Preliminaries

A UME embedder maps a multimodal, instruction-conditioned query or target {q, t} = (Ix , Iv , It ) composed of a task instruction Ix , optionally visual inputs Iv and a text input It , to a single vector (Meng et al., 2025). We build it from a pretrained multimodal LLM: one encoder Fθ processes both queries t and targets t: eθ ({q, t}) = norm Pool(Fθ ({q, t})) , where norm is the normalization operation, and Pool is the pooling operation that produces the final embedding representation (Cui et al., 2025). For a batch of N pairs, we follow the standard one-directional InfoNCE objective (Jiang et al., 2025): Lemb = −

1 X eSii log P Sij , N i je

Sij = cos(qi , tj )/τ

(1)

where τ denotes the temperature, and cos is the cosine similarity function.

Building Sparse MoE UME Models

3.1

MoE Backbone Designs

We explore three ways to build a sparse MoE-based embedder: (1) Native MoE: we reuse a backbone whose experts and router are learned in standard language model pretraining; (2) Dense-to-sparse Upcycling: we convert a dense MLLM into sparse MoE during contrastive training; (3) modality-level routing: we perform routing on the modality level. (1) Native MoE. We directly reuses a MLLM whose feed-forward blocks are already sparse MoE, composed of a shared expert FFNshared and E sparse experts. For a token’s hidden state h ∈ Rd , the pretrained router Wr ∈ RE×d scores all E experts and keeps the k highest, S = TopK(Wr h, k). Their scores are renormalized over S, while a learned vector wshared ∈ Rd and the sigmoid σ gate the shared expert: MoE(h) = γ FFNshared (h) +

αj = P e αj FFNj (h),

(Wr h)j

u∈S e

(Wr h)u

, (2)

γ = σ wshared h .

j∈S

This variant retains MoE pretrained from language modeling. Empirically we find this setting leads to the best performance. We therefore used as default setting for all the subsequent experiments.

Token-level Routing (Ours)

x1 x2 ...

∑

e3 ...

∑

router

(2) Dense-to-MoE Upcycling. We explore upcycling a dense MLLM into a sparse MoE embedder by splitting its feed-forward block and replicating the pieces into E experts, then learn a router over them from scratch. We empirically find its performance to lag behind the native MoE setting. Therefore, we report its best variant in Tab. 4 and provide further details in App. D.1.

Modality-level Routing Modality A

Modality B

Fixed Router

e3 ... en

...

Figure 4 Token- vs. modality-level routing.

(3) Modality-level Experts. As shown in Fig. 4, we explore an alternative routing setting, where we replicate the dense feed-forward blocks into modality-level experts, where a fixed router forwards all tokens from one modality to its corresponding expert. This is in contrast to the regular token-level routing, where each token is dynamically routed to its top-k experts. We find this variant slightly improves over the dense baseline, but lags behind the native MoE setting 4.

3.2

Pooling, router adaptation, and training dynamics

Beyond determining an effective, MoE backbone design, adapting a pretrained MoE model to a contrastive embedding objective also requires other design and training choices. In this work we explore three aspects: pooling, router adaptation, and training dynamics for MoE-UME. Pooling Strategy. Previous UME works (Cui et al., 2025) have discovered that pooling via last token often outperforms other pooling methods like mean (Lee et al., 2025) or latent attention pooling (Lee et al., 2024). Building on this observation, we only compare two designs: a simple last token pooling, versus a designated <emb> token. Empirically we find the latter to perform better (Tab. 5). Router Adaptation and Auxiliary Objectives. Because pre-trained routers establish delicate token-to-expert assignments, downstream contrastive fine-tuning can destabilize learned expert allocation. To systematically evaluate router stability versus adaptability, we compare frozen and trainable router configurations alongside three auxiliary objectives: (1) load-balancing and z-losses to prevent expert collapse; (2) a conflict-aware loss to mitigate cross-task gradient conflict by penalizing routing distribution overlap across conflicting tasks; and (3) task conditioning, which incorporates discrete task labels or continuous instruction embeddings alongside token hidden states into router logit computation. Full details for each formulation appear in App. B.2. Empirically, we find that the simple frozen router instead yields the best performance, which indicates that the routers learned during language pre-training is also near-optimal for contrastive objective (Tab. 5). Adaptation Dynamics. We systematically isolate the hyperparameter settings governing contrastive adaptation in sparse backbones: global batch size (in-batch negative volume), logit scaling, LoRA adaptation rank, learning rate, and training duration (Fig. 6).

Adaptive Computation for MoE-UME

While sparse MoE improves the quality–compute tradeoff, it still runs a fixed per-token compute budget with potential computation redundancy. To further improve efficiency and scalability for MoE-based UME models, we explore various adaptive computation actions. Fig. 5 illustrates the six types of adaptive computing actions we study, covering both training-based (§4.1) and inference-only (§4.2) settings. Tab. 1 summarizes different methods we studied. Due to the space limit, we discuss the most effective methods in the following sections, and leave the rest to App. F and E.1.

4.1

Training-Based Adaptive Computation Methods

Token pruning. We study two types of token pruning: one-shot and gradual pruning. A one-shot pruning makes a single pruning decision at an early layer Q(Huang et al., 2025a) K; a gradual pruning applies a per-token gate at every layer, so a token’s alive mass αt,ℓ = j<ℓ (1 − pdrop t,j ) decays with depth and its downstream compute is removed progressively. We train one-shot pruning with a straight-through estimator (STE). For gradual pruning, we explore both STE and an AViT-style (Yin et al., 2022) halting mechanism, where both methods failed due to unstable training (App. E.1). Therefore, we realize gradual pruning with the GRPO policy discussed in §4.1, which samples the discrete action instead of differentiating through it. Both settings distill from a separate teacher embedding obtained without adaptive computing: efull . Given the target budget ρ⋆ and the model’s current keep rate ρkeep , the overall loss becomes: 2 Ldrop = Lemb + λd 1 − cos(e, efull ) + λb ρkeep − ρ⋆ . 4

(a) Token pruning ... x2

x2 ...

routed experts

attn

...

Ln-1

Ln+1

...

Ld : pruning layer

routed experts

(e) Adaptive top-k routing x1 ' x 2'

Ln+1

exit

(d) Expert Skipping x1

x1 x2 ...

e2 e2 e3 e4

(f) Expert pruning ...

x1 en

x2 ...

router

x1 x2 ...

(b) Layer skipping

pruned experts

e1 e2 e3 ... en'

en-2 en-1

Figure 5 We consider six adaptive computing actions in M O EMB. The top-three (a-c) are general adaptive computing actions, while the bottom-three (d-f) are specific to MoEs. Table 1 Summarization for different designs we explored for each adaptive computing action. Regime is when the decision is made: Train is training-based, Inf. means inference-only. Basis names the work a method is adapted from, or ours where we know of no prior counterpart. † marks those detailed in Apps. F and E.1 instead of in the main text.

Action

Method

Regime Basis

Token pruning

One-shot pruner after decoder block n (Gumbel/STE) Layer-wise gradual drop gate, packed varlen (GRPO)† GDN β-ranked one-shot drop

Train Train Inf.

Huang et al. (2025a) ours ours

Expert skipping

Per-token three-way skip gate (soft fade) GDN β-ranked expert skip Per-layer Otsu split on β or router logit†

Train Inf. Inf.

ours ours Otsu (1979)

Adaptive top-k

Null experts, load-balance–scheduled† Dynamic modality thresholding Fixed router-score threshold†

Train Inf. Inf.

Zeng et al. (2024) Huang et al. (2025b) ours

Layer skip & exit Trained layer-skip gate (STE, distillation)†

Train

ours

Expert pruning

Routing frequency, router probability mass MAN, MSAN, REAP saliency

Inf. Inf.

Lu et al. (2024) Liu et al. (2026); Lasby et al. (2025)

Combined

Joint actions under one FLOPs budget (Gumbel/STE)† Discrete and unified action rollouts (GRPO) Adaptive top-k with token pruning

Train Train Inf.

ours ours ours

Learned expert skipping. Inspired by Mixture-of-Depths (Raposo et al., 2024), we add a skip gate pskip t,ℓ ∈ [0, 1] over existing routers, which determines whether to skip the entire experts computation: L

Lskip = Lemb + λc 1 − ρskip ,

ρskip =

1 X X skip pt,ℓ , L |T | ℓ=1 t∈T

with L being the number of MoE layers, |T | denoting the number of tokens, and pskip t,ℓ the normalized probability produced by each skip gate. The skip budget ρskip applies to the entire model, which allows the skip rate to differ across layers (Fig. 19). As a hard skip gate is non-differentiable, we train it through a soft fade. Let ut,ℓ be the output of the routed experts for token t at layer ℓ, and vt,ℓ the rest of that layer’s output; the routed output ht,ℓ is scaled by the non-skip probability 1 − pskip t,ℓ : ht,ℓ = vt,ℓ + 1 − pskip ut,ℓ . t,ℓ Combining adaptive computation actions with STE. Enabling several actions at once may not effectively accumulate their savings: the actions compete for the same budget, and could cause unstable training and thereby degrade performance. We summarize four training strategies below, with full details in App. E.1: (1) a single global 5

FLOPS reduction budget over all actions trained together, which leaves the allocation entirely to the optimizer. (2) a separate budget for each action, trained simultaneously. (3) the same per-action budgets as in (2) with actions optimized in alternation, which keeps one update from being confounded by another gate moving at the same time. (4) the same per-action budgets as in (2,3), with actions trained one at a time and frozen once finished. GRPO for combined adaptive computation actions. Differentiating through multiple discrete actions with STE can be unstable, so we also explore adaptive computation as policy optimization and adapt GRPO (Shao et al., 2024). Each efficiency gate (e.g. an expert skipping gate) predicts a distribution over its actions. For each input, we sample a group of outputs {ai }G i=1 , where each rollout samples one action from every gate, producing an embedding eai . As adaptive computation has no pretrained reference policy to regularize toward, we replace the KL term with a distillation loss toward a teacher embedding efull , computed by a full-compute greedy forward pass. To avoid reward hacking, we gate the efficiency reward S(ai ) on the success of the in-batch retrieval recall@G(ai ) ∈ [0, 1]: " # G 1 X L = −E min ri Âi , clip ri , 1−ϵ, 1+ϵ Âi + λd E 1 − cos(ea , efull ) , G i=1 ri =

R(ai ) − Rµ πθ (ai ) , Âi = , R(ai ) = recall@G(ai ) s̄+ (ai ) − s̄− (ai ) +λs S(ai ) , {z } | {z } | πθold (ai ) Rσ recall gate

ranking margin

where Rµ , Rσ are the mean and standard deviation of the reward over the group. s̄+ (ai ) and s̄− (ai ) denote the similarity between positive and negative pairs. We provide full details in App. E.1.

4.2

Inference-Based Efficiency

Token importance without an attention map. Inference-only token pruning conventionally ranks visual tokens by the attention mass they receive (Chen et al., 2024), which presumes the attention matrix is available. However, as the efficient FlashAttention kernels (Dao et al., 2022) does not materialize the attention map, we need to find an alternative training-free signal for token importance. In this work, we propose to leverage the writing strength β in Gated DeltaNet (GDN) (Yang et al., 2024), which is used in recent frontier MLLMs (Qwen Team, 2026; Team et al., 2026). At each GDN layer the gated delta rule updates a recurrent memory S using a token-dependent write strength. Within one head of one GDN layer, the rule keeps a memory St ∈ Rdv ×dk and updates it as St = St−1 αt (I − βt kt kt⊤ ) + βt vt kt⊤ , βt = sigmoid Wβ⊤ xt ∈ (0, 1), (3) where xt is the token’s hidden state, kt ∈ Rdk and vt ∈ Rdv its ℓ2 -normalized key and its value, αt ∈ (0, 1) the forget gate, and I the identity. Thus, βt interpolates between leaving the memory untouched (βt →0) and fully overwriting its kt direction with vt (βt →1), which provides a natural surrogate for token importance. Averaging βt over those layers and over heads gives each token an importance score, which drives every inference-only token and expert decision below. Token pruning. We utilize the β score to prune visual tokens. After decoder block K we rank the visual tokens by βt , keep the highest-scoring fraction, and discard the rest for every layer that follows. Adaptive top-k. Let αℓt(1) ≥ · · · ≥ αℓt(k) denote the sorted routing weights for token t at layer ℓ (Eq. 2). Adaptive top-k routing selects a dynamic number of experts kℓt ≤ k per token via a predefined condition. For instance, MoDES (Huang et al., 2025b) keeps an expert m if its scaled routing weight satisfies sℓ αℓt(m) ≥ τ , for which the global threshold τ and layer-scaling factors sℓ are estimated during a calibration pass (App. E.2). Here, τ acts as a uniform decision threshold, while sℓ upweights critical layers to retain more experts where capacity matters the most. Total expert pruning. We consider pruning experts from the model permanently, which reduces memory instead of compute. Each layer keeps the E ′ experts with the largest saliency ϕj , measured on a calibration pass. We compare two families of saliency. Router-based scores read the router alone (Lu et al., 2024), either as the fraction of calibration tokens routed to expert j (routing frequency) or as its mean router probability. Output-based scores instead read the output produced by the expert, averaged over the tokens routed to it, ϕj = mean wtj , wtj ∈ ∥otj ∥, ∥otj ∥2 , αℓtj ∥otj ∥ , otj = FFNj (hℓt ), (4) t→j | {z } | {z } | {z } MAN

MSAN

REAP

where otj is the expert’s output and αℓtj its routing weight from Eq. 2. The three output-based scores differ only in how they weight that output. MAN and MSAN (Liu et al., 2026) take its norm and squared norm, ignoring the router entirely, so an expert is judged purely by how large an output it produces. REAP (Lasby et al., 2025) additionally scales the norm by the routing weight. Empirically, we find this variant to generally perform better (Fig. 7).

4.3

Realizing Wall-Clock Savings

A reduction in logical computation is not a reduction in QPS, and the gap decides which actions are worth having. However, a large body of adaptive computation works focus the former and ignore the latter (Zeng et al., 2024; Huang et al., 2024). In this work, we address this gap by providing implementation for effectively converting the computation saving into actual batched inference acceleration. We then conduct detailed benchmarking for both computation and wall-clock savings. For token pruning-based actions we adopt a packed variable-length approach: at each layer the surviving tokens are gathered into contiguous segments with updated cumulative sequence lengths, and a segment index keeps the attention operations from mixing adjacent samples. This procedure allows for batched adaptive computation where token numbers vary across layers, which is a nontrivial implementation compared to one-shot pruning (Huang et al., 2025a; Shang et al., 2024). For MoE-specific methods, many report efficiency without an inference wall-clock measurement (Zeng et al., 2024; Huang et al., 2024; Li et al., 2023), do not supply the implementation that would realize the saving (Huang et al., 2025b), or rely on a naive implementation that loops over experts (Lu et al., 2024), which we find to be 5.7–8.0× slower than a fused kernel, so any speedup measured against it is overstated. In this work, we run the wall-clock benchmarks through tile-aware grouped-GEMM kernels (Guo et al., 2025) alongside the batched grouped GEMM (Gale et al., 2022), which exposes where per-token expert sparsity stops gaining wall-clock speedup from FLOPS saving.

Experiments

5.1

Experimental Setup

Models and training. We adapt the Qwen3.5-35B-A3B and Qwen3.5-122B-A10B multimodal MoEs (Qwen Team, 2026). All training are done with two epochs at length 8192 in BF16, global batch 512, AdamW at learning rate 10−4 with ten warmup steps and cosine decay, and LoRA of rank 64 and scale 128. We use Nvidia Megatron Core MoE (Liu et al., 2025) with Expert Parallelism for accelerated MoE-UME training. App. B gives full implementation details. Benchmarks. We evaluate our models on two comprehensive and large-scale multimodal retrieval benchmarks: MMEB V2 (Meng et al., 2025) and MRMR (Zhang et al., 2025a). MMEB-V2 (Meng et al., 2025) comprises 78 tasks spanning image, video, and visual document (VisDoc) retrieval tasks. MRMR (Zhang et al., 2025a) is a reasoning-intensive multimodal retrieval benchmark, which requires multi-step logical inference and complex domain-specific interpretation across multimodal contexts (e.g., cross-modal knowledge retrieval, theorem identification, and contradiction detection). We follow the official evaluation setup for both benchmarks.

5.2

Results and Ablations on MoE Design and Training Dynamics for MoE-UME

Main results. Tab. 2 shows that M O EMB achieves superior performance on MMEB V2, outperforming all public-data baselines across model scales. At 3.1B active parameters, M O EMB-A3B achieves 70.4 overall accuracy, surpassing Embed-RL-4B (68.1) and UME-R1-7B (65.6), despite those methods activating 4× more parameters and executing an autoregressive reasoning pass prior to embedding generation. Scaling to M O EMB-A10B further raises accuracy to 72.4, outperforming Embed-RL-4B by 4.3 points and UME-R1-7B by 6.8 points. Fig. 1 compares our proposed expert scaling against test-time reasoning scaling along the accuracy–throughput Pareto frontier. Scaling the number of total experts from E = 32 to E = 256 improves M O EMB-A3B accuracy by +10.4 points while QPS drops negligibly (34.2 → 29.3) QPS, as top-8 routing holds active FLOPs bounded regardless of total expert count. In contrast, TTE-based approaches incurs severe throughput tradeoffs for marginal gains. For instance, enabling reasoning in Embed-RL-2B yields a marginal +1.5 point gain but collapses throughput significantly from 41 to 0.11 QPS, a 370× slowdown. As MMEB-V2’s evaluation suite is largely out-of-domain (OOD) relative to its training data, many top-performing baselines rely on external corpora that mirror the evaluation formats. This inflates their scores through domain-wise 7

Table 2 MMEB-V2 per-category results. bold/underline mark best/second within the public-data group. † Embed-RL pairs a shared 8.77B autoregressive reasoner with a 4.44B/2.13B embedder, giving 13.2B/10.9B active parameters. ‡ Octen-VL-Embedding-Large and DME-Large do not disclose an active parameter count. CLS: classification, QA: question answering, RET: retrieval, GD: grounding, MRT: moment retrieval; V1/V2: ViDoRe-v1/v2, VR: VisRAG, OOD: out-of-domain. Model

Act. (B)

# Datasets→

Image

Video

VisDoc

All

CLS QA RET GD

CLS QA RET MRT Ov

VR OOD Ov

Models adapted with external data ∼2–4B active params DME-2B Qwen3-VL-Embedding-2B RzenEmbed-v1-2B Ops-MM-Embedding-v1-2B ≥8B active params DME-Large‡ Octen-VL-Embedding-Large‡ DME-Medium Qwen3-VL-Embedding-8B IFM-TTE-7B WeMM-Embedding-8B RzenEmbed-v2-7B Ops-MM-Embedding-v1-7B E5-Omni-7B GME-Qwen2-VL-7B

2.2 2.1 2.2 2.2

73.0 70.3 65.3 68.1

73.8 74.3 61.7 65.1

76.2 74.8 73.8 69.2

90.7 88.5 77.8 80.8

76.3 75.0 68.5 69.0

82.2 71.9 45.6 53.6

57.6 64.9 47.5 55.6

53.5 53.9 38.3 41.7

45.6 53.3 36.7 33.7

61.3 61.9 42.6 47.6

86.0 84.4 87.0 76.4

59.3 65.3 57.6 53.2

92.5 86.4 85.4 77.6

72.0 69.4 67.1 65.1

80.9 79.2 78.4 71.0

74.2 73.2 65.6 64.7

– – 9.4 8.1 8.3 8.8 8.3 8.3 8.0 8.3

75.4 75.7 74.5 74.2 76.7 73.5 70.6 69.6 66.7 57.6

82.9 84.5 80.9 81.1 78.5 76.1 71.7 69.6 68.5 34.6

79.6 80.6 78.2 80.2 74.6 78.6 78.5 73.1 73.0 71.2

95.2 94.4 94.5 92.3 89.3 92.9 92.1 87.2 83.9 59.5

81.1 81.9 79.8 80.1 77.9 78.1 75.9 72.7 71.2 55.9

90.2 87.1 87.7 78.4 60.5 66.5 58.8 59.7 46.6 37.3

75.7 82.1 71.0 71.0 67.9 71.7 63.5 62.2 52.9 50.3

64.5 68.0 61.0 58.7 51.7 56.4 51.0 45.7 36.7 28.3

62.2 60.4 58.5 56.1 54.9 55.2 45.5 43.2 34.2 37.5

74.4 76.0 70.8 67.1 59.2 63.2 55.7 53.8 43.5 38.4

89.1 85.9 87.6 87.2 85.2 89.5 89.7 80.0 87.6 89.6

61.5 66.3 57.8 69.9 71.5 59.3 60.7 59.6 62.4 55.5

93.8 86.6 94.5 88.7 92.8 90.4 88.7 79.3 87.5 85.0

75.1 72.2 73.5 73.3 67.3 35.1 69.4 67.5 34.6 69.4

83.4 80.5 82.0 82.4 81.8 75.6 81.2 74.4 74.5 79.4

80.2 80.1 78.4 77.8 74.8 73.9 72.9 68.9 65.8 59.1

72.1 72.4 74.4 71.1 20.6 85.0

49.8 46.2 44.6 38.6 13.2 57.8

61.6 61.2 59.1 59.0 43.9 70.4

Models adapted only with public MMEB-family data ∼2–4B active params PLUME UME-R1-2B VLM2Vec V2-2B BToks VLM2Vec V1-2B M O EMB-A3B ≥8B active params Embed-RL-4B† Embed-RL-2B† UME-R1-7B UniME-V2 VLM2Vec V1-7B M O EMB-A10B

2.1 2.2 2.2 2.2 2.2 3.1

66.5 64.8 62.9 64.3 58.6 60.6

59.2 62.8 56.4 59.8 49.2 73.8

67.6 67.6 69.6 68.8 65.0 72.4

79.7 77.2 77.1 77.4 73.0 92.4

66.3 66.6 64.9 66.0 59.7 71.8

45.0 44.3 39.2 43.7 33.3 53.8

52.3 50.9 34.7 47.0 30.7 65.9

33.5 32.9 28.4 33.0 20.4 45.3

46.7 39.7 37.5 33.6 30.7 45.8

44.1 42.2 34.7 39.9 28.5 53.5

78.1 79.2 79.3 81.3 52.2 88.5

57.4 57.8 62.5 38.1 46.9 82.8

67.5 67.3 68.7 62.7 31.7 81.0

13.2 10.9 8.3 8.0 8.3 10.2

63.7 70.5 71.3 91.3 71.2 57.6 58.4 45.1 62.8 67.9 68.6 90.4 69.2 57.0 55.9 45.1 67.1 69.2 71.9 84.9 71.2 48.6 60.7 38.2 65.6 68.7 73.1 90.8 71.8 37.2 50.6 28.9 62.8 56.5 69.4 81.9 65.4 39.0 30.1 29.1 64.4 75.4 74.7 92.7 74.0 58.1 67.6 47.3

49.5 49.4 39.3 39.6 39.2 49.4

53.0 80.2 53.4 84.9 52.1 79.9 52.0 84.6 47.5 75.7 50.5 83.7 39.0 61.8 42.0 70.5 33.8 20.0 9.2 58.9 56.3 85.5 62.7 89.1

67.1 65.7 58.3 58.3 57.1 82.9

74.7 68.1 74.1 66.8 70.6 65.6 60.1 60.6 34.1 48.5 82.2 72.4

data leakage. To ensure a fair comparison, we split results in Tab. 2 by training data regimen (public vs. external data). Notably, on the few in-domain evaluation splits where MMEB V2’s public training set provides in-domain supervision, such as ViDoRe and VisRAG, M O EMB achieves SOTA performance without external adaptation data. For instance, M O EMB-A3B achieves 81.0 with 3.1B active parameters, outperforming all sub-4B external-data models (e.g., DME-2B’s 80.9 and Qwen3-VL-Embedding-2B’s 79.2). M O EMB-A10B reaches 82.2, surpassing larger baselines like IFM-TTE-7B, which leverages a larger external reasoner, and is within 0.2 points of Qwen3-VL-Embedding-8B, a SoTA MLLM embedder trained with massive external and in-domain data with complicated training strategies. On MRMR (Tab. 3), M O EMB-A3B achieves the highest overall score of 51.6 while activating only 3.1B parameters. This outperforms all baseline systems despite their significantly larger active parameter footprints (7B–13.2B). Specifically, M O EMB exceeds all TTE-based methods which are specifically designed for reasoning-intensive retrieval with significantly more active parameters and compute budgets: PLUME (+11.6), UME-R1-7B (+3.4), and Embed-RL-4B (+4.8). This confirms that scaling model capacity via MoE is a more effective approach that scaling via CoT. Ablations on MoE Design. We first compare structural MoE strategies against a dense baseline at equal active compute (Tab. 4). We can observe upcycling a dense backbone (App. D.1) does not outperform the dense backbone. We conjecture this is due to that the limited scale for contrastive training over the MMEB V2 dataset is not able to train a strong router, compared to

Table 4 Ablation on different expert scaling design.

Method Qwen3.5-2B (dense) Dense-to-MoE upcycling Modality-wise MoE M O EMB-A3B (native)

Act.

Image

Video

VisDoc

Overall

2.2B ∼3.0B 2.2B 3.1B

64.4 62.6 65.1 71.8

43.0 45.3 43.6 53.5

77.7 77.3 78.3 81.0

63.6 63.1 64.2 70.4

Table 3 Reasoning-intensive retrieval on MRMR (Zhang et al., 2025a), grouped by active parameters. Following the benchmark, we report NDCG@10 for all subtasks except Negation (Hit@1); Avg is the 11-subtask mean. † reported by the MRMR benchmark, with active parameters taken from the public model releases; other rows are our evaluations. Bold/underline mark best/second per column.

Knowledge Model

Theorem

Contradiction

Act. (B)

Art

Med.

Sci.

Hum. Math Phy. Eng. Bus. Neg. Design Traffic Avg

2.2 2.2 2.1 2.2 3.0 0.2 4.0 3.1

72.0 70.8 69.9 53.1 36.1 21.3 53.5 80.3

48.2 48.9 47.7 28.9 29.9 27.8 22.4 62.6

62.2 62.3 60.6 57.8 63.6 56.7 43.4 39.9 42.7 29.2 32.6 17.0 36.7 24.0 73.3 76.3

22.5 21.3 18.6 11.7 7.3 18.8 2.1 28.2

36.1 31.5 30.3 22.3 17.5 17.1 2.8 35.3

29.1 27.2 27.7 22.5 13.5 17.3 2.8 34.3

49.0 47.0 44.5 43.0 34.6 28.6 2.9 54.0

14.5 11.5 9.5 11.0 28.5 20.0 11.5 11.0

46.4 33.7 43.2 8.3 19.4 20.2 5.6 63.6

36.0 30.1 27.5 25.0 18.2 9.4 18.3 49.2

43.5 40.0 39.9 28.1 25.2 20.9 18.1 51.6

8.3 8.3 7.0 13.2 8.0 8.0 7.0 8.0

72.7 79.2 79.3 73.5 71.0 65.6 54.3 25.1

57.1 58.2 52.5 53.1 49.7 53.0 40.1 11.7

69.8 71.8 70.0 60.5 58.4 63.5 46.8 16.6

29.9 28.2 27.7 32.9 25.1 23.6 28.8 2.1

44.4 40.2 39.5 45.6 39.8 30.8 36.0 3.4

38.0 32.7 30.1 35.1 28.4 27.4 30.2 2.5

49.4 6.5 50.3 6.0 52.3 8.0 52.6 6.0 51.0 7.0 44.9 7.0 45.1 15.0 5.2 11.5

66.9 57.6 55.9 54.3 37.0 23.8 26.3 3.7

39.8 39.5 45.8 31.7 34.6 34.9 29.6 2.1

48.9 48.2 48.1 46.8 42.0 39.8 36.2 8.6

≤4B active parameters BToks UME-R1-2B PLUME VLM2Vec-V2 ColPali† VISTA† VLM2Vec† M O EMB-A3B ≥7B active parameters LaME-7B UME-R1-7B Ops-MM-Embedding† Embed-RL-4B UniME-V2-7B MM-Embed† GME-Qwen2-VL† E5-V†

63.3 66.1 67.8 69.8 59.6 62.8 45.6 10.8

the native obtained routers from massive language pretraining. Meanwhile, modality-wise MoE shows only marginal gains over the dense baseline (+0.6). In contrast, fine-tuning a native pretrained sparse MoE backbone delivers substantial gains, raising the overall benchmark average by +6.8 points. Ablations on Pooling and Routing Designs. Using the native sparse backbone, we next evaluate architectural and algorithmic adaptation choices for pooling and router designs. (Tab. 5): (1) Pooling Strategy: Replacing standard last-token Table 5 Ablations on pooling designs and router adaptions for MoE-based UME. App. B.2 defines the load-balancing and z losses, the conflict-aware pooling with a dedicated <emb> token provides the objective, and the task-ID and instruction priors. single largest improvement, raising the overall average from 69.50 to 70.34 (+0.84 points). (2) Router UpRouter Pooling Routing objective Image Video VisDoc Overall trained dating vs. Freezing: Keeping the pretrained router frozen performs identically to updating router paramelast ✗ ✓ 70.7 51.6 81.1 69.50 last LB + z ✓ 65.6 53.7 77.8 66.62 ters during fine-tuning (70.38 vs. 70.34). (3) Routing last conflict-aware ✓ 70.4 53.2 81.3 69.80 Objectives: Modifying the routing objective yields <emb> ✗ ✓ 71.1 54.5 81.1 70.34 <emb> task-ID prior ✓ 69.9 53.5 81.0 69.55 no benefit and often hurts performance. Auxiliary <emb> instruction prior ✓ 70.1 54.1 80.3 69.55 load-balancing and z-losses severely degrade accuracy <emb> ✗ ✗ 71.8 53.5 81.0 70.38 (66.62 vs. 69.50) by forcibly pulling tokens away from pre-trained expert assignments. Conflict-aware routing yields a negligible difference (+0.30 points, 69.80), while injecting task-level priors (task IDs or instructions) degrades overall accuracy (69.55). Both (2) and (3) indicates that the routing assignments learned during language pre-training remain effective for contrastive adaptation. Ablations on Training Hyperparameters. We systematically evaluate training sensitivity around this reference protocol on the 35B backbone (Fig. 6): (1) Batch Size: Global batch size is the most critical training factor. Halving the global batch size from 512 to 256 drops overall performance by 4.52 points, underscoring the reliance of contrastive learning objectives on large pools of in-batch negatives. (2) LoRA and temperature: Halving the LoRA rank reduces

Learning rate 72

MMEB-V2

Epochs

71.4

Batch size

LoRA rank

Logit scale

71.4 70.3

-1.02

70.3

-0.55 -1.10

-2.91

-4.52 5e-5

1e-4

256

512

Figure 6 Hyperparameter ablations on the 35B backbone. Each panel varies one setting.

performance by 2.91 points, while lowering the logit scale parameter from 50 to 20 incurs a 1.10-point penalty. (3) Learning Rate & Epochs: Reducing the learning rate from 10−4 to 5 × 10−5 improves overall accuracy from 70.34 to 71.36 (+1.02 points), advancing performance on 56 of 78 tasks. Training for a second epoch at this lower learning rate yields an additional +0.55 point gain.

5.3

Results and Analysis for Adaptive Computation on MoE-UME

Table 6 Summarized results on adaptive-computation. We show a representative budget for each adaptive compute type. Action

Method

Budget

Image

Video

VisDoc

Overall

FLOPs↓%

QPS↑ ×

—

No reduction

top-8 / keep all

71.0

54.5

81.1

70.3

—

1.00

68.2 (−2.8) 69.7 (−1.3) 69.4 (−1.6) 70.0 (−1.0) 70.1 (−0.9) 64.5 (−6.5) 67.3 (−3.7) 66.8 (−4.2) 66.3 (−4.7)

49.1 (−5.4) 53.4 (−1.1) 54.2 (−0.3) 52.6 (−1.9) 53.0 (−1.5) 46.8 (−7.7) 50.5 (−4.0) 51.2 (−3.3) 51.1 (−3.4)

79.3 (−1.8) 81.1 (+0.0) 80.5 (−0.6) 81.3 (+0.2) 81.4 (+0.3) 78.5 (−2.6) 79.4 (−1.7) 78.6 (−2.5) 78.6 (−2.5)

67.2 (−3.1) 69.4 (−0.9) 69.3 (−1.0) 69.5 (−0.8) 69.6 (−0.7) 64.7 (−5.6) 67.1 (−3.2) 66.8 (−3.5) 66.6 (−3.7)

— 12 32 ≈20 22.5 — — — —

— — — ≈1.40 1.46 — — — —

Training-based methods Adaptive top-k Expert skip Token pruning Layer skip

Combined

AdaMoE (App. E.1) Learned skip gate One-shot drop Trained gate (App. E.1) GRPO (gradual drop + skip) Global budget (App. E.1) Per-action, simultaneous (App. E.1) Per-action, alternating (App. E.1) Per-action, sequential (App. E.1)

2.94 experts dynamic drop 70% skip 20% dynamic 80/40% target drop 70 / skip 30 / layer 20 drop 70 / skip 29 / layer 20 drop 70 (frozen) + skip 49

Inference-only methods Adaptive top-k

Uniform top-k MoDES (DMT) (Huang et al., 2025b) 50% skip Random

70.5 (−0.5) 53.6 (−0.9) 80.9 (−0.2) 69.8 (−0.5) 70.4 (−0.6) 53.6 (−0.9) 80.9 (−0.2) 69.8 (−0.5) 63.2 (−7.8) 46.4 (−8.1) 77.5 (−3.6) 63.7 (−6.6)

11.6 13.8 11.6

1.19 1.18 1.19

Expert pruning

REAP (Lasby et al., 2025) MSAN (Liu et al., 2026) MAN (Liu et al., 2026) Router-prob

E=128

70.3 (−0.7) 53.8 (−0.7) 69.9 (−1.1) 54.0 (−0.5) 70.1 (−0.9) 53.6 (−0.9) 68.8 (−2.2) 52.7 (−1.8)

69.5 (−0.8) 69.4 (−0.9) 69.4 (−0.9) 68.7 (−1.6)

≈0

1.16

Token pruning

GDN-β Random

keep 50%

69.7 (−1.3) 53.5 (−1.0) 80.2 (−0.9) 69.2 (−1.1) 69.0 (−2.0) 52.8 (−1.7) 79.4 (−1.7) 68.4 (−1.9)

22.8

1.39

Combined

MoDES + GDN-β

30% skip, keep 50%

69.7 (−1.3) 53.4 (−1.1) 80.0 (−1.1) 69.1 (−1.2)

28.9

1.44

80.1 (−1.0) 80.2 (−0.9) 80.3 (−0.8) 80.4 (−0.7)

Main results. We find that adaptive computation generally yields strong accuracy–compute trade-offs for MoE-based UME. As shown in Tab. 6, all evaluated compute-reduction actions maintain performance within 1.2 points of the unreduced baseline (70.3) while substantially lowering execution cost. For instance, reducing top-k experts activation from eight to four uniformly incurs a minor 0.5-point accuracy drop while removing 11.6% of total FLOPs and increasing throughput by 1.19×. For sequence-level reduction, one-shot token pruning removes 70% of visual tokens, saving 32% FLOPs with a < 1.0-point loss. Similarly, static expert pruning reduces total expert capacity by half (256 → 128) with a < 1.0-point drop. These results confirm that MoE-based embedding models possess substantial execution redundancy across both sequence and parameter dimensions. Token pruning and adaptive top-k provide the strongest accuracy–compute efficiency. Strategies that reduce compute per token or per sequence dominate performance in Tab. 6. Adaptive top-k routing is the most compute-efficient per-token action: uniform top-4 routing costs only 0.5 points, which is closed to calibration-based 10

Adaptive top-k

Token pruning

Expert skipping

Expert pruning

MMEB-V2 accuracy (%)

70 68 66 64 62 60 58

Uniform top-k MoDES (DMT)

56 0

20 40 60 80 routed expert work skipped (%)

Learned skip gate (train) β-threshold β-Otsu Max-logit Otsu

GDN-β One-shot drop (train) GRPO (train) 0

20 40 60 visual tokens dropped (%)

REAP MAN MSAN Router-prob

10 20 30 40 50 routed expert work skipped (%)

256

192 128 experts retained E

Figure 7 Budget–accuracy curves per adaptive-computation action. More experiments and details are reported in Tab. 6 and App. F.

adaptive top-k methods such as MoDES (69.8). Empirically we find that this accuracy relies heavily on the pretrained router’s ranking quality: randomly selecting four experts cost scores to degrade notably (App. E.2). Token pruning trades slightly higher accuracy loss for significantly greater compute reduction: keeping 50% visual tokens costs 1.15 points while reducing FLOPs by 22.8% and accelerating throughput by 1.39×. Inference-only adaptive computation approaches are competitive with training-based methods. For adaptive top-k, inference-only methods outperform training-based methods: a trained adaptive top-k method (Zeng et al., 2024) converges to an average of 2.94 experts per token (63.3% expert skip) and scores 67.2, whereas inference-only MoDES achieves 69.8 at a 57.1% skip rate and 68.8 at a 77.5% skip rate (+1.6 higher despite activating less experts). This indicates that the pretrained router’s internal logit distribution already provides a near-optimal ranking without requiring dedicated training. For token pruning, our proposed GDN β-based heuristics remain highly competitive with trained pruning gate at moderate saving budgets: the inference-only GDN-β policy removes 22.8% of FLOPs for a 1.1-point drop, performing close to the trained one-shot dropper (32% FLOP reduction for a 1.0-point drop). On the other hand, training-based approaches provide a distinct advantage primarily under extreme pruning regimes (70–75% visual tokens removed), where the trained dropper maintains a 1.0 drop compared to 4.3 for the heuristic policy (Fig. 7). Our implementations effectively converts FLOPS-saving to QPS improvement. As shown packed varlen 55.0 in Fig. 8, our two implementations are able to con× masked 60 52.5 vert most of the logical compute reduction into wall× × clock savings: packed varlen converts 63–82% of the 40 50.0 SonicMoE FLOPs it removes and SonicMoE 56–58%, against 41% grouped GEMM 47.5 for unmodified grouped GEMM and nothing at all for naive expert loop 20 × 45.0 masking. Meanwhile, the two baselines approaches for adaptive top-k are less effective due to two reasons. 0 0 10 20 30 0 10 20 30 (1) Unmodified grouped GEMM is tile-bound: it runs ↑ each expert’s assigned tokens in tiles of at least 128 Figure 8 Transferring FLOPs saving into wall-clock speedup under rows, so an expert holding nj tokens costs ⌈nj /128⌉ baseline (grey) and our implementations. Left: token pruning; Right: tiles however few of them survive, and an expert-skip adaptive top-k. policy saves time only once it removes enough of them to eliminate a whole tile; the gain also saturates past k ≈ 4, since what remains is attention, the shared expert and dispatch, so aggressive adaptive-k budgets cost accuracy for almost no latency. (2) The naive HuggingFace expert loop responds strongly to sparsity, converting 163–174%, yet runs 5.7–8.0× slower, so it exaggerates the benefit of expert. ↑

1.24

1.16

QPS

1.20

2.49

FLOPs reduced (%)

Depth-wise adaptive compute allocation across modalities. Fig. 9 plots each action’s firing rate under each modality across layers. Token pruning acts primarily at initial layers and exhibits strong modality sensitivity. High temporal and spatial redundancy enables immediate pruning of video and image tokens. Conversely, visual documents contain dense layout text, and text queries encode task instructions that must be preserved early to contextualize representations before gradual pruning. Expert skipping behaves differently between early (< 24) and late layers: the skip rate vary more in early layers, where text tends to have higher skip rate: as skipping removes only expert computation, text tokens remain present for attention while relying on shared experts for boilerplate instructions; in later layers, almost all modalities are being skipped at a similar rate. Interestingly, the skip rate is almost zero at the final layer, to construct the output embedding. On the other hand, adaptive top-k tends to allocate more experts for text modalities, especially in earlier layers.

Token pruning (GRPO)

0.10 0.05

0.8 0.8

mean skip rate

0.15

1.0

mean skip rate

mean prune rate

0.20

Adaptive top-k (MoDES)

Expert skipping image text video visdoc

0.6 0.4

0.0 0

0.4 0.2

0.2

0.00

0.6

0.0 0

decoder layer

Figure 9 Depth-wise adaptive compute allocation across modalities.

GDN-β vs. emb attention β vs. emb attention by token type β functions as an independent salience sig0.8 GDN (left axis) Spearman 0.025 0.7 emb attention (right axis) Pearson 0.7 nal. GDN β provides an informative token pruning 0.6 0.020 0.6 signal that operates independently of standard atten0.5 0.5 0.015 tion mechanisms. While attention distributions are 0.4 0.4 spiky and heavily concentrated, β tends to be more 0.3 0.010 0.3 0.2 spatially and temporally smooth. Moreover, we find β 0.2 0.005 0.1 0.1 to have a more even focus across different modalities 0.0 0.0 0.000 in M O EMB, compared to attention scores, which are Text Image Text Image Video Visdoc Video mostly concentrated on text tokens. Despite β’s distinct Figure 10 β vs. attention score. Left: per-modality agreement (Spearman properties from attention scores, pruning tokens by β ρ and Pearson r); Right: mean β (left axis) and mean <emb> attention performs comparably to the trained token pruning gate (right axis) by token type. (Fig. 7), confirming its utility as an effective salience metric. App. F reports further analysis of β: its per-layer profile, its agreement with the <emb> attention row and with received attention mass, and its spread by token type. ⟨

⟩

⟨

⟩

⟨

⟩

0.42

0.28

0.34

0.32

mean ⟨emb⟩ attention

agreement with emb⟩ attention ⟨

0.19

mean write strength β

0.36

0.31

Limitations and future work

While M O EMB achieves SOTA performance over models trained with public MMEB-V2 training data, it’s currently lagging behind models trained with external data. Also, it currently does not support audio as input modality. Our future work include training M O EMB under larger scale of data, and to include more input modalities.

Conclusion

We presented M O EMB, which scales the capacity of universal multimodal embedder along the expert axis rather than embedding dimension or test-time reasoning. M O EMB adds over 10 accuracy points at near-constant active compute, establishing a new state of the art among public-data models on MMEB-V2 and MRMR. Furthermore, our systematic study of adaptive computation demonstrates that MoE embedders can eliminate up to half of their logical compute for under a one-point accuracy drop, supported by Gated DeltaNet write strength β as a zero-cost token salience signal and custom serving kernels that successfully convert theoretical FLOP savings into deployed throughput gains.

AI use statement Anthropic Claude Code (Opus 4.8 and Opus 5) was used extensively for drafting code, aggregating results, and drafting manuscript. The authors verified experiment artifacts, numerical claims, and citations and take responsibility for the content.

References Sikai Bai, Haoxi Li, Jie Zhang, Zicong Hong, and Song Guo. DiEP: Adaptive mixture-of-experts compression through differentiable expert pruning. arXiv preprint arXiv:2509.16105, 2025. https://arxiv.org/abs/2509.16105. Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to ponder. arXiv preprint arXiv:2107.05407, 2021. https://arxiv.org/abs/2107.05407. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. https://arxiv.org/abs/2403.06764. Jianpeng Cheng, Xian Wu, Jiangfan Zhang, Wentao Bao, Chaitanya Ahuja, Shlok Kumar Mishra, Hanchao Yu, Yang Gao, Fan Xia, Qi Guo, Shaodan Zhai, Xiangjun Fan, and Jun Xiao. TTE-Flash: Accelerating reasoning-based multimodal representations via think-then-embed tokens. arXiv preprint arXiv:2605.16638, 2026. https://arxiv.org/abs/2605.16638. Xuanming Cui, Jianpeng Cheng, Hong-you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xichen Pan, Chaitanya Ahuja, Shlok Kumar Mishra, Yonghuan Yang, Jun Xiao, Qi Guo, Ser-Nam Lim, Aashu Singh, and Xiangjun Fan. Think then embed: Generative context improves multimodal embedding. arXiv preprint arXiv:2510.05014, 2025. https://arxiv.org/abs/2510.05014. Xuanming Cui, Hong-You Chen, Hao Yu, Hao Yuan, Zihao Wang, Shlok Kumar Mishra, Hanchao Yu, Yonghuan Yang, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng, Qi Guo, and Xiangjun Fan. Reason to contrast: A cascaded multimodal retrieval framework. arXiv preprint arXiv:2602.23369, 2026. https://arxiv.org/abs/2602.23369. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024. https://arxiv.org/abs/2307.08691. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. https://arxiv.org/abs/2205.14135. DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. https://arxiv.org/abs/2412. 19437. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. ColPali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. https://openreview.net/forum?id=ogjBpZ8uSi. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Efficient sparse training with mixture-of-experts. arXiv preprint arXiv:2211.15841, 2022. https://arxiv.org/abs/2211.15841. Tiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang, Weidong Cai, Jiankang Deng, and Lidong Bing. UniME-V2: MLLM-as-a-judge for universal multimodal embedding learning. arXiv preprint arXiv:2510.13515, 2025. https://arxiv.org/abs/2510.13515. Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, and Tri Dao. SonicMoE: Accelerating MoE with IO and tile-aware optimizations. arXiv preprint arXiv:2512.14080, 2025. https://arxiv.org/abs/2512.14080. Chenwei He, Xiangzhao Hao, Tianyu Yang, Yuxiang Ma, Yuheng Jia, Lingxiang Wu, Chaoyang Zhao, Haiyun Guo, and Jinqiao Wang. PLUME: Latent reasoning based universal multimodal embedding. arXiv preprint arXiv:2604.02073, 2026. Haohang Huang, Xuan Lu, Mingyi Su, Xuan Zhang, Ziyan Jiang, Ping Nie, Kai Zou, Tomas Pfister, Wenhu Chen, Wei Zhang, Xiaoyu Shen, and Rui Meng. MMEB-V3: Measuring the performance gaps of omni-modality embedding models. arXiv preprint arXiv:2604.23321, 2026. https://arxiv.org/abs/2604.23321. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in MoE models. arXiv preprint arXiv:2403.07652, 2024. https://arxiv.org/abs/2403.07652. Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. Dynamic-LLaVA: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876, 2025a. https://arxiv.org/abs/2412.00876.

Yushi Huang, Zining Wang, Zhihang Yuan, Yifu Ding, Ruihao Gong, Jinyang Guo, Xianglong Liu, and Jun Zhang. MoDES: Accelerating mixture-of-experts multimodal large language models via dynamic expert skipping. arXiv preprint arXiv:2511.15690, 2025b. https://arxiv.org/abs/2511.15690. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. https://arxiv.org/abs/2401.04088. Haonan Jiang, Yuji Wang, Yongjie Zhu, Xin Lu, Wenyu Qin, Meng Wang, Pengfei Wan, and Yansong Tang. Embed-RL: Reinforcement learning for reasoning-driven multimodal embeddings. arXiv preprint arXiv:2602.13823, 2026. https://arxiv.org/abs/ 2602.13823. Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2Vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2025. https://arxiv.org/abs/2410.05160. Omar Khattab and Matei Zaharia. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. arXiv preprint arXiv:2004.12832, 2020. https://arxiv.org/abs/2004.12832. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 30233–30249, 2022. https://arxiv.org/abs/2205.13147. Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. UME-R1: Exploring reasoning-driven generative multimodal embeddings. arXiv preprint arXiv:2511.00405, 2025. https://arxiv.org/abs/2511.00405. Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. REAP the experts: Why pruning prevails for one-shot MoE compression. arXiv preprint arXiv:2510.13999, 2025. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024. https: //arxiv.org/abs/2405.17428. Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, et al. Gemini embedding: Generalizable embeddings from Gemini. arXiv preprint arXiv:2503.07891, 2025. https://arxiv.org/abs/2503.07891. Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2025. https: //arxiv.org/abs/2410.05993. Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. Adaptive gating in mixture-of-experts based language models. arXiv preprint arXiv:2310.07188, 2023. https://arxiv.org/abs/2310.07188. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-VL-Embedding and Qwen3-VL-Reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026. https://arxiv.org/abs/2601.04720. Xianming Li, Zongxi Li, Jing Li, Haoran Xie, and Qing Li. 2D matryoshka sentence embeddings. arXiv preprint arXiv:2402.14776, 2024. https://arxiv.org/abs/2402.14776. Ziyue Li and Tianyi Zhou. Your mixture-of-experts LLM is secretly an embedding model for free. arXiv preprint arXiv:2410.10814, 2024. https://arxiv.org/abs/2410.10814. Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-Embed: Universal multimodal retrieval with multimodal llms. arXiv preprint arXiv:2411.02571, 2025. https://arxiv.org/abs/2411. 02571. Dennis Liu, Zijie Yan, Xin Yao, Tong Liu, Vijay Korthikanti, Evan Wu, Shiqing Fan, Gao Deng, Hongxiao Bai, Jianbin Chang, Ashwath Aithal, Michael Andersch, Mohammad Shoeybi, Jiajie Yao, Chandler Zhou, David Wu, Xipeng Li, and June Yang. Moe parallel folding: Heterogeneous parallelism mappings for efficient large-scale moe model training with megatron core. arXiv preprint arXiv:2504.14960, 2025.

Zongfang Liu, Jinghui Zhang, Zijian Ma, Guangyi Chen, and Xin Yuan. How to score experts for one-shot MoE expert pruning: A unified formulation and selection principle. arXiv preprint arXiv:2606.15716, 2026. Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. arXiv preprint arXiv:2402.14800, 2024. https://arxiv.org/abs/2402.14800. Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, and Aixin Sun. Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings. arXiv preprint arXiv:2506.04997, 2025. https://arxiv.org/abs/2506.04997. Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. VLM2Vec-V2: Advancing multimodal embedding for videos, images, and visual documents. arXiv preprint arXiv:2507.04590, 2025. https://arxiv.org/abs/2507.04590. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2023. https: //arxiv.org/abs/2210.07316. Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. OLMoE: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024. https://arxiv.org/abs/2409.02060. Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9 (1):62–66, 1979. Qwen Team. Qwen3.5, 2026. https://qwen.ai/blog?id=qwen3.5. Official model cards for Qwen3.5-35B-A3B and Qwen3.5-122B-A10B. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024. https: //arxiv.org/abs/2404.02258. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024. https://arxiv.org/abs/2403.15388. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. https://arxiv.org/abs/2402.03300. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. https://arxiv.org/ abs/1701.06538. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H. S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jusen Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao, Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochen Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanming Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun

Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiezhong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaochen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yuye Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jinjing Xu, L. H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B. H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijing Zhang, Bin Zhao, Chenguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang, and Xinxing Zu. Kimi k3: Open frontier intelligence, 2026. https://arxiv.org/abs/2607.24653. Raghuveer Thirukovalluru, Rui Meng, Ye Liu, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, and Bhuwan Dhingra. Breaking the batch barrier (B3) of contrastive learning via smart batch mining. arXiv preprint arXiv:2505.11293, 2025. https://arxiv.org/abs/2505.11293. Yebo Wu, Feng Liu, Ziwei Xie, Zhiyuan Liu, Changwang Zhang, Jun Wang, and Li Li. TSEmbed: Unlocking task scaling in universal multimodal embeddings. arXiv preprint arXiv:2603.04772, 2026. https://arxiv.org/abs/2603.04772. Zilin Xiao, Qi Ma, Mengting Gu, Chun-cheng Jason Chen, Xintao Chen, Vicente Ordonez, and Vijai Mohan. MetaEmbed: Scaling multimodal retrieval at test-time with flexible late interaction. arXiv preprint arXiv:2509.18095, 2025. https: //arxiv.org/abs/2509.18095. Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. https://github.com/fla-org/flash-linear-attention. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024. https://arxiv.org/abs/2412.06464. Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809–10818, 2022. Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. AdaMoE: Token-adaptive routing with null experts for mixture-of-experts language models. arXiv preprint arXiv:2406.13233, 2024. https://arxiv.org/abs/2406.13233. Longxiang Zhang, Weilong Dai, Guanghao Zhang, Hao Jiang, and Pipei Huang. Think when needed: Adaptive reasoning-driven multimodal embeddings with a dual-LoRA architecture. arXiv preprint arXiv:2605.14448, 2026. https://arxiv.org/abs/ 2605.14448. Siyue Zhang, Yuan Gao, Xiao Zhou, Yilun Zhao, Tingyu Song, Arman Cohan, Anh Tuan Luu, and Chen Zhao. MRMR: A realistic and expert-level multidisciplinary benchmark for reasoning-intensive multimodal retrieval. arXiv preprint arXiv:2510.09510, 2025a. https://arxiv.org/abs/2510.09510. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025b. https://arxiv.org/abs/2506.05176. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022. https://arxiv.org/abs/ 2202.08906.

Appendix A Related Work

B Implementation Details

B.1 Model Architecture and Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.2 Router Adaptation Objectives and Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C Measurement Protocol for FLOPs and Throughput

C.1 Counting Canonical FLOPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Accuracy–Throughput Figure Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.3 Converting FLOP Savings into Wall-Clock Speedups . . . . . . . . . . . . . . . . . . . . . . . . . .

D Additional Studies on MoE Design

D.1 Dense-to-MoE Upcycling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.2 Capacity Ablation on Specialist and Generalist Models . . . . . . . . . . . . . . . . . . . . . . . . .

D.3 Expert Routing Behavior Across Modality and Task . . . . . . . . . . . . . . . . . . . . . . . . . . .

E Additional Details for Adaptive Computation

E.1 Training-Based Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.2 Inference-Based Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F Additional Analysis on GDN Write Strength β

G Per-Task MMEB-V2 Results

Related Work

Universal Multimodal Embeddings. Recent advances in multimodal representation learning adapt vision-language models into general-purpose encoders evaluated on broad benchmark suites such as MMEB and MMEB-V2 (Jiang et al., 2025; Meng et al., 2025). Prior efforts to improve universal embedding performance primarily focus on data supervision, including hard-negative mining (Thirukovalluru et al., 2025; Lin et al., 2025; Gu et al., 2025) and multi-stage reranker distillation (Li et al., 2026; Cui et al., 2026). Another emerging direction leverages test-time reasoning, optimizing autoregressive thought generation prior to embedding emission (Lan et al., 2025; Jiang et al., 2026; Cheng et al., 2026; Zhang et al., 2026). In contrast to data-centric scaling or test-time generation overhead, we examine the complementary axis of encoder parameter capacity, evaluating how native sparse expert scaling improves contrastive representations. Our work also differs fundamentally from existing MoE-based embedding approaches. TSEmbed (Wu et al., 2026) employs fixed task- or modality-wise LoRA routing, which caps expert capacity by the task taxonomy. We instead adapt native token-level sparse MoE backbones, allowing capacity to scale with expert count via dynamic, learned routing. Furthermore, unlike MoEE (Li and Zhou, 2024), which extracts off-the-shelf text embeddings directly from frozen router probabilities, we fine-tune sparse multimodal architectures to pool high-quality representation states. Sparse Mixture-of-Experts Backbones. Sparsely gated MoE architectures scale model capacity while holding per-token compute constant (Shazeer et al., 2017; Fedus et al., 2022). Modern foundation architectures increasingly adopt fine-grained sparse routing to expand representation power, as demonstrated in Mixtral (Jiang et al., 2024), DeepSeek-V3 (DeepSeek-AI, 2024), OLMoE (Muennighoff et al., 2024), Kimi K3 (Team et al., 2026), Qwen3.5 (Qwen Team, 2026), and Aria (Li et al., 2025). However, these routing mechanisms are pre-trained exclusively under next-token prediction. We study how these pre-trained sparse routing distributions transfer when adapted contrastively to produce fixed-length vector representations. Efficient Embeddings and Adaptive Computation. Adjustable-cost embedding methods have been studied via Matryoshka dimension truncation (Kusupati et al., 2022; Li et al., 2024), variable late-interaction vectors (Faysse et al., 2025; Xiao et al., 2025), and visual patch reduction (Ma et al., 2025). However, these works focus on reducing retrieval and storage cost, not the active computation for calculating the embedding. In parallel, adaptive computation frameworks reduce inference latency via dynamic depth skipping (Banino et al., 2021; Raposo et al., 2024), visual token pruning (Chen et al., 2024; Huang et al., 2025a), and adaptive top-k routing (Huang et al., 2025b), but these works mostly focus on visual classification and next token prediction (text generation) tasks, are mostly explored separately, and have not been thoroughly tested on embedding task. We unify these paradigms by systematically comparing token-, expert-, and layer-level execution controls within a single MoE multimodal embedder, identifying which mechanisms lead to best accuracy-efficiency tradeoff, and which best translate theoretical FLOP reductions into deployed wall-clock speedups.

Implementation Details

B.1

Model Architecture and Experimental Setup

Backbone Configurations.We evaluate two primary variants built upon native sparse MLLM backbones: M O EMBA3B (3.1B active parameters) and M O EMB-A10B (10.2B active parameters). Both variants employ E = 256 routed experts per layer with top-8 routing (k = 8) and a single shared expert. M O EMB-A3B uses a hidden dimension of d = 2048, 40 decoder layers (30 Gated-DeltaNet and 10 full-attention blocks), and an expert intermediate dimension of dffn = 512. M O EMB-A10B expands model capacity with d = 3072, 48 decoder layers, and dffn = 1024. Training Data. Models are fine-tuned on a multi-task mixture comprising 24 sampling streams across image, video, and visual-document retrieval. This includes 17 image-text tasks from MMEB (Jiang et al., 2025), 4 visual-document streams from ViDoRe and VisRAG, and 3 video retrieval streams derived from LLaVA-Hound. Following our standard protocol, we exclude generated chain-of-thought sequences from training targets. Complete dataset sampling weights are detailed in Tab. 7. Training setups. We train for two epochs using BF16 precision and the AdamW optimizer (β1 = 0.9, β2 = 0.98, weight decay = 0.01). The learning rate warms up to 10−4 over 10 steps, followed by a cosine decay to 10−5 . Training uses a global batch size of 512 across 64 H200 GPUs with a maximum sequence length of 8,192 tokens. To

enable parameter-efficient adaptation, we apply LoRA (r = 64, α = 128) to the vision encoder, attention projections, and expert feed-forward layers, yielding 73.6M trainable parameters for M O EMB-A3B and 196.6M for M O EMB-A10B. Table 7 Training subsets. Weights are relative and do not sum to one. Subset A-OKVQA DocVQA MSCOCO i2t NIGHTS VisDial VisualNews t2i ViDoRe subset 2 Video caption

B.2

Weight 0.26 0.84 2.58 0.23 3.75 3.49 5.00 5.27

Subset

Weight

ChartQA ImageNet-1K MSCOCO image pairs OK-VQA Visual7W WebQA VisRAG subset 1 Video QA

0.35 2.25 3.78 0.25 1.68 0.23 6.00 4.38

Subset CIRR InfographicsVQA MSCOCO t2i SUN397 VisualNews i2t ViDoRe subset 1 VisRAG subset 2 Video retrieval

Weight 0.43 0.31 2.32 0.22 2.91 5.00 6.00 5.76

Router Adaptation Objectives and Priors

We detail the formulations for the router adaptation variants evaluated in Tab. 5. Load-Balancing and z-Loss. To evaluate whether standard pre-training regularizers aid contrastive adaptation, we PE include a joint auxiliary loss Laux = λlb Llb + λz Lz . The load-balancing objective Llb = E j=1 fj Pj encourages uniform expert allocation (Fedus et al., 2022), where fj is the fraction of tokens routed to expert j and Pj is the mean PB PE router probability assigned to expert j. The router z-loss Lz = B1 b=1 (log j=1 egbj )2 penalizes large pre-softmax router logits g to stabilize routing dynamics (Zoph et al., 2022). We set weighting coefficients to λlb = 0.01 and λz = 0.001. Conflict-Aware Routing. To mitigate multi-task expert interference, this objective forces task pairs with opposing router gradients onto distinct experts. For task t at layer ℓ, we maintain exponential moving averages of its expert (ℓ) (ℓ) assignment distribution π̄t ∈ RE and router gradient gt . The gradient conflict between current task s and candidate (ℓ) (ℓ) (ℓ) task t is defined as Cst = clamp(− cos(gs , gt ), 0, 1). The objective maximizes the conflict-weighted Jensen– Shannon (JS) divergence between routing distributions across layers: L

1 X 1 X (ℓ) (ℓ) , Cst JS π̄s(ℓ) ∥ π̄t Lconf = − |P| L t∈P

(5)

ℓ=1

where P is the set of tasks with non-zero conflict relative to task s, and JS(p ∥ q) = 21 KL(p ∥ m) + 12 KL(q ∥ m) for mixture distribution m = 12 (p + q). This pushes conflicting task pairs toward disjoint expert subsets while exerting no penalty on cooperative or orthogonal pairs. Task-ID and Instruction Priors. We test conditioning router assignments on task information by injecting an additive (ℓ) bias vector b(ℓ) ∈ RE into the router logits: g (ℓ) = Wr h + b(ℓ) . We test two setups: (1) task-ID prior: b(ℓ) is obtained from a learned task embedding table with hidden size 256 and dropout 0.3. (2) instruction prior, b(ℓ) is projected from the mean pooling of instruction tokens at layer ℓ, updating the routing bias dynamically across network depth.

Measurement Protocol for FLOPs and Throughput

C.1

Counting Canonical FLOPs

Logical FLOP counts are computed using the same query/target (whichever side contains multimodal input) subset per task across all 78 MMEB-V2 tasks. As the regular FlopsCounter does not capture fused kernels such as FlashAttention (Dao, 2024) and linear attention (Yang and Zhang, 2024), their FLOPS are added analytically. Grouped expert GEMMs are computed based on physically executed row volumes to prevent double-counting pruned expert allocations.

C.2

Accuracy–Throughput Figure Construction

In this section we provide details for constructing Fig. 1. 19

M O EMB Configurations. The expert-scaling curve for M O EMB-A3B evaluates the performance with different number of total experts E ∈ {32, 64, 128, 192, 256}, constructed by pruning the full E = 256 model using REAP (Lasby et al., 2025) under the reconstruction-error criterion (Eq. 4). M O EMB-A10B represents the full E = 256 backbone evaluated under the identical protocol. The single adaptive compute marker corresponds to the GRPO-trained policy (§4.1). Baseline Models. Autoregressive think-then-embed systems, such as Embed-RL (Jiang et al., 2026) and UME-R1 (Lan et al., 2025), are evaluated with reasoning generation both enabled and disabled to highlight their accuracy-throughput tradeoff via reasoning. PLUME (He et al., 2026) scales test-time compute across latent reasoning steps K ∈ {4, 6, 8}, under which throughput drops from 4.5 to 3.2 queries/sec. VLM2Vec-V2 (Meng et al., 2025) serve as single-vector dense references.

C.3

Converting FLOP Savings into Wall-Clock Speedups

This section details the empirical latency measurements supporting §4.3 and Fig. 8. Sequential Expert Looping. Sequential expert execution used in some existing MoE-efficiency methods (Lu et al., 2024) incurs heavy kernel launch overhead, running 5.7–8.0× slower than fused grouped GEMMs (1189.5 ms vs. 210.2 ms for grouped GEMM at B = 8, S = 1024; Tab. 8). Although sequential loops exhibit high sensitivity to expert sparsity (reducing latency by 32.3–59.9%), this relative gain stems from high baseline launch overheads rather than efficient execution. Table 8 Different throughput and FLOPS corresponding to different expert skip rate, for different MoE kernels. Throughput is queries per second on one H200. Policy

Throughput (QPS)

Mean k

Skip (%)

FLOPs ↓%

Naive loop

Grouped GEMM

SonicMoE

8.00 4.02 2.35 1.70 1.30 1.10

0.0 49.8 70.6 78.8 83.8 86.2

0.0 19.8 28.1 31.4 33.4 34.3

6.7 9.9 12.7 14.7 16.0 16.8

38.1 41.4 43.0 43.7 44.1 44.3

53.6 60.5 63.7 65.2 66.1 66.4

Tile-Bound Constraints in Grouped GEMMs. Grouped GEMM kernels (Gale et al., 2022) process each expert’s tokens in tiles of at least M = 128 rows, and a partly filled tile costs mostly the same time as a full one. An expert holding nj routed rows therefore requires ⌈nj /M ⌉ tiles, and skipping tokens does not produce actual wall-clock saving until it empties a whole tile. Whether it does depends on how many rows an expert holds to begin with. Under top-8 routing a batch of T tokens produces 8T routed token–expert pairs, which spread over E = 256 experts give nj ≈ 8T /E = T /32 rows apiece when the router is balanced, so the count grows with sequence length. At T = 4,096 tokens that is roughly 128 rows per expert. A 52.9% skip policy cuts that to about 50, but both amounts occupy a single tile, so the kernel executes the work it would have executed anyway while the model still spends time on the additional skip gates, which results in a 0.91× slowdown. At T = 8,192 each expert instead holds around 256 rows, which is two tiles. There the same kind of policy at 69.6% skip cuts the count to about 78 rows, one tile (⌈256/128⌉ = 2 → ⌈78/128⌉ = 1), and removing that second tile is a genuine saving of 1.11×. This suggests that the wall-clock speedup for adaptive top-k or expert skipping also depends on the sequence length or the batch size. Backend Customization and Speedup Saturation. Hardware speedups are primarily driven by backend kernel design (Fig. 8b). Replacing standard grouped GEMM with SonicMoE (Guo et al., 2025) kernel yields an 1.40× speedup at fixed k = 8 (210.2 ms vs. 149.2 ms at S = 8192). Within SonicMoE, lowering active expert width from k = 8 to k ≈ 4 provides an additional 1.11–1.24× speedup. Further reduction to k ≈ 1.1

Table 9 Token pruning under masking and physical compaction. Masked

Packed varlen

Keep

QPS

Speedup

FLOPs ↓%

QPS

Speedup

1.00 0.75 0.50 0.25

45.7 44.1 44.0 44.0

1.061 1.025 1.023 1.023

0.0 9.9 19.7 29.6

45.7 48.6 53.4 54.9

1.024 1.088 1.194 1.228

yields diminishing returns (1.17× total for grouped GEMM, 1.74× for SonicMoE; Tab. 8) as execution becomes dominated by non-routed operations, including self-attention, shared experts, and token dispatch. Physical Sequence Compaction vs. Spatial Masking. Token pruning requires physical sequence compaction to achieve wall-clock latency reductions. Spatial masking preserves activation tensor shapes ([B, S, H]), resulting in a flat 1.02× execution speedup across all pruning budgets (Fig. 8a). In contrast, physically packing remaining sequence tokens post-pruning via FlashAttention-2 varlen function (Dao, 2024) shortens activation tensors across downstream layers, yielding speedups of 1.088×, 1.194×, and 1.228× at 75%, 50%, and 25% visual token keep ratios (Tab. 9).

Additional Studies on MoE Design

This section expands the design and capacity ablations of §5.2.

D.1

Dense-to-MoE Upcycling

To evaluate whether expert capacity can be synthesized during downstream fine-tuning, we upcycle a dense 2B multimodal encoder into a sparse MoE architecture by varying structural granularities and router initialization strategies. Structural Granularity. A virtual group count G determines how the dense feed-forward network (FFN) is partitioned. For coarse upcycling (G = 1), each expert is a full replica of the original FFN (dexpert = dffn ), where top-1 routing reproduces the dense baseline’s FLOP footprint. For fine-grained upcycling (G > 1), the FFN is sliced along its intermediate dimension into G narrower experts (dexpert = dffn /G), requiring k = G active experts per token to match the capacity of the dense layer. Router Initialization. Because the router gate is newly introduced, its initialization controls initial expert activation patterns. We test three initialization methods: (1) random initialization (N (0, 0.02)) breaks symmetry between identical experts; (2) tiled initialization replicates the dense gate row across all experts; and (3) zero+bias sets initial pre-softmax logits to zero with a positive bias, ensuring every FFN slice is selected from the first step so the layer matches dense output prior to adaptation. Table 10 Dense-to-MoE upcycling on the 2B encoder. Image, Video and VisDoc are the 36, 18 and 24 task averages; Overall is over all 78. E

Init

Act.

Dense reference 4 4 32 32 8 8

1 1 1 4 4 4

2 1 1 4 4 4

tiled random random zero+bias zero+bias tiled

2× 1× 1× 1× 1× 1×

Image

Video

VisDoc

Overall

62.5

42.6

77.2

62.41

62.6 63.8 61.0 60.3 59.5 59.9

45.3 40.0 38.8 40.0 40.4 34.9

77.3 77.2 75.2 73.4 73.8 72.1

63.12 62.44 60.26 59.64 59.50 57.88

Results. As shown in Tab. 10, upcycling fails to improve representation quality over the dense baseline under same active compute. Coarse top-1 upcycling (E = 4, G = 1) performs comparably to the dense reference (62.44 vs. 62.41), while increasing expert count (E = 32) causes performance to drop to 60.26. Fine-grained partitioning (G = 4) similarly reduces overall accuracy below 60.0. Outperforming the dense baseline requires doubling the active compute per token (2×, top-2 routing), reaching 63.12. These results confirm that synthesizing expert capacity during contrastive fine-tuning is ineffective, highlighting the necessity of native MoE backbones pre-trained at scale.

D.2

Capacity Ablation on Specialist and Generalist Models

Experimental Protocol. To evaluate how model capacity impacts multi-task adaptation, we evaluate multi-task generalists against task-group specialists across three backbones: Qwen3.5-2B (dense, 2B active), Qwen3.5-9B (dense, 9B active), and Qwen3.5-35B-A3B (MoE, 3.1B active). Specialist models are fine-tuned independently on individual task groups (Image Classification and Image Retrieval), whereas generalists are fine-tuned on the full dataset mixture. We measure multi-task capacity degradation using the generalist-specialist performance gap. All configurations share 21

identical training hyperparameters (global batch size 256, learning rate 5 × 10−5 , LoRA rank 64, α = 128, <emb> pooling, 1 epoch per setting).

Cross-task gradient conflict is negligible mean 0.020 · max 0.112 (scale 0–1) .03

.01

.03

.02

.05

.00

.05

.01

.09

.08

.01

.03

.11

.08

.06

.03

.04

.01

.04

.06

.01

.02

.00

Img-Cls

.03

Img-VQA

.05

T2I

.01

.00

I2T

.03

.05

.09

I2I

.01

.08

Vid-Ret

.02

.01

.06

.04

Vid-Cls

.03

.01

.06

Vid-QA

.01

.02

.00

Moment

.11

.01

0.14

gradient conflict C = clamp(−cos(gi, gj), 0, 1)

Visdoc

0.12

0.10

0.08 0.06

.02

0.04 0.02

.02

I2 Vi d I -R et Vi d -C ls Vi d -Q A Mo me nt

T2 I I2T

Vi s do c Im g-C Im ls g-V QA

0.00

Figure 11 Pairwise conflict C = clamp(− cos(gi , gj ), 0, 1) on router parameters.

Minimal Cross-Task Gradient Conflict. A primary hypothesis for multi-task performance degradation is destructive gradient interference across task streams. To test this, we compute the pairwise cosine conflict C = clamp(− cos(gi , gj ), 0, 1) between per-category router parameter gradients g across the ten MMEB-V2 categories. As shown in Fig. 11, router gradient conflict is negligible (mean 0.020, max 0.112). Because categories do not exert opposing gradient updates on router parameters, generalist degradation stems primarily from total capacity constraints rather than inter-task gradient competition. Expanding expert capacity directly addresses this limitation 3. Table 11 Specialist vs. generalist by backbone. Group scores are unweighted averages over the group’s tasks; Mean is the flat mean of the two group scores and is what Fig. 3 plots. Backbone

D.3

Arm

Image-CLS

Image-Ret.

(10 tasks)

(16 tasks)

Mean

Qwen3.5-2B (dense, 2B act.)

specialist generalist ∆

60.69 56.53 −4.16

73.06 71.17 −1.89

66.88 63.85 −3.02

Qwen3.5-9B (dense, 9B act.)

specialist generalist ∆

63.50 61.34 −2.16

77.49 75.81 −1.68

70.50 68.58 −1.92

Qwen3.5-35B-A3B (MoE, 3.1B act.)

specialist generalist ∆

59.73 58.97 −0.76

74.04 72.54 −1.50

66.89 65.75 −1.13

Expert Routing Behavior Across Modality and Task

To analyze expert routing patterns across modalities and task domains, we profile top-8 routing assignments across all 40 MoE layers using 10 MMEB-V2 categories covering all modalities and tasks, with 16 samples per category. The routing divergences are measured using Jensen–Shannon (JS) distance over the 256-expert distribution. 22

Modality Preference Over Task Taxonomy. Layer-averaged JS distance is significantly lower within modalities (0.114) than between modalities (0.323). Conversely, grouping subset pairs by task category yields no statistically significant separation (z = −0.66, p = 0.73), as task categories span modality boundaries. Image and video inputs form a unified visual cluster, exhibiting an inter-modality JS distance of 0.117—lower than intra-image distance (0.136). Notably, this modality specialization emerges naturally without active load-balancing or auxiliary z-loss constraints.

effectively in use vision- or text-leaning

175 150

0.2

0.30 0.25

video

125 100

0.20

0.15

L38: 13

0.1

0.35

text

0.3

0.40

0.10

25 0.0

0.05

0 0

decoder layer

routing JS distance (bits)

0.4

image

200

sampling noise floor between modality within modality

experts (of 256)

routing JS distance (bits)

Depth-Wise Modality Convergence. We score each expert j of a layer by a preference index πj = (vj − tj )/(vj + tj ) ∈ [−1, 1], where vj and tj are its mean shares of the routing mass over the vision and the text subsets, and call it modality-preferring when |πj | > 0.5, the point at which one modality routes three times the mass of the other. Modality divergence peaks in early decoder layers (layer 4 JS = 0.339, where 172 of 256 experts display strong modality preference) and steadily diminishes in deeper layers (layer 38 JS = 0.099, where only 43 experts remain modality-dependent; Fig. 12). Inputs across all modalities thus converge toward a compact shared expert sub-population in deep layers, before briefly re-diverging at layer 39 to construct the final sequence embedding.

image

video

text

0.00

decoder layer

Figure 12 Expert routing differs by modality, and the divergence disappears with depth. (a) Within- and betweenmodality JS distance per layer, with the sampling noise floor shaded. (b) Modality-preferring experts, meaning |πj | > 0.5 on the index defined above, and the number effectively in use, exp(H) of the routing entropy. (c) The 20 subsets at the layer mean, ordered by modality, where the image and video blocks are close to indistinguishable. Grouping the same 190 pairs by task category gives no separation (z = −0.66, p = 0.73). Subsets mix workloads, so the separation shown is a lower bound.

Additional Details for Adaptive Computation

This section gives implementation details for every adaptive-computation method of §4.1 and Tab. 6, grouped by whether the decision is learned during training or taken at inference.

E.1

Training-Based Methods

All training-based variants extend the contrastive framework described in Appendix B. Models are trained on 64 NVIDIA H200 GPUs (expert parallel size 64) in BF16 precision using AdamW (weight decay = 0.01, gradient clip norm 1.0, logit scale 50, maximum sequence length 8,192). LoRA adapters (r = 64, α = 128, dropout 0.05) are applied to visual and language projection layers (linear_qkv, linear_proj, linear_fc1, linear_fc2). Global batch size is set to 512, except for GRPO, which uses a global batch size of 256 with 8 rollouts per input. Initialization and Distillation Regimes. Token pruning, layer skipping, AdaMoE, and GRPO are initialized from the pretrained M O EMB-A3B checkpoint and fine-tuned for one epoch. Token pruning and layer skipping train with a learning rate of 10−5 and an L2 distillation loss (weight = 1.0) against cached full-compute teacher hidden states to stabilize representations during gate optimization. AdaMoE is fine-tuned at a learning rate of 10−4 without distillation. GRPO uses a learning rate of 5 × 10−5 and integrates distillation directly into its reward function. Expert skipping is initialized directly from the base backbone and trained jointly with contrastive adaptation over two epochs at a learning rate of 10−4 without distillation. Token Pruning. Pruning decisions are evaluated after decoder block 2, targeting a 30% visual token retention rate via an L1 budget loss (weight = 0.5, 15% warmup). The Gumbel-Softmax temperature is linearly annealed from 4.0 to 0.5 over 2,000 steps, with predictor weights updated at 40× the backbone learning rate. Pruning is restricted exclusively to visual tokens. 23

Expert Skipping. Expert skipping penalizes active compute via an L1 penalty weight of 0.02 on (1 − ρskip ). This penalty is held at zero for the first 15% of training steps and linearly ramped to full weight over the subsequent 5% of steps. Layer Skipping. A gate module at each block evaluates the hidden state at the <emb> position to produce a binary keep-or-skip decision for the entire block. When a block is skipped, its residual passes through unmodified, bypassing both attention and MoE sub-layers. The gate is trained via a straight-through estimator targeting a 20% skip rate under unit loss weight, applied from decoder block 3 onward. In practice, the policy converges to a 20.5% empirical skip rate concentrated across 9 of the 36 eligible blocks, leaving the remaining 27 blocks unskipped. Adaptive Top-k. Following AdaMoE (Zeng et al., 2024), we augment the router with 256 null experts alongside the 256 physical experts and perform a joint top-8 selection across all 512 candidates. A token’s effective width corresponds to the number of top-8 slots allocated to physical experts, eliminating manual threshold tuning. A load-balancing penalty (weight = 0.02) maintains the mean activation near four physical experts during the first half of training, after which it drops to 10−4 to allow unconstrained routing convergence. The policy settles at an average of 2.94 physical experts per token at evaluation (63.3% reduction in routed expert execution); we compare this setting against MoDES at an equivalent target budget in §5.3. Group Relative Policy Optimization (GRPO). GRPO optimizes a lightweight three-way classification head per MoE layer that emits categorical decisions over {ROUTE, SKIP, DROP} for each token. Native expert routing remains greedy, meaning the policy dictates only execution state without altering expert selection logic. Auxiliary routing losses are disabled. The policy is trained on 8 rollouts per sequence sampled at temperature 1.0 using a learning rate of 5 × 10−5 without KL penalties or value baselines, utilizing group-standardized advantages across rollouts. Log-probabilities sum categorical policy decisions across active tokens and MoE layers, normalized by the total decision count to equalize variable sequence lengths. To manage memory overhead, gradients are replayed one rollout at a time. Updates are computed on-policy (r(i) ≡ 1, rendering ratio clipping inactive). To avoid distorting core feature spaces, policy gradients flow exclusively to the efficiency gate heads; base backbone and LoRA weights remain fixed at their contrastive values (Lan et al., 2025; Jiang et al., 2026). The overall reward combines an efficiency score S(a) and a cosine distillation reward against teacher embeddings (weight = 1.0, softmax temperature 0.2 for negative pairs). The efficiency reward credits each action by the compute it removes at each layer: E[FLOPs(a)] S(a) = 1 − , (6) FLOPsdense h i XX drop E[FLOPs(a)] = αt,ℓ fℓattn + fsh + (1 − pskip − p )f , (7) rt t,ℓ t,ℓ ℓ

where αt,ℓ is the probability that token t reaches layer ℓ, and fℓattn , fsh , frt represent layer-specific attention, shared expert, and routed expert FLOP costs. Expert skipping removes frt at that layer and token dropping removes all downstream per-token compute. Action Combination Strategies. We evaluate four joint execution schemes for combining discrete efficiency actions, where ρa and ρ⋆a denote the realized and target reduction rates for action a: • Shared Target, Simultaneous: We use a single expected FLOP penalty defined as λf (Eθ [FLOPs]/FLOPsfull −b)+ and update all action gates simultaneously. Because individual action rates are unconstrained, optimization favors the lowest-cost action; token dropping dominates early training, leaving expert and layer gates under-optimized. P • Per-Action Targets, Simultaneous: Replacing the single target with decoupled quadratic penalties a λa (ρa −ρ⋆a )2 enforces individual target rates. However, simultaneous optimization causes distribution shifts as gates re-adapt to concurrent updates. • Per-Action Targets, Iterative: Gates are updated one at each training step while holding other gates fixed, mitigating co-adaptation instabilities at the cost of increased training duration. • Per-Action Targets, Sequential: Actions are trained sequentially according to a predefined order and permanently frozen upon reaching their target rate ρ⋆a . Each subsequent gate optimizes over a fixed compute reduction profile, matching the inference execution pipeline. 24

Explored Variants and Negative Results. We additionally evaluated A-ViT and PonderNet-style dynamic halting mechanisms (Yin et al., 2022; Banino et al., 2021) based on learned hidden states, GDN-β token scores, and early exit heads. Training combined a ponder penalty and a geometric distribution prior over steps under a joint Gumbel-Softmax / STE loss formulation (Jang et al., 2017): Lhalt = Lemb + λp E[Nhalt ] + λkl KL (qhalt ∥ Geom(γ)) . Table 12 summarizes the observed failure modes. Aggressive regularization degraded target representation quality, whereas conservative settings caused gate gradients to saturate near zero reduction. Joint pruning techniques such as DiEP (Bai et al., 2025) similarly failed to achieve competitive accuracy-efficiency trade-offs. We hypothesize that credit assignment through non-differentiable or delayed attention paths introduces high gradient variance during early training, causing efficiency penalties to dominate before representation alignment can stabilize. Table 12 Summary of non-convergent or suboptimal baseline implementations. Variant

Observed behavior

Projected A-ViT gate

Training is initially healthy, then drop exceeds roughly 85%; the embedding token loses visual context and both representation and gradients collapse. The compute reward drives nearly all halting to the first layer; the distribution prior does not offset the incentive to drop earliest. With a stable negative sign (low β drops first), the gate frequently becomes inert and returns toward zero drop; more aggressive settings destabilize training. Exit probabilities converge to the final layer, producing negligible savings under stable quality.

Cumulative drop logit GDN-β halting Learned early exit

E.2

Inference-Based Methods

This subsection details the training-free dynamic efficiency heuristics evaluated in §4.2. Token Pruning via GDN-β. Visual tokens are ranked using the Gated-DeltaNet write strength β evaluated at decoder block 4. The lowest-scoring tokens are pruned, and remaining sequences are gathered to reduce compute across all subsequent blocks. Text tokens are fully preserved. Expert Skipping via GDN-β. Using the same β signal, tokens ranking in the lower fraction per MoE block skip routed experts. Static Expert Pruning. Methods including REAP (Lasby et al., 2025), MAN (Liu et al., 2026), MSAN (Liu et al., 2026), and router logit thresholding evaluate expert importance scores over a static calibration set of 16 balanced samples. The lowest-ranked experts are permanently removed across all layers. Because routing maintains top-8 selection over reduced expert pools, this strategy reduces total parameter without reducing dynamic FLOPs per token. Adaptive Top-k Routing. Uniform top-k keeps the k highest-scoring of the native eight experts at every layer, with no calibration and no learned gate. The pretrained router’s ranking is what makes the reduced budget cheap: at k = 4 accuracy falls by 0.5 points, while drawing four of the eight at random costs 6.6 at the identical 11.6% FLOPs reduction (Tab. 6). The ordering the router induces over its top-8 therefore accounts for most of what a halved budget preserves. MoDES Calibration. MoDES derives layer-wise scaling parameters sℓ and a global activation threshold τ using a calibration subset from the ImageNet-1K evaluation split. Adaptive Otsu Thresholding for Expert Skipping. Rather than enforcing uniform skip ratios across layers, we explore dynamically computing per-layer thresholds using Otsu’s method (Otsu, 1979). For each layer, the score distribution is partitioned into two classes by maximizing inter-class variance; tokens in the lower partition skip routed experts. We evaluate two scoring signals: (1) GDN write strength β across 30 GDN layers, and (2) maximum pre-softmax router logits across all 40 MoE layers. A dispersion threshold (coefficient of variation > 0.05–0.40) restricts partitioning to layers with sufficiently bimodal score distributions, subject to a maximum 70% per-layer skip cap. Neither variant requires pre-calibration.

Additional Analysis on GDN Write Strength β

Depth Dynamics and Attention Comparison. We evaluate the layer-wise behavior of GDN write strength β using three representative samples per modality. Across all evaluated tokens, the layer-wise mean of β spans [0.196, 0.596], with coefficients of variation ranging from 0.065 to 0.496. In an eight-example diagnostic pairing each of the 10 GDN layers immediately preceding a full-attention layer with its corresponding attention block, the Spearman correlation between β and incoming attention mass reaches 0.546, 0.619, 0.491, and 0.524 for image, text, video, and VisDoc modalities, respectively. However, attention allocation exhibits considerably higher spatial variance, yielding a mean token-wise coefficient of variation of 2.316, compared to 0.226 for β.

GDN write strength β across network depth (pretrained delta-rule write gate, read at inference; high early, low mid-deep) mean β

0.6 0.4 0.2

0.5

Image

0.4

Video

0.3

Visdoc

write strength β

0.6

Text

0.2

transformer layer index (GDN layers)

Figure 13 The GDN write strength is depth-structured. Per-token Gated-DeltaNet delta-rule write strength β (a gate of the pretrained backbone, read at inference), averaged per layer (top: network-wide marginal; bottom: per modality). Tokens write strongly in the early blocks and weakly through the middle-to-deep blocks. The middle panel of Fig. 9 plots the same quantity beside the two learned actions.

Spatial and Temporal Consistency. Qualitative overlays (Figures 15–17) and layer-wise comparisons across blocks 2, 22, and 38 (Figure 14) demonstrate three key properties of β: 1. Semantic Alignment: β highlights semantically informative regions—such as foreground objects in natural images or plot/text elements in documents—forming contiguous spatial clusters rather than isolated activation noise. 2. Depth Stability: β distributions remain stable across network layers, whereas attention maps shift dynamically. Computing cross-layer cosine similarity yields an average of 0.989 (minimum 0.973) for β, compared to 0.752 (range [0.68, 0.86]) for <emb> attention. Tokens identified as high-value in early blocks preserve high scores deeper in the network, even as attention redirects focus. 3. Temporal Persistence: Across video frames within a given layer, β maps exhibit high cross-frame cosine similarity (0.992 vs. 0.744 for attention). This spatial and temporal stability makes β a reliable candidate for early single-shot pruning, ensuring that token removal decisions remain valid throughout subsequent layers. Pruning Decision Depth. Table 13 evaluates sensitivity to the layer at which the single-shot pruning mask is generated. Executing the decision at layer 2 rather than layer 4 alters accuracy by +0.04 at a 75% keep rate and +0.21 at a 25% keep rate, yielding modest incremental FLOP savings of 0.35 and 1.03 percentage points, respectively. These minor variations indicate that pruning effectiveness is broadly insensitive to the exact decision layer.

Table 13 Inference-only GDN-β decision-depth diagnostic. Layer indices are zero-based. Keep

layer

Accuracy

∆

FLOPs ↓

75% 75% 25% 25%

2 4 2 4

70.07 70.03 66.26 66.05

−0.25 −0.29 −4.06 −4.27

6.47% 6.12% 19.36% 18.33%

GDN β cosine between a signal's own maps

(a) image 1.0

0.998

0.987

<emb> attention

(b) visual document 0.997

0.989

0.980

0.973

0.998

(d) video, across frames 0.999

0.989

0.997

0.981

0.9 0.855

0.838

0.8

0.779

0.843

0.768 0.745

0.712

0.7

0.721

0.705 0.686

0.680

0.666

0.6 L2–L22

L2–L38

L22–L38

L2–L22

L2–L38

L22–L38

L2–L22

L2–L38

L22–L38

L22

L38

Figure 14 Figures (a)–(c): cosine similarity across layers; Figure (d) compares cosine similarity across video frames within one layer.

Visual Comparison of Trained and Inference-Only Token Pruning and Expert skipping Methods. Figures 18 and 19 compare trained adaptive computing methods (token pruning and adaptive top-k / expert skipping) with their inference-only counterparts evaluated on identical inputs. As shown in Fig. 18, the single-shot token dropper and the GDN-β policy select nearly identical token subsets. Conversely, Fig. 19 reveals that the trained expert-skipping gate and adaptive top-k routing diverge notably in their allocation decisions. layer 2

layer 22

input

layer 38 0.60 0.55

GDN-β

0.50 0.45

0.40 0.35 0.30 0.014

<emb> attention

0.012

0.008

attn

0.010

0.006 0.004 0.002

Figure 15 Image: GDN-β vs. the <emb> attention. The clean input is at left; the panels overlay GDN-β (top) and the <emb> token’s attention onto the visual tokens (bottom) at layers 2, 22, and 38. Cell colour encodes magnitude against the colourbars, with the image kept faintly underneath for spatial reference. β is strong and near-uniform early and localises smoothly with depth, whereas the <emb> attention is sparse and scattered and only strengthens in the deeper layers.

layer 2

layer 22

layer 38

0.60 0.55

GDN-β

0.50

input

0.45

0.40 0.35 0.30 0.25

0.006

<emb> attention

0.005

0.003

attn

0.004

0.002 0.001

Figure 16 Visual document: GDN-β vs. the <emb> attention. Same layout as Fig. 15, on a document page. As on natural images, β is the smoother, more distributed signal across depth.

GDN-β layer 2

layer 22

<emb> attention layer 38

layer 2

layer 22

layer 38

0.60

0.007

0.55

0.006

0.50

0.005

GDN-β

0.45

0.004

0.40

0.003

0.35

0.002

0.30

0.001

frame 2

attn

frame 1

frame 0

input

Figure 17 Video: GDN-β vs. the <emb> attention. Rows are frames: the clean frame at left, then GDN-β and the <emb> attention side by side, each at layers 2, 22, and 38. The <emb> attention is nearly empty at layer 2 and sharpens into scattered per-frame peaks with depth, whereas β is strong early and spatially smooth, so the two signals carry complementary information.

dropped

One-shot drop (trained, 70%)

GDN-β (inference-only, 75%)

70% of visual tokens dropped

75% of visual tokens dropped

70% of visual tokens dropped

75% of visual tokens dropped

76% of visual tokens dropped

67% of visual tokens dropped

71% of visual tokens dropped

74% of visual tokens dropped

67% of visual tokens dropped

79% of visual tokens dropped

Video (frame 2)

Video (frame 1)

Video (frame 0)

Visual document

Natural image

input

Figure 18 Trained vs. inference-only token dropping, at an equal budget.

MoDES (adaptive top-k)

Trained expert-skip gate

mean skip 62%

mean skip 47%

0.65 0.60 0.55

mean skip 61%

mean skip 41%

0.50 0.45

per-token routed-expert skip rate

Visual document

Natural image

input

Video (frame 0)

0.40 0.35

mean skip 55%

mean skip 62%

mean skip 49%

Video (frame 1)

mean skip 62%

Figure 19 How the two expert-skipping policies allocate their budget. Per-token fraction of the 40 MoE layers whose routed-expert branch is skipped, for MoDES (adaptive top-k, inference-only) and the trained expert skipping, at the same overall skip rate.

Per-Task MMEB-V2 Results

Table 14 Per-task MMEB-V2 scores. Single-pass Task Image classification (10 tasks) Country211 HatefulMemes ImageNet-1K ImageNet-A ImageNet-R N24News ObjectNet Place365 SUN397 VOC2007 Image VQA (10 tasks) A-OKVQA ChartQA DocVQA GQA InfographicsVQA OK-VQA ScienceQA TextVQA Visual7W VizWiz Text-to-image retrieval (6 tasks) EDIS MSCOCO_t2i VisDial VisualNews_t2i WebQA Wiki-SS-NQ Image-to-text retrieval (2 tasks) MSCOCO_i2t VisualNews_i2t Image-to-image retrieval (8 tasks) CIRR FashionIQ MSCOCO NIGHTS OVEN RefCOCO RefCOCO-Matching Visual7W-Pointing Video classification (5 tasks) Breakfast HMDB51 Kinetics-700 SmthSmthV2 UCF101 Video QA (5 tasks) ActivityNetQA EgoSchema MVBench NExTQA Video-MME Video retrieval (5 tasks) DiDeMo MSR-VTT MSVD VATEX YouCook2 Moment retrieval (3 tasks) Charades-STA MomentSeeker QVHighlight Visual-document retrieval (24 tasks) MMLongBench-doc

TTE-based

M O EMB

VLM2Vec UniME UME-R1 Embed UME-R1 Embed V2-2B V2-7B BToks PLUME 2B RL-2B 7B RL-4B A3B A10B 62.9 25.8 55.8 80.8 47.6 89.3 73.0 65.1 36.1 70.9 84.9 56.4 44.0 48.1 90.1 65.4 59.1 51.7 38.1 71.6 52.8 43.3 79.3 84.2 75.9 82.7 74.7 90.6 67.6 74.7 71.1 78.3 64.8 57.3 19.6 66.2 68.4 64.8 87.0 86.3 69.0 39.2 14.8 40.2 38.2 43.0 60.0 34.7 53.0 35.0 33.6 20.9 30.8 28.4 30.0 27.8 47.3 26.2 10.6 37.5 20.1 42.9 49.7 68.7 –

65.6 19.0 66.1 78.9 49.0 89.1 66.0 73.2 43.8 78.8 92.5 68.7 71.5 59.2 92.4 69.0 67.1 71.7 55.2 84.4 62.7 53.4 82.0 88.5 80.0 84.8 77.3 90.2 70.9 77.3 74.6 80.1 74.3 67.0 27.0 81.3 68.3 68.5 95.3 92.8 94.0 37.2 18.0 42.8 38.0 25.2 61.8 50.6 64.6 51.6 42.2 58.8 35.8 28.9 31.5 27.6 57.5 22.5 5.6 39.6 30.0 32.0 56.8 60.1 40.8

64.3 25.2 62.5 80.5 46.9 85.6 74.0 68.5 38.7 74.9 85.7 59.8 48.6 51.2 91.9 63.9 61.6 61.8 40.1 79.5 49.2 49.8 78.6 85.4 73.9 78.4 72.7 90.5 70.5 72.4 69.1 75.8 64.8 54.0 19.1 66.4 68.1 67.8 87.2 86.4 69.4 43.7 18.0 47.1 43.1 41.0 69.3 47.0 64.8 37.0 45.4 47.9 39.9 33.0 33.0 33.8 56.0 27.6 14.5 33.6 18.0 40.4 42.2 62.7 41.4

66.5 25.0 75.5 74.1 50.8 87.5 81.1 61.5 42.4 76.9 86.1 59.2 49.9 49.8 89.9 69.1 59.6 60.5 42.9 78.9 47.6 46.5 76.2 81.8 74.1 72.6 71.3 89.1 68.6 71.2 69.8 72.7 66.0 54.6 20.3 66.9 68.0 68.4 86.5 88.4 74.9 45.0 20.1 51.2 42.2 44.8 66.5 52.3 69.2 47.8 47.4 57.3 40.0 33.5 32.7 36.2 56.1 28.2 14.5 46.7 19.4 63.5 57.1 67.5 32.0

64.8 23.4 75.2 75.3 50.4 88.7 81.1 52.0 42.6 79.4 80.0 62.8 51.1 64.9 92.2 67.3 67.7 62.4 42.7 78.6 54.1 46.8 77.2 88.0 75.1 76.6 71.7 90.0 62.0 71.5 68.9 74.2 64.2 53.7 17.1 69.5 67.2 66.9 83.3 84.4 71.5 44.3 20.1 54.4 35.8 44.1 67.2 50.9 57.8 45.4 49.9 60.0 41.7 32.9 32.4 34.3 55.4 29.9 12.7 39.7 20.4 41.2 57.5 67.3 39.7

62.8 20.0 65.0 78.0 59.2 88.5 44.9 74.8 43.9 75.4 78.7 67.9 54.7 80.7 92.4 64.9 76.7 61.4 57.3 83.8 52.7 54.5 79.2 84.5 79.4 81.5 71.9 89.3 68.9 74.4 75.3 73.6 70.1 47.6 24.0 92.9 66.3 61.4 94.9 85.8 88.0 57.0 36.7 56.7 55.8 56.7 79.3 55.9 74.8 53.0 50.8 53.9 47.1 45.1 45.3 45.7 67.2 43.6 23.5 49.4 26.4 50.9 70.7 74.1 50.3

67.1 25.0 79.0 80.4 53.9 90.1 82.3 42.3 46.8 80.3 90.8 69.2 58.7 75.1 93.8 69.3 79.2 71.7 53.7 83.5 55.2 51.6 81.9 92.0 78.3 80.7 76.8 90.9 72.5 76.7 71.4 82.0 69.7 55.3 23.4 72.7 68.1 71.4 91.4 91.1 84.2 48.6 21.5 58.3 42.8 50.4 70.0 60.7 76.0 52.4 58.2 69.6 47.3 38.2 40.0 38.9 60.7 32.6 18.5 39.3 21.9 41.1 54.8 70.6 41.3

63.7 60.6 19.4 21.4 66.2 59.8 79.5 78.1 58.1 52.9 88.2 87.9 48.3 39.5 75.4 69.1 43.1 44.1 79.2 72.9 79.5 80.8 70.5 73.8 59.3 63.1 80.9 79.0 94.3 96.0 68.5 76.3 77.5 80.0 67.3 73.9 61.6 63.6 84.3 88.5 55.3 59.7 56.2 58.0 80.8 83.2 87.4 91.9 78.9 78.7 84.9 85.1 73.7 76.4 90.5 89.1 69.6 77.8 75.1 80.3 76.3 76.9 73.9 83.8 73.2 72.4 61.2 52.5 31.9 21.2 93.6 92.3 66.4 67.9 60.7 68.0 95.9 96.1 88.0 91.3 87.9 89.8 57.6 53.8 33.0 30.0 60.1 47.5 56.8 51.6 59.5 59.4 78.5 80.7 58.4 65.9 74.4 77.4 52.8 60.4 55.9 63.0 58.2 73.7 50.5 55.0 45.1 45.3 46.8 48.9 46.2 45.9 65.8 66.9 43.4 40.5 23.3 24.6 49.5 45.8 25.0 21.3 49.9 49.3 73.6 66.8 74.7 81.0 50.7 86.1

64.4 26.3 69.8 79.5 55.2 88.4 50.5 73.7 45.3 74.3 81.3 75.4 64.6 82.5 96.0 77.6 82.6 75.1 67.4 89.6 58.8 60.1 85.4 92.9 79.7 88.4 81.9 90.9 78.8 80.9 77.7 84.1 74.1 54.9 26.7 91.2 69.2 71.0 96.6 92.7 90.4 58.1 36.5 54.4 55.6 61.8 82.1 67.5 80.8 60.6 64.8 74.9 56.6 47.3 50.9 47.4 67.5 42.5 28.1 49.4 30.7 48.6 69.1 82.2 86.4

Continued on next page

Table 14 – continued from previous page Single-pass Task MMLongBench-page-fixed ViDoRe_arxivqa ViDoRe_biomedical_lectures_v2_multilingual ViDoRe_docvqa ViDoRe_economics_reports_v2_multilingual ViDoRe_esg_reports_human_labeled_v2 ViDoRe_esg_reports_v2_multilingual ViDoRe_infovqa ViDoRe_shiftproject ViDoRe_syntheticDocQA_artificial_intelligence ViDoRe_syntheticDocQA_energy ViDoRe_syntheticDocQA_government_reports ViDoRe_syntheticDocQA_healthcare_industry ViDoRe_tabfquad ViDoRe_tatdqa ViDoSeek-doc ViDoSeek-page-fixed VisRAG_ArxivQA VisRAG_ChartQA VisRAG_InfoVQA VisRAG_MP-DocVQA VisRAG_PlotQA VisRAG_SlideVQA

TTE-based

M O EMB

VLM2Vec UniME UME-R1 Embed UME-R1 Embed V2-2B V2-7B BToks PLUME 2B RL-2B 7B RL-4B A3B A10B 44.7 78.9 44.6 37.1 42.3 45.8 45.7 82.7 61.0 89.1 86.3 85.6 91.1 87.8 44.3 – 80.3 76.7 84.2 85.9 71.8 65.9 91.4

– 51.9 33.8 38.2 36.0 54.7 43.6 73.2 45.4 76.8 77.3 79.9 81.7 57.7 35.5 75.8 – 53.1 83.7 82.6 66.4 51.0 86.4

– 76.5 39.1 37.2 39.7 40.0 35.7 80.9 62.3 79.9 85.3 81.0 85.3 80.4 42.0 78.3 – 76.9 86.2 88.7 78.6 65.6 91.8

39.9 72.6 48.2 36.2 49.6 52.1 49.0 79.0 64.8 83.8 82.6 83.2 91.1 88.8 36.6 76.9 80.6 71.6 80.8 85.7 74.9 66.2 88.9

– 73.9 46.1 37.9 45.7 50.2 42.6 76.2 66.8 85.9 83.3 82.6 90.8 86.1 40.6 75.9 – 74.3 86.0 84.4 75.6 68.0 87.1

47.7 86.1 51.0 45.7 53.0 56.9 46.9 86.8 70.7 94.0 86.7 89.0 91.1 94.5 54.6 82.6 82.0 84.9 88.3 90.0 79.1 73.0 92.3

– 73.6 50.7 41.1 57.8 50.4 43.2 80.8 65.0 89.5 85.7 89.8 94.3 90.2 46.7 75.3 – 80.5 84.9 89.2 83.4 72.7 91.5

51.0 71.4 88.7 90.9 50.1 59.7 47.5 54.0 53.9 61.9 59.8 58.0 49.7 51.5 86.9 91.5 69.0 78.7 91.6 96.3 88.1 91.5 90.7 94.4 90.4 95.5 94.7 94.4 54.8 62.8 82.4 89.0 84.4 84.5 86.9 88.9 88.5 87.4 89.6 94.2 79.3 89.2 72.4 75.3 92.6 95.8

72.3 91.2 61.3 54.1 65.4 68.7 55.4 91.5 79.0 96.5 93.2 94.8 95.6 95.5 63.9 89.1 83.8 89.7 90.9 94.4 89.1 74.2 96.3

Record · ID 668062 · SHA-256 0593030bc217fa55
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.