arXiv CSopen access

Online Dynamic Batching with Formal Guarantees for LLM Training

Online Dynamic Batching with Formal Guarantees for LLM Training

arXiv:2606.19989v1 [cs.DC] 18 Jun 2026

Dian Li*,†

Zekun Wang* Yaoru Wang Jiahong Yan Tencent {goodli,zekunwang,yaoruwang,redyan}@tencent.com * Equal contribution. † Corresponding author.

Abstract Modern LLM training breaks a core assumption behind offline batch samplers: the true training cost of a sample is only observable after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion. Unless one pays for a preprocessing- and augmentation-dependent length cache, batch construction is therefore blind to the quantity that determines padding, memory use, and GPU saturation. We introduce Online Dynamic Batching (ODB), a DataLoader-side drop-in system that moves batch formation to this point of accurate observability while preserving DDP step alignment. We formalize this synchronization requirement as the Distributed Group Alignment Problem and prove deadlock-free bounded termination with default join-mode identity coverage and opt-in non-join sample-quota closure. ODB requires no model, optimizer, or attention-kernel changes and is released as online-dynamic-batching with lightweight trainer adapters. Across public 2B/8B Qwen3-VL runs on UltraChat/LLaVA/ShareGPT4o, ODB improves literal emitted-sample throughput vs. fixed-batch Standard by 1.58–2.51× on single-node Full FT/LoRA and 1.71– 3.78× on two-node Full FT, with Standard-comparable quality; production MMMix reaches 4.43×. Against GMT/BMT offline token-budget oracles, ODB is within 15% on UltraChat/LLaVA and faster on high-CV ShareGPT4o: 2.24– 2.39× single-node Full FT/LoRA and 3.06–3.69× two-node Full FT. Together, ODB occupies the online/drop-in regime for high-heterogeneity LLM fine-tuning: large throughput gains at Standard-comparable quality, formal DGAP guarantees, and no length-cache precompute or kernel rewrites.

1

Introduction

Training-time batching has an observability problem. In modern LLM and multimodal finetuning stacks, the cost of a sample is not a static dataset attribute: it is realized only after preprocessing, augmentation, chat templating, tokenization, and visual-token expansion. A sampler that runs before this point either ignores the quantity that determines padding and memory use, or pays for a length cache that is tied to a specific transform/template/cutoff policy. The natural systems move is therefore to form batches exactly where the true length becomes observable. The difficulty is that runtime variable-size batching breaks a contract that fixed-batch DDP gets for free: all ranks must execute the same number of gradient-reduction steps. We measure length heterogeneity by CV = σ/µ, the coefficient of variation of post-pipeline tokenized sample lengths. Three conditions for efficient training. We frame any batching strategy as needing to satisfy three conditions simultaneously: (1) spatial efficiency—padding ≈ 0; (2) compute saturation— each step’s effective workload (tokens × FLOPs) saturates the GPU; (3) temporal efficiency—data prep overlaps GPU compute. Fixed-batch baselines violate at least one condition on variable-length Preprint.

data: small bs avoids padding but underfills the GPU, while large bs raises work per step by mixing unequal lengths and therefore pays padding or OOM cost. Sequence packing can remove padding, but for multimodal training it is a model/kernel-level intervention rather than a pure DataLoaderlevel batching method (§5). Existing HuggingFace length grouping [25], token-budget [21], and prebucketed [12, 20] approaches either require offline preprocessing/augmentation assumptions or use fixed batch sizes; in the high-CV setting, the fixed-batch window that survives the longest examples is often the throughput bottleneck. The Distributed Group Alignment Problem. Moving batch formation to the point of accurate observability creates a distributed synchronization problem. When each rank independently forms variable-size groups from its local realized lengths, the number of groups naturally differs across ranks, but DDP requires all ranks to call AllReduce the same number of times. We call the task of aligning these runtime group counts without sample loss or deadlock the Distributed Group Alignment Problem (DGAP). Prior length-aware batchers and static micro-batch/DDP systems [25, 21, 12, 20, 24, 22, 16] avoid this regime by fixing batch composition or sharding statically; ODB’s novelty is the runtime DDP-aware DataLoader regime, not another offline ordering heuristic. Contributions. We make four contributions. (i) We introduce Online Dynamic Batching (ODB), a DataLoader-side drop-in system that observes preprocessing-/augmentation-dependent post-pipeline lengths and requires no model, optimizer, attention-kernel, or length-cache precompute. (ii) We formalize DGAP and give a Max-Based Bidirectional Group Alignment protocol that provides DDP step alignment, strict identity coverage in default join mode, sample-quota closure in opt-in non-join mode, and deadlock-free bounded termination (Theorems 1–3). (iii) We evaluate against Standard, Sorted, Packing, and offline GMT/BMT/HFG oracle baselines across 2B/8B Full FT and LoRA plus a production MM-Mix case study, showing 1.58–2.51× single-node and 1.71– 3.78× two-node public throughput, plus 4.43× in the production case study, while keeping validation/benchmark metrics in the Standard-comparable band. (iv) We identify regimes where online observability matters—augmentation-policy churn, multimodal preprocessing, and high-CV long tails—and release the open-source online-dynamic-batching package with lightweight trainer adapters.

2

System Design

2.1

Architecture Overview

ODB wraps the PyTorch DataLoader iterator boundary, leaving the Dataset and Model untouched; a lightweight trainer-side integration consumes emitted-sample metadata for accounting, token-level loss scaling, and optional sample-quota stopping (§2.4; Figure 1). Workers run with a null collate (passing single samples), and a dedicated collate process drains the worker queue into a configured grouping buffer (1024 samples in our experiments). On each round it sorts by length, greedily forms variable-size batches (Section 2.2), and synchronizes group counts via a single Gloo all_gather, then runs split/overflow to align ranks to a common target Tgrp (Section 2.3). Aligned groups pass through the configured collate_fn and are placed in the output queue; under-filled slots are padded with IDLE_DATA sentinels that the main process transparently skips, leaving real batches step-aligned across active ranks. The collate process owns its own Gloo group, fully isolated from the main process’s NCCL group. 2.2

Dynamic Batch Sizing

ODB keeps per-batch token count roughly constant via a user-specified budget Lmax . For a realized post-pipeline sample length l, B(l) denotes the target local group size: B(l) = max (⌊Lmax /l⌋ , 1)

so that

B(l) l ≈ Lmax .

(1)

Within each rank, buffered samples are sorted ascending and iterated from longest to shortest with a running group-size threshold t (initially 1): each sample is appended to the current group, and when its size reaches t the group is finalized and t ← B(l) for the last-added (shortest) sample. Successive groups naturally hold more samples since shorter l yields larger B(l), so per-group token counts converge to Lmax (worked example in Appendix D). 2

Rank 0

Rank 1

Rank 0

Rank 1

Worker ×nw augment+template → tokenize

Worker ×nw augment+template → tokenize

Worker ×nw augment+template → tokenize

Worker ×nw augment+template → tokenize

collate_fn(B) pad to lmax

collate_fn(B) pad to lmax

Collate Process sort → group → align

single samples

fixed bs=B

Collate Process sort → group → align

bs=3

fixed bs=B NCCL

Trainer

single samples Gloo

bs=2 NCCL

Trainer

Trainer

(a) Standard DataLoader

Trainer

(b) ODB DataLoader

Figure 1: Architecture. (a) Standard collates inside workers with a fixed bs on all ranks. (b) ODB collates in a dedicated process, groups by length, and aligns group counts via Gloo; each rank may have a different local bs while real batches remain step-aligned across active ranks. 2.3

Cross-Rank Alignment, Termination, and Loss Scaling

Alignment target and bidirectional adjustment. Let W be world size and N the dataset identity count/full-epoch quota. Different ranks generally produce different group counts Gr . ODB com+ + putes a global group-count target Tgrp = max(min(maxr: Gr >0 Gr , Cmin , Smin ), 1) over active + + ranks, where Cmin is the minimum positive output-slot capacity on any active rank and Smin is the minimum positive buffered-sample count on any active rank. It then splits groups by extracting singleton samples when Gr < Tgrp , or overflows groups and recirculates extras when Gr > Tgrp , until every active rank reports Tgrp groups. The full algorithm appears in Appendix A; odb_join_mode only changes termination: default join gives strict per-iteration identity coverage by draining outstanding sampler views before global completion, while opt-in non-join gives no-leak quota closure. Theorem 1 (Strict Zero-Discard, default join mode). Under odb_join_mode = true with DistributedSampler(drop_last=False), each rank emits its remaining and outstanding sampler views before advertising local finish, and the collate subprocess keeps participating in the shared Gloo protocol until all ranks advertise local finish. Therefore the emitted sampler-view multiset equals the sampler multiset M = W ⌈N/W ⌉, its identity projection covers all N dataset identities, and per-iteration ηlogical = 0 by construction. Theorem 2 (No-Leak & Sample-Quota Closure, opt-in non-join). Let Smax be the largest realized global emit count of one aligned trainer step, and let Semit be the global cumulative trainerside emitted-sample count at termination. Under non-join termination, every sampler view remains in exactly one protocol state (emitted, collate buffer, worker queue, or sampler pending), and the trainer-side control logic chains logical iterations until N ≤ Semit ≤ N + Smax . Hence ηquota := max(0, 1 − Semit /N ) = 0. This is a cumulative sample-count guarantee; non-join identity audits are reported in App. C.6. Corollary 1 (Empirical sample-quota closure). ηquota,emp = 0 across all 18 evaluation runs and 6 synthetic distributions (terminal epoch rounded to four decimals ∈ {1.0000, 1.0001}, with only final-step overshoot; App. C.5). Unified loop and termination. Let pf and nw denote the DataLoader prefetch factor and worker count, and let D = max(pf·nw, buffer_size) be the per-rank outstanding-depth envelope. A single all_gather per round exchanges [idx_budgetr , n_groupsr , sizesr ] and, when token-level loss scaling is enabled, tokensr , with n_groupsr ∈ {n>0, 0, −1} encoding “produced n groups”, “insufficient data”, or “finished”. In default join mode, ranks drain outstanding sampler views before the shared completion signal; in opt-in non-join termination, a logical DistributedSampler iteration ends when any rank emits −1 and the trainer-side control logic launches subsequent logical iterations until Semit ≥ N (per early stop, at most W ·D already-fetched sampler views are not delivered to the trainer; Lemma 4). Exact loss scaling is a separate accounting option: a deterministic all-rank predicate can trigger a second all_gather to re-broadcast postalignment token counts. Theorem 3 (Bounded Termination and Deadlock-Freedom). Each logical iteration terminates in O(N/W ) + O(D) rounds and no rank blocks on all_gather, given a finite sampler and uniformcall invariant (Appendix C.3). Appendix C gives the per-rank transition rules and proofs for sample-quota closure and bounded termination; Appendix E visualizes the non-join state machine; Appendix Q proves the default joinmode identity contract and reports its throughput cost. The coordination channel uses a dedicated 3

Gloo group inside the collate subprocess (isolated from NCCL), ∼128 KB per round at W =8— orders of magnitude below gradient AllReduce—and overlaps GPU compute (< 2% overhead). Appendix F additionally validates empty-rank deadlock-freedom and reports two-node 2B/8B Full FT results. P 1 Loss scaling. ODB’s per-rank batches differ in token counts tr , so naive DDP averaging W r L̄r P P 1 ℓ (T = t ). Preis a biased estimate of the per-token reference loss L⋆ = Ttok r,i,k r,i,k P tok r r scaling each rank’s loss by W · wr makes DDP’s post-averaging output equal r wr L̄r ; the unique weight that recovers L⋆ bit-precisely is the token-level weight X tr wr = , so wr L̄r = L⋆ . (2) Ttok r Sample-level weighting instead weights ranks by sample count; it matches the per-token reference L⋆ only when the average tokens per sample tr /nr is identical across ranks, a condition ODB’s group-of-groups alignment generally does not impose. All main experiments therefore use tokenlevel scaling; App. B gives the full derivation and approximate-vs-exact comparison. 2.4

API and Packaging

ODB is implemented as an in-place PyTorch DataLoader wrapper with no changes to the Model or Dataset. The reference LLaMA-Factory integration consumes ODB step metadata for emitted-sample accounting and token-level loss scaling; the same metadata interface can be ported to other Trainer frameworks. ODB is released as a standalone open-source Python package (online-dynamic-batching).

3

Evaluation

3.1

Experimental Setup

Hardware. Unless labeled otherwise, the main experiments use one 8×H20 node (96 GB/GPU), DeepSpeed ZeRO-2, and bf16. Labeled multi-node experiments use two 8×H20 nodes (16 GPUs total) with the same ZeRO-2/bf16 training stack; these results and the empty-rank audit appear in Appendix F. Datasets and models. We evaluate on three public datasets spanning text and multimodal modalities, reporting the CV defined in Section 1: UltraChat-200K [6] (text-only SFT, 208K, CV=0.48), LLaVA-150K [19] (multimodal, 158K, CV=0.29), and ShareGPT4o [14] (multimodal, 57K, CV=1.00; the GPT-4o-curated subset distributed with LLaVA-OneVision), plus six synthetic distributions for correctness audits. We use Qwen3-VL-2B/8B-Instruct checkpoints [2] under both Full FT and LoRA (rank=8, target=all); earlier Qwen-VL reports document the lineage of the released stack [3, 1]. Per-dataset cutoff_len is above the observed maximum (UltraChat 8192/Max 4471; LLaVA 2048/Max 1260; ShareGPT4o 16384/Max 12110), ensuring zero truncation and shared across methods (Table 10). Standard, Sorted, and ODB are mathematically insensitive to cutoff_len above the longest realized sample, whereas Packing’s per-step memory cost and oracle max-token feasibility depend on it; keeping a uniform cutoff_len is therefore essential for a fair Packing/oracle comparison. Baselines and training hyperparameters. Baselines are Standard (fixed batch size, random sampling), Sorted (online length-grouped fixed batch), Packing (HuggingFace sequence packing; text-only in our stack, §5) [11], GMT-/BMT-oracle fairseq-style global/bucketed max-token batchers [21] with a one-time scalar cache of post-pipeline len(input_ids), HFG-oracle randomized fixed-batch group_by_length, and ODB (ours). Oracle caches are used only for batch construction: training still executes the same online preprocessing, augmentation, templating, tokenization, and visual-token expansion path. The cache is per-(dataset, transform policy, template, cutoff_len), so it must be rebuilt when those policies change; precompute cost is reported in Appendix I. All methods use AdamW+cosine, lr 10−5 , warmup_ratio=0.03, grad-clip 4.0, one epoch, bf16+ZeRO-2, and no gradient accumulation. 4

Table 1: Full Fine-Tuning Results. 8B/2B Full FT on 8×H20, selected config per method, 3seed mean±std. sam/s is emitted samples divided by wall-clock. Packing is text-only here; GMT/BMT/HFG use scalar length caches for batch construction and exclude cache construction (App. I). Bold marks the highest emitted-sample throughput among online/no-cache rows; offline oracles are unbolded comparators. Score is MMLU for UltraChat and MMMU-MC choice likelihood for multimodal tasks. 8B (Full FT) sam/s

Speedup

Score

2B (Full FT)

Dataset

Method

Val Loss

sam/s

Speedup

UltraChat CV=0.48 MMLU

Standard 5.77 ± 0.01 Sorted 8.09 ± 0.02c Packing 10.46 ± 0.00 GMT-oraclef 10.94 ± 0.02 BMT-oraclef 10.31 ± 0.02 HFG-oraclef 7.33 ± 0.01 ODB 10.23 ± 0.03b

1.00× 72.85 ± 0.40% 0.8487 ± 0.0013 20.98 ± 0.04 1.00× 58.17 ± 0.06% 0.9980 ± 0.0017 1.40× 74.72 ± 0.33% 0.8839 ± 0.0020 28.31 ± 0.08 1.35× 59.02 ± 0.10% 1.0287 ± 0.0012 1.81× 75.18 ± 0.06% 1.1819 ± 0.0088d 36.61 ± 0.07 1.75× 59.68 ± 0.07% 1.1947 ± 0.0008d 1.90× 75.14 ± 0.09% 0.8350 ± 0.0013 39.44 ± 0.03 1.88× 59.16 ± 0.09% 0.9987 ± 0.0020 1.79× 75.26 ± 0.08% 0.8351 ± 0.0013 35.84 ± 0.04 1.71× 59.15 ± 0.16% 0.9772 ± 0.0013 1.27× 73.70 ± 0.14% 0.8404 ± 0.0013 27.28 ± 0.07 1.30× 58.30 ± 0.13% 0.9974 ± 0.0016 1.77× 74.75 ± 0.11% 0.8558 ± 0.0014 36.91 ± 0.19 1.76× 58.98 ± 0.18% 1.0030 ± 0.0014

Score

Val Loss

Standard 14.38 ± 0.03 Sorted 20.46 ± 0.03c LLaVA f GMT-oracle 26.65 ± 0.05 CV=0.29 f MMMU-MC BMT-oracle 25.70 ± 0.05 HFG-oraclef 21.84 ± 0.04 ODB 24.87 ± 0.09b

1.00× 55.88 ± 0.72% 1.0552 ± 0.0013 47.92 ± 0.05 1.00× 43.06 ± 0.77% 1.1814 ± 0.0225 1.42× 55.84 ± 0.38% 1.1781 ± 0.0028 66.37 ± 0.07 1.39× 39.53 ± 0.54% 1.3318 ± 0.0017 1.85× 53.53 ± 0.66% 1.0630 ± 0.0011 79.44 ± 0.39 1.66× 43.06 ± 0.40% 1.2102 ± 0.0018 1.79× 54.24 ± 0.65% 1.0630 ± 0.0012 75.64 ± 0.13 1.58× 43.49 ± 0.18% 1.2102 ± 0.0013 1.52× 54.71 ± 0.65% 1.0590 ± 0.0012 69.52 ± 0.08 1.45× 42.59 ± 0.96% 1.2007 ± 0.0205 1.73× 54.08 ± 0.53% 1.0944 ± 0.0013 82.42 ± 0.53 1.72× 43.18 ± 0.31% 1.2189 ± 0.0021

Standard 2.37 ± 0.00a 2.44 ± 0.00a ShareGPT4o Sorted f GMT-oracle 2.57 ± 0.01 CV=1.00 f 2.50 ± 0.01 MMMU-MC BMT-oracle f HFG-oracle 2.82 ± 0.01 ODB 5.83 ± 0.04b

1.00× 52.43 ± 0.44% 1.1913 ± 0.0056 6.51 ± 0.01 1.00× 41.29 ± 0.71% 1.2910 ± 0.0058 1.03× 52.82 ± 0.31% 1.2175 ± 0.0059 6.71 ± 0.01 1.03× 39.33 ± 0.37% 1.3191 ± 0.0062 1.09× 53.57 ± 0.71% 1.1904 ± 0.0057 7.03 ± 0.02 1.08× 41.18 ± 0.12% 1.2930 ± 0.0059 1.06× 52.51 ± 0.88% 1.1908 ± 0.0055 7.03 ± 0.00 1.08× 40.79 ± 0.24% 1.2931 ± 0.0059 1.19× 52.31 ± 0.76% 1.1913 ± 0.0058 7.71 ± 0.04 1.18× 40.83 ± 1.31% 1.2909 ± 0.0061 2.46× 53.88 ± 0.20% 1.2269 ± 0.0059 16.09 ± 0.21 2.47× 40.12 ± 0.36% 1.3341 ± 0.0062

a For ShareGPT4o 8B, Standard and Sorted use bs=1; wider fixed batches are infeasible or slower on the long-tail length distribution. b ODB’s (Lmax , pf, buffer) is selected per dataset; tuples appear in App. I. c Sorted bs is the largest value that completes a full epoch; larger profiled settings OOM on longest-tail batches. d Packing val_loss uses a packed-sequence denominator and is not comparable to per-sample val_loss; MMLU is comparable. e MMMU-MC reports the 850/900 rows with A–H letter ground truth; generated-answer analyses are not mixed into Score. f Oracle baselines use scalar length caches for batch construction; reported throughput excludes cache construction.

Method-specific parameters and configuration search. For Standard/Sorted, the throughput knob is batch_size (bs); we sweep {1, 2, 4, 8, 16} where memory permits, and additionally profile larger values to establish OOM/full-epoch survival when noted. For ODB, per-worker bs is always 1; the throughput knobs are Lmax (per-step token budget, B(l) = max(⌊Lmax /l⌋, 1)) and prefetch_factor (pf, with D = max(pf×nw, buffer_size)); we sweep Lmax over a feasibility-tested token-budget grid up to 32768, including powers-of-two anchors and intermediate tuned values, then pf at the selected budget. For GMT-/BMT-oracle, we sweep max_tokens_budget over the corresponding feasible token-budget grid; HFG-oracle sweeps fixed bs under the same survival protocol used for Sorted. Each search runs for 20 minutes to obtain steady-state throughput, and a candidate is eligible only if it produces stable numeric throughput without OOM in the profiling window. HP selection uses training-only profiling signals: completed non-OOM candidates are restricted to a near-fastest throughput band, then deterministic stability/resource tie-breakers are applied; validation loss and benchmark scores are never used. The provisional selection must also complete the final full-epoch training run; if it OOMs or otherwise fails to finish the full epoch, we fall back to the next-ranked eligible configuration under the same profiling rule. Fairness principle. Quality is measured at each method’s selected full-epoch-surviving configuration, not at an arbitrary shared setting; for Sorted, the selected bs must also complete a full epoch (Table 1, footnote c ). Quality benchmarks. UltraChat uses MMLU [7]; multimodal tasks use MMMU-MC [27], a parser-free choice-likelihood score over letter-labeled multiple-choice validation items. MMMUMC excludes non-letter ground-truth rows and avoids generation/parsing by scoring the assistant first-token likelihood of option letters A–H; generated-answer or parser-based analyses are never mixed into the Score column. 3.2

Main Results

Deployment-class reading. The main comparison is whether an online DataLoader-level method can approach stronger offline/model-side throughput while preserving multimodal deployment properties. Packing is text-only in our stack; GMT/BMT/HFG are favorable oracle comparators with 5

exact post-pipeline scalar lengths, no cache cost charged in the throughput column, and the same runtime preprocessing path during training. ODB instead observes post-pipeline lengths online, requires no cache, and achieves the highest emitted-sample throughput among online/no-cache rows on every 8B Full FT dataset. 8B Full FT. ShareGPT4o isolates the long-tail case: ODB reaches 2.46×, while Sorted remains at 1.03× because the longest examples force bs=1. Offline oracles remain sample-count limited by construction rather than by missing length visibility: HFG keeps a fixed batch size, and GMT/BMT constrain padded token area, yielding only 11.4/9.0 samples per update versus ODB’s 52.8 (Table 13). On LLaVA, ODB reaches 1.73× with low padding and higher throughput than Sorted, while avoiding the Sorted validation-loss and answer-format sensitivity discussed in §3.3; on UltraChat, Packing and GMT-oracle are strong non-drop-in/offline comparators, while ODB is the highest-throughput online/no-cache row. 2B Full FT and LoRA. At 2B, ODB again leads the online/no-cache rows: it reaches 2.47× on ShareGPT4o, 1.72× on LLaVA, and 1.76× on UltraChat. The narrow exceptions relative to all comparators clarify the boundary of the claim: GMT-oracle is faster on 2B UltraChat because global offline construction can use full-dataset length information, whereas ODB greedily groups within an online buffer; on 2B LLaVA, ODB is both the fastest and the lowest-padding online/drop-in row. Under LoRA, ODB reaches 1.58–2.51× (App. M). Appendix F adds two-node Full FT validation, where ODB remains the highest-throughput online/no-cache row and reaches 1.71–3.78×. Takeaway. ODB brings online/drop-in batching into the oracle token-budget regime. The remaining raw-throughput wins require assumptions outside this deployment class: text-only packing support, offline length-cache precompute, full-dataset visibility, or cache rebuilds under augmentation/template/cutoff changes. 3.3

Training Quality

Quality is evaluated at the speed-selected configuration. ODB intentionally changes batch shape: its efficiency mechanism is to increase useful tokens and samples per optimizer update while controlling padding. For every method, the reported score is therefore measured at the configuration selected by the same training-only profiling protocol, which restricts completed non-OOM candidates to a near-fastest throughput band and then applies deterministic stability/resource tie-breakers. This makes the quality comparison a direct test of whether the faster update geometry harms the trained model. Discriminative case: 8B Full FT LLaVA. ODB stays in the Standard-comparable operating band, though not tied with Standard on every metric: val_loss 1.0944 ± 0.0013 vs. Standard 1.0552 ± 0.0013 and MMMU-MC 54.08 ± 0.53% vs. 55.88 ± 0.72%, while delivering 1.73× throughput. Sorted reaches 1.42× but regresses by +11.6% on val_loss. Appendix L explains why generatedanswer MMMU is not mixed into the main benchmark: length-sorted training can bias answer format, whereas ODB groups only within each online buffer. Other quality cells. On 8B UltraChat, ODB MMLU 74.75 ± 0.11% remains in the same band as Packing (75.18 ± 0.06%), GMT (75.14 ± 0.09%), and BMT (75.26 ± 0.08%); val_loss remains within 0.008 of Standard and below Sorted. On 2B Full FT, ODB reports UltraChat MMLU +0.81 pp over Standard, LLaVA MMMU-MC +0.12 pp, and ShareGPT4o MMMU-MC −1.17 pp. Under LoRA, ODB remains within the Standard band on LLaVA at both scales and is nominally above Standard on ShareGPT4o at both scales (App. M). Oracle rows are offline, cache-based comparators, not direct online baselines. Their raw-throughput differences reflect update geometry rather than a different data path: on 8B LLaVA, GMT/BMT use about 251–254 samples/update and 591–597 updates/epoch, while ODB uses 177.6 samples/update and 844 updates; on 2B LLaVA, ODB is both faster and uses 119.0 samples/update versus about 60 for GMT/BMT (Tables 13–14). We therefore read all rows as throughput–quality operating points, with ODB adding online/drop-in batching, DDP step alignment, sample-quota closure, and default join-mode identity coverage. 6

5 8B Standard

8B Full FT

2B Standard

ODB Speedup (×)

Throughput vs. bs=1 (×)

3

8B ODB 2B ODB

2

1

0

10

20

8B LoRA (App.)

4

2B LoRA (App.)

Padding Rate (%)

2B MM-Mix

3

8B MM-Mix

2

1

30

2B Full FT

0

0.2

0.4

0.6

0.8

1

Coefficient of Variation (CV)

(a) ShareGPT4o 20-min profiling comparison (CV=1.00), normalized to fixed-bs=1; Table 1 reports the selected 3-seed full-epoch rows. Fixed-bs=2 reaches 25–28% padding and ≈0.60× throughput, while the selected ODB HP points keep padding at 1.27% (8B) and 0.36% (2B) with 2.58×/2.49× profiling throughput.

(b) ODB speedup vs. CV for main-table ODB rows plus MM-Mix markers. CV alone is insufficient: MM-Mix (CV=0.80, fs ≈ 0.37) exceeds ShareGPT4o (CV=1.00, fs ≈ 0.01).

Figure 2: ODB speedup mechanisms. (a) On high-CV ShareGPT4o, fixed batching moves rightward/downward as bs grows, while ODB occupies selected low-padding/high-throughput points. (b) Across workloads, speedup tracks length heterogeneity but is amplified by short-sample mass; LoRA points are appendix context.

3.4

Speedup Analysis

ODB speedup comes from jointly improving the three efficiency conditions in Section 1: spatial efficiency (little padding), compute saturation (enough useful token/FLOPs per step), and temporal efficiency (overlapping input preparation with GPU compute). Fixed-batch training on variablelength data cannot optimize all three simultaneously: bs=1 avoids padding but underfills the GPU, while larger bs increases per-step work only by mixing unequal lengths and therefore pays padding or OOM cost. ODB improves throughput because it attacks the three terms together: online length grouping reduces spatial waste, token-budget updates increase useful work per step, and bounded outstanding depth hides the remaining input latency. Spatial efficiency. Figure 2a shows the spatial effect on ShareGPT4o: fixed bs=2 introduces 25– 28% padding and drops to ≈0.60× of bs=1, while the selected ODB HP markers stay near the low-padding/high-throughput corner. This profiling view is consistent with the selected full-epoch rows in Table 1, where ODB reaches 2.46×/2.47× on 8B/2B. The gain is therefore not merely larger batches, but larger useful batches. Compute saturation. Fixed bs=1 avoids padding on long-tail workloads but gives the GPU too little useful work per update; larger fixed batches add work by adding padding. ODB instead increases real samples/tokens per update under a token budget, so the GPU sees denser useful work without the fixed-batch padding penalty. The effect is nearly scale-invariant within each dataset: UltraChat reaches 1.77×/1.76× on 8B/2B, LLaVA reaches 1.73×/1.72×, and ShareGPT4o reaches 2.46×/2.47×. This pattern suggests that model scale is not the primary driver in these cells; Tables 13–14 relate the throughput differences to batch-shape statistics. Short-sample leverage. CV flags padding pressure, but short-sample mass fs flags recoverable compute density. MM-Mix has lower CV than ShareGPT4o (0.80 vs. 1.00) but much larger fs (≈ 0.37 vs. ≈ 0.01), allowing ODB to aggregate short OCR/VQA and captioning examples into compute-dense updates and reach 4.43× at 2B and 2.92× at 8B. Lmax raises useful work per step until memory/step time saturate, while D hides input latency until pipeline overlap saturates; Section 3.5 locates these operating ranges. 7

Table 2: Ablation: per-batch token budget Lmax at fixed D=1024 (default join-mode ODB bs=1, nw=4, pf=256, buffer=1024, 8×H20, Qwen3-VL-8B Full FT, single-seed 20-minute windows). Speedups use the Table 1 8B Standard baselines. Bold marks the fastest stable row; failed denotes no stable numeric throughput in the profiling window. UltraChat (CV=0.48)

LLaVA (CV=0.29)

ShareGPT4o (CV=1.00)

Lmax

sam/s

spd

sam/s

spd

sam/s

spd

2048 4096 8192 12288 14336 16384 32768

8.48 9.23 9.44 10.21 10.08 9.90 failed

1.47× 1.60× 1.64× 1.77× 1.75× 1.72× —

20.25 22.66 24.17 24.53 24.44 24.88 failed

1.41× 1.58× 1.68× 1.71× 1.70× 1.73× —

5.48 5.74 5.96 6.11 6.17 5.86 failed

2.31× 2.42× 2.52× 2.58× 2.60× 2.47× —

Table 3: Ablation: outstanding depth D (default join-mode ODB, 8×H20, Qwen3-VL Full FT, single-seed 20-minute windows; nw=4, buffer=1024). For each scale/dataset, Lmax is fixed to the selected ODB budget used for the corresponding main-table cell. ovrlap denotes pipeline_overlap in percent; bold uses unrounded sam/s. D=1024 Scale/Dataset

D=2048

D=4096

D=8192

sam/s ovrlap sam/s ovrlap sam/s ovrlap sam/s ovrlap

2B UltraChat 36.12 99.6 36.21 100.0 36.24 100.0 36.18 100.0 2B LLaVA 78.65 94.7 82.64 100.0 81.28 100.0 80.13 100.0 2B ShareGPT4o 15.77 97.0 15.80 100.0 15.46 100.0 14.17 100.0 8B UltraChat 9.93 99.9 9.93 100.0 9.93 100.0 9.92 100.0 8B LLaVA 24.65 98.5 24.85 100.0 24.85 100.0 24.68 100.0 8B ShareGPT4o 6.06 100.0 5.90 100.0 5.86 100.0 5.73 100.0

3.5

Ablation Study

Table 2 varies Lmax at fixed D=1024 to expose batch-shape saturation; Table 3 varies D = max(pf × nw, buffer_size) to expose temporal overlap. Both are single-seed 20-minute profiling windows with default join-mode ODB; full-epoch throughput and quality claims remain in Table 1. Reading Table 2. Throughput rises as Lmax fills each update, then regresses once memory pressure and longer steps dominate. The best stable Lmax in the 8B sweep differs by dataset: 12288 for UltraChat, 16384 for LLaVA, and 14336 for ShareGPT4o. The 32768 setting is outside the stable profiling envelope. Appendix H gives the low-CV quality sensitivity; main-table configurations additionally tune D and validate full-epoch quality. Outstanding depth. After Lmax fixes batch shape, D controls how much prepared work can overlap GPU compute; at nw=4, pf<256 is clamped to D=1024 by buffer filling (App. P). Reading Table 3. At D=1024, overlap is already 94.7–100.0%; D=2048 helps mainly on LLaVA, and larger values are flat or regressive once overlap saturates. Once overlap saturates or throughput stops improving, increasing D adds little benefit and can be left at the smaller setting. Buffer and loss defaults. On ShareGPT4o, most buffer gains appear by buffer=500; 1024– 2000 is the high-throughput range for 2B (16.77–17.10 sam/s) and 1024 for 8B (6.38 sam/s), with padding at or below 0.6% at buffer≥1024 (App. N). The three loss-scaling modes have similar throughput in short profiling windows, within about 0.2% on 2B and 1.0% on 8B relative to samplelevel scaling; tune Lmax and D, not loss scaling. 3.6

DataLoader I/O Sensitivity

Because ODB raises per-step sample count, input demand differs from fixed-bs Standard. The full num_workers sweep (App. O, Table 19) shows the reported cells are not primarily worker-starved: 8

ODB remains above Standard at every worker count, and by nw=4 rows are within about 5% of their best values. We use nw=4 as a portable default and tune pf from 256. 3.7

Case Study: Production Deployment

On production MM-Mix (273K samples from 7 OCR/VQA/captioning corpora, ≈545K sampleviews over 2 epochs; CV≈0.8, bimodal; App. I), two-node Qwen3-VL-2B full-epoch runs reach Standard at 17.85 ± 0.15 sam/s, Sorted at 20.62 ± 2.08 sam/s (1.15×), and ODB Lmax =12288 at 79.15 ± 4.16 sam/s (4.43×)—our largest speedup (Table 12). The high short-sample fraction (fs ≈0.37 vs. ShareGPT4o’s 0.01) activates the compute-density mechanism: ODB aggregates OCR/VQA short examples into dense batches, exceeding what CV alone would suggest; the 8B profiling sweep shows the same qualitative pattern (22.07 sam/s, 2.92×). The corresponding full-epoch rows show Standard-comparable parser-free MMMU-MC behavior: ODB reaches 46.31 ± 0.44, while Standard is 43.33 ± 2.24 and Sorted is 43.65 ± 2.32. Its validation loss is higher than Standard but far below Sorted’s degradation, and the benchmark score remains in the Standard-comparable range. GMT/BMT-oracle obtain higher MMMU-MC scores than ODB, but their validation loss is not lower than Standard’s; we treat this as benchmark-specific variation and retain them as offline cache-based comparators. The measured MM-Mix oracle-cache construction alone takes 299.9 s for 272,589 samples on one H20, a favorable churn-inclusive lower bound that excludes sample-store scans, order/materialization, cache validation, distributed staging, and rebuilds after recipe changes (App. I).

4

Practical Guidance

ROI. Estimate CV and fs = Pr[l < Lmax /4] on a 1k–5k sketch: CV≳ 0.8 or fs > 0.2 flags high-ROI regimes; CV≈ 0.3–0.5 gives smaller, model-dependent gains (App. K). Tuning. In short profiling runs, sweep Lmax upward from 2–4× mean length and choose the smallest stable setting in the near-fastest emitted-sample throughput band; keep the default input depth unless a small pf/D sweep gives a clear throughput gain (Table 3; App. P). After speed-first profiling, run fullepoch quality/stability validation; validation and benchmark scores are not HP-selection inputs. Use default join mode for final training runs; App. Q shows that its strict identity-coverage contract has negligible throughput cost. The relaxed non-join termination is retained only for constrained runtime integrations that cannot support the join-style drain-before-finish protocol. Method choice. Use packing when text-only varlen attention/boundaries are wired; use ODB for multimodal or augmentation-heavy stacks with online lengths and costly cache/rewrite upkeep.

5

Related Work

Packing and length-aware batching. Sequence packing removes padding by concatenation/masking [11]. In transformer stacks, efficient contamination-free packing needs boundary-aware masks/loss handling and often varlen attention/kernel support [5, 4] plus framework boundary plumbing; in our Qwen-VL/DeepSpeed/LLaMA-Factory stack it is a model-side intervention, not a pure DataLoader swap. Token-budget batchers in fairseq/NeMo/OpenNMT/Composer [21, 12, 9, 20] and GMT-/BMT-oracles build offline length caches that rebuild under augmentation/template/cutoff changes. PyTorch samplers [22] and HuggingFace length grouping [25] do not solve runtime variable group-count alignment across DDP ranks. ODB occupies the runtime DDP-aware DataLoader slot: no offline cache or model rewrite. Complementary training-system techniques. Sequence/context parallelism [10, 17, 18] shards one long sample across ranks; ODB batches variable-length samples. Inference continuous batching [26, 13] lacks DDP step-count constraints. DeepSpeed Data Efficiency [15] handles sampling/curriculum routing. ODB operates at the DataLoader boundary and composes with training-stack mechanisms such as DeepSpeed [23], DDP [16], gradient accumulation, LoRA-style adapters [8], and LLaMA-Factory [28]. 9

6

Limitations

ODB’s clamped memory rule B(l)= max(⌊Lmax /l⌋, 1) is a first-order activation-memory proxy, so Lmax must be swept per model, optimizer, precision, and attention stack. Gains are data-dependent: near-uniform/low-CV workloads leave less long-tail slack, and we do not isolate an iso-token or matched-update causal ablation; reported speedups are batch-system operating points combining padding reduction, useful-batch growth, and input overlap. Empirical claims cover multimodal fine-tuning and MM-Mix, mainly single-node H20 runs, with two-node validation in App. F. The metadata alignment exchange should be revalidated at larger world sizes or heterogeneous nodes. ZeRO-3 and FSDP remain outside the evaluation scope; gradient accumulation follows the same aligned micro-step schedule and is supported by isolated validation runs. Default join mode gives strict identity coverage with negligible measured cost; opt-in non-join relaxes termination to cumulative sample-quota closure (App. Q). ODB complements, rather than replaces, model-side packing or varlen-attention systems when those are available.

7

Conclusion

We introduced ODB, an online dynamic batcher that observes realized lengths after runtime preprocessing, tokenization, and multimodal expansion, then forms DDP-safe variable-size batches at the DataLoader boundary without changing the model, optimizer, attention kernel, or dataset. MaxBased Bidirectional Group Alignment provides strict identity coverage under default join mode, sample-quota closure under opt-in non-join mode, and deadlock-free bounded termination. Empirically, ODB delivers 1.58–2.51× throughput across public 2B/8B Full FT and LoRA rows and 4.43× on production MM-Mix with Standard-comparable validation and benchmark metrics. On 8B Full FT it reaches 1.77× on UltraChat, 1.73× on LLaVA, and 2.46× on ShareGPT4o; under LoRA it reaches up to 2.51×. These results narrow the gap between fixed-batch training and stronger offline or model-side batching methods while avoiding scalar length caches and model-side packing rewrites.

References [1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. [2] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. [3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. [4] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. [5] Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [6] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. UltraChat: Scaling alignment data for large language models with multi-round chat. arXiv preprint arXiv:2305.14233, 2023. [7] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. 10

[8] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. [9] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, 2017. [10] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. In Proceedings of Machine Learning and Systems (MLSys), 2023. [11] Mario Michael Krell, Matej Kosec, Sonia P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. In arXiv preprint arXiv:2107.02027, 2021. [12] Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Belber, Sandeep Subramanian, Vitaly Huang, et al. NeMo: A toolkit for building AI applications using neural modules. In arXiv preprint arXiv:1909.09577, 2019. [13] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023. [14] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. [15] Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, and Yuxiong He. DeepSpeed data efficiency: Improving deep learning model quality and training efficiency via efficient data sampling and routing. arXiv preprint arXiv:2212.03597, 2022. [16] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. In Proceedings of the VLDB Endowment, 2020. [17] Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021. [18] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023. [19] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [20] MosaicML. Mosaicml composer: A pytorch library for efficient neural network training. https://github.com/mosaicml/composer, 2022. [21] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019. [22] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. [23] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020. 11

[24] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [25] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: Stateof-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020. [26] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022. [27] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [28] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Ma, and Yongqiang Ma. LLaMA-Factory: Unified efficient fine-tuning of 100+ language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. [29] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.

A

Cross-Rank Group Alignment Protocol

This appendix gives the full algorithm and supporting details summarized in Section 2.3. Let Gr be rank r’s current candidate-group list, Gr = |Gr |, and A = {r : Gr > 0} the active-rank set in a protocol round. When A is nonempty, ODB’s alignment target is computed only over active ranks:     + + Tgrp = max min max Gr , Cmin , Smin , 1 (3) r∈A

+ where Cr is rank r’s output-slot capacity, Sr is its buffered-sample count, and Cmin = + minr∈A, Cr >0 Cr and Smin = minr∈A, Sr >0 Sr are the positive minima over active ranks. Excluding zero-capacity/zero-sample ranks prevents an empty rank from collapsing the target to zero.

After computing Tgrp , each active rank adjusts: Split (upward, Gr < Tgrp ): scanning groups in reverse order, the first group with ≥ 2 samples is found and its last sample is extracted to form a new singleton; repeat until Gr =Tgrp . Overflow (downward, Gr > Tgrp ): the Tgrp largest groups are retained and samples from removed groups are returned to the buffer for reuse. This overflow recirculation ensures no samples are permanently discarded. Communication overhead. The primary metadata round performs one all_gather of (2 + 2 buffer_size) · W · sizeof(int64) bytes (≈128 KB at W =8, buffer=1024). Exact token-level loss scaling can trigger a second token-count gather under the deterministic all-rank predicate described in Section 2.3. The Gloo channel runs on CPU and overlaps with GPU compute.

B

Loss Scaling Derivation

Let rank , W −1} hold nr samples with tr valid tokens and per-token losses ℓr,i,k . Write P r ∈ {0, . . .P N = r nr , Ttok = r tr . The reference loss (single-rank pass under per-token mean reduction) is L⋆ =

1 X Ttok

r,i,k

12

ℓr,i,k .

(4)

Algorithm 1 Max-Based Bidirectional Group Alignment + + Require: Candidate group lists {G0 , . . . , GW −1 }, active set A = {r : |Gr | > 0}, Cmin , Smin (positive values only) Ensure: All active ranks have exactly Tgrp groups; inactive ranks remain idle; overflow samples returned to buffer + + 1: Tgrp ← max(min(maxr∈A |Gr |, Cmin , Smin ), 1) 2: for each active rank r ∈ A do 3: if |Gr | < Tgrp then 4: while |Gr | < Tgrp and ∃ group with |g| ≥ 2 do 5: Scan from last group backward; find first g ∗ with |g ∗ | ≥ 2; extract its last sample as a new singleton group 6: end while 7: else if |Gr | > Tgrp then 8: Sort groups by size (descending); keep top-Tgrp ; return remaining samples to data buffer 9: end if 10: end for

P P 1 ⋆ Each rank locally computes L̄r = (1/tr ) i,k ℓr,i,k ; naive DDP yields W r L̄r , which equals L only in the degenerate case tr ≡ Ttok /W . For weights P {wr } summing to one, replacing L̄r with L̄r · wr · W makes DDP’s post-averaging output equal r wr L̄r ; expanding shows the unique exact choice is wr = tr /Ttok . Sample-level weighting (wr = nr /N ) is exact only when the average tokens per sample tr /nr is identical across ranks. Approximate vs. exact mode. Eq. 2 requires per-group token counts consistent with the postalignment grouping. The primary all_gather piggybacks pre-alignment counts without an extra orig communication round; in approximate mode adjusted counts are estimated from τ̄r =torig r /nr . Exact mode uses the primary counts when alignment is a no-op and otherwise triggers a deterministic second all_gather to re-broadcast post-alignment counts, preserving deadlock-freedom. Empirically, in the ShareGPT4o profiling sweep, approximate and exact token-level modes have unscaled losses within 0.004, but only exact mode satisfies Eq. 2 bit-precisely; all main experiments use exact token-level scaling.

C

Formal Proofs

This appendix gives explicit, full proofs of Theorems 2 and 3 using (i) a per-rank state machine that tracks every place a sampler view may reside, and (ii) an explicit Lyapunov potential function that strictly decreases on emission rounds while skip rounds are finite. Sampler views are conserved as multiset membership at all times, so “no leak” is by inspection of the transition rules; identity coverage is handled by the view-to-identity projection. C.1

Per-Rank State and Transition Rules

Fix world size W , rank index r ∈ {0, . . . , W − 1}, and dataset identity set I = {0, . . . , N − 1}. DistributedSampler(drop_last=False) produces a per-rank sampler-view sequence Dr of size |Dr | = ⌈N/W ⌉ after padding the global shuffled index list to M := W · ⌈N/W ⌉ views and stride-sharding it across ranks. View positions are disjoint across ranks; their identity projection covers I, with P := M − N deterministic tail-padding views that cyclically re-use boundary identities to make per-rank counts equal. We track the view multiset Dr (so an identity that appears as a padding view is counted as a distinct sampler element); identity-level coverage is the cardinality of the corresponding identity set. At protocol round k, rank r’s state is
(k) (k) s(k) = Rr(k) , Q(k) , r r , Br , Er where the four pairwise disjoint components partition Dr : (k)

• Rr — sampler-pending: views Dr has not yet yielded. (k) • Qr — worker queue: views in flight from worker subprocesses to the collate process. 13

(k)

• Br — collate buffer: views received by collate but not yet emitted to the trainer. (k) • Er — emitted: views already delivered to the trainer in some prior batch. The protocol exposes three transition primitives, each of which moves sampler views between two components without creation or destruction: F ETCHr (F ) : F ⊆ Rr(k) , δ = |F |;

Rr(k+1) = Rr(k) \ F, Q(k+1) = Q(k) r r ⊎F

(k) D RAINr : Q(k) r → Br

E MITr (g) : Br(k) → Er(k) , moves group g ⊆ Br(k) Let D := max(prefetch_factor · num_workers, buffer_size) denote the configured per-rank outstanding-depth envelope. The iterator schedules fetch/drain so that the fetched-but-not-emitted view set Qr ⊎ Br stays within this envelope; Br contains the current collate buffer plus any overflow groups recirculated after alignment. No transition deletes elements: every transition is one of the three above or the local-rank no-op of a skip_output round. Lemma 1 (No-Leak Invariant). At every round k and on every rank r, (k) (k) Rr(k) ⊎ Q(k) = Dr . r ⊎ B r ⊎ Er (0)

(0)

(0)

(0)

Proof. Base case k = 0: Rr = Dr , Qr = Br = Er = ∅. Inductive step: each transition primitive moves a (possibly empty) subset between two components on the left-hand side, leaving the disjoint union invariant. C.2

Lyapunov Potential and Bounded Termination

Define the global potential Φ(k) :=

W −1 X

(k) |Rr(k) | + |Q(k) r | + |Br |

= M−

X

|Er(k) |,

r

r=0

where M = W · ⌈N/W ⌉ is the total sampler view count (App. C.1). Φ(k) ∈ {0, 1, . . . , M } is a non-negative integer. We track its evolution. (k)

Lemma 2 (Bounded-round progress). On any active outer round k (some rank has |Br | > 0), (k),+ (k),+ (k) let Gcur = maxr: Gr >0 Gr , and let Cmin and Smin be the round-k positive output-slot and (k),+ (k),+ (k) buffered-sample minima over active ranks. Define tk = max(1, min(Gcur , Cmin , Smin )). Local alignment may split groups before emission, but it does not move views into Er and performs finitely (k) many splits, bounded by the number of buffered views/groups in Br because each split only isolates existing views. After local alignment, the outer round guarantees one of the following: (a) (E MIT round.) Φ(k+1) ≤ Φ(k) − 1. (b) (S KIP round.) Φ(k+1) = Φ(k) and no view is emitted; the round only fetches/drains bounded in-flight views or observes sampler exhaustion. Skip rounds are finite because the fetched-but-not-emitted set is bounded by the outstanding-depth envelope D and the sampler is finite; local split operations are internal to the outer round and do not affect the communication-round count. Proof. Local split adjustment extracts views from existing groups but leaves them in Br , so it preserves Φ and is bounded by the finite number of buffered views/groups. In a normal P aligned case (Gr = tk for all active r), each active rank emits exactly tk groups containing g |g| ≥ tk ≥ 1 views; hence Φ strictly decreases by at least one. In an overflow case (Gr > tk ), the top-tk groups are emitted and the remainder is recirculated into Br ; the emitted groups again contain at least one view, so Φ decreases. If no active rank has enough material to emit, the round is a skip_output fetch/drain round; it cannot repeat indefinitely because outstanding in-flight views are bounded by D and the sampler is finite. Theorem 4 (Round-Count Bound). The protocol terminates in at most ⌈N/W ⌉ + O(D) rounds. 14

Proof. Let q = ⌈N/W ⌉ be the per-rank sampler-view quota under DistributedSampler with drop_last=False. In every E MIT outer round, aligned emission is not serialized by rank: after the bounded fetch/drain prefix, each unfinished active rank emits at least one sampler view in the shared step, while a rank with no remaining material is represented by the mode-specific finished state. Local split/overflow adjustment is internal to that round and does not add communication rounds. Therefore the shared protocol has at most q emitting outer rounds before local sampler-view quotas are drained and the mode-specific termination predicate can be raised. Non-emitting skip_output rounds only fetch/drain the bounded outstanding set (|Qr | + |Br | ≤ D by the configured envelope) or observe sampler exhaustion, so they contribute an O(D) epilogue. Thus each logical iteration terminates in at most q + O(D) = ⌈N/W ⌉ + O(D) outer rounds. This is a termination/quota bound only; identity-level coverage for non-join is not claimed by the proof and is handled empirically in App. C.6, while join mode gives the construction-level identity guarantee (Theorem 1). C.3

Uniform all_gather and Deadlock-Freedom

Lemma 3 (Uniform all_gather invariant). Every rank executes one unconditional primary all_gather per outer iteration. If exact token-level loss scaling needs post-alignment token counts, the optional second all_gather (Appendix B) is gated by a deterministic predicate ϕ(all_n_groups, all_idx_budgets) computed identically on every rank, so it is either executed by all or by none. Proof. The protocol executes the primary all_gather before any branch can return. The optional second all_gather’s entry predicate is a pure function of the same broadcast tensor, so all ranks evaluate it identically. Proof of Theorem 3 (Bounded Termination + Deadlock-Free). By induction on iteration i: if all ranks enter iteration i together (base i=0 trivial), Lemma 3 ensures all execute the same all_gather call(s) and observe identical broadcast tensors. The mode-specific termination predicate—all ranks advertise local finish in default join mode, or any rank advertises n_groupsr = −1 in opt-in non-join—is computed from those shared tensors and therefore evaluated identically; hence all enter iteration i+1 together or all exit together. No rank can block on all_gather because all ranks reach it. Bounded termination follows from Theorem 4. C.4

Sample-Quota Closure

Proof of Theorem 2 (No-Leak + Sample-Quota Closure). The no-leak claim is exactly Lemma 1. Sample-quota closure. The trainer side maintains an emitted-sample counter, accumulates the realized global per-step emitted-sample count, and terminates once the cumulative emit count reaches N . When a logical DistributedSampler iteration ends early (Theorem 3), the outer training loop starts the next logical iteration with a re-shuffled sampler; the stopping condition is unaffected by these chained iterations. Let Smax be the largest realized global emit count of one aligned trainer step under the configured outstanding-depth envelope; the final quota crossing occurs in one such step, so Semit − N ≤ Smax . Lemma 4 (Logical-discard bound, non-join). Within one logical sampler iteration under non-join termination, let Ur := Qr ⊎ Br be the fetched-but-not-emitted outstanding set on rank r; samplerpending views Rr are not fetched and are not counted P in Ur . Then the configured outstandingdepth envelope gives |Ur | ≤ D, hence ηlogical := N1 r |Ur | ≤ W · D/N . This is a per-iteration envelope, not a terminal identity-coverage statement. Non-join provides cumulative sample-quota closure (Theorem 2), and Appendix C.6 reports terminal identity coverage for the evaluated Full FT cells. Strict per-iteration logical zero-discard and identity coverage are obtained by join mode (Theorem 1), where |Ur | = 0 at termination by drain-then-signal. Proof. At a non-join stop point, fetched-but-not-emitted views can reside only in the worker queue Qr or collate buffer Br . ODB’s outstanding-depth envelope bounds this set by D on every rank, including overflow groups recirculated into Br after alignment. Therefore |Ur | = |Qr | + |Br | ≤ D for every rank at the non-join stop point. Summing over W ranks gives the stated bound. 15

C.5

Empirical ηquota

We instantiate Theorem 2 on the audited public Full-FT ODB runs by computing ηquota,emp := max(0, 1 − Semit /N ) from terminal trainer state. Across the 18 audited runs (3 datasets × 2 ODB configurations × 3 seeds), ηquota,emp = 0 uniformly with terminal epoch in {1.0000000, 1.0000334, 1.0000735}; the bounded overshoot is the final batch crossing the quota threshold (Theorem 2). The same quota check on the six 1000-sample synthetic distributions listed in App. I gives ηquota,emp = 0 with terminal epoch in {1.0000, 1.0001}. Per-iteration logical bounds ηlogical ≤ W · D/N for representative configurations are reported in Table 4; we use them only as worst-case protocol envelopes, while terminal measured quota and identity metrics are reported separately. N

W

D

ηlogical bound

157,712 207,865 207,865 207,865 54,424 545,178 545,178

8 8 8 8 8 8 8

4,096 1,024 4,096 2,048 4,096 1,024 8,192

20.8% 3.9% 15.8% 7.9% 60.2% 1.5% 12.0%

Configuration LLaVA 8B (D=4096) UltraChat 8B (ml8k pf256 buf256) UltraChat 8B (ml8k pf1024 buf1024) UltraChat 8B (ml16k pf512 buf1024) ShareGPT4o 8B (ml4k pf1024) MM-Mix 8B (ml8k pf256) MM-Mix 8B (extreme, ml4k pf2048)

Table 4: Per-iteration logical-discard upper bound ηlogical ≤ W · D/N (Lemma 4). The bound is a worst-case envelope on per-iteration un-emitted sampler views; cumulative-count ηquota is driven to 0 by the trainer-side emitted-sample counter (Theorem 2) in the audited runs, and per-sample ηidentity is empirically 0 on four terminal-state Full FT cells (Ultra and SGPT, 2B and 8B; App. C.6, Table 5; surplus emits matching DistributedSampler’s deterministic tail-padding count, with the same 2B/8B surplus). Strict per-iteration ηlogical = 0 with identity coverage by construction (independent of straggler asymmetry) is reserved for join mode (Theorem 1).

C.6

Terminal Identity Coverage

The ηlogical bound of Table 4 S is a worst-case per-iteration envelope. We also report the terminal identity metric ηidentity := 1−| r IDsr |/N , the fraction of dataset identities not emitted by the union of ranks at training termination. We measure it on four full-epoch Full FT cells: Ultra and SGPT at both 2B and 8B. These cells use the same protocol-relevant settings as their §3 counterparts—world size W =8, ODB knobs, seed, batch policy, and DistributedSampler(drop_last=False)— while using the full unsplit datasets for unambiguous N accounting. S Terminal-state result. P All four measured cells have ηidentity = 0: | r IDsr | = N . The surplus emit count r |emitsr | − N ∈ {4, 7} matches the deterministic tail padding of DistributedSampler with drop_last=False: W − (57,284 mod 8) = 4 for SGPT and W − (207,865 mod 8) = 7 for Ultra. Matched 2B and 8B runs have the same surplus count on each dataset (Table 5). Measured invariants. Two empirical invariants, both observed in all four measured cells, are sufficient to conclude ηidentity = 0: Shard-bounded emit, no extra duplicates. Every emitted ID is a valid dataset identity (a member of r’s own DistributedSampler shard Sr ), and the union of per-rank emitted-ID records contains no cross-rank duplicates beyond the P = W ⌈N/W ⌉ − N deterministic sampler-padding reuses (4 for SGPT, 7 for Ultra). Per-rank emit count. By termination each rank has emitted exactly q = ⌈N/W ⌉ sampler views (7,161 each for SGPT; 25,984 each for Ultra; same in matched 2B/8B runs). These two invariants then yield identity coverage as a deterministic implication: Proposition 1 (Identity closure from measured invariants). Consider a non-join ODB run with DistributedSampler(drop_last=False) over N dataset identities and world size W . Let q = ⌈N/W ⌉, M = W q, and P = M − N be the deterministic sampler-padding surplus. If the terminal emitted-ID logs satisfy: (a) all emitted IDs are valid dataset identities and contain no 16

duplicates beyond the P deterministic sampler-padding reuses; and (b) each rank emits exactly q sampler views by termination, then the terminal identity coverage is exact: ηidentity = 0. Proof. By condition (b), the run emits M = W q = N + P sampler views in total. By condition (a), the number of duplicate identity emissions is at most the deterministic padding surplus P . Therefore the number of unique emitted dataset identities is at least M − P = N . Since all emitted IDs are valid dataset identities (and the dataset has only N identities), the unique-emit set is a subset of S size atSleast N of an N -element set, hence equals it: r IDsr = {0, . . . , N − 1}, and ηidentity = 1 − | r IDsr |/N = 0. The argument is conditional: it converts the two measured invariants into a closed-form identityclosure conclusion for these cells, but does not promote the empirical premises into a non-join formal guarantee in general (which is reserved for join mode, Theorem 1). Measured cell ODB setting

N

D

emits / rank total emits dup vs W −N mod W ηidentity

Terminal-state Full FT cells (full epoch) SGPT 2B ml4k pf1024 57,284 4,096 7,161 (×8) SGPT 8B ml4k pf1024 57,284 4,096 7,161 (×8) Ultra 2B ml8k buf256 207,865 1,024 25,984 (×8) Ultra 8B ml8k buf256 207,865 1,024 25,984 (×8)

4 vs 4 ✓ 4 vs 4 ✓ 7 vs 7 ✓ 7 vs 7 ✓

57,288 57,288 207,872 207,872

0% 0% 0% 0%

Table 5: Terminal identity coverage on four Full FT cells (Ultra and SGPT, 2B and 8B, full epoch). The “ODB setting” column names the two audited configurations; “dup vs W −N mod W ” compares observed surplus emits with the deterministic DistributedSampler(drop_last=False) tail-padding count. All rows cover N unique dataset identities; matched 2B/8B runs have the same surplus count on each dataset.

Relation to the ηlogical envelope. The distinction is that ηlogical bounds sampler views that may remain un-emitted within one logical iteration, whereas ηidentity measures the union of emitted dataset identities at termination. In the four measured Full FT cells, the union covers all N identities. The construction-level identity guarantee for arbitrary configurations is provided by join mode (Theorem 1).

D

Grouping Algorithm Example

We illustrate the grouping algorithm (Section 2.2) with a concrete example. Suppose Lmax = 1000 and a rank’s buffer contains four samples with lengths {100, 200, 500, 800}. Sort ascending, [100, 200, 500, 800], initialize threshold t = 1, and iterate from longest to shortest: 1. Sample 800: group = [800]. Size = 1, t = 1; size ≥ t, so finalize G1 = [800]. Update t ← B(800) = ⌊1000/800⌋ = 1. 2. Sample 500: group = [500]. Size = 1, t = 1; size ≥ t, so finalize G2 = [500]. Update t ← B(500) = ⌊1000/500⌋ = 2. 3. Sample 200: group = [200]. Size = 1, t = 2; size < t, continue. 4. Sample 100: group = [100, 200]. Size = 2, t = 2; size ≥ t, so finalize G3 = [100, 200]. Update t ← B(100) = ⌊1000/100⌋ = 10. Result: three groups, ordered from short to long: Group

Samples (lengths)

Padded to

Padded tokens

G3 G2 G1

100, 200 500 800

200 500 800

400 500 800

The threshold carry-over is the key mechanism: G2 contains only one sample because the threshold inherited from G1 is t = B(800) = 1, which is already met. Meanwhile, the updated t = B(500) = 2 requires G3 to accumulate two samples before finalizing, grouping the two shortest sequences 17

together. With more samples of similar lengths (the typical case), each group’s padded-token cost approaches Lmax .

E

Protocol State Machine

Figure 3 renders the non-join per-iteration state machine. It is the executable transition system abstracted in Appendix C: the Lyapunov potential of Appendix C.2 contracts on active-emission rounds after bounded local split/overflow adjustment, and is unchanged on finite skip rounds (Lemma 2). The transition to Logical Stop occurs when any rank signals gr = −1; the trainer-side quota counter then chains logical iterations until the sample quota is met (Theorem 2). In the figure, gr denotes rank r’s gathered group-status signal and br denotes rank r’s idx_budget. active emit: ∀r : gr ≥ 0 minr br > 0 Φ(k+1) ≤ Φ(k) − 1

∃r : gr = −1

Running

Logical Stop

SENTINEL exit

skip_output: ∀r : gr ≥ 0 minr br = 0 Φ unchanged, |R| drains

Figure 3: Per-iteration state machine of the Unified Loop Protocol in non-join mode. Transition guards reference the per-rank state of Appendix C.1; gr is rank r’s gathered group-status signal, br is rank r’s idx_budget, and Φ is the Lyapunov potential of Appendix C.2. The active self-loop contracts Φ on emission after bounded local split/overflow adjustment (Lemma 2); the skip_output self-loop is finite by sampler exhaustion.

F

Two-Node Validation

Empty-rank join-mode audit. We evaluate a two-node H20 empty-rank multi-node case with 8 GPUs per node (16 ranks) outside the equal-samples premise of Theorem 2. Because this setting intentionally violates that premise, we use it only as a liveness audit for deadlock-freedom and bounded termination, not as a quota- or identity-coverage audit. We run the DataLoader-level ODB path in join mode. The one-node 8-rank companion run terminates with PASS world=8; the 2-node run uses 8 H20 GPUs per node (16 ranks total), sets global rank 15 as the exhausted empty rank, and terminates with PASS world=16 empty_rank=15. Table 6 reports the per-rank sampler assignment, trainer-side outputs, and post-join liveness flags. In both audits all active ranks emit batches, the exhausted empty rank emits zero batches, and every collate subprocess exits. Table 6: Two-node empty-rank join-mode audit: per-rank sampler views and trainer-side outputs. Steps are optimizer updates per rank; Emitted is the number of sampler views emitted to the trainer, so Emitted can exceed Steps under dynamic batching. Done=1 indicates clean collate-subprocess exit after join. Rank

0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15

Assigned 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 Steps 35 35 33 33 33 33 31 31 31 31 29 29 29 29 27 Emitted 46 46 42 42 42 42 38 38 38 38 34 34 34 34 31 Done 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

0 0 0 1

The 21-view assigned–emitted difference is expected in this unequal-partition audit outside the theorem premise and is not interpreted as a coverage metric: the empty-rank construction breaks 18

the equal per-rank sampler-quota premise of Theorem 2, and cross-rank alignment gates emission through the global target Tgrp in Eq. 3. The measurement therefore targets liveness—no deadlock, bounded termination, and clean subprocess exit—rather than quota or identity closure. These audits exercise the join-mode drain-then-signal path for exhausted empty ranks. They are liveness audits rather than the source of the main-table throughput rows, and they do not claim support for an active rank that advertises zero samples before it has reached the finished state. Two-node Full FT training with MMMU-MC likelihood scores. We also report a twonode RDMA-enabled 16-rank Qwen3-VL Full FT validation. Multimodal Score cells use MMMU-MC-choice-likelihood-v1; training and validation-loss metrics use the same aggregation protocol as Table 1. All 72 two-node MMMU-MC seed cells pass the protocol validation checks (evaluated=850, excluded=50, total=900; no non-finite score audit failures). This table is multi-node validation evidence rather than the single-node headline setting of Table 1. On 2B, ODB reaches 1.71× on LLaVA, 3.76× on ShareGPT4o, and 2.86× on UltraChat; under MMMU-MC likelihood, ODB remains in the Standard/oracle quality band on both two-node 2B multimodal rows. On 8B, ODB reaches 1.80× on LLaVA and 3.78× on ShareGPT4o; it is highest on ShareGPT4o and within 0.08 pp of the best LLaVA score. Table 7: Two-node Full FT Results with MMMU-MC likelihood scores. 2 nodes × 8 H20 GPUs (16 ranks), RDMA-enabled Qwen3-VL Full FT, DeepSpeed ZeRO-2/bf16. Rows are 3-seed mean±std. LLaVA and ShareGPT4o Score cells use MMMU-MC-choice-likelihood-v1; UltraChat uses MMLU. Oracle rows use scalar length caches for batch construction and exclude cache construction from reported training throughput. 8B (2-node Full FT) sam/s

Speedup

Score

2B (2-node Full FT)

Dataset

Method

Val Loss

UltraChat CV=0.48 MMLU

Standard 6.82 ± 0.40 1.00× 74.73 ± 0.17% 0.856 ± 0.001 Sorted 11.36 ± 1.32 1.67× 75.14 ± 0.17% 0.883 ± 0.002 Packinga 20.19 ± 0.66 2.96× 75.91 ± 0.01% 1.172 ± 0.006 GMT-oracleb 17.38 ± 1.54 2.55× 75.10 ± 0.19% 0.856 ± 0.001 BMT-oracleb 17.92 ± 0.99 2.63× 75.56 ± 0.10% 0.859 ± 0.001 HFG-oracleb 10.75 ± 0.12 1.58× 75.19 ± 0.17% 0.857 ± 0.001 ODB 18.80 ± 0.03 2.76× 75.47 ± 0.10% 0.859 ± 0.001

sam/s 24.56 ± 0.12 47.66 ± 2.63 72.23 ± 1.44 68.44 ± 1.44 64.05 ± 3.02 35.93 ± 1.54 70.15 ± 1.86

Speedup

Score

Val Loss

1.00× 58.82 ± 0.30% 0.971 ± 0.001 1.94× 59.04 ± 0.07% 1.018 ± 0.001 2.94× 60.00 ± 0.09% 1.174 ± 0.001 2.79× 59.28 ± 0.10% 0.992 ± 0.001 2.61× 59.16 ± 0.20% 0.984 ± 0.001 1.46× 59.05 ± 0.34% 0.980 ± 0.001 2.86× 59.09 ± 0.11% 1.016 ± 0.002

Standard 25.42 ± 0.13 1.00× 54.63 ± 0.07% 1.032 ± 0.001 87.10 ± 0.64 1.00× 40.08 ± 0.56% 1.172 ± 0.002 Sorted 33.53 ± 2.89 1.32× 54.82 ± 0.61% 1.131 ± 0.002 119.81 ± 0.42 1.38× 38.63 ± 0.14% 1.294 ± 0.001 LLaVA b GMT-oracle 47.98 ± 0.56 1.89× 54.63 ± 0.98% 1.074 ± 0.002 155.91 ± 2.91 1.79× 38.90 ± 0.07% 1.203 ± 0.002 CV=0.29 b MMMU-MC BMT-oracle 47.16 ± 0.29 1.86× 54.51 ± 0.18% 1.074 ± 0.002 152.81 ± 2.42 1.75× 39.21 ± 0.34% 1.203 ± 0.002 HFG-oracleb 33.75 ± 0.33 1.33× 55.10 ± 0.07% 1.069 ± 0.002 113.67 ± 0.51 1.31× 40.78 ± 0.89% 1.174 ± 0.002 ODB 45.72 ± 0.35 1.80× 55.02 ± 0.47% 1.105 ± 0.001 148.80 ± 0.78 1.71× 40.24 ± 0.83% 1.239 ± 0.002 Standard 2.73 ± 0.41 1.00× 53.06 ± 0.85% 1.218 ± 0.006 3.09 ± 0.02 1.13× 53.29 ± 0.12% 1.240 ± 0.006 ShareGPT4o Sorted b GMT-oracle 3.37 ± 0.02 1.23× 54.35 ± 1.16% 1.217 ± 0.006 CV=1.00 b MMMU-MC BMT-oracle 3.13 ± 0.32 1.15× 53.61 ± 0.74% 1.218 ± 0.006 HFG-oracleb 2.91 ± 0.35 1.07× 53.33 ± 0.95% 1.217 ± 0.005 ODB 10.32 ± 0.46 3.78× 55.25 ± 0.47% 1.230 ± 0.006

8.33 ± 0.05 8.69 ± 0.11 8.50 ± 0.01 9.01 ± 0.07 9.59 ± 0.08 31.35 ± 0.37

1.00× 41.53 ± 0.20% 1.293 ± 0.006 1.04× 39.84 ± 0.25% 1.320 ± 0.006 1.02× 41.10 ± 0.65% 1.292 ± 0.006 1.08× 41.18 ± 1.14% 1.293 ± 0.006 1.15× 40.90 ± 0.80% 1.293 ± 0.006 3.76× 41.45 ± 0.67% 1.356 ± 0.005

a

Packing is UltraChat-only and not drop-in for multimodal training in our stack; bold throughput/speedup marks ODB among online DDP batchers. Packing val_loss uses the packed-sequence denominator and is not directly comparable to per-original-sample rows. b GMT-/BMT-/HFG-oracle rows use scalar oracle length caches for batch construction; reported throughput excludes cache construction. MMMU-MC scores are choice-likelihood scores over the 850/900 letter-labeled MMMU validation rows; the 50 non-letter rows are excluded by protocol.

G

Auxiliary Instruction-Following Evaluation

As an additional supervised fine-tuning (SFT) quality check, we evaluate the same Full FT UltraChat and ShareGPT4o checkpoints on IFEval [29], a rule-based instruction-following benchmark with verifiable text constraints. This check is intended to complement the MMLU/MMMU-MC and validation-loss evidence in the main paper: MMLU-style multiple-choice accuracy is not the only proxy for SFT behavior, while IFEval measures whether a generated response follows explicit formatting and content constraints. We use greedy decoding with max 1024 new tokens and the standard strict/loose IFEval rule checks. The ShareGPT4o rows are checkpoints trained on multimodal data, but IFEval itself is text-only; we therefore interpret it as an instruction-following auxiliary evaluation rather than a visual/multimodal benchmark. Table 8 reports this auxiliary check alongside the corresponding main-task score for the same checkpoint group. 19

Table 8: Auxiliary IFEval instruction-following check. Values are mean±std over three matched seeds, reported as percentages. Main is MMLU for UltraChat and MMMU-MC choice-likelihood accuracy for ShareGPT4o; IFEval uses 541 prompts and 834 instructions. Scale Data 2B 2B 2B 2B 8B 8B 8B 8B

Method

Main

P-strict

P-loose

I-strict

I-loose

UltraChat Standard 58.17±0.06 25.02±2.67 29.82±3.15 37.81±2.37 42.97±2.43 UltraChat ODB 58.98±0.18 27.91±1.51 30.75±1.30 41.45±1.34 44.16±1.20 ShareGPT4o Standard 41.29±0.71 45.78±1.41 50.34±0.77 56.83±1.18 61.43±0.78 ShareGPT4o ODB 40.12±0.36 52.19±1.02 57.18±0.21 63.39±0.39 68.15±0.39 UltraChat Standard 72.85±0.40 10.54±0.67 13.43±0.11 23.18±0.69 27.10±0.42 UltraChat ODB 74.75±0.11 13.68±0.67 16.51±1.05 26.50±0.86 30.82±1.05 ShareGPT4o Standard 52.43±0.44 71.53±0.67 77.14±0.59 79.62±0.48 83.77±0.60 ShareGPT4o ODB 53.88±0.20 77.82±0.67 81.95±0.47 84.33±0.30 87.33±0.28

On this auxiliary instruction-following check, ODB remains in the same quality band as Standard and has positive mean differences on the four IFEval metrics in each of the four matched settings. At the seed level, prompt-level strict accuracy is positive for ODB in all 12 matched pairs, while prompt-level loose accuracy is positive in 11/12 pairs (the remaining 2B UltraChat seed has a small negative loose-score difference). This auxiliary check does not indicate degraded SFT instructionfollowing behavior in these matched checkpoints, while the primary quality evidence remains the validation-loss and task-benchmark results reported in Tables 1, 7, and 16.

H

Quality Hyperparameter Sensitivity

The quality results in Tables 1 and 16 use ODB configurations selected per dataset by the protocol in §3. Since ODB fixes bs = 1, the primary quality-sensitive knob is Lmax ; prefetch_factor controls outstanding depth and is treated as a throughput/overlap knob. For low-CV datasets (e.g., LLaVA, CV=0.29), Lmax significantly affects quality because most samples are short (<2048 tokens): a large Lmax causes ODB to pack many short samples into a single step, creating an effective batch size that deviates from standard training. Table 9 shows an auxiliary generated-answer MMMU sensitivity check for the LLaVA LoRA (8B) Lmax sweep (one validation seed). This auxiliary generated-answer setting is not mixed with the main MMMU-MC tables. In this auxiliary generated-answer setting, Lmax =4096 matches standard training quality (28.2% vs. 28.4%), while the throughput-optimal configuration (Lmax =16384) is 1.3 pp lower. Table 9: Auxiliary LLaVA ODB Lmax sensitivity check (Qwen3-VL-8B, LoRA, generated-answer MMMU validation, one seed). ODB bs=1 (fixed), pf=256 (default). Standard baseline: bs=8, generated-answer MMMU=28.4%. Main multimodal benchmark tables use parser-free MMMUMC. Lmax

Gen-answer MMMU

∆ vs. Std (pp)

16384 8192 4096

27.1% 27.6% 28.2%

−1.3 pp −0.8 pp −0.2 pp

Lmax controls the dynamic batch size via B(l) = ⌊Lmax /l⌋; larger values pack more short samples per step, shifting the effective batch composition further from standard training. The 8192 result is the mean of two independent runs (27.7%, 27.4%). All main multimodal benchmark scores use MMMU-MC.

The pattern is consistent: on low-CV data where samples are homogeneous, ODB should use a conservative Lmax to keep the dynamic batch composition close to standard training. Accordingly, the high-CV ShareGPT4o rows use the selected throughput configurations without a quality-specific Lmax override, while remaining within the paper’s Standard-comparable quality framing.

I

Experimental Setup Details

Per-sample tokenized length statistics (full-pass under the Qwen3-VL tokenizer with vision tokens expanded at load time) and the resulting cutoff_len: 20

Table 10: Per-sample tokenized length statistics. ShareGPT4o has the longest tail (Max 12K) and the highest CV. Dataset

Samples

Mean

Median

P95

P99

Max

cutoff_len

UltraChat LLaVA ShareGPT4o

207,865 157,712 57,284

1,196 508 1,511

1,104 463 977

2,239 814 4,584

3,065 914 8,937

4,471 1,260 12,110

8192 2048 16384

The 6 synthetic distributions used in correctness audits (1000 samples each): uniform-narrow U[64,512], uniform-wide U[64,2048], longtail (90% short / 10% long), bimodal (50/50), all-long U[1800,2048], and all-short U [32,64]. GMT/BMT distributed batching (§3.2, §5). GMT and BMT are rank-replicated oracle samplers: every rank computes the same global batch list (GMT: ascending-length sort plus greedy packing against a max-token budget; BMT: epoch-seeded shuffle, sample-count buckets, withinbucket length sort, greedy packing, then batch shuffle). Following fairseq-style max-token batching, feasibility is computed on padded token area, maxi∈b li ·|b| ≤ max_tokens_budget, with singleton overflows allowed in our oracle implementation to preserve zero truncation and full-epoch coverage. The list is then padded to a multiple of the world size by wrap-around repetition of the leading batches and assigned to ranks by striding, guaranteeing identical step count on every rank. Lastbatch handling is by construction: the wrap-around padding adds at most W −1 repeated batches per epoch, the offline analog of ODB’s wrap-around padding (§2.3). All ranks read the same scalar lengths array from the oracle cache during DataLoader construction, so no per-step length broadcast is required at training time. Oracle length cache for GMT-oracle / BMT-oracle (§3.2, §5). To make the GMT / BMT comparison maximally favorable to the offline token-budget family, we precompute a per-(dataset, transform policy, template, and cutoff) scalar cache of len(input_ids) for every sample by a single forward pass through the LLaMA-Factory preprocessing/augmentation, templating, tokenization, and visual-token-expansion pipeline; no token IDs are stored or reused at training time. Constructiontime precompute cost on a single H20 was 31 s for UltraChat (207,865 samples; ∼6,700 sam/s, pure tokenization), 137 s for ShareGPT4o (57,284 samples; ∼418 sam/s, image-blob + tokenization), and 55 min for LLaVA (157,712 samples; ∼48 sam/s, image-IO-bound under per-sample JPEG decode). The same runtime path therefore also appears during oracle cache construction, but the reported training throughput excludes this one-time cost; during training, GMT/BMT still perform normal online preprocessing, augmentation, tokenization, and visual-token expansion. The cache is invalidated and must be rebuilt on any preprocessing or augmentation-policy change; ODB requires no separate length precompute because it forms batches from lengths observed online in the DataLoader path. MM-Mix composition (case study, Section 3.7). The production multimodal mixture aggregates 7 open-source datasets (272,589 unique samples; 2 training epochs ⇒ 545,178 sample-views) spanning OCR, VQA, and image captioning. The bimodal length distribution (many short OCR/VQA labels alongside long captioning samples) yields CV≈0.8 and short-sample fraction fs ≈0.37. Table 11: MM-Mix composition. All datasets are publicly available under their original licenses; “LO” = LLaVA-OneVision distribution. Dataset

Source

Samples

Task / Modality

IIIT5K ORAND-CAR-A BCTR-Splice (scene) A-OKVQA VQAv2 Image-Textualization (filtered) ShareGPT4o (caption subset)

LO LO BCTR LLaVA GeneralVQA LO LO

1,990 1,999 11,904 17,056 82,783 99,573 57,284

English scene-text OCR Synthetic digit OCR Chinese scene-text OCR Multiple-choice visual QA Open-ended visual QA Image→text captioning Long-form captioning / dialogue

272,589

7 datasets, 2 epochs = 545,178 sample-views

Total (unique)

21

8B MM-Mix throughput sweep (Section 3.7). On Qwen3-VL-8B-Instruct (8×H20, 20-min profiling window per configuration), Standard bs=1 reaches 7.55 sam/s, Sorted bs=2 reaches 15.80 sam/s (2.09×), and ODB peaks at Lmax =8192 with 22.07 sam/s (2.92×). The Lmax sweep at default I/O (pf=256, nw=4) is single-peaked: Lmax =4096 gives 2.42×, Lmax =6144 gives 2.87×, Lmax =8192 gives 2.92×, Lmax =12288 gives 2.80×. Aggressive prefetch values (pf≥1024, lifting outstanding-depth D = pf×nw to ≥ 4096) regressed sample throughput on this 8B workload—with Lmax =4096, pf=1024/2048/4096 decreasing to 1.58 × /1.29 × /0.98×—because larger in-flight buffers stress 8B activation memory through wider length-grouped batches that lengthen step time more than they raise per-step sample count, consistent with the first-order memory model discussed in Limitations. We therefore use the default ODB configuration as the starting point for this workload. MM-Mix churn-inclusive accounting and benchmark quality. For churn-inclusive accounting, we distinguish churn-exclusive training throughput, which excludes offline preparation, from a conservative cache-inclusive cost lower bound that adds only the measured scalar oracle-cache construction time. The MM-Mix GMT/BMT oracle cache was built on one H20 for 272,589 unique samples in 299.9 s (309 s end-to-end wall-clock), producing the per-sample post-pipeline len(input_ids) cache used for oracle batch construction. This charge is still favorable to offline methods: it does not separately account for broader production overheads such as sample-store scans outside the measured prepass, materializing and validating sorted/bucketed orders, staging metadata across workers, or rebuilding artifacts when the mixture, template, transform policy, or cutoff changes. ODB has no separate scalar-length precompute and forms batches from lengths observed online in the DataLoader path. Table 12: MM-Mix full-epoch 2B case-study results. Two nodes × 8 H20 GPUs, Qwen3-VL-2B Full FT, 3-seed mean±std. sam/s is train-split emitted samples divided by wall-clock; for ODB we recompute literal throughput from the train split and runtime because the fixed-batch counter is not meaningful for dynamic batches. Score is MMMU-MC choice-likelihood accuracy.

Method

sam/s

Speedup

Score

Val Loss

Standard Sorted GMT-oracle BMT-oracle HFG-oracle

17.85 ± 0.15 20.62 ± 2.08 26.68 ± 2.09 28.66 ± 0.29 19.46 ± 2.48

1.00× 1.15× 1.49× 1.61× 1.09×

43.33 ± 2.24 43.65 ± 2.32 49.14 ± 0.65 48.27 ± 0.60 43.49 ± 1.67

0.9674 ± 0.0035 1.4028 ± 0.0312 0.9731 ± 0.0028 0.9732 ± 0.0032 0.9656 ± 0.0028

ODB

79.15 ± 4.16

4.43×

46.31 ± 0.44 1.0137 ± 0.0029

Per-config ODB hyperparameters (Table 1, footnote b ). tuples used in Table 1:

The per-config (Lmax , pf, buffer)

• UltraChat 8B: (12288, 1024, 1024) • UltraChat 2B: (16384, 1024, 1024) • LLaVA 8B: (12288, 256, 1024) • LLaVA 2B: (8192, 1024, 1024) • ShareGPT4o 8B: (12288, 256, 1024) • ShareGPT4o 2B: (4096, 256, 1024) These tuples are the selected full-epoch configurations after the speed-first profiling rule in §3; fullepoch quality is reported separately in Table 1. The resulting updates-per-epoch and batch-shape statistics are reported below rather than constrained to a fixed universal step-count target. Throughput decomposition (8B Full FT). Table 13 decomposes throughput into the per-step factors that dynamic batching changes. The columns are reported per-cell as 3-seed means: sam/s = train-split emitted samples / wall-clock; tok/s = real unpadded tokens / wall-clock; upd/ep = optimizer updates per epoch; sam/upd = emitted samples/update and tok/upd = real 22

P P unpadded tokens/update; pad% = 1− Lreal / Lcompute (cumulative padding fraction); dl-wait% and compute% = unhidden DataLoader wait and GPU-compute fractions of elapsed wall time (remainder is pipeline overlap and other; we report the unhidden component because pipeline overlap, when high, makes raw nvidia-smi utilization a misleading proxy). HuggingFace Trainer’s native train_samples_per_second reports world_batch × updates/runtime, which double-counts under dynamic batching where each update consumes a variable real-sample count. Table 13: Throughput decomposition on 8B Full FT. For the Standard, ODB, GMT-oracle, BMToracle, and HFG-oracle cells of the 8B side of Table 1, this table reports literal sam/s, updates per epoch, emitted samples/update, real unpadded tokens/update, cumulative padding, and unhidden DataLoader-wait / compute fractions. For dynamic-batch rows, sam/s is recomputed as train-split emitted samples divided by wall-clock time, matching Table 1. Dataset

Method

sam/s

UltraChat

Standard ODB GMT-oracle BMT-oracle HFG-oracle

LLaVA

ShareGPT4o

tok/s

upd/ep

sam/upd

tok/upd

pad%

dl-wait%

compute%

5.77 6,901 24,684 10.23 12,234 2,692 10.94 13,087 1,890 10.31 12,346 1,901 7.33 9,127 12,342

8.00 9,566 73.36 87,725 104.48 124,981 103.90 124,205 16.00 19,932

0.0 1.3 0.0 0.5 12.8

0.01 0.00 0.00 0.00 0.01

99.9 99.8 99.8 99.8 99.9

Standard ODB GMT-oracle BMT-oracle HFG-oracle

14.38 24.87 26.65 25.70 21.84

7,298 12,621 13,549 13,068 11,184

2,342 844 591 597 1,171

63.97 32,469 177.59 90,133 253.51 128,886 250.97 127,597 127.95 65,523

32.5 2.1 0.0 0.6 15.8

0.00 1.62 0.00 0.00 0.00

99.7 97.5 99.3 99.3 99.5

Standard ODB GMT-oracle BMT-oracle HFG-oracle

2.37 5.83 2.57 2.50 2.82

3,535 8,705 3,876 3,795 4,323

6,803 1,030 4,762 6,057 6,803

8.00 52.82 11.43 8.98 8.00

0.0 0.9 1.8 0.0 0.0

0.01 1.78 0.00 0.01 0.01

99.9 97.3 99.8 99.8 99.8

11,949 78,891 17,236 13,619 12,284

Three patterns are visible. (i) Padding is the binding constraint on LLaVA, sample-count on UltraChat / ShareGPT4o. Standard hits 32.5% padding on LLaVA, whereas bs=1 Standard avoids padding on UltraChat and ShareGPT4o at the cost of only eight samples/update. ODB attacks the limiting factor in each regime: it reduces LLaVA padding to 2.1% and raises sam/upd to 177.6, while increasing sam/upd by 9.2× on UltraChat and 6.6× on ShareGPT4o. (ii) Oracle baselines differ in update geometry. On LLaVA, GMT/BMT-oracle process roughly 251–254 samples/update and only 591–597 updates per epoch, versus ODB’s 177.6 samples/update and 844 updates; HFG’s randomized fixed-batch construction instead uses 128.0 samples/update with 15.8% padding. On ShareGPT4o, HFG falls back to the bs=1 shape, while ODB reaches 52.8 samples/update. These regimes are interpreted jointly with validation loss and MMMU-MC rather than as throughput alone. (iii) Pipeline starvation is not the dominant explanation. Standard and oracle rows are compute-bound, and ODB’s multimodal rows show only a small unhidden DataLoader-wait fraction (≤ 1.78%); thus the throughput differences primarily reflect batch shape and update geometry, not raw I/O efficiency. Throughput decomposition (2B Full FT). Table 14 reports the same decomposition for the 2B side of Table 1. It uses the same literal-throughput convention: sam/s is train-split emitted samples divided by wall-clock time, while token/update and DataLoader-wait fields are aggregated from the corresponding full-epoch runs. The 2B decomposition shows the same batch-shape mechanism as the 8B table, with scale-specific details. Standard’s bottleneck again flips between sam/upd (UltraChat / ShareGPT4o, bs=1) and padding (LLaVA, 24.2%). ODB attacks the cell-specific bottleneck: on LLaVA it reduces padding to 1.6% and raises samples/update from 32.0 to 119.0; on UltraChat and ShareGPT4o it raises the fixed-bs Standard rows from eight samples/update to 98.6 and 20.0 respectively. The oracle rows achieve competitive throughput through offline length-aware grouping, but with different update geometry: on LLaVA, GMT/BMT use about 60 samples/update, HFG keeps the fixed-batch shape at 64.0 samples/update with 14.5% padding, and ODB uses 119.0 samples/update; on UltraChat, HFG keeps the bs=1 shape while GMT/BMT widen updates to 49.6/103.9 samples. DataLoader wait remains negligible except for ShareGPT4o ODB (3.1%), where throughput is still dominated 23

Table 14: Throughput decomposition on 2B Full FT. For the Standard, ODB, GMT-oracle, BMToracle, and HFG-oracle cells of the 2B side of Table 1, this table reports literal sam/s, updates per epoch, emitted samples/update, real unpadded tokens/update, cumulative padding, and unhidden DataLoader-wait / compute fractions. Rows are 3-seed means over full-epoch runs. Dataset

Method

sam/s

tok/upd

pad%

dl-wait%

compute%

UltraChat

Standard ODB GMT-oracle BMT-oracle HFG-oracle

20.98 36.91 39.44 35.84 27.28

25,087 24,684 44,129 2,003 47,153 3,983 42,848 1,901 33,447 24,684

tok/s

upd/ep

sam/upd

8.00 9,566 98.59 117,886 49.58 59,272 103.90 124,224 8.00 9,808

0.0 1.7 0.0 0.4 0.0

0.04 0.01 0.01 0.00 0.05

99.9 99.6 99.8 99.8 99.8

LLaVA

Standard ODB GMT-oracle BMT-oracle HFG-oracle

47.92 82.42 79.44 75.64 69.52

24,319 41,830 40,375 38,456 35,837

4,683 1,259 2,496 2,502 2,342

31.99 119.01 60.02 59.89 63.97

16,238 60,399 30,505 30,448 32,977

24.2 1.6 0.0 0.1 14.5

0.02 0.01 0.02 0.01 0.01

99.6 98.5 99.3 99.4 99.4

ShareGPT4o

Standard ODB GMT-oracle BMT-oracle HFG-oracle

6.51 16.09 7.03 7.03 7.71

9,717 24,027 10,599 10,731 11,833

6,803 2,719 4,762 4,763 6,803

8.00 20.01 11.43 11.43 8.00

11,949 29,892 17,236 17,447 12,284

0.0 0.4 1.8 2.1 0.0

0.01 3.08 0.01 0.01 0.01

99.9 95.5 99.8 99.9 99.8

by the larger emitted-sample/update count rather than by I/O starvation. Thus the 2B table sharpens the 8B conclusion: ODB’s gain is a batch-shape effect, not a Trainer-accounting artifact or raw I/O speed.

J

HFG-oracle Randomized Fixed-Batch Baseline

HFG-oracle instantiates the HuggingFace group_by_length family as a randomized fixed-batch oracle baseline. Each epoch samples a random permutation, partitions it into megabatches, sorts each megabatch by the oracle post-tokenization length, concatenates the megabatches, pads the index list to a multiple of world size, and stride-shards it across ranks. It uses the same scalar length cache as GMT/BMT-oracle but keeps a fixed batch size, separating epoch-level randomization from max-token scheduling while avoiding globally sorted epoch order. For HFG-oracle, bs is selected as the largest full-epoch-safe fixed batch size under the same profiling and full-epoch survival protocol used for Sorted; speedups normalize to the matching Standard rows in Table 1. The HFG-oracle rows are reported directly in Table 1 rather than duplicated in a second appendix table. They show that randomized length grouping controls for fully sorted epoch order without changing the main conclusions: under MMMU-MC, HFG remains in the same quality band as the other non-sorted LLaVA methods, but its fixed batch size cannot exploit the high-CV ShareGPT4o tail where ODB is 2.07× faster at 8B (5.83 vs. 2.82 sam/s) and 2.09× faster at 2B (16.09 vs. 7.71 sam/s). Like GMT/BMT-oracle, HFG inherits the cache-rebuild limitation under preprocessing, template, cutoff, or augmentation changes, whereas ODB observes lengths online after those transformations.

K

CV/fs Two-Feature Decomposition: Phenomenological Reference

Section 4 uses CV and the short-sample fraction fs qualitatively. We report here the explicit twoanchor pinning that motivated their selection, together with the methodological caveat that prevents us from positioning it as a predictive model. The minimal two-feature linear form, Ŝ(CV, fs ) = 1 + α CV + β fs ,

(5)

pinned on the two 2B Full FT workloads with both features measured (ShareGPT4o: CV=1.00, fs ≈0.01, S=2.47; MM-Mix: CV=0.80, fs ≈0.37, S=4.43), gives α≈1.41, β≈6.23. 24

Scope of the two-feature fit. (CV, fs ) is a workload/configuration descriptor: CV summarizes the tokenized length distribution, while fs = Pr[ℓ < Lmax /4] measures short-sample mass under the selected token budget. The headline cells are dominated by four dataset-level workload families rather than 14 independent locations in the (CV, fs ) plane: 8B Full FT ×3 + 2B Full FT ×3 + 8B LoRA ×3 + 2B LoRA ×3 + MM-Mix at 2B/8B vary model scale or finetuning regime around a small set of length-distribution families. Treating all cells as independent would mostly add replication noise from model scale and finetuning regime. A leave-one-out cross-validation (LOOCV) R2 computed over those cells would reflect cell-noise variance, not the predictive power of the two-feature form. Scope. Eq. 5 should be read as a phenomenological reference within the calibrated range CV ∈ [0.80, 1.00], fs ∈ [0.01, 0.37]; it captures the separation between MM-Mix and ShareGPT4o that CV alone cannot, but it is neither a predictor nor a standalone contribution. Section 4 instead uses CV ranking plus the fs deviation screen as qualitative guidance for the deployment recipes (ROI screen, outstanding-depth tuning, Lmax binding).

L

Auxiliary Generated-Answer MMMU Format-Degradation Analysis

We empirically test the answer-format-degradation hypothesis (Section 3.3) for LLaVA 8B Full FT Sorted, whose generated-answer MMMU score collapses to 5.52±0.18% despite val_loss being only +11.6% above Standard. This appendix explains a parser-sensitive analysis and motivates the parser-free MMMU-MC protocol used in the main tables; it is not a main benchmark result. We sample 120 MMMU validation questions across four subjects (Art, Math, Computer Science, History) and generate raw model outputs from representative LLaVA 8B Full FT checkpoints: Standard (bs=8), Sorted (bs=16; the longest-tail-safe fixed batch after bs=32 OOMs), and ODB at a conservative auxiliary setting (Lmax =4096, pf=1024). These generated-answer checkpoints are used only to study parser sensitivity; the main benchmark cells use MMMU-MC likelihood scoring. Decoding is identical greedy with max 256 new tokens; Table 15 reports response-length and answer-format statistics. Table 15: Raw-output analysis on 120 MMMU samples under the generated-answer protocol. 1letter% is the fraction of stripped responses that are exactly a single A–H letter; verbose% is responses > 20 chars after <think> stripping; extracted-acc% is the subset accuracy under the diagnostic answer-extraction rule. Method Standard (bs=8) Sorted (bs=16) ODB

mean chars

1-letter%

verbose%

extracted-acc%

102.5 110.3 115.6

75.0% 9.2% 76.7%

23.3% 90.8% 21.7%

22.5% 3.3% 20.0%

The 120-sample subset accuracies (Std 22.5%, ODB 20.0%, Sorted 3.3%) track the generatedanswer full-evaluation pattern from the same checkpoints (Std 22.30%, ODB 22.00%, Sorted 5.52%) in rank ordering and gap magnitude. These generated-answer numbers are not used in the main Score columns, which use MMMU-MC likelihood. Format-degradation pattern. Standard and ODB answer in MMMU’s expected single-letter format ∼76% of the time; Sorted answers in single-letter format only 9.2% of the time, with 90.8% of responses being verbose continuations. Inspecting the verbose Sorted outputs, the failure mode is not “the model answers correctly with extra explanation”—instead, the model degenerates into echoing chat-template structure or asking a follow-up question rather than producing the multiplechoice answer. Representative raw outputs (Art subject, ground truth “C”). • Standard / ODB: "C" (single letter, MMMU regex extracts “C” → correct). • Sorted: "user\nWhat is the subject of the painting?" (template echo + follow-up question; no valid A–H answer is extracted, yielding an incorrect prediction). 25

This pattern is consistent across all four subjects (Art, Math, Computer Science, History): verbose Sorted responses on MMMU questions are typically meta-questions or partial assistant utterances rather than usable answer letters. Conclusion. These measurements support the format-degradation hypothesis: Sorted’s val_loss is only +11.6% above Standard because next-token likelihood on natural-language continuations remains plausible, but > 90% of Sorted’s generated MMMU responses are not in the answer format MMMU’s exact-match scoring expects. The generated-answer MMMU drop from 22.30% (Std) to 5.52% (Sorted) therefore primarily reflects output-format drift under parser-extracted scoring, not catastrophic divergence in language modeling. ODB’s length-grouped (rather than lengthsorted) formulation does not exhibit this drift (76.7% single-letter rate, on par with Standard 75.0%) in this auxiliary setting. The main evaluated throughput–quality comparison in Section 3.3 therefore uses parser-free MMMU-MC likelihood rather than parser-extracted generated answers; this appendix explains why the generated-answer analysis is not the benchmark result.

M

LoRA Results

Table 16: LoRA Results. Same comparison under LoRA fine-tuning (rank=8, target=all). 8B and 2B LoRA: 3-seed mean±std on the same 8×H20 setup as Table 1. Bold marks the highest emittedsample throughput among online/no-cache rows; offline-oracle throughput cells are unbolded comparators. Score and Val Loss are reported as reference quality checks without bolding. 8B (LoRA) Speedup

Score

2B (LoRA)

Method

UltraChat CV=0.48 MMLU

Standard 7.87 ± 0.00 Sorted 10.15 ± 0.01 Packing 12.57 ± 0.01 GMT-oraclec 13.48 ± 0.02 c BMT-oracle 12.27 ± 0.02 HFG-oraclec 10.79 ± 0.03 ODB 12.45 ± 0.03

1.00× 1.29× 1.60× 1.71× 1.56× 1.37× 1.58×

76.12 ± 0.06% 0.858 ± 0.001 25.16 ± 0.07 76.25 ± 0.01% 0.877 ± 0.001 32.27 ± 0.04 76.49 ± 0.04% 1.122 ± 0.000 42.14 ± 0.06 76.27 ± 0.04% 0.867 ± 0.001 45.49 ± 0.06 76.36 ± 0.03% 0.875 ± 0.001 40.25 ± 0.10 76.12 ± 0.13% 0.858 ± 0.001 29.81 ± 0.09 76.34 ± 0.07% 0.888 ± 0.001 41.63 ± 0.11

1.00× 1.28× 1.68× 1.81× 1.60× 1.18× 1.65×

59.90 ± 0.05% 1.021 ± 0.003b 60.04 ± 0.03% 1.043 ± 0.003 60.61 ± 0.05% 1.068 ± 0.003 59.99 ± 0.05% 1.034 ± 0.003 60.25 ± 0.04% 1.061 ± 0.003 60.05 ± 0.08% 1.030 ± 0.003 60.10 ± 0.10% 1.045 ± 0.003

Standard 18.95 ± 0.04 Sorted 24.91 ± 0.03d LLaVA GMT-oraclec 32.34 ± 0.08 CV=0.29 c MMMU-MC BMT-oraclec 29.99 ± 0.02 HFG-oracle 26.67 ± 0.05 ODB 30.98 ± 0.07

1.00× 1.31× 1.71× 1.58× 1.41× 1.63×

55.76 ± 0.48% 1.098 ± 0.001 53.98 ± 0.05 56.39 ± 0.41% 1.221 ± 0.002 74.04 ± 0.30 56.82 ± 0.43% 1.135 ± 0.001 94.77 ± 0.53 56.63 ± 0.25% 1.121 ± 0.001 90.86 ± 0.34 57.17 ± 0.31% 1.124 ± 0.001 79.86 ± 0.08 56.47 ± 0.00% 1.151 ± 0.001 94.60 ± 1.01

1.00× 1.37× 1.76× 1.68× 1.48× 1.75×

45.02 ± 0.65% 1.299 ± 0.003 44.82 ± 0.12%e 1.339 ± 0.003e 43.84 ± 0.38% 1.376 ± 0.004 44.04 ± 0.38% 1.374 ± 0.004 43.84 ± 0.07% 1.331 ± 0.003 44.63 ± 0.18% 1.301 ± 0.003

2.83 ± 0.01 2.92 ± 0.00 2.98 ± 0.02 2.98 ± 0.02 3.40 ± 0.01 6.81 ± 0.04

1.00× 1.03× 1.05× 1.05× 1.20× 2.41×

54.24 ± 0.00% 1.217 ± 0.006 7.08 ± 0.02 54.55 ± 0.41% 1.235 ± 0.005 7.30 ± 0.03 54.55 ± 0.27% 1.218 ± 0.006 7.44 ± 0.03 54.35 ± 0.20% 1.218 ± 0.006 7.46 ± 0.03 54.19 ± 0.27% 1.217 ± 0.006 8.28 ± 0.03 54.98 ± 0.18% 1.235 ± 0.006 17.75 ± 0.23

1.00× 1.03× 1.05× 1.05× 1.17× 2.51×

41.77 ± 0.20% 1.346 ± 0.005b 42.47 ± 0.12% 1.357 ± 0.005 41.92 ± 0.24% 1.347 ± 0.005 41.76 ± 0.20% 1.347 ± 0.005 41.53 ± 0.31% 1.346 ± 0.005 44.16 ± 0.44% 1.382 ± 0.004

Standard Sorted ShareGPT4o GMT-oraclec CV=1.00 c MMMU-MC BMT-oraclec HFG-oracle ODB

sam/s

Val Lossa

Dataset

sam/s

Speedup

Score

Val Loss

a

8B LoRA throughput, benchmark, and Val Loss use 3-seed LoRA means. Packing uses the packed-sequence denominator as in Table 1, footnote d . b 2B LoRA UltraChat/ShareGPT4o throughput, downstream benchmark, and Val Loss are 3-seed means; Val Loss uses the same checkpoints as the throughput/benchmark aggregation. c GMT-/BMT-/HFG-oracle rows use the scalar oracle length cache for batch construction; speedups normalize to the matching Standard row in this table. d LLaVA 8B LoRA Sorted: full 3-seed mean; I/O sensitivity is discussed in Section 3.6. MMMU-MC uses choice-likelihood scoring on the 850 letter-labeled validation rows. ODB remains in the same Standard-comparable band under LoRA on LLaVA (8B: 56.47% vs 55.76%; 2B: 44.63% vs 45.02%), is nominally above Standard on ShareGPT4o at both scales, and the 8B UltraChat ODB row reports MMLU 76.34 ± 0.07%. e 2B LoRA LLaVA throughput, Score, and Val Loss are aggregated from the 3-seed LoRA evaluation. The main multimodal comparison uses parser-free MMMU-MC and should be read together with Val Loss.

N

Additional Ablations: Buffer Size and Loss Scaling Mode

Buffer size. The grouping buffer determines how many samples the collate worker accumulates before forming groups; larger buffers enable tighter length-based grouping. Table 17 sweeps the buffer on ShareGPT4o (CV=1.00, the most grouping-pressured dataset). Throughput improves sharply up to buffer=500 and remains in the high-throughput range around 1024–2000 on 2B (16.77–17.10 sam/s) and peaks at 1024 on 8B (6.38 sam/s), while padding is at or below 0.6% at buffer≥1024. Lower-CV datasets place weaker demands on the buffer, so we do not ablate them separately. This profiling sweep is used for throughput/shape diagnosis; full quality claims remain in 26

the main and LoRA result tables, and the default buffer=1024 remains a low-padding main-sweep setting. Table 17: Ablation: buffer size on ShareGPT4o profiling windows (Lmax =4096 for 2B, 8192 for 8B; 8×H20, single seed). Speedups normalize to the matching 20-minute Standard profiling baseline. Scale

Buffer

padding%

sam/s

vs Std

Loss

2B 2B 2B 2B 2B 2B 8B 8B 8B 8B 8B 8B

10 50 100 500 1024∗ 2000 10 50 100 500 1024∗ 2000

3.0% 1.6% 1.0% 0.5% 0.4% 0.3% 7.8% 4.5% 2.8% 0.9% 0.6% 0.5%

8.46 12.20 13.73 15.69 16.77 17.10 2.87 4.21 5.03 5.25 6.38 5.98

1.28× 1.84× 2.07× 2.37× 2.53× 2.58× 1.25× 1.83× 2.19× 2.29× 2.78× 2.61×

1.364 1.375 1.350 1.346 1.341 1.366 1.285 1.283 1.283 1.274 1.269 1.268

∗

default configuration.

Loss scaling mode. ODB supports three gradient scaling strategies: (1) Sample-level Lscaled = L · (nlocal /ntotal ) · W , with ntotal piggybacked on the first all_gather; (2) Approximate token-level same form with tlocal /ttotal , post-alignment tokens estimated as tadj ≈ nadj t̄; (3) Exact token-level uses the primary counts when alignment is a no-op and otherwise performs a deterministic second all_gather of one max_groups-length token-count vector per rank to re-broadcast post-alignment counts. All main experiments use mode (3); Table 18 shows that the three modes have similar throughput in short profiling windows, with differences within about 0.2% on 2B and 1.0% on 8B relative to sample-level scaling. Exact token-level scaling remains the conservative default for final runs because it satisfies Eq. 2 bit-precisely. Table 18: Loss scaling ablation (ShareGPT4o profiling windows, 8×H20, single seed).

O

Scale

Scaling mode

Loss

Unscaled loss

sam/s

Speed vs sample

2B 2B 2B 8B 8B 8B

Sample-level Approx token Exact token Sample-level Approx token Exact token

1.342 1.292 1.341 1.269 1.211 1.269

1.317 1.317 1.317 1.283 1.279 1.283

16.70 16.70 16.73 6.39 6.45 6.37

— +0.0% +0.2% — +1.0% -0.3%

DataLoader I/O Sensitivity (Full Sweep)

Table 19: I/O sensitivity: single-seed 20-minute profiling throughput (sam/s) vs. num_workers for Standard and default-join ODB across datasets and scales (8×H20). This diagnostic sweep isolates input-pipeline depth at fixed configurations; main-table cells use the full-epoch protocol in §3. nw

0

1

Standard 2 4

8

0

1

ODB 2

4

8

2B UltraChat 20.6 20.9 20.9 20.9 20.9 36.8 36.5 36.7 36.8 37.0 2B LLaVA 44.7 48.7 48.9 48.6 48.4 80.2 72.7 78.3 80.6 85.1 2B ShareGPT4o 4.9 6.6 6.6 6.6 6.5 16.8 16.1 16.3 16.7 17.0 8B UltraChat 5.7 5.8 5.8 5.8 5.8 9.9 10.2 9.9 9.9 9.9 8B LLaVA 14.0 14.4 14.4 14.4 14.0 25.1 24.2 24.8 25.2 25.6 8B ShareGPT4o 2.1 2.3 2.3 2.3 2.3 6.4 6.4 6.4 6.4 6.4

27

Across the sweep, Standard is mostly flat after one or two workers, and ODB remains faster than Standard at every worker count. The strongest worker sensitivity is the expected nw=0 penalty for fixed-batch Standard on ShareGPT4o and a small ODB dip at nw=1–2 on LLaVA; by nw=4 the ODB rows are within about 5% of their best values. This supports the operational rule: keep nw≥4 as a portable default, start from prefetch_factor=256, and tune per configuration rather than assuming that more workers alone explain ODB’s throughput.

P

Outstanding Depth Clamp Validation

Section 3.5 defines the outstanding depth as D = max(pf × nw, buffer_size). When pf × nw < buffer_size, ODB’s reset logic injects extra indices into the worker queue so that the collate process can assemble a full group—effectively clamping the in-flight sample count to buffer_size. This appendix empirically confirms the clamp behaviour. Fixing buffer_size=1024 and nw=4, we measured pipeline_overlap for pf ∈ {32, 64, 128} on all three datasets and both scales. Every such pf has nominal depth pf × nw ∈ {128, 256, 512}, all strictly below the buffer, so all three points share the same effective D=1024. Table 20 reports the observed variation under this clamp. Table 20: Clamp validation: pipeline_overlap for pf< 256 (nw=4, buffer=1024). The three clamped pf values share the same effective depth; small differences reflect profiling noise/workload variation rather than a change in effective depth. Scale/Dataset

pf=32

pf=64

pf=128

std

2B UltraChat 2B LLaVA 2B ShareGPT4o 8B UltraChat 8B LLaVA 8B ShareGPT4o

0.9967 0.9426 0.9663 0.9992 0.9856 1.0000

0.9967 0.9441 0.9671 0.9993 0.9863 1.0000

0.9960 0.9562 0.9626 0.9993 0.9843 1.0000

0.0004 0.0074 0.0024 0.0000 0.0010 0.0000

Because low-pf points are equivalent under ODB’s clamp, we start the D sweep in Section 3.5 from D=1024 (the smallest non-clamped depth at nw=4, buffer=1024).

Q

Join Mode: Strict Per-Iteration Zero-Discard and Throughput Trade-off

This appendix elaborates Theorem 1: ODB’s default join-mode termination gives a strict periteration ηlogical =0 guarantee (identity-level zero-discard), while non-join termination gives the cumulative-count ηquota =0 guarantee of Theorem 2. We quantify the throughput difference below. Formal proof of Theorem 1. Let M = W · ⌈N/W ⌉ be the total sampler view count and P = M −N the deterministic tail-padding overhead, so each rank starts with quota q = M/W = ⌈N/W ⌉ (DistributedSampler, drop_last=False; App. C.1). Define the per-rank invariant emittedr + outstandingr + remainingr = q, where outstandingr = |Qr | + |Br | and remainingr = |Rr |; this holds at every state transition by Lemma 1 (No-Leak). In join mode, the done_event is set inside an all_gather barrier whose predicate is ∀r : remainingr = 0 ∧ outstandingr = 0 (implemented by the P drain-then-signal join predicate). Therefore at termination emittedr = q for every rank, and r emittedr = W · q = M sampler views. By Lemma 1, each emitted view appears in exactly one batch, so ηlogical = 0 over the sampler-view multiset within a single logical iteration. S The corresponding identity-level statement is r ids(emittedr ) = I (every one of the N dataset identities is emitted at least once); the P surplus emits are deterministic padding views (re-uses of shuffled-prefix identities), not distinct dataset content. Bounded termination (Theorem 3) is preserved because the join-mode all_gather predicate is computed from the same broadcast tensor on all ranks (Lemma 3); the only difference vs. non-join is that the rank that first finishes its quota waits on the same barrier instead of returning −1. Empirical throughput cost. We compare default join and opt-in non-join on representative fullepoch ODB configurations from the main training stack. For each configuration, both modes use the 28

same model, dataset, seed, Lmax , D, launch mode, and training hyperparameters, changing only the termination flag. Table 21 reports literal emitted-sample throughput; the main result tables report benchmark and validation-loss metrics under default join-mode ODB. Table 21: Default join vs. opt-in non-join on representative full-epoch ODB training configurations (3 seeds, identical hyperparameters except termination flag). Throughput cells are mean±std literal emitted-sample sam/s; Join/Non and ∆ are seed-paired ratios averaged before rounding the throughput columns. ∆

Setting

Scale Dataset

Default join Opt-in non-join Join/Non

1n FFT 1n FFT 1n FFT 1n FFT 1n FFT 1n FFT

2B 2B 2B 8B 8B 8B

LLaVA ShareGPT4o UltraChat LLaVA ShareGPT4o UltraChat

82.42±0.53 16.09±0.21 36.91±0.19 24.87±0.09 5.83±0.04 10.23±0.03

82.73±0.36 16.10±0.04 36.90±0.15 24.91±0.06 5.83±0.04 10.23±0.03

0.9963 -0.37% 0.9995 -0.05% 1.0002 +0.02% 0.9982 -0.18% 0.9993 -0.07% 1.0001 +0.01%

1n LoRA 1n LoRA 1n LoRA 1n LoRA 1n LoRA 1n LoRA

2B 2B 2B 8B 8B 8B

LLaVA ShareGPT4o UltraChat LLaVA ShareGPT4o UltraChat

94.60±1.01 17.75±0.23 41.63±0.11 30.98±0.07 6.81±0.04 12.45±0.03

93.84±0.83 17.69±0.26 41.28±0.27 31.00±0.10 6.81±0.03 12.45±0.06

1.0081 1.0033 1.0084 0.9995 1.0000 1.0002

2n FFT 2n FFT 2n FFT 2n FFT 2n FFT 2n FFT

2B 2B 2B 8B 8B 8B

LLaVA 148.80±0.78 ShareGPT4o 31.35±0.37 UltraChat 70.15±1.86 LLaVA 45.72±0.35 ShareGPT4o 10.32±0.46 UltraChat 18.80±0.03

150.52±1.88 31.13±0.25 70.49±1.58 45.75±0.35 10.62±0.06 18.35±1.23

0.9886 -1.14% 1.0070 +0.70% 0.9955 -0.45% 0.9993 -0.07% 0.9708 -2.92% 1.0278 +2.78%

MM-Mix

82.54±3.13

0.9612

2n MM-Mix 2B

79.15±4.16

+0.81% +0.33% +0.84% -0.05% -0.00% +0.02%

-3.88%

Interpretation. Across the 19 workload-level rows, the average join/non-join ratio is 0.9981 (mean ∆ = −0.19%). Single-node rows range from −0.37% to +0.84%; the wider two-node/MMMix range is −3.88% to +2.78%. Thus the drain-before-finish barrier is not a material throughput bottleneck in these main training settings. We therefore use join mode as the default for the reported training rows; non-join remains an opt-in termination choice when cumulative sample-quota closure is needed but the training stack cannot support the join-style drain-before-finish protocol. Deployment guidance. The reported ODB rows in the main result tables use join mode so that identity coverage is part of the main experimental contract. The two-tier guarantee still maps cleanly onto deployment choices: (default join) for workloads where per-iteration sample composition is part of the algorithm contract (e.g. curriculum schedules, RLHF rollouts, bandit-style data selection); (opt-in non-join) only for constrained training-stack integrations that cannot support the drainbefore-finish join protocol while still requiring cumulative sample-quota closure by Theorem 2 (with empirical checks in Cor. 1). The throughput deltas in Table 21 are small (mean workload-level delta −0.19%, range −3.88% to +2.78% across the 19 main ODB cells), while new workloads should re-profile the selected D and Lmax under the chosen termination mode.

29