Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

1

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

arXiv:2610.32215v1 [cs.DC] 23 Oct 2026

Ziyue Luo1 , Jiaxuan Cai1* , Cedric Le Denmat1* , Srijith Nair1* , Fatemeh Nourzad1* , Rohith Krishnan Sudha1* , Qinhang Wu1* , Jifan Zhang2* , Sungjae Lee1 , Zhe Li3 , Peiwen Qiu1 , Siddharth Shah4 , Rishabh Sharma2 , Sundararajan Srinivasan5 , Yinglun Xia1 , Xue Zheng1 , Zidong Liu6 , Bicheng Ying7 , Kaushik Chowdhury5 , Gauri Joshi4 , Yingbin Liang1 , Robert Nowak2 , Srinivasan Parthasarathy1 , Saurav Prakash8 , Balaraman Ravindran8 , Sanjay Shakkottai5 , Ness B. Shroff1 , Haibo Yang3 , Aylin Yener1 , and Jia Liu1

Abstract—The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smaller institutions that hold relevant private data but lack comparable infrastructure. In this paper, we organize our positions on distributed LLM training around the pretraining and fine-tuning paradigms, covering federated learning, split learning, and decentralized approaches, and we assess their readiness for geo-distributed settings with heterogeneous hardware and unreliable connectivity. We then present a unified system architecture built around a cloudhosted parameter server that coordinates heterogeneous clients over outbound gRPC connections, reusing one communication, compression, and fault-tolerance stack for both pretraining and a family of fine-tuning methods. We validate the design with real geo-distributed experiments on GPT-2 Medium and Llama31B pretraining across multiple university sites, showing stable convergence under realistic conditions. Finally, we chart the road ahead, identifying open problems that span frontier model support, elastic fault tolerance, adaptive compression, federated alignment, privacy and trust, incentive design, and system– algorithm co-design, all of which must be addressed to make democratized LLM training a reality.

Scaling laws indicate that model quality keeps improving with more data and more parameters [7], [8], which only sharpens the incentive to train ever larger models. Yet two bottlenecks increasingly restrict who can participate. The first is data availability and governance: much of the world’s useful data is private, and centralizing it raises privacy, bandwidth, and storage costs that many data owners cannot or will not pay. The second is computational resources: a single training run often demands hundreds of high-end GPUs joined by high-throughput interconnects, an infrastructure that lies well beyond the reach of most universities and smaller firms. Distributed and privacy-preserving paradigms such as federated learning [9] and split learning [10] offer a natural starting point for loosening these constraints. Extending LLM training to geo-distributed clusters of heterogeneous GPUs, however, exposes a harder problem: real wide-area networks are slow and unreliable, and the institutions that own the GPUs sit behind firewalls, NATs, and incompatible software stacks that no single operator controls. This tension motivates the central question of this paper.

Index Terms—Distributed machine learning systems, large language models, federated learning, geo-distributed training, parameter server, fault tolerance.

Key Question: Can we make LLM training broadly accessible by distributing it across a network, so that training can happen “anywhere and everywhere”?

I. I NTRODUCTION

To answer this question, we observe that LLM development decomposes into two phases, pretraining and fine-tuning, each with its own objectives and constraints.

Recent progress in Large Language Models (LLMs) [1]– [4] and foundation models [5], [6] has been driven almost entirely by a few large technology companies operating vast, centralized computing centers. This concentration of compute carries three costs. First, it limits transparency, since the most capable models are developed behind closed doors, which constrains independent scientific review and replication. Second, it raises trust and security concerns, because closed and proprietary training pipelines are difficult to audit. Third, it imposes environmental costs, as hyperscale data centers consume enormous amounts of energy and contribute substantially to carbon emissions. 1 The Ohio State University; 2 University of Wisconsin–Madison; 3 Rochester Institute of Technology; 4 Carnegie Mellon University; 5 The University of Texas at Austin; 6 ComboCurve; 7 Google; 8 Indian Institute of Technology Madras. * Co-primary authors with equal contribution. This work is supported in part by NSF Grant 2112471.

Pretraining across Geo-distributed GPU Clusters: Pretraining optimizes a foundation model [1], [2] from scratch, or from a weak initialization, over massive text corpora. We aim to run this phase on underutilized GPUs spread across many institutions. Each site runs a client process that trains locally, for example with Fully Sharded Data Parallelism (FSDP) [11], while coordination across sites takes place over wide-area networks. • Fine-tuning across Geo-distributed GPU Clusters: Finetuning adapts a foundation model to downstream tasks and adds constraints related to data governance, privacy, and site-specific objectives. Here, each institution fine-tunes on its own data while a central service coordinates the updates, with no need for direct peer-to-peer connectivity between sites. •

2

These two phases differ in their learning objectives, but we argue that a single system structure can serve both. That structure is a cloud-hosted parameter server (PS) that holds an authoritative global snapshot, together with a set of heterogeneous clients that perform local updates, periodically push parameter deltas to the PS, and pull refreshed snapshots in return. This abstraction lets us reuse one connectivity model, one fault-tolerance and rejoin path, and one set of communication optimizations for both pretraining and finetuning. The two phases differ not in the machinery but in the demands they place on it. Pretraining is dominated by throughput and stability: the system must maximize token throughput under stragglers and WAN delay while maintaining a steady trajectory over very long runs. Fine-tuning is dominated by governance and personalization: updates come from non-i.i.d. data, must respect institutional boundaries, and are often parameter-efficient (for example, LoRA) to reduce communication and memory cost. Despite these differences, both phases run into the same systems obstacles once training crosses institutional boundaries over the public Internet. We group them into three challenges that any practical cross-institution training system must confront. Challenge 1: Computation Heterogeneity. The first challenge is orchestrating heterogeneous resources, GPUs above all. Even inside a single production cluster, GPU fleets are rarely uniform: new generations arrive quickly, and replacing a cluster wholesale is prohibitively expensive, so operators add newer hardware while keeping older devices in service. The Ohio Supercomputer Center (OSC) [12], for example, runs H100s, A100s, and V100s side by side. Using such mixed fleets well already improves training efficiency [13], and collaboration across institutions only amplifies the problem, since sites differ in GPU architecture, memory capacity, and software environment. We separate this challenge into hardware and software heterogeneity, then summarize its algorithmic consequences. Hardware Heterogeneity. Consider one site with a pair of 32 GB V100 GPUs, a second contributing a single 24 GB RTX 4090, and a third offering a bank of A100s with native BF16 support. Running identical work on every machine turns the slower GPUs into stragglers that stretch the wall-clock training time and leave faster devices idle. Mixing numerical formats such as FP32, FP16, and BF16 can also produce silent errors when gradients are combined, which in turn risks model divergence. Software Heterogeneity. Software stacks diverge as well. Driver versions, CUDA toolkits, and NCCL libraries vary across administrative domains; an optimizer built for CUDA 12.2 may fail on CUDA 11.7, and certain vendor BF16 optimizations are simply absent on older cards. Coordinating upgrades across many universities is slow and error-prone, especially when each cluster serves dozens of unrelated research groups. Algorithmic Implications. Synchronous data-parallel training assumes every step finishes at roughly the same time. Heterogeneous compute breaks that assumption and pushes

the system toward asynchronous aggregation. FSDP, designed for homogeneous clusters, does not readily support different shard sizes per device; in practice we instead vary the perdevice batch size so that faster GPUs do more work, all while preserving the conditions needed for stable convergence. Challenge 2: Latency and Bandwidth Variability. The second challenge is the unpredictable latency and fluctuating bandwidth of a geo-distributed deployment. Variability of WAN links. Wide-area links between distant institutions and cloud infrastructure are variable and prone to disruption. Packet loss and intermittent connectivity break transfers mid-flight, leaving model synchronization incomplete and training iterations inconsistent. The result is delayed convergence, longer runs, and a less effective system overall. Staleness versus Compression. Aggressive compression shrinks the bytes on the wire but can introduce bias that slows or destabilizes convergence. Waiting for every worker, on the other hand, lets the slowest link dictate throughput. Prior work on gradient staleness shows that accuracy degrades quickly once delays exceed a few steps for large transformers, yet WAN jitter alone can introduce delays of several seconds. The system must therefore navigate the space between these two failure modes rather than commit to either. Challenge 3: Heterogeneous Network Configurations. The third challenge is the diversity of institutional network policies, which we group into three forms. Security Middleboxes and Policy Heterogeneity. Many institutional clusters sit behind enterprise firewalls or Network Address Translation (NAT) devices that block inbound connections and restrict outbound ports. Edge servers on 5G or LTE change IP addresses often, and campus Wi-Fi may rewrite or cap long-lived TCP connections. Under such varied policies, a simple peer-to-peer overlay is not realistic. Unreliable and Asymmetric Paths. WAN paths fail for mundane reasons, from fiber cuts to scheduled maintenance, and wireless links fade. Strongly asymmetric bandwidth, where upload and download speeds differ sharply, complicates any protocol that leans on acknowledgments. A reset connection can break a standard gRPC stream and force a costly re-dial that risks dropping in-flight data. Operational Complexity. Contributors should not have to configure port forwarding or stand up VPN tunnels to participate. Truly democratized participation across institutions and edge sites requires a simple “click-to-join” experience. Together, these three challenges define the systems problem of democratized LLM training. Addressing them requires three design principles. First, heterogeneity awareness: the system uses asynchronous aggregation that decouples local training from global synchronization, so that stragglers do not stall faster nodes. Second, asynchronous robustness: the system bounds staleness and uses fault-tolerant communication, on the assumption that WAN links fail by default rather than as the exception. Third, network compatibility: the system relies on a cloud-hosted rendezvous with outbound-only connectivity, which removes the need for peer-to-peer overlays, port

3

forwarding, or VPN setup, and thereby lowers the barrier to entry. Guided by these principles, this paper makes the following contributions. • We identify and characterize the systems challenges that arise when LLM pretraining and fine-tuning move beyond centralized data centers, namely computation heterogeneity, WAN latency and bandwidth variability, and diverse network configurations, and we argue that these challenges remain unsolved at the scale that democratized training requires. • We survey the landscape of distributed LLM training along two axes, training paradigm (pretraining versus finetuning) and coordination style (synchronous, asynchronous, and infrequent-synchronization). We examine how existing methods handle compute, network, and statistical heterogeneity, and where they fall short once deployed in geodistributed, WAN-constrained settings with diverse institutional networks. • We propose a unified two-level system architecture in which a cloud-hosted parameter server, talking to heterogeneous clients only over outbound gRPC connections, supplies shared building blocks (periodic delta synchronization, quantization-based compression, and fault-tolerant rejoin) that a federated pretraining paradigm and a federated finetuning paradigm reuse without modification. Preliminary results on GPT-2 Medium and Llama3-1B pretraining show stable convergence and competitive wall-clock efficiency under realistic WAN conditions. • We describe a set of open problems for the community. These span scaling to frontier and multimodal models, fault tolerance under elastic membership, adaptive communication compression, federated alignment, privacy and trust at LLM scale, incentive design for sustained participation, and system-algorithm co-design for dynamic WAN environments. To facilitate reproducibility and future research, we make the implementation of our unified distributed LLM training framework publicly available at https://github.com/ai-edge-i nstitute/geo-distributed-llm-training. The rest of the paper is organized as follows. Section II surveys distributed learning and existing systems for distributed LLM pretraining and fine-tuning. Section III presents our unified architecture, describes the shared infrastructure and the two paradigms it supports, and reports preliminary experiments. Section IV discusses open problems and the research agenda ahead. II. BACKGROUND : D ISTRIBUTED LLM T RAINING In this section, we review existing methods for distributed LLM training, grouped into two broad categories: pretraining and fine-tuning. The underlying distributed optimization techniques, such as federated learning and split learning, are shared across both categories, but the optimization process and the computation and communication requirements differ markedly between them. The training category also shapes

specific system choices, including the use of compression, whether the model is pipelined or split, and the kind of network messages exchanged. A. Preliminaries a) Distributed learning vs. federated learning: what changes when the boundary is institutional.: Distributed learning trains a shared model across multiple workers, traditionally in data-center settings with centralized coordination, fast intra-cluster communication, and increasingly sophisticated sharding and parallelism stacks for large models [11], [14]–[18]. Federated learning (FL), by contrast, is defined by organizational boundaries: data remain local to multiple owners, communication is constrained, and training proceeds through exchanged model updates rather than centralized data collection [9], [19]. As a result, FL must treat participation, availability, heterogeneity, and system-level issues such as scheduling, fault tolerance, and secure aggregation as firstclass concerns [20], [21], making the optimization method inseparable from the communication protocol that governs it. b) Parameter servers: a foundational abstraction for many-to-one coordination.: A natural abstraction for largescale training is the parameter server (PS): a logically centralized service maintains model parameters, while workers pull parameters, compute updates, and push them back for aggregation. Early systems such as DistBelief already followed this many-to-one pattern in practice [14], and later work formalized PS as a general framework for distributed learning with flexible consistency models, elastic scaling, and fault tolerance [15]. PS-style coordination is especially attractive when direct peer-to-peer communication is difficult or undesirable. Because workers communicate primarily with a coordinator rather than a dense all-to-all group, this abstraction fits dynamic membership, heterogeneous workers, and settings with asymmetric or policy-constrained connectivity. It also provides a natural control point for enforcing synchronous, asynchronous, or bounded-staleness update rules [15], [22]. Although modern data-center training often emphasizes collective communication, systems such as BytePS show that PSstyle and collective designs can be viewed within a unified design space [23]. For WAN-scale and cross-domain learning, the PS abstraction remains especially appealing because it aligns with restricted connectivity, variable worker availability, and explicit server-side control over synchronization. c) Synchronous, asynchronous, and infrequentsynchronization training: consistency models as the hidden design choice.: A central axis in both distributed learning and FL is the synchronization model, since it determines both when workers communicate and which version of the model they optimize against. In synchronous training, workers compute in parallel and then coordinate a global update at a barrier, as in data-parallel SGD or one FedAvg communication round [9]. This yields clean semantics because all workers update from the same parameter snapshot, but wall-clock progress is limited by the slowest worker and communication path.

4

In asynchronous training, workers instead push updates whenever they finish local computation. The idea predates modern deep learning: early work on asynchronous iterative methods and distributed optimization established convergence under delayed information [24]–[26]. Later, H OGWILD ! showed that lock-free asynchronous stochastic updates can work well in suitable regimes [27], while systems such as DistBelief and the parameter-server line brought asynchronous and partially synchronous execution to large-scale learning [14], [15]. The main benefit is higher utilization under stragglers and heterogeneity; the main cost is staleness, since an update may be computed from parameters that are already outdated when applied. This tradeoff motivates bounded-staleness models that lie between strict synchronization and unconstrained asynchrony. Stale Synchronous Parallel (SSP) is the canonical example: workers may proceed asynchronously, but only up to a bounded lag, making staleness an explicit system knob [22]. More recent analyses continue to study asynchronous SGD under heterogeneous computation and communication patterns [28]. In practice, similar ideas also appear through buffered aggregation, time-based synchronization, and staleupdate filtering. The same spectrum appears in FL. Standard FL is usually organized into synchronous rounds, where clients train locally and then send updates to a coordinator; FedAvg is the canonical example and already reflects the idea of amortizing communication through multiple local steps [9]. Asynchronous FL removes this round barrier and allows clients to contribute whenever they finish local work. Asynchronous Federated Optimization explicitly incorporates stale client updates into FL [29], while buffered asynchronous aggregation provides a practical middle ground [30]. A closely related design choice is infrequent synchronization: instead of communicating often, workers perform more local computation between exchanges. This includes local SGD / periodic averaging, with FedAvg as the FL counterpart [9], [31]. The systems advantage is lower coordination cost; the optimization risk is greater drift across replicas, especially under non-IID data, heterogeneous local work, or delayed communication. Recent large-model training systems revisit this idea explicitly in low-communication regimes, with DiLoCo as a representative example [32]. Overall, synchronous execution, asynchronous execution, bounded staleness, and infrequent synchronization are best viewed as points along a broader consistency–communication tradeoff : tighter synchronization reduces staleness but amplifies waiting, while looser or less frequent synchronization improves utilization at the cost of noisier and more drift-prone updates. d) Heterogeneity is the default: compute, network, and statistical heterogeneity.: Heterogeneity is a defining feature of federated and WAN-scale learning. It appears at least in three forms: compute heterogeneity, where clients differ in hardware, memory, workload, and availability; network heterogeneity, where bandwidth, latency, jitter, and reliability vary across links; and statistical heterogeneity, where clients hold non-IID data with different sample sizes or label distributions. These forms are tightly coupled: slower devices and weaker

links create stragglers and stale updates, while non-IID data amplifies the optimization error caused by delayed or uneven participation. One line of work addresses systems heterogeneity. FedProx is a canonical example because it explicitly models FL under both systems and statistical heterogeneity, allowing variable local work while stabilizing training with a proximal term [21]. TiFL mitigates stragglers through tier-based client grouping [33], while Oort improves time-to-accuracy by selecting clients that are both system-efficient and statistically useful [34]. FedScale complements these methods by exposing such effects through realistic large-scale benchmarking [35]. A second line of work targets statistical heterogeneity, especially client drift under non-IID data. FedProx partially addresses this through regularization [21]; SCAFFOLD uses control variates to correct drift [36]; FedNova addresses objective inconsistency caused by unequal local work [37]; and MIME reduces the gap between federated and centralized optimization under heterogeneous data [38]. For WAN-scale cross-silo training, the key lesson is that these heterogeneities should not be treated in isolation. Systems heterogeneity affects who participates, network heterogeneity affects how delayed updates become, and statistical heterogeneity determines how harmful those delays are after aggregation. The relevant question is therefore not simply how to “handle heterogeneity,” but which form of heterogeneity a method addresses, through what mechanism, and under what communication and consistency assumptions. B. Distributed LLM Pretraining Pretraining is a supervised learning method that trains the parameters of an LLM on large text corpora [8], [39], [40]. It can be interpreted as a pattern recognition problem, where the model learns to generate text in a given language, sequentially via auto-regressive generation [41]. Mathematically, suppose the LLM is defined as f θ (·), parametrized by θ, that takes as input the length (k − 1) sequence of tokens {xi }k−1 i=1 and outputs the next token xk for some integer k ≥ 2. At the next step, xk+1 is generated by xk+1 = f θ ({xi }ki=1 ), and so on. Given a large text corpus consisting of P passages of p Kp lengths {Kp }P p=1 each {xi }i=1 , the pretraining problem can be formulated as:   Kp P X X 1  1  (1) ℓCE (f θ ({xpi }k−1 min L(θ) := i=1 ), xk ) θ P p=1 Kp k=2

where ℓCE (·, ·) is the cross-entropy loss function. a) Key Challenges with Centralized Pretraining: While pretraining is a straightforward application of supervised learning from an optimization perspective, the main challenge with LLM pretraining is that the training data and the model parameter sizes are unprecedentedly large when compared to the ML models and datasets used just a decade ago [8], [40], [42], [43]. In fact, the current state-of-the-art (SOTA) LLMs simply cannot be pretrained on single hardware accelerators (e.g., GPU, TPU or HPU) or a single stand-alone machine, and instead require several hours (often, days) of distributed

5

training across powerful data centers with hundreds of highperformance devices. Besides the optimization process, an often ignored overhead in the centralized pretraining pipeline, is data collection and curation [44]. For these reasons, centralized LLM pretraining has driven a sharp rise in energy expenditure and is affordable only to the largest technology firms. Almost all SOTA LLMs are therefore proprietary and maintained by a handful of corporations, which leaves LLM pretraining research dominated by industry and public academic work well behind. Centralized training faces a further limit: scaling laws [7] imply that the supply of publicly available data will be insufficient to meet LLM training demands by 2028 [45]. b) Distributed Systems: These concerns together call for a collaborative effort to build genuinely distributed and decentralized systems for training LLMs at scale. Towards this end, academics and startups [32], [46]–[50], and large corporations [11], [32], [51] have proposed to leverage existing distributed ML systems as a means to facilitate a more democratic approach to LLM training. Federated Learning (FL) [9], [21], [36], [52] and Decentralized FL (DFL) [53]– [56] democratize LLM pretraining on edge devices by forming a loose federation of participating devices, which in turn train the LLM locally, in parallel, while aggregating models trained by others, either via a centralized parameter server, or in a decentralized manner. DiLoCo [32] is the closest to the system we propose here. Its two defining choices are the use of Nesterov momentum as the outer optimizer and time-based synchronization in place of the step-based synchronization common in conventional FL. We refer the reader to the surveys [57], [58] for comprehensive coverage of FL and DFL. There are several key benefits to using such collaborative and decentralized training paradigms: • From a systems perspective, FL and DFL avoid the cost of collecting and centralizing data, since they operate on data held locally by design. •

By avoiding data sharing, these methods are intrinsically privacy-preserving, enabling training on the vast pool of private and sensitive data without compromising privacy [59].

•

Many FL algorithms enjoy linear speedup [60], meaning that their theoretical convergence rate improves as more devices, or clients, participate.

Combined with Split Learning (SL) [10], [61], smaller clients with much lesser compute and memory, can still participate in the training process where they train only the initial few layers of the model, leaving the bulk of processing to the server. c) Open-source Systems: Several projects such as Ray Anyscale [62], Amazon SageMaker [63], Google Vertex AI [64], NVIDIA NeMo [65], and several others, offer commercial plans or solutions for distributed LLM training, but are proprietary and run on private and paid clusters. Recently, various open-source systems have been designed with support for LLM pretraining and fine-tuning, such as PrimeIntellect [66]. Flower [48], [49], is perhaps the largest such open-source endeavor, with a large community of contributors, but many parallel projects have also been initiated in recent years like •

[50], [51], [67]. Some other projects offer decentralized FL systems [68], [69], and with support for model pipelining and split learning [70]. Most of these systems address a single concern, such as federated or distributed training, or pipelining. What is increasingly needed, however, is a system that adapts dynamically to system and data heterogeneity and to the geographical layout of the federation. An ideal system would automatically split or pipeline models that exceed a device’s capacity, while performing system-aware and network-aware load balancing and straggler mitigation. C. Distributed LLM Fine-tuning Fine-tuning is typically the second stage in a two-stage LLM training pipeline, where the pretrained model is additionally trained, using supervised learning or reinforcement learning (RL), on a downstream task. Based on the set of parameters updated, fine-tuning is often classified into 1) full fine-tuning, where all the LLM parameters are trainable; and, 2) parameter efficient fine-tuning (PEFT) [71]–[75], where either a fraction of the parameters or a lower-dimensional representation of the LLM parameters are trainable while the others remain frozen. Mathematically, the supervised finetuning method can be formulated as (1) with the key difference being the reparametrization of θ using a lower dimensional representation, ϕ:   Kp P min L θ(ϕ), {{e xpk }k=1 }p=1 (2) ϕ

where L is the same objective function as in (1), Kp P {{e xpk }k=1 }p=1 represents the fine-tuning dataset, ϕ is typically much smaller in size than θ, and θ(ϕ) is the corresponding parameter set in the original parameter space. The key characteristic of PEFT is that the gradients computed using backpropagation are typically of much smaller size, thus requiring lesser memory. Low Rank Adaptation (LoRA) [71] is, by far, the most heavily researched PEFT method. Introduced in a centralized learning setup, the approach has also gained a lot of popularity in FL [76], [77]. The main idea is to replace weight matrices in a model W i ∈ Rm×n with low-rank updates ∆W i : W i + ∆W i = W i + B i Ai m×r

(3)

where, the lifting matrices B i ∈ R and the projection matrices Ai ∈ Rr×n for all layers i form the trainable set of parameters, ϕ. The LoRA rank r is typically much smaller than m and n. a) FL for LoRA fine-tuning: Fine-tuning under the FL framework has gained substantial traction lately, largely because training large models on edge or IoT devices with scarce communication and computation budgets is increasingly relevant [78]–[80]. Distributed pretraining and finetuning share several properties, including local and private data and the absence of a central authority. Fine-tuning in constrained networks, however, calls for additional systemlevel support such as model splitting [10], [61] or pipeline parallelism [81], [82], since many edge devices cannot store or process an entire LLM and must instead spread it across the

6

Client 1 (Master Client)

2

Parameter Servers on Public Cloud

Initial parameter upload Model Parameters

5

Client relaunch

In this section, we describe how the vision of Section I is realized as a practical, fault-tolerant training system. The system has two levels. The unified building blocks (Section III-B) provide the shared infrastructure for communication, synchronization, compression, and fault tolerance. The two application paradigms, pretraining and fine-tuning (Section III-C), build on this infrastructure to meet their respective objectives. Throughout the section, we also present preliminary evaluation results that validate the design under realistic geo-distributed conditions.

A. Design Goals Our system design is guided by three main goals to address the challenges of distributed LLM training: a) G1: Use heterogeneous GPUs without stragglers: The system must keep all participating clusters actively working at the same time, even if they have different computing power (FLOPS), memory capacity, or support for mixed-precision calculations. This goal directly addresses Challenge 1 (computation heterogeneity). Section I b) G2: Handle WAN latency and bandwidth variations: Model synchronization cannot assume fast, sub-millisecond network connections. Instead, it must function effectively with links where speeds can change by a large margin. This goal directly addresses Challenge 2 (latency and bandwidth variability). c) G3: Survive firewall/NAT asymmetry and preemptions: Clients should only need to make outbound connections. New or re-launched nodes must be able to rejoin the training quickly with a single remote procedure call (RPC) to the parameter server (PS). This goal directly addresses Challenge 3 (diverse network configurations).

B. Level 1: Unified System Building Blocks We first describe the shared infrastructure that is common to both pretraining and fine-tuning paradigms. This comprises the architecture overview, the communication protocol, the network compression scheme, and the fault-tolerance mechanism.

6

Latest parameter retrieval

4

Client 4 Client 3 Client 2

Pe sy par riod nc am ica hr on ete l iza r tio n

Model initialization

Ini tia dis l pa t r i r am bu tio eter n

1

Failed Client

III. A U NIFIED S YSTEM FOR G EO - DISTRIBUTED LLM T RAINING

Gradient Optimizer

Periodical 4 parameter synchronization

3

network. Robustness to system, data, and model heterogeneity, along with tolerance for random client dropouts, are further considerations when designing an FL fine-tuning system for such settings. b) RL-based fine-tuning: Fine-tuning methods requiring multi-turn generation are often formulated as reinforcement learning (RL) problems, like alignment to human values [83]– [87], reasoning for mathematics and coding [88], [89], longhorizon and tool-calling agentic systems using MCP [90], retrieval-augmented generation (RAG) [91], etc. Such systems often require distributed RL across multiple devices [92] for the same reasons as pretraining and fine-tuning.

Restored Client

Fig. 1: Cross-institution unified training system. A master client initializes the model and uploads it to a cloud parameter server (PS), which maintains the global model and optimizer state. Worker clients periodically synchronize parameters with the PS during training and can recover the latest model after failures. Circled numbers correspond to the protocol steps described in Section III-B5.

Fig. 2: Training with pipeline parallelism. Each stage is trained using data parallelism with AllReduce within each cluster.

1) Architecture Overview: Figure 1 illustrates our crossinstitution unified training system. A cloud-hosted parameter server (PS) holds the authoritative global model parameters and a gradient optimizer. A designated master client (Client 1) bootstraps the model, and any number of additional clients located at campus or edge sites perform local model updates. a) Design evolution: from pipeline parallelism to PS: Our initial architectural vision involved “pipeline parallelism,” where different levels of the LLM would reside on separate clusters as depicted in Figure 2. This design was attractive because it allowed scaling very large models beyond the memory capacity of a single cluster and promised reduced communication overhead compared to full data parallelism. However, this pipeline design inherently necessitates direct peer-to-peer (P2P) connections between participating clusters. In practice, the firewalls and network address translation (NAT) policies commonly enforced in institutional networks make direct P2P difficult to implement reliably. This led us to evolve our architecture toward the centralized PS model in Figure 1, which provides a more robust and manageable communication infrastructure. b) Parameter server (PS): We set up our PS as a lightweight gRPC service on Amazon Web Services (AWS). This PS receives parameter differences (deltas) from the clients, applies federated averaging (FedAvg) [9] to combine

7

these updates, and sends out new global model snapshots. All communication, including control messages and model tensors, uses outbound connections from clients to the PS, avoiding the need for complex inbound firewall configurations at client sites. The PS aggregates tensor updates in its memory, efficiently streaming data to minimize computing and memory overhead. Why a centralized PS? Beyond sidestepping the P2P difficulties above, the centralized relay brings two concrete benefits. It reduces the connectivity structure from O(N 2 ) peer links to O(N ) client-to-server links, and it provides a single place to manage TLS encryption and access control for the whole federation. c) Multi-PS distributed learning: A single central PS can itself become a coordination bottleneck. We therefore offer a multi-PS mode as a configurable option: clients form clusters, typically by geographical proximity, and each cluster is coordinated by its own PS. Training within a cluster proceeds as in Algorithm 1, with the local PS sharing the global model, the clients training on their local data, and the PS aggregating periodically. The PSs then synchronize with one another via ring all-reduce. This setup balances load and mitigates the latency imposed by the geographical layout of the federation, directly addressing goal G2 and partly addressing goal G1. d) Client: Each client starts an FSDP [11] worker. The size of its micro-batches and its mixed-precision mode are automatically adjusted to match the client’s local memory and computing power (FLOPS). Clients store their optimizer states locally. Only the differences (deltas) from the last global model snapshot are sent over the WAN. e) Synchronization cadence: Instead of synchronizing after every training step, each client sends updates after a configurable wall-clock interval (e.g., 10 minutes). This limits how “stale” the model updates can become while also controlling the load on the WAN. 2) Communication Protocol: Our initial implementation used PyTorch RPC for cross-institution communication. While PyTorch RPC is highly efficient within a single, tightly controlled cluster, its design mandates a mesh-like topology for distributed processes. This typically requires dynamic allocation of numerous ports, potentially leading to O(N 2 ) connections for N clients. This design conflicted directly with the strict firewall policies common in institutional networks, which often block dynamic port ranges or require explicit, static rules for inbound connections. In response, we transitioned to the gRPC protocol. gRPC allowed us to explicitly manage ports and precisely define packet formats using Protocol Buffers. A key advantage of gRPC is its ability to operate over a single, fixed, userspecified port via persistent, outbound client connections, thereby circumventing inbound firewall restrictions at client sites. We leveraged gRPC’s bidirectional streaming RPCs (e.g., for S TREAM M ODEL and P USH D ELTA calls) to enable continuous exchange of model updates without constantly re-establishing connections. This strategic change effectively addressed the practical and technical constraints we faced, enabling reliable and secure communication for our distributed training framework.

3) Network Compression: To reduce network traffic and latency, we compress the gradient payload (addressing Challenge 2). Compression proceeds in two stages: quantization followed by Huffman encoding. Quantization maps each 32bit floating-point gradient to an 8-bit or 16-bit integer, for compression factors of 4 and 2, respectively. We use minmax quantization for its low overhead, since it requires storing only the minimum and maximum of the gradient distribution. Huffman encoding then compresses the quantized gradients further. The receiver reverses the pipeline, decoding and then dequantizing to reconstruct the payload. We observed that maintaining min-max quantization pairs for each multi-dimensional tensor resulted in high storage overhead. Our solution was to flatten all multi-dimensional tensors into a single one-dimensional tensor and then perform quantization. Treating each tensor separately for Huffman encoding would not give enough gradient values to obtain a skewed distribution, and flattening the quantized tensors solved this issue. 4) Fault Tolerance and Rejoin Logic: Interruptions, such as a dropped gRPC stream, signal a preemption. When a client relaunches, it authenticates with a one-time token, then fetches the latest global model (Θ) and its last saved optimizer state from the PS. Because this path retrieves the live model directly, a client typically resumes within roughly 10 seconds. The mechanism preserves convergence guarantees while hiding the failure from the rest of the federation. For durability beyond a single relaunch, we also checkpoint model and optimizer state to disk as a single file. Checkpointing operates at two levels: a checkpoint is written after every synchronization to guarantee fault tolerance, and a checkpoint manager separately retains the ten checkpoints with the lowest evaluation loss, refreshed every 1000 steps. Recovering from a cold disk checkpoint, rather than from the live PS model, takes about one minute. All checkpoints are stored locally on the client. If the PS itself fails, we relaunch the federation and Client 1 loads its saved checkpoint and streams it to the PS to resume training. 5) Protocol Walk-through: We illustrate the step-by-step flow of our system using the numbered arrows in Figure 1. Model initialization. A designated master client • 1 (Client 1) initializes the starting model (Θ0 ) from a pretrained checkpoint or random weights and prepares its local FSDP shards. • 2 Initial parameter upload. Client 1 streams Θ0 to the PS using a single P USH D ELTA call. The PS stores Θ0 in host memory alongside its gradient optimizer state. • 3 Initial parameter distribution. The PS distributes Θ0 to all other clients through their open S TREAM M ODEL pipes. This ensures every client starts from an identical model state. • 4 Periodic parameter synchronization. Each client trains on its local data, then after the configured wallclock interval, (i) computes the difference (∆) between its local model (θ) and the global model (Θ), (ii) quantizes ∆ based on the observed network link rate, and (iii) sends the quantized ∆ to the PS. The PS de-quantizes the received delta, applies FedAvg, and streams the updated

8

Level 1: Unified System Building Blocks

Level 2: Application Paradigms

Parameter Servers Pretraining Paradigm gRPC Comm. Protocol

Network Compression Finetuning Paradigm Fault Tolerance

Fig. 3: Two-level architecture of our unified system. The unified building blocks (Level 1) provide shared infrastructure for communication, synchronization, compression, and fault tolerance. Both pretraining and fine-tuning (Level 2) build on the same infrastructure, configuring it for their respective objectives.

global model (Θ) back to the client. The local runtime adjusts the micro-batch size and mixed-precision settings to fully utilize each client’s GPU without risking out-ofmemory errors. • 5 Client relaunch. If a client is preempted or crashes, it relaunches and re-authenticates with the PS using a one-time token. • 6 Latest parameter retrieval. The restored client fetches the latest global model (Θ) from the PS, loads its last saved optimizer states, and resumes training as described in Section III-B4. C. Level 2: Application Paradigms The unified building blocks in Figure 3 support two learning paradigms that share one infrastructure but differ in objective. This section explains how each paradigm configures that infrastructure, presents a high-level algorithm, and discusses paradigm-specific considerations. We first give a system-level meta-algorithm that serves both pretraining and fine-tuning, then describe the two paradigm implementations in turn. 1) A meta-algorithm for distributed LLM training: Algorithm 1 states a general meta-algorithm for training LLMs in a distributed manner. The same underlying system drives distributed pretraining as well as fine-tuning with LoRA (FedIT [93]), Hybrid LoRA, RAVAN [94], and zeroth-order optimization [95]. The meta-algorithm follows conventional FedAvg [9]: the parameter server (PS) initializes the global LLM and reparametrizes it into a combination of frozen and trainable parameters. In LoRA [71], for instance, each weight matrix is reparametrized as in (3). The same formulation covers pretraining trivially by setting ϕ(·) = θ(·) = I(·), the identity operator. The PS distributes only the trainable parameters to the clients. Each client freezes the underlying LLM, trains the trainable parameters on mini-batches sampled from its local

Algorithm 1 General meta-algorithm for Distributed Training of LLMs 1: Input: LLM with trainable params θ 0 , local steps H, sync timeout T , global learning rate η PS 2: Parameter server (PS) initializes θ ← θ 0 PS 3: Reparametrize and extract trainable parameters from θ : PS PS ϕ := ϕ(θ ) 4: repeat 5: PS distributes ϕPS to clients. 6: for each client c in parallel do 7: Initialize ϕc ← ϕPS 8: Locally update ϕc until H steps or timeout T , whichever comes first 9: Compute and send deltas δ c := ϕc − ϕPS to PS 10: If interrupted: Request ϕPS from PS and restart local training 11: end for 12: PS receivesPand aggregates all client models ϕPS = N ϕPS + η. N1 i=1 δ c 13: Restore and checkpoint global model θ PS = θ(ϕPS ) 14: until terminated PS 15: Output: Global model θ

dataset, and returns the trainable deltas δ c to the PS. The PS aggregates these deltas and updates the global model with a global learning rate. This process repeats until a convergence criterion is met. 2) Paradigm 1: Federated Pretraining: Our pretraining system is directly derived from the meta-algorithm in Algorithm 1, by replacing the reparametrization in lines 3 and 13 with the identity transformation, i.e., all parameters are communicated and trained. Within each client, we leverage gradient accumulation and FSDP [11] to distribute training across multiple GPUs, allowing each worker to process multiple micro-batches and aggregate them into a single minibatch before performing a parameter update. This allows each client to select a micro-batch size that fits its hardware constraints, avoiding out-of-memory errors while maintaining a consistent effective mini-batch size across heterogeneous clients (addressing Challenge 1). Before transmission, gradients are quantized to BF16 or INT8 depending on the observed network link speed (addressing Challenge 2). If a failure occurs during communication, the system detects the error and initiates immediate retransmission (addressing Challenge 3). Additionally, to prevent stragglers, i.e., weaker clients with low compute speeds, from slowing down the overall process, we also introduce a timeout T at the PS. If some clients time out before performing H optimization steps locally, they share their current set of parameters to the PS. Figure 4 shows the geographical distribution of the GPU servers used for pretraining alongside a real-time Grafana dashboard that tracks training progress together with network latency and compression statistics. a) Preliminary results on GPT-2 Medium pretraining: We evaluate convergence behavior when pretraining GPT-2 Medium [2] on OpenWebText using two university sites as workers: OSU (one worker, 2× NVIDIA H100) and UW–

9

Total Data Received

Total Data Sent

Latest RTT

60 GB

UW–Madison Madison, WI

OSU

Columbus, OH

4 min

40 GB

40 GB 20 GB

20 GB

0

0

16:00

17:00

Worker 0 Last: 25.7 GB

18:00

19:00

Worker 1 Last: 30.4 GB

Data Received Throughput

17:00

18:00

19:00

Worker 1 Last: 21.2 GB

40 MB/s 16:00

Worker 0

17:00

0

16:00

17:00

Worker 0 · latest Last: 2.76 min

18:00

19:00

Worker 1 · latest Last: 2.67 min

Data Sent Throughput

50 MB/s 0

2 min

16:00

Worker 0 Last: 25.9 GB

18:00

0

19:00

Worker 1

Worker 0 · avg

16:00

Worker 0

17:00

Worker 0 Overhead %

Worker 0 Payload Size

Worker 0 Quantization time

0.500

47.6

0.0000476

2357

Worker 1 Compression Ratio

Worker 1 Overhead %

Worker 1 Payload Size

0.500

0.0000476

2357

18:00

19:00

Worker 0 · latest

Worker 1 · avg

Worker 1 · latest

2.98 min 2.76 min 2.84 min 2.67 min

Worker 1

Worker 0 Compression Ratio

Training Loss Across Clients 12 10

GPU Utilization

Contribution (Latest)

100%

Worker 1 Quantization time

8

39.7

6

Contribution (Cumulative)

4 2

100

50%

961

16:30

Worker 0

Worker 1

Worker 0

Worker 1

Worker 0

Current Memory

UT Austin Austin, TX

(a) Geo-locations

17:00

18:00

19:00

Worker 1 Last: 3.27 Min: 2.98

Validation Loss Across Clients

18:00 Worker 1

16:00 Worker 0 Last: 3.29 Min: 3.02

1832

195 0%

6 4

20 GB

2

10 GB 16:00

17:00

Worker 0 Last: 18.2 GB Max: 18.2 GB

18:00 Worker 1 Last: 14.7 GB Max: 14.7 GB

19:00

0

16:00 Worker 0 Last: 3.40 Min: 0

17:00

18:00

19:00

Worker 1 Last: 3.40 Min: 0

(b) Grafana-based Dashboard

Fig. 4: A real geo-distributed deployment of our system, pretraining GPT-2 Medium across three nodes. Panel (a) shows the physical locations of the GPU servers across the United States, and panel (b) shows the live training metrics tracked during the run.

Madison (two workers, each with 2× NVIDIA RTX 3090), coordinated through an AWS EC2 t2.xlarge instance. Model updates are synchronized every 10 minutes. We compare with three representative baselines: (1) FedAvg [9]; (2) DiLoCo [32], which reduces synchronization frequency while leveraging Nesterov momentum [96]; (3) FedNova [97], which normalizes local updates to account for heterogeneous worker speeds. FedAvg and DiLoCo follow a synchronous training paradigm, where all clients are required to complete the same number of local steps before aggregation, making the overall process vulnerable to stragglers, whereas FedNova allows asynchronous updates across clients, similar to our design. Figure 5 shows the training loss trajectories. Our design converges stably and reduces the loss faster in wall-clock time than the baselines. DiLoCo, in contrast, diverges quickly under its recommended hyperparameters. This illustrates a more general point: algorithms tuned for high-bandwidth, homogeneous data-center environments often perform poorly in geo-distributed deployments, where the latency variance and compute heterogeneity are much higher. These results indicate that a design must account explicitly for heterogeneous, geodistributed conditions. b) Preliminary results on cross-site communication latency: We characterize cross-site communication latency to quantify the practical challenges of geo-distributed training. Figure 6 plots the gradient receive latency measured between our AWS instances and the university sites. We observe two patterns. First, the network conditions fluctuate substantially over time, due to transient congestion and routing changes. Second, latency and bandwidth differ across sites, reflecting differences in local network infrastructure and institutional policies. Together, these effects produce both high average latency and high variance, which create stragglers and destabilize synchronization. They also motivate two of our design choices: a centralized relay that simplifies connectivity under firewall constraints, and communication compression that reduces the payload size and thus the WAN overhead. c) Preliminary results on Llama3-1B pretraining: We evaluate our system on Llama3-1B [1] to demonstrate that the design scales to more demanding workloads. We use OSU (one worker, 2× H100) and UT Austin (one worker, 4× H100),

coordinated through two AWS EC2 r5.2xlarge instances, using our multi-PS setup in Section III-B1c. Model updates are synchronized every 20 minutes with gradient quantization from FP32 to UINT8. Figure 7 plots the training loss trajectory. Despite the larger model and the heavier communication load, the loss decreases steadily and without divergence. This shows that the system remains stable for larger models under realistic WAN conditions. 3) Paradigm 2: Federated Fine-tuning: We implement and evaluate five distributed fine-tuning algorithms, each offering different trade-offs in communication efficiency, computation efficiency and privacy. At the system level, the first four methods use the same system backbone as in Algorithm 1, with different reparametrizations in steps 3 and 13. The last method is a Federated Split Learning (FSL) algorithm, which implements specific system changes and diverges from the basic framework introduced in Algorithm 1. For performance evaluation, we fine-tune the GPT2-medium model for the finetuning task of question answering, using the Stanford Question Answering Dataset (SQuAD) [98]. As an example, Figure 8 shows the geographical locations of the GPUs used for LoRA fine-tuning, together with a real-time Grafana dashboard tracking training, validation, and network metrics. a) Federated LoRA-based fine-tuning: In this algorithm, we implement a version of FedIT [93]. Instead of updating the full model, we update and exchange only the low-rank adapter matrices A and B associated with LoRA [71] (c.f. (3)). This method can be realized by using ϕ = {(Al , B l )}L l=1 for L layers in the model, as the reparametrization in step 3 in Alg. 1. In Algorithm 1, at each federated round, the server aggregates and distributes only A and B in step 3. Note that the naive aggregation step in FedIT is known to be inconsistent. However, our system is easily extendable to other variants of FL fine-tuning, such as [79]. To strengthen privacy in federated LoRA fine-tuning, we also implement Hybrid LoRA, which separates global from local updates. The shared reparametrization ϕ is the identity, exactly as in pretraining, so that the base parameters θ are the only quantities exchanged with the PS. Each client additionally maintains private LoRA adapters A and B that never leave the client. During a round, the client updates the base parameters for Hbase steps and the private adapters for Hlora steps, then returns only the base parameters for aggregation. The private adapters let each site personalize without exposing its adaptation to the federation. b) Multi-headed LoRA Adapters (RAVAN): We can extend our system backbone to also implement a variant of LoRA known as RAVAN [94]. In RAVAN, the weights are reparametrized by: ! h X W + ∆W = W + A H i B, (4) i=1

where H i are trainable heads, and W , A, and B are frozen and only shared once in the beginning. The reparametrization ϕ = {H i }i extends Algorithm 1 to RAVAN. c) Zeroth-order federated fine-tuning: Zeroth-order optimization eliminates the need to communicate gradients back

10

3.1

4

3.0 800

1000

1200

3 0

300

600

900

12

0 UW-Madison

200

2 min

100

~1.9 h ~58 min

5

OSU

10

Training Loss

FedNova FedAvg DiLoCo Ours

Comm. Latency (s)

Training Loss

6

10

Ours (Llama3-1B) 2× auto-recovery

8 6 4 2

0 06:00

1200

10:00

Time (HH:MM)

Wall-Clock Time (min)

14:00

0

500

1000 1500 2000 2500

Wall-Clock Time (min)

Fig. 5: GPT-2 Medium training loss. Fig. 6: Gradient receive latency across Fig. 7: Llama3-1B pretraining loss. geo-distributed sites.

Total Data Received

Total Data Sent

10 GiB

10 GiB

5 GiB

5 GiB

0

0

GPU Utilization

Contribution (Latest)

100%

UW–Madison Madison, WI

OSU

Columbus, OH

23:40 00:00 00:20 00:40 01:00

IIT Last: 13.1 GiB UT Last: 13.1 GiB

OSU Last: 12.6 GiB UW Last: 13.1 GiB

0% IIT UW

23:40 00:00 00:20 00:40 01:00

IIT Last: 12.5 GiB UT Last: 12.5 GiB

OSU Last: 13.0 GiB UW Last: 12.5 GiB

00:00

Contribution (Cumulative)

01:00

OSU

UT

OSU 32% UW 32%

UT 32% IIT 4%

OSU 33% UW 33%

UT 33% IIT 2%

Validation Loss Across Clients 8

Latest RTT

6 4

1.42 min

10.5 s

17.3 s

37.2 s

IIT

OSU

UT

UW

2 23:40

00:00

OSU Last: 1.45 Min: 1.35

00:20

00:40

UT Last: 1.45 Min: 1.34

01:00

UW Last: 1.46 Min: 1.35

Training Loss Across Clients

Latest Idle Duration

8 4

1.34 min

1.95 min

IIT

OSU

Average RTT

1.89 min

2.24 s

UT

UW

Average Idle Duration 11.1 s

Chennai, India

1.46 min

UT Austin Austin, TX

(a) Geo-locations

1.80 min

IIT 1.46 min OSU 11.1 s

UW 37.9 s

UT 15.2 s

23:40

00:00

IIT Last: 1.00 Min: 0.355 UW Last: 0.605 Min: 0.393

1.95 min 37.9 s

IIT Madras

2 1 0.5 0.25

1.10 min

15.2 s

OSU 1.95 min IIT 1.10 min

UT 1.80 min UW 2.20 s

apply the system to LoRA fine-tuning of GPT-2 Medium for question answering on SQuAD [98]. To our knowledge, this is the first geo-distributed framework for scalable federated split learning.

00:20

00:40

OSU Last: 0.846 Min: 0.395

01:00

UT Last: 1.29 Min: 0.405

Evaluation: Exact Match

Evaluation: F1 Score

40

50

IV. T HE ROAD A HEAD

20 0

23:40 OSU

00:00 UT

00:20 UW

00:40

01:00

0

23:40 OSU

00:00 UT

00:20

00:40

01:00

UW

(b) Grafana-based Dashboard

Fig. 8: A real geo-distributed deployment of our system, Hybrid LoRA fine-tuning of GPT-2 Medium across three nodes. Panels are read as in Figure 4.

and forth from the server, by using random directional gradients to approximate true gradients. Consider the reparametrization ϕ = (s, g), where s, a random number generator (RNG) seed, and g a scalar directional gradient. The idea is to simply transmit the RNG seed from the PS to clients. The clients then update the parameters using a directional derivative gc along a random direction, obtained using the shared seed s. The clients share gc with the server which aggregates it, and then applies the update along the chosen random direction. This drastically minimizes the overall communication costs from O(d) for d parameters, to O(1). We refer the interested reader to [95] for details on zeroth-order FL fine-tuning. d) Federated split learning (FSL-SAGE): Conventional FL assumes that every client can store and process the full model, an assumption that breaks down for very large models on resource-constrained clients. Federated Split Learning (FSL) [10], [61] addresses this by partitioning the model between the client and the server. To cut both the communication load and the number of synchronization cycles, we adopt FSLSAGE [99], a provably convergent, communication-efficient FSL algorithm built around auxiliary models. FSL-SAGE uses two server processes: an F -server orchestrates federated aggregation of the client-side models, and an S-server trains the server-side model and periodically realigns each client’s auxiliary model. Because auxiliary-model alignment runs far less often than the per-iteration synchronization that vanilla SL requires, FSL-SAGE substantially lowers communication cost and lets the clients run in parallel. We deploy the F server and S-server on separate AWS instances, with clients on single-GPU servers at OSU, UMich, UT, and IITM, and

In the previous sections, we described a unified system that runs both pretraining and fine-tuning across geo-distributed, heterogeneous GPU clusters, and we showed that it converges stably under realistic WAN conditions. The system is, however, only a first step. Several questions remain before such an approach can train frontier-scale models, absorb the churn of an open federation, and reward participants in a way that sustains their contribution over long runs. In this section, we describe these open problems. We organize them from the models one would like to train, to the systems and algorithms needed to support them, and finally to the economic structures that determine whether a federation can be sustained at all. A. Scaling to Frontier and Multimodal Models Our experiments consider models up to 1B parameters. Scaling to frontier sizes, on the order of 70B parameters and beyond, will require new strategies rather than larger versions of the present ones. One promising direction is Mixture-ofExperts (MoE) architectures [42], [43], whose sparse activation reduces the per-client compute and memory footprint. In a geo-distributed setting, however, the placement and routing of experts across WAN-connected sites becomes a difficult systems problem in its own right, since a naive cross-site partition can inflate communication, unbalance load, and destabilize routing. A second direction is to extend the system to multimodal foundation models such as LLaVA [100]. Such models appear compatible with our framework, but they introduce additional heterogeneity in their modality-specific encoders, in data availability, and in communication cost that the current design does not yet address. B. Fault Tolerance under Elastic Membership Fault tolerance in large-scale training has received considerable attention. Bamboo [101] tolerates frequent spot-instance preemptions by filling pipeline bubbles with redundant layer computation; Oobleck [102] pre-computes a set of heterogeneous pipeline templates so that survivors can quickly recover

11

a working configuration; and Gemini [103] reduces recovery latency by more than 13× using multi-tiered in-memory checkpointing. These systems assume managed clusters with reliable, high-bandwidth interconnects and homogeneous hardware. These assumptions do not hold at the open edge. In our setting, disruption is the common case rather than the exception. Clients leave and rejoin between rounds, WAN links fluctuate without warning, and new participants arrive during training with unfamiliar hardware. Supporting such elastic membership requires reassigning work as clients come and go, moving model state over high-latency links, and bringing new joiners up to the current state without stalling the rest of the federation. We believe that lightweight, WAN-aware failure detection and state reconciliation, designed for open and heterogeneous membership rather than for closed clusters, are the main pieces still missing. C. Adaptive Communication Compression Our system already compresses traffic using quantization and Huffman coding (Section III-B3). Communication will only become more important as models grow. Error-feedback mechanisms [36] and adaptive quantization [104] can reduce traffic further. Interestingly, the “Square-Cube Law” observed in SWARM parallelism [105] shows that in some regimes larger models are in fact more communication-efficient, because computation grows faster than communication volume. The open question is how to select the compression strategy adaptively, choosing bit-widths, sparsification ratios, and synchronization frequencies from real-time network measurements. We expect that doing so will require co-designing the compression scheme with the outer optimizer [32], so that the quantization schedule and the synchronization cadence are chosen together rather than separately. D. Federated Alignment and Reinforcement Learning Our system currently supports supervised fine-tuning (Section III-C3). A natural next step is to support federated alignment, since modern alignment pipelines rely heavily on reinforcement learning from human feedback (RLHF) [83]. RLHF does not fit easily into a federation: the reward model must be trained across institutions whose preference data is private and non-IID, a setting in which centralized pipelines do not directly apply. Preference-based objectives such as Direct Preference Optimization (DPO) [84] may be better suited here, since they avoid explicit reward modeling and therefore reduce both the number of shared components and the volume of communication. Even so, preference data remains heterogeneous and institution-specific, and orchestrating such diverse preferences privately, without averaging away their diversity, is an open problem. A further difficulty is distributed rollout generation. RLHF repeatedly samples responses from the policy during training, and recent work identifies this generation step as the dominant systems bottleneck, which has motivated disaggregated execution [106]. Performing rollouts across heterogeneous, WAN-connected sites, while keeping the sampled trajectories fresh enough for stable policy optimization, will require an

execution model quite different from the co-located RLHF systems used today. E. Privacy, Security, and Trust Federated learning is often motivated by privacy. Nevertheless, the shared updates can still leak information about local data. Incorporating formal guarantees such as differential privacy (DP) [59], [107] is therefore an important next step. The utility-privacy tradeoff at LLM scale is not yet well understood, since the noise level that a small model tolerates can substantially degrade a billion-parameter one. Privacy is not the only concern. In an open federation whose participants are not all trusted, Byzantine robustness, the ability to defend against clients that submit corrupted updates, becomes essential [108]. Whether robust aggregation rules such as the coordinate-wise median or the trimmed mean retain their guarantees at LLM scale is an open question that our setting makes pressing. F. Incentive Design and Participation Economics A democratized training ecosystem works only if participants have a reason to contribute compute, data, and bandwidth. Existing schemes range from compute-centric rewards based on GPU hours or raw FLOPs [109] to Shapley-value methods [110] that estimate each participant’s marginal contribution to model quality. Both overlook the temporal and staleness effects that dominate asynchronous, geo-distributed training. A contribution that arrives during a critical convergence phase is worth far more than the same compute delivered after the loss has plateaued, yet neither metric distinguishes the two. Shapley scores are also unstable across aggregation strategies [111], and any deployable rule must account for strategic behavior by participants. Without incentives that reflect timing, data quality, and the possibility of gaming, a federation is vulnerable to free-riding and adverse selection. Designing incentive rules that are temporally aware, robust to manipulation, and inexpensive enough to enforce in real time is an open problem at the intersection of distributed systems and mechanism design. G. System-Algorithm Co-Design Building this system showed us that the algorithmic and systems choices cannot be made in isolation. The synchronization cadence, the compression strategy, and the fault-tolerance policy all interact with the optimizer, the learning-rate schedule, and the aggregation rule. Lengthening the synchronization interval, for example, saves communication but increases staleness, which may in turn call for a different outer optimizer, as DiLoCo [32] demonstrates with Nesterov momentum. We therefore envision an auto-tuning layer that monitors network conditions, compute heterogeneity, and convergence diagnostics, and adjusts the system parameters (compression ratio, synchronization frequency, timeout thresholds) together with the algorithmic hyperparameters (learning rate, local steps H, timeout T , aggregation weights). Such co-design would let the system adapt to the changing conditions of a geo-distributed deployment on its own, and would reduce the operational burden on participating institutions.

12

H. Concluding Remarks In this work, we have demonstrated that LLM training need not remain confined to a handful of centralized data centers. The unified architecture presented in this paper supports both pretraining and fine-tuning over a single cloudhosted parameter server and a set of heterogeneous clients, communicating only through outbound connections, and our preliminary experiments show that it converges under realistic WAN conditions. The same design tolerates client failures and accommodates GPUs of different generations across several institutions. At the same time, the open problems described above show that much work remains. The barrier to participating in frontier model development is no longer only a matter of raw compute. Increasingly, it is a question of whether the systems, algorithms, and incentives exist to pool the compute that already sits idle across the world’s universities, laboratories, and edge sites. We believe such pooling is achievable, but it will require sustained effort across systems, machine learning, security, and economics, together with shared open platforms and benchmarks so that progress can be measured and reproduced. We hope this paper, and the system and open problems it describes, is a call to arms for the community to take up this effort, so that the most capable models, and the science of building them, do not remain the preserve of a few well-resourced institutions. R EFERENCES [1] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2302.13971 [2] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186. [4] OpenAI, J. Achiam et al., “Gpt-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774 [5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763. [6] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022. [Online]. Available: https://arxiv.org/abs/2204.06125 [7] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [8] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “An empirical analysis of compute-optimal large language model training,” Advances in neural information processing systems, vol. 35, pp. 30 016–30 030, 2022. [9] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in AISTATS, 2017. [10] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar, “Split learning for health: Distributed deep learning without sharing raw patient data,” arXiv preprint arXiv:1812.00564, 2018.

[11] Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer et al., “Pytorch fsdp: Experiences on scaling fully sharded data parallel,” arXiv preprint arXiv:2304.11277, 2023. [12] Ohio Supercomputer Center, “Cluster computing,” 2025. [Online]. Available: https://www.osc.edu/resources/technical support/supercomp uters [13] F. Strati, Z. Zhang, G. Manos, I. S. Périz, Q. Hu, T. Chen, B. Buzcu, S. Han, P. Delgado, and A. Klimovic, “Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters,” in Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, 2025, pp. 204–220. [14] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker et al., “Large scale distributed deep networks,” in NeurIPS, 2012. [15] M. Li, D. G. Andersen, J. W. Park, A. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, “Scaling distributed machine learning with the parameter server,” in OSDI, 2014. [16] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “Zero: Memory optimizations toward training trillion parameter models,” arXiv preprint arXiv:1910.02054, 2020. [17] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2020. [18] L. Zheng et al., “Alpa: Automating inter- and intra-operator parallelism for distributed deep learning,” in OSDI, 2022. [19] J. Konečnỳ, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: Distributed machine learning for on-device intelligence,” arXiv preprint arXiv:1610.02527, 2016. [20] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konecný, S. Mazzocchi, H. B. McMahan et al., “Towards federated learning at scale: System design,” in MLSys, 2019. [21] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in MLSys, 2020. [22] Q. Ho, J. Cipar, H. Cui, J. K. Lee, J. K. Kim, P. B. Gibbons, G. Gibson, G. R. Ganger, and E. P. Xing, “More effective distributed ml via a stale synchronous parallel parameter server,” CMU, Tech. Rep., 2013. [23] Y. Jiang and et al., “A unified architecture for accelerating distributed dnn training in heterogeneous gpu/cpu clusters,” in OSDI, 2020. [24] D. Chazan and W. Miranker, “Chaotic relaxation,” Linear Algebra and its Applications, vol. 2, no. 2, pp. 199–222, 1969. [25] J. N. Tsitsiklis, D. P. Bertsekas, and M. Athans, “Distributed asynchronous deterministic and stochastic gradient optimization algorithms,” IEEE Transactions on Automatic Control, vol. 31, no. 9, pp. 803–812, 1986. [26] D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods. Prentice Hall, 1989. [27] B. Recht, C. Ré, S. Wright, and F. Niu, “Hogwild!: A lock-free approach to parallelizing stochastic gradient descent,” in Advances in Neural Information Processing Systems, 2011. [28] A. Koloskova, S. U. Stich, and M. Jaggi, “Sharper convergence guarantees for asynchronous sgd for distributed and federated learning,” Advances in Neural Information Processing Systems, 2022. [29] C. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” arXiv preprint arXiv:1903.03934, 2019. [30] J. Nguyen, K. Malik, H. Zhan, A. Yousefpour, M. Rabbat, M. Malek, and D. Huba, “Federated learning with buffered asynchronous aggregation,” in AISTATS, 2022. [31] S. U. Stich, “Local sgd converges fast and communicates little,” arXiv preprint arXiv:1805.09767, 2018. [32] A. Douillard, Q. Feng, A. A. Rusu, R. Chhaparia, Y. Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen, “Diloco: Distributed low-communication training of language models,” CoRR, vol. abs/2311.08105, 2023. [Online]. Available: https://arxiv.org/abs/ 2311.08105 [33] Z. Chai, A. Ali, S. Zawad, S. Truex, A. Anwar, N. Baracaldo, Y. Zhou, H. Ludwig, F. Yan, and Y. Cheng, “Tifl: A tier-based federated learning system,” in HPDC, 2020. [34] F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury, “Oort: Efficient federated learning via guided participant selection,” in OSDI, 2021. [35] F. Lai, Y. Dai, S. Singapuram, J. Liu, X. Zhu, H. Madhyastha, and M. Chowdhury, “Fedscale: Benchmarking model and system performance of federated learning at scale,” in ICML, 2022. [36] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learn-

13

ing,” in International conference on machine learning. PMLR, 2020, pp. 5132–5143. [37] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” in NeurIPS, 2020. [38] S. P. Karimireddy, S. J. Reddi, S. U. Stich, M. Jaggi, S. Kale, and S. Kumar, “Mime: Mimicking centralized stochastic algorithms in federated learning,” in ICML, 2021. [39] D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y. Wang et al., “Olmo: Accelerating the science of language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 789–15 809. [40] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. [41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [42] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in International Conference on Learning Representations, 2017. [Online]. Available: https: //openreview.net/forum?id=B1ckMDqlg [43] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024. [44] N. Kandpal and C. Raffel, “Position: The most expensive part of an LLM *should* be its training data,” in Forty-second International Conference on Machine Learning Position Paper Track, 2025. [Online]. Available: https://openreview.net/forum?id=L6RpQ1h4Nx [45] P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, and M. Hobbhahn, “Position: will we run out of data? limits of llm scaling based on human-generated data,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024. [46] S. Gugger, L. Debut, T. Wolf, P. Schmid, Z. Mueller, and S. Mangrulkar, “Accelerate: Training and inference at scale made simple, efficient and adaptable.” https://github.com/huggingface/accelerate, 2022. [47] Z. Zhu, C. Giannoula, M. Andoorveedu, Q. Su, K. Mangalam, B. Zheng, and G. Pekhimenko, “Mist: Efficient distributed training of large language models via memory-parallelism co-optimization,” in Proceedings of the Twentieth European Conference on Computer Systems, 2025, pp. 1298–1316. [48] D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y. Gao, L. Sani, H. L. Kwing, T. Parcollet, P. P. d. Gusmão, and N. D. Lane, “Flower: A friendly federated learning research framework,” arXiv preprint arXiv:2007.14390, 2020. [49] L. Sani, A. Iacob, Z. Cao, R. Lee, B. Marino, Y. Gao, D. Cai, Z. Li, W. Zhao, X. Qiu et al., “Photon: Federated llm pre-training,” arXiv preprint arXiv:2411.02908, 2024. [50] R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y. Xu, Y. Du, Y. Wang, and S. Chen, “Openfedllm: Training large language models on decentralized private data via federated learning,” in Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, 2024, pp. 6137–6147. [51] H. R. Roth, Y. Cheng, Y. Wen, I. Yang, Z. Xu, Y.-T. Hsieh, K. Kersten, A. Harouni, C. Zhao, K. Lu, Z. Zhang, W. Li, A. Myronenko, D. Yang, S. Yang, N. Rieke, A. Quraini, C. Chen, D. Xu, N. Ma, P. Dogra, M. Flores, and A. Feng, “NVIDIA FLARE: Federated Learning from Simulation to Real-World,” IEEE Data Eng. Bull., Vol. 46, No. 1, Mar. 2023. [52] S. J. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=LkFG3lB13U5 [53] A. Lalitha, S. Shekhar, T. Javidi, and F. Koushanfar, “Fully decentralized federated learning,” in Third workshop on bayesian deep learning (NeurIPS), vol. 2, 2018, pp. 1–9. [54] S. Savazzi, M. Nicoli, and V. Rampa, “Federated learning with cooperating devices: A consensus approach for massive iot networks,” IEEE Internet of Things Journal, vol. 7, no. 5, pp. 4641–4654, 2020. [55] D. Monschein, J. A. P. Pérez, T. Piotrowski, Z. Nochta, O. P. Waldhorst, and C. Zirpins, “Towards a peer-to-peer federated machine learning environment for continuous authentication,” in 2021 IEEE Symposium on computers and communications (ISCC). IEEE, 2021, pp. 1–6.

[56] V. P. Chellapandi, A. Upadhyay, A. Hashemi, and S. H. Żak, “On the convergence of decentralized federated learning under imperfect information sharing,” IEEE Control Systems Letters, vol. 7, pp. 2982– 2987, 2023. [57] W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang, D. Niyato, and C. Miao, “Federated learning in mobile edge networks: A comprehensive survey,” IEEE communications surveys & tutorials, vol. 22, no. 3, pp. 2031–2063, 2020. [58] L. Yuan, Z. Wang, L. Sun, P. S. Yu, and C. G. Brinton, “Decentralized federated learning: A survey and perspective,” IEEE Internet of Things Journal, vol. 11, no. 21, pp. 34 617–34 638, 2024. [59] K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. Quek, and H. V. Poor, “Federated learning with differential privacy: Algorithms and performance analysis,” IEEE transactions on information forensics and security, vol. 15, pp. 3454–3469, 2020. [60] H. Yang, M. Fang, and J. Liu, “Achieving linear speedup with partial worker participation in non-IID federated learning,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=jDdzh5ul-d [61] C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “Splitfed: When federated learning meets split learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8485– 8493. [62] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan et al., “Ray: A distributed framework for emerging {AI} applications,” in 13th USENIX symposium on operating systems design and implementation (OSDI 18), 2018, pp. 561–577. [63] C. Karakus, R. Huilgol, F. Wu, A. Subramanian, C. Daniel, D. Cavdar, T. Xu, H. Chen, A. Rahnama, and L. Quintela, “Amazon sagemaker model parallelism: A general and flexible framework for large model training,” arXiv preprint arXiv:2111.05972, 2021. [64] J. Bhatia and K. Chaudhary, The Definitive Guide to Google Vertex AI. Packt Publishing, 2023. [65] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al., “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577, 2019. [66] S. Jaghouar, J. M. Ong, M. Basra, F. Obeid, J. Straube, M. Keiblinger, E. Bakouch, L. Atkins, M. Panahi, C. Goddard et al., “Intellect-1 technical report,” arXiv preprint arXiv:2412.01152, 2024. [67] G. A. Reina, A. Gruzdev, P. Foley, O. Perepelkina, M. Sharma, I. Davidyuk, I. Trushkin, M. Radionov, A. Mokrov, D. Agapov et al., “Openfl: An open-source framework for federated learning,” arXiv preprint arXiv:2105.06413, 2021. [68] Z. Cheng, R. Sun, J. Sun, and Y. Guo, “Scaling decentralized learning with flock,” arXiv preprint arXiv:2507.15349, 2025. [69] I. Singh, E. Vissol-Gaudin, A. Otung, and M. Sekiya, “Learning to collaborate: An orchestrated-decentralized framework for peer-to-peer llm federation,” arXiv preprint arXiv:2601.17133, 2026. [70] A. Borzunov, D. Baranchuk, T. Dettmers, M. Riabinin, Y. Belkada, A. Chumachenko, P. Samygin, and C. Raffel, “Petals: Collaborative inference and fine-tuning of large models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2023, pp. 558–568. [Online]. Available: https://arxiv.org/abs/2209.01188 [71] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9 [72] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023. [73] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 4582–4597. [74] E. Ben Zaken, Y. Goldberg, and S. Ravfogel, “BitFit: Simple parameter-efficient fine-tuning for transformer-based masked languagemodels,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1–9. [Online]. Available: https://aclanthology.org/2022.acl-short.1/ [75] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych, “Adapterfusion: Non-destructive task composition for transfer learning,” in

14

Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume, 2021, pp. 487– 503. [76] Y. Wu, C. Tian, J. Li, H. Sun, K. Tam, Z. Zhou, H. Liao, J. Xiong, Z. Guo, L. Li, and C. zhong Xu, “A survey on federated fine-tuning of large language models,” Transactions on Machine Learning Research, 2026, survey Certification. [Online]. Available: https://openreview.net/forum?id=rnCqbuIWnn [77] N. Yan, Y. Su, Y. Deng, and R. Schober, “Federated fine-tuning of llms: Framework comparison and research directions,” IEEE Communications Magazine, vol. 63, no. 10, pp. 52–58, 2025. [78] Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving loRA in privacy-preserving federated learning,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=NLPzL6HWNl [79] Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li, “FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id=TcCorXxNJQ [80] J. Bai, D. Chen, B. Qian, L. Yao, and Y. Li, “Federated fine-tuning of large language models under heterogeneous tasks and client resources,” Advances in Neural Information Processing Systems, vol. 37, pp. 14 457–14 483, 2024. [81] D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 1–15. [Online]. Available: https://doi.org/10.1145/3341301.3359646 [82] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu et al., “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” Advances in neural information processing systems, vol. 32, 2019. [83] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022. [84] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in neural information processing systems, vol. 36, pp. 53 728–53 741, 2023. [85] F. Nourzad, A. Roknilamouki, E. Ekici, J. Liu, and N. Shroff, “Firm: Federated in-client regularized multi-objective alignment for large language models,” 2026. [Online]. Available: https: //arxiv.org/abs/2511.16992 [86] F. Nourzad, D. Sow, Y. Liang, M. Shi, M. Zhang, Y. Li, E. Ekici, and N. Shroff, “Barriers to pareto steerability in preference-conditioned llm alignment,” in ICLR 2026 Workshop on I Can’t Believe It’s Not Better, 2026. [Online]. Available: https://openreview.net/forum?id=GVlh3flsqF [87] ——, “One model, many goals: Meta-learning preference-conditioned alignment for lifelong LLM agents,” in ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving, 2026. [Online]. Available: https://openreview.net/forum?id=2FVShQSae2 [88] J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin, “Large language models for mathematical reasoning: Progresses and challenges,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, 2024, pp. 225–237. [89] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” ACM Transactions on Software Engineering and Methodology, vol. 35, no. 2, pp. 1–72, 2026. [90] X. Hou, Y. Zhao, S. Wang, and H. Wang, “Model context protocol (mcp): Landscape, security threats, and future research directions,” ACM Transactions on Software Engineering and Methodology, 2025. [91] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, H. Wang et al., “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, vol. 2, no. 1, p. 32, 2023. [92] J. Qi, Q. Zhou, L. Lei, and K. Zheng, “Federated reinforcement learning: Techniques, applications, and open challenges,” arXiv preprint arXiv:2108.11887, 2021. [93] J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y. Chen, “Towards building the federatedgpt: Federated instruction tuning,” in ICASSP 2024-2024 IEEE international conference on

acoustics, speech and signal processing (ICASSP). IEEE, 2024, pp. 6915–6919. [94] A. Raje, B. Askin, D. Jhunjhunwala, and G. Joshi, “Ravan: Multi-head low-rank adaptation for federated fine-tuning,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=gyn4n8oC9B [95] Z. Li, B. Ying, Z. Liu, C. Dong, and H. Yang, “Achieving dimension-free communication in federated learning via zerothorder optimization,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https: //openreview.net/forum?id=omrLHFzC37 [96] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in International conference on machine learning. pmlr, 2013, pp. 1139–1147. [97] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” Advances in neural information processing systems, vol. 33, pp. 7611–7623, 2020. [98] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383–2392. [Online]. Available: https://aclanthology.org/D16-1264/ [99] S. Nair, M. Lin, P. Ju, A. Talebi, E. S. Bentley, and J. Liu, “FSLSAGE: Accelerating federated split learning via smashed activation gradient estimation,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/f orum?id=HnwcrtoDd4 [100] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Advances in Neural Information Processing Systems, 2023. [101] J. Thorpe, P. Zhao, J. Eyolfson, Y. Qiao, Z. Jia, M. Zhang, R. Netravali, and G. H. Xu, “Bamboo: Making preemptible instances resilient for affordable training of large DNNs,” in 20th USENIX Symposium on Networked Systems Design and Implementation, 2023. [102] I. Jang, Z. Yang, Z. Zhang, X. Jin, and M. Chowdhury, “Oobleck: Resilient distributed training of large models using pipeline templates,” in ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [103] Z. Wang, Z. Jia, S. Zheng, Z. Zhang, X. Fu, T. S. E. Ng, and Y. You, “Gemini: Fast failure recovery in distributed training with in-memory checkpoints,” in ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [104] A. Ramezani-Kebrya, F. Faghri, I. Markov, V. Aksenov, D. Alistarh, and D. M. Roy, “NUQSGD: Provably communication-efficient data-parallel SGD via nonuniform quantization,” Journal of Machine Learning Research, vol. 22, no. 114, pp. 1–43, 2021. [105] M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov, “SWARM Parallelism: Training Large Models Can Be Surprisingly CommunicationEfficient,” in International Conference on Machine Learning. PMLR, 2023, pp. 29 416–29 440. [106] Y. Xiao, Z. Zhou, F. Mao, W. Wu, S. Zhao, L. Ju, L. Liang, X. Zhang, and J. Zhou, “An adaptive placement and parallelism framework for accelerating RLHF training,” arXiv preprint arXiv:2312.11819, 2024. [107] Z. Xu, Y. Zhang, G. Andrew, C. Choquette, P. Kairouz, B. Mcmahan, J. Rosenstock, and Y. Zhang, “Federated learning of gboard language models with differential privacy,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), 2023, pp. 629–639. [108] X. Cao, M. Fang, J. Liu, and N. Z. Gong, “FLTrust: Byzantine-robust federated learning via trust bootstrapping,” in Network and Distributed System Security Symposium, 2021. [109] S. R. Pandey, L. D. Nguyen, and P. Popovski, “FedToken: Tokenized incentives for data contribution in federated learning,” in Workshop on Federated Learning: Recent Advances and New Challenges (FLNeurIPS), 2022. [110] A. Ghorbani and J. Zou, “Data Shapley: Equitable valuation of data for machine learning,” in International Conference on Machine Learning. PMLR, 2019, pp. 2242–2251. [111] A. Geimer, B. Fiz, and R. State, “On the volatility of Shapleybased contribution metrics in federated learning,” in International Joint Conference on Neural Networks, 2025.

Related documents

• Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead#1447582
• LLM Post-Training: A Deep Dive Into Reasoning Large Language Models#1274281
• You're Hired: Strategic Model Selection for LLM Collaboration#1143417
• Accelerated Dataflow: scaling inference in streaming data analytics with efficiency#1386703
• Accelerated Dataflow: scaling inference in streaming data analytics with efficiency#1375008