Understanding Communication Backends in Cross-Silo Federated Learning Amir Ziashahabi† , Chaoyang He‡ , Salman Avestimehr†
arXiv:2604.10859v1 [cs.DC] 12 Apr 2026
†
Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA, USA ‡ TensorOpera, Inc., Palo Alto, CA, USA [email protected], [email protected], [email protected]
Abstract—Federated learning (FL) has emerged as a practical means for privacy-preserving distributed machine learning. FL’s versatile design makes it suitable for various training settings, from IoT edge devices in cross-device FL to powerful servers in cross-silo FL. A key consequence of this versatility is the high level of diversity found in the networking configuration of FL applications. Coupled with the rising demand for large-scale models such as large language models, well-informed selection and configuration of communication backends become crucial for ensuring optimal performance in FL systems. This work focuses on cross-silo federated learning, presenting in-depth benchmarks of various communication backends, including MPI, gRPC, and PyTorch RPC. In addition, we introduce gRPC+S3, a hybrid backend designed to overcome the limitations of existing approaches, particularly for transmitting large models across geo-distributed deployments, achieving up to 3.8× end-to-end speedup over gRPC. Our benchmarks examine point-to-point and end-to-end performance for a broad range of model sizes running under realistic network conditions. Our findings provide practical insights for selecting and configuring suitable communication backends tailored to the specific federated learning tasks and network configurations. Index Terms—Federated learning, cross-silo, communication backends, MPI, gRPC, wide-area networks (WAN), cloud computing
I. I NTRODUCTION Cross-silo federated learning (FL) is a distributed machine learning paradigm that enables collaboration among multiple organizations or data silos to train machine learning models while preserving data privacy [1]. A critical component in an FL system is its communication backend, which manages the exchange of model updates and control messages among participants. The performance of the communication backend has a profound impact on several key facets of the FL process, including convergence speed, communication overhead, fault tolerance, and resource utilization. As the demand for FL systems grows and their applications expand, there is an increasing need to understand the performance implications and trade-offs associated with different communication backends, particularly in cross-silo FL scenarios, where options are numerous. This paper aims to provide an in-depth overview and performance comparison of various communication backends employed in cross-silo FL systems, including Message Passing Interface (MPI) [2], gRPC [3], and PyTorch RPC [4]. Additionally, in response to the limitations of these communication backends in the
LAN
Geo-Proximal
FL Server FL Client Communication Backend (MPI, gRPC, Pytorch RPC, ...)
Geo-Distributed
Fig. 1: Three common deployment environments for federated learning: (1) LAN: participants in a single building; (2) GeoProximal: participants in close geographical proximity; (3) Geo-Distributed: participants in distant locations. Contextaware selection of communication backends is crucial for optimizing performance in each setting.
context of FL systems, we propose a novel communication backend that combines the capabilities of gRPC and Amazon Simple Storage Service (S3) [5]. This hybrid approach aims to leverage the strengths of both gRPC and Amazon S3 to deliver enhanced performance, reliability, and scalability, particularly in situations where existing communication backends exhibit shortcomings. We study communication backends for cross-silo FL through point-to-point and end-to-end experiments that vary message size and concurrency across three deployment environments: LAN, Geo-Proximal, and Geo-Distributed. In summary, this work offers the following contributions: 1) Introduction of Hybrid gRPC+S3 backend. A design that routes metadata over RPC and model weights via object storage. It is particularly useful for scenarios that involve transmitting large models across a geodistributed network. 2) Benchmarking suite across regimes. A comprehensive benchmark suite that includes a variety of models, metrics, and communication backends tailored specifically for cross-silo FL settings. This enables researchers and practitioners to evaluate the performance of different
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
backends across a diverse set of scenarios. 3) Realistic deployment footing. Experiments are conducted in realistic settings that are representative of cross-silo FL environments. We utilize GPU-powered machines and consider various training environments to ensure our results are applicable and insightful to realworld FL tasks.
Region North California Oregon North Virginia Hong Kong Stockholm Sao Paulo Bahrain
II. BACKGROUND AND M OTIVATION
B. Communication Backends for Distributed ML Communication backends move models, gradients, and control messages. Hence, the backend choice strongly affects runtime, scalability, and robustness. We study three widely used options: MPI [2] (HPC-standard message passing with efficient point-to-point and collectives), gRPC [3] (HTTP/2based RPC with Protobuf and support for streaming, and TLS), and PyTorch RPC [4] (PyTorch framework-integrated RPC supporting sync/async patterns). C. Motivation (1) FL’s diverse training environments and lack of benchmark studies. Cross-silo FL runs over vastly different networks, from fast intra-site fabrics to high-latency wide area networks (WANs), yet systematic evaluations of communication backends across these regimes are scarce. We quantify this variability on two platforms: a LAN with InfiniBand (up to 5 GB/s bandwidth and 3.17 µs latency; TCP fallback about 1 GB/s and 16.8 µs) and AWS EC2 with g4dn.2xlarge instances. Table I reports single- and multi-connection bandwidth and latency between North California and seven regions: intra-region links reach ∼3 GB/s and 0.44 ms, while inter-region links are much slower. The gap between singleand multi-connection throughput widens as latency increases, making connection-level concurrency a first-order factor. This work addresses the benchmarking gap by evaluating backends under LAN, geo-proximal, and geo-distributed conditions with both microbenchmarks and end-to-end training. (2) FL network characteristics and limitations of existing backends. Cross-silo FL deployments require backends that (i) perform under heterogeneous networks with minimal assumptions, (ii) tolerate dynamic participation and failures without global aborts, (iii) scale in both client count and model size while keeping CPU/memory bounded, and (iv) support secure operation over untrusted WAN. Given these needs, we revisit the backends in §II-B for FL. MPI, while efficient, assumes
Bandwidt h (MB/s)
Federated learning (FL) trains a shared model across decentralized clients without revealing raw data [1], [6], [7]. There are two primary categorizations of FL: cross-device (many resource-constrained clients, e.g., phones/IoT) and cross-silo (fewer but well-provisioned participants such as hospitals, banks, or data centers). A server coordinates rounds of training by selecting clients, distributing the current global model, collecting locally computed updates (e.g., via SGD [8]), and aggregating them. This work focuses on cross-silo FL.
592 133 39.4 16.3 11.4 8.27 6.90
Latency (ms)
2946 573 557 513 495 491 444
0.44 11 32.3 83.3 90.9 90.9 111
TABLE I: EC2 bandwidth (single vs. multiconnection) and latency between North California and other regions (g4dn.2xlarge). 400
Mem ory (GB)
A. Federated Learning
Bandwidth (MB/s) Single Multi
20
300 200 100 0 25 15 10 5 0
10
20
30
40
50
# of concurrent dispat ches
Fig. 2: Effect of concurrent dispatch on gRPC: bandwidth (top) and memory (bottom) for North California → Bahrain.
static membership and homogeneous, tightly managed networks. Moreover, SSH/rsh-based process management complicates deployment across independently administered sites, and failure handling often lacks the fault isolation needed when participants can come and go. PyTorch RPC integrates cleanly with the framework but is framework-specific, expects open and stable paths between peers, and offers less control over transport/topology than MPI/gRPC, which can be restrictive under stricter cross-organization security and networking constraints. gRPC is a popular choice for FL applications due to its flexibility and security features. However, it is not optimized for heavy data transfers. For instance, its standard Python implementation multiplexes all traffic over a single HTTP/2 connection. Where multi-connection throughput greatly exceeds single-connection (Table I), a single channel underutilizes the link. Using many channels can recover throughput for large broadcasts yet raises server memory roughly linearly with concurrency because each send buffers its own copy (Fig. 2). Motivated by these constraints, especially for large message transfers across heterogeneous, untrusted networks, we propose gRPC+S3, a communication backend optimized for federated learning scenarios. III. gRPC+S3: A H YBRID C OMMUNICATION BACKEND FOR F EDERATED L EARNING To address the limitations of common communication backends in the context of FL systems, we propose a novel communication backend named gRPC+S3. This approach combines the flexibility and security benefits of gRPC with the highly
Serialized model Serialized model
gRPC+S3 Sender
gRPC Server Receive Queue
gRPC Client Send Queue Metadata + S3 object key
Serialized model
gRPC+S3 Receiver Storage Manager gRPC+S3 Receiver
gRPC Server
Storage Manager S3 object key
gRPC+S3 Receiver
S3 Bucket
Sender Message Handler FL message (model+metadata)
Abstract Communication Layer FL Application
Storage Manager serialized model
Metadata + S3 object keyServer gRPC
Receive Queue
Protobuf message
MetadataReceive + Receiver S3 objectQueue key
Storage Manager serialized S3 model Object key Serialized
Message Handler modelS3 Object key Metadata + S3 Receiver object key Reconstructed FL message Message Handler S3 object key Reconstructed Receiver Message Layer Handler FL message Abstract Communication Reconstructed
message AbstractFLCommunication Layer FL Application Abstract Communication Layer FL Application
FL Application
Fig. 3: Architecture of the proposed gRPC+S3 backend.
scalable and reliable Amazon S3 storage to provide an efficient and FL-friendly communication method. A. Architecture Each FL message can be split into a small metadata record (round, type, sender ID, object key) and a large parameter payload (the model). Transfer using gRPC+S3 is done in two phases. Sender side: (1) upon a send, the Sender Message Handler separates metadata from the model; (2) if the model is new, it is serialized and uploaded to an S3 bucket via the Storage Manager, which returns the S3 object key; repeated sends of the same model reuse a cached key; (3) the handler constructs a compact Protobuf containing metadata and the object key and enqueues it for the gRPC client, which delivers it to the receiver’s gRPC server. Receiver side: (1) the gRPC server enqueues incoming records; (2) the Receiver Message Handler extracts the object key and fetches the serialized model via the Storage Manager; (3) the model is deserialized and recombined with metadata to reconstruct the original FL message. This split keeps RPC messages small and shifts the heavy data path to S3, where receivers can pull in parallel using independent connections. B. Advantages Efficiency. Large parameters bypass the single-connection constraint typical of a Python gRPC channel and leverage S3’s multi-connection downloads; gRPC carries only lightweight control, reducing per-transfer CPU and copying on the sender. Scalability. Broadcasts become single-upload/multidownload: the server uploads once, and all clients fetch concurrently. The server’s peak memory during broadcast is independent of the number of receivers because it does not need to buffer N concurrent copies. Versatility. Deployments can fall back to pure gRPC when the payload is small, the environment lacks object storage, or S3 round-trips would dominate. Reliability and fault tolerance. gRPC provides robust request handling and retries for metadata. S3 offers durable object storage and independent client retrieval, so late or failed receivers can re-fetch without forcing the sender to re-transmit.
Privacy and security. Both legs support encryption in transit (TLS for gRPC; HTTPS for S3). Access to model objects can be controlled via scoped credentials or time-limited, pre-signed URLs. IV. B ENCHMARKING M ETHODOLOGY We analyze communication backend performance across common cross-silo FL scenarios. We report peer-to-peer and end-to-end results over diverse models and deployment environments. Below we summarize the environments and workloads used in our benchmarks. A. Environments We evaluate three deployment regimes that frequently arise in cross-silo FL. Local Area Network (LAN). This setting represents a highbandwidth, low-latency environment typical of a single data center. Our testbed consists of two machines, each with 8 NVIDIA Quadro RTX 5000 GPUs, connected via a 5 GB/s InfiniBand link. Geo-Proximal. Participants are in nearby sites (e.g., same metro/region) with moderate latency and high available bandwidth.We emulate it using Amazon EC2 g4dn.2xlarge instances in different availability zones within the North California region. This setup provides high-speed connectivity, with measured bandwidth of 592 MB/s for single-threaded and 2946 MB/s for multi-threaded communication. Geo-Distributed. This simulates collaboration across continents over a high-latency WAN. We use Amazon EC2 g4dn.2xlarge instances, placing the server in North California and clients in seven other regions: North California, Oregon, North Virginia, Hong Kong, Stockholm, Sao Paulo, and Bahrain. Please see Table I for the detailed network characteristics of this environment. B. Models and Datasets We evaluate both image and text-based tasks to cover common FL workloads, using CNNs and Transformers across four payload-size tiers. Image tasks use GLD-23K [9]; text tasks use 20 Newsgroups [10]. The tiers are: Small (ResNet56 [11], 591,322 parameters, 2.39 MB), Medium (MobileNetV3 [12], 5,152,518 parameters, 19.85 MB), Big (DistilBERT [13], 66,362,880 parameters, 253.19 MB), and Large (ViT-Large [14], 307,432,234 parameters, 1,243.14 MB). These tiers are representative of FL payloads in practice. Even when targeting Large Language Models (LLMs) with billions of parameters, practical deployments typically reduce communication with parameter-efficient fine-tuning techniques (e.g., LoRA) [15], [16], so the transferred state is closer to these tiers than to full model checkpoints. C. Peer-to-Peer Benchmarks We measure backends in isolation to examine how message size and environment shape transport behavior. Metrics include latency (CPU-to-CPU), impact of concurrent sends, and memory utilization. Backends are gRPC, PyTorch
0
10
20
30
40
# of concurrent messages
50
3 2 1 0
1.5 1.0 0.5 0.0
Speed-up
Speed-up
0.44 0.08 0.09 0.16 0.07 1.12 0.27 0.09 0.24 0.11 6.72 4.79 0.18 1.74 0.19 42.25 20.18 0.81 6.16 1.36 1.09 0.46 0.24 2.06 1.23 3.53 4.78 0.77 3.24 2.21 10.16 41.01 4.75 14.04 7.38 38.38 151.18 16.74 43.18 17.09
0.13 0.02 0.07 0.01
4.20 0.21 1.30 0.11
SmallMedium Big Large Model Size (c)
0.31 0.03 0.12 0.02
SmallMedium Big Large Model Size
Model Size: Small
1.5 1.0 0.5 0.0
MPI_MEM_BUFF
(b)
Env: LAN
16.96 0.75 5.31 0.49
10 1
MPI_GENERIC
Env: Geo-Proximal
0.75 0.28 0.18 0.88 0.46 2.30 2.13 0.26 1.80 0.47 7.54 21.61 1.34 11.01 4.55 33.36 91.94 7.06 21.11 10.74
101
PyTorch RPC
Env: Geo-Distributed (CA-HK)
10 1 103
Method
gRPC
Env: Geo-Distributed (CA-VA)
101
Memory (GB)
Time (s, log)
Time (s, log)
Env: LAN
103
gRPC+S3
(a)
6 4 2 0
Model Size: Medium
Env: Geo-Distributed (CA-VA)
Env: Geo-Distributed (CA-HK)
Small Medium Big Large Model Size
Small Medium Big Large Model Size
Model Size: Big
20 10 0
10
20
30
40
# of concurrent messages
50
0
0
Env: Geo-Proximal
10
20
30
40
# of concurrent messages
50
Model Size: Large
100 50 0
0
10
20
30
40
# of concurrent messages
50
Fig. 4: Peer-to-peer results across backends, environments, and model sizes. (a) CPU-to-CPU latency (log scale). (b) Speedup of concurrent over sequential transmission for 10 messages (Large uses 5). (c) Peak sender memory during concurrent broadcast.
RPC, gRPC+S3, and two MPI variants: MPI_GENERIC, which serializes and transforms arbitrary Python objects, and MPI_MEM_BUFF, which uses pre-allocated buffers and achieves near C-level speeds but can only communicate bufferlike objects. To capture the variability of Geo-Distributed deployments, we measure two scenarios: (1) intra-continent communication between North California and North Virginia (CA–VA) and (2) inter-continent communication between North California and Hong Kong (CA–HK). gRPC+S3 is excluded from LAN experiments, since its reliance on external object storage would make S3 access latency the dominant factor, masking backend performance. D. End-to-End Benchmarks For realistic behavior, we take the best configuration per backend, integrate it into FedML [17], and run full FL training for each model tier. We record wall-clock time for communication, CPU–GPU migration, serialization, and waiting on both clients and the server. Client measurements also include training time, while the server adds aggregation time. Waiting corresponds to idle or blocked states, and serialization applies only to gRPC and gRPC+S3. We report client times averaged across all clients. As above, gRPC+S3 is omitted on LAN. E. Implementation and Configuration Details PyTorch RPC. We use the Distributed RPC Framework [4] with TensorPipe. CUDA RPC is enabled (GPU-to-GPU transfers via device maps) for suitable end-to-end experiments. To make it compatible with multi-region AWS configuration, nodes were placed in VPCs with pairwise peering. gRPC / gRPC+S3. gRPC uses grpcio [3]. Unary and Streaming modes performed similarly in peer-to-peer tests,
so results apply to either. gRPC+S3 uses Streaming RPC for metadata/control and boto3 for S3. MPI. We use CUDA-aware Open MPI [18] over UCX [19] using mpi4py [20] interface. MPI_GENERIC uses lowercase calls (e.g., send) to serialize arbitrary Python objects. MPI_MEM_BUFF uses uppercase calls (e.g., Send) to transfer buffer-like objects. V. P EER - TO -P EER E XPERIMENT R ESULTS As shown in Figure 4(a), the optimal communication backend is highly dependent on the network environment. In low-latency LAN and Geo-Proximal settings, backends that avoid serialization overhead, such as MPI_MEM_BUFF and PyTorch RPC, consistently deliver the best performance. The impact of serialization is significant; for instance, in the LAN environment, serialization accounts for up to 86% of gRPC’s total latency and is the primary reason for MPI_GENERIC’s inferior performance compared to MPI_MEM_BUFF. As we move to high-latency GeoDistributed environments, proficiency in utilizing multiple connections becomes the dominant factor for performance. Here, PyTorch RPC consistently outperforms other backends across most model sizes. In contrast, gRPC’s lack of a built-in mechanism for multi-connection transfers causes its performance to degrade substantially with larger models in this setting. As illustrated in Figure 4(b), concurrent message transmission, a common scenario for an FL server distributing a global model, generally enhances performance. This is particularly true in Geo-Distributed settings where it mitigates the effect of high network latency and improves bandwidth utilization, leading to substantial speedups for all backends.
Small
P
3 C+S gRPgRPC RPC rch I yTo MP
Geo-Distributed Geo-Proximal
LAN
P
C gRP PC R rch yTo MPI
0
10
0
10
3 C+S gRPgRPC C P hR orc PI PyT M 0
20
20
Medium
20
30
30
40
40
60
Time (seconds)
Train
40
50
80
0
5
10
15
60
0
5
10
15
100
0
20
40
60
Aggregation
Comm.
Time (seconds)
Big
20
20
80
Serialization
0
25
100
0
0
CPU-GPU Migration
25
50
50
75
Large
100 125 150
100
150
100 200 300 400 500 600 Time (seconds)
Wait
Client (Solid)
0
0
0
100 200 300 400 500 600
200
400
600
500 1000 1500 2000 2500 Time (seconds)
Server (Hatched)
Fig. 5: Per-state duration in end-to-end LAN experiments (communication, CPU-GPU migration, serialization, waiting; plus training on clients and aggregation on server).
gRPC and MPI_MEM_BUFF see performance gains of up to 7× in these conditions. However, there is an exception in the LAN environment, where MPI-based backends experience a performance decline with concurrent transmissions, likely due to multithreading overheads offsetting communication gains. Memory utilization during concurrent dispatches, detailed in Figure 4(c), reveals stark differences between backends. gRPC+S3 is exceptionally memory-efficient, as it uploads the model to S3 only once and subsequently sends lightweight metadata messages, resulting in constant memory usage regardless of the number of receivers. In contrast, memory consumption for all other backends increases almost linearly with the number of concurrent dispatches. This issue is most severe for gRPC and MPI_GENERIC, which create a separate copy of the data for each message, making concurrent dispatches of large models impractical. MPI_MEM_BUFF and PyTorch RPC are more memory-efficient as their design allows them to read directly from buffer memory, avoiding data duplication. VI. E ND - TO -E ND E XPERIMENTS In our end-to-end experiments, we evaluated the system performance using one server and seven clients, with clients training for only a single epoch in each round to emphasize the impact of communication backends. The server was configured to use concurrent transmission to distribute the global model, and each backend was set to its optimal configuration (e.g., GPU-Aware MPI and environment-specific settings for PyTorch RPC). The results are shown in Figure 5. In LAN and GeoProximal environments, the total execution time for small and medium-sized models is heavily dominated by the client training time, leading to comparable performance across most backends. The primary exception is gRPC+S3, whose twostep transfer mechanism introduces overhead that makes it less competitive in these settings. As model sizes increase, communication becomes the bottleneck, and the efficiency of MPI and PyTorch RPC allows them to significantly outperform other backends. In the LAN setting, gRPC-based training was approximately 9× slower for large models.
In the Geo-Distributed environment, communication latency is the primary performance driver, leading to high variance in client completion times, resulting in long waiting periods. PyTorch RPC delivered the lowest execution time for most model sizes, with MPI performing closely and even surpassing it for large models. The most notable result in this setting is the performance inversion between gRPC and gRPC+S3. While gRPC is competitive for small models, it struggles with larger ones because it utilizes available bandwidth ineffectively. This limitation is overcome by gRPC+S3, which delegates the large data transfer to S3, making it 3.5–3.8× faster than gRPC for large models and a much more suitable choice for geodistributed deployments. VII. D ISCUSSION The optimal communication backend hinges on a trade-off between the network environment and practical deployment requirements. In low-latency trusted networks (LAN, GeoProximal), performance is limited by CPU overheads like serialization, making memory-buffer backends (MPI, PyTorch RPC) superior. Conversely, in high-latency Geo-Distributed settings, efficient connection concurrency is paramount, giving PyTorch RPC a general advantage. The high-latency context also reveals a critical bottleneck in standard gRPC for large models, a limitation our gRPC+S3 backend resolves by offloading transfers to cloud storage, improving both throughput and memory efficiency. Practically, while MPI and PyTorch RPC are fastest, their unsuitability for untrusted WANs makes the versatile gRPC+S3, with gRPC fallback for small payloads (< 10 MB), the most robust choice for many real-world FL deployments. VIII. R ELATED W ORK Numerous federated learning (FL) frameworks have been proposed to benchmark tasks and models [17], [21], [22]. These studies, however, typically focus on computational efficiency and model accuracy while overlooking the impact of the communication layer. Most benchmarks rely on a single default backend (usually gRPC) and lack a comparative
analysis of alternative transport mechanisms, a gap our work directly addresses. Our research also connects to two broader areas in distributed machine learning (DML). The first is work on characterizing and enhancing communication backends [4], [23]. These studies are generally confined to high-performance, trusted networks, whereas our analysis extends to untrusted, geo-distributed settings. The second area involves algorithmic techniques to reduce communication overhead, such as quantization [24] and sparsification [25]. These methods are orthogonal to our study and can be used in conjunction with any backend. Similarly, research in geo-distributed training has focused on algorithmic improvements like finding optimal topologies [26] rather than providing a system-level analysis of the underlying communication backends, which is the focus of our paper. IX. C ONCLUSION In this study, we analyzed various communication backends (MPI, gRPC, and PyTorch RPC) for cross-silo FL. We employed peer-to-peer and end-to-end benchmarks to examine their performance under different network conditions and with varying model sizes. We also introduced a novel communication backend, gRPC+S3, to overcome the shortcomings of existing backends, particularly when transmitting large models in untrusted environments. In the context of federated learning, gRPC+S3 demonstrated remarkable memory efficiency, and our Geo-Distributed experiments show that gRPC+S3 significantly outperforms gRPC for messages larger than 10 MB (up to 3.8×), making it a practical choice in such settings. Finally, this study offers clear guidelines and insights, helping researchers and practitioners choose and configure the appropriate communication tools for their federated learning tasks to build efficient systems. X. ACKNOWLEDGMENT This work was supported in part by Qualcomm Technologies, Inc. The opinions, findings, and conclusions or recommendations expressed are those of the author(s) and do not necessarily reflect the views of the sponsor. R EFERENCES [1] P. Kairouz, H. McMahan, B. Avent, A. Bellet, M. Bennis, A. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al., “Advances and open problems in federated learning. arxiv,” arXiv preprint arXiv:1912.04977, 2019. [2] M. P. Forum, “Mpi: A message-passing interface standard,” 1994. [3] “gRPC: A high performance, open source universal rpc framework,” https://grpc.io, accessed: 2023-09-15. [4] P. Damania, S. Li, A. Desmaison, A. Azzolini, B. Vaughan, E. Yang, G. Chanan, G. J. Chen, H. Jia, H. Huang et al., “Pytorch rpc: Distributed deep learning built on tensor-optimized remote procedure calls,” Proceedings of Machine Learning and Systems, vol. 5, 2023. [5] “Amazon simple storage service,” https://aws.amazon.com/s3/, accessed: 2023-09-15. [6] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282.
[7] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492, 2016. [8] H. Robbins and S. Monro, “A stochastic approximation method,” The annals of mathematical statistics, pp. 400–407, 1951. [9] T. Weyand, A. Araujo, B. Cao, and J. Sim, “Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2575–2584. [10] K. Lang, “Newsweeder: Learning to filter netnews,” in Machine learning proceedings 1995. Elsevier, 1995, pp. 331–339. [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [12] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al., “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1314–1324. [13] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019. [14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021. [15] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [16] S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-B. Song, M. ElKhamy, and S. Avestimehr, “Slora: Federated parameter efficient finetuning of language models,” arXiv preprint arXiv:2308.06522, 2023. [17] C. He, S. Li, J. So, X. Zeng, M. Zhang, H. Wang, X. Wang, P. Vepakomma, A. Singh, H. Qiu et al., “Fedml: A research library and benchmark for federated machine learning,” arXiv preprint arXiv:2007.13518, 2020. [18] E. Gabriel, G. E. Fagg, G. Bosilca, T. Angskun, J. J. Dongarra, J. M. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine et al., “Open mpi: Goals, concept, and design of a next generation mpi implementation,” in Recent Advances in Parallel Virtual Machine and Message Passing Interface: 11th European PVM/MPI Users’ Group Meeting Budapest, Hungary, September 19-22, 2004. Proceedings 11. Springer, 2004, pp. 97–104. [19] P. Shamis, M. G. Venkata, M. G. Lopez, M. B. Baker, O. Hernandez, Y. Itigin, M. Dubman, G. Shainer, R. L. Graham, L. Liss et al., “Ucx: an open source framework for hpc network apis and beyond,” in 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects. IEEE, 2015, pp. 40–43. [20] L. Dalcin and Y.-L. L. Fang, “mpi4py: Status update after 12 years of development,” Computing in Science & Engineering, vol. 23, no. 4, pp. 47–54, 2021. [21] D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y. Gao, L. Sani, K. H. Li, T. Parcollet, P. P. B. de Gusmão et al., “Flower: A friendly federated learning research framework,” arXiv preprint arXiv:2007.14390, 2020. [22] F. Lai, Y. Dai, S. Singapuram, J. Liu, X. Zhu, H. Madhyastha, and M. Chowdhury, “Fedscale: Benchmarking model and system performance of federated learning at scale,” in International Conference on Machine Learning. PMLR, 2022, pp. 11 814–11 827. [23] S. Xu, A. Shafi, H. Subramoni, and D. K. Panda, “Arm meets cloud: A case study of mpi library performance on aws arm-based hpc cloud with elastic fabric adapter,” in 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 2022, pp. 449–456. [24] D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “Qsgd: Communication-efficient sgd via gradient quantization and encoding,” Advances in neural information processing systems, vol. 30, 2017. [25] J. Wangni, J. Wang, J. Liu, and T. Zhang, “Gradient sparsification for communication-efficient distributed optimization,” Advances in Neural Information Processing Systems, vol. 31, 2018. [26] O. Marfoq, C. Xu, G. Neglia, and R. Vidal, “Throughput-optimal topology design for cross-silo federated learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 19 478–19 487, 2020.