Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models Qianpiao Ma
Xiaozhu Song
Junlong Zhou∗
Nanjing University of Science and Technology Nanjing, Jiangsu, China [email protected]
Nanjing University of Science and Technology Nanjing, Jiangsu, China [email protected]
Nanjing University of Science and Technology Nanjing, Jiangsu, China [email protected]
Yue Zeng
Jianchun Liu
Huaqing Tu
Nanjing University of Science and Technology Nanjing, Jiangsu, China [email protected]
University of Science and Technology of China Hefei, Anhui, China [email protected]
Zhejiang Gongshang University Hangzhou, Zhejiang, China [email protected]
arXiv:2607.12583v1 [cs.DC] 14 Jul 2026
Abstract Federated fine-tuning (FedFT) enables adapting pre-trained large vision models (LVMs) on distributed, privacy-sensitive devices, while its practical deployment is hindered by three critical challenges: resource constraints, system heterogeneity, and non-IID data. While prior studies partially address these issues, e.g., by pre-training initial models on synthetic images to mitigate the adverse effects of non-IID data, or leveraging parameter-efficient finetuning (PEFT) methods like low-rank adaptation (LoRA) to reduce resource consumption, they remain inadequate and fragmented. Specifically, existing synthetic image generation methods fail to capture device-specific feature distributions, while current PEFT-based FedFT methods often undervalue weaker devices that may provide critical information. More importantly, the separate optimization of pre-training and FedFT neglects their inherent connection, lacking a holistic perspective to maximize training efficiency. To overcome these limitations, we propose FeDiSyn, a unified framework that holistically considers the interplay between pre-training and FedFT to minimize the overall LVM training time. Specifically, FeDiSyn introduces: (i) a scaling law for FedFT pre-training to determine the optimal number of synthetic images, balancing pre-training benefit against generation/pre-training cost, (ii) diffusion-based synthetic image generation that captures device-specific feature distributions for pre-training to tackle non-IID data, and (iii) a contributionaware LoRA configuration and bandwidth allocation algorithm for FedFT to ensure that informative devices are effectively utilized while addressing system heterogeneity. Experimental results on the real-world testbed demonstrate that FeDiSyn reduces completion ∗ Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ICPP ’26, Singapore © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
time by over 52.5% and communication cost by over 97.2%, while achieving comparable accuracy to state-of-the-art solutions.
CCS Concepts • Computing methodologies → Distributed artificial intelligence; • Human-centered computing → Mobile computing.
Keywords Federated fine-tuning, Synthetic image, Pre-training, Scaling law, Large vision models ACM Reference Format: Qianpiao Ma, Xiaozhu Song, Junlong Zhou, Yue Zeng, Jianchun Liu, and Huaqing Tu. 2026. Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models. In Proceedings of 55th International Conference on Parallel Processing (ICPP ’26). ACM, New York, NY, USA, 11 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Since the landmark success of AlexNet in 2012 [16], deep learning powered by large-scale labeled datasets has revolutionized the field of image classification. Initially, most vision models are trained via supervised learning on relatively small datasets. More recently, however, a two-stage paradigm has emerged for training large vision models (LVM): first pre-training on massive datasets, then fine-tuning on downstream task-specific data. In particular, pure transformer-based architectures such as the vision transformer (ViT) [5] have demonstrated more outstanding performance than the convolutional networks when pre-trained at scale. In parallel, the Internet of Things (IoT) [48] has grown explosively, generating massive amounts of data from edge devices that offers a rich source for training LVMs. However, centralized training on large-scale datasets is increasingly limited by privacy regulations (e.g., GDPR) and by bandwidth bottlenecks when accessing local data. This motivates the adoption of federated fine-tuning (FedFT) [46], a technique that integrates model fine-tuning into the federated learning (FL) framework [24]. In a typical FedFT setting, a parameter server first broadcasts a pre-trained global model to a set of distributed devices. Each device fine-tunes the model on its local data and sends the updated parameters back to the server for
ICPP ’26, September 28–October 1, 2026, Singapore
aggregation. This cycle of local fine-tuning and global aggregation repeats until the model converges or reaches a target accuracy. Challenges of FedFT for LVMs. Although FedFT offers a promising solution for fine-tuning LVMs leveraging distributed data, it still faces several practical challenges: (1) Resource Constraints. Modern LVMs like ViT and SwinT [21], usually contain billions of parameters, requiring significant computation/communication resources for fine-tuning/transmission. For example, ViT-L/16 requires roughly 3.1GB of traffic per communication and over 2,900 TFLOPs of computation. In contrast, device resources are typically limited [37]. For example, devices like the NVIDIA Jetson TX2 offer less than 2TFLOPs of computation capacity, while network bandwidth often falls below 100Mbps, rendering full-parameter FedFT infeasible on devices. (2) System Heterogeneity. Devices differ widely in hardware capabilities and network conditions. For example, the computational power of an iPhone 16 amounts to only about 12% of that of an RTX 3090 desktop GPU [18], while network condition varies significantly, ranging from 4G (∼ 100Mbps), 5G (∼ 500Mbps) to fiber connections (∼ 1Gbps). As a result, the time required for fine-tuning and transmitting parameters can differ substantially across devices. Faster devices may idle while waiting for slower ones, leading to a straggler problem [22] that reduces overall FedFT efficiency. (3) Non-IID Data. Local data on devices are often nonindependent and non-identically distributed [24], which involves two aspects. First, the label distribution varies across devices, e.g., some classes may be overrepresented or missing entirely on certain devices. Second, even for the same label, the feature distributions may differ due to variations in style or domain, e.g., cartoon vs. natural images [40]. Status Quo and Limitations. To address resource constraints in FedFT, some studies [8, 28, 29, 33, 34, 38, 39] have deployed the parameter-efficient fine-tuning (PEFT) methods (e.g., Adapter tuning, LoRA and Prompt tuning). PEFT inserts lightweight trainable modules into the LVM while keeping the pre-trained model frozen. Since these modules typically account for less than 5% of the pre-trained model’s parameters, PEFT significantly reduces the computation and communication overhead. However, most of these works [8, 28, 34, 38, 39] simply transplant centralized PEFT methods into distributed settings without accounting for system heterogeneity across devices. A few studies [29, 33] attempt to mitigate heterogeneity by assigning varying LoRA depths based on devices’ capabilities. This strategy, however, overlooks the potential importance of weaker devices in FedFT [2]: assigning them smaller LoRA depths can significantly degrade fine-tuning performance. Moreover, these FedFT approaches are typically initialized with off-the-shelf pre-trained models without synthetic data augmentation, indicating that convergence performance could be further improved [15]. Recent theoretical work [12] has shown that a carefully initialized pre-trained global model can mitigate the adverse effects of non-IID data and accelerate convergence in FedFT. However, models pre-trained on public datasets (e.g., ImageNet [4]) may not reflect the device-specific data distributions [3], making them suboptimal as initialization points for FedFT. Several studies [43, 45] have introduced diffusion models (DM) [32] to generate synthetic images that approximate the label distribution on each device. These synthetic images are then used to pre-train a global model, serving as
Ma et al.
the initialization for FedFT. However, these approaches rely solely on text-conditioned synthesis (e.g., labels or text embeddings) and may fail to capture the unique feature distribution of each device [47], leading to a style gap between synthetic and real data that diminishes the effectiveness of augmentation. Overall, none of these approaches adequately addresses the challenges outlined above, summarized in Table 1. More importantly, these approaches treat synthetic pre-training and FedFT as isolated stages, and ignore their synergy: insufficient images weaken pre-training effectiveness for FedFT, whereas excessive images introduce substantial computation overhead, thereby prolonging generation and pre-training time. Therefore, the key challenge lies in determining the optimal number of generated images, so as to balance pre-training benefit for FedFT against generation/pre-training cost. Overview of the Proposed Approach. To this end, we propose FeDiSyn, a unified framework that holistically considers pretraining and FedFT to minimize the overall completion time for LVM training, which is built upon “one law, two optimizations”. First, we introduce a scaling law for synthetic pre-training under FedFT setting, which captures the relationship among accuracy, total training time, and the synthetic data scaling factor, providing quantitative guidance for determining the optimal number of generated images. Second, for pre-training optimization, each device extracts features by encoding local images into latents, which are then uploaded to the server. By combining these latents with semantically enhanced labels via a DM, the server generates synthetic images to pre-train a global model. Since the synthetic images capture cross-device label and feature distributions, the pre-trained model yields a superior initialization for FedFT, effectively addressing both label and feature distribution skews. Third, for FedFT optimization, FeDiSyn introduces a contribution-aware LoRA configuration and bandwidth allocation algorithm. By configuring potentially important devices sufficient LoRA depth and balancing completion times via bandwidth allocation, the FedFT performance can be further improved while addressing system heterogeneity. Our main contributions are summarized as follows. • We propose FeDiSyn, a unified framework that holistically considers the close interplay between pre-training and FedFT to reduce the overall LVM training time. To our knowledge, this is the first work to explore the scaling law for synthetic pre-training under FedFT setting, and systematically optimize these two stages in a unified manner. • We develop a synthetic pre-training method that tackles nonIID data from both label and feature distribution perspectives. In addition, we design a contribution-aware LoRA configuration and bandwidth allocation algorithm to improve FedFT performance while addressing system heterogeneity. • We implement the proposed FeDiSyn framework and related algorithms on a real-world testbed. Experimental results on various models and datasets show that the proposed solution outperforms the state-of-the-art solutions.
2 Background and Motivation 2.1 Pre-Training Model Initialization for FedFT The Importance of Pre-Training and Initialization. As a central challenge in FedFT, non-IID data is typically addressed through
Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models
ICPP ’26, September 28–October 1, 2026, Singapore
Table 1: Comparison of FedFT methods for LVMs. Method PerAda [39] pFedSeq [28] CaFPT [8] FedPrefix [34] SFLF [38] FedRA [33] CAFF[29] FPS [3] GPT-FL [45] FGL [43] FeDiSyn (Ours)
Synthetic Image for Pre-training No No No No No No No Yes Yes Yes Yes
PEFT Method Adapter Adapter Prompt Prompt LoRA LoRA LoRA LoRA
Handling Resource Constraints ✓ ✓ ✓ ✓ ✓ ✓ ✓ × × × ✓
various aspects, such as device sampling [2] and aggregation mechanism [22, 23], while the impact of model initialization is often overlooked in most studies. Several recent works have highlighted the critical importance of pre-training for the initial model in FedFT. Nguyen et al. [26] observe that models initialized with pre-trained weights typically exhibit significantly lower training loss values at initialization compared to those initialized randomly. Karimireddy et al. [15] further demonstrate that this lower initial loss resulting from pre-training facilitates faster convergence during subsequent FedFT. Chen et al. [3] note that pre-training shapes a better conditioned loss surface and steers the model into a location closer to the optimal solution, enhancing the stability of model aggregation. These phenomena are theoretically supported by [12], which proves that pre-training effectively reduces sensitivity to certain hyperparameters within FedFT by diminishing misaligned filters at initialization. Synthetic Image for Pre-Training LVMs. Although pre-training initialization can enhance FedFT performance, models pre-trained on public datasets (e.g., ImageNet) show limited gains due to domain mismatch between the public data and the device-specific data. For example, in chest X-ray analysis, an ImageNet pre-trained model achieves only 0.2% accuracy improvement over random initialization, whereas a domain-relevant synthetic image pre-trained model yields a 5.9% gain [3]. To address this, GPT-FL [45] and FGL [43] upload labels or text descriptions of device datasets to the server for synthetic image generation. However, these approaches rely solely on text-conditioned generation, which fails to capture devicespecific feature distributions. As a result, the synthetic images may not align well with real local data, limiting their improvement for FedFT. In contrast, FeDiSyn uploads latents encoding local data features to generate synthetic images, as elaborated in Section 3. For example, generated images for Caltech-101 [7] in Fig. 1 show that FeDiSyn can obtain synthetic images that better align with the feature distributions of raw data compared to GPT-FL and FGL.
2.2
Scaling Law of Synthetic-Image Pre-Training for FedFT
Following the groundbreaking success of large-scale models, scaling laws have emerged as a critical framework for quantifying the power-law relationships between model/data scale and performance [14]. Prior studies have demonstrated that pre-training data
Handling System Heterogeneity × × × × × ✓ ✓ × × × ✓
Image
Chair
Handling Label Distribution Skew × × × × × × × ✓ ✓ ✓ ✓
Cup
Ewer
Handling Feature Distribution Skew × × × × × × × × × × ✓
Ferry
Ibis
Raw
Fig. 1: Comparison of FeDiSyn and other synthetic image generation methods on Caltech-101. volume has a significant impact on model performance [27, 42], which has motivated recent extensions of these principles to synthetic data, exploring how generative data scale influences convergence [6, 13]. However, these investigations focus exclusively on centralized training paradigms and are therefore not directly applicable to the FedFT setting, where the capabilities of servers for pre-training and edge devices for fine-tuning jointly determine the overall training time. As illustrated in Fig. 2 on Caltech-101, the synthetic scaling factor 𝜆 introduces a trade-off in FedFT: increasing 𝜆 accelerates convergence but incurs substantial overhead in image generation and pre-training. Therefore, the total completion time follows a U-shaped curve, which, in this example, attains its minimum near 𝜆 = 0.5. To provide guidance for synthetic pre-training in the FedFT setting, we propose a novel scaling law that quantifies the relationship between synthetic volume, target accuracy, and total training time, enabling the analytical determination of the optimal scaling factor 𝜆, as elaborated in Section 5.
2.3
Federated Fine-Tuning with LoRA for LVMs
LoRA [11] fine-tuning is a specific PEFT approach via low-rank matrix decomposition. In LoRA-based FedFT, each device only finetunes and uploads the lightweight inserted LoRA layers while keeping the pre-trained model frozen. The server aggregates the uploaded LoRA parameters and then distributes them back to all
ICPP ’26, September 28–October 1, 2026, Singapore