Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

, ,

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions QINGLIN YANG, Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, China

CHEN QIU, Iwate Biotechnology Research Center, Japan HONGYUAN ZHANG, Nanjing University of Posts and Telecommunications, China PENGDENG LI, YUAN LIU∗ , and ZHIHONG TIAN, Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, China

arXiv:2608.13844v1 [cs.LG] 14 Aug 2026

Large language models (LLMs) have become core components of cloud-based intelligent services in academia and industry, yet their training and deployment are hindered by high computational costs, data centralization, and privacy concerns. Federated learning (FL) offers a decentralized training paradigm that enables clients to collaboratively train a learning model without sharing raw data, making it a promising solution for privacy-preserving LLM training and reasoning. This paper presents a comprehensive survey of federated prompt learning (FPL) to review recent advances in integrating the federated learning paradigm and large language models, answering the following research questions: RQ1: The fundamental motivations, characteristics, and enabling technologies of FPL, and how it differs from conventional FL and full-model federated fine-tuning; RQ2: The trade-offs FPL approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling; RQ3: The remaining security, privacy, robustness, and system challenges, along with key future research directions. To this end, we systematically examine existing FPL methods across the full model lifecycle: pre-training, fine-tuning, and practical applications, while discussing security, privacy, and robustness issues and summarizing existing defense mechanisms. Finally, we highlight open challenges and future directions, aiming to help readers understand how the insights drive research in FPL. CCS Concepts: • Computing methodologies → Natural language processing; Machine learning; • Security and privacy → Privacy-preserving protocols; • Computer systems organization → Distributed architectures. Additional Key Words and Phrases: Large Language Models, Collaborative Learning, Prompt Learning, Federated Fine-tuning, ParameterEfficient Fine-tuning, Security ∗

The Corresponding author.

Authors’ Contact Information: Qinglin Yang, yangqinglin@gzhu.edu.cn, Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, Guangzhou, Guangdong, China; Chen Qiu, c-qiu@ibrc.or.jp, Department of Horticultural Science, Iwate Biotechnology Research Center, Iwate, Japan; Hongyuan Zhang, b23110420@njupt.edu.cn, Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu, China; Pengdeng Li; Yuan Liu; Zhihong Tian, {pdli, yuanliu,tianzhihong}@gzhu.edu.cn, Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, Guangzhou, Guangdong, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM

1

2

Q.Yang C.Qiu, H.Zhang et al.

ACM Reference Format: Qinglin Yang, Chen Qiu, Hongyuan Zhang, Pengdeng Li, Yuan Liu, and Zhihong Tian. 2026. Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions. J. ACM 0, 0, Article 1 (August 2026), 35 pages. https://doi.org/xxx

1 Introduction Large language models (LLMs) have become the cornerstone of modern artificial intelligence, demonstrating remarkable capabilities in natural language understanding, reasoning, and generation across a wide range of domains, including healthcare, finance, software engineering, and education [4]. However, the rapid scaling of LLMs has exposed fundamental limitations of the prevailing centralized training and deployment paradigm. For training, the reliance on massive centralized datasets raises severe privacy and regulatory concerns, while the growing scarcity of high-quality public data and the prohibitive computational and communication costs increasingly constrain further model scaling. These challenges motivate the exploration of new collaborative and privacy-preserving learning paradigms for the development and adaptation of LLMs. FL has emerged as a promising decentralized framework that enables multiple data owners to collaboratively train machine learning models without sharing raw data [59]. When FL meets LLMs, it raises some diverse challenges compared with conventional FL patterns. (i) The distinctive significance of FL for LLMs is the stronger demand for multisource heterogeneous data. (ii) Conventional LLMs usually rely on centralized training, in which massive amounts of data are aggregated onto a unified platform before pre-training or fine-tuning is conducted. However, in real-world scenarios, many forms of high-value data cannot be centralized, such as internal enterprise documents, medical data, financial data, legal data, personal device data, and cross-institutional business data. (iii) In terms of model architecture, FL promotes the evolution of LLMs from a single unified model toward a structure in which shared capabilities and personalized capabilities coexist. Therefore, the integration of LLMs with FL faces greater technical challenges than traditional deep learning. LLMs usually have enormous parameter scales, often reaching billions or even hundreds of billions of parameters, which is several orders of magnitude larger than conventional deep learning models (e.g., GoogLeNet, AlexNet, VGG, and ResNet). If the full model parameters are directly transmitted, both communication costs and storage pressure become extremely high. Hence, the core issue is no longer merely how to aggregate models from multiple clients, but how to transmit and update only the most critical, minimal, and effective parameters under extremely large model scales. To this end, prompt learning and parameter-efficient fine-tuning (PEFT) have gained significant attention as lightweight alternatives to full-model fine-tuning. By freezing the backbone of a pretrained foundation model and optimizing only a small number of task-specific parameters—such as soft prompts, prefixes, or low-rank adapters (LoRA) [21], training cost is drastically reduced while preserving the generalization power of large models. Importantly, the compactness and modularity of prompt-based updates make them inherently well-suited to federated environments. This observation has led to a new paradigm, federated prompt learning (FPL), which leverages prompt learning as the primary interface between LLMs and federated optimization. FPL fundamentally reshapes how LLMs are trained, adapted, and deployed in distributed settings. Instead of federating the entire model or large subsets of parameters, clients collaboratively optimize lightweight prompt or adapter modules on top of a shared frozen foundation model. This design dramatically reduces communication costs, mitigates client-side resource constraints, and enhances robustness to non-IID data, while maintaining strong privacy guarantees. In consequence, federated prompt learning has rapidly evolved from early proof-of-concept studies to a rich ecosystem Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions Conventional Federated Learning Users directly train models on local data

3

Federated Prompt Learning Users train prompt adapters; local data stay private

Parameter Server Aggregate global model

Central server improves global model with prompt update

Local model training

Local prompt adapter training

Local model updates

Prompt adapter updates Only small prompt adapters are trained

Global model training requires local data Private Local Data

User B

User A

Enhancements with custom prompts

User C

User D

Fig. 1. The Difference between conventional FL and federated prompt learning.

of methods spanning prompt-based fine-tuning, personalization, split and off-site architectures, multimodal learning, and real-world applications. 1.1 Motivations and Contributions Despite this rapid progress, the existing literature remains fragmented. Prior surveys [14, 43, 46, 53, 55, 56] have primarily focused on either federated learning in general, federated fine-tuning of LLMs, or the security and privacy of LLMs. Despite the rapid growth of this field, its literature remains fragmented across machine learning, distributed systems, cloud–edge computing, multimodal learning, and cybersecurity. Existing surveys typically examine general FedLLM architectures, federated fine-tuning, IoT-oriented deployments, or LLM security independently. Prompt- and adapter-based methods are often treated as auxiliary implementation techniques rather than as a distinct distributed optimization and service-delivery paradigm. Moreover, previous reviews rarely connect the complete model-service lifecycle, including training, personalization, inference, deployment, application, and protection, or systematically compare methods in terms of computational overhead, communication cost, scalability, dependability, and privacy. A systematic and unified treatment that centers on federated prompt learning as a distinct paradigm, examines its role across the full LLM lifecycle, and synthesizes insights from algorithmic, system, application, and security perspectives is still lacking. Without such a comprehensive view, it is difficult for researchers and practitioners to understand the design trade-offs, identify open challenges, and navigate future research directions in this emerging field. To address this gap, this paper presents a comprehensive survey of FPL for LLMs that develops a unified, lifecycle-oriented understanding of federated prompt learning from both algorithmic and cloud-system perspectives. It organizes existing approaches by their technical mechanisms and deployment roles, and examines how prompt-based adaptation interacts with cloud coordination, heterogeneous edge resources, distributed inference, multimodal applications, and Manuscript submitted to ACM

4

Q.Yang C.Qiu, H.Zhang et al.

adversarial environments. Particular attention is given to performance, communication efficiency, client-side computation and memory, scalability, personalization, privacy, robustness, and practical deployability. It is critically important to answer the following research questions(RQs) comprehensively: (1) RQ1: What are the fundamental motivations, characteristics, and enabling technologies of federated prompt learning, and how does it differ from conventional federated learning and full-model federated fine-tuning? (2) RQ2: What trade-offs do federated prompt learning approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling? (3) RQ3: What security, privacy, robustness, and system challenges remain, and what future research directions are most important? Our contributions are summarized as follows:

• The core contribution of this survey lies in the experimental validation of representative federated prompt learning models and frameworks, through which we derive key empirical findings and practical insights. These results are intended to provide guidance and inspiration for researchers to advance research in this emerging area. • We analyze the security, privacy, and robustness challenges that arise when prompt learning meets federated optimization, and summarize existing attack models and defense mechanisms tailored to federated LLMs. • We distill key lessons from current research and outline open challenges and promising future directions to guide subsequent studies in federated prompt learning. Through this survey, we aim to provide a structured and coherent reference for researchers and practitioners seeking to understand how prompt learning can effectively bridge large language models and federated learning, enabling scalable, efficient, and privacy-preserving collaboration in the era of foundation models. 1.2

Methods

This review combines a systematic literature review with taxonomy-based comparative analysis. Relevant studies are retrieved from databases, such as “Web of Science”, “Google Scholar”, and “dblp”, using predefined search terms related to federated learning and prompt learning. Explicit inclusion and exclusion criteria are applied according to publication period, document type, research relevance, and methodological completeness. After duplicate removal, title and abstract screening, and full-text assessment, 52 high-quality studies were retained for subsequent analysis. The selected studies are then systematically coded in terms of research objectives, core methods, model architectures, datasets, evaluation metrics, empirical results, and reported limitations. Based on their underlying technical mechanisms, the reviewed approaches are organized into a structured taxonomy and compared with respect to performance, efficiency, communication cost, computational overhead, privacy protection, scalability, and application scenarios. Finally, the review synthesizes the major advances, unresolved challenges, and potential directions for future research. 2 Related Work and Background 2.1

Related work

Recent research has increasingly investigated the integration of LLMs and FL as a response to the escalating challenges of data scarcity, privacy preservation, communication cost, and the prohibitive cost of centralized model training. Nikolaou et al. [37] establish a fundamental theoretical property of decoder-only Transformer language models, showing that they are almost surely injective and hence exactly invertible with respect to their input prompts. Contrary to Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

5

the prevailing intuition that nonlinearities, normalization, and attention mechanisms inevitably lead to information loss, the authors prove that distinct input prompts map to distinct hidden representations with probability one under standard architectures, continuous parameter initialization, and finite-step gradient-based training. Leveraging tools from real analysis, they show that collisions can only occur on measure-zero parameter sets and that common training procedures cannot reach such pathological configurations. Beyond theory, Nikolaou et al. introduce SIPIT, the first provably correct and efficient algorithm that reconstructs the exact input prompt from internal hidden states in linear time, exploiting the causal structure of Transformers. Cheng et al. [10] provide a comprehensive, structured overview of federated large language models (FedLLMs), elucidating motivations, methodologies, and future directions at the intersection of FL and LLMs. They highlight FedLLMs as a promising paradigm for addressing fundamental LLM challenges, including exhaustion of high-quality public data, stringent privacy requirements, continuous model updating, and prohibitive computational and communication costs. By systematically reviewing the full FedLLM lifecycle, from pre-training and fine-tuning to deployment and application, the paper categorizes existing approaches such as parameter-efficient fine-tuning, prompt-based and split learning methods, personalized federated LLMs, and backpropagation-free techniques, analyzing their performance trade-offs under data, system, and model heterogeneity. The survey further identifies critical challenges, including communication bottlenecks, synchronization and straggler issues, non-IID data, and amplified security and privacy threats such as poisoning, backdoors, and inference attacks, and summarizes current defense mechanisms and their limitations. For data scarcity, Villalobos et al. [46] quantitatively demonstrate that the supply of high-quality public humangenerated text is finite and likely to be exhausted by frontier LLMs between 2026 and 2032 under current scaling trends. This finding underscores a fundamental limitation of centralized data-driven scaling and highlights the necessity of leveraging decentralized, privately held data sources, thereby motivating privacy-preserving collaborative learning paradigms (e.g., FL). On the systems side, Fan et al. [14] propose FATE-LLM, an industrial-grade FedLLM framework that shows how parameter-efficient fine-tuning (PEFT), off-site tuning, and knowledge distillation can substantially reduce communication overhead while approaching centralized fine-tuning performance. These core techniques of the work demonstrate the practical feasibility of federated LLM training in enterprise environments, but primarily focus on system implementation rather than a unified methodological or conceptual abstraction. Chen et al. [7] systematically review the emerging integration of LLMs and federated learning, highlighting their complementarity in addressing data scarcity, privacy constraints, and scalability. It organizes prior work into a unified framework spanning: (i) incorporating LLM sub-technologies, pre-training and prompt engineering, into FL to improve convergence speed, robustness to nonIID data, personalization, and domain generalization; (ii) applying FL sub-technologies, distributed computing and privacy-preserving mechanisms, to mitigate LLMs’ high computational cost, limited data coverage, and privacy risks; and (iii) holistic federated LLM (FedLLM) systems that jointly optimize training, instruction tuning, and alignment under decentralized settings. Several surveys have attempted to organize the rapidly growing literature at the intersection of LLMs and FL. For example, Thakur et al. [43] provide a broad unifying analysis of FL–LLM fusion, categorizing prior work into paradigms that either use LLM capabilities to enhance FL, apply FL to improve LLM adaptation, or jointly design holistic FedLLM systems. Nevertheless, this review adopts a macro-level perspective and does not systematically analyze prompt- or adapter-based learning as a first-class federated optimization interface. Yao et al. [56] focus on security and privacy issues of LLMs, offering an extensive taxonomy of risks and defenses, but largely decouple these concerns from federated training dynamics. Yang et al. [55] further expand the scope by reviewing the joint integration of IoT, LLMs, Manuscript submitted to ACM

6

Q.Yang C.Qiu, H.Zhang et al.

and FL, emphasizing architectural synergies in edge systems, yet their analysis is application-driven and not centered on LLM adaptation mechanisms. At a finer granularity, Yan et al. [53] conduct a comparative study of federated finetuning paradigms for LLMs, including FedLLMs, KD-FedLLMs, and Split-FedLLMs. Their results reveal inherent tradeoffs between model accuracy, communication cost, and client-side computation, illustrating that no single federated framework is universally optimal. Despite these substantial advances, existing studies and surveys exhibit a notable gap: the lack of a dedicated, systematic treatment of FPL as a distinct and unifying paradigm for LLM–FL integration. Prompt- and adapter-based methods have emerged as the dominant practical mechanism for adapting large language models under resource, communication, and privacy constraints, yet they are often discussed only as auxiliary techniques within broader FedLLM frameworks or system implementations. Different from the existing surveys, our review places FPL at the center of analysis, systematically reviewing how prompt- and adapter-based techniques enable efficient, scalable, and privacy-preserving collaboration of large language models under federated settings. By organizing existing methods along the full LLM lifecycle and jointly examining algorithmic, system-level, application-driven, and security-related dimensions, this survey constructs a unified framework that provides answers to the questions: Why do LLMs need FL?, How does FPL work?, What can FL and LLMs do for each other?, What kinds of challenges can FPL meet?, and What are the potential research directions of FPL? 2.2

Background and Fundamentals

This subsection introduces the fundamental concepts underlying federated prompt learning. We first describe large language models and their adaptation requirements, followed by conventional federated learning, prompt learning, and parameter-efficient fine-tuning. We then formalize federated prompt learning as a lightweight interface for adapting foundation models using decentralized private data. 2.2.1

Large Language Models and Model Adaptation. LLMs are typically built on transformer [20] architectures and

pretrained on large-scale text corpora using self-supervised objectives. Through large-scale pretraining, LLMs acquire general-purpose representations that support a broad range of downstream capabilities, including language understanding, text generation, reasoning, question answering, and instruction following [4]. These pretrained models can subsequently be adapted to specific tasks, domains, organizations, or users. LLMs containing billions of parameters impose substantial computation, memory, communication, and storage requirements. In distributed settings, repeatedly transmitting full-model parameters or gradients further creates a major communication bottleneck. Full fine-tuning may also require each participant to possess sufficient hardware resources to store and optimize the complete model. The adaptation of LLMs is further complicated by the decentralized nature of high-value data. Enterprise documents, medical records, financial information, source code, personal-device data, and institution-specific knowledge are frequently distributed across independent data owners. Privacy requirements, regulatory restrictions, and commercial confidentiality often prevent these data from being transferred to a centralized training platform. These requirements motivate the combination of federated learning with lightweight model-adaptation techniques. 2.2.2

Conventional Federated Learning. As illustrated in Fig. 1, a typical FL system consists of a central server and

a set of participating clients. Each client maintains a local dataset and performs model optimization within its own trusted environment. The server coordinates training by distributing model parameters, collecting local updates, and Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

7

aggregating them into a global model. A conventional FL round generally contains the following three steps: global model broadcast, local model training and transmission, and model aggregation [26]. Let 𝐾 denote the number of clients, 𝐷𝑘 the private dataset of client 𝑘 , and 𝑛𝑘 = |𝐷𝑘 | its number of samples. Conventional FL can be expressed as the minimization of a weighted global objective: 𝐾

𝑛𝑘 𝐹𝑘 (w), 𝐾 𝑘=1 ∑𝑗=1 𝑛𝑗

min 𝐹 (w) = ∑ w

(1)

where 𝑤 represents the shared model parameters and 𝐹𝑘 (w) is the local objective evaluated on 𝐷𝑘 . In FedAvg, the server aggregates the locally optimized parameters as:

𝑛𝑘 , w𝑡+1 𝑘 ∑ 𝑗∈𝑆𝑡 𝑛𝑗 𝑘∈𝑆

w𝑡+1 = ∑

(2)

𝑡

where 𝑆𝑡 is the set of clients participating in communication round 𝑡 . The clients in conventional FL are usually heterogeneous, including non-independent and identically distributed data, computation capacities, storage, and communication. This case becomes more severe when FL is directly applied to LLMs. Transmitting and optimizing an entire LLM may exceed the computation, memory, and bandwidth available to many clients. Therefore, federated LLM systems generally require parameter-efficient adaptation, compression, model partitioning, or off-site training rather than straightforward full-model federated optimization. 2.2.3 Prompt Learning. The prompt plays a role in bridging the downstream task and the knowledge encoded in the pretrained model. Prompt learning adapts a pretrained model by modifying or learning the conditioning context while keeping most or all backbone parameters frozen [63]. This paper treats “prompt tuning” as a class of methods rather than a particular method. Prompts can generally be divided into discrete and continuous forms. A discrete prompt consists of human-readable tokens, task descriptions, demonstrations, or instructions inserted into the model input. Continuous prompt learning instead represents the prompt as a set of trainable embedding vectors. In prompt tuning, learnable embeddings are appended or prepended to the input sequence. Prefix tuning introduces trainable vectors into multiple transformer layers, commonly through the attention [45] mechanism. In vision or visionlanguage models, visual prompts may be introduced into image patches or intermediate visual representations, while textual prompts condition the language branch. Let 𝑓𝜃 denote a pretrained model with frozen parameters 𝜃 , and let 𝑝 denote a trainable prompt. Prompt learning optimizes: min ℒ (𝑓𝜃 (x; p), 𝑦) , p

(3)

where (𝑥, 𝑦) is a downstream training example. Because 𝜃 remains fixed and the dimension of 𝑝 is substantially smaller than that of the backbone, prompt learning reduces the computation, storage, and communication required for adaptation. However, prompt learning remains sensitive to the quality of the pretrained model and the distance between the pretraining and downstream distributions. 2.2.4 Parameter-Efficient Fine-Tuning. Parameter-efficient fine-tuning (PEFT) [15] refers to a broader family of adaptation techniques that freeze most pretrained parameters and optimize only a small number of task-specific parameters. Manuscript submitted to ACM

8

Q.Yang C.Qiu, H.Zhang et al.

Prompt learning (2.2.3) is an important form of PEFT, while other representative techniques include adapters [22], LoRA, prefix-based modules [27], and bias-only tuning [58]. PEFT is particularly suitable for federated LLM adaptation. Because only compact parameter subsets are transmitted, communication overhead can be reduced from full-model scale to prompt- or adapter-module scale. Clients can maintain a common frozen backbone while learning local adaptation modules, and the server can aggregate only the lightweight updates. 2.2.5

Federated Prompt Learning (FPL). FPL integrates federated optimization with prompt learning or closely related

PEFT methods [11]. It mainly federates lightweight adaptation modules rather than the complete foundation model. FPL can substantially reduce communication and client-side optimization costs because only compact modules are trained and exchanged. In FPL, the frozen foundation model preserves broadly transferable knowledge and reduces the risk of catastrophic forgetting during local adaptation. Meanwhile, modular prompts and adapters enable the coexistence of shared and personalized capabilities. A typical FPL training round contains four stages: (1) The server distributes the current global prompt or adapter parameters to selected clients. (2) Each client attaches the received module to a shared frozen foundation model and performs local optimization using its private data. (3) Clients transmit only their updated prompt or adapter parameters to the server. (4) The server aggregates the lightweight updates and constructs a new global module. Let 𝜙 denote the federated prompt or adapter parameters, while 𝜃 denotes the frozen backbone. The FPL objective can be represented as: 𝐾

𝑛𝑘 𝐹𝑘 (𝜃, 𝜙). 𝐾 𝑘=1 ∑𝑗=1 𝑛𝑗

min ∑ 𝜙

After local optimization, the server may perform weighted aggregation: 𝑛𝑘 𝜙 𝑡+1 = ∑ 𝜙𝑘𝑡+1 . ∑ 𝑛 𝑗 𝑗∈𝑆 𝑘∈𝑆 𝑡

(4)

(5)

𝑡

Different from conventional FL, in which the communicated variable 𝑤 may contain all model parameters, FPL communicates only 𝜙 , whose size is generally much smaller than that of the foundation model. For example, PROMPTFL [18] replaces conventional full-model training with cooperative optimization of continuous prompts over a frozen foundation model, while FedPrompt [61] extends soft prompt tuning to federated pretrained language models and supports prompt tuning, P-tuning [30], and prefix tuning. Nevertheless, FPL introduces challenges that differ from those of conventional FL. Prompt parameters learned by heterogeneous clients may encode semantically different concepts, making position-wise averaging ineffective. Data heterogeneity can therefore cause prompt misalignment, client drift, and negative transfer. A single global prompt may favor dominant clients and provide limited adaptation to minority domains. Personalized FPL addresses this issue by separating global knowledge from client-specific prompts or by generating customized prompts according to client updates. 2.2.6

Privacy, Security, and Robustness Considerations. Although keeping raw data local is an important privacy prop-

erty of FPL, lightweight parameter exchange does not eliminate security risks. Prompt or adapter updates may still Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

9

reveal information about local examples through gradient inversion, membership inference, or representation reconstruction. A malicious client may poison its local data or manipulate its prompt updates to insert a backdoor into the aggregated module. Conversely, a malicious or compromised server may distribute manipulated prompts, infer client properties, or exploit intermediate activations in split architectures. Answers to RQ1: FPL emerges from combining FL with prompt-based tuning of large pretrained (often visionlanguage) models, motivated primarily by the practical limitations of applying conventional FL to foundation models: high communication overhead, prohibitive on-device compute and memory demands, non-IID data heterogeneity across clients, and risks of overfitting or catastrophic forgetting under full-model fine-tuning. Rather than exchanging full model gradients or weights, FPL keeps the pretrained backbone frozen at both server and clients, and collaboratively learns only lightweight, task-specific prompts, such as continuous soft-prompt embeddings, prefix vectors, or prompt-generating networks.Then these prompts are aggregated via standard or adapted FL protocols (e.g., FedAvg, FedProx, personalized FL). This design is enabled by advances in parameter-efficient fine-tuning (e.g., prompt tuning, prefix-tuning, P-tuning v2) and foundation models such as CLIP, whose strong pretrained representations can be effectively steered through prompts alone. Different from conventional FL, FPL reduces communication and computation by orders of magnitude, supports client-level personalization by decomposing prompts into shared and local components, and better preserves general pretrained knowledge. Compared to full-model federated fine-tuning, FPL avoids destructive weight-averaging under heterogeneous data, lowers system requirements by eliminating the need to store or transmit full gradients and optimizer states, and offers greater flexibility for supporting multiple tasks or domains without maintaining separate full model copies, positioning FPL as a parameter-efficient, communication-friendly paradigm for adapting foundation models in federated settings. 3 Federated Learning for Large Language Models From the perspective of the interaction between FL and LLMs, this survey broadly categorizes the existing research into three types: (i) federated learning for large language models; (ii) large language model-enhanced federated learning; (iii) the synergistic integration of federated learning and large language models. For the first category, federated learning is applied to address the challenges of training, adaptation, and deployment of large language models on distributed private data. In this relationship, the large language model is the primary object of learning and optimization, while FL provides a collaborative training mechanism that keeps data local. Based on the lifecycle of LLMs, this category can be further divided into federated pretraining, federated fine-tuning, federated instruction tuning, federated alignment, federated prompt learning, federated model compression, and federated continual learning, among other directions. 3.1 Larger Language Models Optimization through Federated Learning Chen et al[6] address the fundamental difficulty of applying FL to large language models LLMs, namely the prohibitive communication and computation costs incurred when updating massive model parameters under non-IID decentralized data. To overcome the limitations of existing prompt-based FL methods such as performance degradation, inefficient training, and client drift, the authors propose FedPepTAO by integrating parameter-efficient prompt tuning with a communication-efficient adaptive optimization strategy. FedPepTAO lies in a layer-importance scoring and lossless selection mechanism, which identifies and synchronizes only a subset of influential prompt layers while keeping other prompts locally updated, thereby significantly reducing communication overhead without sacrificing accuracy. FedPepTAO is evaluated through extensive experiments on 10 NLP benchmarks using RoBERTa-Large and Manuscript submitted to ACM

10

Q.Yang C.Qiu, H.Zhang et al.

multiple decoder-based LLMs (e.g., GPT-2, LLaMA-3B, LLaMA-7B). The results demonstrate that FedPepTAO consistently achieves state-of-the-art accuracy up to 60.8% improvement over baselines, while reducing training time by up to 97.59%. Raje [41] systematically investigates communication-efficient training of LLMs in FL settings, addressing the dual bottlenecks of limited client-side computation and expensive wireless communication that hinder practical federated LLM deployment. Considering that fine-tuning updates of pretrained LLMs are intrinsically low-rank, the work integrates LoRA-based parameter-efficient fine-tuning with communication-only sparsification, proposing federated LoRA with simple sparsity (FLoSS). Unlike prior approaches that prune adapters during training, FLoSS applies unstructured top-k sparsity exclusively during download and upload phases, while preserving dense local optimization to maintain model utility. To address the largely overlooked limitation of existing federated instruction tuning methods for large language models, Qin et al. [39] propose FedHDS. Considering the resource-constrained edge devices and overfitting to narrow client-specific domains, FedHDS constructs a hierarchical selection strategy that combines cross-layer feature fusion from multiple transformer layers with density-based clustering, enabling both local redundancy removal and global coordination across clients in a privacy-preserving manner. FedHDS establishes the first systematic solution for federated data-efficient instruction tuning and highlights the importance of redundancy-aware data selection for scalable, efficient, and generalizable federated LLM training. Furthermore, FedDQC [13] tackles a fundamental yet underexplored challenge in federated instruction tuning of large language models: data quality heterogeneity across decentralized clients, which can severely degrade global model performance despite privacy preservation. To address the lack of global visibility and the impracticality of centralized data filtering in FL, the authors [13] propose FedDQC. It is a novel federated data quality control framework that operates entirely on the client side with minimal overhead. The kernel of FedDQC is the instruction response alignment (IRA) metric, a lightweight and privacy-preserving quality estimator that measures how well an instruction conditions its response by comparing conditional and unconditional inference losses. Building on IRA, FedDQC introduces a quality-aware hierarchical federated training strategy that progressively fine-tunes the model from high-IRA (easy, high-quality) samples to lower-IRA (harder, noisier) data, mirroring human curriculum learning. To address the data heterogeneity issue in federated prompt tuning, Chen et al.[9] propose DualFPT, a federated visual prompt-tuning framework that jointly pursues generalized and personalized adaptation by decomposing learnable prompts into shared global prompts and client-specific local prompts. Because feature shifts and class imbalance impair the transferability of globally aggregated prompts while purely personalized methods risk overfitting to client-specific distributions. DualFPT is compose of two mechanisms: (i) feature sharing (FS); (ii) prompt composition scheme (PCS). First, FS uses a variational autoencoder to disentangle classification-sensitive features from redundant information and applies differential-privacy noise before sharing selected features, narrowing inter-client distribution gaps to improve global-prompt generalization. Second, the PCS employs a lightweight adaptive network to estimate similarity between each test instance and client distributions, dynamically combining local prompts into an instance-specific composite prompt. A two-stage alternating strategy optimizes global parameters, local prompts, and the adaptive network. Nevertheless, DualFPT requires an added feature-extraction stage, exchange of differentially private intermediate features, and training of auxiliary VAE, classifier, and adaptive-network components, increasing implementation complexity relative to prompt-only aggregation. Evaluation is also limited to image classification with a ViT-B/16 backbone, controlled feature- and class-heterogeneity settings, fixed prompt configurations, and relatively short federated training schedules. Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

11

To address the task and client resource heterogeneity in cross-device FL, Bai et al. [3] propose FlexLoRA, a federated fine-tuning framework for LLMs. Motivated by the “bucket effect” in conventional FL where all clients are constrained by the least-capable participant, the authors allow clients to apply heterogeneous LoRA ranks to match their local resources, enabling more powerful clients to contribute richer, less task-specific knowledge. FlexLoRA aggregates client updates by reconstructing full LoRA weight matrices, averaging them on the server, and applying SVD-based decomposition to redistribute rank-adaptive LoRA parameters back to clients, without additional hyperparameters. Theoretical analysis links higher local ranks and larger client populations to improved generalization bounds. Gao et al. [16] address the memory overhead and training latency of federated LoRA fine-tuning for LLMs on resource-constrained, system-heterogeneous edge devices by proposing FAH-QLoRA, a framework that combines heterogeneous base-model quantization with dynamically adjusted LoRA ranks. Its core mechanism follows a two-stage rank-allocation strategy: first determining the average LoRA rank that maximizes loss-reduction rate per unit wallclock time, then assigning device-specific ranks based on heterogeneous computation and communication capabilities, allocating lower ranks to slower devices to mitigate the straggler effect. Truncation and zero-padding further support local training and global aggregation of heterogeneous LoRA modules. The main contribution lies in jointly optimizing model precision and LoRA rank allocation to improve time and memory efficiency in federated LLM fine-tuning, complemented by a convergence analysis under non-convex, non-IID settings. 3.2 Larger Language Models Application via Federated Learning FedMRG [5] is the first comprehensive framework for LLM-driven medical report generation (MRG) under federated learning, tackling two key obstacles to multi-center collaboration: prohibitive LLM communication costs and severe multi-modal data heterogeneity across institutions. To enable privacy-preserving, scalable training, FedMRG employs LoRA to substantially reduce communication overhead. It further addresses heterogeneity at both visual and textual levels: on the encoder side, Hierarchical Contrasting and Prompting (HCP) combines client-aware contrastive learning with diagnosis-aware prompting to capture globally consistent yet locally distinctive visual features; on the decoder side, a Dual-adapter Mutual Boosting (DMB) mechanism harmonizes global reporting knowledge with client-specific linguistic styles via bidirectional knowledge distillation. Otoum et al. [38] present an LLM-driven federated learning framework for scalable and secure IoT management, motivated by the latency, privacy, energy, and scalability limitations of centralized cloud-based IoT architectures. The authors integrate LLMs with FL in a hybrid edge–cloud architecture, enabling privacy-preserving decision-making directly on IoT devices while leveraging cloud resources for global coordination. A key contribution is the gradient sensing federated strategy (GSFS), which adaptively regulates client participation and asynchronous update uploads based on performance shifts and gradient magnitudes, reducing redundant communication and improving convergence efficiency over classical methods, such as FedAvg and FedOpt. Evaluations on the IoT-23 dataset show GSFS achieves higher accuracy and F1-scores for both central and client models, while significantly cutting response latency (up to 51% on the client side) and improving energy efficiency. Results further indicate that edge-based LLM inference enables real-time IoT analytics with reduced cloud dependency, while federated coordination preserves data privacy across heterogeneous devices. Agarwal et al. [2] provide a critical empirical investigation into whether FL with pretrained language models (PLMs) genuinely achieves domain adaptation and personalization, or whether its apparent success stems largely from pretraining-induced regularization. Through systematic experiments on three NLP tasks: sentiment classification (SST2), sequence tagging (OntoNotes), and text generation (Gigaword), the authors analyze key confounders in federated Manuscript submitted to ACM

12

Q.Yang C.Qiu, H.Zhang et al.

NLP, including the role of pretrained weights, client size imbalance, data partition strategies, and the trade-off between server generalization and client personalization. Results show that with PLMs, the performance gap between federated and centralized training stays small even after aggressively ablating client updates, suggesting FL often fails to meaningfully adapt to local client distributions and instead benefits from pretraining’s strong semantic priors. The proposed personalization–generalization slope (𝑚Δ𝑃 ) further indicates that local-data learning provides limited or neutral gains to global generalization, challenging common claims about personalization in federated NLP. While uniform client data distributions improve convergence speed and stability, they do not fundamentally resolve the lack of true domain adaptation. Zhang [60] addresses the resource and memory heterogeneity encountered in federated foundation-model finetuning, where resource-constrained clients may be unable to update all LoRA layers efficiently. It proposes Fed-HeLLo, a federated LoRA framework that assigns different subsets of trainable LoRA layers to clients according to their computational capabilities and layer importance. Its core design combines Fisher information matrix–based allocation (FIMHLA) for dynamically estimating layer importance with geometrically defined and randomized allocation strategies (GD-HLA/RGD-HLA) for stabilizing early-stage training. The principal contribution therefore lies in jointly exploiting client resource heterogeneity and layer-wise importance for efficient federated LoRA fine-tuning. Nevertheless, the framework still relies on a server-side proxy dataset, retains non-negligible activation-memory requirements, lacks explicit privacy protection, and does not directly resolve data heterogeneity. 3.3

Empowering the Reasoning Process of LLMs in Federated Settings

eFedLLM [12] is an FL-based framework for LLM inference, targeting the prohibitive computational, memory, and bandwidth requirements that limit LLM accessibility. Departing from conventional data-parallel FL, it adopts a transformerbased model-parallel architecture, distributing different transformer layers across heterogeneous participants to enable collaborative inference without requiring any single user to host the full model. To ensure reliability in this chained execution setting, eFedLLM introduces a trust-based incentive and verification mechanism that evaluates intermediate layer outputs and filters malicious or low-quality contributors. It further incorporates transformer-tailored optimizations, including hierarchical memory access strategies to reduce global memory reads and SVD-based low-rank compression of weight matrices to lower communication bandwidth while preserving accuracy. Analytical and numerical evaluations show these optimizations reduce memory access and bandwidth usage by up to 60% under practical compression ratios, significantly improving inference efficiency. Lin [29] introduces SplitLoRA, the first split learning-based parameter-efficient fine-tuning framework for LLMs, addressing the prohibitive computation and communication costs limiting federated LLM adaptation on distributed private data. Motivated by the scarcity of high-quality public data and the impracticality of full-model federated finetuning, SplitLoRA combines split federated learning (SFL) with LoRA-based PEFT, partitioning the LLM between clients and a central server so clients train only shallow layers with lightweight LoRA adapters while the server handles most computation. This substantially reduces client-side resource demands and mitigates data heterogeneity by centralizing deeper representations. SplitLoRA provides a practical balance between performance, efficiency, and scalability. Meanwhile, it establishes an open-source benchmark for split LLM fine-tuning. However, open challenges remain in optimal model splitting, handling heterogeneous client resources, and strengthening privacy guarantees against split-learning inference attacks, pointing to promising directions for future research.

Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

13

Table 1. Research trajectories of FL for LLMs.

Research

Representative

Core Idea

Main Challenges

Trajectory

Works

ParameterEfficient

FedPepTAO [6],

Replace full-model federated

How to minimize communication

FLoSS [41]

fine-tuning with lightweight

overhead while preserving the

Adaptation and

prompt- or LoRA-based adaptation. effectiveness and stability of local

Communication

optimization.

Compression Federated

FedHDS [39],

Hierarchical data selection,

How to identify which

Instruction Data

FedDQC [13]

cross-layer feature fusion,

decentralized training samples are

Selection and

density-based clustering, and

informative and reliable when raw

Quality Control

quality-aware curriculum learning.

client data cannot be globally inspected.

Data

DualFPT [9],

Heterogeneity

Agarwal et al. [2] distributions through global–local

Address non-IID client

How to balance globally transferable knowledge and

and

prompt decomposition and

client-specific adaptation without

Personalization

personalized adaptation.

sacrificing either generalization or personalization.

System and

FlexLoRA [3],

Allow heterogeneous clients to

How to prevent

Resource

FAH-QLoRA

train different LoRA ranks,

resource-constrained clients from

Heterogeneity

[16],

quantization precisions, or subsets

becoming system bottlenecks

Fed-HeLLo [60]

of LoRA layers according to

while fully exploiting the clients

devices’ capabilities.

with high capabilities.

Model

eFedLLM [12],

Partition LLMs across clients,

How to achieve efficient and

Partitioning and

SplitLoRA [29]

servers, or multiple participants so

trustworthy collaborative

Collaborative

that no single resource-constrained execution while controlling

Inference

device is required to host or

intermediate-feature leakage,

execute the full model.

communication latency, participant heterogeneity, and model-partitioning overhead. Continued on next page

Manuscript submitted to ACM

14

Q.Yang C.Qiu, H.Zhang et al. Table 1 (continued)

Research

Representative

Trajectory

Works

Application-

FedMRG [5],

Core Idea

Main Challenges

Integrate parameter-efficient

How to adapt general-purpose

Driven Integrated GSFS [38]

adaptation, heterogeneity handling, federated LLM techniques to

Optimization

communication optimization, and

domain-specific data distributions,

edge–cloud collaboration

system constraints, privacy

according to the requirements of

requirements, and real-time

specific domains, such as

deployment conditions.

healthcare and IoT.

As summarized in Table 1, the research focus of federated LLMs has gradually shifted from simply reducing the cost of full-model federated optimization toward a more fine-grained design space involving what to train, what to communicate, where to execute, and what knowledge should be shared. Early efforts primarily exploit parameter-efficient adaptation and communication compression, whereas more recent studies increasingly consider data quality, client heterogeneity, and collaborative model execution in a unified manner. This evolution suggests that federated LLM optimization is no longer merely an aggregation problem, but rather a joint optimization problem across model parameters, data, communication, computation, and privacy. 4 Large Language Model-enhanced Federated Learning In LLM-enhanced FL, LLMs serve as knowledge auxiliary tools to improve the performance and automation level of conventional federated learning. In this context, the model being federated is not necessarily a large language model itself, but may instead be a classification model, a vision model, or another lightweight model. LLMs can participate in local data generation, data augmentation, automatic annotation, and pseudo-label construction to alleviate issues of insufficient client data and class imbalance. They can also transfer knowledge to federated models through knowledge distillation, feature transfer, and model initialization. Furthermore, LLMs can assist with client selection, aggregation strategy design, hyperparameter configuration, training log analysis, anomalous update detection, and privacy risk interpretation, thereby improving the optimization efficiency, manageability, and intelligence level of federated learning systems. As summarized in Table 2, the role of large pretrained models in federated learning is gradually evolving from being directly federated toward serving as reusable knowledge priors for lightweight collaborative adaptation. Accordingly, the main optimization target shifts from complete model parameters to prompts, adapters, low-rank modules, and compressed updates. This evolution further relaxes several conventional assumptions of federated learning. In particular, clients are no longer necessarily required to optimize the same parameter space or even possess the full foundation model. Instead, pretrained knowledge can be accessed through parameter-efficient adaptation, off-site tuning, lightweight emulators, or domain-specific instruction tuning. Consequently, LLM-enhanced federated learning increasingly becomes a joint design problem involving knowledge reuse, parameter efficiency, model accessibility, data quality, and communication efficiency. Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

15

Table 2. Research trajectories of large language model-enhanced federated learning.

Research

Representative

Trajectory

Works

Core Idea

Main Challenges

Foundation

PROMPTFL [18],

Exploit the transferable knowledge

How to leverage strong pretrained

Model-Assisted

FedPrompt [61]

encoded in pretrained foundation

knowledge to reduce

Federated

models while freezing the backbone

communication and optimization

Adaptation

and collaboratively optimizing only

costs while retaining sufficient

lightweight prompt parameters.

adaptability to heterogeneous and highly specialized downstream tasks.

PEFT-Based

FederatedScope-

Apply parameter-efficient

How to select and configure PEFT

Practical

LLM [25],

fine-tuning techniques, including

mechanisms that provide a

Federated

Malaviya et al.

LoRA [21], Prefix-Tuning, Adapters,

favorable trade-off among model

Learning

[35]

BitFit [58], and Prompt-Tuning, to

utility, communication efficiency,

reduce the trainable and

client-side resource consumption,

communicated parameter space.

and robustness to non-IID data.

Model-Access-

Relax the assumption that each

How to simultaneously protect

Constrained

FedBiOT [51]

client must access or deploy the full

model intellectual property, reduce

and Off-Site

foundation model. The original

client-side computation and

Federated

model is represented through

memory requirements, and preserve

Tuning

lightweight emulator and adapter

adaptation quality under

components, while client-side

distribution mismatch between

adapter optimization is coordinated

server-side public data and

with server-side emulator

client-side private data.

distillation through bi-level optimization. Data- and

FIT with Feature

Combine parameter-efficient

How to exploit pretrained

Application-

Diversity [8]

federated instruction tuning with

knowledge and diversified training

Aware

systematic augmentation of

data to improve downstream

LLM-Assisted

instruction-following data.

generalization while preserving

Federated

Foundation-model knowledge and

privacy and avoiding excessive

Learning

feature diversity are exploited to

dependence on application-specific

compensate for scarce annotations.

data augmentation strategies. Continued on next page

Manuscript submitted to ACM

16

Q.Yang C.Qiu, H.Zhang et al. Table 2 (continued)

Research

Representative

Trajectory

Works

Core Idea

Main Challenges

Communication- FLM-TopK [40]

Further compress PEFT updates

How to further reduce

Aware

through joint gradient sparsification

communication after the trainable

Lightweight

and quantization. Intervalized TopK

parameter space has already been

Adaptation

sparsification reduces not only the

substantially reduced by PEFT,

number of transmitted gradient

while controlling the optimization

values but also the position-index

errors introduced by sparsification

overhead associated with

and quantization.

conventional TopK compression.

4.1

Using LLMs to Empower Federated Learning Settings

Motivated by the severe communication overhead, slow convergence, data scarcity, and non-IID sensitivity of classical FL, Guo et al. [18] leverage the strong generalization capability of large pretrained vision–language models (e.g., CLIP) and propose a paradigm shift in FL, named PROMPTFL. PROMPTFL allows distributed clients to collaboratively optimize only a small set of continuous (soft) prompt parameters on top of a frozen foundation model instead of conventional full-model training. This design dramatically reduces trainable parameters and communication cost while preserving privacy, as clients share only prompt updates rather than raw data or full model gradients. Theoretically, PROMPTFL can reach a convergence rate of 𝒪(1/√𝑇 ) under the setting where partial clients are selected at each round, and local data distributions are Non-IID. Extensive experiments across diverse vision benchmarks demonstrate that PROMPTFL achieves competitive or superior accuracy and robustness. Particularly in heterogeneous and datascarce scenarios, it uses orders of magnitude fewer parameters and fewer communication rounds than FedAvg and fine-tuning-based baselines. Its performance depends on the availability of strong foundation models and may be less effective for highly specialized tasks. FedPrompt [61] extends parameter-efficient learning to the federated setting by integrating soft prompt tuning with federated learning for large pre-trained language models (PLMs). Considering the prohibitive communication and memory costs of full-model fine-tuning in FL, FedPrompt freezes the PLM backbone and collaboratively trains only a small set of soft prompt parameters using a split-aggregation strategy, reducing communication overhead to approximately 0.01% of the original model size with minimal accuracy degradation. The method is compatible with standard FedAvg and supports multiple prompt paradigms (e.g., prompt tuning, P-tuning, prefix-tuning). Extensive experiments on diverse NLP tasks under both IID and Non-IID data distributions demonstrate that FedPrompt achieves competitive performance compared to full fine-tuning while being significantly more communication-efficient. Furthermore, robustness analyses show that FedPrompt is resistant to backdoor attacks based on data poisoning, as prompt aggregation mitigates malicious updates, and optional LDP can be incorporated to further enhance privacy at the cost of moderate accuracy loss. Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

17

4.2 Prompt Adapter for Federated Learning FederatedScope-LLM (FS-LLM) [25] is a comprehensive, open-source framework designed to bridge the gap between general-purpose FL systems and the unique requirements of federated fine-tuning of LLMs. Motivated by the prohibitive communication cost, computation burden, data heterogeneity, and model intellectual-property constraints inherent to LLMs, FS-LLM provides an end-to-end solution that integrates benchmarking, algorithm support, and system-level optimization. It offers a standardized federated LLM benchmarking pipeline with curated datasets and evaluation tasks, enabling fair comparison across domains such as code generation, general language understanding, and reasoning. To address efficiency, FS-LLM incorporates a rich suite of parameter-efficient fine-tuning (PEFT) methods (e.g., LoRA, P-/prompt-tuning) and supports federated offsite tuning for scenarios where clients cannot access full models, achieving orders-of-magnitude reductions in communication while maintaining competitive performance. At the system level, FS-LLM integrates accelerating and resource-efficient operators (mixed precision, compression, DeepSpeed, offloading) and flexible training modes, making federated LLM fine-tuning feasible on limited hardware. Malaviya et al. [35] investigate how parameter-efficient fine-tuning (PEFT) can make FL with pretrained language models (PLMs) practical under strict privacy constraints and limited communication budgets. Motivated by the prohibitive cost of transmitting full PLM parameters and the frequent lack of labeled client data, the authors systematically evaluate four PEFT methods: Prefix-tuning, Adapters, BitFit, and LoRA, under both supervised FedAvg and a more realistic semi-supervised scenario (FedFAME). Only a small subset of trainable “𝛿 ” parameters is exchanged between clients and server while the PLM backbone stays frozen, reducing communication from about 420 MB to below 50 MB, and in some cases under 1 MB. Experiments on six GLUE tasks under varying non-IID conditions show PEFT methods achieve competitive performance, with Prefix and Adapter tuning generally outperforming BitFit and LoRA, and FedFAME exhibiting substantially higher robustness than FedAvg under severe heterogeneity (𝛼 = 0.1). The paper further demonstrates strong task-level transferability of PEFT parameters, particularly between semantically related tasks, enabling effective zero-shot adaptation when target-task labels are scarce. While the approach assumes the frozen PLM fits on client devices and is sensitive to hyperparameters like learning rate, the work provides compelling evidence that PEFT is a viable, scalable foundation for communication-efficient, privacy-preserving federated NLP. To relieve the two critical bottlenecks in federated LLM fine-tuning: clients’ lack of access to full model parameters and prohibitive computation/communication costs, Wu et al. [51] propose FedBiOT, a resource-efficient, privacypreserving framework that avoids full-model exposure. Building on the off-site-tuning paradigm, FedBiOT compresses the LLM into an emulator, aligned server-side on public data to mimic the original model, and an adapter, which captures domain-specific knowledge and is fine-tuned locally by clients using lightweight LoRA modules. To handle the distribution mismatch between server-side public data and client-side private data, the authors formulate a bi-level optimization problem alternating between client-side adapter updates and server-side emulator distillation. Experiments on LLaMA-2 across math problem solving, code generation, and question answering under both IID and non-IID settings show FedBiOT consistently outperforms offsite-tuning and FedOT, achieving higher accuracy and robustness while cutting client-side computation by up to 50% and avoiding transmission of full model parameters. Though it incurs slightly higher communication overhead than prior offsite methods due to emulator updates, this cost remains negligible compared to full LLM transfer. Chen et al [8] propose federated instruction tuning (FIT) with a complementary feature diversity (FD) strategy to enable privacy-preserving and data-efficient deployment of large language models for IoT-based autonomous driving. Motivated by the scarcity of high-quality annotated driving data and stringent privacy constraints in vehicular IoT Manuscript submitted to ACM

18

Q.Yang C.Qiu, H.Zhang et al.

environments, FIT leverages federated learning to collaboratively fine-tune only lightweight adapter parameters of a frozen multimodal LLM, thereby substantially reducing communication and computational overhead while preserving sensitive local data. To further address limited scenario coverage and improve generalization, the FD strategy systematically augments instruction-following data along key dimensions, including time, weather, and occlusion, enriching both visual and textual diversity and enhancing robustness in open-world driving conditions. Extensive experiments on the DriveLM benchmark using LLaMA-Adapter backbones and multiple FL algorithms demonstrate that FIT combined with FD, consistently outperforms local training and achieves strong gains in accuracy, language quality, and semantic alignment, while synchronizing only a tiny fraction of model parameters. 4.3

Prompt Tuning for Federated Learning

Qiu et al. [40] solve the substantial communication overhead incurred when exchanging high-dimensional LoRA updates during federated large language model fine-tuning, particularly because conventional TopK sparsification requires transmitting position identifiers for every retained gradient. To this end, the authors propose FLM-TopK, a communication-efficient framework that partitions gradients into multiple intervals and independently applies TopK sparsification within each interval, thereby reducing the number of bits required to encode gradient positions. Its core design models both original and intervalized fine-tuning gradients using a zero-mean Gaussian distribution and formulates compression as a joint optimization of interval size, packet allocation, and interval-specific sparsification rates under a fixed communication budget. Because the resulting optimization problem is non-convex, the authors decompose it into alternating convex subproblems and combine sparsification with unbiased quantization to balance discarded-gradient and quantization errors. Nevertheless, the formulation relies on an empirically derived Gaussiangradient assumption and a fixed per-round communication budget, while the evaluation adopts a single LoRA rank, simulated participation of 10 out of 100 clients, and limited network-level measurements. 5 The Synergistic Integration of FL and LLMs The synergistic integration of FL and LLMs no longer exhibits a one-directional service relationship but instead jointly forms a distributed intelligent system with bidirectional interaction capabilities. As summarized in Table 3, the synergistic integration of federated learning and large language models extends collaboration beyond conventional modelparameter aggregation. The objects of federation progressively evolve from shared model components toward personalized prompts, prompt mixtures, cross-task adapters, semantic representations, and black-box optimization statistics. This evolution also relaxes several assumptions commonly adopted in traditional federated learning. Participants are no longer necessarily required to maintain identical model parameterizations, perform the same downstream task, or possess white-box access to the underlying foundation model. Instead, collaboration can occur at the level of adaptation modules, semantic concepts, multimodal task knowledge, or prompt evaluation information. Consequently, synergistic FL–LLM systems exhibit a broader transition from parameter-level federation toward knowledge-level federation, where the central problem is no longer merely how to aggregate distributed updates, but how to identify, align, personalize, and securely transfer useful knowledge across heterogeneous participants.

Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

19

Table 3. Research trajectories of the synergistic integration of FL and LLMs. Research

Representative

Trajectory

Works

Core Idea

Main Challenges

Personalized Foundation-

FedPerfix [42],

Personalization can further be generated

How to retain transferable knowledge

pFedPG [54]

from client updates through a server-side

from the shared foundation model while

Model

prompt generator instead of relying on a

providing communication-efficient

Adaptation

single globally aggregated adaptation

personalization for clients with severely

module.

non-IID data, limited local samples, and heterogeneous resource constraints.

Mixture-Based

Model personalized adaptation either as a

How to benefit from cross-client

and Global–Local FedPGP [11]

pFedMoAP [33],

mixture of local and non-local prompt

knowledge transfer without sacrificing

Prompt

experts or as a decomposition of a

local specialization or degrading the

Personalization

globally shared prompt and a locally

generalization capability inherited from

retained low-rank component.

pretrained foundation models.

Domain-

PLAN [17],

Use lightweight prompts as a medium for

How to jointly balance personalization,

Generalized and

DP-FPL [44]

transferring domain knowledge across

cross-domain generalization, and privacy,

Privacy-Aware

clients and further integrate personalized

since aggressive local adaptation may

Prompt Learning

prompt adaptation with low-rank

impair transferable knowledge while

decomposition and DP to support

stronger privacy protection may

unseen-domain generalization and formal introduce substantial utility degradation. privacy protection. Task- and client-specific adapters extract

How to enable beneficial knowledge

Task-

specialized knowledge, while cross-task

sharing among clients performing

Heterogeneous

adapter mixtures and task-aware

different multimodal tasks while

Federated

aggregation promote selective knowledge

suppressing negative transfer caused by

Adaptation

transfer.

task, modality, and data heterogeneity.

Multimodal and

Pilot [52]

How to construct globally useful prompt

Semantic and

PFPT [50], Wang et Move beyond position-wise parameter

Robustness-

al. [49],

averaging by aligning local prompts at the representations when local prompts may

Oriented Prompt

FOCoOp [28]

semantic or concept level, systematically

be semantically misaligned, client

characterizing aggregation behavior

distributions differ, domain shifts occur,

under heterogeneous client distributions,

and previously unseen distributions must

and explicitly optimizing both

be handled at inference time.

Aggregation

in-distribution performance and out-of-distribution robustness. Continued on next page

Manuscript submitted to ACM

20

Q.Yang C.Qiu, H.Zhang et al. Table 3 (continued)

Research

Representative

Trajectory

Works

Black-Box

FedPOB [31]

Core Idea

Main Challenges

Bandit optimization[1], preference

How to achieve query-efficient,

Collaborative

feedback, summary-statistic aggregation,

communication-efficient, and

Prompt

and event-triggered communication are

privacy-preserving collaborative prompt

Optimization

used to coordinate black-box prompt

optimization when the underlying

search.

foundation model is proprietary and only black-box access and limited feedback are available.

Privacy, Security,

FL-GLM [62],

Combine split learning, encryption, TEEs, How to strengthen privacy and robustness

and Trusted

Huang et al. [23],

differential privacy (DP), secure

without introducing prohibitive

Execution

ROFED-LLM [48]

aggregation, model pruning, and

computation, communication, or system

communication-layer defenses to protect

complexity.

federated LLM training against information leakage and adversarial attacks.

5.1

Federated pruning learning for personalized model learning

FedPerfix [42] advances personalized federated learning (PFL) for vision transformers by systematically addressing where and how to personalize ViT models under heterogeneous data. Through an empirical sensitivity analysis, the authors identify the self-attention layers and classification head as the most distribution-sensitive components of ViTs, motivating partial personalization rather than full local training. Building on this insight and drawing an analogy to transfer learning, FedPerfix introduces prefix-based personalization with parallel attention, where lightweight, clientspecific prefix plugins adapt the global self-attention representations while the backbone remains globally aggregated. Yang et al. [54] propose pFedPG, a personalized federated learning framework that uses client-specific prompt generation to efficiently adapt large pretrained foundation models under severe data heterogeneity, communication constraints, and limited client resources. Instead of sharing or averaging full model parameters, pFedPG freezes the backbone (e.g., ViT) and alternates between local personalized prompt adaptation and server-side personalized prompt generation, where a lightweight cross-attention-based generator infers client-specific optimization directions from prompt updates and produces customized prompts for each client. This design addresses key challenges of federated learning with foundation models—performance degradation under non-IID data, excessive communication cost, and overfitting with limited local data—while retaining strong representational power. For the personalization issues under severe data heterogeneity in federated prompt learning, Luo et al. [33] argue that distributing only a single globally aggregated prompt fails to fully exploit the lightweight nature of prompt parameters. Hence, the authors propose personalized federated mixture of adaptive prompts (pFedMoAP) that reformulates locally trained prompts as specialized experts and lets clients download multiple pre-aggregated prompts from other participants as fixed non-local experts. The design maintains a server-side prompt pool and uses prompt-space Knearest-neighbor selection to assign relevant non-local experts to each client; a client-specific attention-based gating Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

21

network then takes image features as queries and local/non-local text features as keys and values to produce instanceadaptive text representations. Final predictions combine the mixture-enhanced representation with the locally trained prompt, while only the local prompt is uploaded and the gating network stays private. Ablations show jointly retaining local and mixture-based logits improves performance, moderate local-prompt weighting generally suffices, and more experts help but with diminishing returns. Gating-feature dimensions of 128–256 balance representation capacity and parameter efficiency. However, the method requires a server-side expert pool, multiple prompt downloads, and a client-specific gating network. Thus, communication, storage, and computation shall grow with expert count despite prompts being lightweight. In federated prompt learning, the personalization and generalization are hard to balance. Because a shared prompt may inadequately represent heterogeneous client distributions while aggressive local adaptation erodes the transferable knowledge of pretrained vision-language models. To this end, Cui et al. [11] design FedPGP, a framework that preserves CLIP’s generalization while adapting the global prompt to client-specific data, representing each personalized prompt as the sum of an aggregated global prompt and a locally retained low-rank adaptation term. FedPGP further introduces a prompt-wise contrastive objective treating global and handcrafted CLIP prompts as positive pairs to preserve category-agnostic knowledge, while treating global and personalized prompts as negative pairs to encourage client-specific representations; only the global prompt is communicated and aggregated, while low-rank adaptation parameters remain local. Ablations confirm this design: full-rank adaptation improves local accuracy but substantially degrades base-to-novel generalization, while removing either contrastive relation reduces overall performance. Nevertheless, the framework is evaluated only on CLIP-based image classification, relies on fixed prompt and bottleneck dimensions, and lacks a theoretical explanation for why low-rank adaptation achieves the observed generalization– personalization trade-off. Federated domain generalization means that models trained collaboratively on heterogeneous source domains must generalize to unseen domains without exposing sensitive client data. The conventional approaches exchange domain knowledge directly derived from local samples such as feature statistics, class prototypes, or frequency information, which might bring privacy risks. Therefore, Gong et al. [17] propose Prompt Learning and AggregatioN (PLAN), which utilizes locally learned prompts as a parameter-efficient and potentially less data-revealing medium for cross-client knowledge transfer. PLAN lies in combining reference-guided local prompt learning with attention-based multimodal prompt aggregation to improve unseen-domain generalization without exchanging raw data or directly extracted domain statistics. It follows a two-stage procedure each round. First, every client learns multi-layer text and visual prompts on local data while aligning predictions with the previous round’s global prompts via a KL-divergence regularizer, indirectly synchronizing client prompts and reducing local-domain overfitting. Second, lightweight attentionbased aggregators estimate the relative importance of local prompts and selectively combine them into global text and visual prompts, rather than relying on fixed or uniform averaging. Only prompt tokens and aggregator parameters are optimized, while the pretrained CLIP backbone stays frozen. However, the privacy advantage rests mainly on the indirect nature of prompt learning rather than formal guarantees or leakage analysis. Distributing all local prompts to clients and performing two communication phases per round may also constrain scalability as participant numbers grow, and evaluation is restricted to CLIP-based image classification with a shared label space between source and target domains. While Tran et al. [44] investigate the competing requirements of personalization, generalization, and privacy in FPL for multimodal LLMs. Because personalized prompts can overfit heterogeneous local data and generalize poorly to unseen inputs, while differentially private training may substantially reduce both local and cross-domain performance. Manuscript submitted to ACM

22

Q.Yang C.Qiu, H.Zhang et al.

Thus, Tran et al. proposes DP-FPL, a differentially private personalized federated prompt-learning framework designed to protect prompts against membership-inference attacks while preserving their utility. In DP-FPL, each personalized prompt combines the global prompt with a local component that is iteratively decomposed into two low-rank factors and a residual term. The low-rank factors restrict the optimization space and improve generalization and robustness to privacy noise, whereas the residual compensates for information lost during factorization and preserves personalized expressiveness. Unlike approaches that perform factorization only once, DP-FPL [44] repeats the decomposition during every training iteration and reconstructs the local-prompt gradient from the low-rank gradients. Privacy is enforced through a hybrid mechanism: LDP and global differential privacy (GDP). Both mechanisms employ clipped gradients and Gaussian perturbation, with the noise levels selected to satisfy formal (𝑒𝑝𝑠𝑙𝑖𝑜𝑛, 𝛿 )-LDP and GDP guarantees. The residual is used in the forward computation but is not directly perturbed because it is excluded from the local-prompt gradient reconstruction. The results show that the residual term benefits both local and neighboring-class performance, particularly under low-rank and high-noise conditions. They also indicate that moderate privacy noise may act as a regularizer and improve generalization, although excessive noise eventually degrades overall utility. Nevertheless, the evaluation is limited primarily to CLIP-based visual classification, a small set of benchmark datasets, and simulated pathological or Dirichlet client distributions. Moreover, the claimed regularization effect and protective role of the residual term are largely supported empirically rather than through a complete theoretical account. 5.2

Multimodal Fusion through LLMs and FL

Pilot [52] introduces the federated multimodal instruction tuning (FedMIT) task and provides a principled framework for collaboratively fine-tuning multimodal LLMs under severe task and data heterogeneity. Unlike prior federated instruction tuning approaches that focus on unimodal NLP tasks, FedMIT addresses the more challenging setting where clients perform different multimodal tasks (e.g., VQA [32], captioning, and visual grounding) on decentralized data. To tackle this heterogeneity, Pilot adopts a two-stage “adapter-on-adapter” design within the vision–language connector: i) a task-specific and client-specific feature mining stage that disentangles task-relevant and personalized visual representations via orthogonality constraints, and ii) a cross-task interaction stage that introduces a cross-task mixture-of-adapters (CT-MoA) module to selectively integrate knowledge from heterogeneous tasks. On the server side, Pilot further proposes task-aware aggregation for visual adapters and an adaptive distance-based aggregation strategy for text adapters, mitigating negative interference across clients. In federated prompt tuning under non-IID, locally learned prompt sets may encode concepts in arbitrary orders, causing position-wise averaging to combine semantically unrelated prompts into an uninformative global representation. Therefore, Weng et al. [50] propose probabilistic federated prompt tuning (PFPT), which reframes federated aggregation as a distributed set-modeling problem rather than conventional parameter averaging. PFPT models each client’s prompts as samples from a hierarchical generative model parameterized by a server-maintained set of global summarizing prompts. Each round, a client selects a relevant subset of global prompts via a Bernoulli point-process model and fine-tunes them locally with a frozen pretrained vision transformer and a personalized classification head, with resulting local prompts modeled as Gaussian perturbations around their associated summarizing prompts. The server then infers latent local–global prompt associations and aggregates prompts encoding similar contextual information, aligning at the concept level rather than by prompt position. The resulting mixed discrete–continuous problem is solved via alternating optimization: (1) Generative-model and summarizing-prompt parameters are updated by gradient descent given associations; (2) Association inference reduces Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

23

to weighted bipartite matching solved via the Hungarian algorithm (cubic complexity in the maximum number of local prompts matched) given parameters. The global prompt pool is non-parametric, unused prompts are pruned. While heterogeneous clients can expand it to represent new concepts. Nevertheless, evaluation is limited to vision classification with a pretrained ViT under largely simulated heterogeneity and imbalance. In addition, matching cost and pool size may grow with client diversity. The prompt capacity needed for larger pretraining and downstream domain shifts is neither quantified nor theoretically bounded. Meanwhile, despite reduced communication relative to full-model fine-tuning, prompt tuning still requires storing intermediate gradients through the frozen backbone, imposing non-trivial client-side memory overhead. Rather than proposing a new aggregation algorithm for federated vision-language model adaptation, Wang et al. [49] provide a systematic empirical analysis of federated prompt learning under label skew, domain shift, and their coexistence, examining sensitivity to communication rounds, aggregation strategies, client scale, and prompt length. Using a frozen CLIP model with a ViT-B/16 encoder, the study federatively trains textual or visual prompts via sample-weighted aggregation, evaluated on CIFAR-100 (Dirichlet-based label skew) and Office-Home, Office31, and DomainNet (domain shift). The authors also analyze similarity of optimization directions across clients and between local and global prompts to explain the distinct behaviors of LPT and VPT. Results show FPL is relatively stable across communication rounds, aggregation rules, client counts, and prompt lengths. Equal aggregation performs slightly better under label skew, while sample-weighted aggregation is preferable under domain shift; more clients modestly reduce performance under label skew but show no consistent effect under domain shift, and longer prompts yield only incremental gains at added computational cost. However, evidence is confined to CLIP-based image classification with a single ViT-B/16 backbone, four datasets, simulated heterogeneity, fixed configurations, and one random seed; evaluation mainly compares internal prompt variants rather than benchmarking against recent federated prompt-learning methods, and the optimization-directionsimilarity explanation remains empirical, without establishing a causal or theoretical link to generalization. Considering the trade-off between predictive performance and out-of-distribution (OOD) robustness in federated prompt learning for vision-language models, Liao et al. [28] design FOCoOp, a federated OOD-aware context optimization framework jointly modeling global in-distribution prompts, client-specific local prompts, and OOD prompts. At each client, Bi-level OOD Separations (BOS) optimizes the three prompt sets to establish class-level separation between categories and distribution-level separation between ID and OOD samples. It employs bi-level distributionally robust optimization to perturb global and OOD prompts within optimal-transport uncertainty sets, thereby exposing the model to challenging prompt distributions without requiring real OOD training images. At the server, Globalview OOD Consistency (GOC) aligns aggregated global prompts with client-generated OOD prompts through semiunbalanced optimal transport. OOD prompts that closely resemble global ID prompts are used to calibrate the latter, whereas distant prompts are retained and redistributed as globally consistent OOD references. FOCoOp consistently achieves the strongest overall balance between classification and OOD robustness. Under the ten-client pathological setting, it reaches 93.85% ID accuracy, 91.47% covariate-shift accuracy, 19.50% FPR95, and 95.42% AUROC on CIFAR-100, and 88.35% ID accuracy, 83.56% covariate-shift accuracy, 21.73% FPR95, and 96.56% AUROC on TinyImageNet. Ablations confirm BOS and GOC are complementary, with excluding GOC generally causing larger degradation, underscoring the importance of globally consistent ID–OOD discrimination. Similarity visualizations further show strong within-class image prompt alignment versus weaker ID–OOD prompt matching.

Manuscript submitted to ACM

24

Q.Yang C.Qiu, H.Zhang et al. Lu et al. [31] study black-box prompt optimization for large language models under three practical constraints:

expensive model queries, limited access to proprietary model parameters, and the need for privacy-preserving collaboration among multiple users. The authors propose federated prompt optimization via bandits (FedPOB) and a preference-based extension, FedPOB-Pref, which allow multiple agents to search for effective discrete prompts without sharing their local prompt-evaluation histories. FedPOB encodes discrete prompts using a pretrained text encoder and models prompt performance with a linear reward function. Each agent selects prompts using a LinUCB explorationexploitation policy and periodically sends summary statistics to a central server, which aggregates and redistributes this information. Communication is event-triggered, meaning synchronization only happens once newly collected information passes a preset threshold. FedPOB-Pref extends this idea to pairwise feedback using federated linear dueling bandits: it picks one prompt for exploitation and another for exploration, learns from Bradley-Terry-Luce preference comparisons, and uses dynamic regularization to reduce optimization drift across heterogeneous agents. That said, the framework assumes all agents are optimizing prompts for the same task, with heterogeneity captured only through different candidate prompt spaces. Hence, it is unclear how well it would handle agents with different tasks or conflicting preferences. The methods also assume a linear relationship between pretrained prompt embeddings and rewards, and FedPOB-Pref further assumes a Bradley-Terry-Luce preference model; both assumptions may not hold for more complex, non-linear prompt-performance relationships. In addition, preference feedback is simulated from validation scores rather than gathered from real users, so it remains unclear how robust the method is to noisy or manipulated preferences. Finally, while raw prompts and evaluation histories are kept private, agents still send learned statistics to a trusted central server, and the paper does not offer DP, secure aggregation, or any analysis of potential information leakage. Answers to RQ2: FPL approaches exhibit systematic trade-offs across performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling, with no single method dominating on all dimensions simultaneously. The survey’s unified benchmark (Section 7)shows that LoRA achieves the highest and most stable accuracy across IID and non-IID settings but incurs substantially larger communication costs, whereas prompt tuning minimizes communication and trainable parameters at the expense of lower and more variable accuracy. Notably, P-tuning’s larger parameter count does not translate into greater stability, indicating that parameterization structure matters more than raw scale for robustness under heterogeneous optimization. Methods that improve personalization through richer mechanisms (e.g., multi-expert prompt mixtures or global-local prompt decomposition) gain adaptability at the cost of increased communication, storage, and computation. While aggressive local adaptation risks eroding the generalization inherited from pretrained backbones, a tension partially mitigated through low-rank restrictions and contrastive regularization. Similarly, naive position-wise prompt averaging often fails under non-IID data due to semantic misalignment across clients, motivating concept-level or attention-based aggregation strategies that improve heterogeneity handling but introduce additional computational overhead and remain validated mainly under simulated conditions. 6 Security and Privacy Issues in FPL 6.1

Security Issues in FPL

Safety alignment can be severely compromised by a simple, low-cost, and highly stealthy data poisoning attack in federated instruction tuning (FedIT) of LLMs [57]. Unlike traditional FL poisoning attacks, the proposed attack does not introduce conflicting optimization signals. Instead, malicious clients are trained on harmful instruction: response Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

25

pairs. These pairs remain optimization-consistent with benign training data. As a result, the attack exploits the intrinsic alignment objective of instruction tuning. This makes existing model-level FL defenses largely ineffective. Extensive experiments demonstrate that such attacks can reduce safety metrics by up to 70%, while classical robust aggregation methods provide only marginal protection. To address this gap, the authors propose a post-hoc, server-side defense strategy that decouples safety recovery from client-side training by automatically generating aligned and normal instruction–response data and performing lightweight fine-tuning on the aggregated model. This defense is shown to be plug-and-play, scalable, and capable of restoring safety performance by up to 69% without significantly degrading helpfulness. Zheng et al. [62] propose FL-GLM, a secure and efficient split-learning-based federated learning framework for LLMs, addressing the incompatibility of conventional FL (e.g., FedAvg) with the heavy computation and privacy risks of large models. The framework places the embedding layer and the first and last transformer blocks on the client, keeping intermediate blocks on the server to prevent embedding-gradient leakage, while encrypting all client–server “smashed data” via public-key encryption to guard against peer-client eavesdropping. To mitigate split learning’s low efficiency, FL-GLM introduces client-batch and server-hierarchical parallelism, cutting training time by over 48% versus serial execution. While limited to ChatGLM-scale models and still incurring communication overhead, FL-GLM offers a practical blueprint showing that encrypted, parallelized split architectures can jointly reconcile utility, efficiency, and privacy in federated LLM training. Huang et al. [23] propose a fast, accurate, and security-preserving distributed training framework for LLMs that addresses parameter and data leakage risks in federated settings. Motivated by the limitations of prior defenses: DP’s accuracy loss and the heavy overhead of multi-party computation, the authors design a model slicing-based secure architecture integrating trusted execution environments (TEEs) with lightweight encryption, via two complementary schemes (i) Method 1; (ii) Method 2. Method 1 targets consumer-grade, small-memory TEEs (e.g., Intel SGX) by isolating sensitive fine-tuned components (LoRA or P-Tuning v2 embeddings) within client- and server-side TEEs, protecting intermediate activations exchanged with GPUs via one-time-pad encryption.Method 2 improves efficiency and accuracy further by splitting the LLM by layers, deploying deeper layers in a large-memory server-side TEE (Intel TDX/SGX) while freezing client-side layers. A novel sparsification parameter fine-tuning (SPF) strategy selectively updates key attention heads and combines LoRA for MLP layers. FedShield-LLM [36] is a secure, scalable federated fine-tuning framework for LLMs that addresses the trade-off between strong privacy protection and high model utility in cross-silo FL. Motivated by the vulnerability of standard federated LLM fine-tuning to inference attacks (e.g., gradient inversion, membership inference) and the performance degradation caused by noise-based defenses like DP, FedShield-LLM integrates PEFT (LoRA) with unstructured pruning and Fully Homomorphic Encryption (FHE). Clients fine-tune only low-rank LoRA adapters on local data, prune less informative adapter weights to reduce attack surface and communication cost, and encrypt updates via the CKKS scheme, enabling the server to aggregate encrypted parameters without accessing plaintext. The authors provide theoretical convergence guarantees, showing the encrypted, sparsified FedAvg procedure retains sublinear convergence, and formal security arguments demonstrating resistance to gradient inversion and inference attacks under standard cryptographic assumptions. Experiments on LLaMA-2 (7B and 13B) across medical, financial, mathematical, and general instruction-tuning benchmarks demonstrate FedShield-LLM consistently outperforms Vanilla FL and DP-LoRA in convergence speed, training stability, and text quality, achieving BERTScore gains while approaching the response quality of GPT-4-class models. Manuscript submitted to ACM

26

Q.Yang C.Qiu, H.Zhang et al. However, in federated graph learning, the participants may differ not only in graph structures and feature distribu-

tions but also in downstream tasks, including node-, edge-, and graph-level prediction. To address this multifaceted heterogeneity in federated graph learning, Guoet al. [19] propose FedGPL, an asymmetric federated graph promptlearning framework that separates universal graph representation learning from client-specific prompting and prediction. FedGPL integrates a server-hosted GNN with locally maintained graph prompts and task heads. Meanwhile, the hierarchical directed transfer aggregator (HiDTA) estimates directional transferability between clients and performs personalized aggregation to encourage beneficial cross-task knowledge transfer while suppressing potentially harmful contributions. The Virtual Prompt Graph (VPG) adaptively adds informative virtual structures and removes or neutralizes redundant nodes and edges, thereby preserving domain-specific patterns while reducing graph-data divergence. FedGPL additionally employs split learning and differential-privacy noise to protect the graph embeddings and gradients exchanged between clients and the server. The principal contribution therefore lies in jointly addressing graph-task and graph-data heterogeneity through asymmetric transferability-aware aggregation and adaptive structural prompting, while retaining parameter and communication efficiency. The accompanying theoretical analysis provides conditional support for heterogeneity reduction: HiDTA requires positive bidirectional transferability, whereas the VPG analysis assumes particular graph-representation distributions and similar prompting ratios across clients. Nevertheless, the empirical evaluation is restricted to five benchmark graph datasets, generally uses three clients per task level, assumes full participation and one local training step per round, and adopts a semi-honest threat model. Moreover, stronger privacy constraints reduce predictive accuracy, indicating a non-trivial privacy–utility trade-off. Khan et al. [24] address the largely unexplored security risks of FPL, showing that communication-efficient prompt updates remain vulnerable to targeted backdoor attacks despite a frozen vision-language backbone. It introduces a learnable, visually imperceptible noise trigger that malicious clients inject into local images to shift their CLIP representations toward an attacker-chosen target class, preserving accuracy on clean inputs while causing systematic misclassification of triggered samples. To counter this, the paper proposes selective and accurate backdoor rejection for federated prompt learning (SABRE-FL), a lightweight server-side defense that detects and removes poisoned client updates via embedding-space anomaly detection. The attack adapts the centralized BadCLIP mechanism to a federated, prompt-only setting: each compromised client jointly optimizes a local prompt and an additive noise trigger, relabeling triggered images to a fixed target class. Although the CLIP encoders stay frozen, the trigger shifts image embeddings toward the target-class text representation, and the resulting malicious prompt updates propagate through FedAvg aggregation. Under the default threat model, the adversary controls 25% of clients, knows the public backbone and training process, and can modify local data and labels but not server aggregation directly. The theoretical argument also depends on triggered and clean embeddings maintaining a stable separation margin and on the auxiliary detector achieving near-zero training error, so robustness against adaptive attackers minimizing this separation, alternative trigger types, and direct model-poisoning attacks remains unestablished. The method further assumes a boundable number of malicious clients and is evaluated mainly with a single CLIP ViT-B/16 backbone, with the authors flagging model poisoning, broader threat models, and additional vision-language architectures as directions for future work.

Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

27

6.2 Privacy Issues in FPL Luo et al. [34] systematically examine FL as a privacy-preserving paradigm for fine-tuning LLMs in automated program repair (APR), motivated by the growing reliance of LLM-based repair methods on high-quality industrial code that cannot be centrally shared due to confidentiality constraints. Using a private industrial dataset (TutorCode) for training and the leakage-resistant EvalRepair-Java benchmark for evaluation, the authors investigate three core challenges: whether federated fine-tuning can effectively enhance repair performance, how code heterogeneity (feature-skewed Non-IID data) influences generative repair tasks, and how different FL algorithms impact optimization. By adopting parameter-efficient fine-tuning (QLoRA), the framework significantly reduces communication and memory overhead while preserving data locality. Extensive experiments across six state-of-the-art code LLMs show that federated finetuning consistently and substantially improves bug-fixing performance over local fine-tuning and often approaches, or even surpasses, centralized training, with gains of up to 16.6% in Top@10 and 18.4% in Pass@10. Surprisingly, heterogeneous code distributions do not degrade performance and can even enhance generalization, highlighting the robustness of LLMs in feature-skewed federated generative tasks. While FedAvg emerges as the most stable and effective aggregation strategy, personalized FL remains challenging for LLM-based repair. ROFED-LLM [48] is a robust, privacy-preserving federated learning framework for training LLMs in adversarial wireless environments, addressing the vulnerability of conventional federated LLM training to jamming attacks, device heterogeneity, and privacy leakage. It integrates split federated learning with a multi-modal defense spanning both model and communication layers: on the learning side, LLMs are partitioned between clients and server, with local differential privacy (LDP), secure aggregation, and dynamic parameter pruning reducing information leakage and computational burden; on the communication side, adaptive beamforming, jamming detection, and resource allocation optimization enable real-time mitigation of interference and adversarial attacks over wireless channels. The authors provide convergence guarantees and privacy analysis, showing ROFED-LLM achieves sublinear convergence while maintaining formal DP. Experiments across diverse natural language processing (NLP) benchmarks (e.g., general language understanding evaluation benchmark (GLUE) [47], SQuAD, WikiText-103, FinancialPhraseBank) show it consistently outperforms FedAvg, FedProx, SCAFFOLD, and SplitFed. ROFED-LLM prioritizes FedLLM methods with a 47% lower privacy leakage and significantly greater robustness under severe jamming, with only marginal communication overhead. 7 Experimental Benchmarking and Empirical Insights Besides of the systematic analysis for the existing work on the FPL topic, we conduct a unified empirical benchmark to complement the methodological taxonomy with reproducible quantitative evidence. A major difficulty in this area is that published results are often obtained under disparate task formulations, model backbones, data partitions, and evaluation protocols, rendering direct comparison unreliable. Therefore, rather than pursuing state-of-the-art performance, this benchmark evaluates representative federated PEFT strategies under a strictly consistent experimental protocol, thereby enabling a fair and controlled comparison across methods originally tested in heterogeneous settings. 7.1 Unified Benchmark Evaluation Table 4 reports the core benchmark results on AG News using TinyLlama-1.1B-Chat as the backbone. All methods were trained with four clients over 100 communication rounds and evaluated on the same held-out test set using the post hoc candidate-label scoring protocol. We report the mean and standard deviation over three random seeds. Manuscript submitted to ACM

28

Q.Yang C.Qiu, H.Zhang et al.

Table 4. Unified benchmark results of federated PEFT methods on AG News using TinyLlama-1.1B-Chat. Values are reported as mean ± standard deviation over three random seeds.

Setting

Method

Predictive Performance

Adaptation Scale

System Efficiency

Acc. (%) ↑

Test Loss ↓

Params (M) ↓

Trainable (%) ↓

Download (MB) ↓

FL Time (min) ↓

IID

LoRA Prompt tuning P-tuning

88.78 ± 0.63 82.57 ± 7.28 81.74 ± 1.89

0.091 ± 0.001 0.133 ± 0.048 0.115 ± 0.009

1.13 0.20 12.63

0.1023 0.0186 1.1351

5.04 0.11 0.39

76.80 ± 0.53 110.23 ± 0.84 82.46 ± 0.89

LDA 𝛼 = 0.1

LoRA Prompt tuning P-tuning

88.85 ± 0.12 83.43 ± 3.85 77.56 ± 13.24

0.093 ± 0.002 0.142 ± 0.049 0.151 ± 0.049

1.13 0.20 12.63

0.1023 0.0186 1.1351

5.04 0.11 0.39

78.55 ± 3.19 111.80 ± 5.85 89.19 ± 13.00

LDA 𝛼 = 0.5

LoRA Prompt tuning P-tuning

88.95 ± 0.35 83.10 ± 6.28 73.45 ± 18.28

0.091 ± 0.002 0.145 ± 0.077 0.116 ± 0.014

1.13 0.20 12.63

0.1023 0.0186 1.1351

5.04 0.11 0.39

86.95 ± 13.08 113.90 ± 6.48 82.23 ± 0.44

LDA 𝛼 = 1.0

LoRA Prompt tuning P-tuning

88.76 ± 0.08 79.69 ± 11.77 83.65 ± 3.63

0.094 ± 0.004 0.142 ± 0.054 0.116 ± 0.009

1.13 0.20 12.63

0.1023 0.0186 1.1351

5.04 0.11 0.39

78.94 ± 3.62 113.43 ± 3.91 89.56 ± 11.30

Test Loss is the server-side weighted-average test loss reported by the federated framework. Download denotes the average download volume per worker reported by the framework-level system monitor; the original values are converted to MB for consistent presentation. All experiments use four clients, full client participation, 100 communication rounds, and the same held-out evaluation set.

Several consistent observations emerge from the benchmark results. First, LoRA provides the most stable performance among the compared methods. Its accuracy remains around 88–89% across IID and all LDA-based non-IID settings, with consistently small standard deviations. These results suggest that LoRA is comparatively robust to the evaluated data partition changes, although it incurs a larger communication payload than prompt-based methods. Second, prompt tuning achieves the lowest trainable parameter ratio and the smallest average download volume, but its predictive performance is more sensitive to random seed and data partition. This suggests that prompt tuning provides a strong communication-efficiency advantage, but its optimization stability is weaker than LoRA under the current federated setting. Third, P-tuning does not consistently outperform prompt tuning despite using substantially more trainable parameters. In particular, it exhibits large variance under LDA 𝛼 = 0.1 and 𝛼 = 0.5, indicating that increasing the promptencoder parameter scale does not automatically translate into more stable federated adaptation. This observation is important because it suggests that the practical value of a parameter-efficient method should be judged jointly by accuracy, stability, trainable scale, and communication behavior rather than by parameter count alone. 7.2

Empirical Analysis of Performance, Robustness, and Reproducibility

Fig. 2 complements the quantitative results in Table 4 by visualizing the empirical characteristics of representative federated PEFT methods from three complementary perspectives, namely communication efficiency, robustness under heterogeneous federated settings, and reproducibility across random seeds. Several important empirical observations can be derived from the benchmark results. (1) The compared methods exhibit substantially different robustness profiles under federated heterogeneity. Among all evaluated methods, LoRA consistently demonstrates the strongest stability across IID and non-IID settings. Its task accuracy remains highly stable across different random seeds and LDA partition conditions, while its test loss also remains comparatively well-controlled. This behavior highlights that low-rank parameter adaptation provides a relatively robust optimization path for federated LLM fine-tuning, even when client distributions become heterogeneous. Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

(a)

(b)

29

(c)

Fig. 2. Comparison of representative federated PEFT methods under the unified benchmark from three complementary perspectives. (a) Performance–efficiency trade-off between predictive accuracy and communication cost. (b) Robustness under IID and progressively heterogeneous federated data partitions. (c) Reproducibility across random seeds, where each point denotes one method under one partition setting and is positioned according to its mean accuracy and cross-seed standard deviation. Error bars indicate the standard deviation over three random seeds.

(2) Communication-efficient methods do not necessarily preserve stable predictive behavior under federated optimization. Prompt tuning achieves the smallest trainable parameter ratio and the lowest communication volume among all evaluated methods. However, its performance variance becomes noticeably larger under certain non-IID settings, indicating that extreme parameter compression may increase optimization sensitivity in federated environments. In particular, several prompt-tuning runs show substantial seed-dependent fluctuations despite using the same backbone model and evaluation protocol. (3) Larger trainable parameter scales do not automatically guarantee stronger robustness. Although P-tuning introduces considerably more trainable parameters than prompt tuning, it exhibits the greatest instability across several heterogeneous settings, including occasional performance collapse with moderate non-IID partitions. This observation suggests that federated prompt learning performance is influenced not only by the scale of trainable parameters but also by how the parameterization interacts with heterogeneous local optimization dynamics. In summary, our benchmark demonstrates that federated PEFT methods must be evaluated as a multi-objective design problem rather than through predictive accuracy alone. Specifically, these findings suggest that communicationefficient adaptation should not be evaluated solely by parameter reduction; indeed, a strategy with superior parameter efficiency may suffer from heightened optimization instability, whereas a more stable method may incur substantially larger communication overhead. Consequently, practical federated deployment requires balancing these distinct tradeoffs—spanning effectiveness, optimization robustness, parameter footprint, and communication cost—all of which constitute complementary dimensions in the design space of federated prompt learning instead of being optimized in isolation. 8 Insights and Future Research Although we have reviewed many related studies, the integration of FL and LLMs remains at an early stage of maturity when judged against the recurring limitations acknowledged throughout the surveyed studies: simulated heterogeneity, small or moderate client populations, single-backbone evaluation, informally justified privacy claims, and fragmented experimental protocols. These limitations become sharper still as the field moves from static model adaptation toward Manuscript submitted to ACM

30

Q.Yang C.Qiu, H.Zhang et al.

federated agentic systems, where models act, retrieve, and communicate autonomously rather than simply producing text. Synthesizing these gaps, we organize future research into nine directions spanning evaluation, theory, systems, and increasingly central trustworthiness. 8.1

Standardized, Reproducible, and Multi-Objective Evaluation

As our unified benchmark in Section 7 demonstrates, methods with comparable accuracy can differ sharply in stability, communication cost, and seed-to-seed variance—dimensions that most published results, obtained under disparate backbones, partitions, and protocols, do not jointly report. Future work should move toward shared benchmark suites with multi-seed reporting and standardized non-IID partition generators, so that communication efficiency, robustness, and parameter footprint are treated as co-equal objectives rather than being subordinated to a single accuracy figure. 8.2

Theoretical Foundations for Federated Prompt and PEFT Optimization

Convergence guarantees remain largely confined to simplified regimes (e.g., PromptFL’s 𝒪(1/√𝑇 ) bound [18]), while empirical phenomena such as the generalization–personalization trade-off in FedPGP [11], the concept-drift dynamics in PFPT [50], or the ID–OOD separation margin in FOCoOp [28] are supported only empirically. A principled theory connecting prompt rank, capacity, and domain shift to generalization bounds under non-IID, nonlinear transformer dynamics is still missing, and would let practitioners choose prompt configurations analytically rather than by grid search. 8.3

Heterogeneity- and Resource-Aware Optimization at Realistic Scale

FlexLoRA [3], FAH-QLoRA [16], and Fed-HeLLo [60] all address resource heterogeneity, but under simulated conditions, moderate model sizes, and limited client counts. Real deployments will involve thousands of genuinely heterogeneous devices and clients pursuing different tasks—not merely different label or domain distributions—so validating these mechanisms on larger LLM backbones and authentic edge testbeds is a necessary next step. 8.4

LLMs as Intelligent Controllers of Federated Systems

Section IV’s LLM-enhanced FL largely covers data augmentation and pseudo-labeling; a less explored direction is using LLMs as adaptive controllers—for client selection, aggregation-weight design, hyperparameter tuning, and anomalousupdate detection—closing the loop between “FL for LLMs” and “LLMs for FL” within a single self-improving system. 8.5

Federated Agentic Ecosystems: Retrieval, Multi-Agent Collaboration, and Verifiable Trust

Federated RAG and federated multi-agent systems were introduced in Section I as part of the synergistic-integration category, yet neither is represented by a dedicated study in Section V-B, marking a clear literature gap. Open problems include collaboratively optimizing retrievers or knowledge representations across clients that each retain private knowledge bases, and designing protocols for LLM agents to share task experience or tool-use knowledge without exposing local memory or data. As such systems mature, trust cannot rest on single-round update verification alone: unlike the anomaly-detection defenses developed for prompt-only settings such as SABRE-FL [24] and FedGPL [19], an agent’s trustworthiness must be assessed over extended, multi-step interaction trajectories, where one manipulated tool call or retrieved document can silently corrupt downstream reasoning. Future work should therefore pair federated RAG and multi-agent architectures with trajectory-level verification mechanisms—potentially combining lightweight Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

31

cryptographic attestation, verifiable computation, or provenance logging—that certify agent behavior without requiring centralized visibility into private client data. 8.6 Privacy Protection Beyond Training: From Formal Guarantees to Inference-Time and Agentic Leakage Several methods justify privacy only indirectly—e.g., PLAN [17] argues that prompts are a compact, less data-revealing medium without formal leakage bounds, and SABRE-FL [24] requires exchanging raw embeddings without quantifying the resulting risk. Even DP-FPL [44], despite offering formal ( 𝑒𝑝𝑠𝑖𝑙𝑜𝑛, 𝛿) guarantees, evaluates them against a single shadow-model attack. This gap is sharpened by the theoretical result of Nikolaou et al. [37] that decoder-only Transformers are almost surely invertible, implying that exchanged hidden representations may be reconstructible in principle. The risk extends beyond training: agentic deployments introduce persistent memory, retrieved documents, and inter-agent messages exchanged at inference time, which can leak sensitive local context even when model weights stay private. Future work should derive leakage bounds specific to prompt-, embedding-, and memory-space exchange, and test defenses—privacy-preserving memory management, encrypted or federated retrieval—against adaptive, reconstruction-aware adversaries rather than fixed, known attacks. 8.7 Safety Alignment and Accountability in Federated and Agentic Deployments Ye et al. [57] show that safety alignment achieved during centralized instruction tuning can be substantially eroded by a small fraction of poisoned federated clients, with conventional robust aggregation providing only marginal protection. This risk is amplified in agentic settings, where a misaligned or jailbroken local model may issue harmful tool calls or propagate unsafe behavior to other agents, rather than merely producing an unsafe text response. Extending post-hoc safety-recovery strategies—such as the server-side realignment fine-tuning in [57], to agentic action spaces, and moving from one-time post-training checks to continuous, round-wise safety auditing, remain open problems. This challenge is compounded by an accountability gap: because no single party observes the complete federationwide interaction trace, attributing a faulty or harmful agent decision to a specific client, prompt, or aggregation step is substantially harder than in centralized settings, particularly in regulated domains such as healthcare (FedMRG [5]) and IoT management (Otoum [38]) already surveyed in this paper. Future research should explore lightweight, privacycompatible audit trails and explanation mechanisms that preserve data locality while still enabling post-hoc accountability and regulatory compliance. 8.8 Incentive Design and Real-World Deployment Most surveyed methods assume full or voluntary client participation, yet clients holding valuable proprietary data or compute have limited reason to contribute without appropriate incentives. Coupling incentive-mechanism design with energy- and communication-aware scheduling, and validating results through real-world case studies rather than simulation alone, is needed to translate the substantial methodological progress reviewed in this survey, together with the trustworthiness safeguards outlined above, into deployable, accountable federated LLM systems. Answers to RQ3: Despite substantial methodological progress, federated prompt learning continues to face significant security, privacy, robustness, and system challenges. On the security front, federated instruction tuning remains vulnerable to stealthy data-poisoning attacks that erode safety alignment while evading conventional robust aggregation, and prompt-only systems remain susceptible to backdoor attacks via imperceptible triggers even when Manuscript submitted to ACM

32

Q.Yang C.Qiu, H.Zhang et al.

the backbone stays frozen, with existing defenses (e.g., SABRE-FL, FedGPL) yet to be validated against adaptive attackers or broader threat models. Privacy protections are often justified only informally—relying on the implicit assumption that prompts are less data-revealing—rather than through formal guarantees, and even methods offering DP bounds (e.g., DP-FPL) are tested against limited attack models; this gap is compounded by theoretical evidence that transformer representations may be invertible in principle, and by largely unexamined leakage risks arising from persistent memory and inter-agent communication in agentic deployments. Robustness remains constrained by simulated (rather than real-world) heterogeneity, small client populations, single-backbone evaluation, and fragmented experimental protocols, while accountability is undermined by the absence of any single party observing the complete federation-wide interaction trace. Synthesizing these gaps, the survey outlines future directions spanning standardized multi-objective evaluation, theoretical foundations connecting prompt capacity to generalization under non-IID conditions, heterogeneity- and resource-aware optimization at realistic scale, LLMs as intelligent controllers of federated systems, trustworthy federated agentic ecosystems with trajectory-level verification, privacy protection extending beyond training to inference-time and memory-space leakage, continuous safety auditing and accountability mechanisms for agentic deployments, and incentive design coupled with real-world validation—collectively underscoring that the field remains at an early stage of maturity as it transitions from static model adaptation toward autonomous, agentic federated systems. 9 Conclusion This survey systematically reviewed federated prompt learning across model adaptation, personalization, inference, applications, and security. Existing studies show that prompts and lightweight adapters can substantially reduce the communication, computation, and storage costs of federated LLM fine-tuning while keeping raw data local. However, methods differ considerably in their ability to handle non-IID data, resource heterogeneity, personalization, scalability, and privacy. The unified benchmark further indicates that federated PEFT should be evaluated as a multi-objective problem. LoRA generally offers more stable performance, whereas prompt-based methods achieve lower communication costs but may be more sensitive to heterogeneous optimization. Future research should focus on resource-aware and adaptive optimization, stronger personalization and generalization, standardized evaluation, and formal protection against privacy leakage, poisoning, backdoor, and reconstruction attacks. Acknowledgments This work was partially supported by National Natural Science Foundation of China under Grant No.U25B2030; National Natural Science Foundation of China under Grant No.T2522011; GuangDong Basic and Applied Basic Research Foundation under Grant No. 2025B1515020022, 2026A1515010183. References [1] Alekh Agarwal, Dean P Foster, Daniel J Hsu, Sham M Kakade, and Alexander Rakhlin. 2011. Stochastic convex optimization with bandit feedback. Advances in neural information processing systems 24 (2011). [2] Ankur Agarwal, Mehdi Rezagholizadeh, and Prasanna Parthasarathi. 2023. Practical takes on federated learning with pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023. 454–471. [3] Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. 2024. Federated fine-tuning of large language models under heterogeneous tasks and client resources. Advances in Neural Information Processing Systems 37 (2024), 14457–14483. [4] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15, 3 (2024), 1–45. Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

33

[5] Haoxuan Che, Haibo Jin, Zhengrui Gu, Yi Lin, Cheng Jin, and Hao Chen. 2025. Llm-driven medical report generation via communication-efficient heterogeneous federated learning. IEEE Transactions on Medical Imaging (2025). [6] Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. 2023. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 7871–7888. [7] Chaochao Chen, Xiaohua Feng, Yuyuan Li, Lingjuan Lyu, Jun Zhou, Xiaolin Zheng, and Jianwei Yin. 2024. Integration of large language models and federated learning. Patterns 5, 12 (2024). [8] Jiao Chen, Jiayi He, Fangfang Chen, Zuohong Lv, Jianhua Tang, and Yunjian Jia. 2024. Empowering IoT-based autonomous driving via federated instruction tuning with feature diversity. IEEE Internet of Things Journal (2024). [9] Yuliang Chen, Xi Lin, Chao Sang, and Xiu Su. 2025. DualFPT: Handling Data Heterogeneity in Federated Prompt Tuning from both Generalized and Personalized Perspective. In Proceedings of the 33rd ACM International Conference on Multimedia. 2890–2899. [10] Yujun Cheng, Weiting Zhang, Zhewei Zhang, Chuan Zhang, Shengjin Wang, and Shiwen Mao. 2024. Towards federated large language models: Motivations, methods, and future directions. IEEE Communications Surveys & Tutorials (2024). [11] Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. 2024. Harmonizing generalization and personalization in federated prompt learning. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 381, 16 pages. [12] Shengwen Ding and Chenhui Hu. 2024. efedllm: Efficient llm inference based on federated learning. arXiv preprint arXiv:2411.16003 (2024). [13] Yaxin Du, Rui Ye, Fengting Yuchi, Wanru Zhao, Jingjing Qu, Yanfeng Wang, and Siheng Chen. 2025. FedDQC: Data quality control in federated instruction-tuning of large language models. In Findings of the Association for Computational Linguistics: ACL 2025. 15267–15291. [14] Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049 (2023). [15] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient finetuning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 12799–12807. [16] Zhidong Gao, Zhenxiao Zhang, Yuanxiong Guo, and Yanmin Gong. 2025. Federated adaptive fine-tuning of large language models with heterogeneous quantization and lora. In Ieee infocom 2025-ieee conference on computer communications. IEEE, 1–10. [17] Shuai Gong, Chaoran Cui, Chunyun Zhang, Wenna Wang, Xiushan Nie, and Lei Zhu. 2026. Federated domain generalization via prompt learning and aggregation. IEEE Transactions on Information Forensics and Security (2026). [18] Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. 2023. Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model. IEEE Transactions on Mobile Computing 23, 5 (2023), 5179–5194. [19] Zhuoning Guo, Ruiqian Han, and Hao Liu. 2024. Against multifaceted graph heterogeneity via asymmetric federated prompt learning. arXiv preprint arXiv:2411.02003 (2024). [20] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021. Transformer in transformer. Advances in neural information processing systems 34 (2021), 15908–15919. [21] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9 [22] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing. 5254–5276. [23] Wei Huang, Yinggui Wang, Anda Cheng, Aihui Zhou, Chaofan Yu, and Lei Wang. 2024. A fast, performant, secure distributed training framework for llm. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 4800–4804. [24] Momin Khan, Yasra Chandio, and Fatima Anwar. 2026. SABRE-FL: Selective and accurate backdoor rejection for federated prompt learning. In International Conference on Learning Representations, Vol. 2026. 157155–157176. [25] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5260–5271. [26] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37, 3 (2020), 50–60. [27] Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers). 4582–4597. [28] Xinting Liao, Weiming Liu, Jiaming Qian, Pengyang Zhou, Jiahe Xu, Wenjie Wang, Chaochao Chen, Xiaolin Zheng, and Tat-Seng Chua. 2025. FOCoOp: enhancing out-of-distribution robustness in federated prompt learning for vision-language models (ICML’25). JMLR.org, Article 1489, 27 pages. [29] Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, and Yue Gao. 2024. Splitlora: A split parameter-efficient fine-tuning framework for large language models. arXiv preprint arXiv:2407.00952 (2024). [30] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to finetuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Manuscript submitted to ACM

34

Q.Yang C.Qiu, H.Zhang et al.

61–68. [31] Pingchen Lu, Zhi Hong, Zhiwei Shang, Zhiyong Wang, Yikun Ban, Yao Shu, Min Zhang, Shuang Qiu, and Zhongxiang Dai. 2025. FedPOB: SampleEfficient Federated Prompt Optimization via Bandits. arXiv preprint arXiv:2509.24701 (2025). [32] Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. 2023. The multi-modal fusion in visual question answering: a review of attention mechanisms. PeerJ Computer Science 9 (2023), e1400. [33] Jun Luo, Chen Chen, and Shandong Wu. 2025. Mixture of experts made personalized: Federated prompt learning for vision-language models. In International Conference on Learning Representations, Vol. 2025. 58061–58080. [34] Wenqiang Luo, Jacky Keung, Boyang Yang, He Ye, Claire Le Goues, Tegawende F Bissyande, Haoye Tian, and Xuan Bach D Le. 2024. When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair. ACM Transactions on Software Engineering and Methodology (2024). [35] Shubham Malaviya, Manish Shukla, and Sachin Lodha. 2023. Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning. In Conference on Lifelong Learning Agents. PMLR, 456–469. [36] Md Jueal Mia and M Hadi Amini. 2025. FedShield-LLM: A Secure and Scalable Federated Fine-Tuned Large Language Model. arXiv preprint arXiv:2506.05640 (2025). [37] Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, and Emanuele Rodolà. 2025. Language models are injective and hence invertible. arXiv preprint arXiv:2510.15511 (2025). [38] Yazan Otoum, Arghavan Asad, and Amiya Nayak. 2025. Llms meet federated learning for scalable and secure iot management. arXiv preprint arXiv:2504.16032 (2025). [39] Zhen Qin, Zhaomin Wu, Bingsheng He, and Shuiguang Deng. 2025. Federated data-efficient instruction tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2025. 15550–15568. [40] Wenqi Qiu, Yipeng Zhou, Jinzhi Wang, Quan Z Sheng, and Laizhong Cui. 2025. Flm-topk: Expediting federated large language model tuning by sparsifying intervalized gradients. In IEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 1–10. [41] Arian Raje. 2024. Communication-efficient LLM training for federated learning. Ph. D. Dissertation. Carnegie Mellon University Pittsburgh, PA. [42] Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, and Chen Chen. 2023. Fedperfix: Towards partial model personalization of vision transformers in federated learning. In Proceedings of the IEEE/CVF international conference on computer vision. 4988–4998. [43] Dipanwita Thakur, Antonella Guzzo, and Giancarlo Fortino. 2025. Analyzing the Fusion of Federated Learning and Large Language Model. In 2025 IEEE 5th International Conference on Human-Machine Systems (ICHMS). IEEE, 282–288. [44] Linh Tran, Wei Sun, Stacy Patterson, and Ana Milanova. 2025. Privacy-preserving personalized federated prompt learning for multimodal large language models. In International Conference on Learning Representations, Vol. 2025. 27362–27380. [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [46] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Forty-first International Conference on Machine Learning. [47] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. 353–355. [48] Haoyu Wang, Zilong Yin, Bin Chen, Yujie Zeng, Xiyue Yan, Chenyu Zhou, and Anji Li. 2025. Rofed-LLM: robust federated learning for large language models in adversarial wireless environments. IEEE Transactions on Network Science and Engineering (2025). [49] Zhihao Wang, Wenke Huang, Tian Chen, Zekun Shi, Guancheng Wan, Yu Qiao, Bin Yang, Jian Wang, Bing Li, and Mang Ye. 2025. An empirical study of federated prompt learning for vision language model (IJCAI ’25). Article 1188, 9 pages. doi:10.24963/ijcai.2025/1188 [50] Pei-Yau Weng, Minh Hoang, Lam M Nguyen, My T Thai, Tsui-Wei Weng, and Trong N Hoang. 2024. Probabilistic federated prompt-tuning with non-IID and imbalanced data. Advances in neural information processing systems 37 (2024), 81933–81958. [51] Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. 2024. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3345–3355. [52] Baochen Xiong, Xiaoshan Yang, Yaguang Song, Yaowei Wang, and Changsheng Xu. 2025. Pilot: Building the federated multimodal instruction tuning framework. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 21716–21724. [53] Na Yan, Yang Su, Yansha Deng, and Robert Schober. 2025. Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions. arXiv preprint arXiv:2501.04436 (2025). [54] Fu-En Yang, Chien-Yi Wang, and Yu-Chiang Frank Wang. 2023. Efficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 19159–19168. [55] Hongming Yang, Hao Liu, Xin Yuan, Kai Wu, Wei Ni, J Andrew Zhang, and Ren Ping Liu. 2025. Synergizing Intelligence and Privacy: A Review of Integrating Internet of Things, Large Language Models, and Federated Learning in Advanced Networked Systems. Applied Sciences 15, 12 (2025), 6587. [56] Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing 4, 2 (2024), 100211. Manuscript submitted to ACM

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

35

[57] Rui Ye, Jingyi Chai, Xiangrui Liu, Yaodong Yang, Yanfeng Wang, and Siheng Chen. [n. d.]. Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models. In International Workshop on Federated Foundation Models in Conjunction with NeurIPS 2024. [58] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked languagemodels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 1–9. [59] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. 2021. A survey on federated learning. Knowledge-Based Systems 216 (2021), 106775. [60] Zikai Zhang, Ping Liu, Jiahao Xu, and Rui Hu. 2025. Fed-hello: Efficient federated foundation model fine-tuning with heterogeneous lora allocation. IEEE Transactions on Neural Networks and Learning Systems 36, 10 (2025), 17556–17569. [61] Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. 2023. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. [62] Jia-Ying Zheng, HaiNan Zhang, LingXiang Wang, WangJie Qiu, Hong-Wei Zheng, and Zhi-Ming Zheng. 2024. Safely learning with private data: A federated learning framework for large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 5293–5306. [63] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional prompt learning for vision-language models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 16795–16804.

Manuscript submitted to ACM

Related documents

• Pathwise Individual Rationality in Federated Learning: A Mechanism-Architecture Co-Design#1084534
• Resource-Aware Federated Mixture-of-Experts with Adaptive Pruning for Onboard Learning in LEO Satellite Constellations#1115888
• Decentralized Federated Learning: A Survey on Security and Privacy#177251
• Privacy and Fairness in Federated Learning: On the Perspective of Tradeoff#177281
• Bias in Federated Learning: A Comprehensive Survey#156445