T OWARDS THE N EXT F RONTIER OF LLM S , T RAINING ON P RIVATE D ATA : A C ROSS -D OMAIN B ENCHMARK FOR F EDERATED F INE -T UNING
arXiv:2605.13936v1 [cs.LG] 13 May 2026
Sherpa.ai [email protected]
A BSTRACT The recent success of large language models (LLMs) has been largely driven by vast public datasets. However, the next frontier for LLM development lies beyond public data. Much of the world’s most valuable information is private, especially in highly regulated sectors such as healthcare and finance, where data include patient histories or customer communications. Unlocking this data could represent a major leap forward, enabling LLMs with deeper domain expertise and stronger real-world utility. Yet, these data cannot be shared because they are distributed across institutions and constrained by privacy, regulatory, and organizational barriers. Moreover, institutional datasets are typically non-independent and identically distributed (non-IID), differing across sites in population characteristics, data modalities, documentation patterns, and task-specific label distributions. In this paper, we demonstrate a practical approach to unlocking private and distributed institutional data for LLM adaptation through federated collaboration across data silos. Built on the Sherpa.ai Federated Learning platform, our framework enables nodes to jointly fine-tune a shared LLM without exchanging private data. We evaluate this approach through a cross-domain benchmark in healthcare and finance, using four closed-ended question answering and classification datasets: MedQA, MedMCQA, FPB, and FiQA-SA. We compare three parameter-efficient fine-tuning (PEFT) strategies—LoRA, QLoRA, and IA3—across pretrained backbones under non-IID settings reflecting institutional data heterogeneity. Our results show that federated fine-tuning performs close to centralized training and outperforms isolated single-institution learning. From a Green AI perspective, QLoRA and IA3 improve efficiency with limited accuracy degradation, supporting federated PEFT as a viable approach for adapting LLMs where data cannot be shared. SHERPA.AI federated
MODEL ACCURACY
92%
SILO 2
(Accuracy)
85%
SILO 1
(Accuracy)
33%
SILO 3
(Accuracy)
55%
SILO 5
(Accuracy)
84%
SILO 4
(Accuracy)
88%
Figure 1: Global map illustrating institution-level FiQA-SA accuracies for the best federated model. 1
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
1
Introduction
Large language models (LLMs) have become a strong foundation for a wide range of natural language processing tasks, including classification, closed-ended question answering (QA), instruction following, and domain-specific reasoning in areas such as medicine and finance [1, 2, 3]. In practice, however, high performance in specialized domains still depends on task-specific adaptation through fine-tuning, i.e., further training a pretrained model on labeled task data so that it better captures the terminology and reasoning patterns of the target domain. For example, a general-purpose LLM may require additional fine-tuning to answer medical exam questions correctly or to classify the sentiment of financial texts. Parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) [4], Quantized Low-Rank Adaptation (QLoRA) [5], and infused adapters by inhibiting and amplifying inner activations (IA3) [6] have rendered model adaptation substantially practical by reducing the number of trainable parameters and memory requirements while retaining competitive performance. Thus, PEFT has become a natural choice for adapting LLMs in real-world environments with limited computational resources. This is especially relevant in high-stakes domains such as medicine and finance, where specialized terminology, task formats, and reasoning patterns differ substantially from general-purpose pretraining corpora. Recent work (e.g., [2, 7]) has shown that centralized fine-tuning can yield strong performance in both domains, as evidenced by large-scale medical and financial evaluation efforts. However, the centralized setting assumes that all task data can be pooled in a single location, which is often unrealistic in practice. In many real deployments, data are distributed across institutions (nodes) and cannot be shared freely due to privacy constraints, confidentiality requirements, or regulatory restrictions. Federated learning (FL) [8] provides a natural alternative by allowing multiple nodes to collaboratively fine-tune a shared model without exchanging raw data (see Figure 1). This paradigm is particularly attractive for LLM adaptation in sensitive domains, but it also introduces new challenges, including non-independent and identically distributed (non-IID) data across nodes, which can degrade model performance, as well as communication overhead and memory constraints. Although recent studies such as FedLLM-Bench [9] and FlowerTune [10] have advanced the benchmarking of federated LLM fine-tuning, the literature still lacks sufficiently controlled comparisons across training scenarios, domains, task families, and efficiency dimensions under the same experimental protocol. 1.1
Motivation
Although recent work has demonstrated the potential of centralized [11, 12, 13, 14, 15] and federated LLM finetuning [10, 9, 16, 17], their comparative behavior across domains, benchmarks, and PEFT strategies remains incompletely understood. Existing studies often focus on a single domain, a single benchmark, or a single adaptation strategy, making it difficult to determine how different PEFT methods behave across training paradigms under comparable conditions. This lack of controlled comparison is particularly important in sensitive domains such as medicine and finance, where model deployment must be assessed not only in terms of predictive quality, but also with respect to practical efficiency. In these environments, communication cost, memory footprint, and robustness to non-IID data can be as relevant as accuracy itself when deciding whether a fine-tuning strategy is viable in practice. These considerations motivate the benchmark presented in this paper. We aim to provide a controlled comparison of centralized, single-institution, and federated PEFT across multiple domains and task types, while also incorporating efficiency metrics that are critical for realistic deployment. 1.2
Contribution
This paper does not propose a new FL algorithm or optimization method. Instead, it presents an experimental benchmark of PEFT for LLMs under centralized, single-institution, and federated settings. The main contributions of this benchmark are as follows: • We compare PEFT-based LLM adaptation across three training scenarios: single-institution, centralized, and federated learning. All scenarios are evaluated under a common experimental protocol using three representative PEFT methods: LoRA, QLoRA, and IA3 adapters. • We evaluate the different methods and scenarios across two task families, namely closed-ended QA and classification, and two sensitive domains, medicine and finance, enabling a controlled cross-domain and cross-task comparison. • We complement accuracy results with efficiency metrics, namely communication cost and memory footprint, and explicitly simulate non-IID federated conditions using the Dirichlet [18] partition protocol to create the nodes.
2
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
The remainder of this paper is organized as follows: Section 2 presents the problem formulation. Section 3 describes the privacy-preserving machine learning (ML) solution, including the LLM fine-tuning setting and the fundamentals of FL. Section 4 details the datasets, preprocessing pipeline, and centralized architecture, while Section 5 describes the federated benchmark setup and the creation of nodes. Section 6 reports the experimental setup and results. Section 7 presents a discussion of the obtained results. Finally, Section 8 concludes the paper.
2
Problem Formulation
This section presents the problem addressed in this paper: fine-tuning LLMs for closed-ended question answering and classification. We first introduce LLMs and LLM fine-tuning, and then formalize the supervised learning problem. 2.1
LLMs and LLM Fine-Tuning
An LLM is a neural network designed to process, generate, and reason over natural language at scale. Modern LLMs are typically based on the transformer architecture, which enables them to capture long-range dependencies and complex patterns in text through self-attention mechanisms. By pretraining on massive text corpora, these models acquire broad linguistic knowledge and general-purpose capabilities that can later be adapted to specific application domains and tasks. LLM fine-tuning is the process of adapting a pretrained language model to a specific task by further optimizing it on labeled task data. As illustrated in Figure 2, this process starts from a pretrained LLM obtained through large-scale pretraining on a massive general-purpose corpus. The model is then adapted using a domain-specific labeled dataset, while retaining the general knowledge acquired during pretraining. During this stage, task-relevant parameters are optimized so that the model better captures the terminology, structure, and decision patterns required by the target domain. The result is a fine-tuned LLM specialized for the task. In this work, the target tasks are closed-ended question answering and classification, where the model must predict a correct answer option or class label from a finite set of valid outputs.
PRE-TRAINING
LARGE PUBLIC DATASET
EXTRA-COMPONENT OR PART OF PRE-TRAINED LLM
GETTING
PRE-TRAINED LLM
DOMAIN-SPECIFIC DATASET
TRAINING
FINE-TUNED LLM
Figure 2: Overview of the simplified LLM fine-tuning process from pre-training to domain-specific adaptation. Although pretrained LLMs capture broad linguistic knowledge, they do not necessarily perform optimally in specialized domains such as medicine and finance. Fine-tuning addresses this limitation by exposing the model to task-specific examples, thereby improving its ability to follow the target format and produce valid domain-specific outputs. Unlike classical ML methods based on fixed-dimensional feature vectors, LLM fine-tuning operates directly on tokenized text sequences. The model receives a prompt, and it is trained to generate the correct completion, which in our setting corresponds to a label or answer option. Because fully fine-tuning all model parameters is often computationally expensive, this work focuses on PEFT methods, such as LoRA, QLoRA, and IA3 adapters, which update only a small set of task-specific parameters while keeping the pretrained backbone fixed or mostly frozen. These methods were selected because they are among the most widely adopted and representative approaches for efficient LLM adaptation in the recent literature, offering a practical trade-off between performance and computational cost [19, 5, 4]. This makes them particularly suitable for both centralized and federated training settings. 2.2
Related Work
This subsection reviews the lines of work most relevant to this paper, namely, centralized and federated fine-tuning of LLMs. We emphasize prior studies on parameter-efficient adaptation, sensitive-domain applications, and realistic evaluation settings. This context helps situate the benchmark and comparative analysis presented in our work.
3
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
2.2.1
Centralized Fine-Tuning of LLMs
Recent work has shown that centralized fine-tuning remains a strong baseline for adapting LLMs to domain-specific tasks, especially in high-risk areas such as medicine and finance. In the medical domain, [2] highlights the breadth of clinical evaluation settings and shows that robust assessment should go beyond a single benchmark or prompting strategy. Similarly, [13] studies fine-tuning strategies for medical LLMs and compares full-parameter and parameterefficient adaptation, showing that PEFT can be competitive while substantially reducing training cost. More recently, [20] further demonstrates that targeted fine-tuning can improve medical reasoning ability, reinforcing the importance of specialized adaptation pipelines for healthcare tasks. In finance, recent studies confirm that generic LLMs benefit substantially from domain adaptation. [14] studies instruction fine-tuning for financial classification tasks and shows that fine-tuned models can outperform untuned baselines in specialized financial language understanding. [15] shows that parameter-efficient and quantized adaptation is particularly attractive in finance, where local deployment and memory efficiency are often required. At a broader scale, [7] demonstrates that fine-tuning can yield substantial gains across a wide range of financial tasks when evaluated on a unified benchmark, while [11] emphasizes the need to compare PEFT methods systematically in high-stakes financial scenarios. More generally, [12] suggests that adaptation can improve task performance while also altering reasoning behavior, making careful benchmark design especially important. Despite these advances, most centralized studies focus on a single domain, a single training regime, or a narrow set of PEFT strategies. In contrast, our work evaluates multiple PEFT methods under a unified testbed across both financial and medical closed-ended benchmarks, comparing several backbone models under the same evaluation metrics. This provides a controlled view of how centralized fine-tuning behaves across domains and creates a stronger baseline for comparison with its privacy-preserving counterpart. 2.2.2
Federated Fine-Tuning of LLMs
Federated fine-tuning has recently emerged as a promising alternative when domain data are distributed and cannot be pooled centrally. FedLLM-Bench [9] is one of the first realistic benchmarks for FL of LLMs, showing that prior evaluations often relied on overly artificial settings and that realistic client diversity is essential for fair comparison. Independently, FlowerTune [10] proposes a cross-domain benchmark for federated LLM fine-tuning spanning across general NLP, finance, medicine, and coding, hence highlighting the need for evaluation suites that go beyond a single application domain. A complementary line of work studies algorithmic and system aspects of federated LLM adaptation. [16] organizes the area into parameter-sharing, knowledge-distillation, and split-learning based approaches, and clarifies the efficiency-privacy trade-offs among these families. In healthcare, [17] underlines the importance of domain-specific federated evaluation under non-IID client distributions, particularly in non-IID medical settings. Earlier work, such as [21], established the basic feasibility of federated fine-tuning for language models. It focused on reducing memory, communication, and edge-computing costs. Our work builds on these recent benchmarks, but addresses a different experimental question.FedLLM-Bench studies federated LLM training under instruction-tuning and preference-alignment settings, with a particular emphasis on large-scale client diversity. FlowerTune provides leaderboard-style evaluation of federated fine-tuning across various fields, evaluating multiple pretrained backbones and FL configurations. However, because FlowerTune is designed as a broad, model-agnostic benchmark, its results on demanding medical and financial tasks remain relatively far from those of the strongest proprietary or domain-specialized reference systems reported in the literature, such as GPT-4, Med-PaLM 2 [22], and medical-specialized LLMs. Therefore, an additional question remains open: under a more focused sensitive-domain setup, can federated PEFT remain close to centralized fine-tuning while improving over isolated single-institution training and avoiding raw data centralization? This paper addresses that question through a more controlled benchmark. We focus on closed-ended medical QA and financial classification, using MedQA, MedMCQA, FPB, and FiQA-SA as both task-specific adaptation and evaluation targets. We compare centralized, single-institution, and federated training under the same preprocessing pipeline, training budget, backbone models, PEFT methods, and evaluation metrics. This design allows us to quantify both the loss with respect to full centralization and the gain over isolated institutional training, while also evaluating three representative PEFT strategies: LoRA, QLoRA, and IA3 adapters. In addition to predictive performance, we report communication cost and memory footprint, enabling a joint comparison of accuracy and system efficiency on a concrete FL platform. In this sense, our contribution is complementary to previous benchmarks: it does not propose a broader leaderboard or a new FL algorithm, but provides a focused experimental benchmark for high-performing federated PEFT in sensitive domains.
4
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
2.3
Problem Definition
We now formalize the supervised learning setting considered in this work. Each training instance consists of a tokenized textual input and an associated target output. The input may correspond to a question or an instruction, while the target corresponds to the correct class label or answer option from a finite set of valid outputs. Since LLMs operate on token sequences rather than fixed-dimensional feature vectors, the problem is naturally expressed at the level of input tokens and discrete output labels. Specifically, let N be the total number of training instances. For each instance i ∈ {1, . . . , N }, we define an input sequence xi = [xi,1 , xi,2 , . . . , xi,pi ], where xi,j denotes the j-th token of the input and pi is the length of the sequence for instance i. Depending on the task, xi may represent a question, an instruction, a document-question pair, or any structured prompt provided to the language model. The corresponding target output for each instance is denoted by yi . Since we focus on closed-ended question answering and classification, yi belongs to a finite set of valid responses: yi ∈ Y = {c1 , c2 , . . . , cK }, where Y is the set of candidate classes or answer options, and K = |Y| is the number of possible outputs. The complete dataset can be represented as y = [y1 , y2 , . . . , yN ]⊤ ∈ Y N .
X = {x1 , x2 , . . . , xN },
Let fθ denote a pretrained large language model parameterized by θ. In PEFT, the pretrained parameters remain fully or mostly frozen, and only a small set of task-specific parameters ϕ is optimized. These trainable parameters may correspond to low-rank adaptation matrices (LoRA/QLoRA) or adapter module parameters. The resulting model is denoted by fθ,ϕ . For each input xi , the model defines a conditional distribution over the candidate output space: P (y | xi ; θ, ϕ),
y ∈ Y.
The prediction for instance i is then given by ŷi = arg max P (y | xi ; θ, ϕ). y∈Y
The objective is to learn the task-specific parameters ϕ such that the model correctly maps each input sequence xi to its target label or answer yi . This yields a supervised learning problem over discrete outputs: min L(ϕ) = − ϕ
N X
log P (yi | xi ; θ, ϕ),
i=1
where L is the standard cross-entropy loss over the closed set of candidate answers. Therefore, the task consists of adapting a pretrained LLM to predict the correct class or answer option from a finite output space, given an input prompt xi , using a PEFT strategy.
3
Privacy-preserving ML Solution
This section introduces the centralized and federated learning settings used in the benchmark. We adopt the notation and supervised probabilistic formulation introduced in Section 2.3. 3.1
Classical ML Approach
In the centralized setting, all training examples are pooled into a single dataset and used to adapt a pretrained large language model to the task by optimizing only a small set of task-specific parameters through PEFT. More specifically, the available labeled data are partitioned into disjoint training, validation, and test subsets, typically after preprocessing: {1, . . . , N } = Itrain ⊔ Ival ⊔ Itest , 5
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
with corresponding subsets Dtrain = {(xi , yi )}i∈Itrain ,
Dval = {(xi , yi )}i∈Ival ,
Dtest = {(xi , yi )}i∈Itest .
Let fθ,ϕ denote the pretrained large language model adapted to the target task, where θ represents the pretrained backbone parameters and ϕ the trainable task-specific parameters introduced by the PEFT strategy. In the PEFT setting considered here, θ remains fixed or mostly frozen, while only ϕ is optimized. Then, training consists of minimizing the empirical risk on Dtrain : X 1 J(ϕ) = L(P (· | xi ; θ, ϕ), yi ) , |Itrain | i∈Itrain
where L is typically the cross-entropy loss over the finite output space. The resulting adapted model is selected using Dval and evaluated on Dtest . 3.2
Introduction to FL
FL enables multiple nodes to collaboratively adapt a shared model without exchanging raw data [8]. In our setting, the shared model is a pretrained large language model with fixed base parameters θ and trainable task-specific parameters ϕ. Each node k keeps its local dataset and performs local fine-tuning of the model fθ,ϕk using its own data only. Periodically, each node transmits the trainable parameters ϕk , or updates derived from them, to a central server (the aggregator). The server combines these local updates into a new global set of task-specific parameters ϕ, which is then redistributed to all nodes. This process is repeated until convergence. FL supports data locality by ensuring that input sequences and labels remain within each institution. However, this does not by itself provide formal privacy guarantees against information leakage from model updates. Additionally, it introduces several challenges. One of the most important is the presence of non-IID data across nodes, which may cause local updates to diverge and degrade the performance of the global model. Thus, in this work, the focus is on non-IID scenarios, as samples and labels are distributed across different nodes according to a Dirichlet partition protocol [23, 24, 25] (see Section 5.1). In the LLM setting, the aggregation step acts on the task-specific trainable parameters ϕk , while the pretrained backbone parameters θ remain fixed or mostly frozen. 3.2.1
FL Paradigms
In general, there are two FL paradigms depending on the different data-distribution scenarios: • Horizontal FL (HFL): In this paradigm, all nodes share the same input and output spaces, but each node contains different samples. In our setting, this means that all institutions fine-tune the same pretrained LLM architecture on the same task, using local datasets of the form (xi , yi ), where xi is a tokenized input sequence and yi ∈ Y is a class or closed-ended answer option. Example: multiple financial institutions use the same prompt structure and target label space, but each institution owns different clients, documents, or question-answer pairs. • Vertical FL (VFL): Nodes hold complementary information about the same underlying instances. In the LLM setting, this may correspond to different institutions holding different textual or structured attributes associated with the same sample or client. Example: multiple financial institutions collaborate on the same anti-money-laundering investigation, with one of them holding the target labels, while each institution holds different textual information associated with the same customer or case, such as transaction-related narratives, customer communications, internal compliance notes, or suspicious-activity reports. In this work, we focus exclusively on HFL. After training is complete in an HFL setup, the resulting global adapted model is typically shared with all participating parties. This allows each node to download the final global parameters ϕ and perform inference locally and independently with the model fθ,ϕ , without requiring further interaction or exchange of raw data. 3.2.2
HFL
Under HFL, each node k has a local dataset k Dk = {(xki , yik )}N i=1 ,
where each input sequence is of the form k
xki = [xki,1 , xki,2 , . . . , xki,pk ] ∈ V pi , i
6
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
and each target satisfies yik ∈ Y. All nodes share the same vocabulary space V, the same output space Y, and the same pretrained model architecture, although the number of local samples Nk and the empirical data distribution may differ between nodes. Training proceeds as follows: 1. Local update: Each node k receives the current global task-specific parameters ϕ, initializes its local model as fθ,ϕ , and performs local fine-tuning on Dk . The corresponding local empirical risk is N
Jk (ϕ) =
k 1 X L P (· | xki ; θ, ϕ), yik , Nk i=1
where L is typically the cross-entropy loss over the label space Y. After local optimization, node k obtains updated trainable parameters ϕk . 2. Aggregation: Each node sends its updated task-specific parameters ϕk , or an equivalent update ∆ϕk , to the server. 3. Global update: The server aggregates the collection of local parameters {ϕk } into a new federated global set of task-specific parameters ϕ. If M denotes the number of participating nodes, then a standard weighted averaging scheme can be written as M X Nk ϕ← ϕk . PM ℓ=1 Nℓ k=1 4. Broadcast: The server distributes the updated global parameters ϕ back to all nodes, and the process starts again. Therefore, in the HFL setting for parameter-efficient LLM fine-tuning, collaboration happens via the exchange and aggregation of the trainable adaptation parameters ϕ, while the raw sequences xki and labels yik remain local to each node.
4
Centralized Datasets and Preprocessing
In this section, we describe the datasets, outline the preprocessing steps, and present the architecture of the centralized solution to our problem. 4.1
Description of the Datasets
The experiments reported in this work are conducted on four closed-ended QA and classification benchmarks spanning the financial and medical domains: FiQA-SA [26], FPB [27], MedQA [28], and MedMCQA [29]. These datasets are well-suited to the evaluation of PEFT methods for LLMs, since they formulate the problem either as closed-ended question answering or as label prediction over a finite output space. FiQA-SA is a dataset for financial sentiment analysis composed of financial news headlines and microblog posts, where each instance is annotated with a sentiment value on a continuous scale from −1 (most negative) to 1 (most positive) [26]. In practice, this benchmark evaluates whether the model can correctly infer the sentiment conveyed by short financial texts, which makes it suitable for classification-oriented fine-tuning. FPB (Financial PhraseBank) is a benchmark for financial sentiment classification built from English financial news sentences [27]. The dataset contains 4,840 sentences annotated by multiple annotators with financial backgrounds, and each sentence is labeled according to its polarity from an investor perspective, as positive, negative, or neutral. This benchmark is particularly relevant for evaluating sentiment classification in a specialized financial language. MedQA is a medical question answering benchmark collected from professional medical board exams [28]. It is a multiple-choice dataset designed to assess medical knowledge and reasoning and covers three languages: English, simplified Chinese, and traditional Chinese. In the English setting, it is commonly associated with USMLE-style questions, making it a standard benchmark for closed-ended medical QA. MedMCQA is a large-scale multi-subject medical multiple-choice question answering dataset constructed from realworld medical entrance examinations [29]. It contains more than 194,000 questions that span 21 medical topics and around 2,400 healthcare topics. Each instance includes a question, the correct answer, and distractor options, which makes it especially suitable for evaluating domain-specific reasoning and answer selection. 7
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
Together, these four datasets provide a diverse but coherent evaluation setting for LLM fine-tuning. FiQA-SA and FPB evaluate financial sentiment classification, whereas MedQA and MedMCQA assess closed-ended medical question answering and reasoning. 4.2
Preprocessing of the Datasets
All datasets are preprocessed for supervised fine-tuning by converting each example into a prompt-completion pair and tokenizing it with the tokenizer of the selected base model. When no padding token is available, the end-of-sequence token is used instead. The loss is computed only on the completion tokens, while the prompt and padding tokens are masked. In addition, all inputs are truncated to a maximum sequence length of 512 tokens, while ensuring that sufficient space remains for the target completion. For FiQA-SA, the official train, validation, and test splits are used. Because the original labels are continuous sentiment scores, they are mapped into three classes (negative, neutral, and positive) using thresholds at −0.05 and 0.05, and the targets are represented as labels in {A, B, C}. For FPB, the official training split is randomly shuffled, and 10% of the examples are reserved for validation. The test set corresponds to the official test split. The task is defined as a three-class sentiment classification with outputs A, B, and C. For MedQA, the original training split is randomly shuffled, and 10% of the examples are reserved for validation. The official test split is used for the final evaluation. Each example is converted into an exam-style multiple-choice prompt, and the target is the correct answer letter. Finally, for MedMCQA, the original training split is randomly shuffled, and 10% of the training examples are reserved as an internal validation set for model selection. Because the dataset does not provide a publicly labeled test split, the final evaluation is conducted on the official validation split. Moreover, the dataset is filtered to retain only valid single-choice questions with non-empty answer options and a valid correct option index. Each example is then converted into an exam-style multiple-choice prompt whose target is the correct answer letter. 4.3
Centralized Architecture
ALL PRIVATE DATA Public Cloud Private and local environment
PRIVATE DATA EXCHANGED
Private data
Private data Private data
Private data
Private data
PRIVATE DATA
PRIVATE DATA
PRIVATE DATA
PRIVATE DATA
PRIVATE DATA
INSTITUTION 1
INSTITUTION 2
INSTITUTION 3
INSTITUTION 4
INSTITUTION 5
Figure 3: Classical architecture for centralized training. Figure 3 depicts the centralized architecture. In this setting, all local datasets are pooled into a single training repository, and model training is performed with access to the combined training data. The resulting model is therefore learned under the assumption that data from all participating institutions can be shared and processed jointly. Given the size of the backbone model, training is performed through PEFT, where the pretrained base parameters are kept fixed or mostly frozen, and only a reduced set of task-specific parameters is optimized. This allows the model to adapt to the closed-ended QA and classification tasks while significantly reducing the number of trainable parameters and the associated computational cost.
8
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
5
Proposed Privacy-preserving Solution through FL
This section describes the federated benchmark setup and the procedure used to create the participating nodes. Trained Model
Privacy-Preserving AI Platform
Private Cloud Private and local environment
NO PRIVATE DATA EXCHANGED
Learnings
Learnings
Learnings
Learnings
Learnings
PRIVATE DATA
PRIVATE DATA
PRIVATE DATA
PRIVATE DATA
PRIVATE DATA
INSTITUTION SILO 2 1
INSTITUTION SILO 2 2
INSTITUTION SILO 3 3
INSTITUTION SILO 4 4
INSTITUTION SILO N 5
Figure 4: Proposed architecture for federated fine-tuning with privacy-preserving orchestration. The node architecture used for FL is shown in Figure 4. In this scenario, raw training data are not shared across nodes; each institution keeps its local dataset on site while exchanging only trainable model parameters or updates. A central server coordinates the training process by distributing the current global task-specific parameters, collecting the locally updated parameters from the participating nodes, aggregating them, and broadcasting the updated global model back to the nodes. Given the large size of the backbone model, training is carried out through PEFT: the pretrained base parameters remain fixed or mostly frozen, and only a reduced set of task-specific parameters is updated. This substantially decreases the computational and memory requirements of the federated process while preserving the model’s ability to adapt to the target closed-ended QA and classification tasks. 5.1
Creation of Nodes
To emulate a federated setting with heterogeneous local datasets, the training split of each benchmark is partitioned across five nodes using a Dirichlet-based protocol [18]. This procedure creates controlled label-distribution shifts across nodes, reflecting the type of statistical heterogeneity commonly observed in real-world federated environments. It also provides a reproducible protocol for evaluating federated fine-tuning under comparable non-IID conditions across all benchmarks. For the multiple-choice QA datasets, the resulting heterogeneity is defined over the target answer labels, so that different nodes receive different proportions of examples whose correct option is A, B, C, or D. More specifically, for each dataset, the partition is generated with concentration parameter α. Following [18], which uses a Hellinger distance of 0.5 as a reference threshold associated with noticeable non-IID effects, we select α so that the resulting partitions reach at least this level of non-IID data. In particular, we set α = 0.7 for FPB and α = 1.0 for FiQA-SA, MedQA, and MedMCQA. Figure 5 illustrates the resulting label distributions across the five institutions for the four datasets. The plots show that the Dirichlet-based partitioning induces heterogeneous class proportions across institutions. For the medical QA datasets, the labels correspond to correct answer options, whereas for the financial classification datasets, they correspond to sentiment categories.
9
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
(a) MedQA
(b) MedMCQA
(c) FPB
(d) FiQA-SA
Figure 5: Label distribution across institutions (INS) for the non-IID partitions used in each dataset. Each stacked bar shows the proportion of class labels within one institution.
6
Experiments
In this section, we describe the experimental setup, including the evaluation metric and the training configurations, and present the main results. We evaluate the models under three training scenarios on FiQA-SA, FPB, MedQA, and MedMCQA: 1. Centralized: All training data are pooled into a single centralized repository, and the model is fine-tuned using the complete training split. Evaluation is then performed on the corresponding evaluation set of each dataset. 2. Federated: The training split of each dataset is partitioned across multiple nodes according to the Dirichletbased protocol described in Section 5. Each node performs local fine-tuning on its own private data, and the local updates are aggregated by the central server over successive communication rounds. The resulting global model is evaluated on the corresponding evaluation set. 3. Single-institution: Each node fine-tunes its own model independently using only its local partition of the training data, without any collaboration or parameter aggregation. Each locally trained model is evaluated on the corresponding evaluation set. For each model, PEFT method, and dataset, the single-institution value reported in the tables corresponds to the mean of the accuracies over the five different nodes. These three scenarios enable a direct comparison between full data centralization, fully isolated local training, and collaborative training under data-locality constraints. In this way, the experiments quantify the extent to which FL provides an effective compromise between avoiding raw data sharing and still benefiting from the distributed knowledge available across nodes. 6.1
Evaluation Metrics
We evaluate predictive performance using accuracy, a standard metric for closed-ended question answering and classification tasks. It measures the proportion of examples for which the predicted output exactly matches the ground-truth label.
10
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
Formally, accuracy is defined as Accuracy =
N 1 X I ŷi = yi , N i=1
(6.1)
where N is the total number of evaluated examples, yi is the ground-truth label, ŷi is the predicted label, and I(·) is the indicator function, which takes value 1 if its argument is true and 0 otherwise. In the federated setting, we also report the communication cost per round and the memory footprint. The communication cost per round is defined as the bidirectional transmitted volume of trainable parameters between the server and the participating nodes in a single communication round: 2KBtrain , (6.2) 10243 where K is the number of participating nodes per round and Btrain is the number of communicated trainable bytes per client. Communication cost per round =
The memory footprint is defined as the mean peak reserved GPU memory as reported by PyTorch across participating nodes: K
Memory footprint =
1 X Mk , K 10243
(6.3)
k=1
where Mk denotes the peak reserved GPU memory in bytes observed for node k during training. 6.2
Reproducibility Details
All experiments are carried out on a machine with 219 GB of disk storage, an AMD EPYC 7R13 processor with 48 cores, 372 GB of RAM, Debian GNU/Linux 13 (Trixie), and four NVIDIA L40S GPUs, each with 46,068 MB of memory and Python 3.11. All models are trained using the Sherpa.ai FL platform. Unless otherwise stated, all experiments use a fixed random seed of 42. The optimization process uses the paged_adamw_8bit optimizer, a learning rate of 10−4 , gradient accumulation of 1, and a per-device batch size of 4. The maximum input length is 512 tokens. Gradient checkpoints are enabled, the maximum gradient norm is set to 1.0, and the learning-rate scheduler is constant with 10 warmup steps. Mixed-precision training is used whenever supported by the hardware, with bf loat16 preferred over f loat16. For the PEFT configuration, LoRA and QLoRA use rank r = 32, scaling factor α = 21, and dropout 0.1. Their target modules are inferred automatically from the model architecture at runtime. For the adapter-based setting, we use IA3 adapters applied to the modules k_proj, v_proj, and down_proj, with down_proj specified as the feedforward module. In the federated setup, all available nodes participate in every communication round, and model aggregation is performed with FedAvg over 4 communication rounds, using 1 local epoch per round. Thus, each node performs a total of 4 local epochs in the federated scenario. To ensure a fair comparison across training settings, the centralized and single-institution baselines are also trained for 4 epochs. The models employed in this work are II-Medical-8B-1706 [30], Qwen/Qwen3-8B [31], Qwen/Qwen3-4B-Instruct2507 [31], Qwen/Qwen2.5-1.5B-Instruct [32], meta-llama/Llama-3.1-8B-Instruct [33], meta-llama/Meta-Llama-3-8BInstruct [34], hongzhouyu/FineMedLM-o1 [20], google/gemma-2-9b-it [35], and google/gemma-3-4b-it [36]. These models were selected because they either showed strong performance in prior studies or correspond to recent versions of model families that have consistently achieved competitive results. 6.3
Results
Table 1 summarizes our experimental results for the medical datasets. It shows a clear advantage of the medicalspecialized backbone II-Medical-8B-1706, which achieves the best accuracy in all three training scenarios and for both medical benchmarks. In particular, its LoRA configuration reaches 0.740 and 0.763 in the centralized setting for MedQA and MedMCQA, respectively, while also obtaining the strongest federated results, namely 0.714 and 0.742. This suggests that domain specialization is particularly beneficial in the medical setting, where models explicitly designed for clinical reasoning transfer more effectively across PEFT strategies and training scenarios. A second relevant trend is that federated training often remains close to centralized training while improving over the single-institution setting in most cases. This pattern is especially visible for stronger backbones such as Qwen3-8B, 11
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
Model
MedQA
PEFT
MedMCQA
Singleinstitution
Centralized
Federated
Singleinstitution
Centralized
Federated
II-Medical-8B-1706
LoRA QLoRA IA3 Adapters
0.659 0.650 0.657
0.740 0.725 0.705
0.714 0.686 0.692
0.593 0.582 0.571
0.763 0.755 0.735
0.742 0.741 0.694
Qwen3-8B
LoRA QLoRA IA3 Adapters
0.554 0.530 0.550
0.696 0.663 0.655
0.683 0.669 0.629
0.547 0.543 0.493
0.746 0.720 0.694
0.718 0.683 0.656
Qwen3-4B-Instruct-2507
LoRA QLoRA IA3 Adapters
0.562 0.546 0.555
0.650 0.644 0.627
0.646 0.629 0.612
0.523 0.519 0.493
0.704 0.650 0.698
0.686 0.672 0.633
Qwen2.5-1.5B-Instruct
LoRA QLoRA IA3 Adapters
0.427 0.403 0.383
0.512 0.480 0.486
0.532 0.500 0.468
0.439 0.433 0.379
0.620 0.586 0.556
0.592 0.564 0.528
Llama-3.1-8B-Instruct
LoRA QLoRA IA3 Adapters
0.579 0.567 0.616
0.656 0.634 0.635
0.651 0.639 0.638
0.547 0.531 0.436
0.687 0.692 0.665
0.710 0.684 0.622
Meta-Llama-3-8B-Instruct
LoRA QLoRA IA3 Adapters
0.548 0.532 0.588
0.633 0.628 0.615
0.637 0.622 0.616
0.542 0.554 0.534
0.669 0.680 0.697
0.712 0.689 0.690
FineMedLM-o1
LoRA QLoRA IA3 Adapters
0.476 0.467 0.474
0.634 0.613 0.549
0.611 0.589 0.528
0.467 0.458 0.351
0.646 0.667 0.542
0.527 0.642 0.527
gemma-2-9b-it
LoRA QLoRA IA3 Adapters
0.568 0.541 0.594
0.675 0.677 0.645
0.665 0.653 0.634
0.515 0.492 0.510
0.703 0.714 0.665
0.691 0.693 0.657
gemma-3-4b-it
LoRA QLoRA IA3 Adapters
0.421 0.397 0.408
0.533 0.521 0.460
0.529 0.502 0.473
0.437 0.429 0.394
0.608 0.537 0.555
0.573 0.575 0.579
Table 1: Accuracy comparison across models, fine-tuning methods, and training scenarios in the medical domain. Results are reported for Single-institution, Centralized, and Federated scenarios. The highest performance is indicated in bold. Llama-3.1-8B-Instruct, and Meta-Llama-3-8B-Instruct, where federated training preserves a substantial portion of the centralized performance. Therefore, FL appears to provide an effective balance between isolated local training and full data centralization in the medical domain across many of the evaluated configurations. Regarding PEFT methods, LoRA is generally the strongest option in the medical benchmarks, with QLoRA remaining a close second in most cases, and IA3 adapters usually trailing behind. The gap is not substantial for the strongest models, but it becomes more visible for smaller or less specialized backbones, which indicates that LoRA has the most robust accuracy profile in the domain. Figure 6 provides a focused view of II-Medical-8B-1706, the strongest medical backbone in our experiments. The figure illustrates the same trend observed in Table 1: centralized training usually achieves the highest accuracy, federated training remains close, and single-institution training is consistently lower. This confirms that the federated setting captures useful cross-node information even under non-IID partitions. The figure also highlights that LoRA is the most effective PEFT method for this model, while QLoRA provides nearly the same predictive quality with a small reduction in accuracy. IA3 adapters remain competitive, but their performance is systematically lower than that of LoRA and QLoRA, particularly on MedMCQA. This indicates that, for highly specialized medical models, LoRA offers the best accuracy, whereas QLoRA may be preferred when memory efficiency is a priority. Table 2 summarizes our experiments on the financial datasets. The results are more heterogeneous than those of the medical domain, since no single model dominates all metrics and scenarios. In the centralized setting, the strongest results are obtained by Qwen2.5-1.5B-Instruct on FPB with LoRA (0.873) and by Qwen3-8B on FiQA-SA with LoRA 12
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
(a) MedQA
(b) MedMCQA
Figure 6: Accuracy for the Single-institution, Centralized, and Federated scenarios for the best model for the medical domain (II-Medical-8B-1706). (0.915). In the federated setting, the best FPB result is achieved by Qwen3-8B (0.863), while the best FiQA-SA is obtained by gemma-2-9b-it with LoRA (0.923). These results indicate that financial-domain performance is more task-dependent and that the preferred backbone may vary across benchmarks. Model
FPB
PEFT
FiQA-SA
Singleinstitution
Centralized
Federated
Singleinstitution
Centralized
Federated
Qwen3-8B
LoRA QLoRA IA3 Adapters
0.691 0.664 0.437
0.872 0.868 0.809
0.863 0.863 0.809
0.650 0.634 0.416
0.915 0.897 0.808
0.902 0.910 0.822
Qwen3-4B-Instruct-2507
LoRA QLoRA IA3 Adapters
0.636 0.678 0.670
0.870 0.869 0.789
0.847 0.842 0.779
0.655 0.740 0.854
0.897 0.880 0.863
0.885 0.876 0.859
Qwen2.5-1.5B-Instruct
LoRA QLoRA IA3 Adapters
0.637 0.580 0.663
0.873 0.851 0.764
0.831 0.832 0.762
0.684 0.679 0.803
0.859 0.855 0.803
0.859 0.838 0.829
Llama-3.1-8B-Instruct
LoRA QLoRA IA3 Adapters
0.612 0.654 0.628
0.866 0.857 0.776
0.846 0.846 0.758
0.771 0.691 0.718
0.897 0.902 0.846
0.910 0.889 0.880
Meta-Llama-3-8B-Instruct
LoRA QLoRA IA3 Adapters
0.621 0.641 0.696
0.868 0.864 0.774
0.841 0.845 0.765
0.718 0.700 0.735
0.902 0.910 0.799
0.910 0.902 0.812
gemma-2-9b-it
LoRA QLoRA IA3 Adapters
0.661 0.604 0.746
0.857 0.871 0.808
0.852 0.861 0.805
0.702 0.764 0.822
0.906 0.906 0.850
0.923 0.915 0.889
gemma-3-4b-it
LoRA QLoRA IA3 Adapters
0.576 0.612 0.677
0.865 0.859 0.763
0.844 0.835 0.732
0.711 0.697 0.856
0.863 0.859 0.842
0.880 0.872 0.868
Table 2: Accuracy comparison across models, fine-tuning methods, and training scenarios in the financial domain. Results are reported for Single-institution, Centralized, and Federated scenarios. The highest performance is indicated in bold. Similar to the medical domain, federated training is close to centralized training in most configurations, while it often outperforms single-institution training. In particular, the federated results remain competitive for several models, especially Qwen3-8B, Llama-3.1-8B-Instruct, and gemma-2-9b-it. These results show that FL can preserve a large fraction of the predictive performance even in financial tasks with highly non-IID node distributions. In terms of PEFT, LoRA again provides the most stable overall performance, although QLoRA remains highly competitive and in some cases nearly matches the LoRA results. IA3 adapters are less consistent: they are clearly weaker on some settings, but they also yield the best single-institution results for FPB and FiQA-SA with gemma-2-9b-it
13
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
and gemma-3-4b-it. This suggests that the relative effectiveness of PEFT methods in finance depends more on the backbone and the dataset when compared to their effectiveness in the medical domain.
(a) FPB
(b) FiQA-SA
Figure 7: Accuracy for the Single-institution, Centralized, and Federated scenarios for a representative financialdomain model (Qwen3-8B). Figure 7 illustrates the behavior of a representative financial domain model, namely Qwen3-8B. For both FPB and FiQA-SA, federated training remains close to centralized training, sometimes matching or slightly exceeding it, while outperforming the single-institution setting. This confirms that the collaborative setting is beneficial in finance as well, even though the absolute gains over single-institution training are somewhat benchmark-dependent. The figure also shows that LoRA and QLoRA perform similarly for Qwen3-8B, especially in the federated scenario, where their FPB results are identical, and the FiQA-SA difference is small. IA3 adapters, in contrast, lead to a more pronounced drop in accuracy. Therefore, for this backbone, QLoRA appears particularly attractive because it preserves most of the predictive power of LoRA while offering a substantially lower memory footprint, as shown in Table 3. Table 3 reports the communication cost and memory footprint of federated training. It highlights a clear trade-off between predictive performance and system efficiency. First, LoRA and QLoRA exhibit the same communication cost per round for a given model, which is expected because both methods communicate task-specific low-rank updates of similar size. In contrast, IA3 adapters consistently require much less communication, often by two orders of magnitude, making them the most communication-efficient option in the federated setting. Moreover, QLoRA is systematically the most memory-efficient among the high-performing PEFT methods. For example, with Qwen3-8B, GPU memory decreases from approximately 28.7 GB under LoRA to 20.7 GB under QLoRA, with similar reductions across the remaining backbones. This confirms that quantization substantially reduces resource requirements while preserving competitive accuracy, making QLoRA particularly suitable for resourceconstrained federated deployments. Finally, IA3 adapters and LoRA tend to have relatively similar memory footprints, even though their communication costs differ substantially. This is consistent with the fact that, in both cases, the frozen backbone remains the dominant contributor to the total memory usage, whereas the size of the trainable PEFT module only has a secondary effect on the overall footprint. Consequently, from a systems perspective, IA3 adapters are primarily advantageous for reducing communication, whereas QLoRA is a better option for reducing memory.
(a) Medical
(b) Financial
Figure 8: Memory footprint (GB) in the Federated scenario for the five best-performing models under QLoRA.
14
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
Model
PEFT
Communication cost per round (GB)
Memory footprint medical (GB)
Memory footprint financial (GB)
MedQA
MedMCQA
FPB
FiQA-SA
II-Medical-8B-1706
LoRA QLoRA IA3 Adapters
1.510 1.510 0.012
35.878 24.275 35.135
35.896 24.177 35.137
NA NA NA
NA NA NA
Qwen3-8B
LoRA QLoRA IA3 Adapters
1.510 1.510 0.018
28.712 20.750 28.519
28.713 20.756 28.520
28.713 20.570 28.518
28.712 20.616 28.517
Qwen3-4B-Instruct-2507
LoRA QLoRA IA3 Adapters
0.944 0.944 0.011
16.940 11.756 16.481
16.948 11.739 16.488
16.952 11.778 16.489
16.952 11.672 16.488
Qwen2.5-1.5B-Instruct
LoRA QLoRA IA3 Adapters
0.440 0.440 0.005
9.519 8.521 9.419
10.872 8.582 9.423
9.524 8.536 9.424
9.523 8.598 9.424
Llama-3.1-8B-Instruct
LoRA QLoRA IA3 Adapters
1.342 1.342 0.016
27.903 18.593 27.403
29.151 17.946 27.405
27.914 18.354 27.402
27.915 18.302 27.401
Meta-Llama-3-8B-Instruct
LoRA QLoRA IA3 Adapters
1.342 1.342 0.025
27.903 18.339 27.403
27.910 18.117 27.405
27.914 18.097 27.402
27.915 17.967 27.401
FineMedLM-o1
LoRA QLoRA IA3 Adapters
1.340 1.340 0.020
34.490 21.371 33.883
34.505 21.334 33.885
NA NA NA
NA NA NA
gemma-2-9b-it
LoRA QLoRA IA3 Adapters
1.541 1.541 0.018
43.499 27.915 42.905
43.504 27.581 42.908
43.506 26.977 42.909
43.506 27.984 42.909
gemma-3-4b-it
LoRA QLoRA IA3 Adapters
0.891 0.891 0.010
26.541 21.205 26.133
24.800 19.457 24.800
24.185 18.951 23.761
24.185 18.856 23.765
Table 3: Communication cost per round and memory footprint across models and fine-tuning methods for the Federated scenario. The lowest system overhead is indicated in bold. NA denotes not applicable to that domain. Figure 8 focuses on the memory footprint of the five best-performing QLoRA models in each domain. In the medical setting, the best model, II-Medical-8B-1706, also incurs the highest memory consumption, whereas Qwen3-8B, Llama3.1-8B-Instruct, and Meta-Llama-3-8B-Instruct provide a balance between accuracy and memory usage. In particular, Qwen3-4B-Instruct-2507 offers the lowest memory footprint among the top medical candidates, although at a moderate cost in predictive performance. In the financial setting, we can see a similar pattern. Gemma-2-9b-it is among the strongest models in terms of accuracy, but it is also the most memory-demanding. By contrast, Qwen3-8B, Llama-3.1-8B-Instruct, and Meta-Llama-3-8BInstruct provide competitive performance at noticeably lower memory cost, while Qwen3-4B-Instruct-2507 stands out again as the most lightweight alternative among the top-performing models. Overall, the figure suggests that QLoRA enables a meaningful reduction in memory footprint, but the final model choice still depends on the desired trade-off between efficiency and predictive quality.
7
Discussion
The results presented in Section 6 reveal several consistent patterns across domains, training scenarios, and PEFT methods. In particular, the federated setting achieves performance close to the centralized setting in most cases, often reproducing a large fraction of its predictive performance despite operating under data locality constraints. At the same time, it tends to outperform the single-institution scenario, showing that collaborative fine-tuning across distributed nodes can effectively exploit complementary information without requiring raw data sharing. A second important observation is that the most suitable model depends on the deployment objective. When the main goal is to maximize predictive performance in the medical domain, II-Medical-8B-1706 is the best option, as it achieves
15
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
the best results across MedQA and MedMCQA in all training scenarios. In contrast, for the financial domain, the results are more heterogeneous, with Qwen3-8B, Qwen2.5-1.5B-Instruct, and gemma-2-9b-it excelling on different benchmarks and scenarios. Therefore, if the priority is raw predictive accuracy, domain-specialized models appear preferable in medicine, whereas in finance, the best choice is more task-oriented. From a practical perspective, the preferred PEFT strategy depends on the deployment constraint. When the objective is maximum accuracy and computational cost is secondary, LoRA is generally the most reliable choice, as it provides strong and stable performance across most evaluated settings. When memory is the main constraint, QLoRA offers the best trade-off, consistently reducing memory usage while remaining close to LoRA in accuracy. When communication cost is the bottleneck, IA3 adapters are particularly attractive because they substantially reduce communication per round, although often at the cost of lower predictive performance. Finally, when accuracy and efficiency are equally important, models such as Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct-2507 provide a balanced compromise between predictive quality and resource requirements. Our medical and financial domain results are also useful when viewed alongside previously reported results on MedQA, MedMCQA, FPB, and FiQA-SA, including recent medical [2, 17, 13, 20, 12] and financial [15, 14, 7, 11, 10] fine-tuning works, as well as reported results for strong proprietary or domain-specific systems such as GPT-4 and Med-PaLM 2 in the medical leaderboard [22]. However, these references should be interpreted only as contextual points of comparison, not as baselines that match directly, since evaluation protocols, model classes, prompting settings, data splits, and training regimes may differ across studies. This distinction is especially important because our evaluation includes federated experiments under controlled non-IID conditions, where no raw data are centralized. Therefore, rather than claiming direct superiority or equivalence, our results indicate that federated PEFT can achieve strong performance on demanding medical QA and financial classification benchmarks under data-local federated constraints. Another important conclusion concerns the effect of non-IID data. In this study, the node partitions are explicitly controlled through a Dirichlet-based protocol with a minimum Hellinger distance threshold, which ensures a realistic but measurable degree of non-IID data. Despite this, the federated results remain close to the centralized ones in many cases, especially for stronger backbones and with LoRA or QLoRA. This suggests that, under the non-IID regimes considered here, LLM fine-tuning is affected by statistical heterogeneity but it is not severely degraded by it. In other words, non-IID data do not prevent effective federated adaptation, provided that the underlying model and PEFT strategy are sufficiently robust. This observation is aligned with recent benchmarks and systems studies in federated LLM fine-tuning, which also report that realistic client heterogeneity might be challenging but it is manageable under appropriate configurations [9, 10, 17]. It is also important to clarify the privacy scope of this work. FL reduces the need to centralize raw data by keeping training examples local to each node; however, this should not be interpreted as providing formal privacy guarantees by itself. Model updates may still leak information under certain threat models, for example, through gradient inversion, reconstruction attacks, or other forms of update-level inference. In this benchmark, we focus on the empirical behavior of PEFT-based LLM fine-tuning under federated, centralized, and single-institution settings, and we do not evaluate defenses against such attacks. In particular, mechanisms such as differential privacy, secure aggregation, trusted execution environments, or cryptographic protocols are outside the scope of the present study. Therefore, the results should be understood as evaluating federated fine-tuning under a data-locality assumption, rather than as demonstrating end-to-end privacy protection against adversarial leakage. Finally, our results also give insights from a Green AI perspective. Recent work argues that the evaluation of AI systems should account not only for predictive performance but also for computational efficiency and environmental cost [37]. Our benchmark supports this view by showing that the choice of PEFT method can substantially alter the memory and communication requirements of federated fine-tuning. In particular, QLoRA provides a clear reduction in memory footprint with negligible loss in accuracy, while IA3 adapters sharply reduce communication overhead. These results indicate that greener LLM adaptation is not only a matter of choosing smaller models, but also of selecting the appropriate fine-tuning strategy according to the deployment constraints. Therefore, efficiency-oriented reporting should be considered as an essential component of future benchmarks for federated LLM adaptation.
8
Conclusions
Recent advances in LLMs have largely relied on publicly available data for model training, but one of the next major leaps in their performance improvement will depend on access to private, institutionally held datasets. In sensitive and highly regulated domains such as healthcare and finance, these datasets contain critical knowledge that could substantially improve LLM performance and real-world utility. However, they are distributed across institutions and cannot be centralized due to privacy, regulatory, confidentiality, and organizational constraints. This paper demonstrated
16
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
that federated PEFT provides a practical mechanism to unlock these private data silos for collaborative LLM adaptation without requiring raw data sharing. To study this setting, we introduced a unified benchmark for PEFT of LLMs under centralized, single-institution, and federated training. The benchmark covers closed-ended medical and financial tasks under controlled non-IID conditions, using MedQA, MedMCQA, FPB, and FiQA-SA. Our results show that federated fine-tuning generally reaches performance close to centralized training while clearly outperforming isolated single-institution learning. This suggests that institutions can benefit from collaborative model adaptation even when their data remain local. We further show that model and adapter selection play a key role in practical deployment. Domain-specialized models are especially effective in medicine, while financial tasks exhibit more task-dependent behavior. LoRA tends to achieve the highest accuracy, whereas QLoRA provides the best balance between accuracy and memory efficiency, and IA3 offers the strongest communication efficiency. Taken together, these findings position federated PEFT as a competitive and scalable approach for data-local LLM adaptation in settings where raw data cannot be centralized. Beyond demonstrating feasibility, our results suggest that FL can help enable the next leap in LLM development by allowing models to learn from private, heterogeneous, and institutionally distributed data that would otherwise remain inaccessible.
Contributions and Acknowledgments Daniel M. Jimenez-Gutierrez Enrique Zuazua Georgios Kellaris Joaquin del Rio Oleksii Sliusarenko Xabi Uribe-Etxebarria The authors are presented in alphabetical order by first name.
References [1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730--27744, 2022. [2] Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, et al. Large language models in the clinic: a comprehensive benchmark. arXiv preprint arXiv:2405.00716, 2024. [3] Shengyuan Colin Lin, Felix Tian, Keyi Wang, Xingjian Zhao, Jimin Huang, Qianqian Xie, Luca Borella, Matt White, Christina Dan Wang, Kairong Xiao, Xiao-Yang Liu Yanglet, and Li Deng. Open finllm leaderboard: Towards financial ai readiness, 2025. [4] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. [5] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115, 2023. [6] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. [7] Varun Rao, Youran Sun, Mahendra Kumar, Tejas Mutneja, Agastya Mukherjee, and Haizhao Yang. Llms meet finance: Fine-tuning foundation models for the open finllm leaderboard. arXiv preprint arXiv:2504.13125, 2025. [8] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273--1282. PMLR, 2017. 17
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
[9] Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. Fedllm-bench: realistic benchmarks for federated learning of large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. [10] Yan Gao, Massimo Roberto Scamarcia, Javier Fernandez-Marques, Mohammad Naseri, Chong Shen Ng, Dimitris Stripelis, Zexi Li, Tao Shen, Jiamu Bai, Daoyuan Chen, et al. Flowertune: A cross-domain benchmark for federated fine-tuning of large language models. arXiv preprint arXiv:2506.02961, 2025. [11] Dannong Wang, Jaisal Patel, Daochen Zha, Steve Y. Yang, and Xiao-Yang Liu. Finlora: Benchmarking lora methods for fine-tuning llms on financial datasets, 2025. [12] Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. On the impact of fine-tuning on chain-of-thought reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11679--11698, 2025. [13] Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, et al. Med42--evaluating fine-tuning strategies for medical llms: full-parameter vs. parameter-efficient approaches. arXiv preprint arXiv:2404.14779, 2024. [14] Sorouralsadat Fatemi, Yuheng Hu, and Maryam Mousavi. A comparative analysis of instruction fine-tuning large language models for financial text classification. ACM Trans. Manage. Inf. Syst., 16(1), February 2025. [15] Dannong Wang, Daniel Kim, Bo Jin, Xingjian Zhao, Tianfan Fu, Steve Yang, and Xiao-Yang Liu. Finlora: Finetuning quantized financial large language models using low-rank adaptation. arXiv preprint arXiv:2412.11378, 2024. [16] Na Yan, Yang Su, Yansha Deng, and Robert Schober. Federated fine-tuning of llms: Framework comparison and research directions. IEEE Communications Magazine, 63(10):52--58, 2025. [17] Zeyu Chen, Yun Ji, Bowen Wang, Liwen Shi, Zijie Zeng, and Sheng Zhang. Flow of knowledge: Federated fine-tuning of llms in healthcare under non-iid conditions. arXiv preprint arXiv:2510.00543, 2025. [18] Daniel M Jimenez-Gutierrez, Aris Anagnostopoulos, Ioannis Chatzigiannakis, and Andrea Vitaletti. Fedartml: A tool to facilitate the generation of non-iid datasets in a controlled way to support federated learning research. IEEE Access, 2024. [19] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024. [20] Hongzhou Yu, Tianhao Cheng, Ying Cheng, and Rui Feng. Finemedlm-o1: Enhancing the medical reasoning ability of llm from supervised fine-tuning to test-time training, 2025. [21] Agrin Hilmkil, Sebastian Callh, Matteo Barbieri, Leon René Sütfeld, Edvin Listo Zec, and Olof Mogren. Scaling federated learning for fine-tuning of large language models. In International Conference on Applications of Natural Language to Information Systems, pages 15--23. Springer, 2021. [22] Aaditya Ura, Pasquale Minervini, and Clémentine Fourrier. The open medical-llm leaderboard: Benchmarking large language models in healthcare. https://huggingface.co/blog/leaderboard-medicalllm, April 2024. Hugging Face blog post, accessed 2026-04-10. [23] Daniel M Jimenez-Gutierrez, David Solans, Mikko Heikkila, Andrea Vitaletti, Nicolas Kourtellis, Aris Anagnostopoulos, and Ioannis Chatzigiannakis. Non-iid data in federated learning: A systematic review with taxonomy, metrics, methods, frameworks and future directions. arXiv e-prints, pages arXiv--2411, 2024. [24] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pages 965--978. IEEE, 2022. [25] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263, 2021. [26] Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941--1942, 2018. [27] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782--796, 2014. 18
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
[28] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. [29] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR, 2022. [30] Intelligent Internet. Ii-medical-8b: Medical reasoning model, 2025. [31] Qwen Team. Qwen3 technical report, 2025. [32] Qwen Team. Qwen2.5: A party of foundation models, September 2024. [33] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ammar Al-Dahle, Adam Letman, Anant Mathur, Alan Schelten, Angela Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [34] AI@Meta. Llama 3 model card, 2024. [35] Gemma Team. Gemma, 2024. [36] Gemma Team. Gemma 3, 2025. [37] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54-63, 2020.
19