LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI∗ Technical Report Daniele Lotito

Alexandru Iosup

IBM Research Europe Dublin, Ireland daniele.lotito@ibm.com

Vrije Universiteit Amsterdam Amsterdam, The Netherlands a.iosup@vu.nl

Serverless execution of Large Language Models (LLMs) is becoming important, reflecting their wide societal use. An increasingly present stage in LLM pipelines is fine-tuning, which specializes trained models and improves inference performance. Serverless operation of LLM fine-tuning workloads shifts operational concerns from the user to the provider. For these new workloads, providers face important operational complexities, such as performance issues, resource waste, and even service failure due to misconfiguration. However, optimizing such workloads is still at an early stage, and, relevant to this work, relatively little is currently known about how LLM fine-tuning workloads run in practice. In this work, we address the knowledge gap by characterizing the execution of LLM fine-tuning workloads on clusters equipped with different GPUs, running LLM models at various scales. We analyze IBM’s LLMFineTuningBench, a large-scale, diverse profiling dataset of 30,920 fine-tuning experiments that records both successful and failing configurations. We provide quantitative insight into throughput (expressed in tokens/s), scaling, failure, and potential for performance optimization for four commonly used GPU types, in setups of 1 to 32 GPUs. We find that (1) job feasibility (run or fail) depends on workload properties, with failure ratios rising from 52% for small batch sizes (16-64) up to 95% for the largest (1,024); (2) throughput grows logarithmically with batch size; (3) scaling is near-linear when the workload is deployed intra-node, and optimizations such as RoCE substantially improves efficiency for inter-node deployment (e.g., 0.72 vs. 0.26 efficiency without RoCE); and (4) system-level optimizations can outweigh application-level ones (RoCE, up to 2.9×, vs. Fast MoE, 1.7×). CCS Concepts: • Computer systems organization → Cloud computing. Keywords: datacenters, LLM fine-tuning, workload characterization, availability, performance, serverless, IBM LLMFineTuningBench

∗ Technical report of the article published at Middleware WoAIS ’26 [43].

THIS WORK

arXiv:2610.40983v1 [cs.DC] 29 Oct 2026

Abstract

BEFORE

Radu Nicolae Vrije Universiteit Amsterdam Amsterdam, The Netherlands r.nicolae@vu.nl

7

1

2

á provider: by intuition

A

guess GPUs, nodes, and batch size

Job

5

4

A Job

LLMFine TuningBench (from [33])

3

8

á provider: with evidence Run? (§3.1)

How fast? (§3.2)

Scale? (§3.3)

Faster? (§3.4)

this work (§3): dataset analysis (30,920 entries)

6

9 opportunities & challenges (§4)

Figure 1. Fine-tuning as a service configured blindly, based on intuition ( 1 - 3 ), compared to evidence-based ( 4 - 9 ). We characterize LLMFineTuningBench [33] in §3 and offer actionable insights in §4.

1

Introduction

Our society and economy are increasingly reliant on AI services, especially Large Language Models (LLMs), which are increasingly adopted and embedded across academia [1, 32, 37], government [5, 45, 48], and industry [21, 40]. Subject to cost and scalability pressures, the typical training-inference two-step pipeline has evolved: cheaper and smaller-scale than training, and placed in the pipeline in-between training and inference, LLM fine-tuning is becoming a common approach to specializing LLMs and improving their inference performance. As LLM fine-tuning adoption increases, serverless execution could bring important performance improvements, efficiency gains, and operational transparency [29]. Serverless LLM fine-tuning raises known operational concerns, including throughput, scalability, resource use, related efficiency trade-offs, and whether an automatically configured job is feasible (completes or fails). This raises the challenge that, as a new workload, LLM fine-tuning has largely unknown execution characteristics. Although current characterizations of GPU clusters analyze production traces of deep-learning jobs [17, 24, 58] and of LLM pretraining and evaluation [18], only anecdotal evidence exists on LLM finetuning performance and feasibility [50, 61]. Addressing this challenge, in this work we characterize LLM fine-tuning workloads in GPU-based clusters.

Radu Nicolae, Daniele Lotito, and Alexandru Iosup

Currently, massive-scale and powerful LLMs [3, 35], such as Llama [55], Mistral [25], and Granite [39], are foundational for LLM operations. However, retraining LLMs from scratch is rapidly pricing out even the largest non-Big Tech enterprises [46], leaving fine-tuning among the few performanceand cost-sustainable techniques for enterprise or smallerscale specialization [6]. Consequently, platforms such as Amazon Bedrock [2], Databricks Mosaic AI [27], and IBM’s watsonx.ai [19] increasingly offer fine-tuning as a service. The community is working towards serverless operations for LLM, based on quantitative evidence about how LLM workloads run in practice. As illustrated in Figure 1 for finetuning, the vision is to replace the current approach to configuring LLM fine-tuning jobs, which is based on anecdotal evidence and intuition, with a data-driven approach based on shared profiling information and (as in this work) characterization. Anecdotes and intuition are unlikely to overcome the rapid increase in complexity and unpredictability of finetuning workloads [6, 16]. As a lesson learned by operators across decades, relying on users to provide accurate estimates of resource use has proven infeasible [30]. As supported by the evidence presented in this work (Figure 1, component 8 , detailed in Section 3) jobs configured using these approaches have a high likelihood of failure (boxes marked with × in 3 ) or high performance degradation (orange boxes). Conversely, changing to an evidence-based approach should alleviate performance and failure ( 6 ). Section 4 discusses opportunities and challenges related to this change. Configuring LLM jobs is not easy. Configurations need to account for diverse GPU types for cost and efficiency reasons [13]. Misconfiguration could trigger worse performance or less efficient use of (GPU) resources. The most important effect, job failure, occurs for example when settings lead to out-of-memory crashes [63, 64]. Worse, when one job fails it could, in turn, cause further, correlated failures [11, 62]. Operators of LLM infrastructure are already collecting data about fine-tuning in practice. IBM released in September 2026 an important dataset [33] ( 7 , Figure 1), which is the largest-of-its-kind profiling effort covering 30,920 finetuning experiments across 33 LLMs and 4 commonly used GPU types. The FAIR [60] dataset LLMFineTuningBench covers the configuration space systematically, via step-wise exploration, and includes failed configurations. In contrast to production traces, LLMFineTuningBench facilitates comprehensive characterization, but records no job arrival patterns. Related work: Fine-tuning studies measure a few LLMs on a few GPU types [50, 61], and fine-tuning schedulers, e.g., Ymir [12], use these measurements as input. Trace archives for workloads [22], failures [28], and workflows [56] have proven the value of data collection and characterization, but do not cover LLM fine-tuning. IBM’s AutoConf [20] uses LLMFineTuningBench to predict configuration feasibility, but its core ML-model does not improve understanding

Table 1. Summary of LLMFineTuningBench [33], per GPU type. #Exp. = experiment count; ✓ = ratio of jobs that ran successfully, ; = rejected before launch, p = failed at runtime. Dataset = entire dataset. Rows sum to 100%. GPU type Mem. #Exp. ✓[%] ;[%] p[%] LLMs𝑏 GPUs Nodes A100-SXM𝑎 80 GB 27,837 36.6 31.5 31.9 24 1-32 1-4 A100-PCIe 80 GB 1,818 42.2 11.3 46.5 13 1-4 1 L40S 48 GB 1,068 30.8 15.5 53.7 9 1-4 1 H100-PCIe 80 GB 197 51.3 1.0 47.7 3 1-4 1 Dataset 30,920 36.8 29.6 33.6 33 1-32 1-4 𝑎 A100-SXM is the only GPU type with multi-node jobs, RoCE, and GPTQ-LoRA. 𝑏 Different combinations of the 33 LLMs tested overall are tested for each GPU type.

Table 2. Four questions under analysis, by subsection (§). ID Question (§) Metric Analysis target Q1 Will the job run? (3.1) failure ratio [%] per factor Q2 How fast? (3.2) throughput [tokens/s] median per batch size Q3 Will it scale? (3.3) efficiency intra- vs. inter-node Q4 How to go faster? (3.4) speedup [×] matched pairs

of configuration effects. Lotito et al. [34] propose early MLmodels for performance. However, no prior work on LLM fine-tuning matches this work’s focus on characterization, scope, scale, and diversity. We make a four-fold contribution: 1. We propose an analysis method (§2), structured around questions and statistical approaches to answer them. 2. We conduct the analysis and provide a detailed characterization of LLM fine-tuning workloads in practice (§3): whether jobs run and the key levers to job feasibility, job performance, and job scalability, and how to make them faster. 3. We present actionable insights related to the characterization (§4). We envision an online recommender system that helps auto-configure LLM fine-tuning workloads as a key part of the serverless infrastructure, and discuss its challenges and opportunities. 4. Open science: We release the reproducibility capsule for this project, which includes the analytical tools and results: github.com/atlarge-research/llm-finetuningworkload-characterization

2

Dataset Analysis Method

Our method has three stages: (1) data collection, here, already done by IBM ( 7 , Figure 1), (2) processing, and (3) analysis, done in this work ( 8 , Figure 1). For stage 1, Lotito et al. at IBM collect LLMFineTuningBench [33] with the SFTTrainer actuator of the open-source ado framework [26], which runs fms-hf-tuning jobs [8] over a systematic sweep of configurations. The dataset covers 33 LLMs of various scales (0.35B-405B parameters), 3 fine-tuning methods, total batch sizes of 20 -210 , and 29 -213 tokens per sample (sequence length). The fine-tuning methods are full [7], LoRA [16], and GPTQ-LoRA [10], which applies LoRA to a GPTQ-quantized LLM. The dataset also covers three optimizations, whose impact we analyze: (1) Fast Kernels, fused

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

(a) Batch Sizes 518 1,409 1,281 2,331 2,221 1,864 1,395 1,398 1,838 2,400 2,884

1024 512 256 128 64 32 16 8 4 2 1

0

50

100

(b) Sequence Lengths

(c) GPU Models

8192

6,330

L40S

4096

2,027

2048

5,257

A100 SXM A100 PCIe H100 PCIe

1024

1,623

512

4,302

0

50

100

(d) Fine-tuning Methods

739

Full

11,177

GPTQ

776

LoRA

7,586

17,654 1,050 96

0

Failure Ratio [%]

50

100

0

rejected before launch

50

100 failed at runtime

Figure 2. Failure ratios per (a) total batch size, (b) sequence length, (c) GPU model, and (d) fine-tuning method (GPTQ=GPTQLoRA). Panels (c) and (d) are sorted by failure ratio. Rows depict the failure ratio (bar and %) and the number of failing experiments (count), e.g., 69% (739) for (c) L40S. Hatched: rejected before launch; solid: failed at runtime. low-level kernels [54]; (2) Fast MoE, kernels of mixture-ofexperts with expert parallelism [15]; and (3) RoCE, RDMA over Converged Ethernet, which lets nodes exchange data directly between their memories, over Ethernet, without involving the CPU or the operating system [65]. As Table 1 summarizes, for this dataset, LLMs run on 4 GPU types: L40S as an enterprise GPU, A100 with two interfaces as a powerful GPU, and H100 as the expensive but fastest-in-class GPU. (2) Data processing: We keep all 30,920 experiments from the dataset, unchanged. For each experiment, we extract its outcome (valid, rejected before launch, or failed at runtime), its batch size per GPU, and its optimization technique. We infer from system policies external to the dataset that invalid experiments are rejected before launch if the batch size is not a multiple of the GPU count (5,360 experiments); or, when the model uses parallel experts, if the GPU count is not a multiple of the expert-parallel (EP) degree (3,784). (3) Data analysis: We define four key questions a provider encounters when deploying LLM fine-tuning jobs (Table 2), each with a metric and a target for analysis.

3

LLM Fine-Tuning Characteristics

We follow the method introduced in Section 2, use the questions from Table 2, and conduct a detailed characterization of LLM fine-tuning workloads running in GPU-based clusters. For each question, we begin with the most important findings, labeled Finding i for the ith finding, then present our detailed analysis. 3.1

(Q1) Failures: Will the Job Run?

Job failures, especially unrecoverable ones, delay execution, can cause queue bottlenecks, and, if failing at runtime, also waste GPU time, energy, and ultimately money [28, 51]. We compute the failure ratio across four factors and illustrate the results in Figure 2. All failures are unrecoverable. Finding 1: 63% experiments yield no valid run. Batch size and sequence length strongly correlate with runtime failure.

Finding 2: Failures follow two batch-related trends: small batches are mostly rejected before launch because they cannot be evenly split across GPUs, and large batches mostly fail at runtime, likely because they exceed GPU memory. Overall, 19,539 experiments (63%) yield no valid run, with 9,144 rejected before launch and 10,395 failed at runtime. The high failure ratio indicates that naïve configurations should not be used. However, the distribution of failures across batch size and sequence length, Figure 2a and b, respectively, suggests that profiling can identify values for these factors where the failure likelihood is significantly lower. Failures vs. batch size: The failure ratio is U-shaped in the batch size: from 86% at batch size 1, which are mostly rejected before launch, it falls to 42% at 8, then rises to 95% at 1,024. Thus, a profiling approach efficiently offers, through sampling, insights into failure and success patterns. Failures vs. sequence length: Similarly, failures grow with sequence length, but follow a logarithmic trend (Figure 2b, vertical axis is power-of-2) rather than a U-shape, because both extreme batches (low or high) and extreme sequence lengths (high) tend to overflow GPU memory. However, the main growth with higher sequence length comes from runtime failures, which increase from 21% to 46% of experiments. The experiments rejected before launch stay within 26-31%, because the rules of the job runner (actuator) in the IBM experiments do not involve sequence length [26]. Failures vs. GPU type: When computational resources are scarce, feasibility under low memory becomes a primary concern alongside performance degradation [33]. The dataset shows that the 48 GB L40S has the highest failure ratio (69%, Figure 2c) in 739 of its 1,068 experiments, mostly at runtime (54%), while A100-PCIe, with 80 GB, shows 58% failures in 1,050 of its 1,818 experiments, consistent with its larger memory, though model mixes differ. Failures vs. fine-tuning method: Methods that need less memory lower hardware requirements [16]; full fine-tuning, the most memory-hungry, fails 67% of runs (Figure 2d), while

A100-SXM

A100-PCIe

L40S

H100-PCIe

10,000 7,500 5,000 2,500 0

1

2

4

8

16

32

64

128

Median Throughput [tokens/s]

Median Throughput [tokens/s]

Radu Nicolae, Daniele Lotito, and Alexandru Iosup

2.4k 2.5k 3.3k 5k 5.3k 8k 12k 16k

512

1,024

3.3k 4.2k 7.2k 13k 17k 22k 33k 42k 70k

2.9k 4k 5.1k 7.5k 9.1k 18k 29k 44k

2,048

4,096

4.6k 6.7k 11k 20k 29k 38k 55k 75k

80k 50k 30k 15k 5k 0

Median Throughput [tokens/s]

Batch Size [#]

Figure 3. Throughput of fine-tuning workloads deployed on a single-GPU cluster, by GPU type. 1.4k 1.9k 2.6k 4.2k 7.6k 12k 17k 22k 39k 56k 64k

no valid run no experiment

8,192

Sequence Length [tokens] Figure 4. Throughput of all 11,381 valid runs, per total batch size (rows) and sequence length (columns). Empty cells: no valid run; hatched: not tested. LoRA and GPTQ-LoRA train only a fraction of the parameters and fail less often. At runtime, LoRA fails less often than full fine-tuning (30% vs. 36%), consistent with lower memory needs for LoRA. However, memory is not always the cause; job distribution and the hardware used also matter: GPTQ-LoRA, for example, fails most at runtime (40%), yet ran only on A100-SXM. Implications: Automatic pre-launch checks can alleviate both classes of failures: rejection before launch and runtime failure. A rule check for feasibility can repair a configuration, instead of rejecting it, for example by adjusting the batch size to a multiple of the number of GPUs. A memory check can limit the batch size and sequence length based on available GPU memory, helping prevent runtime failure [33]. 3.2

LoRA

GPTQ-LoRA

30,000 20,000 10,000 0

1

2

4

1 node

2–4 nodes

8

16

32

Number of GPUs [#]

Batch Size [#]

1 2 4 8 16 32 64 128 256 512 1,024

Full

(Q2) Performance: How Fast Will the Job Run?

Throughput directly influences how many GPU hours a finetuning job costs and the derived metrics of the job (e.g., energy consumption, financial cost, cluster usage, time in queue). The dataset measures throughput as the number of tokens a job processes per second; for the batch-size trend, to exclude the effect of adding GPUs, we use only the 2,320 valid runs on a single GPU. Finding 3: Throughput grows logarithmically with the batch size: on one A100-SXM GPU, batch size 8 vs. 1 increases throughput by 2.3×, yet batch 128 (vs. 8) increases throughput only by 1.5×.

Figure 5. GPU scaling per workload (1 node = 8 GPUs).

Throughput vs. batch size: Figure 3 illustrates a logarithmic relationship between throughput and batch size: each doubling of the batch size adds approximately the same amount, about 1,000 tokens/s on A100-SXM. For this GPU, the throughput rises from 2,847 tokens/s at batch 1 to 6,616 at 8, but only to 9,627 at 128, where the memory starts to saturate; beyond batch size 128, no single-GPU run succeeded. Throughput vs. GPU type: Because each GPU type ran different models, we can compare GPU types only on paired experiments where the only variable is the GPU type. In all 103 pairs, the 48 GB L40S is slower than the 80 GB A100-PCIe, with a median throughput 32% lower. The A100-SXM, in turn, is 18% faster than A100-PCIe in all 49 pairs. However, we cannot pair H100-PCIe because the dataset has no identical job on H100-PCIe and another GPU type. Throughput vs. sequence length: Throughput mostly grows with both batch size and sequence length, as Figure 4 shows, from 1.4k tokens/s at (batch size, sequence length) of (1, 512) to 75k at (128, 8 192) on 16 or 32 GPUs. However, increasing further leads to infeasible, failing jobs. Implications: Failed or only narrowly faster runs are expensive, which affects small budgets (e.g., academic groups) [51]. Large batches only narrowly increase throughput but raise failure risk (Finding 1), leaving moderate batches as the allaround choice.

3.3

(Q3) Scalability: Will the Job Scale with GPUs?

The cost of more GPUs is worth it only if throughput amortizes it, and the extra GPUs do not sit idle or waste time communicating [34]. Figure 5 shows median throughput versus the number of GPUs (1 to 32) across homogeneous nodes with 8 GPUs each; up to 8 GPUs, the jobs run intra-node and, otherwise, inter-node. Because different LLMs ran at different GPU counts, we also analyze the weak-scaling efficiency on matched pairs, jobs that differ only in the number of GPUs and keep the same batch size per GPU (efficiency of 1 means that doubling the GPUs also doubles the throughput).

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

Finding 4: Intra-node, throughput scales almost linearly with the number of GPUs (efficiency of 0.94-1.00). Across nodes, scaling is also influenced by the run setup and the network: from 8 to 16 GPUs (1 to 2 nodes), efficiency is 0.98 for runs without a time limit, but drops to 0.26 without RoCE and to 0.72 with RoCE for runs capped at 600 s. Within a node: Up to 8 GPUs, all three methods scale nearlinearly: median throughput grows from 3,100-4,300 tokens/s on one GPU to 28,700-33,500 on 8, and matched efficiency stays between 0.94 and 1.00 for every intra-node doubling. With identical total batch (strong scaling), efficiency drops to only 0.85-0.88 (104-207 pairs), mainly because individual GPUs receive less work. Across nodes: From 8 to 16 GPUs, thus, on 2 nodes, GPUs also need to communicate over the network, which can affect efficiency. However, runs without a time limit (fms-hf-tuning 2.4.0) gain 1.95-1.96× throughput with and without RoCE, while runs capped at 600 s (fms-hf-tuning 2.7.1) gain only 1.45× with RoCE. The same 600 s-capped jobs, without RoCE, slow to 0.52×, equivalent to doubling the compute power and halving the throughput. From 16 to 32 GPUs (2 to 4 nodes), the same 600 s-capped jobs gain 1.88× without RoCE and only 1.90× with it, showing that when jobs already run distributed across nodes, adding more nodes brings close to linear gains, with efficiency at 0.94-0.95. Scaling vs. fine-tuning method: Up to 8 GPUs, the three methods scale alike; above 8 GPUs, the LoRA curve from Figure 5 drops below full fine-tuning because the LoRA runs used much larger models (median 47B parameters) than the full fine-tuning runs (8.1B). However, when the same job runs with both methods, LoRA is only 9% faster than full fine-tuning (median for 3,744 pairs). Implications: Jobs up to 8 GPUs should run within one node, where doubling the GPUs also (nearly) doubles the throughput. The inter-node world is less predictable, with gains depending on the run setup and network: without RoCE, some jobs ran at half their one-node speed even when using twice the GPUs.

3.4

(Q4) Optimization: Faster Fine-Tuning?

Fine-tuning optimizations, such as LoRA, can lower the already high multi-tier cost associated with LLM workloads [34, 61]. LLMFineTuningBench profiles three optimizations (Section 2): (a) RoCE, a system-level optimization that speeds up the data exchange between nodes [14]; (b) Fast MoE, an application-level optimization that speeds up the expert layers of MoE models [9]; and (c) Fast Kernels, an applicationlevel optimization that replaces the standard GPU code of frequent training computations (e.g., loss, normalization layers) with faster, hand-optimized code [9, 54]. Figure 6 depicts speedups of each optimization.

(a) RoCE no time limit 600 s limit

1.03×

241 270

2.89×

(b) Fast MoE Full, EP 1 Full, EP 2 Full, EP 4 Full, EP 8 LoRA, EP 1 LoRA, EP 2 LoRA, EP 4 LoRA, EP 8

2.43× 174 1.86× 1.78× 1.71×

514 364 199 153 446 295 146

1.44× 1.08× 1.02× 0.92×

(c) Fast Kernels

1.12×

all pairs 0

1

2,418

2

3

Speedup [×]

Figure 6. Median speedup over matched pairs of jobs that differ only in one optimization: (a) RDMA over Converged Ethernet, (b) faster code for MoE layers, or (c) faster code for common training operations. EP: expert-parallel degree; whiskers: middle 50%; gray: number of pairs. Finding 5: At system level, RoCE speeds up multi-node jobs, especially runs capped at 600 s: 2.89×, vs. 1.03× without a time limit. Finding 6: At application level, Fast MoE speeds up MoE LLMs by 1.69× when all experts stay single-GPU, but only 1.16-1.33× when the experts spread over 2, 4, and 8 GPUs. Fast Kernels, however, add only 12% throughput (1.12×). System-level, RoCE (Figure 6a): Across 511 job pairs on 2-4 nodes, RoCE speeds up fine-tuning by a median of 2.16×. This median, however, mixes two classes of runs with contrasting results. Runs without a time limit gain only 3% throughput (1.03×, 241 pairs); on 2 nodes, they gain a mere 1%. Runs with a 600 s time limit gain 2.89× (a quarter of them above 3.5×). The two classes also differ in gradient accumulation (mostly 4 vs. 1). Fast MoE (Figure 6b): MoE models send each token only through a small fraction of their expert sub-networks [25], and Fast MoE runs these experts with hand-optimized GPU code [9, 53]. When the experts stay on 1 GPU (EP=1), Fast MoE speeds up fine-tuning by a median of 1.69×: 2.43× for full fine-tuning and 1.44× for LoRA. The speedup gap arises because LoRA leaves the expert weights frozen, as follows from [16], and there is less expert computation left to speed up. When the experts spread over GPUs (EP>1), we identify a decreasing trend, also illustrated in Figure 6b; for example, for LoRA at EP=8, speedup even becomes a slowdown (0.92×), because exchanging tokens between GPUs, which expert parallelism requires [15], takes time the faster code cannot amortize. The higher the EP, the more communication dominates, and slowdowns occur when latency hiding is no longer feasible.

Radu Nicolae, Daniele Lotito, and Alexandru Iosup

Table 3. Actionable insights: techniques and candidate systems that could use the findings of Section 3: (F1) U-shaped, batch-driven failure ratio, (F2) batch-dependent pre-launch and runtime failures, (F3) logarithmic batch-size throughput growth, (F4) near-linear intra-node, variable inter-node scaling, (F5) setup-dependent system-level RoCE speedups, and (F6) application-level MoE and kernel speedups. Each finding informs a technique directly (○), partly (), or not (○). Finding Technique Key evidence 1 2 3 4 5 6 Potentially benefiting systems Objective Admission control 63% of experiments fail ○ ○  ○ ○ ○ AutoConf [20], Ymir [12] Fewer failures Right-sizing Batch 8→128: only 1.5× throughput   ○  ○ ○ Morph. [57], Coastline (§5) [41] Resource use Placement Intra-node efficiency 0.94-1.00 ○ ○ ○ ○  ○ FGD [59], AlpaServe [31] Resource use, latency Optimization choice Speedups up to 2.9× ○ ○ ○  ○ ○ INFaaS [47], Coastline [41] Throughput Provisioning 11-65% fewer GPU hours  ○   ○ ○ SpotServe [38] Cost-efficiency Workflow analysis Tracing Throughput and failure profiles   ○ ○ ○ ○ TraceIT [49] Timing reconstruction Simulation What-if models Matched pairs as ground truth   ○ ○ ○ ○ OpenDC [36], Kavier [42] Long-term planning Digital twinning Online recalibration Calibration data ○ ○ ○ ○ ○ ○ OpenDT [44], ExaDigiT [4] Live operations Domain Operations

Fast Kernels (Figure 6c): They have a smaller but steady speedup of 1.12× over 2,418 pairs of seven models of up to 8B parameters. Half of these pairs gain 5-15%. Implications: We identify that the optimizations mostly save GPU hours (hence, energy and money): 11% saved for Fast Kernels, 65% for RoCE in runs capped at 600 s, but also minor losses such as LoRA at EP=8 (0.92×, a slowdown). Based on these findings, a provider can enable Fast Kernels by default, use Fast MoE for LLM experts that fit on one GPU, and enable RoCE for multi-node jobs after checking that jobs benefit (in the dataset, jobs with a 600 s limit).

4

Actionable Insights

In this section, we present actionable insights that help achieve the vision formulated in the introduction: A serverless infrastructure for LLM fine-tuning, where profiling data and resulting characterization help achieve better configurations. Table 3 links our main findings from Section 3 with four domains to improve toward our vision. For each domain, we identify techniques and specific objectives to be improved by leveraging the key evidence produced in this work. We also identify concrete examples of systems (tools) that can help with the objectives. We now address each domain. Operations: LLMFineTuningBench and the correlated findings can be used as input to improve dynamic decisionmaking, resource management, and scheduling to improve latency [31] or throughput [47], resource use [57, 59], and cost-efficiency [38]. For fine-tuning, analysis of the feasibility boundary (e.g., our F1, F2) or batch sizes (e.g., F3) can guide analysis-driven admission control in tools such as AutoConf [20] or Ymir [12]. Accessibility is also rapidly becoming an operational concern: The ongoing GPU scarcity limits accessibility [34]; moreover, accessible units across the continuum could be subject to specific licensing, regulations, and other concerns that restrict operation [23]. Workflow analysis: Our findings on throughput and failure profiles per configuration (F1-F3) can also be used to enable performance analysis for complex serverless workflows, when combined with other tracing and profiling techniques,

such as TraceIT [49] for timing reconstruction when asynchronous and third-party services are part of the workflow. Simulation: The matched pairs (F4-F6) can serve as measured and processed reality and can be used as input for complex simulations [36, 52] for long-term analysis and optimization processes, for example with Kavier [42], to evaluate resource management and scheduling policies for performance, capacity planning, cost-efficiency, but also for the more emerging sustainability concerns; we envision such analysis, prior to deployment, to become a state-of-the-art procedure. Simulators such as Kavier [42] could leverage LLMFineTuningBench to derive analytical or ML-based models to predict LLM fine-tuning performance, or to calibrate existing models using the measured reality. Digital twinning: As an overarching goal, the IBM dataset can be used as input for digital twinning of live operations [44], leveraging all the aforementioned classes of actions benefiting from LLMFineTuningBench: (1) operations, (2) workflow analysis, or (3) simulation. A digital twin could leverage LLMFineTuningBench as calibration data or as a ground truth (better: measured reality) for the twin to use when making decisions on infrastructure steering, either automated or with a human-in-the-loop [44]. Such a digital twin, applied to LLM fine-tuning workloads, could use Kavier [42] as its simulation engine and Coastline [41] as its advisor, with both helping steer serverless infrastructure toward established service level objectives (SLOs) and agreements (SLAs).

5

Conclusion and Ongoing Work

Serverless LLM fine-tuning is an increasingly common workload class, which transfers the complexity of job configurations to providers. However, only scarce empirical evidence exists on how these workloads deploy, run, and succeed. Using the public dataset LLMFineTuningBench released by IBM, in this work we characterize LLM fine-tuning workloads in GPU-based clusters. We find that (1) failures, frequent due to the sweep design, follow two batch-dependent modes, (2) throughput only grows logarithmically with the

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

batch size, (3) scaling is near-linear within a node, but setupdependent across nodes, and (4) optimizations such as Fast Kernels, RoCE, or Fast MoE could yield significant speedups. Based on our findings, we derive actionable insights (§4). Ongoing work: Toward reliable, fast, and efficient LLM fine-tuning, we pursue three directions of increasing magnitude. First, in the operations domain, we are integrating the feasibility boundary (Findings 1-2) and observed performance trends (Findings 3-6) into Coastline [41], a system that helps users configure fine-tuning jobs. For simulators, we are contributing to Kavier [42], an analytical discrete-event simulator, by expanding its capabilities with fine-tuning, calibrated and validated against LLMFineTuningBench, which serves as measured ground truth (Findings 3-6). Last, for digital twins, we envision that a digital counterpart of the real-world ecosystem could steer toward objectiveoriented outcomes to better meet established SLOs and SLAs. Integrating the operations and simulator tools described earlier could help achieve this vision. Furthermore, this is a call for the entire community to help develop meaningful and versatile digital twins that can become trusted advisors-onthe-side in serverless LLM operations.

Acknowledgments This work was partially supported by EU MSCA AIStars4EU (g.a. 101341806) and partly supported by a National Growth Fund through the Dutch project 6G Future Network Services.

References [1] Adhari AlZaabi, Amira ALamri, Halima Albalushi, Ruqaya Aljabri, and AbdulRahman AalAbdulsalam. 2023. ChatGPT applications in academic research: a review of benefits, concerns, and recommendations. bioRxiv (2023). [2] Anthropic. 2024. Fine-tune Claude 3 Haiku in Amazon Bedrock. https: //www.anthropic.com/news/fine-tune-claude-3-haiku. Accessed: 2026-09-29. [3] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, et al. 2021. On the Opportunities and Risks of Foundation Models. CoRR abs/2108.07258 (2021). [4] Wesley Brewer, Matthias Maiterth, Vineet Kumar, Rafal P. Wojda, Sedrick Bouknight, Jesse Hines, Woong Shin, Scott Greenwood, David Grant, Wesley Williams, et al. 2024. A Digital Twin Framework for Liquid-cooled Supercomputers as Demonstrated at Exascale. In SC. [5] Ziqing Dai. 2024. Applications and Challenges of Large Language Models in Smart Government-From technological Advances to Regulated Applications. In FAIML. [6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In NeurIPS. [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. [8] Foundation Model Stack. 2023. fms-hf-tuning: Tuning Recipes with HuggingFace SFTTrainer and PyTorch FSDP. https://github.com/ foundation-model-stack/fms-hf-tuning. [9] Foundation Model Stack. 2024. fms-acceleration: Libraries to Accelerate Fine-Tuning with fms-hf-tuning. https://github.com/foundation-

model-stack/fms-acceleration. [10] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In ICLR. [11] Matthieu Gallet, Nezih Yigitbasi, Bahman Javadi, Derrick Kondo, Alexandru Iosup, and Dick H. J. Epema. 2010. A Model for SpaceCorrelated Failures in Large-Scale Distributed Systems. In Euro-Par. [12] Wei Gao, Weiming Zhuang, Minghao Li, Peng Sun, Yonggang Wen, and Tianwei Zhang. 2024. Ymir: A Scheduler for Foundation Model Fine-tuning Workloads in Datacenters. In ICS. [13] Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. CoRR abs/2404.14527 (2024). [14] Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. 2016. RDMA over Commodity Ethernet at Scale. In SIGCOMM. [15] Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. 2021. FastMoE: A Fast Mixture-of-Expert Training System. CoRR abs/2103.13262 (2021). [16] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR. [17] Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, and Tianwei Zhang. 2021. Characterization and prediction of deep learning workloads in large-scale GPU datacenters. In SC. [18] Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, et al. 2024. Characterization of Large Language Model Development in the Datacenter. In NSDI. [19] IBM. 2025. IBM watsonx.ai: Model Customization and Tuning. https: //www.ibm.com/products/watsonx-ai. Accessed: 2026-09-29. [20] IBM Research. 2026. AutoConf: GPU Configuration Validity Classifier for LLM Fine-Tuning. https://github.com/IBM/ado/tree/main/plugins/ custom_experiments/autoconf. [21] Shashikant Ilager, Lukas Florian Briem, and Ivona Brandic. 2025. GREEN-CODE: Optimizing Energy Efficiency in Large Language Models for Code Generation. CoRR abs/2501.11006 (2025). [22] Alexandru Iosup, Hui Li, Mathieu Jan, Shanny Anoep, Catalin Dumitrescu, Lex Wolters, and Dick H. J. Epema. 2008. The Grid Workloads Archive. FGCS (2008). [23] Matthijs Jansen, Auday Al-Dulaimy, Alessandro V. Papadopoulos, Animesh Trivedi, and Alexandru Iosup. 2023. The SPEC-RG Reference Architecture for The Compute Continuum. In CCGrid. [24] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In USENIX ATC. [25] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of Experts. CoRR abs/2401.04088 (2024). [26] Michael A. Johnston and Alessandro Pomponio. 2026. ado: a Python framework for computational experimentation and benchmarking. JOSS (2026). [27] Daniel King, Nancy Hung, and Kasey Uhlenhuth. 2024. Introducing Mosaic AI Model Training for Fine-Tuning GenAI Models. https://www.databricks.com/blog/introducing-mosaic-ai-modeltraining-fine-tuning-genai-models. Accessed: 2026-09-29. [28] Derrick Kondo, Bahman Javadi, Alexandru Iosup, and Dick H. J. Epema. 2010. The Failure Trace Archive: Enabling Comparative Analysis of Failures in Diverse Distributed Systems. In CCGrid. [29] Samuel Kounev, Nikolas Herbst, Cristina L. Abad, Alexandru Iosup, Ian T. Foster, Prashant J. Shenoy, Omer F. Rana, and Andrew A. Chien.

Radu Nicolae, Daniele Lotito, and Alexandru Iosup

2023. Serverless Computing: What It Is, and What It Is Not? CACM (2023). [30] Cynthia Bailey Lee, Yael Schwartzman, Jennifer Hardy, and Allan Snavely. 2004. Are User Runtime Estimates Inherently Inaccurate?. In JSSPP. [31] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, et al. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In OSDI. [32] Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. 2024. Mapping the increasing use of LLMs in scientific papers. arXiv preprint arXiv:2404.01268 (2024). [33] Daniele Lotito, Vassilis Vassiliadis, Alessandro Pomponio, Srikumar Venugopal, and Michael Johnston. 2026. LLM Fine-Tuning Benchmark Dataset. Hugging Face Datasets. Revision fcbd42f9. https: //huggingface.co/datasets/ibm-research/LLMFineTuningBench/ [34] Daniele Lotito, Srikumar Venugopal, Vassilis Vassiliadis, Christian Pinto, Alessandro Pomponio, and Michael Johnston. 2026. Quick Acquisition of Accurate Performance Models for New Configuration Space Areas. IEEE Access (2026). [35] Qinghua Lu, Liming Zhu, Xiwei Xu, Zhenchang Xing, and Jon Whittle. 2024. Toward Responsible AI in the Era of Generative AI: A Reference Architecture for Designing Foundation Model-Based Systems. IEEE Softw. (2024). [36] Fabian Mastenbroek, Georgios Andreadis, Soufiane Jounaid, Wenchen Lai, Jacob Burley, Jaro Bosch, Erwin Van Eyk, Laurens Versluis, Vincent van Beek, and Alexandru Iosup. 2021. OpenDC 2.0: Convenient Modeling and Simulation of Emerging Technologies in Cloud Datacenters. In CCGrid. [37] Jesse G Meyer, Ryan J Urbanowicz, Patrick CN Martin, Karen O’Connor, Ruowang Li, Pei-Chen Peng, Tiffani J Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Gonzalez-Hernandez, et al. 2023. ChatGPT and large language models in academia: opportunities and challenges. BioData Mining (2023). [38] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. SpotServe: Serving Generative Large Language Models on Preemptible Instances. In ASPLOS. [39] Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, et al. 2024. Granite Code Models: A Family of Open Foundation Models for Code Intelligence. CoRR abs/2405.04324 (2024). [40] Seungjae Moon, Jung-Hoon Kim, Junsoo Kim, Seongmin Hong, Junseo Cha, Minsu Kim, Sukbin Lim, Gyubin Choi, Dongjin Seo, Jongho Kim, et al. 2024. LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference. CoRR abs/2408.07326 (2024). [41] Radu Nicolae. 2026. Coastline: Exploring the Impact of Multi-Objective, Context-Aware Recommenders on Performance and Sustainability of Datacenters under LLM Fine-Tuning Workloads. MSc thesis. Vrije Universiteit Amsterdam. Work in progress toward a peer-reviewed article. Software artifact: https://doi.org/10.5281/zenodo.23300382. [42] Radu Nicolae, Alexandru Iosup, Animesh Trivedi, and Jesse Donkervliet. 2026. Kavier: Exploring Performance, Sustainability, and Efficiency of LLM Ecosystems under Inference through Cache-Aware Discrete-Event Simulation. CoRR abs/2605.25247 (2026). [43] Radu Nicolae, Daniele Lotito, and Alexandru Iosup. 2026. LLM FineTuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI. In Workshop on AI and Serverless (WoAIS ’26), December 14-18, 2026, Tarragona, Spain. ACM. doi:10.1145/3861043.3861051 [44] Radu Nicolae, Jules van der Toorn, Stavriana Kraniti, Houcen Liu, and Alexandru Iosup. 2026. OpenDT: Exploring Datacenter Performance and Sustainability with a Self-Calibrating Digital Twin. In ICPE Companion.

[45] Hao Qin and Zhi Li. 2024. A Study on Enhancing Government Efficiency and Public Trust: The Transformative Role of Artificial Intelligence and Large Language Models. IJEMR (2024). [46] Marco Ramponi. 2023. Why Language Models Became Large Language Models and the Hurdles in Developing LLM-Based Applications. https://www.assemblyai.com/blog/why-language-modelsbecame-large-language-models. Accessed: 2026-09-29. [47] Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In USENIX ATC. [48] Mehrdad Safaei and Justin Longo. 2024. The end of the policy analyst? testing the capability of artificial intelligence to generate plausible, persuasive, and useful policy analysis. DGOV (2024). [49] Joel Scheuner, Simon Eismann, Sacheendra Talluri, Erwin Van Eyk, Cristina L. Abad, Philipp Leitner, and Alexandru Iosup. 2026. Let’s trace it: Fine-grained serverless benchmarking for synchronous and asynchronous applications. FGCS (2026). [50] Arjun Singh, Nikhil Pandey, Anup Shirgaonkar, Pavan Manoj, and Vijay Aski. 2024. A Study of Optimizations for Fine-tuning Large Language Models. CoRR abs/2406.02290 (2024). [51] Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and Policy Considerations for Deep Learning in NLP. In ACL. [52] Sacheendra Talluri, Dante Niewenhuis, Xiaoyu Chu, Jakob Kyselica, Mehmet Çetin, Alexander Balgavy, and Alexandru Iosup. 2026. Cloud Uptime Archive: Open-Access Availability Data of Web, Cloud, and Gaming Services. IEEE TPDS (2026). [53] Shawn Tan, Yikang Shen, Rameswar Panda, and Aaron C. Courville. 2024. Scattered Mixture-of-Experts Implementation. CoRR abs/2403.08245 (2024). [54] Philippe Tillet, Hsiang-Tsung Kung, and David D. Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In MAPL@PLDI. [55] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR abs/2307.09288 (2023). [56] Laurens Versluis, Roland Mathá, Sacheendra Talluri, Tim Hegeman, Radu Prodan, Ewa Deelman, and Alexandru Iosup. 2020. The Workflow Trace Archive: Open-Access Data From Public and Private Computing Infrastructures. IEEE TPDS (2020). [57] Luping Wang, Lingyun Yang, Yinghao Yu, Wei Wang, Bo Li, Xianchao Sun, Jian He, and Liping Zhang. 2021. Morphling: Fast, Near-Optimal Auto-Configuration for Cloud-Native Model Serving. In SoCC. [58] Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In NSDI. [59] Qizhen Weng, Lingyun Yang, Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. 2023. Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent. In USENIX ATC. [60] Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, JanWillem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data (2016). [61] Yuchen Xia, Jiho Kim, Yuhan Chen, Haojie Ye, Souvik Kundu, Cong Callie Hao, and Nishil Talati. 2024. Understanding the Performance and Estimating the Cost of LLM Fine-Tuning. In IISWC. [62] Nezih Yigitbasi, Matthieu Gallet, Derrick Kondo, Alexandru Iosup, and Dick H. J. Epema. 2010. Analysis and modeling of time-correlated failures in large-scale distributed systems. In GRID. [63] Zuoning Yin, Xiao Ma, Jing Zheng, Yuanyuan Zhou, Lakshmi N. Bairavasundaram, and Shankar Pasupathy. 2011. An empirical study

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

on configuration errors in commercial and open source systems. In SOSP. [64] Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. 2020. An empirical study on program failures of deep learning jobs. In ICSE.

[65] Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion Control for Large-Scale RDMA Deployments. In SIGCOMM.

Related documents

• OrchSLM: Probing the Dynamics of Small Language Model Orchestration#1071256
• TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference#1076809
• Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing#1154686
• EdgePace: Deadline- and Freshness-Aware LLM Scheduling on Unified-Memory Platforms#1468911
• PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows#620367