FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration

FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration Huang Cheng Scott Zhang Aubert Li Meta, Menlo Park, California, USA huangcheng@meta.com

arXiv:2608.19659v1 [cs.LG] 20 Aug 2026

Abstract

of replicas that fit in the same fleet. Consequently, neither always use the smallest TP nor always use the largest TP is generally correct. In our measurements, the SLO-feasible throughput per GPU peaks at the interior choice TP4 for both workloads. Under heavier chat load, however, TP4 violates the completion-time objective while TP8 remains feasible. Thus the relevant target is not a context-free TP ranking; it is the final resource-constrained fleet decision. The conventional solution is to profile every candidate. This is reliable but scales with the Cartesian product of models, TP degrees, request shapes, loads, and serving policies. Generic active-learning and Bayesianoptimization methods reduce measurements, but they commonly target predictive uncertainty or the best isolated configuration. A fleet planner instead needs to know whether another measurement can change a coupled allocation. FleetSieve treats profiling as downstream decision identification. It maintains uncertainty over capacity and tail latency, solves the fleet allocation under conservative and optimistic realizations, and measures configurations that are expected to reduce disagreement between those decisions. It stops only when critical feasibility is resolved and the remaining allocation gap is below a tolerance. This paper makes three contributions: 1. Decision-critical acquisition. Profiling is driven by a measurement’s predicted effect on the downstream resource-coupled allocation rather than by grid coverage or marginal uncertainty alone. 2. Joint capacity and tail modeling. Capacity and the binding tail metric share the decision path, preventing a high-throughput but tail-infeasible configuration from being certified. 3. A conditional decision certificate. The profiler returns a three-state result—certified feasible, undecided, or certified infeasible—under an empirical uncertainty model. A replay that constructs priors from revealed measurements only produces the same stopping point and allocation. Our evaluation covers one 31B model and one H100 platform. FleetSieve reduces aggregate profiling GPUseconds modestly, provides a larger reduction for Chat, and provides no reduction for Code. We therefore interpret the method as selectively useful when unresolved measurements can still change the allocation, rather than

Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty, but measures many configurations that do not affect the final resource allocation. We present FleetSieve, which selects measurements according to their expected effect on a resource-coupled, SLO-aware fleet decision. FleetSieve models capacity and tail latency jointly, compares conservative and optimistic allocations, and stops when their remaining decision gap is below a specified tolerance. On a fixed H100 measurement grid for a 31B-parameter open-weight model, FleetSieve reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling in the fixed comparison. Across 200 random reveal orders, its mean saving over random profiling is 5.4% (95% bootstrap CI: 3.5–7.2%). The fixed-comparison saving is 21.5% for Chat, while FleetSieve does not use the fewest GPU-seconds for Code. Joint capacity and tail modeling also avoids selecting a configuration whose 46.4-second completion p99 violates a 30-second SLO. In a 16-GPU allocation, an incorrect sparse-profile decision loses up to 1.93 requests/s and 12.4 percentage points of max-min fulfillment. Boundary repeats and BurstGPT measurements support the observed load-dependent tail-latency mechanism.

1

Introduction

An LLM serving system rarely serves one homogeneous request stream. Interactive chat, code assistance, ranking, and background generation differ in arrival process, token lengths, and latency objectives. Under a finite GPU resource budget, an operator must choose a parallel configuration and a replica count for each class. These choices interact: assigning more GPUs to one replica may reduce its latency and increase its memory headroom, but it also leaves fewer GPUs for replicas serving other workloads. Tensor parallelism illustrates this tension. Increasing TP shards model state over more GPUs and can improve per-request latency and KV-cache headroom, but it increases collective communication and reduces the number 1

as a universal improvement.

where ciq is sustainable throughput, ℓiq is the relevant tail-latency metric, and siq is success probability. A configuration is feasible only when its tail and success metrics 2 Background and Related Work satisfy the class policy. The allocator chooses an integer replica count niq and LLM serving. PagedAttention in vLLM [1] and contin- served load x . In our implementation each class uses i uous batching in Orca [2] improve memory utilization and one homogeneous configuration, expressed with a binary scheduling. Sarathi-Serve [3], DistServe [4], Splitwise [5], selector y : iq and AlpaServe [6] optimize batching, phase separation, X or placement. SLOs-Serve [7] allocates serving resources yiq = 1, niq ≤ M yiq , (1) across multiple objectives, while Nitsum [8] adapts TP q at runtime for tiered requests. These systems motivate X xi ≤ niq ciq , (2) a rich configuration space; FleetSieve addresses the comq plementary question of which measurements are needed X before committing a fleet configuration. niq gq ≤ G, (3) i,q

0 ≤ xi ≤ λi ,

Configuration search and decision-focused measurement. SCOOT [9] applies constrained Bayesian optimization to LLM inference-engine tuning. Active learning more generally chooses informative samples [10]; targeted active learning explicitly optimizes information about a downstream Bayesian decision [11]. Fixedconfidence combinatorial exploration, such as CombGapE [12], identifies an optimal combinatorial action with few samples. FleetSieve borrows this decision-focused perspective but instantiates it for correlated serving measurements, multiple SLO metrics, and an integer, resourcecoupled fleet allocation. We do not claim that active learning, Bayesian optimization, elimination, or fixedconfidence stopping is independently new.

niq ∈ Z≥0 ,

yiq ∈ {0, 1},

yiq = 1 ⇒ ℓiq ≤ Li , siq ≥ smin . i

(4) (5)

The satisfaction ratio is ri = xi /λi . The objective is lexicographic: (1) enforce ri ≥ δi for critical classes whenever feasible, (2) maximize mini ri , (3) maximize total served goodput, and (4) minimize fragmentation. Profiling uses a separate compute budget measured in GPU-seconds. If all θiq were known, exhaustive profiling would produce the oracle allocation. The profiling problem is to approach that decision while revealing only a subset of the table.

4

FleetSieve

Public traces. Our primary replay uses the public Azure LLM inference traces accompanying Dy- 4.1 Joint uncertainty state namoLLM [13]; the dataset is distributed under CC BY. After round t, FleetSieve maintains capacity intervals BurstGPT [14] supplies an independent conversation trace [c , c ] and tail intervals [ℓ , ℓ ]. The intervals iq,t iq,t iq,t iq,t for a mechanism-level robustness check. combine an interior-knee structural prior with measured residuals. They are empirical rather than distribution-free; Resource-constrained selection. Resource- Section 6.4 describes the audit that bounds this claim. constrained selection also appears in other infrastructure The conservative allocation A− t uses lower capacity and domains. Cheng et al. [15], for example, formulate upper tail bounds. The optimistic allocation A+ t uses continuous roadside coverage as a knapsack-constrained upper capacity and lower tail bounds. Both use the same Steiner-tree problem. FleetSieve addresses a different GPU constraint, SLO policy, and lexicographic objective. system and uses a different algorithm, but shares the broader objective of concentrating a finite resource budget 4.2 Decision certificate on decisions that determine system-level utility.

3

The state is Certified-Infeasible if even A+ t cannot meet the critical floor. It remains Undecided if critical feasibility is unresolved or the optimistic and conservative objectives differ by more than ε. It is CertifiedFeasible when A− t satisfies the critical policy, every committed tail upper bound is known and within SLO, and the stage-wise allocation gap is at most ε. This certificate is conditional on the uncertainty set containing the relevant performance table. We therefore use “empirical” or “conditionally calibrated,” not an unconditional correctness guarantee.

Problem Formulation

Let workload class i ∈ W have offered demand λi , an SLO Li , and priority pi . Priorities define a set of critical classes C and minimum fulfillment floors δi for i ∈ C. A serving configuration q ∈ Qi specifies a TP degree and an admission/load setting. It consumes gq GPUs per replica. Its unknown performance vector is θiq = (ciq , ℓiq , siq ), 2

Algorithm 1 FleetSieve profiling loop Table 1: Peak SLO-feasible throughput (requests/s/GPU). 1: Initialize a common sparse design for every method. Workload TP2 TP4 TP8 2: while unmeasured candidates remain do 3: Fit joint capacity and tail intervals. Chat 0.798 1.649 1.408 4: A− ← conservative fleet allocation. Code 0.560 2.779 1.392 5: A+ ← optimistic fleet allocation. 6: z ← Certificate(A− , A+ , ε). 7: if z = Certified-Feasible then measured cells: three load scales, two TP degrees, and 8: return A− . three seeds. 9: else if z = Certified-Infeasible then 10: return Infeasible. Baselines and metrics. All methods start from the 11: end if same observations, use the same candidate table, SLOs, al12: Reveal the candidate with maximum decision-gap locator, and empirical stop. We compare uniform random, reduction. fixed grid order, joint maximum uncertainty, a shared13: end while feature uncertainty surrogate, targeted active learning, 14: return the best measured feasible allocation. value of information, constrained Bayesian optimization, and FleetSieve. Primary metrics are GPU-seconds to an εcorrect certified decision (ε = 0.05), decision regret versus 4.3 Acquisition the fully measured oracle, and regret AUC. We also report For each unmeasured experiment e, FleetSieve estimates the resulting 16-GPU allocation, boundary stability, and how revealing its outcome would shrink capacity/tail un- tail-certificate controls. certainty and change the conservative–optimistic allocation gap: 6 Results b t − Γt+1 | e] , scoret (e) = E[Γ where Γt is the stage-wise downstream allocation gap. 6.1 The TP optimum is interior and load The implementation uses a deterministic approximation dependent to this value-of-information objective. Expected profiling GPU-seconds are a strict tie-breaker, not a divisor; con- Table 1 and Figure 1 report peak SLO-feasible throughput sequently we describe the rule as decision-critical rather per GPU. Both workload classes peak at TP4; neither smaller nor larger TP is uniformly best. than universally compute-resource-optimal. Across five repeats at the heavier Chat C128 point, TP4 and TP8 both serve 11.27 requests/s. Their mean 5 Experimental Methodology completion-p99 values are 46.4 and 25.2 seconds, respectively, so only TP8 passes the 30-second objective. Thus System. We evaluate a 31B-parameter open-weight de- the SLO-feasible TP can change even when throughput coder model in FP8 on a single H100 node, using a vLLM- does not. based serving stack with speculative decoding disabled. Candidate TP degrees are {2, 4, 8}. Each cell runs for 300 6.2 Decision-critical profiling is modestly seconds; profiling is measured as duration multiplied by better in aggregate GPUs used, reported in GPU-seconds. Table 2 and Figure 2 report GPU-seconds to the common empirical certificate. On the fixed aggregate comparison, FleetSieve uses 22,200 GPU-seconds, 6.9% below random and 9.8–21.3% below the other listed baselines. Every deterministic run and every random trial reaches zero regret; the oracle TP is 4 for both classes. The class split is important. Against random, the fixed Chat comparison saves 21.5%; against the remaining baselines it saves 30.4–38.5%. Its Chat regret AUC is 0.1256, about half the fixed-grid value of 0.2459. For Code, random, shared-feature uncertainty, and joint uncertainty use fewer GPU-seconds than FleetSieve because TP4 becomes evident early. The table value and the randomized estimate answer different questions. Across 200 random reveal orders, the ratio-of-means advantage over random is 5.4%, with a

Workloads and SLOs. The primary workload uses the public Azure conversation and code traces [13]. Trace records provide arrival times and input/output token counts, not prompt content. The chat policy requires success ≥ 99%, TTFT-p99 ≤ 2 seconds, and completionp99 ≤ 30 seconds. The code policy requires success ≥ 99% and TTFT-p99 ≤ 5 seconds. Load labels such as C96 are ordinal indices into a documented arrival-scale ladder, not literal in-flight request counts. The fixed primary grid contains 21 standardized cells plus four earlier code-TP2 calibration cells. The four cells use a different warm-up split and remain separately labeled; excluding them does not change the code oracle because TP2 is dominated. Boundary experiments repeat eight TP4/TP8 cells over five seeds. BurstGPT adds 18 3

3 Chat Code

2 1 0

2

4 Tensor-parallel degree

8

Chat completion p99 (s)

Peak feasible req/s/GPU

(a) The feasible-throughput knee is interior

(b) Load changes the SLO-feasible choice 46.4

40 30-s SLO 25.2

20 0 TP4

TP8

Figure 1: Why profiling is necessary. (a) Peak SLO-feasible throughput per GPU is non-monotone and peaks at TP4 for both Azure workload classes. (b) At the heavier Chat C128 point, TP4 and TP8 serve the same raw request rate, but only TP8 satisfies the completion-time objective; bars show the five-seed mean. Table 2: GPU-seconds to the same zero-regret certified decision. Lower is better. One row groups three methods that are equivalent on this fixed grid: targeted active learning, value of information, and constrained Bayesian optimization. Method Aggregate Chat Code FleetSieve Uniform random Shared-feature uncertainty Joint maximum uncertainty Fixed grid order Targeted AL / value-VOI / constrained BO Full grid 95% bootstrap CI of 3.5–7.2%. For Chat it is 19.4% (16.2– 22.3%). On a single random draw, FleetSieve uses fewer GPU-seconds only 58% of the time in aggregate and 65% for Chat. The gain is reliable in expectation, not for every ordering. A 500-replicate parametric bootstrap over the repeated boundary cells never changes the oracle decision; this noise analysis covers the repeated cells and holds the remaining grid cells fixed. A heuristic early stop based on a fixed 1.5× ceiling is unsafe: fixed-grid and joint-uncertainty profiling stop with TP8 on Chat and regret 0.2407. Under the common empirical certificate, all methods continue to the correct TP4 frontier. This failure motivates evaluating the stopping rule rather than hindsight “first correct” GPU-seconds.

6.3

22,200 23,844 24,600 25,800 26,400 28,200 29,400

9,600 12,228 14,400 14,400 13,800 15,600 –

12,600 11,616 10,200 11,400 12,600 12,600 –

12.4 points. This is the resource-coupled consequence that an isolated TP-accuracy metric misses. Because every row uses the same 16 GPUs, we do not report a GPU-count saving.

6.4

Tail latency is load-bearing in the certificate

A capacity-only rule selects Chat TP4 at C128 because it serves 11.27 requests/s, but its 46.4-second completionp99 violates the 30-second objective. The joint rule selects TP4 at C96 for the frontier decision, where completionp99 is 15.98 seconds, and uses TP8 when C128 must be served. The certificate refuses to fire until the committed configuration’s tail upper bound is known. A separate fixed-order audit exercises the state machine. A non-critical trajectory certifies at step 20 (27,000 GPUseconds); a trajectory with critical floors of 6 requests/s for Chat and 10 requests/s for Code certifies at step 17 (22,800 GPU-seconds); an intentionally infeasible 1,000-requests/s demand returns Certified-Infeasible. These GPUseconds audit the certificate and are not the acquisitioncomparison GPU-seconds in Table 2. The initial offline audit used an optimistic prior derived from unobserved peak cells early in the trajectory. To test whether it affected the result, we replaced that branch with a revealed-only prior and replayed the audit. The branch is inactive after the initial six probes, and both versions

Profiling changes a constrained 16GPU allocation

We allocate 16 GPUs between Chat and Code at three demand scales (Table 3 and Figure 3). The fully measured oracle and FleetSieve choose two TP4 replicas per class. A sparse independent rule chooses one TP8 Chat replica and two TP4 Code replicas. Both consume eight GPUs per class, but the Chat capacity differs. At 0.7× demand, spare capacity masks the decision and all three serve the full load. At 1.0×, the sparse decision loses 0.73 requests/s and 6.1 percentage points of max-min fulfillment. At 1.3×, the losses grow to 1.93 requests/s and 4

8

(b) The workload split matters GPU-hours to certificate

GPU-hours to certificate

(a) Aggregate fleet decision FS: 6.9% below random

6 4 2 0 FS

ed nd xU Ra Shar Ma

id /BO V T/

Gr

ll Fu

8

Chat Code

6 4 2 0 FS

nd

Ra

d are

Sh

U

x Ma

id BO Gr V/ T/

Figure 2: Profiling GPU-seconds under the common empirical certificate. FS denotes FleetSieve; Rand is uniform random; Shared is the shared-feature uncertainty surrogate; MaxU is joint maximum uncertainty; T/V/BO groups three methods that are equivalent on this grid. FleetSieve is modestly better in aggregate, substantially better for Chat, and not better for Code. Table 3: Effect of the profiling decision on the 16-GPU allocation. MM is max-min fulfillment; goodput is served requests/s. The sparse rule chooses Chat TP8 rather than the oracle TP4. Demand

Method

TP (Chat/Code)

Replicas

MM

Goodput

0.7×

Oracle / FleetSieve Sparse rule

4/4 8/4

2/2 1/2

1.000 1.000

19.60 19.60

1.0×

Oracle / FleetSieve Sparse rule Fixed TP8 Fixed TP2

4/4 8/4 8/8 2/2

2/2 1/2 1/1 3/5

1.000 0.939 0.696 0.350

28.00 27.27 22.40 10.39

1.3×

Oracle / FleetSieve Sparse rule

4/4 8/4

2/2 1/2

0.846 0.722

33.99 32.07

certify at the same step, GPU-seconds, and allocation. We 7 Discussion and Limitations label the certificate empirical and conditionally calibrated; an online implementation should use the revealed-only The evaluation covers one 31B-parameter model, one servpath. ing stack, and one H100 platform. Model architecture, quantization, interconnect, and runtime can change the balance between collective-communication time and KV6.5 Repeated boundaries and a second cache capacity, and may move or remove the observed interior optimum. BurstGPT broadens the token-length public trace and burst distribution, but tests the tail-latency mechaAcross five seeds on eight boundary cells, seven cells retain nism rather than the complete acquisition algorithm. the same feasibility outcome. Chat TP4 at C96 is the The certificate relies on empirically calibrated residexception: it satisfies the policy in four of five runs because ual bands and an interior-knee structural assumption. A one run falls to 94.3% success. The load-bearing C128 multimodal or sharply discontinuous capacity curve could distinction is stable in every run: TP4 fails with a mean violate these assumptions. Measurement noise also matcompletion-p99 of 46.38 seconds, while TP8 passes at ters near an SLO boundary: one of eight repeated cells 25.15 seconds. changes feasibility across seeds, although the C128 cell BurstGPT reproduces the mechanism direction: TP8 that changes the TP choice is stable in every repeat. has lower completion-p99 at all three tested loads, and The primary grid includes four TP2 calibration cells the gap grows with load (Figure 4). It does not reproduce collected with a different warm-up split. They remain a uniquely feasible TP8 cell: both TPs pass at scale 250 labeled separately, and removing them does not change and both fail at 375. The trace’s absolute timescale is the oracle because TP2 is dominated on the reported metcompressed 250–500×, so this is a load-controlled robust- ric. Finally, the fleet experiment fixes the GPU resource ness check, not a timescale-faithful operational replay or allocation at 16. It improves served demand and fairness a second end-to-end acquisition win. without reducing the steady-state GPU count. 5

(b) Total served goodput

1 Requests/s

Minimum fulfillment

(a) Max-min fulfillment

0.8 0.6 0.4

30 20 10

0.2

0.7

1 Demand multiplier FleetSieve / oracle Fixed TP8

1.3

0.7

1 Demand multiplier

1.3

Sparse rule Fixed TP2

Figure 3: A profiling decision changes the resource-coupled fleet outcome. At low demand, spare capacity hides the difference. As demand rises, the sparse TP8/TP4 choice loses both fairness and served load relative to FleetSieve and the fully measured oracle. Every method uses the same 16-GPU resource allocation.

0.5

Completion p99 (s)

1 Allocation gap

(b) BurstGPT tail-latency check

certificate fires

(a) Empirical certificate trajectory

Max-min gap Goodput gap

0

0

2 4 6 Cumulative profiling GPU-hours

8

50

TP4 TP8

40

30-s SLO

30 20 250

375 500 Inter-arrival compression scale

Figure 4: Uncertainty and robustness. (a) A fixed-order audit shows that the conservative and optimistic fleet objectives converge only after the decisive tail measurement; the non-monotonic gaps reflect changes in the optimistic allocation as new cells are revealed. (b) On BurstGPT, TP8 has lower completion p99 at every tested load, but the three-point grid does not contain a TP8-only feasible point.

8

Conclusion

measurement window and are never silently mixed with standardized repeats. Run metadata include model precision, TP, trace window, arrival scale, warm-up, measurement duration, seed, success rate, TTFT, completion time, and failure state. Raw records are immutable; derived feasibility is recomputed from the class policy. The initial observations, candidate set, per-cell GPUseconds, allocator, SLO policy, and oracle table are identical across all profiling methods. Random profiling is evaluated over multiple reveal orders. Failed, rejected, and timed-out requests do not count as SLO goodput.

FleetSieve profiles an LLM serving configuration space according to the fleet allocation that the measurements inform. On the evaluated H100 grid, it reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling in the fixed comparison. The benefit is larger for Chat and absent for Code. Joint capacity and tail modeling prevents a measured 46.4-second SLO violation, while the resulting TP choice recovers up to 1.93 requests/s and 12.4 percentage points of max-min fulfillment in a 16-GPU fleet. These results show that decision-aware profiling can reduce measurement effort when uncertainty affects the allocation, while its value remains workload- and calibration-dependent.

B

Baseline Definitions

• Uniform random samples remaining cells uniformly.

A

Reproducibility Protocol

• Fixed grid traverses a predefined TP/load order.

Each standardized primary cell uses a 30-second warm-up • Joint maximum uncertainty selects the largest and a 270-second measurement window. The four earlier marginal uncertainty from the shared model without calibration cells use a 10-second warm-up and 290-second decision impact. 6

• Shared-feature uncertainty uses an RBF cross-TP surrogate and selects by uncertainty.

[12] S. Nakamura and M. Sugiyama. A fast algorithm for the real-valued combinatorial pure exploration of multi-armed bandit. arXiv:2306.09202, 2023.

• Targeted active learning prioritizes information [13] J. Stojkovic, C. Zhang, I. Goiri, J. Torrellas, and about the current decision. E. Choukse. DynamoLLM: Designing LLM inference • Value of information estimates expected decision clusters for performance and energy efficiency. In improvement with its own compute-resource normalProceedings of HPCA, 2025. ization. [14] Y. Wang et al. BurstGPT: A real-world workload dataset to optimize LLM serving systems. In Proceed• Constrained Bayesian optimization uses indepenings of KDD, 2025. dent per-arm models, expected improvement, and feasibility constraints. [15] H. Cheng, X. Fei, M. Almulla, and A. Boukerche. A knapsack constrained Steiner tree model for continuous coverage over urban VANETs. In Proceedings of References IEEE ICC, pages 130–135, 2014. [1] W. Kwon et al. Efficient memory management for large language model serving with PagedAttention. In Proceedings of SOSP, 2023. [2] G.-I. Yu et al. Orca: A distributed serving system for transformer-based generative models. In Proceedings of OSDI, 2022. [3] A. Agrawal et al. Sarathi-Serve: Efficient LLM serving with chunked prefills. In Proceedings of OSDI, 2024. [4] Y. Zhong et al. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of OSDI, 2024. [5] P. Patel et al. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of ISCA, 2024. [6] Z. Li et al. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of OSDI, 2023. [7] S. Chen, Z. Jia, S. Khan, A. Krishnamurthy, and P. B. Gibbons. SLOs-Serve: Optimized serving of multi-SLO LLMs. arXiv:2504.08784, 2025. [8] V. Srivatsa, Z. He, P. Guo, D. Li, and Y. Zhang. Nitsum: Serving tiered LLM requests with adaptive tensor parallelism. arXiv:2605.05467, 2026. [9] K. Cheng, Z. Wang, W. Hu, T. Yang, J. Li, and S. Zhang. SCOOT: SLO-oriented performance tuning for LLM inference engines. In Proceedings of The Web Conference, 2025. [10] B. Settles. Active Learning. Morgan & Claypool, 2012. [11] L. Filstroff, I. Sundin, P. Mikkola, A. Tiulpin, J. Kylmäoja, and S. Kaski. Targeted active learning for Bayesian decision-making. arXiv:2106.04193, 2021. 7

Related documents

• PerfReasoning: How Well Do LLMs Reason on Hardware Performance?#927763
• Xronos: Heterogeneity-Aware Tensor Parallelism for Collaborative LLM Fine-Tuning on Edge CPUs#1014955
• Xronos: Heterogeneity-Aware Tensor Parallelism for Collaborative LLM Fine-Tuning on Edge CPUs#978396
• Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving#440345
• Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation#175232