Agentic Autoscaling through Worker-Pool Orchestration for LLM-driven Text Classification in Cloud Computing Environments BABLU KUMAR and ANSHUL VERMA✉∗ , the Quantum Cloud Computing and Distributed Systems (qCLOUDS) Laboratory, School of Computing and Information Systems, The University of Melbourne, Australia and the Department of Computer Science, Banaras Hindu University, India
RAJKUMAR BUYYA, the Quantum Cloud Computing and Distributed Systems (qCLOUDS) Laboratory, School of
arXiv:2609.14898v1 [cs.DC] 14 Sep 2026
Computing and Information Systems, The University of Melbourne, Australia The growing adoption of large language model (LLM)-based systems for large-scale text processing has created a critical need for dynamic autoscaling to manage high-latency, bursty, and computationally intensive workloads. This paper proposes an agentic autoscaling framework through worker-pool orchestration for LLM-driven text classification. The framework integrates a priority task queue, a dynamic pool of agent workers, a real-time metrics collector, and an application-layer autoscaler. Its classifier-agnostic design supports both zero-shot and fine-tuned language models without modifying the autoscaling logic. The framework is evaluated using Autoscaling+BART and Autoscaling+DeBERTa against static allocation and standalone RoBERTa and DistilBERT baselines. On the AG News dataset, Autoscaling+BART achieves 84.5% accuracy, while Autoscaling+DeBERTa improves it to 90.5%. On the SMS Spam Collection dataset, Autoscaling+DeBERTa achieves 99.5% accuracy, whereas Autoscaling+BART attains 84.5% accuracy with lower execution time. Overall, the proposed framework consistently outperforms the baseline approaches in resource efficiency while maintaining high classification performance, demonstrating that elastic worker-pool orchestration provides an effective and cost-efficient solution for scalable LLM-driven text classification in cloud environments. CCS Concepts: • Computing methodologies → Artificial intelligence; Distributed artificial intelligence; Intelligent agents. Additional Key Words and Phrases: Large Language Models, Agentic Systems, Cloud Computing, Autoscaling, Text Classification, Resource Management, Dynamic Worker Pool ACM Reference Format: Bablu Kumar, Anshul Verma, and Rajkumar Buyya. 2026. Agentic Autoscaling through Worker-Pool Orchestration for LLM-driven Text Classification in Cloud Computing Environments. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 23 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
The proliferation of large language models (LLMs) as autonomous processing units, commonly referred to as agents, has enabled the automation of data-intensive natural language processing tasks such as text classification, summarization, Authors’ Contact Information: Bablu Kumar, [email protected]; Anshul Verma, [email protected]; [email protected], the Quantum Cloud Computing and Distributed Systems (qCLOUDS) Laboratory, School of Computing and Information Systems, The University of Melbourne, Australia and the Department of Computer Science, Banaras Hindu University, Varanasi, India; Rajkumar Buyya, [email protected], the Quantum Cloud Computing and Distributed Systems (qCLOUDS) Laboratory, School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM
Manuscript submitted to ACM
1
2
B. Kumar, A. Verma, R. Buyya
and information extraction [5, 17, 24]. An agent autonomously receives a task, invokes a language model, and returns a result without human intervention [6, 17]. As organizations increasingly adopt agentic pipelines for applications including customer support, news analysis, spam filtering, and content moderation, efficient resource provisioning has become a critical challenge [2, 22, 26]. Unlike conventional web services, LLM-driven agentic workloads exhibit high inference latency, bursty task arrivals, and significant computational cost, particularly when using GPU-backed infrastructure or commercial inference APIs [2, 4]. Consequently, traditional web-service autoscalers designed around infrastructure-level metrics are often inadequate for these workloads, motivating the need for autoscaling mechanisms specifically designed for agentic systems. A straightforward solution to this provisioning problem is static resource allocation, where a fixed number of worker processes is provisioned according to either average or peak workload. Provisioning for average demand leads to queue buildup and increased latency during traffic bursts, whereas provisioning for peak demand wastes computational resources during low-load periods [25]. Autoscaling addresses this trade-off by dynamically adjusting the worker pool orchestration based on application-level signals, such as queue depth and worker utilization, thereby matching resource allocation to workload demand without prior knowledge of future arrivals [2]. A second challenge is model selection. Modern NLP systems employ diverse language models, ranging from zero-shot models such as BART-large-MNLI and DeBERTa-v3-large to fine-tuned models such as DistilBERT [24]. These models differ in accuracy, latency, and computational cost, yet model selection is often coupled with infrastructure provisioning. In practice, however, these are independent design dimensions: the autoscaling layer determines the number of active workers, whereas the model determines task-level prediction performance [4, 8]. Therefore, an effective agentic framework should support interchangeable language models without modifying the autoscaling strategy. To address these challenges, this paper proposes an adaptive agentic autoscaling framework for LLM-driven text classification based on the cloud environment. The framework introduces a classifier-agnostic application-layer autoscaling mechanism that dynamically adjusts computational resources based on workload conditions while allowing different LLM classifiers to be integrated without modifying the scaling logic. The proposed framework employs asynchronous worker orchestration, queue-aware scheduling, and lightweight threshold-based scaling to improve resource utilization and execution efficiency without requiring external orchestration platforms. By separating infrastructure management from classification logic, the framework enables independent optimization of scalability and model quality across different LLM-driven classification workloads. The main contributions of this work are summarized as follows:
• We propose an adaptive agentic autoscaling framework for cloud-based LLM-driven text classification that explicitly worker-pool orchestration elasticity from language model selection, allowing different classifiers to be integrated without modifying the autoscaling mechanism. • We design and implement a lightweight five-component architecture, comprising a priority task queue, dynamically scalable asynchronous worker pool orchestration, centralized metrics collector, threshold-based dynamic autoscaler, and asynchronous orchestrator. • We develop two complementary algorithms that govern the proposed framework: (i) an asynchronous agent worker loop orchestration for concurrent task execution, metric collection, and graceful worker termination, and (ii) a threshold-based autoscaling control loop that dynamically provisions and deprovisions worker instances according to queue depth and worker utilization while respecting configurable scaling bounds and cooldown intervals. Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
3
• We perform a comprehensive experimental evaluation on two benchmark text-classification datasets, comparing the proposed framework with static allocation and standalone classification baselines. The results demonstrate improved resource efficiency, latency, throughput, and scalability while confirming that infrastructure elasticity and model selection are independent design dimensions. The rest of the paper is organized as follows. Section 2 reviews the related work. Section 3 presents the system architecture of the proposed framework. Section 4 introduces the system model and mathematical formulation. Section 5 presents the implementation methodology, including the system workflow and proposed algorithms. Section 6 presents the experimental setup and benchmark datasets. Section 7 discusses the experimental results. Finally, section 8 concludes the paper and outlines future research directions.
2
Related Work
Recent advances in cloud computing and LLMs have accelerated research in cloud autoscaling, transformer-based text classification, and agentic AI systems. These complementary research directions provide the foundation for scalable intelligent cloud applications. • Cloud and Container Autoscaling: Autoscaling is a key mechanism for maintaining application performance while efficiently utilizing cloud resources. Reactive approaches, such as Horizontal Pod Autoscaler (HPA) [19, 21], adjust service replicas using infrastructure-level metrics including CPU and memory utilization. However, HPA often reacts slowly to latency-sensitive AI workloads [23]. To address this limitation, predictive autoscaling techniques based on time-series forecasting and machine learning have been proposed [1, 18]. Recent studies further exploit Transformer models for workload prediction and hierarchical decentralized autoscaling [15], including Kubernetes custom-resource-based autoscaling [9], multivariate MAPE frameworks [10], and InformerAutoScale for long-term workload forecasting [11]. For LLM serving such as ARAScaler [8], LLMaaS [4] and recent distributed GPU-cluster serving frameworks [13] optimize request batching, scheduling, pipeline execution, and resource allocation to improve inference efficiency. Unlike these approaches, our framework performs application-layer autoscaling using queue depth and worker utilization while remaining independent of external orchestration platforms. • Transformer-Based Text Classification: Transformer architectures have become the dominant paradigm for text classification. Zero-shot models, such as BART-large-MNLI and DeBERTa-v3, achieve competitive performance without task-specific training [16], whereas fine-tuned models such as DistilBERT provide an effective balance between inference speed and predictive accuracy [20]. Existing studies primarily evaluate these models in standalone inference settings, emphasizing prediction accuracy rather than execution efficiency. In contrast, our work evaluates multiple classifiers within a unified autoscaling framework, separating classifier performance from infrastructure efficiency. • Agentic LLM Systems and Multi-Agent Orchestration: Recent LLM advances have enabled agentic AI systems in which autonomous agents perform reasoning, planning, memory management, and tool utilization with minimal human intervention [5, 6, 22]. Existing research has largely focused on multi-agent collaboration and reasoning, while platforms such as LLMaaS [4] and MoESys [25] improve LLM-serving efficiency through scheduling [12], batching, and mixture-of-experts routing. However, these approaches generally assume fixed computational resources and do not consider dynamic autoscaling of agent workers. Our framework instead Manuscript submitted to ACM
4
B. Kumar, A. Verma, R. Buyya
focuses on application-layer worker-pool autoscaling while supporting interchangeable language models through a classifier-agnostic execution layer. Table 1. Comparison of representative cloud autoscaling and LLM-serving approaches with the proposed framework. Approach
Year
Predictive Scaling
Application Layer
LLM Support
Classifier Agnostic
Dynamic Worker Pool
Cloud Deployment
NLP Evaluation
Kubernetes HPA [19, 21]
2025
✗
✗
✗
✓
✗
✓
✗
Predictive Autoscaling [1, 18]
2024
✓
✗
✗
✓
✗
✓
✗
Proactive Scaling with Predictive Models and Custom Resources [9]
2024
✓
✓
✗
✓
✓
✓
✗
Transformer-based MAPE Autoscaling Framework [10]
2025
✓
✓
✗
✓
✓
✓
✗
✗
✓
InformerAutoScale [11]
✗
2025
✓
✓
ARAScaler [8]
2024
✓
✓
✓
✗
✓
✓
✗
LLMaaS [4]
2024
✗
✓
✓
✗
✓
✓
MoESys [25]
2024
✗
✓
✓
✗
✗
✓
G #
✓
✗
✗
✓
✓
✗
✗
✓
G #
✓
✓
✓
✓
# G
✓
G #
✓
Standalone Transformer fiers [16, 20]
Classi- 2024-2025
Agentic LLM Systems [6, 12, 13]
2024-2026
✗
2026
✓
Proposed Framework
✓
✓
✓
✓
✗
Notation: ✓ = Fully addressed; G # = Partially addressed; × = Not addressed;
Despite significant progress in these research areas, they remain largely independent. Existing autoscaling approaches optimize infrastructure resources, LLM-serving frameworks focus on inference efficiency, and transformer-based text classification emphasizes predictive performance. Table 1 summarizes representative approaches and highlights that limited attention has been given to unified frameworks combining application-layer autoscaling, classifier-agnostic execution, and dynamic worker-pool management for LLM-driven text classification. 3
System Architecture
Figure 1 presents the internal architecture of the proposed agentic autoscaling framework. Incoming benchmark data or user requests are first received through the ingestion interface and converted into tasks submitted to the priority task queue, where tasks are ordered according to priority and sequence number. The queued tasks are then dispatched to the dynamic worker pool, which combines task execution agents with multiple asynchronous workers. The asynchronous workers, worker pool, and autoscaling controller agent collectively constitute the Worker and Autoscaling Controller component. After a task is received from the dynamic worker pool, each task execution agent coordinates its execution with an available worker, invokes the selected LLM/classifier, and tracks the task status and result. The worker acts as the actual execution unit, processing the assigned task asynchronously, invoking the selected model for inference, and returning the result to the agent. The worker pool maintains multiple asynchronous workers and dynamically expands or contracts according to workload demand. Each worker invokes the pluggable LLM/classifier interface, which supports BART-large-MNLI, RoBERTa-large-MNLI, DeBERTa-v3-large, and DistilBERT-base without modifying the worker or autoscaling logic. During execution, the framework continuously monitors application-level runtime conditions, including queue depth, worker utilization, latency, and throughput. These runtime states are collected by the dynamic worker pool and passed to the autoscaling controller, which contains an autoscaling agent responsible for observing and analyzing the current workload and worker state. The autoscaling agent evaluates queue depth 𝑞(𝑡) and worker utilization 𝑈 (𝑡) together with information maintained in the Knowledge/Experience Base, which stores historical workload states, scaling actions, and observed outcomes. This historical information provides contextual knowledge for assessing previous scaling responses and supporting the autoscaling decision process. Based on the observed state and defined scaling policy, the autoscaling agent determines whether to scale up or scale down. Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
1) Ingestion Interface
5
2) Priority Task Queue Min Heap-based Scheduler
Incoming Request
Benchmark Dataset and User/client Request
---
Configure model and System Settings
7) System Output
---
Ordered by (Priority pi, Sequence sj) Dequeue Task
Worker and Autoscaling Controller Agent 3) Dynamic Worker Pool 4) Pluggable LLM/Classifier
Workers (Execution Units) Worker 1 The actual running process/thread that hosts executes the task. Executes the assigned task Calls the LLM model for inferences Returns the result to the agent.
DeBERTa-v3-large (Zero-shot)
Worker 2
BART-large-MNLI (Fine-tuned)
Worker 3
RoBERTa-large-MNLI (Zero-shot)
---
DistilBERT-base (Fine-tuned)
.... Other Models
Worker n
Application-level runtime metrics Scale Out (+ Worker) Scale In (- Worker) 5) Autoscaling Controller Queue & Load Analyzer Analyze q(t), U(t) Detect workload phase (burst/steady/idle)
Worker Provisioning (Actuator) Spawn new workers Terminate idle workers Update pool size w(t)
6) Knowledge/Experience Base Historical States Scaling Actions Prompt-Output Pairs Used by Controller for Continuous Learning & Better Decisions
Threshold Decision Scale up if q(t) > θq or U(t) > θU+ Scale down if q(t) = 0 and U(t) < θU−
Data Flow
Scaling Policy Scaling bounds nmin ≤ n(t) ≤ nmax Step size η Cooldown Interval τ
Control / Scaling signal
Supports adaptive decision-making based on observed workload patterns
Feedback Loop
Fig. 1. Architecture of the proposed agentic autoscaling framework
It enforces the minimum and maximum worker bounds, scaling step size, and cooldown interval before issuing a scaling action. The resulting decision is communicated back to the dynamic worker pool through the autoscaling control loop. A scale-out action creates additional asynchronous workers when workload demand increases, whereas a scale-in action terminates eligible idle workers when demand decreases, without interrupting active tasks. The resulting worker state and execution outcomes are fed back to the autoscaling agent and the Knowledge/Experience Base, enabling Manuscript submitted to ACM
6
B. Kumar, A. Verma, R. Buyya
continuous interaction between workload observation, scaling decisions, and system outcomes. Thus, the task execution agents coordinate individual classification tasks, while the autoscaling agent observes workload conditions and manages worker-pool capacity, with the Knowledge/Experience Base providing historical context for the control process. Finally, the framework produces classification results and system-level performance measures, including accuracy, latency, throughput, active workers, resource usage, and resource efficiency. 4
System Model
Figure 2 illustrates the complete execution flow of the proposed agentic autoscaling framework. Starting from dataset ingestion, the pipeline performs task construction, priority scheduling, asynchronous worker execution, dynamic autoscaling, and performance evaluation. The mathematical formulations introduced in section 5 are implemented through the following eight processing stages. 𝑁 . Experiments are conducted on • Stage 0 – Dataset Input: The pipeline accepts a labeled dataset D = {(𝑥𝑖 , 𝑦𝑖 )}𝑖=1
two benchmark datasets: AG News for multi-class news classification and SMS Spam Collection for binary spam detection. • Stage 1 – Stratified Sampling: A balanced evaluation subset is generated using stratified sampling to preserve the original class distribution. The sampled instances are subsequently transformed into executable task objects while maintaining strict separation between training and testing data. • Stage 2 – Task Construction: Each sampled document is converted into a task object 𝜏𝑖 = (𝑥𝑖 , 𝑝𝑖 , 𝑒𝑖 ), where the priority value and enqueue timestamp are assigned before insertion into the scheduling queue. • Stage 3 – Priority Queue Management: Task objects are inserted into the priority queue according to the ordering defined in section 5. Incoming requests follow a burst-then-steady workload pattern, allowing the framework to evaluate autoscaling performance under realistic workload fluctuations. • Stage 4 – Worker Pool Initialization: The worker pool orchestration is initialized with a minimum number of asynchronous workers. Each worker operates independently and continuously retrieves tasks from the shared priority queue while remaining bounded by the minimum and maximum pool-size constraints. • Stage 5 – Agent Worker Execution: Each worker repeatedly dequeues a task, invokes the selected language model, records the prediction latency, and returns the classification result. Since the worker implementation is independent of the underlying classifier, BART-large-MNLI, DeBERTa-v3-large, RoBERTa-large-MNLI, DistilBERT, or any compatible model can be substituted without modifying the execution logic. Runtime exceptions are isolated at the worker level to prevent failures from propagating to other concurrent workers. • Stage 6 – Runtime Monitoring: The metrics collector continuously updates queue depth, worker utilization, latency, throughput, and task completion statistics, providing the runtime state required for autoscaling decisions. • Stage 7 – Dynamic Autoscaling: The autoscaler periodically monitors queue depth and worker utilization. Based on predefined scale-out, scale-in, and cooldown thresholds, it dynamically adjusts the worker pool while ensuring that active tasks are not interrupted. This feedback loop enables the framework to respond efficiently to workload orchestration fluctuations. • Stage 8 – Performance Evaluation: After all tasks have been processed, the framework computes classification accuracy, per-category accuracy, latency, throughput, and resource-efficiency metrics. These processing stages collectively implement the proposed agentic autoscaling framework and provide the basis for the experimental evaluation presented in section 7. Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
7
Stage 0: Raw Input
Stage 5: Agent Worker Loop (Algorithm 1) - runs concurrently for every wₖ ∈ W(t) 5a. TryDequeue(Q, timeout=1s)
Dataset D={ (x1,y1), (x2,y2), …, (xn,yn) } xi ∈ X=text document yi ∈ Y=label set
bounded-timeout dequeue prevents indefinite block allows stop-signal check between polls
Dataset A: Y = {ham, spam} , |Y| = 2 · Dataset B, Y = {World, Sports, Business, Sci/Tech} , |Y|=4 Stage 1: Stratified Sampling
task τᵢ
Stratified sample S ⊂ D, N_c = N / |Y| per class c ∈ Y (ensures balanced class representation) Dataset A: N=200, N_c=100 · Dataset B: N=200, N_c=50 DistilBERT train split: 4,000 disjoint rows, no overlap with S
No
returned?
5b. Stop
5c. σₖ ← busy, t_start ← Now() worker state transitions from idle to busy · start latency timer
For each (xᵢ, yᵢ) ∈ S, construct task object τᵢ, τᵢ = ( xᵢ , pᵢ , eᵢ )
xᵢ = text payload · pᵢ ∈ Z⁺ = priority (lower = higher priority) · eᵢ = enqueue timestamp priority assignment: pᵢ = 1 (spam/urgent) or 3 (ham/normal) or by category rank
5d. Invoke classifier: (ŷᵢ, δᵢ) ← f(xᵢ) f : X → Y is the pluggable classifier callable (BART / DeBERTa / RoBERTa / DistilBERT) ŷᵢ ∈ Y = predicted label · δᵢ = Now() - t_start = service time for this invocation
Stage 3: Priority Queue Q Enqueue τᵢ as 3-tuple: (pᵢ, sᵢ, τᵢ)
sᵢ = next(counter) - monotonically increasing number Ordering: τᵢ ≺ τⱼ ⟺ (pᵢ, sᵢ) <_lex (pⱼ, sⱼ) Queue depth at time t: q(t) = |{ τᵢ ∈ Q : enqueued, not yet dequeued }| Burst phase λ_burst (t ∈ [0, 0.35T]) Steady phase λ_steady (t ∈ (0.35T, T])
5e. ŷᵢ ← error log
exception raised?
Yes
continue to be isolated. does not affect wₖ
No
Stage 4: Worker Pool W(t) Initialisation
W(t) updated → continue processing Q
Yes
Stage 2: Task Object Construction
5f. Construct result rᵢ = (τᵢ, ŷᵢ, δᵢ, k, success)
W(t) = { w₁, w₂, …, w_n(t) } · n(t) = |W(t)|
enqueue rᵢ → result queue Q_out σₖ ← idle · return to 5a
Initialise with n(0) = n_initial workers (2 in all experiments) Bounds enforced at all times: n_min ≤ n(t) ≤ n_max n_min ∈ {1,2} · n_max ∈ {6,8} each worker wₖ is an asyncio coroutine
Stage 6: Metrics Collector M(t) — asyncio Lock protected, sliding window ω=200 M(t) = ( q(t), n(t), U(t), L₉₅(t), c(t), φ(t) )
Final metrics: Reported for every (dataset × approach × scaling-policy) combination Overall accuracy
Acc = (1/N) Σᵢ 𝟙[ŷᵢ = yᵢ] Per-class accuracy
worker utilization—U(t) = |{ wₖ ∈ W(t) : σₖ = busy }| / n(t) ∈ [0, 1] P95 latency - L₉₅(t) = 95th-percentile of last ω=200 latencies { δᵢ } c(t) = cumulative completed tasks · φ(t) = cumulative failed tasks Stage 7: Autoscaler A — control rule R evaluated every Δt seconds at polling instants t_j = j·Δt 7a. Cooldown gate: check (t_j − t_last) ≥ τ
τ = 4–10s prevents oscillation · if cooldown active → skip to next poll (no scale event)
Acc_c = Σ_{i:yᵢ=c} 𝟙[ŷᵢ=yᵢ] / |{i:yᵢ=c}| Avg latency L̄ = (1/N) Σ δᵢ
q(t_j) > θq
OR U(t_j) > θU⁺ θq=3–5, θU⁺=0.80
P95 latency L₉₅ = 95th pct δᵢ
No
Total runtime T_runtime (wall-clock)
q(t_j) = 0 AND U(t_j) < θU⁻ θU⁻ = 0.20
Yes
Yes
7b. SCALE UP Δn = min(η, n_max−n)
7c. SCALE DOWN Δn = min(η, n−n_min)
Efficiency E (Acc × 100 / (T/60)) No Stage 8: Output Metrics (computed after all N tasks complete)
7d. Apply pool-size update with clipping
n(t_j+1) = clip( n(t_j) + Δn, n_min, n_max ) · update t_last if Δn ≠ 0
Fig. 2. End-to-end workflow of the proposed agentic autoscaling pipeline with dynamic worker scaling and performance evaluation
Manuscript submitted to ACM
8 5
B. Kumar, A. Verma, R. Buyya METHODOLOGY
This section presents the implementation methodology of the proposed agentic autoscaling framework, including the system workflow, asynchronous agent worker algorithm, and threshold-based autoscaling algorithm.
5.0.1 Task Stream and Arrival Process: Let D = {𝑥 1, 𝑥 2, . . . , 𝑥 𝑁 } denote a dataset of 𝑁 input documents to be classified, where each 𝑥𝑖 is paired with a ground-truth label 𝑦𝑖 ∈ Y drawn from a label set Y (for AG News, Y = {World, Sports, Business, Sci/Tech}, |Y| = 4; for the SMS Spam Collection, Y = {ham, spam}, |Y| = 2). Each document is wrapped in a task object 𝜏𝑖 = (𝑥𝑖 , 𝑝𝑖 , 𝑒𝑖 ) where 𝑝𝑖
∈ Z+ is a priority value (lower values indicate higher priority; in our implementation, priority is assigned by
category to allow time-sensitive classes to be processed first) and 𝑒𝑖 is the enqueue timestamp. Tasks arrive into the system according to a time-varying arrival process 𝜆(𝑡), the instantaneous arrival rate at time 𝑡. We model arrivals as occurring in two phases to reflect realistic bursty traffic: 𝜆(𝑡) =
𝜆burst, 𝑡 ∈ [0, 𝛽𝑇 ] 𝜆steady, 𝑡 ∈ (𝛽𝑇 ,𝑇 ]
𝜆burst ≫ 𝜆steady
where 𝑇 is the total observation window and 𝛽 ∈ (0, 1) is the fraction of the window during which burst-phase arrivals occur (in our experiments, 𝛽 ≈ 0.35–0.40, with 𝜆burst approximately 4–7× larger than 𝜆steady , constructed to stress-test the autoscaler’s response to a sudden load spike followed by a return to baseline demand).
5.0.2 Priority Task Queue: Tasks are buffered in a priority queue 𝑄, implemented as a binary min-heap ordered first by priority 𝑝𝑖 and, to break ties deterministically and avoid undefined comparison behaviour between non-orderable payloads, by a monotonically increasing sequence number 𝑠𝑖 assigned at enqueue time: 𝜏𝑖 ≺ 𝜏 𝑗 ⇐⇒ (𝑝𝑖 , 𝑠𝑖 ) <lex (𝑝 𝑗 , 𝑠 𝑗 ) We define the instantaneous queue depth as 𝑞(𝑡) = |{𝜏𝑖 ∈ 𝑄 : 𝜏𝑖 enqueued, not yet dequeued at time 𝑡 }| The queue depth 𝑞(𝑡) reflects workload intensity and is used by the autoscaler together with worker utilization.
5.0.3 Agent worker: An agent worker 𝑤𝑘 is a long-lived asynchronous process that repeatedly executes the loop: dequeue a task 𝜏𝑖 from 𝑄; invoke a classification function 𝑓 : X → Y on the task payload 𝑥𝑖 ; record the result and elapsed time; enqueue the result. We treat 𝑓 as an opaque, pluggable callable - the agentic abstraction at the center of this paper’s contribution – satisfying only the type signature 𝑓 (𝑥𝑖 ) ↦→ (𝑦ˆ𝑖 , 𝛿𝑖 ) where 𝑦ˆ𝑖 ∈ Y is the predicted label and 𝛿𝑖 ∈ R+ is the wall-clock service time for that invocation. In our experiments, 𝑓 is instantiated as one of: a zero-shot natural-language-inference classifier (BART-large-MNLI, DeBERTa-v3-large, RoBERTa-large-MNLI), or a supervised fine-tuned classifier (DistilBERT). Crucially, no component of the queue, worker pool, metrics collector, or autoscaler inspects or depends on which 𝑓 is in use; this is the formal statement of the Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
9
classifier-agnostic property claimed in Contribution. At any time 𝑡, each worker is in one of two states, busy or idle: 𝜎𝑘 (𝑡) ∈ {busy, idle} A worker is busy for the duration of one 𝑓 invocation, 𝛿𝑖 seconds, and idle while waiting on the (possibly empty) queue. 5.0.4
Worker pool: The worker pool at time 𝑡 is the set of currently active workers, 𝑊 (𝑡) = {𝑤 1, 𝑤 2, . . . , 𝑤𝑛 (𝑡 ) },
𝑛(𝑡) = |𝑊 (𝑡)|
We refer to 𝑛(𝑡) as the pool size. The pool is bounded: 𝑛 min ≤ 𝑛(𝑡) ≤ 𝑛 max
∀𝑡
where 𝑛 min, 𝑛 max ∈ Z+ are operator-configured lower and upper bounds (in our experiments, 𝑛 min ∈ {1, 2} and 𝑛 max ∈ {4, 6, 8} depending on the run). Pool size changes only at discrete control instants determined by the autoscaler (section 5); between control instants, 𝑛(𝑡) is constant. We define worker utilization at time 𝑡 as 𝑈 (𝑡) =
|{𝑤𝑘 ∈ 𝑊 (𝑡) : 𝜎𝑘 (𝑡) = busy}| ∈ [0, 1] 𝑛(𝑡)
The worker utilization 𝑈 (𝑡) reflects processing capacity and, together with queue depth, guides autoscaling decisions. 5.0.5 Autoscaling Control Model: We formalize the autoscaler as a control loop that samples the system state at a fixed polling interval Δ𝑡 and applies a control function 𝑅 to determine pool-size adjustments. Let 𝑡 𝑗 = 𝑗 · Δ𝑡 for 𝑗 = 0, 1, 2, . . . denote the sequence of polling instants. At each 𝑡 𝑗 , the autoscaler observes the state pair s 𝑗 = (𝑞(𝑡 𝑗 ), 𝑈 (𝑡 𝑗 )) and computes a pool-size delta Δ𝑛 𝑗 ∈ Z according to the rule + min(𝜂, 𝑛 max − 𝑛(𝑡 𝑗 )), Δ𝑛 𝑗 = − min(𝜂, 𝑛(𝑡 𝑗 ) − 𝑛 min ), 0, The auxiliary conditions are defined as
if 𝐶 up, if 𝐶 down, otherwise.
𝐶 up = 𝑞(𝑡 𝑗 ) > 𝜃𝑞 ∨ 𝑈 (𝑡 𝑗 ) > 𝜃𝑈+ ∧ (𝑡 𝑗 − 𝑡 last ) ≥ 𝜏, 𝐶 down = 𝑞(𝑡 𝑗 ) = 0 ∧ 𝑈 (𝑡 𝑗 ) < 𝜃𝑈− ∧ (𝑡 𝑗 − 𝑡 last ) ≥ 𝜏 . Here 𝜃𝑞 ∈ Z+ is the queue-depth scale-up threshold, 𝜃𝑈+ ∈ (0, 1) and 𝜃𝑈− ∈ (0, 1) are the scale-up and scale-down utilization thresholds respectively (with 𝜃𝑈− < 𝜃𝑈+ ), 𝜂 ∈ Z+ is the maximum step size per control instant, 𝜏 ∈ R+ is a cooldown period, and 𝑡 last is the timestamp of the most recent non-zero Δ𝑛 scaling action. The pool size then evolves as 𝑛(𝑡 𝑗+1 ) = clip 𝑛(𝑡 𝑗 ) + Δ𝑛 𝑗 , 𝑛 min, 𝑛 max and 𝑡 last is updated to 𝑡 𝑗 whenever Δ𝑛 𝑗 ≠ 0. The cooldown term (𝑡 𝑗 −𝑡 last ) ≥ 𝜏 is the mechanism that prevents oscillatory scaling – without it, a pool that scales up in response to a transient queue spike could immediately scale back down at the very next polling instant once that single burst is absorbed, only to scale up again moments later, a pattern that wastes the overhead of worker spin-up/spin-down without net benefit. In our experiments, Δ𝑡 ∈ [2, 5] seconds and 𝜏 ∈ [4, 10] seconds, with 𝜃𝑞 ∈ [3, 5], 𝜃𝑈+ = 0.80, 𝜃𝑈− = 0.20, and 𝜂 = 2 as the default scale-step. Note that 𝑅 Manuscript submitted to ACM
10
B. Kumar, A. Verma, R. Buyya
depends only on (𝑞(𝑡 𝑗 ), 𝑈 (𝑡 𝑗 )) – application-layer signals intrinsic to the task-processing semantics of the pipeline – and not on any infrastructure-layer signal such as host CPU or memory utilization. This is the formal distinction from container-level autoscalers (Section 3): 𝑅 is well-defined and meaningful even when the underlying workers are bound by GPU computation or network-bound API calls whose host-level CPU footprint may be misleadingly low. 5.0.6 Metrics collector: The metrics collector 𝑀 maintains a bounded sliding window of the most recent 𝜔 completed task latencies (we use 𝜔 = 200), L (𝑡) = {𝛿𝑖 : 𝜏𝑖 completed in (𝑡 − 𝜔 ′, 𝑡]} and exposes, at any query time 𝑡, a snapshot 𝑀 (𝑡) = 𝑞(𝑡), 𝑛(𝑡), 𝑈 (𝑡), 𝐿95 (𝑡), 𝑐 (𝑡), 𝜙 (𝑡)
where 𝐿95 (𝑡) is the 95th-percentile latency over L (𝑡), 𝑐 (𝑡) is the cumulative count of successfully completed tasks, and 𝜙 (𝑡) is the cumulative count of failed tasks (task failures are isolated at the worker level: an exception raised during one 𝑓 (𝑥𝑖 ) invocation is caught, logged, and reported as a failed result without terminating the worker or affecting any other in-flight task, which is the basis of the framework’s fault-isolation property). 𝑀 (𝑡) is the single source of truth consumed by the autoscaler at every polling instant and is also the basis for all accuracy, latency, and throughput statistics reported in section 7. 5.0.7 Performance Metrics: For a completed run over dataset D, let 𝑦ˆ𝑖 denote the predicted label for input 𝑥𝑖 as returned by 𝑓 . We report overall accuracy 𝑁
Acc =
1 ∑︁ ⊮[𝑦ˆ𝑖 = 𝑦𝑖 ] 𝑁 𝑖=1
and per-category accuracy for each 𝑐 ∈ Y, Í Acc𝑐 =
𝑖:𝑦𝑖 =𝑐 ⊮[𝑦ˆ𝑖 = 𝑦𝑖 ]
|{𝑖 : 𝑦𝑖 = 𝑐}|
For the binary spam-detection task we additionally report the recall on the minority/positive class explicitly, since, as shown in section 7, overall accuracy alone can mask a near-complete failure on one class when the dataset is classbalanced but the model’s predictions are not: recall on class 𝑐 is Acc𝑐 restricted to 𝑐 = spam. To compare approaches that trade off accuracy against runtime, we define an accuracy-per-time efficiency score 𝐸=
Acc × 100 𝑇runtime /60
measured in accuracy-percentage-points per minute of total runtime. 𝐸 penalizes both low accuracy and long runtime in the same scalar quantity, allowing a fair comparison between, for example, a fast-but-less-accurate zero-shot pipeline and a slow-but-more-accurate fine-tuned standalone model. 5.1
Agent Worker Algorithm
Algorithm 1 describes the execution logic of an asynchronous agent worker within the proposed framework. Each worker repeatedly retrieves a task from the shared priority queue, performs text classification using the selected language model, records execution metrics, and returns the prediction result. The algorithm incorporates timeout control and exception handling to ensure robust execution and fault isolation, allowing multiple workers to operate concurrently without affecting one another. This asynchronous and classifier-agnostic design enables efficient parallel Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
11
task processing while supporting dynamic worker-pool management under varying workloads. Algorithm 1 executes independently for every active worker in the pool. Initially, Lines 1-2 initialize the worker state and repeatedly execute the worker loop until a stop signal is received. Lines 3-6 retrieve the highest-priority task from the shared queue using a bounded timeout; if no task is available, the worker continues waiting without busy-waiting. Lines 7-14 change the worker status to busy, record the start time, invoke the selected language model, and handle any execution exceptions through a try-catch mechanism to ensure fault isolation. Lines 15-18 compute the task execution latency, package the prediction result together with execution metadata, and insert the completed task into the output queue. Finally, Lines 19-20 update the runtime metrics and return the worker to the idle state, allowing it to process the next task. Since the worker implementation is classifier-agnostic, BART-large-MNLI, DeBERTa-v3-large, RoBERTa-large-MNLI, DistilBERT, or any compatible language model can be integrated without modifying the execution logic. Consequently, infrastructure elasticity and model selection remain independent, enabling different language models to be evaluated under the same autoscaling framework. Algorithm 1 Agent worker loop Require: Task queue 𝑄 , result queue 𝑄 out , metrics collector 𝑀 , classification function 𝑓 , stop signal Stop 1: 𝜎𝑘 ← idle 2: while ¬Stop do 3: 𝜏𝑖 ← TryDeqeue (𝑄, timeout = 1s ) 4: if 𝜏𝑖 = None then 5: continue {re-check stop signal, avoid busy-wait} 6: end if 7: 𝜎𝑘 ← busy 8: 𝑡 start ← Now ( ) 9: Try 10: ( 𝑦ˆ𝑖 , _ ) ← 𝑓 (𝑥𝑖 ) 11: success ← True 12: catch (exception 𝜖 ) 13: 𝑦ˆ𝑖 ← Error 14: success ← False 15: Log (𝑘, 𝜏𝑖 , 𝜖 ) 16: end try 17: 𝛿𝑖 ← Now ( ) − 𝑡 start 18: 𝑟𝑖 ← (𝜏𝑖 , 𝑦ˆ𝑖 , 𝛿𝑖 , 𝑘, success ) 19: Enqeue (𝑄 out , 𝑟𝑖 ) 20: 𝑀 .Record (𝑟𝑖 ) 21: 𝜎𝑘 ← idle 22: end while
5.2
Autoscaling Algorithm
Algorithm 2 presents the threshold-based control strategy used to dynamically manage the agent worker pool orchestration. The autoscaler periodically monitors queue depth and worker utilization to estimate the current workload and processing capacity. Based on predefined upper and lower thresholds, it performs scale-up or scale-down operations while enforcing cooldown intervals to prevent frequent oscillations. Since scaling decisions rely exclusively on application-level metrics rather than infrastructure utilization, the proposed approach provides responsive and efficient resource management for LLM-driven text classification workloads. Algorithm 2 executes continuously throughout system operation. Line 1 initializes the autoscaler parameters, including the monitoring interval, cooldown period, scaling thresholds, scaling step size, and worker-pool bounds. Lines 2-5 periodically collect a snapshot of the current system state and skip the scaling cycle if the cooldown interval has not yet expired. Lines 6-10 evaluate the scale-up condition by checking whether the queue depth or worker utilization exceeds the predefined thresholds; if satisfied, Manuscript submitted to ACM
12
B. Kumar, A. Verma, R. Buyya
additional workers are provisioned while respecting the maximum pool size. Lines 11-15 evaluate the scale-down condition when the queue becomes empty and worker utilization falls below the lower threshold, releasing idle workers without violating the minimum pool size. The control loop then repeats, continuously adapting the worker pool orchestration to workload variations while maintaining stable system operation. Algorithm 2 Autoscaler control loop + , 𝜃 − , step 𝜂 , bounds 𝑛 Require: Metrics collector 𝑀 , worker pool 𝑊 , poll interval Δ𝑡 , cooldown 𝜏 , thresholds 𝜃𝑞 , 𝜃𝑈 min , 𝑛 max 𝑈 1: 𝑡 last ← −∞ 2: while ¬Stop do 3: Sleep (Δ𝑡 ) 4: (𝑞, 𝑛, 𝑈 , 𝐿95 , . . .) ← 𝑀 .Snapshot ( ) 5: if Now ( ) − 𝑡 last < 𝜏 then 6: continue {cooldown active, skip this cycle} 7: end if + then 8: if 𝑞 > 𝜃𝑞 ∨ 𝑈 > 𝜃𝑈 9: Δ𝑛 ← min(𝜂, 𝑛 max − 𝑛) 10: if Δ𝑛 > 0 then 11: 𝑊 .ScaleUp (Δ𝑛) 12: 𝑡 last ← Now ( ) 13: end if 14: else if 𝑞 = 0 ∧ 𝑈 < 𝜃𝑈− then 15: Δ𝑛 ← min(𝜂, 𝑛 − 𝑛 min ) 16: if Δ𝑛 > 0 then 17: 𝑊 .ScaleDown (Δ𝑛) 18: 𝑡 last ← Now ( ) 19: end if 20: end if 21: end while
The proposed methodology establishes a complete framework for LLM-driven agentic text classification by integrating queue-aware task scheduling, asynchronous worker execution, and application-layer autoscaling within a unified cloud architecture. The following section evaluates the proposed framework through extensive experiments on benchmark datasets, comparing classification performance, execution efficiency, scalability, and resource utilization against representative baseline approaches. 6
Experimental Setup
This section describes the computational environment, software configuration, model integration, dataset preparation, framework initialization, autoscaling configuration, execution procedure, and output generation used to reproduce the experiments. The complete experimental configuration is illustrated in Figure 3. The procedure is organized as a sequence of practical steps so that the proposed agentic autoscaling framework can be reproduced under the same experimental conditions. (1) Compute Environment and Software Setup: All experiments were conducted using Google Colaboratory as the execution environment. A new Colab notebook was initialized using Python 3.12. The framework uses Python’s standard-library asyncio module for asynchronous task execution and does not require Kubernetes, an external task-queue service, or a dedicated cloud orchestration platform. Two runtime configurations were used according to the computational requirements of the evaluated classifiers. BART-large-MNLI and RoBERTa-largeMNLI were executed using the CPU runtime, whereas DeBERTa-v3-large inference and DistilBERT fine-tuning were performed using an NVIDIA T4 GPU runtime. The Colab environment provides approximately 12.7 GB of system memory and session-local temporary storage, while the T4 configuration provides approximately 15 GB of GPU memory. Since Colab resources are session-based and hardware availability may vary, the experiments were Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
13
Layer 1: Compute Environment Python 3.12, asyncio standard library, No external orchestration service—Free cost
CPU-only runtime — zero-shot BART / RoBERTa runs NVIDIA T4 GPU runtime—DeBERTa / DistilBERT fine-tuning Layer 2: Datasets Dataset A — SMS Spam Collection
Dataset B — AG News
full corpus: 5,574 SMS messages
full corpus: 127,600 articles (train + test)
labels: Y = {ham, spam} · |Y| = 2
Labels: World / Sports / Business / Sci/Tech
avg length: 10–15 words per message
Avg length: 40–50 words per article
separability: HIGH, distinctive spam lexicon (FREE, WIN, urgent)
Separability: MODERATE, Business / Sci/Tech vocabulary overlap Layer 3: stratified sampling procedure
Balanced evaluation set S · N = 200 per dataset
N_c = N / |Y| examples per class (equal representation, controls for imbalance) Dataset A: 100 ham + 100 spam = 200 drawn from held-out eval slice
Dataset B: 50 × 4 categories = 200 drawn from official test split
DistilBERT train split: 4,000 disjoint rows, No overlap with S, Deduplicated Layer 4 — autoscaler configuration Autoscaler — fixed parameters
Δt = 2–5 s poll interval
τ = 4–10 s cooldown period
η = 2 max scale step
θ_q = 3–5 queue threshold
pool bounds: n_min ∈ {1,2} · n(0) = 2 · n_max ∈ {6,8} · metrics window ω = 200
θ_U⁺ = 0.80 scale-up util θ_U⁻ = 0.20 scale-down util
Layer 5 — four pluggable classifier configurations f : X → Y Classifier f — swapped without changing any pipeline code Pipeline runs (with autoscaler)
BART-large-MNLI zero-shot 407M params
DeBERTa-v3-large zero-shot 435M params
Standalone runs (no autoscaler, n≡1)
RoBERTa-large-MNLI zero-shot ⚠ spam recall
DistilBERT fine-tuned 67M params
candidate labels: descriptive phrases (e.g. "politics and world news") mapped back to Y after prediction Layer 6 — procedure (per run) Sample S →build tasks τᵢ →submit via λ(t) → collect results rᵢ →compute Acc, Acc_c, L₉₅, T, E
all 5 approach × 2 dataset combinations evaluated on the same stratified sample S
Fig. 3. Experimental setup of the proposed agentic autoscaling framework
performed within a single active runtime for each execution. Reported wall-clock latency therefore corresponds to the native execution environment and is not normalized across CPU and GPU configurations. The required Python libraries were installed before execution using the standard package manager. The main dependencies were transformers, torch, datasets, pandas, and accelerate. The framework itself uses asyncio for worker concurrency and asynchronous orchestration. In Colab, the framework is executed using await because the notebook environment already maintains an active event loop. Manuscript submitted to ACM
14
B. Kumar, A. Verma, R. Buyya (2) Language Model Access and Configuration: The evaluated transformer models were loaded through the Hugging Face transformers library. The models used in the experiments were BART-large-MNLI (facebook/ bart-large-mnli), DeBERTa-v3-large (MoritzLaurer/deberta-v3-large-zeroshot-v2.0), RoBERTa-largeMNLI (roberta-large-mnli), and DistilBERT-base-uncased (distilbert-base-uncased). BART-large-MNLI, DeBERTa-v3-large, and RoBERTa-large-MNLI were used for zero-shot classification, whereas DistilBERT-baseuncased was fine-tuned for supervised classification. The publicly available models can be downloaded directly by the transformers library during the first execution and cached locally within the Colab session. An optional Hugging Face access token can be configured when authenticated access is required; the token is not required for the publicly available models used in this study. For zero-shot classification, the Hugging Face pipeline interface was used with the zero-shot-classification task. CPU execution was specified using device=-1, whereas GPU execution used the corresponding GPU device. Candidate labels were represented using descriptive natural-language phrases rather than only class names. For AG News, the candidate phrases were "politics and world news," "sports and athletics," "business and finance," and "science and technology." For SMS Spam Collection, the candidate phrases were "spam message" and "legitimate message." The candidate phrase receiving the highest NLI entailment score was selected as the prediction and mapped to the corresponding dataset label. (3) Dataset Access and Preparation: Two benchmark datasets were used. The SMS Spam Collection1 [3, 14] dataset was loaded as a tab-separated dataset containing the fields label and text. The dataset contains 5,574 messages, including 4,827 ham and 747 spam messages. AG News2 [7, 27] dataset was loaded through the Hugging Face datasets library using its test split. The test split contains 7,600 articles distributed across four categories: World, Sports, Business, and Sci/Tech. To ensure a fair comparison across classifiers and scaling configurations, a balanced stratified evaluation subset of 𝑁 = 200 samples was generated independently for each dataset. The number of evaluation samples per class was calculated as 𝑁𝑐 =
𝑁 , |Y|
resulting in 100 samples per class for SMS Spam Collection and 50 samples per class for AG News. The same 200 evaluation samples were used for every experimental configuration. For the supervised DistilBERT experiments, an additional disjoint training subset of 4,000 samples was selected from the remaining dataset samples. The training subset was kept separate from the evaluation subset to prevent train–test leakage. (4) Framework and Worker Initialization: After dataset preparation, each evaluation sample was converted into a task object containing the input text, priority information, and enqueue metadata. The proposed framework was initialized through the asynchronous orchestration mechanism with an initial worker count of 𝑛(0) = 2. The worker pool was constrained by predefined minimum and maximum worker limits. Each worker was implemented as an independent asynchronous execution unit and launched through the framework’s asynchronous event loop. Workers repeatedly wait for tasks from the shared priority queue, process the assigned task, invoke the selected classification function, and return the prediction and execution statistics. The task execution logic is classifier-agnostic. The classification function is supplied to the worker through a common interface, allowing BART-large-MNLI, DeBERTa-v3-large, RoBERTa-large-MNLI, or DistilBERT to be substituted without modifying 1 SMS Spam Collection dataset is available at https://github.com/justmarkham/pycon-2016-tutorial/blob/master/data/sms.tsv 2 AG News dataset is available at https://github.com/mhjabreel/CharCnn_Keras
Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
15
the worker implementation or autoscaling policy. Each worker processes tasks independently, and exceptions generated during an individual inference operation are isolated at the worker level so that a failed task does not terminate other concurrent workers. (5) Agent and Model Execution: The task execution agent coordinates the processing of each queued task with an available asynchronous worker. After a task is retrieved from the priority queue, the agent assigns the task for execution and invokes the selected classifier through the common model interface. The worker records the task start time, executes the classifier, captures the prediction and execution status, and returns the result to the agent. For zero-shot models, model inference is executed through the Hugging Face classification pipeline. Because transformer inference can block the asynchronous event loop, the inference operation is dispatched through an executor so that other workers can continue processing tasks concurrently. For DistilBERT, supervised fine-tuning was performed using the Hugging Face Trainer API with mixed-precision training (fp16=True) on the NVIDIA T4 GPU. A batch size of 16 was used, with one training epoch for AG News and two training epochs for SMS Spam Collection. Training and evaluation samples were kept disjoint. After training, the resulting classifier was integrated into the same worker execution interface used by the other models. (6) Workload Generation and Task Scheduling: The 200 evaluation tasks for each dataset were submitted according to the two-phase workload model consisting of a burst-arrival phase followed by a steady-state phase. During the burst phase, tasks were introduced at a rate exceeding the initial processing capacity, producing queue pressure and triggering the autoscaling mechanism. During the subsequent steady-state phase, the arrival rate was reduced, allowing the queue to drain and enabling the autoscaler to release unnecessary workers. All tasks were inserted into the priority queue and ordered using the priority value and monotonically increasing sequence number. Workers continuously retrieved available tasks from the queue and processed them asynchronously. The same task sequence and evaluation samples were maintained across the compared pipeline configurations to ensure consistent experimental conditions. (7) Autoscaling Configuration: The autoscaler was configured using fixed parameters throughout the pipelinebased experiments. The queue-depth scale-out threshold was set within 𝜃𝑞 ∈ [3, 5], while the worker-utilization thresholds were set to 𝜃𝑈+ = 0.80 for scale-out and 𝜃𝑈− = 0.20 for scale-in. The maximum scaling step was set to 𝜂 = 2 workers per scaling event. The autoscaler polling interval was configured within Δ𝑡 ∈ [2, 5] seconds, and the cooldown period was set within 𝜏 ∈ [4, 10] seconds. The initial worker pool contained two workers, with 𝑛 min ∈ [1, 2] and 𝑛 max ∈ [6, 8] depending on the evaluated classifier. At each polling interval, the autoscaling agent obtains the current application state from the runtime metrics and evaluates the queue depth 𝑞(𝑡) and worker utilization 𝑈 (𝑡). A scale-out decision is generated when 𝑞(𝑡) > 𝜃𝑞
∨
𝑈 (𝑡) > 𝜃𝑈+ ,
𝑞(𝑡) = 0
∧
𝑈 (𝑡) < 𝜃𝑈− .
whereas scale-in is considered when
The scaling step is bounded by 𝜂, and the worker count is maintained within [𝑛 min, 𝑛 max ]. A cooldown interval is enforced after each scaling event to prevent repeated or oscillatory scaling actions. (8) Runtime Monitoring and Autoscaling Verification: During execution, the framework continuously records queue depth, active worker count, worker utilization, latency statistics, completed tasks, failed tasks, and throughput. The autoscaling controller periodically obtains a snapshot of these application-level metrics and Manuscript submitted to ACM
16
B. Kumar, A. Verma, R. Buyya
evaluates the current workload condition. The runtime output includes timestamped status information that allows the scaling behavior to be directly verified. For example, a runtime state may report the current queue depth, number of workers, active workers, utilization, and latency percentile. Scale events are explicitly recorded when workers are added or removed. During the burst-arrival phase, an increase in queue depth or worker utilization should result in scale-out events, whereas after the queue has drained and utilization decreases, eligible idle workers are removed through scale-in events. These runtime logs provide direct evidence that the dynamic worker-pool mechanism is active during the experiment. (9) Knowledge/Experience Interaction: The Knowledge/Experience Base maintains historical workload states, scaling actions, and observed outcomes associated with previous execution states. The autoscaling agent can access this information as contextual knowledge when evaluating workload behavior and previous scaling responses. The current experimental scaling action remains governed by the predefined threshold-based policy, while the stored experience provides a record of the relationship between observed workload states, scaling actions, and resulting system behavior. The worker state and execution outcomes are subsequently fed back to the Knowledge/Experience Base, maintaining a continuous observation–decision-action cycle. (10) Performance Measurement and Output: After all 𝑁 = 200 evaluation tasks have been completed, the framework aggregates the task-level results and computes the classification and system-performance metrics. Overall accuracy and per-category accuracy are calculated from the predicted and ground-truth labels. Average latency and the 95th-percentile latency are obtained from the recorded task execution durations, while throughput and total execution time quantify processing efficiency. Queue depth and worker utilization are retained to characterize autoscaling behavior, and the resource-efficiency metric is calculated from the achieved classification accuracy and runtime. The framework generates a structured final report containing the classification results and runtime statistics. Task-level results are stored with the task identifier, true label, predicted label, correctness status, execution duration, and worker identifier. The resulting data are subsequently used to generate the performance plots, comparison tables, worker-scaling traces, queue-depth curves, latency and throughput analyses, and resource-efficiency results presented in Section 7. (11) Baseline and Comparative Evaluation: The proposed framework was evaluated using Autoscaling+BART and Autoscaling+DeBERTa. These configurations were compared against Static+BART, Standalone RoBERTa, and Standalone DistilBERT. Static+BART uses the same BART-large-MNLI classifier with a fixed two-worker allocation and therefore isolates the effect of dynamic worker management. Standalone RoBERTa performs sequential zero-shot classification without the proposed queue management or autoscaling mechanism, while Standalone DistilBERT uses the fine-tuned model without dynamic worker-pool orchestration. All configurations use the same evaluation samples and consistent performance metrics. This setup enables the effects of classifier selection and worker-pool elasticity to be examined independently. 7
Results and Discussion
This section evaluates the proposed agentic autoscaling framework through comprehensive experiments on the AG News and SMS Spam Collection benchmark datasets. The proposed framework is compared with representative static and standalone baselines under identical experimental settings. The discussion is organized into four aspects: (i) dynamic worker scaling and queue evolution, (ii) latency and throughput, (iii) classification performance, and (iv) resource efficiency analysis. Together, these results demonstrate the effectiveness of the proposed application-layer autoscaling framework for scalable LLM-driven text classification. Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
17
The experimental evaluation is first motivated by the workload behavior illustrated in Figure 4, which compares static worker allocation with the proposed agentic autoscaling strategy under bursty workloads. With a fixed worker pool, an increase in the task arrival rate beyond the available processing capacity causes queue accumulation, increased waiting time, and higher execution latency, while low-demand periods result in worker underutilization. In contrast, the proposed framework continuously observes application-level conditions, including queue depth and worker utilization, and dynamically adjusts the worker pool according to workload demand. Thus, the motivation for dynamic autoscaling is to increase processing capacity during workload bursts and reduce unnecessary worker allocation during low-demand periods, thereby improving responsiveness and resource efficiency. Let 𝜆(𝑡) denote the task arrival rate and let each worker process tasks with an average service rate 𝜇. For a fixed worker pool of size 𝑛, the processing capacity is 𝐶 (𝑡) = 𝑛𝜇. Since 𝑛 remains constant under static allocation, burst periods with 𝜆(𝑡) > 𝐶 (𝑡) cause tasks to arrive faster than they can be processed, resulting in continuous queue growth. The queue dynamics are given by 𝑑𝑞(𝑡)/𝑑𝑡 = 𝜆(𝑡) − 𝐶 (𝑡), indicating that queue depth increases whenever the arrival rate exceeds the available processing capacity. Consequently, waiting time and end-to-end latency increase, whereas during steady-state operation (𝜆(𝑡) ≪ 𝐶 (𝑡)), the fixed worker pool becomes underutilized, wasting computational resources.
Fig. 4. Static worker allocation versus the proposed agentic autoscaling framework under bursty workloads
To overcome this limitation, the proposed agentic autoscaling framework dynamically adjusts the worker pool according to application-level workload conditions. The processing capacity becomes 𝐶 (𝑡) = 𝑛(𝑡)𝜇, where 𝑛(𝑡) denotes the dynamically allocated worker pool. The autoscaling agent aims to maintain 𝜆(𝑡) ≈ 𝐶 (𝑡) by continuously observing queue depth and worker utilization, thereby minimizing queue growth during bursty workloads while avoiding unnecessary resource allocation under light workloads. 7.1
Dynamic Worker Scaling and Queue Evolution
Figures 5(a)–(b) and 6(a)-(b) illustrate the runtime behavior of the proposed framework under bursty workloads. In both figures, the x-axis represents the workload observation interval (𝑡 1 –𝑡 40 ), while the y-axis denotes either the number of Manuscript submitted to ACM
18
B. Kumar, A. Verma, R. Buyya
active workers or the queue depth. Figure (a) shows that both Autoscaling+BART and Autoscaling+DeBERTa initialize with four workers, expand to six workers at approximately 𝑡 5 , and reach a maximum of eight workers during the burst period (𝑡 9 –𝑡 36 ). As the workload decreases, the worker pool is gradually reduced to seven workers, demonstrating adaptive resource provisioning. In contrast, the Static configuration maintains a constant pool of two workers throughout execution, while the standalone RoBERTa and DistilBERT baselines execute sequentially with a single worker. Figure (b) presents the corresponding queue-depth evolution. Under the static and standalone configurations, the queue grows continuously throughout the workload, reaching approximately 190-225 pending tasks by the end of execution because the fixed worker pool cannot match the incoming request rate. In contrast, the proposed autoscaling framework limits the queue to approximately 110 tasks during the burst phase and gradually drains it to zero by the end of execution through dynamic worker provisioning. These results demonstrate that application-layer autoscaling effectively matches processing capacity to workload demand, substantially reducing queue buildup and waiting time. 7.2
Latency and Throughput
Figures 5(c)–(d) and 6(c)-(d) compare the execution latency and throughput of the evaluated approaches. In both subfigures, the x-axis represents the workload observation interval, while the y-axis corresponds to latency per task (seconds) and throughput (tasks/min), respectively. For the AG News dataset, Autoscaling+BART consistently maintains the lowest latency of approximately 5–8 s throughout execution, whereas Autoscaling+DeBERTa initially exhibits higher latency (approximately 160 s) owing to the computational complexity of DeBERTa-v3-large before gradually decreasing to about 40 s as additional workers are provisioned. The Static configuration continues to experience increasing latency, reaching approximately 80 s by the end of execution because of persistent queue buildup. Similar behavior is observed on the SMS Spam dataset, where Autoscaling+BART maintains latency below 10 s, while the Static and DistilBERT baselines increase to approximately 90 s and 220 s, respectively. Figure (d) shows that Autoscaling+BART achieves the highest throughput, increasing from approximately 18 tasks/min to 35 tasks/min during the burst period before stabilizing around 31 tasks/min. Autoscaling+DeBERTa achieves approximately 24 tasks/min on AG News and about 1.5 tasks/min on SMS Spam because of the higher inference cost of the larger language model. In contrast, the static and standalone approaches maintain nearly constant throughput throughout execution, indicating their inability to adapt to workload fluctuations. Overall, dynamic worker provisioning significantly improves execution efficiency by increasing parallelism while reducing waiting time. 7.3
Classification Performance
This subsection evaluates the classification effectiveness of the proposed framework on the AG News and SMS Spam Collection datasets. The analysis focuses on overall and per-category classification accuracy while comparing the proposed autoscaling framework with static and standalone baseline configurations. 7.3.1 AG News Dataset. Figure. 5(e) summarize the classification results on the AG News dataset. Using the same BART-large-MNLI classifier, the proposed autoscaling framework improves overall accuracy from 75.2% under static allocation to 84.5%, while simultaneously reducing execution time through dynamic worker management. Replacing BART with DeBERTa-v3-large further increases overall accuracy to 90.5% without modifying the queue management, worker execution, or autoscaling strategy, demonstrating that the proposed framework remains independent of the underlying language model. The per-category results show that Autoscaling+DeBERTa consistently achieves the highest accuracy across all four news categories, followed by Autoscaling+BART. In comparison, the static BART configuration Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
19
Fig. 5. Experimental results on the AG News dataset. The dashboard presents (a) worker scaling, (b) queue depth, (c) latency, (d) throughput, (e) per-category accuracy, and (f) resource efficiency.
exhibits reduced performance, while the standalone DistilBERT and RoBERTa baselines achieve lower overall accuracy. These results indicate that the proposed application-layer autoscaling framework preserves or improves classification performance while enabling interchangeable language models within the same execution environment. 7.3.2 SMS Spam Collection. Figure 6(e) present the classification results for the SMS Spam Collection dataset. Owing to the strong lexical distinction between legitimate and spam messages, most evaluated approaches achieve relatively high overall accuracy. Among all methods, Autoscaling+DeBERTa achieves the highest overall accuracy of 99.5%, whereas Autoscaling+BART attains 84.5% while providing substantially lower execution time. A notable observation is the poor performance of the standalone RoBERTa classifier, which achieves only 47.5% overall accuracy because of extremely low spam-class recall (1%), despite correctly classifying most ham messages. This result highlights a limitation of NLI-based zero-shot classification for imbalanced binary classification tasks and emphasizes that classifier selection remains critical even when efficient autoscaling is employed. Overall, the proposed framework consistently matches or Manuscript submitted to ACM
20
B. Kumar, A. Verma, R. Buyya
Fig. 6. Experimental results on the SMS Spam Collection dataset. The dashboard presents (a) worker scaling, (b) queue depth, (c) latency, (d) throughput, (e) per-label accuracy, and (f) resource efficiency.
exceeds the classification performance of the corresponding static configurations while maintaining complete classifier independence. 7.4
Resource Efficiency Analysis
Figures 5(f) and 6(f) compare the overall resource efficiency of the evaluated approaches using six complementary dimensions: throughput, latency, resource use (utilization), fault tolerance, scalability, and cost efficiency. On the AG News dataset, Autoscaling+BART achieves the strongest overall efficiency profile, with scores of approximately (90, 85, 78, 92, 95, and 88), while Autoscaling+DeBERTa attains the second-best efficiency profile (70, 65, 70, 95, 90, and 85) together with the highest classification accuracy. In contrast, the static BART configuration exhibits noticeably lower scalability and cost efficiency because its fixed worker pool cannot adapt to workload fluctuations, whereas the standalone RoBERTa and DistilBERT baselines achieve the lowest overall efficiency due to their sequential execution Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
21
model. A similar trend is observed on the SMS Spam Collection dataset. Autoscaling+BART again provides the best balance across throughput, latency, resource utilization, fault tolerance, scalability, and cost efficiency, with scores of approximately (88, 85, 78, 92, 95, and 88). Autoscaling+DeBERTa achieves the second-best efficiency while delivering the highest classification accuracy, with slightly lower efficiency attributable to its higher computational cost. Both autoscaling configurations consistently outperform the static and standalone baselines across the evaluated resourceefficiency dimensions. Overall, the observed changes in worker count and queue depth demonstrate that the proposed agentic worker-pool orchestration can adapt processing capacity to bursty workloads, while the latency and throughput results confirm the effectiveness of asynchronous execution and application-layer autoscaling. The classification results further demonstrate the classifier-agnostic nature of the framework, as different zero-shot and fine-tuned language models can be integrated without modifying the queue management, worker execution, or autoscaling mechanism. At the same time, the comparative results indicate that model selection remains an important factor in determining the accuracy-efficiency trade-off, while the autoscaling mechanism independently manages processing capacity according to workload demand. The resource-efficiency results further show that dynamic worker-pool management provides improved scalability and resource utilization compared with fixed or sequential execution. Collectively, these findings validate the proposed five-component architecture and its two complementary control mechanisms, asynchronous agent-worker execution and threshold-based autoscaling-and demonstrate that worker-pool elasticity and language-model selection can be treated as independent design dimensions for scalable LLM-driven text classification.
8
Conclusions and Future Work
This paper proposed an agentic autoscaling framework for LLM-driven text classification that decouples infrastructure elasticity from model selection through a classifier-agnostic worker-pool orchestration architecture implemented entirely using python’s asyncio library. Our framework integrates priority-based task scheduling, asynchronous agent execution, runtime monitoring, and application-layer autoscaling without requiring external orchestration platforms. Experimental evaluation on the AG News and SMS Spam Collection benchmark datasets demonstrates that Autoscaling+DeBERTa achieves the highest classification accuracies of 90.5% and 99.5%, respectively, while Autoscaling+BART provides a favorable balance between accuracy (84.5% on both datasets) and execution efficiency. Compared with static resource allocation and standalone baselines, the proposed framework consistently improves latency, throughput, scalability, and overall resource efficiency. The results further demonstrated improvements in queue management, latency, throughput, scalability, and resource efficiency compared with static and standalone baselines. These findings validate the proposed architecture and demonstrate that worker-pool elasticity can be managed independently of language-model selection, enabling different zero-shot and fine-tuned classifiers to operate within a common autoscaling framework. The current study is limited to two benchmark datasets, manually configured threshold parameters, and evaluation within a single computing environment. Future work will extend the framework to larger and more diverse workloads and additional language models, while investigating automated parameter optimization and predictive autoscaling to anticipate workload changes before queue buildup occurs. Further research will also investigate distributed cloud-edge deployment, intelligent task placement, and heterogeneous resource allocation, enabling lightweight models to operate at the edge while computationally intensive LLMs are dynamically assigned to cloud resources. These extensions can further improve scalability, adaptability, and resource efficiency for large-scale agentic AI applications. Manuscript submitted to ACM
22
B. Kumar, A. Verma, R. Buyya
Acknowledgments We gratefully acknowledge the support of the Institutions of Eminence (IoE) Scheme of Banaras Hindu University, Varanasi, India under Development Scheme No. 6031 and an Australian Research Council (ARC) Discovery Project grant.
References [1] Muhammad Abdullah, Waheed Iqbal, Josep Ll. Berral, Jordi Polo, and David Carrera. 2023. Burst-Aware Predictive Autoscaling for Containerized Microservices. IEEE Transactions on Services Computing 16, 6 (2023), 4058–4071. doi:10.1109/TSC.2020.2995937 [2] Yahya Al-Dhuraibi, Fawaz Paraiso, Nabil Djarallah, and Philippe Merle. 2018. Elasticity in Cloud Computing: State of the Art and Research Challenges. IEEE Transactions on Services Computing 11, 2 (2018), 430–447. doi:10.1109/TSC.2017.2711009 [3] Tiago A. Almeida, José María Gómez Hidalgo, and Akebo Yamakami. 2011. Contributions to the study of SMS spam filtering: new collection and results. In Proceedings of the 11th ACM Symposium on Document Engineering (DocEng ’11). Association for Computing Machinery, New York, NY, USA, 259–262. doi:10.1145/2034691.2034742 [4] Zinuo Cai, Rongbo Ma, Yicheng Fu, Weishan Zhang, Ruhui Ma, and Haibing Guan. 2024. LLMaaS: Serving Large-Language Models on Trusted Serverless Computing Platforms. IEEE Transactions on Artificial Intelligence 6, 2 (2024), 405–415. [5] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1–45. [6] Sadia Sultana Chowa, Riasad Alvi, Subhey Sadi Rahman, Md Abdur Rahman, Mohaimenul Azam Khan Raiaan, Md Rafiqul Islam, Mukhtar Hussain, and Sami Azam. 2026. From language to action: a review of large language models as autonomous agents and tool users. Artificial Intelligence Review (2026). [7] M. H. Jabreel. 2016. GitHub - mhjabreel/CharCnn_Keras: The implementation of text classification using character level convoultion neural networks using Keras — github.com. https://github.com/mhjabreel/CharCnn_Keras. [Accessed 15-07-2026]. [8] Byeonghui Jeong and Young-Sik Jeong. 2024. Arascaler: Adaptive resource autoscaling scheme using etimemixer for efficient cloud-native computing. IEEE Transactions on Services Computing 18, 1 (2024), 72–84. [9] Bablu Kumar, Anshul Verma, and Pradeepika Verma. 2024. Optimizing resource allocation using proactive scaling with predictive models and custom resources. Computers and Electrical Engineering 118 (2024), 109419. [10] Bablu Kumar, Anshul Verma, and Pradeepika Verma. 2025. A multivariate transformer-based monitor-analyze-plan-execute (MAPE) autoscaling framework for dynamic resource allocation in cloud environment. Computing 107, 3 (2025), 69. [11] Bablu Kumar, Anshul Verma, Pradeepika Verma, and Akram Bennour. 2025. Optimizing resource allocation in cloud-native applications through proactive autoscaling with the InformerAutoScale model. The Journal of Supercomputing 81, 9 (2025), 1077. [12] Yandi Li, Jianxiong Guo, Zhiqing Tang, Xingjian Ding, Juncheng Wang, Tian Wang, and Weijia Jia. 2025. Cloud-Edge System for Scheduling Unpredictable LLM Requests With Combinatorial Bandit. IEEE Transactions on Services Computing 18, 6 (2025), 3567–3580. [13] Yanying Lin, Shijie Peng, Shuaipeng Wu, Yanbo Li, Chengzhi Lu, Kejiang Ye, and Chengzhong Xu. 2025. Serving LLM in Distributed GPU Cluster With Fine-Grain Pipeline Constraints. IEEE Transactions on Services Computing 18, 5 (2025), 3164–3176. [14] J. Markham. 2016. pycon-2016-tutorial/data/sms.tsv at master · justmarkham/pycon-2016-tutorial — github.com. https://github.com/justmarkham/ pycon-2016-tutorial/blob/master/data/sms.tsv. [Accessed 15-07-2026]. [15] EunChan Park, KyeongDeok Baek, and In-Young Ko. 2025. Hierarchical Decentralized Autoscaling for Spatio-Temporal Load Bursts. IEEE Transactions on Services Computing 18, 6 (2025), 3865–3877. [16] Adamu Gaston Philipo, Doreen Sebastian Sarwatt, Jianguo Ding, Mahmoud Daneshmand, and Huansheng Ning. 2025. Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms. IEEE Transactions on Information Forensics and Security 20 (2025), 7602–7616. doi:10.1109/TIFS.2025.3588728 [17] Zoe Porter, Radu Calinescu, Ernest Lim, Victoria Hodge, Philippa Ryan, Simon Burton, Ibrahim Habli, Tom Lawton, John McDermid, John Molloy, et al. 2025. INSYTE: a classification framework for traditional to agentic AI systems. ACM Transactions on autonomous and adaptive systems 20, 3 (2025), 1–39. [18] Olesia Pozdniakova, Dalius Mažeika, and Aurimas Cholomskis. 2024. SLA-Adaptive Threshold Adjustment for a Kubernetes Horizontal Pod Autoscaler. Electronics 13, 7 (2024), 1242. doi:10.3390/electronics13071242 [19] Berta Serracanta, Andor Lukács, Alberto Rodriguez-Natal, Albert Cabellos, and Gábor Rétvári. 2025. On the Stability of the Kubernetes Horizontal Autoscaler Control Loop. IEEE Access 13 (2025), 7160–7166. doi:10.1109/ACCESS.2025.3526751 [20] Sahar Shah, Sara Lucia Manzoni, Farooq Zaman, Fatima Es Sabery, Francesco Epifania, and Italo Francesco Zoppis. 2024. Fine-tuning of distil-bert for continual learning in text classification: An experimental analysis. IEEE Access 12 (2024), 104964–104982. [21] The Kubernetes Authors. 2026. Horizontal Pod Autoscaler. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/. Accessed: 2026-07-13. Manuscript submitted to ACM
Agentic Autoscaling for LLM-driven Text Classification
23
[22] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345. [23] Y. Wang, H. Li, and X. Zhang. 2024. Resource Management for AI Workloads in Cloud Computing: Challenges and Opportunities. IEEE Transactions on Cloud Computing 12, 4 (2024), 2101–2116. [24] Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2024. Give us the facts: Enhancing large language models with knowledge graphs for fact-aware language modeling. IEEE Transactions on Knowledge and Data Engineering 36, 7 (2024), 3091–3110. [25] Dianhai Yu, Liang Shen, Hongxiang Hao, Weibao Gong, Huachao Wu, Jiang Bian, Lirong Dai, and Haoyi Xiong. 2024. Moesys: A distributed and efficient mixture-of-experts training and inference system for internet services. IEEE Transactions on Services Computing 17, 5 (2024), 2626–2639. [26] Xingyu Yuan, He Li, Mianxiong Dong, and Kaoru Ota. 2026. Adaptive scheduling of multimodal large language model in intelligent edge computing. ACM Transactions on Autonomous and Adaptive Systems 21, 1 (2026), 1–22. [27] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems. 649–657.
Manuscript submitted to ACM