Blackbox Performance Analysis of LLM-based Network Control

Blackbox Performance Analysis of LLM-based Network Control Mahdi Alizadeh

Pooria Namyar

University of Southern California

Microsoft Research

Arvin Ghavidel

Ramesh Govindan

University of Southern California

University of Southern California effective in a world of hand-written heuristics or solvers whose logic is authored, stable, and inspectable. LLM-based network control breaks that bargain. Hosted LLMs expose an API, not weights, prompts, decoding logic, or processing steps. Even when weights are public, faithfully reducing billions of parameters and their processing logic to a formal specification or source-level model is intractable. On the other hand, black-box methods such as random search, hill climbing, and simulated annealing need no model of the target and so can be plausibly used for LLM-based network control. However, they are ineffective because they treat candidate inputs independently or exploit only local feedback, and often fail to uncover meaningful adversarial inputs even for simple heuristics [29] (§3). Given the likely future prevalence of LLM-enabled network control, it is imperative to develop performance auditing techniques that scale with opacity instead of assuming it away. As more network management tasks employ LLMs [38], more of their logic will run behind an opaque API rather than in code anyone can read or a model anyone can query for gradients. A performance analyzer that assumes inspectability will have less and less left to analyze. LENS: RL-based search for adversarial inputs. In this paper, we introduce LENS, a black-box performance analyzer for LLM-based network control. LENS, unlike prior work (Table 1), requires no access to model weights, source code, or other solver internals. It requires only a representation of valid inputs, an executable interface to the target LLM solver, and a function for scoring the quality of its decisions (e.g., the gap between the LLM’s decision and the optimum). LENS casts adversarial input generation as a reinforcement learning (RL) problem (§3). Starting from a valid seed input, the RL policy applies a sequence of changes to the input. LENS evaluates the modified input, and uses the resulting target-to-optimal performance gap as the reward. Unlike local black-box search, its search policy reuses experience across evaluations to discover adversarial patterns, such as TE demand configurations that expose bottlenecks consistently mishandled by the target LLM. Challenges and Contributions. Unfortunately, vanilla RL is ineffective for this task for three reasons.

arXiv:2610.24348v1 [cs.NI] 17 Oct 2026

Abstract LLMs are becoming an indispensable part of network control systems due to their impressive reasoning. Before such LLMenabled systems can be deployed, it is imperative to develop tools that can audit their performance on previously unseen inputs. However, auditing these systems is challenging due to the inherent opacity of LLMs. LENS is the first step towards black-box performance analysis of LLM-enabled network control systems. It uses novel methods to extend deep reinforcement learning to guide the search for performance gaps and inputs that make LLMs struggle. LENS finds large gaps for LLM-based traffic engineering and GPU allocation systems, and even matches existing white-box analyzers designed for hand-crafted TE heuristics.

1

Introduction

Large language models (LLMs) are moving beyond conversation and code generation to decision-making [16, 18, 40–42]. Motivated by their reasoning and generalization capabilities, research is starting to use them for network control: allocating paths, compute, and other resources (§2). For example, LMTE [41] uses LLMs to solve traffic engineering (TE) problems, exceeding 99.5% accuracy on held-out demands. We show that despite this, an adversary without access to a single model weight can construct a valid traffic matrix that causes LMTE to leave more than half of a network’s traffic unrouted relative to the optimal solution (§4). WANs today route terabytes of demand over capacity-constrained links [14, 23, 31], and must either drop the resulting unrouted demand, degrading user-perceived performance, or overprovision for it, raising costs. Thus, before LLM-enabled network control is pervasively deployed, we must address a more challenging issue than average-case accuracy: how poorly can it perform on valid, realistic inputs, and how would we even find out? Prior work on performance analyzers answers exactly this question by searching for adversarial inputs, but for a different class of target. White-box tools require a precise mathematical or logical description of the target [20, 29, 30, 37]; gray-box methods require source code, gradients, or access to other system internals [19, 32]. Such tools are 1

Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, and Ramesh Govindan

No Additional No Source Structure Formal Modeling Code Access Aware Search

Tool LENS (ours) White-box analysis [12, 30, 37] Source code analysis [19] Gray-box analysis [32] Traditional black-box [10, 21]

✓ ✗ ✓ ✗ ✓

✓ ✓ ✗ ✓ ✓

✓ ✓ ✓ ✓ ✗

Table 1: Related work on performance analyzers.

First, the input space is high-dimensional. Vanilla RL often discovers the coarse shape of an adversarial input but fails to refine its exact values and can get trapped in a local optimum. LENS addresses this problem with multi-stage refinement. After an initial broad search, it estimates the importance of each input field, freezes the fields responsible for the current gap, and resumes RL over the remaining fields. Thus, each stage reduces the effective search space and ensures the RL policy focuses on specific parts of the input while preserving earlier progress. Second, RL requires many reward observations, but computing each reward is expensive: it requires a call to a remote LLM service and an invocation of an optimization solver. LENS reduces this cost through sparse reward evaluation and result caching. In addition, it increases throughput with parallel actors and improves efficiency by warm-starting optimization solvers from solutions to nearby inputs. Third, unconstrained RL exploration can produce unrealistic inputs. LENS lets operators describe realistic workloads through constraints. It then restricts the search accordingly, projecting each edit into the feasible space. The resulting adversarial inputs remain valid and relevant to the target deployment. Results. We evaluate LENS on two LLM-based solvers (§4). LMTE [41] achieves more than 99.5% accuracy on its test set, yet LENS finds traffic matrices that induce a normalized performance gap of more than 50% without workload constraints and 15.5% under constraints derived from the training set. In the constrained setting, LENS’s gap is about 180% larger than that found by the strongest black-box baseline. Similarly, for cluster scheduling using LLMCoSolver [18], LENS finds an adversarial gap about 28.5× larger than the maximum gap observed on the test set and improves over the strongest black-box baseline by about 130%. Equally importantly, LENS is more general than, or as effective as, some existing white-box analyzers. For a TE heuristic [29], LENS matches the gap found by MetaOpt [30] without modeling the heuristic at all. It also outperforms MetaEase [19] by 3.75× and the strongest traditional blackbox baseline by 7.3× under the same time budget. LENS generalizes to Teal [39], an RL-based TE solver for which white-box methods (or even gray-box methods such as [32])

Figure 1: LLM-based traffic engineering workflow.

cannot be applied, and beats the strongest black-box baseline by about 120%. Looking ahead. The same property that makes LENS useful today — that it needs only an executable interface, not a model — ensures that it can scale and adapt as network control systems evolve to use LLMs. We sketch four directions in §5: scaling to larger problem instances; moving from single high-gap instances to diverse adversarial corpora that can be used for adversarial retraining; repurposing the offline attack policy as an online signal for routing instances to the solver least likely to mishandle them; and, as agentic AI takes on more of network management, using LENS to systematically probe those systems for reliability gaps without ever needing access to their weights or prompts.

2

LLM-based Network Control

In this paper, we consider two LLM-based solvers for network control problems. Both produce high-quality solutions that are near-optimal on test sets. Traffic Engineering. TE determines how to split demands between WAN ingress-egress pairs across multiple paths while (a) respecting link capacities, and (b) optimizing a network-wide objective such as maximizing total flow (MaxFlow) or minimizing the maximum link utilization (MLU). In LMTE [41], an LLM-based approach to solving TE problems (Fig. 1), a controller collects traffic history and network state, converts them into the model’s input format, and sends a query to an LLM. LMTE trains an encoder that outputs an embedding for recent traffic matrices, topology, and link capacities. It inputs this embedding together with a prompt (Fig. 1) describing the task to a frozen language-model backbone, then applies a lightweight post-processing step to produce split ratios. GPU Cluster Scheduling. Allocating GPUs to jobs in a cluster while maximizing throughput and respecting capacity constraints can be formulated as a knapsack problem [36]. We have developed an LLM-based solver for this problem, leveraging LLMCoSolver [18], a generic LLM-based optimization solver. LLMCoSolver must first be fine-tuned on problem instances. Then, given a cluster configuration and a set of jobs, it returns an allocation of GPUs to jobs. 2

State Action Reward

TE

GPU Scheduling

Demand matrix of src–dest traffic volumes. Edit multiple demand entries. Routed flow difference (OPT vs. LLM).

Job matrix of throughput and GPU requirements. Edit throughput/GPU use of multiple jobs. Selected throughput difference (OPT vs. LLM).

Table 2: Mapping RL to specific problem domains.

3

Figure 2: LENS architecture. The policy edits a structured input, the environment projects the edit back into the valid input space, and the 𝑄 OPT (𝐼 ) and 𝑄 LLM (𝐼 ) executors evaluate the result. The measured gap is stored as a reward and reused by the learner.

LENS: Black-box RL-based Search

Adversarial Input Search. Consider the TE problem, and let 𝑄 LLM (𝐼 ) and 𝑄 OPT (𝐼 ) denote the quality (e.g., Max-Flow) of the solutions produced by the LLM and the optimal solver on the traffic matrix 𝐼 , respectively. Given an operator-defined input space X that captures input constraints, the goal of adversarial input search is to find 𝐼 ★, the input that maximizes the difference in solution quality: 𝐼 ★ ∈ arg max𝐼 ∈ X [𝑄 OPT (𝐼 ) − 𝑄 LLM (𝐼 )]. An analogous definition applies to GPU cluster scheduling. A performance analyzer that searches for adversarial inputs must satisfy two requirements: (a) it must treat the entire LLM-based pipeline as a black box; and (b) it must navigate a high-dimensional, constrained input space effectively enough to uncover meaningful adversarial inputs. Exploiting Reinforcement Learning (RL). RL learns a policy by interacting with an environment. At each step, the policy observes the current state, selects an action, transitions to a new state, and receives a scalar reward. Over repeated interactions, the policy learns which actions are likely to lead to a high-reward state. Adversarial input search can be modeled using RL (Table 2). The current candidate input represents the state, an edit to that input constitutes the action, and the performance gap between the LLM-based system and the optimal solver defines the reward. This approach satisfies our two requirements. In RL, the search policy does not need the LLM source code, model weights, or gradients. It can also search efficiently; rather than treating each candidate input as an isolated trial, the policy learns from prior inputs and rewards to propose promising edits that quickly lead to adversarial inputs. Challenges. Despite this natural fit, we find that off-theshelf vanilla RL is sub-optimal for LENS’s adversarial input search. First, the input space is high-dimensional, which hinders exploration; as a result, vanilla RL often fails to find strong adversarial inputs and can become trapped in local optima (§3.2). Second, RL learns through many multi-step trials, but evaluating the reward at each step is expensive because it requires invoking a remote LLM service and an optimization solver (§3.3). Third, unconstrained exploration can produce invalid or implausible inputs irrelevant to deployment (§3.4).

3.1

Architecture Overview

Deep RL learner observes the current input 𝐼𝑡 and proposes an edit 𝑎𝑡 intended to increase the performance gap. During search, it updates its policy from observed transitions (𝐼𝑡 , 𝑎𝑡 , 𝐼𝑡 +1, 𝑟𝑡 ) so that edits that repeatedly produce larger gaps become more likely. Since the learner uses only this input-output feedback, it does not require access to the internal operation of the LLM-based system. For training the learner’s policy, LENS uses Soft ActorCritic (SAC) [13]. Compared with an on-policy learner such as PPO [35], this off-policy design better fits our setting for two reasons. First, SAC’s entropy objective encourages continued exploration, helping the policy avoid local optima. Second, SAC stores transitions in a replay buffer and reuses them across policy updates, whereas on-policy methods discard them after each update. This reuse allows the policy to learn complex adversarial patterns from past experience. The policy’s feature extractor matches the input structure. We use dense graph convolutions for TE demand matrices and an MLP for vector-valued cluster scheduling inputs. Environment maintains the current input and applies the edit proposed by the learner. It then calls the LLM-based solver and the optimization solver to obtain their solutions, computes the performance gap 𝑄 OPT (𝐼𝑡 +1 ) − 𝑄 LLM (𝐼𝑡 +1 ), and stores the observed transition and reward in the replay buffer. In traffic engineering, an edit may increase or decrease the demands between multiple source–destination pairs. In cluster scheduling, it may change the priority, arrival time, duration, or resource requirements of multiple jobs. Replay Buffer stores transitions and rewards (𝐼𝑡 , 𝑎𝑡 , 𝐼𝑡 +1, 𝑟𝑡 ) generated during the search. The policy network reuses these transitions to update its parameters frequently, which allows it to learn complex adversarial patterns from past experience and prevents it from getting stuck in local optima [13]. 3.2

Multi-Stage Refinement

Vanilla RL often learns the coarse-grained structure of an adversarial input but struggles to refine it into a high-impact adversarial input. In TE, for example, the policy may learn 3

Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, and Ramesh Govindan

which demands should increase or decrease to pressure particular bottlenecks, but may fail to identify their precise values. Two factors make this refinement task challenging for RL. First, the input space is high-dimensional, and the performance gap landscape is non-convex. Because each action can modify many fields (e.g., traffic demands, or job GPU allocations), exploring a better value for one field can undo useful assignments elsewhere. Second, the scalar reward conveys only the aggregate gap and does not attribute the gap to specific fields. Using vanilla RL produces gaps 3–5× smaller than those identified by LENS (§4.3). LENS overcomes this limitation with multi-stage refinement. LENS first uses RL to discover a coarse adversarial pattern. It then identifies the fields most responsible for the current gap, freezes their values, and resumes refining the remaining fields. Repeating this process preserves progress from earlier stages while reducing the number of fields that remain subject to exploration. The learner can therefore shift from discovering the overall adversarial pattern to refining the values of the remaining unconstrained fields. To identify critical fields, LENS assigns an importance score to each field in the current input. The score measures how strongly the current gap is sensitive to that field’s value. Formally, let 𝐼 = (𝑣 1, . . . , 𝑣𝑛 ) be the current adversarial input and 𝑅(𝐼 ) = 𝑄 OPT (𝐼 ) − 𝑄 LLM (𝐼 ) be its performance gap. For each field 𝑖, LENS samples 𝑀 valid alternative values, substitutes each one while holding all other fields fixed, and evaluates the resulting gaps 𝑅𝑖,1, . . . , 𝑅𝑖,𝑀 . The importance Í score for field 𝑖 is defined as 𝑅(𝐼 ) − 𝑀1 𝑀 𝑗=1 𝑅𝑖,𝑗 . A large positive score means that perturbing the field typically reduces the performance gap, so LENS freezes its current value in subsequent stages. Fields with scores near or below zero remain open to optimization by the RL policy. LENS then repeats the search-and-freeze process until it exhausts its stage budget.

3.3

reward. This avoids redundant LLM and solver calls while retaining trajectory-level feedback. Reward caching and warm starts. LENS caches each evaluated input and its measured gap. When an identical input reappears under the same solver configuration, LENS reuses the cached gap without re-evaluating the target LLM or the optimization solver. For nearby inputs, it uses the cached optimal solution to warm-start the optimization solver. The optimizer still computes the solution for the new input, but typically converges faster. Together, caching and warm starts minimize redundant computation and reduce latency. Parallel actors. LENS runs independent search instances in parallel from different seed inputs to increase search coverage. The actors explore different regions concurrently and periodically share high-gap candidates or a replay buffer. 3.4

Enforcing Constraints on Inputs

Operators typically seek adversarial inputs consistent with their expected workloads. LENS allows them to define an admissible search space X that captures both validity and the operating conditions relevant to their deployment, then searches only for adversarial inputs within X. For example, since WAN traffic matrices are often sparse and local [1, 30], an operator may constrain demand density and source– destination locality when evaluating LLM-based TE. LENS enforces these requirements through a projection operator, loosely inspired by the projection step in projected gradient descent (PGD [5]). After each edit, it projects the candidate to the nearest feasible input in X, evaluates only the projected input, and uses that input as the state for the next RL step. We have implemented projection operators for constraints that specify bounds on individual input fields or aggregate quantities, as well as sparsity and locality. For example, for locality, LENS sets demands between source– destination pairs that exceed a certain distance threshold to zero. For density, it retains the allowed number of positive demands and sets to zero the smallest remaining entries. Adding a new type of constraint requires only implementing its projection operator; the LLM-based system and optimal solver remain black boxes.

Scaling LENS

In theory, after every proposed edit the RL learner must compute a new performance gap as its reward. Computing this gap can dominate runtime: (1) the target may be a cloudhosted LLM service, and (2) computing the optimal solution can be extremely slow. LENS combines three techniques to accelerate RL search. These optimizations have minimal impact on the quality of the discovered inputs (§4.3). Sparse rewards. Because each edit is bounded, the performance gap often changes slowly between consecutive inputs. LENS therefore evaluates the performance gap every 𝑘 steps, or sooner if the cumulative distance from the last evaluated input exceeds a configured threshold. At intermediate steps, the policy receives the most recently measured gap as its

4

Evaluation

We evaluate LENS on two LLM-based optimization solvers: LMTE [41] for traffic engineering and LLMCoSolver [18] for cluster scheduling (§2). We show that LENS can expose large performance gaps without access to their internal solver pipelines. We then use LENS to analyze a heuristic and a nonLLM learning-enabled system to demonstrate its generality and effectiveness. Experimental setting. We run LENS and each baseline with five different random seeds and report the largest gap found 4

Figure 3: LENS outperforms black-box baselines for (a) traffic engineering targets, LMTE and Teal, and (b) cluster scheduling with LLMCoSolver. Teal is the learned target discussed in §4.2.

Figure 4: LENS matches MetaOpt, while outperforming MetaEase and black-box baselines under the same time budget.

across these runs. All methods are configured to utilize all available CPU threads. Baselines. We compare LENS with generic black-box search methods: random search, simulated annealing, and hill climbing [10, 21]. We also compare against MetaOpt [30], a whitebox analyzer that computes a provably optimal gap, and MetaEase [19], a source-code-based analyzer. 4.1

finds the largest gap at every evaluated problem size, and the difference grows as the number of jobs increases. 4.2

Beyond LLM-based Network Control

LENS matches or exceeds the white-box analyzers for Demand Pinning. Demand pinning (DP) [22, 29] routes demands that fall below a specified threshold along their shortest paths and solves an optimization problem for the remaining large demands. We set its threshold to 5% of the average link capacity and bound each demand by half of the average link capacity. We use B4 and 𝐾 = 4 shortest paths. Both MetaOpt and MetaEase can analyze DP. Fig. 4 shows that LENS eventually matches MetaOpt’s gap while outperforming MetaEase by 3.75× and the strongest black-box baseline by 7.3×. MetaOpt uses a mathematical model of the heuristic to provably compute the worst-case gap. In contrast, LENS requires only black-box input-output access to DP and still reaches the same gap, albeit at a slower rate because it lacks access to DP’s internals. LENS also applies to non-LLM learned systems. Teal [39] is an RL-based TE system that uses a graph neural network to predict traffic splits and ADMM [4] to repair infeasible outputs. This combination of a neural policy and iterative repair cannot be formulated in MetaOpt [30] or analyzed by end-to-end gradient analysis [32]. LENS treats Teal as a black box and can analyze its performance. We train Teal’s open-source implementation on the same dataset used for LMTE, and the resulting model achieves more than 99.5% accuracy on the test set. Under input constraints, LENS finds a normalized gap of 9.4%, outperforming all black-box baselines. Without these constraints, LENS finds a gap of 64.70%.

LLM-Based Network Control

LENS exposes a 57.51% normalized gap in LMTE. We measure the difference between the traffic routed by the optimal solver and LMTE, normalized by the topology’s total link capacity. We evaluate LMTE on B4 [17] and precompute 𝐾 = 4 shortest paths for every source–destination pair. We use Qwen 2.5 8B as LMTE’s backend LM and fine-tune LMTE’s embedding layers on 20K gravity-model traffic matrices until its accuracy exceeds 99.5% on held-out inputs. Without workload constraints, LENS finds a normalized gap of 57.51%. We then restrict the search using bounds derived from the training set: individual demands may not exceed the largest observed demand, and total demand may not exceed the maximum observed aggregate volume. Even so, LENS finds a 15.4% normalized gap. LENS exposes a gap over 25× larger than LLMCoSolver’s test-set gap. Following [36], we formulate cluster scheduling as a 0/1 knapsack problem with a capacity of 200 GPUs; each job requests between 1 and 40 GPUs and has an associated throughput. We measure the difference between the total throughput achieved by the optimal solver and LLMCoSolver. We use Qwen 2.5 8B as the backend LM and fine-tune it on 500K randomly generated instances. The largest gap on the test set is 4 throughput units. We constrain LENS’s search using the normalized nearestneighbor distance from the adversarial input to the training set so that generated instances remain close to the training data. Even under this constraint, LENS finds inputs with a gap exceeding 100 throughput units—more than 25× the largest test-set gap. LENS outperforms all black-box baselines (Fig. 3). For LMTE, LENS finds a gap that is 180% larger than that found by the strongest black-box baseline. For LLMCoSolver, LENS

4.3

LENS’s design choices

Multi-stage refinement. Figure 5 compares LENS to vanilla RL under the same policy and search budget. Multi-stage search finds 50 − 1000% larger gaps on both TE and cluster scheduling problems. Sparse rewards. LENS computes the reward once every 𝑘 steps (§3.3). Figure 6 depicts the discovered gap and the runtime for different values of 𝑘. Increasing the step size 5

Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, and Ramesh Govindan

be reused in production as a routing signal to select among a portfolio of solvers. Given an incoming instance, a selector can predict which heuristic or solver is likely to have the smallest expected gap and dispatch accordingly. This turns LENS from an offline analyzer into an online decision layer for adaptive solver orchestration. Generalizing to other agentic AI systems. Agentic AI is increasingly used in network management [38]. A promising direction is to define domain-specific performance metrics for these agents and use LENS to search for adversarial inputs that expose reliability gaps. Because LENS only needs a black-box execution interface, it can evaluate and stress-test agentic systems across many domains without requiring access to internals such as model weights or prompts.

Figure 5: LENS’s multi-stage refinement finds larger gaps than vanilla RL under the same search setup. (a) Traffic engineering, (b) Cluster scheduling with different job counts.

6

Performance analyzers. MetaOpt [29, 30], XPlain [20], and other domain-customized analyzers [3, 12, 37] require users to formally specify the target and constraints in a specific language. MetaEase [19] reduces this modeling burden by analyzing the source code. These methods remain inapplicable when the complete solver is available only through an API. In contrast, LENS only requires an executable target and a performance scoring function (Table 1). Black-box search. Random search, hill climbing [10], and simulated annealing [21] are general. However, they rely on local feedback and can be trapped in local optima. They often fail to find useful adversarial inputs even for simple heuristics (Fig. 4). Robustness of learned systems. Adversarial ML methods attack neural policies, image classifiers, and graph neural networks [15, 24, 27]. These methods target classification or policy outputs and often assume model or gradient access, rather than gaps relative to an optimum. [11] uses RL to generate adversarial inputs for RL-based network controllers, but does not address high-dimensional, constrained optimization instances. LENS adds multi-stage refinement to make RL effective in this setting. Robustness of LLMs. LLMs are increasingly used for optimization and network management [8, 18, 26, 33, 34, 38, 40, 41]. Existing evaluations report average accuracy, feasibility, or task success, while LLM red-teaming benchmarks [6, 7, 25, 28, 43] focus on unsafe generations or agent behavior. LENS instead searches for valid inputs where the LLM’s decision is substantially worse than the optimal.

Figure 6: Sparse reward frequency 𝑘 trades search quality for runtime on LMTE.

from 𝑘=1 to 𝑘=2 yields an 8% reduction in the discovered gap but provides a 1.5× speedup. For 𝑘 = 4, these numbers are 13% and 2.8×, respectively. Warm starts. We have compared the runtime of LENS with and without warm starting for both LMTE and DP (graphs omitted). For DP, where Gurobi optimization calls dominate the execution, warm-starting yields a substantial 3.1× speedup at 𝑘=1 (reducing runtime from 23.6 to 7.5 minutes). Conversely, LMTE exhibits a negligible 3% speedup because its runtime is dominated by LLM query latency rather than the optimization solver.

5

Related Work

Future Work

Broader and larger-scale domains. LENS is a black-box framework, so we can evaluate it on a broader set of problems. New domains include packet scheduling [2] and caching [9]. To measure adversarial effectiveness and scalability, we can evaluate over larger WAN topologies for TE or more diverse cluster job sets. This would likely unearth opportunities for improving LENS’s efficiency and effectiveness. Improving LLM-solver robustness. LENS currently focuses on finding high-gap instances during search. A natural extension is to generate a diverse corpus of adversarial inputs, including inputs constrained to a target workload distribution. These corpora can help increase the robustness of learned solvers through adversarial retraining and stresstest suites, with explicit tracking of the trade-off between worst-case improvement and average-case performance. From attack policy to online solver selection. The policy learned by LENS maps the given input to an edit that increases the performance gap. The same representation can

7

Conclusion

LENS is a black-box analyzer for finding adversarial inputs that widen the gap between a target LLM-based solver and an optimal baseline. It applies to heuristics, learned pipelines, 6

LLM solvers, and agentic AI systems without requiring models, source code, or gradients. For traffic engineering and cluster scheduling, LENS matches or exceeds prior analyzers on classical targets, outperforms black-box local search, and exposes large gaps in high-accuracy learned and LLM solvers that average-case benchmarks miss. As such, LENS catalyzes the exploration of performance analyzers that improve the auditability of opaque agentic systems.

https://arxiv.org/abs/2301.04205 [13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:1801.01290 [cs.LG] https://arxiv.org/abs/1801.01290 [14] Chi-Yao Hong, Srikanth Kandula, Ratul Mahajan, Ming Zhang, Vijay Gill, Mohan Nanduri, and Roger Wattenhofer. 2013. Achieving high utilization with software-driven WAN. SIGCOMM Comput. Commun. Rev. 43, 4 (Aug. 2013), 15–26. https://doi.org/10.1145/2534169.2486012 [15] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. 2017. Adversarial Attacks on Neural Network Policies. arXiv preprint arXiv:1702.02284 (2017). [16] Zangir Iklassov, Yali Du, Farkhad Akimov, and Martin Takáč. 2024. Self-Guiding Exploration for Combinatorial Problems. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 130569–130601. https://doi.org/10.52202/079017-4150 [17] Sushant Jain, Alok Kumar, Subhasree Mandal, Joon Ong, Leon Poutievski, Arjun Singh, Subbaiah Venkata, Jim Wanderer, Junlan Zhou, Min Zhu, Jon Zolla, Urs Hölzle, Stephen Stuart, and Amin Vahdat. 2013. B4: experience with a globally-deployed software defined wan. SIGCOMM Comput. Commun. Rev. 43, 4 (Aug. 2013), 3–14. https://doi.org/10.1145/2534169.2486019 [18] Xia Jiang, Yaoxin Wu, Minshuo Li, Zhiguang Cao, and Yingqian Zhang. 2025. Large Language Models as End-to-end Combinatorial Optimization Solvers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://arxiv.org/abs/2509.16865 [19] Pantea Karimi, Siva Kesava Reddy Kakarla, Pooria Namyar, Santiago Segarra, Ryan Beckett, Mohammad Alizadeh, and Behnaz Arzani. 2026. MetaEase: Heuristic Analysis from Source Code via Symbolic-Guided Optimization. USENIX Symposium on Networked Systems Design and Implementation 2026. [20] Pantea Karimi, Solal Pirelli, Siva Kesava Reddy Kakarla, Ryan Beckett, Santiago Segarra, Beibin Li, Pooria Namyar, and Behnaz Arzani. 2024. Towards Safer Heuristics with XPlain. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks. 68–76. [21] Scott Kirkpatrick, Jr. Gelatt, C. Daniel, and Mario P. Vecchi. 1983. Optimization by Simulated Annealing. Science 220, 4598 (1983), 671– 680. [22] Umesh Krishnaswamy, Rachee Singh, Nikolaj Bjørner, and Himanshu Raj. 2022. Decentralized cloud wide-area network traffic engineering with BLASTSHIELD. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA, 325–338. https://www.usenix.org/conference/nsdi22/ presentation/krishnaswamy [23] Umesh Krishnaswamy, Rachee Singh, Paul Mattes, Paul-André C. Bissonnette, Nikolaj Bjørner, Zahira Nasrin, Sonal Kothari, Prabhakar Reddy, John Abeln, Srikanth Kandula, et al. 2023. OneWAN is better than two: Unifying a split WAN architecture. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, 515–529. [24] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2018. Adversarial Examples in the Physical World. In Artificial Intelligence Safety and Security. Chapman and Hall/CRC, 99–112. [25] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Ré, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew

References [1] Firas Abuzaid, Srikanth Kandula, Behnaz Arzani, Ishai Menache, Matei Zaharia, and Peter Bailis. 2021. Contracting Wide-area Network Topologies to Solve Flow Problems Quickly. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association, 175–200. https://www.usenix.org/conference/ nsdi21/presentation/abuzaid [2] Albert Gran Alcoz, Balázs Vass, Pooria Namyar, Behnaz Arzani, Gabor Retvari, and Laurent Vanbever. 2025. Everything Matters in Programmable Packet Scheduling. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 1467–1485. https://www.usenix.org/ conference/nsdi25/presentation/alcoz [3] Venkat Arun, Mina Tahmasbi Arashloo, Ahmed Saeed, Mohammad Alizadeh, and Hari Balakrishnan. 2021. Toward Formally Verifying Congestion Control Behavior. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (SIGCOMM ’21). [4] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning 3, 1 (2011), 1–122. [5] Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press. [6] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track. [7] Huanran Chen, Yinpeng Dong, Zeming Wei, Yichi Zhang, and Jun Zhu. 2025. Towards the Worst-case Robustness of Large Language Models. arXiv preprint arXiv:2501.19040 (2025). [8] Shengkai Chen, Zhiguang Cao, Jianan Zhou, Yaoxin Wu, Senthilnath Jayavelu, Zhuoyi Lin, Xiaoli Li, and Shili Xiang. 2026. DRAGON: LLMDriven Decomposition and Reconstruction Agents for Large-Scale Combinatorial Optimization, In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026). arXiv preprint arXiv:2601.06502. [9] Edward G. Coffman and Peter J. Denning. 1973. Operating systems theory. Prentice-Hall. [10] Lawrence Davis. 1991. Bit-Climbing, Representational Bias, and Test Suite Design. In Proceedings of the 4th International Conference on Genetic Algorithms. Morgan Kaufmann, 18–23. [11] Tomer Gilad, Nathan H. Jay, Michael Shnaiderman, Brighten Godfrey, and Michael Schapira. 2019. Robustifying Network Protocols with Adversarial Examples. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks (HotNets ’19). https://doi.org/10.1145/3365609. 3365862 [12] Saksham Goel, Benjamin Mikek, Jehad Aly, Venkat Arun, Ahmed Saeed, and Aditya Akella. 2024. A Performance Verification Methodology for Resource Allocation Heuristics. arXiv:2301.04205 [cs.LO] 7

Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, and Ramesh Govindan

Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. Transactions on Machine Learning Research (2023). [26] Shengcai Liu, Caishun Chen, Xinghua Qu, Ke Tang, and Yew-Soon Ong. 2024. Large Language Models as Evolutionary Optimizers, In 2024 IEEE Congress on Evolutionary Computation (CEC). arXiv preprint arXiv:2310.19046. [27] Jiaqi Ma, Shuangrui Ding, and Qiaozhu Mei. 2020. Towards More Practical Adversarial Attacks on Graph Neural Networks. In Advances in Neural Information Processing Systems, Vol. 33. 4756–4766. [28] Mantas Mazeika, Yangsibo Deng, Maksym Andriushchenko, Andy Zou, Jingfeng Feng, Jiannan Gu, Dan Hendrycks Singh, Steven Kuwajima, Fang Liu, Bo Huang, James Zico Park, and Dan Hendrycks. 2024. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. In Proceedings of the 41st International Conference on Machine Learning. [29] Pooria Namyar, Behnaz Arzani, Ryan Beckett, Santiago Segarra, Himanshu Raj, and Srikanth Kandula. 2022. Minding the gap between fast heuristics and their optimal counterparts. In Proceedings of the 21st ACM Workshop on Hot Topics in Networks. 138–144. [30] Pooria Namyar, Behnaz Arzani, Ryan Beckett, Santiago Segarra, Himanshu Raj, Umesh Krishnaswamy, Ramesh Govindan, and Srikanth Kandula. 2024. Finding Adversarial Inputs for Heuristics using Multi-level Optimization. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 927–949. [31] Pooria Namyar, Behnaz Arzani, Srikanth Kandula, Santiago Segarra, Daniel Crankshaw, Umesh Krishnaswamy, Ramesh Govindan, and Himanshu Raj. 2024. Solving Max-Min Fair Resource Allocations Quickly on Large Graphs. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 1937–1958. https://www.usenix.org/conference/ nsdi24/presentation/namyar-solving [32] Pooria Namyar, Michael Schapira, Ramesh Govindan, Santiago Segarra, Ryan Beckett, Siva Kesava Reddy Kakarla, and Behnaz Arzani. 2024. End-to-End Performance Analysis of Learning-enabled Systems. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks (Irvine, CA, USA) (HotNets ’24). Association for Computing Machinery, New York, NY, USA, 86–94. https://doi.org/10.1145/3696348.3696875 [33] Ioannis Protogeros, Rufat Asadli, Benjamin Hoffman, and Laurent Vanbever. 2026. Benchmarking LLM-Driven Network Configuration Repair. arXiv preprint arXiv:2604.22513. [34] Pratyush Sahu, Rahul Bothra, Venkat Arun, Brighten Godfrey, Akshay Narayan, and Ahmed Saeed. 2026. Assistants, Not Architects: The Role of LLMs in Networked Systems Design. arXiv preprint arXiv:2604.25506. [35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347 [36] Suhas Jayaram Subramanya, Don Kurian Dennis, Virginia Smith, and Gregory R. Ganger. 2025. COpter: Efficient Large-Scale ResourceAllocation via Continual Optimization. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25). 322–340. https://doi.org/10.1145/3731569.3764846 [37] Mina Tahmasbi Arashloo, Ryan Beckett, and Rachit Agarwal. 2023. Formal Methods for Network Performance Analysis. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). [38] Zhaodong Wang, Samuel Lin, Guanqing Yan, Soudeh Ghorbani, Minlan Yu, Jiawei Zhou, Nathan Hu, Lopa Baruah, Sam Peters, Srikanth Kamath, Jerry Yang, and Ying Zhang. 2025. Intent-Driven

Network Management with Multi-Agent LLMs: The Confucius Framework. In Proceedings of the ACM SIGCOMM 2025 Conference (São Francisco Convent, Coimbra, Portugal) (SIGCOMM ’25). Association for Computing Machinery, New York, NY, USA, 347–362. https: //doi.org/10.1145/3718958.3750537 [39] Zhiying Xu, Francis Y. Yan, Rachee Singh, Justin T. Chiu, Alexander M. Rush, and Minlan Yu. 2023. Teal: Learning-Accelerated Optimization of WAN Traffic Engineering. In Proceedings of the ACM SIGCOMM 2023 Conference. 378–393. [40] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409 (2023). [41] Xinyu Yuan, Yan Qiao, Zonghui Wang, Meng Li, and Wenzhi Chen. 2026. LMTE: Putting the “Reasoning” into WAN Traffic Engineering with Language Models. In IEEE INFOCOM 2026. [42] Jie Zhao and Kang Hao Cheong. 2025. Structure-Aware Cooperative Ensemble Evolutionary Optimization on Combinatorial Problems with Multimodal Large Language Models. In Advances in Neural Information Processing Systems, Vol. 38. Curran Associates, Inc., 8730–8758. https://proceedings.neurips.cc/paper_files/paper/2025/ hash/0cda0892c74edef6bc73e46a4480511f-Abstract-Conference.html [43] Yajie Zhou, Jiajun Ruan, Eric S. Wang, Sadjad Fouladi, Francis Y. Yan, Kevin Hsieh, and Zaoxing Liu. 2026. NetArena: Dynamic Benchmarks for AI Agents in Network Automation. In International Conference on Learning Representations.

8

Related documents

• Blackbox Performance Analysis of LLM-based Network Control#1388914
• Provisioning the calibration window: code, score arrays and outputs (v2.0)#1108455
• Verifiable Data Readiness: A Zero-Knowledge Approach to Privacy-Preserving Dataset Auditing [Replication Package]#124839
• Verifiable Data Readiness: A Zero-Knowledge Approach to Privacy-Preserving Dataset Auditing [Replication Package]#124838
• Distributed Resilient State Estimation for Cyber-Physical Systems Against Bit Errors: A Zonotopic Set-Membership Approach#16681