arXiv CS open access

The Frontier LLM Trap in Network Automation

2026 · arxiv_cs

arXiv CS · Papers · 2026

distributed-systems internet networking protocols

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

The Frontier LLM Trap in Network Automation Minhao Jin

Sean Wang

Princeton University Princeton, NJ, USA

Aarti Gupta

Maria Apostolaki

Princeton University Princeton, NJ, USA

arXiv:2608.03080v1 [cs.NI] 4 Aug 2026

ABSTRACT

This paper asks: How can we build a network automation agent that helps operators with tasks such as configuration translation, troubleshooting, verification, and policy enforcement, while being effective, fast, controllable, and tailored to a specific network? More importantly, how can such an agent improve over time without creating a long-term dependency on frontier LLM providers? A natural starting point is to use a small, inexpensive, locally deployable LLM [3, 5, 19] as the interface for network automation, and then teach it the networking expertise it is missing. Such models can be hosted using the operator’s own infrastructure, making them an attractive option: they are fast, cost-efficient, easy to deploy close to the operator or the network, and give the operator full control over the model and its serving environment. However, determining what exactly to teach the model is no easy task. In practice, the relevant expertise is often implicit, scattered across vendor behavior, operator experience, incomplete documentation, and the quirks of a specific network. Training such a model directly is also difficult. Networking lacks large, clean datasets of specific queries paired with correct answers. Even if we could generate queries and candidate answers, correctness is hard to verify, and finding the right answer is non-trivial. More importantly, it is unclear how to reliably teach a model to avoid similar, or even identical, mistakes. Fine-tuning can be opaque and brittle [26, 28]; prompt engineering creates a context problem and is not reliable [20, 31, 39], while constrained decoding [13, 16, 17, 37, 40], which restricts the model’s token choices to what a grammar permits, can overconstrain the model (even preventing it from generating the correct answer) and make the system harder to maintain [8, 32]. Attempts in the networking domain are useful [16, 17] but require the rule grammar to align with tokenization. We argue for a different approach: instead of trying to teach everything at once, teach small models by systematically discovering and correcting their own recurring mistakes. The key idea is to build an offline teaching loop that combines multiple complementary tools: fuzzers to trigger failures in the small model, frontier LLMs to propose candidate answers and explanations, and a validation engine

Large LLMs are powerful tools for network automation, but they are expensive, slow to serve, hard to audit, poorly tailored to individual networks, and create long-term dependencies on a small number of AI providers. Existing alternatives fall short: small open-source models are cheaper but unreliable, while deterministic scripts and verification are controllable but hard to build and maintain. We propose a middle ground. An offline loop composed of fuzzing and validation discovers the recurring mistakes small models make, then resulting networking knowledge is expressed as explicit logic rules. In production, these rules guide a small model on tasks such as configuration translation, yielding automation that is cheaper, lower-latency, auditable, and easier to adapt to a specific network. More critically, network knowledge and operational experience stay and are accumulated where they belong, the network itself, not a rented service.

INTRODUCTION

Networking tasks are both difficult and diverse, spanning configuration, troubleshooting, verification, performance tuning, and policy enforcement. Traditional automation tools struggle to keep up: they often require substantial manual engineering, must be rewritten for each new task or network, and are difficult for operators to adopt, maintain, and extend. Frontier LLMs offer a compelling alternative [1, 4, 6, 7], promising enormous speed-ups for many of these tasks, but have significant limitations in practice. First, they are expensive, often impossible to serve, making them especially hard to justify for smaller networks. Second, they are often slow. We find that GPT-5.5 [30] can require up to four attempts to produce a correct configuration translation, with each attempt taking as long as 2.5 minutes. Finally, they also create a fragile operational dependency on a small number of AI providers: providers may change prices, restrict usage, degrade access, or disappear altogether. As a result, industry is increasingly looking beyond frontier-model-only solutions toward cheaper, more controllable, and more deployable forms of network automation [23]. 1

Minhao Jin, Sean Wang, Aarti Gupta, and Maria Apostolaki

composed of emulators to test behavior in realistic network settings and/or verifiers to check semantic correctness when possible. Each component is insufficient on its own. Frontier LLMs may hallucinate or propose answers that are incorrect, unnecessary, or inappropriate for a particular network; verifiers can be slow or incomplete; emulators are limited by their fidelity; and fuzzers only reveal the failures they can generate. But together, they can continually expose mistakes, infer missing networking principles, and validate candidate corrections while building on an internally owned model. This approach is not traditional model distillation [18, 21, 29, 38, 41]. We do not blindly force the small model to imitate a frontier LLM’s outputs. In fact, we find that this strategy would be suboptimal for networking. First, many networking tasks admit multiple correct answers: for example, several configurations may implement the same intended behavior, and over-constraining the model to match one answer can unnecessarily reduce flexibility. Second, correctness is often contextual or network-specific: a solution that is valid in general may be irrelevant or wrong in a network where a particular protocol, feature, or policy is disabled. Third, frontier models themselves are not reliable ground truth; they can hallucinate, omit important constraints, or produce incorrect configurations. We have experienced all of these limitations firsthand. GPT-5.5 has < 5% one-shot accuracy for configuration translation, and when it finds a correct answer, it often differs from what a human would write, or has consequences that are not yet visible. Thus, we use frontier LLMs not as oracles to imitate, but as hypothesis generators whose outputs must be tested, validated, and converted into explicit rules before they guide the small model. We then use traditional tools the networking community has developed and relied on for years—verifiers, emulators, and tests—to separate useful hypotheses from plausible but incorrect ones. The validated knowledge is captured as explicit logic rules and applied at inference time, rather than buried inside model weights. In this way, frontier LLMs act as capable but fallible temporary teachers, not permanent dependencies or sources of authority. The resulting loop incrementally builds a network-specific layer of auditable, editable rules that improves a small model over time while keeping the system inexpensive, controllable, and, more importantly, internally owned. This paper explores the agenda above through the lens of a particular networking task: configuration translation. We ask whether small models can be made practical not by scaling them up, but by surrounding them with the right symbolic knowledge. More broadly, we argue that the next generation of network automation should not be defined by how often we call frontier models, but by how effectively we can extract, validate, and reuse their expertise.

NETWORK AUTOMATION REQUIREMENTS & CHALLENGES

To better understand the problem, we put ourselves in the shoes of a network engineer tasked with translating configurations from one vendor or configuration language to another. We proceed by trying two approaches. Approach 1: Frontier LLM with Per-Query Validation. We first ask the frontier LLM to translate each configuration, then manually inspect the result and check it using existing tools. At first glance, the generated translations appear plausible, but closer inspection reveals unnecessary configuration statements that preserve behavior but may complicate future network management. For example, when translating a Cisco OSPF routing configuration to Junos we found that GPT-5.5 often added static routes in the translated version even though they did not exist in the original. Using an emulator, we evaluate the correctness of translations, examining router behaviors such as OSPF attributes, LSA metrics, inter-area routes, and static routes. We observe that the frontier model alone is unreliable: many responses are incorrect, exhibiting similar mistakes, others are correct but contain unnecessary statements for some networks, and many tasks admit multiple valid translations. In our experiments, 18.6% of the generated Junos configs fail to even establish connectivity with emulated test routers. Hence, we re-prompt GPT-5.5 with the failure context and ask it to propose a minimal correction. While full validation is achieved within 1.79 repair iterations on average, some corrections take an unpredictable amount of time, often resulting in an ad hoc loop of inspecting the output, modifying the prompt, rerunning the model, and checking again. Furthermore, as network size increases, token costs and serving latency increase, reducing productivity gains. Worse, the workflow now depends on continued access to a frontier model provider whose pricing, rate limits, policies, or availability may change without the network operator’s control. Approach 2: Small Model with Logged Experience. A natural response is to preserve the frontier model’s outputs and pass them on to a small model. We log useful prompts, answers, validation failures, corrections, and examples from previous translations. In effect, we build an internal knowledge base of networking principles and “model operating procedures” for configuration translation. However, this approach also breaks down. The accumulated context quickly becomes too large to include in every prompt, and deciding which pieces of past knowledge are relevant to a new configuration is itself difficult. Even when the right hints are included, a small model may fail to follow them reliably. Indeed, the LLM community has reported significant degradation of instruction following for LLMs as the context 2

The Frontier LLM Trap in Network Automation

Offline training

Production Specification

Query

Query Generator

Local LLM

Knowledge Extraction

Wrong Reply

Rule Mining

Validation

Teacher Feedback

Cheat sheet

Knowledge Enforcement

Local LLM Serving

clients

Rule B Service ...

Auditable rules

Corrected Reply

Rule A Service

1 Figure 1: Offline, ○the query generator produces test queries, the local LLM replies, and validation checks each 2 3 against the specification; ○once failed, the teacher and validation engine propose a fix; ○the resulting 〈query, wrong 4 per-task rule reply, corrected reply〉 triplet feeds Rule Mining, which extracts auditable rules. In production, ○a Mining service applies these rules between the LLMRule serving engine and end users, with all tasks sharing one serving engine. Original

Symbolic Representation

Rule Template

interface Ethernet0/1 ip address 10.0.0.1 255.255.255.0 no shutdown

interface [i1:IFACE] interface [i1] / ip address [addr:IPV4] [mask:IPV4] interface [i1] / no shutdown

Source-Conditioned Rules Target-Conditioned Rules Target-Only Rules

... interface ge-0/0/1.0; ...

... interface [i2:IFACE] ...

... interface ge-0/0/1.0 { priority 1; } ...

... interface [i2:IFACE] interface [i2] / priority [p:NUM] ...

grows [20, 31, 39]. Our experiment also shows that conveystream-like service that enforces them. This section explains ing knowledge through prompts lacks guarantees and is 6% why each stage is hard and what insights make it tractable Wrong rulesgeneral ... Reply less accurate than strict enforcement even with just 15 short Miningand for multiple networking tasks. natural-language rules. Teaching by Failure, Not by Specification. Teaching a Corrected A better approach: Our experiment shows that we need model network automation from scratch is difficult for the Reply a way to capture the productivity benefits of LLMs withsame reason that fully deterministic solutions are difficult: out turning frontier models into a permanent production the required knowledge is implicit, incomplete, and potendependency. A practical network automation agent should tially unbounded. There is no comprehensive specification of satisfy five requirements. First, it should be effective: it must how every network task should be solved, nor a large dataset produce outputs that are mostly correct and useful for real of inputs paired with correct outputs to train on or analyze. networking tasks, not merely plausible. Second, it should be Even human network engineers do not begin with complete network-specific: its behavior should adapt to the protocols, knowledge; they start with general principles and learn the policies, vendors, and operational conventions of the target details of a particular network through experience. We adopt network. Third, it should be low-latency and inexpensive: the same philosophy for small models. Rather than trying to operators should be able to use it interactively, and smaller teach the model everything, we start with an already usable networks should be able to deploy it without paying frontiersmall model and focus on correcting its observed mistakes. model serving costs on every query. Fourth, it should be While intuitive, this approach is hindered by three challenges: controllable and auditable: when the agent makes a recom(i) how do we expose mistakes, given the space of possible mendation, operators should be able to inspect the reasoning, user queries? (ii) how do we know when a small model is acidentify the rules being applied, and override or update them tually wrong? and if it is, (iii) how do we fix it without ground when needed. Finally, it should be maintainable: as the nettruth? To address these challenges, we constrain the search work changes, improving the agent should not require rewritspace to a single task, e.g., configuration translation, rather ing brittle scripts, manually curating ever-growing prompts, than trying to build a universal networking model. Within or repeatedly fine-tuning opaque model weights. In short, we a task, the space of possible queries is structured and can be want an agent that feels like an LLM to the operator, behaves navigated systematically using a fuzzer. While fuzzing can like a network-specific tool, and can improve over time withexpose suspicious or inconsistent model behavior, it does not out locking the organization into a frontier model provider. by itself tell us whether the output is actually wrong, why it is wrong, or what the correct behavior should be. Our insight is that networking already has a rich ecosystem of reliable 3 KNOWLEDGE EXTRACTION tools that can provide this missing signal. Verifiers can check & ENFORCEMENT ARCHITECTURE syntactic and some semantic properties [9, 14, 36], while Our experience with translating configurations motivated a emulators can test behavior under realistic conditions [2]. key change: we need to separate the validation from the inWhen the small model’s output is found to be wrong, we rely ference path. With this idea in mind, we propose NetMentor on a frontier LLM to propose candidate corrections using shown in Figure 1. NetMentor splits the expensive per-query feedback from the validation step. Observe here that we do validation into: (i) an offline loop that identifies concise, not treat the frontier model as an oracle; its suggestions are reusable, and auditable task-specific rules; and (ii) an online Query

IF NOT in src # Source Condition interface [i1:IFACE] / ip ospf priority [n:NUM] AND in tgt # Target Condition interface [i2:IFACE] AND NOT in tgt interface [i2:IFACE] / priority [n:NUM] AND maps_to(i1, i2) # Predicate THEN # Action inject(interface [i2:IFACE] / priority [n:NUM])

IF interface [i1:IFACE] / ip ospf priority [n:NUM] NOT in src AND interface [i2:IFACE] in tgt AND interface [i2:IFACE] / priority [n:NUM] NOT in tgt AND maps_to(i1, i2) THEN inject(interface [i2:IFACE] / priority [n:NUM])

Original

... interface Ethernet0/1 ip address 10.0.0.1 255.255.255.0 no shutdown ...

1 2 3 4 5

Source_cond = { interface [i1:IFACE] / ip ospf priority [n:NUM] } Target_cond = { interface [i2:IFACE], ¬(interface [i2:IFACE] / priority[n:NUM]) } Pred = { maps_to(i1, i2)} action = inject( interface [i2:IFACE] / priority [n:NUM])

hostname R1 interface Ethernet0/1 ip address 10.0.0.1 255.255.255.0 no shutdown ...

Symbolic

1 2 3 4 5

... interface [i1:IFACE] interface [i1] / ip address [addr:IPV4][mask:IPV4] interface [i1] / no shutdown ...

hostname R1 interface [i1:IFACE] interface [i1] / ip address [addr:IPV4][mask:IPV4] interface [i1] / no shutdown ...

Minhao Jin, Sean Wang, Aarti Gupta, and Maria Apostolaki Original

hypotheses that are passed through the same validation loop to decide whether they capture real networking knowledge. While frontier LLMs are not oracles, we find they are great at repairing their own mistakes with guidance. This behavior might mean that they have the knowledge they need (e.g., have been trained on the appropriate documents) but are unlikely to invoke it directly at inference without guidance. Capturing Knowledge through Logical Rules. By repeatedly asking a frontier LLM to correct outputs based on validator feedback, we obtain a corpus of errors and their corresponding corrections, which we call the rule mining dataset. We use this corpus to extract logical rules that capture the networking knowledge needed for the task at hand. Specifically, the rules describe possible repair actions, as well as the conditions under which these actions should be applied. A central challenge is that rules extracted naively from this dataset will fail to generalize to new erroneous outputs, such as network configurations with similar statements, but with different values configured for the parameters. To keep the set of rules manageable and generalizable, our insight is to use a symbolic representation for each learned rule, such that it can apply in many instances with possibly different values for the symbolic parameters. Rules also need to be auditable and efficiently verifiable, i.e., they should be easy to understand and should provide wide coverage over common cases. Our second insight is to propose three domain-specific rule templates based on observed LLM mistakes. The symbolic representations and rule templates are described in §4.1. We learn the rules manually in our current prototype, and plan to automate this using LLMs (guided by synthesis) in future work. Enforcing Knowledge with a Streaming Service. Enforcement must be lightweight, or local deployment forfeits its cost advantage; reliable, in that no returned response violates the active rule set; and minimally invasive, treating the model and serving stack as black boxes. Prompting fails on reliability: instructions are advice, and following degrades as context grows [20, 31, 39]. Constrained decoding [13, 16, 17], which masks each next-token distribution to what a grammar permits, is reliable by construction but puts rule logic inside the decoding loop, and cannot express rules that are not token-local, i.e., rules whose validity depends on tokens generated much later. In Cisco-to-Junos translation, for example, the small model (e.g., Qwen2.5-Coder-32B-Instruct [19]) may generate an OSPF export statement before generating the corresponding policy-options block. A rule such as “drop ’export’ if no ’policy-options’ exists” cannot be decided when the token ‘export‘ first appears, because the required block may be generated later. Thus, token-level decoding would either reject valid intermediate outputs too early or require complex lookahead and global state. We therefore

Wrong Reply

Corrected Reply

1 ... 2 interface ge-0/0/1.0; 3 ...

1 ... 2 interface ge-0/0/1.0 { 3 priority 1; 4 } 5 ...

Query 1 interface Ethernet0/1 2 ip address 10.0.0.1 255.255.255.0 3 no shutdown

Symbolic Representation

Rule Template

1 interface [i1:IFACE] 2 interface [i1] / ip address [addr:IPV4] [mask:IPV4] 3 interface [i1] / no shutdown

Source-Conditioned Rules Target-Conditioned Rules Target-Only Rules

1 ... 2 interface [i2:IFACE] 3 ...

Mining rules ...

1 ... 2 interface [i2:IFACE] 3 interface [i2] / priority [p:NUM] 4 ...

Figure 2: An example of rule mining for configuration translation. Rule Mining converts the 〈query, wrong reply, corrected reply〉 triplet to symbolic form, matches it to a rule template, and captures it as a logical repair rule. enforce rules with a streaming chunk-repair proxy that operates over completed configuration blocks rather than individual tokens, sidestepping token-locality entirely (see §4.2).

4 INITIAL DESIGN 4.1 Rules for Configuration Translation We start by describing the symbolic representations of configurations and rules, and then present three useful templates along with an illustration on the example in Figure 2. Symbolic Grammar for Configurations. To symbolically represent configurations, we adopt a similar approach to previous work [10, 34], which uses abstract patterns to identify similar lines. The full grammar for symbolic configurations and rules is shown in Figure 3. Configurations, ⟨cfg⟩, are sequences of lines with contexts (context-lines), which consist of statements and typed parameters. Contexts describe the position of a line in the configuration’s hierarchy. To illustrate, consider the query in Figure 2, which contains part of a Cisco configuration. In the symbolic representation, the interface name, IP address, and mask are extracted as parameters 𝑖 1, addr, and mask, respectively. Also, lines 2 and 3 are associated with the context "interface [i1]", indicating that they configure the interface 𝑖 1 in the hierarchy. Rules for Repair. A rule consists of three parts: (1) match conditions on context-lines, which capture when a rule is applied, (2) parameter predicates, which capture the relationships between symbolic variables within or across lines and contexts, and (3) a repair action, which describes how to fix an error in the target configuration. 4

The Frontier LLM Trap in Network Automation

replacing a context-line, and may depend on the matched instances of a match condition. Useful templates for rule application. To ensure that rules are interpretable and auditable, we define domain-specific rule templates that target common errors observed during LLM configuration translation. For these templates, match conditions are separated into source conditions, 𝑆, and target conditions, 𝑇 . Intuitively, rule templates identify the instances of lines in a target configuration that need repair. In the first template, called source-conditioned rules, we identify all lines (𝑆𝐿𝑖 ) in the source configuration that match with the source conditions 𝑆, and then look for the existence of lines (𝑇 𝐿 𝑗 ) in the target configuration that match with 𝑇 , and where the predicates 𝑃 hold over 𝑆𝐿𝑖 and 𝑇 𝐿 𝑗 . Targetconditioned rules are defined analogously, where the roles of the source and target conditions are switched. We also define a third template for target-only rules, which contain only target conditions. Rule Templates in Action. Figure 2 shows a common misconfiguration example caused by differing default OSPF priority values between Cisco and Junos [12, 22]. Using the source-conditioned rule template, we can define a rule to find and fix this mistake:

Configurations ⟨cfg⟩ F ⟨ctx_line⟩ + ⟨ctx_line⟩ F ⟨ctx⟩⟨line⟩ ⟨line⟩ F (⟨stmt⟩|⟨param⟩) + ⟨ctx⟩ F emp|(⟨line⟩/) + ⟨stmt⟩ F 𝑠 ∈ 𝑆 ⟨param⟩ F 𝑣 : ⟨type⟩ ⟨type⟩ F NUM|IPV4|PFX4|IFACE|AREA_ID Rules Conditions ⟨C⟩ F ⟨ctx_line⟩|¬⟨ctx_line⟩ Predicates ⟨P⟩ F true|false|𝑣 1 =𝑣 2 |𝑅 |¬⟨P⟩|⟨P⟩∧⟨P⟩|⟨P⟩∨⟨P⟩ Actions ⟨A⟩ F inject(⟨ctx_line⟩)|drop(⟨ctx_line⟩) |replace(⟨ctx_line⟩,⟨ctx_line⟩)

Figure 3: Grammar for Symbolic Representation of Configurations and Rules

𝑆 ={interface [𝑖 1 : IFACE], ¬(interface 𝑖 1 /ip ospf priority [𝑝 : NUM])} 𝑇 ={interface [𝑖 2 : IFACE], ¬(interface 𝑖 2 /priority [𝑝 : NUM])} 𝑃 ={maps_to(𝑖 1,𝑖 2 ),true} 𝑎 =inject(interface 𝑖 2 /priority 1)

Match Conditions. We say that two symbolic lines 𝐿1 and 𝐿2 match, denoted match(𝐿1,𝐿2 ), if they describe the same pattern of statements and parameters. For instance, the line "interface [𝑖 1 : IFACE]" from Figure 2 can only match other lines of the form "interface [𝑥 : IFACE]," where 𝑥 is any variable name. To find matches for similar lines in specific contexts, we extend matching to context-lines, and we say that match((𝐶 1, 𝐿1 ), (𝐶 2, 𝐿2 )) holds if match(𝐿1, 𝐿2 ) and corresponding lines in 𝐶 1 and 𝐶 2 match. For a given context-line (𝐶,𝐿), a match condition can be positive, (𝐶,𝐿), or negative, ¬(𝐶,𝐿). For a configuration, cfg, we say that match(cfg,(𝐶,𝐿)) holds if there exists some line (𝐶 ′,𝐿 ′ ) ∈ cfg such that match((𝐶 ′, 𝐿 ′ ), (𝐶, 𝐿)), while match(cfg, ¬(𝐶, 𝐿)) holds if no such (𝐶 ′,𝐿 ′ ) exists. For positive match conditions, parameter predicates are applied to instances of the matched context-lines, whereas negative match conditions identify omitted parameters/statements. Parameter Predicates. Once the matching (and missing) instances of context-lines have been identified, the relationships among the parameters of the matched lines that must hold or need to be changed are expressed using parameter predicates. The set of predicates, ⟨P⟩, includes the standard equality and logical connectives, as well as domainspecific atomic predicates over configuration parameters. These domain-specific predicates are needed for expressing fine-grained properties, possibly relating lines in the source and target configurations. Repair Actions. The fixes for errors are expressed as repair actions 𝑎 ∈ ⟨A⟩, which can be either injecting, dropping, or

If the source defines an interface 𝑖 1 without an explicit priority and the target defines a matching 𝑖 2 (via maps_to predicate) that also omits priority, the rule injects "priority 1" into interface 𝑖 2 .

4.2

Rule enforcement

We enforce rules with a streaming chunk-repair mechanism. For each network task, a lightweight Rule Service acts as a proxy between the user and the LLM serving platform: it forwards the user’s request, monitors the streamed output, and returns a rule-compliant response. The service inspects syntactically meaningful chunks, such as completed configuration blocks, as they stream and evaluates each against the rule set. On a violation, it interrupts the stream, applies the corresponding repair action, appends the repaired prefix to the original prompt, and resends the request so the LLM continues from the corrected state. This keeps enforcement minimally invasive: the LLM and its serving stack are unchanged, and the compact rule representation guides correction without expensive token-level search. Because each retry reuses the original prompt plus the accepted prefix, serving engines with prefix/KV-cache 5

Minhao Jin, Sean Wang, Aarti Gupta, and Maria Apostolaki

With NetMentor, the small LLM achieves 28× and 20× accuracy boost against the vanilla small model and frontier LLM respectively. Both vanilla Qwen2.5-Coder-32BInstruct and GPT-5.5 achieve low pass rates (<5%) for the validator (Figure 4a). NetMentor raises the accuracy of Qwen2.5Coder-32B-Instruct to 75.2% by enforcing mined rules and 70.8% by injecting rules into the prompt. Thus, LLM errors are not random but recurring and systematically correctable.

1.0

100

75.2% 70.8% (85/113) (80/113)

1.00

0.75

0.5

CDF

Accuracy (%)

lines. These rules are concise and easy to audit, yet capture enough recurring structure to guide translation effectively.

Qwen2.5-Coder-32B-Instruct (NetMentor w. Prompt) Qwen2.5-Coder-32B-Instruct (NetMentor w. Enforcement)

GPT-5.5 Qwen2.5-Coder-32B-Instruct (Vanilla)

0.50 0.25

3.5% (4/113)

2.7% (3/113)

LLM

(a) Accuracy

0.00

0.0

Latency (s)

50 100 150 Latency (s)

(b) Latency

Figure 4: (a) NetMentor raises Qwen’s accuracy by >20× over vanilla Qwen and GPT-5.5, and enforcement beats prompt injection by 6%. (b) NetMentor cuts median latency 81% below GPT-5.5 and shrinks the vanilla model’s latency variance by two orders of magnitude.

NetMentor’s rule enforcement introduces minimal endto-end latency overhead. Vanilla Qwen exhibits a pronounced long tail because for some queries, it repeatedly generates duplicated Junos configuration blocks until reaching the timeout. NetMentor prevents these pathological generations, reducing latency variability by approximately two orders of magnitude while increasing the median latency by only 30% relative to the vanilla model. NetMentor achieves 81% lower median latency while delivering substantially higher accuracy than GPT-5.5. Overall, NetMentor substantially improves correctness and latency predictability while introducing only modest overhead.

support can reuse unchanged prefix blocks and cut regeneration cost [25].

INITIAL PROMISE

We use NetMentor to discover common errors made by a small LLM, then guide the model to avoid them at inference. This raises accuracy by 28× with minimal end-to-end latency overhead.

5.1

Setup

Equivalence testing as a first verification step. In NetMentor, we manually specify the properties for the emulator to test, which puts a significant burden on the user. A broader research question is how to automatically identify the taskrelevant behaviors that determine whether a solution is correct, or whether two versions of behavioral specification (e.g., a high-level intent and its implementation) are equivalent. This also raises a coverage question: which behaviors and corner cases have been tested, and what guarantees can we provide for solutions that pass? Generalizing knowledge-enforcement representations. Our current rule templates generalize corrections across related examples, but the templates and rules are manually constructed through inspection and intuition. An important research direction is to automatically infer symbolic patterns, synthesize candidate rules, and evaluate their correctness and generality. One must also investigate whether rules (hence template design) should align with model tokens or remain expressed over higher-level networking objects. Smarter Fuzzer. Rather than exploring the configuration space uniformly, a smarter fuzzer should target inputs that exploit the model’s learned biases. Vision models, for instance, learn spurious associations: because most pictures of tables include chairs, a model may wrongly infer chairs in a chairs-free scene [27, 33, 35]. Identifying such biases enables targeted guidance. Similarly, because configurations often

We first borrow the fuzzing approach of Metha [11] focusing on static routing and OSPF, producing independent rulemining and test sets of ≈ 50 and ≈ 110 configs, respectively. To check correctness, we build a ContainerLab-based validator [2]. For each Cisco–Junos pair, it constructs parallel star topologies with identical FRRouting [15] peer routers derived from the Cisco configuration, waits for OSPF adjacencies to converge, and compares the observed routing behavior. We serve Qwen2.5-Coder-32B-Instruct [19] as a small model on a single Nvidia H200 GPU instance via vLLM [24] and use GPT-5.5 to correct failed translations, which succeeded for all configurations in the rule-mining set. We then manually mine rules from 〈query, wrong reply, corrected reply〉 triplets using the templates of § 4.1, yielding 16 rules covering OSPF attributes, interface and area configuration, static routing, and Junos syntax. By enforcing these rules, ≈ 85% of rule-mining configurations pass validation.

5.2

FUTURE AGENDA

Results

A few concise rules in NetMentor can correct many LLM errors. We mine 16 rules to correct many recurring translation errors: 11 source-conditioned, 3 target-conditioned, and 2 target-only, averaging 1.5 source and target conditions, 2.9 parameters, and 0.9 predicates each. We define 6 relational domain-specific predicates, relating source lines to target 6

The Frontier LLM Trap in Network Automation

omit default values, a model may learn that omitted parameters can remain omitted across vendors—even when Cisco and Junos assign different OSPF priorities. Detecting such recurring assumptions could guide the fuzzer toward defaultsensitive and rarely co-occurring features, exposing failures faster and reducing the cost of teaching the small model.

(Oakland, CA) (NSDI’15). USENIX Association, USA, 469–483. [15] FRRouting Project. [n. d.]. FRRouting – IP Routing Protocol Suite for Physical and Virtual Routers. https://frrouting.org/. [16] Hongyu Hè and Maria Apostolaki. 2025. Just-in-Time Logic Enforcement: A new paradigm of combining statistical and symbolic reasoning for network management. In Proceedings of the 24th ACM Workshop on Hot Topics in Networks (UMD Campus, College Park, MD, USA) (HotNets ’25). Association for Computing Machinery, New York, NY, USA, 184–192. https://doi.org/10.1145/3772356.3772406 [17] Hongyu Hè, Minhao Jin, and Maria Apostolaki. 2025. Making Logic a First-Class Citizen in Generative ML for Networking. arXiv preprint arXiv:2506.23964 (2025). [18] Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023. 8003–8017. [19] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-Coder Technical Report. arXiv preprint arXiv:2409.12186 (2024). [20] Daniel Jaroslawicz, Brendan Whiting, Parth Shah, and Karime Maamari. 2025. How Many Instructions Can LLMs Follow at Once? arXiv preprint arXiv:2507.11538 (2025). [21] Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of proprietary large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 3134–3154. [22] Juniper Networks. [n. d.]. Junos OS Documentation. h t t p s : //www.juniper.net/documentation/product/us/en/junos-os/. [23] Katherine Li. 2026. Microsoft’s Satya Nadella says every company should build its own AI model. https://www.businessinsider.com/satyanadella-said-every-company-should-build-own-ai-model-2026-6 [24] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. https://doi.org/10.1145/3600006.3613165 [25] Not Lain. [n. d.]. KV Caching Explained: Optimizing Transformer Inference Efficiency. https://huggingface.co/blog/not-lain/kv-caching. [26] Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. 2024. Revisiting catastrophic forgetting in large language model tuning. In Findings of the association for computational linguistics: EMNLP 2024. 4297–4308. [27] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 292–305. https://doi.org/10.18653/v1/2023.emnlp-main.20 [28] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing (2025). [29] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707 (2023). [30] OpenAI. 2026. GPT-5.5 System Card. https://openai.com/index/gpt5-5-system-card/. [31] Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. 2025. Agentif: Benchmarking instruction

CONCLUSION

Small open-source models are cheap but unreliable, while frontier models bring cost, latency, and fragile dependencies. We bridge this gap with a framework that discovers common mistakes, generalizes them into auditable logic rules, and uses these rules to guide a small model at inference time. Preliminary results motivate an agenda for automating and generalizing verification, rule-mining, and data generation.

REFERENCES [1] [n. d.]. Claude. https://claude.ai/. https://github.com/srl[2] [n. d.]. ContainerLab Github repo. labs/containerlab [3] [n. d.]. Gemma. https://deepmind.google/models/gemma/. [4] [n. d.]. Germini. https://gemini.google.com/?hl=ro. [5] [n. d.]. Llama. https://ollama.com/library/llama3.1. [6] [n. d.]. OpenAI. https://openai.com/. [7] [n. d.]. XAI. https://x.ai/. [8] Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. 2025. CRANE: Reasoning with constrained LLM generation. arXiv preprint arXiv:2502.09061 (2025). [9] Ryan Beckett, Aarti Gupta, Ratul Mahajan, and David Walker. 2017. A General Approach to Network Configuration Verification. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (Los Angeles, CA, USA) (SIGCOMM ’17). Association for Computing Machinery, New York, NY, USA, 155–168. https://doi.org/10.1145/3098822.3098834 [10] Ryan Beckett, Francis Y. Yan, Raghunadha Reddy Pocha, Vineesh V. Raj, Ayyub Shaik, and Siva Kesava Reddy Kakarla. 2026. Concord: Learning Network Configuration Contracts. In Proceedings of the 21st European Conference on Computer Systems (McEwan Hall/The University of Edinburgh, Edinburgh, Scotland UK) (EUROSYS ’26). Association for Computing Machinery, New York, NY, USA, 801–818. https://doi.org/10.1145/3767295.3769338 [11] Rudiger Birkner, Tobias Brodmann, Petar Tsankov, Laurent Vanbever, and Martin Vechev. 2021. Metha: Network Verifiers Need To Be Correct Too!. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association, 99–113. https://www.usenix.org/conference/nsdi21/presentation/birkner [12] Cisco Systems. [n. d.]. Cisco IOS Configuration Fundamentals Command Reference. https://www.cisco.com/c/en/us/td/docs/iosxml/ios/fundamentals/command/cf_command_ref.html. [13] Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. 2025. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. In Eighth Conference on Machine Learning and Systems. https://openreview.net/forum?id=rjQfX0YgDl [14] Ari Fogel, Stanley Fung, Luis Pedrosa, Meg Walraed-Sullivan, Ramesh Govindan, Ratul Mahajan, and Todd Millstein. 2015. A general approach to network configuration analysis. In Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation 7

Minhao Jin, Sean Wang, Aarti Gupta, and Maria Apostolaki following of large language models in agentic scenarios. arXiv preprint arXiv:2505.16944 (2025). [32] Avinash Reddy, Thayne T Walker, James S Ide, and Amrit Singh Bedi. 2026. Draft-conditioned constrained decoding for structured generation in llms. arXiv preprint arXiv:2603.03305 (2026). [33] Amir Rosenfeld, Richard Zemel, and John K. Tsotsos. 2018. The Elephant in the Room. arXiv:1808.03305 [cs.CV] https://arxiv.org/abs/1808.03305 [34] Mark Santolucito, Ennan Zhai, Rahul Dhodapkar, Aaron Shim, and Ruzica Piskac. 2017. Synthesizing configuration file specifications with association rule learning. Proc. ACM Program. Lang. 1, OOPSLA, Article 64 (Oct. 2017), 20 pages. https://doi.org/10.1145/3133888 [35] Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. 2020. Don’t Judge an Object by Its Context: Learning to Overcome Contextual Bias. arXiv:2001.03152 [cs.CV] https://arxiv.org/abs/2001.03152 [36] Alan Tang, Siva Kesava Reddy Kakarla, Ryan Beckett, Ennan Zhai, Matt Brown, Todd Millstein, Yuval Tamir, and George Varghese. 2021. Campion: debugging router configuration differences. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (Virtual Event, USA) (SIGCOMM ’21). Association for Computing Machinery, New York, NY, USA, 748–761. https://doi.org/10.1145/3452296.3472925

[37] Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2024. SynCode: LLM generation with grammar augmentation. Transactions on Machine Learning Research (2024). [38] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers). 13484–13508. [39] Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu, and Wei Zhang. 2025. Lifbench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 16445–16468. [40] Kexun Zhang, Hongqiao Chen, Lei Li, and William Wang. 2023. Don’t Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding. arXiv preprint arXiv:2310.07075 (2023). [41] Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024. Universalner: Targeted distillation from large language models for open named entity recognition. In International conference on learning representations, Vol. 2024. 12276–12294.

The Frontier LLM Trap in Network Automation

Analyze & explore related research in Conceptio

Related documents