arXiv CSopen access

Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety

1

Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety Muhammad Bilal , Senior Member, IEEE, Jon Crowcroft , Fellow, IEEE, Ruizhi Wang, Xiaolong Xu, Senior Member, IEEE, and Schahram Dustdar, , Fellow, IEEE

arXiv:2605.12729v1 [cs.NI] 12 May 2026

Abstract Large language models are increasingly being used to support network operations (NetOps) and artificial intelligence for IT operations (AIOps), including incident investigation, root-cause analysis, configuration synthesis, and limited self-healing. In both NetOps and AIOps, this shift is changing how tasks are managed. Agent-based operations work as workflows, from gathering evidence to taking action, following permissions, policies, and checks, and providing rollback options when necessary. This is crucial because operational decisions can have instant impacts. It depends on mediated evidence, controlled change, and accountable recovery. To make the argument concrete, we organise the relevant literature around the hierarchy of autonomy, tool scope, evidence traces, and assurance contracts. These contracts define what an agent may observe, propose, and execute. They also define the checks that must pass before any action is allowed. A consistent pattern appears across work on telemetry query recommendation, diagnosis, root-cause analysis, configuration synthesis, change planning, and limited self-healing. Operational reliability does not come chiefly from the model itself. It depends on the machinery around the model: typed tool interfaces, provenance- and freshness-aware retrieval, explicit budgets and stopping rules, least- privilege access, and verification gates at the write boundary that the agent cannot bypass. We also argue that evaluation should go beyond static question answering. Agentic NetOps and AIOps systems require workflow-centred evaluation, including trace quality, bounded tool use, safe proposal generation, replay in sandboxed environments, and canary trials with rollback-aware scoring. Without these measures, a system may appear robust in demonstration yet remain too fragile when real infrastructure can be changed. Finally, we examine security, privacy, and governance risks that become acute when agents sit close to operational control surfaces. These include prompt injection through operational artefacts, retrieval poisoning, telemetry integrity attacks, excessive agency, and weak auditability. Taken together, the survey concludes that progress in intelligent NetOps and AIOps will depend on treating autonomy as a constrained operational control problem, whose outputs must be reliable, auditable, and securely deployable. Index Terms Large language models, LLM agents, agentic AI, network operations, NetOps, artificial intelligence for IT operations, AIOps, autonomous networks, tool-augmented agents, root-cause analysis, closed-loop control, safety, reliability.

I. I NTRODUCTION Modern networks and cloud services are operated through a dense layer of telemetry streams, tickets, runbooks, dashboards, and configuration artefacts. Over the last decade, this layer has grown faster than the teams that maintain it, largely because service graphs, deployment pipelines, and network policies now change continuously. In parallel, operators have accumulated powerful tools for reasoning about configuration and reachability (for example, offline analysis of routing and policy intent, and data-plane invariant checking) [1], [2], [3]. What is new is not that we automate operations, but that large language models (LLMs) can sit across this entire surface and turn fragmented operational work into a single loop that reads, reasons, and acts. M. Bilal is with School of Computing and Communications Lancaster University, United Kingdom. J. Crowcroft is with the Department of Computer Science and Technology, University of Cambridge, United Kingdom. R. Wang and X. Xu are with the chool of Software, Nanjing University of Information Science and Technology, China. Schahram Dustdar is with the Distributed Systems Group of TU Wien, Austria and ICREA, Barcelona, Spain. Corresponding author: Muhammad Bilal (e-mail: [email protected])

2

This operational burden is being intensified by three related trends: the breadth of the operational tool surface (more systems to query and more sources of truth), the velocity of change (deployments, policy edits, and topology churn), and the volume of interruption (alerts and incident handovers). In the field of SRE (Service-Oriented Responsibility), organizations aim to minimise detection and recovery times amidst increasing change frequency, making evidence collection and change security critical constraints [4], [5]. The main advantage of LLMs lies in their ability to handle both procedural and linguistic tasks, both of which can be aided by tools. This perspective places agentic NetOps and AIOps within the context of autonomic computing. Kephart and Chess described autonomous systems as self-managing systems focused on configuration, remediation, and protection. LLM-based operational agents revisit this goal with modern tools, including natural-language interfaces, planning, and code generation. However, these tools also introduce new failure types, such as incorrect diagnoses, outdated evidence, unsafe tool use, prompt manipulation, and unrestrained execution [6]. The LLMs transform the user interface by introducing natural language as the control layer for state queries and tool invocations. In early deployments, these models were primarily used for event aggregation, post-event analysis, or query building. As systems have grown into intelligent agents, able to plan several steps, call APIs or command-line tools, and change configurations directly, this shift has become more significant. Recent research demonstrates that by combining language inference with operational tools, language models can support event workflows, including monitoring and diagnostics.[7], [8], [9], [10]. This work focuses on this shift and its impact on architecture, assessment, and security. However, LLMs should not be considered the only way to achieve intelligent operations. In various NetOps and AIIOps scenarios, causal reasoning and dependency graph methods have provided interpretable and less computationally intensive fault location and hypothesis generation mechanisms; therefore, a core design question is: how should LLM agents complement rather than mask or replace these causal mechanisms? Consider an on-call incident where a latency SLO breaches for a microservice after a routine deployment. An agent must (i) retrieve recent changes and ownership context, (ii) issue a small number of discriminating log/metric/trace queries, (iii) propose a mitigation such as a feature-flag rollback or traffic shift, and (iv) route any write action through policy checks, invariant validation, and canary rollout with rollback triggers. The same example later illustrates the tool loop, the verification wall, the evaluation ladder, and the threat model. The literature relevant to agentic operations is scattered across networking, systems, software engineering, and security. Networking venues emphasise correctness and invariants under policy and topology dynamics [1], [2], [3], [11]. Systems and SRE practice emphasise incident response latency, operational load, and the brittleness that comes from complex dependencies [4]. Software engineering venues increasingly study how LLMs help operators express intent and retrieve the right operational evidence [7], [8]. Meanwhile, foundation-model research has produced patterns for tool use and iterative reasoning that are directly relevant to operations, but are rarely discussed with operational risk in mind [12], [13], [14]. The survey is timely although researchers are exploring the agentic workflows, but the field lacks shared terms for autonomy, a stable set of evaluation tasks, and a clear threat model that matches real operational tool surfaces. A. Why agentic operations now The structure of a running task is just as important as the functionality of the underlying model. In most cases, tasks involve a chain of queries and actions, often with missing details. An operator may need to check the blast radius, look for evidence in logs or counters, suggest ways to fix issues, make changes, and then watch for any problems that return. These steps follow a clear procedure. Much of the important context appears in work orders, manuals, commit messages, or handover notes. LLMs (Logical Models) fit well here because they can work with both the process and the language found in these records. a) (1) Running tasks require extensive retrieval.: Much of the work in these settings comes down to searching for evidence and correlating information from several tools.

3

Retrieval-enhanced designs address this need by joining model reasoning with dependable access to documents and system data [12]. In practice, the information we need is often present, but it can be hard to find. This is usually because of fragmented tools, changes over time, inconsistent names, or scattered records from past events. b) (2) Tool invocation translates language into an executable plan.: Modern LLM agents can select tools, maintain state across steps, and modify the plan as new evidence emerges.[13], [14]. This approach is similar to the workflow of an experienced operator, but agents can repeat the process more quickly on different interfaces. In incident management, this helps in developing systems capable of recommending investigative queries, summarising monitoring signals, and identifying potential root causes [7], [8], [10]. c) (3) Networks and clouds already have a correctness tradition.: NetOps has mature ways to check whether a change violates invariants, breaks reachability, or conflicts with policy intent [2], [3], [1]. This tradition introduces a beneficial constraint on agent-based systems. An agent may propose action plans, but the entire system must check critical properties before any action is taken and keep changes within safe limits. As network operations have shown, finding new configurations addresses only one aspect of the problem. The real challenge lies in deploying these changes without causing temporary violations or disruptions[11]. However, these advantages also introduce new types of failures. Agentbased systems integrate interpretation and execution into a single operational flow. Misunderstandings of dashboards, unclear operating manuals, or imprecise objectives can all lead to changes that impact system availability. Once languages become the programming means for operating tools, the previous assumption that languages were merely harmless interfaces no longer holds true. d) Main question: measurable operational objectives.: This article explores a very straightforward core question: How should we design, evaluate, and manage NetOps and AIOps agents to maintain or even maximise reliability and security while reducing operational burden? From an operational perspective, the goal is to improve outcomes such as diagnostic and mitigation times while meeting explicit security constraints. These constraints include policy compliance, limiting the scope of impact, and ensuring rollback readiness.[4], [5]. For this reason, the later discussion on evaluation focuses on workflow performance as the primary subject, rather than on static question answering. B. Scope and definitions We use NetOps to mean operational workflows specific to communication networks, including intent, configuration, troubleshooting, traffic engineering, and safe rollout [15], [16], [17]. We use AIOps to mean AI-supported IT operations, including log and metric anomaly detection, incident triage, diagnosis, and remediation [18]. We use agentic to describe a system that maintains state across steps, selects tools or actions, and executes multi-step plans, possibly with human approval at selected stages [19], [20], [21]. a) Autonomy ladder as an operational object.: We model autonomy as a ladder of rungs. Each rung is characterised by which tools the agent may use and which gates
are mandatory: read write Ak = Tk , Tk , Gk , (1) read write where Tk are read-only tools (queries and retrieval), Tk are write-capable tools, and Gk are nonbypassable gates (policy checks, invariants, approvals, canary and rollback rules) [3], [11], [22]. For clarity, we treat “write” as two capabilities: Tkwrite = Tkpropose ∪ Tkexec , (2) so a system may be allowed to draft diffs or change requests (propose) without being allowed to apply them (execute). Later sections use Ak to state what is being evaluated and what safety contract must hold. b) Evidence traces for audit and scoring.: We treat the agent’s interaction with tools as a first-class output:
E = (τ1 , y1 ), . . . , (τn , yn ) , (3) where τi is a typed tool call (tool name plus parameters), and yi is its output. This makes the concept of ”tool grounding” more concrete: the evaluation can determine whether the tracing results are discriminative, whether the cost is limited, and whether it is sufficient to justify the proposed operation. [19], [20], [21].

4

Therefore, a practical approach to interpreting the term ”agentic” in this survey is to view it as a ladder of autonomy. At the bottom is a read-only assistant responsible for aggregating and retrieving information. Mid-level systems can recommend queries, hypotheses, and manual procedures. Systems with higher autonomy can develop mitigation measures, generate configuration discrepancies, and run security checks. At the top is a closed-loop controller that continuously submits changes and verifies results. [8], [10], [22]. The taxonomy in Fig. 1 frames these roles, the tool surfaces they touch, and the guardrails typically needed at each level. We also limited our investigation to the operations tool interface, including: monitoring backends, log storage, tracing systems, configuration repositories, CI/CD pipelines, network controllers, and ticketing systems. This is crucial because security and governance are inextricably linked to the tools that agents can invoke, the permissions they possess, and the checks and controls that are enforced.[18], [8], [10]. C. Contributions of this survey This paper makes four contributions. 1) Workflow-grounded taxonomy. We have constructed a classification system for agent-based NetOps and AIOps design, based on operational phases (observation, diagnosis, decision-making, and action) and levels of autonomy, rather than the internal structure of the model. We associate these roles with specific tool interfaces and failure modes. 2) Architectural patterns and guardrails. We extract recurring design patterns from recent systems, including retrieval and evidence management [12], tool-use loops [13], [14], and operational assistants for monitoring, query recommendation, incident analysis, and network troubleshooting [7], [8], [9], [10]. We map these patterns to guardrails such as verification, privilege minimisation, staging, and human approval. 3) Evaluation practice and benchmark guidance. We synthesise our evaluation methodology into task definitions that reflect actual operational work, including event classification, diagnostic accuracy under partial observability, and successful generation and rollout of safety changes. We also discuss how to design benchmarks that measure end-to-end utility rather than isolated language quality, drawing on emerging agent evaluation practice [23]. 4) Security, governance, and compliance analysis. We analyse concerns that are specific to agentic operations, including unsafe tool invocation, privilege and audit boundaries, and the tension between autonomy and accountability. We emphasise where existing network verification and update abstractions can be repurposed as enforceable safety constraints [2], [3], [1], [11]. D. Organisation of the paper Section 2 reviews NetOps and AIOps backgrounds and the operational tool stack. Section 3 introduces LLM and agent foundations for operations, including retrieval and tool-use loops. Sections 4 to 6 review architectural patterns and representative systems across monitoring, diagnosis, and change execution. Section 7 focuses on evaluation and benchmarking. Section 8 covers security, governance, and compliance. Section 9 outlines open research problems, followed by conclusions. II. O PERATIONAL BACKGROUND : N ET O PS AND AIO PS Operations is best understood as a control problem carried out under uncertainty and organisational constraint. Most incidents are not solved by one brilliant query. They are solved by repeatedly turning partial signals into safe, reversible actions while keeping the blast radius small. That is why “helpful text” is only a small part of the story. Key challenges include the collection of evidence, the proposal and review of changes, and the documentation of decisions for subsequent audit and organizational learning. Early research on Internet operations demonstrated that even minor configuration errors or policy mistakes can result in unexpected and widespread failures [24]. As systems increase in scale and software updates occur more frequently, the likelihood of such issues escalates.

5

Fig. 1: A ladder-of-autonomy taxonomy for agentic NetOps and AIOps. As systems move from read-only assistance to write-capable execution, the engineering centre of gravity shifts from answer quality to safe action: typed tool interfaces, independent verification, approvals, and rollback-ready rollout. A. A unified evidence-to-action control loop Both NetOps and AIOps can be modelled as a partially observed control loop. There is a latent system state xt (network and service state), operators and automation observe it only through tool-mediated observations ot (logs, metrics, traces, reachability checks), and they apply actions at (changes, mitigations) that alter future state [1], [10], [8], [18]. A compact abstraction is: xt+1 ∼ P (xt+1 | xt , at ), ot ∼ O(ot | xt ), at ∈ Ak . (4) The key operational point is the rightmost term. The agent does not act in an unconstrained action space. It acts within an autonomy rung k, which is defined by the tool permissions and the mandatory

6

gates that must be satisfied before any write-side action is executed [3], [11]. Following the introduction, we treat an autonomy rung as an operational object:
(5) Ak = Tkread , Tkwrite , Gk , read write where Tk are read-only tools (queries and retrieval), Tk are write-capable tools, and Gk are nonbypassable gates (policy checks, invariants, approvals, rollout constraints) [3], [11]. This makes “tools as a boundary of trust” concrete: the trust boundary is precisely the interface where a proposal becomes an executed action, and where Gk must hold. A second shared primitive is mismatch between what is intended and what is actually realised [15], [1]. Let xdesired denote the declared desired state (intent, policy, change request), and let xrealised denote t t the realised operational state (what the system actually does, as evidenced by measurements and derived models) [15], [1]. We can express mismatch as:
∆t = d xdesired , xrealised , (6) t t where d(·, ·) is a domain-appropriate distance (for example, a binary violation indicator for an invariant, a count of violated intent clauses, or a normalised diff magnitude for a configuration slice). This is useful later because ∆t is an evaluation axis: good operations reduces ∆t safely, or refuses to act when uncertainty is too high to reduce it without undue risk [3], [11].

Fig. 2: Unified evidence-to-action control loop across NetOps and AIOps. The system is partially observed (ot comes through tools), actions at are constrained by autonomy rung Ak , and write-side execution is mediated by mandatory gates Gk (policy, invariants, approvals, rollout constraints).

7

B. NetOps workflows and artefacts NetOps encompasses operational lifecycle of a communication network, and its workflow typically consists of five stages: capturing intents, integrating device or controller changes, validating against invariants and operational policies, executing phased deployment, and conducting post-change monitoring with rollback as necessary. In programmable networks, the controller has a central role in this process. This change began with Software-Defined Networking (SDN), which separated the control plane from the data plane. This separation gave operators a clearer way to manage the control plane [25]. NetOps is built around artifacts that exist at different levels of operational truth. At the top are desired states, which include policies, intents, templates, and change requests. In the middle are rendered states, such as device configurations, controller rules, access control lists (ACLs), and routing policies. At the bottom are realized states, including forwarding tables, routing adjacencies, counters, and alarms. Operational problems often arise when these layers do not match. In notation (6), xdesired represents intents and policies, while xrealised t t represents the resulting data-plane and control-plane behavior. User-facing intents are often too broad to verify directly, while vendor configurations are precise but difficult to interpret. Intent-based networking fill this gap by employing a closed-loop process of transformation, execution, and monitoring, rather than relying on a single step transformation[15]. a) Why configuration is operationally hard.: Configuration is not merely parameter setting. It is a distributed process across heterogeneous devices, with asynchronous convergence and emergent behaviour. Configuration errors can stem from localised errors, policy conflicts, drift, or fragile sequences of changes. Empirical research on BGP configuration errors demonstrates that even minor policy errors can propagate and cause system-wide impacts, remaining a cautionary tale in the field of automation. [24]. Furthermore, even if the final state is correct, it can still be reached through unsafe intermediate states if the update order is incorrect. This has prompted the adoption of secure update protocols and phased deployment designs to explicitly manage transient inconsistencies. [26]. b) Causal diagnosis as a NetOps primitive.: Not all NetOps diagnoses are suitable for language reasoning or pattern recognition. A complementary research approach employs causal inference, dependency models, and event-related structures to explain the causes of observed symptoms[27], [28], [29]. This is attractive in network operations because faults propagate along known relationships: topology, protocol dependencies, control plane interactions, service dependencies, and time event sequences[27], [29]. Causal approaches help construct fault graphs that are both compact and interpretable. When the underlying causal or dependency structure is known, this approach helps to reduce the need of repeatedly performing same resource-intensive inference based on raw telemetry data and operational manual context. Thus, for network operations, it makes sense to use causal reasoning as a primary tool for Tkread . LLMs can use these models through questioning or summarizing, but the causal graph itself clarifies the main assumptions and demonstrates the relationships between the evidence. This assumption about structured reasoning is crucial for improving practical efficiency [27], [28], [29]. c) Validation and verification as everyday NetOps.: In network operations, a change is normally checked before it is accepted. The reason is simple: a small configuration edit can alter reachability, isolation, waypointing, or policy compliance. This has led to tools that translate configurations into models and inspect these properties before deployment. Batfish is a useful example. It extracts the intended data plane from candidate configurations and supports pre-deployment what-if analysis [1]. Minesweeper builds on this line by offering scalable configuration verification over richer properties and practical analysis pipelines [30]. In controller-managed or fast-changing networks, the same concern appears at runtime. Updates should not break invariants simply because the control plane has moved quickly. VeriFlow addresses this by placing a verification layer between the controller and network devices, catching problematic rules before they reach the data plane [3]. Verification also tests the assumptions held by operators. Teams often carry high-level beliefs about which resources are reachable and who can access them. These beliefs may be partial, dated, or wrong. NoD shows how examining them can expose policy gaps and security risks [31].

8

d) NetOps tools and change governance.: NetOps also rests on ordinary governance machinery: configuration management databases, topology stores, configuration platforms, CI checks, controller APIs, and ticketing systems. These tools decide who can see, approve, and change the network. Read access is often wide, while write access is narrower and leaves an approval and audit trail. Strong verification alone does not ensure adoption. A method has to fit operational practice and remain reliable, scalable, and usable under routine pressure [32]. This point matters for agentic systems because NetOps is already shaped by tools, policies, and human approval. These are not side controls. They define the action surface available to an agent. C. AIOps workflows and artefacts AIOps supports the daily operation of modern network and cloud systems by turning telemetry into incident evidence and response options. Incident work often moves through detection, classification, diagnosis, mitigation, and learning, although the order is seldom tidy. Shift patterns, escalation rules, service ownership, and post-incident review shape what happens in practice. AIOps therefore sits between raw system signals and operational judgement. a) Signals to decisions: why evidence quality dominates.: AIOps systems usually start by processing alerts. These alerts often contain noise or show symptoms that are not clear at first. This initial uncertainty shapes the rest of the incident response. The purpose of triage is to direct the incident to the appropriate responders and to eliminate common causes as quickly as possible. Diagnosis involves connecting observed symptoms to possible failure modes. Ideally, this is supported by tests that can rule out weaker hypotheses. Early systems work on problem determination in large services already stressed that diagnosis must be grounded in measured behaviour, since modern services are dynamic and failures often have several contributing causes [33]. As systems have grown in complexity, the diagnostic process has expanded to include information about provenance and the context of recent changes. Orca illustrates how operational debugging can be improved by tying incidents to code and deployment provenance, so that suspect changes can be ranked and inspected [34]. b) Artefacts: incident timelines, runbooks, and postmortems.: In comparison to NetOps, AIOps produces a broader and more detailed set of narrative artefacts[18], [35]. Incidents generate timelines, chat transcripts, dashboards, on-call notes, runbooks, and postmortems [35], [36]. These artefacts capture both policy and practice. They reflect what the organization considers safe, which actions are permitted, and what evidence must be gathered before any mitigation is attempted [35], [36]. They are also a source of systematic bias, because runbooks may be stale, postmortems may omit sensitive details, and tickets may carry ambiguous or adversarial text [35], [37]. For agentic systems, these artefacts are both a knowledge base and an attack surface, so it is helpful to treat them as governed inputs rather than informal documentation. c) From anomaly detection to actionable RCA.: Early research in AIOps largely focused on incident detection. However, the true operational benefits lie in the actual mitigation of incidents [18], [35]. This early focus led researchers to pay more attention to root cause analysis, often combining different types of data to support actual fault localization [38], [39]. Recent work in automated software engineering argues for multi-modal RCA at finer granularity, combining metrics with other signals to localise causes more precisely in microservice settings [38]. This reinforces a practical point: operations data is incomplete, and single-modality answers are often too fragile [37], [38], [39]. Figure 2 follows the standard incidentresponse loop used in large-scale services and observability stacks, informed by classic service diagnosis and tracing systems [33], [40], [41], [35]. D. Operational artefacts as data types: trust and freshness Agentic operations systems inevitably consume and produce artefacts [35], [18]. Treating these artefacts as data types, instead of as “just text”, makes the trust boundary visible. It also reduces accidental over-trust during incidents, when people and systems are already working under pressure. Two axes are useful.

9

TABLE I: Operational artefacts as data types, with typical trust and freshness concerns. Artefact type

Typical source

Trust class

Common staleness or failure mode

Policy / intent

Authoritative

Telemetry: logs / metrics / traces Tickets / runbooks

policy-as-code, change requests [42], [43] repositories, controllers [26], [44] observability stack [33], [40], [41], [37] ITSM, wikis [36], [35]

Postmortems / timelines

incident documents [35], [45]

Advisory

Policy drift, partial translation into device or service configurations [42], [43]. Drift from reality, unsafe intermediate states during rollout [26], [44]. Missing coverage, misleading signals, sampling bias [37], [18]. Stale steps, ambiguous intent, injection through freeform text [36], [35]. Incomplete detail, hindsight bias, outdated assumptions [35], [45].

Config baselines / diffs

Authoritative Mixed Advisory

a) Trust level.: Some artefacts are authoritative, such as policy-as-code, signed configuration baselines, and CMDB entries with ownership. Some are advisory, such as runbooks, postmortems, and tickets. Other inputs are considered untrusted, including free-form chat logs, externally provided documents, and user-influenced log strings. The operating rule is simple: untrusted artifacts can provide information for hypotheses, but without independent checks in Gk , they must not be used to drive write operations. b) Freshness and staleness risk.: Operational knowledge ages [35], [36]. Runbooks lag behind deployments, ownership rotates, and instrumentation coverage changes [36], [37]. Freshness should therefore be tracked at the artefact level through timestamp, version, and provenance. When a write action is under consideration, retrieval should favour recent and authoritative sources. E. Observability modalities and tool surface Agentic operations must connect varied evidence types to the tools that expose them. Each modality is an imperfect sensor for a distributed system. Each tool is either a read-side interface that produces observations ot , or a write-side actuator that proposes or executes actions at and therefore triggers the gates Gk . a) Logs: rich context, weak structure.: Logs carry useful semantic context, but they are often unstructured, noisy, and inconsistent across components. In practice, log analytics usually begins with parsing and template extraction so that downstream correlation is possible. Drain is a widely used example of online log parsing for streaming environments, which is closer to operational reality than offline batch parsing [46]. For anomaly detection and diagnosis, deep sequence models such as DeepLog demonstrate how systems can learn normal log patterns and flag deviations. However, their reliability in operation depends on robustness to time offsets and on the quality of the upstream parser [47]. Dataset resources such as Loghub help anchor evaluation in varied real logs, and reduce overfitting to the logging style of a single system [48]. b) Metrics: regular, scalable, and ambiguous.: Metrics are cheap to collect and easy to aggregate, which makes them common in alerting systems, but the main weakness is they are ambiguous. Different failure modes may produce the same metric symptoms. Modern metric pipelines also use labels and dimensions to record service topology and deployment context. Standardisation efforts such as OpenTelemetry aim to improve data consistency across services and vendors. In incident handling, this consistency matters because it reduces the amount of data translation required before diagnosis can begin [49]. c) Traces: causal hints with coverage gaps.: Distributed tracing is valuable because it exposes request paths and the factors that affect latency. Traditional tracing designs link underlying events to request context, so that engineers can ask causal questions rather than scan raw logs alone [40]. At scale, traceability is both a systems problem and a data management problem. Canopy is an end-to-end traceability and analysis system designed for production applications and high-volume environments [41]. The practical limit is coverage. Traces are sampled, instrumentation is incomplete, and context propagation can break across service boundaries.

10

Fig. 3: Operational artefacts by trust level and staleness risk. The same artefact can move over time. For example, a runbook can become stale after a logging change. Retrieval therefore needs provenance and freshness signals when actions approach the write boundary. d) Tickets and change records: the policy layer in disguise.: Tickets and change records encode intent, approval, and accountability [35], [50]. They are not telemetry, but they often contain the constraints that decide whether an action is safe [36], [35]. They also help define the ground truth for operational outcomes: whether mitigation was accepted, whether a change was rolled back, and whether a similar incident recurred [50], [51]. For agentic systems, tool interfaces to incident management and change control should therefore be treated as first-class surfaces [35], [52]. e) Tools as a boundary of trust.: Operational tools determine which operations an agent can safely observe and perform. Query tools are used to expose evidence. Change tools can alter the system. This asymmetry is a major reason why operations differ from many general agent benchmarks. Read-only assistants can be useful in situations with moderate risk. Write-enabled agents require the principle of least privilege, phased deployment, and independent verification. Without these controls, it will simply execute failure automatically, which is not the most efficient form of failure. In the unified model (4), this is the difference between tools in Tkread and tools in Tkwrite . This is also where the gate Gk must be located. Network operations literature on verification and security updates provides concrete precedents for this trust boundary [1], [30], [3], [26]. III. LLM FOUNDATIONS FOR OPERATIONS LLMs matter in operations for a different reason than in many general agent demonstrations [14], [13], [53], [54]. Operations work has been coordinated through various tools, including queries, dashboards,

11

Fig. 4: A practical “LLM-in-ops” stack. Reliability comes mainly from typed tool interfaces, an explicit verification wall, and gated operational actions, with persistent evidence logging and governance around approvals. configuration validators, and ticketing systems. Furthermore, it is subject to clear constraints, such as change windows, approval processes, scope limitations, and audit responsibilities. [4], [5], [55], [44], [56], [35]. Therefore, the foundation includes both the model itself and the system in which it exists. This system limits the scope of what the model can perceive, making tool use the default mode of cognition, and treating every operation as a change that must be justified, checked, and reversible.[9], [7], [8], [52], [57], [58]. Figure 4 sketches a practical stack that reflects how mature NetOps and AIOps organisations already operate [4], [5], [18], [35]. The LLM is most valuable as a coordinator of evidence, hypotheses, and proposals, while reliability comes from interfaces and checks that sit outside the model [12], [53], [13], [59], [60]. This section introduces a compact set of primitives that later sections reuse: a workflow contract, budgets and stopping rules, evidence traces, and the verification wall. A. A workflow contract for safe autonomy We model an operational agent as running under an explicit workflow contract. This contract makes “safe autonomy” operational, testable, and auditable, rather than a mood [61], [57], [58]. We reuse the autonomy-rung object from Section 1: Ak = (Tkread , Tkwrite , Gk ), (7)

12

where Tkread and Tkwrite are the read and write tool sets available at rung k, and Gk is the set of mandatory gates (policy checks, invariants, approvals, change windows). We also reuse the evidence-trace object:
E = (τ1 , y1 ), . . . , (τn , yn ) , (8) where τi is a tool call (including arguments) and yi is its output. This style of explicit evidence capture is consistent with long-standing observability and tracing practice in operations [62], [40], [63], [41], [49]. a) Definition (Agentic workflow contract).: A contract is a
tuple C = Ak , Π, I, R, B, S , (9) where Π is an organisational policy (policy-as-code, ACL rules, change governance), I is a set of operational invariants (reachability, isolation, SLO guards, safety constraints), R is a set of required evidence obligations (what must be observed before proposing or committing), B is a budget vector, and S is a set of stop rules (escalate, rollback, or halt). This contract is the simplest way to state what later sections mean by “tool-first + gates”. The agent may reason freely, but it may only act through Ak , and it may only commit by passing Gk while satisfying Π, I, and R [42], [2], [3], [64], [1], [65], [43]. B. Budgets and stopping rules as first-class constraints Operational agents are budgeted, not only for cost control, but because bounded loops are part of the safety posture [66], [67], [68]. We treat budgets as a vector B = (B tool , B tok , B time , B risk ). (10) tool tok time risk Here B limits tool churn, B limits model context and generation, B limits latency, and B limits the probability of executing a violating write action. A convenient budgeted planning constraint is: n n n X X X tool tok c(τi ) ≤ B , ℓi ≤ B , ∆ti ≤ B time , (11) i=1

i=1

i=1

where c(τi ) is a per-tool call cost (or simply 1), ℓi is tokens consumed in step i, and ∆ti is wall-clock time for step i. For risk, we use an explicit limit:
. Pr VΠ,I (at , E) = 1 ≤ ϵ, with ϵ = B risk , (12) where at is a candidate write action (often a diff plus rollout plan), and VΠ,I (·) is a violation indicator under the checker and the modelled domain. This makes “risk sensitivity” measurable: you can log when the agent refused to commit because the estimated risk was above ϵ [61], [57], [69], [70], [71], [72]. a) Stop rules.: Stop rules S should be explicit and mechanically enforceable, for example: (i) budget exhaustion, (ii) missing required evidence R, (iii) failed gate in Gk , (iv) repeated contradictory observations, (v) non-diagnosable ambiguity that requires a human decision. In operations, “knowing when to stop” is as important as knowing what to try [73], [74], [45]. C. Why operations need tool-first design Two operational facts explain why fluent text is a poor success criterion. First, evidence is produced by tools. In incident response, the dependable way to know the system state is to query it. This is why recent systems collect and structure diagnostic artefacts before LLM reasoning, rather than asking the model to infer state from a short prompt [10], [8], [9], [7], [75]. Second, actions must be reviewable and reversible. An answer often implies a change. For this reason, practical systems express actions as structured proposals, check them outside the model, and apply them with limited permissions [76], [77], [44], [42], [43]. Router configuration synthesis shows the same point. Raw LLM output is brittle, and correctness comes from verifiers and repair loops around the generator [59], [78], [79].

13

Fig. 5: Operational agent state machine with mandatory gates and stop conditions. Read-only states can iterate freely under budgets; write intent must pass a non-bypassable verification wall before execution. D. Prompting, schemas, and domain adaptation Prompting is often enough for summarisation, ticket drafting, and first-pass triage, since these tasks are mostly read-only [35], [36], [52]. Once the system proposes tool calls or remediation plans, the target changes. The system must behave steadily under noise, choose tools correctly, follow schemas, and avoid unsafe guesses [13], [14], [53], [54], [68]. Structured outputs act as executable contracts because they can be tested. Typed actions, hypotheses, and diffs can be checked syntactically, tested against policy, replayed, and regression-tested on past incidents [8], [80], [81]. Schemas should therefore be versioned like APIs. When a schema changes, the evaluation set should be rerun, since schema drift is a common failure mode during live incidents. a) Where adaptation helps.: Across operations papers, adaptation mainly helps with format reliability, tool-selection skill, stopping behaviour, and risk sensitivity when uncertainty remains. Networkingspecific adaptation work argues for reusable capability across several networking workloads [82]. Reliabilityfocused agent designs use reusable invariants and checks to constrain action, rather than relying only on more task-specific data [60], [68]. b) Ambiguity handling is part of the foundation.: Operational intent is frequently underspecified. HotNets work on ambiguity in LLM-based network configuration synthesis shows that clarification is necessary for correctness under overlapping rule scopes and priority interactions [83], [59]. E. Retrieval-augmented generation for operational knowledge Operational knowledge changes continuously. Runbooks are revised after incidents, ownership rotates, and baselines drift. Retrieval is therefore a reliability mechanism, since it bounds what the model may assume [12], [84]. The general motivation for retrieval-augmented generation is well established [12]. Operations adds two stricter requirements: freshness and provenance.

14

Fig. 6: Operational budgets as first-class constraints on agent loops. Tool, token, and time budgets bound churn and latency; a risk budget bounds the probability of committing a violating write action under the checker. a) Freshness and time sensitivity.: Many operational questions are time-indexed, even when the prompt does not say so. A runbook step may become wrong after a migration. An incident may be explained by a rollout that took place an hour earlier. This is why retrieval and evaluation should be time-aware, with penalties for answers that rely on stale artefacts when fresher evidence was available [84]. b) Provenance and trust levels.: Retrieved content can shape privileged action, so the retrieval layer is also part of the security boundary [70], [85], [86]. At a minimum, it should enforce allowlists, preserve document provenance, redact secrets, and record what was retrieved [69], [87]. A practical design should also distinguish between authoritative sources, such as policy-as-code, CMDB entries, and validated topology, advisory sources such as runbooks, tickets, and postmortems, and untrusted sources such as free-form chat or externally supplied text. This separation is not just tidy engineering. It affects what the agent may safely believe. Retrieval methods that require explicit attribution therefore serve a useful operational purpose [88]. F. Tool use and controlled action interfaces Tool-using agents can select tools, retain state across several steps, and revise a plan as new evidence appears [14], [13]. In operational settings, however, tool use has to be structured, permissioned, and tested under failure conditions that resemble ordinary practice. Work on tool-augmented agents shows that tool errors are not exceptional cases. They include poor tool choice, malformed arguments, fragile dependence on the wording of tool outputs, and loops that continue long after the useful work has ended [54], [67], [19], [89]. Systems that connect LLMs to large API surfaces make the same point from another direction: invocation must be constrained, and argument formatting must be robust enough for operational use [21], [20].

15

a) Typed tools and least privilege.: Operational tools should be typed, versioned, and permissioned. An agent should not be given free-form shell access by default. Instead, agent should call narrow APIs for privileged or sensitive tasks: running a validator, testing in a sandbox, opening a change request, or deploying a canary. b) Action proposals as diffs.: A robust pattern is to require diffs with preconditions and rollback steps. The executor then applies those diffs only after validation. Verified Prompt Programming illustrates this pattern: generate, check, localise violations, and iterate until verified [59]. G. A minimal verification wall proposition We make the “verification wall” precise as a non-bypassable gate between proposals and commits [2], [3], [1], [30], [90]. a) Proposition (Verification wall).: Let Ak = (Tkread , Tkwrite , Gk ). Assume every write action a, equivalently every τ ∈ Tkwrite , is executed only if it passes a non-bypassable gate g : (a, E, Π, I) 7→ {allow, deny}, g ∈ Gk , (13) and that g is sound for the modelled domain. That is, g = allow implies that a satisfies Π and I as represented by the checker. Then any executed write-action sequence satisfies Π and I in the modelled domain. b) Proof sketch.: Each executed write action must pass g by assumption. Soundness gives that each permitted action satisfies Π and I in the checker domain. Induction over the executed sequence then gives the result. This gives no guarantee outside the modelled domain. Coverage gaps, stale models, or unsound checkers break the implication, which is why Section 8 treats checker scope and bypass resistance as security and governance concerns [32], [65], [76], [91]. H. Planning patterns with explicit budgets and gates Agentic operations converges on a small set of planning patterns because incidents tend to follow a recurring loop: hypothesise, test, mitigate, and validate [4], [5], [73], [10], [9], [8], [75]. The difference from generic agents is that plans are constrained by Π, I, budgets B, and gates Gk [13], [14], [76], [56], [77], [92]. I. Safety foundations that interact with tool use Agentic operations expands the attack surface because it routes untrusted text and telemetry into privileged actions [70], [87], [93], [57]. Safety therefore belongs in the foundations of the architecture [61], [57], [58]. a) Prompt injection and untrusted documents.: Tickets, runbooks, and chat transcripts are attractive carriers for malicious instructions. n terms of operations and maintenance, the rules are simple: treat retrieved documents as untrusted input, isolate tool permissions, and perform independent checks before executing any operation [87], [70], [71]. b) Telemetry manipulation as an integrity attack.: An adversary can manipulate telemetry data, causing agents to make incorrect diagnoses and recommend harmful actions. This constitutes an integrity failure. Even well-behaved models can be manipulated by false input. Recent research has addressed telemetry data manipulation attacks against LLM-based IT operations agents and proposed defense strategies based on structured telemetry data cleaning and cross-checking [72]. This threat also fits a broader risk-management view, in which model safety depends on the integrity of tools, retrieved context, and upstream data channels [61], [57]. c) Guardrails that survive incidents.: Three guardrails are consistently enforceable: least-privilege tool access, independent verification gates, and controlled autonomy through approval, canarying, and rollback readiness [92], [76], [44], [56]. They also make evaluation more meaningful, because they define safe success rather than mere task completion [74], [66], [58].

16

Algorithm 1 Planner–executor loop under a workflow contract 1: Input: incident signal s, contract C = (Ak , Π, I, R, B, S) 2: E ← ∅ 3: H ← ∅

(evidence trace) (hypotheses set)

4: initialise budget counters (btool , btok , btime ) ← (0, 0, 0) 5: while not resolved do

if S triggers on (E, H, s, B) then 7: return escalate or rollback with E 8: end if 9: p ← P LAN(s, H, E, Π, I, B) (typed plan) 10: if not P OLICY C HECK(p, Π) then 11: return escalate with E 12: end if 13: for step i in p do 14: choose (τi , argsi ) with τi ∈ Tkread ∪ Tkwrite 15: update budgets: btool ← btool + c(τi ) 16: update (btok , btime ) similarly 17: if (btool , btok , btime ) exceeds B then 18: return stop and escalate with E 19: end if 20: if τi ∈ Tkwrite then 21: if not G ATE C HECK(τi , argsi , E, Π, I, Gk , B risk ) then 22: return deny action, propose safe alternatives, and escalate with E 23: end if 24: end if 25: yi ← τi (argsi ) 26: append (τi , yi ) to E 27: H ← U PDATE H YPOTHESES(H, E) (keep alternatives and disconfirming tests) 28: s ← U PDATE S IGNAL(s, E) 29: end for 30: end while 31: return diagnosis (top-k) + next safe actions + evidence trace E 6:

IV. AGENTIC ARCHITECTURE PATTERNS FOR OPERATIONS Agentic NetOps and AIOps systems vary mainly in what they are allowed to do [35], [52], [60]. A useful way to describe this is a ladder of autonomy, where each rung is a capability contract, not a product label. At low autonomy, the agent reads and organises evidence. At higher autonomy, it proposes or executes changes, which forces explicit permissions, gates, and rollback semantics [9], [7], [8], [60]. A. Autonomy rungs as capability contracts We make the rung definition operational by tying it to tool scopes and non-bypassable gates. Let the tools available at rung k be Tk = Tkread ∪ Tkpropose ∪ Tkexecute . (14) We reuse the rung object introduced earlier and
write Ak = Tkread , Tkwrite , Gk , Tkwrite = Tkpropose ∪ Tkexecute , (15) where Gk is the set of mandatory gates (policy checks, invariant checks, approvals, change windows). For any candidate action a produced by the agent, we model gating as a total function gk (a, E, Π, I) ∈ {0, 1}, (16)

17

TABLE II: A ladder of autonomy for agentic NetOps/AIOps written as capability contracts. Higher autonomy shifts the engineering burden from answer quality to gated action, least privilege, and rollback [92], [56], [76], [77], [94]. Rung k

Tool scopes Tk

Mandatory gates Gk

Artefacts produced

Success criteria (examples)

Copilot (read-only)

Tkread only: search, retrieve,

Redaction; provenance rules; citation requirement; rate limits [69], [87], [88]

Evidence trace E; incident summary; query candidates [8], [7], [49]

Faster evidence location; fewer missed signals; low false citations [8], [7], [68].

summarise, and query suggestion; no execution [8], [7], [52] Analyst (readmostly)

Read tools plus diagnosis tools, including causal graphs, dependency models, eventcorrelation engines, and telemetry queries; executes read queries only[9], [10], [75]

Citation to tool outputs; contradiction checks; stop rules; escalation policy [10], [52], [58]

E; hypothesis set H; RCA report with evidence pointers [10], [38], [39], [95]

Correct top-k localisation; low tool churn; calibrated uncertainty [38], [39], [96], [66].

Planner–executor (write-limited)

Read tools plus propose tools for diff synthesis, plus execute tools limited to safe actuators [59], [60], [79]

Non-bypassable verification wall; approvals; canary constraints; rollback triggers [1], [3], [64], [65], [76], [56]

E; change diff ∆; preconditions; rollout plan; rollback plan [77], [76], [94], [44]

No invariant violation in the checked domain; safe rollout; quick rollback on regression [1], [3], [30], [77], [95].

Closed-loop (selfhealing)

Continuous read plus execute within a narrow action envelope [76], [77], [97]

Strict action envelope; automated rollback; continuous monitors; periodic audits [92], [57], [58], [98]

E (continuous); signed action log; recovery certificates where possible [49], [99], [98]

Improved MTTR with bounded regressions; stable long-run drift ∆t [76], [77], [97], [5].

which returns 1 only when the action is permitted under policy Π, passes the relevant invariants I (in the modelled domain), and satisfies the gate conditions in Gk [3], [1], [64], [65], [43]. The evidence trace E is the structured log of tool calls and outputs:
E = (τ1 , y1 ), . . . , (τn , yn ) . (17) This framing lets us state autonomy precisely. A system is not ”safe” because it sounds cautious. It is safe only if its write actions are constrained by Tkexecute and cannot bypass gk (·) [92], [57], [58]. B. Ladder of autonomy with explicit tools, gates, artefacts, and success Table II summarises four common levels. For each level, we list (i) tool scope, (ii) mandatory levels, (iii) generated artifacts, and (iv) success criteria. The purpose of this is to ensure that subsequent architecture and evaluation sections refer to the same objects, rather than restating autonomy in words. C. Reference architecture and evaluation observables Figure 4 gives the reference architecture used throughout this survey. It condenses the safe-autonomy commitments that recur across incident assistants, tool-grounded diagnosis systems, and verification-heavy NetOps workflows [8], [7], [10], [9], [1], [3], [26], [30], [60], [52], [57], [58]. The practical value of this architecture is that it makes later evaluation concrete. It exposes observables that can be measured across tasks, traces, and deployment stages [66], [68], [100]. Examples include: • Evidence quality: completeness and correctness of E, including missing tool calls, incorrect citations, and stale artefacts [88], [84]. • Gate behaviour: pass and fail rates of gk (·), together with the reasons for rejection, such as policy violation, invariant failure, missing approval, or budget exhaustion [1], [3], [64], [65], [57]. • Proposal quality: validity of diffs ∆, satisfaction of preconditions, and behaviour under staged rollout [59], [79], [77], [94].

18

•

Operational outcomes: time to mitigate, rollback frequency, and post-change drift or mismatch metrics, for example ∆t from Section II [5], [35], [95], [50].

D. Human review as a concrete object In guarded planner–executor systems, the human does not review a chat transcript. They review a bundle that can be scored, archived, and replayed [74], [44], [94], [58]. We model the
review bundle as R = ∆, Pre, Checks, Canary, Rollback, Ptrs(E) , (18) where ∆ is the proposed change (diff or action plan), Pre are preconditions, Checks are required validations (policy-as-code, invariants, sandbox), Canary is the staged rollout plan, Rollback defines triggers and steps, and Ptrs(E) are evidence pointers into the trace [1], [3], [76], [77], [94]. Figure 7 sketches this review flow, which is the practical interface for accountability [58], [57].

Fig. 7: What a human approves in a guarded planner–executor system. The review object is a bundle R, not prose: diff, preconditions, checks, canary plan, rollback triggers, and evidence pointers into E.

E. Failure modes by rung Autonomy changes which failures dominate [74], [66], [57]. At low rungs, the main damage is wasted time and misleading explanations. At high rungs, the main damage is unsafe change and hard-to-audit action [92], [58].

19

We summarise this shift with a heatmap (Figure 8) across common risks: hallucination or story mode, tool misuse, prompt injection via artefacts, drift between verified and realised state, and rollback failure [70], [87], [72], [50], [94]. The heatmap is intended as an illustrative summary rather than an empirical measurement. Cell intensities are assigned qualitatively from each rung’s capability set Tk = Tkread ∪ Tkpropose ∪ Tkexecute and gate strength Gk : risks driven by write capability (tool misuse, drift, rollback failure) increase when Tkexecute ̸= ∅, while strong, non-bypassable gates reduce the likelihood that these risks propagate into executed actions [1], [3], [64], [65], [92]. The qualitative takeaway is that gating and verification are not optional extras at higher rungs. They are binding constraints once Tkexecute is non-empty [57], [58]. F. Pattern summaries tied back to the contract The following pattern is designed to align with the contract object (Tk , Gk , E, Π, I). a) Pattern A: Co-pilot (Read-only): The Co-pilot pattern improves operator throughput by compressing and navigating evidence. Its security depends primarily on the source, editing, and faithful referencing of the tool’s output. The measure of success should be finding evidence faster and missing fewer signals, rather than the level of polish in the summary.[8], [7], [52]. b) Pattern B: Analyst (tool-grounded diagnosis).: The Analys agent runs read queries, maintains competing hypotheses, and generates a structured root cause analysis (RCA) report containing evidence clues. The primary failure mode is the story pattern: a seemingly coherent explanation that has a weak connection to the system under investigation. Therefore, the design should require tool-based citations, contradiction checks, and explicit termination conditions, as well as escalation if uncertainty persists [9], [10], [75], [38], [39], [95]. c) Pattern C: Planner–executor (write-limited).: This is the first rung where safe autonomy becomes a property of the whole system. The agent proposes diffs and plans, while execution is constrained by least privilege and by a verification wall that checks policy and invariants [1], [30], [3], [64], [65], [60]. Success is not only whether the change was applied. It is whether the change was applied without violating checked properties, and whether the system could return quickly to a known-good state if reality objected [77], [94], [95]. d) Pattern D: Closed-loop (self-healing).: A closed-loop system aims to integrate detection, diagnosis, mitigation, and recovery verification into a continuous process. At this rung, since the system is part of the control loop, the range of action must be very limited, and rollback needs to happen automatically. Thus, the main evaluation metric should be the system’s long-term stability in the face of faults and changes, rather than the outcome of any single event [76], [77], [97], [57]. V. AGENTIC N ET O PS NetOps is a natural target for agentic systems because many operational tasks already follow a repeatable loop: (i) gather evidence from measurement and control-plane state, (ii) form a hypothesis about a failure or a policy violation, (iii) propose a change, and (iv) validate the change before rollout [101], [16], [15], [35], [44]. At the same time, NetOps has a higher safety bar than many AIOps settings because a single incorrect change can propagate quickly, trigger large blast-radius outages, or create hard-to-debug transient behaviours during convergence [26], [11], [102], [103]. Empirical studies show that operational failures in networks often arise from simple causes, including component faults, misconfigurations, and side effects from other configuration changes. Although these causes may appear simple, they are difficult to predict without systematic testing [102], [103], [104], [24], [50]. This combination of structured workflows and high consequence makes NetOps a useful stress test for agentic LLM design. The LLM is rarely the correct place to “decide truth” about network state. Instead, the LLM is most valuable as a controller of workflow: it translates intent into queries, routes evidence through verifiers, proposes minimally invasive diffs, and documents preconditions and rollback [9], [59], [60]. In other words, NetOps rewards agentic systems that are tool-first and verification-first, rather than

20

Fig. 8: Autonomy–risk coupling and the gated action envelope. As the agent moves from copilot to closed-loop control (left), the severity of key failure modes rises (right). A proposed action a is therefore routed into an execution policy: allow-listed low-risk actions may proceed only through a non-bypassable gate g(a, E, Π) with explicit checks and a rollback plan, while high-risk actions remain proposal-only by default.

21

Fig. 9: Agentic NetOps loop specialised to high-consequence change. The LLM is useful as a workflow controller: it gathers evidence, clarifies intent, proposes minimally invasive diffs, and routes every write through an explicit verification wall, staged rollout, and rollback path. conversationally fluent [1], [3], [65], [32]. This separation also leaves room for specialised, network-native control mechanisms such as NOS [105], where fast adaptation is handled by the control layer rather than by the language model itself. For example, in O-RAN scheduling, the lower-layer manages latency tails and spectrum targets. Meanwhile, higher-level systems interpret intent, select policies, and oversee auditable changes [106]. This positioning is consistent with practical NetOps verification and safe-change practice, where the hard part is not proposing a change but justifying and checking it before it is trusted [1], [30], [3], [64], [107]. A. NetOps property definitions (targets for the verification wall) To make “verification” concrete, we fix a compact model and a small palette of properties that recur across the NetOps literature [2], [64], [1], [30]. Let the network be a directed graph G = (V, E). Let x denote the realised control/forwarding state (routing adjacencies, FIB entries, ACL tables, tunnel state, etc.). For a traffic class c (e.g., 5-tuple predicate, VRF, DSCP class), let Fx (v, c) denote the next-hop relation induced by x. This induces a (possibly empty) forwarding path Pathx (s, c) obtained by iterating Fx from source s until termination (deliver, drop) or a loop. These abstractions match the intent of data-plane and control-plane analysis systems that operationalise reachability and policy reasoning from configurations [2], [1], [64], [31].

22

We use the following predicates as reusable invariants (Figure 10): Reachx (s, t, c) ≡ Pathx (s, c) terminates at t without drop or loop, (19) Isolatex (s, t, c) ≡ ¬Reachx (s, t, c), (20) Waypointx (s, t, w, c) ≡ Reachx (s, t, c) ∧ w ∈ Pathx (s, c), (21) LoopFreex (s, c) ≡ Pathx (s, c) contains no repeated node. (22) These cover the most common operational questions: “can A reach B”, “must A never reach B”, “must traffic pass through a firewall/proxy”, and “are we creating transient loops”. They also align with what configuration analysers and invariant checkers are typically asked to prove or refute at scale [1], [30], [3], [65], [17], [107].

Fig. 10: Common NetOps invariants that can form the verification wall. The point is not the notation; it is to make the targets explicit and reusable across synthesis, rollout safety, and autonomy gating.

B. Intent, ambiguity, and when clarification is mandatory Natural language intent is typically underspecified, even when expressed by experienced operators [101], [16], [15], [83]. Intent often omits operational constraints that matter for safety, such as acceptable blast radius, maintenance window, traffic class priorities, failure-domain boundaries, or whether a waypoint is “preferred” versus “mandatory”. Intent-based networking is typically conceptualised as a closed-loop system that provides continuous assurance, rather than as a single compilation step [15], [101], [16]. Prior to the advent of LLMs, research approached configuration as a synthesis problem defined by

23

explicit correctness criteria, which typically focused on translating high level policies into device-level configurations, ensuring invariants are maintained, and protocol semantics are respected [108], [109], [17], [110]. For agentic NetOps, this suggests a safe division of labour: the LLM handles ambiguity resolution and produces structured candidates, while synthesis and checking back-ends provide the authority on correctness [59], [83], [79]. Recent work makes ambiguity explicit and studies how an LLM-driven assistant should ask questions before generating configurations [83], [59]. A practical and checkable condition for mandatory clarification is that there exist multiple distinct diffs that satisfy the current checks but induce different forwarding semantics for some traffic class. Let S be the set of diffs ∆ that pass the current policy and invariant checks. We have an ambiguity witness when: ∃ ∆1 , ∆2 ∈ S, ∆1 ̸= ∆2 : ∃(s, t, c) s.t. (23) Pathx⊕∆1 (s, c) ̸= Pathx⊕∆2 (s, c). In practice, the most common sources of such witnesses are rule overlap and priority conflicts, where an operator’s intent does not specify precedence even though the configuration must [83], [17], [110]. Under these conditions, the agent should not “pick a reasonable default”. It should ask a small typed set of disambiguating questions, commit the answers to the intent record, and only then generate diffs [83], [59]. Finally, synthesis is not only about producing configs but also about supporting review and post-incident learning. Work on producing localised explanations for synthesised configurations supports this direction and fits naturally with agentic workflows where explanations are linked to specific lines, policies, and checker outputs [78], [79]. C. Update safety: correctness during rollout, not only at rest Update safety is not only about the final configuration being correct. Even correct end states can be reached through unsafe intermediate states, such as loops, blackholes, or policy violations during rollout, because updates are applied under asynchronous convergence and partial deployment [11], [26], [25]. Agentic NetOps should treat “apply change” as a protocol, not a single tool call [11], [56], [44]. Let xt denote the realised network state during rollout, as partial deployments and convergence events take effect. Transient safety requires every intermediate state to satisfy the relevant invariants: ∀t ∈ [0, T ] : I(xt ) = 1, (24) where I combines the required reachability, isolation, waypointing, and loop-freedom checks. This separates final-state correctness from rollout correctness, which is the key lesson of consistent updates and staged operations[11], [56], [111]. D. Change risk scoring and autonomy gating NetOps practitioners assess risk using criteria that align with autonomy decisions, such as the number of affected devices, novelty of the change, the adequacy of verification procedures, and the history of similar changes causing incidents[56], [44], [50]. Agentic systems benefit from making that reasoning explicit because it links verification coverage to permissioning and gating, rather than treating coverage as an afterthought [112], [57]. Let ∆ be a candidate diff (or change bundle). A minimal risk score that is easy to compute and easy to review is: Risk(∆) = α Blast(∆) + β Novelty(∆)
(25) + γ 1 − Coverage(∆) . where Blast(∆) approximates potential impact, Novelty(∆) measures distance from known-safe patterns, and Coverage(∆) measures how well tests and checks exercise the configuration lines and behaviours touched by ∆. Recent coverage work shows why this is operationally important: “we have tests” is not the

24

Fig. 11: Update as a protocol, not a tool call. Canary and staged expansion turn transient-safety requirements into observable gates, with explicit rollback triggers tied to telemetry. same as “we have assurance for the lines we have just changed” [112], [65]. In practice, when coverage is low for the touched lines, autonomy should be reduced rather than expanded [112], [56]. This connects directly to the earlier ladder-of-autonomy and gate definitions. A simple execution gate can include a risk threshold: h gk (∆, E, Π, I) = 1 PolicyCheck(∆, Π) = 1 i (26) ∧ Verify(∆, I) = 1 ∧ Risk(∆) ≤ θk . This makes ”validation barriers” an important principle. The scope of coverage and the risks decide if an agent can carry out a plan, not just if they can have a major effect [1], [3], [64], [112], [57]. E. Troubleshooting: from symptoms to hypotheses with reproducible traces Network troubleshooting is typically a search on an incomplete set of evidence. This is because signals are often noisy, telemetry data may be missing, and symptoms can deviate from the root cause due to retries, buffering, and control plane convergence. Therefore, practical application relies on a standardized workflow: collecting discriminative observations to narrow down the fault, and then validating hypotheses through targeted testing[113]. The same idea carries over to agentic NetOps. The agent should be rewarded for producing a reproducible evidence trail E, not for writing a smooth account of events. Recent dialoguedriven diagnosis in data centre networks points in the same direction: success depends on workflow discipline, tool sequencing, and explicit escalation when confidence remains low [9]. Cross-layer cases are still the most difficult. Application symptoms may originate from routing policy, transport behaviour, middlebox policy, or transient forwarding states during updates. Agentic NetOps systems should therefore represent hypotheses at several layers and test them with independent signals. A practical rule is to maintain competing hypotheses and require at least one disconfirming test before committing to any change with a broad blast radius. F. Causal inference for explainable NetOps diagnosis Causal inference provides a useful counterweight to purely LLM-centred NetOps diagnosis. Many network failures are not isolated events; they propagate through topology, protocol state, control-plane

25

dependencies, service dependencies, and time-ordered event streams [27], [29]. A causal diagnosis method aims to recover this propagation structure, so that the output is not only a ranked root cause but also an explanation path linking symptoms to candidate causes [28], [29]. Earlier NetOps work already reflects this idea. G-RCA models service-quality problems in large IP networks using dependency relationships, temporal and spatial event correlation, and reasoning logic, showing that operator-facing RCA benefits from explicit dependency models rather than opaque prediction alone [27]. Work on mining causes of network events from log data uses causal inference to go beyond co-occurrence and recover likely causal relationships among network events [28]. Later work further combines causal inference with protocol-layer and topology knowledge, reducing spurious causal edges and producing information that is more useful for troubleshooting [29]. A closely related cellular-network line of work studies causal inference for estimating or explaining the operational effect of radio-access parameter and software changes. For example, causal models have been used to estimate the impact of handover-parameter adjustments and other cellular-network configuration interventions, including transmission-power and cell-offset changes, on subsequent service and performance time series [114], [115], [116]. This is especially relevant to agentic NetOps because it shows that some root-cause and change-impact questions can be answered through structured causal models before an LLM is asked to reason over large telemetry and runbook contexts. For agentic NetOps, these methods should be treated as diagnostic back-ends rather than competitors to the agentic framing. A useful division of labour is: causal inference constructs or updates a fault graph; the LLM turns operator intent into queries, requests additional evidence, explains the causal path, and prepares a reviewable mitigation proposal [27], [28], [29]. This hybrid design has two advantages. First, it improves explainability, because the diagnosis is grounded in a causal path rather than a free-form narrative. Second, it can reduce repeated reasoning over raw telemetry and runbook context, because the causal graph narrows the candidate space before the LLM is asked to summarise, compare, or prepare an action proposal [28], [29]. G. Repair as an iterative workflow There is growing interest in combining localisation, repair, and validation into an integrated loop. Automatic Configuration Repair is illustrative: it frames misconfiguration handling as localise–fix–validate, which matches how operators actually de-risk changes during incidents [79]. This maps well to agentic design because it encourages iterative, checkable steps and makes rollback and validation central rather than decorative. H. Putting it together: why NetOps is the sharpest testbed NetOps makes agent-safety claims concrete because it offers: (i) checkable invariants, (ii) a known failure mode of transient rollout unsafety, and (iii) a principled link between test coverage and autonomy. A conservative autonomy policy then follows, with broad read access, narrow write access, and closedloop behaviour only for low-impact actions with strong verification support. This is consistent with the NetOps literature on outage causes and the limits of informal reasoning in complex control planes[102], [103], [104], [32]. VI. AGENTIC AIO PS AIOps has a richer public literature than NetOps because software services produce abundant observability data and incident work is already tool-mediated through telemetry stores, dashboards, incident history, and runbooks[37], [49], [35]. What changes with agentic LLMs is not that the system can describe an incident, rather, it is that the system can conduct the investigation as a structured workflow: select the next discriminating query, interpret partial results, maintain competing hypotheses, and draft an action proposal that is tied to evidence, approvals, and rollback [8], [10], [75], [52].

26

Fig. 12: Workflow view of agentic AIOps. The central loop is query-driven diagnosis: plan the next discriminating query, collect evidence via tools, update hypotheses, then draft mitigation proposals that are gated by policy, approvals, and rollback-ready verification. Two background facts from the systems and SRE literature explain why AIOps is a natural home for agents. First, modern production diagnosis is fundamentally query-driven, and good teams treat incident response as repeated hypothesis testing under partial observability [4], [63], [73]. Second, tracing and analysis stacks in large organisations are designed to support rapid, iterative diagnosis at scale, because static dashboards are not enough [41], [62], [40], [37]. Agentic AIOps should therefore be judged by whether it respects this operational reality: tool-first investigation, not free-form explanation. Figure 12 captures the AIOps lifecycle and highlights where tool-grounded RCA and mitigation planning must be coupled to explicit gating and rollback [63], [41], [8], [10], [75], [35], [57]. a) AIOps invariants.: We can relate AIOps with NetOps verification by expressing AIOps mitigation safety as formal conditions over the current service state. Let st represent the observed state at time t, which includes SLOs, error budget, dependency health, replica status, rollout state, and tenant impact. A candidate action at is acceptable only if the relevant service conditions are satisfied: ϕSLO (st , at ) Pr[Lp99 (st+1 ) ≤ Lmax ] ≥ 1 − ϵ, (27) ϕbudget (st , at ) Berr (t + 1) ≥ Bmin , (28) ϕblast (st , at ) |Uaffected (at )| ≤ βmax , (29) t+1 ϕdep (st , at ) Reachable(Gdep , Scritical ) = true. (30) Together, these conditions address latency, error budget, blast radius, and dependency health. This concept goes beyond just log analysis or ticket handling. Once an agent suggests an action, the system must verify that service-level requirements are still met before the action can proceed. A. A causal framing for RCA under partial observability AIOps RCA is rarely a single-step prediction problem. It is better modelled as ranking and testing over a dependency structure, under incomplete sensors and changing context [33], [117], [118], [39].

27

Let the service dependency graph be Gs = (S, E), (31) where nodes u ∈ S are components (services, databases, queues, hosts), and edges encode dependencies (RPC, data, or resource coupling). Given an incident, the agent observes multi-modal evidence through tools: metrics, traces, logs, and recent change context. A compact way to express RCA is as a ranking problem:
score(u) = f metrics, traces, logs, change(u) , (32) where the agent returns a top-k list along with the evidence for each of the candidate. This framing aligns well with how production debugging systems are used in practice, through writing discriminating query, refine the candidate set, and then write the next query [63], [41], [33], [34]. Some recent studies have provided stronger evidence for this argument by clarifying causal structures. These systems learn or construct event-based causal graphs and use these graphs to generate interpretable root cause analysis (RCA) candidate models. They do this rather than simply relying on seemingly convincing correlations[117], [118], [119], [39]. For agentic AIOps, the architectural lesson is direct. The ”answer” cannot be found without using tools to collect evidence, nor can it be found without testing to rule out other explanations [10], [75], [58]. B. Log understanding and anomaly detection with LLMs Logs remain the most common operational signal, and also one of the least forgiving [46], [47], [120], [121]. They are verbose, uneven across systems, and liable to drift as code, dependencies, and deployment habits change. For this reason, AIOps pipelines rarely work with raw logs in their original form. They usually introduce at least three intermediate steps: parsing or templating, which turns free text into structured events, aggregation, which forms sequences or distributions at service, host, or request level, and detection or scoring, which identifies anomalies that deserve escalation [46], [122], [47], [48]. a) What LLMs change, and what they do not.: A sensible way to place LLMs in log-based AIOps is to treat them as semantic adapters, rather than as replacements for classical detectors. Deep sequence models such as DeepLog showed that event-sequence modelling can detect anomalies without hand-written rules [47]. In real deployments, however, this earlier promise depends on rather plain conditions: good log quality, stable parsing, manageable workload shift, and templates that do not change faster than the detector can learn them [120], [122], [48]. Practitioner studies sharpen the point. Operators need methods that remain useful under drift, produce evidence-linked outputs, and degrade safely when signals are missing [121]. LLMs can help here by mapping heterogeneous log language into more stable schemas, condensing long bursts into incidentfocused summaries, and suggesting the next query or filter needed to test a hypothesis [8], [52], [10]. This is useful only when the system constrains the output format and preserves provenance [88], [57]. If an LLM summary cannot be traced back to log lines and timestamps, it becomes a familiar operational nuisance: neat enough to circulate, and unsafe when someone has to act on it [69], [58]. b) Parsing and benchmarking remain foundational.: Because parsing errors cascade into every downstream step, log parsing and its evaluation remain first-class background. LogPai’s work on tools and benchmarks for automated log parsing helped standardise evaluation practice and made clear that parser choice can dominate downstream quality [122]. LogHub consolidated commonly used datasets into a shared collection, enabling more reproducible comparisons across systems and domains [48]. For agentic AIOps, the implication is direct: if the agent is trained or evaluated on cleanly parsed logs, but deployed on noisy and drifting templates, autonomy will be miscalibrated [120], [121]. This argues for explicitly reporting (i) parsing quality, (ii) drift sensitivity, and (iii) behaviour when parsers disagree [122], [48]. C. Incident triage and RCA with tool-grounded agents Incident response has an implicit structure that is easy to state and hard to operationalise: detect, triage, diagnose, mitigate, and learn [35], [45]. Agentic systems add value in the gaps between these stages,

28

where humans spend time translating between representations: ticket text to telemetry queries, telemetry results to hypotheses, hypotheses to mitigations, mitigations to change records [8], [10], [75], [52]. a) From natural language to telemetry queries.: A high-leverage capability is converting incident descriptions into concrete telemetry queries. Xpert studies the role of query writing in a large-scale incident workflow and proposes LLM-supported query recommendations to reduce time-to-signal [8]. This is a key agentic building block because it reframes “LLM reasoning” as evidence acquisition. If the system cannot reliably propose the next useful query, it will default to narrative [8], [7], [68]. b) Evidence fidelity beats narrative plausibility.: LLM-driven RCA systems increasingly emphasise that explanations must be grounded in collected diagnostic evidence. RCACopilot is a clear example: it couples incident-specific data collection workflows with an LLM component for category prediction and explanation, evaluated on real incidents and deployed in practice [10]. Two lessons matter here. First, data collection is the anchor that keeps the model inside the observed world. Second, the explanation is operationally meaningful only when accompanied by a trace of what was inspected and why [10], [88], [58]. c) Agents and tool-augmented RCA.: A parallel line of work frames RCA as an autonomous, tool-augmented agent problem. RCAgent uses tool-augmented action trajectories for data collection and analysis, and evaluates across multiple RCA targets (root cause, solution, evidence, ownership) [75]. For AIOps, the point is that RCA quality depends on tool sequencing and stopping rules, not only on language [75], [66], [68]. An agent that cannot stop when evidence is insufficient will burn budget on noisy queries and eventually “decide” based on fragile cues [75], [57]. D. Mitigation planning under a constrained action envelope Once an agent proposes mitigations (restart, failover, throttling, config toggle), the system becomes a socio-technical control loop. The failure mode is no longer wrong answer but wrong change at the wrong time [4], [55], [97], [74]. Safe autonomy therefore begins by formalising what actions are even on the table [57], [58]. A useful example is queue-aware streaming intrusion detection [123], where online evidence accumulation triggers bounded mitigation rather than open-ended configuration change. Such designs illustrate the kind of controlled action surface that agentic NetOps and AIOps systems should expose to LLM planners. The distributed nature of modern services limits the flexibility of AIOps operations. Restarts, failovers, replica changes, traffic shifts, and rollbacks can involve multiple components, and their security depends on consistency, replication, coordination, and recovery rules [124], [125]. Stateless frontend failovers are generally low-risk. However, replaying logs, or changing leaders can disrupt ordering, persistence, or client-visible consistency. Therefore, AIOps systems must consider state, arbitration, and leader election rules and should avoid remedial operations that could compromise the required protocols. Let A be the permitted mitigation set, and partition it into low-risk versus high-risk actions: A = Alow ∪ Ahigh , (33) where Alow includes actions with well-tested rollback paths, while Ahigh includes changes that alter correctness assumptions or have long-tailed side effects (schema migrations, broad policy changes, permanent config edits). This partition should be organisation specific and must be part of policy Π, it should not be left to prompt text [92], [57]. Mitigation selection can then be stated as a constrained optimisation problem defined over possible actions:   ⋆ a = arg min Impact(a) + λ Risk(a) s.t. g(a, E, Π) = 1, (34) a∈A

where E is the collected evidence trace and g(·) is the non-bypassable gate that enforces policy, approvals, and required checks. This makes the intended separation explicit; which means that the agent proposes, but the system decides what is permitted and when [76], [77], [57], [58]. E. Workflow artefacts as adversarial inputs AIOps depends heavily on workflow artefacts, such as, tickets, runbooks, on-call notes, postmortems, and chat transcripts [45]. These artefacts are valuable because they record local practice, but they are also

29

untrusted input channels. They may be stale, incomplete, or may also contain malicious instructions. For that reason, prompt injection and integrity threats are not peripheral “LLM application” issues, rather, they belong to the main design surface [69], [126], [87], [70], [93]. Therefore, in agentic AIOps, operational rules need to be strict, for example, artefacts should not directly trigger privileged actions [87], [57]. Particularly, in a tool-first design, the agent might see these artefacts as leads, so these leads must be confirmed or rejected using evidence from the tools before suggesting any fixes [10], [75], [88]. This requirement should also show up in evaluations. Benchmarks created only from clear incident descriptions are likely to exaggerate safe autonomy [66], [68]. F. Datasets and benchmarks for agentic AIOps Advancement in the field has been limited by the reliance on proprietary incident corpora, heterogeneous telemetry infrastructures, and insufficiently robust ground truth [127], [18]. While public datasets contribute to progress, agentic evaluation imposes more rigorous requirements thosetypically are missing from them. Agentic evaluation necessitates not only labeled incidents but also authentic tool interfaces and success criteria that align with real-world operational practices [66], [68], [57]. a) Logs: shared corpora and realism.: LogHub facilitates more reproducible evaluations [48]; however, reproducibility does not equate to realism. Empirical investigations into practical log anomaly detection highlight challenges such as missing events, evolving templates, and ambiguous labels [120]. Studies involving practitioners similarly emphasise that benchmarks should prioritise robustness and actionable outcomes rather than focusing solely on accuracy [121]. In the context of agentic AIOps, these findings support the development of benchmark variants that incorporate data drift, partial observability, and noisy metadata, such as incorrect tickets and outdated runbooks [120], [121], [68]. . b) Traces and multi-modality.: Trace-based RCA benchmarks capture causal structure more directly. TraceRCA provides a benchmark for microservice root cause localisation based on traces and related signals [119]. TADBench targets trace anomaly detection, which is directly relevant when agents must decide whether an anomaly warrants escalation or mitigation [128]. Causal and event-graph RCA work suggests a direction for richer ground truth: not only “which service”, but also “which event chain and which tests disconfirmed alternatives” [117], [118], [39], [38]. c) Agentic benchmark requirements (tool interfaces are not optional).: A practical way to state benchmark requirements is to treat an agentic AIOps benchmark as an environment: B = (D, T , Π, A, M), (35) where D provides multi-modal observations, T exposes tool calls (queries and controlled actuators), Π encodes policy and approvals, A is the permitted action envelope, and M defines metrics (accuracy, time-to-diagnosis, cost, and safety violations). Without T and Π, evaluation collapses into story quality [66], [68], [67], [20]. Emerging agent benchmarks for IT automation are starting to move in this direction by including realistic tasks and interfaces rather than only static text inputs [66], [68]. For AIOps specifically, the missing pieces are still common: (i) tool-call traces and intermediate hypotheses, (ii) explicit action outcome simulation (success, regression, rollback), and (iii) scoring that penalises unsafe actions even when the final diagnosis is correct [127], [57], [58]. These requirements connect directly to Section III: safe autonomy is only measurable when tools, gates, and action envelopes are part of the benchmark [57], [66], [68]. VII. E VALUATION AND BENCHMARKING Evaluation is where many agentic operations papers still under-specify the claim [129], [130], [131]. The operational objective is not only to be correct, but to be correct under constraints: partial observability, drift, heterogeneous tool surfaces, and strict risk controls on actions [120], [132], [133], [134], [135]. As a result, evaluation must move beyond static question answering and towards workflow performance:

30

TABLE III: Representative systems and what they contribute (illustrative, not exhaustive) [35], [18], [66]. System

Domain

Core idea

Notes / limits

DeepLog [47], [120]

AIOps (logs)

Xpert [8], [35]

AIOps (triage/diag.)

RCACopilot [10], [88]

AIOps (RCA)

RCAgent [75], [66]

AIOps (RCA agents)

Sequence modelling for log anomaly detection [47] LLM-assisted telemetry query recommendation for incident workflows [8] Tool-driven diagnostic data collection + LLM category prediction and explanations [10] Autonomous, tool-augmented agent trajectories for RCA subtasks [75]

TraceRCA [119], [39]

AIOps (traces)

TADBench [128], [37]

AIOps (traces)

Strong baseline; realism depends on parsing quality and drift handling [120], [121], [122]. Shifts focus to evidence acquisition; quality tied to DSL/tool schema and feedback loops [8], [68]. Highlights that workflows are the anchor; narrative must remain evidence-linked and auditable [10], [58]. Raises evaluation needs: stopping rules, tool budgets, and safety constraints beyond accuracy [75], [68], [57]. Trace-based root cause localisation Useful for causal structure; still needs richer benchmark [119] agent traces and action outcomes for autonomy [117], [118]. Benchmark and empirical study for Important for escalation policies; benchmark trace anomaly detection [128] design should include drift and partial observability [128], [68].

TABLE IV: Task taxonomy for agentic NetOps and AIOps, stated at the tool and action level [129], [130], [136]. Task

Inputs

Tools (surface)

Success criteria (operational)

Incident triage & routing

alerts, tickets, short context

Evidence acquisition (querying) Root cause analysis (RCA)

incident description, symptoms logs/metrics/traces, change history

Remediation planning

runbooks, policy, failure domain info

Net intent-to-config synthesis

intent, inventory, topology

Change safety & rollout control

proposed diff, risk context

ticketing, CMDB/service correct ownership/team, low time-to-first-meaningful-action, no map, on-call rota [35], privacy leaks [51], [69] [52] log search, metrics DSL, discriminating queries early, bounded tool budget, reproducible trace queries [8], [131] query trail [8], [129], [130] correlation, dependency correct root cause in top-k, evidence-linked explanation, graph, repo metadata [10], calibrated uncertainty [10], [75], [96] [75], [117], [118] change API wrappers, safe plan as diffs/typed steps, explicit preconditions, rollback feature flags, runbook trigger defined [10], [137] retriever [10], [137] config generators, correct diff, invariants hold under failures, ambiguity resolved compilers, verifiers [1], before synthesis [1], [30], [83] [30], [83] sandbox, canary safe intermediate states, bounded blast radius, automatic controller, rollback hooks rollback on guardrail breach [11], [56] [11], [56], [136]

what evidence was gathered, whether the agent stopped at the right time, and whether any proposed or executed change respected policy and rollback [75], [129], [130], [136]. This reflects a long-standing lesson from production reliability engineering, where trusted systems leave artefacts that can be reviewed, act cautiously under uncertainty, and recover safely when events do not follow the plan [55], [11], [97], [4]. Table IV sets out the terminology by defining the tasks, their tool surfaces, and the meaning of operational success in each case. Table V then makes the metrics explicit, and Figure 13 gives a minimal evaluation report card that every agentic operations paper should include. Finally, we show how to score an agent trace directly, so that process quality becomes measurable rather than merely asserted [129], [130], [131] (Figure 14).

31

TABLE V: Metrics glossary for agentic NetOps/AIOps. Definitions are stated so papers report the same objects, even when tool stacks differ [96], [135], [134]. Metric

Definition (what to compute)

MTTD (proxy)

Time from incident start to first correct triage signal or first Measures time-to-signal, not prose quality [8], [35] discriminating query Time to verified mitigation in sandbox or replay, or time to Matches workflow reality: fix is gated and observable [55], approved plan in human-gated setting [4], [11] Root cause in top-k candidate list (with a fixed ground-truth Reflects shortlist inspection behaviour [119], [38] mapping policy) Ranking quality for RCA candidates or next-step Distinguishes “first useful” from “eventually found” [8], recommendations [75], [129] Count or rate of attempts to use forbidden tools, scopes, or Safety is measurable and should be reported explicitly [92], unsafe actions [131], [137] Fraction of rollouts that trigger rollback under guardrail Captures real-world dynamics and recovery readiness [11], breach [97] Actions taken that do not improve outcome (or violate “least Noise increases on-call load and risk [74], [35] change” rules) P Budgets bind autonomy in practice [20], [21], [131] i c(τi ) over tool calls τi (tokens, API cost, retries) End-to-end wall-clock time (tool time included), reported with Determines operational usability [136], [139] percentiles How often the agent refuses when evidence is insufficient, Separates safe caution from paralysis [129], [138] versus when evidence is sufficient P |B | ECE = b nb acc(Bb ) − conf(Bb ) “Wrong with confidence” is operationally dangerous [96] Score degradation under controlled drift: missing telemetry, Drift is the default case in operations [120], [132], [133], template shifts, stale runbooks [135] Whether the returned explanation contains a valid symptom- Distinguishes causal explanation from plausible narrative to-cause path, supported edges, and disconfirming tests . [28], [29], [117]. Cost split across telemetry queries, causal-graph construcSeparates lightweight causal diagnosis from expensive tion/update, retrieval, and LLM calls. repeated model reasoning, and makes the cost of tool use and agentic reasoning explicit [29], [100], [131].

MTTR (proxy) RCA@k MRR/nDCG PolicyViolations RollbackRate UnnecessaryActions ToolCost Latency Abstention quality ECE Drift robustness CausalPathQuality ComputeBreakdown

Why it matters

A. Metrics glossary and a consistent scoring vector Operational success is inherently multi-objective and should be represented as a vector rather than a single aggregate score [129], [131], [138]. A compact form that covers both diagnosis and action is as follows: m = RCA@k, MRR, Violations,
(36) Rollbacks, ToolCost, Latency . The key is that some dimensions are “hard” constraints (policy violations should be near zero), while others are trade-offs (tool cost versus time-to-signal). For hybrid causal–LLM systems, evaluation should also report where computation is spent: causal-graph construction or update, telemetry querying, retrieval, and LLM calls. This distinction matters because a causal model may amortise diagnostic cost across incidents, while an LLM-heavy design may repeatedly consume context and compute for each investigation. Explainability should likewise be scored separately from answer fluency, for example by checking whether the system returns a causal path, the evidence supporting each edge, and the tests that would falsify the proposed cause. A simple stop-decision metric that is easy to report is: StopScore = Pr(stop | insufficient evidence) − Pr(stop | sufficient evidence). (37) This directly penalises agents that conclude too early, while rewarding agents that stop when they should [75], [129]. B. A standard evaluation report card (what every paper should report) Many results are hard to compare because papers omit the operational interface. Figure 13 is a minimal report card: datasets and splits, tool surfaces, budgets, gates, trace logging, drift tests, and cost and latency [135], [134], [129]. It is intentionally boring, because boring is what lets the field make scientific progress [133], [132], [135].

32

Fig. 13: Evaluation ladder with matched reporting burden. Left: an evaluation ladder from offline corpora to canary-in-production studies, where realism and operational risk increase. Right: the minimum information that must be reported to make claims at each rung (tasks/data, tool surface, budgets, gates, trace logging, robustness). The connector highlights that moving down the ladder requires stricter evidence, not just better headline scores.

33

C. Worked example: scoring an agent trace (process quality is observable) Let an agent produce a trace E = (τ1 , . . . , τn ) where each τi is a tool call, a retrieved artefact, or a proposed action. A practical trace score decomposes into: (i) discriminating value, (ii) policy compliance, and (iii) budget use [129], [131], [130]. One minimal form is: n   X TraceScore(E) = Gain(τi ) − λc(τi ) − µI{τi violates policy} , (38) i=1

paired with outcome metrics such as RCA@k and RollbackRate. Here Gain(τi ) can be instantiated as “reduces hypothesis entropy” or “matches a gold evidence trace”, depending on the benchmark. The important point is that policy violations and budget overruns are no longer hidden behind a fluent final paragraph [129], [137]. D. Offline evaluation: datasets, metrics, and protocols Offline evaluation is necessary, but it must be designed to avoid over-claiming [135], [134], [129]. a) (i) Leakage and contamination.: Incident corpora contain near-duplicates (recurring alerts, repeated signatures, repeated playbooks). At minimum, splits should be time-aware and duplicate-aware, with overlap checks reported. This matters even more for LLM systems because pretraining contamination can silently inflate benchmark performance, and contamination checks are now part of mainstream LLM evaluation practice [140], [23], [134]. b) (ii) Metrics must match operational intent.: For RCA, top-k accuracy is useful, but ranking metrics (MRR, nDCG) often match engineering practice better. For action planning, report policy violations, unnecessary actions, and rollback completeness. For confidence gating and abstention, calibration matters, because wrong-but-confident agents create the worst operational incidents [96], [129]. c) (iii) Tool interfaces and traces are first-class.: Agentic ops evaluation should not treat tools as an implementation detail. Benchmarks for tool use and long-horizon control provide useful templates for reporting tool schemas, trajectories, and budgets [19], [20], [21], [131], [141], [139]. A pragmatic protocol is to run offline evaluation in a stubbed tool environment with deterministic responses, then score both outcome and trace (Equation 38) [130], [129]. d) Robustness under drift and partial observability.: Operational signals drift continuously. Benchmarking should include stress variants: missing telemetry, delayed telemetry, contradictory signals, and evolving templates [120], [135], [134]. If an agent becomes more autonomous under these stressors, it is usually a sign that the evaluation is too forgiving. e) LLM-as-a-judge is not enough.: LLM graders can help with structure and readability, but they are not a substitute for tool-grounded correctness and safety scoring. Use LLM judges as an auxiliary signal, never as the only metric [140], [135]. E. Online evaluation: sandboxes, canaries, and rollback tests Online evaluation becomes necessary once an agent proposes actions, because action quality depends on the behaviour of the system being changed [97], [136], [139]. A useful first stage is incident replay, where telemetry queries are real but actions are simulated. This fits NetOps pre-change verification pipelines [1], [30], and AIOps replay over frozen trace and log stores when the agent also leaves an auditable evidence trail [10], [75], [129]. A later stage is then a controlled rollout. A change may be correct in its final state yet unsafe while it is being applied, so the agent should stage the change, run a canary, validate the result, and only then widen the rollout [11], [56], [136]. a) Guardrail-triggered rollback as an evaluation primitive.: An evaluation that cannot tell whether the agent would have rolled back is not yet measuring safe autonomy. A stronger protocol specifies the guardrails, the telemetry used to detect breaches, the rollback action, and the allowed response time [97], [137]. Rollback aware learning further indicates that reversibility should be incorporated into the action model [142].

34

Fig. 14: Process quality is scorable. A trace is evaluated by discriminating value (did the query narrow hypotheses), policy compliance (did it attempt forbidden actions), and rollout discipline (were guardrails and rollback triggers specified). b) Fault injection and chaos-style tests.: Agents should be evaluated on controlled fault scenarios, because real incidents are rare, costly, and unevenly observed. Chaos engineering provides a disciplined way to inject failures and test whether detection and mitigation workflows behave as expected [97]. F. Benchmarks for RCA and operations It shapes the scientific claim itself, as discussed earlier [129], [130], [136]. For AIOps, useful anchors include shared log corpora and trace-based RCA datasets [48], [119]. In NetOps, configuration analysis and verification form the foundation. This ecosystem allows operators to check network invariants and test changes in a repeatable way, which helps ensure that updates do not introduce unexpected problems[1], [30], [112]. Agentic evaluation introduces two requirements often missing from existing benchmarks. These requirements reflect the growing practical needs arising from the evolution of evaluation methodologies. First, the tool interface should be clearly defined. Second, it should be able to acquire intermediate supervision information, such as by recovering evidence traces from task setup methods[129], [131], [130], [138]. When these are absent, a plausible narrative can too easily stand in for a sound investigation. G. Cost, latency, and operational usability In practice, agentic systems often face limitations due to excessive tool invocations, retries, and the use of verbose context prompts. These factors are typically more critical than the performance of any

35

single model invocation. Therefore, evaluations should encompass not only end-to-end latency but also the number of tool invocations, failure rate, token usage, cost estimates, and the time required for operator involvement[136], [139], [141]. Usability should be judged through the artefacts operators actually use: diffs, evidence trails, guardrail monitors, and rollback plans[4], [55], [129]. VIII. S ECURITY, PRIVACY, SAFETY, AND GOVERNANCE The introduction of agent operations has altered the risk profile of NetOps and AIOps. Large language models are no longer limited to generating text; they can now interact with the envi- ronment in a more direct way. Agents can now coordinate access to privileged tools, propose and even implement changes, and choose which evidence to rely on. This is a stark contrast to their previous, more passive role. Previous assumptions led to a classic agent obfuscation problem: attackers didn’t need to compromise the tool itself, but only needed to influence authorised agents to abuse it[92], [70], [143], [144]. A sound engineering response begins with architectural choice. This approach determines how the system will operate in the real-world environment and lays the foundation for subsequent improvements. In practice, this means treating operational artifacts as potential adversaries from the outset. The system should enforce the principle of least privilege at every tool boundary, requiring all operations to pass through independent approval gates that the model itself cannot bypass. As complexity increases, these measures help control risk and maintain control[71], [145], [146]. Figure 15 makes the trust boundaries explicit. We then map threats to controls and to measurable tests in Table VI, and state a concrete evaluation protocol for security claims in agentic operations systems in Section VIII-G. This structure is deliberate. Here, security is treated as a programme of enforceable controls and falsifiable tests, rather than as a catalogue of hazards [144], [143]. A. Attacker model, assets, and trust boundaries A usable threat model for agentic operations should name assets, attacker capabilities, and the concrete surfaces where those capabilities apply [70], [144]. a) Assets at risk.: Beyond confidentiality of secrets in logs and tickets, the primary assets are: (i) integrity of operational truth (what is happening in the system), (ii) integrity of changes (what was applied, where, and why), and (iii) availability of the workflow (can on-call staff still diagnose and act). In NetOps, a wrong config or unsafe rollout can create wide blast-radius outages. In AIOps, the more common failure is accelerating a bad mitigation, or delaying a correct mitigation through repeated refusal or mis-triage [72], [86]. b) Attacker capabilities and goals.: We model an attacker by the tuple A = (C, G, S), where C is the set of capabilities, G is the set of goals, and S is the set of reachable surfaces. Capabilities commonly include: (1) writing or influencing text that enters prompts (tickets, chat, wiki pages), (2) inserting or modifying documents in retrieval stores (poisoning or jamming), (3) influencing telemetry strings (user-controlled payloads that appear in logs), and (4) inducing unsafe tool use by shaping the agent’s context. Goals include unsafe action, denial of workflow (refusal storms), and data exfiltration. These are now well evidenced in prompt-injection and RAG-security work [70], [85], [86], [143], [147]. c) Trust boundaries in agentic ops.: Operationally, it helps to split the system into four zones: 1) Untrusted inputs: tickets, external docs, user-influenced log lines, chat transcripts. 2) Semi-trusted evidence stores: internal runbooks, postmortems, baselines; high value, but stale or poisonable. 3) Privileged tools: change APIs, controller interfaces, restart and drain actions, ticketing actions. 4) Verification and governance: policy-as-code, validators, simulators, approvals, tamper-evident auditing. The design requirement is simple: zone (4) must be non-bypassable by the model [92], [144].

36

Fig. 15: Threat model for agentic NetOps and AIOps. Untrusted artefacts (tickets, runbooks, dashboards, telemetry strings) can inject instructions or distort evidence. The main control is a non-bypassable boundary between reasoning and action: least-privilege tools, policy and invariant gates, verification (including crosssignal checks), and audit trails that make decisions reviewable.

37

TABLE VI: Threat-to-control matrix for agentic operations. Controls must be enforceable at the tool boundary, and they should come with measurable tests (attack success, unsafe tool calls, bypass attempts, and refusal storms). Threat class

Typical surface and goal

Primary controls (enforceable)

Prompt injection (direct / indirect)

Injected instructions in tickets, runbooks, or chats that steer tool use or bypass policy

Instruction–data separation; contextbuilder sanitisation; tool allow-lists; nonbypassable gates

RAG poisoning (targeted)

Corrupt runbook snippets or “known fix” documents that induce a wrong change proposal

RAG jamming / denial

Blocker documents induce refusal loops, slow the workflow, or stop incident handling altogether

Telemetry manipulation (integrity)

Single-channel dishonest signals steer diagnosis and mitigation

Tool misuse and excessive agency

Wrong tool, wrong arguments, unsafe sequencing, or broad blast radius

Data exfiltration

Prompt or tool output coerces disclosure of secrets or memorised data

Measurable tests

Attack success rate; unsafe tool-call rate; bypass attempts blocked [70], [71], [143], [147], [145], [146] Provenance and allow-lists; signed or Targeted answer-flip rate; reviewed documents; retrieval filters by source-trust violation owner and freshness; evidence citation rate; retrieval provenance coverage [85] Refusal budget; fallback retrieval; Refusal-storm rate; timedocument-level risk scoring; safe degraded to-first-meaningful-action; mode false-positive refusal rate [86] Cross-signal consistency checks; Cross-signal inconsistency independent measurements; “trust but detection rate; wrongverify” for each claim mitigation rate under adversarial signals [72] Least privilege by tool scope; typed tools; Forbidden-call rate; staged execution; approvals for writes argument-safety violations; rollback completeness and timeliness [144] Pre-model redaction; role-based retrieval; Secret-leak rate; policy output filtering; tight audit on data access violations on data access; retention and deletion compliance [148], [149], [150]

B. Threats mapped to controls and measurable tests Table VI links the most common threat classes to concrete controls and to tests that can be run in evaluation. C. Telemetry integrity via cross-signal consistency In operations, failures of integrity often matter more than failures of confidentiality. An agent may still act wrongly, even when it is grounded, if the evidence on which it relies is false or incomplete [72]. Telemetry is not a neutral view of the system. Sampling policies, missing instrumentation, device-side manipulation, delayed reports, aggregation choices, and model drift over time can all distort the evidence seen by an agent [37], [120], [132], [133]. In adversarial settings, compromised or user-controlled devices may inject misleading log fields, counters, or protocol events that bias diagnosis and mitigation [72], [69]. In non-adversarial settings, sparse sampling and stale models can create similar effects. Agentic systems should therefore treat telemetry as evidence with provenance, coverage, and uncertainty, rather than as ground truth. We formalise a minimal cross-signal consistency check by extracting features from three independent channels (metrics, traces, logs) over a time-aligned window: ϕm = Φm (ometrics ), ϕt = Φt (otraces ), ϕℓ = Φℓ (ologs ). (39) We then define a consistency predicate   Consistent(ometrics , otraces , ologs ) = ⊮ d(ϕm , ϕt , ϕℓ ) ≤ δ , (40)

38

where d(·) can be a simple max-deviation or a learned disagreement score, and δ is a policy threshold. This predicate is not “truth”. It is a cheap integrity guard that forces high-impact actions to wait for corroboration, or to downshift autonomy when signals disagree [72]. D. Non-bypassable policy gates and the propose–commit split Security and safety become enforceable once every write action must pass a gate that the model cannot bypass. We model the gate as   g(a, E, Π) = ⊮ Π(a) = 1 ∧ I(E) = 1 , (41) where a is the proposed action (or diff), E is the evidence trace (tool calls and outputs), Π is policy (who can do what, when), and I(E) is a verifier that checks invariants and integrity conditions (including crosssignal consistency). This results in two explicit design choices explicit: (i) the LLM proposes but does not commit, and (ii) evidence and verification are first-class inputs to approval. The OWASP “excessive agency” and “insecure tool design” risks are, in practice, failures to implement this split cleanly [144]. E. Privacy, retention, and compliance as system properties Operational data frequently contains secrets, customer identifiers, and sensitive architecture details. Agentic systems increase exposure because sensitive data can appear in retrieval context, intermediate tool outputs, and audit traces [149], [150]. a) Privacy beyond leakage.: The privacy risks of operational agents go far beyond the accidental disclosure of secrets. Telemetry may include user identifiers, location traces, device behaviour, traffic patterns, and service-usage metadata. Even when direct identifiers are removed, linkage attacks and rareevent patterns can make anonymisation weak in practice [151]. Agentic NetOps and AIOps systems should therefore minimise data before retrieval, restrict access by role and incident scope, prefer aggregated or anonymised views where suitable, and record why sensitive telemetry was accessed. Regulatory duties also matter. The system design should make retention, deletion, purpose limitation, and auditability explicit, rather than leaving them to prompt-level behaviour [61]. b) Redaction before model access.: Redaction should take place in the context builder, before data reaches the model. This includes secret scanning, role-based filtering, and tenant isolation. Post-generation filtering is too late, since the model has already processed the material [149]. c) Leakage and memorisation risks.: Even with careful retrieval, large models may disclose sensitive memorised content or be pushed towards regurgitation. Training-data extraction studies make this risk concrete [148]. For operations, the safer stance is to minimise sensitive context, avoid long-lived memory by default, and treat each incident as a retention boundary [149], [150]. F. Governance as artefacts: approvals and audit integrity Governance is strongest when it appears as enforceable artefacts, not only as policy text. a) Governance as a sociotechnical process.: Governance is strongest when it appears as enforceable artefacts, not only as policy text. b) Governance as a sociotechnical process.: Governance in agentic operations is more than a technical gate. Policy checks, approval gates, and audit trails matter, but they only work when they fit the operating process. That process includes who is on call, who may approve change, when escalation is required, and how responsibility is assigned after an incident. Human approval can fail in ordinary ways, through fatigue, misplaced trust in automation, or unclear ownership [152], [153], [154], [74]. Higher-autonomy systems should therefore make the approval path explicit, including the evidence to be reviewed and the conditions under which weak or repeated approval requests are suppressed. c) Audit trails that support forensics.: For agentic ops, the minimum audit record includes: policy version, retrieved evidence identifiers and provenance, tool calls with arguments and outputs, proposed and executed diffs, and verification results that permitted action. If audit logs are not integrity-protected, post-incident accountability collapses. Tamper-evident logging is a natural fit for this requirement [98].

39

G. Security evaluation protocols Security claims should be evaluated with explicit attacker inputs and explicit tool surfaces, not only with benign accuracy tests. A practical protocol is: (i) define the permitted tool set and policy Π, (ii) define attacker surfaces S (tickets, docs, telemetry strings), (iii) run injected and poisoned variants alongside clean variants, and (iv) report both task outcomes and security outcomes [150], [143], [144], [149]. We recommend reporting at least: attack success rate, unsafe tool-call rate, bypass attempts blocked, refusal-storm rate (false positives that halt the workflow), and cost/latency impact under attack [86], [150], [144]. IX. O PEN PROBLEMS AND RESEARCH AGENDA Agentic NetOps and AIOps sits at an awkward intersection. On one side, the tool surfaces (telemetry queries, configuration validators, deployment APIs, ticketing systems) are mature and operationally meaningful. On the other side, LLM agents are still difficult to bound, hard to evaluate under realistic drift, and easy to mislead when they treat untrusted text as instruction [155], [156], [157]. This section focuses on unresolved problems and transforms them into contract satisfaction questions. Instead of listing general goals, we identify potential areas for failure and outline specific methods for judging success. a) Autonomy levels (used throughout).: We distinguish five deployment levels: L0 read-only copilot; L1 evidence-gathering analyst (read-only tools); L2 proposal agent (writes proposals/diffs only); L3 bounded executor (allow-listed low-risk actions with canary-first rollback); L4 closed-loop self-healing under runtime invariants and continuous rollback monitoring. Cross-cutting research axes include verification, benchmarking, security, human factors, standards, and operational economics. A. Problem framing: autonomy levels and operational assurance contracts A recurring mistake is to treat autonomy as a binary choice. In practice, operations teams already implement graded autonomy through access control, approvals, change windows, and progressive rollout. A deployable research agenda should therefore make autonomy a first-class variable and report results at multiple rungs. General agent frameworks motivate the separation between reasoning and acting, but operations demands that the separation is enforceable and auditable rather than merely prompt-stated [14], [13], [155], [157]. a) Assurance contract (rung-indexed).: We define an assurance
contract at rung k as a tuple: Ck = Tk , Rk , Gk , Uk , Bk , (42) where Tk is the permitted tool surface (including any write capability), Rk is the required evidence to collect, Gk is the set of non-bypassable gates and required checks, Uk is the rollout protocol, and Bk are budgets (tool cost, latency, and action-risk budgets). We keep the same tool partition used earlier: Tk = Tkread ∪ Tkpropose ∪ Tkexecute . (43) execute The shift highlighted in Section 4 is exactly the regime Tk ̸= ∅, where gating and verification become binding constraints rather than optional hygiene [157], [158]. b) Contract satisfaction as a trace property.: Let τ denote an agent trace: retrieved artefacts (with provenance), tool calls (arguments and outputs), hypotheses, proposed diffs, and (if enabled) executed actions. We say τ |= Ck if the trace contains the mandated evidence, respects budgets, and never bypasses the gate for executed actions: Evidence(τ ) ⊇ Rk ∧ Budget(τ ) ≤ Bk , (44) execute ∧ ∀a ∈ τ ∩ Tk : g(a, E, Π) = 1, (45) ∧ Rollout(τ ) follows Uk ∧ Audit(τ ) satisfies Gk . (46) Here g(a, E, Π) ∈ {0, 1} is the same non-bypassable policy gate interface used earlier, with evidence E and policy Π. This turns “deployable autonomy” into a measurable claim: we can count missing evidence, budget overruns, and gate violations, and we can compare systems by how often they satisfy Ck under drift and adversarial inputs [159], [160].

40

B. Verification, guarantees, and compositional safety across layers and time The central technical challenge for safe autonomy is that operations properties are layered and timedependent. A NetOps change can satisfy final-state reachability and still create transient loops or blackholes during rollout. An AIOps mitigation can reduce error rate while violating tail-latency SLOs and triggering cascading retries. The research problem is therefore not only verification, but compositional safety across layers and across the rollout timeline. a) Network-level invariants remain the clearest foothold.: NetOps has a strong tradition of proactive verification and safe update reasoning. Batfish and Minesweeper show how to analyse real configurations against protocol semantics and invariants before deployment [1], [30]. Complementary systems emphasise fast, online detection of policy violations and the ability to reason about forwarding behaviour and changes at scale [3], [2], [64]. Work on consistent updates formalises the core risk: correct end states can be reached through unsafe transient states, so deployment must be treated as a protocol [11]. b) Service-level safety needs testable operational semantics.: For AIOps, safety targets are naturally expressed as SLO constraints, error budgets, and blast-radius limits, and they often require monitored rollouts and realistic failure scenarios. Large-scale systems experience shows that production failure modes are subtle, and validation must include representative workloads and fault patterns [161], [162]. Benchmark suites such as DeathStarBench provide reproducible microservice workloads that can support end-to-end evaluation of diagnosis and mitigation policies under stress [163]. c) Compositional safety in a shared invariant interface.: To stay coherent with Sections 5–7, we keep invariants as predicates over system state x. Write the overall invariant as: I(x) = Inet (x) ∧ Islo (x), (47) and reuse rollout safety as a constraint over intermediate states: ∀t ∈ [0, T ] : I(xt ) = 1. (48) Then the “layers and time” target is: Inet ∧ Islo ∧ Irollout . (49) It reports pre-change verification, rollout monitoring, and how often each layer caught what the other missed [11], [1]. C. Benchmark realism and reproducibility for contract-based evaluation Agentic evaluation fails when it becomes narrative scoring. Benchmarks must record tool use, intermediate hypotheses, budgets, and safety constraints. Without these records, they cannot separate disciplined investigation from confident guessing. General agent benchmarks help with multi-step interaction and tool use, but operations needs stricter safety and audit scoring[67], [19], [53], [164]. SWE-bench gives a related warning: evaluation changes when tasks involve real artefacts under constraints, rather than isolated question answering [23], [164]. A contract-based view gives a simple benchmark rule. Each instance should be scorable as τ |= Ck or not, with partial credit for the clauses satisfied. Broader evaluation frameworks support this earlier point, since robustness, calibration, and efficiency are not optional in high-stakes settings [100], [159], [160]. D. Hybrid causal–LLM operations A key open problem is how to combine causal diagnostic models with LLM-based agents without weakening either side. Causal models provide structured explanations, smaller hypothesis spaces, and lower repeated diagnostic cost once a dependency graph has been built or learned [27], [28], [29], [165]. LLM agents add a different strength: they can coordinate heterogeneous tools, handle operator intent, summarise evidence, and draft mitigations for review [8], [10], [75], [66], [68]. The stronger architecture is therefore unlikely to be purely causal or purely generative. Causal inference should narrow and explain the fault space, while the LLM manages evidence acquisition, operator interaction, and safe proposal generation [29], [165], [75].

41

This integration leaves several open questions. Causal graphs must remain useful under topology, workload, and service drift [120], [132], [133]. Agents also need criteria for deciding when a causal explanation is strong enough to justify a mitigation proposal [28], [29], [165]. Evaluation should go beyond root-cause ranking and measure the quality of the causal path, the evidence used to support it, and the operational cost of reaching it [27], [28], [29], [100], [131], [66], [68]. E. Security and integrity as contract clauses, not add-ons Security in agentic operations is not mainly a privacy problem. The deeper risk is confused-deputy behaviour, where untrusted content such as tickets, runbooks, or telemetry fields steers an agent that has access to privileged tools. Recent work shows that prompt injection and tool hijacking can be effective in tool-integrated applications, while prompt-only defences remain brittle [93], [166], [167], [155], [156]. Evidence poisoning creates a related risk for retrieval and telemetry pipelines, making provenance, allowlists, and integrity checks part of the control plane rather than optional hardening [168]. Structured query interfaces and strict delimitation of untrusted inputs are promising, but they remain difficult to enforce across realistic tool chains [71], [157], [158]. In the contract language, these safeguards belong inside Gk : non-bypassable gates, provenance checks, audit integrity, and measurable limits on unsafe tool calls, bypass attempts, and excessive blocking. F. Human factors, accountability, and organisational fit Agentic operations will fail in practice if they ignore how incident response actually works. On-call work is collaborative, interrupt-driven, and full of handoffs. This makes artefacts central: evidence bundles, proposed diffs, checks, ownership records, and rollback triggers. Empirical studies show that coordination and information foraging dominate real incident work [73], while trust in automation remains contextual and can be weakened by inconsistent or opaque behaviour [74]. In contract terms, the question is which Gk artefacts support fast, correct human intervention and post-incident learning, rather than merely producing persuasive prose [159]. G. Standardisation and minimal interoperable schemas Interoperability is an enabling constraint. Without shared schemas for tool calls, evidence pointers, diffs, approvals, rollbacks, and audit logs, systems remain bespoke and results remain hard to compare. There is a practical opportunity to align agent evidence collection with OpenTelemetry as a common substrate for logs, metrics, and traces [49]. For networking management, intent and telemetry standards and typed management APIs (such as gNMI) provide scaffolding for safe, auditable tool calls [101], [99], [169]. These do not solve agent safety, but they reduce accidental diversity so that traces can be replayed and compared [170]. a) Minimal schema set.: A single universal standard is unlikely in the near term. A more practical route is a small set of interoperable schemas that preserve local system design while making traces checkable, replayable, and auditable. • Tool call: tool name, version, typed arguments, timestamp, result pointer, and error code. • Evidence pointer: source ID, provenance, trust tier, freshness timestamp, and citation anchors such as query IDs, offsets, or row IDs. • Diff: structured change representation, including configuration diffs, feature-flag toggles, or deployment patches, with scope and target inventory. • Approval: approver role, decision, and reviewed bundle, including the diff, checks, and rollout plan. • Rollback: triggers, time bounds, rollback actions, and post-rollback verification queries. • Audit log: hash-chained events linking prompts, policies, retrieved evidence, tool calls, diffs, approvals, and outcomes. Together, these schemas provide the minimum structure needed to assess τ |= Ck and replay traces in reference sandboxes [170].

42

TABLE VII: Open problems for agentic NetOps/AIOps as contract-aligned research questions, each paired with a dominant failure mode and a measurable evaluation handle. Category

Research question (contract view)

Dominant failure mode

Evaluation handle

Safe autonomy & guarantees

How to choose (Rk , Gk , Bk ) so that τ |= Ck is achievable yet non-trivial?

Refusal storms or overpermissive execution

Compositional safety

How to bind Inet , Islo , Irollout into Uk and make it non-bypassable?

Pr[τ |= Ck ], gate-violation rate, refusal rate under missing telemetry [60], [83], [157] Pre-change vs. runtime catch rates; false-assurance rate [11]

Benchmark realism

Trace metrics (tool efficiency, stop quality) + outcome metrics under drift variants [67], [19], [53], [164], [159] How to reduce confused-deputy tool misuse under Unsafe tool call that Attack success rate, unsafe toolinjection and poisoning attacks? appears tool-grounded call rate, bypass attempts, overblocking rate [166], [167], [168], [71], [155], [156] What review bundles and uncertainty signals improve Over-trust or under-trust; Handover success, intervention intervention quality? poor handover appropriateness, time-to-keyevidence [73], [74], [159] What minimal schema set enables replay, auditing, Bespoke systems that Schema coverage, portability and comparison across stacks? cannot be replayed across back-ends, replay fidelity consistently [49], [101], [169], [170] How to maintain safety and calibration as tools, Silent degradation; rising Safety trends over time, driftsignals, and prompts drift? tool cost; confident trigger accuracy, cost/latency per wrong actions incident [61], [100], [160]

Security & integrity

Human factors

Interoperability

Continual ops & cost

Unsafe transient states despite a “correct” final state What benchmark artefacts are minimally sufficient to Offline over-claim; score traces, not only outcomes? brittle tool use under drift or budgets

H. Continual operation: drift, learning, governance, and cost Agent operations should be considered as a continuous process, not as a one-time deployment. Models and prompts can age, tool APIs may change, disappear, or acquire new behaviour, the signals underneath the system also drift, and all this sometimes happens quietly. Agent systems should therefore be managed as operational services, with routine evaluation, and where needed, retraining or recalibration. Governance frameworks such as the NIST AI Risk Management Framework provide a useful vocabulary for mapping risks, measuring performance, and monitoring systems over time. This vocabulary fits well with contractbased deployment and phased release practices, since both require the system boundary and its obligations to be made explicit [61]. General evaluation frameworks that track several dimensions and publish comparable artefacts are a useful starting point. For NetOps and AIOps, however, the record needs to be more operationally specific. An “agent card” should report tool scopes Tk , contract clauses, approval rules, rollback conditions, and safety performance across time [100], [160], [159]. I. Roadmap summary Table VII summarises the main research questions, linking each one to a failure mode and a suitable evaluation method. The common requirement is practical evidence. When a paper claims a higher level of autonomy, it should define what that level permits and then measure whether the system actually satisfies it. It is not enough to say that the agent seems to work. Even a clock is right twice a day, and we would not put it in charge of a production network. X. C ONCLUSION In NetOps and AIOps, agents do much more than generate operational documentation. They are involved in actual tasks while maintaining reliability, security, and accountability. In network and cloud operations, agents operate within a controlled process that links evidence with action. They read artifacts, use available

43

tools, formulate hypotheses, propose actions, and sometimes execute those actions.This stepwise approach helps maintain operational integrity. In this context, the surrounding control system matters more than the smoothness of model execution. Recent work suggests that safer agentic design depends less on allowing models to act independently and more on defining authority boundaries, tool constraints, and control mechanisms. For this reason, we treat autonomy as a ladder of operational commitments, supported by guarantee contracts and auditable artefacts. Read-only assistance, evidence gathering, proposal generation, bounded execution, and self-healing are not variants of the same capability. Each level exposes different tools, failure modes, and accountability requirements. The difference matters most at the write boundary. An agent may recommend a configuration change, mitigation, or policy update, but it should not carry that step into the running system without model agnostic checks. Small operational errors do not always remain local: a misplaced rule or premature deployment can spread into wider service failure. In NetOps, reachability, isolation, loop freedom, and update safety provide tests before change. In AIOps, the parallel discipline is diagnostic: possible causes must be tied to dependencies and observed facts. Evaluation must therefore look beyond whether the system produced the expected answer. It should show what was investigated, which tools were used, what evidence was lost, and whether the causal path can be inspected. Policy violations, latency, rollback behaviour, robustness, and auditability matter because they describe the workflow, not only the answer. A credible evaluation path should move from offline, time-aware corpora to replay, sandbox testing, and controlled online trials. Security and governance follow the same logic. Operational artefacts carry local knowledge, but they are not neutral inputs: they may be stale, sensitive, or attacker-influenced. We therefore treat trust as part of the workflow rather than as an afterthought. Prompt injection, retrieval poisoning, telemetry tampering, excessive tool use, and refusal storms are not isolated failure cases. They reveal weaknesses in how evidence, access, action, and recovery are controlled. Accountability is the other side of the same problem. If an agent operates near a control surface, its behaviour must be explainable after the event. Otherwise, the system may be useful, but it is not governable in any serious operational sense. This also limits current evidence. Public benchmarks rarely reproduce the drift, partial observability, stale documentation, tool failure, and adversarial inputs that make real operations difficult. Claims about closed-loop autonomy in NetOps and AIOps should therefore remain modest. Read-only support already has clear value, and narrow self-healing may be practical, but broad self-healing in larger and less predictable systems remains a harder step. Therefore, In this survey we adopt a contract-based view of autonomy. At each level, the system should state the permitted tools, required evidence, required checks, and arrangements for rollback and audit. Higher autonomy depends on a dependable link between evidence, permitted action, and safe recovery. Without that link, a system may look autonomous in a demonstration but remain brittle in the operational setting that matters. ACKNOWLEDGEMENTS The authors thank Hamed Haddadi for his helpful comments on earlier drafts of this survey, particularly on governance, telemetry integrity, privacy, and operational risks in agentic systems. The authors acknowledge the use of AI-assisted tools for language editing and structural refinement. The authors remain fully responsible for the technical content, citations, arguments, and final text. R EFERENCES [1] A. Fogel, S. Fung, L. Pedrosa, M. Walraed-Sullivan, R. Govindan, R. Mahajan, and T. Millstein, “A general approach to network configuration analysis,” in Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15). USENIX Association, 2015, pp. 469–483. [2] P. Kazemian, G. Varghese, and N. McKeown, “Header space analysis: Static checking for networks,” in Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2012. [3] A. Khurshid, X. Zou, W. Zhou, M. Caesar, and P. B. Godfrey, “VeriFlow: Verifying Network-Wide invariants in real time,” in 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13). USENIX Association, Apr. 2013, pp. 15–27. [4] N. R. Murphy, B. Beyer, C. Jones, and J. Petoff, Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016.

44

[5] N. Forsgren, J. Humble, and G. Kim, Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press, 2018. [6] J. Kephart and D. Chess, “The vision of autonomic computing,” Computer, vol. 36, no. 1, pp. 41–50, 2003. [7] Z. Yu, M. Ma, C. Zhang, S. Qin, Y. Kang, C. Bansal, S. Rajmohan, Y. Dang, C. Pei, D. Pei, Q. Lin, and D. Zhang, “Monitorassistant: Simplifying cloud service monitoring via large language models,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. ACM, 2024, pp. 38–49. [8] Y. Jiang, C. Zhang, S. He, Z. Yang, M. Ma, S. Qin, Y. Kang, Y. Dang, S. Rajmohan, Q. Lin, and D. Zhang, “Xpert: Empowering incident management with query recommendations via large language models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. Association for Computing Machinery, 2024. [9] H. Wang, A. Abhashkumar, C. Lin, T. Zhang, X. Gu, N. Ma, C. Wu, S. Liu, W. Zhou, Y. Dong, W. Jiang, and Y. Wang, “Netassistant: Dialogue based network diagnosis in data center networks,” in Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI ’24). USENIX Association, 2024. [10] Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, J. Zeng, S. Ghosh, X. Zhang, C. Zhang, Q. Lin, S. Rajmohan, D. Zhang, and T. Xu, “Automatic root cause analysis via large language models for cloud incidents,” in Proceedings of the 19th European Conference on Computer Systems (EuroSys ’24). ACM, 2024, pp. 674–688. [11] M. Reitblatt, N. Foster, J. Rexford, C. Schlesinger, and D. Walker, “Abstractions for network update,” in Proceedings of the ACM SIGCOMM 2012 Conference. ACM, 2012, pp. 323–334. [12] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems (NeurIPS 2020), 2020. [13] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: language models can teach themselves to use tools,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23. Red Hook, NY, USA: Curran Associates Inc., 2023. [14] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in Proceedings of the International Conference on Learning Representations, ser. ICLR 2023, 2023. [Online]. Available: https://openreview.net/forum?id=WE vluYUL-X [15] A. Leivadeas and M. Falkner, “A survey on intent-based networking,” IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 625–655, 2023. [16] M. Falkner and J. Apostolopoulos, “Intent-based networking for the enterprise: a modern network architecture,” Communications of the ACM, vol. 65, no. 11, pp. 108–117, 2022. [17] A. El-Hassany, P. Tsankov, L. Vanbever, and M. T. Vechev, “NetComplete: Practical network-wide configuration synthesis with autocompletion,” in Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2018, pp. 579–594. [18] P. Notaro, J. Cardoso, and M. Gerndt, “A survey of aiops methods for failure management,” ACM Trans. Intell. Syst. Technol., vol. 12, no. 6, 2021. [19] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig, “Webarena: A realistic web environment for building autonomous agents,” in The Twelfth International Conference on Learning Representations, 2024. [20] M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li, “API-bank: A comprehensive benchmark for tool-augmented LLMs,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 3102–3116. [21] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,” in Advances in Neural Information Processing Systems (NeurIPS 2024), vol. 37, 2024, pp. 126 544–126 565. [22] B. Tian, X. Zhang, E. Zhai, H. H. Liu, Q. Ye, C. Wang, X. Wu, Z. Ji, Y. Sang, M. Zhang et al., “Safely and automatically updating in-network acl configurations with intent language,” in Proceedings of the ACM Special Interest Group on Data Communication, 2019, pp. 214–226. [23] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, 2024. [24] R. Mahajan, D. Wetherall, and T. Anderson, “Understanding BGP misconfiguration,” in Proceedings of the ACM SIGCOMM 2002 Conference, 2002, pp. 3–16. [25] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner, “Openflow: enabling innovation in campus networks,” SIGCOMM Comput. Commun. Rev., vol. 38, no. 2, p. 69–74, 2008. [26] M. Reitblatt, N. Foster, J. Rexford, and D. Walker, “Consistent updates for software-defined networks: change you can believe in!” in Proceedings of the 10th ACM Workshop on Hot Topics in Networks, ser. HotNets-X, 2011, pp. 1–6. [27] H. Yan, L. Breslau, Z. Ge, D. Massey, D. Pei, and J. Yates, “G-rca: A generic root cause analysis platform for service quality management in large ip networks,” IEEE/ACM Transactions on Networking, vol. 20, no. 6, pp. 1734–1747, 2012. [28] S. Kobayashi, K. Otomo, K. Fukuda, and H. Esaki, “Mining causality of network events in log data,” IEEE Transactions on Network and Service Management, vol. 15, no. 1, pp. 53–67, 2018. [29] S. Kobayashi, K. Otomo, and K. Fukuda, “Causal analysis of network logs with layered protocols and topology knowledge,” in 2019 15th International Conference on Network and Service Management (CNSM), 2019, pp. 1–9. [30] R. Beckett, A. Gupta, R. Mahajan, and D. Walker, “A general approach to network configuration verification,” in Proceedings of the ACM SIGCOMM 2017 Conference, 2017, pp. 155–168. [31] N. P. Lopes, N. Bjørner, P. Godefroid, K. Jayaraman, and G. Varghese, “Checking beliefs in dynamic networks,” in Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15), 2015, pp. 499–512.

45

[32] M. Brown, A. Fogel, D. Halperin, V. Heorhiadi, R. Mahajan, and T. Millstein, “Lessons from the evolution of the batfish configuration analysis tool,” in Proceedings of the ACM SIGCOMM 2023 Conference, 2023, pp. 122–135. [33] M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. A. Brewer, “Pinpoint: Problem determination in large, dynamic internet services,” in Proceedings of the International Conference on Dependable Systems and Networks (DSN). IEEE, 2002, pp. 595–604. [34] R. Bhagwan, R. Kumar, C. S. Maddila, and A. A. Philip, “Orca: Differential bug localization in large-scale services,” in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18), 2018, pp. 493–509. [35] Z. Chen, Y. Kang, L. Li, X. Zhang, H. Zhang, H. Xu, Y. Zhou, L. Yang, J. Sun, Z. Xu et al., “Towards intelligent incident management: why we need it and how we make it,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1487–1497. [36] J. Jiang, W. Lu, J. Chen, Q. Lin, P. Zhao, Y. Kang, H. Zhang, Y. Xiong, F. Gao, Z. Xu et al., “How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20), 2020, pp. 1410–1420. [37] B. Li, X. Peng, Q. Xiang, H. Wang, T. Xie, J. Sun, and X. Liu, “Enjoy your observability: An industrial survey of microservice tracing and analysis,” Empirical Software Engineering, vol. 27, no. 1, p. 25, 2022. [38] Y. Wang, Z. Zhu, Q. Fu, Y. Ma, and P. He, “MRCA: Metric-level root cause analysis for microservices via multi-modal data,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24), 2024, pp. 1057–1068. [39] Z. Zhu, C. Lee, X. Tang, and P. He, “Hemirca: Fine-grained root cause analysis for microservices with heterogeneous data sources,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, 2024. [40] R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica, “X-trace: A pervasive network tracing framework,” in Proceedings of the 4th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’07), 2007, pp. 271–284. [41] J. Kaldor, J. Mace, M. Bejda, E. Gao, W. Kuropatwa, J. O’Neill, K. W. Ong, B. Schaller, P. Shan, B. Viscomi, V. Venkataraman, K. Veeraraghavan, and Y. J. Song, “Canopy: An end-to-end performance tracing and analysis system,” in Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP ’17), 2017, pp. 34–50. [42] R. Opdebeeck, M. Alfadel, A. Rahman, Y. Kashiwa, J. F. Ferreira, R. G. Kula, and C. D. Roover, “An empirical study of policy as code: Adoption, purpose, and maintenance,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026. [43] D. Sokolowski, D. Spielmann, and G. Salvaneschi, “Automated infrastructure as code program testing,” IEEE Transactions on Software Engineering, vol. 50, no. 6, pp. 1585–1599, 2024. [44] M. Al-Fares, V. Beauregard, K. Grant, A. Griffith, J. Hasan, C. Huang, Q. Leng, J. Li, A. Lin, Z. Liu, A. Mansy, B. Martinusen, N. Mehta, J. C. Mogul, A. Narver, A. Nigham, M. Obenberger, S. Smith, K. Steinkraus, S. Sun, E. Thiele, and A. Vahdat, “Change management in physical network lifecycle automation,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23), 2023, pp. 635–653. [45] J. Sillito and M. Pope, “Learning from lessons learned: Preliminary findings from a study of learning from failure,” in Proceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering, 2024, pp. 97–102. [46] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in 2017 IEEE International Conference on Web Services (ICWS), 2017, pp. 33–40. [47] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017, pp. 1285–1298. [48] J. Zhu, S. He, P. He, J. Liu, and M. R. Lyu, “Loghub: A large collection of system log datasets for ai-driven log analytics,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), 2023, pp. 355–366. [49] OpenTelemetry Authors, “Opentelemetry specification,” Cloud Native Computing Foundation (CNCF), 2024, accessed: 2026-02-02. [50] Y. Wu, B. Chai, Y. Li, B. Liu, J. Li, Y. Yang, and W. Jiang, “An empirical study on change-induced incidents of online service systems,” in Proceedings of the IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2023, pp. 234–245. [51] Y. Chen, X. Yang, H. Dong, X. He, H. Zhang, Q. Lin, J. Chen, P. Zhao, Y. Kang, F. Gao, Z. Xu, and D. Zhang, “Identifying linked incidents in large-scale online service systems,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020, pp. 304–314. [52] P. Las-Casas, A. G. Kumbhare, R. Fonseca, and S. Agarwal, “Llexus: an ai agent system for incident management,” SIGOPS Oper. Syst. Rev., vol. 58, no. 1, 2024. [53] Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, X. Zhou, Y. Huang, C. Xiao, C. Han, Y. R. Fung, Y. Su, H. Wang, C. Qian, R. Tian, K. Zhu, S. Liang, X. Shen, B. Xu, Z. Zhang, Y. Ye, B. Li, Z. Tang, J. Yi, Y. Zhu, Z. Dai, L. Yan, X. Cong, Y. Lu, W. Zhao, Y. Huang, J. Yan, X. Han, X. Sun, D. Li, J. Phang, C. Yang, T. Wu, H. Ji, G. Li, Z. Liu, and M. Sun, “Tool learning with foundation models,” ACM Comput. Surv., vol. 57, no. 4, Dec. 2024. [54] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” in The Twelfth International Conference on Learning Representations, 2024. [55] J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. AddisonWesley Professional, 2010. [56] O. Alipourfard, J. Gao, J. Koenig, C. Harshaw, A. Vahdat, and M. Yu, “Risk based planning of network changes in evolving data centers,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 414–429. [57] C. Autio, R. Schwartz, J. Dunietz, S. Jain, M. Stanley, E. Tabassi, P. Hall, and K. Roberts, “Artificial intelligence risk management framework: Generative artificial intelligence profile,” Tech. Rep., 2024-07-26 04:07:00 2024.

46

[58] C. C. Phiri, “Creating characteristically auditable agentic ai systems,” in Proceedings of Intelligent Robotics FAIR 2025 (IntRob ’25), 2025, pp. 1–14. [59] R. Mondal, A. Tang, R. Beckett, T. Millstein, and G. Varghese, “What do llms need to synthesize correct router configurations?” in Proceedings of the 22nd ACM Workshop on Hot Topics in Networks (HotNets ’23). Association for Computing Machinery, 2023, pp. 189–195. [60] Y. Zhou, K. Hsieh, S. K. Mani, S. Kandula, and Z. Liu, “Meshagent: Enabling reliable network management with large language models,” Proc. ACM Meas. Anal. Comput. Syst., vol. 9, no. 3, Dec. 2025. [61] E. Tabassi, “Artificial intelligence risk management framework (ai rmf 1.0),” Tech. Rep., 2023-01-26 05:01:00 2023. [62] B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag, “Dapper, a large-scale distributed systems tracing infrastructure,” Google, Inc., Tech. Rep., 2010, technical report (widely circulated). [63] J. Mace, R. Roelke, and R. Fonseca, “Pivot tracing: Dynamic causal monitoring for distributed systems,” in Proceedings of the 25th Symposium on Operating Systems Principles (SOSP ’15). Association for Computing Machinery, 2015, pp. 378–393. [64] P. Kazemian, M. Chang, H. Zeng, G. Varghese, N. McKeown, and S. Whyte, “Real time network policy checking using header space analysis,” in Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation, 2013, p. 99–112. [65] F. Ye, D. Yu, E. Zhai, H. H. Liu, B. Tian, Q. Ye, C. Wang, X. Wu, T. Guo, C. Jin, D. She, Q. Ma, B. Cheng, H. Xu, M. Zhang, Z. Wang, and R. Fonseca, “Accuracy, scalability, coverage: A practical configuration verifier on a global wan,” in Proceedings of the ACM SIGCOMM 2020 Conference, 2020, pp. 599–614. [66] S. Jha, R. Arora, Y. Watanabe, T. Yanagawa, Y. Chen, J. Clark, B. Bhavya, M. Verma, H. Kumar, H. Kitahara, N. Zheutlin, S. Takano, D. Pathak, F. George, X. Wu, B. O. Turkkan, G. Vanloo, M. Nidd, T. Dai, O. Chatterjee, P. Gupta, S. Samanta, P. Aggarwal, R. Lee, J.-w. Ahn, D. Kar, A. Paradkar, Y. Deng, P. Moogi, P. Mohapatra, N. Abe, C. Narayanaswami, T. Xu, L. R. Varshney, R. Mahindru, A. Sailer, L. Shwartz, D. Sow, N. C. M. Fuller, and R. Puri, “Itbench: evaluating ai agents across diverse real-world it automation tasks,” in Proceedings of the 42nd International Conference on Machine Learning, ser. ICML’25, 2025. [67] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang, “Agentbench: Evaluating llms as agents,” in ICLR 2024, 2024. [68] S. Yao, N. Shinn, P. Razavi, and K. R. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” in International Conference on Learning Representations (ICLR 2025), 2025. [Online]. Available: https://openreview.net/forum?id=roNSXZpUDN [69] I. McCormack. (2025, Jan.) Preserving integrity in the age of generative AI. National Cyber Security Centre (NCSC). [Online]. Available: https://www.ncsc.gov.uk/blog-post/preserving-integrity-in-age-generative-ai [70] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proceedings of the 2023 Workshop on Artificial Intelligence and Security (AISec ’23). Association for Computing Machinery, 2023, pp. 79–90. [71] S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “StruQ: Defending against prompt injection with structured queries,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 2383–2400. [72] D. Pasquini, E. M. Kornaropoulos, G. Ateniese, O. Akgul, A. Theocharis, and P. Efstathopoulos, “When AIOps become “AI oops”: Subverting LLM-driven IT operations via telemetry manipulation,” arXiv:2508.06394, 2025. [73] J. Sillito and E. Kutomi, “Failures and fixes: A study of software system incident response,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2020, pp. 185–195. [74] D. G. Widder, L. Dabbish, J. D. Herbsleb, A. Holloway, and S. Davidoff, “Trust in collaborative automation in high stakes software engineering work: A case study at nasa,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2021. [75] Z. Wang, Z. Liu, Y. Zhang, A. Zhong, J. Wang, F. Yin, L. Fan, L. Wu, and Q. Wen, “RCAgent: Cloud root cause analysis by autonomous agents with tool-augmented large language models,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM), 2024, pp. 4966–4974. [76] J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu, “Acto: Automatic end-to-end testing for operation correctness of cloud system management,” in Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP). ACM, 2023, pp. 96–112. [77] B. Grubic, Y. Wang, T. Petrochko, R. Yaniv, B. Jones, D. Callies, M. Clarke-Lauer, D. Kelley, S. Demetriou, K. Yu, and C. Tang, “Conveyor: One-Tool-Fits-All continuous software deployment at meta,” in 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 2023, pp. 325–342. [78] A. Nazari, Y. Zhang, M. Raghothaman, and H. Chen, “Localized explanations for automatically synthesized network configurations,” in Proceedings of the 23rd ACM Workshop on Hot Topics in Networks (HotNets), 2024, pp. 52–59. [79] X. Liu, P. Zhang, A. Abhashkumar, J. Chen, and W. Jiang, “Automatic configuration repair,” in Proceedings of the 23rd ACM Workshop on Hot Topics in Networks (HotNets), 2024, pp. 213–220. [80] Y. Lu, H. Li, X. Cong, Z. Zhang, Y. Wu, Y. Lin, Z. Liu, F. Liu, and M. Sun, “Learning to generate structured output with schema reinforcement learning,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 4905–4918. [81] S. Geng, H. Cooper, M. Moskal, S. Jenkins, J. Berman, N. Ranchin, R. West, E. Horvitz, and H. Nori, “JSONSchemaBench: A rigorous benchmark of structured outputs for language models,” 2025. [82] D. Wu, X. Wang, Y. Qiao, Z. Wang, J. Jiang, S. Cui, and F. Wang, “Netllm: Adapting large language models for networking,” in Proceedings of the ACM SIGCOMM 2024 Conference, 2024, pp. 661–678. [83] R. Mondal, N. Bjørner, T. D. Millstein, A. Tang, and G. Varghese, “Tackling ambiguity in user intent for llm-based network configuration synthesis,” in Proceedings of the 24th ACM Workshop on Hot Topics in Networks (HotNets ’25), 2025, pp. 176–183, workshop paper; PDF available from HotNets.

47

[84] S. Zhang, Y. Xue, Y. Zhang, X. Wu, A. T. Luu, and C. Zhao, “MRAG: A modular retrieval framework for time-sensitive question answering,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 3080–3118. [85] W. Zou, R. Geng, B. Wang, and J. Jia, “PoisonedRAG: Knowledge corruption attacks to Retrieval-Augmented generation of Large Language Models,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 3827–3844. [86] A. Shafran, R. Schuster, and V. Shmatikov, “Machine against the RAG: Jamming retrieval-augmented generation with blocker documents,” in 34th USENIX Security Symposium (USENIX Security 25). USENIX Association, 2025, pp. 3787–3806. [87] D. Chismon. (2025, Dec.) Prompt injection is not SQL injection (it may be worse). National Cyber Security Centre. Accessed: 12 Jan 2026. [Online]. Available: https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection [88] J. Qi, G. Sarti, R. Fernández, and A. Bisazza, “Model internals-based answer attribution for trustworthy retrieval-augmented generation,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), 2024, pp. 6037–6053. [89] T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou, “Large language models as tool makers,” in ICLR 2024, 2024. [90] A. Gember-Jacobson, C. Raiciu, and L. Vanbever, “Integrating verification and repair into the control plane,” in Proceedings of the 16th ACM Workshop on Hot Topics in Networks (HotNets-XVI). ACM, 2017, pp. 129–135. [91] R. Shiiba, S. Kobayashi, O. Akashi, and K. Fukuda, “Refining specifications for configuration repair with side effect diagnosis,” in Proceedings of the 2nd Workshop on Formal Methods Aided Network Operation (FMANO ’25). ACM, 2025, pp. 43–48. [92] J. H. Saltzer and M. D. Schroeder, “The protection of information in computer systems,” Proceedings of the IEEE, vol. 63, no. 9, pp. 1278–1308, 1975. [93] Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng et al., “Prompt injection attack against llm-integrated applications,” arXiv preprint arXiv:2306.05499, 2023. [94] I. Weber, H. Wada, A. D. Fekete, A. Liu, and L. Bass, “Supporting undoability in systems operations,” in Proceedings of the 27th Large Installation System Administration Conference (LISA ’13). USENIX Association, 2013, pp. 75–88. [95] G. Yu, P. Chen, Z. He, Q. Yan, Y. Luo, F. Li, and Z. Zheng, “Changerca: Finding root causes from software changes in large online systems,” Proc. ACM Softw. Eng., vol. 1, 2024. [96] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning (ICML), vol. 70. PMLR, 2017, pp. 1321–1330. [97] A. Basiri, L. Hochstein, N. Jones, and H. Tucker, “Automating chaos experiments in production,” in Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP ’19. IEEE, May 2019, pp. 31–40. [98] S. A. Crosby and D. S. Wallach, “Efficient data structures for tamper-evident logging,” in 18th USENIX Security Symposium (USENIX Security 09). USENIX Association, 2009, pp. 317–334. [99] H. Song, F. Qin, P. Martinez-Julia, L. Ciavaglia, and A. Wang, “Network Telemetry Framework,” RFC 9232, May 2022. [100] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. A. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda, “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023. [101] A. Clemm, L. Ciavaglia, L. Z. Granville, and J. Tantsura, “Intent-Based Networking - Concepts and Definitions,” RFC 9315, Oct. 2022. [102] D. Turner, K. Levchenko, A. C. Snoeren, and S. Savage, “California fault lines: Understanding the causes and impact of network failures,” in Proceedings of the ACM SIGCOMM 2010 Conference, 2010, pp. 315–326. [103] P. Gill, N. Jain, and N. Nagappan, “Understanding network failures in data centers: Measurement, analysis, and implications,” in Proceedings of the ACM SIGCOMM 2011 Conference, 2011, pp. 350–361. [104] T. Xu and Y. Zhou, “Systems approaches to tackling configuration errors: A survey,” ACM Comput. Surv., vol. 47, no. 4, 2015. [105] M. Bilal, “Network-optimised spiking neural network for event-driven networking,” arXiv preprint arXiv:2509.23516, 2025. [106] M. Bilal and X. Xu, “Network-optimised spiking neural network (nos) scheduling for 6g o-ran: Spectral margin and delay-tail control,” arXiv preprint arXiv:2510.11291, 2025. [107] A. Horn, A. Kheradmand, and M. Prasad, “Delta-net: Real-time network verification using atoms,” in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). Boston, MA: USENIX Association, Mar. 2017, pp. 735–749. [108] R. Beckett, R. Mahajan, T. Millstein, J. Padhye, and D. Walker, “Don’t mind the gap: Bridging network-wide objectives and devicelevel configurations,” in Proceedings of the ACM SIGCOMM 2016 Conference, 2016, pp. 328–341. [109] ——, “Network configuration synthesis with abstract topologies,” in Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2017, pp. 437–451. [110] A. El-Hassany, P. Tsankov, L. Vanbever, and M. Vechev, “Network-wide configuration synthesis,” in International Conference on Computer Aided Verification, 2017, pp. 261–281. [111] L. Vanbever, S. Vissicchio, C. Pelsser, P. Francois, and O. Bonaventure, “Lossless migrations of link-state igps,” IEEE/ACM Transactions on Networking, vol. 20, no. 6, pp. 1842–1855, 2012. [112] X. Xu, W. Deng, R. Beckett, R. Mahajan, and D. Walker, “Test coverage for network configurations,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). Boston, MA: USENIX Association, Apr. 2023, pp. 1717–1732. [113] N. Handigol, B. Heller, V. Jeyakumar, D. Mazières, and N. McKeown, “I know what your packet did last hop: Using packet histories to troubleshoot networks,” in 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). Seattle, WA: USENIX Association, Apr. 2014, pp. 71–85.

48

[114] X. Hua, Q. Li, Z. Zhang, X. Fan, L. Zhu, R. Yu, Y. Zhou, Y. Zhang, J. Feng, C. Deng, and X. Yuan, “Csdnet:causal inference aided handover parameter adjusting effect estimation in cellular networks,” in ICC 2023 - IEEE International Conference on Communications, 2023, pp. 5304–5309. [115] Y. Zhang, Q. Li, X. Hua, R. Yu, Z. Zhang, X. Fan, L. Zhu, T. Sha, and Y. Zhang, “Dcdn: Estimating handover parameter adjusting effect with causal inference,” in 2023 IEEE 98th Vehicular Technology Conference (VTC2023-Fall), 2023, pp. 1–5. [116] M. Li, Y. Sun, X. Hua, R. Yu, X. Fan, L. Zhu, J. Feng, and D. Pei, “Predicting parameter change’s effect on cellular network time series,” 2024. [117] A. Ikram, S. Chakraborty, S. Mitra, S. Saini, S. Bagchi, and M. Kocaoglu, “Root cause analysis of failures in microservices through causal discovery,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 31 158–31 170. [118] Z. Yao, C. Pei, W. Chen, H. Wang, L. Su, H. Jiang, Z. Xie, X. Nie, and D. Pei, “Chain-of-event: Interpretable root cause analysis for microservices through automatically learning weighted event causal graph,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE-Companion). ACM, 2024, pp. 50–61. [119] Z. Li, J. Chen, R. Jiao, N. Zhao, Z. Wang, S. Zhang, Y. Wu, L. Jiang, L. Yan, Z. Wang, Z. Chen, W. Zhang, X. Nie, K. Sui, and D. Pei, “Practical root cause localization for microservice systems via trace analysis,” in Proceedings of the IEEE/ACM 29th International Symposium on Quality of Service (IWQoS ’21). IEEE, 2021, pp. 1–10. [120] N. Zhao, H. Wang, Z. Li, X. Peng, G. Wang, Z. Pan, Y. Wu, Z. Feng, X. Wen, W. Zhang, K. Sui, and D. Pei, “An empirical investigation of practical log anomaly detection for online service systems,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’21). Association for Computing Machinery, 2021, pp. 1404–1415. [121] X. Ma, Y. Li, J. Keung, X. Yu, H. Zou, Z. Yang, F. Sarro, and E. T. Barr, “Practitioners’ expectations on log anomaly detection,” IEEE Transactions on Software Engineering, vol. 51, no. 9, pp. 2455–2471, 2025. [122] J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, “Tools and benchmarks for automated log parsing,” in Proceedings of the IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’19). IEEE, 2019, pp. 121–130. [123] M. Bilal, O. Tariq, and H. Ahmed, “Nos-gate: Queue-aware streaming ids for consumer gateways under timing-controlled evasion,” IEEE Transactions on Consumer Electronics, pp. 1–1, 2026. [124] L. Lamport, “Time, clocks, and the ordering of events in a distributed system,” Communications of the ACM, vol. 21, no. 7, pp. 558–565, 1978. [125] F. B. Schneider, “Implementing fault-tolerant services using the state machine approach: a tutorial,” ACM Comput. Surv., vol. 22, no. 4, p. 299–319, Dec. 1990. [126] R. Pedro, M. E. Coimbra, D. Castro, P. Carreira, and N. Santos, “Prompt-to-SQL injections in LLM-integrated web applications: Risks and defenses,” in Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, pp. 1768–1780. [127] Z. Li, N. Zhao, S. Zhang, Y. Sun, P. Chen, X. Wen, M. Ma, and D. Pei, “Constructing large-scale real-world benchmark datasets for AIOps,” arXiv preprint arXiv:2208.03938, 2022. [128] Y. Sun, M. Shao, X. Nie, K. Yang, X. Li, B. Hao, S. Zhang, C. Pei, D. He, Y. Li, and D. Pei, “A comprehensive benchmark and empirical study of trace anomaly detection,” IEEE Transactions on Services Computing, vol. 18, no. 6, pp. 3364–3377, 2025. [129] C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong, and J. He, “The thirty-eight conference on neural information processing systems datasets and benchmarks track,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2024/file/877b40688e330a0e2a3fc24084 208dfa-Paper-Datasets and Benchmarks Track.pdf [130] L. Gioacchini, G. Siracusano, D. Sanvito, K. Gashteovski, D. Friede, R. Bifulco, and C. Lawrence, “Agentquest: A modular benchmark framework to measure progress and improve LLM agents,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations). Association for Computational Linguistics, 2024, pp. 185–193. [131] Z. Guo, Y. Huang, and D. Xiong, “CToolEval: A chinese benchmark for LLM-powered agent evaluation in real-world API interactions,” in Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 2024, pp. 15 711–15 724. [132] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems 28, 2015, pp. 2503–2511. [133] E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ml test score: A rubric for ml production readiness and technical debt reduction,” in 2017 IEEE International Conference on Big Data (IEEE BigData 2017), 2017, pp. 1123–1132. [134] C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive label errors in test sets destabilize machine learning benchmarks,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. [Online]. Available: https://openreview.net/forum?id=XccDXrDNLek [135] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with CheckList,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020, pp. 4902–4912. [136] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu, “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” 2024. [137] L. Gioacchini, A. Delsanto, I. Drago, M. Mellia, G. Siracusano, and R. Bifulco, “AutoPenBench: A vulnerability testing benchmark for generative agents,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics, 2025, pp. 1615–1624.

49

[138] T. Xu, L. Chen, D.-J. Wu, Y. Chen, Z. Zhang, X. Yao, Z. Xie, Y. Chen, S. Liu, B. Qian, A. Yang, Z. Jin, J. Deng, P. Torr, B. Ghanem, and G. Li, “CRAB: Cross-environment agent benchmark for multimodal language model agents,” in Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 2025, pp. 21 607–21 647. [139] R. Kapoor, Y. P. Butala, M. A. Russak, J. Y. Koh, K. Kamble, W. AlShikh, and R. Salakhutdinov, “Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web,” in Proceedings of the European Conference on Computer Vision (ECCV), 2024, pp. 161–178. [140] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NeurIPS 2023, 2023. [141] J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024, pp. 881–905. [142] A. Sorstkins, O. Tariq, and M. Bilal, “Learning to undo: Rollback-augmented reinforcement learning with reversibility signals,” 2025. [143] Q. Zhan, R. Fang, H. S. Panchal, and D. Kang, “Adaptive attacks break defenses against indirect prompt injection attacks on LLM agents,” in Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang, Eds. Association for Computational Linguistics, 2025, pp. 7116–7132. [144] H. Xia, H. Wang, Z. Liu, Q. Yu, Y. Guo, and H. Wang, “Safetoolbench: Pioneering a prospective benchmark to evaluating tool utilization safety in llms,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 2025, pp. 17 643–17 660. [145] Y. Chen, H. Li, Z. Zheng, D. Wu, Y. Song, and B. Hooi, “Defense against prompt injection attack by leveraging attack techniques,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2025, pp. 18 331–18 347. [146] R. Zhang, D. Sullivan, K. Jackson, P. Xie, and M. Chen, “Defense against prompt injection attacks via mixture of encodings,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). Association for Computational Linguistics, 2025, pp. 244–252. [147] Y. Chen, H. Li, Y. Sui, Y. Song, and B. Hooi, “Backdoor-powered prompt injection attacks nullify defense methods,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 2025, pp. 4508–4527. [148] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel, “Extracting training data from large language models,” in 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, 2021, pp. 2633–2650. [149] Y. Song, R. Liu, S. Chen, Q. Ren, Y. Zhang, and Y. Yu, “Securesql: Evaluating data leakage of large language models as natural language interfaces to databases,” in Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, 2024, pp. 5975–5990. [150] J. Wang, T. Yang, R. Xie, and B. Dhingra, “Raccoon: Prompt extraction benchmark of llm-integrated applications,” in Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 2024, pp. 13 349–13 365. [151] A. Narayanan and V. Shmatikov, “Robust de-anonymization of large sparse datasets,” in 2008 IEEE Symposium on Security and Privacy (sp 2008), 2008, pp. 111–125. [152] L. Bainbridge, “Ironies of automation,” Automatica, vol. 19, no. 6, pp. 775–779, 1983. [153] R. Parasuraman and V. Riley, “Humans and automation: Use, misuse, disuse, abuse,” Human Factors, vol. 39, no. 2, pp. 230–253, 1997. [154] J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,” Human Factors, vol. 46, no. 1, pp. 50–80, 2004. [155] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,” in Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 2024, pp. 10 471–10 506. [156] Z. Wang, V. Siu, Z. Ye, T. Shi, Y. Nie, X. Zhao, C. Wang, W. Guo, and D. Song, “AGENTVIGIL: Automatic black-box red-teaming for indirect prompt injection against llm agents,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 2025, pp. 23 159–23 172. [157] F. Jia, T. Wu, X. Qin, and A. Squicciarini, “The task shield: Enforcing task alignment to defend against indirect prompt injection in llm agents,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2025, pp. 29 680–29 697. [158] T. Wen, C. Wang, X. Yang, H. Tang, Y. Xie, L. Lyu, Z. Dou, and F. Wu, “Defending against indirect prompt injection by instruction detection,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 2025, pp. 19 472–19 487. [159] Q. Zeng, C. Jin, X. Wang, Y. Zheng, and Q. Li, “AIRepr: An analyst-inspector framework for evaluating reproducibility of llms in data science,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 2025, pp. 10 170–10 201. [160] D. Kim, G. Shim, Y. Chun, M. Kim, C. Park, and H. Lim, “Benchmark profiling: Mechanistic diagnosis of llm benchmarks,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2025, pp. 15 635–15 650. [161] K. Veeraraghavan, J. Meza, D. Chou, W. Kim, S. Margulis, S. Michelson, R. Nishtala, D. Obenshain, D. Perelman, and Y. J. Song, “Kraken: Leveraging live traffic tests to identify and resolve resource utilization bottlenecks in large scale web services,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 2016, pp. 635–651. [162] X. Sun, W. Luo, J. T. Gu, A. Ganesan, R. Alagappan, M. Gasch, L. Suresh, and T. Xu, “Automatic reliability testing for cluster management controllers,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, pp. 143– 159.

50

[163] Y. Gan, Y. Zhang, D. Cheng, A. Shetty, P. Rathi, N. Katarki, A. Bruno, J. Hu, B. Ritchken, B. Jackson, K. Hu, M. Pancholi, Y. He, B. Clancy, C. Colen, F. Wen, C. Leung, S. Wang, L. Zaruvinsky, M. Espinosa, R. Lin, Z. Liu, J. Padilla, and C. Delimitrou, “An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems,” in Proceedings of the twenty-fourth international conference on architectural support for programming languages and operating systems, 2019, pp. 3–18. [164] L. Hu, G. Chen, X. Shang, S. Cheng, B. Wu, G. Li, X. Zhu, W. Zhang, and N. Yu, “CompileAgent: Automated real-world repolevel compilation with tool-integrated llm-based agent system,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2025, pp. 2078–2091. [165] M. Li, Z. Li, K. Yin, X. Nie, W. Zhang, K. Sui, and D. Pei, “Causal inference-based root cause analysis for online service systems with intervention recognition,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 3230–3240. [166] Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in 33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1831–1847. [167] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, “MASTERKEY: Automated jailbreaking of large language model chatbots,” in Proceedings of the Network and Distributed System Security Symposium, ser. NDSS 2024, 2024. [168] N. Carlini, M. Jagielski, C. A. Choquette-Choo, D. Paleka, W. Pearce, H. S. Anderson, A. Terzis, K. Thomas, and F. Tramèr, “Poisoning web-scale training datasets is practical,” in 2024 IEEE Symposium on Security and Privacy (SP), 2024, pp. 407–425. [169] P. Borman, M. Hines, C. Lebsack, C. Morrow, A. Shaikh, R. Shakir, W. B. Li, and D. Loher, “gRPC Network Management Interface (gNMI),” Specification, 2023, accessed 2026-02-02. [Online]. Available: https://github.com/openconfig/reference/blob/master/rpc/g nmi/gnmi-specification.md [170] Model Context Protocol, “Model context protocol specification,” Official specification, 2025, protocol revision 2025-06-18; accessed 2026-11-25. [Online]. Available: https://modelcontextprotocol.io/specification/2025-06-18