Building AI That Works: ESnet’s Pragmatic Approach to AI-Driven Operational Excellence – A progress report on developing Operations Response & Business Intelligence Toolkit (ORBIT) Bin Dong, Sukhada Gholba, Brooklin Gore, Shawn Kwang, David Mitchell, Samuel Oehlert, Garrett Stewart, Brendan White, Luke Baker, Ed Balas, Britt Gathright, Chin Guok, Jon-Paul Heron, John MacAuley, Scott Richmond, Chris Robb, Chris Tracy, Kesheng Wu
arXiv:2607.22948v1 [cs.NI] 24 Jul 2026
Energy Science Network
June 2026
Executive Summary The ORBIT (Operations Responses & Business Intelligence Toolkit) project was initiated to explore the potential of agentic AI for upcoming ESnet 7 initiative. This report summarizes the project’s objectives, design, key findings, and recommendations, providing a comprehensive overview of the operational and strategic value of AI-driven tools in a production environment. ORBIT is grounded with a present business needs to address persistent pain points in the NOC workflow. These include slow information retrieval from siloed data sources, lengthy and difficult-to-parse incident tickets, and context loss during shift handoffs. These challenges result in increased cognitive load on operators and longer incident resolution times. To address these issues, the ORBIT project set out to develop an AI-powered system that could automate routine tasks, synthesize information from multiple sources, and provide operators with actionable insights directly within their existing workflows. ORBIT is an agentic AI system integrated into ServiceNow, the NOC’s primary incident management platform. The system is built on a modular, layered architecture that includes a centralized reasoning hub, a set of specific tools (MCPs) for accessing ESnet data sources, a semantic search layer, and a chat interface. The ORBIT project has adopted the industry best practice of using “skills” to manage the complexity and stochasticity of the AI toolchain. Skills are versioned, tested, and iteratively refined sets of instructions that guide the AI in performing specific tasks, ensuring greater reliability and predictability. The ORBIT project has yielded several important findings. The system successfully delivered on all six of its initial tasks, and its flexible architecture enabled the rapid development of two additional tasks proposed by NOC engineers. We observed significant organic adoption of the project’s general-purpose infrastructure components, with the chat interface and LiteLLM model gateway attracting a large number of users and a high volume of requests from outside the project. This demonstrates a clear demand for accessible, well-supported AI tools within ESnet. Furthermore, our experiments with “skills” have shown that this approach can dramatically improve the performance and reliability of the AI, reducing the number of steps required to complete a task and eliminating errors. Based on these findings, we recommend that ESnet adopt the four general-purpose components of ORBIT as officially supported services. This will require a dedicated productization effort to establish clear ownership, governance, and support structures. We also recommend continued investment in a comprehensive skill library to further improve the reliability and effectiveness of the system. Additional future work may include a systematic evaluation of mission-oriented outcomes, the development of a unified, identity-aware tool gateway, and a continued focus on data governance and operational sustainability.
work operations demands more than model novelty; it requires measurable improvements to operational workflows under realistic constraints [1, 7, 8, 33, 45]. Research and engineering groups managing highperformance research networks face recurring challenges in incident handling: finding the right resolution context across heterogeneous knowledge sources, communicating status to multiple audiences, and maintaining operator control over AI outputs [11, 19, 23, 27, 45]. These challenges are compounded by the inherent stochasticity of AI tool chains, which can lead to unpredictable and unreliable behavior. These requirements motivate a shift from benchmarking model capabilities in isolation to considering business needs, technology trends, and staff engagement in production settings [31, 36, 41]. Our work began with a systematic review of business needs [11], followed by a careful consideration of available resources and technical expertise [29]. We then selected a set of tasks from ESnet’s Network Operation Center (NOC) workflow to drive a data integration and platform development effort (see Figure 1 for an illustration). Following this decision, the planning team worked with engineering staff to develop concrete tasks from NOC operations. Based on considerations such as data availability, staff availability, task impact, and task feasibility, the team selected the six specific NOC tasks listed in Table 1. The project had a limited duration (January through June 2026) with three primary objectives: (1) develop the Operations Responses & Business Intelligence Toolkit (ORBIT) to accomplish the six specific tasks, (2) engage ESnet staff in exploring AI tools, and (3) gather performance and usability statistics to inform future AI efforts. This report summarizes the ORBIT project, detailing our software system, usage observations, and system performance measures in support of all three project goals. ORBIT is a pragmatic AI system integrated into a Network Operation Center (NOC) workflow to assist with incident handling. Our design is stakeholderdriven and developed through incremental deployment, guided by an explicit goal: augment operator capabilities without introducing additional process overhead. ORBIT is built to fit existing operational tooling, most notably ServiceNow incident work notes, while preserving human-in-the-loop decision-making throughout. This work makes three contributions, one aligned with each project objective:
Contents 1 Introduction
1
2 Background
3
3 System Overview 3.1 Layered components and data flow . . 3.2 Orchestration and event loops . . . . 3.3 Safety and Security . . . . . . . . . . 3.4 Governance, and evaluation . . . . . . 3.5 Implementation Flexibility . . . . . . .
5 5 5 6 7 7
4 Main Components 4.1 ServiceNow Client through Agentic API 4.2 ORBIT Agentic Service . . . . . . . . 4.3 Chat Interface . . . . . . . . . . . . . 4.4 Semantic Search . . . . . . . . . . . . 4.5 Tools for Direct Data Accesses . . . . 4.6 Model Access Gateway . . . . . . . .
7 7 8 12 12 13 14
5 Performance and Usability 5.1 Statistics from ServiceNow AI Actions 5.2 ORBIT Server Usage . . . . . . . . . 5.3 Chat Interface Usage . . . . . . . . . 5.4 AI Model Usage . . . . . . . . . . . . 5.5 User Interviews . . . . . . . . . . . .
15 16 16 17 18 20
6 Lessons and Discussion 6.1 From Prompts to Skills: Managing Complexity in AI Tool Chains . . . . . 6.2 Managing Stochasticity: Same Prompt, Different Answer . . . . . . . 6.3 Emergent Use Cases: The Best Ones Find You . . . . . . . . . . . . . . . . 6.4 Organic Infrastructure Adoption: Build It and They Will Come . . . . . . . . 6.5 Cognitive Barriers to AI Adoption . . .
21
7 Review of the State-of-the-art
23
8 Conclusion and Future Work 8.1 Project Outcomes and Lessons Learned 8.2 Recommendations . . . . . . . . . . . 8.3 Future work . . . . . . . . . . . . . .
25 25 26 27
1
21 21 22 22 23
Introduction
While advances in large language models make AI assistance increasingly accessible, deploying AI in net1
LangChain
NOC Ticket Test Suite
(1) Agentic Actions Ticket Lifecycle
- Generate timeline - Recommend RFO - Provide bulleted view -……
(2) ORBIT Server
Search forLibreChat Related Doc
(3) Chat
Dispatch request, Call RAG, Retrieve extra data, …
Web Server Forward requests and results to and from ORBIT Server
- Suggest procedure - Recommend priority - …… (4) Semantic Search
(5) Direct Data Accesses Vector DB on Docs
Data access capability
Update, reindex, RAG, …
Data Sources
ESDB
ServiceNow
Wiki
Figure 1: A logical view of ORBIT where the key NOC functions sit on top of a set of tools that integrate data from multiple sources.
Table 1: List of NOC tasks targeted for the ORBIT project.
Activity Summarize incident for hand-off
Data ServiceNow
2
Generate ticket timeline
ServiceNow, ESDB
3
Suggest alarm procedure
ESDB, ESnet Wiki, Stardust, ServiceNow
4
Recommend alarm priority
5
Propose final RFO
6
NOC chat bot
ESnet Wiki, ESDB, ServiceNow ESDB, ESnet Wiki, ServiceNow ESnet Wiki, ESDB
1
Description When a ticket is handed off to another team or escalated, parse the ticket and generate a history of the actions taken so far in a summary that is easy for the next person to process. On demand, generate a timeline of the ticket with all actions taken and resolution information for inclusion in an after action report. Place all the NOC process documentation behind AI and use the alarm data in a given ticket to automatically populate it with relevant potential next steps from our standard processes when the ticket is created. Place this information inline in the ticket with references to the documentation cited. Use NOC process documentation to recommend any changes to the default alarm priority given the alarm profile and impact. Using NOC process documentation and our RFO codes, process the ticket narrative and determine a suggested RFO code and close note summary. Place all the NOC process documentation behind AI and create a chat bot that enables a user to ask questions of our documentation along with references to the source documentation.
2
1. A workflow-integrated agentic system for NOC incident handling. ORBIT delivers two categories of production-oriented capability. Adaptive Incident Summarization (Tasks 1–2 in Table 1) generates audience-specific summaries from ServiceNow incident narratives to streamline handoffs and stakeholder communication. Intelligent Resolution Guidance (Tasks 3–5) retrieves and ranks resolution procedures from operational knowledge bases using incident context, surfacing relevant steps and supporting evidence when operators would otherwise rely on manual documentation search. Both capabilities are grounded in cross-source synthesis across ServiceNow, Confluence, Slack, ESDB, and Stardust via purpose-scoped MCP tools and hybrid semantic search integrated directly into the ServiceNow incident workflow. Beyond the six originally scoped tasks, the platform’s composability enabled two additional tasks—Caused-By-Change Analysis and Change Analysis—proposed by NOC engineers and implemented within hours.
1
2
3
4
5
Alarm Fires
Open Ticket
Hunt for Context
Fix It & Document
Do It Again
Figure 2: A schematic drawing of a simplified NOC workflow.
3. Operational measurements and engineering lessons from a test deployment. We report per-task execution statistics, skill-refinement evidence (engineered skills reduced agent actions from 10 to 4 and eliminated retries), workloadlevel cost breakdowns, and structured user interviews with NOC operators. Five engineering lessons distilled from this evidence address skill versioning and testing, stochasticity in tool selection, emergent use cases, organic infrastructure adoption, and the cognitive barriers to operational AI adoption. Together, these measurements and lessons illustrate how operational AI systems can be evaluated against workflow requirements rather than relying on benchmark scores or subjective impressions of helpfulness.
2
2. Reusable AI infrastructure that attracted adoption beyond the original project scope. Four components—(1) the LiteLLM model gateway, (2) the agentic application service, (3) the chat interface (chat.es.net), and (4) the MCP tool layer—were designed as generalpurpose services rather than project-specific utilities. The agentic service exposes a uniform API through which any client can compose taskspecific workflows from pluggable tools, versioned prompt templates, and selectable language models, making it reusable as a generalpurpose agentic composition service beyond incident handling. During the evaluation period, the gateway served over 714,000 requests consuming more than 21 billion tokens, of which roughly 95% came from coding-assistant workloads initiated by engineers outside the ORBIT project. Six of seven NOC operators adopted the chat interface organically, generating 72 conversations and over 1,200 tool invocations without prescribed workflows—substantially more interaction than the 12 user-initiated ServiceNow AI Actions recorded in the same period. This infrastructure realizes the AI Sandbox called for in ESnet’s Data and AI report [11].
Background
Operational teams increasingly adopt AI assistance to reduce the time and cognitive load associated with incident response [42, 15]. In practice, success depends less on demonstrating model novelty and more on achieving measurable improvements in operator workflows under real-world constraints: heterogeneous data sources, variable incident narratives, strict governance requirements, and the need for human-in-theloop decision-making [1, 7, 8, 33, 45]. Network Operations Centers (NOCs) face recurring bottlenecks in context gathering and communication. When alarms trigger, operators must quickly determine what happened, identify the relevant components and procedures, and translate ticket histories into accurate status updates for different stakeholders. Much of this effort is spent searching across disjoint knowledge bases rather than executing resolution steps (see Figure 2). In practice, the pain concentrates in four areas that compound one another: • Finding information is slow. Operators search across documentation, past tickets, and Slack conversations to locate the relevant procedure, often re-finding the same material they located weeks earlier. The absence of a unified retrieval 3
path turns every new alarm into a repetitive, from multiple sources [2, 18, 35], and a unified data time-consuming scavenger hunt. access mechanism is needed for more than half of the work-packages identified from our workshop [11, 29]. • Tickets become novels. ServiceNow incidents The tasks in Table 1 exercise a small set of input accumulate pages of work notes, comments, and sources: the NOC process and procedure documenstatus updates over their lifetimes. Understand- tation captured as ESnet Wiki pages in Confluence ing the state of an incident means reading every- (“Wiki/Confluence data”), ServiceNow incident rething; there is no reliable shortcut for extracting ports (“tickets”), Stardust for ESnet telemetry [3], and the salient facts from a lengthy narrative. ESDB (ESnet Database) for physical and logical at• Handoffs lose context. When tickets move be- tributes of network components. We limited the numtween shifts or are escalated to another team, ber of sources to ensure the evaluation would primarily the receiving operator effectively starts from characterize data integration and synthesis, while proscratch. Critical context—what was tried, what viding coverage of realistic operational content. A fundamental design constraint is that AI assiswas ruled out, what the customer was told—is embedded in the ticket narrative but not sur- tance must integrate seamlessly into the ServiceNowfaced in a form that supports rapid onboarding. centric operational workflow without creating additional overhead or shifting accountability away from • Knowledge is everywhere and nowhere. Pro- operators. Consequently, ORBIT is explicitly not decedures live in Confluence, real-time context signed to autonomously decide or execute resolution lives in Slack, incident history lives in Servi- actions. Instead, it generates evidence-grounded drafts ceNow, and equipment details live in ESDB. and suggestions inserted into ticket work notes and These sources are almost never consulted to- editable fields, enabling operators to accept, refine, or gether, forcing operators to mentally stitch to- reject AI output. This supports governance and also gether a coherent picture from fragments scat- enables evaluation: operator decisions and edits protered across systems. vide observable signals of usefulness and failure modes. Another key motivation is measurability. Rather At ESnet, these challenges motivated the ex- than relying on subjective impressions of “helpfulness,” ploratory project ORBIT 1 . The project originated from the project requires instrumentation supporting usabilESnet’s internal review of high-impact AI use cases, ity and performance evaluation for deployment readiemphasizing workflows that are time-consuming, diffi- ness and broader investment decisions. ORBIT, therecult to execute consistently, and feasible to integrate fore, collects operationally relevant data such as LLM into operational tooling [11, 29]. Through collabora- call counts, latency, and resource consumption (with tion with NOC operators, we narrowed an initial set tokens used as a proxy for cost), along with user feedof candidate use cases down to six that share a com- back gathered from the same interface used by opermon theme: operators already possess the necessary ators. A guiding principle is that software capability knowledge, but it is buried across multiple systems alone is insufficient: the team must gather enough and formats. The four pain points above map di- evidence to characterize the operational impact and rectly onto the ORBIT task set: Adaptive Incident inform whether ORBIT should proceed to broader use Summarization addresses the “tickets are novels” and or similar AIOps initiatives. Stakeholder feedback also “handoffs lose context” problems by generating con- identified evaluation pitfalls, including contamination cise, audience-appropriate summaries, while Intelligent of tickets when AI actions are repeated. This drives Resolution Guidance targets the “finding information requirements for mechanisms to separate AI-generated is slow” and “knowledge is everywhere” problems by content from original a ticket text so that performance retrieving and ranking relevant procedures from op- measurement remains interpretable. erational knowledge bases using incident context. AcFinally, ORBIT serves as ESnet’s initial operational cordingly, ORBIT targets these two categories of func- test deployment of AI components. Beyond improving tionality (see Table 1). incident handling, the project aims to build organizaA key driver for this work is data integration. AI tional capability for designing, integrating, and govtools are known to be well-suited for integrating data erning AI systems in production. This includes secu1 rity evaluation under ESnet’s AI and cloud policies, Some earlier documents refer to this as “Ops Assistant.” 4
careful handling of business-sensitive network configuration data in ESDB, and structured cross-department collaboration to establish a foundation for responsible AI operations via both technical integration patterns and an evaluation methodology grounded in operational measurements.
tool access, and user evaluation workflows (see subsection 4.3). With the required data sources, this interface also serves as the solution to Task 6 in Table 1.
ServiceNow MID server workflow integration ServiceNow controls the event loop by invoking the agent at appropriate times (e.g., on new incident tasks or specific business-rule triggers). The MID server receives agent outputs and writes them into designated ticket fields using discrete beta fields during development phases to preserve operational safety. This component ensures minimal disruption to existing incident response processes and enables accurate instrumentation of agent execution outcomes, more in 4.1.
3.2
Model Accesses via LiteLLM The architecture routes LLM calls through an enterprise proxy that standardizes access, monitors token and other resource usage, and supports cost accounting. This design en3 System Overview sures that usability and performance evaluation can ORBIT is implemented as an agentic, workflow- include operational cost signals, not just generation integrated system built to support operator-led inci- quality (see subsection 4.6). dent handling from within ServiceNow. The overall architecture follows a layered design that separates or- Agentic application service The core reasoning chestration, retrieval, generation, and integration with and composition logic resides in a dedicated agentic existing operational tools (see Figure 1 for a logical application service, accessed by ServiceNow AI Acview and Figure 3 for an implementation view). This tions (and other clients) through a REST-based API. separation is necessary to achieve two properties simul- Given a request type (e.g., “resolution guidance” or taneously: (i) the ability to ground AI outputs in op- “incident summary”), the agent orchestrates a seerational evidence drawn from multiple ESnet sources, quence of tool calls: querying MCP endpoints for and (ii) operational reliability through controlled tool ticket context, retrieving evidence from the semantic search layer, optionally incorporating additional coninvocation and human oversight. textual entities such as configuration item (CI) identifiers, and performing LLM-based synthesis constrained 3.1 Layered components and data flow by the retrieved evidence. Prompt templates are verThe implementation comprises six functional compo- sioned alongside the agent code and selectable per renents, the first four of which are likely to be used in quest, so that prompts are managed as engineered, other AI-related projects, while the last two are more testable artifacts rather than ad hoc experiments (see specific to the particular tasks listed in Table 1. The Lesson 1 in Section 6). Further architectural details following is a brief description of these six components. are described in subsection 4.2.
Orchestration and event loops
The key operational principle is a controlled event loop: ServiceNow requests information from the agent as tickets are created or updated, and the agent responds by performing retrieval and generation steps under explicit tool constraints. This approach keeps the NOC workflow stable – operators see AI outputs as structured additions within the incident record rather than relying on a separate decision-making system. It also supports reproducibility: each AI response correChat interface A locally hosted web interface sponds to a request type, ticket context snapshot, and (chat.es.net) provides the environment for interactive evidence retrieval trace. development and operator-facing experimentation. It is implemented with an open-source tool known as Li- MCP tools for accessing ESnet data sources ORbreChat 2 . It supports prompt iteration, MCP-based BIT’s agentic layer reaches operational data through a set of Model Context Protocol (MCP) tools, each 2 Available at https://www.librechat.ai/ exposing a narrowly scoped interface to a specific ESnet data source. A ServiceNow MCP tool provides read access to incident narratives, work notes, and 5
Section 15
Agent
Layer
ServiceNow MidServer
RooCode, Cline, etc.
Librechat
FastAPI Agents
Agents
chat.es.net
agent-dev.es.net
Inference
Layer
Atlassian
LiteLLM Model Proxy
Tool Proxy litellm.es.net
Similarity Search
OpenAI
ServiceNow Orchestrator Tools (MCP) Stardust ESDB NSO
Chat
Vectorized
Confluence
Vectorized
ServiceNow
Vectorized
Slack
Anthropic
Vectors Google Gemini Embedding Model
Retrieval Augmented Generation (RAG)
Figure 3: Detailed view of the ORBIT system, with data access functions expanded to show each MCP service available as of June 2026. The ServiceNow AI Actions, invoked through the MID server, are the only ORBIT-specific component; all other components are shared infrastructure designed to support future use cases as well.
alarm/context fields required by each use case, as well as controlled write-back pathways through predefined ServiceNow mechanisms rather than ad hoc ticket modifications. An ESDB MCP tool returns equipment and configuration information for network devices referenced in an incident. There are a small number of MCP tools for various other networking functionalities, not called out in Table 1. Complementing these structured-data interfaces, a unified enterprise semantic search layer vectorizes and indexes textual knowledge sources, including Confluence procedures, ServiceNow operational records, and Slack discussions, to support both semantic similarity and hybrid search strategies that improve context matching beyond keyword retrieval. By combining dense and sparse embeddings with lexical retrieval and reciprocal rank fusion, the search layer handles query styles ranging from free-text procedural questions to exact identifier lookups. The current deployment shares infrastructure with the Stardust network telemetry system [3] but indexes only textual sources; incorporation of structured network telemetry into the retrieval layer remains future work. The search architecture and query pipeline are detailed in subsection 4.4.
3.3
Safety and Security
Security is an active area of development for ORBIT, with ongoing work toward alignment with ESnet’s institutional AI guidance for Large Language Models operating within enterprise environments. Given the access-control and audit requirements that come with those environments, security controls fall into two categories: those already implemented in the current ORBIT Server and those planned for future implementation. Implemented controls The current ORBIT Server provides immutable logging of all system interactions, tool configurations, and invocation arguments, providing a full audit record suitable for forensic review and ongoing performance monitoring. It also includes prompt injection mitigations, such as input validation and output filtering, to prevent sensitive material from appearing in model responses. Pending implementation Service accounts are pending scoping to a predefined set of validated actions rather than broad operational permissions, with write scope to be restricted to prevent ad hoc or unau6
thorized modifications to system state. Data retrieval will similarly be bound to what is relevant to the requesting user’s context, preventing privileged information from surfacing where it does not belong.
3.4
Governance, and evaluation
Several design elements support evaluation and governance. First, the system is human-in-the-loop by construction: it provides suggestions and draft text, while operators remain responsible for acceptance and final decision-making. Second, it includes feedback mechanisms in the ServiceNow workflow so that operator judgments of usefulness and relevance can be collected without separate tooling. Third, it logs LLM calls and exports usage/performance telemetry using OpenTelemetry, enabling time-series analysis of system behavior and cost signals over the research period. These logs support deployment decisions by characterizing not just whether the system runs, but how reliably it executes tasks, what resources it consumes, and how operators interact with its outputs. Finally, security requirements shape integration choices. Communications between components occur over HTTPS with ESnet-approved authentication methods, and ESDB access is protected via filtering in MCP interfaces and controlled network access patterns during development. Write operations occur only through the ServiceNow MID server’s secured workflows with attribution, auditability, and predefined action sets. Together, these constraints support safe operational deployment while preserving the ability to instrument the system for rigorous usability and performance measurement.
3.5
Figure 4: A sample AI Action Form.
proach. In particular, as the underlying data integration layer becomes available and more stable, we will be better positioned to reassess the effort required to realize the task set in Table 1. This flexibility allows us to pivot within the allocation—prioritizing the tasks that show the strongest usability and performance evidence, while potentially incorporating additional highimpact tasks that can be implemented without extensive development time.
4
Main Components
Following the high-level overview provided in the previous section, this section provides more details on the six key components of the ORBIT system: (1) the ServiceNow client, (2) the ORBIT agentic service, (3) the chat interface, (4) semantic search, (5) the MCP tools for direct data access, and (6) LiteLLM for model routing and accounting.
4.1
ServiceNow Client through Agentic API
A central design principle of ORBIT is that AI assistance must meet operators where they already work— inside ServiceNow—rather than introducing another tool. Our primary design goal with the ServiceNowORBIT integration was therefore to provide a transparent interface for AI action tooling, including prompt design and trigger points 3 throughout the ServiceNow platform (see Figure 4). Users with a specific role can create, test, modify, and evaluate AI actions in real-time with real data. ServiceNow AI actions are context-aware, which enables field values to be inserted into each prompt to reference the specific ticket
Implementation Flexibility
To support exploratory work within the fixed project duration while enabling incremental learning, ORBIT follows a phased development model. The team first iterates on query/prompt development and evaluation, then integrates the resulting capabilities into the agent workflow and performs system-level testing. Across phases, we refine the prompts, retrieval patterns, and tool orchestration based on both operator feedback and measured execution behavior, so that evaluation results can be rapidly translated into prompt optimization and near-term decisions. 3 For developer documentation, see https: As an exploratory project, the team also remains //www.servicenow.com/docs/r/build-workflows/ explicitly flexible in scope and implementation ap- workflow-studio/flow-triggers.html. 7
or data element the prompt is running against. Once Avoiding response pollution Originally, AI action crafted and optimized, AI actions are integrated into results were written to the work_notes field of incident records. Because work notes are a key input to the system through one of three mechanisms: actions such as “Summarize Incident,” a previous AI • Business rules that fire automatically on de- response would be included in a subsequent invocafined conditions (e.g., creation of a new incident tion, polluting the output with its own prior generatask), requiring no operator action; tions. As a solution, we created a custom AI Notes journal field to store AI action results. This resolved • Scheduled jobs that run at predetermined inthe response pollution problem and also helped optervals; or erators distinguish AI-generated content from human• UI actions that can be triggered on demand by authored entries in the ticket activity log—a separation that is equally important for preserving the integrity of a NOC operator. evaluation measurements (Section 5). When an AI action is triggered on a specific record, a prompt is composed from the prompt template as- 4.2 ORBIT Agentic Service sociated with the action, resolving any field variables it contains, and is passed to ORBIT via a REST-based The ORBIT agentic application service functions as API through a ServiceNow MID Server. The current the system’s centralized reasoning hub, responsible templates are listed in Table 2. The result is then for integrating data from multiple operational sources, stored in a field defined in the AI Action and appears orchestrating agentic workflows, and delivering AIdirectly in the ticket’s activity log, visible to the entire generated outputs to downstream consumers. Its core team. Responses are logged as AI Intentions, provid- responsibilities include querying MCP endpoints for ing a body of knowledge that can be reviewed for ac- ticket context and equipment metadata, performing curacy and ranked (Good, Meh, Bad) with reviewer semantic searches across vector databases, managcomments. This built-in feedback mechanism sup- ing interactions with large language models through ports systematic quality tracking, enabling the team to LiteLLM, and synthesizing the results into actionmeasure output quality over time and identify where able summaries and resolution guidance via RetrievalAugmented Generation. prompts or model selection require refinement. The service is designed for extensibility: its architecture supports the incremental addition of new data Model selection Each AI action can be configured connectors and query services, enabling broader ESnet to invoke a specific model drawn from a table of availoperational tasks beyond the initial ORBIT scope. Beable models, all routed through the LiteLLM gateway cause the four elements required for each execution— (Section 4.6). The current default for most production tools, prompt template, language model, and agent actions is Claude Opus. Because model choice is conexecutor—are all selected per request rather than comfigurable per action, the team is collecting performance piled into the service, the same server functions as statistics to inform cost-capability trade-offs, for exa general-purpose agentic composition service. New ample, assigning a lighter-weight model to straightforworkflows can be introduced by registering additional ward summarization tasks while reserving more capaMCP tools and prompt templates without modifying ble models for nuanced resolution guidance. the service itself. By hosting this infrastructure within ESnet’s established environment, the system maintains Operator control Throughout this integration, the strict compliance with security policies and access conoperator always remains in control. AI-generated con- trols for sensitive data. A dedicated server also abtent is presented as a suggestion that operators can stracts integration complexity from client applications, accept, edit, or reject; the system functions as an as- allowing ServiceNow AI Actions, the chat interface, sistant, not a decision-maker. This human-in-the-loop and future consumers to invoke the same reasoning design preserves accountability and supports the gov- capabilities through a uniform API. ernance requirements described in Section 3.3.
8
Table 2: Prompt templates currently in use. The first five are for the tasks specified in Table 1. The remaining ones were developed during software development (c.f. subsection 6.4). In these templates, the variable ${number} will be substituted by the ticket/incident number and ${incident} refers to the parent incident from which the new task is derived.
Action name Summarize incident for hand-off Generate ticket timeline Suggest alarm procedure Recommend incident priority Propose Final RFO
Caused-ByChange Analysis
Change Analysis
Prompt Template Summarize ${incident} for escalation task ${number}. Format the response as plain text. For ${number} in ServiceNow, generate a timeline of the ticket with all actions taken and resolution information for inclusion in an after action report. Search the wiki docs for alarm procedure that are relevant for ${number}. Also review the history and emails in the ticket to see what has already been done. Suggest the next step. Be brief. Format the results as plain text. For ${number} in ServiceNow. Also review the documentation in the wiki about ticket and alarm priorities. Recommend if any changes are needed to the priority of the incident or if the current priority is correct. Respond with a one sentance answer. For incident ${number} in ServiceNow, propose the required fields for resolving the Incident. Consult the appropriate Confluence wiki documentation in making the determination. Specifically, state a proposed value for RFO, Resolved By, and a brief one or two sentence summary for Resolution Notes. Respond in plain text with no markdown formatting. Look up the Incident ${number}. Also look up the CHG which is referenced in the Caused by Change field (which is named ’caused_by’ in the API). If that field is blank, just report that and be done. Otherwise, list if the Incident was created during the planned start and end times of the CHG and report that. Include the timezones. Next, check to see if the Configuration Item of the Incident is included in the list of Affected CI’s for the CHG. Use sys_id’s for that analysis. If you find that the Incident was opened during the window and the Incident CI was in the Affected CI list of the Change then double check the results. Report on that as well. In the outputs, never include the raw ’sys_id’ fields. Always use a short name or description or something meaningful to a human. Format the response in plain text. Do not use markdown. Look up the "Incidents Caused by Change" related list of incidents. list if each of the Incidents was created during the planned start and end times of the CHG and report that. Include the timezones. Next, check to see if the Configuration Item of the Incident is included in the list of Affected CI’s for the CHG. Report on that as well. In the outputs, never include the raw ’sys_id’ fields. Always use a short name or description or something meaningful to a human. Format the response in plain text. Do not use markdown.
9
Figure 5: Overview of the ORBIT application service and its web dashboard. Through a series of connectors, the service interacts with MCP tools for data access and with AI models for reasoning and planning based on input prompts.
(a) An example agentic action trace with 12 steps, including per-step time measurement and token usage.
(b) Detail of a single reasoning step, including the chain of thought, tool call with parameters, and tool output (omitted for brevity).
Figure 6: A sample execution trace from the web dashboard for an instance of the task 3 “Suggest alert procedure” for a ticket.
10
4.2.1
External Interactions
As illustrated in Figure 5, the ORBIT application service comprises three internally developed components, interfaces with three existing ESnet infrastructure services, and connects to five external MCP or Elasticsearch endpoints. On the client-facing side, the service provides the backend for chat.es.net, enabling interactive ticket resolution, and receives API calls from the ServiceNow MID server via a Python client or direct POST requests. Because the MID server is pre-existing infrastructure, integration required coordinating with the ESnet ServiceNow team to implement targeted feature extensions rather than deploying new middleware. On the data-facing side, the service gathers operational context through a set of specialized MCP connectors and search interfaces: a ServiceNow MCP server for reading ticket narratives, work notes, and alarm fields; an ESDB MCP server for equipment and configuration metadata; Stardust MCP endpoints for router and interface telemetry; and dedicated semantic search engines backed by Elasticsearch for retrieving correlated Wiki pages and historically similar tickets. The reasoning agent autonomously selects which of these tools to invoke at each step. To reduce invocation overhead, the architecture explores importing MCP tool definitions directly into the reasoning loop via specialized adapters, eliminating the need for intermediary adapter servers on the host machine. All LLM calls are routed through the LiteLLM gateway (Section 4.6). 4.2.2
Internal Components
FastAPI server and request handling The service is built on FastAPI [32], which handles client requests, invokes the core reasoning algorithm, and enforces security parameters. It exposes distinct endpoints for OpenAI-compatible chat requests and MID server API requests. Payloads encapsulate user inputs and vary with dynamic system factors—for example, whether the server fetches ticket information dynamically from a supplied identifier or expects the client to provide full ticket context upfront, and whether the end user may select a specific language model. The FastAPI server routes processed inputs to an agent executor, which returns a final response formatted as a plaintext string for chat interfaces or as a structured JSON payload for the MID server.
Security and authentication Access control follows the principle of least privilege. At the network level, the server is deployed within a protected ESnet segment accessible only through the corporate VPN. At the application level, machine-to-machine authentication uses static, token-based bearer credentials; authorized clients receive unique, long-lived secret tokens stored in approved secret managers rather than in source code. For authorization, the server associates each request with the initiating end user and forwards credentials as JSON Web Tokens. Downstream services perform independent authentication and authorization, returning standard HTTP 403 responses for unauthorized actions. All communication between clients and the server is encrypted using Transport Layer Security in accordance with ESnet policy. 4.2.3
Reasoning Algorithm and Tooling
The reasoning engine implements the ReAct algorithm [47], which interleaves reasoning traces with action steps to iteratively resolve ticket queries. Built on LangChain [16], the architecture is model-agnostic, though it retains the flexibility to adopt model-specific optimizations should a highly specialized model be warranted. Each execution requires four elements: a suite of operational tools, a versioned prompt template tailored to the task, a designated language model, and an agent executor. The system prompt governs the model’s operational logic and tool-selection strategy and is maintained separately from the raw user inputs received through the chat interface. To support concurrent users, the FastAPI server combines an asynchronous event loop with a stateless agent design: each incoming request instantiates an independent agent with no persistent session history. While session-based or agent-pool architectures could offer persistent memory or improved resource utilization, the stateless approach ensures clean, isolated execution for every request in the production environment. 4.2.4
Execution Traces and Continuous Monitoring
Every tool invocation and reasoning step is recorded by the service’s web dashboard (Figure 6), which provides a detailed execution trace for each agent run. For every request, the dashboard captures the sequence of tools
11
called, the data accessed, token consumption, steplevel timing, and an estimated execution cost. This instrumentation serves two purposes: it gives developers the information needed to analyze and debug individual agent runs, and it surfaces system-wide patterns, such as frequently invoked tools, latency bottlenecks, and cost outliers, that guide prompt optimization and resource planning. Beyond ad hoc debugging, the traces also underpin a continuous monitoring regime. Drawing on feedback from NOC engineers across more than 200 instances of ServiceNow AI actions, the team distilled a regression suite of approximately four dozen tests. These tests are executed daily against productionrepresentative tickets, providing longitudinal performance metrics that detect prompt regressions, toolselection drift, and latency changes before they affect operators.
Figure 7: A high-level illustration of the semantic search pipeline in the Elasticsearch platform.
exploration—while keeping interactions inside ESnet’s trust boundary rather than on external commercial services. It therefore plays a dual role: a productivity tool for the wider organization, and a low-risk proving ground where the prompts, agents, and integrations that support ORBIT are developed and refined.
4.4 4.3
Chat Interface
Our team has also implemented a chat interface hosted on chat.es.net, a conversational AI platform built on the open-source LibreChat project. Within the context of ORBIT, its primary contribution has been to provide an accessible, interactive interface for experimentation: it lets users iterate on prompts, compare model behavior, and validate agent and tool integrations without writing code or standing up their own client. This lowers the barrier to participation, allowing both engineers and non-developers to explore how large language models respond to operational network data and to refine the prompting strategies that downstream ORBIT components ultimately depend on. The platform brokers model access through the LiteLLM proxy (Section 4.6) and exposes ESnet data sources through MCP server integrations (Section 4.5). Notably, because access is gated by ESnet’s single sign-on, chat.es.net is currently the one path in which tool invocations can be bound to an authenticated end-user identity rather than a shared service credential—a distinction we return to in Section 4.5. This makes the platform a natural staging ground for testing tool integrations before they are promoted into the more tightly scoped, production-facing agents described in Section 4.2. More broadly, chat.es.net serves as ESnet’s general-purpose AI workbench, giving staff a sanctioned, secure environment for everyday tasks— summarization, drafting, code assistance, and data
Semantic Search
ESnet’s Hybrid Semantic Search platform unifies operational knowledge currently scattered across ServiceNow, Confluence (wiki), and Slack into a single, AIpowered retrieval layer. The system is built on Elasticsearch and served through existing interfaces (e.g., LibreChat and MCP servers), enabling users to issue natural-language queries without manually selecting a target system (see Figure 7). 4.4.1
Indexing pipeline
Data is ingested into source-dedicated Elasticsearch indices to preserve source-specific structure and allow per-source tuning. • ServiceNow content is exported via a custom script that captures resolved/closed incidents and key fields (e.g., short description, narrative, resolution notes). • Confluence and Slack are synchronized using Elastic’s native connectors. For semantic retrieval, the platform generates embeddings using carefully selected fields rather than embedding all text. Field selection is critical for precision: ServiceNow emphasizes incident descriptions and resolution text, while Confluence emphasizes page titles and bodies, and Slack emphasizes message text and thread context. The embedding stage runs two complementary models:
12
fields. Because term queries return no results when the query does not match a stored identifier exactly, this component is naturally inert for free-text queries and contributes only when structurally appropriate.
1. ELSER (Elastic Learned Sparse EncodeR) produces sparse representations by expanding text into weighted token distributions over a highdimensional vocabulary, capturing learned associations that enable retrieval across vocabulary mismatch.
4.4.3 Fusion and final ranking 2. E5 (Multilingual-E5-Small) produces dense vector embeddings that encode semantic meaning, The four ranked lists are merged using Reciprocal Rank enabling similarity search based on conceptual Fusion (RRF). RRF combines rankings using relative proximity rather than term overlap. positions rather than raw score magnitudes, avoiding score-scale incompatibilities across retrievers. The reThese embeddings are indexed to support both sparse sult is a single consensus ranking that remains robust semantic and dense semantic retrieval, complementing across diverse operational query styles—from free-text the lexical retrieval provided by BM25. questions to structured identifier lookups. 4.4.2
Hybrid querying architecture
4.5
The system issues a single query containing four subretrievers, each optimized for different query characteristics. Elasticsearch executes these sub-retrievers internally and merges their ranked lists via RRF before returning results: 1. ELSER sparse retrieval uses a learned sparse encoder to expand both documents and queries into weighted token distributions over a highdimensional vocabulary. These expansions capture learned associations rather than synonyms, enabling retrieval of semantically relevant incidents even when no query terms appear verbatim in historical records — without requiring explicit synonym configuration.
Tools for Direct Data Accesses
ORBIT’s reasoning agent accesses operational data through a set of purpose-specific tools, each scoped to a well-defined data domain rather than exposed as a general-purpose query interface. The current tool set comprises: • A ServiceNow tool that retrieves incident payloads—including narratives, work notes, and alarm/context fields—for a given ticket identifier; • An ESDB tool that returns equipment and configuration metadata for network devices referenced in an incident;
• Semantic search tools that retrieve correlated 2. E5 dense retrieval performs approximate nearestConfluence Wiki pages and historically similar neighbor (ANN) search for semantic similarity, tickets (Section 4.4). even when no relevant terms overlap. Vectors are indexed using BBQ-HNSW, which comEach tool is implemented as an MCP tool, ensuring bines 1-bit quantized candidate retrieval with that the agent receives structured, validated responses automatic oversampled rescoring against fullwhile data owners retain explicit control over which precision vectors to preserve recall under comfields and operations are reachable. Restricting each pression. tool to a narrowly defined operation—rather than ex3. BM25 with field boosting provides lexical preci- posing a broad query surface—is a deliberate security sion for exact technical terminology, with boosts decision: it minimizes the attack surface and elimireflecting that certain fields (e.g., incident sum- nates the need for fragile input validation on free-form maries or root-cause fields) are more semanti- queries. Because these tools were developed incrementally cally informative. alongside existing ESnet systems, they are reached 4. Exact identifier matching is a lightweight through three distinct access mechanisms, each sekeyword-based retriever for structured identifiers lected to match the requirements of a particular in(e.g., incident/ticket IDs, circuit IDs, CI names), tegration context: implemented as exact term queries on keyword 13
Gateway architecture The LiteLLM gateway is deployed as a GitOps-managed Kubernetes application on ESnet infrastructure, with continuous delivery handled by ArgoCD. It exposes an OpenAI-compatible API endpoint, providing broad compatibility with existing AI tooling and developer workflows without requiring • Gateway-brokered tools are accessed through custom client integrations. the LiteLLM MCP gateway (Section 4.6), which Model traffic is routed across four provider groups: centralizes access control, usage accounting, and • Anthropic Claude models (Haiku, Sonnet, Opus) observability across the broader AI ecosystem, via Google Vertex AI, making each tool reusable by applications beyond ORBIT. • Google Gemini models via Vertex AI, • Identity-gated tools are surfaced through • third-party commercial and open-source models chat.es.net (Section 4.3), which inherits the available through Vertex AI, and platform’s OpenID Connect (OIDC) authentication and binds each tool invocation to an au• on-premises models hosted by Lawrence Berkethenticated end user. ley National Laboratory’s CBORG service. • In-process tools are imported directly into the agent’s reasoning loop (Section 4.2), avoiding the overhead of an intermediary adapter and providing low-latency access for tightly coupled, high-frequency invocations.
Each mechanism provides adequate security and functionality within its own scope; collectively, they reflect the practical reality that operational AI tooling must balance competing concerns—invocation latency, centralized governance, and end-user identity propagation—that no single integration pattern satisfactorily addresses today. Identity propagation illustrates this tension: requests traversing machine-tomachine boundaries carry a forwarded JSON Web Token that the receiving service validates independently (Section 4.2), whereas requests originating from an interactive session arrive already bound to an authenticated user. Converging these mechanisms behind a unified, identity-aware access layer is an area of active interest both at ESnet and across the broader community; we discuss this further in Section 8.
4.6
Model Access Gateway
ORBIT operates within a centralized AI gateway ecosystem built around LiteLLM, an open-source model proxy and routing layer that provides unified access to large language models and AI tools across ESnet’s infrastructure [14]. Rather than requiring each application to manage its own provider credentials, model selection logic, and usage accounting, the gateway consolidates these concerns behind a single authenticated entry point. This allows for enforcing usage policies, enabling cost accountability across users, projects, and organizational groups, and ensuring consistent security governance. The key components of this ecosystem are shown in Figure 3.
Commercial requests are routed through FedRAMPcompliant Google Cloud and AWS infrastructure, providing data-handling protections consistent with institutional requirements. On-premises CBORG models offer zero-cost alternatives suitable for routine workloads, while cloud-hosted commercial models serve more demanding or complex tasks. This tiered arrangement allows the team to balance capability against cost on a per-task basis. Authentication and access control The gateway issues virtual API keys scoped per user, application, or project. Each key carries role-based access control (RBAC) policies that govern which models, tools, and data resources it may reach. Machine-to-machine workflows are supported through dedicated API keys, enabling automated pipelines and agentic systems to authenticate without user intervention. Fine-grained downstream key management allows ESnet to retain control over upstream provider credentials while distributing access broadly: when backend resources change, such as when new models are added, tools are updated, or providers are rotated, end users and applications require no re-keying, as the gateway handles routing transparently. MCP tool integration Beyond model access, the gateway proxies MCP tools that expose internal ESnet data sources, monitoring systems, documentation repositories, and operational APIs to LLM-enabled applications. These tools are aggregated into a unified namespace accessible through a single authenticated
14
endpoint. The gateway enforces tool-level authorization policies, ensuring that each application can reach only the specific tools its API key permits. This capability addresses a significant governance gap, as many MCP tool implementations lack native authentication mechanisms. Tool sets can further be filtered to the subset relevant to a given application, reducing unnecessary context consumption during LLM interactions. Usage tracking and cost accounting The gateway records all model invocations and MCP tool calls, attributing resource consumption to individual users, API keys, teams, and projects. Usage data is persisted in a Patroni-managed PostgreSQL cluster, providing durable records for reporting and auditing. Administrative dashboards offer real-time visibility into spending patterns, model utilization, and tool invocation rates, while configurable spending limits and alerts enable project managers to enforce budgets proactively. This accounting infrastructure is particularly important given the mixed cost profile of the model portfolio: CBORG on-premises models carry no direct cost, whereas commercial cloud models are billed per token and must be tracked against institutional contracts and per-user spending limits. Security and compliance The gateway serves as a centralized security enforcement point, consolidating request and response logging, prompt-injection protection, data-leak prevention, and a comprehensive audit trail for all model and tool interactions. This unified visibility enables security teams to detect anomalous usage patterns and enforce data-handling policies consistently across all AI-enabled applications, both within ORBIT and across the broader ESnet environment.
5
Performance and Usability
This section reports the experimental results for ORBIT, an AI assistant integrated into an NOC incident workflow to support six operator-selected use cases, with emphasis on Intelligent Resolution Guidance and Adaptive Incident Summarization. The evaluation is designed to address a core concern in production AIOps: improvements must be demonstrated through operational usability and measurable performance, not through model novelty alone. Accordingly, our experiments quantify (i) how effectively ORBIT reduces operator effort, (ii) how reliably it executes
the intended workflow actions under real operational constraints, and (iii) how prompt and orchestration choices affect both response quality and system behavior. Usability and adoption measurements are collected through structured user evaluation with NOC operators. In addition to survey-based feedback (e.g., relevance/usefulness ratings), we also use observational studies to capture how operators interact with ORBIT outputs in ServiceNow—particularly whether the assistant helps operators find correct context faster, whether it changes the operator’s search-and-retrieval process, and whether outputs are accepted, edited, or rejected. These measurements are paired with interface-level signals that reflect real workflow engagement, such as acceptance patterns and user feedback on generated suggestions. Performance measurements focus on actionable system characteristics that are observable during workflow execution. We track end-to-end responsiveness for ORBIT actions, the number of underlying agent/tool steps executed per request, and resource consumption proxies such as LLM token usage (and related cost considerations). While global metrics like MTTR may be difficult to attribute cleanly due to noisy incident data and limited sample sizes, we instead measure targeted timing that directly reflects operator work—most notably the time operators spend researching and validating ticket context. We also instrument latency and retry behavior to characterize the reliability of agent execution in the presence of stochastic model outputs. Finally, the experiments include a dedicated analysis of prompt optimization. We treat prompts as executable artifacts (“prompts are code”) and evaluate prompt variants using controlled incident replays. Results show that engineered, stepwise prompts can reduce the number of agent actions and improve firsttry correctness, while careful orchestration mitigates stochastic failures (e.g., incorrect tool selection) observed when prompts and tool descriptions are ambiguous. Together, these measurements characterize ORBIT along dimensions that matter for deployment decisions: not only whether outputs are plausible, but whether they are consistently useful, grounded, and operationally efficient enough to inform next-step deployment and broader AIOps investment.
15
It is important to note that these invocation counts and timing statistics do not capture output quality. As part of the evaluation, NOC engineers have reviewed the AI-generated outputs for hundreds of incidents, asDEV 41 sessing whether the summaries, procedure suggestions, NOC 12 and priority recommendations were accurate and operSYS 116 ationally useful. That review process is ongoing, and a more systematic quality evaluation—covering output accuracy, operator acceptance rates, and task-by5.1 Statistics from ServiceNow AI Actions task effectiveness—is planned for the coming months We begin our experimental measurements with the as part of the project’s next evaluation phase. Next, AI Actions from the ServiceNow system for handling we examine the components supporting these AI Acincidents reported to ESnet. The evaluation period tions in more detail. covered in this section runs from January through June 2026 (approximately six months). As of the writ- 5.2 ORBIT Server Usage ing of this report, the ServiceNow system records a total of 169 AI Actions over that period. The statistics reported here were gathered during the development and initial deployment phase and largely reflect activity to date; further changes are expected as prompt optimization and broader operator engagement continue. In subsection 4.1, we provided a detailed description of how the ServiceNow AI Actions might be triggered, for example by an individual user through UI Actions or by systems through Business Rules. Table 3 shows that nearly 70% of the total 169 AI Actions are triggered automatically (116 out of 169). Only about Figure 8: Daily total requests handled by the ORBIT Server 7% of AI Actions are triggered through UI Actions, and from mid-May to June 2026. The generally increasing use is many users appear to have tried it only once. Nearly a primarily due to the addition of new automated testing. quarter of the AI Actions are triggered by developers, Figure 8 shows the overall utilization of the ORBIT which is not surprising given that these statistics were server environment over time. It highlights peak usgathered during the development phase of the project. Among the currently recorded AI Actions, Table 4 age periods, which generally correlate with active troushows a breakdown by defined tasks (see Table 1 bleshooting sessions or testing phases, giving a clear and Table 2). Note that the time reported here are view of when the system experiences the highest load. recorded by ServiceNow and more detailed time break- Throughout May 2026, more regression test cases were down would be available from the agentic application developed based on NOC operator feedback. The total number of these test cases reached about four dozen service, as described next in subsection 5.2. The most frequently invoked task is Task #3 (sug- by June. These test cases are executed on a daily gest alarm procedure), followed closely by Task #4 schedule. Since the number of requests to the OR(recommend alarm priority). Both are recommenda- BIT Server stabilized around 400, which is much larger tion tasks that require the ORBIT system to corre- than four dozen, our observation is that there are more late information from multiple sources and thus ex- requests from other sources beyond ServiceNow AI Acsubsection 5.1), such as chat.es.net and hibit large variation in execution time. From Table 4, tions (see 4 we see that their standard deviations (StdDev) ex- Slack . Figure 9 illustrates the spread of total request times ceed their average execution times. In contrast, the from start to finish. Most agent executions cluster summarization tasks (Tasks #1 and #2) have smaller 4 standard deviations relative to their average execution In collaboration with the Charles Shiflett of the Platform time, which indicates that these tasks involve more Engineering group, ORBIT is available as an AI Agent in Slack starting in mid-May 2026. predictable data sources and run more consistently. Table 3: The number of times ServiceNow AI Actions are invoked by different types of users. Note “DEV” for developers, “NOC” for NOC engineers, and “SYS” for automated actions triggered by ServiceNow rules.
16
Table 4: Statistics of ServiceNow AI Actions available as of June 2026. The numbered actions are those defined in Table 1, while the un-numbered ones were recommended by NOC engineers after the start of the project.
Task 1 2 3 4 5
Count
Summarize incident Generate time line Suggest alarm procedure Recommend alarm priority Propose Final RFO Caused-By-Change Analysis Change Analysis
29 6 55 48 8 21 2
Duration (s) Average StdDev 30.4 16.3 61.3 48.0 50.5 59.0 33.1 50.8 27.4 14.1 35.0 17.5 88.5 122.3
Figure 9: Distribution of total request durations on the ORBIT Server, highlighting typical execution latency and long-tail outliers.
Figure 10: Distribution of execution steps per request on the ORBIT Server, representing the depth of the agent’s iterative reasoning loop.
around a predictable median duration of a few seconds, while a noticeable long tail indicates complex queries requiring extensive tool use, retry logic, or prolonged language model generation time. Figure 11 reveals the agent’s reliance on specific external systems to achieve its goals. The ServiceNow ticket retrieval tool and various Confluence standard operating procedure searches are the most frequently invoked actions, underscoring the agent’s primary operational role in gathering incident context and crossreferencing internal documentation. Figure 10 provides insight into the agent’s reasoning complexity. As shown in Table 2, every ServiceNow AI Action accesses some incident and therefore the MCP tool for accessing the content of an incident is the most used tool, logging a total of over 2500 uses (see Figure 11). The next most popular MCP tool is for querying Confluence documents for NOC procedures, which is used over 1000 times. Besides these commonly used tools, a variety of other MCP tools are used. This actual list of MCP tools is longer than the logical list shown in earlier discussions. One complication we observed with a long list of tools is that the
AI model might have a hard time selecting the right one to use (see a discussion in subsection 6.1), which creates a situation where a request may need multiple steps alternating between language model processing and tool execution to find the right tools to gather information from multiple sources and synthesize the right information to complete the request. Figure 12 contrasts the total time spent on language model generation versus external tool execution. Overall, 27.4% of total execution time is spent on calling MCP tools for data access, while 72.6% of the time is spent on invoking AI models. This suggests that we should investigate the time spent on AI models to understand how this might be optimized to reduce both the execution time and monetary cost.
5.3
Chat Interface Usage
The chat interface is used by the ORBIT team for tasks such as exploring how to construct suitable prompt templates, and by NOC engineers for ad hoc exploration of tickets and ticket-handling procedures (i.e., Task #6 in Table 1). Figure 13 shows the usage statis-
17
Figure 11: Most frequently used MCP tools on the ORBIT Server are for accessing incidents in ServiceNow and procedure documentation in Confluence.
operators supports retaining it as a standalone service independent of the ORBIT project’s outcome.
5.4
Figure 12: Overall time spent comparing large language model (LLM) generation versus external tool execution on the ORBIT Server.
tics of NOC operators through June. The LibreChat system captured 72 conversations over the same period in which ServiceNow recorded only 12 UI Actions, as shown in Table 3. This disparity can be partly explained by the fact that ServiceNow AI Actions cover only a handful of tasks, while the chat interface gives operators much more freedom to explore AI capabilities. Regardless, we observe considerably more interest in the chat interface than in the ServiceNow UI Actions. These patterns suggest that chat.es.net has value beyond its role in ORBIT. Its flexibility makes it a natural general-purpose AI workbench for ESnet staff, and the organic adoption observed among NOC
AI Model Usage
The LiteLLM gateway described in this section realizes the AI Sandbox (WP19) called for in ESnet’s Data and AI report [11]. By providing a single authenticated entry point for model access, usage accounting, and access control, it enables the organization to evaluate demand for shared AI infrastructure across multiple workload types. Figure 14 is a screenshot of our LiteLLM installation showing the cost and usage of requests going through the chat interface (i.e., the component labeled (3) in Figure 1) and the agentic service behind ORBIT (i.e., the component labeled (2) in Figure 1). The time period covered is the whole development period. Figure 14 shows that the agentic service consumed about 876 million tokens while the chat interface consumed about 322 million tokens. Despite the difference in tokens consumed, their costs in dollars are nearly the same. This indicates that it is worthwhile to select the right models to control costs for the enterprise. Table 5 breaks down the LiteLLM usage by workload category. A little over 56% of the requests going through the LiteLLM server come from the chat service and the agentic service, yet these requests consumed less than 5% of the total tokens, which indicates that the remaining workloads consume far more
18
Figure 13: chat.es.net usage by NOC operators by June 2026. Six of seven members were active, exchanging 448 messages across 72 conversations and invoking ORBIT-relevant tools (ESDB, ServiceNow, semantic search over Confluence and Slack, Stardust) 1,220 times without prescribed workflows.
Figure 14: LiteLLM statistics about usage and cost from the chat.es.net and the project ORBIT. Together these two services served just over 404,000 requests using about 1 billion tokens. During the period of time, this LiteLLM server has served more than 714,000 requests consuming over 21 billion tokens.
19
Table 5: LiteLLM usage by workload category over the evaluation period. Coding assistants dominate token consumption despite fewer requests than chat and agentic workloads combined.
Category Chat (chat.es.net) Agentic (ORBIT) Coding assistants Total
Requests (000) 19 372 323 714
Tokens (000,000) 322 876 20,244 21,422
Cost ($) 1,045 1,014 20,516 22,620
tokens per request. Our inquiry into this revealed that most of these requests originate from code assistants that often send complex context for code refactoring and generation. This initial AI Sandbox deployment has attracted considerable attention from ESnet staff, with several engineers reporting perceived productivity improvements in both coding and operational tasks (see also Section 5.5). ESnet management has reviewed the usage statistics and authorized a $500 per month cap for each ESnet staff.
5.5
User Interviews
it as the feature with the greatest immediate impact. • Experience level shapes what “assistance” means. Advanced engineers used AI sparingly but found value in targeted analytical tasks (cause-by-change analysis, incident updates, metric reviews), while less experienced users valued scaffolding such as command syntax hints and procedural reminders. Both groups found the system useful, but for different task types. • Discoverability of data sources is a usability bottleneck. Users were often unaware that certain data (e.g., email context) was already accessible via existing MCP pathways, and they selected among MCP tools without a clear mental model. This friction reduced the realized value of capabilities that were technically available. • Evidence-grounded, actionable outputs are valued. Users cited concrete value when AI produced a correct, operationally actionable insight (e.g., identifying that a peer had recently turned up a relevant circuit). Generic summaries without operational grounding were less appreciated.
During the evaluation phase of the ORBIT work, we conducted structured interviews with our NOC enDirections for improvement identified by operagineers to assess system usability and gather operators. tional feedback. The interviews produced two cate• Workload-level roll-ups, not only per-ticket gories of finding: observations about what operators assistance. A recurring unmet need was the experienced using the current system, and directions ability to summarize all active incidents (e.g., for improvement that operators identified as high-value 15–30 items) with provider updates and next acnext steps. Both categories are summarized below. tions, particularly for shift handoff. This points The overarching usability finding is consistent to a workload-level summarization capability beacross all interviews: operators experience AI as most yond the current per-ticket scope. valuable when it reduces the effort of stitching together cross-system incident context into a coherent, action• Better handling of repeated and recurring able picture. Adoption patterns varied by experience incidents. When tickets reappear without level, but the core value proposition—reducing the root-cause closure, operators re-investigate from cognitive overhead of multi-source context assembly— scratch because historical resolution context is was recognized uniformly. not aggregated into an actionable form. Operators requested that ORBIT surface prior resoluObservations about the current system. tion attempts for recurring incident patterns. • Context stitching is the dominant pain point. Operators must manually reconstruct incident history from multiple disconnected tools (ServiceNow, Slack, email, and various monitoring systems). ORBIT’s cross-source aggregation directly addresses this, and operators recognized 20
• Consolidation of alarming systems. Operators repeatedly requested a unified conversational interface across DNA, Spectrum, Alerta, and TNMS. Integrating additional alarming data sources would be the highest-leverage next integration step.
• Catch-up and notification triage. Operators described recurring “what did I miss” workflows across Slack and email, and flagged noise from non-operational notifications. A digest and triage capability would reduce daily overhead considerably.
for any individual task. Table 6: Comparison of naïve versus engineered prompts for incident summarization. Note that the engineered prompt is about 26.6% faster.
Naïve Prompt 10 78 s 3
Engineered Prompt 4 57 s 0
Metric • Externalization of institutional knowledge. Agent actions Key knowledge—CLI syntax quirks, topology Completion time and vendor specifics such as satellite-based cirRetries to correct query cuit details—was not reliably accessible from static documentation. Operators requested that This result reinforces a practical guideline: verORBIT’s retrieval layer be extended to surface sion your prompts, test them against representative this tacit knowledge more reliably. incidents, and iterate systematically. Within ORBIT, prompts are maintained as versioned artifacts alongside the agent code, and prompt variants are evalu6 Lessons and Discussion ated using controlled incident replays (Section 5) so 6.1 From Prompts to Skills: Managing that regressions are caught before deployment.
Complexity in AI Tool Chains The current industry best-practice to deal with the stochastic behavior in AI tool chain is to emply a technique known as “skills” which is to provide a common set of directives about how to use a specific set of MCP tools to in an application context such as ORBIT. A central engineering insight from ORBIT development is that prompts must be treated with the same rigor as source code: versioned, tested, and iteratively refined. The difference between a marginally useful AI response and a genuinely operational one is almost always traceable to prompt engineering rather than model selection. To illustrate, Table 6 contrasts a naïve prompt with an engineered, stepwise variant for the same incidentsummarization task. The naïve prompt (“Please summarize this ticket with comments and work notes”) required 10 agent actions, 78 s to complete, and 3 retries before the agent located the correct query path. The engineered prompt—which explicitly instructs the agent to retrieve the sys_id first, specifies the fields of interest, and directs parallel queries—completed in 4 agent actions and 57 s (a 26.6% reduction in wallclock time) with correct execution on the first attempt. The existing experiment described in Section 6.1 is to refine the instruction for a specific task. Based on the industrial best-practice, we are in the process of developing “skills” for ORBIT so that shared knowledge such as which tool to use to turn a ticket number into text version of the ticket content is always available does not need to be write into the prompt template
6.2
Managing Stochasticity: Same Prompt, Different Answer
AI models are inherently stochastic—randomness in token sampling is a feature of the architecture, not a bug. In practice, this means that the same prompt applied to the same incident can produce subtly (or notso-subtly) different outputs on successive runs. Two additional sources of non-determinism compound the problem in agentic systems such as ORBIT: • Tool-selection ambiguity. When MCP tool names or descriptions are similar, the model occasionally selects the wrong tool, leading to divergent execution paths even when the prompt is identical. • Retrieval variability. Semantic search via RAG introduces its own layer of non-determinism: slight differences in embedding scores or index state can alter which documents are surfaced and, consequently, the content of the generated response. Several practices help mitigate—though never fully eliminate—this variability:
21
• Consistent, descriptive tool naming with documentation. Clear, unambiguous MCP tool names and rich tool descriptions reduce the likelihood of mis-selection by the reasoning model.
prompt/tool framework, the team created the conditions for rapid, user-driven innovation. The highestimpact use case may not be the one originally planned; it may instead surface from operators who recognize a painful workflow that the platform can now address with minimal additional effort. This experience also • Ongoing refinement of templates and model led to the “Caused-By-Change Analysis” and “Change selection per task type. Different tasks may Analysis” actions visible in Table 4, which were not benefit from different models or temperature part of the original six-task scope. settings; matching these systematically reduces output variance for high-stakes actions. • Richer prompts that guide tool selection. Stepwise instructions that explicitly name the tools to invoke (as in Lesson 1) narrow the model’s decision space and improve first-try correctness.
6.4
The ORBIT team is adopting the industry best practice of encapsulating these mitigation strategies into reusable “skills.” These skills provide a structured way to manage the stochastic nature of AI tool chains by providing a consistent set of instructions for the AI to follow. Stochasticity never fully disappears. The practical takeaway is to design around it: use deterministic anchors (explicit tool routing, structured output schemas, and retry-with-validation logic) wherever possible, and instrument the system to detect when outputs diverge beyond acceptable bounds.
6.3
Emergent Use Cases: The Best Ones Find You
Not all high-impact use cases can be anticipated during project planning. One of ORBIT’s most effective deployments emerged organically from an existing operational ritual: the weekly metrics meeting. Each week, the NOC reviews incidents that fall outside service-level specifications. A recurring analysis task during this meeting is identifying all incidents that resulted from a planned change and then verifying two conditions in real time: (1) whether the associated maintenance window was exceeded, and (2) whether all affected configuration items (CIs) were included in the change record. Performing this cross-referencing manually during a live meeting was slow and errorprone. A NOC engineer collaborated with the Business Automation team to add a ServiceNow UI action that automates this analysis using ORBIT’s AI infrastructure. The implementation took approximately one hour—enabled by the fact that the underlying platform, data integrations, and prompt patterns were already in place. The broader lesson is architectural: build the platform, then watch what your users do with it. By investing in composable data integrations and a flexible
Organic Infrastructure Adoption: Build It and They Will Come
A recurring theme in AI deployment guidance is that infrastructure investments should be justified by projected demand. ORBIT’s experience suggests the opposite sequence: build shared infrastructure for a concrete use case, instrument it, and let demand reveal itself. The LiteLLM gateway (subsection 4.6) was deployed to serve ORBIT’s agentic service and the chat interface – two workloads with a combined 1.2 billion tokens over the evaluation period (Table 5). Without additional promotion, the same gateway attracted coding-assistant workloads that consumed over 20 billion tokens – roughly 95% of total token volume from engineers who discovered the endpoint through word of mouth. Similarly, the chat interface (subsection 5.3) saw 6 of 7 NOC operators adopt it organically, generating 72 conversations and over 1,200 tool invocations (Figure 13), compared with only 12 user-initiated ServiceNow AI Actions in the same period (Table 3). In both cases, the heaviest usage came from workloads and interaction patterns that were not part of the original project scope. Two practical implications follow. First, shared AI infrastructure, including model gateways, authenticated tool endpoints, usage accounting, should be designed from the outset as general-purpose services rather than project-specific utilities; the marginal cost of supporting additional workloads is low once the platform exists, and usage data (Figure 14) provides the evidence base for sustaining the investment. Second, per-workload cost visibility matters: the data in Table 5 show that coding assistants and agentic services have dramatically different token-per-request profiles, and therefore different cost structures; without workload-level accounting, the organization cannot make informed model-selection or budget decisions as adoption scales.
22
6.5
• Focus on security, but do not overcorrect. Overly restrictive policies can stifle the exploratory culture that makes operational AI successful. The goal is to establish security boundaries (e.g., write-scope restrictions, audit logging, role-based access) that protect sensitive data while preserving the freedom to iterate on prompts, tools, and workflows.
Cognitive Barriers to AI Adoption
Perhaps the most unexpected lesson was not technical but organizational. AI is sufficiently new in operational settings that most staff have difficulty envisioning concrete use cases. The prevailing mental model treats AI as a glorified chatbot, and few practitioners are tracking the rapid evolution of agentic capabilities, tool integration, and retrieval-augmented generation. This creates a cognitive barrier: moving from constraint-based thinking (“what can this chatbot do?”) to possibility-based thinking (“what workflows could an AI agent with access to our data sources transform?”) is genuinely difficult without concrete, relatable examples. However, once operators were given a hands-on demonstration of what ORBIT could do with their own tickets and data—particularly the experiences described in Lessons 1–3—the ideation barrier dropped rapidly. New use-case proposals began to flow from the NOC team itself, several of which were implemented within hours. This dynamic carries several practical implications for teams deploying operational AI: • Do not expect your initial use cases to be your strongest. Early use cases serve as existence proofs and learning vehicles; the most impactful applications typically emerge after the team has internalized what the platform can do. • Infrastructure begets opportunity. The more data sources are integrated and the more tools are exposed, the larger the combinatorial space of feasible use cases becomes. Each new MCP server or indexed knowledge base multiplies the options available to both the agent and the human designers. • Embrace rapid iteration. AI development rewards fast cycles of prompt engineering, testing, and deployment. Teams should be organizationally prepared for this tempo, including lightweight approval processes for new AI actions and prompt variants. • Develop organizational policies early. Clear policies for AI usage—covering data handling, model selection, output review, and accountability—provide the guardrails that enable experimentation without unacceptable risk.
7
Review of the State-of-the-art
AI/ML is widely expected to assist Network Operations [10, 23]. However, widely publicized studies document the difficulty of transferring successes from demonstration projects to operational environments [31, 36, 41]. It is therefore highly desirable to ground demonstrations in realistic business use cases, incorporating proper business processes and engaging the right staff [36]. This report primarily focuses on the technical aspects of the ORBIT work, but also touches on data governance and staff engagement. There are many publications showing successful uses of AI technology in various use cases [15]. Here we briefly summarize one specifically focuses on network operations in cloud infrastructure [42]. A featured case study demonstrates the tangible impact on network operations, showing a 60% reduction in false positives and a decrease in Mean Time to Recovery (MTTR) from three hours to 55 minutes. In many of these studies, a "human-in-the-loop" approach is needed to ensure operational safety and trust [4, 26, 40]. In a network operations case study, site reliability engineers validated AI-suggested remedies and restricted auto-remediation to highconfidence predictions [42] to ensure operational safety, prevent "black box" errors, and leverage expert feedback to refine models. Retrieval-Augmented Generation (RAG) for Enterprise Knowledge Retrieval-Augmented Generation (RAG) is a technique that enhances AI language model responses by dynamically fetching relevant, up-to-date information from external knowledge sources at query time—grounding the model’s outputs in real data rather than relying solely on what was learned during training [17]. While this process reduces hallucinations and improves accuracy, particularly in specialized technical domains, it also introduces significant challenges [44]. Key among these
23
are ensuring the model remains robustly grounded, implementing reliable citation and attribution, and addressing limitations when retrieved evidence is incomplete, conflicting, or must be synthesized across multiple sources [49]. Semantic Search over Siloed Operational Data Network operations at ESnet generate vast amounts of heterogeneous data across disparate systems like ticketing platforms, wikis, and chat logs. This proliferation of data silos hampers efficient knowledge discovery and decision-making, leaving critical information underutilized and unsuitable for integrated querying [24]. To address this, there is a pressing need to not only break down these enterprise data silos but also to provide a simple, unified query interface [22]. Such an interface requires robust semantic query understanding to accommodate the natural variations in ESnet’s operational workflows—seamlessly interpreting synonyms, technical jargon, and identifier-based searches to provide operators with a single, coherent view of all relevant institutional knowledge.
[28] showed that LLMs could simulate believable human behavior in sandbox environments through memory retrieval, reflection, and planning mechanisms. These early efforts were later unified by comprehensive surveys—Wang et al. [43] established a fourmodule framework (Profiling, Memory, Planning, Action), while Xi et al. [46] offered a cognitive BrainPerception-Action paradigm that extended the discourse to multi-agent societies and human-agent collaboration. Evaluation Methodologies for Operational AI Recent research and development in large language model (LLM)-based agent evaluation has shifted toward more comprehensive and dynamic assessment frameworks. Yehudai et al. [48] provide a systematic review of agent benchmarking, categorizing evaluation approaches from atomic core capabilities such as planning and tool-use to application-specific and generalist assessments, while emphasizing the growing importance of continuously updated dynamic benchmarks to address the rapid evolution of agent capabilities. Complementing this, Luo et al. [20] present a methodologycentered taxonomy synthesizing over 300 papers, linking architectural foundations to deployment challenges and practical tooling considerations. Domain-specific evaluation has also emerged as a critical area, exemplified by Sun et al. [39], who survey “Data Agents” capable of autonomous statistical reasoning, code generation, and tool-augmented retrieval for data science pipelines. Together, these works highlight a maturing field that increasingly demands multi-dimensional evaluation spanning reasoning, tool integration, real-world applicability, and domain expertise.
Hybrid Retrieval Architectures and Rank Fusion To overcome the limitations of any single retrieval method, our approach utilizes a hybrid architecture that combines the complementary strengths of lexical (e.g., BM25), dense, and sparse retrieval models [9]. The resulting ranked lists are merged using rank fusion, with methods like Reciprocal Rank Fusion (RRF) being particularly effective[9, 6]. RRF enhances overall robustness by rewarding documents that rank well across multiple retrievers — even if no single retriever ranks them first — mitigating the weaknesses of any individual approach. This hybrid system is further augmented by a high-precision strategy for exact identifier match- Measuring Usability, Reliability, and Cost Serving, which handles the critical edge case of structured, ing AI models in an enterprise environment requires a multi-layered, Zero-Trust security posture that adknown-item queries common in operational data. dresses threats from the infrastructure to the model Prompting and Orchestrating LLM Tool Use interaction layer [12, 21, 34]. These approaches typi(“Agentic” Systems) The foundational research in cally start by assuming the traditional network perimeLLM-based autonomous agents emerged from several ter is obsolete, instead enforcing continuous verificaseminal works: ReAct [47] introduced the paradigm tion of identities, devices, and workloads with adapof interleaving reasoning traces with actions, enabling tive, AI-driven access policies [25, 34]. Critically, this models to dynamically plan and adjust based on en- extends to the LLM itself through a dedicated "Secure vironmental feedback. AutoGPT [38] demonstrated AI Gateway," which treats all incoming user prompts fully autonomous task execution by chaining LLM as untrusted [5, 37]. Our current implementation calls with memory and tool use, sparking widespread uses LiteLLM for enforcing Role-Based Access Control interest in self-directed agents. Generative Agents (RBAC), preventing data exfiltration, and detecting 24
adversarial attacks like prompt injection before they reach the model [14]. Beyond robust security, successful operational deployment hinges on comprehensive measurement and governance. This requires diligent instrumentation and telemetry to continuously evaluate system performance against key enterprise metrics: usability (e.g., task success, user satisfaction), reliability (e.g., timeouts, error rates), and cost (e.g., token consumption) [13, 21]. Capturing this data enables reproducible evaluations, provides a clear accounting of the system’s value and resource utilization, and establishes the foundation for effective data governance and iterative improvement in a production setting. Distinctiveness of ORBIT The works surveyed above advance individual aspects of operational AI, anomaly detection and incident triage, retrievalaugmented generation, hybrid search, agentic orchestration, and evaluation methodology, but they largely address these concerns in isolation and evaluate them on curated benchmarks rather than in production operator workflows. ORBIT targets the gap between these demonstrated capabilities and their integrated, measurable use in a real NOC setting. Specifically, (i) it synthesizes evidence across multiple siloed operational data sources (ServiceNow tickets, Confluence procedures, Slack discussions, and network databases) within a single agentic action, rather than retrieving from a single corpus; (ii) it embeds AI outputs directly into the existing ServiceNow incident workflow so that operator acceptance, editing, and rejection provide naturalistic evaluation signals without additional tooling; (iii) it treats prompts as versioned, testable engineering artifacts and evaluates prompt variants through controlled incident replays, making prompt optimization an explicit part of the system evaluation rather than an offline exercise; and (iv) it reports both system-level performance metrics (request latency, token consumption, tool-call depth) and operator-level usability evidence (structured interviews, adoption patterns, feedback signals) from a sustained deployment, providing the kind of workflow-aligned evaluation that prior work calls for but rarely demonstrates end to end.
8
Conclusion and Future Work
This report presented ORBIT, an agentic AI system integrated into ESnet’s NOC incident workflow, and reported on the system design, operational measurements, and engineering lessons gathered during its initial deployment.
8.1
Project Outcomes and Lessons Learned
The project was organized around three objectives: build software to accomplish six specific NOC tasks, engage ESnet staff in exploring AI tools, and gather performance and usability statistics to inform future AI efforts. We revisit each in turn. Software and task coverage ORBIT delivered all six originally scoped NOC tasks (Table 1), and the platform’s composability enabled two additional tasks—Caused-By-Change Analysis and Change Analysis—proposed by NOC engineers and implemented within hours (see Section 6.4 for details, and Table 4 for invocation counts). The system recorded 169 AI Actions in ServiceNow over the evaluation period, of which nearly 70% were triggered automatically by business rules (Table 3). Recommendation tasks (suggest alarm procedure, recommend alarm priority) were invoked most frequently but exhibited high variance in execution time, with standard deviations exceeding their means; summarization tasks showed more predictable performance (Table 4). These patterns confirm that cross-source synthesis tasks are both the most demanded and the most sensitive to prompt and tool-selection quality—a finding reinforced by the skill-refinement results in Section 6, where engineered skills reduced agent actions from 10 to 4 and eliminated retries (Table 6). Output quality is supported by a systematic continuous monitoring regime. Drawing on NOC engineer review of more than 200 ServiceNow AI action instances, the team distilled a regression suite of approximately four dozen tests executed daily against production-representative tickets. This provides longitudinal evidence that the system maintains consistent behavior across prompt updates and model changes, and that regressions are detected before they reach operators (Section 5.2). Staff engagement Adoption data show organic uptake beyond the original project scope. Six of seven
25
NOC operators used the chat interface, generating 72 conversations and over 1,200 tool invocations without prescribed workflows (Figure 13)—substantially more interaction than the 12 user-initiated ServiceNow AI Actions recorded in the same period (Table 3). The LiteLLM gateway attracted coding-assistant workloads that consumed over 20 billion tokens—roughly 95% of total token volume—from engineers who were not part of the ORBIT project (Table 5). User interviews (Section 5.5) identified cross-source context aggregation as the primary value of AI assistance and highlighted tool discoverability as the main usability bottleneck: operators were often unaware which data sources were already reachable and selected among MCP tools without a clear mental model. Performance and cost visibility The instrumentation built into ORBIT—execution traces, per-step timing, token accounting, and the LiteLLM usage dashboard—provided the quantitative foundation for all evaluation results reported in Section 5. A practical finding is that workload-level cost visibility is essential: the agentic service and chat interface together consumed less than 5% of a project’s total tokens, yet accounted for over 56% of requests, while coding assistants dominated token consumption at dramatically different cost-per-request profiles (Table 5). Without per-workload accounting, the organization could not make informed model-selection or budget decisions as adoption scales. Lessons and implications Five engineering lessons emerged from the deployment (see section 6). The ORBIT team has begun to adopt the industry best practice of encapsulating procedural knowledge into versioned, testable “skills”; this approach helps mitigate the inherent stochasticity of AI toolchains. The team also found that the highest-impact use cases may emerge from operators after the platform is available rather than from initial planning, and that staff adoption accelerates sharply once operators see concrete demonstrations with their own data. Together, these lessons argue for investing in composable, generalpurpose AI infrastructure—model gateways, authenticated tool endpoints, usage accounting—and then instrumenting it to let demand and use-case evolution guide subsequent development.
8.2
Recommendations
Based on the evidence gathered, we offer the following recommendations for next steps with AI infrastructure and NOC tooling. The four general-purpose components introduced by ORBIT: the LiteLLM model gateway, the agentic application service, the chat.es.net interface, and the MCP tool layer, have demonstrated sustained demand well beyond the original project scope. The LiteLLM gateway served over 714,000 requests and 21 billion tokens, with 95% of that volume coming from engineering workloads outside ORBIT; six of seven NOC operators adopted the chat interface organically; and the MCP tool layer enabled rapid implementation of new use cases proposed by operators. We therefore recommend that ESnet formally adopt these four components as officially supported organizational services, with dedicated operational ownership, documented SLAs, and integration into ESnet’s standard access-control and cost-accounting infrastructure. The ServiceNow AI Actions, by contrast, recorded only 12 user-initiated invocations from NOC operators during the evaluation period, with most automated actions attributable to business-rule triggers rather than active operator engagement. Output quality evaluation is still in progress, and the evaluation period predates several prompt optimizations. We therefore recommend that a decision on the long-term status of the AI Actions be deferred pending the structured quality evaluation and extended usage collection planned for the coming months (Section 5.1). Finally, transitioning any of these components from exploratory project infrastructure to officially supported services requires careful productization planning. This includes establishing clear operational ownership and on-call responsibilities, defining incidentresponse procedures for service degradation, documenting dependency chains (particularly the LiteLLM gateway’s dependencies on external model providers), and setting up capacity planning aligned with observed growth in token consumption. The engineering lessons in Section 6 provide a starting point, but a dedicated productization plan—covering staffing, governance, security review, and user-facing SLAs—should be developed before any component is formally promoted to production service status.
26
8.3
Future work
The following items address both the technical evolution of the ORBIT platform and the strategic business considerations that will determine whether its infrastructure can be sustained and expanded responsibly. Systematic Evaluation of Mission-Oriented Outcomes. The current evaluation rightly focuses on system performance and usage metrics, but a critical missing piece is a formal, systematic evaluation of the quality and accuracy of the AI-generated outputs, and its impact on ESnet’s operational mission. Without this, the true operational value and potential risks of ORBIT remain unquantified. Future evaluation cycles must focus on understanding the low adoption of the ServiceNow AI Actions and measure outcomes that connect directly to ESnet’s network reliability mission: operator time-to-context before and after ORBIT assistance, reduction in repeated investigative effort for recurring incidents, and shift-handoff quality as assessed by the receiving operator. This framework is a prerequisite for moving ORBIT from a promising prototype to a trusted, production-grade operational tool. Development of a Skill Library. To further mitigate the impact of stochasticity and to ensure the consistent application of best practices, future work should focus on the development of a comprehensive skill library. This library will contain a collection of versioned, tested, and documented skills for each of the NOC tasks that ORBIT supports. This will not only improve the reliability and predictability of the system, but will also serve as a valuable resource for training new staff and for sharing knowledge across the organization. Unified identity-aware tool gateway. A recurring theme across the system design and operator feedback is that ORBIT’s value depends less on any single model or tool than on how cleanly its tools are exposed, discovered, and governed. ORBIT currently spans three distinct tool-access paths—in-process, gatewaybrokered, and identity-gated (Section 4.5)—that trade off latency, centralized governance, and end-user identity propagation differently, with no single path resolving all three concerns simultaneously. A natural next step is to converge these paths onto a unifying, identity-aware tool gateway: a single brokerage point through which tools are discovered, governed, and invoked, and through which the authenticated user’s identity propagates to the data-owning service so that authorization decisions are made against the
real end user rather than a shared service credential. The relevant building blocks are emerging—OIDC token exchange (RFC 8693) for delegating identity across service hops, and the evolving MCP authorization specification—but the standards remain young and reference implementations are still maturing. Operational sustainability and knowledge concentration. The depth of knowledge required to maintain, prompt-optimize, and extend ORBIT currently resides in a small development team. If key personnel leave or shift to other projects, operational continuity is at risk. Future work should include structured knowledge transfer: documented runbooks for common failure modes, a maintained library of annotated prompt examples covering each task type, and cross-training of at least one NOC-side maintainer on prompt development and regression testing workflows. This is a prerequisite for the formal service adoption recommended above. Data Governance and Provider Dependency. ORBIT processes live ServiceNow incident records and ESDB network configuration data. While the LiteLLM gateway routes commercial requests through FedRAMP-compliant infrastructure, there is currently no formal classification policy governing which data categories may be sent to external model providers and under what contractual terms. Future work should establish a data classification scheme aligned with ESnet’s information security policy, implement promptand response-level filtering to prevent sensitive fields (e.g., device credentials, circuit topology) from being included in external model calls, and audit existing AI Action prompts against that classification. Furthermore, many AI workloads currently route through commercial model providers, which could be an operational risk. Future work should establish a systematic model-equivalence evaluation framework that enables workloads to be migrated across providers— or to on-premises alternatives—with measured quality impact, and should define the criteria under which onpremises models become the default for cost-sensitive or security-sensitive tasks.
Acknowledgment This work was supported in part by the Office of Advanced Scientific Computing Research, Office of Science, of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231, and used resources
27
[10] J. Dobie and R. Holder. Network system of systems manager. In 2024 Integrated Communications, Navigation and Surveillance Conference (ICNS), pages 1–14, 2024.
of the Energy Science Network (ESnet).
References
[1] N. Acharya. Artificial intelligence: real challenge [11] C. Guok, E. Balas, S. Balasubramanian, J. Balor boon for network operation center and network cas, J. Daneshamooz, S. Gholba, M. Haberman, security. In ITM Web of Conferences, volume 65, S. Kwang, J. MacAuley, S. Moats, M. Nikahd, page 03001. EDP Sciences, 2024. S. Oehlert, C. Rotermund, C. Robb, G. Stewart, J. Tian, C. Tracy, A. Wiedlea, J. Wu, X. Yang, [2] C. Althati, M. Tomar, and L. Shanmugam. Enand S. Yu. Esnet data and ai workshop report. hancing data integration and management: the Technical report, ESnet, 2025. role of ai and machine learning in modern data platforms. Journal of Artificial Intelligence Gen- [12] R. Huijts and K. Suilen. Institutional ai eral science (JAIGS) ISSN: 3006-4023, 2(1):220– sovereignty through gateway architecture: Imple232, 2024. mentation report from fontys ict, 2025.
[3] E. Balas, A. Lake, S. Faci, J. Kafader, J. Ran- [13] M. Z. Islam and A. Dhanekula. Measuring the dolph, and K. Turner. Stardust network measuresecurity impact of zero trust access controls: A ment components (esnet-stardust-components) mixed-methods study of identity-based policies v1. Technical report, Lawrence Berkeley Na(cisco ise+ ad) and incident reduction. American tional Laboratory (LBNL), Berkeley, CA (United Journal of Data Science and Analytics, 4(06):01– States), 2022. 42, 2023. [4] K. Besigomwe. Human-in-the-loop self-healing [14] I. Jaff, S. Kankute, Y. Jiang, and et al. LiteLLM. systems: Integrating human oversight for auhttps://www.liteLLM.ai/, 2026. tonomous failure detection, repair and system optimization. Cognizance Journal of Multidisci- [15] M. Joy, S. Venkataramanan, M. Ahmed, M. Mark, L. Gudala, M. Shaik, A. K. Pamidi plinary Studies, 5(3):254–267, 2025. Venkata, and V. K. Reddy Vangoor. AIops in [5] I. Brett. Simplified and secure mcp gateways for action: Streamlining it operations through arenterprise ai integration, 2025. tificial intelligence. International Journal of Intelligent Systems and Applications in Engineer[6] S. Bruch, S. Gai, and A. Ingber. An analysis of ing, 12(23s):2175–2185, 2024. also available at fusion functions for hybrid retrieval. ACM Trans. http://dx.doi.org/10.2139/ssrn.5257975. Inf. Syst., 42(1), Aug. 2023. [7] J. Bughin, J. Deakin, and B. O’beirne. Digital transformation: Improving the odds of success. McKinsey Quarterly, 22:1–5, 2019.
[16] LangChain, Inc. LangGraph: Agent orchestration framework for reliable ai agents. https: //www.langchain.com/langgraph, 2024. Accessed: 2025.
[8] R. G. Cooper. Why AI projects fail: Lessons [17] P. Lewis, E. Perez, A. Piktus, F. Petroni, from new product development. IEEE EngineerV. Karpukhin, N. Goyal, H. Küttler, M. Lewis, ing Management Review, 52(4):15–21, 2024. W.-t. Yih, T. Rocktäschel, et al. Retrievalaugmented generation for knowledge-intensive [9] G. V. Cormack, C. L. A. Clarke, and S. Buettcher. nlp tasks. Advances in neural information proReciprocal rank fusion outperforms condorcet and cessing systems, 33:9459–9474, 2020. individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference [18] J. Lipkova, R. J. Chen, B. Chen, M. Y. Lu, on Research and Development in Information ReM. Barbieri, D. Shao, A. J. Vaidya, C. Chen, trieval, SIGIR ’09, page 758–759, New York, NY, L. Zhuang, D. F. Williamson, et al. Artificial inUSA, 2009. Association for Computing Machintelligence for multimodal data integration in onery. cology. Cancer cell, 40(10):1095–1110, 2022. 28
New esnet report looks at ai[19] S. Long, J. Tan, B. Mao, F. Tang, Y. Li, M. Zhao, [29] B. Powell. powered future for scientific networking. https: and N. Kato. A survey on intelligent network //www.es.net/news-and-publications/ operations and performance optimization based esnet-news/2025/ai-data-report/, Aug. on large language models. IEEE Communications 2025. Surveys & Tutorials, 27(6):3915–3949, 2025. [20] J. Luo et al. Large language model agent: A sur- [30] Z. Rackauckas. Rag-fusion: A new take on retrieval augmented generation. International Jourvey on methodology, applications and challenges. nal on Natural Language Computing, 13(1):37– arXiv preprint, 2025. 47, Feb. 2024. [21] Y. Mao, X. Ma, and J. Li. Research on api security gateway and data access control model for [31] G. Rajan. Pilots to scaleT M : A framework for diagnosing and closing the enterprise AI execution multi-tenant full-stack systems. Preprints, 2025. gap, 2025. Available at SSRN 6576378. [22] M. Masmoudi, S. Ben Abdallah Ben Lamine, Fastapi. https://fastapi. M. H. Karray, B. Archimede, and H. Baaza- [32] S. Ramirez. tiangolo.com, 2018. Python web framework for oui Zghal. Semantic data integration and querybuilding APIs. ing: A survey and challenges. ACM Comput. Surv., 56(8), Apr. 2024. [33] M. Raza, Z. Jahangir, M. B. Riaz, M. J. Saeed, and M. A. Sattar. Industrial applica[23] S. Min and B. Kim. Adopting artificial intelligence tions of large language models. Scientific Reports, technology for network operations in digital trans15(1):13755, 2025. formation. Administrative Sciences, 14(4), 2024. [24] K. C. Nagabhyru. From data silos to knowl- [34] K. K. C. Reddy, T. C. Lekha, T. P, M. Arjun, K. Radhika, and P. Naresh. Enterprise secure edge graphs: Architecting crossenterprise ai soluai execution engine: A zero-trust security gatetions for scalability and trust. Available at SSRN way for enterprise LLM systems. In 2026 9th In5697663, 2023. ternational Conference on Inventive Computation [25] P. R. Nangi, C. K. R. N. Obannagari, and S. SetTechnologies (ICICT), pages 2015–2021, 2026. tipi. A multi-layered zero-trust security framework for cloud-native and distributed enterprise [35] N. R. S. Reddy and M. Adapa. Ai-driven data integration: Transforming enterprise data pipelines systems using ai-driven identity and access intellithrough machine learning. Journal of Computer gence. International Journal of Emerging Trends Science and Technology Studies, 7(12):110–119, in Computer Science and Information Technol2025. ogy, 4(3):144–153, 2023. [26] A.-R. O. Ottun and H. Flores. Trustwor- [36] A. S. Shaik. The AI capability trap: Why enterprise ai investments fail to scale beyond pilots, thy ai in practice: A comprehensive review 2026. Available at SSRN 6514144. of human oversight and human-in-the-loop approaches. TechRxiv, Oct. 2025. [37] A. Sharma. Securing enterprise LLM gateways & misconfigured proxy exposure detection+ mit[27] J. Pan, L. Cai, S. Yan, and X. S. Shen. Network igation framework for exposed LLM endpoints, for ai and ai for network: Challenges and opporapi gateways, model download hooks. Internatunities for learning-oriented networks. IEEE Nettional Journal of Advance Industrial Engineering, work, 35(6):270–277, 2021. 12(04):1–15, 2024. [28] J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, Autogpt. https: P. Liang, and M. S. Bernstein. Generative agents: [38] Significant Gravitas. //github.com/Significant-Gravitas/ Interactive simulacra of human behavior. In ProAutoGPT, 2023. ceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023.
29
[39] M. Sun et al. A survey on large language modelbased agents for statistics and data science. arXiv preprint, 2025. [40] S. Tariq, M. B. Chhetri, S. Nepal, and C. Paris. Alert fatigue in security operations centres: Research challenges and opportunities. ACM Comput. Surv., 57(9), Apr. 2025. [41] J. Vallone. Reassessing AI pilot failure rates: A scoping review and managerial implications, 2025. Available at SSRN 5459054. [42] S. P. Veluru. Leveraging ai and ml for automated incident resolution in cloud infrastructure. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 2(2):51–61, 2021. [43] L. Wang, C. Ma, X. Feng, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023. [44] X. Wang, Z. Wang, X. Gao, F. Zhang, Y. Wu, Z. Xu, T. Shi, Z. Wang, S. Li, Q. Qian, R. Yin, C. Lv, X. Zheng, and X. Huang. Searching for best practices in retrieval-augmented generation. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17716–17736, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. [45] J. Westenberger, K. Schuler, and D. Schlegel. Failure of AI projects: understanding the critical factors. Procedia computer science, 196:69–76, 2022. [46] Z. Xi, W. Chen, X. Guo, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2025. [47] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [48] A. Yehudai et al. A survey on evaluation of large language model-based agents. arXiv preprint, 2025. [49] S. Yu, M. Cheng, Q. Liu, D. Wang, J. Yang, J. Ouyang, Y. Luo, C. Lei, and E. Chen. 30
Multi-source knowledge pruning for retrievalaugmented generation: A benchmark and empirical study. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, page 3931–3941, New York, NY, USA, 2025. Association for Computing Machinery.