arXiv:2605.19627v1 [cs.NI] 19 May 2026
How Helpful is LLM Assistance in Network Operations? A Case Study at a Large Demonstration Network Ryo Nakamura
Koshi Eguchi∗
Information Technology Center The University of Tokyo Tokyo, Japan [email protected]
Graduate School of Information Science and Technology The University of Tokyo Tokyo, Japan [email protected]
Abstract—This paper reports on a real-world case study in which over 100 network engineers assessed how a Large Language Model (LLM) can assist in building and operating a network. The versatility of LLMs has accelerated their adoption across a wide range of domains, and assisting network operations is one such promising application. LLMs are probabilistic models, unlike deterministic protocols and configurations; therefore, clarifying their capabilities—how and to what extent LLMs can help in network operations—is a crucial step toward adopting LLMs. To offer practical insights into this issue, we conducted an extensive experiment on a large demonstration network built for a public exhibition, consisting of 21 racks with heterogeneous network devices. In the experiment, a total of 105 network engineers used an LLM-based chatbot while building and operating the network. The chatbot was equipped with three external functions: retrieval-augmented generation for domainspecific knowledge, CLI control of network devices running on the network, and access to a ticket system. The participants gave evaluations for the chatbot’s responses on a best-effort basis. Analysis of the chat histories shows that 68.1% of the evaluations were positive, indicating a quantitative baseline of the LLM’s helpfulness in network operations. Our results also demonstrate that understanding the capabilities of the chatbot is important for eliciting better responses. Moreover, we provide detailed use case analyses while sharing actual user–chatbot interactions. Index Terms—Network Operations, Large Language Models, Model Context Protocol, Retrieval-Augmented Generation
I. I NTRODUCTION Operating network infrastructure has become increasingly complex and challenging over the years. There are various types of networks, e.g., Internet service provider, data center, enterprise, and campus networks. They have their own requirements, and so suitable network designs also vary in each environment. To maintain the network infrastructure, operators need to possess a wide range of specialized skills and expertise—understanding of router and switch architectures, configurations, protocols, network design, and troubleshooting—and make continuous efforts to keep the networks running smoothly. ∗ This work was done while the author was with the University of Tokyo; the author is now with Sakana AI.
To alleviate the burden on network operators, applying Artificial Intelligence is a promising approach. Recent significant advancements have enabled Large Language Models (LLMs) to perform various tasks [1]; software engineering is a popular one [2]–[4]. Network communities also seek ways to leverage LLMs for networking purposes. Using LLMs to process complex configurations of network devices [5]–[7] may reduce the effort required for device setup and help in understanding the structures of running networks. Intentbased networking [8] has been exploring the use of LLMs to decompose high-level goals and constraints, often written in natural language, into low-level configurations to simplify operation and management processes [9]–[12]. A fundamental challenge underlying these attempts is to clarify how and to what extent LLMs can assist in network operations. Consider the following extreme example: an operator instructs an LLM, “The communication speed is slow; please fix it,” and the LLM responds, “Okay, I will reboot the routers on the path.” Such behavior is, of course, unacceptable. Understanding the capabilities and limitations of LLMs is crucial for applying them effectively in operations and for exploring future applications. Moreover, such efforts should be pursued through both evaluations conducted in sandbox environments [13], [14] and studies across diverse practical use cases [15], [16]. This paper presents a real-world case study on applying an LLM to network operations. We conducted an experiment in which over 100 network engineers used an LLM-based chatbot and assessed how the chatbot could assist in building and operating a large demonstration network. The chatbot we developed for the experiment was equipped with three functions in addition to an ordinary chat interface: (1) searching and retrieving knowledge specific to the network, e.g., design documents, (2) accessing and controlling command line interfaces (CLIs) of devices running in the network, and (3) accessing a ticket system. The experimental environment was ShowNet [17], one of the largest demonstration networks in the world, which is built and operated temporarily during an annual exhibition of network technologies. In 2025, the
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
ShowNet network consisted of 21 racks with heterogeneous network devices from multiple vendors. Our experiment aimed to reveal how a current LLM can assist operators through such an extensive network construction and operation. During the experiment, 105 network engineers used the chatbot over a two-week period, resulting in chat histories comprising 815 threads. We analyzed these conversations from two perspectives: the extent to which the LLM was helpful in building and operating the network, and how the engineers used the chatbot and for what purposes. Based on these analyses, we derived the following key findings regarding the capabilities of the LLM from the viewpoint of network operations: • The current LLM (GPT-4.1), integrated with the three functions, achieved a 68.1% rate of positive evaluations, indicating a quantitative baseline of how helpful LLMs can be in network operations. • If users do not understand the scope of the chatbot’s capabilities and knowledge, it is difficult to elicit effective responses from the chatbot. • The current LLM can generally control network device CLIs across multiple vendors, executing 85.1% of the commands without syntax errors in the experiment. Meanwhile, it occasionally issues incorrect commands or fails to handle complex operations. • Use cases of the chatbot vary with users’ expertise. Skilled engineers tended to use it directly for operational tasks, whereas junior engineers benefited from it as a supplementary tool for learning and problem solving. The rest of the paper is organized as follows: Section II describes the experiment we conducted and the chatbot structure. Section III summarizes the chat history data, and Section IV presents analysis results and findings. Section V discusses related work, and Section VI concludes the paper. II. A SSISTING N ETWORK O PERATORS WITH AN LLM- BASED C HATBOT The purpose of our experiment was to investigate, through practical use by network engineers, the effectiveness of an LLM-based chatbot in network operations and its usage patterns. This section describes the experimental environment, Interop Tokyo ShowNet, and the chatbot structure. Finally, we mention the ethical considerations of collecting chat histories. A. Experimental Environment: ShowNet Interop Tokyo [18] is an annual exhibition of network technologies in Japan. In 2025, the exhibition was held from June 11 to 13 with 532 exhibitor booths, and 136,875 people visited the exhibition. ShowNet (AS290) [17] is a demonstration network built at the exhibition venue. The ShowNet network provides Internet connectivity for exhibitors and visitors at the event while demonstrating new technologies and conducting various interoperability tests [19]. The network was constructed prior to the Interop Tokyo exhibition. Its construction began on May 30 at the exhibition hall, and the network was taken down on June 13, following the exhibition.
Fig. 1: A snapshot of the ShowNet booth at Interop Tokyo 2025, before the exhibition opened on June 11. The network where we conducted the experiment was built on this booth. ShowNet is a large-scale demonstration network that covers a broad range of network technologies. In 2025, the network comprised approximately 700 devices in 21 full-height racks and nine points of presence (PoPs) to cover the exhibition halls. Figure 1 is a snapshot of the ShowNet booth where the network was built. The backbone network of ShowNet was also designed as a demonstration and for interoperability testing; it was composed of Segment Routing over IPv6, a recent routing mechanism primarily designed for carrier networks, while using Ethernet VPN and VXLAN for its access networks. The network was a multi-vendor environment composed of devices and software provided by Interop exhibitors. For instance, routers and switches spanned eight different vendors, including Cisco, Juniper, and Huawei. To build this network, 828 engineers were involved in the project. We provided the chatbot to assist network engineers who built and operated the network. Chatbot usage was expected to differ depending on the users’ level of expertise. Here, we describe the three categories of engineers who were involved in the ShowNet project and participated in the experiment: • Network Operation Center (NOC) members: the NOC members lead the ShowNet project; they design the network and demonstrations and are responsible for the operations. The group consists of 30 experts from academia, xSPs, vendors, and other organizations. NOC members have a deep understanding of the technologies and designs of the network. • Junior engineers: ShowNet hosts an educational program that accepts junior engineers, e.g., university students and junior staff members from industry, as volunteer members who help build and operate ShowNet. These junior engineers are relatively less skilled and not well-versed in the technologies used in the network. • Vendor specialists: ShowNet is composed of products temporarily provided by vendors exhibiting at the event. Vendor specialists from those companies support its construction with their expertise in products. In total, 105 engineers participated in the experiment: 27 NOC members, 41 junior engineers, and 37 vendor specialists.
B. An LLM-based Chatbot for Network Operations We developed and provided an LLM-based chatbot to help the engineers build and operate ShowNet. In addition to an ordinary chat interface like ChatGPT, the chatbot had the following external functions: • Retrieval-Augmented Generation (RAG): RAG [20] enables the LLM to search ShowNet-specific documents and leverage their contents to generate more accurate responses. • Controlling CLIs of the network devices: The LLM can access CLIs of routers and switches running on the network and execute arbitrary operational commands, such as show interfaces and show ip route. • Accessing Tickets: The LLM can access a ticket system used for ShowNet, and list and view the tickets. These naive functions enable the chatbot to behave like a virtual assistant with ShowNet-specific knowledge, control of network devices, and reading of tickets. Figure 2 illustrates the structure of the chatbot system. The chatbot employs a GPT-4.1 model deployed on the Azure OpenAI Service. The chatbot front end, running on a Linux machine, provides users with an ordinary chat interface; users send prompts, and then responses from the LLM appear. The LLM can invoke the three external functions. This capability of LLMs to use external functions is called function calling or tool calling [21]. In addition to the RAG function, two Model Context Protocol servers (described later) provide the other two functions. The LLM autonomously invokes the appropriate functions based on ongoing chat contexts. The source code of the chatbot is available at [22]. RAG is an approach where a model retrieves information from an external knowledge source (typically a vector database) and uses it to generate more accurate responses. In this experiment, we used the RAG functionality provided by the Azure OpenAI Service [23]. We stored three types of documents in a vector database: the ShowNet design documents for this year, converted from PPTX files into Markdown format (8,427 lines in total); the ShowNet operation guide for this year, consisting of 1,350 lines of Markdown; and the configuration files of 166 network devices from last year’s ShowNet. The LLM searches the vector database when needed and uses the retrieved contents to generate responses. Controlling CLIs from the LLM is implemented as a Model Context Protocol (MCP) [25] server. MCP is an open protocol that defines a procedure for LLMs to invoke external functions. The CLI MCP server exposes two tools to the LLM via MCP: one to list accessible network devices along with their OS types, and the other to send commands to specified devices. When the LLM decides to operate a network device, it calls the latter tool via the front end, passing a target device name and a command to be executed. The CLI MCP server then establishes an SSH connection to the device, executes the command, and returns its output. Note that the CLI MCP server does not inform the LLM what commands can be executed on each device; instead, the LLM generates commands based
Azure OpenAI Service LLM (GPT-4.1) Prompt
Response
Prompt
Function-MCP Bridging
Response
MCP
Browser
CLI MCP server SSH
Operator
Vector Database
Function Calling
Chatbot Front end
Chat UI
Search (RAG)
Network Devices
MCP
Ticket System MCP server REST
Ticket System
Experimental Environment (ShowNet)
Fig. 2: The chatbot system structure. A GPT-4.1 model deployed on the Azure OpenAI Service serves as the back-end LLM. The chatbot front end based on Chainlit [24] running on a Linux machine provides the chat interface to the operators and relays function calling from the LLM to the MCP servers.
on its own knowledge. The CLI MCP server implementation is open source [26]. Although the implementation is capable of changing running configurations of devices, this capability is disabled during the experiment to avoid unintended or faulty changes that could disrupt ShowNet services. As the construction of the network progressed, the number of network devices accessible through the MCP server increased. Finally, the LLM operated the CLIs of 69 devices at least once, across 29 products from Cisco, Juniper, Huawei, and A10 Networks. Ticket access from the LLM is also implemented as an MCP server. The Ticket System MCP server exposes two tools to the LLM: one to list tickets, and the other to view the full content of a specified ticket. In ShowNet, operators manage tasks through tickets. One ticket may cover the initial setup of a router, while another ticket may handle a troubleshooting task. In 2025, 384 tickets were opened over the two weeks. We developed this functionality so that the chatbot could be leveraged for operational support from a project management perspective. The ticket system used in ShowNet has been developed in-house; therefore, the Ticket System MCP server is not released as open source. System prompts instruct LLMs how to behave, such as by stating “you are a senior engineer.” Previous literature shows that optimizing system prompts would improve the performance of LLMs [13], although our goal is not to find better system prompts. The system prompt we used was therefore simple. It first stated, “You are a professional network engineer and an assistant who provides technical support to the members building ShowNet,” and then explained how to use the external functions. In addition, we embedded simplified topology information—connectivity between backbone routers and their groupings, e.g., core routers—into the system
# of threads
prompt. This embedding allowed the LLM to acquire a certain level of understanding of the connections between the devices. The embedded topology information was represented using the Mermaid graph notation [27]. The system prompt we used is available in Appendix. On May 30, 2025, the people involved in ShowNet assembled at the venue and began building the network. On May 31, the chatbot became available, and participants in the experiment began using it. We explained the chatbot structure and how to use it to the NOC members and junior engineers, while the vendor specialists were given presentation materials instead, due to on-site scheduling constraints. The participants used the chatbot freely and gave either positive ( ) or negative ( ) evaluations to the responses on a best-effort basis. Positive evaluations indicate the degree of assistance: the response was helpful in solving a problem or providing support for network construction and operation. Negative evaluations indicate that the response was faulty, such as including misinformation. Dialogues between the participants and the chatbot, called threads, were kept on the front-end database. On June 13, the chatbot stopped along with the shutdown of ShowNet.
50 NOC Members Junior Engineers Vendor Specialists
25 0
1
50 Participant
100
Fig. 3: The number of threads per participant. Thread
… …
About Protocol X
… [Call CLI]
Topic Segments
…
…
About Router A
…
👍
… …
About a Ticket
C. Ethical Considerations This experiment collected chat histories as experimental data. Accordingly, we prepared an informed consent form explaining the purpose of the study and the data management procedures. The form stated that chat histories and evaluations would be collected and retained in accordance with institutional data management policies. It also clarified that the data would be used only for research purposes and that any prompts or responses presented in publications would be anonymized. The participants began using the chatbot after agreeing to these conditions. The study procedure was reviewed and approved by the ethics committee of the authors’ organization. III. DATA We collected chat histories, comprising 815 threads created by 105 participants between May 31 and June 13, 2025. Figure 3 shows the number of threads per participant. Overall, the junior engineers used the chatbot most frequently, followed by the NOC members in terms of thread count, and then the vendor specialists. The junior engineers actively asked the chatbot to obtain technical explanations and configuration guidance. This point is described in Section IV-D. Before analyzing the chat histories, we processed them to decompose each thread into groups of exchanges of prompts and responses concerning a single topic, called topic segments. The participants asked the chatbot at will. Since a single thread can span multiple unrelated topics, thread-level analysis is not suitable. We therefore split the threads into topic segments. Figure 4 illustrates an example of a thread. A participant asks the chatbot about a protocol and then lets the chatbot check a router’s status. The participant gives a positive evaluation at the end of the conversation about the router, and the thread proceeds to the next topic. This thread can be decomposed into three topic segments.
Fig. 4: Thread and topic segment. A thread consists of multiple topic segments, each of which also consists of one or more exchanges of prompts and responses.
Splitting a dialogue into topic segments is a long-standing issue in natural language processing [28]–[30], and LLMs have potential in this area too [31]. We used GPT-4.1 to split the threads into topic segments. We passed each thread to the LLM via the OpenAI API, along with an instruction on how to split. The instruction outlines the experimental context and rules for splitting threads—splitting whenever a new topic appears and keeping exchanges in a segment when the user refers back to previous messages. The following is an actual example: within a single thread, a participant first asked the chatbot about configuring BGP Flowspec on an IOS-XR router, and then, in a subsequent message, asked why a BGP neighbor is down on an A10 Networks device. These interactions were identified as different topics and were split into separate segments because the subsequent message addressed a different device and did not refer to the previous message. Table I lists the number of threads, their topic segments, exchanges, and evaluations. We obtained a total of 1,267 segments from 815 threads. Of the 4,007 prompt–response exchanges, 389 responses received evaluations; the participants evaluated 9.7% of all responses. Figure 5 shows the CDF of the number of exchanges within a segment. 95% of segments consist of fewer than 10 exchanges concerning a single topic. The longest segment (42 exchanges) is a dialogue where a junior engineer performs the initial configuration of a Cisco Catalyst switch, consulting the LLM about each command, for example, adding users, setting up NTP and the time zone, enabling remote login, and configuring ACLs.
TABLE I: The number of threads, topic segments, exchanges, and evaluations given for the responses.
TABLE II: The overall result of evaluations given for the chatbot’s responses.
Participant Type
# of Threads
# of Segments
# of Exchanges
# of Evaluations
Participant Type
All NOC Members Junior Engineers Vendor Specialists
815 199 506 110
1267 259 829 179
4007 673 2798 536
389 125 232 32
All NOC Members Junior Engineers Vendor Specialists
CDF
1.0
0.5
0.0
0
10 20 30 # of exchanges in a segment
40
Fig. 5: CDF of the number of exchanges of prompts and responses per segment. IV. R ESULTS This section shows analysis results—cases where the chatbot exhibited its helpfulness (§IV-A and §IV-B), the faulty behaviors observed (§IV-C), and analysis based on the users’ intent in using the chatbot (§IV-D). The interactions between the participants and the chatbot were mostly in Japanese. The chat examples presented in the rest of the paper are English translations of the actual conversations. A. Overall Usefulness of the Chatbot Table II shows the overall number of evaluations. The participants provided evaluations for a total of 389 responses, of which 265 were positive and 124 were negative. The proportion of positive evaluations, which indicate the responses were helpful for building and operating the network, is 68.1%. This value can be regarded as a quantitative baseline of the effectiveness of LLM support in network operations at present. This value is consistent with results from other LLM benchmarks in related domains. SWE-bench [32] and BFCLbench [33], which evaluate agentic coding and function calling capabilities, respectively, report performance levels of around 70%. The close alignment suggests that our results are in line with broader performance trends of current LLMs. Takeaway: The LLM with the naive three functions provided helpful support in 68.1% of the evaluated cases, indicating its practical usefulness in assisting network operations at present. The vendor specialists gave a lower score of 43.8%, compared with the NOC members (71.2%) and the junior engineers (69.8%). Although the number of evaluations given by the vendor specialists was not large, their relatively low score can be attributed to differences in how deeply they understood the chatbot’s capabilities. As mentioned in Section II-B, we
# of Evaluations Positive Negative 265 89 162 14
124 36 70 18
Positive Rate 68.1% 71.2% 69.8% 43.8%
had no opportunity to explain how to use the chatbot in detail to the vendor specialists, other than providing the presentation material. As a result, the vendor specialists used the chatbot without a clear understanding of the scope of its capabilities and knowledge. They tended to ask the chatbot to perform tasks that it could not do. We share two examples from conversations that received negative evaluations. One asked the chatbot about the product name of a device with a given hostname; however, the chatbot did not know the specific mapping between hostnames and products in the first place. Another asked the chatbot to check the reachability of an IP address. The LLM itself had no ability to check such reachability; instead, it issued a ping command from a router via the CLI MCP server, which failed due to a routing issue. Takeaway: Queries beyond an LLM’s capabilities or knowledge can lead to unintended or faulty behavior. Understanding the scope of the chatbot’s capabilities and knowledge is important to elicit effective responses. B. Function Usages We next analyze the topic segments from the viewpoint of function usage. Table III lists the number of evaluations given for segments involving the use of the functions. The results show that, for segments using RAG and/or CLI, positive evaluation rates range from 61.4% (RAG only) to 79.1% (RAG + CLI). These results also represent the current level of the LLM’s effectiveness when naively connected to domain knowledge and CLIs of network devices. On the other hand, segments involving the Ticket function received a lower positive rate (<61%). A reason for this low rate lies in the design of the Ticket MCP server. The Ticket MCP server can only read tickets, which limits its usage to simple tasks such as summarizing currently open tickets. As a result, the Ticket function was used less frequently, and its evaluations were lower compared to other functions. Here, we share two examples of actual uses of the functions. Figure 6 is a conversation involving the use of the RAG function. The participant pastes the current configuration of router bgp on a Cisco Nexus switch and asks a question: only a single route is active in the routing table, although there are multiple paths. Then the chatbot correctly answers that the current configuration lacks maximum-paths and multipath-relax while referring to the vector database. Beyond this BGP example, many participants raised questions
TABLE III: The number and proportion of evaluations in segments that involve function uses. Functions
# of Segments
RAG CLI Ticket RAG + CLI RAG + Ticket CLI + Ticket RAG + CLI + Ticket
562 267 50 101 18 8 4
# of Evaluations Positive Negative 78 80 14 53 9 4 3
49 29 9 14 9 5 2
Positive Rate 61.4% 73.4% 60.9% 79.1% 50.0% 44.4% 60.0%
Prompt: I’d like to enable eBGP multipath on a Cisco Nexus switch, but with the configuration below only one route is installed. Are there additional configurations needed to enable multipath? router bgp 65353 router-id 45.0.0.53 log-neighbor-changes address-family ipv4 unicast redistribute direct neighbor 2001:3e8:0:211::5
[…]
Response: To enable eBGP multipath on the Nexus series, you can consider the following configuration. 1. Under router bgp, it is necessary to configure the maximumpaths setting to specify the number of available parallel paths. For example, it can be placed as follows. router bgp 65353 address-family ipv4 unicast maximum-paths 64 address-family ipv6 unicast maximum-paths 64
2. If multipath is required across routes with different AS numbers, this command must also be added under the router bgp configuration. router bgp 65353 bestpath as-path multipath-relax
[…]
Fig. 6: The chatbot successfully suggests the fix for a multipath configuration on a Cisco Nexus switch.
Prompt: Please check the BGP status of the core routers, and report if any problems are found. Response: The BGP status of the core routers is summarized below. ## BGP status on MX204 (mx204.noc) BGP status: Peers (total): 11 Peers down: 5 Main peers down: IP: 192.0.2.0 (AS: 65099) – State: Idle IPv6: 2001:200:0:8950::9c4:0, … (total 5 peers) – State: Connect, Active Reason: The IPv6 peers remain in the Connect or Active state, indicating unstable link establishment. […]
Fig. 7: A participant had the chatbot check the BGP status. These outputs correctly reflected the status at that time.
The ShowNet network consisted of network devices from multiple vendors, and thus proper commands varied depending on the devices; for example, the command for displaying an IPv6 routing table is show route table inet6.0 in Juniper devices and show route ipv6 unicast in Cisco IOS XR devices. In the experiment, the LLM was able to handle most of these command differences and issue appropriate commands. We counted the number of commands executed through the CLI MCP server and the number of failed commands due to syntax errors. As a result, 1,427 commands were executed, of which 213 failed due to syntax errors, corresponding to 14.9%. The LLM operated devices from Cisco, Juniper, Huawei, and A10 Networks, whose CLI documentation is available and likely included in its training data. The results would differ if the LLM operated devices with unseen CLIs. Takeaway: The LLM is generally able to issue correct commands across devices with different CLI structures. In our experiment, it succeeded in 85.1% of the cases. C. Cases where the Chatbot Failed
about a wide range of configurations, such as initial device settings (e.g., hostname, login users), monitoring (e.g., logging, SNMP), and routing configurations. The CLI function was primarily used for verifying the operational state of network devices. Figure 7 shows a conversation in which a participant asks the chatbot to summarize the current BGP peer status of the core routers. The chatbot understood which routers were classified as core routers thanks to the system prompt. It successfully issued commands to those routers via the CLI MCP server and checked the status of their BGP peers. Other participants also utilized the CLI function for various purposes, including monitoring interfaces, verifying IGP status, checking whether specific prefixes exist in routing tables, and reviewing logs for events such as stormcontrol detection on access switches.
The chatbot performed well in around 70% of the evaluations; at the same time, as expected, there were cases where it did not. In contrast to the successful case shown in Figure 6, the LLM sometimes generated incorrect configurations. For example, a participant asked the chatbot to show basic configuration commands for a Cisco 8000 series router. The LLM falsely assumed that its OS is IOS XE and generated incorrect configurations because Cisco 8000 series is IOS XR, which has a different CLI structure from IOS XE. Additionally, the chatbot often generated erroneous configurations for devices whose configuration documentation is not publicly available. Figure 8 shows another case where the LLM fails to construct proper CLI commands. The prompt is: “Please find any links on the core routers where IS-IS is configured but the MTU is not set to 9000.” This task consists of two steps: (1) identifying the interfaces running IS-IS, and (2) finding
Prompt: Please find any links on the core routers where IS-IS is configured but the MTU is not set to 9000. Response: I attempted to run commands to check the IS-IS link MTU settings on the core routers, but it seems that the commands did not work correctly on some devices: 1. Juniper (mx204.noc, ptx10002.noc): The command `show isis interface | match inet | match MTU` was executed, but no clear output was obtained. 2. Cisco IOS-XR (ncs57c3.noc, cisco8712.noc, cisco8201-32fh.noc): The command used, `show clns interface brief | include MTU`, was recognized as invalid. As a result, it is necessary to investigate the cause of these errors and consider an alternative approach or a more appropriate command. […]
Fig. 8: The LLM failed to construct proper commands to fulfill the instruction by a participant.
those interfaces whose MTU size is not 9000 bytes. However, the LLM could not decompose the instruction into two steps, resulting in inappropriate commands that combine both steps into one with match or include statements. Such failures are to be expected when managing CLIs with LLMs, as LLMs are probabilistic models. Reasoning approaches [34], [35] would improve performance in handling CLIs. Takeaway: The LLM may issue incorrect commands in CLI operations. The capabilities of LLMs to handle CLIs should be further evaluated in the future. Hallucination—where an LLM generates plausible but incorrect information—also occurred in the experiment. In addition to plausibly generating incorrect configurations or commands, the LLM often generated misinformation when the participants asked questions that the LLM could not answer. Queries about things an LLM could not possibly know may trigger hallucinations. In an example case, a participant asked the chatbot about the names of specific NOC members, and the chatbot responded with entirely incorrect names. There was no information about the names of NOC members in the knowledge base that the LLM could access. A more significant case occurred when a participant asked the chatbot about tickets assigned to an organization, and all the tickets listed in the response were nonexistent. We have not been able to identify the cause of this behavior. As in other research areas, addressing such hallucinations from the perspective of network operations is an important challenge. D. Use case Analysis This section analyzes the segments based on participants’ intent in using the chatbot. Through an examination of the chat histories and their segments, we defined the following six categories of participants’ intent in dialogue. 1) Status Monitoring involves real-time verification of device and network status. Example: checking the chassis
Status Monitoring Troubleshooting
NOC Members
Configuration Analysis Knowledge Support
Project Management Other
14.3
18.9
23.6
Junior Engineers
8.3
Vendor Specialists
7.8
23.5
0
26.6
35.0
7.3
25
10.4
6.2
44.0
34.6
50 Percentage (%)
10.6
75
12.3
11.7
100
Fig. 9: Proportion of segments classified into each category.
status of core routers, verifying link conditions, and confirming BGP neighbor establishments. 2) Troubleshooting focuses on fault diagnosis and resolution. Example: investigating the reason for a link-down event, analyzing VXLAN tunnel establishment failures, and locating MTU mismatches. 3) Configuration Analysis covers the retrieval and review of configurations for investigation and validation. Example: inspecting routing configurations on specific devices and checking firewall filter rules. 4) Knowledge Support refers to educational use of the chatbot, including explanations of specific technologies and command references. Example: describing the mechanism of SRv6, and explaining differences between family inet, inet6, and iso in Juniper devices. 5) Project Management involves coordination and progress tracking for supporting project workflow and team management. Example: summarizing open tickets, reporting overall progress, and suggesting which tasks should be prioritized. 6) Other includes conversations not classified in the above categories. Example: “This is a test.” We used GPT-4.1 via the OpenAI API to classify all the segments into the six categories in a similar manner to the topic segmentation described in Section III. Figure 9 shows the proportion of segments in each category. As shown, chatbot usage differs across the participant types. The NOC members, who have a deep understanding of the network, were more likely to use the chatbot for direct operational tasks; status monitoring and troubleshooting together accounted for 37.9% of their segments. In contrast, the junior engineers mainly used the chatbot for knowledge support (44%) and configuration analysis (35%). Figure 10 is an actual prompt from a segment classified as Knowledge Support. Similar uses of the chatbot for explanatory purposes were observed frequently throughout the experiment. Operators’ understanding of the target network changes how they employ the LLM: experienced operators treat it as a practical assistant for real-time operations, whereas junior engineers use it as a tutor. Table IV shows the number of evaluations given for the responses in the segments for each category. The highest proportion of positive evaluations is Status Monitoring (77%). The ability to check and verify the status across multiple network devices through natural language instructions appears
Prompt: You are an experienced network engineer explaining SRv6 to a junior engineer. The junior engineer has a basic knowledge in networking but little of SRv6. Below is part of the configuration for the mx304.noc. Using this configuration as an example, carefully explain how packet delivery works in SRv6. source-packet-routing { srv6 { block shownet-usid-block fcbb:bb00::/32
[…]
Fig. 10: A case of Knowledge Support. A junior engineer asks the chatbot to explain SRv6 with an actual configuration. TABLE IV: The number of evaluations given to segments in each category. # of Evaluations Positive Negative
Category Status Monitoring Trouble Shooting Configuration Analysis Knowledge Support Project Management Other
47 27 111 59 16 5
14 19 50 25 16 0
Positive Rate 77.0% 58.7% 68.9% 70.2% 50.0% 100.0%
to be useful. In contrast, Troubleshooting has a relatively lower positive rate of 58.7%. Troubleshooting requires advanced reasoning and multiple steps, e.g., forming hypotheses about potential causes, checking the status, and isolating problems. Complex tasks remain difficult for the LLM when used without additional reasoning, as shown in Figure 8. Meanwhile, Knowledge Support (70.2%) and Configuration Analysis (68.9%), which were actively utilized by the junior engineers, indicate that the chatbot was also effective as a supplementary tool for enhancing their technical skills. Takeaway: The way operators use the chatbot varies with their expertise; experienced engineers leverage it for direct network operations, while junior engineers benefit from it as a supplementary tool for learning and problem solving. V. R ELATED W ORK An LLM-integrated chatbot is a straightforward approach for assisting network operators. ByteDance operates such a system [15], which answers network diagnosis questions from users by invoking external functions and accessing monitoring data. Nickel et al. [11] propose exposing NetBox [36], a DCIM software, to an LLM to provide network information. Abane et al. [37] attempt vendor-agnostic support by combining an LLM with a knowledge graph generated from device configurations and states. Employing LLMs for root cause analysis is also a popular approach [16], [38], [39], since troubleshooting is a time-consuming task where LLM will reduce human workload. Our chatbot is naive compared with these systems. Enabling LLMs to use more sophisticated external tools would enhance their performance and capabilities.
Configuration analysis and generation are attractive tasks for leveraging LLMs. Network configurations are often complex to interpret and debug; thus, LLMs are expected to alleviate this operational burden. To generate accurate configurations, Mondal et al. [5] propose a loop where an LLM generates configurations and then verifiers validate them, complemented by slower manual inspection by humans. NetConfEval [6] is a benchmark suite to examine the capability of LLMs in translating high-level natural language requirements into low-level representations, including configurations. It shows that GPT4-Turbo can achieve near-100% accuracy in simple translation tasks, but its performance drops with complex requirements. ConfAgent [7] further leverages multiple LLMs integrated into domain-specific tools for generating configurations. This paper presents a case study at a temporary network, and the results themselves are inherently difficult to reproduce. Meanwhile, recent studies have developed reproducible benchmarks to clarify how well LLMs can understand networks. Donadel et al. [13] propose a framework to assess whether LLMs can correctly answer questions when provided with topology information. They show that the best-performing model, Bing, achieves an average accuracy of 79% across three topologies. Aykurt et al. [40] present NETLLMBENCH, which evaluates LLMs in network configuration tasks on the Kathara [41] emulator. Zhou et al. [14] propose a more extensive benchmark, called NetPress, that dynamically synthesizes benchmark scenarios within emulated environments. VI. C ONCLUSION This paper presents a real-world case study in which 105 network engineers used and assessed an LLM-based chatbot, equipped with RAG and two MCP servers for CLI control and ticket access, for building and operating a large demonstration network. Analysis of the chat histories shows that 68.1% of the evaluations given for the chatbot responses were positive, indicating that the chatbot was moderately useful for helping network operations. The results also demonstrate that effective use of the chatbot depends on users’ understanding of the chatbot’s capabilities and knowledge. Moreover, the LLM exhibited its ability to handle network device CLIs, e.g., analyzing configurations and verifying network status. 85.1% of issued commands were executed without syntax errors on various devices from multiple vendors. We consider the results presented in this paper as a quantitative baseline for understanding how and to what extent current LLMs can assist in network operations. The results also demonstrate that further investigation is needed in various directions, such as enabling LLMs to control network devices more accurately and developing strategies for operators to interact more effectively with LLM-based assistants. ACKNOWLEDGMENT We would like to thank all the people involved in Interop Tokyo ShowNet in 2025. In this research work, we used the UTokyo Azure (https://utelecon.adm.utokyo.ac.jp/en/research computing/utokyo azure/).
R EFERENCES [1] M. Raza, Z. Jahangir, M. B. Riaz, M. J. Saeed, and M. A. Sattar, “Industrial applications of large language models,” Scientific Reports, vol. 15, no. 1, p. 13755, Apr 2025. [Online]. Available: https://doi.org/10.1038/s41598-025-98483-1 [2] D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, and B. Myers, “Using an llm to help with code understanding,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639187 [3] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” IEEE Trans. Softw. Eng., vol. 50, no. 4, p. 911–936, Apr. 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3368208 [4] Q. Zhang, C. Fang, Y. Xie, Y. Zhang, Y. Yang, W. Sun, S. Yu, and Z. Chen, “A survey on large language models for software engineering,” 2024. [Online]. Available: https://arxiv.org/abs/2312.15223 [5] R. Mondal, A. Tang, R. Beckett, T. Millstein, and G. Varghese, “What do llms need to synthesize correct router configurations?” in Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, ser. HotNets ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 189–195. [Online]. Available: https://doi.org/10.1145/3626111.3628194 [6] C. Wang, M. Scazzariello, A. Farshin, S. Ferlin, D. Kostić, and M. Chiesa, “Netconfeval: Can llms facilitate network configuration?” Proc. ACM Netw., vol. 2, no. CoNEXT, Jun. 2024. [Online]. Available: https://doi.org/10.1145/3656296 [7] S. Li, Z. Gan, J. Liu, C. Gao, F. Li, S. Wu, P. Hu, and F. Li, “Confagent: Towards intelligent network configuration via llm agent,” in 2025 IEEE/ACM 33rd International Symposium on Quality of Service (IWQoS), 2025, pp. 1–10. [8] A. Clemm, L. Ciavaglia, L. Z. Granville, and J. Tantsura, “Intent-Based Networking - Concepts and Definitions,” RFC 9315, Oct. 2022. [Online]. Available: https://www.rfc-editor.org/info/rfc9315 [9] A. Mekrache, A. Ksentini, and C. Verikoukis, “Intent-based management of next-generation networks: an llm-centric approach,” IEEE Network, vol. 38, no. 5, pp. 29–36, 2024. [10] N. Van Tu, J.-H. Yoo, and J. W.-K. Hong, “Towards intent-based configuration for network function virtualization using in-context learning in large language models,” in NOMS 2024-2024 IEEE Network Operations and Management Symposium, 2024, pp. 1–8. [11] L. I. Nickel, L. Hohmann, N. Stolbov, L. Gerstacker, and S. Rieger, “Integrating llms with netbox and netmiko for vendor-agnostic intentbased networking,” in NOMS 2025-2025 IEEE Network Operations and Management Symposium, 2025, pp. 1–6. [12] K. Dzeparoska and A. Leon-Garcia, “Kpi assurance and llms for intentbased management,” in NOMS 2025-2025 IEEE Network Operations and Management Symposium, 2025, pp. 1–9. [13] D. Donadel, F. Marchiori, L. Pajola, and M. Conti, “Can LLMs Understand Computer Networks? Towards a Virtual System Administrator,” in 2024 IEEE 49th Conference on Local Computer Networks (LCN). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2024, pp. 1–10. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/LCN60385.2024.10639641 [14] Y. Zhou, J. Ruan, E. S. Wang, S. Fouladi, F. Y. Yan, K. Hsieh, and Z. Liu, “Netpress: Dynamically generated llm benchmarks for network applications,” 2025. [Online]. Available: https://arxiv.org/abs/2506.03231 [15] H. Wang, A. Abhashkumar, C. Lin, T. Zhang, X. Gu, N. Ma, C. Wu, S. Liu, W. Zhou, Y. Dong, W. Jiang, and Y. Wang, “NetAssistant: Dialogue based network diagnosis in data center networks,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). Santa Clara, CA: USENIX Association, Apr. 2024, pp. 2011–2024. [Online]. Available: https://www.usenix.org/conference/nsdi24/presentation/wang-haopei [16] C. Wang, X. Zhang, R. Lu, X. Lin, X. Zeng, X. Zhang, Z. An, G. Wu, J. Gao, C. Tian, G. Chen, G. Liu, Y. Liao, T. Lin, D. Cai, and E. Zhai, “Towards llm-based failure localization in production-scale networks,” in Proceedings of the ACM SIGCOMM 2025 Conference, ser. SIGCOMM ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 496–511. [Online]. Available: https://doi.org/10.1145/3718958.3750505
[17] T. Tomine, R. Nakamura, and R. Motobayashi, “ShowNet at Interop Tokyo: A Continuously Evolving Demonstration Network,” The Internet Protocol Journal, vol. 28, no. 1, pp. 2–12, 2025. [Online]. Available: https://ipj.dreamhosters.com/wp-content/uploads/2025/04/281-ipj.pdf [18] “Interop Tokyo 2025.” [Online]. Available: https://www.interop.jp/2025/en/ [19] R. Nakamura, H. Nakamura, K. Okada, and R. Kato, “Technology Highlights of ShowNet 2024,” The Internet Protocol Journal, vol. 28, no. 2, pp. 2–13, 2025. [Online]. Available: https://ipj.dreamhosters.com/wp-content/uploads/2025/08/282-ipj.pdf [20] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020. [21] “Function Calling – OpenAI Platform.” [Online]. Available: https://platform.openai.com/docs/guides/function-calling [22] “upa/llmexp-chatbot: Chatbot for our experiment: Assisting Network Operators with an LLM.” [Online]. Available: https://github.com/upa/llmexp-chatbot [23] “How to use Azure OpenAI Assistants file search - Azure OpenAI — Microsoft Learn.” [Online]. Available: https://learn.microsoft.com/enus/azure/ai-foundry/openai/how-to/file-search [24] “Chainlit/chainlit: Build Conversational AI in minutes,” 2025. [Online]. Available: https://github.com/Chainlit/chainlit [25] “What is the Model Context Protocol (MCP)?” [Online]. Available: https://modelcontextprotocol.io/docs/getting-started/intro [26] “upa/mcp-netmiko-server: An MCP server that enables LLMs interacting with your network devices,” 2025. [Online]. Available: https://github.com/upa/mcp-netmiko-server [27] “Diagram Syntax — Mermaid.” [Online]. Available: https://mermaid.js.org/intro/syntax-reference.html [28] M. A. Hearst, “Text tiling: Segmenting text into multi-paragraph subtopic passages,” Computational Linguistics, vol. 23, no. 1, pp. 33–64, 1997. [Online]. Available: https://aclanthology.org/J97-1003/ [29] L. Xing and G. Carenini, “Improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring,” in Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, H. Li, G.-A. Levow, Z. Yu, C. Gupta, B. Sisman, S. Cai, D. Vandyke, N. Dethlefs, Y. Wu, and J. J. Li, Eds. Singapore and Online: Association for Computational Linguistics, Jul. 2021, pp. 167–177. [Online]. Available: https://aclanthology.org/2021.sigdial-1.18/ [30] I. Ghinassi, L. Wang, C. Newell, and M. Purver, “Recent trends in linear text segmentation: A survey,” in Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 3084–3095. [Online]. Available: https://aclanthology.org/2024.findings-emnlp.174/ [31] Y. Fan, F. Jiang, P. Li, and H. Li, “Uncovering the potential of ChatGPT for discourse analysis in dialogue: An empirical study,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, Eds. Torino, Italia: ELRA and ICCL, May 2024, pp. 16 998–17 010. [Online]. Available: https://aclanthology.org/2024.lrecmain.1477/ [32] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66 [33] S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez, “The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=2GmDdhBdDk [34] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” 2023. [Online]. Available: https://arxiv.org/abs/2205.11916 [35] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903
[36] “netbox-community/netbox: The premier source of truth powering network automation.” [Online]. Available: https://github.com/netboxcommunity/netbox [37] A. Abane, A. Battou, and M. Merzouki, “An adaptable ai assistant for network management,” in NOMS 2024-2024 IEEE Network Operations and Management Symposium, 2024, pp. 1–3. [38] D. Roy, X. Zhang, R. Bhave, C. Bansal, P. Las-Casas, R. Fonseca, and S. Rajmohan, “Exploring llm-based agents for root cause analysis,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, ser. FSE 2024. New York, NY, USA: Association for Computing Machinery, 2024, p. 208–219. [Online]. Available: https://doi.org/10.1145/3663529.3663841 [39] A. Shan, J. Kaur, R. Singh, T. Banka, R. Yavatkar, and T. Sridhar, “Rca copilot: Transforming network data into actionable insights via large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2507.03224 [40] K. Aykurt, A. Blenk, and W. Kellerer, “Netllmbench: A benchmark framework for large language models in network configuration tasks,” in 2024 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), 2024, pp. 1–6. [41] G. Bonofiglio, V. Iovinella, G. Lospoto, and G. Di Battista, “Kathará: A container-based framework for implementing network function virtualization and software defined networks,” in NOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium, 2018, pp. 1–9.
A PPENDIX Figure 11 shows the original system prompt of the chatbot developed for the experiment. Here, file_search, netmiko server, and ttdb refer to the RAG function, the CLI MCP server, and the Ticket System MCP server, respectively. There are two prompts that refer to files whose names start with specific strings. These prompts appeared because our understanding of RAG was limited at the time of development, and they likely had little or no actual effect. However, we include the original prompts used during the experiment to present the implementation as it was. The simplified topology information, shown in Figure 12, follows the prompt in Figure 11. The actual Mermaid block is quite long; therefore, redundant lines are omitted and represented by [...]. Devices and external connections, i.e., IXPs and transit providers, are represented as nodes. Subgraphs represent groups of devices based on their roles and locations. In addition, we included link speed information as link texts. We manually described this topology information based on the network design.
* You are a professional network engineer and an assistant who provides technical support to the members building ShowNet. Your name is "ShowNet Chatbot." * ShowNet is a large-scale demonstration network built for Interop Tokyo, an exhibition of network technology. # You will supports ShowNet members by, for example: - Providing examples of network device configurations referring stored files (file_search). - Answering questions about the construction and operation of the ShowNet network. - Supporting network construction, operation, and troubleshooting by accessing network devices via `netmiko server` tools. - Accessing Trouble Ticket Database, which is the ticket system for ShowNet, via `ttdb` tools and providing support for ticket operations. The following information should be kept internally as guidelines for answering questions. It does not need to be disclosed to the user. * When answering specific questions about ShowNet, use `file_search` and base your answers on the content of the retrieved files. * Files starting with `past-shownet-config_` are configuration files for past ShowNet equipment. Refer to these files when providing configuration examples. * Files starting with `operation-guide_` are files that describe guidelines for building ShowNet. Refer to these files when answering general questions about ShowNet operation and construction. * If the `netmiko server` tool is available, operate the running network devices via the `netmiko server` as needed. When operating network devices, you must use CLI commands or configuration commands appropriate for the device_type of the devices. * The Mermaid graph information below describes the connections of routers and switches that form the ShowNet backbone network. You can refer the diagram as needed. * Each node is a router or switch, and its name corresponds to its hostname. * Connections between nodes are links. Links are labeled with their link speed. * Each subgraph indicates the area to which the routers contained within it belong.
Fig. 11: The system prompt provided to the chatbot. ```mermaid graph TD %% --- INTERNET --subgraph INTERNET %% Internet Exchange Points (IXP) and Internet Gateways KDDIInternetGateway JPIX [...] end %% --- NOC / Core --%% (Core Routers & FW) subgraph NOC direction TB mx204.noc ncs57c3.noc cisco8712.noc [...] end [...] %% --- Media-over-IP --subgraph MOIP acx7100.moip fx-2.moip end %% *** INTERNET *** mx204.noc ---|100G| KDDIInternetGateway mx204.noc ---|100G| JPIX [...] %% *** L2/L3 links
(bandwidth in link label)
***
%% NOC Core Interconnects mx204.noc ---|100G| ptx10002.noc mx204.noc ---|100G| cisco8201-32fh.noc
```
ncs57c3.noc ---|400G| cisco8201-32fh.noc ncs57c3.noc ---|400G| ptx10002.noc [...]
Fig. 12: The topology information embedded in the system prompt.