Can Large Language Models Assist the Comprehension of ROS2 Software Architectures? Laura Duits
Bouazza El Moutaouakil
Ivano Malavolta
Vrije Universiteit Amsterdam The Netherlands [email protected]
Vrije Universiteit Amsterdam The Netherlands [email protected]
Vrije Universiteit Amsterdam The Netherlands [email protected]
arXiv:2604.21699v1 [cs.SE] 23 Apr 2026
Abstract Context. The most used development framework for robotics software is ROS2. ROS2 architectures are highly complex, with thousands of components communicating in a decentralized fashion. Goal. We aim to evaluate how LLMs can assist in the comprehension of factual information about the architecture of ROS2 systems. Method. We conduct a controlled experiment where we administer 1,230 prompts to 9 LLMs containing architecturally-relevant questions about 3 ROS2 systems with incremental size. We provide a generic algorithm that systematically generates architecturallyrelevant questions for a ROS2 system. Then, we (i) assess the accuracy of the answers of the LLMs against a ground truth established via running and monitoring the 3 ROS2 systems and (ii) qualitatively analyse the explanations provided by the LLMs. Results. Almost all questions are answered correctly across all LLMs (mean=98.22%). gemini-2.5-pro performs best (100% accuracy across all prompts and systems), followed by o3 (99.77%), and gemini-2.5-flash (99.72%); the least performing LLM is gpt-4.1 (95%). Only 300/1,230 prompts are incorrectly answered, of which 249 are about the most complex system. The coherence scores in LLM’s explanations range from 0.394 for “service references” to 0.762 for “communication path”. The mean perplexity varies significantly across models, with chatgpt-4o achieving the lowest score (19.6) and o4-mini the highest (103.6). Conclusions. There is great potential in the usage of LLMs to aid ROS2 developers in comprehending non-trivial aspects of the software architecture of their systems. Nevertheless, developers should be aware of the intrinsic limitations and different performances of the LLMs and take those into account when using them. ACM Reference Format: Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta. 2026. Can Large Language Models Assist the Comprehension of ROS2 Software Architectures?. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
The Robot Operating System (ROS) can be considered as the defacto standard for developing robotics systems [28]. ROS supports Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, Scotland, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
modular development and integration of third-party components [28, 35]. In 2004, more than 531M ROS packages were downloaded [15]. ROS2 provides standardised communication mechanisms, including topics, services, and actions, allowing components to seamlessly exchange information. ROS nodes can be created and removed dynamically, allowing flexibility and reconfigurability. Additionally, ROS provides reusable libraries (e.g., tf2, rclcpp) and tools (e.g., rviz, rqt_graph), which accelerate development and facilitate integration on a variety of robotic platforms [1, 29]. The capabilities of Large Language Models (LLMs) are advancing rapidly and are being adopted across multiple domains [11]. LLMs are already used for various software engineering tasks [37], such as code generation [41], automated documentation [14], test case generation [42], and assisting software architects in design decision making [12, 17]. However, the potential of LLMs to support the comprehension of the software architecture of a robotic system remains largely unexplored. In this context, the ability of LLMs to process large amounts of data and internalise cross-domain knowledge makes them good candidates for comprehending the software architecture of complex systems like those based on ROS. Before these models can be effectively used to aid ROS developers in tasks such as system design, verification, or optimisation, it is necessary to establish their comprehension about ROS systems. Our goal is to investigate the ability of LLMs to aid architects in comprehending the software architecture of ROS2 robotics systems. To this end, we carry out a controlled experiment involving 9 commercial LLMs and 3 third-party ROS2 systems. By defining and integrating an automated question-generation algorithm with a structured prompt design methodology, a total of 29,406 prompts are systematically generated. We sample 1,230 questions, prompt them to each of the 9 LLMs, and compare the 11,070 responses of all LLMs to answer questions about the three different ROS2 systems. The responses are compared against a predefined ground truth and further analysed for general patterns, emerging topics, length, and perplexity score. The results of our study show that LLMs can reach good accuracy in comprehension tasks about the architecture of ROS2 systems. However, there are aspects where the knowledge or interpretation of the models differs. Specifically, the majority of questions were answered correctly across all LLMs (mean = 98.22%), with even a 100% accuracy on the simplest system. gemini-2.5-pro performs best (100% accuracy across all prompts and systems), followed by o3 (99.77%), and gemini-2.5-flash (99.72%); the least performing LLM is gpt-4.1 (95%). Over the 11,070 prompts, only 300 were incorrectly answered, of which 249 are about the largest system. About 99% (297/300) of the incorrect answers refer to one specific type of question, i.e., the one about the presence of a message exchange between two ROS nodes. The coherence score of
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
the main topics present in the explanations provided by the LLMs ranges from 0.394 for the “service references” topic to 0.762 for the “communication path” topic. There is a wide disparity in the mean perplexity of the LLMs with chatgpt-4o having the lowest score (19.6) and o4-mini having the highest (103.6). The main contributions of this study are: (i) an empirical evaluation of the ability of LLMs to assist architects in comprehension tasks about ROS2 systems; (ii) an algorithm for systematically generating questions about a ROS2 system topology; (iii) an in-depth discussion and contextualization of the obtained results for both academics and ROS developers; (iv) the replication package of the study including the source code for extracting the topologies, generating questions, establishing the prompts, and prompting the LLMs, as well as the raw data, and data analysis scripts [26].
2
Background
ROS [35] is a widely adopted framework for robotic software. Despite its popularity, ROS1 was limited in terms of security, reliability, and scalability. To address these concerns, ROS2 was redesigned from scratch [28]. Key architectural differences are the usage of the Data Distribution Service (DDS) [34] as middleware, allowing for more reliable and scalable communication, and the inclusion of DDS-Security [7]. ROS1 has reached end-of-life on 31 May 2025; consequently, this work focusses exclusively on ROS2, and in this context, ROS will be considered synonymous with ROS2. ROS systems comprise executable programs called nodes, each performing a specific task inside the system. Nodes can be organised into packages that can be distributed and reused. There are three ways for nodes to communicate with each other. Topics can be used for asynchronous messaging between publishers and subscribers. Services, on the other hand, enable synchronous communication in which a client sends a request to a service server and, upon completion, receives a response in return. Finally, it is also possible to use actions by having a client sending a goal to an action server, who sends feedback and, upon reaching the goal, the result. However, as of today, actions are less frequently used. To allow for the configuration of specific variables, ROS nodes have a parameter feature. Parameters can be configured at system launch, but also updated at runtime by a user or other nodes. The parameter changes are node specific; however, this information might be relevant for other nodes as they might want to react to these changes as well. For this purpose, there is a special topic called ’/parameter_events’ on which all parameter changes are published.
Figure 1: Computation graph of the pubsub system. A computation graph (or topology) illustrates the network of nodes and their communication interfaces within a ROS system. Figure 1 presents an example of a computation graph for a simple system consisting of two nodes (i.e., publisher and subscriber), a topic for their communication, and the system-wide ’/parameter_events’ topic [33]. For the purpose of this study, we exclude action servers from the analysis, meaning that we exclusively consider topics and service servers as interfaces for the nodes.
Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta
In addition, we consider the system-level topic ’/parameter_events’ as a valid communication path that links two nodes with each other.
3
Related Work
Various architectural tasks can be supported by LLMs [37]. Referring to Table 1, Dhar et al. [12] investigated the feasibility of LLMs to generate design decisions for Architecture Design Records (ADRs). They compared the performance of two types of LLMs: GPT for decoder-only models and T5 for encoder-decoder models. The results indicate that LLMs achieve an overall good BERTScore on the 95 ADRs prompted (e.g., GPT-4 shows a precision of 0.847 and an F1 of 0.849 in the 0-shot), indicating the potential to help architects document design decisions. Nevertheless, the authors report that the quality of the responses is insufficient, necessitating human involvement in the process. Where [12] used a single-shot, DíazPace et al. [13] consider a step-wise approach using five LLM-based copilots that allow user interaction to generate design decisions. In addition, the work included the usage of RAG strategies to improve the results. Nevertheless, the copilots struggled to formulate the decisions due to a lack of context. LLMs can also support architects in their understanding of a system. Soliman and Keim [39] evaluated the ability of GPT-3.5 to answer architectural knowledge questions about a software system embedded in the model. A zero-shot approach was used to prompt the questions. The results were compared against a ground truth to determine the model’s accuracy and were manually assessed on the quality and trustworthiness. The responses were considered moderately accurate while often including irrelevant information. However, they did analyse the potential to help practitioners understand the architecture of existing software systems. This work was extended by [38] to compare the performance of seven prominent LLM when answering the same set of AK questions. The findings align with previous work on the accuracies of the models. Moreover, while there are often similarities between the responses of some of the LLMs, these similarities do not extend to all seven models. Amalfitano et al. [3] assessed whether LLMs are able to automate software architecture recovery (SAR) based on source code. Their approach considered 3 types of tasks related to class diagrams, architectural and design patterns, and architectural styles. They considered 4 widely adopted LLMs: GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Mistral Large. The study included a self-reflection mechanism to encourage the LLMs to evaluate the correctness of their answers. Although the models were generally effective in identifying high-level architectural styles and basic structural elements, they struggled with implementation-specific analyses and less conventional architectural styles. Self-reflection improved performance on tasks involving structural and conceptual abstractions, but had little effect on fine-grained pattern recognition. In conclusion, to the best of our knowledge, our study is the first to evaluate the architectural knowledge of LLMs concerning ROS2 systems. To this end, we extend the findings of [39] and [38] by considering different and multiple software systems. Moreover, we consider both the answers and the explanations provided by the LLMs, providing insight into the reasoning capabilities of LLMs.
Can Large Language Models Assist the Comprehension of ROS2 Software Architectures?
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 1: Overview of the most recent related work on LLMs applied to software architecture tasks Study Amalfitano et al. Automated Software Architecture Design Recovery from Source Code Using LLMs [3]
Year 2026
Soliman et al. LLMs for Software Architecture Knowledge: A Comparative Analysis Among Seven LLMs [38]
2026
Soliman & Keim. Do Large Language Models Contain Software Architectural Knowledge? [39] Dhar et al. Can LLMs Generate Architectural Design Decisions? [12]
2025
Díaz-Pace et al. Helping Novice Architects to Make Quality Design Decisions Using an LLM-Based Assistant [13]
2024
2024
Target task Automated recovery of architectural knowledge (AK) from source code Answer questions about the AK of Apache Hadoop HDFS
LLMs GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Mistral Large
Approach 0-shot combined with role and emotion prompting followed by self-reflection of the responses compared to a pre-defined ground truth Qwen2-Instruct, 0-shot responses compared DeepSeek-R1-Distill-Llama, GPT-4o, to a pre-defined ground GPT-4o-mini, Llama-3.3-Instruct, truth TinyLlama-Chat-v1.0, Mistral-Large-Instruct-2407 Answer ques- GPT-3.5 0-shot responses compared tions about the to a pre-defined ground AK of Apache truth Hadoop HDFS Generate Archi- GPT2, GPT2-medium, GPT2-large, 0-/few-shot and fine-tuning tecture Design GPT2-xl, ada, davinci, comparisons for ADR DeciRecords (ADRs) text-davinci-003, GPT-3.5-turbo, sion generation based on a GPT-4, T5-small, T5-base, T5-large, given decision context T5-3b, T0-3b, Flan-T5-small, Flan-T5-base, Flan-T5-large, Flan-T5-xl LLM-based agent GPT-3.5 0-shot and Retrievalfor architectural Augmented Generation decision-making (RAG) for prompting archiactivities tectural questions
Key findings & Limitations LLMs partially recover architectures and styles but struggle with deeper abstractions and semantic consistency, partly improved by self-reflection; Comparison based on correctness There is a high number of false positives but some similarity between the LLMs although not uniform across all models; Usage of a single software system causing it to be less generalisable Moderately accurate answers, often containing irrelevant information; Usage of a single software system and LLM causing it to be less generalisable LLMs can produce plausible ADR decisions but quality is insufficient for autonomous use; Focus on text generation rather than reasoning correctness
Usage of RAG improves results using a 0shot approach but a lack of context forms a bottleneck; The LLM is integrated into a pipeline, which prevents its results from being analysed in isolation.
Table 2: Overview and main characteristics of the analysed ROS2 systems. System pubsub turtlebot
Maintainer ROBOTIS
Year 2018
panda
IFRA Group
2022
ROS Packages gazebo, cartographer, navigation2 gazebo, moveit, ros2_control, ros2_controllers, gripper_controllers, xacro
4 Study Design 4.1 Research Questions According to the template by Basili et al. [5], the goal of this study is to analyse the responses of LLMs for the purpose of evaluating their output size, accuracy, perplexity, and mentioned topics from the point of view of ROS2 developers in the context of ROS2 software architectures. The research questions of this study are: RQ1 – What is the output size of LLMs when answering questions related to the ROS2 software architectures? This RQ sets the stage and helps ROS developers understanding the verbosity and (possible) resource consumption of LLMs, which directly impact usability and cost in real-world ROS2 development scenarios. RQ2 – What is the accuracy of LLM when answering questions related to the ROS2 software architectures? Measuring accuracy is essential to determine whether LLMs can reliably provide correct architectural information, ensuring their practical applicability for supporting ROS2 system design and validation. RQ3 – What is the level of perplexity of LLMs when answering questions related to the ROS2 software architectures? This RQ complements RQ2 since perplexity helps assess the confidence and coherence of LLM explanations, offering insights into the LLMS’ reasoning quality beyond accuracy alone. RQ4 – What are the topics most frequently mentioned by LLMs when answering questions related to the ROS2 software architectures? Identifying frequently mentioned topics reveals the focus and coverage of LLM reasoning in the context of ROS2
Nodes 2 8
Topics 3 15
Services 12 53
Connections 4 64
40
33
276
402
GitHub Repo ros2_documentation turtlebot3 ros2_RobotSimulation
architectures, highlighting potential biases and gaps in architectural knowledge relevant to ROS2 systems.
4.2
ROS2 Systems Selection
To analyse the ability of LLMs to support the comprehension of the software architecture of ROS2 systems, we have selected three systems with computation graphs of different sizes. A description of the selected systems is provided below and in Table 2: (i) pubsub: a baseline example of a ROS2 system consisting of a publisher and a subscriber node that are connected through a single topic [33]; (ii) turtlebot: a modular mobile robot platform widely used in ROS2 education and reseach [25, 27, 32] that can be run in simulation using Gazebo and RViz [36]; (iii) panda: a manipulator system integrating Gazebo and MoveIt 2 frameworks for simulation and motion planning, developed by the IFRA Group at Cranfield University to provide ready-to-use ROS2 robot simulation packages [23]. The systems analysed in this study consist of (i) an official ROS2 tutorial example (i.e., pubsub) and (ii) two third-party robot platforms (i.e., turtlebot and panda); we use those systems since (i) their software architecture is heterogenous in terms of topology and size and (ii) they are developed by third parties, better supporting the external validity of our study. In this study, simulations are used in place of physical robots for reproducibility. The ROS2 Humble distribution is used for the whole experiment, as it is the only ROS2 release officially supported by turtlebot at the time the experiments were conducted.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
4.3
Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta
LLMs Selection
The choice of LLMs is based on the best-performing LLMs according to the LMArena leaderboard [10]. Specifically, the top ten unique models from the Arena Overview as ranked on June 16th (see Table 3). While gpt-4.5-preview was originally in the list, this model has been deprecated and removed by OpenAI and is thus excluded from the experiment. For all models, we used the default settings, meaning no model parameters were specified during prompting. Table 3: Overview of the selected LLMs LLM name
Developer
Release date
Context
gemini-2.5-pro o3-2025 chatgpt-4o claude-opus-4 gemini-2.5-flash gpt-4.1 grok-3 claude-sonnet-4 o4-mini
Google OpenAI OpenAI Anthropic Google OpenAI xAI Anthropic OpenAI
Jun 5, 2025 Apr 16, 2025 Mar 26, 2025 May 14, 2025 May 20, 2025 Apr 14, 2025 Feb 24, 2025 May 14, 2025 Apr 16, 2025
128k tokens 128k tokens 8k tokens 100k tokens 64k tokens 8k tokens 16k tokens 100k tokens 4k tokens
4.4
Cost ($/MTok) Input Output $2.50 $10.00 $2.00 $8.00 $5.00 $20.00 $15.00 $30.00 $1.00 $4.00 $4.00 $16.00 $3.00 $12.00 $3.00 $12.00 $1.00 $4.00
JSON Encoding of ROS Computation Graphs
Computation graphs are commonly used to reason about ROS systems. Existing tools can extract this information, for example, from bag files [9] or in real time during system execution using the rqt_graph package. In our study, we convert the computation graph to JSON following the structure proposed by Maruyama et al. [24, 30]. A JSON file is generated for each ROS2 system.
4.5
Prompt Generation and Evaluation
Central to the experiment is the prompt plan, a general outline for the formulation of the prompts. Figure 2 explains the structure of the prompts used in this experiment. To limit the influence of the prompt on the response of the LLMs, the prompt is treated as a controlled variable. However, no two generated prompts are identical as there are three placeholders in the prompt for the JSON topology, answer instruction, and question, respectively.
4.6
Experimental Variables and Design
The independent variables of this study are: (i) the LLMs and (ii) the ROS2 systems. The dependent variables are the: (i) number of output tokens, (ii) accuracy, (iii) perplexity, and (iv) the topics mentioned in the contents of the LLMs’ response. Accuracy is determined by comparing each answer with a predefined ground truth, yielding a binary evaluation of ‘correct’ or ‘incorrect’. The accuracy for each combination of LLM and ROS2 system is calculated as the ratio of correct answers to the total number of questions. The number of output tokens is directly provided by the LLMs’ APIs and it is meant to measure the verbosity of LLMs’ responses. Topics and perplexity are described in Section 4.8. The experiment follows a repeated measures design, in which each LLM answers the same sampled set of questions for each ROS2 system, allowing direct comparison of the performance of the models. Each question is prompted to the LLMs in an identical way, following the prompt template in Listing 2. The prompt contains three placeholders corresponding to (i) the system topology, (ii) the answer instruction, and (iii) the question itself. The specific prompt used in this study was established and refined through several iterations. Initial tests were carried out to compare different
Listing 2: Prompt template used in this study. # System Prompt The Robot Operating System (ROS) is a set of software libraries and tools for building robot applications. ROS2 is a middleware framework built on the Data Distribution Service (DDS) protocol. You are an AI assistant specialising in ROS2 robotic systems. You can analyse and reason about robotic systems. You aid ROS2 architects by answering questions about a given ROS2 system. ROS2 robotic systems are presented as system topologies in JSON format, including references to the ROS2 entities: nodes, topics, and services. Communication between nodes is achieved through the anonymous publish/subscribe system or through the request/response mechanism between clients and service servers. Answer questions solely based on the explicit content of the input. Do not infer, guess, or assume any information that is not present in the data. If the question lacks sufficient context, state what additional information is required to answer the question effectively. Respond by providing the direct answer to the question and a reference to where this can be found in the context provided. Be honest about the limitations of the data. Clarify uncertainty respectfully and avoid misleading conclusions. # Preamble The following is a JSON topology of a ROS2 robotic system. It lists the nodes, their publishers, subscribers, clients and service servers. # Instruction Analyse the <json> topology and answer the <question> strictly based on the provided <json> topology. <json>{json}<json> Please provide your answer between the <answer></answer> tags and the explanation between the <explanation></explanation> tags. Your output must include {answer instruction}, and must not exceed the limit of 100 words. <question>{question}<question> Let’s think step by step to be sure we have the right answer.
ways of representing the ROS2 systems. The best results were achieved using the JSON topology, as LLMs are fundamentally textual models with a fixed context window. Trial runs were used to determine the formulation of the different types of questions and to specify the system’s instructions. Finally, the Chain-of-Thought instruction was added to encourage the LLMs to reason step by step, improving the clarity and reliability of their responses. We ultimately conducted the experiment by executing 1,230 prompts to each LLM, resulting in a total dataset of 1,230 x 9 = 11,070 prompts (and corresponding answers).
4.7
Experiment Execution
Before running the experiment, the topologies of the three ROS2 systems were retrieved in isolation. We opted to follow the guidelines as described by the maintainers for the simulations, specifically: (1) pubsub: Create a package and include the code for the talker and listener nodes. Build the system and start the talker node. Then, start the listener node. Wait 10s before running the topology generation script. (2) turtlebot: Follow the ‘Quick Start Guide’ to configure TurtleBot3 for ROS2 Humble. Build the system. Install the simulation package. Launch the simulation using an empty world. Start RViz2. Wait 10s before running the topology generation script. (3) panda: Install the project following the steps for the ROS2 Humble Environment. Import and install the improved move_group_
Can Large Language Models Assist the Comprehension of ROS2 Software Architectures?
interface.h file. Build the system. Launch Gazebo + MoveIt!2 Environment + ROS2 Robot Triggers/Actions. Select the empty world. Wait 10s before running the topology generation script.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Based on these topologies, questions were generated independently for each system according to algorithm 1. The prompt plan (see Figure 2) then sampled these questions following Equation 1 and generated a structured JSON object to query the models. The experiment was carried out between 30 June and 16 July, 2025 using APIs provided by the organisations. Each question was prompted one at a time for each LLM, and each question was presented only once. A script was used to run the experiment, iterating over the LLMs defined in a configuration file containing the specific settings of each LLM (e.g., organisation, model name, batch support). The API requests included a specification of the specific model and the prompt. No configuration parameters were provided. The models’ responses were logged as .TXT files, and the extracted message text was exported to a CSV file.
computing it (GPT-2 in our case). Lower perplexity values indicate that the model generates more coherent explanations, reflecting higher internal consistency in the provided answer. We used topic modelling [21] for identifying mentioned topics in the models’ reasoning across responses. The analysis focusses exclusively on the text enclosed within the <explanation> tags. Before modelling, entity names and question-related terminology (e.g., ‘topology’, ‘node’, ‘topic’) are removed. The remaining text is subsequently cleaned, tokenised, and lemmatised before applying BERTopic [19]. BERTopic was selected due to its ability to extract novel insights from short and unstructured text and consistent performance in different domains [16]. A manual examination of a sample of 234 responses revealed recurring patterns about the use of list structures, references to the topology, and the inclusion of instructional cues. To quantitatively assess the prevalence of these patterns, a set of regular expressions was applied to systematically identify and count their occurrences within the results.
4.8
5
Data Analysis
We extract the number of tokens used by each LLM from log files and then we compute descriptive statistics for models and systems. For assessing LLMs’ accuracy, their answer within the <answer> tag is automatically extracted using a regex and compared to the ground truth. Comparisons are performed automatically on the basis of the general patterns identified. In total, 300 answers were labelled ‘incorrect’ and we manually reviewed all of them to detect edge cases (e.g., the closing </answer> tag was missing in 2 answers), resulting in the identification of four additional ‘correct’ answers. We also sampled 273 correct answers, following a 95% confidence interval and a 5% error rate, and manually checked their correctness. For each combination of the LLM and ROS2 system, the response accuracy is determined by dividing the number of correct answers by the total number of questions for the respective system. Moreover, the computation graphs of the systems are plotted to depict the nodes and their connections through topics and services. The graphs are coloured according to the number of incorrect answers made concerning the specific entities. Perplexity is used as a complementary metric to evaluate the quality of the explanations generated by the LLMs. The perplexity of the models’ responses is used to measure the certainty of the LLMs. Perplexity of an answer 𝑎 is defined as Perplexity(𝑎) = 2
1 −𝑀
Í𝑀
𝑗 =1 log 𝑃 (𝑡 𝑗 |𝑡 < 𝑗 ,𝑝 )
, where: • 𝑎 = (𝑡 1, 𝑡 2, . . . , 𝑡𝑀 ) is the answer produced by the LLM; • 𝑀 = number of tokens in the answer; • 𝑡 𝑗 = the 𝑗-th token in the answer; • 𝑡 < 𝑗 = all tokens before 𝑡 𝑗 in the answer; • 𝑝 = the prompt provided to the LLM; • 𝑃 (𝑡 𝑗 | 𝑡 < 𝑗 , 𝑝) = probability assigned by the LLM to token 𝑡 𝑗 given the prompt and previous tokens. We compute 𝑃 by using a pre-trained GPT-2 model and tokenizer since (i) it is efficient in terms of time and computational resources, (ii) its logits and probabilities for each token are already available via the Hugging Face Transformers library [22], (iii) it provides a consistent tokenization for all outputs of all 9 LLMs used in this study. Intuitively, perplexity indicates how predictable a sequence produced by a model (i.e., one of the 9 selected LLMs) is to the model
Questions Generation
To ensure coverage of the various architectural components of the ROS systems, six categories of questions were established that cover entities in general, the different types of interface, and the connection between nodes. The type of questions can be multiplechoice (MCQ), boolean (BOOL), or open-ended (OPEN). Based on the JSON-encoding of a ROS2 computation graph (see Section 4.4), a set of questions is generated according to Algorithm 1. First all entities (i.e., unique nodes, topics, and service servers) are extracted from the topology. For each entity, a question is generated to determine whether it is an entity or not, and a question on what kind of entity it is. Moreover, fake entities names are generated based on the different parts of actual entity names. The number of fake entities generated is equal to the number of real entities in the system. For these questions, only a question is included as to whether it is an entity in the ROS system. Then, for each node, boolean questions are generated for each possible topic or service to cheque whether the node publishes/subscribes to the topic or provides the service or is client to the service server. There is also an open question for the publishers, subscribers, services, and clients of each node. In addition, for every combination of two nodes, a boolean question is included on whether there is a communication path from one to the other. Lastly, for each service server and topic, an open question is generated on what the type is. The total number of questions generated for each system 𝑠 is |𝑄𝑠 | ≈ |𝑁𝑠 | × (|𝑁𝑠 | + 3|𝑆𝑠 | + 3|𝑇𝑠 | + 5) + 2|𝑆𝑠 | + 2|𝑇𝑠 | + |𝑅 𝑓 ,𝑠 | (where 𝑁𝑠 the set of all nodes in 𝑠, 𝑆𝑠 the set of all services in 𝑠, 𝑇𝑠 the set of all topics in 𝑠, and 𝑅 𝑓 ,𝑠 the set of false entities generated for 𝑠). For example, for the pubsub system (with |𝑁𝑝𝑠 | = 2, |𝑆𝑝𝑠 | = 12, and |𝑇𝑝𝑠 | = 3 ), this results in |𝑄 𝑝𝑠 | ≈ 2 × (2 + 3 × 12 + 3 × 3 + 5) + 2 × 12 + 2 × 3 + 2 = 136 generated questions. 30, if ⌊0.1 × 𝐺⌋ < 30 𝑆 = ⌊0.1 × 𝐺⌋ , if 30 ≤ ⌊0.1 × 𝐺⌋ ≤ 100 (1) 100, ⌊0.1 if × 𝐺⌋ > 100 Since the number of questions generated is in Ω(𝑁 2 ), we opted to sample the questions according to Equation 1. The sample size 𝑆 was determined for each category and question type, where 𝐺 represents the total number of generated questions. A minimum of
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta
Algorithm 1: Systematic generation of questions about a ROS2 computation graph input: R = ROS2 computation graph output: Q = { Level, Category, Type, Question, Answer } 1 begin 2 𝑁 ← ROS2 nodes defined in 𝑅 ; 𝑆 ← ROS2 services defined in 𝑅 ; 𝑇 ← ROS2 topics defined in 𝑅 ; 3 foreach 𝑒 ∈ 𝑅 𝑓 do 4 𝑄 ← 𝑄 ∪ { (0, ENTITY, BOOL, “Is there a ROS2 entity called <e>?”, “No” ) } ; 5 6 7
8 9 10 11 12 13 14 15 16 17 18 19 20 21
22 23 24 25
foreach 𝑒 ∈ 𝑅 do 𝑄 ← 𝑄 ∪ { (0, ENTITY, BOOL, “Is there a ROS2 entity called <e>?”, “Yes” ) } ; 𝑄 ← 𝑄 ∪ { (0, ENTITY, MCQ, “What kind of ROS2 entity is <e>? Possible answers: 1- a ROS topic, 2- a ROS service, 3- a ROS node.”, find_answer() ) } ; foreach 𝑛 ∈ 𝑁 do foreach 𝑡 ∈ 𝑇 do 𝑄 ← 𝑄 ∪ { (1, PUBLISH, BOOL, “Does node <n> publish to topic <t>?”, find_answer() ) } ; 𝑄 ← 𝑄 ∪ { (1, SUBSCRIBE, BOOL, “Is node <n> subscribed to topic <t>?”, find_answer() ) } ;
𝑄 ← 𝑄 ∪ { (1, PUBLISH, OPEN, “To which topics can node <n> publish?”, find_answer() ) } ; 𝑄 ← 𝑄 ∪ { (1, SUBSCRIBE, OPEN, “To which topics is node <n> subscribed?”, find_answer() ) } ; foreach 𝑠 ∈ 𝑆 do 𝑄 ← 𝑄 ∪ { (1, SERVICE, BOOL, “Does node <n> provide service <s>?”, find_answer() ) } ; 𝑄 ← 𝑄 ∪ { (1, CLIENT, BOOL, “Does node <n> use service <s> as a client?”, find_answer() ) } ; 𝑄 ← 𝑄 ∪ { (1, SERVICE, OPEN, “Which services does node <n> provide?”, find_answer() ) } ; 𝑄 ← 𝑄 ∪ { (1, CLIENT, OPEN, “Which services does node <n> use as a client?”, find_answer() ) } ; foreach 𝑛𝑖 ∈ 𝑁 do if 𝑛𝑖 ! = 𝑛 then 𝑄 ← 𝑄 ∪ { (2, MESSAGE, BOOL, “Is there a communication path from node <n> to node <𝑛𝑖 > via a topic or service?”, find_answer() ) } ; foreach 𝑠 ∈ 𝑆 do 𝑄 ← 𝑄 ∪ { (1, SERVICE_TYPE, OPEN, “What is the type of service <s>?”, find_answer() ) } ; foreach 𝑡 ∈ 𝑇 do 𝑄 ← 𝑄 ∪ { (1, TOPIC_TYPE, OPEN, “What is the type of topic <t>?”, find_answer() ) } ;
30 questions were sampled when available; otherwise, all questions were included. In general, we selected 10% of the questions generated, with an upper limit of 100. Table 4 shows the population and corresponding sample sizes for each type of question per system.
Answer to RQ1 – The number of output tokens increases with the complexity and size of the architecture of the system. o3 and gpt-4o-mini tend to have higher variance across all systems, with the latter generating more tokens than others.
6.2 6 Results 6.1 Output Size (RQ1 ) An overview of the number of output tokens used per LLM and ROS2 system is shown in Figure 3. In general, the number of tokens used increases with the complexity of the systems with an overall mean of 150, 177, and 293 for the pubsub, turtlebot, and panda, respectively. Only in the case of gemini-2.5-flash is the average number of tokens used to answer the questions related to pubsub the highest. In all other cases, an increase in system complexity results in an increase in tokens used by the models. The responses of gpt-4o were generally longer with on average 386 tokens, and more dispersed with a standard deviation is 195. The distribution of the number of tokens used per type of question (i.e., category and type) shows that especially the MESSAGE_BOOL questions result in a large number of tokens used (see Figure 4). The tokens necessary also increases with the size of the system, from a mean of 226 for pubsub to on average 273 for turtlebot and 308 for panda. Figure 5 shows the number of tokens used for each response compared to the cost of the responses related to the distribution of correct and incorrect responses.
Accuracy (RQ2 )
The vast majority of questions were answered correctly (97.9%), with even a 100% accuracy on the pubsub questions across all LLMs. In total, 300 questions were incorrectly answered. There were 51 incorrect answers related to turtlebot, and 249 regarding panda. More than 98% (297/300) of the incorrect answers refer to the MESSAGE_BOOL questions. Of the 132 questions total in this category, 108 were incorrectly answered at least once. The other three incorrect answers were concerning the CLIENT_OPEN questions by chatgpt-4o and o4-mini, and SUBSCRIBE_OPEN by gemini-2.5-flash, all in the panda system. Table 5 shows the accuracy of the LLMs per ROS system. Analysis of the incorrect answers reveals that both gemini-2.5-flash and o4-mini extracted topics and services that were not in the topology for the SUBSCRIBE_OPEN and CLIENT_OPEN questions respectively. In the case of gemini-2.5-flash the correct topics were included in the answer; however, an additional topic was added to the list. o4-mini included a service while none was listed in the topology. chatgpt-4o, on the other hand, extracted the correct list of clients but excluded several on purpose, resulting in an incomplete list in the answer. The LLM reasoned that, because “we are only interested in services being used as clients”, only entries
Can Large Language Models Assist the Comprehension of ROS2 Software Architectures?
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 4: Overview of the number of questions generated for each system, with sample sizes given in (parentheses). Category Type pubsub turtlebot panda TOTAL
Entity BOOL MCQ 34 17 (30) (17) 152 76 (30) (30) 698 349 (70) (35) 884 442 (130) (82)
Publish BOOL OPEN 6 2 (6) (2) 120 8 (30) (8) 1,320 40 (100) (30) 1,446 50 (136) (40)
Subscribe BOOL OPEN 6 2 (6) (2) 120 8 (30) (8) 1,320 40 (100) (30) 1,446 50 (136) (40)
Service BOOL OPEN 24 2 (24) (2) 424 8 (43) (8) 11,040 40 (100) (30) 11,488 50 (167) (40)
Client BOOL OPEN 24 2 (24) (2) 424 8 (43) (8) 11,040 40 (100) (30) 11,488 50 (167) (40)
Message BOOL 2 (2) 58 (30) 1,560 (100) 1,620 (132)
S_Type OPEN 12 (12) 53 (30) 276 (30) 341 (72)
T_Type OPEN 3 (3) 15 (15) 33 (30) 51 (48)
TOTAL 136 (132) 1,472 (313) 27,796 (785) 29,406 (1,230)
Figure 3: Distribution of the output tokens used by the LLMs across the ROS2 systems.
Figure 4: Distribution of the output tokens used for the different questions across the ROS2 systems. Table 5: Overall accuracies of the models per system. LLM gemini-2.5-pro o3 chatgpt-4o claude-opus-4 gemini-2.5-flash gpt-4.1 grok-3 claude-sonnet-4 o4-mini Mean
Figure 5: Number of output tokens against the costs.
with the term “srv” would qualify, and the ones included “action” were therefore excluded from the final answer.
pubsub 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
turtlebot 100.0% 99.68% 96.49% 99.04% 99.68% 95.53% 94.89% 99.04% 99.36% 98.19%
panda 100.0% 99.62% 91.34% 98.98% 99.49% 90.45% 91.97% 97.83% 98.60% 96.48%
Mean 100.0% 99.77% 95.94% 99.34% 99.72% 95.33% 95.62% 98.96% 99.32% 98.22%
The LLMs mainly struggle with the MESSAGE_BOOL questions, specifically the ones where the topic ’/parameter_events’ provides the sole communication path. This behaviour is expected, since all nodes publish by default to this topic. The dataset used in the study included a total of 963 such questions (107 unique), and 293 times these questions were incorrectly answered. In the cases where it was not considered a valid path, more than half of the time the analysis was incomplete (152, 52.1%) because not all
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
interfaces were identified. However, when ’/parameter_events’ was correctly listed as one of the interfaces, the connection was often ignored (63, 21.6%) or identified as an invalid path (58, 19.9%) because it is a ‘global’, ‘system-wide’ topic that ‘does not constitute a unique communication path’. Various times the LLMs mixed up the list of subscribers and publishers, resulting in a wrong conclusion (16, 5.5%). Three times the question was answered in the wrong direction (1.0%). In the other 5 cases, the LLMs identified a communication path, while this was not the case; claude-sonnet-4 made a directionality error twice and once by claude-opus-4, gemini-2.5-flash extracted a list of subscribers that did not exist, and claude-opus-4 confused a correctly retrieved subscriber for a publisher, resulting in a non-existent path.
Figure 6: LLMs’ accuracy when the only possible communication path is through ’/parameter_events’.
In total, each model received 107 MESSAGE_BOOL questions where the only possible path is through ’/parameter_events’. While gemini-2.5-pro consistently answered these questions positively, all other models varied in their analysis (see Figure 6). gpt-4.1, grok-3, and chatgpt-4o more often said there was not a communication path, 89, 79, and 78 times, respectively. The other LLMs did analyse a communication path at least 83.2% of the times. In 15 of the cases where the LLM answered positively, the response did not include a reference to the topic ’/parameter_events’. There are 14 cases where the LLM stated that it could not determine whether a node used a specific service or client because the client information was missing. Based on this, LLMs responded negatively to the possibility of a connection, and the responses provided were evaluated as correct since the missing field indicates that the node is not a client for any service server. The analysis of the computation graphs (see our replication package [26]) tell us that for pubsub all nodes and topics are correctly analysed by the LLMs, whereas for turtlebot and panda we observe at least one incorrect answer for 25.0% and 22.5% of the nodes, respectively. If we do not consider the ’/parameter_events’ topic, the majority of the incorrect answers is related to a very low number of nodes within the 3 studied systems, bringing the maximum number of incorrect answers per node down to only two.
Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta
Answer to RQ2 – Accuracy is high across LLMs (mean=98.22%), with perfect accuracy on the simplest system (i.e., pubsub). gemini-2.5-pro performs best (100%), followed by o3 (99.77%) and gemini-2.5-flash (99.72%), while gpt-4.1 shows the lowest accuracy (95.33%). Of the 11,070 prompts executed in this study, only 300 are answered incorrectly, mostly involving the largest system (i.e., panda) and a single question type (message exchange detection involving parameter changes).
6.3
Perplexity (RQ3 )
There is a wide disparity in the mean perplexity across LLMs. o4-mini has the highest score with 103.6, while the lowest is achieved by chatgpt-4o with 19.6. The Anthropic models achieve similar scores with claude-sonnet-4 20.6 and claude-opus-4 20.0. The Google models resulted in a score of 33.5 for gemini-2.5-pro and 48.7 for gemini-2.5-flash, and xAI’s grok-3 score is 35.5. The mean perplexity for o3 is 70.4 and for gpt-4.1 it is 27.5. A common pattern identified in the explanations of the LLMs is the usage of lists. Numbered lists were used 472 times by different models, and more than 25% of the responses included dotted or dashed lists (2,849/11,070). Especially claude-opus-4 and claude-sonnet-4 used both types of lists often. In contrast, grok-3 did not include any type of list in its response. In addition, all models occasionally reference the message or service types in their responses to questions other than SERVICE_TYPE_OPEN and MESSAGE_TYPE_OPEN. Some models include an explicit instruction that they “need to check” in their response, in the case of claude-sonnet-4 the instruction is included in 52.2% of the responses. Answer to RQ3 – The mean perplexity exhibited by LLMs varies substantially, from 19.6 (chatgpt-4o) to 103.6 (o4-mini), indicating notable differences in the stability and predictability of the generated explanations for ROS2 software architecture-related answers.
6.4
Mentioned Topics (RQ4 )
In total, six distinct topics were identified in the explanations provided by the LLMs. Table 6 includes the specific words associated with each topic, the coherence of the topic, and the total number of occurrences in all responses. Table 6: Topics identified in the explanations of the LLMs. 0 1 2 3 4 5
Topic Words Coherence service refer- [‘server’, ‘service’, ‘list’, ‘provides’, 0.394 ences ‘provided’, ‘client’, ‘type’, ‘array’] publisher ref- [‘publisher’, ‘type’, ‘publishes’, ‘list’, 0.409 erences ‘message’, ‘topic’, ‘msg’, ‘published’] system entities [‘entity’, ‘ro’, ‘name’, ‘server’, ‘node’, 0.720 ‘type’, ‘named’, ‘provided’] subscriber ref- [‘subscriber’, ‘subscribed’, ‘list’, ‘topic’, 0.406 erences ‘subscription’, ‘msg’, ‘type’, ‘listed’] communication [‘communication’, ‘path’, ‘subscribes’, 0.762 path ‘publishes’, ‘via’, ‘direct’, ‘topic’, ‘node’] interfaces [‘client’, ‘server’, ‘publisher’, ‘sub0.536 scriber’, ‘listed’, ‘service’, ‘field’, ‘us’]
Count 3,710 2,019 1,793 1,574 1,182 792
Table 7 shows how the topics are distributed in the different LLMs. In general, the distribution is uniform to global counts. However, the topic service references occur significantly less in the responses of claude-opus-4 and claude-sonet-4, while the interfaces are more present in the answers provided by those two LLMs.
Can Large Language Models Assist the Comprehension of ROS2 Software Architectures?
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 7: Distribution of the topics per LLM. LLM gemini-2.5-pro o3-2025 chatgpt-4o claude-opus-4 gemini-2.5-flash gpt-4.1 grok-3 claude-sonnet-4 o4-mini
0 405 433 435 337 447 434 420 339 460
1 224 218 224 241 220 225 226 224 217
2 208 195 213 207 192 211 206 211 150
3 175 178 177 156 180 172 177 175 184
4 132 131 132 132 130 133 133 132 127
5 86 75 49 157 61 55 68 149 92
Although there are more categories than topics (see Table 8), categories can be grouped into the identified topics. For service, publish, entities, subscribe, and message questions, the mapping is one-to-one with the topics. In the other cases, categories are strongly related (e.g., client with service) or a subcategory (e.g., S_Type of Service). The T_Type questions, however, are mostly amongst publisher references topic while this category of questions is also closely related with the subscriber references. Table 8: Distribution of the topics per category. Category Entity Publish Subscribe Service Client Message S_Type T_type
0 123 5 1 1,734 1,208 0 638 1
1 18 1,558 31 0 0 0 0 412
2 1,753 1 4 12 11 0 9 3
3 4 16 1,537 0 0 1 0 16
4 2 1 0 0 0 1,179 0 0
5 8 3 11 117 644 8 1 0
Answer to RQ4 – Topic coherence ranges from 0.394 (service references) to 0.762 (communication path). The distribution of topics tends to be uniform across LLMs, with the only exception of Claude models which tend to mention more interfaces and less service references.
7
Discussion
General reflections. Our results show that LLMs performed well as assistants for architecture comprehension tasks on ROS2 systems. All models provided correct answers about the pubsub system. However, the accuracy of most models decreased as system complexity increased, particularly for panda. The majority of the mistakes made were related to communication paths, indicating that the models struggle with reasoning about implicit or indirect communication paths. Specifically, there was no consensus amongst the models regarding the purpose of the ’/parameter_events’ topic. The variation in handling the system-level topic between models indicates differences in internal reasoning and training exposure to ROS2 conventions. Although gemini-2.5-pro consistently accepted it as a valid link, other models, including gpt-4.1 and chatgpt-4o, tended to reject it. This variation indicates that LLMs differ in how they conceptualise global communication topics, which may affect their reliability in more complex reasoning tasks. There were several occasions in which the models hallucinated. For example, both gemini-2.5-flash and o4-mini subtracted a list of topics or services that were not in the topology. In addition, some models occasionally hallucinated a communication path through ’/parameter_events’. However, this was also the result of interpreting the direction of the question incorrectly.
The results of topic modelling highlight that LLMs tend to focus on enumerating structural entities rather than abstract relationships or behaviours. The strong presence of references in the explanations, along with frequent list-based reasoning, reinforces that the models are more confident in retrieval-based reasoning than in inferring implicit dependencies. Moreover, the presence of procedural markers such as “need to check” indicates that models often simulate reasoning through a stepwise instruction-like discourse. The mean perplexity scores reveal substantial differences in the models’ ability to generate confident responses to architecturallyrelevant questions. o4-mini exhibits the highest perplexity, indicating that its output is the least predictable and potentially less fluent, while chatgpt-4o achieves the lowest score, suggesting more consistent and coherent responses. The differences suggest that perplexity can provide an additional lens for evaluating not only the correctness of responses but also the fluency and internal consistency of the explanations generated by each model. From a computational perspective, the increase in output tokens with system complexity reflects greater reasoning effort by the models. However, longer responses did not always correlate with higher accuracy. Moreover, LLMs that cost more tend to have shorter responses, but also slightly more incorrect answers. The dispersion in token usage, particularly by chatgpt-4o, also reflects variability in how models balance confidence and explanation detail. Implications for ROS developers. Our results highlight both the potential and limitations of current LLMs as tools for supporting the comprehension of system topologies and supporting design validation. Although models can reliably extract explicit information from the topology, they show greater difficulty in reasoning about implicit or system-wide communication paths. Moreover, the results highlight the need for caution when prompting the models, as they have an inherent bias with respect to, for instance, system-level topics such as /parameter_events. Based on ones own preferences, developers might prefer to use gemini-2.5-pro, for instance, over gpt-4.1, or vice versa. Our results also emphasise the need for further exploration of different prompting strategies, evaluate them using multiple ROS systems and LLMs. Further practical implications of our study include: (i) if cost is a top priority, gemini-2.5-flash is the LLM producing the lowest number of output tokens and it is the cheapest in terms of both input and output tokens, still with excellent results (see Table 5); (ii) if the system under analysis is highly complex and accuracy of the LLMs’ answers is a top priority, then developers should use an LLM in the Gemini family, like gemini-2.5-pro (see Table 5); (iii) developers should pay special attention to system-level entities like the parameter_events topic when prompting LLMs since they might strongly influence the accuracy of the outcomes (see Figure 6); (iv) we suggest to use multiple LLMs during a comprehension task since different LLMs provide answers with different reasoning structures and levels of perplexity, which can help architects getting a more nuanced understanding of the architecture of their ROS2 system. Implications for researchers. Our results point to promising avenues for fine-tuning or instruction alignment specifically targeted at improving reasoning tasks in the context of ROS2 systems. Specifically, building on the evidence provided in this work that LLMs perform reasonably well in comprehending basic facts about
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
the software architecture of a ROS2 system, the software engineering community might explore the boundaries of such comprehension capabilities; for example, by answering the following research questions: To what extent can LLMs comprehend more complex facts about a ROS2 system (e.g., presence of architectural antipatterns [31] or opportunities for refactoring [4])? How can telemetry data be integrated into the architecture description prompted to the LLM (e.g., via RAG or a tool-based AI agent) and used for extracting facts related to non-functional properties (e.g., energy consumption [2], performance [20])? How can LLMs be effectively integrated into a (mesh of) AI agents for supporting architecting activities [40]? Another line of research with potential is the representation of architectural tactics [6] into a knowledge based that LLMs can use to provide timely recommendations about applicable tactics either at development time or at runtime, basically leading to LLM-supported architectural self-adaption for ROS2 systems [18].
8
Threats To Validity
Internal Validity. We are aware about the LLMs’ sensitivity to prompt phrasing. This concerns both the prompt plan and the formulation of the questions. To mitigate this risk, the prompt and questions were treated as a controlled variable: all models were evaluated using the same standardised prompts, which were iteratively refined until consistent responses were observed. External Validity. The chosen ROS2 systems might not represent the diversity of ROS2 deployments in real-world applications. To mitigate this threat, three systems of different sizes were considered to capture a range of system topologies and interface configurations, ensuring that the evaluation included both minimal setups and more elaborate architectures. All three systems were developed by third parties and were not created by the authors, which strengthens this study by ensuring that the evaluation reflects systems that could realistically be encountered in practice. The choice of used LLMs form another threat to validity. However, the choice to only use the highest-rated models was made because the goal of this study is to evaluate current state-of-the-art capabilities in reasoning about ROS2 systems. In addition, the usage of nine models from four different providers strengthens the diversity of the evaluation. Construct Validity. Ambiguous or underspecified questions posed a potential threat to construct validity, as they could influence the quality and completeness of the model responses. To address this potential threat, multiple phrasings were iteratively tested and evaluated and a final version was selected that minimised ambiguity, while preserving the intended construct. Moreover, the prompt included an instruction to use no more than 100 words in the response to limit the costs for this study. This restriction introduced a potential threat to validity, as it constrained the ability of some models to fully express their reasoning. The specific representation of the architecture of a ROS2 system used in this study represents a potential threat to validity. To mitigate this threat, several approaches were considered and compared. However, the best performance was achieved in the trial runs with the JSON topology, as included in the current work. The correctness of the LLMs’ answers was assessed by automatically extracting the text enclosed in the <answer> tags and comparing it with a predefined ground truth using regexes. Although this approach ensures consistency and scalability, it also
Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta
introduces a potential threat to the validity of the construct. Subtle differences in phrasing, formatting, or minor variations could be flagged as incorrect, potentially misinterpreting the answer of the model. To mitigate errors introduced by automated evaluation, all answers initially labelled as incorrect were manually reviewed. This step reduced the risk of misclassification due to edge cases, such as missing closing tags or minor formatting differences. By combining automated and manual checks, the evaluation aimed to accurately reflect the correctness of each response while maintaining the integrity of the construct being measured. Another threat concerns the interpretation of the ROS2 topic /parameter_events. This topic can be understood as solely a system-internal mechanism or also as a topic available for communication between nodes. Such ambiguity may lead LLMs to produce responses that are logically valid under one interpretation but appear incorrect under the evaluation criteria. To address this issue, responses concerning these questions were more thoroughly analysed to identify the models’ underlying motivations. Nevertheless, this issue illustrates how ambiguity in domain-specific concepts can affect whether the evaluation accurately measures the intended construct of reasoning about the architecture of ROS2 systems. Conclusion Validity. Due to the differences in sizes between the considered systems, there was a wide disparity between the number of questions generated for each of the systems. To balance the questions per system and keep the experiment manageable, we decided to use a sampling strategy to select the questions used. For each system, question category and type, at least 30 and at most 100 questions were randomly sampled, if available. The minimum of 30 questions was chosen to leverage the Central Limit Theorem, which suggests that sample means tend to approximate a normal distribution for sample sizes of 30 or more, supporting more reliable statistical analysis. However, especially for the smaller pubsub system, there were often not 30 questions to sample from. In those cases, all questions were selected. However, smaller sample sizes can limit statistical power, increase variability, and reduce confidence in the conclusions drawn from the data. Although this approach ensured feasibility, it can affect the reliability of statistical comparisons between systems and models.
9
Conclusion
This study presents initial results on the ability of large language models (LLMs) to support the comprehension of ROS2 systems. We analysed the answers of nine LLMs to a total of 1,230 questions related to three different ROS2 systems of varying complexity. The responses were compared with a predefined ground truth to assess correctness, and topic modelling was applied to analyse the reasoning patterns. The findings indicate that LLMs have the capability of supporting the comprehension of ROS2 concepts and can accurately answer architecturally-relevant questions. This suggests that they have potential to assist robotics architects, particularly in system exploration and validation tasks. However, differences between models and recurring misinterpretations of system-level communication reveal that their reasoning about structural relationships remains limited. As future work, we will replicate this study using a broader range of (possibly industrial, large-scale) ROS2 systems and more recent
Can Large Language Models Assist the Comprehension of ROS2 Software Architectures?
LLMs to assess the generalisability of the findings. Additionally, the analysis will be extended to include other system entities, such as action servers, which serve as important interfaces for node interactions. Further investigations will explore the impact of prompt engineering techniques on LLMs’ accuracy. Alternative representations of ROS2 architectures (e.g., using rospec [8] will be used to evaluate the effect on the reasoning of the LLMs. We will also involve humans in verifying LLM-provided answers to strengthen the validity of the qualitative analysis and mitigate potential bias introduced by models’ self-justification. We also plan to ground LLM queries in ROS2 developers’ GitHub issues and pull request to have a deeper assessment of the capabilities of LLMs concerning ROS software architectures.
Acknowledgments This work is supported by the InnoGuard Marie Skłodowska-Curie Doctoral Network (Grant Agreement No. 101169233).
References [1] Michel Albonico, Milica Dordevic, Engel Hamer, and Ivano Malavolta. 2023. Software engineering research on the Robot Operating System: A systematic mapping study. Journal of Systems and Software 197 (2023), 111574. doi:10.1016/ j.jss.2022.111574 [2] Michel Albonico, Andreas Wortmann, and Ivano Malavolta. 2026. Tuning ROS 2 for Energy-Efficient Navigation: Empirical Insights from Costmap 2D Configurations. In 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE. [3] Domenico Amalfitano, Marco De Luca, Tiziano Santilli, Patrizio Pelliccione, and Anna Rita Fasolino. 2026. Automated Software Architecture Design Recovery from Source Code Using LLMs. In Software Architecture, Vasilios Andrikopoulos, Cesare Pautasso, Nour Ali, Jacopo Soldani, and Xiwei Xu (Eds.). Springer Nature Switzerland, Cham, 73–89. [4] Abdulrahman Ahmed Bobakr Baqais and Mohammad Alshayeb. 2020. Automatic software refactoring: a systematic literature review. Software Quality Journal 28, 2 (2020), 459–502. [5] Victor R Basili and H Dieter Rombach. 1988. The TAME project: Towards improvement-oriented software environments. IEEE Transactions on software engineering 14, 6 (1988), 758–773. [6] Len Bass, Paul Clements, and Rick Kazman. 2021. Software architecture in practice. Addison-Wesley Professional. [7] Kai Beckman and Jonas Reininger. 2018. Adaptation of the DDS Security Standard for Resource-Constrained Sensor Networks. In 2018 IEEE 13th International Symposium on Industrial Embedded Systems (SIES). 1–4. doi:10.1109/SIES.2018.8442103 [8] Paulo Canelas, Bradley Schmerl, Alcides Fonesca, and Christopher Steven Timperley. 2025. The Usability Argument for ROS-based Robot Architectural Description Languages. In Proc. the 15th annual workshop on the intersection of HCI and PL. [9] Zhuojun Chen, Michel Albonico, and Ivano Malavolta. 2023. Automatic Extraction of Time-windowed ROS Computation Graphs from ROS Bag Files. arXiv:2305.16405 [cs.RO] https://arxiv.org/abs/2305.16405 [10] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning. [11] Zina Chkirbene, Ridha Hamila, Ala Gouissem, and Unal Devrim. 2024. Large language models (llm) in industry: A survey of applications, challenges, and trends. In 2024 IEEE 21st International Conference on Smart Communities: Improving Quality of Life using AI, Robotics and IoT (HONET). IEEE, 229–234. [12] Rudra Dhar, Karthik Vaidhyanathan, and Vasudeva Varma. 2024. Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study. arXiv:2403.01709 [cs.SE] https://arxiv.org/abs/2403.01709 [13] J. Andrés Díaz-Pace, Antonela Tommasel, and Rafael Capilla. 2024. Helping Novice Architects to Make Quality Design Decisions Using an LLM-Based Assistant. In Software Architecture, Matthias Galster, Patrizia Scandurra, Tommi Mikkonen, Pablo Oliveira Antonino, Elisa Yumi Nakagawa, and Elena Navarro (Eds.). Springer Nature Switzerland, Cham, 324–332. [14] Colin Diggs, Michael Doyle, Amit Madan, Eric O. Scott, Emily Escamilla, Jacob Zimmer, Naveed Nekoo, Paul Ursino, Michael Bartholf, Zachary Robin, Anand Patel, Chris Glasz, William Macke, Paul Kirk, Jasper Phillips, Arun Sridharan, Doug Wendt, Scott Rosen, Nitin Naik, Justin F. Brunelle, and Samruddhi Thaker.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom 2025. Leveraging LLMs for Legacy Code Modernization: Evaluation of LLMGenerated Documentation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). 177–184. doi:10.1109/LLM4Code66737. 2025.00027 [15] Open Robotics Discourse. 2025. 2024 ROS Metrics Report. https://discourse. openrobotics.org/t/2024-ros-metrics-report/42354. Forum post, accessed 22 October 2025. [16] Roman Egger and Joanne Yu. 2022. A Topic Modeling Comparison Between LDA, NMF, Top2Vec, and BERTopic to Demystify Twitter Posts. Frontiers in Sociology Volume 7 - 2022 (2022). doi:10.3389/fsoc.2022.886498 [17] Matteo Esposito, Xiaozhou Li, Sergio Moreschini, Noman Ahmad, Tomas Cerny, Karthik Vaidhyanathan, Valentina Lenarduzzi, and Davide Taibi. 2025. Generative ai for software architecture. applications, challenges, and future directions. Journal of Systems and Software (2025), 112607. [18] David Garlan, Bradley Schmerl, and Shang-Wen Cheng. 2009. Software architecture-based self-adaptation. In Autonomic computing and networking. Springer, 31–55. [19] Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794 [cs.CL] https://arxiv.org/abs/2203.05794 [20] Engel Hamer, Michel Albonico, and Ivano Malavolta. 2025. Resource Utilization of 2D SLAM Algorithms in ROS-Based Systems: an Empirical Evaluation. Journal of the Brazilian Computer Society (JBCS) 31, 1 (2025), 229–261. doi:10.5753/jbcs. 2025.4343 [21] Mustapha Hankar, Mohammed Kasri, and Abderrahim Beni-Hssane. 2025. A comprehensive overview of topic modeling: Techniques, applications and challenges. Neurocomputing 628 (2025), 129638. doi:10.1016/j.neucom.2025.129638 [22] Hugging Face. 2025. Transformers. https://huggingface.co/docs/transformers/ index [23] IFRA Group, Cranfield University. 2022. ros2_RobotSimulation: Ready-to-use ROS 2 Gazebo + MoveIt!2 simulation packages for industrial and collaborative robots. https://github.com/IFRA-Cranfield/ros2_RobotSimulation. [24] iRobot ROS. 2019. iRobot ROS 2 Performance Evaluation Framework. https: //github.com/irobot-ros/ros2-performance. Accessed: 27/03/2025. [25] Abhishek Kumar Kashyap and Kavya Konathalapalli. 2025. Autonomous navigation of ROS2 based Turtlebot3 in static and dynamic environments using intelligent approach. International Journal of Information Technology (2025), 1–23. [26] Laura Duits and Bouazza El Moutaouakil and Ivano Malavolta. 2026. Replication package of this study. https://figshare.com/s/bb59cd6b6e03a5e861ca [27] Athul Krishna M J, Ajai V Babu, Suraj Damodaran, Rekha K James, Muhammed Murshid, and Tripti S Warrier. 2024. ROS2 - Powered Autonomous Navigation for TurtleBot3: Integrating Nav2 Stack in Gazebo, RViz and Real-World Environments. In 2024 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES). 1–6. doi:10.1109/SPICES62143.2024.10779642 [28] Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. 2022. Robot operating system 2: Design, architecture, and uses in the wild. Science robotics 7, 66 (2022), eabm6074. [29] Ivano Malavolta, Grace A. Lewis, Bradley Schmerl, Patricia Lago, and David Garlan. 2021. Mining guidelines for architecting robotics software. Journal of Systems and Software 178 (2021), 110969. doi:10.1016/j.jss.2021.110969 [30] Yuya Maruyama, Shinpei Kato, and Takuya Azumi. 2016. Exploring the performance of ROS2. 2016 International Conference on Embedded Software (EMSOFT) (2016), 1–10. https://api.semanticscholar.org/CorpusID:9703204 [31] Ran Mo, Yuanfang Cai, Rick Kazman, Lu Xiao, and Qiong Feng. 2019. Architecture anti-patterns: Automatically detectable violations of design principles. IEEE Transactions on Software Engineering 47, 5 (2019), 1008–1028. [32] José Francisco Molina Santiago, José-Armando Fragoso-Mandujano, Samuel Gómez-Peñate, Victor David Castillo González, and Francisco-Ronay LópezEstrada. 2023. Trajectory Tracking and Obstacle Avoidance with Turtlebot 3 Burger and ROS 2. In 2023 XXV Robotics Mexican Congress (COMRob). 93–98. doi:10.1109/COMRob60035.2023.10349744 [33] Open Robotics. [n. d.]. Writing a Simple Publisher and Subscriber (Python). https://docs.ros.org/en/humble/Tutorials/Beginner-Client-Libraries/ Writing-A-Simple-Py-Publisher-And-Subscriber.html. [34] G. Pardo-Castellote. 2003. OMG Data-Distribution Service: architectural overview. In 23rd International Conference on Distributed Computing Systems Workshops, 2003. Proceedings. 200–206. doi:10.1109/ICDCSW.2003.1203555 [35] Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. Kobe, 5. [36] ROBOTIS Co., Ltd. [n. d.]. TurtleBot3 Simulation Documentation. https://emanual. robotis.com/docs/en/platform/turtlebot3/simulation/. [37] Larissa Schmid, Tobias Hey, Martin Armbruster, Sophie Corallo, Dominik Fuchß, Jan Keim, Haoyu Liu, and Anne Koziolek. 2025. Software Architecture Meets LLMs: A Systematic Literature Review. arXiv:2505.16697 [cs.SE] https://arxiv. org/abs/2505.16697 [38] Mohamed Soliman, Elia Ashraf, Kamel M. K. Abdelsalam, Jan Keim, and Ashwin Prasad Shivarpatna Venkatesh. 2026. LLMs for Software Architecture Knowledge:
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
A Comparative Analysis Among Seven LLMs. In Software Architecture, Vasilios Andrikopoulos, Cesare Pautasso, Nour Ali, Jacopo Soldani, and Xiwei Xu (Eds.). Springer Nature Switzerland, Cham, 99–115. [39] Mohamed Soliman and Jan Keim. 2025. Do Large Language Models Contain Software Architectural Knowledge? : An Exploratory Case Study with GPT. In 2025 IEEE 22nd International Conference on Software Architecture (ICSA). 13–24. doi:10.1109/ICSA65012.2025.00012 [40] Karthik Vaidhyanathan and Henry Muccini. 2025. Software Architecture in the Age of Agentic AI. In European Conference on Software Architecture. Springer,
Laura Duits, Bouazza El Moutaouakil, and Ivano Malavolta
41–49. [41] Jianxun Wang and Yixiang Chen. 2023. A Review on Code Generation with LLMs: Application and Evaluation. In 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI). 284–289. doi:10.1109/MedAI59581.2023.00044 [42] Quanjun Zhang, Chunrong Fang, Siqi Gu, Ye Shang, Zhenyu Chen, and Liang Xiao. 2025. Large Language Models for Unit Testing: A Systematic Literature Review. arXiv:2506.15227 [cs.SE] https://arxiv.org/abs/2506.15227