arXiv CSopen access

Automated Summarization of Software Documents: An LLM-based Multi-Agent Approach

Noname manuscript No. (will be inserted by the editor)

Automated Summarization of Software Documents: An LLM-based Multi-Agent Approach

arXiv:2606.24689v1 [cs.SE] 23 Jun 2026

Duc S. H. Nguyen · Minh T. Nguyen · Phuong T. Nguyen · Juri Di Rocco · Davide Di Ruscio

the date of receipt and acceptance should be inserted later

Abstract Large Language Models (LLMs) and LLM-based Multi-Agent Systems (MAS) are revolutionizing software engineering (SE) by advancing automation, decision-making, and knowledge processing. Their recent application to SE tasks has already shown promising results. In this paper, we focus on summarization as a key application area. We present Metagente, an LLM-based MAS designed to generate concise and accurate summaries of software documentation. Metagente employs a Teacher–Student architecture where multiple LLM agents collaborate to enhance relevance and precision of produced summaries. An empirical evaluation on real-world datasets demonstrates Metagente’s effectiveness in streamlining workflows, outperforming the considered baselines. The evaluation provides evidence that Metagente improves summarization for requirements analysis and technical documentation. Our findings underscore the transformative potential of these technologies in SE, while identifying challenges and future research directions for their seamless integration. Duc S. H. Nguyen Hanoi University of Science and Technology, Vietnam E-mail: [email protected] Minh T. Nguyen Hanoi University of Science and Technology, Vietnam E-mail: [email protected] Phuong T. Nguyen University of L’Aquila, Italy E-mail: [email protected] Juri Di Rocco University of L’Aquila, Italy E-mail: [email protected] B Davide Di Ruscio University of L’Aquila, Italy E-mail: [email protected]

2

Duc S. H. Nguyen et al.

Keywords Summarization · LLMs · Multi-Agent Systems · GitHub · Google Play

1 Introduction In Software Engineering, there has been an increasing use of documents to provide detailed descriptions for software artifacts. Among others, README files in GitHub are written in the Markdown format, containing information about a repository, e.g., instructions, help, or updates. A well-written README facilitates reading comprehension, helping visitors to grasp the scope of a repository. This is also the case of mobile app stores like Google Play Store,1 in which apps are normally equipped with a long HTML document, allowing visitors to understand the apps’ functionalities. An empirical study (Liu et al., 2022) showed that README is often the very first item that visitors take a look at when it comes to becoming acquainted with a GitHub repository. Similarly, apps’ descriptions are of high importance, as they can be used as sources to mine domain knowledge (Liu et al., 2017). By several GitHub repositories and Google Play apps, instruction files are usually lengthy, and reading and understanding them can require time and effort, thus discouraging visitors from continuing with the repositories/apps. Therefore, apart from README, GitHub also allows its users to add a short description called “About” to each repository, offering a succinct summary of the main functionalities. In a similar manner, besides a full README file, apps in Google Play Store feature a brief description, resembling the “About” summary in GitHub repositories. Unfortunately, in these platforms, while being useful, this field is usually overlooked and left unfilled, posing difficulties for those who want to explore the repositories, on the fly. Recently, GitSum (Doan et al., 2023) has been proposed as the first approach to summarize long README files to yield brief but concise “About” descriptions. Being built on top of BART and T5, GitSum successfully produces relevant summaries starting from long README files. However, while obtaining an encouraging recommendation performance, the tool still suffers from a low accuracy for input files that are of mixed fields, including code and text. This triggers the need for a more effective way to summarize README files. Large Language Models (LLMs) have been applied in Software Engineering (Ozkaya, 2023; Nguyen et al., 2024) to solve a wide range of tasks. To name but a few, researchers have employed LLMs in code summarization (Sun et al., 2024; Haldar and Hockenmaier, 2024), debugging (Lee et al., 2024; Tian et al., 2024), testing (Arawjo et al., 2024; Li et al., 2025a), or code generation (Gu, 2023; Fakhoury et al., 2024; Huang et al., 2024). However, LLMs are not without limitations: Existing work (He et al., 2025) has revealed that while they are gaining popularity, their performance on tasks requiring domain-specific expertise or complex multi-step reasoning remains limited. In this context, 1

https://play.google.com/store/apps

An LLM-based MAS for the Summarization of Software Documents

3

prompt-tuning is a practical approach, in which prompts are iteratively refined to improve the performance of the pre-trained language model without modifying its internal design. Noteworthy, the prompt-tuning process can be prone to subjective bias and scalability issues, making it difficult to generalize across diverse tasks (White et al., 2023). Moreover, it is not easily accessible for several developers as it might require advanced technical expertise. To cope with the limitations of single LLMs, multi-agent systems have been proposed to enable specialized LLMs to collaborate within a shared framework (He et al., 2025; Wang et al., 2024a). These systems magnify the unique strengths of various LLMs, where agents specialize in tasks such as code generation, debugging, or domain-specific problem-solving (He et al., 2025; Xia et al., 2024). Challenges such as effective coordination, efficient communication, and the overhead of integrating multiple agents persist, even though dedicated frameworks like LangChain2 or LLamaIndex3 are now increasingly adopted in Software Engineering to mitigate such issues. Nevertheless, MASs represent a compelling alternative to relying solely on the capabilities of individual models (Wang et al., 2024b). In our previous work (Nguyen et al., 2025), we developed Metagente, a multi-agent framework composed of four LLM-based agents, which interact in a collaborative manner using a teacher-student loop, enabling prompt optimization with minimal supervision. An evaluation using datasets collected from GitHub demonstrated that Metagente obtains a good recommendation performance, showing that the proposed approach is suitable for generating summaries for software documents. In this paper, we enhance Metagente across several dimensions. First, to improve efficiency, we propose a dynamic iteration strategy that adaptively halts processing of low-potential samples, reducing computational cost without compromising quality. Second, we extend the framework to incorporate multiple LLMs as the recommendation engine, enhancing its flexibility. Third, we broaden the evaluation to include summarization of structured HTML-based application descriptions into concise texts. This task is particularly challenging due to the heterogeneous content of HTML files, which can complicate summarization. Our evaluation on curated datasets demonstrates that Metagente achieves strong semantic alignment and competitive ROUGE scores while requiring fewer training iterations, outperforming various baselines. In summary, the main contributions of this work include the novel dynamic iteration strategy, the integration of diverse LLMs, and the expanded evaluation scope for complex summarization tasks, elaborated as follows. – Solution. We introduced an extended version of Metagente, a collaborative LLM-based multi-agent system for summarizing app descriptions. Aiming for efficiency, a dynamic stopping mechanism has been deployed, so as to optimize the fine-tuning process. 2 3

https://www.langchain.com/ https://www.llamaindex.ai/

4

Duc S. H. Nguyen et al.

– Evaluation. Using datasets collected for mobile apps’ documents, we conducted an empirical evaluation to study the performance of Metagente. Moreover, we also compared dynamic vs. non-dynamic strategies, evaluating generalization across test subsets. – Reproducibility. The replication package including code, data, and prompts has been released to foster reproducibility.4 Structure. Section 2 provides some background related to the importance of README and “About” descriptions in Google Play. Section 3 introduces Metagente, our proposed approach to the summarization of README files using LLM-based MAS. The evaluation is elaborated in Section 4, and the results are reported and analyzed in Section 5. In Section 6, there are discussion on the impacts, and the threats to the validity of the findings. We review the related work in Section 7. Finally, Section 8 sketches future work, and concludes the paper.

2 Motivation and Background Section 2.1 presents a motivating example to demonstrate the necessity of “About” descriptions in the Google Play ecosystem. Afterwards, the ROUGE scores are presented in Section 2.2 as a base for the presentation of Metagente in Section 3.

2.1 Motivating Examples Figure 1(a) shows the Telegram app, seen from the frontpage on Google Play Store. Apart from essential information including the number of reviews (15M with 4.0 as rating), number of downloads (>1B), there is also a short description, which is actually the “About” field, describing the main functionalities, i.e., “Telegram is a messaging app with focus on speed and security.” Such a description is brief but informative, and it helps users–especially first-time visitors–gain a quick orientation to decide whether to continue with the app, without reading the corresponding long README, which resides behind the frontpage shown in Figure 1(b). Figure 2 depicts another example of Courtside 18915 that is also a wellmaintained app. However, while having a long and detailed README (see Figure 2(b)), the app does not feature any short description (see Figure 2(a)). Due to such a lack, visitors need to spend time reading the README in order to understand the app’s functionalities. Essentially–in some extent–this may discourage visitors from continuing with the app. The examples motivate us to come up with an approach to summarize long README documents to yield a short description. In our previous work (Doan 4 5

https://github.com/MDEGroup/Metagente https://play.google.com/store/apps/details?id=com.pl.fiba&hl=en

An LLM-based MAS for the Summarization of Software Documents

(a) Front page

5

(b) README

Fig. 1 Telegram features a brief description for the app in its front page.

et al., 2023), we conceived GitSum as a first approach to translate long GitHub README files to obtain “About” descriptions. Using BART and T5 as the recommendation engine, GitSum can generate relevant summaries for GitHub repositories. However, while earning a high accuracy in different testing samples, GitSum still struggles with input files composed of miscellaneous fields, including code and text. In this paper, we aim to overcome the current limitations, leveraging the synergies of different task-specific LLMs that are properly orchestrated to enhance the recommendation quality. In particular, we employ an LLM to perform the extraction of the input data, filtering out noisy fields and retaining the most meaningful ones. More importantly, by means of a Teacher-Student architecture applying collaborative LLMs, we iteratively refine a set of prompts to produce the best fit prompt, which then can be used to generate the final summary. The next subsection describes the metrics used by Metagente to steer the adjustment phase of LLMs-based agents.

6

Duc S. H. Nguyen et al.

(a) Front page

(b) README

Fig. 2 Despite a lengthy README file, Courtside 1891 does not offer any informative descriptions.

2.2 Metrics for Evaluating Summarization Tasks Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004) is a set of metrics applied in evaluating automatic summarization and machine translation (Chen et al., 2021; Zhang et al., 2022). Given a pair of a produced summary and a ground-truth text, ROUGE metrics judge their similarity based on the overlap between them. ROUGE metrics are in the range of 0 and 1, in which a higher value corresponds to a higher similarity between the automatically produced summary and the reference one. The metrics are computed as follows: P P CountG (gramn ) R,G∈S Pgramn ∈R P recisionrouge−n = P R,G∈S gramn ∈G CountG (gramn ) P P CountG (gramn ) R,G∈S Pgramn ∈R Recallrouge−n = P R,G∈S gramn ∈R CountR (gramn ) F 1rouge−n =

2 ∗ P recisionrouge−n ∗ Recallrouge−n P recisionrouge−n + Recallrouge−n

where R, G, and S are the reference summary, generated summary, and the test set, respectively. gramn is an n-gram phase, where n is the length of

An LLM-based MAS for the Summarization of Software Documents

7

a word sequence; CountR (gramn ) and CountG (gramn ) are the occurrence number of gramn in R and G. ROUGE-1 P and ROUGE-2 are calculated with N = 1, 2, i.e., uni-gram and bi-grams. gramn ∈R CountG (gramn ) represents the number of N-grams existing in both the ground-truth summary and generated summary. The aforementioned equations are interpreted as follows. Given two sets of N-grams produced from a generated summary and its original summary, respectively, Precision evaluates the proportion of N-grams in the first set that is found in the second set, while Recall measures the proportion of N-grams in the second set that exists in the first set. ROUGE F1-score is based on Longest Common Subsequence (LCS) defined as follows: LCS(R, G) ; Recallrouge−l = length(G) 2 ∗ P recisionrouge−l ∗ Recallrouge−l F 1rouge−l = P recisionrouge−l + Recallrouge−l

P recisionrouge−l =

LCS(R, G) length(R)

where LCS(R, G) represents the length of the longest common subsequence of reference summary and its generated summary (R and G), respectively. ROUGE-L indicates the natural similarity between the two given sequences in sentence-level structure. It is also considered to be useful in evaluating summarization performance. In our proposed approach, ROUGE scores are used as means to guide the learning process, allowing the recommendation engine to refine the summaries, making them more relevant. The detailed architecture of Metagente together with its constituent components is described in the next section.

3 Proposed Approach This section introduces the Metagente approach to the summarization of software documents using LLMs-based agents. To deal with noise in the input data, an LLM has been deployed as an extractor component, filtering out noisy and retaining the most meaningful fields. More importantly, by means of a Teacher-Student architecture applying collaborative LLMs, we iteratively refine a set of prompts to produce the best fit prompt, which then can be used to generate the final summary. As shown in Fig. 3, there are two main phases, i.e., Optimization and Evaluation, explained as follows.

3.1 Optimization The pipeline shown in Fig. 3(a) involves four specialized LLM-powered agents: Extractor Agent, Summarizer Agent, Teacher Agent, and Prompt Creator Agent. These agents collaborate through prompt-driven communication, and

8

Duc S. H. Nguyen et al.

(a) Optimization Pipeline

(b) Evaluation Pipeline Fig. 3 Pipelines used in Metagente.

their outputs are evaluated using ROUGE metrics to iteratively improve the quality of produced summaries as presented below. Extractor Agent. This agent is responsible for isolating the most informative parts of a raw description that can be in different formats, e.g., HTML and Markdown. An application description is typically structured text document that may contain sections such as introduction, key features, advantages, detailed use cases, setup instructions, compatibility. Extractor Agent filters out irrelevant or boilerplate content, focusing only on the segments that explain the core function and value proposition of the application. This filtered version, rather than the entire original document, is then passed on for downstream processing, significantly improving both computational efficiency and focus during training. Summarizer Agent. Given the refined description, this agent generates a concise and meaningful “About” statement for the app. Starting from an initial prompt, it iteratively produces summaries that are then improved over multiple rounds using updated prompts generated by Teacher Agent. By means of a dedicated prompt, Summarizer Agent focuses on capturing the app’s primary goal, context, and differentiating features, while excluding peripheral or overly detailed information. The final result is a short and clear description that accurately represents the essence of the application.

An LLM-based MAS for the Summarization of Software Documents

9

Teacher Agent. It evaluates the quality of each generated summary using the following four elements: (1) the current summarization prompt; (2) the generated short description; (3) the ground-truth target description; and (4) the computed ROUGE-L scores. Based on these inputs, it produces an improved prompt for Summarizer Agent. This optimization follows a multi-step reasoning structure designed to pinpoint mismatches and refine instructions accordingly. Prompt Creator Agent. After multiple training iterations across diverse samples, this agent aggregates and generalizes the refined prompts into a single, robust instruction set. It extracts recurring patterns and conditional logic to form a unified prompt suitable for inference. This final prompt helps Summarizer Agent adapt to varying input description styles and content structures, ensuring consistent output quality. Metagente has been designed so as to allow for a flexible integration of different LLMs. In particular, for this implementation, five LLMs are considered as the engine, including: GPT-4o, GPT-3.5-turbo, Gemma, Mistral, and Llama. For the configuration with GPT, to optimize resource usage, Extractor Agent and Summarizer Agent are implemented using the lightweight GPT-4o-mini version, which offers an efficient balance of performance and cost. In contrast, Teacher Agent and Prompt Creator Agent leverage the more capable GPT-4o model to perform higher-level evaluation and reasoning. This hierarchical configuration enables the system to scale effectively while maintaining strong overall performance. As illustrated in Figure 3(a), the workflow of the agents unfolds as follows: 1. A description is provided to Extractor Agent, which generates a concise version of the text to be used as input by Summarizer Agent. 2. Optimization Loop: (a) Summarizer Agent receives the extracted text and produces a summary based on its current prompt, shown below. (b) A ROUGE-L score is then calculated by comparing the generated summary with the ground truth summary. If the score is equal to or exceeds a predefined threshold, the optimization loop is interrupted, and the current prompt used by Summarizer Agent is saved as a candidate prompt. (c) If the ROUGE-L score is below the threshold, Teacher Agent receives the inputs from Extractor Agent and Summarizer Agent along with the ROUGE-L score. It uses this information to generate a new prompt, which is then passed to Summarizer Agent for the next iteration. The prompt used by Teacher Agent is as follows. (d) The loop continues until one of the following three conditions is satisfied: (i) the ROUGE-L score meets or exceeds the threshold; (ii) the maximum number of predefined iterations is reached; or (iii) the ROUGE-L score stops improving or begins to decline. 3. Once the loop ends, a new HTML description is passed through the same pipeline to generate another candidate prompt.

10

Duc S. H. Nguyen et al. Extractor Agent Prompt

Your task is to shorten and extract only the introduction and description information from an app. You are given the following description for an app: <Description> readme text </Description> # Steps - **Identify the structure of the app’s description**: The app’s description is a structure text file that might contains many sections such as introduction, description, key features, advantages, detailed use case, setup instructions, compatibility,... - **Remove all sections that are not relevant to the app’s description**: Irrelevant sections might include technical guidance (installing/running/specification... instruction), compatibility, troubleshooting,... - **Remove all unnecessary links/tags**: Identify all links/tags that DO NOT contribute to the description of the app. You must remove all of these reference links and tags. - **Return only text that is relevant to the description of the app**: The output should only contains the text that is relevant to the introduction/description of the app, including the app name/title, app feature description/purpose statement/overview. DO NOT include any output identifications such as: ”Here’s the ...” or ”Extracted App’s description:” ”””

Summarizer Agent Prompt Your task is to shorten and extract only the introduction and description information from an app. You are given the following description for an app: “”” Summarize the following extracted text from an app’s description into a short term/phrase introducing the app: <EXTRACTED APP’S DESCRIPTION> $extracted text </EXTRACTED APP’S DESCRIPTION> The output should include only a short term/phrase introducing the app. “””

4. After all HTML descriptions have been processed, Prompt Creator Agent collects all candidate prompts that achieved a ROUGE-L score above the defined threshold and aggregates them into a single, final prompt. The agent performs its activities by means of the following prompt.

3.2 Orchestration of Agents Metagente operates as a collaborative pipeline in which multiple LLM-based agents interact toward a common objective. The orchestration of these agents is structured around three main phases: Agent Communication, Iterative Refinement, and Prompt Consolidation, explained as follows. – Agent Communication. Each agent functions not in isolation but through controlled interaction with its environment and its peers. To facilitate this, we utilize the LangChain framework as the communication backbone. In addition, outputs are structured in a machine-readable format, enabling

An LLM-based MAS for the Summarization of Software Documents

11

Teacher Agent Prompt You are a professional Prompt Engineer. You are working on a system using a Large Language Model (LLM) to help developers automatically generate a short Description term/phrase contain key concept/idea from an extracted text of the description of an app. Your task is to modify and improve the current prompt of the LLM based on the result of testing on a data include a description and a ground truth description. # Steps: - **Analyze the data for testing**: Analyze the following data include an extracted text from a description and a ground truth description from an app: <EXTRACTED TEXT> $extracted text </EXTRACTED TEXT> <GROUND TRUTH DESCRIPTION> $description </GROUND TRUTH DESCRIPTION> - **Review the current result**: Review the generated description using the extracted text its ROUGE score on the ground truth description to identify improvements that could be made: <GENERATED DESCRIPTION> $generated about </GENERATED DESCRIPTION> <ROUGE SCORE> $rouge score </ROUGE SCORE>

Prompt Creator Agent Prompt You are a professional Prompt Engineer. You are working on a system using a Large Language Model (LLM) to help developers automatically generate a short Description term/phrase contain key concept/idea from an extracted text of the description of an app. Your task is to combine several candidate prompts for the LLM into a final prompt. # Steps: - **Review all candidate prompts**: Analyze the following prompts to identify common parts to be included in the final prompt and also includes specific details or conditional key points from these prompts to be included in the final prompt <CANDIDATE PROMPTS> $summarizer list </CANDIDATE PROMPTS> - **Generate a final prompt**: Based on the common parts and conditional key points, generate a final prompt for the LLM. # Output Format: Do not include any reasoning/explanation like ”Based on the result of the above review:”, ”Here’s the”, ... or any output identifiers like ”Prompt:”, ”New Prompt”, ... The output should only include a string representing the prompt for the LLM ”””

agents to reliably pass information to one another. This design ensures consistency and seamless cooperation throughout the pipeline. – Iterative Refinement. The process begins with Summarizer Agent receiving a filtered version of the application’s HTML description from Extractor Agent. Using the current summarization prompt, it generates a candidate “About” summary. The result is evaluated using the ROUGE-L metric, which serves as a primary indicator of semantic and structural alignment

12

Duc S. H. Nguyen et al.

with the ground-truth summary. This feedback, along with the current prompt and outputs, is passed to Teacher Agent, which formulates an improved prompt for the next round. Unlike traditional fixed-iteration loops as proposed in our previous work (Nguyen et al., 2025), in this paper we implement a dynamic stopping criterion to optimize the training efficiency as follows: 1. Each sample starts with a baseline allowance of 15 iterations. 2. If the ROUGE-L score stagnates across 3 consecutive iterations or decreases in 2 iterations (with each drop also counted as stagnation), the process halts early for that sample. 3. If ROUGE-L reaches a predefined threshold (0.7), then the process stops. We expect this adaptive mechanism prevents unnecessary computation on low-potential samples, while still offering more iteration budget to promising ones. In addition, we examine the impact of this dynamic iteration strategy in comparison to a non-dynamic (fixed-loop) setting to evaluate gains in performance and efficiency. – Prompt Consolidation. After the optimization process completes, we gather a pool of prompt versions, each tailored to a specific training instance. These prompts are analyzed by Prompt Creator Agent, which extracts common structures and key conditional instructions to synthesize a generalized summarization prompt. This final prompt is then used during inference to ensure that Summarizer Agent performs effectively across a diverse range of HTML description inputs, maintaining both coherence and relevance in the generated “About” summaries.

3.3 Evaluation Pipeline Once the optimized prompt has been obtained from the Optimization Pipeline, the evaluation phase can then be initiated. As illustrated in Figure 3(b), the Evaluation Pipeline is comparatively simple and consists of only two agents, i.e., Extractor Agent and Summarizer Agent. The former uses the same prompt and performs the same function as in the optimization phase, processing the HTML description to extract a concise description. The latter, however, uses the final optimized prompt produced in the previous phase for all summarization. After the summarization, ROUGE-1, ROUGE-2, ROUGEL and cosine similarity were collected to evaluate the quality of the generated summaries against their respective ground truth references. These metrics are collected from the evaluation of several HTML descriptions so that the optimization task could be assessed.

An LLM-based MAS for the Summarization of Software Documents

13

4 Evaluation This section presents the empirical study conducted to evaluate the proposed approach’s performance. We introduce the research questions in Section 4.1, and describe the datasets in Section 4.2.

4.1 Research Questions To evaluate Metagente we answer the following research questions, and compare it with baselines. – RQ1 : How does Metagente perform compared to single LLM-based agents? We investigate if the use of a multi-agent architecture is really needed, given that a single agent might already be sufficient to get a decent recommendation. For this RQ, we consider Mixtral-8x7B-Instruct-v0.1, Llama-2-7b-hf, GPT-4o, and Gemma-2-2b-it as baselines for comparison as they have been widely used in summarization tasks. – RQ2 : Does the dynamic iteration strategy contribute to training efficiency? This RQ evaluates whether the newly proposed dynamic iteration strategy is beneficial to the generation of short summaries, i.e., effective and/or efficient, compared to the static one conceived in our previous work (Nguyen et al., 2025). – RQ3 : Which LLM contributes to a better performance of Metagente? The previous version of Metagente relies on Original LLM set including GPT-4o and GPT-4o-mini as the only engine for agents. In this paper, we further extended the architecture to sustain other LLMs, including Mistral-7B-Instruct-v0.3, Llama-3.2-3B-Instruct, and Gemma-2-2b-it, to validate the extensibility of Metagente.

4.2 Datasets In this section, we explain in detail the process conducted to curate, clean, and restructure the datasets used in the evaluation. 4.2.1 Data Curation To assess the effectiveness of Metagente in a domain beyond traditional software documentation, we curated a novel dataset of mobile applications derived from the AndroZoo6 repository (Allix et al., 2016), a large-scale archive comprising over 25 million Android applications mined from various sources. We initially selected a representative sample of 20,000 applications from the 5GB AndroZoo summary metadata file, filtering for apps that (i) were sourced from the Google Play Store; and (ii) had a release date after January 1st, 6

https://androzoo.uni.lu/

14

Duc S. H. Nguyen et al.

2020 (we consider apps in the most recent 5 years). Application metadata was retrieved programmatically through the AndroZoo API, which provides multiple versions for each application along with associated attributes (e.g., name, description short, description html, version string). The following filtering criteria were applied to refine the corpus: – Apps with only one version available were excluded. – For multi-version apps, only the metadata of the most recent version was retained. – Apps without a star rating (i.e., no user reviews) were discarded. – Using the langdetect7 library for language identification, we excluded apps whose description html or description short fields are not written in English. After filtering, the final dataset consisted of 2,980 mobile applications. For each application, we used the description html field as input and the corresponding description short as the ground-truth summary. Table 1 reports the number of applications falling into predefined rating intervals. Table 1 Distribution of applications by average star rating. Star Rating Range 0–1 1–2 2–3 3–4 4–5

Number of Applications 0 68 361 854 1,696

As shown in the table, the majority of the apps in the dataset have a rating above 3.0, with over half being rated between 4 and 5. This distribution reflects a natural skew toward well-rated apps, which are also more likely to have informative and well-structured descriptions. Apps without any rating were excluded as part of our data quality filtering process, and the statistics are depicted in Table 2. Table 2 Summary statistics of the mobile application dataset (token-level estimates). Metric Average Length Median Length Minimum Length Maximum Length Standard Deviation

description html 289 tokens 244 tokens 2 tokens 798 tokens 198 tokens

description short 10 tokens 11 tokens 1 token 17 tokens 3 tokens

A distributional analysis reveals that input texts exhibit high variance and right-skewness, consistent with the verbose and heterogeneous nature of mobile 7

https://github.com/Mimino666/langdetect

An LLM-based MAS for the Summarization of Software Documents

15

app descriptions. In contrast, the ground-truth summaries are concise, with low variance and a narrow range, confirming the suitability of this dataset for evaluating the summarization capabilities of Metagente. 4.2.2 Data Preprocessing Data quality plays a crucial role in the fine tuning and inference phases. To assess the quality of these pairs, we conducted an exploratory data analysis (EDA) by computing ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) and Cosine Similarity between description html and description short, and the scores are shown in Figure 4.

Fig. 4 The relationship between Cosine Similarity and ROUGE scores.

As seen in the figure, there is a clear positive correlation between ROUGE and Cosine Similarity scores. In other words, a higher semantic similarity tends to coincide with a denser n-gram overlap. This observation allows us to come up with the following data filtering strategy. – Training set. Given the goal of building a system that learns effectively from a small amount of high-quality data, we selected training samples with 0.85 ≤ Cosine Similarity < 1.0. This range ensures that each training pair is highly relevant yet not identical, thereby reducing noise and enhancing learning efficiency. – Test set. It was constructed from the remaining samples not used in training. The following filtering rule was applied: (ROUGE-1 ≥ 0.1) or (ROUGE-2 ≥ 0.1) or (ROUGE-L ≥ 0.1) and Cosine Similarity ≥ 0.4 These thresholds were selected based on empirical inspection of samples at various cutoff points, ensuring a reasonable level of summary relevance while maintaining diversity. This filtered set contains over 800 samples, and it is used to derive the following final sets for the evaluation.

16

Duc S. H. Nguyen et al.

– D1 : 400 randomly sampled test cases. – D2 : 600 randomly sampled test cases. – D3 : Top 200 samples with the highest Cosine Similarity scores. – D4 : Top 200 samples with the highest ROUGE-L scores. These subsets enable us to evaluate the performance of Metagente across both general and high-quality cases, facilitating robust and diverse performance analysis. The next section elaborates on the results obtained from the empirical evaluation.

5 Experimental Results We report and analyze the experimental results by answering the research questions introduced in Section 4.1.

5.1 RQ1 : How does Metagente perform compared to single LLM-based agents? In our previous work (Nguyen et al., 2025), Metagente was compared with various studies, and the results demonstrated that it consistently outperformed the baselines in generating repository descriptions from GitHub README files. Recent studies have evaluated the performance of different LLMs on summarization tasks across various domains (Aly et al., 2025; Takeshita et al., 2025; Keya et al., 2025; Khan et al., 2025). These studies highlight that models such as Mixtral-8x7B-Instruct-v0.1 and Llama-2-7b-hf achieve competitive or superior results, especially in zero-shot settings. To further deepen the comparison with single-agent LLMs, we extended our evaluation to include two widely adopted general-purpose LLMs that are frequently used in summarization tasks including GPT-4o and Gemma-2-2b-it. We evaluate Metagente against the baselines using two datasets, i.e., D1 and D2 and the results are shown in Figure 5. Overall, the generated summaries exhibit higher or comparable semantic similarity (cosine) and competitive ROUGE scores compared to ground-truth summaries. As shown in Figure 5, it is evident that both the baselines, i.e., Mixtral-8x7B-Instruct-v0.1 and Llama-2-7b-hf, achieve a lower and less stable performance on the two datasets. An extremely low recommendation quality is seen by Mixtral-8x7B-Instruct-v0.1, i.e., most of the ROUGE scores are smaller than 0.15. Their outputs also show a greater variance–some summaries are overly generic, while others omit key aspects of the original descriptions. This underscores the baselines’ limitations when used for summarization without iterative refinement or agent collaboration. The figure shows that Metagente consistently outperforms both baselines across all the evaluation metrics, i.e., it delivers more consistent and higherquality results across both random subsets. The advantage is especially clear in

An LLM-based MAS for the Summarization of Software Documents

17

ROUGE-L, which captures structural and semantic similarity with the reference summaries. In particular, by the D1 dataset (see Figure 5(a)), most of the ROUGE-L scores obtained by Metagente range from 0.3 to 1.0. Meanwhile by Mixtral-8x7B-Instruct-v0.1 and Llama-2-7b-hf, the corresponding scores are much smaller, e.g., less than 0.2. The same trend is witnessed with D2 (see Figure 5(b)), in which the ROUGE scores obtained by Metagente are always superior to those of Mixtral-8x7B-Instruct-v0.1 and Llama-2-7b-hf. This suggests that Metagente generates summaries that are not only concise but also more faithful to the original content, reflecting both the overall purpose and the unique details of the applications described. These improvements point to the value of the multi-agent optimization loop in producing summaries that are easier to read and more informative. The stability of these improvements across metrics and datasets provides strong evidence that the Teacher–Student refinement loop and prompt consolidation strategy not only improve accuracy but also enhance robustness and generalizability in diverse summarization settings. Answer to RQ1 : Metagente consistently outperforms various single LLMbased agents in terms of all the ROUGE scores, demonstrating the benefit of collaboratively joint LLM-based agents.

18

(a) D1

Duc S. H. Nguyen et al.

(b) D2 Fig. 5 Comparison between Metagente and LLM-based single agents on the D1 and D2 datasets.

An LLM-based MAS for the Summarization of Software Documents

(a) Comparison of generated summary with ground-truth description short

(b) Comparison of generated summary with ground-truth description html Fig. 6 Dynamic vs. Non-Dynamic strategies. 19

20

Duc S. H. Nguyen et al.

5.2 RQ2 : Does the dynamic iteration strategy contribute to training efficiency? We conducted a comparative analysis between the dynamic and non-dynamic prompting strategies using ROUGE scores and cosine similarity. The results are depicted in Figure 6. Quantitatively, the non-dynamic approach shows slightly higher averages in ROUGE-2 and ROUGE-L, suggesting stronger n-gram overlap with reference summaries. In contrast, the dynamic method performs on par in ROUGE-1 and demonstrates stronger semantic alignment in cosine similarity, highlighting its ability to capture meaning beyond surface-level overlap. Interestingly, despite comparable performance, the dynamic strategy completed training in only 260 iterations, while the non-dynamic method requires 620 iterations to converge. The comparison of efficiency between non-dynamic and dynamic strategies is shown in Figure 7.

(a) Average Flow Duration

(b) Average Step Duration

(c) Number of input tokens

(d) Number of output tokens

Fig. 7 Comparison of efficiency between non-dynamic and dynamic strategies.

The figure shows that compared to the non-dynamic method, the dynamic one is always efficient with respect to the timing and resources used. This suggests that the dynamic method is significantly more efficient in learning useful prompts, adapting early based on intermediate feedback. Figure 6 further highlights that both strategies follow similar trends in ROUGE and cosine similarity distributions. However, the dynamic version tends to produce outputs with higher semantic consistency and broader generalization, whereas the non-dynamic one exhibits sharper peaks in overlap-based metrics. The evaluation reveals that, despite producing results comparable to the non-dynamic setting, the dynamic iteration strategy requires significantly fewer

An LLM-based MAS for the Summarization of Software Documents

21

training steps. This means that adaptive stopping criteria help avoid wasted computation while still capturing meaningful improvements. Such efficiency is especially important in real-world deployments, where computational resources are often limited. Answer to RQ2 : The dynamic method creates summaries comparable to the those generated by the non-dynamic ones. However, the dynamic method needs much fewer training steps (260 vs. 620). This means that it improves efficiency by reducing unnecessary training steps while maintaining comparable quality, making the framework suitable for resource-constrained environments in real-world scenarios. 5.3 RQ3 : Which LLM contributes to a better performance of Metagente? To evaluate the compatibility and adaptability of our pipeline across various LLMs, we implemented Metagente using multiple models–ranging from lightweight open-source LLMs to commercial APIs–on the same datasets and measured their performance using ROUGE metrics. In particular, the following models are considered in our evaluation: Original LLM set (a combination of GPT-4o and GPT-4o-mini), GPT-3.5-turbo, Gemma-2-2b-it, Mistral-7b-Instruct-v0.3, and Llama-3.2-3B-Instruct. Figure 8 and Figure 9 depict violin boxplots, comparing the average ROUGE1, ROUGE-2, and ROUGE-L scores across four evaluation subsets: two random test sets, i.e., D1 and D2 with 400 and 600 samples, respectively, and two top-ranked subsets based on Cosine Similarity and ROUGE-L, D3 and D4 each with 200 samples. As shown in the figures, our pipeline demonstrates compatibility with all tested LLMs although performance varies–with the original LLM set consistently outperforming others–the system remains operational and effective with all models. The framework remains robust across different LLM backbones, indicating strong portability and adaptability in diverse real-world scenarios. Even lighter models such as Gemma-2-2b-it and Mistral-7B-Instruct-v0.3 were able to produce summaries meeting minimum ROUGE thresholds, indicating robustness in prompt processing and adaptation.

22

(a) D1

Duc S. H. Nguyen et al.

(b) D2 Fig. 8 Performance comparison on D1 and D2 .

23

(b) D4 Fig. 9 Performance comparison on D3 and D4 .

An LLM-based MAS for the Summarization of Software Documents

(a) D3

24

Duc S. H. Nguyen et al.

In Figure 8, concerning the ROUGE scores obtained on D1 , we can see that using Gemma-2-2b-it, Metagente yields a mediocre performance compared to that when using the other LLMs. Mistral-7B-Instruct-v0.3 as the agent is better than Gemma-2-2b-it, still it underperforms the remaining LLMs including Original LLM set, GPT-3.5-turbo and Llama-3.2-3B-Instruct. Among others, the Original LLM set with two GPT-4o agents and GPT-4o-mini agents contributes to the best performance to Metagente. The same trend is seen on D2 (shown in Figure 8(b)). These findings suggest that our pipeline does not depend on a specific backbone model, making it portable and flexible for different deployment scenarios, including low-resource settings. We further investigated how the system improves summary quality over time using the proposed Teacher-Student architecture. For this, we keep track of the ROUGE scores across iterations for each model and computed the average learning curves for ROUGE-L, ROUGE-2, and ROUGE-1 respectively, up to iteration 7. The final results are depicted in Figure 10. The figure reveals consistent upward trends in ROUGE-L, ROUGE-1, and ROUGE-2 for most models within the early iterations. Original LLM set exhibits the most stable and effective self-improvement behavior, quickly reaching and maintaining high scores. Other models such as Mistral-7B-Instruct-v0.3 and GPT-3.5-turbo also improve over iterations, though with more fluctuations. Llama-3.2-3B-Instruct and Gemma-2-2b-it show limited progress, suggesting that lower-capacity models may struggle to utilize refined prompts effectively. This pattern validates the effectiveness of our prompt refinement loop: after each round, Teacher Agent leverages feedback (including ROUGE scores and summary outputs) to update the prompt for Summarizer Agent, leading to consistent performance gains. The early stopping mechanism also helps terminate training for stagnant samples, reducing redundant computation. The cross-model evaluation demonstrates that Metagente achieves reasonable performance across a wide spectrum of LLMs, including lightweight open-source models and high-capacity proprietary systems. This implies that the framework is not bound to a specific engine, but rather can flexibly adapt to available resources and deployment contexts. Answer to RQ3 : Using Original LLM set as the agent engine helps Metagente obtain a superior recommendation performance. The selfimprovement process is effective across the datasets, with most models showing iterative gains in ROUGE scores. Higher-capacity LLMs benefit more from the prompt refinement mechanism, but even smaller models improve initially, validating the robustness of our training loop.

6 Discussion This section provides discussion related to the implications and possible extensions of our work, as well as the threats to validity of the findings.

An LLM-based MAS for the Summarization of Software Documents

(a) ROUGE-1

(b) ROUGE-2

(c) ROUGE-L Fig. 10 Learning curves.

25

26

Duc S. H. Nguyen et al.

6.1 Implications Our experimental results provide several implications for both research and practice as follows. Usage. The findings demonstrate that Metagente, as a multi-agent framework, consistently outperforms various single-agent LLMs in summarization tasks. This suggests that the collaborative Teacher–Student refinement loop and the dynamic stopping strategy are effective mechanisms for generating concise and faithful summaries. By assigning specialized roles to different agents and enabling iterative refinement, Metagente is able to generate summaries that are both more accurate and more faithful to the original content. This indicates that the benefit of multi-agent systems lies not only in diversity of models but also in the structured cooperation among them. In practice, such a system could be integrated into SE workflows to automatically produce short descriptions of lengthy documentation, thereby reducing cognitive load for developers and end users. Use cases. While our evaluation focuses on Markdown files and application descriptions, the approach could potentially be extended to other domains. For example, the framework could support software documentation summarization, where lengthy requirements or technical reports are converted into concise and accessible descriptions. It will also enhance developer support platforms, such as automatically generating short “About” sections for repositories or applications, improving accessibility for first-time users. Furthermore, it might benefit community-driven Q&A platforms by condensing verbose answers into shorter and more searchable forms. Broader implications. Beyond the specific task of summarizing README and application descriptions, our findings suggest that LLM-based multi-agent systems could serve as a general approach to improve the reliability, scalability, and controllability of AI-assisted summarization. The principles of dynamic prompt refinement and agent specialization can also inspire solutions in adjacent tasks such as code recommendation, bug report summarization, and automated knowledge management. 6.2 Concerns and Limitations While the empirical evaluation demonstrates the effectiveness of Metagente, there are various concerns and limitations as follows. – Diversity of datasets and application contexts. The evaluation primarily focused on GitHub README files and Google Play application descriptions. Although these are representative sources of software documentation, they may not capture the full diversity of real-world documents such as technical reports, requirement specifications, or bug reports. Consequently, the generalizability of Metagente to heterogeneous domains still requires further investigation.

An LLM-based MAS for the Summarization of Software Documents

27

– Task-oriented and user-centered validation. Our evaluation relied on benchmark datasets and quantitative metrics (e.g., ROUGE, cosine similarity). While these measures have been widely used, they may not fully reflect the utility of summaries in practical scenarios where developers, or end-users consume documentation. Conducting user studies, for instance through crossover experiments comparing human- vs. agent-generated summaries, would provide a deeper understanding of Metagente’s real-world impact. – Dependence on training data and potential biases. Although our dataset curation aimed to ensure quality and diversity, the reliance on app descriptions from the Google Play ecosystem may introduce inherent biases, such as domain-specific language or stylistic conventions. Moreover, overlaps with LLM pretraining corpora could unintentionally inflate performance, raising concerns about overfitting. To address this, future evaluations should incorporate datasets from independent sources, including industrial documentation and student-generated artifacts. – Computational cost and scalability. Despite the efficiency gains achieved by the dynamic iteration strategy, the orchestration of multiple LLM-based agents inevitably introduces computational overhead. Scaling Metagente to larger datasets or integrating it into production pipelines may require further optimization or hybrid strategies to balance accuracy and resource consumption.

6.3 Future Research Directions Future work on Metagente can be extended along several promising directions: – Expanding application domains. While the current evaluation has focused on Markdown files and application descriptions, future work could apply Metagente to additional domains such as code summarization, technical documentation, engineering reports, or biomedical texts. Such extensions would allow the framework to demonstrate its adaptability to highly diverse and domain-specific documents. – Reinforcement learning from feedback. Another direction is to incorporate reinforcement learning with human feedback (RLHF) or agent feedback (RLAIF) to optimize the prompt refinement and summarization process. This could provide stronger alignment with user expectations and continuously improve the quality of generated summaries. – Retrieval-augmented generation. Integrating external retrieval mechanisms into the multi-agent loop would allow agents to access contextual information beyond the input document. This hybrid approach could reduce the risk of missing critical background knowledge and enhance factual accuracy. – Cost-aware optimization. Future work should also explore efficient allocation of computational resources by combining lightweight open-source

28

Duc S. H. Nguyen et al.

LLMs with larger proprietary ones in a hybrid manner. Such resourceaware scheduling could reduce cost while sustaining high-quality results, making the system more practical in production environments. – Explainability and transparency. Finally, improving interpretability by designing logging, tracing, and decision-explanation mechanisms for each agent could strengthen trust in industrial deployments. Providing insights into why a specific summary was produced would enhance both reliability and accountability in real-world software engineering workflows.

6.4 Threats to Validity – Internal Validity is related to the factors within the experimental design that may bias or distort the results. In our approach, the Teacher–Student refinement loop and dynamic stopping strategy are sensitive to hyperparameters such as iteration thresholds and ROUGE cutoffs. Suboptimal settings might have influenced convergence speed and output quality. Another internal factor is the selection of baseline models: stronger or weaker baselines may lead to different perceived relative improvements. Moreover, single-agent baselines are highly sensitive to model setup (e.g., prompt design, hyperparameter configuration), which may lead to variability in their performance and affect the fairness of comparison. Future experiments should incorporate cross-validation with randomized seeds, multiple runs per configuration, and more systematic baselines to minimize such risks. – External Validity concerns the generalizability of our findings to other domains, document types, and usage contexts. While GitHub README and Google Play descriptions are representative forms of software documentation, they do not capture the broader diversity of real-world documents such as industrial requirement specifications, technical reports, or bugtracking entries. The language style, structural heterogeneity, and domainspecific jargon in these contexts may pose additional challenges not addressed by our current evaluation. Similarly, the experiments were limited to English-language datasets; generalizing to multilingual documentation or low-resource languages remains unexplored. – Construct Validity is whether the employed measures and datasets adequately capture the concept of summarization quality. In our evaluation, we relied primarily on ROUGE and cosine similarity metrics, which are wellestablished but may fail to reflect semantic fidelity, factual consistency, or user-perceived usefulness of the generated summaries. For instance, a summary with high n-gram overlap may still be misleading in meaning. Additionally, we adopted the short “About” descriptions from Google Play as ground truth references; however, such descriptions are often written with a marketing or promotional purpose, rather than serving as strictly informative summaries. This introduces the risk that models learn to mimic stylistic tendencies rather than optimizing for clarity or completeness. While we

An LLM-based MAS for the Summarization of Software Documents

29

attempted to mitigate this by curating high-quality subsets and applying filtering thresholds, human-centered evaluations (e.g., user studies with developers or app users) would provide a more comprehensive assessment of summary utility.

7 Related Work Recently, there has been a surge in the applications of LLM-based MAS in Software Engineering. This section reviews some of the most notable applications, as well as MAS for summarization and prompt optimization in agent collaborations.

7.1 LLM-based Multi-Agent Systems in Software Engineering A number of representative applications highlight the potential of LLM-based multi-agent systems (LaMAS) across the SE lifecycle. In program synthesis, systems like CodePori (Rasheed et al., 2024) and AgentMesh (Khanzadeh, 2025) demonstrate how assigning specialized roles to multiple agents—such as manager, developers, reviewers, and verification agents–enables the generation of functional, production-level software directly from natural language requirements. CodePori shows strong performance on the HumanEval benchmark and further scales to end-to-end applications spanning thousands of lines of code, while AgentMesh showcases artifact-centric communication and iterative plan–code–review loops for reliable development. Moving beyond coding, MAGIS (Tao et al., 2024) addresses repository-level issue resolution on GitHub by coordinating multiple agents to locate files, handle long contexts, and implement complex fixes, significantly outperforming single-agent LLMs on the SWE-bench benchmark. In industrial contexts, GoNoGo (Khoee et al., 2024) introduces a planneractor setup to support automotive software release decisions, already deployed at Volvo to improve accuracy and reduce manual overhead in risk-sensitive processes. Similarly, CogniSim (Cinkusz and Chudziak, 2025) integrates cognitive agents into Agile and DevOps workflows, showing improvements in backlog refinement, testing, and deployment, though it also raises questions about transparency and human–agent collaboration. At higher abstraction levels, frameworks such as MAAD (Li et al., 2025b) and multi-agent design/refactoring models automate architecture generation, design trade-offs, and quality balancing by assigning domain-specialized agents (e.g., performance, security, UI/UX) and employing structured evaluation protocols. LaMAS is emerging as a promising way to streamline software engineering– from requirements to deployment–while also highlighting challenges such as scalability, reliability, security, and the role of human oversight in collaborative, intelligent development.

30

Duc S. H. Nguyen et al.

7.2 Summarization Metagente exemplifies the growing shift toward multi-agent systems (MAS) to enhance the quality, factual consistency, and controllability of LLM-based summarization. By assigning specialized roles, i.e., Extractor Agent, Summarizer Agent, and Teacher Agent, Metagente decomposes the summarization task into modular subtasks with tailored prompts. This role-based specialization is complemented by adaptive strategies, including early halting of low-potential samples and real-time prompt adjustments, enabling efficient and high-quality generation. This design builds upon prior MAS approaches like MAMM-REFINE (Pan et al., 2024), which introduced a Detect-Critique-Refine pipeline where collaborative agents iteratively improve summaries. Notably, its Multi-Agent Single-Model (MASM) configuration with reranking significantly outperforms single-agent baselines. Metagente extends these ideas by integrating adaptive prompt control and dynamic role coordination throughout the generation process. Other MAS-based summarization systems further demonstrate the versatility of collaborative agents. Chain-of-Agents (Zhang et al., 2024) processes long-form input via chained agents under manager oversight, while D&R (Zhou et al., 2025b) distills stronger models through structured debates and preference optimization. MADRA (Wang et al., 2023) enhances agent reasoning with retrieval-augmented evidence to reduce hallucination, and SR-DCR (Zhou et al., 2025c) leverages asymmetric debates and token-level confidence for arbitration. Even in adjacent domains, such as legal argumentation, the Reflective Multi-Agent framework (Zhang and Ashley, 2025) highlights the benefits of iterative role-driven collaboration for improving factual and ethical integrity. Beyond summarization, several MAS frameworks illustrate the power of structured agent collaboration across broader reasoning and task-solving settings. OPRO (Yang et al., 2024) employs different LLMs as optimizers, using natural language prompts to iteratively generate and refine solutions. APE (Zhou et al., 2023) adopts a similar philosophy by generating candidate instructions and refining them via semantic similarity and evaluation metrics. Camel (Li et al., 2024) introduces a role-playing framework where agents are guided by inception prompting, enabling instruction-following cooperation and generation of multi-agent conversational data. MetaGPT (Hong et al., 2024) embeds human workflows into modular agent behaviors to address hallucination and coordination challenges. Complementary to these systems, AutoGen (Wu et al., 2024) and LangChain8 facilitate the development of LLM-based applications via conversation-driven agents and modular programming abstractions. AutoGen promotes human-in-the-loop interactions through multi-turn dialog and agent orchestration, while LangChain streamlines integration with external tools and data sources. Altogether, these studies contextualize and reinforce Metagente’s contributions: dynamic agent specialization, modular workflows, and adaptive control 8

https://docs.langchain.com

An LLM-based MAS for the Summarization of Software Documents

31

as key mechanisms for building faithful and effective multi-agent summarization systems.

7.3 Prompt Optimization in Agent Collaboration Metagente emphasizes prompt optimization as a core mechanism for effective multi-agent collaborations. Its architecture incorporates dynamic, role-specific prompt refinement—most notably via Teacher Agent, which continuously updates summarization prompts based on ROUGE-L feedback. This closed-loop adaptation enables agents to specialize and respond to diverse inputs with increased coherence, factuality, and control. This design is in line with broader efforts to optimize prompts across agent-based systems. MASS (Zhou et al., 2025a) introduces a framework that jointly searches for both communication topologies and agent prompts, demonstrating that interleaving local and global prompt tuning leads to better system-wide coordination. Similarly, NEXUSSUM (Kim and Kim, 2025) applies modular prompt designs across hierarchical agent roles–Preprocessor, Summarizer, and Compressor—using Chain-ofThought reasoning and few-shot examples to guide generation. Further advances in Automated Prompt Optimization (APO) support the principles underpinning Metagente. MARS (Zhang et al., 2025) proposes a multi-agent dialogue framework, where agents like Planner and Teacher-CriticStudent collaboratively evolve prompts. Shen et al. (Shen et al., 2025) explored feedback-driven refinement, showing that group-level, online optimization consistently improves collaboration quality. Meanwhile, SEE (Cui et al., 2025) and EXPO (Kong et al., 2025) treated prompt tuning as a high-dimensional or dynamic problem, applying metaheuristics and bandit learning to refine both instructions and exemplars. Collectively, these studies reinforce the design choices in Metagente: prompt modularity, agent-level specialization, and adaptive optimization loops are essential to building robust, controllable LLM-based multi-agent systems.

8 Conclusion and Future Work This paper presented Metagente–a novel approach to summarization of software documents leveraging LLM-based multi agents. Using datasets collected for mobile apps’ documents, we conducted an empirical evaluation to study the performance of Metagente. Moreover, we also compared dynamic vs. nondynamic strategies, evaluating generalization across test subsets. The experimental results showed that Metagente is able to generate highly relevant summaries, outperforming single LLMs. Our work showcases the potential of LLM-based MAS in the generation of software documents. Future work will focus on the deployment of the proposed architecture to a wider range of application domains, beyond the scenarios investigated in this study. In particular, we plan to explore tasks such as code recommendation,

32

Duc S. H. Nguyen et al.

automated code comment generation, and other software engineering activities where collaborative LLM-based agents can provide added value. A key challenge in these extensions will be to refine and optimize the architecture of the LLM-based MAS, with the dual objective of improving the overall effectiveness of the system and reducing its computational cost, thus making it more suitable for real-world integration in developer workflows. In addition, an important research direction concerns the definition of benchmarking framework tailored to LLM-based MAS in Software Engineering. Such a framework should not only enable rigorous comparisons across different agent architectures and coordination strategies, but also include metrics that capture aspects such as accuracy, efficiency, robustness, scalability, and fairness. Establishing these benchmarks will be instrumental for the community to assess progress in this emerging area and to guide the development of more reliable, efficient, and trustworthy multi-agent solutions.

Declarations Conflict of interest. All the authors declare that they have no conflict of interest. Furthermore, they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Ethical approval. This article does not contain any studies with human participants or animals performed by any of the authors. Acknowledgements This paper has been partially supported by the MOSAICO project (Management, Orchestration and Supervision of AI-agent COmmunities for reliable AI in software engineering) that has received funding from the European Union under the Horizon Research and Innovation Action (Grant Agreement No. 101189664). The work has been also partially supported by the European Union–NextGenerationEU through the Italian Ministry of University and Research, Projects PRIN 2022 PNRR “FRINGE: context-aware FaiRness engineerING in complex software systEms” grant n. P2022553SL. We acknowledge the Italian “PRIN 2022” project TRex-SE: “Trustworthy Recommenders for Software Engineers,” grant n. 2022LKJWHC.

References K. Allix, T. F. Bissyandé, J. Klein, and Y. Le Traon. Androzoo: Collecting millions of android apps for the research community. In Proceedings of the 13th International Conference on Mining Software Repositories, MSR ’16, pages 468–471, New York, NY, USA, 2016. ACM. ISBN 978-1-4503-41868. doi: 10.1145/2901739.2903508. URL http://doi.acm.org/10.1145/ 2901739.2903508. W. M. Aly, T. H. A. Soliman, and A. M. AbdelAziz. Cross-domain evaluation of large language models for abstractive text summarization: An empirical perspective. International Journal of Advanced Computer Science

An LLM-based MAS for the Summarization of Software Documents

33

and Applications, 16(6), 2025. doi: 10.14569/IJACSA.2025.0160695. URL http://dx.doi.org/10.14569/IJACSA.2025.0160695. I. Arawjo, C. Swoopes, P. Vaithilingam, M. Wattenberg, and E. L. Glassman. Chainforge: A visual toolkit for prompt engineering and llm hypothesis testing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–18, 2024. S. Chen, X. Xie, B. Yin, Y. Ji, L. Chen, and B. Xu. Stay professional and efficient: Automatically generate titles for your bug reports. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE ’20, page 385–397, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450367684. doi: 10.1145/3324884. 3416538. URL https://doi.org/10.1145/3324884.3416538. K. Cinkusz and J. A. Chudziak. Agile software management with cognitive multi-agent systems. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence (ICAART 2025) - Volume 1, pages 385–392. SCITEPRESS, 2025. ISBN 978-989-758-737-5. doi: 10.5220/0013153000003890. W. Cui, Z. Li, H. Sun, D. Lopez, K. Das, B. Malin, S. Kumar, and J. Zhang. See: Strategic exploration and exploitation for cohesive in-context prompt optimization. arXiv preprint arXiv:2402.11347, 2025. URL https:// arxiv.org/abs/2402.11347. T. T. H. Doan, P. T. Nguyen, J. Di Rocco, and D. Di Ruscio. Too long; didn’t read: Automatic summarization of GitHub README.MD with Transformers. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, EASE ’23, page 267–272, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700446. doi: 10.1145/3593434.3593448. URL https://doi.org/ 10.1145/3593434.3593448. S. Fakhoury, A. Naik, G. Sakkas, S. Chakraborty, and S. K. Lahiri. Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering, 2024. Q. Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 2201–2203, 2023. R. Haldar and J. Hockenmaier. Analyzing the performance of large language models on code summarization. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 995–1008, 2024. J. He, C. Treude, and D. Lo. Llm-based multi-agent systems for software engineering: Literature review, vision and the road ahead. ACM Trans. Softw. Eng. Methodol., 2025. ISSN 1049-331X. doi: 10.1145/3712003. URL https://doi.org/10.1145/3712003. Just Accepted. S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative frame-

34

Duc S. H. Nguyen et al.

work. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id= VtmBAGCN7o. D. Huang, J. M. Zhang, Q. Bu, X. Xie, J. Chen, and H. Cui. Bias testing and mitigation in llm-based code generation. ACM Transactions on Software Engineering and Methodology, 2024. F. Keya, M. Y. Jaradeh, and S. Auer. Leveraging LLMs for Scientific Abstract Summarization: Unearthing the Essence of Research in a Single Sentence. Association for Computing Machinery, New York, NY, USA, 2025. ISBN 9798400710933. URL https://doi-org.univaq.idm.oclc.org/10.1145/ 3677389.3702588. R. Khan, S. Sharma, and D. Upadhyay. Extracting abstractive summaries through generative ai models. In 2025 3rd International Conference on Disruptive Technologies (ICDT), pages 671–676, 2025. doi: 10.1109/ICDT63985.2025.10986332. S. Khanzadeh. Agentmesh: A cooperative multi-agent generative ai framework for software development automation, 2025. URL https://arxiv.org/ abs/2507.19902v1. A. G. Khoee, Y. Yu, R. Feldt, A. Freimanis, P. A. Rhodin, and D. Parthasarathy. Gonogo: An efficient llm-based multi-agent system for streamlining automotive software release decision-making. arXiv preprint arXiv:2408.09785, 2024. URL https://arxiv.org/abs/2408.09785. H. Kim and B.-H. Kim. Nexussum: Hierarchical llm agents for long-form narrative summarization, 2025. URL https://arxiv.org/abs/2505.24575. M. Kong, Z. Wang, Y. Shu, and Z. Dai. Meta-prompt optimization for llmbased sequential decision making. arXiv preprint arXiv:2502.00728, 2025. URL https://arxiv.org/abs/2502.00728. J. Y. Lee, S. Kang, J. Yoon, and S. Yoo. The github recent bugs dataset for evaluating llm-based debugging applications. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 442–444. IEEE, 2024. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for ”mind” exploration of large language model society. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, 2024. Curran Associates Inc. Article 2264, 18 pages. M. Li, D. Li, J. Liu, J. Cao, Y. Tian, and S.-C. Cheung. Enhancing differential testing with llms for testing deep learning libraries. ACM Transactions on Software Engineering and Methodology, 2025a. R. Li, Y. Zhang, X. Zhou, P. Liang, W. Sun, J. Xuan, Z. Jin, and Y. Liu. Maad: Automate software architecture design through knowledge-driven multi-agent collaboration. ACM Transactions on Software Engineering and Methodology, 0(0):Article 0, 2025b. doi: 10.1145/nnnnnnn.nnnnnnn. URL https://arxiv.org/abs/2507.21382v1. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004.

An LLM-based MAS for the Summarization of Software Documents

35

Association for Computational Linguistics. URL https://aclanthology. org/W04-1013. Y. Liu, L. Liu, H. Liu, X. Wang, and H. Yang. Mining domain knowledge from app descriptions. Journal of Systems and Software, 133:126–144, 2017. ISSN 0164-1212. doi: https://doi.org/10.1016/j.jss.2017.08.024. URL https:// www.sciencedirect.com/science/article/pii/S0164121217301784. Y. Liu, E. Noei, and K. Lyons. How readme files are structured in open source java projects. Information and Software Technology, 148:106924, 2022. ISSN 0950-5849. doi: https://doi.org/10.1016/j.infsof.2022.106924. URL https: //www.sciencedirect.com/science/article/pii/S0950584922000775. D. S. H. Nguyen, B. G. Truong, P. T. Nguyen, J. Di Rocco, and D. Di Ruscio. Teamwork makes the dream work: LLMs-Based Agents for GitHub README.MD Summarization. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, FSE Companion ’25, page 621–625, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712760. doi: 10.1145/3696630.3728511. URL https://doi.org/10.1145/3696630.3728511. P. T. Nguyen, J. Di Rocco, C. Di Sipio, R. Rubei, D. Di Ruscio, and M. Di Penta. GPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT. Journal of Systems and Software, 214:112059, 2024. ISSN 0164-1212. doi: https://doi.org/10.1016/j.jss.2024.112059. URL https: //www.sciencedirect.com/science/article/pii/S0164121224001043. I. Ozkaya. Application of large language models to software engineering tasks: Opportunities, risks, and implications. IEEE Softw., 40(3):4–8, 2023. doi: 10.1109/MS.2023.3248401. URL https://doi.org/10.1109/ MS.2023.3248401. L. Pan, Y. Wu, Y. Liu, M. Sun, Y. Gong, Z. Yang, and D. Jiang. Mammrefine: Faithful multi-agent summarization with automatic mistake mining and self-refinement. arXiv preprint arXiv:2503.15272, 2024. URL https: //arxiv.org/abs/2503.15272. Z. Rasheed, A. M. Sami, K.-K. Kemell, M. Waseem, M. Saari, K. Systä, and P. Abrahamsson. Codepori: Large-scale system for autonomous software development using multi-agent technology. Information and Software Technology, 2024. URL https://arxiv.org/abs/2402.01411. Preprint, arXiv:2402.01411. M. Shen, R. Shu, A. Pratik, J. Gung, Y. Ge, M. Sunkara, and Y. Zhang. Optimizing llm-based multi-agent system with textual feedback: A case study on software development. In AI Agents: Capabilities and Safety Workshop @ Conference on Language Modeling (COLM), 2025. URL https://arxiv.org/abs/2505.16086. W. Sun, Y. Miao, Y. Li, H. Zhang, C. Fang, Y. Liu, G. Deng, Y. Liu, and Z. Chen. Source code summarization in the era of large language models. arXiv preprint arXiv:2407.07959, 2024. S. Takeshita, S. P. Ponzetto, and K. Eckert. IRSum: One model to rule summarization and retrieval. In O. Arviv, M. Clinciu, K. Dhole, R. Dror, S. Gehrmann, E. Habba, I. Itzhak, S. Mille, Y. Perlitz, E. Santus, J. Se-

36

Duc S. H. Nguyen et al.

doc, M. Shmueli Scheuer, G. Stanovsky, and O. Tafjord, editors, Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 262–275, Vienna, Austria and virtual meeting, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-261-9. URL https://aclanthology.org/2025.gem-1.23/. W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang, and Y. Cheng. Magis: Llmbased multi-agent framework for github issue resolution. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2024. R. Tian, Y. Ye, Y. Qin, X. Cong, Y. Lin, Y. Pan, Y. Wu, H. Haotian, L. Weichuan, Z. Liu, et al. Debugbench: Evaluating debugging capability of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 4173–4198, 2024. H. Wang, X. Du, W. Yu, Q. Chen, K. Zhu, Z. Chu, L. Yan, and Y. Guan. Apollo’s oracle: Retrieval-augmented reasoning in multi-agent debates. arXiv preprint arXiv:2312.04854, 2023. URL https://arxiv.org/abs/ 2312.04854. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6): 186345, 2024a. doi: 10.1007/S11704-024-40231-1. URL https://doi.org/ 10.1007/s11704-024-40231-1. L. Wang, Y. Zhou, H. Zhuang, Q. Li, D. Cui, Y. Zhao, and L. Wang. Unity is strength: Collaborative llm-based agents for code reviewer recommendation. In Proceedings of the 39th IEEE/ACM ASE, ASE ’24, page 2235–2239, New York, NY, USA, 2024b. ACM. ISBN 9798400712487. doi: 10.1145/3691620. 3695291. URL https://doi.org/10.1145/3691620.3695291. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt. In Proceedings of the 30th Conference on Pattern Languages of Programs, PLoP ’23, USA, 2023. The Hillside Group. ISBN 9781941652190. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang. Autogen: Enabling next-gen llm applications via multi-agent conversations. In Proceedings of the First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=BAakY1hNKS. C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying llm-based software engineering agents. CoRR, 2024. doi: 10.48550/ARXIV.2407.01489. URL https://doi.org/10.48550/arXiv.2407.01489. C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen. Large language models as optimizers. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), Vienna, Austria, 2024. URL https://openreview.net/forum?id=Bb4VGOWELI. J. Zhang, Z. Wang, H. Zhu, J. Liu, Q. Lin, and E. Cambria. Mars: A multiagent framework incorporating socratic guidance for automated prompt op-

An LLM-based MAS for the Summarization of Software Documents

37

timization, 2025. URL https://arxiv.org/abs/2503.16874. L. Zhang and K. D. Ashley. Mitigating manipulation and enhancing persuasion: A reflective multi-agent approach for legal argument generation. In Proceedings of the Workshop on Legally Compliant Intelligent Chatbots at ICAIL 2025, Chicago, IL, USA, 2025. URL https://arxiv.org/abs/ 2506.02992. T. Zhang, I. C. Irsan, F. Thung, D. Han, D. Lo, and L. Jiang. itiger: An automatic issue title generation tool. ESEC/FSE 2022, page 1637–1641, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450394130. doi: 10.1145/3540250.3558934. URL https://doi.org/ 10.1145/3540250.3558934. Y. Zhang, R. Sun, Y. Chen, T. Pfister, R. Zhang, and S. O. Arik. Chain of agents: Large language models collaborating on long-context tasks. arXiv preprint arXiv:2406.02818, 2024. URL https://arxiv.org/abs/2406. 02818. Preprint. Under review. Work done in part at Google Cloud AI Research. H. Zhou, X. Wan, R. Sun, H. Palangi, S. Iqbal, I. Vulić, A. Korhonen, and S. Ö. Arık. Multi-agent design: Optimizing agents with better prompts and topologies. arXiv preprint arXiv:2502.02533, February 2025a. URL https://arxiv.org/abs/2502.02533. Preprint submitted to arXiv. X. Zhou, H. Huang, and L. Liao. Debate, reflect, and distill: Multi-agent feedback with tree-structured preference optimization for efficient language model enhancement. arXiv preprint arXiv:2506.03541, 2025b. URL https: //arxiv.org/abs/2506.03541. Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models are human-level prompt engineers. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), Kigali, Rwanda, 2023. URL https://openreview.net/forum?id= 92gvk82DE-. Z. Zhou, F. Wu, S. Talaei, H. Zhao, C. Meixin, T. Xu, A. Saberi, and Y. Choi. When to trust context: Self-reflective debates for context reliability. arXiv preprint arXiv:2506.06020, 2025c. URL https://arxiv.org/abs/2506. 06020.