arXiv CSopen access

Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation

arXiv:2605.14563v1 [cs.SE] 14 May 2026

Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation

Suyoung Bae1 , Jaehoon Lee1 , Changkyu Choi2 YunSeok Choi1∗, Jee-Hyong Lee1∗ 1 Sungkyunkwan University, South Korea 2 University of Oslo, Norway

Abstract Automated code documentation is essential for modern software development, providing the contextual grounding that both human developers and coding agents rely on to navigate large codebases. Existing repository-level approaches process components independently, causing redundant retrieval and conflicting descriptions across documents while producing outputs that lack hierarchical structure. Therefore, we propose MemDocAgent, a long-horizon agentic framework that generates documentation within a single, integrated context spanning the entire repository. It combines two components: (i) Dependency-Aware Traversal Guiding that predetermines a traversal order respecting dependency and granularity hierarchies; (ii) Memory-Guided Agentic Interaction, in which the agent interacts with RepoMemory, a shared memory accumulating prior work traces through read, write, and verify operations. Through an in-depth multi-criteria evaluation, MemDocAgent achieves the best performance over both open and closed-source baselines and demonstrates practical applicability in real software development workflows. 2

1

Introduction

Repository-level documentation requires understanding over an entire codebase to generate coherent natural-language descriptions that span individual function behaviors, dependency structures, and overall architectural patterns [1–3]. As software repositories scale in size and complexity, such documentation serves as a critical interface for both human developers and coding agents, providing the contextual grounding needed for code navigation and collaboration [4–6]. Manual writing at this scale is prohibitively expensive, motivating a new line of work on automating repository-level documentation [7–12]. Automating this task is inherently a long-horizon: each document depends not only on the local code being summarized, but also on information retrieved, interpreted, and written in earlier documentation steps. However, existing repository-level documentation systems, ranging from open-source frameworks [1–3] to closed-source products [13, 14], decompose a repository into components and document each component independently, treating each as a self-contained task without shared state across the repository. As a result, the same source files are repeatedly retrieved across iterations, with an average overlap of 50% across existing systems as shown in Figure 1. More critically, prior documentation outputs are not reused across iterations, leaving the system unable to detect dependency conflicts. This leads to contradictory descriptions of the same entity or architectural logic across documents, resulting in an average cross-document inconsistency rate of 13% across existing systems (as shown in Figure 1). ∗ Corresponding authors 2 Code and Data are available at https://github.com/bsy99615/MemDocAgent.

Preprint.

Redundant Source-File Retrieval Rate (%)

Cross-Document Inconsistency Rate (%)

Rate

Closed-source 70% 60% 50% 40% 30% 20% 10% 0%

Lower is better

Open-source

64.9%

52.1% 31.5% 17.3%

13.4%

15.3%

8.3%

8.9%

0.0% 3.1% MemDocAgent Claude-Code DeepWiki Prompting RepoAgent DocAgent Figure 1: Limitations in existing documentation systems. While existing systems suffer from substantial repeated source-file retrieval and cross-document inconsistencies, MemDocAgent eliminates repeated retrieval and reduces inconsistency by 75.5%. N/A denotes closed-source systems for which repeated source-file retrieval cannot be measured because their internal pipelines are inaccessible. N/A

N/A

Beyond these procedural limitations, existing methods lack hierarchical structure in their documentation outputs. High-quality repository documentation should provide a multi-level view of a codebase, connecting fine-grained component behavior, module-level responsibilities, and repository-wide architectural design into a coherent whole [15]. However, existing methods typically document only a narrow slice of this hierarchy: component-level approaches generate localized docstrings but overlook architectural context [1, 3], whereas repository-level approaches summarize global structure but omit fine-grained implementation details [2]. Consequently, the resulting documentation is either too local to explain system-level design or too coarse to support concrete code understanding. This hierarchical gap reinforces the need for a long-horizon documentation process that can accumulate, preserve, and connect information across many interdependent documentation steps. Existing systems treat repository documentation as short-horizon, component-local tasks, whereas the task requires accumulating and reconciling repository-wide knowledge across many documentation steps. We therefore formulate repository-level documentation as a long-horizon agentic problem [16], where a single agent maintains persistent state, reuses prior retrievals and outputs, and builds a coherent hierarchical documentation of the codebase. To this end, we propose MemDocAgent, a long-horizon agentic framework in which a single agent reasons over the entire repository within one continuous trajectory, with persistent access to its accumulated retrievals and outputs. Along this trajectory, the agent produces hierarchical documentation spanning the component, module, and repository levels. This design is supported by two core components: Dependency-Aware Traversal Guiding and Memory-Guided Agentic Interaction. Dependency-Aware Traversal Guiding predetermines a traversal order that respects dependency relations and the granularity hierarchy, and provides it as guidance to the agent to ensure hierarchical structure and coverage completeness. In Memory-Guided Agentic Interaction, the agent interacts with RepoMemory, a shared memory that accumulates prior work traces throughout the trajectory. These traces include retrieved components, intermediate reasoning, and generated documentation, and are accessed through agent’s actions to improve efficiency and cross-document consistency. To validate the effectiveness of MemDocAgent, we evaluate it against existing baselines along four dimensions: completeness, truthfulness, helpfulness, and information sufficiency. Existing metrics capture coverage, accuracy, and usefulness, but overlook implementation-level sufficiency for practical code understanding. We therefore introduce information sufficiency as a quantitative metric that measures whether the generated documentation alone provides enough information to reproduce the original source code. Empirically, MemDocAgent achieves the best performance across all four criteria, demonstrating that its documentation captures both the breadth and the depth required to support real software development workflows.

2

Related work

2.1

Long-horizon coding agents

Recent coding agents have advanced from solving isolated programming tasks to executing extended trajectories with numerous sequential tool calls and reasoning steps [17–25]. This long-horizon regime builds on agentic foundations such as ReAct [26], Reflexion [27], and multi-agent frame2

works [28–30]. A key challenge is that working memory accumulates across turns and exceeds the context window, causing agents to lose track of prior objectives or propagate errors throughout the trajectory [31–33]. To handle long contexts, prior work has explored external memory systems that maintain and dynamically organize context outside the model [34–40], along with summarization and compression based techniques for context management [41–43]. However, these methods are designed for single output settings where memory supports a single final answer, and fall short when multiple sequentially generated outputs must remain mutually consistent. 2.2

Repository-level code documentation

Early code documentation methods focused on function-level summarization, training models to produce short natural language descriptions for individual code units [44–48]. With the emergence of LLMs, this scope expanded to summarization at the class and project levels [49–55], and recent systems have begun to operate over entire repositories. RepoAgent [3] extracts dependency relations and generates component-level docstrings in topological order through an LLM pipeline, while DocAgent [1] adopts a multi-agent pipeline for the same component-level objective. CodeWiki [2], in contrast, hierarchically decomposes a repository and recursively synthesizes documentation via a multi-agent pipeline targeting repository-level abstractions. While these methods successfully extend documentation generation to the repository scale, they treat each component as an independent unit, thereby failing to maintain cross-document consistency or dependency-aware reasoning across the long-horizon documentation trajectory. This per-component processing leads to redundant retrievals and inter-document contradictions that undermine the practical utility of the resulting documentation.

3

Preliminaries

Repository-level hierarchical documentation. Given a repository R with a hierarchical directory structure and multiple source files, our task is to produce a hierarchical documentation set D over three granularities: components (functions, methods, classes), modules (directories), and the repository itself. The corresponding documentation units are: (c)

(m) Nm }j=1

c U = {ui }N i=1 ∪ {uj

∪ {u(r) },

(1)

Where Nc and Nm are the numbers of components and modules, and the superscripts (c), (m), (r) indicate the granularity level, respectively. The system processes every u ∈ U to generate a corresponding document d ∈ D, yielding |D| = Nc + Nm + 1 documents in total. Long-horizon reasoning trajectories. We model the documentation generation process as a longhorizon reasoning trajectory, formally defined as τ = (t0 , a0 , o0 , t1 , a1 , o1 , . . . , tT , aT , oT ), where each turn i comprises a thought ti , an action ai , and an observation oi following the ReAct pattern [26], and T denotes the total number of turns in the trajectory. In repository-level documentation generation, a single trajectory typically spans over a thousand turns as the agent processes hundreds of components, dozens of modules, and the repository itself.

4

MemDocAgent

4.1

Overview

MemDocAgent frames repository-level documentation as a cumulative long-horizon process, where a single agent processes every documentation unit within one continuous trajectory and reuses information accumulated along the way. This design is supported by two key components. First, dependency-aware traversal (§4.2) that orders the trajectory so that each unit is documented after its dependencies and child units, ensuring both context-grounded generation and repository coverage. Second, memory-guided agentic interaction (§4.3), where the agent interacts with a shared external memory to retrieve, reuse, and update information across sub-tasks. Figure 2 illustrates our framework. 3

<Long-Horizon Workflow>

Repository (𝑈) datasets evaluation eval.py

parser.py

Subtask 1 (𝑢1 )

Action Space

[Observation] Continue generating the next sub task.

READ

[Thought] Let me analyze what I need to find out.

[Action] READ <FUNC>utils.ensure_tuple, </FUNC> […] Subtask 2 (𝑢2 )

Memory Search

[Observation] [retrieved context]

…

utils.py

RepoMemory

Miss

Graph Traverse

Committed Docs

API calls

𝑀 = {𝑑1 , 𝑑2 , … , 𝑑𝑖−1 }

[Thought] Let me write the documentation

models

…

[Action] WRITE <DOC>[draft documentation]</DOC> [Observation] Draft stored. Proceed to Verify Repository Module Component Depends on

WRITE Document Generation

[Thought] Now I need to verify this documentation, […]

Subtask 𝑖 (𝑢𝑖 )

Dependency Flow Graph

Save

Draft 𝑑෡𝑖

Check

[Action] WRITE <DOC>[draft documentation]</DOC>

Traversal Ordering

[Observation] Draft stored. Proceed to Verify

Final 𝑑𝑖

[Action] FINISH

module1.md

comp1.md

0.92 0.94 0.91 Verify score

comp3.md module2.md comp1.md module3.md comp1.md

External Store API calls / Web retrieval

Refresh working context

[Observation] Continue generating the next sub task.

Move to next subtask

…

(A) Dependency-Aware Traversal Guiding

Subtask 𝑛−1 [Thought] The documentation now passed […] (𝑢𝑛−1 )

Subtask 𝑛 (𝑢𝑛 )

2. Self-Evaluation

FINISH

…

Ordered Subtasks

0.94 0.91 Verify score

Component Store 𝑀𝑐

[Thought] […] Let me revise to be more precise […]

repo.md

comp2.md

Module Store 𝑀𝑚

1. Cross Doc Conflict Check

[Observation] (final 0.82, threshold 0.9) Conflict with […] Therefore, Revise the draft.

𝜋: 𝑈 → (𝑢1 , 𝑢2 , … , 𝑢𝑛 )

Subtask 1 (𝑢 1) ) Subtask 2 (𝑢 2 Subtask 𝑛 (𝑢𝑛 )

0.95 0.98 Verify score

VERIFY

[Action] VERIFY

Repository Store 𝑀𝑟

Hierarchical Documentation

(B) Memory-Guided Agentic Interaction

Figure 2: Overview of MemDocAgent. (A) Dependency-aware traversal guiding first computes a traversal order that respects both dependency relations and the granularity hierarchy. Following this order, (B) memory-guided agentic interaction treats each unit as a multi-turn sub-task, where the agent interacts with RepoMemory through R EAD, W RITE, and V ERIFY to generate di and commits the verified document upon F INISH. The local context is refreshed before moving to the next sub-task, and this loop repeats until all units are documented, producing the final hierarchical documentation. Table 1: RepoMemory structure. Memory store Documentation store External store

4.2

component_store module_store repo_store search_cache

Key

Value

component_id module_path repo_name query

path, document, claims, depends_on, source code, type, verification_score path, document, claims, child_units, verification_score path, document, claims, child_units, verification_score search result

Dependency-aware traversal guiding

To structurally guarantee the completeness of D, this module predetermines a traversal order π = (u1 , . . . , uN ) over U and injects it into the agent as sequential sub-tasks. We first construct a directed dependency graph G = (V, E) over R, where each node v ∈ V corresponds to a documentation unit and a directed edge (u, v) ∈ E indicates that u depends on v. Unlike DocAgent [1], which uses a topological order reflecting only component-level dependencies, π jointly satisfies two constraints: (1) component-level dependency order, where ui precedes uj whenever uj depends on ui ; and (2) granularity-level hierarchical order, where each module is processed only after all of its components and sub-modules are documented, and the repository unit is processed only after all modules are complete. By jointly satisfying both, π ensures that the agent always processes a unit with all of its dependent and child units already documented. Detailed graph construction and the traversal order algorithm are provided in Appendix A.1. 4.3

Memory-guided agentic interaction

To ensure both retrieval efficiency and cross-document consistency throughout the long-horizon trajectory, this section introduces a centralized shared memory, RepoMemory, and designs an interaction mechanism in which every action of the agent interacts with this memory. RepoMemory. As shown in Table 1, RepoMemory consists of a Documentation store and an External store. The documentation store organizes generated documents by hierarchy level, storing component, module, and repository-level records with their identifiers, paths, documents, decomposed claims, verification scores, source code and type (function, method, class) for components, child units for module and repository units. The external store caches search results by query to avoid redundant retrievals during the trajectory. 4

Action space. Unlike prior repository-level documentation systems whose actions are executed independently, every action in MemDocAgent is grounded in interaction with RepoMemory: R EAD retrieves context from RepoMemory, W RITE produces a document using the retrieved context, V ERIFY cross-validates the generation against documents already committed to RepoMemory, and F INISH commits the verified document to RepoMemory. At each turn, the agent observes the result of its previous action and autonomously selects the next action among these four operations to complete the current documentation sub-task. All instructions for each action are defined in the agent’s system prompt (Appendix A.2 for the full system prompt and Appendix A.4 for trajectory examples). READ: Memory-guided adaptive retrieval. Through R EAD, the agent adaptively decides whether additional context is needed for each sub-task ui . If so, it calls the retrieval tool with two types of structured requests: internal requests that specify related component identifiers within the repository, and external requests that pose natural-language queries about external algorithms or libraries. The retrieval tool first checks RepoMemory by key: component IDs for internal requests and query strings for external requests. If the key exists, the stored value is returned at O(1) cost; otherwise, the tool retrieves the information from the codebase via the dependency graph G or from the external retrieval API, and stores it for later reuse. For module and repository-level sub-tasks, the retrieval tool additionally retrieves the documentation of all direct child units from the Documentation Store. Since the traversal order π guarantees that these children have already been documented, this step reuses existing documentation without additional codebase retrieval. WRITE: Hierarchical document generation. When the agent thinks that sufficient context has been collected, it performs W RITE to generate a draft document dˆi . The generation is guided by a granularity-specific format, defined in the agent’s system prompt, that specifies the structure and required content of the output for each granularity. VERIFY: Self-evaluation and cross-document conflict verification. Repository-level documentation often describes the same functionality from multiple perspectives across files, so without verification these descriptions easily drift apart and degrade overall consistency. To address this, V ER IFY evaluates dˆi in two ways. First, it performs self-evaluation on factual consistency, completeness, and helpfulness, with each score assigned on [0, 1]. Second, it applies a cross-conflict verification tool to detect inconsistencies across dependency-related documents. Inspired by NLI-based inconsistency detection for text summarization [56–58], we verify crossdocument conflicts by comparing the current draft dˆi against documents already committed to RepoMemory, which are guaranteed to have passed V ERIFY and thus serve as reliable references. The verifier decomposes dˆi into atomic claims and uses a local NLI model to check each claim against the verified documents of units that have a dependency relation with ui . The resulting contradiction rate serves as the cross-conflict score. Details of the verification process are provided in Appendix A.3. dˆi passes V ERIFY only if the average of two verified scores exceeds a threshold. Otherwise, the conflicts detected during this stage are returned as a conflict report in the next turn’s observation, prompting the agent to either determine an additional R EAD or re-W RITE the draft. FINISH: Memory commit and trajectory continuation. Once dˆi passes V ERIFY, the agent finalizes it as di , invokes F INISH to commit di to the corresponding granularity store, and refreshes its local context for the next sub-task. This loop repeats until all units in U are documented, yielding the final hierarchical documentation set D.

5

Experimental setups

5.1

Evaluation metrics

We evaluate MemDocAgent along four criteria. Beyond completeness, truthfulness, and helpfulness, which are established quality indicators, we further propose information sufficiency. This criterion quantifies whether the documentation contains enough information to reproduce source code that integrates and executes coherently within the repository. We outline the evaluation metrics below; full details are in Appendix B.1. 5

Completeness. Completeness measures the extent to which the generated documentation adheres to standard structural conventions (Section Presence) and includes essential components (e.g., function, class) expected for a given code element (Entity Coverage). The final completeness score is computed as the average of the section presence and entity coverage scores. Section Presence Score. For each documentation, we measure the proportion of required sections that are present, using rule-based pattern matching. Entity Coverage Score. We extract the set of core entities from the target source code via abstract syntax tree (AST) analysis, and measure the proportion of these entities that are referenced in the documentation. The core entities are defined according to granularity. Helpfulness. We assess the semantic quality and practical utility of each documentation through an LLM-as-a-judge protocol following DocAgent [1]. The judge rates each required section on a 5-point Likert scale using a section-specific rubric, and the normalized section scores are averaged into a single helpfulness score. Truthfulness. Truthfulness measures whether the generated documentation faithfully reflects the actual code, covering both factual correctness (Consistency) and contextual relevance (Relevance). We adopt a fine-grained documentation evaluation framework that decomposes generated documentation and evaluates each segment using predefined evaluation criteria [59, 60] to compute these two scores. The final truthfulness score is computed as the product of the consistency and relevance scores. Consistency Score. We measure the proportion of claims in the documentation that are factually supported by the source code, identified at fine-grained segment level. Relevance Score. We measure the proportion of documentation content that pertains to the source code, filtering out off-topic or extraneous descriptions. Information Sufficiency. Beyond surface-level documentation quality, we also evaluate whether the documentation contains sufficient information to support practical code reconstruction. We operationalize this by measuring code regeneration performance: if a model can regenerate the original component using only the documentation and the function signature, the documentation is likely to preserve the behavior, dependencies, and implementation details needed for downstream use. To evaluate information sufficiency, we collect 564 component-level test cases from DevEval [61]. The evaluation proceeds as follows: (1) remove each target function body while keeping its signature; (2) provide the signature and documentation context to a code-generation model; (3) regenerate the missing function body; and (4) reinsert the generated code into the repository and evaluate it with functional correctness (Pass@k) and structural similarity (CodeBLEU). 5.2

Baselines

We compare MemDocAgent against four open-source and two closed-source systems. Details are in the Appendix B.2. Open-source baselines. RepoAgent [3], DocAgent [1], and CodeWiki [2] are agent-based systems for repository-level documentation generation. We additionally include Prompting, a baseline that prompts the LLM to document each target unit independently, with first-hop dependency context. Closed-source baselines. DeepWiki [13] is a commercial system widely used in industry for automated repository-level documentation. We also include Claude-Code [14], treating the README.md produced by its /init command as the repository-level documentation. 5.3

Implementation details

To evaluate the effectiveness of MemDocAgent on repository-level documentation generation, we select 20 Python repositories from DevEval [61] via stratified sampling, applying three criteria: (i) balanced coverage across the 10 domains in DevEval, (ii) the presence of unit tests to enable information sufficiency evaluation, and (iii) substantial structural depth and dependency complexity at the repository level. Detailed statistics are provided in Appendix B.3. 6

Table 2: Main results on documentation quality: (a) Overall results on average completeness, truthfulness, and helpfulness and (b) the same metrics separated by documentation granularity. Open-source baselines use two backbones, Qwen3-Coder and GPT-5-mini. Within each backbone group, the best result is bold and the second-best underlined. * marks in section scores set to 1.0 by default, as their internal format instruction is inaccessible. “–” denotes non-applicable granularities. (a) Overall results on average completeness, truthfulness, and helpfulness. Backbone

Method

Completeness

#Doc Section Score

Coverage Score

Truthfulness

Complete.

Consistency

Relevance

Truthful.

Helpfulness

Open-Source Baselines

Qwen3-Coder

Prompting RepoAgent DocAgent CodeWiki MemDocAgent

3,323 3,650 3,198 80 3,323

0.838 0.659 0.792 0.770 0.959

0.837 0.967 0.898 0.473 0.998

0.838 0.813 0.845 0.622 0.979

0.809 0.910 0.900 0.906 0.968

0.803 0.918 0.889 0.852 0.946

0.649 0.835 0.800 0.772 0.916

0.458 0.391 0.556 0.628 0.690

GPT-5-mini

Prompting RepoAgent DocAgent CodeWiki MemDocAgent

3,323 3,650 3,198 80 3,323

0.790 0.555 0.776 0.975 0.916

0.473 1.000 0.944 0.579 1.000

0.632 0.778 0.860 0.777 0.958

0.883 0.898 0.898 0.847 0.983

0.840 0.947 0.925 0.771 0.968

0.741 0.850 0.831 0.653 0.952

0.617 0.449 0.671 0.708 0.800

20 404

1.000* 1.000*

0.845 0.815

0.869 0.935

0.734 0.762

0.696 0.755

Closed-Source Baselines Claude-Code (/init) DeepWiki

0.488 0.840

0.744 0.920

(b) Results separated by documentation granularity (component, module, and repository). Backbone

Method

Component-Level Complete.

Truthful.

Module-Level

Helpful.

Complete.

Repository-Level

Truthful.

Helpful.

Complete.

Truthful.

Helpful.

Open-Source Baselines

Qwen3-Coder

Prompting RepoAgent DocAgent CodeWiki MemDocAgent

0.886 0.813 0.845 – 0.987

0.694 0.835 0.800 – 0.962

0.577 0.391 0.556 – 0.691

0.870 – – 0.630 0.975

0.666 – – 0.769 0.902

0.313 – – 0.606 0.702

0.758 – – 0.613 0.975

0.587 – – 0.775 0.883

0.483 – – 0.650 0.677

GPT-5-mini

Prompting RepoAgent DocAgent CodeWiki MemDocAgent

0.631 0.778 0.860 – 0.952

0.761 0.850 0.831 – 0.974

0.627 0.449 0.671 – 0.750

0.626 – – 0.809 0.974

0.766 – – 0.597 0.924

0.626 – – 0.737 0.858

0.640 – – 0.744 0.949

0.696 – – 0.708 0.959

0.598 – – 0.679 0.791

We implement MemDocAgent with two backbone LLMs: Qwen3-Coder-30B-A3B (open-source from HuggingFace) and GPT-5-mini (closed-source from OpenAI). Each repository is documented with a maximum of 10 steps per sub-trajectory, up to 2 revisions per unit, and a verify threshold of 0.9. For evaluation, we use Claude Haiku 4.5 as the LLM judge for truthfulness and helpfulness, and GPT-4o-mini and Claude Haiku 4.5 as code generators for evaluating information sufficiency. To mitigate self-preference bias, we deliberately use evaluator models that differ from any backbone used by MemDocAgent or the baselines. More implementation details are in Appendix B.4.

6

Experimental results

6.1

Comparisons with baselines

Documentation quality comparisons. Table 2 presents a performance comparison between MemDocAgent and existing baselines. Note that the number of generated documents (#Doc) differs because baselines use different documentation granularities and targets. Table 2a shows that MemDocAgent achieves the best overall performance across both open-source backbones. With Qwen3-Coder, it improves completeness, truthfulness, and helpfulness to 0.979, 0.916, and 0.690, outperforming the strongest baselines by 15.9%, 9.7%, and 9.9%, respectively; with GPT-5-mini, it further raises truthfulness and helpfulness to 0.952 and 0.800 while maintaining high completeness at 0.958. Moreover, existing baselines exhibit clear granularity trade-offs. RepoAgent attains high completeness but lower helpfulness, whereas CodeWiki produces concise repositorylevel documents but misses fine-grained details. In contrast, MemDocAgent generates hierarchical 7

Pass@1

0.6

0.46 0.46

0.21

0.44

0.1 0.0

Qwen3-Coder-30B

GPT-5-mini Prompting

★

★

0.5

0.42

0.3 0.26

0.26

0.23

0.47

0.47 0.47

0.42

0.4

0.33 0.35

0.29

0.3 0.2

0.42

0.40

0.4

Pass@3

0.6

★

★

0.5

0.46

0.27

0.24

0.3 0.2

0.1

0.1

Qwen3-Coder-30B

RepoAgent

MemDocAgent (C)

GPT-5-mini

DocAgent

MemDocAgent (C+M)

0.0

★

★

0.5

0.2

0.0

CodeBLEU

0.6

0.4

0.35 0.36

0.30

0.49

0.33 0.31

0.37

0.41 0.42

0.38

0.42 0.43 0.43 0.35

0.27 0.19

Qwen3-Coder-30B

No docs

GPT-5-mini

Ground-truth

MemDocAgent (C+M+R)

Best score

Figure 3: Pass@1, Pass@3 and CodeBLEU when regenerating code from each method’s documentation alone, averaged over GPT-4o-mini and Claude Haiku 4.5, with the x-axis denoting the backbone used for each open-source baseline. Table 3: Ablation study on key components of MemDocAgent. All ablations are conducted under the Qwen3-Coder backbone. Values in parentheses indicate the drop in percentage points relative to ours. Documentation Quality

Information Sufficiency

Method

Completeness

Truthfulness

Helpfulness

Pass@1

Pass@3

CodeBLEU

MemDocAgent w/o think w/o memory system w/o conflict verify

0.979 0.917 (−6.2) 0.903 (−7.6) 0.900 (−7.9)

0.916 0.777 (−13.9) 0.768 (−14.8) 0.781 (−13.5)

0.690 0.583 (−10.7) 0.593 (−9.7) 0.592 (−9.8)

0.461 0.312 (−14.9) 0.324 (−13.7) 0.282 (−17.9)

0.467 0.319 (−14.8) 0.333 (−13.4) 0.290 (−17.7)

0.419 0.225 (−19.4) 0.190 (−22.9) 0.233 (−18.6)

documentation with shared memory, improving coverage, consistency, and utility over both open and closed-source systems. Second, MemDocAgent’s improvements generalize across documentation granularities. As shown in Table 2b, MemDocAgent achieves the best performance at the component, module, and repository levels on both backbones. While existing baselines cover only parts of this hierarchy, MemDocAgent consistently maintains high completeness, truthfulness, and helpfulness across all levels, demonstrating the benefit of long-horizon memory-guided documentation. Per-repository results and further analysis are provided in Appendix C.1, C.2 and C.5. Information sufficiency comparisons. We measure whether documentation alone provides sufficient information to reproduce the original code by feeding the documentation as the only context to two generation models (GPT-4o-mini and Claude Haiku 4.5) and evaluating Pass@1, Pass@3, and CodeBLEU against ground-truth code. To examine whether progressively richer hierarchical context helps the model regenerate code that integrates coherently and executes correctly within the repository, we evaluate three MemDocAgent variants that progressively widen the supplied context: (C) the target component document only; (C+M) the component with its direct parent module document; and (C+M+R) the component with parent module and repository document. We additionally include two reference settings: No docs (signature only) and Ground-truth (signature with the human-written component docstring obtained from DevEval [61]). Figure 3 shows the evaluation result (see Appendix C.3 for full result). Across both backbones, every existing baseline surpasses the No docs setting, confirming that automatically generated documentation provides useful signal for code reproduction. Among them, MemDocAgent achieves the strongest performance across all settings. Generated documentation can score higher than the Ground-truth reference because human-written docstrings are intended as short and concise summaries, whereas the baselines and our method produce richer descriptive documentation. Compared among three MemDocAgent variants, performance increases monotonically as granularity levels are added, with (C+M+R) attaining the best score in every setting. That higher-level documents consistently help indicates that our framework produces cross-level consistent documentation whose repository-wide context and architectural information enable the model to regenerate code that executes correctly within the repository. 6.2

Ablation study

We conduct an ablation study to assess the contribution of each proposed component. Specifically, w/o think removes the agent’s think step before each action, w/o memory system replaces RepoMemory 8

with a stateless cache that is cleared after each documentation unit, and w/o conflict verify disables cross-document conflict verification. Table 3 shows that removing any single component causes substantial drops across all metrics, confirming that every component is essential. The three components affect different aspects of documentation. Together, ReAct-style reasoning, RepoMemory, and conflict verification each contribute a complementary dimension of quality, and only their combination yields MemDocAgent’s full hierarchical documentation capability. 6.3

Analysis of efficiency

70

Total: 60.86s

Read time

Time per document (s)

60 Other time As shown in Figure 4, the effect of eliminating redundant sourceTotal: 50.51s 50 file retrieval is consistently observed in retrieval cost. DocAgent 37.42 performs on average 23.44 seconds, whereas MemDocAgent com40 36.75 pletes the same task in 13.76 seconds, reducing read time by 41%. 30 These reductions hold consistently across all 20 repositories used 20 23.44 in our evaluation (Appendix C.4), demonstrating that RepoMem10 13.76 ory preserves the documentation information accumulated along the 0 DocAgent MemDocAgent long-horizon trajectory in an efficiently retrievable form, thereby structurally improving the retrieval efficiency of hierarchical docu- Figure 4: Time comparison bementation without compromising quality. tween DocAgent and MemDocAgent. 6.4 Effectiveness of verification

Self verify score

Verification score

Cross document conflict verify score Figure 5 shows how self-evaluation and cross-document conflict scores change over repeated V ERIFY attempts. 63.8% of components 0.98 1.00 0.97 0.98 pass on the first attempt with an average score of approximately 0.97, 0.95 0.95 0.96 0.95 while the remaining 24.4% and 11.3% are resolved within two and 0.90 Threshold = 0.90 0.86 three attempts, respectively, with documentation quality improving 0.85 0.86 0.83 at each additional attempt. When a component receives a low score 0.82 0.80 at the initial verification, MemDocAgent autonomously refines it, 0.79 0.75 leading to higher self-evaluation and cross-document consistency 0.70 0.72 scores in subsequent attempts and thereby ensuring accurate and con0.65 sistent documentation. These results demonstrate that the V ERIFY Verify 1 Verify 2 Verify 3 action effectively improves documentation quality through targeted re-verification, applied only to components with detected conflicts Figure 5: Verification scores or self-evaluation failures. across V ERIFY attempts.

7

Conclusion

We introduced MemDocAgent, a long-horizon agentic framework for repository-level documentation in which a single agent reasons over the entire repository by combining dependency-aware traversal guiding and memory-guided agentic interaction. MemDocAgent preserves repository-wide context across hierarchical documentation steps, enabling consistent documentation from component-level implementation details to repository-level architecture. Across evaluations, MemDocAgent outperformed existing baselines, improving completeness, truthfulness, helpfulness, and information sufficiency. These results demonstrated the ability of our framework to produce documentation that is more useful for developers and coding agents in real software development workflows.

Limitation Long-horizon agentic documentation task requires computational resources because both input context and generated outputs can grow large over the trajectory. In addition, because the agent depends on the model’s reasoning ability, it may occasionally enter a repeated loop without completing the sub-task. Although the overall impact is minimal, with only 0.08% of components terminating without meeting the verification threshold in our experiments, this limitation should still be considered. Future work should explore more efficient trajectory control, adaptive stopping criteria, and stronger recovery mechanisms.

9

References [1] Dayu Yang, Antoine Simoulin, Xin Qian, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, and Grey Yang. DocAgent: A multi-agent system for automated code documentation generation. In Pushkar Mishra, Smaranda Muresan, and Tao Yu, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 3: System Demonstrations), pages 460–471. Association for Computational Linguistics, 2025. [2] Anh Nguyen Hoang, Minh Le-Anh, Bach Le, and Nghi DQ Bui. Codewiki: Evaluating ai’s ability to generate holistic documentation for large-scale codebases. arXiv preprint arXiv:2510.24428, 2025. [3] Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. RepoAgent: An LLM-powered open-source framework for repository-level code documentation generation. In Delia Irazu Hernandez Farias, Tom Hope, and Manling Li, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pages 436–464. Association for Computational Linguistics, 2024. [4] David Lorge Parnas. Precise documentation: The key to better software. In The Future of Software Engineering, 2010. URL https://api.semanticscholar.org/CorpusID:38934599. [5] Golara Garousi, Vahid Garousi-Yusifoğlu, Guenther Ruhe, Junji Zhi, Mahmoud Moussavi, and Brian Smith. Usage and usefulness of technical software documentation: An industrial case study. Information and Software Technology, 57:664–682, 2015. ISSN 0950-5849. doi: https://doi.org/10.1016/j.infsof.2014.08.003. URL https://www.sciencedirect.com/ science/article/pii/S095058491400192X. [6] Lea Katalina Kivinen. Ai-driven chatbot as a support tool for developers during the onboarding process. 2023. [7] David Lorge Parnas. Software engineering (extended abstract) an unconsummated marriage. ACM SIGSOFT Software Engineering Notes, 22(6):1–3, 1997. [8] Andrew Forward and Timothy C Lethbridge. The relevance of software documentation, tools and technologies: a survey. In Proceedings of the 2002 ACM symposium on Document engineering, pages 26–33, 2002. [9] Sergio Cozzetti B De Souza, Nicolas Anquetil, and Káthia M De Oliveira. A study of the documentation essential to software maintenance. In Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information, pages 68–75, 2005. [10] Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. Software documentation issues unveiled. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 1199–1210, 2019. URL https://api.semanticscholar.org/CorpusID:174800564. [11] Junji Zhi, Vahid Garousi-Yusifoğlu, Bo Sun, Golara Garousi, Shawn Shahnewaz, and Guenther Ruhe. Cost, benefits and quality of software development documentation: A systematic mapping. Journal of Systems and Software, 99:175–198, 2015. [12] Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E Hassan, and Shanping Li. Measuring program comprehension: A large-scale field study with professionals. IEEE Transactions on Software Engineering, 44(10):951–976, 2017. [13] Cognition AI. DeepWiki. https://deepwiki.com/, 2025. [14] Anthropic. Claude Code. https://www.anthropic.com/claude-code, 2025. [15] Golara Garousi, Vahid Garousi, Mahmoud Moussavi, Guenther Ruhe, and Brian Smith. Evaluating usage and quality of technical software documentation: an empirical study. In Proceedings of the 17th international conference on evaluation and assessment in software engineering, pages 24–35, 2013. 10

[16] Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks. arXiv preprint arXiv:2503.09572, 2025. [17] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. [18] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024. [19] Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643–13658, 2024. [20] Huy Nhat Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2409.16299, 2024. [21] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 2188–2200. IEEE, 2025. [22] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. [23] Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, et al. Ultrahorizon: Benchmarking agent capabilities in ultra long-horizon scenarios. arXiv preprint arXiv:2509.21766, 2025. [24] Rui Ye, Zhongwang Zhang, Kuan Li, Huifeng Yin, Zhengwei Tao, Yida Zhao, Liangcai Su, Liwen Zhang, Zile Qiao, Xinyu Wang, et al. Agentfold: Long-horizon web agents with proactive context management. arXiv preprint arXiv:2510.24699, 2025. [25] Shukai Liu, Jian Yang, Bo Jiang, Yizhi Li, Jinyang Guo, Xianglong Liu, and Bryan Dai. Context as a tool: Context management for long-horizon swe-agents. arXiv preprint arXiv:2512.22087, 2025. [26] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. [27] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. [28] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations, 2023. [29] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 15174–15186, 2024. [30] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First conference on language modeling, 2024. 11

[31] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638. URL https://aclanthology.org/2024.tacl-1.9/. [32] Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The illusion of diminishing returns: Measuring long horizon execution in LLMs. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/ forum?id=3lm8lWYxiq. [33] Guangya Wan, Mingyang Ling, Xiaoqi Ren, Rujun Han, Sheng Li, and Zizhao Zhang. Compass: Enhancing agent long-horizon reasoning with evolving context. arXiv preprint arXiv:2510.08790, 2025. [34] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. [35] Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. CoRR, abs/2310.08560, 2023. URL https://doi.org/10.48550/arXiv.2310.08560. [36] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. [37] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 19724–19731, 2024. [38] Bernal J Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems, 37:59532–59569, 2024. [39] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. Advances in Neural Information Processing Systems, 2025. [40] Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32779–32798, 2025. [41] Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, et al. Resum: Unlocking long-horizon search intelligence via context summarization. arXiv preprint arXiv:2509.13313, 2025. [42] Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. Acon: Optimizing context compression for long-horizon llm agents. arXiv preprint arXiv:2510.00615, 2025. [43] Mo Li, LH Xu, Qitai Tan, Long Ma, Ting Cao, and Yunxin Liu. Sculptor: Empowering llms with cognitive agency via active context management. arXiv preprint arXiv:2508.04664, 2025. [44] Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori Pollock, and K VijayShanker. Automatic generation of natural language summaries for java classes. In 2013 21st International conference on program comprehension (ICPC), pages 23–32. IEEE, 2013. [45] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using a neural attention model. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073–2083, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1195. URL https://aclanthology.org/P16-1195/. 12

[46] Alexander LeClair and Collin McMillan. Recommendations for datasets for source code summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3931–3937, 2019. [47] Yunseok Choi, Cheolwon Na, Hyojun Kim, and Jee-Hyong Lee. Readsum: retrieval-augmented adaptive transformer for source code summarization. IEEE Access, 11:51155–51165, 2023. [48] Bibek Poudel, Adam Cook, Sekou Traore, and Shelah Ameli. Documint: Docstring generation for python using small language models. arXiv preprint arXiv:2405.10243, 2024. [49] Junaed Younus Khan and Gias Uddin. Automatic code documentation generation using gpt3. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–6, 2022. [50] Vadim Lomshakov, Andrey Podivilov, Sergey Savin, Oleg Baryshnikov, Alena Lisevych, and Sergey Nikolenko. ProConSuL: Project context for code summarization with LLMs. In Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 866–880, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-industry.65. URL https://aclanthology.org/ 2024.emnlp-industry.65/. [51] Vladimir Makharev and Vladimir Ivanov. Code summarization beyond function level. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 153–160. IEEE, 2025. [52] Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–13, 2024. [53] Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. Automatic semantic augmentation of language model prompts (for code summarization). In Proceedings of the IEEE/ACM 46th international conference on software engineering, pages 1–13, 2024. [54] Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, et al. Code needs comments: Enhancing code llms with comment augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13640–13656, 2024. [55] Liuwen Cao, Hongkui He, Hailin Huang, Jiexin Wang, and Yi Cai. Rethinking-based code summarization with chain of comments. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 3043–3056, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/ 2025.coling-main.204/. [56] Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177, 2022. [57] Alessandro Scirè, Karim Ghonim, and Roberto Navigli. Fenice: Factuality evaluation of summarization based on natural language inference and claim extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14148–14161, 2024. [58] Joonho Yang, Seunghyun Yoon, ByeongJeong Kim, and Hwanhee Lee. FIZZ: Factual inconsistency detection by zoom-in summary and zoom-out document. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.3. URL https://aclanthology.org/2024.emnlp-main.3/. 13

[59] Suyoung Bae, CheolWon Na, Jaehoon Lee, Yumin Lee, YunSeok Choi, and Jee-Hyong Lee. Referee: Reference-free and fine-grained method for evaluating factual consistency in real-world code summarization. arXiv preprint arXiv:2604.10520, 2026. [60] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.741. URL https://aclanthology.org/ 2023.emnlp-main.741/. [61] Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. DevEval: A manually-annotated code generation benchmark aligned with real-world code repositories. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 3603–3614, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.214. URL https://aclanthology.org/ 2024.findings-acl.214/. [62] Robert Tarjan. Depth-first search and linear graph algorithms. SIAM journal on computing, 1 (2):146–160, 1972.

14

A

Additional details about MemDocAgent

A.1

Algorithms of dependency-aware traversal guiding

Algorithm 1 describes dependency graph construction, and Algorithm 2 presents the topological traversal used for hierarchical generation. Algorithm 1 Building the unified dependency graph G Require: Components C with DEPENDS _ ON(c) and PATH(c); repo id r Ensure: Graph G : V → 2V over V = C ∪ M ∪ {r}, where u → v means “u depends on / contains v” # Tier 1: component-level dependency edges 1: for c ∈ C do 2: G[c] ← DEPENDS _ ON(c) ▷ call, inheritance, attribute access, import 3: end for # Module hierarchy induced from file paths 4: Derive M from { DIRNAME( PATH (c)) : c ∈ C} and its prefix closure 5: CHILDREN (m) ← { c ∈ C : DIRNAME ( PATH(c)) = m } ∪ { m′ ∈ M : DIRNAME(m′ ) = m } # Tier 2: module-level containment edges 6: for m ∈ M do 7: G[m] ← CHILDREN(m) 8: end for # Tier 3: repository-level containment edges 9: G[r] ← { v ∈ M ∪ C : v has no parent module } 10: return G Algorithm 2 Calculating dependency-aware traversal order π Require: Unified graph G (Alg. 1) Ensure: Order π such that (i) every component precedes its dependents, and (ii) every module follows its children, with r last 1: G ← C ONDENSE SCC(G) ▷ Tarjan [62]: collapse cycles into super-nodes 2: roots ← {v ∈ G : v has no incoming edge} 3: visited ← ∅; π ← [ ] 4: function V ISIT(v) ▷ iterative post-order; emits v after all G[v] 5: for d ∈ G[v] with d ∈ / visited do 6: V ISIT(d) 7: end for 8: visited ∪= {v}; π.APPEND(v) 9: end function 10: for v ∈ roots ∪ (G \ visited) do ▷ cover isolated nodes too 11: if v ∈ / visited then V ISIT(v) 12: end for 13: return π A.2

System prompts used in MemDocAgent

Table 4: Agent’s system prompt <ROLE> You are an expert code documentation worker agent operating on a software repository. Your goal is to generate a high-quality documentation that are both complete and helpful for developers. </ROLE> continued on next page

15

Table 4 – continued from previous page <OBJECTIVES> Generate documentation sufficient for a developer to reconstruct a functional implementation from scratch using only this documentation. Each level must answer the following questions: - Repository-level : “What does this system do, and how are its parts organized?” - Module-level : “What does this module do, and how do its components interact?” - Component-level: “How do I implement this function/class/method correctly?” </OBJECTIVES> <DOCUMENTATION_STRUCTURE> Generate documentation following this structure depending on the target level: 1. REPO : Repository-level Documentation: - Brief introduction and purpose of the overall system - Architecture overview with diagrams - High-level functionality of each sub-module including references to its documentation file - Link to other module documentation instead of duplicating information - Do not duplicate content covered in MODULE or COMPONENT docs. Focus on the big picture, not implementation details 2. MODULE: Module-level Documentation: - Explanation of the module’s role within the system and its internal design, so a developer can understand *how* its components fit together before reading individual component details - Responsibility and boundaries of the module - List of core components with a one-line description each - Component interaction diagram or call flow (if non-trivial) - Key data structures and state shared across components - Links to component-level documentation for each component 3. COMPONENT: Component-level Documentation: - Providing enough detail to *reimplement* the function, method, or class correctly — covering inputs, outputs, behavior, edge cases, and constraints - Summary of what the component does and why it exists (not how it works) </DOCUMENTATION_STRUCTURE> <WORKFLOW> 1. You will first receive a sub-task, which includes the type of task (COMPONENT, MODULE, or REPO), the target component/module/repo to document, and other relevant information. 2. Analyze the provided code components or module structure, explore the not given dependencies between the components if needed. 3. For COMPONENT tasks, generate the documentation for the specific component, and save the documentation in memory with the name of ‘component_id’. 4. For MODULE tasks, synthesize the documentations of sub-components and generate the module-level documentation, and save the documentation in memory with the name of ‘module_id’. 5. For REPO tasks, synthesize the documentations of all modules and generate the repository-level documentation, and save the documentation in memory with the name of ‘repo_id’. 6. For each task, you perform thought-action-observation loops to iteratively improve the documentation until it passes verification, then save the final documentation to memory and return. - At every turn, you MUST follow this structure: Thought: 〈your reasoning about what to do next, what information you need〉 Action: Choose exactly one action from the list below and provide the necessary input. (Observation will be provided after each action to inform your next step. Do NOT generate this yourself.) - Repeat until all target component is fully documented and verified, then call Finish to save and exit. - The required sequence of actions is typically: Read → Write → Verify → (optional Read or Write) → Finish </WORKFLOW> <AVAILABLE_TOOLS> For each turn, first think about what you should do, then take exactly one of the four actions below: - READ: Request specific information about the code component or module structure. - WRITE: Generate documentation for the target component/module/repo based on the provided information and context. - VERIFY: Evaluate the quality of the generated documentation using an external evaluation tool. Use the feedback to iteratively improve the documentation if needed. continued on next page

16

Table 4 – continued from previous page - FINISH: Save the final documentation to memory and end the task. Only call this after VERIFY passes. </AVAILABLE_TOOLS> <DETAILED_TOOL_USAGE> 1. READ: If you think more information is needed to generate high-quality documentation of the target component, use this action to request relevant information. - During think step, you should analyze the current code and context, and explain what additional information might be needed (if any) - You have access to three types of information sources: 1) Sub-components or sub-modules (from memory): - If the target is a MODULE or REPO, you can request the documentation of its sub-components or sub-modules that have already been documented from memory. - This is the primary source of information for MODULE and REPO tasks, since the module/repo-level documentation should be synthesized based on the already generated sub-component/sub-module documentations. - If you think you need the information of sub-components or sub-modules from memory, use <RETRIEVE>YES</RETRIEVE> tag to request the specific documentations from memory. - If the target is a REPO, you should first read the documentation of all top-level modules and components using READ<RETRIEVE>YES</RETRIEVE>, then decide if you need to read the documentation of lower-level sub-modules based on the gaps you observe in the module-level documentations. 2) Internal Codebase Information (from local code repository): For Functions: - Code components called within the function body - Places where this function is called For Methods: - Code components called within the method body - Places where this method is called - The class this method belongs to For Classes: - Code components called in the __init__ method - Places where this class is instantiated - Complete class implementation beyond __init__ 3) External Open Internet retrieval Information: - External Retrieval is extremely expensive. Only request it when understanding an external third-party API or library is essential for accurate documentation, and that information cannot be found within the target codebase. - Use the import statements in <IMPORT_INFORMATION_IN_THE_FILE> to identify candidates for retrieval. If the component relies on a third-party API (e.g. from openai import OpenAI, import torch, from stripe import Webhook), you may request retrieval to understand the external API’s expected behavior, parameters, or return types. - Also request retrieval for novel, state-of-the-art algorithms or techniques not self-explanatory from the code alone (e.g. NDCG Loss, Cohen’s Kappa, specialized metrics). - Each query should be a clear, natural language question targeting specific API behavior or concept. - If more information is needed, end your response with a structured request in XML format: <RETRIEVE>YES or NO</RETRIEVE> <REQUEST> <INTERNAL> <CLASS>class1,class2</CLASS> <FUNCTION>func1,func2</FUNCTION> <METHOD>self.method1,instance.method2,class.method3</METHOD> </INTERNAL> <RETRIEVAL> <QUERY>query1</QUERY> </RETRIEVAL> </REQUEST> - Important rules for structured request: - If no items exist for a category, use empty tags (e.g., <CLASS></CLASS>) - Each external QUERY should be a concise, clear, natural language search query. - Use comma-separated values without spaces for multiple items. - For METHODS, keep dot notation in the same format as the input. continued on next page

17

Table 4 – continued from previous page - Only first-level calls of the focal code component are accessible. Do not request information on code components that are not directly called by the focal component. - External Retrieval is extremely expensive. Only request external open internet retrieval information if the entity involves a novel, state of the art, recently-proposed algorithms or techniques. (e.g. computing a novel loss function (NDCG Loss, Alignment and Uniformity Loss, etc), certain novel metrics (Cohen’s Kappa, etc), specialized novel ideas that can not be searched within the target codebase) - Each query should be a clear, natural language question - Only request internal codebase information that you think is necessary for docstring generation task. For some components that is simple and obvious, you do not need any other information for docstring generation. - For module and repository-level documentation, you MUST request <RETRIEVE>YES</RETRIEVE> first before any other information source. The memory-stored summaries are the primary basis for module and repository-level documentation and should always be retrieved at the start. - For module and repository-level documentation, Sources 2) and 3) are supplementary. Since sub-components or sub-modules have already been individually documented in prior steps, their internal or external information has already been gathered. Only request additional internal codebase or external retrieval information if there is a specific gap that cannot be addressed by the memory-stored summaries. - Example response: Thought: put your thought here. Action: READ <RETRIEVE>NO</RETRIEVE> <REQUEST> <INTERNAL> <CLASS></CLASS> <FUNCTION>execute_query,connect_db</FUNCTION> <METHOD>self.process_data,data_processor._internal_process</METHOD> </INTERNAL> <RETRIEVAL> <QUERY></QUERY> </RETRIEVAL> </REQUEST> 2. WRITE: If you think you have collected sufficient context, use this action and generate the documentation for the target task type. - General guidelines for high-quality documentation: - Make documentations actionable and specific: Focus on practical usage. - Use clear, concise language: Avoid jargon unless necessary, use active voice, and be direct and specific. - Type Information: Include precise type hints, note any type constraints, and document generic type parameters. - Context and Integration: Explain component relationships, note any dependencies, and describe side effects. - Follow Google docstring format: Use consistent indentation, maintain clear section separation. - You MUST include the documentation in a <DOCUMENTATION>...</DOCUMENTATION> block including required contents. - The <DOCUMENTATION> block must contain ONLY the documentation text — section headers and their content. - NEVER include any of the following inside <DOCUMENTATION>: - Python source code (def, class, return, import statements) - Triple-quoted docstrings (triple double-quotes or triple single-quotes) - Code fences (“‘) - The original function/class definition - Any comment syntax (# or //) - CORRECT example: Thought: put your thought here. Action: WRITE <DOCUMENTATION> ## Summary: Reads a file and returns its content as a list of lines. ## Description: continued on next page

18

Table 4 – continued from previous page Opens the specified file in read mode using the given encoding. ## Args: path (str): Absolute or relative path to the target file. encoding (str): Character encoding. Defaults to ‘utf-8’. ## Returns: list[str]: A list of strings, one per line, with trailing newlines removed. ## Raises: FileNotFoundError: If the file does not exist. </DOCUMENTATION> - WRONG example (do NOT do this): <DOCUMENTATION> def read_lines(path, encoding=‘utf-8’): """Reads a file and returns its content as a list of lines.""" return open(path, encoding=encoding).readlines() </DOCUMENTATION> 3. VERIFY: After generating a documentation, use this action to self-evaluate the documentation quality along three criteria, each scored from 0.00 to 1.00 (two decimal places). - Verification Process: - First read the target task information (source code and related information) as if you’re seeing it for the first time. - Read the generated documentation and evaluate each of the following criteria. - Be a harsh, skeptical reviewer. Assume the documentation is flawed until proven otherwise. - A perfect 1.00 is reserved for documentation that cannot be improved in any way. - Scoring discipline: - Start each criterion at 0.50 and adjust up or down based on concrete evidence found during review. - Deduct at least 0.05 for each distinct minor flaw (e.g. imprecise wording, missing edge case). - Deduct at least 0.10 for each distinct major flaw (e.g. wrong type, fabricated parameter, missing parameter entirely). - A score above 0.85 requires an explicit justification in your Thought — list what the documentation does well. - Never round up. If you are unsure whether a claim is correct, treat it as incorrect. - Avoid anchoring to your own prior output — evaluate the text as if someone else wrote it. - Evaluation criteria: 1) Consistency: - Line-by-line cross-check: for every factual statement in the documentation, locate the corresponding evidence in the source code. If you cannot point to a specific line that supports the claim, deduct points. - Every parameter name, type, and description must exactly match the actual function signature; no parameter is mislabeled, mistyped, or given a fabricated default value. - Described behavior (preconditions, side effects, return values, raised exceptions) must be directly verifiable from the source code — nothing is fabricated or assumed. - No statement in the documentation may contradict what the code actually does (e.g., wrong return type, incorrect exception condition, swapped parameter roles). - Code examples, if present, must be syntactically valid Python and produce the stated output when executed against the implementation. - Deduct points for: vague hedge words (“typically”, “usually”, “may”) used instead of precise statements; claims that go beyond what the source code shows; any detail copied from a different component; invented functionality not present in the code. 2) Completeness: - Walk through the source code top-to-bottom and verify that every visible element is documented: * Every parameter: name, type, and purpose. Is any parameter silently omitted? * Return value: type and meaning. Is it documented? * Every exception explicitly raised: type and trigger condition. Is any missing? - Significant edge cases, constraints, and preconditions visible in the source code (e.g. if-guards, assertions, boundary checks) must be reflected in the documentation. - For classes: are all public attributes and __init__ parameters documented? For modules/repos: are all public components listed with one-line role descriptions? - Deduct points for: any parameter, return value, or raised exception present in the source but absent from the documentation; missing edge-case that is visible in the code; undocumented class attributes or module-level exports. continued on next page

19

Table 4 – continued from previous page 3) Helpfulness: - Read the Summary line in isolation. Does it immediately convey what the component does and why it exists — or does it merely restate the function/class name in slightly longer words? - Imagine you are a developer who has never seen this codebase. Can you understand how and when to use this component from the documentation alone, without reading the source code? - Descriptions must explain purpose and typical use cases (WHY / WHEN), not just restate the code logic line-by-line (HOW). - For non-trivial interfaces: are usage examples included? Do they demonstrate realistic, end-to-end scenarios, or are they trivially obvious (e.g. just calling the function with no context)? - Deduct points for: summary that paraphrases the function name without adding insight; descriptions that merely narrate the implementation step-by-step; missing examples for complex interfaces; examples that show only the happy path without edge cases. - You MUST include a <SCORE> block immediately after the action line with a structured request in XML format: <SCORE> <CONSISTENCY>score1</CONSISTENCY> <COMPLETENESS>score2</COMPLETENESS> <HELPFULNESS>score3</HELPFULNESS> </SCORE> - The three scores are combined into a final score to determine whether the documentation meets the verification. - If the observation of the VERIFY action is PASS, proceed to Finish. If the verdict is REVISE, consider whether more context is needed (READ) or whether you can directly rewrite based on the feedback (WRITE). - If the documentation does not pass verification, it may be revised and re-verified up to {max_revisions} times. - Example response: Thought: put your thought here. Action: VERIFY <SCORE> <CONSISTENCY>0.90</CONSISTENCY> <COMPLETENESS>0.85</COMPLETENESS> <HELPFULNESS>0.80</HELPFULNESS> </SCORE> 4. FINISH: If you think the documentation is complete and of high quality, and has passed verification, use this action to save the documentation to memory and end the task. Only call this after VERIFY passes. - Example response: Thought: put your thought here. Action: FINISH </DETAILED_TOOL_USAGE>

Table 5: Component sub-task injection prompt Now, generate comprehensive documentation for the function component named bplustree.utils.pairwise using the provided source code and file path. The generated function typed component-level documentation should be included following contents: continued on next page

20

Table 5 – continued from previous page ## Summary: - One-line description focusing on WHAT the function does - Avoid repeating the function name - Emphasize the outcome or effect ## Description: - List known callers within the codebase and the context in which they call this function: Describe the typical trigger condition or pipeline stage - Explain why this logic is extracted into its own function rather than inlined: Describe the respon sibility boundary it enforces ## Args (if present): - name, type, allowed range/values, default value - Note any interdependencies between parameters ## Returns: - Explain what the return value represents - Include all possible return values and edge-case return values ## Raises: - exception type and exact condition in the code that triggers it ## Constraints: - Preconditions: what must be true before calling this function - Postconditions: what is guaranteed to be true after it returns ## Side Effects: - Any I/O (files, network, stdout) - External state mutations (global variables, database writes, cache updates) - External service calls ## Control Flow: - Mermaid flowchart (flowchart TD) illustrating the main decision branches and loops ## Examples (if public and non-trivial): - Show realistic end-to-end usage including error handling <FILE_PATH> bplustree/utils.py </FILE_PATH> <IMPORT_INFORMATION_IN_THE_FILE> import itertools from typing import Iterable </IMPORT_INFORMATION_IN_THE_FILE> <SOURCE_CODE> def pairwise(iterable: Iterable): a, b = itertools.tee(iterable) next(b, None) return zip(a, b) </SOURCE_CODE>

Table 6: Module sub-task injection prompt Now, generate comprehensive documentation for the bplustree module using the provided module tree. The generated module-level documentation should be included following contents: continued on next page

21

Table 6 – continued from previous page ## Tree: - Directory/file hierarchy of this module (indented tree format) ## Role: - Single-responsibility description: the one thing this module owns - Avoid repeating the module name ## Description: - Describe where and when this module is used within the repo: List the primary consumers (other modules or entry points that import it) - Explain why these components are grouped into a separate module: Describe the cohesion principle (shared concept, layer boundary, etc.) ## Components: - List all public classes, functions, and constants with their signatures - For each: one-line role description - Mermaid dependency graph showing relationships among internal components ## Public API: - The interfaces this module exposes to the rest of the repository - For each public symbol: signature, brief description, usage note ## Dependencies: - Internal imports (other repo modules) and their purpose - External imports (third-party libraries) and why they are needed ## Constraints: - Constraints callers must respect when using this module - Thread-safety, ordering requirements, initialization prerequisites <MODULE_TREE> bplustree/ __init__.py const.py entry.py memory.py node.py serializer.py tree.py utils.py </MODULE_TREE>

Table 7: Repository sub-task injection prompt Now, generate comprehensive documentation for the bplustree repository using the provided repository tree. The generated repository-level documentation should be included following contents: continued on next page

22

Table 7 – continued from previous page ## Tree: - Full top-level directory hierarchy (2–3 levels deep, indented tree format) - Annotate each major directory with its responsibility ## Purpose: - What problem this repository solves and why it matters - Target users and the scenarios they use it in - Position in the broader ecosystem (standalone tool, library, service) ## Architecture: - End-to-end data flow diagram using Mermaid (flowchart TD or sequence diagram) - Key abstractions and architectural patterns (pipeline, agent loop, plugin, etc.) ## Entry Points: - CLI commands, importable APIs, or service endpoints - For each: what it exposes, required arguments, and target audience ## Core Features: - List the key capabilities the repo provides - For each feature: one-line description + the implementing module(s)/component(s) ## Dependencies: - Key external dependencies and the role each plays - Version constraints or compatibility requirements that affect architecture Conditional sections:

## Configuration: - Config files, environment variables, runtime parameters - Include only if configuration meaningfully affects system behavior ## Extension Points: - How to extend the system (plugins, hooks, subclassing, config-driven behavior) - Include only if extensibility is a first-class concern <REPO_TREE> bplustree/ bplustree/ setup.py </REPO_TREE>

Table 8: Repository-level documentation format ## Tree: - Full top-level directory hierarchy (2–3 levels deep, indented tree format) - Annotate each major directory with its responsibility ## Purpose: - What problem this repository solves and why it matters - Target users and the scenarios they use it in - Position in the broader ecosystem (standalone tool, library, service) ## Architecture: - End-to-end data flow diagram using Mermaid (flowchart TD or sequence diagram) - Key abstractions and architectural patterns (pipeline, agent loop, plugin, etc.) ## Entry Points: - CLI commands, importable APIs, or service endpoints - For each: what it exposes, required arguments, and target audience ## Core Features: - List the key capabilities the repo provides continued on next page

23

Table 8 – continued from previous page - For each feature: one-line description + the implementing module(s)/component(s) ## Dependencies: - Key external dependencies and the role each plays - Version constraints or compatibility requirements that affect architecture Conditional sections: ## Configuration: - Config files, environment variables, runtime parameters - Include only if configuration meaningfully affects system behavior ## Extension Points: - How to extend the system (plugins, hooks, subclassing, config-driven behavior) - Include only if extensibility is a first-class concern

Table 9: Module-level documentation format ## Tree: - Directory/file hierarchy of this module (indented tree format) ## Role: - Single-responsibility description: the one thing this module owns - Avoid repeating the module name ## Description: - Describe where and when this module is used within the repo: List the primary consumers (other modules or entry points that import it) - Explain why these components are grouped into a separate module: Describe the cohesion principle (shared concept, layer boundary, etc.) ## Components: - List all public classes, functions, and constants with their signatures - For each: one-line role description - Mermaid dependency graph showing relationships among internal components ## Public API: - The interfaces this module exposes to the rest of the repository - For each public symbol: signature, brief description, usage note ## Dependencies: - Internal imports (other repo modules) and their purpose - External imports (third-party libraries) and why they are needed ## Constraints: - Constraints callers must respect when using this module - Thread-safety, ordering requirements, initialization prerequisites

Table 10: Component-level (Function) documentation format ## Summary: - One-line description focusing on WHAT the function does - Avoid repeating the function name - Emphasize the outcome or effect ## Description: - List known callers within the codebase and the context in which they call this function: Describe the typical trigger condition or pipeline stage - Explain why this logic is extracted into its own function rather than inlined: Describe the responsibility boundary it enforces continued on next page

24

Table 10 – continued from previous page ## Args (if present): - name, type, allowed range/values, default value - Note any interdependencies between parameters ## Returns: - Explain what the return value represents - Include all possible return values and edge-case return values ## Raises: - exception type and exact condition in the code that triggers it ## Constraints: - Preconditions: what must be true before calling this function - Postconditions: what is guaranteed to be true after it returns ## Side Effects: - Any I/O (files, network, stdout) - External state mutations (global variables, database writes, cache updates) - External service calls ## Control Flow: - Mermaid flowchart (flowchart TD) illustrating the main decision branches and loops ## Examples (if public and non-trivial): - Show realistic end-to-end usage including error handling

Table 11: Component-level (Method) documentation format ## Summary: - One-line description focusing on WHAT the method does - Avoid repeating the method name - Emphasize the effect on the object’s state ## Description: - List known callers and the context in which they call this method: Describe the lifecycle stage or pipeline step where this method is invoked - Explain why this logic is its own method rather than inlined or placed elsewhere ## Args (if present): - name, type, allowed range/values, default value ## Returns: - type, possible values, and edge-case return values ## Raises: - exception type and exact condition that triggers it ## State Changes: - Attributes READ: list self.<attr> fields this method reads - Attributes WRITTEN: list self.<attr> fields this method modifies ## Constraints: - Preconditions: what must be true about the object/args before calling - Postconditions: what is guaranteed on self or the return value after the call ## Side Effects: - I/O, external service calls, or mutations to objects outside self

Table 12: Component-level (Class) documentation format ## Summary: continued on next page

25

Table 12 – continued from previous page - One-line description focusing on WHAT the class represents - Focus on the core purpose or responsibility ## Description: - Scenarios where this class should be instantiated; known callers/factories that create instances - Motivation for this class as a distinct abstraction; the responsibility boundary it enforces ## State: - Each attribute: name, type, valid range/values, invariant it participates in - For __init__ parameters: note default value and any constraints for the caller - Class invariants: conditions that must always hold between method calls ## Lifecycle: - Creation: how to instantiate (required args, factory methods) - Usage: which methods are called in what order; any required sequencing - Destruction: cleanup responsibilities (context manager, close(), etc.) ## Method Map: - Mermaid diagram (flowchart or graph) showing method call dependencies and typical invocation order ## Raises: - Exceptions raised by __init__ and their trigger conditions ## Example: - Demonstrate creation, typical method sequence, and cleanup

A.3

Self-evaluation and cross-document conflict verification: detailed procedure

Self-evaluation score: The agent evaluates the candidate document dˆi along three criteria: factual consistency, completeness, and helpfulness. For each criterion c ∈ {cons, comp, help}, the agent returns a score sc (dˆi ) ∈ [0, 1]. The overall self-evaluation score is defined as the mean of the three:
sself (dˆi ) = 13 scons (dˆi ) + scomp (dˆi ) + shelp (dˆi ) . (2) NLI-based cross-document conflict evaluation score: We verify cross-document conflicts by utilizing existing NLI-based inconsistency detection [56–58], which decomposes a candidate text into fine-grained units and checks each unit against a reference source using a pre-trained NLI model. The process consists of the following steps: Step 1. Reference set construction. We determine the candidate reference set V(ui ) ⊆ RepoMemory from the dependency relations of ui , with the granularity of ui dictating which records are included: • if ui ∈ C is a component, V(ui ) contains the component records connected to ui via the depends_on relation; • if ui ∈ M is a module, V(ui ) contains the records of its direct children, the components and sub-modules pointed to by Cm ; • if ui = r is the repository, V(ui ) contains the records of root modules pointed to by Cr . We then filter V(ui ) by retaining only the records whose verification score exceeds a threshold (0.9), yielding the trusted reference set Dref (ui ). If Dref (ui ) = ∅, meaning ui has no dependencies or none of the candidate references meet the threshold, conflict verification is skipped and we set sconflict = 0. Step2. Hypothesis filtering. The verifier first decomposes dˆi into atomic claims via an LLM (prompt in Table 13), and retains only those that explicitly mention a component or module name covered by the reference documents Dref (ui ), forming the hypothesis set Hi . This filtering both prevents claims unrelated to the references (e.g., descriptions of ui ’s own behavior) from inducing spurious entailment during NLI, and reduces verification cost by reducing NLI calls to reference-relevant claims.

26

Table 13: Claim extraction prompt for cross-doc conflict verification You are a technical documentation analyst. Extract atomic factual claims from the given documentation. Each claim should be a single, verifiable statement about the code’s behavior, parameters, return values, side effects, or design. Exclude subjective opinions, vague descriptions, and section headers. Documentation: {documentation} Extract atomic factual claims from this documentation. Return a JSON array of strings. Each string is one claim. Return ONLY the JSON array, no other text. Example output: ["The function returns None when the input is empty", "The timeout parameter defaults to 30 seconds"]

Step 3. Pair-wise NLI verification. For each hypothesis h ∈ Hi and each reference document dj ∈ Dref (ui ), we use a pre-trained NLI model3 to test whether dj entails h. A pair (h, dj ) is judged unsupported if the entailment score falls below a threshold τnli . A hypothesis that remains unsupported across every dj ∈ Dref (ui ) is classified as a conflict, since by construction h shares a component or module with the references (Step 2) yet none of them confirms it. Table 14 shows representative judgments produced by the NLI model. Table 14: Examples of NLI judgments used in conflict detection. Premise: “DataProcessor.process is a static method that returns a list of integers [1, 2, 3]; it takes no arguments.” Hypothesis (from dˆi )

Label

“process_data calls DataProcessor.process, which returns a dictionary.” “process_data retrieves data by calling DataProcessor.process, which returns a list.” “process_data stores the result in self.data for later retrieval.”

Reason

contradiction

return type mismatch (list vs. dict)

entailment

consistent with premise

neutral

outside premise’s scope

Step 4. Conflict score aggregation. We aggregate the per-hypothesis judgments from Step 3 into a document-level conflict score for dˆi . Let Hiconf ⊆ Hi denote the subset of hypotheses classified as conflict; the conflict score is defined as conf

|Hi | sconflict (dˆi ) = ∈ [0, 1], |Hi |

(3)

where larger values indicate greater inconsistency between dˆi and the reference documents Dref (ui ). We set sconflict (dˆi ) = 0 by convention when Hi = ∅. The final verify score combines the self-evaluation score sself (dˆi ) with the conflict-free fraction 1 − sconflict (dˆi ) via averaging:
verification_score(dˆi ) = 21 sself (dˆi ) + (1 − sconflict (dˆi )) ∈ [0, 1]. (4) A.4

Agent trajectory examples

3 MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli.

27

Subtask #1 — COMPONENT: misc.generate_authors.drop_recurrences File: misc/generate_authors.py Thought1: Function appears to be a generator that removes duplicates from an iterable while preserving order; no caller context available. Action1: READ <RETRIEVE>NO</RETRIEVE> Observation1: No information retrieved. Thought2: I have the function source code and imports available. The function ‘drop_recurrences‘ is straightforward - it removes duplicates from an iterable while preserving order. Since I don’t have information about callers or usage context in the broader codebase, I’ll document what I can see from the function itself. Let me write the documentation for this function based on the source code analysis. Action2: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation2: Draft stored. Proceed to Verify. Thought3: Now I need to verify the quality of the documentation I’ve written. Let me check each criterion Action3: VERIFY <SCORE> <CONSISTENCY>0.90</CONSISTENCY> <COMPLETENESS>0.75</COMPLETENESS> <HELPFULNESS>0.85</HELPFULNESS> </SCORE> Observation3: Weighted avg = 0.8333, Conflict = 1.00, Final = 0.83 → REVISE (below threshold 0.90) Thought4: The verification shows that completeness needs improvement. Looking at the issues: 1. The constraint about hashability wasn’t properly emphasized - this is a critical limitation of the function 2. I should be more explicit about what happens with unhashable types (which would cause runtime errors) 3. The examples could be more comprehensive Let me revise the documentation to address these issues. Action4: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation4: Draft stored. Proceed to Verify. Thought5: Let me verify the revised documentation against the same criteria. Action5: VERIFY <SCORE> <CONSISTENCY>0.95</CONSISTENCY> <COMPLETENESS>0.9</COMPLETENESS> <HELPFULNESS>0.9</HELPFULNESS> </SCORE> Observation5: Verification PASS (final=0.92, threshold=0.90) Proceed to FINISH action. Thought6: The documentation has now passed verification with strong scores across all criteria. The function ‘drop_recurrences‘ is well-documented with attention to its key characteristics: - It removes duplicates while preserving order - It requires hashable elements - It raises TypeError for unhashable types - It’s implemented as a generator for memory efficiency - It includes clear examples showing both proper usage and edge cases All requirements have been met and the documentation is ready to be saved. Action6: FINISH Subtask #31 — COMPONENT: pysnooper.variables.BaseVariable File: pysnooper/variables.py Thought1: I need to document the BaseVariable class from pysnooper/variables.py. Let me analyze what I know about this class and what I need to find out. Action1: READ <RETRIEVE>NO</RETRIEVE> <REQUEST> <INTERNAL> <CLASS></CLASS> <FUNCTION>utils.ensure_tuple,needs_parentheses</FUNCTION> <METHOD></METHOD> </INTERNAL> <RETRIEVAL>

28

(continued from previous page) <QUERY></QUERY> </RETRIEVAL> </REQUEST> Observation1: [Retrieved Context] Thought2: I now have information about the utility functions, but I still need to understand what pycompat.ABC is. Action2: READ <RETRIEVE>NO</RETRIEVE> <REQUEST> <INTERNAL> <CLASS></CLASS> <FUNCTION></FUNCTION> <METHOD></METHOD> </INTERNAL> <RETRIEVAL> <QUERY>pysnooper pycompat module ABC definition</QUERY> </RETRIEVAL> </REQUEST> Observation2: [Retrieved Context] Thought3: Let me write the documentation based on what I can definitively determine. Action3: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation3: Draft stored. Proceed to Verify. Thought4: Now I need to verify this documentation against the source code to ensure consistency, completeness, and helpfulness. Action4: VERIFY <SCORE> <CONSISTENCY>0.95</CONSISTENCY> <COMPLETENESS>0.9</COMPLETENESS> <HELPFULNESS>0.85</HELPFULNESS> </SCORE> Observation4: Verification REVISE (final=0.82, threshold=0.90) Dependency Information No-Conflict : 0.75 - Conflict with [pysnooper.variables.BaseVariable.items] (support=0.00): YOUR claim : Usage: Call items(frame, normalize=False) to evaluate and process the variable in a given frame context DEP states : No direct I/O operations or external service calls Fix the above conflicts by aligning your documentation with the dependency’s documented behavior. Score 0.82 is below threshold 0.90. Revise the documentation. Revise the draft (revision 1/2). Thought5: I see there’s a conflict regarding the usage description. The issue is that I’m claiming the items method “evaluates and processes the variable in a given frame context” but the dependency information says there are no direct I/O operations or external service calls. However, looking more carefully at the actual implementation, the items method does perform evaluation (which is a form of processing) and it does take a frame parameter, so my description isn’t completely wrong. But let me reconsider what’s really happening. The method does: 1. Takes a frame parameter 2. Evaluates code in that frame context 3. Calls _items on the result 4. Returns the result of _items The conflict seems to be about terminology - “processes” might be too vague or misleading. Let me revise to be more precise about what the method actually does. Action5: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation5: Draft stored. Proceed to Verify. Thought6: Let me verify this revised documentation to make sure it addresses the previous concerns while maintaining accuracy. Action6: VERIFY <SCORE> <CONSISTENCY>0.95</CONSISTENCY> <COMPLETENESS>0.95</COMPLETENESS> <HELPFULNESS>0.9</HELPFULNESS> </SCORE> Observation6: Verification PASS (final=0.97, threshold=0.90) Dependency Information No-Conflict : 1.00 Proceed to FINISH action.

29

(continued from previous page) Thought7: The documentation has now passed verification with a strong score of 0.97. All criteria have been met. Action7: FINISH Subtask #67 — MODULE: pysnooper Path: pysnooper/ Thought1: I need to generate module-level documentation for the pysnooper module based on the provided directory structure. First, I should retrieve the existing documentation for the sub-modules to understand their individual contributions, then synthesize them into a coherent module-level documentation. Let me start by retrieving the documentation for each submodule that has already been documented in memory. Action1: READ <RETRIEVE>YES</RETRIEVE> <REQUEST> <INTERNAL> <CLASS></CLASS> <FUNCTION></FUNCTION> <METHOD></METHOD> </INTERNAL> <RETRIEVAL> <QUERY></QUERY> </RETRIEVAL> </REQUEST> Observation1: [Retrieved Context] Thought2: Let me write the documentation based on what I can definitively determine. Action2: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation2: Draft stored. Proceed to Verify. Thought3: Now I need to verify the quality of the generated module-level documentation to ensure it meets the required standards for consistency, completeness, and helpfulness. Action3: VERIFY <SCORE> <CONSISTENCY>0.90</CONSISTENCY> <COMPLETENESS>0.85</COMPLETENESS> <HELPFULNESS>0.80</HELPFULNESS> </SCORE> Observation3: Verification REVISE (final=0.90, threshold=0.90) Dependency Information No-Conflict : 0.94 - Conflict with [pysnooper.pycompat.ABC] (support=0.00): YOUR claim : abc: Standard library for abstract base class functionality DEP states : Abstract base class compatibility wrapper for cross-Python-version support. - Conflict with [pysnooper.utils.WritableStream] (support=0.00): YOUR claim : Output destinations must be one of the supported types (None, str, PathLike, callable, or WritableStream) DEP states : An abstract base class defining the interface for writable streams that must implement a write method. Fix the above conflicts by aligning your documentation with the dependency’s documented behavior. Score 0.90 is below threshold 0.90. Revise the documentation. Revise the draft (revision 1/2). Thought4: Action4: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation4: Draft stored. Proceed to Verify. Thought5: Let me verify once more to see if this latest version resolves the remaining conflicts. Action5: VERIFY <SCORE> <CONSISTENCY>0.95</CONSISTENCY> <COMPLETENESS>0.95</COMPLETENESS> <HELPFULNESS>0.9</HELPFULNESS> </SCORE> Observation5: Verification PASS (final=0.92, threshold=0.90) Dependency Information No-Conflict : 0.94. Proceed to FINISH action. Thought6: All criteria have been met. Action6: FINISH #68 — REPO: PySnooper

30

(continued from previous page) Path: /home/sybae/project/Data/Source_Code_Original/Software-Development/PySnooper Thought1: I need to generate comprehensive repository-level documentation for the PySnooper repository. Action1: READ <RETRIEVE>YES</RETRIEVE> <REQUEST> <INTERNAL> <CLASS></CLASS> <FUNCTION></FUNCTION> <METHOD></METHOD> </INTERNAL> <RETRIEVAL> <QUERY></QUERY> </RETRIEVAL> </REQUEST> Observation1: [Retrieved Context] Thought2: I now have the documentation for the main pysnooper module. I need to also check the misc directory to understand what’s in there, as it might contain important information like README or other documentation files. Let me also get information about setup.py to understand the package configuration. Action2: READ <RETRIEVE>YES</RETRIEVE> <REQUEST> <INTERNAL> <CLASS></CLASS> <FUNCTION>read_file</FUNCTION> <METHOD></METHOD> </INTERNAL> <RETRIEVAL> <QUERY></QUERY> </RETRIEVAL> </REQUEST> Observation2: [Retrieved Context] Thought3: Let me write the documentation based on what I can definitively determine. Action3: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation3: Draft stored. Proceed to Verify. Thought3: Now I need to verify this repository-level documentation to ensure it’s accurate and complete. Action3: VERIFY <SCORE> <CONSISTENCY>0.85</CONSISTENCY> <COMPLETENESS>0.80</COMPLETENESS> <HELPFULNESS>0.85</HELPFULNESS> </SCORE> Observation3: Verification REVISE (final=0.81, threshold=0.90) Dependency Information No-Conflict : 0.78 - Conflict with [pysnooper] (support=0.00): YOUR claim : CLI Commands: None - PySnooper is a library, not a command-line tool DEP states : Provides comprehensive code tracing and variable inspection capabilities for debugging Python applications - Conflict with [pysnooper] (support=0.00): YOUR claim : PySnooper operates with minimal configuration: DEP states : Provides comprehensive code tracing and variable inspection capabilities for debugging Python applications Fix the above conflicts by aligning your documentation with the dependency’s documented behavior. Score 0.81 is below threshold 0.90. Revise the documentation. Revise the draft (revision 1/2). Thought4: I now have the documentation for the main pysnooper module. I need to also check the misc directory to understand what’s in there, as it might contain important information like README or other documentation files. Let me also get information about setup.py to understand the package configuration. Action4: READ <RETRIEVE>YES</RETRIEVE> <REQUEST> <INTERNAL> <CLASS>Tracer</CLASS>

31

(continued from previous page) <FUNCTION>snoop</FUNCTION> <METHOD>_call__, __enter__, __exit__</METHOD> </INTERNAL> <RETRIEVAL> <QUERY></QUERY> </RETRIEVAL> </REQUEST> Observation4: [Retrieved Context] Thought5: Action5: WRITE<DOCUMENTATION>[..draft]<DOCUMENTATION> Observation5: Draft stored. Proceed to Verify. Thought6: Let me verify once more to see if this latest version resolves the remaining conflicts. Action6: VERIFY <SCORE> <CONSISTENCY>0.95</CONSISTENCY> <COMPLETENESS>0.9</COMPLETENESS> <HELPFULNESS>0.9</HELPFULNESS> </SCORE> Observation6: Verification PASS (final=0.90, threshold=0.90) Dependency Information No-Conflict : 0.88. Proceed to FINISH action. Thought6: All criteria have been met. Action6: FINISH

B

Additional details about experimental setups

B.1

Evaluations

B.1.1

Completeness

Completeness measures the extent to which the generated documentation adheres to standard structural conventions and includes essential code entities expected for a given target. It is computed as the average of two sub-scores: Completeness =

Section Presence Score + Entity Coverage Score ∈ [0, 1] 2

(5)

Section Presence Score: For each documentation, we measure the proportion of required sections that are present, using rule-based pattern matching over section headers. The set of required and conditional sections differs per granularity level, reflecting the structural conventions adopted by prior systems (DocAgent, RepoAgent, CodeWiki) and our own framework. Conditional sections are not penalized when absent if they are not applicable to the target (e.g., a function with no arguments is not required to contain an Arguments section). The required and conditional sections per granularity are summarized in Table 16. The Section Presence score is computed as: Section Presence Score =

|required sections present| |required sections|

(6)

Entity Coverage Score: We extract the set of core entities from the target source code via abstract syntax tree (AST) analysis, and measure the proportion of these entities that are referenced in the documentation via regular-expression matching. The core entity set is defined per granularity: • Component-level: Function, class, method, and variable names that appear within the target source code. 32

Table 16: Required and conditional sections for documentation in each granularity. Granularity Required Sections

Conditional Sections

Function or Method

Summary, Description, Returns

Arguments, Exceptions, Side Effects, Control Flow, Usage Examples

Class

Summary, Description, Returns

Arguments, Exceptions, Usage Examples, Side Effects

Module

Module Structure (Tree), Role, Description, Components, Public API, Dependencies

-

Repository Repository Structure (Tree), Purpose, Architecture, Entry Points, Core Features, Dependencies

Configuration, Extension Points

• Module-level: Names of the child components contained in the module. • Repository-level: Names of the top-level modules together with the component names along their immediate paths. The Entity Coverage score is computed as: Entity Coverage Score = B.1.2

|mentioned entities| |total entities|

(7)

Helpfulness

We assess the semantic quality and practical utility of each documentation through an LLM-as-judge protocol, following the design adopted in DocAgent [1]. The judge rates each required section on a 5-point Likert scale using a section-specific rubric, and the normalized section scores are averaged into a single Helpfulness score: Helpfulness =

Score − 1 ∈ [0, 1] 4

(8)

We use Claude Haiku 4.5 (Anthropic API, temperature = 0.0, max tokens = 256) as the judge model. The section-specific rubrics in each granularity (component, module, and repository) are described in Table 17, Table 18, and Table 19, respectively. B.1.3

Truthfulness

Truthfulness measures whether the content of the generated documentation faithfully reflects the actual source code. It is decomposed into two sub-scores: Consistency, which measures factual correctness with respect to the code, and Relevance, which measures the proportion of code-relevant content within the documentation. We adopt a fine-grained documentation evaluation framework that decomposes generated documentation and evaluates each segment using predefined evaluation criteria [59, 60] to compute above two scores. Step1: Claim extraction. From each documentation, we extract natural-language sentences while excluding non-assertive elements (section headers, directory trees, code blocks or diagrams). The remaining text is decomposed into atomic claims following the FactScore [60]. Claim extraction is performed by GPT-4o-mini (temperature = 0.0, max tokens = 1024). Step2: Reference context retrieval. For each documentation target, we collect the relevant source code and its dependency context as the reference, using a granularity-specific strategy: • Component: The target source code together with the source of its dependency components. • Module: The source code of all components contained in the module. 33

Table 17: Component-level helpfulness evaluation criteria. Args and Returns apply to functions and methods only; Summary and Description apply to all component types (function, method, class). Aspect

Score

Summary

1 2 3 4 5

Merely restates the component name; no additional value. Adds one minor detail but still vague about purpose. Some context about what it does, but WHY/WHEN is unclear. Clearly conveys WHAT and WHY in practical terms. Perfectly balances conciseness and purpose; developer immediately understands when/why to use this.

1 2 3

Only repeats what the code does line-by-line; no context. Mentions callers or motivation superficially. One of WHERE/WHEN or WHY is well-explained but the other is missing. Both addressed clearly. Both addressed with specific detail; a new developer would understand the design rationale immediately.

Description

4 5

Args

1 2 3 4 5

Returns

1 2 3 4 5

Criterion

Just restates parameter types already visible from signature. Adds minimal description; constraints/allowed values missing. Describes purpose of most params but lacks constraints or edge cases. Each param has purpose, type, and constraints; interdependencies noted. Comprehensive: every param’s purpose, constraints, interdependencies, and edge values documented. Only restates the return type annotation. Brief description; edge-case return values missing. Normal return value explained; some edge cases missing. All return values and their meanings explained, including edge cases. Every possible return value, meaning, and edge condition documented.

• Repository: A name-index–based dynamic retrieval, where component and module names appearing in each claim are matched against the repository index to retrieve the relevant files on demand. Step3: Claim-level fine-grained evaluation. Each atomic claim is evaluated as one of three categories: consistent, inconsistent, or irrelevant, conditioned on the target source code with related information. Evaluation is performed by Claude Haiku 4.5 (temperature = 0.0, max tokens = 16), separately from the extraction model, so that the extraction and judgment LLMs are decoupled. Step4: Calculating truthfulness. The Consistency score therefore measures the proportion of verifiable claims (excluding off-topic content) that are factually supported by the source code, while the Relevance score measures the proportion of the documentation that is on-topic and amenable to verification at all. The two sub-scores are then combined multiplicatively: Consistent Consistent + Inconsistent

(9)

Irrelevant Consistent + Inconsistent + Irrelevant

(10)

Consistency = Relevance = 1 −

Truthfulness = Consistency × Relevance ∈ [0, 1] B.1.4

(11)

Information sufficiency

Figure 6 illustrates the overall pipeline of our information sufficiency evaluation. DevEval [61] provides human-curated unit tests aligned with reference code, enabling functional verification at the function level. From this resource, we collect 564 test cases targeting 312 component-level functions across our 20 benchmark repositories, and use them as the evaluation set. The evaluation proceeds in four stages. 34

Table 18: Module-level helpfulness evaluation criteria. Aspect

Score 1 2 3 4

Role

Just restates the module name or directory path. Vague purpose; no distinction from other modules. Clear responsibility described, but scope boundaries unclear. Precisely describes the one thing this module owns, clearly distinct from neighboring modules. Single-responsibility stated so clearly that a developer can immediately decide whether to use this module.

5

Description

Public API

Criterion

1 2 3 4 5

Neither primary consumers nor cohesion principle addressed. One is mentioned superficially. One of consumers or cohesion is well-explained. Both addressed clearly. Both addressed with specific detail; developer understands the architectural motivation.

1 2 3 4

Just lists names with no descriptions. Minimal descriptions; usage notes missing. Most symbols described; some usage notes present. All public symbols with signatures, brief descriptions, and usage notes. Complete API reference with guidance on which symbol to use for which task.

5

Table 19: Repository-level helpfulness evaluation criteria. Aspect

Score

Purpose

1 2 3 4 5

Architecture

1 2 3 4 5

Core Features

1 2 3 4 5

Criterion Vague; could describe any software project. Names the domain but not the specific problem solved. Clear problem statement, but target users or ecosystem position unclear. Problem, target users, and position in ecosystem all described. A new contributor can immediately understand what, who, and why. No diagram or abstract description only. High-level description without data flow. Data flow described in text; diagram absent or incomplete. End-to-end data flow diagram present with key abstractions identified. Complete architecture diagram with flow + abstractions + key design patterns. Just a list of module names without descriptions. Feature descriptions are vague. Most features described with implementing modules. Each feature has a clear one-line description and implementing module(s). Feature-module mapping complete and clear; a developer can navigate to any feature immediately.

Step1: Code body removal. We remove the body of each target component-level function, retaining only its signature. Step2: Documentation context injection. A code-generation LLM is provided with the function signature together with documentation context drawn from the hierarchical documentation tree, and we consider three progressively richer configurations. C provides only the component’s own documentation d(c) ; C+M additionally includes the parent module documentation d(m) ; and C+M+R further adds the repository-level documentation d(r) , yielding the full hierarchical context {d(c) , d(m) , d(r) }. This design isolates the contribution of each documentation level to the model’s ability to reconstruct the original function. Step3: Function regeneration. Given the injected context, the LLM regenerates the body of the function. 35

Original Source Code

Repository

Test Cases

def section_by_title( self, title: str, ) -> Optional[WikipediaPageSection]:

Wikipedia-API

if not self._called["extracts"]: self._fetch("extracts")

evaluation eval.py

sections = self._section_mapping.get(title)

tests/extract_html_format_test.py::TestHtmlFormatExtracts:: test_subsubsection test_subsection_by_title_return_last test_with_erroneous_edit test_subsection_by_title_with_multiple_spans test_subsection_by_title

if sections:

parser.py

return sections[-1]

…

return None

Hierarchical Documentation repo.md module1.md comp1.md comp2.md

Step1: Code body removal

Three Types [C]

{Comp2.md}

Step3: Code Regeneration

[C+M]

def section_by_title( self, title: str, )

{Comp2.md + module1.md} [C+M+R]

…

{Comp2.md + module1.md + repo.md}

-> Optional[WikipediaPageSection]: def section_by_title(self, title: str,) -> Optional[WikipediaPageSection]:

if not self._called["extracts"]: self.fetch("extracts")

Step4. Evaluation

Pass@k Test Cases via PyTest

sections = self.mapping.get(title)

{Document}

if sections: return sections[-1]

Step2: Documentation context injection

return None

CodeBLEU With Original Code

Figure 6: Overview of evaluating information sufficiency. Step4: Evaluation. The regenerated function is reinserted into its original location in the repository, after which we (i) execute the corresponding unit tests to verify functional correctness (Pass@k) and (ii) measure the structural similarity between the regenerated and the original source code (CodeBLEU). B.2

Baselines

We compare MemDocAgent against six baselines spanning two categories: open-source documentation systems whose internal pipelines are accessible for measurement, and closed-source commercial systems that operate as black boxes. Open-source baselines. • Prompting: A non-agentic baseline that prompts an LLM once with the target component and its first-hop dependencies as context, generating documentation in a single forward pass without iterative refinement or cross-document verification. We apply this baseline independently at the component, module, and repository levels. • RepoAgent [3]: An agentic system that generates function and class-level docstrings by traversing the repository’s dependency graph constructed via static analysis. While effective for component-level cataloging, it does not synthesize parent module or repository-level documentation from the underlying components. • DocAgent [1]: A multi-agent framework consisting of Reader, Searcher, Writer, Verifier, and Orchestrator agents. It traverses the code dependency graph in topological order and generates component-level docstrings in a bottom-up manner. Since each component is processed in an isolated context, cross-document consistency with previously generated documents is not explicitly enforced. • CodeWiki [2]: A wiki-style documentation system that produces hierarchical pages at the module and repository levels by clustering related components into structured wiki articles. It targets high-level overviews and does not generate fine-grained component-level documentation. Closed-source baselines. • DeepWiki [13]: A commercial system widely used in industry for automated repositorylevel documentation. As the system’s internal pipeline is not publicly accessible, we evaluate only its final output. • Claude Code [14]: A command-line coding assistant. We treat the README.md produced by its /init command as the repository-level documentation. 36

B.3

Datasets

Table 20 summarizes the statistics of the 20 repositories used in our evaluation. Table 20: Repository statistics across 20 repositories spanning 10 domains. #Test fn. and #Test cases indicate the number of component-level functions covered by human-curated unit tests and the total number of test cases collected from DevEval [61], respectively. # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Domain Communications Database Internet Multimedia Scientific-Engineering Security Software-Development System Text-Processing Utilities

Repository

Files

Classes

Methods

Functions

Components

#Test fn.

#Test cases

IMAPClient Wikipedia-API bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

28 4 10 42 37 44 92 43 25 47 9 14 8 194 10 29 8 20 9 17

13 5 1 53 22 34 11 56 28 47 1 9 10 125 15 52 11 13 6 24

36 47 30 174 84 209 45 373 107 74 1 43 37 209 35 117 49 17 32 54

35 7 2 98 63 67 52 240 31 70 40 35 17 342 26 38 19 58 17 28

84 59 33 325 169 310 108 669 166 191 42 87 64 676 76 207 79 88 55 106

33 13 19 42 17 30 11 52 4 3 18 11 3 17 16 4 3 3 7 6

71 46 27 59 40 65 16 60 12 9 25 28 3 25 23 15 3 13 11 13

690

536

1,773

1,285

3,594

312

564

Total

B.4

Implementation details

Backbone models. All experiments on open-weight models are served through a vLLM server running on two NVIDIA RTX Pro 6000 GPUs. Closed-weight models are accessed via their official APIs. We instantiate MemDocAgent with two backbones to verify that the framework is not tied to a particular model family: Qwen3-Coder-30B-A3B (temperature: 0.1, max input tokens: 32000, max output tokens: 4096) as an open-weight backbone, and GPT-5-mini (temperature: 1.0, max input tokens: 100000, max output tokens: 4096) as a closed-weight backbone. Both backbones share the same agent scaffolding, prompts, and hyperparameters; only the underlying generator differs. MemDocAgent configuration. External knowledge retrieval is delegated to the Perplexity API, which is invoked only when local repository evidence is insufficient. We use the Perplexity API (https://www.perplexity.ai/) for external query requests. The sub-task loop is capped at 10 steps, the verify loop at 2 steps, and the verify acceptance threshold is set to 0.9. All remaining hyperparameters follow the default decoding settings of each backbone.

C

Additional experimental results

C.1

Repository-wise experimental results

While Section 6.1 reports the overall documentation quality averaged across the 20 repositories, Tables 21, 22, and 23 present the per-repository breakdown of completeness, truthfulness, and helpfulness, respectively, comparing MemDocAgent against the open-source baselines. MemDocAgent consistently outperforms the baselines across all three criteria, and its advantage holds uniformly across repositories regardless of domain or scale. 37

Table 21: Per-repository Completeness scores across 20 repositories. Top: Qwen3-Coder-30B-A3BInstruct backbone. Bottom: GPT-5-mini backbone. Best score per row in bold. Domain

Repository

Prompting

RepoAgent

DocAgent

CodeWiki

MemDocAgent

(a) Qwen3-Coder-30B-A3B-Instruct Communications Database Internet Multimedia Scientific-Eng. Security Software-Dev. System Text-Processing Utilities

Wikipedia-API IMAPClient bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

Average

0.807 0.827 0.892 0.894 0.847 0.846 0.844 0.800 0.831 0.839 0.772 0.832 0.797 0.855 0.843 0.839 0.868 0.856 0.803 0.870

0.809 0.837 0.823 0.847 0.781 0.856 0.770 0.851 0.787 0.838 0.750 0.845 0.779 0.830 0.770 0.843 0.784 0.825 0.789 0.843

0.951 0.835 0.892 0.858 0.855 0.906 0.869 0.777 0.825 0.599 0.878 0.873 0.870 0.815 0.867 0.838 0.907 0.862 0.796 0.829

0.557 0.550 0.633 0.624 0.602 0.600 0.664 0.612 0.676 0.613 0.675 0.632 0.690 0.660 0.699 0.621 0.567 0.580 0.622 0.568

0.998 0.978 0.974 0.972 0.985 0.976 0.987 0.971 0.970 0.966 0.943 0.990 0.987 0.989 0.979 0.985 0.989 0.989 0.962 0.983

0.838

0.813

0.845

0.622

0.979

(b) GPT-5-mini Communications Database Internet Multimedia Scientific-Engineering Security Software-Development System Text-Processing Utilities

Wikipedia-API IMAPClient bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

Average

C.2

0.630 0.634 0.645 0.615 0.572 0.568 0.544 0.568 0.825 0.784 0.330 0.622 0.558 0.626 0.669 0.715 0.646 0.535 0.784 0.763

0.809 0.770 0.828 0.783 0.781 0.793 0.770 0.789 0.787 0.765 0.750 0.777 0.779 0.760 0.770 0.776 0.784 0.724 0.789 0.781

0.925 0.848 0.859 0.881 0.866 0.914 0.882 0.893 0.834 0.684 0.896 0.900 0.864 0.843 0.717 0.873 0.901 0.897 0.874 0.839

0.746 0.780 0.710 0.794 0.764 0.768 0.676 0.765 0.797 0.827 0.749 0.807 0.829 0.729 0.854 0.646 0.841 0.769 0.877 0.819

0.979 0.947 0.958 0.952 0.938 0.973 0.960 0.952 0.960 0.954 0.902 0.966 0.959 0.982 0.955 0.958 0.967 0.942 0.973 0.974

0.632

0.778

0.860

0.777

0.958

Scalability analysis

Table 24 reports how each system’s performance shifts as repository size increases. We sort the 20 repositories by file count and split them into a smaller half (Stage 1) and a larger half (Stage 2), then compare performance across the two splits. The open-source baselines (Prompting, RepoAgent, DocAgent) consistently degrade across all four metrics, with vanilla Prompting showing the largest drop in Effectiveness (-0.086), indicating that single-shot long-context generation and stateless multi-agent decomposition both fail to capture cross-file dependencies as repositories grow. In contrast, MemDocAgent records the smallest absolute change |∆| on every metric and even shows slight improvements on the three documentationquality measures (Completeness +0.001, Truthfulness +0.010, Helpfulness +0.006). MemDocAgent maintains documentation quality even as the repository grows larger, showing that memory-guided hierarchical documentation scales reliably to large repositories. 38

Table 22: Per-repository Truthfulness scores across 20 repositories. Top: Qwen3-Coder-30B-A3BInstruct backbone. Bottom: GPT-5-mini backbone. Best score per row in bold. Domain

Repository

Prompting

RepoAgent

DocAgent

CodeWiki

MemDocAgent

(a) Qwen3-Coder-30B-A3B-Instruct Communications Database Internet Multimedia Scientific-Eng. Security Software-Dev. System Text-Processing Utilities

Wikipedia-API IMAPClient bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

Average

0.680 0.657 0.679 0.623 0.645 0.593 0.661 0.547 0.657 0.668 0.606 0.542 0.766 0.584 0.679 0.698 0.755 0.648 0.648 0.640

0.784 0.812 0.838 0.808 0.811 0.831 0.863 0.801 0.864 0.847 0.822 0.864 0.883 0.837 0.747 0.900 0.892 0.814 0.841 0.837

0.782 0.804 0.766 0.866 0.722 0.774 0.849 0.792 0.880 0.822 0.734 0.878 0.724 0.824 0.710 0.880 0.802 0.857 0.779 0.758

0.823 0.882 0.926 0.916 0.542 0.423 0.783 0.781 0.970 0.831 0.509 0.310 0.850 0.852 0.970 0.634 0.947 0.782 0.872 0.834

0.919 0.920 0.913 0.940 0.963 0.876 0.945 0.845 0.950 0.875 0.934 0.897 0.894 0.932 0.816 0.941 0.960 0.962 0.944 0.897

0.649

0.835

0.800

0.772

0.916

(b) GPT-5-mini Communications Database Internet Multimedia Scientific-Eng. Security Software-Dev. System Text-Processing Utilities

Wikipedia-API IMAPClient bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

Average

C.3

0.721 0.787 0.772 0.794 0.777 0.731 0.671 0.671 0.737 0.818 0.810 0.708 0.764 0.734 0.718 0.702 0.750 0.711 0.743 0.707

0.858 0.791 0.823 0.881 0.877 0.815 0.867 0.798 0.901 0.863 0.864 0.888 0.875 0.840 0.835 0.825 0.870 0.877 0.835 0.821

0.878 0.847 0.873 0.875 0.849 0.788 0.826 0.795 0.827 0.837 0.811 0.817 0.838 0.833 0.783 0.869 0.896 0.844 0.785 0.754

0.752 0.312 0.544 0.578 0.462 0.623 0.882 0.500 0.695 0.833 0.537 0.855 0.823 0.500 0.824 0.839 0.713 0.500 0.684 0.600

0.956 0.943 0.972 0.992 0.944 0.977 0.929 0.922 0.949 0.989 0.968 0.937 0.958 0.984 0.964 0.984 0.942 0.901 0.933 0.895

0.741

0.850

0.831

0.653

0.952

Details of information sufficiency

Figure 3 compares the average results across two code-generation models, GPT-4o-mini and Claude Haiku 4.5, and Table 25 provides the full per-model results. C.4

Efficiency analysis

We further conduct efficiency analysis through Table 26 and Figure 7. On both Qwen3-Coder30B-A3B-Instruct and GPT-5-mini, MemDocAgent achieves a lower average generation time per document than DocAgent (53.16s vs.60.86s and 56.72s vs.69.89s, respectively), while producing the most complete documentation hierarchy among all baselines (3,323 and 3,497 documents covering all three levels). API cost per document on GPT-5-mini is also marginally lower than DocAgent ($0.0215 vs.$0.0219). Figure 7 confirms that this efficiency gain is not driven by a few favorable repositories: MemDocAgent consistently reduces both read time and the number of read calls per component across all 20 repositories. We attribute this consistent improvement to RepoMemory, which avoids redundant source-file retrievals by reusing verified dependency documents, replacing repeated rawcontext reads with focused memory lookups. Together, these results show that MemDocAgent attains 39

Table 23: Per-repository Helpfulness scores across 20 repositories. Top: Qwen3-Coder-30B-A3BInstruct backbone. Bottom: GPT-5-mini backbone. Best score per row in bold. Domain

Repository

Prompting

RepoAgent

DocAgent

CodeWiki

MemDocAgent

(a) Qwen3-Coder-30B-A3B-Instruct Communications Database Internet Multimedia Scientific-Eng. Security Software-Dev. System Text-Processing Utilities

Wikipedia-API IMAPClient bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

Average

0.474 0.406 0.370 0.481 0.394 0.568 0.554 0.550 0.539 0.441 0.397 0.594 0.453 0.459 0.398 0.301 0.435 0.502 0.308 0.538

0.356 0.390 0.369 0.393 0.363 0.385 0.362 0.354 0.404 0.460 0.354 0.380 0.366 0.384 0.435 0.436 0.393 0.396 0.383 0.454

0.613 0.613 0.623 0.609 0.542 0.529 0.606 0.348 0.620 0.651 0.610 0.516 0.541 0.506 0.562 0.516 0.548 0.521 0.501 0.553

0.250 0.212 0.750 0.550 0.533 0.523 0.583 0.521 0.667 0.682 0.733 0.724 0.667 0.523 0.500 0.423 0.667 0.745 0.583 0.521

0.686 0.632 0.787 0.682 0.601 0.692 0.674 0.623 0.669 0.769 0.723 0.745 0.648 0.782 0.606 0.612 0.720 0.768 0.609 0.780

0.458

0.391

0.556

0.568

0.690

(b) GPT-5-mini Communications Database Internet Multimedia Scientific-Eng. Security Software-Dev. System Text-Processing Utilities Average

Wikipedia-API IMAPClient bplustree datasette Jinja2 sumy hypertools mingus csvkit folium zxcvbn-python trailscraper PySnooper ydata-profiling exodus-bundler flower parsel online-judge-tools mackup PyJWT

0.583 0.634 0.592 0.618 0.627 0.592 0.598 0.586 0.618 0.679 0.613 0.608 0.632 0.618 0.611 0.625 0.631 0.647 0.594 0.638

0.413 0.427 0.402 0.455 0.421 0.446 0.426 0.406 0.471 0.505 0.431 0.434 0.449 0.460 0.488 0.504 0.449 0.463 0.460 0.468

0.639 0.722 0.568 0.713 0.701 0.687 0.688 0.559 0.637 0.710 0.727 0.633 0.703 0.756 0.616 0.655 0.657 0.688 0.680 0.674

0.817 0.583 0.650 0.750 0.650 0.743 0.750 0.667 0.650 0.750 0.667 0.833 0.750 0.733 0.633 0.693 0.633 0.717 0.750 0.733

0.851 0.762 0.793 0.757 0.764 0.793 0.864 0.722 0.733 0.851 0.787 0.850 0.762 0.862 0.866 0.749 0.762 0.793 0.823 0.861

0.617

0.449

0.671

0.708

0.800

the broadest documentation coverage while remaining more time- and cost-efficient than the strongest agentic baseline. C.5

More examples of generated documentation

Figures 8 and 9 show example screenshots of documentation generated by MemDocAgent.

40

Table 24: Scalability analysis under Qwen3-Coder-30B-A3B-Instruct backbone: performance change from Stage 1 (10 small-sized repositories) to Stage 2 (10 large-sized repositories), with ∆ = S2 − S1 . For each metric, the smallest absolute change |∆| is in bold. Completeness

System Prompting RepoAgent [3] DocAgent [1] MemDocAgent

Truthfulness

Helpfulness

Effectiveness (CodeBLEU)

S1

S2

∆

S1

S2

∆

S1

S2

∆

S1

S2

∆

0.620 0.785 0.862 0.954

0.603 0.775 0.857 0.955

−0.017 −0.010 −0.005 +0.001

0.761 0.872 0.827 0.741

0.734 0.857 0.814 0.750

−0.027 −0.015 −0.013 +0.010

0.625 0.462 0.687 0.751

0.610 0.441 0.645 0.756

−0.015 −0.021 −0.042 +0.006

0.423 0.313 0.209 0.282

0.337 0.298 0.156 0.268

−0.086 −0.015 −0.053 −0.014

Table 25: Information sufficiency evaluation via downstream code reproducibility GPT-4o-mini Method

Pass@1

Pass@3

Claude Haiku 4.5

CodeBLEU

Pass@1

Pass@3

Average

CodeBLEU

Pass@1

Pass@3

CodeBLEU

0.218 0.281

0.131 0.175

0.163 0.222

0.180 0.251

0.341 0.335 0.289 0.403 0.441 0.446

0.209 0.292 0.231 0.404 0.461 0.461

0.261 0.305 0.242 0.424 0.469 0.467

0.327 0.312 0.269 0.369 0.409 0.419

0.408 0.351 0.204 0.472 0.479 0.477

0.334 0.349 0.262 0.419 0.441 0.469

0.347 0.359 0.271 0.422 0.458 0.492

0.379 0.348 0.190 0.423 0.428 0.428

Reference baselines No docs Ground-truth

0.107 0.151

0.129 0.191

0.142 0.221

C C C C C+M C+M+R

0.156 0.231 0.158 0.273 0.354 0.349

0.196 0.253 0.168 0.297 0.362 0.356

0.314 0.290 0.249 0.335 0.376 0.392

C C C C C+M C+M+R

0.226 0.345 0.265 0.390 0.401 0.428

0.233 0.359 0.279 0.395 0.411 0.430

0.350 0.345 0.175 0.373 0.376 0.379

0.154 0.199

0.198 0.253

Backbone: Qwen3-Coder-30B-A3B-Instruct Prompting RepoAgent DocAgent MemDocAgent

0.262 0.352 0.305 0.535 0.568 0.573

0.325 0.357 0.315 0.550 0.575 0.578

Backbone: GPT-5-mini Prompting RepoAgent DocAgent MemDocAgent

0.442 0.352 0.259 0.448 0.480 0.509

0.461 0.359 0.263 0.449 0.505 0.553

Table 26: Document generation time and cost analysis across baselines. Component, Module, and Repo report the number of documents produced at each level of the documentation hierarchy. Time/doc the average per-document generation time and Cost/doc the average API cost per document. Baseline

Component Module Repo Total docs Runtime (s) Time/doc (s) Total cost ($) Cost/doc ($) Generation model: Qwen3-Coder-30B-A3B-Instruct

ChatLLM RepoAgent DocAgent CodeWiki MemDocAgent

3,226 3,650 3,198 0 3,226

77 0 0 60 77

20 0 0 20 20

3,323 3,650 3,198 80 3,323

39,780 11,340 194,640 7,740 176,640

11.97 3.11 60.86 96.75 53.16

– – – – –

– – – – –

30.35 5.98 69.89 48.75 56.72

30.35 5.98 69.89 48.75 75.25

0.0091 0.0016 0.0219 0.6094 0.0215

Generation model: GPT-5-mini ChatLLM RepoAgent DocAgent CodeWiki MemDocAgent

3,226 3,650 3,198 0 3,410

77 0 0 60 67

20 0 0 20 20

3,323 3,650 3,198 80 3,497

100,860 21,840 223,500 3,900 198,360

41

Read calls / component Read time / component (s)

DocAgent

MemDocAgent

30

DA avg 23.44

20

MEM avg 13.76

10 0 5 4

DA avg 3.79

3

MEM avg 2.62

2 1 0

l I t t r r r r -AP tree Jinja2 rtools csvki ython oope undle parse ackup Clien asette sumy ingus folium crape ofiling flowe -tools PyJWT dia bplus e P dat m s a-pr -p ySn s-b m ge l p A i n d y a M b h I P odu v tr dat e-ju n y zxc i l ex on

ipe Wik

Figure 7: Per-repository read time (s) and read calls per component for DocAgent (DA) and MemDocAgent (MEM). Dashed lines mark the averages across 20 repositories.

42

Figure 8: Repo, Module, Component-level documentation example on Sumy Repository 43

Figure 9: Repo, Module, Component-level documentation example on Datasette Repository 44