Natural Language-Focused Software Engineering via Code-Documentation Equivalence
arXiv:2606.22247v1 [cs.SE] 20 Jun 2026
ARYAZ EGHBALI, CISPA Helmholtz Center for Information Security, Germany ZHONGXIN LIU, Zhejiang University, China MICHAEL PRADEL, CISPA Helmholtz Center for Information Security, Germany Source code documentation is an integral part of software development and maintenance, as it helps in understanding the code and facilitates communication among developers. However, existing documentation is often incomplete, outdated, or inaccurate, which can lead to misunderstandings and errors. In the era of large language models (LLMs), which are being extensively used for software engineering tasks, the quality of documentation becomes even more critical, as documentation provides important context for the models. In this paper, we introduce the notion of documentation-to-code equivalence, a novel property that captures whether documentation accurately and completely describes the code it documents. We present a novel approach, called Documentary, to automatically generate equivalent documentation for a given code snippet. Our evaluation shows that Documentary can generate equivalent documentation for 53.4% of the evaluated function-level code snippets. To show the benefits of documentation-to-code equivalence, we describe and evaluate two software engineering tasks: code understanding and code editing. Our results show that documentation-to-code equivalence allows an LLM to predict the output of a function with 12.8–24.5% higher accuracy, when compared to human-written documentation and documentation generated by a baseline approach. Furthermore, human developers consider documentation generated by Documentary to be more useful for understanding and editing code than the original human-written documentation. CCS Concepts: • Software and its engineering → Documentation; Software maintenance tools. Additional Key Words and Phrases: documentation, LLM, equivalence ACM Reference Format: Aryaz Eghbali, Zhongxin Liu, and Michael Pradel. 2026. Natural Language-Focused Software Engineering via Code-Documentation Equivalence. Proc. ACM Softw. Eng. 3, FSE, Article FSE204 (July 2026), 22 pages. https://doi.org/10.1145/3808211
1
Introduction
Source code, written in formal programming languages, is often difficult to read and understand, especially for complex codebases or when the code is written by someone else. Hence, most programming languages support natural language documentation in the form of comments and docstrings, allowing developers to describe code intent more clearly. Such documentation has been shown to be beneficial in improving code comprehension and facilitating collaboration among developers [12, 15]. However, time constraints and the continuous evolution of code can lead to documentation that is incomplete, outdated, or inaccurate [10, 12, 15, 17]. Such documentation, or simply no documentation at all, can hinder the understanding of the code and delay the development process. Authors’ Contact Information: Aryaz Eghbali, CISPA Helmholtz Center for Information Security, Stuttgart, Germany, [email protected]; Zhongxin Liu, Zhejiang University, Hangzhou, China, [email protected]; Michael Pradel, CISPA Helmholtz Center for Information Security, Stuttgart, Germany, [email protected].
This work is licensed under a Creative Commons Attribution 4.0 International License. © 2026 Copyright held by the owner/author(s). ACM 2994-970X/2026/7-ARTFSE204 https://doi.org/10.1145/3808211 Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:2
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
... def pbr(self) -> None: 741 "Pretty print has a line break" 742 if self.p_p == 0: 743 self.p_p = 1
739 740
744 745 746 747
def p(self) -> None: "Set pretty print to 1 or 2 lines" self.p_p = 1 if self.single_line_break else 2
748 749 750 751 752
def soft_br(self) -> None: "Soft breaks" self.pbr() self.br_toggle = " "
753
def o(self, data: str, puredata: bool = False, force: Union[bool, str] = False) -> None: """ 756 Deal with indentation and whitespace 757 """ 758 if self.abbr_data is not None: 759 self.abbr_data += data 760 ... 754 755
Fig. 1. Part of the source code from the Alir3z4/html2text repository.
As an example of suboptimal documentation, consider the function soft_br from the repository Alir3z4/html2text, shown in Fig. 1. This function is only four lines long, yet a developer or a large
language model (LLM) unfamiliar with the codebase would struggle to understand its purpose and how it works without spending time or tokens reading other parts of the codebase. For example, the function calls the pbr function, which itself sets some state in the object, and the effect of this state change is not clear without reading the code of other functions that use this state. The soft_br function also modifies the br_toggle attribute, which is used in other parts of the code to determine how to format line breaks, and this is equally unclear without reading those functions. Since this is only a short excerpt from a file exceeding 1,000 lines, identifying the relevant context is time-consuming. Prior work on improving documentation focuses either on summarizing code [1, 4, 14, 16, 27, 28, 31] or on ensuring that documentation remains consistent with the code during software evolution [17, 18, 22, 23]. While useful, these approaches do not ensure that the documentation is complete and accurate. A summary typically omits details, and hence, does not ensure completeness. Consistent documentation only guarantees soundness—it does not contradict the code—but makes no claim about completeness. To illustrate the limitations of summarization and consistency, again consider the function soft_br in Fig. 1. The docstring of the function is consistent with the code, and the function is short enough to not require any summarization. However, the main limitation is that the docstring does not provide a complete description of the function’s behavior, particularly in terms of its side effects and interactions with other parts of the code. Another limitation of summarization-focused and consistency-focused approaches is that they do not provide a clear and objective way to evaluate the quality of the documentation. The reason is that one side of the evaluation of the documentation-code relation is in natural language, and therefore unstructured and vague. Prior work in these areas [16–18] has focused on evaluating metrics that measure how close the documentation is to human-written documentation. However, such metrics of textual Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:3
Table 1. Comparison of documentation-code properties.
Doc-code property Consistency [11] Summarization [4] Equivalence (this work)
Sound
Complete
Evaluation of property
✓ ✓ ✓
✗ ✗ ✓
Code vs. natural language Code vs. natural language Code vs. code
similarity may not accurately capture the semantic similarity of documentation and are limited because human-written documentation is often incomplete. Yet, because of the lack of an objective measure for documentation quality, comparison to human-written documentation has become the de facto standard. To address the limitations of prior work, we introduce a novel property called documentationto-code equivalence, which captures whether a piece of documentation accurately and completely describes the corresponding code. Informally, this property means that the documentation contains enough information to re-implement the same functionality as the code that it documents (see Section 2.1 for a formal definition). This is evaluated by checking whether an LLM can generate code that is equivalent to the original code, based on the documentation and the surrounding context. Table 1 summarizes the differences between the existing properties of documentation-code relations and our proposed property of documentation-to-code equivalence. A key difference is that, in addition to ensuring the soundness of the documentation, i.e., that the documentation does not contradict the code, documentation-to-code equivalence also ensures the completeness of the documentation, i.e., that the documentation fully describes the code. Moreover, it enables the evaluation of the documentation-code relation by evaluating the equivalence of two code snippets, which is a more well-defined problem than textual similarity and can be evaluated automatically. Figure 8 shows an equivalent docstring for the same soft_br function from Fig. 1, which provides a comprehensive explanation of the function’s purpose, behavior, and interactions with other components of the codebase. While documentation-to-code equivalence is a strong property, existing documentation often does not satisfy it, as illustrated by the example above. To address this problem, we present a novel approach, called Documentary, to automatically generate equivalent documentation for a given code snippet. The key idea of Documentary is to iteratively refine the documentation by generating code from the documentation and comparing it to the original code, until the generated code is equivalent to the original code. To determine the equivalence of two code snippets, we use an LLM as a judge [2, 5], which allows our approach to be language- and platform-agnostic. We envision documentation-to-code equivalence to be a useful property for various software engineering tasks, performed either by humans or LLMs. In this paper, we explore two such tasks: code understanding and code editing. For code understanding, we consider the task of predicting the output of a function for a given input, based on the function signature and its documentation. For code editing, we consider the task of modifying code based on a natural language description of the desired change. Our evaluation applies our ideas to function-level docstrings of real-world Python projects. The results show that Documentary effectively generates equivalent docstrings for 53.4% of the functions in our dataset, as determined by running the tests. Regarding the code understanding task, we find that equivalent docstrings allow an LLM to predict the output of a function with 12.8–24.5% higher accuracy, when compared to human-written documentation and documentation Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:4
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
generated by a baseline approach. For the code editing task, we find that equivalent docstrings help developers understand the code better than the human-written documentation. These results demonstrate the benefits of documentation-to-code equivalence in improving the performance of LLMs and human developers in software engineering tasks. With the recent surge in the use of LLMs for software engineering tasks, documentation is now also used by LLM-based tools and agents. This means that the documentation should be useful not only for human developers, but also for LLMs. Furthermore, the medium of communication between developers and LLMs is to a large extent natural language. Therefore, we see documentation-tocode equivalence as an important step toward natural language-focused software engineering, i.e., performing software engineering tasks partially or fully based on natural language documentation as a significant source of information. In summary, this paper makes the following contributions: • A novel property (documentation-to-code equivalence) that describes a strong semantic relationship between code and its documentation. • An iterative approach, called Documentary, to automatically generate equivalent documentation for a given code snippet. • Empirical evidence of the benefits of equivalent documentation in two common software engineering tasks, code understanding and code editing. 2
Approach
We first define the novel concept of documentation-to-code equivalence as a property of a piece of documentation relative to a code snippet (Section 2.1). Then, we present our Documentary approach to automatically generate equivalent documentation for a given code snippet (Section 2.2). 2.1
Definition of Documentation-to-Code Equivalence
Suppose we have a piece of code 𝑐, a piece of documentation 𝑑 that is intended to document 𝑐, and some surrounding context ctx that provides additional information about the code, such as other functions, classes, or modules in the same file or project. Furthermore, let 𝐿 be a large language model (LLM) that can generate code from natural language prompts. The documentation-to-code equivalence property is defined as follows: The documentation 𝑑 is equivalent to the code piece 𝑐, with respect to an LLM 𝐿 and the surrounding context ctx, if 𝐿 can generate 𝑐 from (𝑑, 𝑐𝑡𝑥). If this property holds, we say that 𝑑 is an equivalent documentation for 𝑐. The above formulation is general and can be instantiated with different kinds of code, documentation, context, and LLMs. In the remainder of this paper, we focus on function-level docstrings as the documentation, function bodies as the code, and the whole source code file as the surrounding context. 2.2
Generating Equivalent Documentation
We present an automatic approach, called Documentary, to generate equivalent documentation. Given a function signature, its body, and the surrounding context, Documentary produces a docstring that is equivalent to the function body, according to the definition in Section 2.1. Figure 2 shows an overview of Documentary. Documentary starts by generating code from the existing documentation (if none exists, it uses only the surrounding context). Then, using the same LLM that generated the code as a judge [2, 5], the approach compares the generated code with the original code to determine whether they are equivalent. If they are not equivalent, the approach generates a brief description of the differences between the two code snippets. This description is then used to generate new documentation that is more specific and accurate. Documentary repeats Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:5
Fig. 2. Overview of the iterative approach to generate equivalent documentation.
this process until the generated documentation is equivalent to the given code snippet, i.e., the code generated from the documentation is judged to be equivalent to the original code, or until a maximum number of iterations is reached. This design is motivated by two observations about the capabilities of LLMs. First, we observe that LLMs are good at generating code and documentation, but they sometimes miss some important details or, because of missing context, their generated code or documentation may not be equivalent to the ground truth. The approach builds on this observation by leveraging LLMs to generate code and documentation in a roundtrip manner, which allows us to iteratively refine the documentation. Second, we observe that LLMs are good at comparing pieces of code and identifying semantic differences between them. Documentary leverages this ability to identify the differences between code generated from candidate docstrings and the original code, and to use any differences to guide the model to generate more complete and accurate documentation. 2.2.1 Core Algorithm. The core of our approach is an iterative process that generates and refines a docstring for a given function. Algorithm 1 summarizes this process. We describe the details of each part of the algorithm in the following sections. The first iteration of the main loop (lines 6–15) starts with generating the body of the function (described in more detail in Section 2.2.2) from the surrounding context and the existing docstring, if any. The reason for starting with the existing docstring is that it may already be equivalent to the code, in which case we can stop immediately. Even if the existing docstring is not equivalent, it can serve as a basis for generating a new docstring. The generated body is then compared with the original body to check if they are equivalent, which determines if we need to continue with the approach or can stop and return the docstring (line 10). The equivalence check is described in more detail in Section 2.2.3. If the generated body is not equivalent to the original body, we generate a brief description of the differences between the two code snippets, as described in Section 2.2.4. These differences help the model incorporate the missing information into the new docstring in the next step (line 18). The new docstring is generated based on the surrounding context, the original body, the generated body, the differences between the two bodies, and the previous docstring. This completes the main loop, with the generated docstring being used in the next iteration to generate a new body. The main loop runs for at most a configurable number of iterations (default: five), or until an equivalent docstring is found. If the approach reaches the maximum number of iterations but does Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:6
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
Algorithm 1 Algorithm to generate an equivalent docstring. 1: Input: function_header, function_body, surrounding_context, size_limit, max_iterations 2: Output: equivalent docstring or failure
3: docstring = original_docstring or "" 4: iterations = 0 5: equivalent = False 6: while iterations < max_iterations do ⊲ Iterative refinement of docstring 7: generated_body = generate_body(surrounding_context, function_header, docstring)
equivalent = check_equivalence(function_body, generated_body) if equivalent then return docstring end if differences = generate_differences(function_body, generated_body) docstring = generate_docstring(docstring, differences) 14: iterations += 1 15: end while 16: if not equivalent then ⊲ Final attempt to refine docstring 17: docstring = refine_docstring(docstring, function_body) 18: generated_body = generate_body(surrounding_context, function_header, docstring) 19: equivalent = check_equivalence(function_body, generated_body) 20: if equivalent then 21: return docstring 22: end if 23: end if 24: return failure 8: 9: 10: 11: 12: 13:
not find an equivalent docstring (line 16), the generated docstring may contain information that does not match the given implementation. For example, the docstring may mention checking for a certain exception, whereas the implementation does not check for that exception. To handle such cases, the algorithm includes a final refinement step (line 17). In this step, we prompt the LLM to remove any information from the generated docstring that does not correspond to any code in the correct implementation. After this step, we ask the LLM to do the first two steps of the main loop (generating the body, and evaluating the equivalence) once again (lines 18–19). If the generated body is judged to be equivalent to the correct implementation, the approach considers the docstring equivalent. Otherwise, the approach gives up and reports a failure. 2.2.2 Code Generation. The first step in the main loop is generating the body from the current candidate docstring, the function signature, and the surrounding context. To this end, Documentary prompts the LLM based on the template shown in Fig. 3. The docstring is initially the original docstring, i.e., the docstring written by the developer, if it exists, or an empty string otherwise. In subsequent iterations, Documentary uses the docstring generated in the previous iteration. The surrounding context includes the whole source code file, except for the code of the function being documented. Given the full file context, the LLM generates the body of the function soft_br, shown in Fig. 4, which is different from the original body shown in Fig. 1. Note that although the generated body is different, it may still be functionally equivalent to the original body. Therefore, we need to continue Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:7
In the file containing ```python 3 {context} 4 ``` 5 implement the body of the function 6 ```python 7 {function_header} 8 {docstring} 9 ``` 10 Only output the body, with correct indentation. Do not repeat the function signature or docstring. 1 2
Fig. 3. Template of the prompt for generating the function body.
1 2 3 4
if self.blockquote > 0: self.o("\n> ") else: self.o("\n")
Fig. 4. Generated body for the function soft_br from the original docstring.
with the next step, which is the equivalence check, to determine if the generated body is indeed equivalent to the original body. 2.2.3 Equivalence Check. To determine whether Documentary has achieved its goal of generating an equivalent docstring, the approach checks if the code generated from the current docstring is equivalent to the original code. This can be done in various ways, for example by executing both code snippets on a set of inputs and comparing the outputs, by using string matching, or by employing an LLM to judge their equivalence. String matching is too strict and can cause false negatives, as two code snippets can be equivalent while being syntactically different. Executing the code snippets is a more robust way to check for equivalence, but it requires a set of inputs that cover the behavior of the function, which may not be available, and are hard to generate automatically. Moreover, running an arbitrary piece of code is not always feasible [26]. Using an LLM as a judge is a more flexible approach that can be applied to any code snippet. It also allows the approach to be language and platform agnostic. Therefore, Documentary compares the generated code with the original code using an LLM as judge, which assesses whether the two code snippets are equivalent. To compare the two code snippets, Documentary prompts the LLM with a prompt, as shown in Fig. 5. The prompt contains both code snippets and asks the LLM to first describe the differences between the two snippets if there are any, and then answer whether the two snippets are equivalent or not. The prompt also asks the LLM to provide an example of input that results in different behavior in the two code snippets, if possible. The explanation of differences and input examples provide a chain-of-thought reasoning for the LLM, which has been shown to improve the quality of the responses [30]. Finally, the prompt directs the model to ignore any implicit errors and exceptions. This is because the LLM does not know about the domain of valid inputs for the function, and may consider two code snippets to be different if one of them raises an exception for an input that is not valid for the function. To avoid such false negatives, we ask the model to ignore implicit errors and exceptions. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:8
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
Are the following two code pieces equivalent? Correct implementation: 3 ```python 4 {function_header} 5 {function_body} 6 ``` 7 Alternative implementation: 8 ```python 9 {function_header} 10 {generated_body} 11 ``` 12 Briefly explain why the alternative implementation is or is not equivalent to the correct implementation, at a low level. 13 Phrase the differences as "The correct implementation checks for X, but the alternative implementation does not", "The correct implementation calls Y, but the alternative implementation calls Z", etc. 14 Only mention each difference once. 15 If possible, give an example case where they behave differently. 16 Ignore all implicit errors and exceptions. 17 If there are cases where they behave differently, output "DIFFERENT" at the end of the response. 18 Otherwise, if they behave exactly the same for all inputs, then they are equivalent, and you should output "EQUIVALENT" at the end of the response. 19 The output must contain one of "DIFFERENT" or "EQUIVALENT", never both. 1 2
Fig. 5. Template of the prompt for checking the equivalence of two code snippets.
If the LLM decides that the two code snippets are equivalent, the process is terminated and the docstring is considered equivalent to the code. Otherwise, we move to the next step, which is generating a new docstring. Fig. 6 shows the response from the LLM when comparing the generated body in Fig. 4 with the original body in Fig. 1. It first describes three differences between the two implementations, namely, calling different functions, modifying different state, and checking for a condition that the other implementation does not check for. Then, based on the last point of difference (the condition check), it provides an example of an input that results in different behavior in the two implementations. Finally, it outputs DIFFERENT, indicating that the two implementations are not equivalent. Since the verdict of Documentary is that the two implementations are not equivalent, the approach continues with the main loop, which will generate a new docstring. 2.2.4 Docstring Generation. If the generated code is not equivalent to the original code, our approach attempts to generate a new, revised docstring. The prompt includes the surrounding context, the developer’s implementation of the function, and the docstring and body generated in the previous iteration. We also calculate the diff of the correct and generated function body, so that the LLM can focus on the differences. In this prompt, as seen in Fig. 7, we ask the LLM to first describe the differences between the generated body and the correct body. As in the equivalence check, the LLM is prompted to explain the differences, providing chain-of-thought reasoning. We ask the LLM to then identify the part of the docstring responsible for the code that is different and modify the docstring to match the correct implementation. For differences that have no correspondence with the docstring, we ask the model to add the necessary information to the docstring. We also add instructions for the LLM to limit the length of the docstring to the provided size limit. We specify the size limit in terms of a Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
1
FSE204:9
Not equivalent.
2
- The correct implementation calls self.pbr(), but the alternative implementation calls self.o(...) to write a newline (and potentially a blockquote prefix in the alternative path). 5 - The correct implementation assigns self.br_toggle = \" \", but the alternative 6 implementation does not modify br_toggle. 7 - The correct implementation does not check self.blockquote, but the alternative 8 implementation checks if self.blockquote > 0 to decide between \"\\n> \" and \"\\n\".
3
4
9
Example where they differ: If self.blockquote is greater than 0, the alternative will output a line starting with \"> \" after the newline, whereas the correct version does not emit 12 that blockquote prefix (and may affect subsequent formatting relying on br_toggle or other 13 state). This can lead to different rendered output after soft_br(). 10 11
14 15
DIFFERENT
Fig. 6. Response from the LLM when comparing the generated body with the original body.
multiple of the original function length, which defaults to 1x, i.e., the same length as the original function. Given the prompt described above, the model generates a new docstring. Documentary checks whether the docstring adheres to the size limit. If the response exceeds the size limit, we prompt the LLM again to summarize the docstring to fit in the size limit. For the example function soft_br, the LLM generates the docstring shown in Fig. 8. This docstring now explains what the function does, and how it works. In the next iteration of the main loop, this docstring is used to generate a new body, which matches the original body. The equivalence check of Documentary also evaluates the two code pieces as equivalent, and so the docstring is returned. 3
Applications
We consider two common software engineering tasks—output prediction and code editing—and in our evaluation, show how equivalent documentation benefits both human developers and LLMs when performing these tasks. 3.1
Code Understanding via Output Prediction
The first task we consider is code understanding, where we use output prediction as a proxy task. In this task, given a function signature and its documentation, the goal is to predict the output of the function for a given input. This task reveals how much behavioral information the documentation contains about the function. Since many downstream software engineering tasks, such as test generation, bug detection, and code editing, require understanding the behavior of the code on a certain input, improvements in this task can also benefit those tasks. More formally, we define the task as follows. Given the signature of a function under test 𝑓 , its documentation 𝑑, and a test prefix 𝑝 that exercises 𝑓 and ends with an assertion statement with one missing value 𝑣, the goal is to predict 𝑣. An example of such a function is shown in Fig. 9, which is the constructor of a middleware class that reads an environment variable to set the allowed origins for CORS requests. An example of a test prefix that exercises this function is shown in Fig. 10, which tests the behavior of the function when the environment variable is not set. The test prefix ends with an assertion statement that checks the value of the allow_origins attribute, but the expected value is missing. The goal is to predict the missing value, which in this case is an empty tuple, or () in Python. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:10
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
In the file containing ```python 3 {context} 4 ``` 5 the correct implementation of the function 6 ```python 7 {function_header} 8 ``` 9 is 10 ```python 11 {function_header} 12 {function_body} 13 ``` 14 However, using the docstring 15 ```python 16 {new_docstring} 17 ``` 18 the body was generated as 19 ```python 20 {function_header} 21 {generated_body} 22 ``` 23 Here is the diff between the generated body (before) and the correct implementation (after): 24 ```diff 25 {diff} 26 ``` 27 First, explain all the ways in which the correct implementation is different from the generated body. 28 Then, for each difference, identify which part of the docstring is responsible for that difference (if any) and fix that part of the docstring to represent the correct implementation. 29 The docstring must not mention the generated body. 30 If there are any differences that are not related to any part of the docstring, add a sentence to the docstring explaining the correct implementation of that part, such as custom functions, specific checks, or other details that are not in the generated body. 31 Do not add any new information that is not in the correct implementation. 32 The wording of the docstring must be as if it is describing the correct implementation without mentioning the phrases like "correct implementation" or "correct body". 33 The docstring must be in natural language and start and end with triple double-quotes. 34 Output the docstring inside triple backticks with correct indentation (`{indentation}`). Do not include the function header or body. 35 Limit the new docstring to a maximum of {int(size_limit * orig_function_length)} lines. 1 2
Fig. 7. Template of the prompt for generating a new docstring. """Soft line breaks are produced by signaling a paragraph boundary and preparing a two-space continuation. 2 It first marks a paragraph break, then assigns br_toggle to two spaces, so the next line ends with two spaces before the newline. 3 This preserves Markdown-style line breaks within paragraphs.""" 1
Fig. 8. Generated docstring for the function soft_br after the first iteration.
Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
1 2 3 4 5 6 7 8
FSE204:11
def __init__(self, app: ASGIApp) -> None: allow_origins_str = os.getenv('PERMITTED_CORS_ORIGINS') if allow_origins_str: allow_origins = tuple(origin.strip() for origin in allow_origins_str.split(',')) else: allow_origins = () super().__init__(app, allow_origins=allow_origins allow_credentials=True, allow_methods=['*'],allow_headers=['*'],)
Fig. 9. Example of a function from the All-Hands-AI/OpenHands repository.
def test_localhost_cors_middleware_init_without_env_var(): """Test that the middleware works correctly without PERMITTED_CORS_ORIGINS environment variable.""" 3 with patch.dict(os.environ, {}, clear=True): 4 app = FastAPI() 5 middleware = LocalhostCORSMiddleware(app) 6 # Check that allow_origins is empty when no environment variable is set 7 assert middleware.allow_origins == 1 2
Fig. 10. Example of a test prefix that exercises the function in Fig. 9. The correct value for the incomplete assertion is (), i.e., an empty tuple.
1 2 3 4 5
+ + 8 + 9 + 6 7
10 11 12
if len(size) != 2: raise ValueError( "Expected `size` to be a tuple of 2 integers. " f"Received: size={size}" ) if size[0] <= 0 or size[1] <= 0: raise ValueError( f"`size` must have positive height and width. Received: size={size}" ) if len(images.shape) < 3 or len(images.shape) > 4: raise ValueError( "Invalid images rank: expected rank 3 (single image) "
Fig. 11. Example of a code change from the keras-team/keras repository.
3.2
Code Editing
As the second task, we consider code editing. This task represents a scenario in which a developer wants to modify a function based on a natural language description of the change. In this task, given a code snippet 𝑐𝑜𝑙𝑑 with its documentation 𝑑𝑜𝑙𝑑 and a change description 𝑚, the goal is to modify the code to 𝑐𝑛𝑒𝑤 . For example, consider commit c718248 in the keras-team/keras repository, which describes the change in the code as "Validate positive height and width in image resize". Figure 11 shows the change in function resize, where a check for negative dimensions is added that raises a ValueError if either dimension is negative. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:12
4
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
Evaluation
We evaluate our approach and the utility of the generated documentation in downstream tasks, addressing the following research questions: RQ1 How effective is our approach in generating equivalent docstrings? RQ2 How transferable are equivalent docstrings across different LLMs? RQ3 How do different components and parameters of the approach affect the generated docstrings? RQ4 What computational costs does the approach incur? RQ5 How much do equivalent docstrings improve LLM performance on the code understanding task? RQ6 How much do equivalent docstrings improve human performance on the natural languagebased code editing task? 4.1
Experimental Setup
4.1.1 Datasets. We use two datasets from previous work as the basis for our evaluations. The first dataset is CoDocBench [21], which is a dataset of pairs of code and docstring changes in the history of 203 popular Python projects on GitHub. The second dataset is DyPyBench [7], which is a dataset of 50 popular projects across various domains that have executable tests. From the union of these datasets (252 unique projects), we create three datasets for our evaluation. The first dataset, which we call the code-docstring equivalence dataset, is used for RQs1–4, where we want to evaluate the behavioral equivalence of the generated code piece with the human-written code. Therefore, we use test execution as the ground truth for behavioral equivalence. To this end, for each project in the union of the datasets, we try to run the test suite by installing the dependencies using the pipreqs package1 . From the executable projects, we take the functions that are covered at least 50% by the tests. This results in 2437 functions from 12 projects. We then sample at most 25 functions from each project, which results in a total of 238 functions. These functions range from 2 to 168 lines of code, with an average of 16 and median of 9 lines of code. Most functions lack docstrings; only 47% are documented. For RQ5, we use the same set of 252 projects described above. Then we extract the tests that have assertions with a comparison, where one side of the comparison is either a literal or an explicit collection, for example assert foo() == 42 or assertEqual(foo(), [1, 2, 3]). We then use the heuristic by Watson et al. [29] to find the function-under-test for each test. We exclude test cases where gpt-4.1-nano is not able to predict all the assertion values correctly given the implementation of the function under test, as we consider these cases too hard for the LLM to predict. This results in 253 test functions, with a total of 480 assertions, which we call the output prediction dataset. Finally, we compile a third dataset, which we call the user study dataset for RQ6. From five popular projects, namely, keras, marshmallow, celery, requests, and click, we select the latest commit (from keras the last two commits) where the code change is isolated to one function with fewer than 10 lines of change, the change is not trivial from the commit message, and there is a test that exercises the changed function. This process ensures that there is a need for understanding the code to perform the change, that we can use test execution to check the correctness of the change, and that the participants are less likely to have seen the change before. 4.1.2 Baseline. We implement a baseline for our evaluations, which is prompting the LLM to generate a docstring for the given function in a single step. We use the same LLM as in Documentary. 1 https://github.com/bndr/pipreqs
Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:13
In the file containing ``` 3 {context} 4 ``` 5 generate a Python docstring for function: 6 ``` 7 {function_code} 8 ``` 9 Only output the docstring with correct indentation. Do not repeat the function signature or the function body. 1 2
Fig. 12. The template of the prompt for the baseline approach.
The prompt, as shown in Fig. 12, includes the surrounding context, the function signature, and the function body. 4.1.3 LLMs and Hardware. We run all experiments on an Ubuntu 22.04 machine with Intel Xeon CPU with 48 cores running at 2.20GHz and 256GB of RAM. The experiments are run inside Python 3.12 Docker containers. We use the gpt-4.1-nano-2025-04-14 as the main LLM through the OpenAI API. 4.2
RQ1: Generating Equivalent Documentation
We run Documentary on each function in the dataset to obtain an equivalent docstring. For cases where Documentary fails to generate an equivalent docstring, we use the last generated docstring as its best effort. Then we prompt the LLM to generate the body of the function, based on the context, the function signature, and the generated docstring. Finally, we run the tests to check if the generated code behaves the same as the original code. If the test results of the generated code match the test results from the original code, we consider the generated docstring to be equivalent to the code. We report the success rate, and precision, which show how often Documentary generates a docstring equivalent to the code, and how often docstrings that Documentary claims to be equivalent are actually equivalent, respectively. As shown in Table 2, Documentary generates equivalent docstrings in 53.4% of the cases. When claiming that a generated docstring is equivalent to the code, then for 66.4% of these cases the generated docstrings are indeed equivalent to the code. Figure 13 shows how the length of the function, measured in lines of code (LoC), affects the success rate of generating equivalent documentation. Unsurprisingly, the success rate decreases as the function length increases, which is expected as longer functions are more complex and harder to describe in a docstring. However, Documentary consistently outperforms the baseline across all function lengths except one, where there is only one function that the baseline is able to generate equivalent docstring. 4.3
RQ2: Model Independence
To evaluate how useful equivalent documentation is across different models, we run an experiment where we generate equivalent documentation using one LLM, and then use that documentation to generate code using different LLMs. We then run the tests to check if the generated code is equivalent to the original code. If the test results match the test results from the original code, we consider the generated code to be equivalent to the original code. For the model used in Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:14
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
Fig. 13. Distribution of successful generation of equivalent documentation using Documentary and the baseline for different function lengths. Table 2. Effectiveness in generating equivalent documentation. Success shows how often the approach generates an equivalent documentation based on test execution. Total is the number of functions in the dataset, and Precision shows the ratio of true positives to all cases that Documentary claims to have generated equivalent documentation.
Approach Baseline Documentary
Success / Total (%) Precision 81 / 238 (34.0%) 34.0% 127 / 238 (53.4%) 66.4%
Documentary, we use gpt-4.1-nano, and we use gpt-5-nano, and gemini-2.5-flash-lite as the other LLMs used for generating code from the equivalent documentation. For each function, we first generate equivalent documentation using gpt-4.1-nano and our approach. Then, for cases where Documentary declares the generated documentation as equivalent, we use the generated documentation to generate code using gpt-5-nano, and gemini-2.5-flashlite. Finally, we run the tests to check if the generated code is equivalent to the original code. For gpt-5-nano, 89.6% of the generated code are equivalent to the original code, and for gemini-2.5flash-lite, 74.7% of the generated code are equivalent to the original code. Comparing these results with Documentary’s precision of 88.9% (obtained when using gpt-4.1-nano for code generation), we see that other LLMs generate equivalent code at a similar rate. These results show that equivalent documentation can be generated and used by different LLMs. More importantly, this also shows that equivalent documentation can persist in the code and provide its benefits while the LLMs evolve. 4.4
RQ3: Ablation Study
We study the effects of the number of iterations and the size limit on the effectiveness of generating equivalent documentation. For the number of iterations, we run our approach with a maximum of 10 iterations and a size limit of 2x, and report the distribution of the iteration count when the approach decides that the generated documentation is equivalent to the code. If the original Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:15
Fig. 14. Distribution of the number of iterations when generating equivalent docstrings.
Fig. 15. Success rate and precision of generating equivalent docstrings for different size limits.
docstring is already equivalent to the code, the iteration count is zero. Iteration counts greater than zero mean that the approach generates the docstring after that many attempts. As shown in Fig. 14, most (75%) equivalent documentation is generated within five iterations, and the median number of iterations is two. For the size limit, we run our approach with a maximum of 10 iterations and size limits of 0.5x, 1x, 2x, and 3x. Figure 15 shows the success rate and the precision of the generated equivalent documentation for different size limits. Our results show that a size limit of 1x provides the best precision and success rate. We suspect that a size limit greater than 1x encourages the LLM to generate verbose docstrings that contain unnecessary information, which can confuse the LLM in generating the code from the docstring. 4.5
RQ4: Costs
We measure the runtime and monetary costs of generating equivalent documentation using Documentary. This experiment measures the time it takes to run Documentary per function in seconds using Python’s time.perf_counter, as well as the number of input and output tokens when running Documentary on a function. We then calculate the cost of generating an equivalent docstring, based on the current pricing of gpt-4.1-nano. Regarding time, it takes 33.8 seconds to generate an equivalent docstring on average per function. Figure 16 shows that for most functions it takes less than a minute to generate an equivalent docstring. Regarding token consumption, Documentary uses 78,144 input tokens and 3,173 output Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:16
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
Fig. 16. Distributions of the number of input and output tokens, and the runtime of generating equivalent docstrings.
tokens, on average, for an attempt at generating an equivalent docstring for a function. Given the pricing of gpt-4.1-nano at $0.10 per 1M input tokens and $0.40 per 1M output tokens in February 2026, the average cost of generating an equivalent docstring for a function is $0.009. As shown in Fig. 16, most functions require less than 200K input tokens and 10K output tokens, resulting in a cost of less than $0.03 per function. Furthermore, for large codebases to adopt equivalent documentation, the cost can be reduced by batching the generation and using cached input tokens as the context for multiple functions that contain the same prefix. Given the reasonable costs of our current approach, we leave such optimizations for future work. 4.6
RQ5: Equivalent Documentation for LLMs (Code Understanding)
We run an experiment to evaluate how well equivalent docstrings help LLMs understand the behavior of a function without seeing the code, as specified in Section 3.1. For this experiment, we use the output prediction dataset described in Section 4.1.1. For each test in the dataset, we first generate an equivalent docstring for the function-under-test using Documentary. If Documentary fails to generate an equivalent docstring, we fall back to using a docstring generated by the LLM, as in the baseline. Then, we prompt the LLM with the function signature, the equivalent docstring, and the test code, and ask it to predict the output of each assertion in the test as a valid Python expression. Finally, we compare the AST of the predicted output with the AST of the actual output in the assertion statement, and consider a match as a correct prediction. The metrics we report are the number of correctly predicted assertions, the average percentage of correctly predicted assertions per test, and the number of tests where all assertions are correctly predicted. We compare the results of Documentary with the predictions from the original docstring and the baseline defined in Section 4.1.2. Table 3 shows that using equivalent docstrings improves the number of correctly predicted assertions by 4.0–15.9, the average percentage of correctly predicted assertions per test by 0.6–17.3, and the number of tests where all assertions are correctly predicted by 5.6–19.0 absolute percentage points, compared to using human-written docstrings, and LLM generated docstrings. The results show that an equivalent docstring allows the LLM to not only implement the correct behavior, as shown in Section 4.2, but also predict the behavior of the function. For the example in Fig. 9, Documentary generates the equivalent docstring shown in Fig. 17. This docstring provides a detailed explanation of the function’s behavior, including how it initializes the Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:17
Table 3. Effectiveness in output prediction. Correct Assertions shows how many assertions are correctly predicted, and Correct Tests shows how many tests have all assertions correctly predicted.
Correct Assertions Approach Original docstring Baseline Documentary
Count
%
Average Correct Assertions per Test
280 / 480 337 / 480 356 / 480
58.3% 70.2% 74.2%
53.5% 70.2% 70.8%
Correct Tests Count
%
105 / 253 139 / 253 153 / 253
41.5% 54.9% 60.5%
""" This initialization method retrieves the list of permitted CORS origins from the environment variable 'PERMITTED_CORS_ORIGINS'. 3 If this variable is set, it splits the string by commas and strips whitespace from each origin, converting it into a tuple. 4 If the variable is not set, it defaults to an empty tuple, allowing all origins. 5 super().__init__ is called with the application instance and the list of allowed origins, along with standard CORS settings: 6 allow_credentials is enabled, and all methods and headers are permitted. 7 Additional configuration, such as environment-based origin control, is handled within this setup. 8 """ 1 2
Fig. 17. The equivalent docstring generated by Documentary for the function __init__ in Fig. 9. allow_origins, which results in a correct prediction of the assertion value, which is an empty tuple. The LLM, however, predicts an empty list as the assertion value for both the original docstring and the baseline generated docstring.
4.7
RQ6: Equivalent Documentation for Human Developers (Code Understanding and Code Editing)
We evaluate the effectiveness of equivalent docstrings for human developers by conducting a user study with 11 participants. The demographic statistics of the participants are shown in Table 4. The study consists of six tasks, which we ask the participants to perform in a web-based interface. We give a maximum of ten minutes per task. Each task starts with showing a Python file to the participant, with a target function in that file. The participants are free to read any part of the file, including the target function, its docstring if it exists, and any other functions or classes in the file. Once they are satisfied with understanding the function, the participants are asked to answer the following question, to be answered on a Likert scale from 1 to 5: • How much did the docstring help you in understanding the code? Next, the participants proceed to a new page that shows the same file and the same target function and, in addition, now also shows a natural language description that specifies a change to the code of the target function. We directly take the relevant part of the commit message as the change description, which is written by the developers of the code change. The participants are asked to modify the code of the target function according to the change description. Once they are done with the change, they submit the modified code. The study interface then shows them three further questions about the task, each to be answered on a Likert scale from 1 to 5: • How much did the docstring help you in modifying the code? Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:18
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
Table 4. Demographic statistics of the participants in the user study. (a) Numerical attributes.
Attribute Age Years of Python experience
Min
Max
Mean
Median
23 0
31 6
26 3.18
25 4
(b) Categorical, personal attributes (countries of the participants are anonymized to maintain their anonymity).
Education
Gender
Country
Bachelor’s (27.3%) Master’s (63.6%) PhD (9.1%)
Male (72.7%) Female (18.2%) Prefer not to say (9.1%)
Country 1 (45.5%) Country 2 (45.5%) Country 3 (9.1%)
(c) Categorical, professional attributes.
AI-tools experience
Code understanding in daily work
Code editing in daily work
Regularly (54.5%) Occasionally (27.3%) Rarely (9.1%) Only a few times (9.1%)
Almost every day (9.1%) Once/twice a week (81.8%) Fewer than once a week (9.1%)
Almost every day (27.3%) Once/twice a week (63.6%) Fewer than once a week (9.1%)
• How difficult was this task? • How confident are you in your solution? If at any point during a task the ten minute timer runs out, the participant is automatically moved to the post-task questions, and the modified code is submitted as it is at that moment. We use a within-subject, randomized design, where each participant performs three tasks with the original docstring (control), and three tasks with the docstring generated by Documentary (treatment). The decision of treatment-first or control-first is randomized for each participant. In addition to the questions, we also measure the time it takes the participants from seeing the code for the first time until deciding to move to the change page, and the time it takes from seeing the change description until submitting the modified code. We perform the Mann-Whitney U test for all metrics to check if the differences in the metrics between the control and treatment groups are statistically significant. Our results show that participants found the equivalent docstrings generated by Documentary to be more helpful in both understanding and modifying the code. These results show that developers not only find the equivalent docstrings useful, but also prefer them over the original docstrings. Although the participants were more confident in their tasks and found the tasks easier when using the equivalent docstrings, these differences were not statistically significant. Considering that the tasks in the user study are from complex projects, and that the participants were not familiar with the code, we envision equivalent documentation to be helpful for contributors to open source projects, who frequently work in unfamiliar codebases. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:19
Table 5. The average values for the Likert scale questions, the time taken for understanding and changing the code, and the total time. ↑ means higher is better; ↓ means lower is better. Bold values indicate statistically significant differences between the control and treatment groups.
Metric Help in understanding ↑ Help in modifying ↑ Confidence ↑ Difficulty ↓ Understanding time (min) ↓ Modifying time (min) ↓ Total time (min) ↓ 5 5.1
Original
Documentary
p-value
2.58 2.15 2.73 3.21 1.94 4.53 6.48
3.76 2.85 3.30 2.82 2.18 4.30 6.48
0.0004 0.0425 0.0698 0.2057 0.6535 0.7388 1.000
Related Work Improving Documentation
There have been multiple studies on improving the consistency of documentation with respect to the code [17, 18, 23], but their focus is on resolving inconsistencies after code changes. In contrast, our work aims to generate equivalent documentation, even if the existing documentation is consistent with the code. Another recent work [25] has proposed to add natural language outlines to code, which are natural language descriptions that explain smaller code pieces inside functions or classes. These outlines have been shown to assist developers in understanding the code and performing some tasks like code changes. Although these outlines can be regarded as another type of equivalent documentation, we do not have empirical evidence of the equivalence of these outlines to the code. 5.2
LLM Evaluation via Self-consistency
Recent work [3, 19] propose evaluating LLMs via self-consistency in a roundtrip sequence, where the LLM is prompted to generate documentation from code, and then the documentation is used to generate code again. The generated code is then compared to the original code. That approach is similar to our evaluation of equivalent documentation, but we focus on the generation of documentation that is equivalent to the code, rather than just evaluating the LLM’s ability to generate both modalities in a roundtrip correctly. 5.3
LLM-based Software Engineering
LLMs and LLM-based agents have shown promise in various software engineering tasks [24], such as code comprehension [20], code generation [9], and program repair [6, 13]. Recent studies have also explored the use of LLMs and agentic approaches for automating even more software development processes, such as executing arbitrary code [8, 26]. Our work builds towards enhancing the usability of these approaches by generating equivalent documentation that can be used by LLMs to understand and modify code. 6
Threats to Validity
We acknowledge the following threats to the validity of our findings. First, our evaluation relies on the test cases to determine the equivalence of code snippets. If the test cases are not comprehensive, they may miss important aspects of the code’s behavior. Furthermore, flaky tests may lead to incorrect conclusions about the equivalence of code snippets. We try to mitigate this threat, Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:20
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
which is a difficult task and an ongoing research challenge [8, 26], by using large datasets for our evaluations. Second, our evaluation focuses on Python code and docstrings. The effectiveness of our approach may vary for other programming languages or documentation styles. However, the principles of our approach are language-agnostic and can be applied to other languages. Third, for our user study, we were only able to recruit 11 participants, which did not allow us to observe statistically significant differences for some metrics. 7
Conclusion
In this paper, we present a novel property, called documentation-to-code equivalence, in which the documentation of a code snippet is considered equivalent if it contains sufficient information to generate the same code snippet. Equivalent documentation helps LLMs perform software engineering tasks that require understanding the code, such as output prediction. It also allows developers to understand code better. We propose an iterative approach, called Documentary, to generate equivalent documentation. Our evaluation on two datasets of Python projects shows that Documentary can generate equivalent documentation for 53.4% of functions, and that using equivalent documentation improves code comprehension for both human developers and LLMs. Furthermore, we show that equivalent documentation can be used by LLMs other than the one used to generate them, and that the cost of generating equivalent documentation is low enough to be practical for large codebases. 8
Data Availability
Our code, datasets, and scripts to reproduce the results are available at https://github.com/solast/Documentary, and as a stable snapshot also at https://doi.org/10.5281/zenodo.19514143. Acknowledgments This work was supported by the German Research Foundation (DFG; projects 492507603, 516334526, and 526259073). References [1] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A Transformer-based Approach for Source Code Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL. 4998–5007. doi:10.18653/v1/2020.acl-main.449 [2] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 526–538. doi:10.1109/MSR66628.2025.00086 [3] Miltiadis Allamanis, Sheena Panthaplackel, and Pengcheng Yin. 2024. Unsupervised Evaluation of Code LLMs with Round-Trip Correctness. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview.net/forum?id=YnFuUX08CE [4] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In Proceedings of the 33nd International Conference on Machine Learning, ICML. 2091– 2100. [5] Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs Instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. arXiv:2406.18403 [cs] doi:10.48550/arXiv.2406.18403 [6] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 2188–2200. doi:10.1109/ICSE55347.2025.00157 [7] Islem Bouzenia, Bajaj Piyush Krishan, and Michael Pradel. 2024. DyPyBench: A Benchmark of Executable Python Software. In ACM International Conference on the Foundations of Software Engineering (FSE). Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
FSE204:21
[8] Islem Bouzenia and Michael Pradel. 2025. You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects. Proceedings of the ACM on Software Engineering 2, ISSTA (June 2025), 1054–1076. doi:10.1145/3728922 [9] Jizheng Chen, Kounianhua Du, Xinyi Dai, Weiming Zhang, Xihuai Wang, Yasheng Wang, Ruiming Tang, Weinan Zhang, and Yong Yu. 2025. DebateCoder: Towards Collective Intelligence of LLMs via Test Case Driven LLM Debate for Code Generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 12055–12065. [10] Sergio Cozzetti B. De Souza, Nicolas Anquetil, and Káthia M. De Oliveira. 2005. A Study of the Documentation Essential to Software Maintenance. In Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting & Designing for Pervasive Information. ACM, Coventry United Kingdom, 68–75. doi:10.1145/1085313. 1085331 [11] Beat Fluri, Michael Würsch, Emanuel Giger, and Harald C. Gall. 2009. Analyzing the Co-Evolution of Comments and Source Code. 17, 4 (2009), 367–394. doi:10.1007/s11219-009-9075-x [12] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2020. Deep Code Comment Generation with Hybrid Lexical and Syntactical Information. Empirical Software Engineering 25, 3 (May 2020), 2179–2217. doi:10.1007/s10664-019-09730-9 [13] Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. InferFix: End-to-End Program Repair with LLMs. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, San Francisco CA USA, 1646–1656. doi:10.1145/3611643.3613892 [14] Alexander LeClair, Aakash Bansal, and Collin McMillan. 2021. Ensemble Models for Neural Source Code Summarization of Subroutines. CoRR abs/2107.11423 (2021). arXiv:2107.11423 https://arxiv.org/abs/2107.11423 [15] Bo Lin, Shangwen Wang, Zhongxin Liu, Xin Xia, and Xiaoguang Mao. 2023. Predictive Comment Updating With Heuristics and AST-Path-Based Neural Learning: A Two-Phase Approach. IEEE Transactions on Software Engineering 49, 4 (April 2023), 1640–1660. doi:10.1109/TSE.2022.3185458 [16] Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021. Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=zv-typ1gPxA [17] Zhongxin Liu, Xin Xia, David Lo, Meng Yan, and Shanping Li. 2021. Just-in-time obsolete comment detection and update. IEEE Transactions on Software Engineering 49, 1 (2021), 1–23. [18] Zhongxin Liu, Xin Xia, Meng Yan, and Shanping Li. 2020. Automating just-in-time comment updating. In Proceedings of the 35th IEEE/ACM International conference on automated software engineering. 585–597. [19] Marcus J. Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2023. Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain. In The Twelfth International Conference on Learning Representations. [20] Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to Help With Code Understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machinery, New York, NY, USA, 1–13. doi:10.1145/3597503.3639187 [21] Kunal Pai, Premkumar Devanbu, and Toufique Ahmed. 2025. CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE Computer Society, 451–455. doi:10.1109/MSR66628.2025.00077 [22] Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J. Mooney. 2021. Deep Just-In-Time Inconsistency Detection Between Comments and Source Code. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, 427–435. https://ojs.aaai.org/index.php/AAAI/article/view/16119 [23] Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond J. Mooney. 2020. Learning to Update Natural Language Comments Based on Code Changes. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 1853–1868. doi:10.18653/v1/2020.acl-main.168 [24] Abhik Roychoudhury, Corina Păsăreanu, Michael Pradel, and Baishakhi Ray. 2025. Agentic AI software engineers: programming with trust. Commun. ACM (2025). [25] Kensen Shi, Deniz Altınbüken, Saswat Anand, Mihai Christodorescu, Katja Grünwedel, Alexa Koenings, Sai Naidu, Anurag Pathak, Marc Rasi, Fredde Ribeiro, Brandon Ruffin, Siddhant Sanyam, Maxim Tabachnyk, Sara Toth, Roy Tu, Tobias Welp, Pengcheng Yin, Manzil Zaheer, Satish Chandra, and Charles Sutton. 2025. Natural Language Outlines for Code: Literate Programming in the LLM Era. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE Companion ’25). Association for Computing Machinery, New York, NY, USA, 150–161. doi:10.1145/3696630.3728541 Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.
FSE204:22
Aryaz Eghbali, Zhongxin Liu, and Michael Pradel
[26] Beatriz Souza and Michael Pradel. 2025. Treefix: Enabling Execution with a Tree of Prefixes. arXiv:2501.12339 [cs] doi:10.48550/arXiv.2501.12339 [27] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, Marianne Huchard, Christian Kästner, and Gordon Fraser (Eds.). ACM, 397–407. doi:10.1145/3238147.3238206 [28] Yu Wang, Fengjuan Gao, and Linzhang Wang. 2021. Demystifying Code Summarization Models. CoRR (2021). https://arxiv.org/abs/2102.04625 [29] Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On Learning Meaningful Assert Statements for Unit Test Cases. In ICSE. [30] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, 24824–24837. [31] Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020. Retrieval-based Neural Source Code Summarization. In ICSE.
Received 2025-09-12; accepted 2026-03-24
Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE204. Publication date: July 2026.