arXiv CSopen access

Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions

Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions Zhanna Kaufman

Yuriy Brun

[email protected] University of Massachusetts Amherst Amherst, MA, USA

[email protected] University of Massachusetts Amherst Amherst, MA, USA

Adithya Murali

Madeline Endres

[email protected] University of Wisconsin–Madison Madison, WI, USA

[email protected] University of Massachusetts Amherst Amherst, MA, USA

arXiv:2607.08885v1 [cs.SE] 9 Jul 2026

Abstract

this burden, researchers have proposed shifting developer review toward smaller reliability-oriented artifacts such as tests [17, 41], specifications [40, 81], or assertions [16]. For example, Endres et al. propose using LLMs to generate executable assertions that allow developers to review intended program behavior rather than the full implementation, reducing the amount of code that must be inspected while enabling run-time validation of generated code. Similarly, Lahiri envisions a future in which LLMs help formalize user intent through behavioral artifacts such as tests, assertions, and formal specifications [40]. By formalizing intended program behavior, such artifacts have the potential to help developers determine if generated code captures their intent and, in some cases, validate that intent at run time. A key assumption underlying these approaches is that developers can accurately evaluate machine-generated reliability artifacts. Yet this assumption remains largely unexamined. While a growing body of work studies developers’ interactions and understanding of LLM-generated code (see Hou et al. [28], Table 10), comparatively little empirical evidence exists regarding how effectively developers evaluate generated specifications, tests, or assertions. This uncertainty extends to one of the most common mechanisms used to support comprehension of such artifacts: natural-language explanations [31, 48]. Modern LLMs routinely generate comments and explanations alongside code and specifications. However, like the code itself, such explanations may be incorrect, incomplete, vague, or irrelevant [11, 36]. Consequently, it is unknown whether naturallanguage explanations improve programmer judgment or instead introduce additional sources of error. In this paper, we investigate two questions: (1) How effective are programmers at judging the correctness and completeness of LLMgenerated assertions about code? and (2) How does the presence and quality of natural-language comments affect these judgments? We focus on postcondition assertions for several reasons. First, postconditions possess well-defined formal semantics, allowing us to rigorously characterize metrics such as correctness and completeness. Second, LLMs are fairly proficient (but not yet perfect) at generating valid assertions from documentation and have been explicitly proposed as part of a human-in-the-loop software reliability workflow [16]. This allows us to study a close-to-ideal setting, in which LLMs are generally capable of generating useful assertions, but are not sufficiently reliable to eliminate the need for careful human review. Finally, postconditions can express rich behavioral

Code comprehension and code review are already critically important software engineering tasks, and the rising use of AI code generation tools is only increasing that importance. Generative AI has the possibility of supporting these activities, for example by augmenting code with assertions and natural-language explanations describing code behavior. However, little is known about how effective such support may be. We conduct a controlled experiment with 86 Python programmers and a follow-up think-aloud study to examine developers’ ability to assess the correctness and completeness of generated assertions of varying quality, and to investigate how natural-language explanations influence these assessments. While programmers can somewhat accurately judge correct assertions (74% accuracy), they perform poorly when shown incorrect assertions (49% accuracy), despite reporting similar levels of confidence in both judgments. This difference in judgment accuracy is statistically significant (𝑝 < 0.001): the odds of a developer accurately judging a correct assertion was nearly three times higher than the odds of accurately judging an incorrect assertion (OR = 2.94). Surprisingly, natural-language explanations of assertions provided no overall benefit. Furthermore, low-quality explanations could impair specification assessment accuracy (𝑝 = 0.037, 𝑂𝑅 = 0.58) while simultaneously increasing developer confidence (𝑝 = 0.005, 3.99/5 vs. 4.25/5). Our findings suggest that, contrary to common assumptions, AI assistance may not improve the reliability of code comprehension and review. More broadly, our findings highlight the importance of helping developers evaluate machine-generated reliability artifacts, in addition to generating them.

1

Introduction

Generative AI tools have transformed software engineering, and are now a routine part of virtually every aspect of development including code generation [63], architectural design [34], debugging [63], and verification [20]. This makes code comprehension and code review, already important parts of the software engineering lifecycle, even more important, as humans are asked to be part of the loop, reviewing, adapting, and adopting automatically generated code [55, 78]. However, reviewing code is difficult, time-consuming, and cognitively demanding [5, 6]. Programmers often struggle to understand non-self-authored code [74], including LLM-generated code [69], as such code may be incorrect or misaligned with user intent [17, 77], which can be difficult to fix in practice [80]. To reduce 1

properties while remaining compact enough to support controlled experimental evaluation. We conduct a controlled experiment with 86 Python programmers spanning a range of professions and experiences. We construct a corpus of Python functions paired with LLM-generated postcondition assertions and natural-language comments, extending datasets used in prior work on assertion generation [10, 16, 45]. Participants were shown a function and postcondition, then asked to assess whether the postcondition correctly captured the function’s behavior and how completely it did so. To evaluate how explanation quality affects developers’ reasoning about generated reliability artifacts, we included four types of comments that described the assertions: correct, incorrect, under-specified, and over-specified. We complemented the controlled experiment with a think-aloud study with 10 developers. This qualitative analysis allows us to characterize how developers reason about generated assertions and explanations. Our findings reveal a substantial asymmetry in developer reasoning. Participants were much better at identifying correct assertions than identifying incorrect ones (74% vs. 49% accuracy, 𝑝 < 0.001, 𝑂𝑅 = 2.94), despite reporting similarly high confidence in both cases (approximately 4/5 for both, 𝑝 > 0.1). Surprisingly, naturallanguage explanations led to no improvement in judgment accuracy (𝑝 > 0.05), and low-quality explanations could even reduce accuracy (𝑝 = 0.037, 𝑂𝑅 = 0.58) while simultaneously increasing confidence compared to no comment at all (𝑝 = 0.005, 4.25/5 vs. 3.99/5). Simultaneously, developers’ assessments of assertion completeness correlated with an independent mutation-based measure of how many bugs it could detect (Spearman’s 𝜌 = 0.18, 𝑝 < 0.001), and participants rated correct assertions as significantly more complete than incorrect ones (𝑝 < 0.001). Together, this suggests that developers can distinguish stronger specifications from weaker ones even when they struggle to identify logical errors. Providing insight into these findings, our think-aloud study suggests that developers frequently rely on direct comparisons between assertion clauses and documentation, helping explain why some incorrect assertions are difficult to detect. We make the following contributions:

We conclude by discussing implications for AI-assisted software reliability, suggesting that future tools should move beyond generating reliability artifacts to helping developers verify that they faithfully capture intended program behavior.

2

Motivating Example

Consider Juan, a developer who is asked to write and test the function has_close_elements(numbers: List[float], threshold: float) (this example comes from the HumanEval [10] dataset). Juan is given a natural-language docstring description of the desired functionality: 1

" Check if in given list of numbers , are any two numbers closer to each other than given threshold . "

Juan decides to use an LLM to generate and test the code. In generating unit tests, the biggest challenge lies in generating assertions to act as oracles [50]. Luckily, LLMs may be able to help. For example, ChatAssert [27] generates test oracles using code summaries and examples, and then uses static and dynamic analyses to repair incorrect assertions. But LLMs sometimes oversimplify specifications, and at other times fabricate them entirely [43]. Thus, to effectively use LLMs to generate tests, Juan needs to review and assess the correctness of the generated oracle assertions. They ask an LLM to generate a test oracle for this docstring in the form of an assert with function input numbers and function return ret. The model returns: 1 2

assert not ret or any ( abs (j - i ) < threshold for i in numbers for j in numbers if i != j )

This postcondition asserts that if the function returns True, then some pair of non-equivalent elements in the input list must have an absolute difference less than the given threshold. This statement may seem satisfactory at first glance, but contains two errors. The first is an issue of completeness: it does not perform any checks in cases where ret is False. The second is an issue of correctness: the if i != j check excludes all elements with equivalent values, rather than equivalent indices. Comprehending assert statements can be difficult in practice. If Juan misses the completeness error, they fail to test half of their intended functionality. If Juan misses the correctness error, they will end up developing a function that is entirely mismatched with their intended functionality. A natural question thus arises: if Juan prompts the LLM to also generate a comment explaining the test oracle, can the resulting natural language translation of the test oracle help them catch such errors? Furthermore, if the LLM’s comment is misaligned with the logic of the oracle (i.e., of poor quality), how will this misalignment affect Juan’s understanding of the test? Unfortunately, a dearth of research exists into developer understanding of LLM-generated test oracles, whether LLM-generated comments can aid this understanding, and whether misaligned comments can hurt this understanding. This work aims to bridge this gap by investigating 1) how reliably developers can identify postcondition correctness, 2) whether LLM-generated comments can aid in this task, and 3) whether the quality of these comments matters. We augment this investigation with a qualitative analysis of developer reasoning during postcondition comprehension tasks.

• A controlled experiment with 86 Python programmers investigating how developers evaluate LLM-generated postconditions, a representative class of machine-generated specifications used in AI-assisted software reliability workflows. • Evidence that developers are much better at recognizing correct assertions than incorrect ones (74% vs. 49% accuracy, 𝑝 < 0.001, 𝑂𝑅 = 2.94), despite reporting similarly high confidence in both cases (approximately 4/5, 𝑝 > 0.1). • An investigation of natural-language explanations, showing that they provide no overall improvement in developers’ ability to evaluate assertions, while low-quality explanations can reduce accuracy (𝑝 = 0.037, 𝑂𝑅 = 0.58) compared to high-quality ones, and simultaneously increase developer confidence (𝑝 = 0.005) compared to no comment at all. • A think-aloud study with 10 developers identifying five recurring reasoning strategies for evaluating generated assertions and providing qualitative insight into why some classes of incorrect assertions may be systematically overlooked. 2

3

Experimental Methodology

(2) Incorrect: There is at least one input/output pair that aligns with requirements in the function docstring where the postcondition raises an error.

To understand how programming logic, natural language, and human reasoning interact, we conducted two user studies:

Figure 1 has an example stimulus, including the target function and associated postcondition conditions.

(1) Controlled Experiment: A large-scale experiment with 86 participants. Participants assessed if a series of logical postconditions accurately captured the a function’s behavior. We varied stimuli in a controlled manner to understand the impact of assertion correctness or the quality of an accompanying natural-language comment on human judgment. (2) Qualitative Think-Aloud: For a nuanced understanding of developer reasoning, we conducted a think-aloud with 10 participants. Participants interacted with a series of postconditions and functions while verbalizing their reasoning.

Comment Conditions: For each postcondition–function pair, our corpus contained five stimuli: one for each comment condition. In our definitions, the ‘failure’ of an postcondition indicates that the Python assert statement will raise an error. ‘passing’ indicates that the Python assert statement will not error. When we refer to the ‘failure’ or ‘passing’ of a comment, we are referring to a direct translation of the comment into a Python assert statement. (1) Exact: Comment is equivalent to the postcondition; For all function input/output pairs, there are no cases where the comment passes and the postcondition fails, or vice versa. We consider exact comments to be high-quality comments. (2) Over-specified: Comment is stricter than the postcondition; There exists a function input/output where the comment fails and the postcondition passes, but not vice versa. (3) Under-specified: Comment is less strict than the postcondition; There is a function input/output where the comment passes and the postcondition fails, but not vice versa. (4) Incorrect: There exist function input/output pairs where the comment fails and the postcondition assertion passes, and there exist function input/output pairs where the comment passes and the postcondition assertion fails. (5) No Comment: No natural-language comment was included in the stimulus. The stimulus consisted entirely of the target function and candidate postcondition.

These studies combine the strengths of controlled analysis with insight into participant reasoning, enabling a deeper understanding of how programmers interpret specifications.

3.1

Study 1: Controlled Experiment

To study how assertion correctness and the presence or quality of an associated natural-language comment influence human judgement, we conducted a controlled experiment with 86 participants. Participants reviewed a series of stimuli, each consisting of a function and associated docstring, postcondition assertion, and (optionally) a natural-language comment describing the assertion. Participants assessed whether the postcondition accurately described the function’s behavior, assuming the provided implementation was correct. Participants also rated the completeness of the postcondition—that is, how fully they felt it captured the function’s behavior. For both correctness and completeness judgments, participants reported confidence using a 0–5 Likert-style scale. Our experimental design was guided by two considerations: (1) balancing experimental power against participant attention, and (2) isolating the effects of postcondition correctness and comment quality. Each participant completed 10 stimuli. We chose 10 after a small pilot study where we varied the number and found that participants began to lose attention after 10. The study was hosted on Qualtrics, took 45–60 minutes, and contained 3 attention-checks. Stimuli were balanced within and across participants with respect to two postcondition correctness conditions (correct and incorrect) and five comment conditions (exact, over-specified, underspecfied, incorrect, and no-comment). We discuss these conditions below.

Figure 2 shows an example of an exact, over-specified, underspecified, and incorrect comment for the correct postcondition from Figure 1. 3.1.2 Stimulus Corpus Construction. We now describe how we constructed a corpus instantiating the experimental conditions described in Section 3.1.1. Each stimulus consisted of a function, a postcondition assertion, and (optionally) a natural-language comment describing the postcondition. For functions, we used a subset of the HumanEval [10] dataset developed by OpenAI for evaluating code generation models. We selected this dataset because it was previously used in prior work on LLM-generated postconditions by Endres et al. [16]. In addition, HumanEval functions are sufficiently complex to permit non-trivial postconditions while remaining simple enough for inclusion in an online survey without overloading participants. In their previous work, Endres et al. used GPT-3.5-turbo and GPT4 to generate postcondition/comment pairs for the 164 HumanEval functions and evaluated the resulting postconditions for correctness and completeness [16]. From this dataset, we filtered for functions that had at least one correct and one incorrect postcondition. We required that the incorrect postcondition be logically incorrect, rather than syntactically. We further filtered out cases where either postcondition was logically trivial, prohibitively long, or referenced out-of-context functions. We identified 29 functions that fit our requirements.

3.1.1 Experimental Conditions. To isolate the effects of programming logic and natural-language explanations, we systematically varied two aspects of each stimulus: (1) postcondition correctness and (2) the presence and quality of an accompanying naturallanguage comment. This led to ten variants of every function (2 correctness × 5 comment conditions). Correctness Conditions: For each function, our corpus included two postconditions, one correct with respect to the function, and one incorrect: (1) Correct: For all function input/output pairs that align with the requirements in the function docstring, there are no cases where the postcondition raises an error. 3

(a) HumanEval function 58 1 2 3 4 5 6 7 8 9

def common ( l1 : list , l2 : list ) : """ Return sorted unique common elements for two lists . common ([1 ,4 ,34 ,653 ,2 ,5] , [5 ,7 ,1 ,5 ,653 ,121]) -> [1 , 5, 653] common ([5 ,3 ,2 ,8] , [3 ,2]) -> [2 ,3] " " " shared = set () for e1 in l1 : for e2 in l2 : if e1 == e2 : shared . add ( e1 ) return sorted ( list ( shared ) )

(b) Correct postcondition: ensures that all returned elements appear in both input lists and that the return value is sorted and contains only unique elements. 1 2

assert all ( i in l1 and i in l2 for i in ret ) and ret == sorted ( set ( ret ) )

(c) Incorrect postcondition: appears plausible, but fails for some correct return values. When l1 = [1,1,2], l2 = [1,4], and return_val = [1], it evaluates to [1] == [1,1] as duplicate elements from l1 are preserved. 1 2

assert ret == sorted ([ value for value in l1 if value in l2 and ret . count ( value ) == 1])

Figure 1. Example stimuli shown to participants, including the target function and associated postcondition conditions. (a) Exact comment 1

# Ensures that all elements in the return value are present in both lists l1 and l2 , and that the return value is a sorted list of unique elements .

(b) Over-specified comment: adds a constraint that does not exist in the postcondition. 1

# Confirm that the return value is a unique set of elements found in both l1 and l2 , sorted in ascending order , that appear in both lists exactly once .

1

# Return value contains elements that are present in both lists l1 and l2 , and the return value is sorted .

(c) Under-specified comment: omits a constraint that exists in the postcondition.

(d) Incorrect comment: both adds and omits constraints compared to the postcondition. 1

# Verify that the return value contains all elements that are common between the two input lists .

Figure 2. Example comments corresponding to the correct postcondition in Figure 1 under each comment-quality condition.

All postconditions in the Endres et al. dataset included an associated LLM-generated comment. Some postconditions already had multiple associated comments spanning several quality categories because the original dataset contained repeated postconditions paired with different comments. When one or more commentquality categories (exact, over-specified, under-specified, and incorrect) were missing for a postcondition, the first author used GPT-3.5-turbo to generate additional candidate comments. These candidates were manually reviewed and annotated to produce one example of each comment-quality condition for every postcondition. Three authors independently reviewed all postconditions and comments for quality and correctness consistency before inclusion in our experimental corpus. At this stage, we removed three additional functions from the corpus due to implementation ambiguity in the original HumanEval dataset. The final corpus contained 26 functions. Each function was paired with one correct and one incorrect postcondition, and each postcondition was paired with all five comment conditions, yielding a total of 260 experimental stimuli.

3.2

our survey questions for 5 curated functions. We curated these questions by ordering our HumanEval functions by the difference between the percentage of participants who accurately identified the correct postcondition as correct, and the percentage of participants who accurately identified the incorrect postcondition as incorrect. We chose the 3 functions with the highest differences between these values, and the 2 functions with the lowest differences between these values. We counterbalanced function order, as well as whether follow-up participants were given correct or incorrect postconditions across questions, such that each participant saw 3 incorrect postconditions and 2 correct postconditions. All stimuli included exact comments.

4

Research Questions and Analysis Approach

We organize our investigation and analysis of programming logic, natural language comments, and human comprehension around four research questions. As described in Section 3, we approach our investigation from the perspective of a programmer assessing postconditions in Python. RQ1—Postcondition Correctness: How does the logical correctness of an LLM-generated postcondition assertion impact programmers’ ability to determine whether the assertion correctly describes the behavior of a Python function? RQ2—Natural-Language Comments: How does the presence and quality of LLM-generated comments affect programmers’ ability to identify whether a postcondition correctly describes the behavior of a Python function?

Study 2: Qualitative Think-Aloud

While our controlled experiment quantified programmer performance, it did not explain why participants succeeded or failed. To better understand how programmers approach and analyzed postconditions, we also conducted a series of think-aloud sessions where participants described their thought process when responding to 4

RQ3—Assertion Characteristics: What features of assertions (e.g., complexity or completeness) correlate with programmer performance when evaluating postconditions?

fixed and random effects, rather than requiring separate post hoc comparisons. Additional analyses, including the feature analyses for RQ3, were exploratory and are labeled as such in our results. Following recommendations for exploratory analyses [58], we do not apply multiple comparison corrections to reduce the likelihood of false negatives. We used RStudio [56] for our analysis. R packages included lme4, lmerTest, stats, car, and psych. We include all analysis scripts and report the results of all conducted statistical tests in our replication package [1].

RQ4—Developer Reasoning: How do programmers reason about postconditions and accompanying natural-language comments? Are particular reasoning strategies associated with successful comprehension? Although our controlled experiment was originally designed to investigate the effects of natural-language comments (RQ2), early analysis revealed a substantially larger-than-anticipated difference between programmers’ ability to recognize correct and incorrect postconditions. This unexpected finding motivated additional analyses examining the role of postcondition correctness (RQ1), postcondition characteristics associated with performance (RQ3), and our qualitative follow-up study of the reasoning strategies underlying these differences (RQ4). To support the original experimental goal, we pre-registered our hypotheses relating to RQ2 on the Open Science Framework [21]. In particular, we hypothesized that natural-language comments would improve participants’ ability to evaluate assertion correctness, but that these effects would depend on comment quality. All pre-registered hypotheses, study materials, analysis scripts, and replication artifacts are included in our replication package [1, 37]. While RQ1, RQ3, and RQ4 were motivated by observations made during the early stages of analysis (and are thus more exploratory), all analyses were conducted using the same controlled experimental design and statistical methodology (see Sections 3 and 4.1).

4.1

4.2

Qualitative Methods and Final Codebook

We address RQ4 through a qualitative think-aloud study designed to better understand how programmers reason about logical assertions and accompanying natural-language comments (Section 3.2). Participants verbalized their reasoning while completing five assertionevaluation tasks. To analyze the think-aloud transcripts, we conducted a directed content analysis [29] using a hybrid deductive– inductive approach. Two authors collaboratively developed and refined a codebook capturing: (1) how participants constructed a mental model of the function and postcondition, and (2) the strategies participants used to determine postcondition correctness. Our mental-model coding scheme was informed by prior work on code comprehension [13, 19]. Specifically, we categorized participant– stimulus pairs as using a top-down, bottom-up, or as-needed comprehension processes: Top-Down: Formed expectations from the docstring or comment, then verified them by reading the code. Bottom-Up: Inferred behavior from the code, then verified it using the docstring or comment. As-Needed: Alternated between the code and the docstring or comment as needed to resolve questions.

Quantitative and Statistical Methods

We address RQ1–RQ3 through quantitative analysis of our controlled experiment with 86 programmers. Across these analyses, our primary dependent variables are participant accuracy (whether the participant correctly assessed the postcondition), response time, and self-reported confidence. Unless otherwise stated, we use mixed-effects models throughout our analyses. We use generalized linear mixed-effects models for binary outcomes (e.g., participant accuracy) and linear mixedeffects models for continuous outcomes (e.g., response time and confidence). Because response times were right-skewed, we logtransformed them prior to analysis following established recommended best practice [12, 72]. All models include participant- and stimulus-level random effects to account for repeated measures, participant ability, and variation in stimulus difficulty. Fixed effects were selected according to the research question. For example, when evaluating the effects of comment quality, we include postcondition correctness as a covariate to account for its potential influence on performance. We used odds ratios for generalized linear model effect sizes, and estimated marginal means for linear models. When comparing binary variables, we additionally report 𝜒 2 tests and Fisher’s exact tests where appropriate. We used a significance threshold of 𝑝 < 0.05. We do not apply a global correction for multiple comparisons because the primary analyses for RQ1 and the pre-registered analyses for RQ2 each centered on only three mixed-effects models: one each for accuracy, response time, and confidence. Such models jointly estimate all

Correctness strategies were identified inductively as they emerged during analysis, and are one of our contributions in RQ4. To improve reliability, two authors jointly coded the transcript of one participant, and refined code definitions by resolving coding disagreements. The first author then applied the finalized codebook to the remaining transcripts.

5

Participants and Recruitment

For both studies, we recruited via computing-specific email lists and Slack channels at three large public universities. We also used snowball sampling seeded by software company contacts. Participants had to (1) be at least 18, (2) able to read and understand Python code, and (3) correctly answer all 3 study attention checks. Participants received a $12 gift card. Controlled Experiment Participants. We collected 106 survey submissions, of which 86 met our inclusion criteria and were retained for analysis. We excluded participants who failed one or more attention checks. To remove implausibly rushed responses, we also excluded stimulus-level data completed in under 20 seconds (18/737 total stimuli); the mean time spent on a non-attention-check question was 133.5 seconds. Participants ranged in age between 18 and 42, with a mean age of 27.1. 66.3% of participants were professional developers. Most 5

Age

Prof. Dev.

Years Exp.

LLM Use

Writes Python

Assertion Exp.

1 2 3 4 5 6 7 8 9 10

31 26 25 27 32 26 21 32 32 29

Yes No Yes Yes No No Yes Yes Yes Yes

10–15 10–15 5–10 10–15 5–10 5–10 1–5 >15 10–15 5–10

Sometimes Never Daily Rarely Often Sometimes Sometimes Rarely Daily Daily

Daily Rarely Daily Daily Often Often Often Often Rarely Daily

Some Very little Moderate Moderate Some Some Very little A lot Some Moderate

Confidence Distribution by Participant and Postcondition Correctness Postcondition: Correct Participant: Correct (73.9%) 60

Percentage of Responses (%)

ID

Figure 3. Think-aloud participant overview.

48.3 29.5

20 0.4

1.5

3.7

3.2

2.1

Postcondition: Incorrect Participant: Correct (49.0%)

37.4

40

0

participants reported between one and ten years of programming experience, moderate or greater Python proficiency, frequent Python use, and at least some prior experience writing logical assertions and using LLMs.

8.4

13.7

Postcondition: Incorrect Participant: Incorrect (51.0%) 43.1

12.1 1.7

1.1

4.6

0

1

2

3

42.0

40.9

4

5

11.6 1.7

2.2

4 5 0 1 Participant Confidence

2

1.7

Postcondition Correct

No

3

Yes

Figure 4. Confidence distributions of participants in four cases. Case 1 (top left): Participant succeeds in identifying a correct postcondition. Case 2 (top right): Participant fails in identifying a correct postcondition. Case 3 (bottom left): Participant succeeds in identifying an incorrect postcondition. Case 4 (bottom right): Participant fails in identifying an incorrect postcondition.

Think-Aloud Participants. To better understand the reasoning processes underlying our quantitative findings, we recruited 10 additional participants for the think-aloud study. Figure 3 shows participant demographics and programming experience. Participants ranged in age from 21 to 32 years. Seven participants were professional programmers and three were university students. Nine participants had more than five years of programming experience, although only one reported significant experience writing logical assertions.

Participants frequently used negative examples when they perceived a postcondition to be incorrect (18/29 cases), typically to confirm an initial judgment formed through clause comparison, logic walk-throughs, or intuition (14/18). We also examined the impact of assertion correctness on response time. When the postcondition was correct, participants took about the same amount of time to respond whether they answered correctly or not (103.4 vs. 108.4 average seconds). In contrast, successfully identifying an incorrect postcondition required substantially more time (123.0 s) than incorrectly accepting it as correct (97.1 s). This difference was significant; a mixed-effects model with participant accuracy and postcondition correctness as predictors shows a significant effect of participant accuracy on response time (𝑝 = 0.014), driven by the interaction effect between participant accuracy and postcondition correctness (𝑝 = 0.002). This suggests that detecting specification errors requires effort, and incorrectly accepting plausible assertions often occurs quickly.

Results

We now present our experimental results, organized around our four research questions (see Section 4).

6.1

9.7

60

20

6

43.2 36.4

40

0

Postcondition: Correct Participant: Incorrect (26.1%)

RQ1: The Impact of Assertion Correctness

Programmers were substantially better at recognizing correct postconditions than identifying incorrect ones. Participants correctly identified 73.9% of correct postconditions as correct, but only 49.0% of incorrect postconditions as incorrect. This difference was significant (𝑝 < 0.001, 𝑂𝑅 = 2.94), indicating that the odds of correctly recognizing a correct postcondition were nearly three times higher than those of correctly identifying an incorrect one. As a qualitative example, our stimulus example in Figure 1 is the function where we saw the largest difference in participant accuracy between the correct and incorrect postcondition (92% vs. 10%, respectively). We hypothesize two potential reasons for this discrepancy. First, developers may have an acceptance bias toward AI-generated specifications; prior work has shown that people are likely to accept code suggestions even when they are incorrect [15]. Second, it may be more cognitively demanding to identify that a logical statement is incorrect than it is to confirm that the statement is correct. We return to this second possibility in RQ3 (Section 6.3) and RQ4 (Section 6.4). Interestingly, neither postcondition correctness nor participant correctness significantly affected self-reported confidence (𝑝 > 0.1). Confidence was uniformly high in all cases (3.98–4.14, 0–5 Likert). Figure 4 summarizes confidence distributions by postcondition and participant correctness.

RQ1: Summary—Impact of Assertion Correctness: Programmers were much better at recognizing correct assertions than identifying incorrect ones. Detecting incorrect assertions also required significantly more time, yet participants remained similarly confident regardless of whether their judgments were correct.

6.2

RQ2: Comment Presence and Quality

We now investigate how natural-language comments impact programmer assessments of postcondition correctness. Surprisingly, we find that (1) comments generally do not improve programer accuracy, and (2) low-quality comments can reduce performance and increase confidence. We fit a mixed effects model predicting participant accuracy from comment condition while controlling for postcondition correctness. 6

Contrary to our pre-registered hypothesis (see Section 4), the presence of a comment (even if that comment was of high-quality) did not significantly improve participant accuracy compared to providing no comment at all (𝑝 > 0.1). Furthermore, these effects did not differ between correct and incorrect postconditions. These findings suggest that simply providing an explanation is insufficient; explanation quality can matter more than explanation presence. Although comments provided no overall benefit, comment quality did matter. When comparing between different comment qualities in the case that a comment exists, participants were significantly less accurate when comments were under-specified than when comments exactly matched the postcondition (𝑝 = 0.037, 𝑂𝑅 = 0.58). Anecdotally, participants were also slightly less accurate with under-specified comments than with no comment at all(54.3% vs. 62.7%), suggesting that incomplete explanations may actively interfere with reasoning rather than simply failing to help. This decrease in performance was accompanied by increased confidence. Participants reported significantly higher confidence when evaluating under-specified comments than when evaluating stimuli without comments (𝑝 = 0.005; emmeans = 4.25/5 vs. 3.99/5). We found no significant relationship between participant confidence and judgment correctness. Thus, under-specified comments may have increased perceived understanding without improving actual comprehension. Comment condition also affected response time (𝑝 = 0.022). Participants took the longest time to respond when looking at exact comments, and the shortest time to respond when there was no comment at all (emmeans = 94.3, 114.5, 105.3, 109.6, 111.8 for no comment, exact comment, incorrect comment, over-specified comment, and under-specified comment, respectively). Participants may have spent additional time comparing the comment with the postcondition, although this additional effort did not improve judgment accuracy.

means = = 2.66 vs. 2.35). Together, these findings suggest that participants could distinguish stronger specifications from weaker ones even when they failed to identify logical errors. Complexity: Correct postconditions exhibited significantly lower cyclomatic complexity than incorrect postconditions (𝑝 < 0.001; emmeans = 4.28 vs. 4.85), a pattern that we find is also true across the larger postcondition corpus from prior work [16] from which we filter for our experimental stimuli. However, mediation analysis found that cyclomatic complexity did not explain the effect of postcondition correctness on participant accuracy (𝑝 = 0.50), and neither Halstead difficulty nor cognitive complexity differed significantly between correct and incorrect postconditions (𝑝 > 0.1). Thus, although incorrect assertions tend to be somewhat more complex, complexity alone does not explain developers’ difficulty detecting them. Assertion Construction Patterns: Using the postcondition taxonomy proposed by Endres et al. [16], we explored whether particular assertion construction patterns were associated with participant performance. Assertions containing type checks (𝜒 2 = 7.99, 𝑝 = 0.005, 𝑂𝑅 = 2.10) or arithmetic equalities (𝜒 2 = 9.64, 𝑝 = 0.002, 𝑂𝑅 = 2.07) were more likely to be judged correctly, whereas implications (𝜒 2 = 3.84, 𝑝 = 0.05, 𝑂𝑅 = 0.69) and element-property assertions (𝜒 2 = 3.26, 𝑝 = 0.07, 𝑂𝑅 = 0.66) were associated with decreased performance. We return to these observations in RQ4 (Section 6.4), where our qualitative analysis suggests these construction patterns differ in the extent to which they support direct clause comparison.

RQ2: Summary—Impact of Comments: Contrary to our pre-registered hypotheses, natural-language comments did not improve programmers’ ability to evaluate postconditions. Furthermore, under-specified comments reduced accuracy relative to exact comments, while increasing developer confidence compared to no comment at all.

RQ3: Summary—Assertion Construction Patterns: Participants could recognize differences in completeness between postconditions. Although incorrect assertions tended to be slightly more complex, complexity did not explain the performance difference observed in RQ1. Some logical structures are easier for developers to evaluate than others.

6.3.1 Demographics. Finally, we found no evidence that developer experience, Python expertise, assertion-writing experience, or LLM usage predicted participant accuracy. This may indicate that programmers of all experience levels could benefit from assistance with parsing postconditions.

6.4 6.3

RQ4: Understanding Developer Reasoning

To provide insight into developer reasoning, we conduct a qualitative think-aloud, where we investigate (1) how participants understand functions and associated postcondition assertions, and (2) how do participants reason about postcondition correctness. We collected think-aloud transcripts for 50 function–postcondition pairs (10 participants, 5 stimuli each). Figure 5 describes these stimuli, which were selected to cover a range of reasoning difficulties (as determined by participant performance in our large quantitative evaluation). Comprehension: Building a mental model: As described in Section 4.2, we code each interaction for its predominant comprehension process (top-down, bottom-up, or as-needed). For the majority of responses (31/50), participants used a topdown approach, first reading the English language docstring or comment, and then confirming their understanding by parsing the

RQ3: Assertion Characteristics and Performance

Because the large asymmetry observed in RQ1 was unexpected, we conducted several exploratory analyses to better understand what characteristics of postcondition assertions were associated with participant performance. Completeness: For correct postconditions, we compared participant completeness ratings against the independently computed mutation-based bug-completeness scores from Endres et al. [16]. Participants’ completeness ratings showed a significant positive correlation with bug-completeness (Spearman’s 𝜌 = 0.18, 𝑝 < 0.001). Participants also rated correct postconditions as significantly more complete than incorrect ones (Wilcoxon rank-sum test, 𝑝 < 0.001; 7

Original Postcondition - Correct

% Identified Function common match_parens has_close_elements any_int choose_num

Incorrect

Correct

Diff

10.0 82.4 17.6 75.0 0.6

92.3 18.2 76.9 82.4 0.611

82.3 64.2 59.3 7.4 1.1

Clause Compare

Perceived Correct

Logic Walk Pos Example Neg Example

Figure 5. Functions curated for our qualitative follow-up study. These functions make up the top three and the bottom two of our stimuli, ordered by the difference between the percentage of participants who accurately identified the correct postcondition as correct and those who accurately identified the incorrect postcondition as incorrect.

Perceived Incorrect

Intuition

Original Postcondition - Incorrect Clause Compare

code. In 7 responses, participants went bottom-up, first reading the code and then confirming with comments. In 10 responses, participants showcased an as-needed approach, bouncing between the documentation and the code. In 2 responses, the participants did not provide enough spoken information for us to make a determination. Reasoning Strategies: We identified five strategies used by participants to determine postcondition correctness:

Perceived Correct

Logic Walk Pos Example Neg Example

Perceived Incorrect

Intuition

(1) Clause Comparison: The participant compared logical clauses between the function and the postcondition, and determined whether they matched. “Now, looking at this assertion, it’s entirely missing the integer criterion.” (P4) (2) Logic Walk-through: The participant traced the flow of data through the postcondition to determine whether it matched an expected data flow with respect to the function. “This is just saying, return whatever the maximum value is in that valid range if X is less than or equal to Y . . . otherwise negative 1. It seems correct.” (P3) (3) Positive Examples: The participant walked through a specific example of an input-output pair for the function that passed the postcondition. “Empty list, that would be fine. [The postcondition] would be vacuously satisfied if the return value is just one thing.” (P4) (4) Negative Examples: The participant walked through a specific example of an input-output pair for the function that failed the postcondition. “If X and Y were 3 and 3, then the return value [of the function] would be negative 1, but ... this assertion would fail.” (P10) (5) Intuition: The participant did not systematically determine postcondition correctness, but rather intuited that the postcondition seemed correct or incorrect. “I’m not totally certain, maybe I’m missing some subtle detail, but it sounds correct. It seems like it’s capturing the correct behavior... I can’t think of how I’m not right.” (P3)

Figure 6. Two Sankey diagrams showing the portion of responses from the think-aloud survey where participants employed each strategy when they perceived a postcondition to be incorrect, and when they perceived a postcondition to be correct. The diagrams are separated by the ground-truth correctness of the original postconditions.

We found that in most cases where a postcondition was perceived to be incorrect (18/29), participants used negative examples. This was often to confirm a perception formed by clause comparisons, logic walk-through, or intuition (14/18). For example, one participant viewing the choose_num function intuited that “the assert doesn’t look right”, then followed a negative example to “think through the whole thing” (P9). Occasionally, participants would attempt to confirm or break their correctness perception by trying to find a counterexample. For instance, one participant stated that the correct postcondition for function match_parens “seems right” (P10), then went on to look for counterexamples with “let me think if I can break this” (P10). Easier postconditions tended to afford clause comparison. In four of the five postconditions with at least 75% accuracy in the original study (Figure 5), every think-aloud participant used clause comparison at least once. The any_int function had the highest average response accuracy across ground-truth correct and incorrect postconditions. For this function, both postconditions afforded direct clause comparison. For example, when viewing the incorrect postcondition, P1 noted that “we are missing the requirement that they are all integers”. When viewing the correct postcondition, P6 noted that “it seems like this assertion checks all three conditions here”. We observed no clear relationship between the number of strategies used and response correctness, nor with stimulus. However,

Participants often used multiple strategies to assess the same stimulus. Figure 6 displays two Sankey diagrams summarizing strategy use across our five think-aloud functions (Figure 5), split by when the participant perceived a postcondition to be correct and when they perceived it to be incorrect. The top diagram summarizes responses for ground-truth correct postconditions, while the bottom diagram summarizes responses for ground-truth incorrect postconditions. 8

we did observe individual differences in reasoning styles: some consistently used only one or two strategies, whereas others regularly applied three or more. For instance, P2 used only 1 strategy in 4 out of 5 of their responses, while participant 8 used 4 or more strategies in 4 out of 5 of their responses. Additionally, certain participants seemed to prefer certain strategies. For instance, P3 always began with a logic walk-through, P6 leaned on clause comparisons, and P8 consistently used examples. Only 12/50 responses involved the use of intuition, with just 3 responses consisting entirely of intuition. However, when intuition was used, participants were wrong more often than they were right, with 7/12 incorrect responses. These made up the majority of the 11 incorrect responses.

complexity to flag potentially incorrect postconditions or inform recommendations to developers.

8

8.1

RQ4: Summary—Developer Reasoning: Participants primarily used top-down comprehension, relying on clause comparison, logic walk-throughs, positive and negative examples, and intuition to assess function–postcondition alignment. Participants more accurately classified postconditions that afforded clause comparison than those that did not. Negative examples were primarily used to confirm suspected incorrectness after other reasoning strategies.

7

Related Work

While recent work has made substantial progress in generating specifications, assertions, and natural-language explanations, comparatively little is known about if developers can reliably evaluate these generated reliability artifacts. Our work helps fill this gap by systematically investigating the human ability to assess machinegenerated assertions and the effect of accompanying natural-language explanations on that ability.

Natural Language Support for AI-Generated Software

Collaboration between developers and LLMs is now standard in development workflows [63]. Although LLMs perform well on many benchmarks, they remain susceptible to vulnerabilities [25], hallucinated APIs [35, 60], and misinterpretation of natural-language instructions [68]. Generated code may be fundamentally misaligned with developer intent, and that misalignment can be hard to detect [69]. Humans also cede agency to artificial intelligence in a way they do not cede it to humans [9], and readily accept incorrect code [15]. Natural-language comments are the standard mechanism for improving code comprehension [32] and can increase developer adoption of LLM-generated code [42]. Early work generated comments using static-analysis heuristics and templates [2, 14, 26, 47, 62]. More recent work formulates comment generation as machine translation from code to natural language [3, 4, 18, 22, 30, 32, 33, 39, 46, 53, 54, 61, 70, 71]. These approaches are typically evaluated using automatic metrics such as BLEU [52], or ROUGE [44], although such metrics do not necessarily correlate with developer understand, and humans may prefer human-written summaries to machine-written ones [64]. As a result, several works evaluate generated comments through human studies of how the comments impact comprehension. Prior work has measured comprehension using short-answer questions [24, 64], multiple-choice questions [73], cloze tests [8], and code writing/debugging tasks [24]. ComCat shows that expertise-guided context improves both generated comments and developer understanding [24]. B"orstler et al. find that comments conveying information beyond direct code translation, such as purpose, improve readability but not necessarily comprehension [8]. Steidl et al. propose and validate a model that evaluates comment quality based on coherence, usefulness, completeness, and consistency [65]. Finally, Wu et al. show that annotations containing formally verified assertions can themselves improve program understanding [73]. In our work, we are interested in programmers’ ability to correctly assess logical assertions. Rather than evaluating generated comments through programmer comprehension tasks, we investigate how comment quality affects programmers’ judgement of generated assertions. Because assertions have precise formal semantics, we can systematically characterize comment–assertion alignment as exact, over-specified, under-specified, or incorrect, enabling controlled study of how different forms of comment quality influence programmer judgment.

Discussion and Implications

Our findings have several implications for tool design. Developers tend to trust ML output even when it is wrong. Humans are significantly worse at recognizing when postconditions are incorrect than when they are correct; that is, they are far more likely to return false positives than false negatives. Furthermore, they trust with high confidence in both cases (see Figure 4). This aligns with previous findings regarding code repair suggestions [15], and indicates that development assistance tools should facilitate developer comprehension of suggested test oracles. Comments are insufficient to explain postconditions to developers. Comments are a standard way to increase code comprehension [32] and facilitate LLM-based development [42]. However, our findings in Section 6.2 indicate that comprehension of postconditions specifically may not increase significantly with comments, and that comments that are insufficiently specified may even harm comprehension. Future work should investigate alternative approaches to aligning human mental models to logical postcondition statements. Developers have an intuitive understanding of completeness. Humans can correctly gauge relative postcondition completeness, and can intuit that correct postconditions are more complete than incorrect postconditions (Section 6.3). This could indicate that intuited bug-completeness can potentially help ground mental models of postcondition correctness, and should be investigated in future work. Incorrect postconditions tend towards higher cyclomatic complexity than correct postconditions. While we did not find that this mediated our observed difference between participant tendency to identify correct and incorrect postconditions (Section 6.3), this is a noteworthy difference observed not only in our filtered dataset but over the entire unfiltered dataset from previous work [16]. Future work should investigate the use of cyclomatic 9

8.2

Aligning Natural Language Intent with Code

other demographics will exhibit different understanding patterns of LLM-generated comments or assertions.

A common approach to ensuring alignment between developer intent and generated code is to formalize human-written naturallanguage specifications. Lahiri argues that future AI-assisted software development should increasingly rely on behavioral artifacts such as tests, assertions, and formal specifications to capture developer intent [40]. Helping realize this vision, Prasad et al. propose a cognitively grounded framework that helps developers verify AI-generated code by reasoning over machine-generated formal artifacts [57]. Together, these works motivate specification-centric, human-in-the-loop development workflows. However, comparatively little is known about how effectively developers can evaluate the generated specifications they are asked to review. A large body of work has focused on generating formal specifications from natural-language documentation. ALICS [51] and Doc2Spec [76] infer formal language specifications from API documentation. Toradocu [23], JDoctor [7], and C2S [75] use comments to generate oracles as precondition or postcondition assertions. Swami generates executable tests from structured natural-language specifications [50]. More recent work leverages LLMs, using context gathered from existing code [49], documentation [27, 43], or GitHub issues [59] to generate specifications. In practice, natural-language documentation or developer intent can be misaligned with implementation. Thus, other approaches attempt to catch or correct such misalignment. iComment extracts implicit program rules from comments and analyzes them for implementation inconsistencies [66]. @TComment generates tests for Javadoc properties to find discrepancies between comments and code [67]. JavadocMiner uses heuristic-based metrics to assess inline documentation quality and code-alignment [38]. Zhou et al. extract logical directives from API documents and use a constraint solver to detect defects [79]. Specine repairs misalignment between LLM-generated specifications and intent [68]. Endres et al. propose empirically and qualitatively validated metrics to measure alignment between natural-language generated postconditions and intent [16]. Finally, TiCoder partially formalizes intent by interactively guiding developers via test generation [17]. Accross this literature, evaluation of alignment between natural language and formal specifications is largely automatic. While recent work has proposed systems that support developers in reviewing generated specifications and formal artifacts [57], there is limited empirical understanding of how accurately programmers can assess specification correctness or how they reason about these artifacts. In our study, we provide participants with both humanwritten intent in the form of function docstrings, and machinegenerated assertion statements. By measuring both participant accuracy at identifying assertion correctness and perceptions of completeness, we provide empirical evidence about the human verification capability underlying specification-driven development.

9

The generation model. We used GPT-3.5-turbo to generate assertion comments, and the postcondition assertions we used were also generated by this model during previous work [16]. More recent models have since been released, with updated code generation capabilities. However, as our work targets human understanding rather than model capability, we posit that our results will generalize to more recent models. The survey dataset. All of our survey questions, including LLMgenerated comments and postconditions, were built with the HumanEval dataset [10]. We chose this dataset as it was written specifically to act as a benchmark for LLM-driven code generation. However, this dataset includes only single-function Python code blocks with functionality that can be mostly summarized in a single assertion statement. We cannot be certain that our results generalize to multi-function code or to multiple assertions, though we posit that more complex code will exacerbate the concerns we have surfaced.

10

Conclusion

We perform a systematic evaluation of humans’ ability to comprehend LLM-generated logical Python postcondition statements, and the effect of associated natural-language comments of varying quality on this comprehension. We further perform an exploratory analysis of postcondition characteristics that affect human comprehension. Our work contributes to the under-studied area of human understanding of LLM-generated logical specifications.

Data Availability Our stimuli, pre-registration, data, analysis scripts, and results are available at a repository on the Open Science Framework (osf.io) [1].

Acknowledgments This work is supported by the National Science Foundation under grant no. CCF-2210243.

References [1] 2026. Replication Package for The impact of comments on the correct understanding of logical code statements. https://osf.io/q6d2m/overview?view_only= 442a1b8d82554de0baa4b4083987a6b6. [2] Nahla J. Abid, Natalia Dragan, Michael L. Collard, and Jonathan I. Maletic. 2015. Using stereotypes in the automatic generation of natural language summaries for C++ methods. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 561–565. doi:10.1109/ICSM.2015.7332514 [3] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. arXiv:2103.06333 [cs.CL] https://arxiv.org/abs/2103.06333 [4] Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 2091–2100. https://proceedings.mlr.press/v48/allamanis16.html [5] Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th international conference on software engineering (ICSE). IEEE, 712–721. [6] Tobias Baum, Kurt Schneider, and Alberto Bacchelli. 2019. Associating working memory capacity and code change ordering with code review performance. Empirical Software Engineering 24, 4 (2019), 1762–1798. [7] Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Sergio Delgado Castellanos. 2018. Translating code comments to procedure specifications (ISSTA 2018). Association for Computing Machinery, New York, NY, USA, 242–253. doi:10.1145/3213846.3213872

Limitations and Threats to Validity

The population. Our participants were recruited exclusively from a population of university students and software company employees in the United States. As such, we were only able to capture participant behavior in this demographic, and it is possible that 10

[8] Jürgen Börstler and Barbara Paech. 2016. The Role of Method Chains and Comments in Software Readability and Comprehension—An Experiment. IEEE Transactions on Software Engineering 42, 9 (2016), 886–898. doi:10.1109/TSE.2016. 2527791 [9] Cindy Candrian and Anne Scherer. 2022. Rise of the machines: Delegating decisions to autonomous AI. Computers in Human Behavior 134 (2022), 107308. doi:10.1016/j.chb.2022.107308 [10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (2021). arXiv:2107.03374 [cs.LG] [11] Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2025. Automated Code Review in Practice. In International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 425–436. doi:10.1109/ICSESEIP66354.2025.00043 [12] Douglas Curran-Everett. 2018. Explorations in statistics: the log transformation. Advances in Physiology Education 42, 2 (2018), 343–347. arXiv:https://doi.org/10.1152/advan.00018.2018 doi:10.1152/advan.00018.2018 PMID: 29761718. [13] A Dunsmore, M Roper, and M Wood. 2000. The role of comprehension in software inspection. Journal of Systems and Software 52, 2 (2000), 121–129. doi:10.1016/ S0164-1212(99)00138-7 [14] Brian P. Eddy, Jeffrey A. Robinson, Nicholas A. Kraft, and Jeffrey C. Carver. 2013. Evaluating source code summarization techniques: Replication and expansion. In International Conference on Program Comprehension. 13–22. doi:10.1109/ICPC. 2013.6613829 [15] Hadeel Eladawy, Claire Le Goues, and Yuriy Brun. 2024. Automated Program Repair, What Is It Good For? Not Absolutely Nothing!. In International Conference on Software Engineering (14–20). Lisbon, Portugal, 1017–1029. doi:10.1145/ 3597503.3639095 [16] Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions? PACMSE 1, FSE (2024), 84:1–84:24. doi:10.1145/ 3660791 [17] Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering 50, 9 (Sept. 2024), 2254–2268. doi:10.1109/tse.2024.3428972 [18] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 1536– 1547. doi:10.18653/v1/2020.findings-emnlp.139 [19] Kathrin Figl, Maria Kirchner, Sebastian Baltes, and Michael Felderer. 2025. The Influence of Code Comments on the Perceived Helpfulness of Stack Overflow Posts. 30, 6 (2025). doi:10.1007/s10664-025-10727-w [20] Emily First, Markus Rabe, Talia Ringer, and Yuriy Brun. 2023. Baldur: WholeProof Generation and Repair with Large Language Models. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (6–8). San Francisco, CA, USA, 1229–1241. doi:10.1145/ 3611643.3616243 [21] Erin Foster and Ariel Deardorff. 2017. Open Science Framework (OSF). Journal of the Medical Library Association 105 (04 2017). doi:10.5195/JMLA.2017.88 [22] Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nie, Xin Xia, and Michael Lyu. 2023. Code Structure–Guided Transformer for Source Code Summarization. ACM Trans. Softw. Eng. Methodol. 32, 1, Article 23 (Feb. 2023), 32 pages. doi:10.1145/3522674 [23] Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Automatic generation of oracles for exceptional behaviors. In International Symposium on Software Testing and Analysis (ISSTA). Saarbrücken, Genmany, 213–224. doi:10.1145/2931037.2931061 [24] Skyler Grandel, Scott Thomas Andersen, Yu Huang, and Kevin Leach. 2026. ComCat: Expertise-Guided Context Generation to Enhance Code Comprehension. 35, 3, Article 82 (Feb. 2026), 30 pages. doi:10.1145/3742475 [25] Burak Gülmez. 2026. Code generation with large language models: a survey from neural program synthesis to autonomous software development. Applied

Intelligence 56, 6 (2026), 200. doi:10.1007/s10489-026-07230-0 [26] Sonia Haiduc, Jairo Aponte, and Andrian Marcus. 2010. Supporting program comprehension with source code summarization. In 32nd International Conference on Software Engineering - Volume 2 (Cape Town, South Africa) (ICSE ’10). Association for Computing Machinery, New York, NY, USA, 223–226. doi:10.1145/1810295.1810335 [27] Ishrak Hayet, Adam Scott, and Marcelo d’Amorim. 2025. ChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance. IEEE Trans. Softw. Eng. 51, 1 (Jan. 2025), 305–319. doi:10.1109/TSE.2024.3519159 [28] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. [29] Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis. Qualitative health research 15, 9 (2005), 1277–1288. [30] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension (Gothenburg, Sweden) (ICPC ’18). Association for Computing Machinery, New York, NY, USA, 200–210. doi:10.1145/3196321.3196334 [31] Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zimmermann. 2022. Practitioners’ expectations on automated code comment generation. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 1693–1705. doi:10.1145/3510003.3510152 [32] Yuan Huang, Shaohao Huang, Huanchao Chen, Xiangping Chen, Zibin Zheng, Xiapu Luo, Nan Jia, Xinyu Hu, and Xiaocong Zhou. 2020. Towards automatically generating block comments for code snippets. Information and Software Technology 127 (2020), 106373. doi:10.1016/j.infsof.2020.106373 [33] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing Source Code using a Neural Attention Model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Katrin Erk and Noah A. Smith (Eds.). Association for Computational Linguistics, Berlin, Germany, 2073–2083. doi:10.18653/v1/P16-1195 [34] Jasmin Jahić and Ashkan Sami. 2024. State of Practice: LLMs in Software Engineering and Software Architecture. In International Conference on Software Architecture (ICSA). 311–318. doi:10.1109/ICSA-C63560.2024.00059 [35] Erik Jones, Hamid Palangi, Clarisse Simões Ribeiro, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Hassan Awadallah, and Ece Kamar. 2024. Teaching Language Models to Hallucinate Less with Synthetic Tasks. In The Twelfth International Conference on Learning Representations. https: //openreview.net/forum?id=xpw7V0P136 [36] Sungmin Kang, Louis Milliken, and Shin Yoo. 2024. Identifying Inaccurate Descriptions in LLM-generated Code Comments via Test Execution. arXiv:2406.14836 [cs.SE] https://arxiv.org/abs/2406.14836 [37] Zhanna Kaufman, Yuriy Brun, Adithya Murali, and Madeline Endres. 2025. The impact of comments on the correct understanding of logical code statements. osf.io/5f73u [38] Ninus Khamis, René Witte, and Juergen Rilling. 2010. Automatic Quality Assessment of Source Code Comments: The JavadocMiner. In Natural Language Processing and Information Systems, Christina J. Hopfe, Yacine Rezgui, Elisabeth Métais, Alun Preece, and Haijiang Li (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 68–79. [39] Junaed Younus Khan and Gias Uddin. 2023. Automatic Code Documentation Generation Using GPT-3. In International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE ’22). Association for Computing Machinery, New York, NY, USA, Article 174, 6 pages. doi:10.1145/3551349.3559548 [40] Shuvendu K Lahiri. 2026. Intent formalization: A grand challenge for reliable coding in the age of AI agents. arXiv preprint arXiv:2603.17150 (2026). [41] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In International Conference on Software Engineering (ICSE). IEEE, 919–931. [42] Changwen Li, Christoph Treude, and Ofir Turel. 2026. Do comments and expertise still matter? An experiment on programmers’ adoption of AI-generated JavaScript code. Journal of Systems and Software 231 (2026), 112634. doi:10.1016/j.jss.2025. 112634 [43] Hui Li, Zhen Dong, Siao Wang, Hui Zhang, Liwei Shen, Xin Peng, and Dongdong She. 2025. Extracting Formal Specifications From Documents Using LLMS for Test Automation. In International Conference on Program Comprehension (ICPC). 1–12. doi:10.1109/ICPC66645.2025.00039 [44] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Annual Meeting of the Association for Computational Linguistics. https://api. semanticscholar.org/CorpusID:964287 [45] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=1qvx610Cu7 11

[46] Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, and Yang Liu. 2021. Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. arXiv:2006.05405 [cs.LG] https://arxiv.org/abs/2006.05405 [47] Paul W. McBurney and Collin McMillan. 2014. Automatic documentation generation via source code summarization of method context. In Proceedings of the 22nd International Conference on Program Comprehension (Hyderabad, India) (ICPC 2014). ACM, New York, NY, USA, 279–290. doi:10.1145/2597008.2597149 [48] Vishal Misra, Jakku Sai Krupa Reddy, and Sridhar Chimalakonda. 2020. Is there a correlation between code comments and issues? an exploratory study. In Proceedings of the 35th Annual ACM Symposium on Applied Computing (Brno, Czech Republic) (SAC ’20). Association for Computing Machinery, New York, NY, USA, 110–117. doi:10.1145/3341105.3374009 [49] Davide Molinelli, Luca Di Grazia, Alberto Martin-Lopez, Michael D. Ernst, and Mauro Pezzè. 2025. Do LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset. In International Conference on Automated Software Engineering (ASE). 278–290. doi:10.1109/ASE63991.2025.00031 [50] Manish Motwani and Yuriy Brun. 2019. Automatically Generating Precise Oracles from Structured Natural Language Specifications. In International Conference on Software Engineering (29–31). Montreal, QC, Canada, 188–199. doi:10.1109/ICSE. 2019.00035 [51] Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring method specifications from natural language API descriptions. In 2012 34th International Conference on Software Engineering (ICSE). 815–825. doi:10.1109/ICSE.2012.6227137 [52] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (Philadelphia, Pennsylvania) (ACL ’02). Association for Computational Linguistics, USA, 311–318. doi:10.3115/1073083.1073135 [53] Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and KaiWei Chang. 2021. Retrieval Augmented Code Generation and Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, MarieFrancine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 2719–2734. doi:10.18653/v1/2021.findings-emnlp.232 [54] Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Annibal, Alec Peltekian, and Yanfang Ye. 2021. CoTexT: Multi-task Learning with Code-Text Transformer. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), Royi Lachmy, Ziyu Yao, Greg Durrett, Milos Gligoric, Junyi Jessy Li, Ray Mooney, Graham Neubig, Yu Su, Huan Sun, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 40–47. doi:10.18653/v1/2021.nlp4prog-1.5 [55] Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good vibrations? A qualitative study of co-creation, communication, flow, and trust in vibe coding. arXiv preprint arXiv:2509.12491 (2025). [56] Posit team. 2025. RStudio: Integrated Development Environment for R. Posit Software, PBC, Boston, MA. http://www.posit.co/ [57] Siddhartha Prasad, Skyler Austen, Kathi Fisler, and Shriram Krishnamurthi. 2026. Meaningful Human-in-the-Loop Checking of GenAI Synthesis for Restricted Languages. In 40th European Conference on Object-Oriented Programming (ECOOP 2026), Vol. 372. 22:1–22:31. doi:10.4230/LIPIcs.ECOOP.2026.22 [58] Kenneth J Rothman. 1990. No adjustments are needed for multiple comparisons. Epidemiology 1, 1 (1990), 43–46. [59] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. SpecRover: Code Intent Extraction via LLMs. In International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE ’25). IEEE Press, 963–974. doi:10.1109/ICSE55347. 2025.00080 [60] Advait Sarkar, Xiaotong, Xu, Neil Toronto, Ian Drosos, and Christian Poelitz. 2024. When Copilot Becomes Autopilot: Generative AI’s Critical Risk to Knowledge Work and a Critical Solution. arXiv:2412.15030 [cs.HC] https://arxiv.org/abs/ 2412.15030 [61] Rishab Sharma, Fuxiang Chen, and Fatemeh Fard. 2022. LAMNER: code comment generation using character language model and named entity recognition. In International Conference on Program Comprehension (Virtual Event) (ICPC ’22). ACM, New York, NY, USA, 48–59. doi:10.1145/3524610.3527924 [62] Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K. VijayShanker. 2010. Towards automatically generating summary comments for Java methods. In International Conference on Automated Software Engineering (Antwerp, Belgium) (ASE ’10). ACM, New York, NY, USA, 43–52. doi:10.1145/ 1858996.1859006 [63] Stack Overflow. 2025. 2025 Stack Overflow Developer Survey. https://survey. stackoverflow.co/2025 Accessed June 2026. [64] Sean Stapleton, Yashmeet Gambhir, Alexander LeClair, Zachary Eberhart, Westley Weimer, Kevin Leach, and Yu Huang. 2020. A Human Study of Comprehension and Code Summarization. In Proceedings of the 28th International Conference on Program Comprehension (Seoul, Republic of Korea) (ICPC ’20). Association for Computing Machinery, New York, NY, USA, 2–13. doi:10.1145/3387904.3389258

[65] Daniela Steidl, Benjamin Hummel, and Elmar Jürgens. 2013. Quality analysis of source code comments. International Conference on Program Comprehension (ICPC) (2013), 83–92. https://api.semanticscholar.org/CorpusID:16657129 [66] Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*icomment: bugs or bad comments?*/. SIGOPS Oper. Syst. Rev. 41, 6 (Oct. 2007), 145–158. doi:10.1145/1323293.1294276 [67] Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. 260–269. doi:10.1109/ICST.2012.106 [68] Zhao Tian and Junjie Chen. 2025. Aligning Requirement for Large Language Model’s Code Generation. arXiv:2509.01313 [cs.SE] https://arxiv.org/abs/2509. 01313 [69] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages. doi:10.1145/3491101.3519665 [70] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. Improving automatic source code summarization via deep reinforcement learning. In International Conference on Automated Software Engineering (Montpellier, France) (ASE ’18). Association for Computing Machinery, New York, NY, USA, 397–407. doi:10.1145/3238147.3238206 [71] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wentau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 8696–8708. doi:10.18653/v1/2021.emnlp-main.685 Best practice in statistics: The use of log [72] Robert M West. 2022. transformation. Annals of Clinical Biochemistry 59, 3 (2022), 162–165. arXiv:https://doi.org/10.1177/00045632211050531 doi:10.1177/00045632211050531 PMID: 34666549. [73] Haoze Wu, Rocky Klopfenstein, Keith Farkas, and Nina Narodytska. 2026. Viverra: Text-to-Code with Guarantees. arXiv:2605.14972 [cs.SE] https://arxiv.org/abs/ 2605.14972 [74] Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. 2018. Measuring Program Comprehension: A Large-Scale Field Study with Professionals. IEEE Transactions on Software Engineering 44, 10 (2018), 951–976. doi:10.1109/TSE.2017.2734091 [75] Juan Zhai, Yu Shi, Minxue Pan, Guian Zhou, Yongxiang Liu, Chunrong Fang, Shiqing Ma, Lin Tan, and Xiangyu Zhang. 2020. C2S: translating natural language comments to formal program specifications (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 25–37. doi:10.1145/3368089.3409716 [76] Hao Zhong, Lu Zhang, Tao Xie, and Hong Mei. 2009. Inferring Resource Specifications from Natural Language API Documentation. In International Conference on Automated Software Engineering. 307–318. doi:10.1109/ASE.2009.94 [77] Li Zhong and Zilong Wang. 2024. Can LLM replace stack overflow? a study on robustness and reliability of large language model code generation (AAAI’24/IAAI’24/EAAI’24). AAAI Press, Article 2437, 9 pages. doi:10.1609/aaai. v38i19.30185 [78] Suzhen Zhong, Shayan Noei, Ying Zou, and Bram Adams. 2026. Human–AI Synergy in Agentic Code Review. arXiv preprint arXiv:2603.15911 (2026). [79] Yu Zhou, Ruihang Gu, Taolue Chen, Zhiqiu Huang, Sebastiano Panichella, and Harald Gall. 2017. Analyzing APIs Documentation and Code to Detect Directive Defects. In International Conference on Software Engineering. 27–37. doi:10.1109/ ICSE.2017.11 [80] Yangtian Zi, Luisa Li, Arjun Guha, Carolyn Anderson, and Molly Q Feldman. 2025. “I Would Have Written My Code Differently’: Beginners Struggle to Understand LLM-Generated Code. In International Conference on the Foundations of Software Engineering (Clarion Hotel Trondheim, Trondheim, Norway) (FSE Companion ’25). Association for Computing Machinery, New York, NY, USA, 1479–1488. doi:10.1145/3696630.3731663 [81] Christo Zietsman. 2026. The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review. arXiv preprint arXiv:2603.25773 (2026).

12