SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References

arXiv:2608.04195v1 [cs.SE] 4 Aug 2026

SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References Simantika Bhattacharjee Dristi

Matthew B. Dwyer

University of Virginia Charlottesville, Virginia, USA nwc8gr@virginia.edu

University of Virginia Charlottesville, Virginia, USA matthewbdwyer@virginia.edu

Abstract—Source code summaries have traditionally been evaluated from a human developer’s perspective, with quality determined by how closely they resemble developer-written references and how well they align with human preferences. But this overlooks a growing reality: LLM-based tools and agents increasingly consume code summaries as inputs for software engineering (SE) tasks, and what makes a summary useful for a consuming agent on a task remains largely unexplored. To bridge this gap, we propose S ONAR, a reference-free framework that evaluates source code summaries along four dimensions: Correctness, Abstraction, Conciseness, and Fluency. Rather than optimizing for a pre-written “gold standard”, S ONAR introduces a novel code regeneration-based approach that uses a summary to regenerate code and leverages that reconstruction as a quality signal of the summary. This provides an empirical grounding that requires neither a reference summary nor the subjective judgment of humans or LLMs. We evaluate S ONAR’s dimensions on their ability to influence LLM performance across four downstream SE tasks. We find that Correctness, followed by Abstraction, significantly correlates with LLM performance, with correlations up to 14× higher than the best baseline. Conciseness and Fluency, though widely valued by human developers, remain mostly insignificant to an LLM consumer, suggesting that what makes a summary useful is task- and consumer-dependent. Through a large-scale evaluation of 11 popular LLMs using S ONAR, we further identify the strengths and weaknesses of different models across each quality dimension, while offering insights to facilitate future research on task-aware summarization. Index Terms—Code summarization, reference-free, taskspecific quality attributes, code summary evaluation

I. I NTRODUCTION Like many other software engineering (SE) tasks, source code summarization – generating natural-language descriptions of code functionality and intent [1] – is increasingly driven by large language models (LLMs) [2], [3]. Good summaries offer developers and automated agents a natural language anchor for reasoning about code by accurately and concisely describing behavior, while abstracting away implementation details. Although many techniques have been proposed for evaluating code summaries [2], [4]–[10], we argue that the true value of a summary should be judged in terms of the tasks it supports and the consumers it serves. The growth of agentic software engineering [11] has reshaped how software engineering tasks are executed and elevated agents as a principal consumer of task-related information – like summaries. From this perspective, prior

evaluation methods for source code summaries share two significant limitations. First, they treat developer-written documentation as a gold standard. From reference-based metrics such as BLEU [4], ROUGE [8], and METEOR [7] to the reference-free metric, SIDE [6], existing evaluation techniques largely optimize toward developer-written documentation, despite prior evidence that it can be incomplete or ambiguous [12]. Second, they assume human developers are the sole consumers of summaries - an outdated assumption on which the entire evaluation paradigm rests. This was a reasonable assumption when summarization research first emerged, with the stated goal of helping developers understand unfamiliar code faster. But the landscape has since shifted; LLM-based tools and agents now use code summaries as inputs for a wide range of downstream SE tasks [13]–[16]. As a result, a developer-only view no longer reflects how summaries are used or how they should be evaluated. These limitations are coupled: current techniques optimize summaries toward a flawed reference using metrics tailored to a class of consumers whose role in specific software engineering tasks is diminishing. To resolve these limitations, in this work, we adopt a three-part strategy. First, we identify distinct quality dimensions across which SE tasks may vary in their sensitivity. Second, we introduce a reference-free evaluation framework that measures multiple summary dimensions using only the source code and the summary itself, without relying on any pre-existing documentation as a gold standard to match. Finally, rather than assessing how important each dimension is to human developers, we measure how they impact LLM performance in practice, shifting code summary evaluation toward an automated consumer’s perspective. We begin with an observation that motivates our framework: for the same source code and prompt, different LLMs generate summaries that vary considerably in accuracy, reference to implementation detail, redundancy, and readability. For example, as Fig. 1 shows, CodeGemma generates a summary that is concise and readable but contains inaccurate behavioral information. Gemini’s summary, on the other hand, correctly describes the function, but includes many implementation details that tightly couple the summary to a single implementation of the underlying program behavior. Mistral offers a more balanced summary that accurately captures the program behavior, avoids unnecessary implementation detail, and remains concise and

Source Code def greatest_common_divisor(a: int, b: int) -> int: while b: a, b = b, a % b return a

Summary by CodeGemma Returns the largest number that evenly divides either of the two integer inputs.

Summary by Gemini This Python function greatest_common_divisor takes two integer arguments (a and b) and calculates their greatest common divisor (GCD). It implements the Euclidean algorithm. The function repeatedly performs the following steps: (1) It replaces the first number (a) with the second number (b). (2) It replaces the second number (b) with the remainder of the division of the original first number by the original second number (a % b). This process continues in a loop until the second number (b) becomes zero. At that point, the first number (a) holds the greatest common divisor, which is then returned.

Summary by Mistral The function returns the largest integer that can evenly divide both integer inputs a and b.

Fig. 1: LLM-generated summaries vary across quality dimensions under the same code and same prompt. Red borders denote incorrect information. fluent enough to be readable. These observations illustrate the multidimensional nature of summary quality, leading us to identify four key dimensions of an LLM-generated code summary: Correctness - the extent to which it accurately captures the underlying code’s behavior; Abstraction - the extent to which it avoids mentioning low-level implementation details; Conciseness - the extent to which it is free of redundant or verbose information; and Fluency - the extent to which it is expressed naturally and readably. To measure each of these dimensions in a principled way, we propose S ONAR, a reference-free, multi-dimensional evaluation framework for source code summaries. S ONAR is built on a novel code regeneration-based approach, where the summary is used to regenerate code(s), and the regenerated code(s) is then analyzed as a signal of summary quality. By going from code to summary and back to code, we leverage the causal relationship between a summary and the code(s) regenerated from it to empirically ground evaluation of code summaries, without relying on a reference, human, or LLM as a judge. As described in §III, we utilize the concept of round-trip correctness (RTC) [17] to measure three of the S ONAR dimensions. For Correctness, given a summary and essential context, an independent code-generator LLM regenerates code, and the degree of functional equivalence between the original and regenerated code determines the summary’s correctness. For Abstraction, we adapt RTC in a novel way to produce a pool of regenerated implementations and then we measure the diversity of that pool. A more abstract summary allows code generators to explore a broader implementation space, resulting in a diverse set of regenerated implementations. For Conciseness, we compress the summary while retaining

functional equivalence as determined by RTC and interpret the largest such degree of compression as a measure of unnecessary information in the original summary. While accuracy (loosely analogous to correctness) and conciseness have previously been recognized as code summary quality aspects [18], their evaluation has remained mostly text-centric with limited consideration of the functional behavior encoded in the summary. Abstraction, meanwhile, has been entirely overlooked in existing literature. As we show through our experimental study in §IV, S ONAR addresses limitations of prior approaches. First, S ONAR effectively captures the quality dimensions of code summaries, achieving a 90% overall agreement rate with ground-truth annotations across dimensions. Second, at least one S ONAR dimension significantly correlates with LLM performance across a set of downstream tasks, increasing the predictive signal by a factor of 1.3 − 14 over the best baseline. Moreover, the most predictive dimension varies by task, calling for a shift toward task-aware summarization. Finally, S ONAR-based prompting provides a lightweight way to improve task-specific LLM summary generation, yielding up to a 10 percentage point (pp) gain in abstraction without any additional training. The primary contributions of this paper lie in: (1) introducing abstraction as a quality attribute of code summaries that is relevant for some tasks; (2) defining code regenerationbased measures for summary correctness, abstraction, and conciseness; (3) defining S ONAR, a reference-free, multidimensional evaluation framework using those measures; (4) demonstrating, through a large-scale evaluation on downstream tasks, that different summary quality dimensions are important for different tasks; and (5) showing that S ONAR-guided prompting can improve LLM-generated summaries along taskrelevant dimensions. II. BACKGROUND AND R ELATED W ORK A. Round-Trip-Correctness (RTC) The core idea is to form a round trip from code to natural language and back to code. A natural language description is first generated from code, new code is synthesized from that description, and the resulting code is compared against the original. RTC has traditionally been used as an evaluation method for code LLMs [17], [19]. B. Evaluation of Code Summaries Reference-based methods have long dominated code summary evaluation [2], [20]. These include exact match-based metrics, such as BLEU [4], METEOR [7], ROUGE [8], that measure word overlap, and semantic match-based metrics, such as BERTScore [21], BLEURT [22], SentenceBERT [23], SimLLM [5], that assess semantic closeness to a reference. While widely used, these methods rely on developer-written documentation as reference summaries, which is not always available, and even when available, is often noisy or low quality [12], [14], [24]. Thus, the reference may be an unreliable oracle, causing high-quality summaries to be unfairly penalized when they differ from it.

Limited research has focused on developing reference-free evaluation methods for code summaries. SIDE [6] reduces reference dependence at inference time, but its training continues to assume developer-written documentation as ground truth, thereby retaining a reference dependence. Moreover, SIDE measures quality only through embedding similarity between code and summary, rather than execution-based evidence. Therefore, its effectiveness in capturing true semantic alignment remains unclear. A more recent line of work employs LLMs to judge code summary quality across dimensions such as content adequacy, coherence, fluency, understandability, and conciseness [9], [10], [18]. However, these evaluations rely solely on the LLM’s internal reasoning about the summary text, with no empirical basis to justify the assigned scores. S ONAR goes beyond all existing evaluation methods by eliminating the need for both training and reference summaries and by introducing regenerated code as a novel empirical anchor to ground the scores assigned to a summary under test. C. Code Summaries in Automated Workflows Natural language code summaries are playing an increasingly active role in automated software engineering workflows, from code translation and test generation to bug localization [13]–[16], [25]–[29]. For example, Ahmad et al. [16] leverage summaries as a pivot for unsupervised code translation, whereas Hossain et al. [14] use LLM-generated summaries as input to guide test oracle generation. Recent work also uses code summaries as proxy specifications for web code generation to evaluate the accessibility of LLM-generated code [29]. Despite these advances and evidence that code comments, and by extension, code summaries, can steer model behavior in meaningful ways [30], evaluation has remained entirely human-centric [2], [5], [6], [20], with no prior work assessing summaries from an LLM’s perspective. III. T HE SONAR A PPROACH S ONAR is built on the principle of “Regenerate, then Evaluate” - given a summary under test and its underlying source code, it first generates additional source code representation from the summary, and then probes those regenerated codes to infer the quality of the summary. Code regeneration provides a behavioral signal for summary quality. For example, if a code summary accurately captures the functional behavior of the underlying source code, then a code generator model, prompted with that summary, should be able to regenerate a functionally equivalent version of the source code. As another example, summary that is decoupled from low-level implementation details should allow a diverse pool of regenerated implementations, while one rich in such implementation details shrinks that space considerably. A. Framework Figure 2 presents an overview of the S ONAR framework, which comprises four independent scorer modules, each mea-

suring one quality dimension. Given a source code, x and its corresponding summary s, S ONAR returns four scores: SONAR(x, s) = (C(x, s), A(x, s), N (x, s), F (s)) where C measures Correctness, N measures Conciseness, A measures Abstraction, and F measures Fluency. Each score lies in [0, 1], where higher is better. In a pre-processing step, the minimal code context, c, required to reconstruct the source code from the summary alone, is extracted from x. For function-level code, this context comprises the function signature - the function name, return type, and parameter names and types, along with any import statements the code depends on. The scorer modules share a code generator LLM, G, which takes the summary s and extracted context c as input and generates a candidate implementation: x̂ = G(s, c) This regenerated code x̂ is analyzed by multiple S ONAR modules to compute the quality scores. Each scorer module is discussed in detail in the sections that follow. 1) Correctness Scorer: This module quantifies correctness under the intuition that the more functionally similar x̂ is to x, the more correct the summary. Definition 1 (Correctness): Given source code x, summary s, and the regenerated code, x̂, the correctness of s with respect to x is: C(x, s) = FuncSim(x, x̂) where FuncSim(x, x̂) measures functional similarity. The scorer first instantiates G with s and c to generate x̂, and then computes functional similarity between x and x̂ using the differential fuzzing-based approach of Dristi et al. [31], [32]. A test input set T is generated on the fly, and both x and x̂ are run on each input. Functional similarity is defined as the ratio of inputs for which both produce identical outputs: FuncSim(x, x̂) = |{t ∈ T | x(t) = x̂(t)}| /|T |

(1)

Prior RTC work has either used unit test execution or reference-based metrics such as CodeBLEU [33] as a functional similarity oracle. However, unit testing is infeasible in our setting since very few code-summary pairs in available datasets [34], [35] come with associated test cases. Referencebased metrics, meanwhile, suffer from surface bias [31], failing to reason about deeper code semantics. Our choice of differential fuzzing as the oracle resolves both issues: the test input set, T , is automatically generated using a byte-level fuzzer with no predefined tests needed, while execution-level comparison provides stronger evidence for the functional similarity score. However, when the source code does not come with sufficient context to be run in isolation, reference-based code evaluation metrics can serve as a proxy for functional similarity.

Fig. 2: The SONAR Framework 2) Abstraction Scorer: This module approximates abstraction in a code summary through the diversity of regenerated implementations: the more implementation details a summary encodes, the more constrained the regeneration space is, and the less diverse the regenerated implementations will be. Definition 2 (Abstraction): Given source code x, summary s, and a set of regenerated implementations, X̂, the abstraction of s with respect to x is: A(x, s) = ImplDiv(X̂) where ImplDiv(X̂) is the implementation-level diversity of X̂. In this module, first, X̂ is constructed using a pool of code generators G = {G1 , . . . , GU }. Since different LLMs exhibit different structural biases and coding styles, cross-model sampling enables broader exploration of the implementation space than any single model could provide. V implementations per code generator are further sampled at different temperatures to capture implementation diversity both within and across models. Given s, and c, this yields U, V

X̂ = {x̂i,j | x̂i,j ∼ Gi (s, c)}i=1, j=1 a total of U × V regenerated implementations. Whereas the correctness scorer (§ III-A1) prompts G without additional instruction, each code generator here is explicitly instructed to preserve any implementation-specific details stated in the summary, while making independent implementation choices only when the summary leaves them unspecified. Thus, abstract summaries allow the generators to explore a wider implementation space, while implementationheavy ones constrain them and reduce diversity in X̂. Next, the abstraction scorer computes the pairwise implementation-level similarity, ImplSim, between each pair of implementations in X̂. Implementation details embedded in a summary can propagate to regenerated code at both structural and lexical levels. As shown in Fig. 3, summary (a) encodes specific data structures and variable names, which

all regenerated implementations mimic. Summary (b), by contrast, describes the function behavior at a high level, allowing different regenerated implementations to use different variable names and data structures. Based on this observation, we decompose ImplSim into two components: SimCPG , which measures structural similarity between two codes using Code Property Graphs (CPGs) extracted via Joern [36], an opensource program analysis tool, and SimToken , which measures lexical similarity through token-level overlap: ImplSim(x̂i , x̂j ) = (SimCPG (x̂i , x̂j ) + SimToken (x̂i , x̂j ))/2 The implementation-level diversity of X̂ is then calculated as the complement of the average pairwise similarity across all implementations in X̂: X 2 ImplDiv(X̂) = 1 − ImplSim(x̂i , x̂j ) k(k − 1) i<j where k = U × V . While prior work has largely relied on Abstract Syntax Trees (ASTs) for structural comparison [31], [37], [38], CPGs subsume ASTs by additionally capturing control flow and data dependencies, providing a more discriminative structural similarity measure. For each implementation, we extract its CPG and represent it as a set of labeled edges. Each edge in this set connects two program elements (statements, variables, etc.) and is labeled by its relation type (e.g., AST, control-flow, or data-dependency). We compute SimCPG as the Jaccard similarity between the two CPG edge sets. Since Jaccard similarity is commonly used in code analysis and clone detection [39], [40], we also use it for SimToken , though the proposed approach to measure abstraction can be used with other similarity measures as well. To ensure Simtoken captures meaningful implementation overlap, we exclude comments and docstrings, as they do not represent implementation content. We also discard languagespecific keywords and tokens appearing in the context, c, since their overlap across implementations is unavoidable and therefore uninformative.

Summary (a) Takes numbers: List[float] and threshold: float. Iterates using enumerate with idx and idx2 . If idx != idx2 , computes absolute distance between elements and returns True if distance is less than threshold, False otherwise.

Summary (b) Returns True if any two elements in the list are closer to each other than the specified threshold, False otherwise.

Implementation 1(a): for idx , elem in enumerate (numbers): for idx2 , elem2 in enumerate(numbers): if idx != idx2 : distance = abs( elem - elem2) if distance < threshold: return True return False

Implementation 2(a): return any( abs(a - b) < threshold for i, a in enumerate(numbers) for j, b in enumerate(numbers) if i != j )

Implementation 1(b): for idx , e1 in enumerate(numbers): for idx2 , e2 in enumerate (numbers): if idx != idx2 : if abs(e1 - e2) < threshold: return True return False

Implementation 2(b): sorted_nums = sorted(numbers) for i in range( len(sorted_nums) - 1): if sorted_nums[i+1] sorted_nums[i] < threshold: return True return False

Fig. 3: Implementation details in a summary affect the diversity of regenerated implementations. Red borders highlight implementation-specific details propagated from the summary to the regenerated codes. 3) Conciseness Scorer: While conciseness is traditionally assessed by measuring redundancy, determining what is redundant in a code summary is non-trivial. This module takes a behavioral approach - content is redundant if its removal still allows functionally equivalent code to be regenerated. Redundancy is thus reflected in how much a summary can be compressed while preserving behavior. Definition 3 (Conciseness): Given source code x, summary s, and a set of compressed summaries S = {s1 , . . . , sn } derived from s, let x̂ = G(s, c) be the code regenerated from the original summary and x̂i = G(si , c) be the code regenerated from each si ∈ S. The conciseness of s with respect to x is: N (x, s) = 1 −

max i∈{1,...,n}

ri

where ri is the compression ratio of si , defined as: ( 1 − |s|s|i | if FuncSim(x̂i , x̂) = 1.0 ri = 0 otherwise

(2)

preserve behavior, this provides stronger evidence that the original summary is already concise. 4) Fluency Scorer: Fluency is the only S ONAR dimension that treats the summary as pure natural language, independent of the source code. With prior works consistently highlighting fluency as a property valued by developers [2], [3], [6], [20], its inclusion in S ONAR allows us to investigate whether LLM agents and human developers prioritize the same aspects of summary quality. Since code regeneration yields no useful signal for a purely textual property, this module follows the established practice of measuring fluency as the inverse of language model perplexity [41], [42]. Definition 4 (Fluency): Given summary s and language model L, the fluency of s is defined as: F (s) = 1/PPL(s) where PPL(s) denotes the perplexity of s under L. IV. E XPERIMENTAL S TUDY

(3)

The conciseness scorer starts by generating x̂ from s using G. At the same time, a compressor LLM, P , is prompted n times independently to compress s while preserving all behavioral information, producing the set of compressed summaries, S. Each si ∈ S is used to generate x̂i via G, which is evaluated for functional similarity with x̂ using Eq. 1. Next, the compression ratio ri (Eq. 3) is calculated to assess how much of s can be removed while preserving behavior. A larger ri reflects greater redundancy, while ri = 0 indicates that si failed to preserve behavior. The conciseness score is then denoted by the complement of the maximum compression ratio in S (Eq. 2). A summary that cannot be compressed further without losing behavior has all ri = 0 and therefore receives a score of 1.0. On the other hand, the more redundancy a summary carries, the greater behaviorpreserving compression it allows, and the lower its conciseness score. We use n compressions rather than one to get a more reliable conciseness estimate. Multiple compressions increase the likelihood of finding the tightest behavior-preserving compression. Conversely, if none of the n compressed summaries

Our experimental study is structured around four research questions : RQ1: How reliably does S ONAR capture different quality dimensions of a code summary? RQ2: How do S ONAR’s quality dimensions correlate with LLM performance in downstream software engineering tasks? RQ3: How do different LLMs compare in terms of summary quality when evaluated using S ONAR? RQ4: How sensitive is S ONAR to the choice of LLM across the framework’s modules? To address these questions, we generate summaries using LLMs, score them with S ONAR, and then analyze the summaries and their corresponding scores. To note, all summaries analyzed in this paper are LLM-generated; while manual curation could yield greater within-dimension variance, it would not reflect how summarization works in practice. This section outlines the summary generation setup, the implementation details of S ONAR, and the evaluation results and analysis. A. Summary Generation 1) Datasets: We select the datasets based on two criteria. First, each dataset must include source code paired with human-written comments, docstrings, or prompts that can serve as reference summaries for comparison with baselines

TABLE I: Agreement rate (%): SONAR vs LLM-only Scorers. Vanilla Prompt Task: Summarize the following Python function in natural language. Instructions: - Output ONLY the summary. - Do NOT include any explanations, comments, or markdown Python function: ⟨Source Code⟩

Dimension-Aware Prompt Task: Summarize the following Python function in natural language, such that your summary satisfies these four qualities: - Correctness: accurately describe the behavior of the function. - Abstraction: avoid low-level implementation details. - Conciseness: avoid redundancy. - Fluency: be grammatically correct, and naturally readable. Instructions: - Output ONLY the summary. - Do NOT include any explanations, comments, or markdown Python function: ⟨Source Code⟩

Few-Shot Prompt Task: Summarize the following Python function in natural language. Instructions: - Output ONLY the summary. - Do NOT include any explanations, comments, or markdown The following examples illustrate the expected output: Example 1: Python function: ⟨Example Source Code 1⟩ Summary: ⟨Example Summary 1⟩ Example N: Python function: ⟨Example Source Code N⟩ Summary: ⟨Example Summary N⟩ Now summarize this function: Python function: ⟨Source Code⟩

Fig. 4: Prompt templates used for summary generation. (§ IV-D). Second, the code must be executable, either standalone or with sufficient context, to support differential fuzzing for Correctness1 and Conciseness scoring (Eq.1), and downstream task evaluation (§ IV-D). However, almost every widely used code summarization dataset [34], [35], [43], [44], while providing reference documentation, contains non-executable code. This gap can be attributed to the text-only nature of existing summary evaluation, where no prior work has used execution-based validation to evaluate summaries. We therefore use popular coding benchmarks instead: HumanEval [45], MBPP [46], BigCodeBench [47], and The Vault [48], all of which meet both selection criteria. We construct a pool of 500 functions, comprising all 164 from HumanEval and 112 sampled from each remaining dataset, for summary generation and subsequent evaluation. While HumanEval and MBPP include simple standalone functions, BigCodeBench and The Vault contain more complex programming tasks, together providing a function pool with varying levels of code complexity. Code and source code are used interchangeably throughout this paper to refer to function-level Python code. 2) Large Language Models (LLMs): As code summarizers, we select 11 popular LLMs spanning a range of model families and sizes (Fig. 5). These include large closed-source models 1 Italicized dimension names (e.g., Correctness) denote S ONAR ’s measurement of that dimension; non-italicized usage denotes the general concept.

Scorer SONAR Gemini 3.1 Pro Claude Opus 4.8

Correctness

Abstraction

Conciseness

Overall

96 51 60

87 76 77

87 62 79

90 63 72

such as Gemini 2.5 Flash and Kimi K2.5, medium-scale open-source models like CodeLlama-13B and Mistral-7B, and smaller models like Phi-3.5 Mini and CodeT5 (220M), a compact Seq2Seq model fine-tuned specifically on the summarization task. For summary generation, all models are run with identical decoding parameters and temperature settings. 3) Prompting Strategies: Of our selected models, 9 are instruction-tuned, excluding CodeT5 and StarCoder2. For the instruction-tuned models, we use three prompting strategies to generate summaries (Fig. 4): Vanilla Prompting, which simply asks the model to summarize a function with no additional guidance; Dimension-Aware Prompting, which explicitly instructs the model to satisfy S ONAR’s four quality dimensions; and Few-Shot Prompting, which augments the prompt with 2 examples to steer the model toward desired summaries. B. Implementation Details of S ONAR To implement S ONAR (See § III-A for notation), we use Gemini-2.5 Flash as G in the Correctness Scorer. In the Abstraction Scorer, we use U = 3 generators in G: Gemini2.5 Flash, Qwen-2.5-Coder-32B, and OLMo-32B-Instruct, all known for their strong code-generation ability [49], [50]. From each model, we sample V = 3 implementations, yielding a pool of 9 regenerated implementations for Abstraction calculation. Gemini-2.5 Flash serves as both the compressor, P , and generator, G, in the Conciseness Scorer, with n = 3 summaries forming S. Finally, following prior work [42], we use GPT-2 as the language model, L, in the Fluency Scorer. C. RQ1: Reliability Assessment of S ONAR’s Scores 1) Experimental Setup: This experiment evaluates the extent to which each S ONAR dimension captures the quality aspect it is designed to measure. Since Fluency follows prior work, we focus on the dimensions for which S ONAR proposes new measurement methods: Correctness, Abstraction, and Conciseness. Because existing works do not empirically evaluate these dimensions [51], establishing a reliable ground truth for validation remains challenging. We therefore validate these dimensions using human preference labels. For each dimension, we sample 100 pairs of summaries (A, B), where both summaries belong to the same code. Two authors independently annotate each pair for the target dimension by choosing “A,” “B,” or “Not Sure,” indicating which summary is more correct, more abstract, or more concise. Only pairs where both annotators unanimously agree on a preference are retained as ground truth labels. This results in 84 Correctness pairs, 86 Abstraction pairs, and 89 Conciseness pairs. We compute agreement rate as the percentage of pairs where S ONAR assigns higher score to the human-preferred summary.

TABLE II: Experimental setup for each downstream SE task. “n” denotes the number of summaries used per task. “Qwen”, “OLMo”, “Gemini” refer to Qwen2.5-Coder-32B, OLMo32B-Instruct, and Gemini-2.5-Flash, respectively. Task

Metric

Models

n

Retrieval

MRR [34]

UniXcoder, CodeT5+, GraphCodeBERT

1414

Qwen, OLMo, Gemini

1414

Qwen, OLMo, Gemini

1085

Qwen, OLMo, Gemini

2152

Translation Optimization Test Oracle Generation

CA@1 [53], Edit Distance [54] Pass@1 [45], Speedup [55] Success Rate [56]

For comparison, we use two independent LLM-based scorers as baselines: Gemini 3.1 Pro and Claude Opus 4.8, both ranking within the top 4% of reasoning models as of mid2026 [52]. Each LLM scorer receives a code, its summary, and a clear definition of the target dimension, and is asked to assign a score between 0 and 1 for that dimension. Agreement rates of S ONAR and the baselines are shown in Table I. 2) Results and Analysis: S ONAR achieves 87–96% agreement across dimensions and 90% overall (Table I), compared to the best baseline, at 72%. This 18 pp improvement over the best baseline underscores S ONAR’s reliability in capturing the intended quality dimension. The highest agreement is for correctness at 96%, where S ONAR benefits from execution-based validation. In contrast, the LLM-only scorers are confined to source code and summary text alone, and hence struggle most with correctness. Although abstraction and conciseness are more perceptible from the summary text, S ONAR still outperforms the best baseline by 10 and 8 pp, respectively. Notably, S ONAR is implemented with Gemini-2.5 Flash, which ranks well below the top-tier baseline models. This suggests that S ONAR ’s advantage comes from its design rather than a stronger underlying model; we further test this in § IV-F. Findings of RQ1: S ONAR reliably captures the intended quality dimensions of code summaries, achieving a 90% overall agreement rate with ground-truth annotations. D. RQ2: Correlation Analysis between S ONAR’s Dimensions and Downstream SE Task Performance 1) Task Selection: To evaluate the impact of S ONAR’s dimensions on LLM performance, we focus on prior work that has used code summaries, comments, or documentation as input for some SE task, and select four representative tasks that collectively capture distinct downstream uses of code summaries: code retrieval [13], [57], [58], code translation [15], [16], code optimization [59], [60], and test oracle generation [14], [26]. Since the selected tasks require test cases or cross-language equivalents, we restrict our experiments to HumanEval and MBPP, which provide these artifacts. We randomly sample 50 source codes from each dataset and collect all summaries generated for them from the pool described in § IV-A. For each task, we retain only summaries whose source

code includes the artifacts required for that task, leading to varying sample sizes across tasks. Table II reports the sample size, evaluation metrics, and models used per task. Zero-shot prompts are used for all tasks to avoid prompt-specific effects. Code Retrieval. Given a summary as the query, the task is to retrieve the most relevant code by matching the summary’s embedding against the embeddings of codes in a candidate pool. Since cross-language retrieval is particularly challenging [13], we focus on the Python-to-Java setting, where a Python function summary is used to retrieve its Java equivalent. We use the same retrieval pool as [13], augmented with the sampled HumanEval and MBPP codes and their equivalent Java implementations. We report MRR [34] as the metric, which in our per-summary setting reduces to RR. Code Translation. Given a Python function summary as specification, an LLM is prompted to implement the same functionality in Java. Alongside translation accuracy, we measure the average edit distance (the diversity metric of [53]) among 5 independently sampled Java implementations from the same LLM, since diverse translations are known to benefit data augmentation [53] and runtime efficiency [61]. Code Optimization. The LLM is tasked with generating the most runtime-efficient implementation of the functionality described by the summary. Execution speedup is measured using the large-input stress tests and experimental setup of [55]. Test Oracle Generation Building on [14], we provide the LLM with the function signature of the method under test (MUT), along with the corresponding summary. We then ask the LLM to generate 10 assert statements to test the MUT. 2) Baselines: Baselines include five popular referencebased metrics: text-based, BLEU-4, METEOR, ROUGE-L, and embedding-based, BLEURT, and BERTScore (F1). Although SIDE [6] is a relevant reference-free baseline, its Javaspecific training makes comparison with S ONAR’s Python implementation infeasible here. For the baselines, we use HumanEval and MBPP’s natural-language descriptions as references, since they are human-written [62] and thus provide reasonable gold standards for comparison. 3) Evaluation Metric: We use Spearman’s rank correlation coefficient [63], r ρ , to measure the correlation between S ONAR’s dimensions and downstream task performance. Spearman is appropriate because the dimension scores and task metrics are not normally distributed (validated with the Shapiro-Wilk normality test [64]), span different value ranges, and are not assumed to be linearly related. Instead, we are interested in whether changes in scores are monotonically associated with increases or decreases in downstream performance, which Spearman captures through rank-based correlation. To account for model-specific variation, we evaluate three LLMs per task, compute rρ separately for each LLM and metric, and aggregate correlations using Fisher-z averaging [65]. Table III reports the mean(±standard deviation) of rρ , with the corresponding significance level. 4) Results and Analysis: As shown in Table III, at least one S ONAR dimension significantly and positively correlates with every task metric, with up to a 14 times improvement

TABLE III: rρ between each method and downstream task performance. ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001; unmarked is non-significant (p ≥ 0.05). Bold values indicate the highest significant correlation per task metric. Task

SONAR

Translation

Optimization

Test Oracle Gen.

MRR

CA@1

Edit Distance

Pass@1

Speedup

Success Rate

Correctness

+0.06(±0.04)

+0.41(±0.07)∗∗∗

−0.14(±0.02)∗∗∗

+0.59(±0.05)∗∗∗

−0.05(±0.04)

+0.41(±0.06)∗∗∗

Abstraction

+0.18(±0.04)∗∗∗

−0.11(±0.05)∗

+0.28(±0.05)∗∗∗

−0.02(±0.01)

+0.11(±0.01)∗∗

−0.21(±0.04)∗∗∗

Conciseness

+0.06(±0.08)

+0.04(±0.03)

+0.05(±0.02)

−0.01(±0.02)

−0.01(±0.03)

+0.03(±0.02)

Fluency

+0.03(±0.06)

−0.02(±0.04)

−0.07(±0.04)∗

+0.06(±0.03)

−0.06(±0.03)

+0.05(±0.03)

BLEU-4

+0.02(±0.08)

−0.09(±0.02)∗

+0.07(±0.02)

+0.07(±0.02)

+0.06(±0.04)

+0.09(±0.01)∗∗

METEOR

+0.02(±0.06)

−0.01(±0.03)

+0.03(±0.01)

+0.14(±0.01)∗∗∗

−0.05(±0.06)

+0.12(±0.03)∗∗∗

∗

Method

Baselines

Retrieval

ROUGE-L

+0.01(±0.08)

−0.11(±0.04)

+0.01(±0.03)

−0.06(±0.01)

+0.09(±0.06)

+0.09(±0.03)∗∗

BLEURT

+0.14(±0.07)∗

+0.02(±0.07)∗

−0.02(±0.05)

+0.15(±0.02)∗∗∗

−0.03(±0.05)

+0.20(±0.03)∗∗∗

BERTScore

+0.08(±0.08)

−0.04(±0.03)

+0.00(±0.04)

+0.08(±0.03)

−0.00(±0.04)

+0.12(±0.02)∗∗∗

in predictive strength over the strongest baseline. For code retrieval, only Abstraction shows significant correlation (rρ = 0.18), suggesting that summaries with fewer implementation details generalize better across languages and more effectively support cross-language retrieval. Since retrieval relies solely on embedding similarity and does not verify functional behavior, Correctness remains insignificant. For translation and optimization where semantic preservation is essential, Correctness is the dominant signal: CA@1 and Pass@1, show moderately strong correlations of rρ = 0.41 and rρ = 0.59, respectively. However, since both metrics are binary, tied ranks may cause Spearman correlation to underestimate the impact of Correctness. We therefore also report Odds Ratios (ORs) [66]. A one standard deviation increase in Correctness makes a semantics-preserving translation and a semantics-preserving optimization 1.90 and 2.62 times more likely, respectively. These represent 71% and 68% improvements over the strongest baselines, respectively. This gain can be attributed to S ONAR’s regeneration-based design for Correctness: if a summary can regenerate code that is functionally equivalent to the original, it is also more likely to support behavior-preserving implementations in another language or as an optimized variant. For translation diversity, measured by edit distance, Abstraction shows a significant positive correlation (rρ = 0.28), indicating that abstract summaries enhance diversity not only within the same language, but also across languages. Optimization speedup is not strongly predicted by either S ONAR or the baselines, though Abstraction shows a slight positive trend (rρ = 0.11), suggesting fewer implementation details may give the LLM more freedom to find faster alternatives. Finally, Correctness is the most significant predictor of success rate in test oracle generation (rρ = 0.41), outperforming the best baseline by 105%. More importantly, any positive correlation for a reference-based metric is tied to the particular reference used; a different reference could yield a different result. S ONAR, by contrast, evaluates the summary under test directly, without any reference, meaning its correlations are genuine signals that the summary itself drives downstream performance. We note that moderate correlations shown in Table III are expected, as LLM-generated summaries cluster

TABLE IV: Partial Spearman correlation between each SONAR dimension and downstream task performance. Task Dimension

Retrieval MRR

Translation CA@1

Optimization

Edit Dist. ∗∗∗

∗∗

Pass@1 ∗∗∗

Test Oracle

Speedup

Success Rate

Correctness

+0.09

+0.37

−0.10

+0.55

−0.05

+0.33∗∗∗

Abstraction

+0.20∗∗∗

−0.07

+0.25∗∗∗

+0.08

+0.10∗

−0.15∗∗∗

Conciseness

+0.03

+0.05

−0.01

−0.03

−0.05

+0.06∗

Fluency

+0.09∗

−0.05

+0.01

+0.06

−0.02

+0.00

in a narrow range for most quality dimensions (see Fig. 5). Interestingly, like the baselines, Fluency and Conciseness show almost no significant impact on downstream tasks, despite being widely valued as important qualities for human developers [9], thereby revealing a contrast between what human and LLM consumers value in a code summary. This finding is consistent with the capabilities of modern LLMs: their large context windows make conciseness less relevant, while training on large text corpora allows them to better tolerate less fluent summaries. S ONAR’s dimensions are not independent. For example, an overly concise summary may omit behavioral details, hurting Correctness, while summaries rich in implementation details may help Correctness, but hurt Abstraction. To account for these inter-dependencies, we report partial Spearman correlations for each dimension (Table IV), controlling for the others. With one exception, each significant correlation from Table III holds in Table IV, confirming that S ONAR dimensions capture task-specific signal beyond their overlap with the others. Although Table IV reports LLM performance on four tasks and six metrics, the results reveal a broader pattern: no single quality dimension is appropriate for all tasks. For behaviorpreserving tasks, correctness matters. However, when the goal extends beyond behavior preservation to exploring alternative implementations, abstraction, alongside correctness, helps. But abstraction may hurt tasks that require source-specific details, such as test oracle generation (Table IV). Again, conciseness and fluency could still be relevant for human consumers in tasks like program comprehension. This is where S ONAR’s multi-dimensional design becomes valuable: rather than collapsing to a single notion of quality, it enables prioritizing dimensions most relevant to the specific task and consumer,

Fig. 5: Per-dimension S ONAR scores for 11 LLMs.

Fig. 6: Effect of prompting strategy on S ONAR dimensions.

thereby facilitating task-aware evaluation of code summaries.

language-like, lowering Fluency. Conciseness varies from 0.42 (Gemini) to 0.90 (CodeT5), indicating substantial differences in average summary length across models. Together, these dimension-specific strengths and weaknesses highlight that summarizer selection should account for the target task and consumer, with preference given to models that perform well on relevant quality dimensions. Motivated by the task-specific role of summary dimensions observed in RQ2 (§ IV-D), we analyze how prompting can steer models toward task-aware summaries. Fig. 6 presents the average score per dimension for all summaries generated under each of the three prompting strategies described in § IV-A3. Dimension-aware and few-shot prompting improve nearly all dimensions over vanilla prompting, except Correctness, which remains stable. Abstraction improves the most, with gains up to ≈ 10 pp, while Conciseness improves by up to ≈ 6 pp. This demonstrates a lightweight mechanism for task-aware summarization. As an example, for diverse code translation, rather than asking an LLM only to summarize the code, explicitly specifying the relevant dimensions, correctness, and abstraction, or providing a few examples of correct and abstract summaries can steer the summarizer LLM toward summaries that better support the target downstream task.

Findings of RQ2: At least one S ONAR dimension, either Correctness or Abstraction, significantly correlates with LLM performance for every task, increasing the predictive signal between 1.3 and 14 times relative to the best baseline. What is important in a summary varies with task, and S ONAR uniquely captures this variation through its multidimensional design, which existing single-score metrics cannot. E. RQ3: Evaluation of LLM-based Source Code Summarization using S ONAR 1) Experimental Setup: Using the setup described in § IV-A, Fig. 5 illustrates each LLM’s average score per S ONAR dimension, computed across all summaries generated by that LLM under Vanilla prompting. Mean scores are reported to highlight overall quality trends; finer-grained analysis of score distribution within each model is left for future work. 2) Results and Analysis: Fig. 5 shows that no single LLM dominates all four dimensions. Most modern LLMs achieve relatively low Abstraction scores, typically between 0.25 and 0.50. Smaller models such as CodeT5, achieve higher Abstraction, but at the cost of Correctness. In contrast, most LLMs score high on Correctness (0.75–1.00), with the much smaller 3.8B model, Phi-3.5 Mini, achieving a score of 0.82, comparable to frontier models like Gemini (0.88) and Kimi (0.87). Surprisingly, these SOTA models fall behind open-source midsized LLMs in Fluency, with Gemini and Kimi scoring 0.14 and 0.10 below the average. To investigate this trend further, we qualitatively analyze the summaries generated by Gemini and find that Gemini often enumerates the implementation step by step in the summary while reusing tokens and keywords from the source code, (Fig. 1). This helps preserve behavior of the code, improving Correctness, but it reduces Abstraction. Moreover, the abundance of code-related tokens and step-bystep procedural descriptions makes the summaries less natural-

Findings of RQ3: LLMs show inherent trade-offs across quality dimensions. Varying prompting strategy provides a practical way to improve task-relevant dimensions, with Abstraction showing up to 10 pp improvement, without any additional training.

F. RQ4: Sensitivity Analysis of S ONAR Since § III-A defines the four S ONAR dimensions through LLM-based interpretation of the summary under test, the scores may be biased by the particular LLM used inside the scorer modules. We test this sensitivity by measuring how much Correctness and Conciseness scores change under different LLM choices. We exclude Abstraction because it

TABLE V: Sensitivity of Correctness and Conciseness to the choice of LLM. “Alt. Model” denotes the alternative LLM substituted for the original implementation model. Dimension

Alt. Model

Generator (G) MAD

rρ

Correctness

Qwen OLMo

0.042 0.055

0.891 0.878

Conciseness

Qwen OLMo

0.037 0.036

0.939 0.943

Compressor (P ) rρ

MAD N/A 0.052 0.047

0.924 0.931

already aggregates multiple samples from a multi-LLM pool, reducing sensitivity to any single model. We also exclude Fluency since S ONAR follows the model choice used in prior work. Thus, we restrict this analysis to the code generator, G used by the Correctness and Conciseness Scorers and the compressor, P used by the Conciseness Scorer. 1) Experimental Setup: In our implementation, Gemini2.5-Flash serves as both G and P in the Correctness and Conciseness Scorers. To analyze sensitivity, we replace Gemini with Qwen2.5-Coder-32B and OLMo-32B-Instruct, two strong open-source LLMs, and recompute Correctness and Conciseness for 1,000 randomly sampled summaries. We quantify the effect in Table V using the Mean Absolute Deviation (MAD) and Spearman correlation relative, rρ , to the original Geminibased scores, where MAD is defined as: MAD =

n 1X

n i=1

|zialt − ziGemini |

Here, ziGemini and zialt denote the scores assigned to summary i by Gemini and the alternative model, respectively. 2) Results and Analysis: Table V shows that replacing G with a different LLM leads to only small deviations from the original Gemini-based Correctness and Conciseness scores, with MAD between 0.04 and 0.06. The alternative models also maintain very strong positive correlations with the original scores (rρ = 0.88–0.94) for both Correctness and Conciseness. This suggests that S ONAR is robust to the choice of the code-generator LLM, G: the low MAD indicates that scores remain numerically close to the original, while the high correlation shows that the relative ranking of summaries is largely preserved even when Qwen or OLMo is used instead of Gemini. Similarly, replacing the compressor P has limited impact on Conciseness, with an absolute deviation of ≈ 0.05 for both Qwen and OLMo and strong correlations of 0.92 and 0.93, respectively. Overall, these results suggest that S ONAR is not tied to a single LLM choice; instead, its regenerationbased design remains stable under model substitution, leaving both absolute scores and summary rankings largely unaffected. Findings of RQ4: S ONAR remains mostly insensitive to the choice of internal LLM, maintaining very strong cross-model consistency (rρ = 0.88-0.94) with small score deviations (0.04-0.06).

V. T HREATS TO VALIDITY Our findings may be affected by both internal and external threats. A. Threats to Internal Validity The tools and scripts used to implement S ONAR and automate our experiments introduce the possibility of implementation bugs. We mitigate this risk by building on widely-adopted tools such as Joern and by rerunning experiments to ensure consistency of results. The validation of S ONAR’s dimensions relies on human-annotated ground truth, which is inherently subject to inter-annotator variation. To address this, we retain only those labels where both annotators unanimously agree, providing a reasonably sound basis for evaluation. Moreover, to mitigate the risk of author bias influencing the annotation, each annotator independently assigns preferences without access to the other’s preferences or S ONAR’s scores for the pair under evaluation. Although S ONAR uses regenerated code to empirically ground each quality dimension, this signal is still an approximation of summary quality. Its accuracy may be affected by the inherent non-determinism of the LLMs used, the influence of their pretrained knowledge, and noise from other framework components, such as the differential fuzzing tool. B. Threats to External Validity The findings on how S ONAR dimensions affect LLM performance may be influenced by the choice of LLM used for the task. We mitigate this threat by employing multiple LLMs and reporting the mean correlation across all models. The experiments in § IV-D rely on MBPP and HumanEval, two established benchmarks for code-related tasks. However, we acknowledge that these benchmarks may not fully reflect the complexity of real-world software, and evaluating S ONAR on more complex datasets would further strengthen the generalizability of our findings. Finally, as S ONAR currently supports Python only, the applicability of our findings to other programming languages may be limited. The core idea, however, is language-agnostic in principle and can be adapted to support additional languages. VI. C ONCLUSION In this work, we identify abstraction as a new dimension of source code summary quality and present S ONAR, a referencefree, multi-dimensional framework that enables task-aware evaluation of source code summaries. S ONAR leverages code regenerated from a summary as an empirical signal to assess quality across multiple dimensions, lifting automatic evaluation beyond references, training, and subjective LLM judgment for the first time. Using S ONAR, we evaluate code summaries from an LLM’s perspective, investigating what aspects of a summary make it useful to an automated consumer. We find that S ONAR reliably captures the intended aspects of summary quality and effectively predicts LLM performance across a range of downstream SE tasks. We also find that what makes a summary useful varies with

the downstream task and consumer, motivating the need for task-aware summarization. Using S ONAR, we evaluate 11 popular LLMs, identify model-specific tradeoffs, and suggest ways to steer code summarization toward specific tasks. As software engineering enters the agentic era, we hope this work broadens the notion of code summarization and its evaluation beyond human needs, opening new opportunities to study summaries from an automated consumer’s perspective.

R EFERENCES [1] T. Sharma, M. Kechagia, S. Georgiou, R. Tiwari, I. Vats, H. Moazen, and F. Sarro, “A survey on machine learning techniques for source code analysis,” 2022. [Online]. Available: https://arxiv.org/abs/2110.09610 [2] W. Sun, Y. Miao, Y. Li, H. Zhang, C. Fang, Y. Liu, G. Deng, Y. Liu, and Z. Chen, “Source code summarization in the era of large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2407.07959 [3] Y. Virk, P. Devanbu, and T. Ahmed, “Calibration of large language models on code summarization,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729400 [4] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 311–318. [Online]. Available: https://doi.org/10.3115/1073083.1073135 [5] X. Jin and Z. Lin, “Simllm: Calculating semantic similarity in code summaries using a large language model-based approach,” Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3660769 [6] A. Mastropaolo, M. Ciniselli, M. D. Penta, and G. Bavota, “Evaluating code summarization techniques: A new metric and an empirical characterization,” 2023. [Online]. Available: https: //arxiv.org/abs/2312.15475 [7] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, J. Goldstein, A. Lavie, C.-Y. Lin, and C. Voss, Eds. Ann Arbor, Michigan: Association for Computational Linguistics, Jun. 2005, pp. 65–72. [Online]. Available: https://aclanthology.org/W05-0909/ [8] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” 01 2004, p. 10. [9] G. Crupi, R. Tufano, A. Velasco, A. Mastropaolo, D. Poshyvanyk, and G. Bavota, “On the effectiveness of llm-as-a-judge for code generation and summarization,” 2025. [Online]. Available: https: //arxiv.org/abs/2507.16587 [10] Y. Wu, Y. Wan, Z. Chu, W. Zhao, Y. Liu, H. Zhang, X. Shi, H. Jin, and P. S. Yu, “Can large language models serve as evaluators for code summarization?” IEEE Transactions on Software Engineering, vol. 51, no. 12, pp. 3205–3217, 2025. [11] I. Ahmed, A. Aleti, H. Cai, A. Chatzigeorgiou, P. He, X. Hu, M. Pezzè, D. Poshyvanyk, and X. Xia, “Artificial intelligence for software engineering: The journey so far and the road ahead,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–27, 2025. [12] A. Vitale, A. Mastropaolo, R. Oliveto, M. D. Penta, and S. Scalabrino, “Optimizing datasets for code summarization: Is code-comment coherence enough?” 2025. [Online]. Available: https://arxiv.org/abs/ 2502.07611 [13] Y. Yang, L. Kuang, J. Liu, Z. Liu, Y. Xia, and D. Lo, “Unicor: Modality collaboration for robust cross-language hybrid code retrieval,” 2026. [Online]. Available: https://arxiv.org/abs/2512.10452 [14] S. B. Hossain, R. Taylor, and M. Dwyer, “Doc2oracll: Investigating the impact of documentation on llm-based test oracle generation,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729354 [15] C.-e. A. Tai, P. Nie, L. Golab, and A. Wong, “Nl in the middle: Code translation with llms and intermediate representations,” in 2025 IEEE International Conference on Collaborative Advances in Software and COmputiNg (CASCON), 2025, pp. 295–300.

[16] W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Summarize and generate to back-translate: Unsupervised translation of programming languages,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein, Eds. Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 1528–1542. [Online]. Available: https://aclanthology.org/2023.eacl-main.112/ [17] M. Allamanis, S. Panthaplackel, and P. Yin, “Unsupervised evaluation of code llms with round-trip correctness,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024. [18] J. Robinson and J. K. Kummerfeld, “Simple and effective baselines for code summarisation evaluation,” 2025. [Online]. Available: https://arxiv.org/abs/2505.19392 [19] M. J. Min, Y. Ding, L. Buratti, S. Pujar, G. Kaiser, S. Jana, and B. Ray, “Beyond accuracy: Evaluating self-consistency of code large language models with identitychain,” 2024. [Online]. Available: https://arxiv.org/abs/2310.14053 [20] D. Roy, S. Fakhoury, and V. Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 1105–1116. [Online]. Available: https://doi.org/10.1145/3468264.3468588 [21] T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr [22] T. Sellam, D. Das, and A. Parikh, “Bleurt: Learning robust metrics for text generation,” in Proceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 7881–7892. [23] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019. [Online]. Available: https: //arxiv.org/abs/1908.10084 [24] F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A large-scale empirical study on code-comment inconsistencies,” in Proceedings of the 27th International Conference on Program Comprehension, ser. ICPC ’19. IEEE Press, 2019, p. 53–64. [Online]. Available: https://doi.org/10.1109/ICPC.2019.00019 [25] G. Denaro and L. Guglielmo, “Automated test generation from program documentation encoded in code comments,” 2025. [Online]. Available: https://arxiv.org/abs/2504.21161 [26] A. Goffi, A. Gorla, M. D. Ernst, and M. Pezzè, “Automatic generation of oracles for exceptional behaviors,” in Proceedings of the 25th International Symposium on Software Testing and Analysis, ser. ISSTA 2016. New York, NY, USA: Association for Computing Machinery, 2016, p. 213–224. [Online]. Available: https://doi.org/10.1145/2931037.2931061 [27] Z. Ni, L. Bo, B. Li, T. Chen, X. Sun, and X. Wu, “An approach of method-level bug localization,” IET Software, vol. 16, no. 4, p. 422–437, May 2022. [Online]. Available: https://doi.org/10.1049/sfw2.12060 [28] A. R. Oskooei, S. S. Yukcu, M. C. Bozoglan, and M. S. Aktas, “Natural language summarization enables multi-repository bug localization by llms in microservice architectures,” 2025. [Online]. Available: https://arxiv.org/abs/2512.05908 [29] H. Suh, M. Tafreshipour, S. Malek, and I. Ahmed, “Human or llm? a comparative study on accessible code generation capability,” 2025. [Online]. Available: https://arxiv.org/abs/2503.15885 [30] A. Imani, M. Moshirpour, and I. Ahmed, “Inside out: Uncovering how comment internalization steers llms for better or worse,” 2025. [Online]. Available: https://arxiv.org/abs/2512.16790 [31] S. B. Dristi and M. B. Dwyer, “Analyzing and mitigating surface bias in code evaluation metrics,” 2025. [Online]. Available: https: //arxiv.org/abs/2509.15397 [32] ——, “A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings,” 2026. [Online]. Available: https://arxiv.org/abs/2602.15761 [33] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/2009.10297

[34] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” 2020. [Online]. Available: https://arxiv.org/abs/1909.09436 [35] A. LeClair, S. Jiang, and C. McMillan, “A neural model for generating natural language summaries of program subroutines,” in Proceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19. IEEE Press, 2019, p. 795–806. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00087 [36] The Joern Team, “Joern - the bug hunter’s workbench,” https://joern.io/, 2026, accessed: 2026-06-29. [37] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk, “Deep learning code fragments for code clone detection,” in 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), 2016, pp. 87–98. [38] H.-H. Wei and M. Li, “Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code,” in Proceedings of the 26th International Joint Conference on Artificial Intelligence, ser. IJCAI’17. AAAI Press, 2017, p. 3034–3040. [39] M. Zakeri-Nasrabadi, S. Parsa, M. Ramezani, C. Roy, and M. Ekhtiarzadeh, “A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges,” J. Syst. Softw., vol. 204, no. C, Oct. 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111796 [40] I. Keivanloo, C. K. Roy, and J. Rilling, “Sebyte: Scalable clone and similarity search for bytecode,” Science of Computer Programming, vol. 95, pp. 426–444, 2014, special Issue on Software Clones (IWSC’12). [Online]. Available: https://www.sciencedirect.com/science/ article/pii/S0167642313002773 [41] A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi, “DExperts: Decoding-time controlled text generation with experts and anti-experts,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 6691–6706. [Online]. Available: https://aclanthology.org/2021.acl-long.522/ [42] Y. Su, T. Lan, Y. Wang, D. Yogatama, L. Kong, and N. Collier, “A contrastive framework for neural text generation,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022. [43] X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred api knowledge,” in Proceedings of the 27th International Joint Conference on Artificial Intelligence, ser. IJCAI’18. AAAI Press, 2018, p. 2269–2275. [44] A. V. M. Barone and R. Sennrich, “A parallel corpus of python functions and documentation strings for automated code documentation and code generation,” 2017. [Online]. Available: https://arxiv.org/abs/1707.02275 [45] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374 [46] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 [47] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, B. Hui, N. Muennighoff, D. Lo, D. Fried, X. Du, H. de Vries, and L. V. Werra, “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” 2025. [Online]. Available: https://arxiv.org/abs/2406.15877

[48] D. Nguyen, L. Nam, A. Dau, A. Nguyen, K. Nghiem, J. Guo, and N. Bui, “The vault: A comprehensive multilingual dataset for advancing code understanding and generation,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 4763–4788. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.316/ [49] Ai2, “Olmo 3: Charting a path through the model flow to lead opensource ai,” https://allenai.org/blog/olmo3, Nov. 2025, accessed: 2026-0629. [50] LM Market Cap, “Bigcodebench (hard) leaderboard,” https://lmmarketcap.com/benchmarks/bigcodebench, 2026, accessed: 2026-06-29. [51] G. Sundaram, B. Venktesh V, and S. K B, “Docstringeval: Evaluating the effectiveness of language models for code explanation through docstring generation,” in 2025 International Conference on Emerging Technologies in Computing and Communication (ETCC), 2025, pp. 1–7. [52] LLM Stats Research, “Best ai for reasoning in 2026,” https://llm-stats. com/leaderboards/best-ai-for-reasoning, 2026, accessed: 2026-06-27. [53] Y. Xie, A. Naik, D. Fried, and C. Rose, “Data augmentation for code translation with comparable corpora and multiple references,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 13 725–13 739. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.917/ [54] P. Liguori, C. Improta, R. Natella, B. Cukic, and D. Cotroneo, “Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators,” Expert Systems with Applications, vol. 225, p. 120073, 2023. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0957417423005754 [55] Y. Peng, J. Wan, Y. Li, and X. Ren, “Coffe: A code efficiency benchmark for code generation,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715727 [56] S. B. Hossain and M. B. Dwyer, “Togll: Correct and strong test oracle generation with llms,” in Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, p. 1475–1487. [Online]. Available: https: //doi.org/10.1109/ICSE55347.2025.00098 [57] X. Li, K. Dong, Y. Q. Lee, W. Xia, H. Zhang, X. Dai, Y. Wang, and R. Tang, “CoIR: A comprehensive benchmark for code information retrieval models,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 22 074–22 091. [Online]. Available: https: //aclanthology.org/2025.acl-long.1072/ [58] K. W. Nafi, B. Roy, C. K. Roy, and K. A. Schneider, “[research paper] crolsim: Cross language software similarity detector using api documentation,” in 2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM), 2018, pp. 139–148. [59] Q. Liang, Z. Sun, Q. Zhu, J. Hu, Y. Zhao, Y. Chen, M. Zhu, G. Wang, and L. Zhang, “Directional diffusion-style code editing pre-training,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12079 [60] N. Tang, D. Meininger, G. Xu, Y. Shi, Y. Huang, C. McMillan, and T. J.J. Li, “Naturaledit: Code modification through direct interaction with adaptive natural language representation,” 2026. [Online]. Available: https://arxiv.org/abs/2510.04494 [61] L. Zhang, J. Wang, C. Hu, B. Liang, J. Li, and M. Zhang, “Bridging functional correctness and runtime efficiency gaps in llm-based code translation,” 2026. [Online]. Available: https://arxiv.org/abs/2606.17683 [62] M. L. Siddiq, S. Dristi, J. Saha, and J. C. S. Santos, “The fault in our stars: Quality assessment of code generation benchmarks,” in 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM), 2024, pp. 201–212. [63] C. Spearman, “The proof and measurement of association between two things,” The American Journal of Psychology, vol. 15, no. 1, pp. 72–101, 1904. [64] S. S. SHAPIRO and M. B. WILK, “An analysis of variance test for normality (complete samples)†,” Biometrika, vol. 52, no. 3-4, pp. 591–611, 12 1965. [Online]. Available: https://doi.org/10.1093/biomet/ 52.3-4.591 [65] D. M. Corey, W. P. Dunlap, and M. J. Burke, “Averaging correlations: Expected values and bias in combined pearson rs and fisher’s z transformations,” The Journal of General Psychology,

vol. 125, no. 3, pp. 245–261, 1998. [Online]. Available: https: //doi.org/10.1080/00221309809595548 [66] S. M. Hailpern and P. F. Visintainer, “Odds ratios and logistic regression: further examples of their use and interpretation,” Stata Journal, vol. 3, no. 3, pp. 213–225, September 2003. [Online]. Available: https://ideas.repec.org/a/tsj/stataj/v3y2003i3p213-225.html

Related documents

• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation#429448
• Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation#1037194
• Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation#1028784
• LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning#31335
• LangSelect: Cost-Aware Target-Language Routing for LLM Code Generation#1013115