FASE: Fast Adaptive Semantic Entropy for Code Quality SHIZHE LIN, University of Waterloo, Canada LADAN TAHVILDARI, University of Waterloo, Canada Multi-agent code generation offers a promising paradigm for autonomous software development by simulating the human software engineering lifecycle. However, system reliability remains hindered by LLM hallucinations and error propagation across interacting agents. While semantic entropy provides a principled way to quantify uncertainty without ground-truth answers, current methods often rely on costly LLM-driven equivalence checks. In this work, we introduce Fast Adaptive Semantic Entropy (FASE), a novel metric that approximates functional correctness based on the minimum spanning tree of structural and semantic dissimilarity graphs. Evaluations
arXiv:2606.09800v1 [cs.SE] 8 Jun 2026
on HumanEval and BigCodeBench demonstrate that FASE outperforms state-of-the-art semantic entropy by LLM entailment, achieving a 25% average improvement in Spearman correlation and a 19% increase in ROCAUC score against Pass@1 from ground-truth test cases when using the Qwen3-Embedding-8B model. Furthermore, by eliminating costly LLM-driven equivalence evaluation, FASE incurs negligible computational overhead, requiring only approximately 0.3% of the runtime cost of traditional semantic entropy approaches. These results position FASE as a practical, cost-effective solution for optimizing uncertainty quantification in real-world multi-agent workflows. CCS Concepts: • Mathematics of computing → Information theory; • Software and its engineering → Automatic programming; • Computing methodologies → Artificial intelligence. Additional Key Words and Phrases: Generative AI, Agentic Code Generation, Adaptive Workflow, Trust in Agent Output
1
Introduction
Recent advances in large language models (LLMs) have accelerated the emergence of autonomous multi-agent systems for software engineering tasks [13, 17, 30]. Rather than relying on a single model to generate code end-to-end, modern frameworks increasingly decompose software development into collaborative workflows involving specialized agents responsible for requirement analysis, planning, coding, testing, debugging, and review [8, 40]. Inspired by human software engineering practices, systems such as MetaGPT [18], CodeCoR [37], and AdaCoder [54] demonstrate that role specialization and structured collaboration can substantially improve code generation quality and task-solving capability for complex programming problems. These agentic workflows further enable LLMs to address repository-level development tasks, long-horizon reasoning, and iterative refinement processes that are difficult for standalone models to handle effectively. Despite these advances, LLM-based software engineering systems remain fundamentally limited by hallucinations and uncertainty in generated outputs [1, 28, 29, 47]. Errors produced during early stages of reasoning or implementation can propagate across agents, leading to cascading failures throughout the development pipeline [36] as shown in Fig. 1. Existing approaches attempt to mitigate these issues through self-reflection [23], retrieval-augmented generation [7, 9], iterative debugging [11, 21], static analysis [32, 46], mutation testing [41, 42], and fuzz testing [4, 6]. Recent frameworks also introduce consensus-based reasoning [45], and repository-aware grounding strategies [26] to improve factual consistency and functional correctness. However, many of these techniques rely on expensive iterative interactions with LLMs, resulting in substantial computational overhead and limited scalability for real-world deployment. Entropy-based uncertainty estimation methods have recently been explored to evaluate the reliability of LLM-generated code without requiring ground-truth Authors’ Contact Information: Shizhe Lin, University of Waterloo, Waterloo, Canada, [email protected]; Ladan Tahvildari, University of Waterloo, Waterloo, Canada, [email protected].
1
2
Lin et al.
Fig. 1. The risk of hallucination from unreliable agents (red) undermines the credibility of coherent agents (blue) and propagates across different workflow architectures.
labels [25]. Semantic entropy [12] measures uncertainty by grouping outputs according to functional equivalence rather than textual similarity, but it requires bidirectional entailment checks using LLMs, which limits scalability in practical multi-agent systems. To improve efficiency, structural entropy [39] has been proposed as an alternative designed specifically for source code. Nevertheless, structural similarity alone cannot fully capture program semantics, as functionally equivalent solutions may have different structures while structurally similar code may still exhibit different behaviours. Building upon the prior works, this paper makes the following contributions: • Lightweight semantic entropy via embedding models: Fast Adaptive Semantic Entropy (FASE) that leverages the semantic signals captured by code embedding models and dynamically adapts clustering according to the unique structure of each programming task’s solution space. FASE enables the estimation of the functional correctness of LLM-generated code without requiring ground-truth test cases while incurring only negligible computational overhead. • Critical evaluation on estimation of code functional correctness: Evaluation on the reliability and accuracy of the proposed FASE entropy against existing LLM-based semantic entropy methods, self-evaluation techniques, and other baseline approaches in terms of their correlation and predictive capability for code functional correctness. • Cost–accuracy trade-off analysis for multi-agent observation: Quantitative comparison of computational runtime between FASE and other baselines, demonstrating that the proposed approaches offer substantially improved efficiency while maintaining useful correctness signals for scalable multi-agent systems. The rest of the paper is organized as follows. Section 2 formalizes the problem and identifies the key research gaps. Section 3 presents the methodology of the proposed FASE approach. Section 4 outlines the research questions addressed in this work, details the experimental setup, reports the results, and provides a comprehensive analysis of the evaluation. The related work and state-of-the-art are listed in Section 5. Section 6 discusses threats to validity. Finally, Section 7 concludes by summarizing the main findings and discussing future research directions. 2
Problem Definition
The increasing adoption of LLM systems for software development creates a growing need for efficient and reliable methods to estimate the quality of generated code without relying on ground-truth test cases. Entropy of LLM Output: One common measure of uncertainty is the predictive entropy of the output distribution, which quantifies the amount of information the model has about the output given the input [27]: 𝑃𝐸 (𝑋 ) = 𝐻 (𝑌 |𝑥) = Í − 𝑃 (𝑦|𝑥) ln 𝑃 (𝑦|𝑥) where the predictive entropy for an input 𝑥 is defined as the conditional entropy of the output
FASE: Fast Adaptive Semantic Entropy for Code Quality
3
random variable 𝑌 given 𝑥. Low predictive entropy reflects a sharply peaked output distribution, implying high model confidence, while high predictive entropy indicates a more uniform distribution in which many possible outputs have comparable likelihoods. The derivation of entropy for a set of code samples 𝑋 given a programming task is presented in Eq. 1, where 𝑐 denotes an equivalence class within the set of all possible classes C. In practice, however, only a finite set of code samples is available, and the computation should be based on individual output 𝑠 observed within the sampled set. " 𝑃𝐸 (𝑋 ) = 𝐻 (C|𝑥) = −
∑︁
𝑃 (𝑐 |𝑥) log 𝑃 (𝑐 |𝑥) = −
𝑐
#
∑︁
∑︁
𝑐
𝑠 ∈𝑐
𝑃 (𝑠 |𝑥) log
"
#! ∑︁
𝑃 (𝑠 |𝑥)
(1)
𝑠 ∈𝑐
Textual, Structural and Semantic Equivalence: The equivalence classes 𝑐 of outputs are determined by clustering the results using an equivalence function 𝐸 (·, ·). For a given equivalence class, all output sentences within the class are pairwise equivalent, ∀𝑠, 𝑠 ′ ∈ 𝑐 : 𝐸 (𝑠, 𝑠 ′ ), meaning they convey the same content. A straightforward approach to determining whether two outputs are equivalent is to compare their textual similarity but it fails to reliably distinguish between coherent and hallucinated responses [12, 39]. Structural entropy [39] addresses this limitation by introducing a higher-level perspective that abstracts away trivial textual differences and groups code based on similarities in their abstract syntax trees. Nevertheless, solutions to programming tasks are not constrained to a single structural form, and functionally equivalent programs may exhibit diverse syntactic representations. To overcome this, recent work has proposed semantic entropy [12] through LLM-based bidirectional entailment, which quantifies uncertainty at the level of meaning rather than tokens, providing a more faithful estimate of a model’s confidence. However, applying this approach directly to programming tasks remains imperfect [35], as LLMs often fail to detect subtle code variations that can result in substantial differences in functionality. There are three limitations of the current state-of-the-art entropy calculation: • The accuracy of bidirectional entailment relies on LLM’s self-evaluation capability; different models often produce inconsistent judgments, making it difficult to identify a reliable and universally optimal evaluator, particularly in the coding domain. • The cost of producing equivalence-class labels grows exponentially with the number of generated outputs for LLM-based semantic entropy, rendering the method impractical for multi-step, multi-agent workflows where time efficiency is critical. • Structural entropy alone fails to account for multiple valid structural representations of a given task, often assigning functionally equivalent code samples to different classes. These limitations motivate the need for more accurate, efficient, and model-agnostic approaches to estimate the consistency of functionality in code generation. 3
Methodology
The proposed FASE entropy for LLM-generated code offers a more efficient approach to computing semantic entropy and estimating code quality without relying on equivalence prediction of LLMs, following the workflow shown in Fig. 21 . By mapping code artifacts into a continuous embedding space, their semantic relationships can be quantified directly, enabling fast and cost-effective assessment of functional similarity and divergence among generated solutions.
1 Fig.1 and Fig.2 contain icons designed using resources from Flaticon.com
4
Lin et al.
Fig. 2. The workflow of computing encoder-based semantic entropy for the samples of code generated for a given task.
Pairwise Semantic Distance via Encoder-Only Embedding Models: The current realization of the semantic equivalence function 𝐸 (·, ·) relies on LLM-based bidirectional entailment [12]. While effective, this approach incurs substantial computational cost from LLM inference. This combination of high per-query cost and O (|𝑥 | 2 ) scaling results in significant overhead, rendering the method impractical for deployment in real-world or large-scale production settings. This paper leverages the low computational cost and high efficiency of embedding models to replace the existing approach, enabling finer-grained estimation of the pairwise semantic distance matrix at substantially reduced cost. For the samples of AI-generated code 𝑥 from the LLM coder, their pairwise semantic distance matrix consists of |𝑥 | 2 values where 𝑑𝑖,𝑗 ∈ [0, 1] represents the semantic difference between the generated code 𝑥𝑖 and 𝑥 𝑗 , 𝑖, 𝑗 ∈ |𝑥 | through cosine distance. As |𝑥 | increases, the estimated distribution converges to the true underlying distribution. The resulting class labels can then be used to compute semantic entropy in the same manner as approaches that rely on LLMs to judge bidirectional functional equivalence. Structural Abstraction using Minimum Spanning Trees: While the full matrix encodes all pairwise distances, it is often redundant and obscures local structure due to its density. Extracting a minimum spanning tree (MST) from a densely connected pairwise distance matrix provides a compact representation of the underlying semantic structure by preserving only the most informative relationships among samples [15]. The MST retains the subset of edges (𝑥𝑖 , 𝑥 𝑗 , 𝑤) for 𝑥𝑖 , 𝑥 𝑗 with 𝑖, 𝑗 ∈ |𝑥 | of weight 𝑤 that connect all samples with minimal total distance, effectively highlighting the strongest semantic affinities while discarding weaker, less informative connections. As a result, the MST reveals the intrinsic topology of the data, including local neighborhoods, cluster boundaries, and the relative dispersion of samples. This abstraction enables efficient downstream analysis while preserving essential structural information. Adaptive Density-Based Clustering for Equivalence Class Construction: With access to a pairwise distance matrix, density-based methods can conveniently be used to compute equivalence clusters. However, fixed reachability or cluster size do not generalize well across tasks since the number and size of semantic equivalence classes are precisely the quantities we aim to estimate for entropy computation. This issue can be addressed by our adaptive density-based clustering algorithm, which leverages key structural properties derived from the pairwise distance matrix and the extracted minimum spanning tree (MST). Empirically, we observe that: (1) even when two sample sets exhibit similar average pairwise distances, those containing meaningful local clusters will yield an MST with a smaller average edge weight; and (2) edges that connect distinct clusters tend to have significantly larger weights than edges within the same
FASE: Fast Adaptive Semantic Entropy for Code Quality
5
cluster. Based on these observations, the adaptive neighborhood threshold 𝜖 is determined using the mode of the MST edge weights 𝑤, thereby reducing the influence of the relatively rare, high-weight inter-cluster edges. The mode is estimated as the maximizer of a Gaussian kernel density estimation computed over 𝑤. The Mo(𝑤) is then scaled by the ratio between the mean edge weight of the MST and the mean value of the full distance matrix as shown in Eq. 2: 𝑤¯ ) (2) 𝑑¯ where 𝛼 is a constant used to control the clustering granularity. The choice of 𝛼 depends on the selected coder LLM and ¯ which are task-specific adaptive variables. embedding model across the entire code generation dataset, unlike 𝑤¯ or 𝑑, 𝜖𝑎𝑑𝑎𝑝𝑡𝑖𝑣𝑒 = 𝛼 · Mo(𝑤) · (1 −
The semantic equivalence class labels of the generated codes become FASE entropy through Eq. 1. To summarize, FASE introduces a novel approach for uncertainty estimation in code generation by replacing expensive LLM-based semantic equivalence checks with an efficient embedding-driven graph analysis framework. By combining semantic embeddings, minimum spanning tree extraction, and adaptive density-based clustering, FASE captures functional uncertainty without ground-truth knowledge and further bridges the gap between fast but structurally limited syntax-based approaches and semantically accurate but computationally expensive LLM-based evaluation, providing a practical and effective solution for reliable code quality estimation for large multi-agent workflows and real-world software engineering scenarios. 4
Evaluation
The evaluation section begins by presenting the research questions, followed by a description of the experimental setup. It then investigates the properties of the PDM and MST in differentiating functional variations among generated code samples. Next, the study demonstrates the necessity of adaptive clustering for semantic entropy estimation. Subsequently, the proposed FASE entropy and its variants are compared against existing baselines. Finally, the section provides a comprehensive analysis of the overhead associated with all evaluated metrics. The development of the proposed FASE entropy aims to answer the following research questions: • RQ1. How effectively can embedding-based graph abstractions in the form of Pairwise Distance Matrix (PDM) and Minimum Spanning Tree (MST) differentiate between tasks with high and low functional consistency in code generation? This research identifies the most influential factors derived from the pairwise distance matrix and the extracted MST that effectively differentiate the quality of generated code samples. • RQ2. To what extent does task-specific adaptive clustering optimize the modeling of semantic equivalence classes compared to fixed-threshold baselines? Given the distance matrix, density-based clustering methods can efficiently group code samples into equivalence classes; however, their performance is highly sensitive to key hyperparameters. Whether a fixed criterion can generalize across different coding tasks remains an open question that requires empirical evaluation. • RQ3. How do FASE and its structural hybrid variants compare against state-of-the-art LLM-based semantic entropy and self-evaluation baselines in predicting code functional correctness within multi-agent workflows? To demonstrate the effectiveness of the proposed method, experiments are conducted comparing it against state-of-the-art structural and semantic entropy approaches, as well as other self-evaluation methods. • RQ4. What are the computational efficiency and scalability gains of FASE compared to traditional LLM-driven bidirectional entailment checks? A quantitative comparison of runtime and resource usage is required to demonstrate the practical value of alternative observation methods.
6
Lin et al.
Answering these research questions provides a comprehensive understanding of both the effectiveness and efficiency of embedding-based uncertainty estimation for code generation. Together, these analyses establish a principled tradeoff between accuracy and efficiency, demonstrating the practicality of FASE for scalable code quality assessment in real-world multi-agent systems. 4.1
Experiment Setup
Two code generation benchmarks are selected to examine the correlation between semantic entropy and code generation quality. • HumanEval [52] is widely used as a foundational benchmark for code-generation models. It consists of 164 handwritten Python programming problems of basic algorithms. • BigCodeBench [55] is a more challenging and practical code-generation benchmark compared to HumanEval. In particular, we use the BigCodeBench-hard subset, which contains 148 Python problems that are closer to real-world programming scenarios. Four open-source 7-billion-parameter LLMs are selected as coders and analysts in the experiments: Mistral-7B [22], CodeLlama-7B [38], DeepSeek-Coder-7B [16], and Qwen2.5-Coder-7B [20]. By evaluating this diverse set of models, this research explores the generality of semantic entropy signals across different model designs, training objectives, and data distributions. The number of parameters is identical to avoid the impact of model scale. Table 1. Embedding Models Used in the Study
Name All-MiniLM-L6-v2 GTE-ModernBERT-base Llama-Embed-Nemotron Qwen3-Embedding Qwen3-Embedding Qwen3-Embedding
# Parameters 0.02B 0.1B 8B 0.6B 4B 8B
Dimensionality 384 768 4096 1024 2560 4096
Context Window 0.2K 8K 32K 32K 32K 32K
For generating semantic embeddings of code and task descriptions, four encoder-only embedding models of different sizes are selected with high performance in text embedding, searching, ranking and clustering tasks as shown in Tab. 1: All-MiniLM-L6-v2 [43], GTE-ModernBERT-base [48], Llama-Embed-Nemotron [2] and Qwen3-Embedding [49]. Their embedding dimensionalities are 384, 768 and 4096 for All-MiniLM, ModernBERT, and Nemotron, respectively. We also explored the impact of parameter size on the same model for Qwen3-Embedding of 0.6B, 4B and 8B parameters with dimensional vector size of 1024, 2560 and 4096. Each encoder differs in underlying training design and domain emphasis, allowing us to test whether semantic representation quality impacts semantic distance, clustering results, and ultimately the estimation of functional correctness. By incorporating this variety of embedding models, we aim to strengthen the soundness and generality of our experimental findings. The experiment is conducted on ASUSTeK ESC4000A-E12 with AMD EPYC™ 9554 processor on a Ubuntu 24.04.2 LTS environment. The GPU we used is NVIDIA H100 NVL with driver version 550.163.01 and CUDA version 12.4. We deployed CodeLlama-7b-Instruct-hf for the agent model. When generating output with nucleus sampling, we set the temperature to 0.5 and top_p to 0.95 for more consistent results. For each task, 10 samples will be generated to compute related metrics. The implementation and experimental artifacts are publicly available at this link2 2 https://github.com/corvolin/CSE4AgenticSoftDev
FASE: Fast Adaptive Semantic Entropy for Code Quality 4.2
7
Effectiveness of Embedding Representations in Capturing Functional Correctness
The overall performance of the selected models on code generation benchmarks is summarized in Fig. 3, which presents results on the HumanEval and BigCodeBench-hard tasks.
Fig. 3. Pass@1 distribution of code generated by Mistral, CodeLlama, DeepseekCoder, Qwen2.5-Coder on HumanEval and BigCodeBench-hard
The green region denotes the number of tasks with pass@1 [5] equal to 1, indicating that the model consistently generates functionally correct solutions on the first attempt. The red region represents tasks with pass@1 equal to 0, where no correct solutions are produced. The orange region captures intermediate cases with 0<pass@1<1, reflecting partial success across multiple attempts. On the HumanEval benchmark, most LLMs are able to generate partially correct solutions, with the number of tasks achieving pass@1 = 1 ranging from 31 for Mistral to 101 for both DeepSeek-Coder and Qwen2.5-Coder. In contrast, performance on the BigCodeBench-Hard benchmark is substantially lower: the majority of models fail to produce any functionally correct solutions. The best performance is achieved by Qwen2.5-Coder, with only 2 tasks reaching pass@1 = 1 and 15 tasks exhibiting partial success with 0<pass@1<1. We also evaluated the effect of introducing an analyst agent that provides semi-structured information on data and control flow based on the task’s functional requirements. The impact on pass@1 is mixed. Across both HumanEval and BigCodeBench-Hard, the number of tasks with pass@1 = 0 and pass@1 = 1 decreases, while the number of tasks with intermediate outcomes of 0<pass@1<1 increases. For instance, Qwen2.5-Coder reduces 9 cases of pass@1 = 0 at the cost of 4 cases of pass@1 = 1. In contrast, other models
8
Lin et al.
tend to exhibit a larger reduction in pass@1 = 1 than in pass@1 = 0. These results suggest that naively introducing an analyst agent without task-specific adaptation does not necessarily improve overall performance, aligning with growing concerns in the research community regarding hallucination propagation in multi-agent systems.
(a) All-MiniLM PDM
(b) All-MiniLM MST
(c) Qwen3-Emb.-0.6B PDM
(d) Qwen3-Emb.-0.6B MST
(e) ModernBERT PDM
(f) ModernBERT MST
(g) Qwen3-Emb.-4B PDM
(h) Qwen3-Emb.-4B MST
(i) Nemoron PDM
(j) Nemotron MST
(k) Qwen3-Emb.-8B PDM
(l) Qwen3-Emb.-8B MST
Fig. 4. The ratio of mean distance in Pairwise Distance Matrix (PDM) and mean edge weight in MST categorized by their connected nodes, Pass-Pass(Blue), Fail-Fail(Purple) and Pass-Fail(Brown), compared to all distances and weights in its samples.
The PDM generated from the embedding models indeed capture meaningful semantic differences among code samples for each task. This research firstly focus on tasks with intermediate pass@1 values and categorize pairwise distances into three groups: (i) between functionally correct codes that passed provided test cases, (ii) between codes that failed the test cases, and (iii) between a correct and a failed code. For each category, the ratio between its mean distance and the overall mean distance of the PDM is computed, and the distribution of these ratios are shown in Fig. 4. Across all embedding models, distances between pairs of functionally correct codes are consistently the smallest. Distances between failed code pairs are slightly larger, likely reflecting diverse failed root cause despite shared task objectives. Notably, both categories yield ratios below 1, whereas distances between correct and failed code pairs consistently exceed 1. This indicates that substantial functional differences correspond to significantly larger semantic distances relative to the task-specific average. After extracting the MST from the PDM, the previously observed pattern remains consistent. Edges connecting functionally correct codes exhibit the smallest ratio, followed by edges between failed codes. In contrast, edges linking correct and failed codes show substantially larger ratio, increasing from approximately 1.7–1.8 in the PDM to 3.4–3.6 in
FASE: Fast Adaptive Semantic Entropy for Code Quality
9
the MST. This amplification indicates that the MST preferentially selects low-weight edges that capture local structure, effectively forming tight clusters among semantically similar codes. Once these intra-cluster connections are established, the higher-weight edges representing connections between functionally distinct groups are incorporated to link the clusters. Consequently, the extracted MST further differentiates the separation of semantic clusters compared to the original PDM.
(a) All-MiniLM
(b) ModernBERT
(c) Nemotron
(d) Qwen3-Emb.-0.6B
(e) Qwen3-Emb.-4B
(f) Qwen3-Emb.-8B
Fig. 5. The ratio between mean distance in Pairwise Distance Matrix (PDM) and mean edge weight in MST categorized for tasks with Pass@1 of 0, between 0 and 1, and 1.
However, as shown in Fig. 3, the majority of code generation tasks across all four LLMs exhibit either pass@1 = 0 or pass@1 = 1. In such cases, there are few or no edges connecting functionally correct and failed codes with substantially higher weights relative to the task average. Moreover, the difference of intra-cluster edges between pairs of correct codes and pairs of failed codes is relatively small. Although not shown, the distributions of pairwise distances and MST edge weights do not differ significantly between tasks with pass@1 = 0 and those with pass@1 = 1. While examining the PDM and MST in isolation does not reveal strong differences between tasks with pass@1 = 1 and pass@1 = 0, comparing edge weights before and after MST extraction provides additional insight. Fig. 5 illustrates the ratio between the mean edge weight of the MST and the mean pairwise distance of the PDM for each task. Across all embedding models, a consistent pattern emerges: tasks with pass@1 = 1 exhibit the lowest ratios, followed by tasks with intermediate pass@1 values, while tasks with pass@1 = 0 have the highest ratios. This indicates that, although overall distance distributions appear similar across tasks, those with pass@1 = 1 contain stronger local structure, reflected by relatively smaller MST edge weights. In contrast, tasks with pass@1 = 0 lack such structure, with their minimum edges remaining comparable to the overall pairwise distances.
10
Lin et al.
Answer to RQ.1: The cosine-distance distributions derived from the PDM and its corresponding MST capture meaningful patterns of functional consistency among generated code samples. In particular, high-weight edges in the extracted MST often correspond to genuine functional differences between correct and incorrect solutions. Although the overall distance distributions for tasks with Pass@1 = 1 and Pass@1 = 0 appear similar, tasks with Pass@1 = 1 consistently exhibit a substantially lower mean MST-to-PDM weight ratio.
4.3
Impact of Adaptive Clustering on Semantic Equivalence Modeling
Given the pairwise distance matrix derived from semantic embeddings of the generated code, an intuitive approach for identifying semantic clusters is to apply density-based clustering techniques [3, 10]. The foundation of adaptive clustering is DBSCAN with a fixed neighborhood radius 𝜖 and HDBSCAN with a fixed minimum cluster size (MCS). The parameter 𝜖 is varied from 0.01 to 0.2, covering the largest range of pairwise semantic distances across different embedding models. The MCS varies from 2 to 9, with a single cluster of size 10 permitted. The adaptive 𝜖 based on the MST/PDM ratio are derived as Eq. 2. The constant 𝛼 ranges from 0.1 to 3 to control clustering granularity.
(a) All-MiniLM
(b) ModernBERT
(c) Nemotron
(d) Qwen3-Emb.-0.6B
(e) Qwen3-Emb.-4B
(f) Qwen3-Emb.-8B
Fig. 6. The Spearman’s 𝜌 correlation measured between the semantic entropy of each method and the pass@1 results when only the coder is involved.
Fig. 6 presents the Spearman’s 𝜌 correlation between semantic entropy and pass@1 in the coder-only setting and all clustering methods show only weak correlation between 0.1 and -0.2. The light and dark grey dashed curves correspond to DBSCAN with a fixed 𝜖and HDBSCAN with a fixed MCS, respectively. Both fixed clustering approaches exhibit weak correlation with the actual functional correctness of the generated code. The blue curves represent adaptive clustering
FASE: Fast Adaptive Semantic Entropy for Code Quality
11
based on the mean pairwise distance of the PDM, where the dashed and dotted variants correspond to approaches without additional scaling, and the solid variant incorporates the proposed scaling mechanism. Similarly, the green curves represent adaptive clustering based on the mean edge weight of the MST. Overall, semantic entropy derived from MST-based statistics demonstrates stronger correlation with functional correctness than approaches based solely on the PDM mean, and the inclusion of scaling further improves the correlation. The solid red curves correspond to semantic entropy computed using the MST mode as the adaptive reference value. This approach consistently achieves the strongest correlation across most clustering granularities, except in extremely fine-grained clustering settings where MST-mean-based approaches occasionally perform better.
(a) All-MiniLM
(b) ModernBERT
(c) Nemotron
(d) Qwen3-Emb.-0.6B
(e) Qwen3-Emb.-4B
(f) Qwen3-Emb.-8B
Fig. 7. The Spearman’s 𝜌 correlation measured between the semantic entropy of each method and the pass@1 results when the analyst agent provides additional instruction to the coder
The adaptive clustering approaches exhibit stronger correlation when an analyst agent is introduced before the coder agent. As shown in Fig. 7, nearly all adaptive clustering methods produce semantic entropy values with improved correlation to pass@1, whereas the fixed clustering approaches remain close to zero correlation. The relative ranking of the adaptive methods remains largely consistent, although the scaled PDM-mean approach now outperforms the MSTmean-based approach. Among all methods, the scaled MST-mode strategy achieves the strongest overall performance across nearly all settings, with the exception of Qwen3-Embedding-0.6B under extremely fine-grained clustering configurations, where alternative approaches perform better.
12
Lin et al.
Answer to RQ.2: Task-specific adaptiveness is essential when clustering code samples represented as semantic embeddings. The strongest correlation between semantic entropy and functional correctness is consistently achieved by density-based clustering methods that use the scaled mode of MST edge weights as the neighborhood radius 𝜖.
4.4
Comparison with LLM-based Semantic Entropy, Self-Evaluation Methods and Other Baselines
To demonstrate the effectiveness of the proposed FASE measurement, this experiment compares it against the stateof-the-art semantic entropy approach based on bidirectional entailment using LLM judgments, as well as alternative entropy formulations derived from textual and structural equivalence. In addition, the evaluation includes widely used LLM self-evaluation baselines such as P(True) and majority voting, where functional correctness is determined according to predictions generated either by the coding LLM itself or by a committee of auxiliary LLMs. The evaluation considers two complementary perspectives: (1) Spearman’s 𝜌 coefficient, which measures the correlation between each uncertainty metric and the ground-truth pass@1 values, and (2) the Area Under the Curve (AUC) score, which evaluates the binary predictive capability of each metric in determining whether a task achieves pass@1 = 1. Table 2. The Spearmnar’s 𝜌 correlation coefficient and AUC score for the baselines and FASE-related entropy using Qwen3-Embedding8B in different workflow and coding LLMs. The entropy measurements are inverted by 1 − entropy to keep correlation positive, as lower entropy indicates greater consistency and better functional correctness. Best scores for each setting are highlighted.
Coder Only Analyst + Coder Mistral CodeLlama DeepSeek Qwen2.5 Mistral CodeLlama DeepSeek Qwen2.5 𝜌 AUC 𝜌 AUC 𝜌 AUC 𝜌 AUC 𝜌 AUC 𝜌 AUC 𝜌 AUC 𝜌 AUC Self P(True) 0.07 0.51 0.22 0.65 0.05 0.52 0.32 0.65 0.02 0.52 0.17 0.84 0.15 0.56 0.22 0.59 Majority Voting 0.13 0.52 0.19 0.61 0.06 0.53 0.28 0.63 0.04 0.51 0.18 0.83 0.07 0.54 0.18 0.59 Textual Entropy 0.06 0.64 0.18 0.74 0.24 0.66 0.36 0.73 0.26 0.6 0.24 0.78 0.4 0.76 0.41 0.67 Structural Entropy 0.2 0.8 0.18 0.76 0.31 0.73 0.37 0.74 0.45 0.95 0.34 0.96 0.56 0.91 0.61 0.86 Semantic Entropy 0.13 0.62 0.35 0.77 0.25 0.69 0.41 0.76 0.15 0.74 0.38 0.85 0.45 0.88 0.7 0.9 Fixed MCS 0.04 0.63 0.12 0.69 0.19 0.63 0.02 0.52 0.07 0.57 0.02 0.51 0.22 0.61 0.13 0.53 Fixed 𝜖 0 0.63 0 0.54 0.04 0.52 0.11 0.53 0.02 0.8 0.04 0.74 0.27 0.78 0.13 0.62 FASE-PDM Mean 0.1 0.57 0.14 0.58 0.19 0.62 0.38 0.73 0.37 0.91 0.24 0.83 0.51 0.86 0.46 0.76 FASE-MST Mean 0.31 0.73 0.22 0.68 0.39 0.73 0.48 0.75 0.37 0.91 0.26 0.83 0.51 0.86 0.46 0.76 FASE-Scaled PDM Mean 0.31 0.73 0.24 0.74 0.39 0.73 0.48 0.75 0.37 0.93 0.28 0.84 0.51 0.86 0.46 0.76 FASE-Scaled MST Mean 0.33 0.75 0.25 0.75 0.4 0.74 0.5 0.76 0.39 0.93 0.28 0.84 0.51 0.86 0.47 0.76 FASE-Scaled MST Mode 0.34 0.75 0.25 0.75 0.4 0.74 0.5 0.76 0.39 0.93 0.3 0.89 0.51 0.86 0.47 0.76 FASE ∪ Structural 0.34 0.77 0.28 0.75 0.4 0.74 0.5 0.76 0.47 0.96 0.34 0.98 0.59 0.91 0.61 0.87 Metric
The results, presented in Tab. 2, demonstrate that FASE entropy consistently outperforms LLM-entailment-based semantic entropy in both correlation and predictive performance across code generation workflows involving either a standalone coder LLM or an analyst–coder pipeline. In particular, combining the semantic equivalence labels generated by FASE using scaled MST-mode clustering with structural equivalence classes through a union operation produces the entropy measure with the strongest overall correlation and predictive capability, outperforming all baseline methods across the evaluated settings. In the coder-only setting, when Mistral is used, all FASE-related variants achieve moderate correlation with functional correctness, although structural entropy attains the highest AUC score of 0.80. For CodeLlama, traditional semantic entropy outperforms the other metrics in both Spearman’s 𝜌 and AUC, while FASE remains highly competitive. In the
FASE: Fast Adaptive Semantic Entropy for Code Quality
13
case of DeepSeek-Coder, FASE with scaled MST Mean/Mode achieves the strongest performance with a 𝜌 of 0.40 and an AUC of 0.74. Similarly, for Qwen2.5-Coder, FASE with scaled MST Mean/Mode again yields the best overall results, reaching a 𝜌 of 0.50 and an AUC of 0.76. When an analyst agent is incorporated into the workflow, FASE ∪ structural entropy provides the most accurate estimation of functional correctness for Mistral-generated outputs, achieving a 𝜌 of 0.47 and an AUC of 0.96. For CodeLlama, the highest correlation is obtained using traditional semantic entropy with a 𝜌 of 0.38, whereas FASE achieves the best predictive performance with an AUC of 0.98. FASE again produces the strongest results for DeepSeekCoder, with a 𝜌 of 0.59 and an AUC of 0.91. For Qwen2.5-Coder, traditional semantic entropy achieves the best overall performance, reaching a 𝜌 of 0.70 and an AUC of 0.90. Answer to RQ.3: FASE and its structural hybrid variants outperforms existing state-of-the-art LLMbased semantic entropy methods by approximately 25% in correlation and 8% in AUC on average, achieving the strongest overall performance across most experimental settings involving all four selected LLMs and both the coder-only and analyst+coder workflows.
4.5
Computational Cost and Scalability Analysis
One of the primary reasons that traditional semantic entropy is impractical in large-scale code generation evaluation is its computational cost from bidirectional entailment queries which scales in O (|𝑥 | 2 ) as code sampling increases. Tab. 3a records and compares the time required for all LLM-related tasks discussed in earlier evaluation steps, and Tab. 3b covers time costs associated with embedding models. Table 3. The time cost for LLM and embedding model tasks measured in seconds. (a) The time cost of LLM-related tasks.
Observation Method Self-Evaluation Majority Voting Semantic Equivalence
Mistral 3.34 31.86 44.23
Codellama 25.4 9.8 110.98
Deepseek-Coder 3.74 31.46 39.24
Qwen2.5-Coder 2.72 32.48 17.63
(b) The time cost of embedding-related tasks.
Embedding Model All-MiniLM ModernBERT Nemotron Qwen3-Emb.-0.6B Qwen3-Emb.-4B Qwen3-Emb.-8B
PDM Generation 0.021 0.073 0.327 0.112 0.264 0.357
MST Extraction 0.0000342 0.0000265 0.0000265 0.0000265 0.0000264 0.0000264
The self-evaluation metric P(True) is relatively efficient to compute compared to more expensive semantic equivalence checks. For a batch of 10 self-evaluation predictions, Mistral, DeepSeek-Coder, and Qwen2.5-Coder require approximately 2–3 seconds, whereas CodeLlama requires substantially longer at 25.4 seconds. Majority voting, which relies on a committee of auxiliary LLMs excluding the coding model itself, incurs significantly higher computational cost, requiring approximately 30 seconds. In comparison, conducting bidirectional semantic equivalence checks across 10 code samples is considerably more expensive, taking 44.23 s, 110.98 s, 39.24 s, and 17.63 s for the respective models.
14
Lin et al.
Tasks involving embedding models instead of LLMs are an order of magnitude faster than LLM-based operations. Computing the PDM for 10 code samples or functionality reviews requires only approximately 0.02–0.3 seconds. Among the evaluated embedding models, All-MiniLM-L6-v2 is generally the fastest, followed by GTE-ModernBERT, while the Nemotron and Qwen3-Embedding series are the slowest, consistent with their larger model sizes. The extraction of the MST from the PDM incurs negligible computational overhead and is effectively independent of the choice of embedding model. Answer to RQ.4: The embedding models provide substantial efficiency gains, reducing the overhead to approximately 0.3% of the traditional pairwise LLM-based equivalence checks on average.
Overall, the experimental results provide strong evidence supporting the effectiveness and efficiency of the proposed framework. The results show that embedding-based representations meaningfully capture functional differences between code samples, and adaptiveness is crucial for reliable clustering, with density-based methods using scaled MST-PDM ratio. FASE consistently outperforms existing LLM-based semantic entropy methods and offers substantial efficiency benefits, reducing computational overhead to roughly 0.3% of that required by traditional LLM-based equivalence checking. Collectively, these findings validate both the predictive strength and practical efficiency of FASE for scalable code quality estimation. 5
Related Work
Several lines of research have explored the use of multiple LLM agents to enhance automated code generation. At the same time, considerable attention has been devoted to understanding their inherent limitations, particularly those arising from code quality issues, hallucination propagation, and strategies for mitigating cascading errors across collaborative agent workflows. Multi-agent Frameworks for Code Generation. Recent advances in large language models (LLMs) have enabled autonomous AI agents for software development, motivating extensive research on agent architectures, collaboration, and evaluation [13, 17, 30, 36]. Prior studies highlight both the potential of these systems to automate complex software engineering tasks and the challenges they introduce, including error propagation, limited self-assessment, and coordination overhead [47]. To improve robustness, multi-agent frameworks have been proposed to encourage specialization and collaboration among agents with distinct responsibilities [40]. Several collaborative agent frameworks have shown promising results for code generation and repair. CodeCoR [37] employs specialized agents for prompt generation, coding, testing, and repair, while MetaGPT [18] integrates human-inspired software engineering workflows into structured multi-agent prompting pipelines. Dong et al. [8] propose a self-collaboration framework with analyst, coder, and tester agents that substantially improves Pass@1 over single-agent generation. AdaCoder [54] further introduces adaptive planning and iterative debugging, achieving better generalizability, faster inference, and lower token consumption across diverse LLMs. Beyond code generation, multi-agent collaboration has also been applied to software requirement compliance verification through retrieval-augmented generation and human-in-the-loop feedback [7]. Collectively, these studies demonstrate the importance of adaptive coordination and reliability estimation in autonomous software engineering systems. Agent Hallucination and Uncertainty. Hallucination in large language models has received significant attention, leading to extensive surveys and empirical studies on hallucination detection and mitigation [19, 50, 51]. Prior work categorizes hallucinations across both natural language and code generation tasks, highlighting the challenge of
FASE: Fast Adaptive Semantic Entropy for Code Quality
15
identifying hallucinations without ground-truth answers. FEWL [44] introduces a principled hallucination metric that weights evaluator LLMs according to estimated expertise. In code generation, HalluCode [28] shows that current LLMs struggle to recognize and classify hallucinations effectively. CodeMirage [1] further establishes one of the first comprehensive taxonomies of code hallucinations and introduces a dedicated benchmark for hallucination detection. Extending this direction, Liu et al. [29] propose a broader taxonomy covering multiple categories of hallucinations, their causes, and impacts across different models and benchmarks. Recent studies also explore improved detection methodologies. Yang et al. [46] propose a hybrid static–dynamic hallucination detection framework that substantially improves detection performance on multiple benchmarks. Foodeei et al. [14] investigate semantic uncertainty under different decoding strategies and show that structured reasoning can improve both semantic diversity and functional correctness. Collectively, these studies motivate the need for efficient semantic-level uncertainty estimation methods for reliable code generation systems. LLM Code Quality Assurance. Several lines of research have explored software quality assurance for LLM-based systems. Self-confidence estimation methods such as P(True) and P(IK) [23] attempt to predict output reliability without ground-truth labels, while self-reflection frameworks improve consistency through iterative feedback and refinement [21]. To reduce hallucinations in code generation, De-Hallucinator [9] iteratively retrieves project-specific API references to ground model predictions, significantly improving API usage and test generation quality. Repositoryaware approaches such as 𝐴3 -CodGen [26] further incorporate local, global, and third-party library context to reduce logical inconsistencies and improve code reuse. At the systems level, multi-agent software engineering frameworks increasingly adopt software engineering practices such as milestone-based collaboration assessment [53], contributionaware agent selection [34], and adaptive coordination strategies. Interactive workflows such as TICODER [11] improve code generation accuracy through iterative intent clarification and test-driven feedback. Recent work also explores LLMassisted software testing and oracle generation. CANDOR [45] proposes a multi-agent framework for automated unit test generation that combines specialized agents and consensus-based reasoning to mitigate hallucinations in generated test oracles, significantly improving oracle correctness and mutation scores over existing approaches. Empirical studies also examine the reliability and maintainability of LLM-generated code. Liu et al. [33] evaluate ChatGPT-generated programs across correctness, complexity, and security dimensions, highlighting issues related to non-determinism and vulnerability generation. Subsequent work [32] shows that iterative self-repair and static-analysis-guided refinement can partially improve maintainability and correctness. In parallel, automated mutation testing [41, 42] and fuzz testing [4, 6] demonstrate strong potential for validating LLM-generated code. Recent semantic representation learning approaches such as Tailor [31] further emphasize the importance of semantic-level reasoning for reliable functional similarity detection and code quality assurance. Entropy of LLM output. Several prior works have investigated uncertainty estimation in LLMs by accounting for code equivalence, recognizing that multiple surface forms may convey the same underlying meaning. To address this challenge, semantic entropy was proposed as an unsupervised uncertainty measure that aggregates model outputs at the level of meaning rather than token sequences [25]. These methods demonstrate robustness across tasks and datasets, generalize to unseen scenarios, and do not rely on prior task knowledge or labeled data [12]. In the context of code generation, structural entropy [39] is specifically designed for source code represented as abstract syntax trees, providing an alternative approach for efficient code quality estimation. FASE provides a scalable uncertainty estimation mechanism that directly addresses a key limitation in current multiagent code generation systems—namely, the lack of efficient and reliable measures for agent-level hallucination and error propagation. By bridging embedding-based semantic reasoning with entropy-driven evaluation, it complements existing
16
Lin et al.
LLM code quality assurance and entropy-based methods, offering a practical alternative for modeling uncertainty in complex multi-agent workflows. 6
Threats to Validity
Despite its strong empirical performance, FASE inherits common limitations of embedding-based and clustering-driven approaches, as well as constraints of experiments: Embedding Representation: A key limitation of textual and structural similarity measures is that the true semantics of a program may change drastically due to a single operator or variable modification. Although modern embedding models often employ attention mechanisms, embeddings of code snippets with opposite functionality may still exhibit high similarity, thereby undermining the validity of pairwise semantic comparisons. The combination of FASE and structural entropy through a union operation achieves the best overall performance in Tab. 2 because abstract syntax trees are highly sensitive to fine-grained token-level mutations that embedding-based semantic representations may overlook. Prompt Engineering: The quality of prompts significantly influences LLM-generated outputs [24]. The experiments conducted in this study employ relatively simple prompting strategies. For code generation tasks, the models are provided with only basic coding instructions with a persona system prompt. For functional correctness and equivalence evaluation, a 2-shot prompting strategy is adopted, consisting of one positive example (functional/equivalent) and one negative example (failed/not equivalent). The extent to which FASE generalizes under more sophisticated prompting strategies remains an open question. Code Generation Scale: Due to computational resource constraints, this study evaluates only open-source models with 7B parameters. The inclusion of additional models with broader architectural diversity and varying parameter scales may influence the observed performance and generalizability of FASE. 7
Conclusion & Future Directions
This research explores the application of FASE entropy in code generation and demonstrates its effectiveness in estimating the functional correctness of generated code in the absence of ground-truth test cases. Given the impractical computational cost of LLM-based bidirectional equivalence checks, this research offers a scalable alternative that combines code semantic and structural consistency. Extensive evaluations across four widely used code LLMs and four embedding models show that this approach achieves high correlation and predictive performance while significantly reducing computation time. By enabling efficient, ground-truth–free monitoring of code quality, the proposed FASE entropy supports multi-agent code generation workflows with reduced risk of error propagation and hallucination. Several directions remain for future work. First, the evaluation can be expanded to include a broader range of foundation LLMs, embedding models, and code generation benchmarks to further validate the robustness and generalizability of FASE. Second, FASE does not explicitly distinguish cases where all generated solutions consistently drift away from the intended functionality with low entropy. Incorporating an additional dimension of semantic alignment with task requirements may improve the detection of functionally incorrect yet semantically consistent outputs. Finally, FASE can be integrated directly into adaptive multi-agent decision-making frameworks, where uncertainty estimates dynamically guide agent selection, workflow orchestration, refinement strategies, or verification depth. References [1] Agarwal, V., Pei, Y., Alamir, S., and Liu, X. Codemirage: Hallucinations in code generated by large language models. arXiv preprint arXiv:2408.08333
FASE: Fast Adaptive Semantic Entropy for Code Quality
17
(2024). [2] Babakhin, Y., Osmulski, R., Ak, R., Moreira, G., Xu, M., Schifferer, B., Liu, B., and Oldridge, E. Llama-embed-nemotron-8b: A universal text embedding model for multilingual and cross-lingual tasks, 2025. [3] Campello, R. J., Moulavi, D., and Sander, J. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining (2013), Springer, pp. 160–172. [4] Cao, D., Hong, Y., Pan, Q., and Wu, J. Program interoperable large language model software testing scheme: a case study on javascript engine fuzzing. IEEE Transactions on Dependable and Secure Computing (2025). [5] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [6] Chen, X., Liu, J., Zhang, Y., Hu, Q., Han, Y., Zhang, R., Ran, J., Yan, L., Huang, B., and Ma, S. Traceawareness and dual-strategy fuzz testing: Enhancing path coverage and crash localization with stochastic science and large language models. Computers and Electrical Engineering 123 (2025), 110266. [7] Das, S., Deb, N., Chaki, N., and Cortesi, A. A multi-agent rag framework for regulatory compliance checking of software requirements. ACM Transactions on Software Engineering and Methodology (2025). [8] Dong, Y., Jiang, X., Jin, Z., and Li, G. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 1–38. [9] Eghbali, A., and Pradel, M. De-hallucinator: Mitigating llm hallucinations in code generation tasks via iterative grounding. arXiv preprint arXiv:2401.01701 (2024). [10] Ester, M., Kriegel, H.-P., Sander, J., Xu, X., et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (1996), vol. 96, pp. 226–231. [11] Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., and Lahiri, S. K. Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. [12] Farqhar, S., Kossen, J., Kuhn, L., and Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature 630, 8017 (2024), 625–630. [13] Ferrag, M. A., Tihanyi, N., and Debbah, M. From llm reasoning to autonomous ai agents: A comprehensive review. arXiv preprint arXiv:2504.19678 (2025). [14] Foodeei, D., Fan, S., and Jaggi, M. Semantic uncertainty in advanced decoding methods for llm generation. arXiv preprint arXiv:2506.17296 (2025). [15] Gagolewski, M., Cena, A., Bartoszuk, M., and Brzozowski, Ł. Clustering with minimum spanning trees: How good can it be? Journal of Classification 42, 1 (2025), 90–112. [16] Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196 (2024). [17] He, J., Treude, C., and Lo, D. Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1–30. [18] Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Yau, S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for a multi-agent collaborative framework. In Proceedings of the International Conference on Learning Representations (2024), vol. 2024, pp. 23247–23275. [19] Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43, 2 (2025), 1–55. [20] Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). [21] Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., and Fung, P. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computational Linguistics (2023), pp. 1827–1843. [22] Jiang, A. Q., Sablayrolles, A., Nadeau, N., Usunier, N., and Lample, G. Mistral 7b. arXiv preprint arXiv:2310.06825 (2023). [23] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 (2022). [24] Khojah, R., de Oliveira Neto, F. G., Mohamad, M., and Leitner, P. The impact of prompt programming on function-level code generation. IEEE Transactions on Software Engineering (2025). [25] Kuhn, L., Gal, Y., and Farqhar, S. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 (2023). [26] Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., and Li, Q. A3-codgen: A repository-level code generation framework for code reuse with local-aware, global-aware, and third-party-library-aware. IEEE Transactions on Software Engineering 50, 12 (2024), 3369–3384. [27] Lindley, D. V. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics 27, 4 (1956), 986–1005. [28] Liu, F., Liu, Y., Shi, L., Huang, H., Wang, R., Yang, Z., Zhang, L., Li, Z., and Ma, Y. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971 (2024). [29] Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., and Ma, Y. Beyond functional correctness: Exploring hallucinations in llm-generated code. IEEE Transactions on Software Engineering (2026). [30] Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., and Lou, Y. Large language model-based agents for software engineering: A survey.
18
Lin et al.
ACM Transactions on Software Engineering and Methodology (2024). [31] Liu, J., Zeng, J., Wang, X., and Liang, Z. Learning graph-based code representations for source-level functional similarity detection. In Proceedings of the 45th International Conference on Software Engineering (2023), IEEE, pp. 345–357. [32] Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., and Lo, D. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–26. [33] Liu, Z., Tang, Y., Luo, X., Zhou, Y., and Zhang, L. F. No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering 50, 6 (2024), 1548–1584. [34] Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling (2024). [35] Maveli, N., Vergari, A., and Cohen, S. B. What can large language models capture about code functional equivalence? arXiv preprint arXiv:2408.11081 (2024). [36] Mohammadi, M., Li, Y., Lo, J., and Yip, W. Evaluation and benchmarking of llm agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 (2025), pp. 6129–6139. [37] Pan, R., Zhang, H., and Liu, C. Codecor: An llm-based self-reflective multi-agent framework for code generation. arXiv preprint arXiv:2501.07811 (2025). [38] Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [39] Song, Y., Sun, T., Tang, X., Rajput, P. K., Bissyandé, T. F., and Klein, J. Measuring llm code generation stability via structural entropy. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (2025), IEEE, pp. 3922–3926. [40] Talebirad, Y., and Nadiri, A. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314 (2023). [41] Tip, F., Bell, J., and Schäfer, M. Llmorpheus: Mutation testing using large language models. IEEE Transactions on Software Engineering (2025). [42] Wang, G., Xu, Q., Briand, L., and Liu, K. Mutation-guided unit test generation with a large language model. IEEE Transactions on Software Engineering (2026). [43] Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems 33 (2020), 5776–5788. [44] Wei, J., Yao, Y., Ton, J.-F., Guo, H., Estornell, A., and Liu, Y. Measuring and reducing llm hallucination without gold-standard answers. arXiv preprint arXiv:2402.10412 (2024). [45] Xu, Q., Wang, G., Briand, L., and Liu, K. Hallucination to consensus: Multi-agent llms for end-to-end junit test generation. ACM Transactions on Software Engineering and Methodology (2026). [46] Yang, B., Dang, J., Liu, H., and Jin, Z. Advancing llm-generated code reliability: A hybrid approach for hallucination detection. IEEE Transactions on Software Engineering (2025). [47] Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., and Shmueli-Scheuer, M. Survey on evaluation of llm-based agents. arXiv preprint arXiv:2503.16416 (2025). [48] Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track (2024), pp. 1393–1412. [49] Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., Huang, F., and Zhou, J. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025). [50] Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics (2025), 1–46. [51] Zhang, Z., Wang, C., Wang, Y., Shi, E., Ma, Y., Zhong, W., Chen, J., Mao, M., and Zheng, Z. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. ACM on Software Engineering 2, ISSTA022 (2025), 481–503. [52] Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2023), pp. 5673–5684. [53] Zhu, K., Du, H., Hong, Z., Yang, X., Guo, S., Wang, Z., Wang, Z., Qian, C., Tang, X., Ji, H., et al. Multiagentbench: Evaluating the collaboration and competition of llm agents. arXiv preprint arXiv:2503.01935 (2025). [54] Zhu, Y., Liu, C., He, X., Ren, X., Liu, Z., Pan, R., and Zhang, H. Adacoder: An adaptive planning and multi-agent framework for function-level code generation. IEEE Transactions on Software Engineering (2025). [55] Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 (2024).
Received 5 June 2026