arXiv CS open access

Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation

arXiv:2609.01601v1 [cs.SE] 1 Sep 2026

Kefeng Duan∗ , Dewu Zheng∗ , Yanlin Wang† , Terry Yue Zhuo, Mingwei Liu, Jianxing Yu, Jiachi Chen, Ensheng Shi, Xilin Liu, Yuchi Ma, and Zibin Zheng

also remain consistent with project-specific APIs, dependencies, data structures, coding conventions, and the broader repository context [7]–[9]. Therefore, repository context is essential for resolving project-specific dependencies and preserving consistency with the existing codebase. However, real-world code repositories often exceed the input length limits of LLMs, making it infeasible to provide the entire repository as input. To address this challenge, existing studies have widely adopted retrievalaugmented generation (RAG), which first retrieves task-relevant context from the repository and then incorporates it into the generation prompt. Representative repository-level RAG methods have improved contextual support from several perspectives, including context quality and retrieval decision-making. To improve context quality, RepoCoder [10] iteratively constructs new retrieval queries using generated results from previous rounds, Cocomic [11] locates relevant code dependencies based on a project context graph, and DRACO [12] retrieves finegrained code entities through dataflow analysis. In retrieval decision-making, RepoFormer [13] determines whether retrieval should be triggered for the current task, RLPG [14] identifies the type of dependency context required by the task, and ProCC [15] dynamically selects suitable retrieval prompt construction strategies. Overall, these methods improve repositorylevel code generation mainly by enhancing how repository Index Terms—Repository-level code generation, large language context is obtained, selected, and organized. models, retrieval-augmented generation, critical token, dense Although the above methods improve the quality of reposiretriever. tory context from the perspectives of retrieval strategy, context source, and retrieval decision-making, they still organize and I. I NTRODUCTION use context mainly at the task level, lacking explicit modeling Recent LLMs are increasingly used in real-world soft- of key token-level decisions during generation. Specifically, ware development [1]–[4]. Unlike standalone code-generation existing methods typically treat the retrieved context as holistic tasks [5], [6], repository-level code generation targets real- support for the entire generation task, implicitly assuming that world development within existing repositories. In this setting, a set of relevant repository contexts can continuously serve generated code must not only satisfy task requirements but the subsequent generation process. However, code generation is a highly path-dependent autoregressive process, and dif∗ K. Duan and D. Zheng are co-first authors with equal contribution. ferent generation positions may require different contextual † Y. Wang is the corresponding author (e-mail: yanlin-wang@outlook.com). information. K. Duan, D. Zheng, Y. Wang, M. Liu, and Z. Zheng are In repository-level code generation, an incorrect API name, with Sun Yat-sen University, Guangzhou 510006, P.R. China (evariable reference, index, or control-flow token can cause mail: duankf@mail2.sysu.edu.cn; zhengdw5@mail2.sysu.edu.cn; yanlinwang@outlook.com; liumw26@mail.sysu.edu.cn; zhzibin@mail.sysu.edu.cn). subsequent generation to deviate from repository semantics J. Yu is with the School of Artificial Intelligence, Sun Yat-sen University, and eventually fail the entire implementation. In other words, Zhuhai 519082, P.R. China (e-mail: yujx26@mail.sysu.edu.cn). T. Y. Zhuo is with Monash University and CSIRO’s Data61 (e-mail: code generation errors are often not accumulated uniformly terry.zhuo@monash.edu). across all tokens, but instead concentrate at a small number J. Chen is with Zhejiang University (e-mail: chenjiachi@zju.edu.cn). of positions that have a decisive impact on the final result. E. Shi, X. Liu, and Y. Ma are with Huawei Cloud Computing TechWe call these positions critical tokens: a small number of nologies Co., Ltd. (e-mail: shiensheng@huawei.com; liuxilin3@huawei.com; mayuchi1@huawei.com). tokens in the LLM inference chain whose correctness plays Abstract—The repository-level code generation task requires synthesizing code that satisfies task requirements while remaining consistent with the target repository context. Since real-world repositories often exceed the input length limits of LLMs, existing approaches commonly adopt retrieval-augmented generation (RAG) to provide repository-specific context. Despite improving repository-context retrieval, existing methods typically provide context as task-level support, without explicitly identifying the critical tokens that require fine-grained repository context during generation. During the autoregressive generation process of LLMs, errors often concentrate at a small number of decisive positions: once such tokens are generated incorrectly, subsequent code may follow an incorrect semantic path and eventually lead to functional failure. We refer to these positions as “critical tokens”. In this paper, we propose ACT O R, an adaptive critical tokenaware retrieval framework for repository-level code generation. ACT O R identifies critical tokens during generation and triggers targeted retrieval on demand to provide repository context at these decisive positions. In addition, we design a position-aware weighting method for dense retrievers to prioritize context that is more informative for generation. We evaluate ACT O R on two representative repository-level benchmarks, RepoExec and CoderEval. Experimental results show that ACT O R consistently outperforms state-of-the-art methods, achieving relative improvements of 8.4% on RepoExec and 15.4% on CoderEval. Beyond performance gains, we systematically quantify the impact of critical tokens, revealing their central role in major generation failures and highlighting the necessity of targeted retrieval strategies. We provide the code and data at https://github.com/DeepSoftwareAnalytics/ACToR.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

a decisive role in the correctness of the final answer, and whose incorrect generation may cause the entire generated result to fail. Therefore, the core limitation of existing retrieval pipelines is not merely that retrieval occurs before generation or remains at the query level, but that they lack a mechanism to identify these critical tokens and provide targeted repository context support at these key positions. Based on this observation, we introduce ACT O R (Adaptive Critical Token-aware Retrieval), a dynamic, fine-grained retrieval framework. Unlike static methods, our framework operates during the generation process. It identifies critical tokens where the model requires additional context and performs targeted, on-demand retrieval for each one. This adaptive mechanism ensures that the model always has the precise information it needs, thereby mitigating the quality degradation and functional failures common in existing systems. Besides, we propose a position-aware weighting method for dense retrievers. This allows us to retrieve context that is more informative for generation. We validate our approach on two standard repository-level code generation benchmarks: RepoExec [16] and CoderEval [7]. ACT O R consistently improves over strong repository-level baselines, with relative gains of 8.4% on RepoExec and 15.4% on CoderEval. Complementing these end-to-end results, we provide a quantitative characterization of critical tokens: they occupy only a small fraction of generated positions (about 5–11% across models), yet their composition and syntactic profiles concentrate much of the model’s error and uncertainty, motivating sparse, generation-time retrieval updates rather than repository context supplied only once up front. Our analysis further ties major failures to this small set of pivotal positions, underscoring the necessity of targeted retrieval strategies like ACT O R. We summarize our contributions as follows: • We introduce the concept of critical tokens for repositorylevel code generation. To our knowledge, this is the first work to systematically quantify the decisive impact these tokens have on the quality of the final generated code. • We propose a novel targeted retrieval framework designed for these critical tokens. This framework performs on-demand retrieval to supply the language model with precise, timely context at the most crucial steps of the generation process. • We design a position-aware weighted pooling method to enhance our retrieval mechanism. This technique refines the context embedding process, significantly boosting the precision of retrieval. • We conduct extensive experiments on representative repository-level code generation benchmarks, demonstrating that our approach significantly and consistently outperforms existing SOTA methods. II. M OTIVATING E XAMPLES This section uses two real examples from CoderEval [7] to illustrate two key properties of critical tokens. First, although any token change may affect subsequent generation, critical tokens are distinctive because their errors often change the model’s core judgment about the current implementation logic,

Docstring

Ground Truth try:

def popitem (self): """ Remove and return the `(key, value)` pair first inserted. """

key = next (iter(self.__order)) except StopIteration: raise KeyError() else: return (key, self.pop(key)) Non-Critical token position

Critical token position next

…

(iter

GT critical token

Try:\nkey

self

…

.buffer

return

…

</END>

value

…

</END>

else:

Key)

)</END>

Wrong predicted critical token try: key = self.buffer.popitem(last=False) value = self.data[key] del self.data[key] except StopIteration: raise KeyError() return (key, self.pop(key))

try: key = next (iter(self._order)) except StopIteration: raise KeyError() else: value = self.pop(key) return (key, value)

Fig. 1. Motivating example from CoderEval task 62b8d22f [7]: an incorrect critical token can cause catastrophic failure in subsequent generations (popitem scenario). Signature + Docstring

Ground Truth

def normalize_cmd (cmd:tuple[str, ...])>tuple[str,...]: ""“ Fixes for the following issues on windows - https://bugs.python.org/issue8557 - windows does not parse shebangs This function also makes deep-path shebangs work just fine """

# Use PATH to determine the executable exe = normexe (cmd[0]) # Figure out the resulting command cmd = parse_filename (exe) + (exe,) + cmd[1:] # This could have given us back another bare executable exe = normexe (cmd[0]) return (exe,) + cmd[1:]

Critical token position parse

Critical token position to

… Path

determine

…

Critical token position

GT critical token exe

…

[0]

…

</END>

)

…

</END>

cmd

Wrong predicted index critical token </END>

Wrong predicted API critical token

with

env

…

…1:]

)</END>

Wrong predicted comment critical token

# Use PATH with env to determine the executable exe = normexe(cmd[0], env=env) # Figure out the resulting command cmd = parse_filename(exe) + (exe,) + cmd[1:] exe = normexe(cmd[0]) return (exe,) + cmd[1:]

# Use PATH to determine the executable exe = normexe(cmd[0]) # Figure out the resulting command cmd = exe + cmd[1:] # This could have given us back another bare executable exe = normexe(cmd[0]) return exe

# Use PATH to determine the executable exe = normexe(cmd[0]) # Figure out the resulting command cmd = parse_filename(exe) + (exe,) + cmd[1:] # This could have given us back another bare executable exe = normexe(cmd) return exe

Fig. 2. Motivating example from CoderEval task 62e4fb4d [7]: critical tokens exhibit diverse syntax and can appear in multiple roles (normalize cmd scenario).

causing later code to follow an incorrect semantic path and eventually leading to function-level failure. Second, critical tokens are not limited to a fixed syntactic form; they may appear as API names, variable references, index expressions, or even semantic cues in comments. Therefore, a single pregeneration retrieval step or a fixed syntactic rule can hardly cover the tokens that truly determine the final generation result. Motivating example 1: Figure 1 shows the difference between a critical-token error and an ordinary token error in terms of their consequences. The goal of the popitem function is to remove and return the earliest inserted item from a cache. This behavior depends on a repository-specific ordering structure: the correct implementation should first obtain the earliest key from self._order, for example through next(iter(self._order)), and then update

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

the cache according to that key. However, at the highlighted position, the model generates self instead of next. This error directly changes the model’s judgment about which repository object the subsequent code should be built around. Instead of constructing an iterator over self._order, the model turns to an incorrect call such as self.buffer.popitem.... As a result, the generated function no longer follows the repository’s insertion-order deletion logic; the remaining code is generated around this wrong operation and eventually deviates completely from the intended behavior. In contrast, the noncritical token error in the figure affects only a local type-related detail and does not break the main control-flow or data-flow structure of the function. This comparison shows that critical tokens are not simply tokens predicted incorrectly; they are positions where an error can make the model depart from the core implementation logic and thereby compromise functional correctness.

III. M ETHODOLOGY A. Overall Pipeline In this section, we introduce an innovative methodology for repository-level code generation, centered on a targeted retrieval mechanism guided by critical tokens. The methodology is divided into two phases: offline training and online inference. In the offline phase, we focus on training a lightweight discriminative ensemble model. Its purpose is to accurately identify critical tokens. During the online inference phase, the methodology enhances code quality through two key techniques: Position-Aware Weighted Retriever and Critical Token-Guided Inference. Figure 3 shows the overall pipeline of our study. B. Offline Training

During the training phase, we design a critical token discriminator that identifies critical tokens and triggers dynamic retrieval based on the model’s hidden states during generation. To obtain high-quality training data that match this scenario, Motivating example 2: Figure 2 further shows the syntactic we follow the data construction pipeline described below (see and semantic diversity of critical tokens. The normalize cmd Figure 3). function needs to normalize a command representation ac1) Data Construction: This subsection describes the procording to the repository’s existing implementation, including cedure for building our training set, which consists of three how executable names are parsed, how command tuples are stages: Repository Filtering, Function Sampling, and Prompt indexed, and which internal helper APIs should be invoked. Construction. Therefore, the task requires not only understanding the function 1) Repository Filtering. We select our data from RepoSTdescription but also following the conventions encoded in the Train [8], a large-scale training dataset specialized for surrounding code. In the comment-related case, an incorrect repository-level code generation, containing 7,415 functions comment-related token misleads the model into inferring that from 824 repositories. To ensure data quality, we filter an additional environment argument is needed, causing it to repositories according to the following criteria: hallucinate an env parameter that is not supported by the ✓ The repository must contain more than fifteen training repository API. In the API-related case, the model fails to samples (functions). generate the correct call to parse_filename and instead ✓ The repository must have more than ten stars on GitHub. falls back to simple string concatenation, such as cmd = exe ✓ The repository must be a real-world software project, + cmd[1:]. This bypasses the repository-defined filename rather than a collection of solutions for algorithmic chalparsing logic and makes the normalization process inconsistent lenges (e.g., LeetCode) or academic assignments. with the project implementation. In the index-related case, ✗ The repository must not appear in our evaluation the model incorrectly uses the entire cmd object instead of benchmarks (RepoExec [16] and CoderEval [7]). accessing a specific element, for example passing cmd directly After applying these filters, we include 102 repositories for to normexe, which expects a single element. This indexing constructing training data. error causes a type mismatch, propagates to the return statement, 2) Function Sampling. After repository filtering, we further and eventually leads to a runtime failure. Although these three sample candidate functions within the selected repositories errors involve comments, API calls, and index expressions, according to the following rules: they share the same property: each occurs at a position that ✓ The sample must include a function signature. determines the direction of the subsequent implementation ✓ The sample must include a complete docstring (used to and causes the generated result to deviate from repository describe function behavior and constraints). semantics. Our training dataset excludes samples that do not meet both conditions. After that, we select 1,203 task samples Together, these examples show that critical tokens have for our training. two important properties. On the one hand, their errors can 3) Prompt Construction. For each repository, we follow make the model continue generation along an incorrect RLCoder [9] and adopt the “Split and Aggregate candidate” semantic path, causing function-level failures that go far strategy to construct repository context candidates. For each beyond a local token error. On the other hand, they are task sample, we concatenate the function signature and docsyntactically diverse and cannot be fully captured by a fixed string to form the initial code sequence X. This sequence X syntactic category or a single pre-generation retrieval step. serves as the query to a dense retriever. The retriever ranks These observations motivate a generation-time mechanism that these candidates by cosine similarity and returns an ordered identifies critical tokens as they arise and provides targeted set of context snippets: C = {C1 , C2 , . . . , Cm }, which is repository context support at those key positions. sorted by descending similarity. We then concatenate the

(b) ONLINE INFERENCE (a) OFFLINE TRAINING

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

(a) Data Construction

(b) Token Evaluation

Raw Tasks Construction

LLM Generation by Token

Programming Task

def quick_sort(arr): pivot = arr[0] less = [i for i in arr[1:] if i <= pivot] greater = [i for i in arr[1:] if i > pivot]

(d) Classifier Training

Train classifier to judge token category

Label

Labeled Data for training Uncertainty

Subsequent Attention Influence

Mismatch

Normal

return quick_sort …

Weighted Retrieval Module

Programming Task

Judge by Token During Generation Continue Generation

Function Signature

Non-Critical Token

def reverse(input_string: str) -> str:

Docstring

Token

""" Returns the reversed string. """

Label

Fine-grained retrieval for more precise context.

Critical Token

Fig. 3. The Pipeline of ACT O R.

contexts and the initial code sequence in retrieval score order to form prompts as: P = [C1 | C2 | . . . | Cm | X], where “|” denotes a separator between contexts. In practice, we use file-path annotations of each context and the target function prompt as explicit separators to preserve context boundaries and improve readability. After that, we treat the corresponding function body as the ground truth and record both the prompt and ground truth for training. 2) Token Evaluation: At this stage, our goal is to determine, for each generated token, whether it is a critical token and to construct a labeled dataset for this classification task. A critical token is a token that the model tends to mispredict and whose erroneous generation would significantly affect subsequent tokens, substantially increasing the risk of functional failure in the generated code. In other words, an error at such a position is not only incorrect locally but also likely to trigger cascading errors or to break program semantics. Based on this definition, we quantify each generated token using the following three criteria: • Token Mismatch. The model’s top-1 prediction at position i differs from the ground-truth token. The formula can be expressed as: xi ̸= x̂i (1) where xi is the ground-truth token at position i and x̂i denotes the model’s top-1 prediction. A top-1 mismatch indicates high prediction difficulty at this position. If a downstream impact accompanies this mismatch, the token is more likely to be critical. • Uncertainty. We introduce entropy to quantify the model’s uncertainty when generating the token at position i: X Hi = − pi (v) log pi (v), (2) v∈V

where pi (v) denotes the probability assigned by the model to token v at position i, and V is the vocabulary of large language models. A larger entropy Hi indicates greater

model uncertainty at that position, and therefore a higher likelihood that the token is critical. • Subsequent Attention Influence. We propose a metric based on the model’s self-attention mechanism [17] to assess the potential influence of a token on subsequent tokens. Selfattention, the core mechanism of Transformer-based LLMs, is computed as: QK ⊤ A = softmax mask √ (3) dk where Aj,i denotes the attention weight from position j (Query) to position i (Key/Value). Considering up to the next k tokens (or until the sequence end), we define the subsequent attention on position i as: amean (i) = Meanj∈{i+1,...,i+k} T opK (Aj,i ).

(4)

Intuitively, a larger amean (i) means several later positions place larger attention on the current position, which means an error at position i is more likely to propagate and amplify, increasing its criticality. In practice, we use K = 5 as an empirical setting. Rather than K = 1, Top-5 is less sensitive to individual outliers or attention noise. Additionally, we do not use the average attention score across all subsequent positions because it would dilute the concentrated dependencies among a few positions. Top-5 strikes a balance between robustness and sensitivity, capturing significant dependencies from later positions while reducing the influence of single extreme attention values. A generated token is labeled as “critical” if it satisfies a token mismatch condition, or if either Hi or amean(i) exceeds a predefined threshold. Otherwise, it is labeled as “Non-critical”. To mitigate the impact of the model’s own prediction errors on the critical-token judgment, we employ a teacher-forcing evaluation method [18]. After assessing the importance of the token at position i, we replace the model’s prediction at position i with the ground-truth token and continue evaluating subsequent tokens conditioned on this corrected context. This procedure suppresses error propagation effects in downstream

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

evaluation and makes the determination of an individual token’s criticality more reflective of its actual value. 3) Label Balance: However, non-critical tokens constitute the majority of the dataset, leading to pronounced class imbalance. To effectively filter low-value, non-critical samples, we measure the token self-information at each generation position. Self-information, also called surprisal or information content, is a fundamental quantity in information theory that measures the information conveyed by a particular event [19]. Prior work has successfully applied it to tasks such as prompt compression by assessing the importance of lexical units [20]. We compute the self-information of token xi as I(xi ) = − log2 P xi | x0 , x1 , . . . , xi−1 (5)

Algorithm 1: Inference with Critical Token Evaluation. Input: TargetFunctionPrompt, ClassifierEnsemble, Threshold p Output: FinalPrediction (A complete sequence of generated code) Prediction ← [ ]; Context ← Retrieve(TargetFunctionPrompt); while not EndOfGeneration() do CurrentPrompt ← Combine(TargetFunctionPrompt, Context,; Prediction) current token, hidden state ← DecodeOneStep(CurrentPrompt); is critical ← CheckIfCritical(hidden state, ClassifierEnsemble, p); if is critical then Query ← ConstructQuery(TargetFunctionPrompt, Prediction,; current token) UpdatedContext ← Retrieve(Query); RerollPrompt ← Combine(TargetFunctionPrompt, UpdatedContext,;

where a large language model estimates the conditional Prediction, current token) probability. A larger I(xi ) indicates that the token is less predictable and carries more information in the given context. current token ← DecodeOneStep(RerollPrompt); Specifically, we sort all tokens labeled as non-critical by // Re-decode I(xi ) in descending order and preferentially retain samples Context ← UpdatedContext; // Keep context with higher self-information until the number of positive end and negative samples reaches balance. From an informationAppend current token to Prediction; theoretic perspective, this procedure selects “hard negatives”, end improving the quality of the training signal and enabling the return Prediction; classifier to better distinguish between critical and non-critical tokens. 4) Classifier Training: To accurately identify critical tokens, we designed and trained an ensemble of critical token judgers. positions and thus can miss richer, generation-relevant context. Each judger is a 3-layer Multilayer Perceptron (MLP). The Motivated by common code-generation scenarios, we propose core mechanism takes the hidden state generated by a base a weighted pooling scheme whose weights follow a doubleCode-LLM for a specific token as input and outputs a binary endpoint Gaussian shape. probability indicating whether the token is critical. Let L denote Let a text span have length L with token indices 0, 1, . . . , L− the number of decoder layers in the Code-LLM. For a sample 1. We define the unnormalized weight at position i as the sum X, we synchronously save the hidden state at position n from of two Gaussian kernels centered at the two endpoints: layer L of the main model and denote it by (i − (L − 1))2 i2 (L) w = exp − + exp − , i hn = LLM(X) layer L, pos n . (6) (9) 2σ 2 L2 2σ 2 L2 i = 0, . . . , L − 1. This hidden state serves as the MLP’s input. The classifier produces a probability distribution over the two classes: ⊤ pn = pcritical , pnon-critical = Classifier h(L) . n

(7)

We optimize the classifier with the cross-entropy loss. For sample n with label yn ∈ {0, 1} (where yn = 1 indicates a critical token), the loss is Ln = − yn log pcritical + (1 − yn ) log(1 − pcritical ) . (8) In our experiments, we use the final-layer hidden state as the input to the classifier. Empirically, the classifier we train ranges from 4.01 to 10.01 million parameters. This is remarkably lightweight, especially when contrasted with the 1B and 13B scales of the backbone LLMs evaluated. C. Online Inference 1) Weighted Dense Retriever: In repository-level code generation, we construct retrieval queries from the targeted function to obtain retrieval-augmented prompts. Typical dense retrievers [21] obtain code embedding by average-pooling token embeddings, which ignores the relative importance of token

where σ > 0 is a hyperparameter that controls the “sharpness” of the peaks. The first term boosts positions near the beginning (e.g., function signatures); the second term boosts positions near the end (e.g., context immediately adjacent to the ungenerated code). When σ is relatively small, the mass concentrates at the ends; when σ is larger, the two Gaussians overlap, and the weight profile becomes smoother or nearly flat. To obtain a proper weighting for pooling, we apply a softmax normalization over the wi : exp(wi ) αi = L−1 , X exp(wj )

i = 0, . . . , L − 1.

(10)

j=0 d Given token embeddings {ei }L−1 i=0 (where ei ∈ R ), the final pooled vector is the weighted sum

v =

L−1 X

αi ei .

i=0

This method offers two types of benefits:

(11)

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

1) Position-aware retrieval: By allocating a higher weight to standardized and reproducible evaluation. In our experiments, both the sequence head (which often contains signatures we use the Python version, which includes 230 representative or definitions) and the sequence tail (which is near the code-generation tasks drawn from real-world open-source ungenerated code), the pooled vector is more likely to retrieve projects. context that is informative for generation. 2) Low online cost: The weights depend only on σ and L, so B. Baselines they can be precomputed and cached for different lengths To evaluate ACT O R, we compare it with four representative at load time, eliminating expensive computations per query baselines covering non-retrieval prompting, vanilla retrieval, and adding negligible runtime overhead. and SOTA repository-level baselines: 2) Critical Token Guidance Inference: To enhance the accuracy of code generation, we introduce a Critical Token-Guided • RawPrompt. A non-retrieval baseline that provides only the function signature and NL description to the generator. Dynamic Correction strategy for inference (see Algorithm 1 • RawRAG. A vanilla RAG baseline commonly used in prior for details). The core of this strategy is to perform real-time repository-level code generation studies [9], [10]. It uses monitoring on each generated token and, upon identifying a the RawPrompt content as retrieval query, retrieves the topcritical token, immediately apply a correction using targeted ranked context snippets, and prepends them to the prompt, retrieval. representing a standard one-shot retrieval setting. The process begins with an initial retrieval using the target • RepoCoder [10]. This baseline is a SOTA iterative RAG function prompt to obtain the initial context, which is then framework specialized for repository-level code generation. concatenated with the prompt to form the model’s input. In its multi-round process, it leverages previously generated Subsequently, the model generates code token by token. At code to construct more targeted retrieval queries, thereby each step, the corresponding hidden state is extracted and fed progressively refining the contextual information. Through into a pre-trained classifier ensemble for monitoring. this iterative context replenishment mechanism, RepoCoder If a token is classified as normal, the generation proceeds. enhances the model’s code-generation capabilities by supHowever, if the ensemble unanimously identifies a “Critical plying targeted repository context. Token”, the system immediately initiates a dynamic correction • RLCoder [9]. This baseline introduces a retrieval optimizaprocess. It first combines the original prompt, the currently tion strategy based on Reinforcement Learning (RL). Its core generated code, and the critical token into a new query. This mechanism enables the retriever to learn to select the most query is then used to perform a second retrieval to obtain valuable context for code generation adaptively, without an updated, more targeted context. Finally, the model rerequiring manually labeled data. Furthermore, RLCoder decodes the current step using the new context to generate incorporates a “stop signal” mechanism to intelligently detera corrected token. The resulting token is appended to the mine which context candidates to retain, thereby striking an generation sequence. In practice, to speed up inference, we effective balance between retrieval efficiency and generation skip the critical token check when generating line breaks and quality. space tokens. Notably, if a correction occurs, the updated context is retained to guide all subsequent generation steps until a new critical C. Evaluation Metrics token appears, ensuring that the model continues to operate Consistent with prior work in the field of code generation [7], with the most relevant information available. [16], [22], we adopt Pass@k (k = 1, 3, 5) as our core evaluation metric. Within our evaluation framework, a generated code sample is considered a “correct” solution only if it passes IV. E XPERIMENTAL S ETUP all associated unit tests. Based on this criterion, the Pass@k A. Benchmarks metric estimates the probability that at least one of the k We evaluate our approach on two repository-level bench- generated samples for a given task is correct. marks: RepoExec [16] and CoderEval [7]. RepoExec focuses on code generation that involves repository-level dependencies. It is designed to evaluate the D. Model Selection ability to handle and integrate cross-file dependencies within In a typical RAG pipeline, there are two core components: an entire codebase while producing functionally correct code. a retriever, which selects relevant context from the repository, The benchmark comprises 355 Python repository-level code and a generator, which produces the final code conditioned on generation tasks. For each task, comprehensive test suites both the natural language description and the retrieved context. were constructed using unit-test generation and coverage To evaluate the effectiveness and generality of our proposed enhancement techniques. method, we select representative models for both components. CoderEval is a standardized benchmark designed for pracRetriever. In our experiments, we employ UniXcoder as the tical application scenarios. It evaluates the capability of code retriever. As a widely adopted dense embedding model [9], [23], generation across varying levels of contextual dependency. Each UniXcoder [21] demonstrates excellent retrieval performance task includes a function signature, a problem description, a by embedding textual queries and candidate code snippets reference solution, and a suite of unit tests; all components into high-dimensional dense vectors and calculating the cosine are executed within a unified Docker environment to ensure similarity between these vector embeddings. This model can

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

efficiently retrieve the most relevant content from large-scale 8.4% on RepoExec and a remarkable 15.4% on CoderEval over repositories. its baseline. Similarly, the CodeLlama-13B model achieves the Generator. We select two mainstream LLM families to highest Pass@5 score in our entire evaluation (39.57%) on validate the effectiveness and generalization capability of our CoderEval, underscoring our method’s ability to unlock the proposed method: the DeepSeekCoder (DSCoder) series [24] full potential of powerful base models. and the CodeLlama series [25]. For the DSCoder series, we As illustrated in Table II, ACT O R achieves a superior utilize the base version with 1.3B and 6.7B parameters. For the balance between computational overhead and generation quality. CodeLlama series, we utilize the Hugging Face1 version with We observe a slight difference in per-token latency (Ttok = 22.6 7B and 13B parameters. Selecting models with varying scales ms, approximately 3.3 ms higher than the baselines), primarily and architectures enables a comprehensive evaluation of our due to the KV cache recomputation required to integrate method’s applicability across different foundational generators. retrieved fragments into the context. However, this operation is invoked only at sparsely identified critical token positions rather than at every generation step, keeping the associated E. Experimental Details overhead highly controllable. The selective and targeted context To ensure fairness and consistency in comparisons, a unified integration mechanism also contributes to more concise code configuration is used across all experiments. In the retrieval generation. Specifically, ACT O R produces the fewest average stage, the context length is limited to 1K tokens, and the tokens (N tok = 94.59), while achieving an Rgen of 83.80%, maximum number of code snippets is 10. For the training indicating that the generated code remains structurally compact and generation stages, the temperature parameter is set as 0.6, and faithful to the ground truth. Compared with the iterative and the maximum number of generated tokens is limited to RepoCoder, which takes 4.50s per sample, ACT O R reduces 512. The thresholds for subsequent attention influence and end-to-end execution time to 2.66s, demonstrating that dynamic, uncertainty during training are configured as 0.05 and 0.8, targeted retrieval is significantly more efficient than multi-pass respectively. All experiments are executed on a Linux server generation strategies. featuring 8 Intel(R) Xeon(R) Gold 6348 CPU cores and 8 NVIDIA A800 GPUs with 80GB of memory. RQ1 Summary: ACT O R outperforms baselines across V. E VALUATION R ESULTS Our experiments intend to answer the following research questions (RQs): • RQ1: How does ACT O R compare against state-of-the-art methods in accuracy and efficiency? • RQ2: What is the performance impact of each of our core components? • RQ3: How sensitive is ACT O R to its hyperparameter settings? • RQ4: What composition and syntactic characteristics do critical tokens exhibit?

scales (up to 15.4%); Pass@5 gains suggest better accuracy and diversity. Retrieval overhead is modest because correction runs only at critical tokens.

B. RQ2: Ablation Study

Rule-level labeling: We evaluate DSCoder-1B by omitting exactly one labeling criterion from Section III-B2, holding all other settings fixed. Table III reports Pass@k relative to full ACT O R. Table III supports two takeaways. First, no single labeling criterion is redundant: dropping any one consistently lowers Pass@k on both benchmarks, so mismatch, uncertainty, and subsequent-attention cues each supply non-overlapping A. RQ1: Overall Performance and Efficiency As detailed in Table I, our framework, ACT O R, delivers supervision. Second, their relative importance is benchmarkconsistent and significant performance improvements across all dependent rather than a fixed ranking. CoderEval behaves like base models on both the RepoExec and CoderEval benchmarks. a setting where errors are localized to concrete mispredictions, This demonstrates the broad applicability and effectiveness of so mismatch-oriented supervision carries the largest margin when it is removed. RepoExec instead stresses long-horizon, our dynamic retrieval approach. A key finding is our method’s impact on multi-pass success cross-snippet structure, where the subsequent-attention signal rates. On the RepoExec dataset, for instance, while the ACT O R shows the largest single-rule gap (on the order of low-tomid-twenties percent relative at Pass@1). Uncertainty remains DSCoder-6.7B variant shows a marginal dip in Pass@1 score compared to the strongest baseline (RLCoder DSCoder-6.7B ), it informative everywhere but is typically the least damaging achieves superior Pass@3 and Pass@5 results. This pattern omission when removed alone, which is consistent with entropy suggests that our framework enhances the diversity of generated partly correlating with the other two signals. Module-level ablation: We next disable one runtime composolutions, resulting in a higher overall success rate with nent at a time: dynamic critical-token inference or the positionincreased attempts. aware weighted retriever (replaced by mean-pooled retrieval The value of our framework is further highlighted by embeddings). Table IV compares each variant to full ACT O R. the CodeLlama models. When integrated with ACT O R, the Table IV shows that ACT O R is neither “retrieval-only” nor CodeLlama-7B model’s Pass@5 score improves by a relative “inference-only”: both stages are needed. Disabling dynamic 1 https://huggingface.co/ inference removes sparse, on-demand context refresh and

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

TABLE I P ERFORMANCE OF FOUR MODELS ON THE R EPO E XEC AND C ODER E VAL BENCHMARKS . S UPERSCRIPTS IN THE PERCENTAGES INDICATE THE IMPROVEMENT OF ACT O R OVER THE CORRESPONDING BEST BASELINE .

RepoExec

Method

CoderEval

Pass@1

Pass@3

Pass@5

Pass@1

Pass@3

Pass@5

RawPrompt DSCoder-1.3B RawRAG DSCoder-1.3B RepoCoder DSCoder-1.3B RLCoder DSCoder-1.3B ACT O R DSCoder-1.3B

7.55 8.79 10.03 10.82 11.27 ↑4.2%

13.07 15.75 17.21 18.03 19.24 ↑6.7%

15.77 20.00 21.13 21.97 23.10 ↑5.1%

10.87 15.65 18.70 16.09 20.00 ↑7.0%

17.83 23.91 27.39 26.96 28.70 ↑4.8%

22.61 27.39 31.74 30.00 33.48 ↑5.5%

RawPrompt DSCoder-6.7B RawRAG DSCoder-6.7B RepoCoder DSCoder-6.7B RLCoder DSCoder-6.7B ACT O R DSCoder-6.7B

10.93 14.08 16.28 16.68 16.45 ↓1.4%

18.03 23.66 24.76 25.07 26.20 ↑4.5%

21.41 28.73 28.45 28.17 30.70 ↑6.9%

14.35 20.00 20.43 22.61 23.04 ↑1.9%

21.30 27.83 31.74 31.30 32.17 ↑1.4%

26.09 31.74 35.65 33.91 37.83 ↑6.1%

RawPrompt CodeLlama-7B RawRAG CodeLlama-7B RepoCoder CodeLlama-7B RLCoder CodeLlama-7B ACT O R CodeLlama-7B

9.75 12.34 13.80 13.80 15.77 ↑14.2%

15.38 21.13 22.11 21.41 23.94 ↑8.3%

18.59 25.07 27.04 25.63 29.30 ↑8.4%

15.65 19.57 22.61 18.26 23.04 ↑1.9%

23.91 27.83 29.13 24.78 31.30 ↑7.4%

26.09 30.57 30.43 30.87 35.65 ↑15.4%

RawPrompt CodeLlama-13B RawRAG CodeLlama-13B RepoCoder CodeLlama-13B RLCoder CodeLlama-13B ACT O R CodeLlama-13B

9.80 15.72 17.01 13.75 18.59 ↑9.3%

16.68 24.60 27.07 22.31 29.01 ↑7.2%

20.00 27.89 31.83 26.76 32.96 ↑3.6%

18.26 24.78 22.61 23.48 26.52 ↑7.0%

25.22 32.17 33.48 30.43 35.65 ↑6.5%

30.43 34.78 35.65 32.17 39.57 ↑11.0%

Rawprompt Rawrag Repocoder Rlcoder ACT O R

19.3 19.1 19.1 19.2 22.6

2.53 2.21 4.50 2.28 2.66

130.82 108.81 215.61 106.66 94.59

83.15 83.38 89.10 95.38 83.80

0.6

0.7

0.8

Uncertainty

0.9

1.0

0.03 0.04 0.05 0.06

29.13 27.39 23.48 21.74 28.51

0.07

0.03 0.04 0.05 0.06

16.96 16.96 14.78 14.78 15.22

Subsequent Attention Influence

16.96 16.09 13.04 12.61 19.74

24.78 25.65 28.70 24.35 24.78

26.96 26.09 24.35 26.09 26.52 0.6

0.7

0.8

Uncertainty

0.9

1.0

0.03

Rgen

15.65 15.65 20.00 15.22 14.35

26.52 23.91 22.61 25.22 23.04

0.04

Ntok

16.09 13.91 15.65 17.39 15.65

30.00 27.83 26.52 29.57 27.39

0.05

Tsam

Pass@5 30.87 30.00 29.57 31.74 30.00

28.70 29.13 33.48 30.43 32.17

0.06

Ttok

Pass@3 26.20 24.35 26.52 26.96 26.52

31.74 29.13 26.96 28.70 30.43

0.07

Method

Pass@1 17.47 15.65 15.22 16.96 16.52

0.07

TABLE II T IME COST ANALYSIS ON C ODER E VAL BENCHMARK A LL USING THE DSC ODER -1B MODEL . Ttok DENOTES THE AVERAGE INFERENCE LATENCY PER TOKEN ; Tsam REPRESENTS THE END - TO - END EXECUTION TIME PER SAMPLE . Ntok MEASURES THE AVERAGE NUMBER OF GENERATED TOKENS . Rgen IS THE TOKEN - LEVEL RATIO BETWEEN GENERATED CODE AND GROUND TRUTH , CALCULATED EXCLUSIVELY FOR SAMPLES THAT PASS ALL TEST CASES .

30.43 28.26 30.43 29.57 31.30

0.6

0.7

0.8

Uncertainty

0.9

1.0

Fig. 4. Performance of DSCoder-1.3B-Base model on the CoderEval benchmark under different hyperparameter configurations.

C. RQ3: Sensitivity Analysis To explore the robustness of the ACT O R framework and produces the clearest quality loss, with CoderEval exhibitits sensitivity to the empirical thresholds used for identifying ing substantially larger relative drops than RepoExec across critical tokens, we conduct a detailed sensitivity analysis on Pass@k (roughly mid-teens at their strongest). Reverting the CoderEval benchmark. As the identification of critical weighted pooling to a mean-pooled retriever yields smaller tokens is determined by whether the uncertainty or subsequent but consistent regressions, indicating that better static context attention influence exceeds specific predefined thresholds, aggregation still matters even when inference is enabled. Figure 4 presents the performance of the DeepSeek-CoderOverall, weighted retrieval sharpens what enters the prompt 1.3B-Base model across a variety of these hyperparameter before and between corrections, while inference closes residual configurations. Our analysis shows that the model achieves peak alignment gaps along the trajectory; neither substitutes fully performance across the evaluation metrics when Subsequent for the other. Attention Influence is set to 0.05 and Uncertainty to 0.8. This configuration is identified as the optimal experimental setting RQ2 Summary: Each labeling signal and each runtime among all evaluated candidates, yielding Pass@1 of 20.00%, stage is non-redundant: rules shift in relative priority across Pass@3 of 28.70%, and Pass@5 of 33.48%. Consequently, we benchmarks, and retrieval weighting complements sparse adopt this specific configuration as the standard setting for all dynamic correction rather than replacing it (Tables III–IV). experiments in our study.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

TABLE III A BLATION ON TOKEN - LEVEL LABELING RULES (R EPO E XEC AND C ODER E VAL , DSC ODER -1B). E ACH ROW REMOVES ONE LABELING CRITERION FROM S ECTION III-B2. S UPERSCRIPTS GIVE THE RELATIVE DECREASE VERSUS FULL ACT O R. RepoExec

Method Pass@1 ACT O R w/o Token Mismatch w/o Uncertainty w/o Succ. Attn. Influence

Pass@3

CoderEval Pass@5

Pass@1

Pass@3

Pass@5

11.27

19.24

23.10

20.00

28.70

33.48

8.96 ↓20.5% 10.70 ↓5.1% 8.51 ↓24.5%

15.63 ↓18.8% 18.76 ↓2.5% 14.99 ↓22.1%

19.15 ↓17.1% 21.97 ↓4.9% 18.03 ↓22.0%

12.43 ↓37.9% 16.09 ↓19.6% 14.09 ↓29.6%

20.48 ↓28.6% 24.87 ↓13.3% 21.83 ↓23.9%

24.35 ↓27.3% 28.70 ↓14.3% 25.65 ↓23.4%

TABLE IV A BLATION ON INFERENCE AND RETRIEVAL MODULES (R EPO E XEC AND C ODER E VAL , DSC ODER -1B). S UPERSCRIPTS GIVE THE RELATIVE DECREASE VERSUS FULL ACT O R. RepoExec

Method

CoderEval

Pass@1

Pass@3

Pass@5

Pass@1

Pass@3

Pass@5

11.27

19.24

23.10

20.00

28.70

33.48

w/o Critical Token Inference

10.42 ↓7.5%

18.31 ↓5.1%

22.25 ↓3.7%

16.96 ↓15.2%

24.78 ↓15.8%

30.00 ↓10.4%

w/o Weighted Retriever

11.04 ↓2.1%

18.28 ↓5.3%

22.54 ↓2.4%

18.26 ↓8.7%

26.52 ↓7.6%

30.43 ↓9.1%

ACT O R

Furthermore, the performance fluctuations across the entire experimental grid are relatively marginal, which supports the claim that the directional trends of the ACT O R framework remain remarkably stable. Across all 25 tested configurations, the performance variance (≈ 2.83 for Pass@1, ≈ 3.42 for Pass@3, and ≈ 2.65 for Pass@5) and standard deviation (≈ 1.68 for Pass@1, ≈ 1.85 for Pass@3, and ≈ 1.63 for Pass@5) remain low, indicating that the method is insensitive to specific hyperparameter choices. This stability underscores the reliability of our dynamic, token-level context refreshing mechanism, demonstrating that ACT O R consistently provides the precise information the model needs, even with varying threshold settings. RQ3 Summary: ACT O R exhibits exceptional robustness to hyperparameter variations, maintaining stable and superior code generation performance with minimal variance across all tested threshold configurations.

D. RQ4: Composition and Syntax of Critical Tokens 1) Critical Token Composition Analysis: We characterize the composition of critical tokens by analyzing their distribution and overlap across different models. As illustrated in Figure 5, critical tokens constitute a minimal fraction of the total, ranging from just 5.08% to 10.62%, with the vast majority being non-critical. A closer examination reveals a consistent overlap between mismatch tokens and high-uncertainty tokens, empirically validating that models are more prone to error when generating uncertain tokens. In contrast, the overlap between high subsequent attention influence and the other two criteria is considerably lower. This suggests that models tend to focus their attention on tokens they are confident about, rather than on those they are likely to mispredict. The proportion of tokens exhibiting all three characteristics is exceptionally low (e.g., just 0.02% in CodeLlama 7B), indicating that models

DSCoder 1.3B (10.62%)

DSCoder 6.7B (8.92%)

CodeLlama 7B (5.08%)

CodeLlama 13B (5.55%)

Fig. 5. Composition and overlap of critical tokens across four models. The figure illustrates the percentage breakdown for tokens classified as Mismatch Tokens (■), High Subsequent Attention Influence Tokens (■), and High Uncertainty Tokens(■). The percentage in each subplot’s title represents the total proportion of critical tokens for that model.

are generally efficient at avoiding deep focus on intractable tokens. Furthermore, by analyzing these trends by parameter scale, Figure 6 reveals that different model series exhibit distinct behaviors. In the DSCoder series, increasing the parameter count from 1B to 7B is associated with a significant decrease in both Token Mismatch (3.29% to 2.43%) and Uncertainty (5.26% to 2.56%), suggesting that larger model scale directly enhances predictive accuracy. Conversely, the CodeLlama series shows broadly stable rates for these metrics as parameter size increases from 7B to 13B, which may indicate that performance improvements have begun to plateau or are occurring elsewhere. Despite these differences, both model series share a common pattern: as parameter counts increase, the models’ “Subsequent Attention Influence” on specific tokens tends to increase. This refined ability to identify and focus on essential information may be one of the core reasons for the improved code generation performance observed in larger-scale models. 2) Critical Token Syntactic Analysis: To further explore the nature of critical tokens, we conduct a detailed analysis of their syntactic characteristics. As illustrated in Figures 7 and 8, critical tokens exhibit significant syntactic diversity and a clear concentration of model attention on specific token types. For these analyses, we filter the dedent and indent syntax types.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

5.26 5

5.39

4.88

100

Mismatch Token High Uncertainty Token High Subsequent Attention Influence Token

Syntax Type identifier keyword delimiter operator number_literal string_literal whitespace other

3.29 3

2.43 2.57

2.76

2.69 2.56

1.64 0.81

1 0

2.48

DSCoder 1.3B

DSCoder 6.7B

CodeLlama 7B

CodeLlama 13B

Fig. 6. Comparative Analysis of Critical Token Prevalence Across Different Models

Figure 7 shows the absolute percentage distribution of syntax types within each critical token condition. A key observation is that identifiers and keywords are the predominant syntax types in most error-related conditions, such as “Mismatch Only” and “Mismatch + Uncertainty”. This suggests that the model’s mistakes and uncertainties most frequently occur when generating variable names and language keywords. In contrast, the “Attention Only” condition has a much more balanced composition, with a high proportion of delimiters and operators, indicating that the model focuses its attention not just on core logic but also on structural elements of the code. Figure 8, which displays the relative distribution, provides deeper insights into where the models’ biases lie. The heatmap highlights which syntax types are disproportionately likely to be critical compared to their normal occurrence rate. For example, keywords are exceptionally over-represented in the “Mismatch + Uncertainty” and “Mismatch Only” conditions (7.64x and 6.05x the baseline, respectively). This indicates that while identifiers are more numerous, keywords are a frequent source of combined error and uncertainty. Similarly, operators are highly over-represented in the “Attention Only” condition (1.86x the baseline), confirming that the model pays special attention to them. In summary, while critical tokens are syntactically diverse, they are not random. Model errors (Mismatch) and confusion (Uncertainty) are heavily concentrated in keywords, while model focus (Attention) is disproportionately directed toward structural operators and delimiters. RQ4 Summary: (1) Composition. Code-LLMs focus attention on tokens they are confident in, not on errorprone ones. As they scale, their ability to attend to essential information becomes more refined, and this improved focus, rather than fewer raw errors, drives better code generation. (2) Syntax. Errors and uncertainty concentrate in critical tokens, mainly keywords, which are more likely to be mispredicted than expected. In contrast, the Code-LLM’s attention centers on structural elements such as delimiters and operators. VI. C ASE S TUDY Figure 9 illustrates a case study of various retrieval strategies in implementing a handler registration decorator, highlighting

Percentage (%)

80 4

Only inty Only ion Only ertainty ttention ttention nditions c t o A A atch a Atten atch + Un match + rtainty + ll Three C Mism Uncert A Mis Unce Mism

Fig. 7. Absolute Percentage Distribution of Syntax Types in critical tokens, using DSCoder-1B model. Mismatch Only 0.37

1.08 6.05 0.38 0.36 0.95 0.38 0.05

Uncertainty Only 0.21

0.95 3.97 0.15 1.48 0.44 0.35 2.63

Attention Only 1.75

0.61 0.70 0.26 1.86 1.64 0.97 1.04

Mismatch + Uncertainty 0.16

1.03 7.64 0.14 1.05 0.26 0.28 0.03

Mismatch + Attention 0.31

1.16 5.48 0.17 0.19 1.44

Uncertainty + Attention 0.46

0.79 3.08

Relative Proportion (x times baseline)

6 5 4 3 2 1

1.91 0.95 0.60 2.79

ord iter eral ifier ator delim ident keywumber_lit oper n

l r othe ng_literahitespace w stri

Fig. 8. Relative Percentage Distribution of Syntax Types in critical tokens, using DSCoder-1B model. LLM Input

RawRAG

def on(self, hook): ""“ Decorator function to add a new handler to the registry. Args: hook (HookType): Hook attribute for which to register the handler.

def decorator(handler): if hook not in self.listeners.keys(): self.listeners[hook] = [handler] else: self.listeners[hook].append(handler) return handler return decorator

Returns: callable: Decorator for registering listeners for the specified hook. """

Ground Truth def decorator(handler): self.registry[hook].append(handler) return handler return decorator\n

ACToR def decorator(coroutine): self.registry[hook].append(coroutine) return coroutine return decorator\n

RLCoder def decorator(func): self.hooks[hook].register(func) return func return decorator

RepoCoder def decorator(handler): if not hasattr(self, hook): setattr(self, hook, []) getattr(self, hook).append(handler) return handler return decorator

Fig. 9. Case Study of Critical Token Across Different Methods

ACT O R’s ability to generate functionally accurate code through dynamic critical token identification. When implementing the on(self, hook) decorator, the model must accurately identify the repository-defined attribute used for handler registration. Traditional RawRAG fails due to the limitations of static retrieval, leading to “hallucinations” of generic variable

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

names like self.listeners. Methods such as RLCoder and RepoCoder also struggle with coarse retrieval granularity or lack dynamic adjustments, resulting in incorrect predictions, such as self.hooks, or overly complex and redundant logic. The ACT O R framework addresses this by identifying “critical tokens” during generation, triggering a precise, dynamic secondary retrieval. This on-demand mechanism allows ACT O R to retrieve the correct attribute, self.registry, in realtime. Consequently, ACT O R generates concise code that aligns perfectly with the Ground Truth, as shown in the comparison below, demonstrating its superiority in resolving deep repository dependencies that static methods often overlook.

progress, repository-level code generation aims to produce implementations that remain consistent with an existing project rather than a standalone snippet [7], [8], [55]–[60]. A. Repository-Level Code Generation

Repository-level code generation requires models to produce code consistent with the real-world repositories [31]. This task requires models to thoroughly understand the repository’s contents and generate accurate implementations that meet the repository’s environment and requirements. Existing work can be grouped into two main categories: retrieval-augmented methods that supply more useful context to the generator, and approaches that leverage compilation or build information for VII. T HREATS TO VALIDITY post-hoc validation and validation-driven re-retrieval. a) Internal Threats: First, due to computational resource High-quality context helps models accurately capture reposconstraints, we evaluate a limited selection of LLMs. However, itory semantics, thereby improving generation quality. Prior we choose representative models of varying scales, and the studies focus on more effective context acquisition, exploring consistent performance gains across these models suggest both retrieval-based and static-analysis-based approaches. In that ACT O R’s effectiveness is not model-specific. Second, to retrieval-based methods, RepoCoder [10] proposes an iterative address the inherent stochasticity of LLMs, we utilize Pass@k pipeline that strengthens each retrieval and generation round ussampling strategies to mitigate the influence of sampling ing the previous generation. RLcoder [9] fine-tunes the retriever stochasticity. Third, whether a token is truly critical depends on using feedback from the generator and introduces a stop signal its downstream impact on the final generated program, which strategy to discard low-relevance contexts. RepoMinCoder [61] cannot be directly observed when the token is generated. We filters and reorders retrieved context based on information loss, therefore approximate critical-token labels using observable preserving salient information while suppressing retrieval noise. proxy signals, including token mismatch, uncertainty, and In static-analysis-driven approaches, RLPG [14] manually subsequent attention influence. Although these proxies may defines ten dependency-based context sources and seven context miss some functionally decisive tokens, we mitigate this types, and trains a classifier to identify the context requirements threat by combining complementary signals, applying teacher of tasks. Cocomic [11] builds a project context graph from forcing to reduce error-propagation effects, and validating their dependency relations and uses graph search to retrieve relevant contribution through ablation studies. code fragments. b) External Threats: A primary concern is the compuAlthough these methods provide valuable contextual support tational overhead introduced by the iterative context-updating for generation, they are generally treated as separate from and KV cache recalculation, which may pose challenges for retrieval, overlooking that models may require different contexts deployment in large-scale, real-time systems. To mitigate this, when generating different tokens. To address this, our method we optimize the retrieval process by avoiding redundant KV triggers retrieval when encountering critical tokens, providing cache computations when the retrieved context remains unmore effective, token-specific contextual support for code changed. Our experiments demonstrate that the actual time cost generation. remains within a controllable range for repository-level tasks. We also plan to integrate advanced KV cache optimization techniques [26]–[28] in the future to further enhance efficiency. B. Error Analysis Of Code Generation Another threat is the scope of benchmarks. Currently, our Recent research has increasingly focused on understanding evaluation is focused on RepoExec and CoderEval. Although and mitigating errors in code generated by LLMs. Several these are standard for repository-level code generation, they key studies have established a foundation for this area. For may not capture the full complexity of all real-world software instance, Wang et al. [62]–[66] conduct a thematic analysis to projects. To improve generalizability, we intend to expand our distill a comprehensive taxonomy of code generation errors. evaluation to broader benchmarks in the future [29]. Their study reveals that LLMs often produce non-trivial, multi-line code-generation errors across various locations and VIII. R ELATED W ORK with multiple root causes. Zhang et al. [67] concentrate In recent years, large language models and LLM-based specifically on the phenomenon of hallucinations in repositoryagents have transformed a wide range of software engineering level code generation. They define and categorize three primary tasks [1]–[4], [30], [31]. Representative applications include hallucination categories: task requirement conflicts, factual code generation [5], [6], [32]–[41], code search [21], [42], knowledge conflicts, and project context conflicts, which can [43], code summarization and comment generation [44], [45], be further divided into eight specific types. Liu et al. [68] and issue resolution [29], [46]–[51]. Related efforts further likewise conduct a thematic analysis of LLM-generated code study code review [52], commit message generation [53], to characterize hallucinations and introduce the HalluCode and repository-level code translation [54]. Building on this benchmark for evaluating models’ ability to recognize them.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

Whereas these studies focus on errors evident in final outputs, our work investigates token-level precursors to such failures. Instead of cataloguing the mistakes themselves, we analyze the underlying properties of “critical tokens” through the dimensions of mismatch, uncertainty, and subsequent attention influence. Our primary contribution is a fine-grained analysis that links these token-level properties to specific syntactic categories. In addition, we propose a token-level RAG framework that targets retrieval based on critical tokens to more effectively mitigate these errors. IX. C ONCLUSION In this paper, we introduce the concept of the “critical token”, a specific point in code generation where a single error can trigger catastrophic failures. To mitigate this, we propose a novel framework that performs targeted, on-demand retrieval upon encountering a critical token, leveraging a weighted mechanism to enhance its effectiveness. In addition to this framework, we are the first to systematically quantify and analyze the properties of these tokens. Experimental results demonstrate that our approach achieves state-of-theart performance in repository-level code generation. ACKNOWLEDGMENTS This work is supported by the National Natural Science Foundation of China (Grant No. 92582202, No. 62302534). R EFERENCES [1] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024. [2] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 2023, pp. 31–53. [3] Z. Zheng, K. Ning, Q. Zhong, J. Chen, W. Chen, L. Guo, W. Wang, and Y. Wang, “Towards an understanding of large language models in software engineering tasks,” Empirical Software Engineering, vol. 30, no. 2, p. 50, 2025. [4] Y. Wang, W. Zhong, Y. Huang, E. Shi, M. Yang, J. Chen, H. Li, Y. Ma, Q. Wang, and Z. Zheng, “Agents in software engineering: Survey, landscape, and vision,” Automated Software Engineering, vol. 32, no. 2, p. 70, 2025. [5] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [6] T. Y. Zhuo, V. M. Chien, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul et al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” in The Thirteenth International Conference on Learning Representations. [7] H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12. [8] Y. Xie, A. Xie, D. Sheth, P. Liu, D. Fried, and C. Rose, “Repost: Scalable repository-level coding environment construction with sandbox testing,” arXiv preprint arXiv:2503.07358, 2025. [9] Y. Wang, Y. Wang, D. Guo, J. Chen, R. Zhang, Y. Ma, and Z. Zheng, “Rlcoder: Reinforcement learning for repository-level code completion,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 1140–1152.

[10] F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J.G. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 2471–2484. [11] Y. Ding, Z. Wang, W. U. Ahmad, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang, “Cocomic: Code completion by jointly modeling in-file and cross-file context,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 3433–3445. [12] W. Cheng, Y. Wu, and W. Hu, “Dataflow-guided retrieval augmentation for repository-level code completion,” arXiv preprint arXiv:2405.19782, 2024. [13] D. Wu, W. U. Ahmad, D. Zhang, M. K. Ramanathan, and X. Ma, “Repoformer: Selective retrieval for repository-level code completion,” arXiv preprint arXiv:2403.10059, 2024. [14] D. Shrivastava, H. Larochelle, and D. Tarlow, “Repository-level prompt generation for large language models of code,” in International Conference on Machine Learning. PMLR, 2023, pp. 31 693–31 715. [15] H. Tan, Q. Luo, L. Jiang, Z. Zhan, J. Li, H. Zhang, and Y. Zhang, “Prompt-based code completion via multi-retrieval augmented generation,” ACM Transactions on Software Engineering and Methodology, 2024. [16] N. Le Hai, D. M. Nguyen, and N. D. Bui, “On the impacts of contexts on repository-level code generation,” in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, pp. 1496–1524. [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [18] R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Neural Computation, vol. 1, no. 2, pp. 270–280, 1989. [19] C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948. [20] Y. Li, B. Dong, F. Guerin, and C. Lin, “Compressing context to enhance inference efficiency of large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 6342–6353. [21] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 7212–7225. [22] L. Guo, Y. Wang, E. Shi, W. Zhong, H. Zhang, J. Chen, R. Zhang, Y. Ma, and Z. Zheng, “When to stop? towards efficient code generation in llms with excess token prevention,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1073–1085. [23] K. Deng, J. Liu, H. Zhu, C. Liu, J. Li, J. Wang, P. Zhao, C. Zhang, Y. Wu, X. Yin et al., “R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models,” arXiv preprint arXiv:2406.01359, 2024. [24] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024. [25] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023. [26] Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan et al., “Cachegen: Kv cache compression and streaming for fast large language model serving,” in Proceedings of the ACM SIGCOMM 2024 Conference, 2024, pp. 38–56. [27] J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang, “Cacheblend: Fast large language model serving for rag with cached knowledge fusion,” in Proceedings of the Twentieth European Conference on Computer Systems, 2025, p. 94–109. [Online]. Available: https://doi.org/10.1145/3689031.3696098 [28] Z. He, J. Zhang, S. Luo, J. Xu, Z. Zhang, and D. He, “Let the code llm edit itself when you edit the code,” arXiv preprint arXiv:2407.03157, 2024. [29] L. Guo, W. Tao, R. Jiang, Y. Wang, J. Chen, X. Liu, Y. Ma, M. Mao, H. Zhang, and Z. Zheng, “Omnigirl: A multilingual and multimodal benchmark for github issue resolution,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 24–46, 2025. [30] Z. Zheng, K. Ning, Y. Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen, “A survey of large language models for code: Evolution, benchmarking, and future trends,” arXiv preprint arXiv:2311.10372, 2023.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, MONTH 20XX

[31] Y. Wang, K. Duan, D. Zheng, E. Shi, F. Zhang, Y. Wang, J. Chen, X. Liu, Y. Ma, H. Zhang et al., “Towards an understanding of context utilization in code intelligence,” arXiv preprint arXiv:2504.08734, 2025. [32] B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,” arXiv preprint arXiv:2207.10397, 2022. [33] E. Dehaerne, B. Dey, S. Halder, S. De Gendt, and W. Meert, “Code generation using machine learning: A systematic review,” Ieee Access, vol. 10, pp. 82 434–82 455, 2022. [34] Y. Wang and H. Li, “Code completion by modeling flattened abstract syntax trees as graphs,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 16, 2021, pp. 14 015–14 023. [35] S. Liu, Y. Huang, M. Liu, J. Chen, E. Shi, Y. Ma, H. Zhang, Y. Zhang, and Y. Wang, “Shortcoder: Knowledge-augmented syntax optimization for token-efficient code generation,” 2026. [Online]. Available: https://arxiv.org/abs/2601.09703 [36] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474, 2022. [37] A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, 2020, pp. 1433–1443. [38] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al., “Competitionlevel code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022. [39] R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161, 2023. [40] A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei et al., “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173, 2024. [41] D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022. [42] J. Gong, Y. Wu, L. Liang, Y. Wang, J. Chen, M. Liu, and Z. Zheng, “Cosqa+: Enhancing code search evaluation with a multi-choice benchmark and test-driven agents,” IEEE Transactions on Software Engineering, vol. 52, no. 1, pp. 206–220, 2025. [43] Y. Wang, L. Guo, E. Shi, W. Chen, J. Chen, W. Zhong, M. Wang, H. Li, H. Zhang, Z. Lyu et al., “You augment me: Exploring chatgpt-based data augmentation for semantic code search,” in 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2023, pp. 14–25. [44] Y. Wang, E. Shi, L. Du, X. Yang, Y. Hu, Y. Wang, D. Guo, S. Han, H. Zhang, and D. Zhang, “Context-aware code summarization with multi-relational graph neural network,” Automated Software Engineering, vol. 32, no. 1, p. 19, 2025. [45] H. Guo, X. Chen, Y. Huang, Y. Wang, X. Ding, Z. Zheng, X. Zhou, and H.N. Dai, “Snippet comment generation based on code context expansion,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 1, pp. 1–30, 2023. [46] W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang, and Y. Cheng, “Magis: Llm-based multi-agent framework for github issue resolution,” Advances in Neural Information Processing Systems, vol. 37, pp. 51 963–51 993, 2024. [47] T. Jiang, Y. Wang, X. He, D. Guo, J. Chen, M. Wen, E. Shi, X. Liu, Y. Ma, and G. Li, “Phoenixrepair: Rethinking repair strategy exploration in software agents,” 2026. [Online]. Available: https://arxiv.org/abs/2607.18859 [48] H. Ye, A. Z. Yang, C. Hu, Y. Wang, T. Zhang, and C. Le Goues, “Adverintent-agent: Adversarial reasoning for repair based on inferred program intent,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 1398–1420, 2025. [49] L. Guo, Y. Wang, C. Li, W. Tao, P. Yang, J. Chen, H. Song, D. Tang, and Z. Zheng, “Swe data construction, automatically!” Proceedings of the ACM on Software Engineering, vol. 3, no. FSE, pp. 525–546, 2026. [50] ——, “Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks,” 2026. [Online]. Available: https://arxiv.org/abs/2506.10954 [51] D. Zheng, R. Ye, Y. Wang, Y. Ye, H. Zhang, E. Shi, X. Liu, Y. Ma, J. Yu, and Z. Zheng, “Swe-prime: Fewer trajectories, better performance,” 2026. [Online]. Available: https://arxiv.org/abs/2608.27449 [52] D. Zheng, Y. Wang, X. Wang, K. Duan, H. Zhang, X. Liu, Y. Ma, and Z. Zheng, “From static to dynamic: Benchmarking

real-world code review with mcr-bench,” 2026. [Online]. Available: https://arxiv.org/abs/2608.27442 [53] E. Shi, Y. Wang, W. Tao, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “Race: Retrieval-augmented commit message generation,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 5520–5530. [54] Y. Wang, Y. Wang, S. Wang, D. Guo, J. Chen, J. Grundy, X. Liu, Y. Ma, M. Mao, H. Zhang et al., “Repotransbench: A real-world multilingual benchmark for repository-level code translation,” IEEE Transactions on Software Engineering, 2025. [55] D. Zheng, Y. Wang, E. Shi, R. Zhang, Y. Ma, H. Zhang, and Z. Zheng, “Humanevo: An evolution-aware benchmark for more realistic evaluation of repository-level code generation,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 1372–1384. [56] Y. Wang, B. Zhang, Y. Wang, D. Guo, T. Y. Zhuo, J. Chen, M. Liu, X. Zhang, and Z. Zheng, “Arkrepobench: A repository-level code completion benchmark for harmonyos development,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 19 409–19 429. [57] Y. Wang, Z. Zhang, C. Wang, X. Xu, M. Liu, Y. Wang, J. Chen, and Z. Zheng, “Realsec-bench: A benchmark for evaluating secure code generation in real-world repositories,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 35 866–35 883. [58] W. Gu, J. Chen, Y. Wang, T. Jiang, X. Li, M. Liu, X. Liu, Y. Ma, and Z. Zheng, “What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond,” arXiv preprint arXiv:2503.20589, 2025. [59] T. Jiang, Y. Wang, Y. Wang, D. Guo, E. Shi, Y. Ma, J. Chen, and Z. Zheng, “Aligncoder: Aligning retrieval with target intent for repository-level code completion,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 971–982. [60] Y. Wang, S. Wang, Y. Wang, B. Zhang, D. Guo, J. Chen, and Z. Zheng, “Reporeasoner: Evaluating repository-level code reasoning ability of long-context language models,” Proceedings of the ACM on Software Engineering, vol. 3, no. FSE, pp. 2790–2812, 2026. [61] Y. Li, E. Shi, D. Zheng, K. Duan, J. Chen, and Y. Wang, “Repomincoder: Improving repository-level code generation based on information loss screening,” in Proceedings of the 15th Asia-Pacific Symposium on Internetware, 2024, pp. 229–238. [62] Z. Wang, Z. Zhou, D. Song, Y. Huang, S. Chen, L. Ma, and T. Zhang, “Towards understanding the characteristics of code generation errors made by large language models,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2025, pp. 717–717. [63] M. De Kruijf and K. Sankaralingam, “Idempotent code generation: Implementation, analysis, and evaluation,” in Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). IEEE, 2013, pp. 1–12. [64] Z. Liu, Y. Tang, X. Luo, Y. Zhou, and L. F. Zhang, “No need to lift a finger anymore? assessing the quality of code generation by chatgpt,” IEEE Transactions on Software Engineering, vol. 50, no. 6, pp. 1548– 1584, 2024. [65] D. Song, Z. Zhou, Z. Wang, Y. Huang, S. Chen, B. Kou, L. Ma, and T. Zhang, “An empirical study of code generation errors made by large language models,” in 7th Annual Symposium on Machine Programming, 2023. [66] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems, vol. 36, pp. 21 558–21 572, 2023. [67] Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng, “Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 481–503, 2025. [68] F. Liu, Y. Liu, L. Shi, H. Huang, R. Wang, Z. Yang, L. Zhang, Z. Li, and Y. Ma, “Exploring and evaluating hallucinations in llm-powered code generation,” arXiv preprint arXiv:2404.00971, 2024.

Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation

Analyze & explore related research in Conceptio

Related documents