arXiv CSopen access

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions MUSLIM CHOCHLOV, University of Limerick, Ireland MICHAEL ENGLISH, University of Limerick, Ireland JIM BUCKLEY, University of Limerick, Ireland Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While

arXiv:2605.25536v1 [cs.SE] 25 May 2026

reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domainaware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs. CCS Concepts: • Software and its engineering → Maintaining software; • Computing methodologies → Artificial intelligence. Additional Key Words and Phrases: systematic literature review, large language models, software engineering, code generating tasks ACM Reference Format: Muslim Chochlov, Michael English, and Jim Buckley. 2026. A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions. 1, 1 (May 2026), 43 pages. https://doi.org/XXXXXXX.XXXXXXX

1

Introduction

Recently, software engineering (SE) has become increasingly supported by contemporary large language models (LLMs) that can, among other tasks, generate code artifacts from natural language, code context, specifications, or other inputs [41]. This seems to bring efficiencies: for example, Peng et al. report that programmers employing Github Copilot completed the task 55.8% faster than the control group [75]. Other evidence, coming from trials at Microsoft, Accenture, and a Fortune 100 company suggests a 26.08% increase in completed coding tasks when AI is equipped [23]. Likewise, in Authors’ Contact Information: Muslim Chochlov, [email protected], University of Limerick, Limerick, Ireland; Michael English, University of Limerick, Limerick, Ireland, [email protected]; Jim Buckley, University of Limerick, Limerick, Ireland, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM

1

2

Trovato et al.

code competitions requiring code generation, AlphaCode has achieved average rankings in the top 54.3% of evaluations [60]. These results indicate that LLM-based CGTs are not merely incremental improvements over prior techniques, but represent a shift toward broadly applicable, generative AI-driven development support. Yet, despite these promising results the current applicability of LLMs towards code generating tasks (CGTs) (i.e., software engineering tasks whose primary outcome is automatically produced executable source code, such as code generation, test generation, or program repair) that go beyond code snippet generation/code competitions is questionable [41]: particularly, the effects and challenges of applying such LLMs for real-world code generating scenarios are still under-reported. The existing related secondary studies and tertiary reviews are either too broad (e.g. looking at the intersection of machine learning (ML) and SE [54]), are focused only on a subset of code generating tasks (e.g. looking at AI in software testing [11]), or are narrowly focused on cross-task aspects of AI4SE like prompt engineering [90]. This makes it difficult to isolate the state of evidence specifically for LLM-based CGTs. To address this gap we synthesize evidence from existing secondary studies in our tertiary review with the goal of discovering the existing landscape, effects of applying LLM-based CGTs mapped to HELM (Holistic Evaluation of Language Model: framework designed for holistic evaluation of LLMs for natural language tasks[61]) measures and scenarios, existing integration challenges, and possible future directions. We follow Kitchenham’s SLR guidance and SEGRESS reporting [50, 51]. The protocol (search strings, screening rules, extraction forms, and analysis scripts) and the curated dataset are publicly released for transparency and reuse [17]. Our search spans multiple recommended digital libraries [37] and Google Scholar, combines database querying with structured snowballing [103], and incorporates a validated semi-automated screening step using a state-of-the-art LLM in line with emerging evidence-synthesis bestpractice [72, 97]. Reliability safeguards include multi-rater audits (Fleiss’ 𝜅 and Cohen’s 𝜅) and an extractor–checker strategy for qualitative evidence extraction and synthesis with Wilson 95% confidence intervals for the resulting proportions, following Kitchenham’s recommendations for secondary studies in SE. We structure the paper around the following research questions (RQs): (1) RQ1: What is the landscape of secondary studies that report (at least in part) on LLM-based CGTs? Sub-questions: (a) What is the distribution of secondary studies by year and by study type (e.g. mapping studies/systematic reviews)? (b) How many primary studies are reviewed in these secondary studies? (c) In what venues are they published? (d) What is AI and SE domain scopes of these studies? (2) RQ2: What HELM measures of LLM-based CGTs are reported in secondary studies? (3) RQ3: What HELM scenarios of LLM-based CGTs are reported in secondary studies? (4) RQ4: What challenges in applying or integrating LLM-based CGTs into software engineering workflows are identified in secondary studies? (5) RQ5: What future directions for LLM-based CGTs are reported on in existing secondary studies? This paper makes the following contributions: • Tertiary synthesis of LLM-based CGTs. Using a Kitchenham/SEGRESS-compliant protocol, we identify 30 secondary studies (2017–2025), quantify the publication landscape (year, study type, venue), map their AI and SE scopes, map the observable effects to the adopted HELM [61] framework, and map their challenges and future directions against our derived (bottom-up) taxonomies grounded in existing frameworks [50, 61, 101]. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions3 • Novel application of LLM towards literature review. We describe and show how LLM (GPT4o) can be used reliably to improve efficiency of snowballing (foe e.g. recursively reviewing large lists of references and citations) as part of a literature review process, potentially saving human-hours. • Novel adoption of the HELM framework. We describe and show how the HELM framework (originally designed for natural language based LLM holistic evaluation) can be adapted to SE contexts by retaining its multi-measure design and redefining scenarios to reflect SE-specific dimensions (task, programming language, application type/domain), consistent with SWEBOK v4 and ISO/IEC 25010 [30, 61, 101]. This paper is structured as follows: Section 2.1 provides CGT/LLM definitions, describes the employed HELM framework and discusses related work. Section 3 details the methodology (RQs, search, selection, quality/bias assessment, data extraction and synthesis). Section 4 reports results and answers for RQ1–RQ5, including the landscape, HELMaligned effects, scenarios, challenges, and future directions with Section 5 providing discussion of these results. Section 6 discusses threats to validity. Section 7 provides a conclusion to this work. 2

Background and Related Work

2.1

Code Generating Tasks

Existing research usually treats code-producing tasks either in isolation (e.g., focusing on code generation, test generation, automated program repair [47, 100, 112]) or in a broader context of software engineering [41], which leaves the field without a shared definition for/vision of code-generation tasks. Therefore, here we adopt a unifying umbrella definition for scoping these tasks in this work: Code-generating task (CGT) is a software engineering task whose primary output is an automatically generated code artifact written in a programming language that is intended to be compiled or interpreted and executed (such output can include tests as code and repair patches). The input for CGTs varies and can include natural language, existing code, documentation, images, and other types of input [9, 41]. (Tasks that solely produce non-executable text (e.g., inline code comments, docstrings, or documentation) are not considered a CGT.) Scale of code produced by CGTs varies as well and can be as small as line/token level (e.g. in code completion tasks) or as large as package/system level (e.g. in program synthesis task). To the best of our knowledge, there is no finite set of CGTs defined in the literature, although several studies provide attempts at compiling comprehensive lists of software engineering tasks[41, 109]. Based on these lists, augmented by existing LLM-based code-generating literature, a set of CGTs tasks can be constructed. Below we give the commonly used names for these CGTs, their input/output (input → output), and short descriptions as they appear in the literature. • Code generation (natural language/examples/formal specifications → code). Produces new code, ideally consistent with user intent [19]. • Program synthesis (natural language/examples/formal specifications → code). Similar to code generation, but at larger scale [35]. • Code completion (code context → next code lines/blocks). Edit-time, multi-token, but small-scale code generation, conditioned on local code context [19, 89]. • Patch generation as part of program repair / vulnerability repair (code → repaired code). Generates code patches that are intended to pass defined tests or fix reported defects / vulnerabilities [64, 68]. • Test generation (natural language/specifications/code → tests as code). Intends to produce unit / integration / other forms of tests as code [31, 32, 83]. Manuscript submitted to ACM

4

Trovato et al. • Code translation (code → code). Migrates code across languages while preserving functionality [56]. • Refactoring (code → functionality-preserving code). Applies functionality-preserving transformations (e.g. rename / re-structure / move / de-duplicate / optimize) to code [13].

2.2

Large Language Models

Contemporary large language models (LLMs) are commonly defined in terms of their architecture, training state, and their size. In terms of the architecture, these models are commonly built using the “Transformer” architecture, a deep multi-layered neural network, introduced by Vaswani et al [41, 94]. Particularly, this architecture facilitattes training parallelism, allowing models to train faster and to scale with more data and parameters. Also, its self-attention mechanism allows it to better capture relationships between (distant) input tokens and to better understand their context. Common subtypes of this architecture are encoder, decoder and encoder-decoder, where the latter two are predominantly used for generative tasks [100]. Transformer-based models commonly begin with a self-supervised pre-training stage: the model learns general knowledge from large unlabeled corpora of natural or programming language(-s). This is achieved by solving objectives, most commonly next-token prediction with a decoder-only transformer (e.g. GPT), masked-token prediction with an encoder (e.g. BERT), or denoising in an encoder-decoder (T5/BART) [15, 25, 59, 78]. Afterward, models can be adapted by fine-tuning, including alignment methods such as instruction tuning [74] and reinforcement learning from human feedback (RLHF) [71]. The size of these models is usually defined by the number of their parameters. At the moment, there seems to be no formal consensus on a parameter threshold that would separate ’large’ language models from language models. Existing research notes that usage is contextual, with many works informally treating models larger than ∼10B parameters as large, while others emphasize “tens to hundreds of billions” [66, 108]. However, some other research treats models with as little as 100M parameters as large [20, 100]: in the real-world language models span orders of magnitude: BERT (125M), LLaMA (7–70B), GPT-3 (175B), and PaLM (540B) [15, 21, 25, 91]. Therefore, for all practical reasons of this paper, we can define LLMs as Transformer-based (with decoder or encoder-decoder components for generative tasks) pre-trained language models, trained on broad corpora of code with self-supervised objectives and having at least 100M parameters. Subsequently, these models can be adapted by fine-tuning (e.g., instruction tuning/RLHF). 2.3

Adapting the HELM Framework for Software Engineering Contexts

To the best of our knowledge, no existing taxonomy has been tailored specifically for LLM-based CGT evaluation. Therefore, here we adopt and adapt the Holistic Evaluation of Language Models (HELM) framework [61] because it offers a holistic, multi-dimensional evaluation methodology that is increasingly recognized in the AI community towards LLM evaluation, originally targeted at natural language tasks. Our adaptation preserves HELM’s design rationale while aligning its constructs with the factors unique to CGTs. The framework [61] supports evaluation of LLMs through two main components: scenarios and measures. Measures. HELM defines a set of 7 evaluation measures capturing aspects of model’s performance. Table 1 summarizes these measures and shows their CGT-specific interpretation. In applying HELM to code-generating tasks, we interpret its dimensions at the level of CGT usage in software engineering workflows, rather than restricting them exclusively to model-internal properties or final software artifacts. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions5 Table 1. Mapping HELM measures to CGT contexts.

HELM Measure

Summary of Original Definition

Accuracy

Task correctness vs. ground truth

Calibration Robustness

Efficiency

Toxicity Fairness

Bias

Interpretation for CGT Context

Functional correctness of generated code (e.g., tests passed, specification compliance) Confidence vs. correctness alignment Alignment between model confidence and actual correctness of code suggestions Stability under perturbation Stability of model behavior under incomplete, ambiguous, or adversarial prompts during code generation Resource usage and latency Computational efficiency of code generation (latency, cost), and—when deployed—its impact on developer workflow (e.g., IDE responsiveness) Offensive or harmful natural language Generation of insecure code patterns, vulnerabilities, or license-violating fragments Human group related performance dispari- Disparities across developer groups or conties texts (e.g., experience level, programming background) Systematic preference for particular coding Undesirable associations/stereotypes styles, APIs, frameworks, or programming languages

As a result, different HELM dimensions naturally map to different evaluation targets: some primarily reflect properties of the generated code (e.g. Accuracy), others properties of the underlying model behavior (e.g. Robustness), and others the interaction between the model and the development process (e.g. Efficiency). Scenarios. HELM defines a scenario as a triple (task, domain, and language), further dividing domain into (what (e.g. text), who (e.g. author), when (e.g. some point in time)). While we retain the most of HELM’s scenario’s structure, the original “what/who/when” decomposition of domain is tailored to natural language contexts and does not directly extend to code. Likewise, in SE, other factors such as the application domain (e.g., embedded systems, enterprise software, scientific computing) and the application type/system type (e.g., mobile application, web service, safety-critical system) are more descriptive than ’domain’. Our adaptation therefore defines scenarios as follows: • Task: Retained, but as CGTs such as code generation, program repair, vulnerability repair, and test generation, as per the definition above. • Language: Retained, referring to the programming language of generated artifacts (e.g., Python, Java, C++). • Domain: Reinterpreted as a couple of application domain and application type/system type, consistent with SE standards such as SWEBOK v4 [101] and ISO/IEC 25010 [30]. With these adaptations, we preserve the methodological structure of the original framework while extending its applicability to CGTs. 2.4

Related Work

Tertiary systematic literature reviews (SLRs) on the intersection of artificial intelligence (AI) and software engineering (SE) research areas are few. Here, we present two such existing tertiary reviews and highlight their differences with our Manuscript submitted to ACM

6

Trovato et al. Table 2. Positioning relative to closely related tertiary studies. Author

AI scope

SE scope

Amalfitano et al. [11] Kotti et al. [54] Our work

AI ML LLM

Software Testing Software Engineering CGTs

Years 1995 - 2022 2009 - 2022 2017 - 2025

work in Table 9. Amalfitano et al. synthesize secondary studies at the intersection of AI and software testing (ST) with studies collected starting from 1995 and going to 2022, drawing on established taxonomies from both areas (AI/ST) to map 20 reviews and to characterize popular applications and research gaps [11]. Independently, Kotti et al. aggregate 83 secondary studies on machine learning (ML) for SE, starting from 2009 and going to 2022, reporting that quality and testing dominate in terms of the SE area and outlining challenges such as stronger empirical validation, better data pipelines, and more industrial studies [54]. Our work significantly differs from these studies, especially in AI scope, SE scope, and the years for which publications were analyzed. On the AI scope, we narrow from AI/ML in general to LLMs. On the SE scope, we move from the entirety of SE (or the single Knowledge Area of software testing, as defined in SWEBOK v4 [101]) to focus on a set of CGTs (somewhat orthogonal to a Knowledge Area) whose primary outputs are intended as compilable/executable/ interpretable code artifacts (code generation, completion, program repair, test generation, translation, refactoring - see Section 2.1). Finally, in terms of years for which studies were collected, we look at the period (2017 - 2025) when contemporary LLM-based approaches started to appear en-masse as opposed to older approaches reviewed by Amalfitano and Kotti (see Table 2). This tighter focus enables us to concentrate on contemporary LLM-based CGTs and to synthesize data for these approaches that prior broad studies could not emphasize. 3

Systematic Review Methodology

To conduct a tertiary SLR, we adopt Kitchenham’s systematic review methodology (for software engineering) [50] and their later SEGRESS guidelines (for structured, transparent reporting) [51]. In line with these guidelines we make an SLR protocol and data collection records publicly available for transparency and reproducibility [17]. 3.1

Research Questions

Following Kitchenham’s SLR methodology, the research questions (RQs) are formulated as: (1) RQ1: What is the landscape of secondary studies that report (at least in part) on LLM-based CGTs? Sub-questions: (a) What is the distribution of secondary studies by year and by study type (e.g. mapping studies/systematic reviews)? (b) How many primary studies are reviewed in these secondary studies? (c) In what venues are they published? (d) What is AI and SE domain scopes of these studies? (2) RQ2: What HELM measures of LLM-based CGTs are reported in secondary studies? (3) RQ3: What HELM scenarios of LLM-based CGTs are reported in secondary studies? (4) RQ4: What challenges in applying or integrating LLM-based CGTs into software engineering workflows are identified in secondary studies? (5) RQ5: What future directions for LLM-based CGTs are reported on in existing secondary studies? Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions7 3.2

Search Strategy

The search process has the following major steps: 0 (Preliminary) Search string construction; 1 Search in digital databases and screening of relevant papers to produce a set of ’seed’ papers; 2 Backward and forward snowballing, starting with ’seed’ papers, following screening of newly found (if any) papers; 3 Full-text screening of papers from step 2 and 3 to identify the final set of secondary studies for review. The process is presented in more detail for steps 1, 2, and 3 (omitting the preliminary step that is discussed separately) in Figure 1 that follows a PRISMA-like [67] format. 3.3

Search String Construction

Constructing queries for searching in digital publication databases can be challenging. Several approaches like QuasiGold Standard (QGS) were proposed to construct such queries in a rigid un-biased manner [49]. QGS can be sensitive to the proper selection of seed publication venues and so here, we used a different approach that employed Google Scholar (GS) to find relevant seed papers. We extracted keywords from these papers for query construction, and then tested the constructed query on a test set of papers (the protocol for this can be found in [17]). Initially, we searched GS using a trial query constructed from domain-relevant concepts that were agreed by all the authors of this paper. GS was selected due to its ability to fetch significant amount of relevant papers from multiple sources [104]. The trial query was constructed drawing from PICOC-style (Population, Intervention, Comparison, Outcome, and Context) query format proposed by Petersen et al. [77] and this application is shown below in its abstract form (Comparison and Outcome not being so relevant for mapping studies): Population: ("code generation" OR "code completion") Intervention: AND ("LLM" OR "LCM" OR "large language model" OR "large code model") Context: AND ("systematic review" OR "meta-analysis" OR "survey") Following this, we looked at the top 20 results returned by GS after running the trial query. All three authors independently screened these top 20 results, looking at abstracts/titles and, if needed, full-text. Each author was tasked with: • Identifying if a paper is relevant based on the inclusion/exclusion criteria provided in Section 3.4. If it did not meet the criteria, the reviewer was asked to specify which inclusion or exclusion criteria were not satisfied (e.g. IC1 or EC3, - see Section 3.4). • Suggesting additional keywords based on the title/abstract/keywords only to expand the initial trial query. They were asked to structure them into these 3 categories of our PICOC type query: Population, Intervention, and Context. All the authors have discussed their set of reviewed papers reaching agreement on 18/20 papers (where 12/18 were found relevant and 6/18 were found irrelevant). The inter-rater agreement was calculated using Fleiss’ kappa [29] (suitable when more than 2 raters are involved) and was 0.85, signaling almost perfect agreement. Next, we randomly split the collected relevant papers into a training set (10/12, used for query construction via aggregated suggested keywords) and a testing set (2/12, held out for validation) using an automated script. Given the Manuscript submitted to ACM

8

Trovato et al.

Search in databases (step 1)

PRISMA-style Paper Identification Flow (with Snowballing)

Records identified via databases: ACM (39), IEEE (219), Scopus (412), WoS (145), DBLP (47) Total: n=862

Additional records from Google Scholar: n=63

Total records before deduplication: n=925

Duplicates removed: n=242

Records screened (title/abstract): n=683

Records excluded: n=646

Full-text review (step 3)

Snowballing (step 2)

Seed papers selected: n=37

Refs/citations from seeds: n=5773

Excluded in deduplication: n=3004

Records screened (title/abstract): n=2769

Excluded in screening: n=2745

Combined seed (37) + snowballing (24) papers: n=61

Duplicates removed: n=1

Full-text eligibility review: n=60

Excluded: - Informal review (28) - Not a secondary study (1) - No evidence for LLM (1) n=30

Final set included in review: n=30

Fig. 1. PRISMA flowchart of the tertiary review process.

small size of the seed set, this split was not intended for statistical performance estimation, but rather as a sanity check to ensure that the constructed query could retrieve relevant studies not explicitly used during query formulation, which is a common practice in query validation for small samples [53]. The final set of query keywords was agreed after discussion by all the authors of the paper. The resultant final abstract search query is shown below (when searching in databases this query was adapted to best leverage the databases’ query format, but preserving semantics): Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions9

Population: ("code generation" OR "code completion" OR "software development" OR "coding process" OR "programming code process" OR "code generation and review" OR "software development lifecycle" OR "SE tasks" OR "program repair" OR "coding" OR "debugging" OR "testing" OR "programming" OR "code development" OR "SDLC" OR "LLM4SE" OR "generating codes" OR "ai-assisted code generation" OR "generated code") Intervention: AND ("LLM" OR "LCM" OR "large language model" OR "large code model" OR "transformer" OR "Encoder-Decoder") Context: AND ("systematic review" OR "meta-analysis" OR "survey" OR "Systematic Literature Review" OR "Systematic Mapping Study" OR "structured methodology" OR "SLR" OR "Mapping Study" OR "literature review")

The query was formulated at the level of broader software engineering and code-related activities rather than enumerating individual code-generating tasks. While specific CGTs such as refactoring or code translation are included in our operational definition, pilot searches and prior secondary studies indicate that such tasks are rarely used as primary indexing terms in review titles or abstracts, and are instead discussed under broader categories such as code generation, program repair, or software development. Consequently, we prioritized concept-level terms to maximize recall and reduce sensitivity to heterogeneous task nomenclature. Relevant secondary studies addressing refactoring or translation were subsequently captured during screening and snowballing. Subsequent validation of this query on the testing set, showed 100% recall.

3.4

Inclusion/Exclusion Criteria

For the goals of this SLR, we will use these inclusion criteria (IC) when selecting studies: (IC1) Secondary studies: SLRs, systematic mapping studies (SMSs), literature surveys, and meta-analyses [50]; (IC2) Studies that focus on, or include, CGTs implemented using large language models (LLMs), as defined in Section 2.1; (IC3) Studies published in peer-reviewed venues and in arxiv (the quality of the latter to be assessed in quality assurance) starting from 2017, inclusive; (IC4) Studies written in English language. We will use these exclusion criteria (EC) to reject studies: (EC1) Primary studies that do not summarize existing research; (EC2) Studies focusing on applying LLMs in software engineering outside of CGTs (e.g., code summarization, code comments, commit messages). (EC3) Grey literature, except arXiv (we include arXiv due to fast pacing in this research area where many articles appear on arXiv first and the majority is later published in peer-reviewed venues[62]); (EC4) Studies focusing on small scale snippet-oriented LLM generated code for teaching or research activities; (EC5) Studies focusing purely on theoretical AI without software applications or on outdated pre-Transformer models (even if these are large); (EC6) Unavailable. Manuscript submitted to ACM

10

Trovato et al.

3.5

Search Databases

The automated search was conducted in the following digital libraries, as suggested for systematic literature reviews [37] and commonly used in software engineering secondary studies: • IEEE Xplore [4] • ACM Digital Library [1] • Scopus/ScienceDirect [6, 7] • DBLP [2] • Web of Science (WoS) [8] • Google Scholar (GS) (as a search engine for additional grey literature like arxiv) [3] 3.6

Study Selection Process

3.6.1 Search in digital databases and screening of seed papers. The search was conducted by the first author of this paper (MC) in the digital libraries (see Section 3.5), adjusting the search string as required for each database. A total of 862 records was returned (excluding those returned in GS), as can be seen in Figure 1. Given that GS retrieves results using a proprietary relevance-based ranking algorithm and does not support reproducible boolean querying or bulk export, we applied a structured yet practical approach to include it in our review protocol. Following established best practices [37, 40], we screened results in descending order of relevance. Screening was conducted in fixed blocks of 50 results, with a stop criterion defined as follows: • Stop screening if the inclusion rate falls below 5% across two consecutive 50-result blocks. • Impose a hard cap of 300 results per query. This strategy aligns with literature suggesting that GS’s precision sharply declines beyond the first 200–300 results, and ensures a balance between sensitivity and feasibility [14, 98]. A total of 63 records were returned and therefore, a total of 925 records (862 from digital databases and 63 from GS) were collected in June 2025. After removing duplicates, 683 were retained for screening. Following this, the first author (MC) screened 683 records based on their titles and abstracts, and full-text only if needed (if a yes/no decision could not be made on the title/abstract alone). The author marked the paper as yes/no and, if no, specified the IC/EC violated. To assess the reliability of our inclusion/exclusion decisions and to ensure transparency and lack of bias in the screening process, we conducted an inter-rater agreement analysis using Fleiss’ Kappa [29]. From the total screening pool of 683 records, we selected a random sample of 30 papers, comprising an equal distribution of 15 included and 15 excluded items (as marked by MC). The two other authors (JB and ME) had to mark this anonymously as yes/no. This balanced sampling design helps mitigate prevalence bias, which can distort kappa estimates in imbalanced datasets [28, 65]. Methodological sources suggest that a sample size of 30 items is sufficient for reliable estimation of Fleiss’ Kappa, particularly when three or more raters are involved [16, 39]. We therefore consider this a sufficiently methodologically-sound compromise between statistical rigor and reviewer workload. Fleiss’ Kappa yielded a value of 0.86, which indicates almost perfect agreement among reviewers [57]. Ultimately, 37 ’seed’ papers were retained after this step. 3.6.2 Backward and forward Snowballing. Snowballing in a literature search is a method used to identify additional relevant papers by exploring the references and citations of a key paper or set of papers (37 ’seed’ papers in our case) [103]. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 11 In particular, we employed an iterative snowballing process, following the structured guidelines of Wohlin [103]. To preempt excessive effort during iterations with diminishing returns, we monitored the inclusion rate across snowballing rounds. Specifically, if an iteration yielded fewer than 5% newly included studies relative to the number of screened references, the process would be terminated early. This threshold is motivated by practical screening guidelines in systematic review literature [37], which note that the marginal value of additional screening falls sharply below this level. While these studies do not explicitly define 5% as a stopping point, they acknowledge that precision commonly drops below this threshold in later phases, indicating low yield and high effort. Therefore, we adopt 5% as a cut-off threshold. In our study, the yield-based stopping criteria was met after the first iteration where only 24 of 2769 records (0.9%) screened were deemed relevant see Figure 1. To manage the screening of 2,769 candidate publications after duplicates were removed, we adopted a hybrid screening approach that combined GPT-4o [5], a novel large language model, with manual validation. This is in-line with recommendations in the evidence synthesis literature on semi-automated screening [72, 97], where the AI system is used as a first-pass screener combined with human evaluation of AI decisions. The model was prompted using our inclusion and exclusion criteria (see Section 3.4) and tasked with classifying all records based on their titles/abstracts into “Include”, “Uncertain”, or “Exclude” categories (see the supplementary material for the prompts [17]). To evaluate the AI system’s reliability, the first author (MC) conducted a validation study on a sample of 100 records drawn from the full corpus of 2,769. The size of a sample aligns with best practices in the literature where 100-200 sample size is commonly used to validate human-machine decisions [93, 96]. It is also consistent with recommendations for Cohen’s kappa estimation (inter-rater agreement), where over 50 samples are suggested [87]. The sample comprised all 31 records that the AI system flagged as relevant (“Include” only as there were no "Uncertain"), together with 69 randomly selected records that the system excluded (“Exclude”). The first author independently screened this validation set. Among the 31 AI-flagged records, 24 were confirmed relevant (precision = 0.77; Wilson 95% CI (confidence interval): 0.60–0.89). All 24 human-identified relevant records were captured by the AI (recall = 1.00; Wilson 95% CI: 0.86–1.00). In the 69 randomly sampled records marked by AI system as "Exclude", none were found relevant (0/69; one-sided exact 95% upper bound on prevalence = 4.25%). Agreement between human and AI screening decisions was almost perfect (Cohen’s kappa = 0.83). 3.6.3 Selection for full-text review. The final set of 37 ’seed’ papers and 24 papers identified during the snowballing, resulted in 61 in total, and they were selected for full-text review. During this review by the first author (MC): • 28 were removed because they lacked a systematic approach to literature review and were informal reviews; • one was a duplicate; • one was found to not be a secondary study, upon closer inspection; • one did not provide any evidence that LLMs were reviewed as part of deep learning. A final, total set of 30 papers was selected for our tertiary study (see Figure 1). 3.7

Quality Assessment

Each selected study was evaluated by the first author using the DARE (Database of Abstracts of Reviews of Effects) criteria, following the questions and scoring criteria proposed by Kitchenham et al. [52]. There are 4 questions, where each question can score either 1/0.5/0 (for exact questions and scoring see Section 2.4 in Kitchenham et al.[52]. We also Manuscript submitted to ACM

12

Trovato et al. Table 3. Quality assessment scores

S

Authors

Q1

Q2

Q3

Q4

Total

Quality

Reference

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21 S22 S23 S24 S25 S26 S27 S28 S29 S30

Vitale et al. Dong et al. Liu et al. Zheng et al. Zhang et al. Gorissen et al. Huang et al. Jiang et al. Zhang et al. Joel et al. Ramirez-Rueda et al. Zhang et al. Hu et al. Dehaerne et al. Huangzhao et al. Wan et al. Lee et al. Zhou et al. Husein et al. Hou et al. Görmez et al. Utomo et al. She et al. Tony et al. Wang et al. Ahmed et al. Ramírez et al. Sasaki et al. Zubair et al. Zheng et al.

1 0.5 0 0.5 0 0.5 0.5 1 0.5 1 0.5 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 0 1 0.5

1 0.5 0.5 0.5 0.5 0.5 0.5 1 0.5 1 1 0.5 1 0 0 0.5 0 1 1 1 0.5 0.5 1 0.5 1 1 1 0.5 1 1

0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0 0 1 0 1 0 0 0 1 0

1 0.5 0.5 0.5 0.5 0.5 1 1 1 1 1 0.5 1 1 0 0.5 0.5 1 1 1 0.5 1 0.5 1 1 1 1 1 1 0.5

3 1.5 1 1.5 1 1.5 2 4 2 3 3.5 2 3 2 0 1 1.5 4 4 4 2 2.5 3.5 2.5 4 3 3 1.5 4 2

High Low Low Low Low Low Medium High Medium High High Medium High Medium Low Low Low High High High Medium Medium High Medium High High High Low High Medium

[95] [27] [63] [109] [105] [33] [43] [47] [107] [48] [80] [106] [42] [24] [44] [99] [58] [111] [46] [41] [34] [92] [86] [90] [100] [9] [79] [82] [112] [110]

include these in our protocol in supplementary material [17]). Reliability was assessed by the other two authors using the same methodology and reviewing 5 different papers each. The resultant quality assessment (QA) scores can be seen in Table 3 where S is a paper identifier, followed by ’Authors’, Q is a question with a score assigned, followed by ’Total’, ’Quality’ is a paper placement into quality bands based n its score, and followed by reference to this paper. The average quality score is 2.45 comparable to other tertiary reviews such as Kitchenham’s et al. [52] where average scores of 2-3 are reported. The quality was assessed to guide synthesis: we did not remove papers based on their scores. Instead, we retained all papers for descriptive mapping (e.g., trends, taxonomy) and note their quality when synthesizing data and drawing conclusions. For this reason, papers were partitioned into a Low [0–<2], Medium [2–<3], and High [3–4] categories according to their total QA scores. These cut-points align with generic quality conventions (low/moderate/high) [11, 54]. Overall, the distribution of quality categories was Low 9/30 (30.0%), Medium 8/30 (26.7%), High 13/30 (43.3%) with high-medium quality paper prevalence (70%). Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 13 For bias assessment, inter-rater reliability on score overlaps was estimated using quadratic-weighted Cohen’s k for each question score and for the total. Agreement was almost perfect (exceeding 0.81 [57]): for MC–JB on the total (n=5, k=0.91) and for MC–ME on the total (n=5, k=0.82). Category-level (low/medium/high) classifications were identical for all MC–JB overlaps (5/5) and for 4/5 MC–ME overlaps (the remaining case differed by 0.5 on the total), yielding substantial agreement overall. Subsequently, discrepancies were discussed, and several of MC’s assessments were adopted after consensus, reflecting convergence toward a shared understanding of the criteria. Given this calibration and the observed high agreement level, MC was deemed a reliable assessor to complete the remaining QA evaluations independently. 3.8

Data Extraction & Synthesis

Data extraction was completed by the first author (MC) and cross-checked by the two other authors each validating a randomly assigned set of 5 papers.

Table 4. Data extraction fields and their in-paper location Data field name

Description

Cardinality

In-paper data location

Coding

The publication year of the paper The AI scope as reported in the paper Reported SE KA (according to SWEBOK) or CGT The name of the venue where a paper was published The type of the venue where a paper was published Ranking of the venue where the paper was published The number of primary studies reviewed in a paper The type of secondary study as reported by the authors

[1] [1] [0..n] [1] [1] [0-1] [1] [1]

Metadata Title/Abstract Full-text Metadata Metadata External (CORE/Scimago) Full-text Full-text

N Y Y (SE scope) N N N N N

Synthesized effects reported for LLM-based CGTs

[0..n]

Results/Discussion/Findings and similar sections

Y

CGT as reported in the paper Programming language(s) as reported in the paper Application type(s) as reported in the paper Application domain(s) as reported in the paper

[0..n] [0..n] [0..n] [0..n]

Full-text Full-text Full-text Full-text

N N N N

Synthesized challenges reported for LLM-based CGTs

[0..n]

Results/Discussion/ Implications/Guidelines

Y

Future direction(s) as reported in the papers

[0..n]

Conclusion/Future Work/ Limitations/Discussion

Y

RQ1 Year of publication AI scope SE KA/Task Venue name Venue type Venue ranking # primary studies Study type RQ2 HELM measure RQ3 HELM scenario Task Language Application Type Application Domain RQ4 Challenges RQ5 Future directions

3.8.1 Data Extraction Fields. For each included study we extracted the data using the extraction form fields presented in Table 4. The table shows data field names of the data, its description, cardinality, where in the paper this data was looked for, and if it needs further coding (e.g. heterogeneous / qualitative data). The cardinality specifies how many values could be extracted for a data field. For example, "Year of publication" can only have one value [1], "SE KA/Task" can have zero or more [0..𝑛], and "Venue ranking" may or may not have a value [0 − 1]. In-paper data location specifies Manuscript submitted to ACM

14

Trovato et al.

where in the paper we looked for the data. In addition to in-paper data, we extracted supplementary metadata (such as "Year of publication", for example) and external data (such as "Venue ranking"). Specifically for the latter, we retrieved venue rankings from CORE [22] (for conferences) and Scimago [84](for journals). Here values were not coded but directly recorded as provided by the external source. For all non-metadata fields, we have recorded the exact location of the identified data (for example, section, page numbers, figures or tables). AI scope was recorded as provided by the authors and cross-checked with standard definitions from AI/ML/DL/LLM taxonomy derived from Russell and Norvig [81]. (All AI scope extracted data matched those in the taxonomy). The "Study type" was recorded as provided by the authors. Finally, when extracting evidence for "HELM measure", "Challenges", and "Future directions" we followed these rules: • We collected synthesized evidence only: Evidence had to be from across multiple techniques, approaches, or studies, providing multiple references for a claim, and summarizing multiple such sources. • We only collected evidence related to LLMs and CGTs (in case when papers had broader AI/SE scope). • We recorded verbatim exact quotes extracted from the papers in support of the effects, challenges and directions (including their location in the paper, as stated earlier). 3.8.2 Data coding. For items flagged “Y” in the "Coding" column of Table 4, we apply predefined schemes to normalize heterogeneous terminology. AI scope was coded with standard definitions (for AI/ML/DL/LLM) derived from Russell and Norvig [81]. In the majority of cases, each study was assigned a single AI scope label reflecting its primary methodological paradigm. In one case, a study made substantive and inseparable use of both ML-based and non-AI techniques: this study was therefore coded with a hybrid label (ML & Non-AI) rather than forcing an artificial dominance decision. For "SE KA/Task" (coded name "SE scope, see Table 4") SWEBOK v4 KAs were used [101]. The mapping protocol was as follows: (1) First we checked if KAs are explicitly provided (e.g. a paper might structure itself explicitly around KAs) aligning those KAs to SWEBOK v4. For example, some KAs have newer names in SWEBOK v4: e.g. "software development" -> "software construction", "software security" KA now encompasses tasks like vulnerability repair. In this case KAs are reported as they appear in the paper (adjusting to SWEBOK v4, only if needed). (2) If KAs were not explicitly provided, we tried to match software engineering tasks discussed in a paper to KAs, using KA and task definitions: e.g. "vulnerability repair" -> "software security", "bug repair" -> "software construction/software maintenance". (3) In other cases, when a match was not apparent, we recorded a "Cannot map". "HELM measure" data was assigned to HELM measure categories as described in Section 2.3. "Challenges" were coded using the categories shown in Table 5. We derived the challenge categories inductively, iteratively, in a bottom-up manner using thematic synthesis tailored to software engineering and grounding categories in existing taxonomies, where possible (see Table 5). The first author (MC) iteratively performed clustering to consolidate codes into higher-order, non-overlapping categories. Two other authors reviewed these categories over several meetings, and disagreements were reconciled/the categories refined. This choice of thematic synthesis is consistent with established SE guidance for heterogeneous bodies of evidence where meta-analysis is infeasible, provided that extraction forms and procedures are reported [50, 51]. Although ’Tooling & Workflow’ issues may indirectly affect efficiency (e.g., developer latency), we treat them separately from HELM’s efficiency dimension, which we reserve for resource and performance-oriented Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 15 Table 5. Challenges categories, their definitions, and their relation to existing taxonomies.

Category

Definition

Relation to existing taxonomies

Data & Context

Mismatch between available context and task needs (e.g., private APIs, repo-scale grounding, token limits, domain shift, early-phase data sparsity).

HELM: Robustness, Accuracy; ISO/IEC 25010: Compatibility, Reliability;

Tooling & Workflow

Integrating LLMs-based CGTs into existing tools or work- ISO/IEC 25010: Interaction Capability; flows.

Evaluation & Benchmark Validity

Benchmarks/metrics not reflecting practice (ambiguous HELM: Robustness, Bias; tasks, single-metric focus, leakage/imbalance, narrow languages/projects).

People & Process

Human/organizational factors (trust issues, review workload, ownership, missing communication).

Legal & Privacy

Privacy and intellectual property risks.

HELM: Toxicity; ISO/IEC 25010: Security;

Security & Safety

Vulnerabilities in generated code; adversarial fragility, data poisoning.

HELM: Toxicity; ISO/IEC 25010: Security;

Economics

Deployment/runtime constraints (latency/throughput, cost, energy, hardware and software limits).

HELM: Efficiency; ISO/IEC 25010: Performance efficiency;

Model/Training

Limits from pretraining/fine-tuning/continual learning and HELM: Robustness, Accuracy; representations (data scarcity, lack of structural information).

Not provided

For studies that explicitly do not state a challenge: retained for methodological transparency.

Other/Uncategorized

For statements that do not fit well into other categories. Kitchenham [50] Necessary to ensure no force-fitting, maintaining methodological rigor.

constraints captured under the Economics category. ’People & Process issues do not map cleanly to existing technical evaluation frameworks such as HELM or ISO/IEC 25010, reflecting a known gap in current taxonomies with respect to human and organizational factors in LLM-based software engineering. Likewise (following the same procedure as for "Challenges"), "Future directions" were assigned to the categories defined in Table 6. Here, specifically, we distinguish ’Dataset improvement’ and ’Benchmark & evaluation improvement’ analytically: dataset improvement concerns the quality and coverage of data artifacts themselves, whereas benchmark and evaluation improvement focuses on task formulation, metrics, and evaluation protocols applied to those datasets. The final set of papers along with their references and data extraction items with single values (e.g., "Year of publication", "Study type") is reported in the Table 7. Items with potentially multiple values (e.g., "SE KA/Task", "HELM measure") are summarized in aggregated form, with the full per-study data available in the supplementary material [17]. 3.8.3 Data synthesis. Using the coded dataset, we aggregate and analyze data using Kitchenham’s guidelines [50] to answer the RQs: • Descriptive summaries (counts, proportions, timelines) for single-valued items (e.g. "Year of publication", "# primary studies"); • Distributions and cross-tabulations for multi-valued items (e.g., "SE KA/Task" coded as "AI scope"; • Thematic grouping of qualitative data such as "HELM measures", "Challenges", and "Future directions". Manuscript submitted to ACM

16

Trovato et al. Table 6. Future direction categories, their definitions, and their relation to existing taxonomies. Category

Definition

Model improvement & training

Covers future work aimed at improving model architec- Justified by SWEBOK (Software Constructures, training strategies, fine-tuning, performance, ex- tion) for code-level improvements and plainability, and prompting techniques. HELM measures such as accuracy, robustness, efficiency. ISO/IEC 25010 also emphasizes performance efficiency. [30, 61, 101]

Benchmark & evaluation improvement

Includes directions for creating benchmarks, metrics, and multi-level evaluation protocols.

Advancing experimental research

Captures the need for more empirical, large-scale, and re- In line with Amalfitano et al. [11] producible studies to strengthen evidence bases.

Dataset improvement

Refers to improving training/evaluation datasets, including quality, representativeness, coverage, and benchmarks.

Grounded in SWEBOK (Software Construction inputs) and ISO/IEC 25010 (data quality as part of product quality). [30, 101]

Ensembling & hybridiza- Future work directions suggesting combinations of models tion (e.g., ensembles, hybrid systems with symbolic methods).

Justified as a standard AI approach and a common pattern in mature research fields, where complementary techniques are combined to mitigate individual limitations [26].

Focuses on deploying LLMs into real-world SE workflows, adoption in industry, and socio-technical integration.

Supported by SWEBOK (Software Engineering Management, Software Process) and ISO/IEC 25010 (maintainability and portability). [30, 101]

Security, Privacy & Trust- Consolidates concerns about vulnerabilities, adversarial worthiness robustness, privacy, and transparency.

Justified by HELM (robustness, toxicity), and ISO/IEC 25010 (security as a core characteristic). [30, 61]

Integration into practice

3.8.4

Relation to existing taxonomies

Justified by SWEBOK (Software Testing), HELM metrics (accuracy, calibration, robustness), and ISO/IEC 25010 (functional suitability and quality in use). [30, 61, 101]

Not provided

For studies that explicitly do not state a future direction: retained for methodological transparency.

Other/Uncategorized

For statements that do not fit well into other categories. Kitchenham [50] Necessary to ensure no force-fitting, maintaining methodological rigor.

Risk of Bias Assessment.

AI-scope. Each study was assigned an AI scope label (nominal categories: LLM, DL, ML, ML & non-AI, AI). MC (the first author) labeled all studies; JB and ME independently labeled pre-specified overlapping subsets (5 papers each, one shared). After the calibration round, near-complete agreement was achieved on the AI-scope field. MC and JB reached almost perfect agreement (using Gwet’s AC1 for highly unbalanced categories [38]) (Gwet’s AC1 = 1.00), while MC and ME reached substantial agreement (Gwet’s AC1 = 0.76). These results confirm that the main extractor’s coding can be considered reliable for the remaining studies. SE KA/Task. Because each study could receive multiple labels here, we looked at how raters agree on sets of KAs per study. For this we employed Jaccard similarity to measure sets overlap and Krippendorff’s 𝛼 for statistical measure of agreement on sets [55]. Across the overlapping subset of studies, the mean Jaccard similarity between raters was 0.85 (MC–JB) and 0.97 (MC–ME), showing that the raters selected largely identical sets of categories. When assessed using Krippendorff’s 𝛼, the coefficient was 0.875, indicating strong, almost-perfect reliability (𝛼 > 0.80). RQ3: scenario. Likewise, all data attributes representing a scenario (i.e, task, language, application type, and application domain) can all have multiple values, therefore we need to assess how raters agree on these using Jaccard and Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 17 Table 7. Final set of secondary studies with extracted single-valued data and references S

Year of publication

AI scope

Venue name (abbreviation)

Venue type

Venue ranking

# primary studies

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21 S22 S23 S24 S25 S26 S27 S28 S29 S30

2025 2024 2024 2023 2023 2024 2023 2024 2023 2024 2024 2024 2025 2022 2024 2024 2025 2025 2025 2024 2024 2024 2023 2024 2024 2023 2024 2024 2025 2024

LLM LLM LLM LLM DL LLM ML & non-AI LLM LLM LLM AI LLM LLM ML DL DL LLM LLM LLM LLM LLM ML DL LLM LLM DL LLM LLM LLM LLM

TOSEM SEA4DQ TAAI arXiv TOSEM HCSE arXiv arXiv arXiv arXiv PCS arXiv arXiv IEEE Access SCIS ACM CSUR arXiv TOSEM CSI TOSEM EuroSPI ICITCOM TOSEM arXiv TSE EPIA CONISOFT COMPSAC CSI ESE

Journal Workshop Conference Preprint archive Journal Conference Preprint archive Preprint archive Preprint archive Preprint archive Journal Preprint archive Preprint archive Journal Journal Journal Preprint archive Journal Journal Journal Conference Conference Journal Preprint archive Journal Conference Conference Conference Journal Journal

Q1 N/A N/A – Q1 N/A – – – – Q4 – – Q1 Q1 Q1 – Q1 Q1 Q1 B N/A Q1 – Q1 National:Portugal N/A B Q1 Q1

107 28 28 134 112 55 140 235 1009 111 20 110 291 37 142 276 41 58 23 395 7 36 67 13 102 29 15 28 41 123

Study type

QA score

Reference

SLR Survey SLR Review Survey/SLR Survey Survey Survey/SLR Survey Survey/SLR SLR SLR SLR Review Survey Survey/Review Survey/SLR SLR SLR SLR SMS SLR Survey/SLR Systematic investigation/SLR Review Survey SLR SLR SLR Review

3 1.5 1 1.5 1 1.5 2 4 2 3 3.5 2 3 2 0 1 1.5 4 4 4 2 2.5 3.5 2.5 4 3 3 1.5 4 2

[95] [27] [63] [109] [105] [33] [43] [47] [107] [48] [80] [106] [42] [24] [44] [99] [58] [111] [46] [41] [34] [92] [86] [90] [100] [9] [79] [82] [112] [110]

Krippendorff’s 𝛼. Here, the mean Jaccard similarity for task between raters was 0.88 (MC–JB) and 0.92 (MC–ME), suggesting that the raters selected almost identical sets of tasks. When assessed using Krippendorff’s 𝛼, the coefficient was 0.84, indicating almost-perfect reliability (𝛼 > 0.80). For language, Jaccard similarity was 0.8 (MC-JB) and 1.0 (MCME) with Krippendorff’s 𝛼 = 0.75, suggesting very high overlap with substantial reliability. Similarly for application type Jaccard was 0.89 (MC-JB) and 1.0 (MC-ME) with 𝛼 = 0.77 indicating very high overlap and substantial reliability. Finally, raters completely agreed on domain with Jaccard of 1.0 for both raters. Because there was no variability in coding decisions (all raters consistently selected no applicable categories), Krippendorff’s 𝛼 cannot be computed: the measure is undefined in the absence of variance. RQ2, RQ4, and RQ5 extraction reliability. To assess the reliability of our data extraction process, we adopted the extractor–checker strategy recommended by [50], where one researcher (the extractor - MC) identifies evidence quotes and assigns labels, and two independent checkers (JB and ME) validate random samples of the extracted data. Each checker inspected an independent random sample of 20 entries drawn from the full dataset for each RQ2 RQ4 and RQ5, classifying each extraction as ACCEPT (quote and label correct), RELABEL (quote correct, label incorrect), or REJECT (quote incorrect). Based on these checker decisions we defined and computed two reliability metrics. The quote correct rate (QCR) quantifies the proportion of correctly extracted quotes (regardless of label) as QCR =

𝐴 + 𝑅𝐿 𝑅𝑋 =1− , 𝑛 𝑛 Manuscript submitted to ACM

18

Trovato et al.

where 𝐴 is the number of ACCEPT items, 𝑅𝐿 the number of RELABEL items, 𝑅𝑋 the number of REJECT items, and 𝑛 the total reviewed items. The label accuracy given correct quote (LAcc) measures labeling correctness conditional on a valid quote: 𝐴 . 𝐴 + 𝑅𝐿 All proportions were subject to 95% Wilson score confidence intervals (CI) [69], which provide more accurate LAcc =

coverage for smaller samples. Across the three research questions, the extractor demonstrated consistently high reliability (see Table 8). For RQ2, both QCR and LAcc were nearly perfect (QCR = 1.00 [0.94–1.00]; LAcc = 0.98 [0.87–1.00]). For RQ4, performance remained excellent (QCR = 1.00 [0.96–1.00]; LAcc = 0.98 [0.87–1.00]). For RQ5, reliability was still high though slightly lower (QCR = 1.00 [0.95–1.00]; LAcc = 0.95 [0.83–0.99]), suggesting minor increases in label ambiguity rather than extraction errors. All but lower CI bounds exceed 0.80, suggesting almost perfect reliability. These results confirm that the extractor (MC) can be considered highly reliable for the remaining extractions. Table 8. Reliability of data extraction across RQ2,4,5 (95% Wilson CI).

RQ

Metric

Value

CI Low

CI High

RQ2

QCR LAcc QCR LAcc QCR LAcc

1.00 0.98 1.00 0.98 1.00 0.95

0.94 0.87 0.96 0.87 0.95 0.83

1.00 1.00 1.00 1.00 1.00 0.99

RQ4 RQ5

4

Results

In this section we present and discuss the results of data analysis to answer the RQs of this work. 4.1

RQ1: What is the landscape of secondary studies that report (at least in part) on LLM-based CGTs?

4.1.1 RQ1a: What is the distribution of secondary studies by year and by study type (e.g. mapping studies/systematic reviews)? The secondary studies are grouped by the year and by the study type in Figure 2. The earliest study in our set of papers dates to 2022, a review published in IEEE Access (S14, see Table 7). In 2023, 1 review, 3 surveys, and 2 survey/SLR appeared, including several in ACM Transactions on Software Engineering and Methodology (TOSEM) and arXiv, 6 total. The year 2024 marked a notable increase, with 17 studies overall (mostly SLRs and surveys). These studies were widely distributed across venues, including journals such as TOSEM, IEEE Transactions on Software Engineering (TSE), and ACM Computing Surveys (ACM CSUR); conferences and workshops such as the International Workshop on Software Engineering and AI for Data Quality in Cyber-Physical Systems/Internet of Things (SEA4DQ), the International Conference on Technologies and Applications of Artificial Intelligence (TAAI), the International Working Conference on Human-Centered Software Engineering (HCSE), and others; and preprint archives (arXiv). Finally, by the time the studies were collected in June 2025, we identified six new studies for the year of 2025 (5 SLRs and 1 survey/SLR), appearing in TOSEM, Empirical Software Engineering (ESE), and Computer Standards and Interfaces (CSI) as well as arXiv. Given that the year is incomplete, further growth towards the 2024 "high-tide" mark is expected. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 19 Studies per Year by Study Type

14 12 10 8

Study Type

13

Review SLR SMS Survey Survey/Review Survey/SLR Systematic investigation/SLR

6

179

346 276 225 7

632

41

4 1178

520 225

2025

134

2023

0

37

2022

2

2024

Number of Studies

16

Publication Year

Fig. 2. Secondary studies per year by study type

Two noticeable trends can be observed here (see Figure 2). First, the data suggests a clear increase of secondary studies including LLM-based CGTs: from a single study in 2022, through 6 studies in 2023, to 17 in 2024, with a leveling off to 6 in the first 6 months of 2025. There is also an observable change in the balance between more exploratory, classification-oriented studies (reviews and surveys) and more directed, synthesis-oriented studies such as SLRs/SMS over time [50]: 0% purely SLR/SMS in 2022 and 2023, 47% in 2024 (8/17), and 83% in 2025 (5/6). This progression indicates a growing maturity of the research area. 4.1.2 RQ1b: How many primary studies are reviewed in these secondary studies? To answer this RQ, we examined the cumulative number of primary studies (non-unique) reported across the included SLRs and SMSs, as reported in Table 7 and visualized in Figure 2 (with numbers inside the boxes for each study type and year). As can be seen, in 2022, the single study reviewed only 37 primary studies. By 2023, coverage expanded significantly: six review/survey/early SLR papers collectively reviewed 1491 primary studies that year. In 2024, the number of primary studies increased further: particularly 7 SLRs reviewed 632 primary studies, 1 SMS covered 7, with the rest of primary studies reviewed in surveys/reviews and mixed type studies, resulting in a total of 1724 that year. Noticeable trend here is the growth of the reviewed papers in total. By mid-2025 (our data collection cut-off in June), 5 SLRs had reviewed 520 primary studies and the other survey/SLR covered 41, producing 561 total for the current year; a surprising finding considering the more general growth in AI4SE. Surveys, reviews and mixed type papers seemed to consistently review a larger number of primary studies than SLRs (e.g. 1092 total in these study types versus 632 total in SLRs for 2024), reflecting their broader and more exploratory aims. This is only different in 2025, but the data for that year is incomplete. Although the number of secondary studies increased nearly threefold from 2023 to 2024 (from 6 to 17), the total number of primary studies reviewed grew less significantly (from 1,491 to 1,724). This trend could suggest both the maturation of the research area (with less primary studies appearing) and the onset of redundancy across secondary studies, with future growth in secondary research likely to stem more from diversification of scope and questions (more SLRs) than from the discovery of entirely new primary studies (less informal reviews and mapping studies). Importantly, these trends are observed over a relatively Manuscript submitted to ACM

20

Trovato et al. Table 9. Distribution of Studies by Venue and Source Type Venue name/type

Venue ranking

Count

Average QA score

Journal TOSEM CSI ESE IEEE Access PCS SCIS ACM CSUR TSE

Q1 Q1 Q1 Q1 Q4 Q1 Q1 Q1

Total (Journals)

5 2 1 1 1 1 1 1

3.1 4 2 2 3.5 0 1 4

13

2.77

1 1 1 1 1 1 1

1.5 3 3 2 1.5 2.5 1

7

2.07

1

1.5

1

1.5

9

2.39

9

2.39

Conference COMPSAC CONISOFT EPIA EuroSPI HCSE ICITCOM TAAI

B N/A National:Portugal B N/A N/A N/A

Total (Conferences) Workshop SEA4DQ

N/A

Total (Workshop) Preprint archive arXiv

—

Total (Preprint archive)

short time span (approximately 3.5 years) and should therefore be interpreted as early indicators rather than long-term trajectories. 4.1.3 RQ1c: In what venues are they published? As can be seen in Table 9, the majority of peer-reviewed secondary studies have been published in journals (13 out of 30). These include 8 venues such as ACM Transactions on Software Engineering and Methodology (TOSEM) having 5 studies and Computer Standards and Interfaces (CSI) having 2 studies. All other venues had one paper. Most of these journals (7/8) are ranked Q1 in Scimago, suggesting that the topic has gained recognition in high-quality venues. One study was published in PCS, which has Q4 ranking in Scimago, however, the QA score of that study was high (4). Conferences account for 7 studies, spanning diverse venues such as COMPSAC, and HCSE. For 4 of these venues rankings could not be obtained and the remaining 3 had variable rankings (B and national-level). Only one workshop paper was identified, where a ranking was not available and a low QA score was derived (1.5). This is consistent with the more exploratory, early-stage nature of workshop publications. Finally, a substantial number of studies (9 out of 30) appeared on the arXiv preprint server. This reflects the fastmoving nature of the field and a drive for rapid dissemination of survey results. Ranking scores are not available for preprints since they bypass peer review. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 21 Unsurprisingly, the average QA score (2.77) was the highest for studies published in journals, reflecting their top-tier nature and rigorous peer-review. Interestingly though, the second highest average QA score (2.39) was recorded for arXiv preprints. This suggests that, despite the lack of peer-review, some of these studies followed a rigorous, systematic literature review approach. Studies published in conferences had comparable but slightly lower average QA score of 2.07 and the workshop paper had the lowest of 1.5.

Table 10. Mapping of papers to AI domain scope.

AI scope

Count

Papers (S identifiers)

LLM

21

DL ML ML & non-AI AI

5 2 1 1

S1, S2, S3, S4, S6, S8, S9, S10, S12, S13, S17, S18, S19, S20, S21, S24, S25, S27, S28, S29, S30 S5, S15, S16, S23, S26 S14, S22 S7 S11

Table 11. Mapping of papers to SE (SWEBOK) domain scope.

SE Scope

Count

Papers (S identifiers)

Software Construction

25

Software Maintenance Software Security Software Testing Software Design Software Requirements Software Engineering Management Software Quality Software Configuration Management Software Engineering Operations

14 11 7 5 5 3 2 1 1

S1, S2, S3, S4, S5, S6, S7, S8, S9, S10, S11, S12, S13, S14, S15, S16, S19, S20, S21, S22, S24, S25, S26, S29, S30 S1, S4, S5, S7, S9, S12, S13, S14, S16, S20, S21, S25, S29, S30 S1, S4, S5, S7, S9, S12, S13, S18, S20, S27, S30 S2, S4, S9, S13, S20, S21, S25 S2, S9, S13, S20, S21 S2, S9, S13, S20, S21 S9, S20, S21 S13, S20 S30 S13

Cannot Map

3

S17, S23, S28

Table 12. Distribution of secondary studies by number of SE scope topics (SWEBOK KAs) covered, excluding “Cannot Map”. # SE KAs

Number of Papers

1 2 3 4 6 7 8

12 3 5 3 1 1 2

Papers (S identifiers) S3, S6, S8, S10, S11, S15, S18, S19, S22, S24, S26, S27 S14, S16, S29 S1, S5, S7, S12, S25 S2, S4, S30 S21 S9 S13, S20

4.1.4 RQ1d: What is AI and SE domain scopes of these studies? To address this RQ, we used the AI scope of each secondary study and SE scope (coded) where the latter can have multiple KAs (according to SWEBOK [101]). Table 10 Manuscript submitted to ACM

22

Trovato et al.

Intersection of AI Scope (ML domain) and SE Scope (SWEBOK) 0

0

1

0

0

0

0

0

0

0

0

0

0

2

0

0

1

0

5

3

1

10

2

5

9

7

ML

0

0

2

0

0

0

1

0

0

0

0

ML & non-AI

0

0

1

0

0

0

1

0

0

1

0

Tes t

cu rity

ftw are

Se ftw are

So

So

en

ty

uir em

So

ftw are

Re q

Qu ali

e ftw are So

Ma

int

en

rat ftw are So

ee gin

ftw are

En

an c

ion

s

nt me

gO pe

gM

an ag e

De

rin ee gin So

En ftw are So

rin

on

ftw are So

Co

So

ftw are

Ma na

ge

ns tru

cti

me

ap tM nn o

ion

Ca

gu rat nfi Co ftw are So

ing

0

17

ts

4

1

sig n

0

2

nt

1

AI Scope

DL LLM

15.0 12.5 10.0 7.5 5.0 2.5 0.0

Count

AI

SE Scope (SWEBOK) Fig. 3. Intersection of AI scope and SE scope

shows secondary studies grouped by their AI scope, table 11 shows secondary studies grouped by SE scope, table 12 shows the distribution of SE KAs, and Figure 3 shows intersection of AI and SE scope in secondary studies. In terms of AI scope, the majority of secondary studies focus on LLMs specifically (21 out of 30, see Table 10). A smaller number review under broader deep learning (DL) scope (5), general machine learning (ML) scope (2), a combination of ML with non-AI scope (1), or generic AI scope (1). LLMs are reviewed in all of these papers, even under broader scopes where they are treated as one technique among several. The dominance of specifically LLM-oriented reviews (21/30) can indicate that the research community has focused on generative AI, with broader scoped surveys used to capture a broader perspective of research in the area. In terms of SE scope, most studies focus on Software Construction (25 studies, see Table 11). This is to be expected since code generation, for example, and related CGTs most directly contribute to software construction activities. Other frequent scopes include Software Maintenance (14), Software Security (11), and Software Testing (7). Less frequently covered are Software Design (5), Software Requirements (5), and Software Engineering Management (3), which are typically discussed in studies with broad KA scope (in addition to "Software Construction, for example") and may contribute little or nothing to CGTs themselves. Three studies could not be mapped to any SWEBOK KA, and only isolated coverage was found for Software Quality (2), Software Configuration Management (1), and Software Engineering Operations (1). Overall, the SE scope shows a strong concentration on software construction, maintenance, and security with software testing following. In terms of SE scope topic distribution, the majority of secondary studies are narrow in scope: 12 cover only 1 SE KA (see Table 12), a group of 11 papers span 2–4 topics, and only 4 studies have broad scope (6–8 topics). These broader studies typically include higher-level KAs (e.g., requirements, design, or management) in addition to, rather than instead of, construction and maintenance-oriented topics. When considering AI and SE scopes together, several patterns emerge: • LLM-focused studies seem to dominate Software Construction, with 17 LLM-focused studies addressing construction tasks such as code generation and completion (see Figure 3). Ten of these also intersect with Software Maintenance (e.g., program repair) and 9 with Software Security (e.g., vulnerability repair). Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 23 • Likewise, DL-oriented reviews (5 studies) also primarily map to Software Construction, with less frequent overlap in Software Maintenance and Software Security. • ML-scoped reviews (2 studies) are narrower, typically addressing tasks in Software Construction and Software Maintenance. • Again, a similar trend is observed in "ML & non-AI" and "AI" studies (2) spanning Software Construction, Software Maintenance, and Software Security. Taken together, these observations suggest that the dominance of construction, maintenance, and security focused SE scope is largely invariant to AI scope. To sum up, the majority of secondary studies are LLM-focused (21). In terms of SE scope, the major topics covered seem to be "Software Construction" (25), "Software Maintenance" (14), and "Software Security" (11). Half of the studies (15) are narrowly focused (covering 1-2 SE KAs at most) with fewer studies (4) having very broad scope (covering 7-8 SE KAs). The intersection analysis reveals a strong LLM × Software Construction cluster, indicating that generative models are predominantly investigated in tasks directly related to producing code artifacts. Secondary clusters are visible in LLM × Software Security and LLM × Software Maintenance, that could indicate growing interest in applications to vulnerability repair, and long-term code evolution. 4.1.5 Answering RQ1. In this section, we synthesize our RQ1a-RQ1d findings and answer the overarching RQ1: What is the landscape of secondary studies that report (at least in part) on LLM-based CGTs? Answering to RQ1, the landscape of secondary studies on LLM-based CGTs can be characterized as follows: • it has quickly expanded (the number of studies tripled from 2023 to 2024), moving from exploratory studies to more focused systematic studies (0 self-reported SLRs/SMSs in 2022/2023 to 83% of SLRs/SMSs in 2025); • it shows signs of stabilization and redundancy in the number of primary studies (while the number of secondary studies tripled from 2023 to 2024, the number of primary studies they cover increased from 1491 to 1724 only); • it is maturing in venue quality, with substantial presence in Q1 journals (12/30 studies); • it is dominated by LLMs-focused studies and Software Construction, with secondary KAs in Software Maintenance and Software Security; • it remains narrow in SE scope for most studies (15/30), with only a handful of reviews providing broad, integrative coverage (4/30).

4.2

RQ2: What HELM measures of LLM-based CGTs are reported in secondary studies?

To answer RQ2, we have extracted HELM measures" from the secondary studies (see Table 4). As can be seen from Table 13, no HELM-related data items were extracted from 6 secondary studies. The majority of the remaining 24 studies (20), touched on 1-3 HELM measures, whereas just a few touched on 4-5 measures (4 studies). Across the 24 secondary studies, the most frequently reported HELM measures were Accuracy (18 studies), Robustness (16), and Efficiency (14), see Table 13. Toxicity was considered in only 8 studies, Bias in 2, while Calibration and Fairness were not explicitly addressed. In the following sections we synthesize findings for each HELM measure, taking into account papers’ quality based on QA scores (see Table 3), where [0 - 2) is low quality, [2 - 3) medium quality, and [3 - 4] high quality as summarized in Table 14. Manuscript submitted to ACM

24

Trovato et al. Table 13. Mapping of secondary studies to HELM measures and Scenarios. A ✓ indicates presence.

Total

✓

✓ ✓ ✓ ✓

✓ ✓

✓

✓

✓

✓

✓ ✓ ✓ ✓

✓

✓

✓ ✓ ✓ ✓ ✓

✓

✓

✓ ✓ ✓

✓

✓ ✓

✓ ✓ ✓

✓ ✓

✓ ✓

✓ ✓

14

29

✓ 18

0

16

0

2

8

✓ ✓ ✓

Application Domain

✓

✓ ✓

✓ ✓ ✓

8

2

Overall

✓ ✓ ✓

✓ ✓ ✓ ✓ ✓

✓

Scenario

✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Application Type

✓

Language

✓

Task

✓

Totals

HELM

✓ ✓ ✓ ✓ ✓

Efficiency

Fairness

Robustness

✓ ✓

Toxicity

✓

Scenario

Bias

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21 S22 S23 S24 S25 S26 S27 S28 S29 S30

Calibration

Study

Accuracy

HELM measures

5 1 0 1 3 2 3 4 2 1 0 3 3 2 2 2 3 2 3 4 0 0 4 2 3 0 1 0 1 1

1 1 2 1 2 3 2 1 1 2 1 2 1 1 1 1 1 1 1 1 1 1 1 2 2 2 1 0 1 1

6 2 2 2 5 5 5 5 3 3 1 5 4 3 3 3 4 3 4 5 1 1 5 4 5 2 2 0 2 2

0

Table 14. Summary of evidence for HELM measures, showing number of studies, quality distribution (based on Total QA score), and representative study IDs.

Measure

# Studies

Accuracy Robustness Efficiency Toxicity Bias

18 16 14 8 2

High [3 - 4]

Medium [2 - 3)

Low [0 - 2)

9 (S1, S8, S10, S13, S18, S19, S20, S23, S25) 8 (S1, S8, S13, S18, S19, S20, S23, S25) 7 (S1, S8, S19, S20, S23, S25, S29) 5 (S1, S8, S13, S23, S27) 2 (S1, S20)

4 (S7, S12, S14, S30) 4 (S7, S9, S12, S24) 4 (S7, S9, S12, S14) 1 (S24) 0

5 (S4, S5, S6, S16, S17) 4 (S2, S5, S15, S17) 3 (S5, S6, S16) 2 (S15, S17) 0

4.2.1 Accuracy. According to our extracted data, accuracy is the most extensively documented measure of LLM performance in CGTs. It is discussed in 18 studies (see Table 13). Of these, 9 are high quality (Total QA 3–4), 4 are medium quality [2 - 3), and 5 are low quality (<2), see Table 14. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 25 Eight studies (mostly medium-low quality papers) report improvements in accuracy or high accuracy for LLM-based CGTs (S5, S6, S7, S12, S14, S16, S19, S30). These gains are often tied to specific benchmarks and tasks and improvements are in comparison to traditional approaches or older DL models, such as recurrent neural networks (RNN), for example. Several works (S7, S16, S19) show that LLMs achieve high pass@k (a widely used accuracy metric that measures the probability that at least one of the top-𝑘 generated solutions for a given task is correct, typically verified against unit tests) scores on HumanEval [19] and MBPP[12], indicating strong functional correctness on short program synthesis tasks. S5 (low quality) report accuracy improvements in program repair benchmarks, where models correctly patch a large fraction of buggy programs. In testing scenarios, S12 and S14 (medium quality) show models generating unit tests that achieve higher coverage than prior baselines. Seven studies (mostly high quality) (S1, S8, S10, S12, S13, S20, S25) adopt a more cautious stance, questioning whether reported accuracy reflects genuine generalization. Several high-quality contributions (S1, S20) warn that benchmark scores may be inflated by dataset issues. For instance, S1 notes that code duplication and data leakage can lead to overly optimistic accuracy estimates. S20 emphasizes that accuracy metrics often fail to capture semantic correctness, producing syntactically correct but semantically incorrect code. Other high-quality papers (S10, S13) further stress that accuracy varies substantially across tasks and languages, with the same model performing well on HumanEval but poorly on more diverse or real-world datasets. S12 (medium quality) reports modest success in automated program repair, where most generated patches remain incorrect. Four studies (mostly high quality) explicitly report poor accuracy outcomes (S10, S18, S23, S30). Here, high-medium quality works (S10, S18, S30) demonstrate that models fail on less-represented programming languages and complex integration tasks, with accuracy dropping well below levels required for deployment (S23). Several recurring themes are reported in the literature that might affect the accuracy of LLM-based CGTs. Three studies (S4, S6, S10) emphasize fine-tuning as an effective way to improve accuracy as compared non-fine-tuned baseline models. Supplying additional context (such as APIs, higher-level project information) (S1, S20) and using few-shot prompting (S6, S10) are also supposed to improve accuracy. The view on the number of model’s parameters is mixed: S4 suggests that models with higher number of parameters achieve better effectiveness, whereas S8 highlights that smaller models can outperform larger ones. Finally, the usage of decoder-only architecture (S4) and larger training dataset size (S19) were linked to improved LLM-based CGTs. Overall, across the 18 studies, 8 lower-quality studies emphasize improvements, whereas 11 higher-quality studies either adopt a critical cautious perspective (7 studies) or report poor accuracy (4 studies). Several methods such as fine-tuning, providing more context and using few-shot prompting were reported as potentially increasing the accuracy of LLM-based CGTs. 4.2.2 Robustness. According to our extracted data, robustness is discussed in 16 studies (S1, S2, S5, S7–S9, S12, S13, S15, S17–S20, S23–S25). Of these, 8 are high quality (Total QA 3–4), 4 are medium quality [2–3), and 4 are low quality (<2), see Table 14. Across these studies, a consistent view emerges: robustness of LLM-based CGTs is frequently overstated and remains fragile under realistic conditions. Eight high-quality studies (S1, S8, S13, S18, S19, S20, S23, S25) provide the strongest and most general evidence for this conclusion. S1 notes that robustness is overstated when training and evaluation datasets overlap, due to duplication / data leakage. S13 reports that widely used benchmarks do not capture real-world scenarios, therefore limiting robustness claims. S20 defines robustness in terms of generalizability and finds that performance does not transfer consistently across tasks, datasets, or domains. S8 describes that robustness must be considered across multiple software engineering Manuscript submitted to ACM

26

Trovato et al.

activities such as package migration, for example, not only in isolated benchmarks. S23 highlights that both dangers of external attacks on the generated code and internal model limitations can undermine reliable behavior, requiring human oversight. S25 reports that decoding parameters of LLMs such as temperature settings strongly affect output stability. Four medium-quality studies (S7, S9, S12, S24) reinforce these findings in more specific contexts. S7 reports that applying off-the-shelf LLMs directly reduces stability in program repair and requires additional heuristics. S9 emphasizes the importance of robustness to adversarial attacks as LLMs are applied more widely in software engineering. S12 shows that test-driven program repair is affected by “test overfitting,” where generated patches pass existing tests but fail on unseen cases, thus limiting generalization. S24 compares robustness across programming languages and notes that results obtained in Python often transfer only partially to C. Four low-quality studies (S2, S5, S15, S17) report similar robustness concerns, albeit with narrower scope. S2 finds that improvements in test generation accuracy do not remove robustness issues, as generated tests fail to adapt when requirements change. S15 raises robustness concerns about data poisoning, where malicious triggers in training data compromise downstream behavior. S5 and S17 note that small changes in context or task formulation can reduce performance significantly. Overall, the majority of papers here (including all high-quality) are cautious regarding issues with robustness of LLM-based CGTs. High-quality papers emphasize that robustness is often overstated by benchmark-limited evaluations and is sensitive to data quality and evaluation design. Medium and low-quality studies confirm robustness issues in specific scenarios such as program repair and test generation. 4.2.3 Efficiency. According to our extracted data, efficiency is discussed in 14 studies (S1, S5–S9, S12, S14, S16, S19, S20, S23, S25, S29). Of these, 7 are high quality (Total QA 3–4: S1, S8, S19, S20, S23, S25, S29), 4 are medium quality [2–3) (S7, S9, S12, S14), and 3 are low quality (<2) (S5, S6, S16), see Table 14. Overall, these studies report efficiency issues associated with application of LLMs. Seven high-quality studies (S1, S8, S19, S20, S23, S25, S29) primarily discuss costs of LLM-based application to CGTs and possible improvements to address efficiency issues. Particularly, 6 studies (S8, S19, S20, S23, S25, S29) mention hardware /computational costs (in terms of time/space required), economical costs (such as high costs of running GPUs), and energy costs (including the impact of CO2 emissions and green energy). Some of the possible solutions discussed involve improving training processes (S1, S19) and utilizing hardware acceleration capabilities (S8, S19). Four medium-quality studies (S7, S9, S12, S14) echo these concerns. S12 and S14 highlight computational and economical costs associated with LLM-based CGT application, whereas S9 note limited applicability on resourceconstrained hardware such as mobile applications, for example. Interestingly, S7 shows that fine-tuning LLMs is still faster and more efficient than training older DL models (non-pretrained) from scratch. Likewise, three low-quality studies (S5, S6, S16) discuss efficiency in similar manner: S5 and S16 highlight all types of costs associated while S6 questions efficiency benefits of LLM-based CGTs, integrated into interactive development environments (IDE). Overall, the majority of studies (across all quality groups) highlight efficiency issues (such as computational / hardware / economical / energy / effort costs) associated with application of LLM-based CGTs. Several studies suggest potential improvements in terms of training strategies and hardware acceleration. 4.2.4 Toxicity. Toxicity is discussed in 8 studies (S1, S8, S13, S15, S17, S23, S24, S27). Of these, 5 are high quality (Total QA 3–4: S1, S8, S13, S23, S27), 1 is medium quality [2–3) (S24), and 2 are low quality (<2) (S15, S17), see Table 14. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 27 Five high-quality studies (S1, S8, S13, S23, S27) report on risks related to harmful, biased, or unsafe outputs produced by LLMs in code-generating contexts. S8 and S23, particularly, mention privacy concerns when private data (such as e-mails, passwords, and usernames for example) can leak from training datasets into code generated by LLMs. S1 highlights that toxicity concerns arise when the quality of identifier names and function names is unaccounted (echoed by S13 for general code vulnerabilities in training datasets) and leak into generated code, increasing vulnerability chances. S27 provides evidence that generated code can be substantially more vulnerable than code created by humans. Two low-quality studies (S15, S17) also raise toxicity concerns in terms of private data leakage into generated code; they also mention copyrighted code fragments appearing in generated data. As a potential solution, one medium-quality study (S24) suggests that vulnerability issues in generated code can be somewhat addressed with few-shot prompting giving secure examples. Overall, the evidence indicates that toxicity is a less frequently studied measure compared to accuracy or robustness, but it is consistently recognized as a risk. Studies emphasize that toxicity remains unresolved and must be evaluated when LLMs are applied to code-generating tasks, primarily linking these issues to training datasets. 4.2.5

Bias. Bias is discussed in just 2 studies (S1, S20), both are high quality (Total QA 3–4), see Table 14.

S1 identifies bias resulting from duplicated or cloned code instances in training datasets. The study reports that such data artifacts can bias the model towards highly represented classes, which is described as a form of long-tailed distribution problem. This type of bias can distort model behavior by over-emphasizing common patterns while underrepresenting rarer but important ones. S20 reports that reliance on limited or biased datasets may cause LLMs to inherit systematic biases, leading to biased or inaccurate predictions in code generation and completion tasks. The study stresses that dataset design and curation are critical to mitigate such issues. 4.2.6 Answering RQ2. ("What HELM measures of LLM-based CGTs are reported in secondary studies?") across secondary studies, the HELM measures of LLM-based CGTs are: • Accuracy: Mixed effects reported. While the majority of studies recognizes the effectiveness of LLM-based CGTs on standard benchmarks such as HumanEval and MBPP, the higher-quality studies remain more cautious emphasizing poor/questionable accuracy on a) less-represented languages, b) integration-heavy tasks, and c) real-world scenarios; • Robustness: widespread fragility has been reported under dataset/prompt/configuration changes and limited transfer across tasks/datasets/languages; • Efficiency: the majority of studies highlight recurring limitations due to computational, economic, and energy costs, with partial mitigation via training and hardware acceleration strategies; • Toxicity: this effect is addressed in fewer studies, but consistent recognition of risk are reported regarding privacy of data in generated code and its security (susceptibility to vulnerabilities); • Bias: reported in a very limited number of studies, but high-quality evidence linking biased outcomes to data artifacts and distributional imbalance of code (e.g. prevalence of clones). 4.3

RQ3: What HELM scenarios of LLM-based CGTs are reported in secondary studies?

To answer RQ3 we extracted data items related to HELM scenario (see Table 13). As can be seen only one study (S28) did not provide any Scenario-related data items: the majority of the remaining 29 studies provided 1-2 scenario-related data items (28 studies) and just one study provided 3. Almost all studies (29) reported the task(-s) under consideration, Manuscript submitted to ACM

28

Trovato et al.

while only a minority specified the languages (8) or application types (2), and none provided details on the application domain. Secondary studies report a wide range of code-related tasks, often using heterogeneous or overlapping terminology. Table 15 presents the original task labels as they appear in the studies, their mapping to our CGT definitions (Section 2.1), and the corresponding counts of studies in which these tasks are reported. When tasks are consolidated according to our definitions, Patch generation / repair emerges as the most frequently reported CGT (22 studies), aggregating terms such as program repair, vulnerability repair, bug repair, and bug fixing. This is followed by code generation (20 studies), and then by code translation and code completion (9 and 8 studies, respectively). Other CGTs, including program synthesis and test generation (reported under several synonymous labels), appear less frequently, while refactoring is mentioned in only a small number of studies. Some rare task labels (e.g., “code co-evolution,” “code editing,” “method name generation,” and “testing repair”) cannot be mapped directly to our CGT definitions but are closely related to refactoring or repair activities. Table 15. Original task terms from secondary studies, mapped to canonical categories; counts are case-insensitive and tasks are expanded from comma-separated lists (sorted by frequency).

Original term (as reported)

Mapping to definitions

Count

code generation code translation code completion program repair vulnerability repair program synthesis test generation code editing refactoring test case generation automated bug fixing bug fix bug repair code co-evolution patch generation program translation programming error repair security vulnerability repair software bug repair software vulnerability repair testing repair unit test case generation method name generation

Code generation Code translation Code completion Patch generation / Repair Patch generation / Repair Program synthesis Test generation — (not mapped) Refactoring Test generation Patch generation / Repair Patch generation / Repair Patch generation / Repair — (not mapped) Patch generation / Repair Code translation Patch generation / Repair Patch generation / Repair Patch generation / Repair Patch generation / Repair — (not mapped) Test generation — (not mapped)

20 9 8 8 6 4 4 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1

A wide range of programming languages is represented. The most frequently mentioned languages include Java, Python, C, C++, and JavaScript, with several studies reporting multi-language contexts. Beyond these mainstream languages, additional coverage includes C#, SQL, PHP, Kotlin, Rust, Go, Verilog, Solidity, Ruby, and OCaml. Studies also note domain-specific and low-resource languages, as well as formal languages such as Isabelle/HOL. Also beyond traditional programming languages, Excel and Power Fx are mentioned. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 29 Table 16. Papers grouped by integration/application challenge category (distinct-paper counts). High: QA ∈ [3, 4], Medium: [2, 3), Low: [0, 2).

Category

Count

High [3-4]

Medium [2-3)

Low [0-2)

Economics Tooling & Workflow Evaluation & Benchmark Validity Data & Context People & Process Security & Safety Model/Training Legal & Privacy

10 7 7 6 5 5 4 4

5 (S13, S19, S23, S25, S29) 3 (S18, S23, S27) 6 (S1, S8, S10, S13, S20, S29) 4 (S8, S19, S20, S25) 3 (S10, S18, S27) 2 (S8, S23) 3 (S8, S20, S25) 2 (S23, S25)

4 (S7, S9, S12, S14) 2 (S21, S24) 0 0 2 (S12, S30) 2 (S9, S24) 1 (S9) 0

1 (S16) 2 (S6, S17) 1 (S5) 2 (S15, S17) 0 1 (S15) 0 2 (S15, S16)

Not provided

7

2 (S11, S26)

1 (S22)

4 (S2, S3, S4, S28)

The set of application types is more heterogeneous and reported in only 2 studies. One study mentions low-code applications, highlighting LLM support for end-user programming. Another reports a broad set of software systems, including mobile applications, DL libraries, compilers, SMT solvers, autonomous driving systems, cyber-physical systems, toolchains, JavaScript engines, quantum computing platforms, and video games. Answering RQ3: What HELM scenarios of LLM-based CGTs are reported in secondary studies? we note that secondary studies mostly report tasks, sometimes languages, and rarely application types, but never domains. Having that said, secondary studies report scenarios spanning a wide range of tasks (most prominently code generation, patch generation and repair, code completion, and code translation), applied across multiple programming languages (with strong representation of Java, Python, C, C++, and JavaScript, and additional coverage of domain-specific, low-resource, and end-user languages), and targeting diverse application types (from low-code platforms to compilers, mobile apps, cyber-physical systems, and quantum computing platforms). 4.4

RQ4: What challenges in applying or integrating LLM-based CGTs into software engineering workflows are identified in secondary studies?

As can be seen from Table 16 the majority of studies discuss challenges related to Economics (5 high, 4 medium, 1 low), Tooling & Workflow (3 high, 2, medium, 2 low), and Evaluation & Benchmark Validity where paper quality is the strongest (6 high, 0 medium , 1 low). These are followed by Data & Context (4 high, 0 medium, 2 low), People & Process (3 high, 2 medium, 0 low), and Security & Safety (2 high, 2 medium, 1 low) categories. Less frequently discussed are challenges in Model/Training (3 high, 1 medium, 0 low) and Legal & Privacy (2 high, 0 medium, 2 low). Finally, 7 studies reported no challenges, including 2 high-quality papers. Economics. Across all studies in this category, there seems to be a consistent mention of computational costs and resource constraints/limitations. The former are mentioned in 3 high-quality studies (S13, S23, S29), 3 medium quality (S7, S12, S14), and one low-quality (S16). The computational costs are primarily associated with various stages of LLM training (pre-training, fine-tuning) (S14, S16, S23, S29), inference (S7, S12, S16), and the need to run larger and more computationally expensive benchmarks (S13). Additionally, financial costs are mentioned if off-the-shelf solutions such as GPT-4 are employed (S12). Challenges arising due to resource limitations/constraints are reported in 4 high-quality (S19, S23, S25, S29) and 1 medium quality study (S9). Here, large sizes of LLMs require more computational power, storage, and can affect latency and energy consumption (S19, S25). This is particularly pronounced in resource-limited Manuscript submitted to ACM

30

Trovato et al.

environments such as on mobile devices where LLMs can struggle to execute, whereas limiting and reducing models’ sizes would result in decreased effectiveness (S23). Training can also be limiting for individuals and organizations without the access to resource-rich infrastructure (S23). Tooling & Workflow. Across studies in this category, two recurrent issues are mentioned: insufficient integration of LLM-based CGTs into existing tools and pipelines, and the lack of operational controls and responsiveness. The lack of integration with existing tools is highlighted by three studies: 1 high-quality (S18), 1 medium (S24), and 1 low quality (S6). They report the lack of integration with existing IDEs/version control tools/ other existing traditional approaches. One medium-quality study (S21) and two high-quality studies (S23, S27) note the absence of accessible controls such as temperature parameters (an LLM parameter controlling the randomness of its output), for example. High-quality evidence further points to responsiveness and throughput as factors for workflow latency reduction, concurrent-user scaling, and memory management to support interactive user workflow (S23). Finally, high-quality studies stress comprehension and personalization: suggestions must be reviewable and understandable by a developer, and tools should adapt to individual coding styles and integrate with developer utilities (S23, S27). A low-quality study adds that adaptive mechanisms are needed to address context-specific hallucinations (S17). Evaluation & Benchmark Validity. Across all evidence in this category, challenges are mostly supported by highquality sources (6 high, 1 low). Multiple studies argue that common benchmarks are unrepresentative of practical development: they emphasize single languages, curated repositories, simplified bugs, and imbalanced or narrow task scopes, which collectively distort difficulty and real-world scenarios (S1, S8, S13). Also, they report poor generalizability: models tuned to a specific dataset or task family transfer poorly across domains, languages, or problem types (S29, S13). Several studies identify specification and measurement issues, noting ambiguous task definitions and single-metric reporting: they propose comprehensive, multi-metric frameworks that extend beyond functional correctness to include non-functional qualities and practice-level outcomes, with cross-language coverage and adaptable protocols (S8, S10, S13, S20). Finally, one low-quality source notes practical unavailability of executable test fixtures needed by dynamic metrics, further limiting reliable assessment (S5). Data & Context. Here, studies emphasize such challenges as unfamiliar/private domains and limitations in how much and what kind of context models can use. High-quality studies report difficulties in capturing local–global dependencies for code completion (S19), challenges adapting to repository and system-level problems beyond function scope (S8), and sensitivity to intent ambiguity and project-specific semantics—requiring additional domain knowledge (S20). They also point to data complexity, that can exceed the ability of LLMs to process (S25). Coverage gaps compound these issues: low-resource and domain-specific languages remain underrepresented, constraining applicability (S8). Lower-quality studies align with these observations, noting mishandling of private APIs/repo-wide reasoning and hard limits from input-token windows (S17), as well as domain shift to unseen libraries/environments with impracticality of full fine-tuning, hence the need for prompting/RAG-style adaptation (S15). People & Process. High-quality studies in this category highlight communication problem between a developer and an LLM (S18), existing reliance on human oversight (for example in bug fixing) (S10); and the risk by novice developers to overly rely on LLM-based CGTs outputs, underscoring the need for safety training and usage guidance (S27). Medium-quality studies position LLMs primarily as assistants rather than replacements (S30) and note limited empirical understanding of how practitioners actually use these tools in situ (S12). Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 31 Security & Safety. Studies in this category call for rigorous validation in the development pipeline (e.g., formal methods, alignment to coding standards, developer review) (S23, S8). The threats such as backdoors, data memorization, and model extraction/stealing are reported together with the possible mitigations such as obfuscation and client-side protection (S23). Medium-quality studies reinforce the practical risk (e.g. complex web-application tasks are prone to security weaknesses), whereas adversarial attacks are a concern in mission-critical settings (S24, S9). Low-quality evidence reiterates data poisoning as a training-time hazard that can manifest as systematic defects at inference (S15). Model/Training. High-quality studies argue for continuous learning to track evolving programming practices (S8), for injecting program structure and domain knowledge via code embeddings, syntax/semantic analyses to improve LLMs (S20), and for automated construction of organization-specific datasets from software repositories to reduce the manual cost of fine-tuning while increasing task fit (S25). Medium-quality study adds to this implying the need for further model tuning or specialization (S9). Legal & Privacy. Studies here indicate potential threat of leaking private information: code-generating models can reproduce training snippets containing personally identifiable information or secret material, with non-trivial leakage rates, motivating privacy-aware evaluation and mitigations (S23). Organizational responses are also reported: due to confidentiality obligations, many teams avoid commercial endpoints and prefer open-source models fine-tuned on in-house data (S25). Low-quality sources align with this picture, noting that web-crawled corpora may include sensitive artifacts (e.g., credentials) and code under restrictive licences (e.g., GPL), which models can later emit during code generating task (S16, S15). Answering RQ4: What challenges in applying or integrating LLM-based CGTs into software engineering workflows are identified in secondary studies? The most frequently reported challenges concern economics: substantial computational and financial costs across training, inference, and evaluation, together with hard resource constraints that affect latency, energy, and feasibility on constrained devices. Tooling and workflow issues center on incomplete integration in existing tools and workflows (e.g. IDE), limited operational controls (e.g., temperature (LLM randomness parameter)), responsiveness and throughput requirements for interactive use, and the need for comprehensible, personalized suggestions. The evaluation validity concerns are methodologically strongest and highlight unrepresentative benchmarks, poor cross-domain generalization, contamination, ambiguous task specifications, and calls for comprehensive, multi-metric, cross-language assessment frameworks. Data and context challenges include weak grounding in private/unfamiliar domains, token-window limits for repository-scale tasks, and gaps for low-resource or domain-specific languages. People and process concerns emphasize reliance on human oversight, novice over-trust, and poor communication between LLMs and humans. Security and safety issues span vulnerabilities in generated code and model-level threats (adversarial, poisoning, extraction), requiring human validation. Less frequent are model/training needs (continual/task-aligned adaptation, org-specific data pipelines) and legal/privacy risks from memorization and disclosure of protected content. Notably, 7 studies reported no challenges. 4.5

RQ5: What future directions for LLM-based CGTs are reported in existing secondary studies?

The reviewed secondary studies outline several categories of future research directions as shown in Table 17. The most frequently mentioned category is model improvement & training (16 studies). Within this category, 6 studies are of high quality (S8, S11, S18, S19, S20, S25), 7 are of medium quality (S7, S9, S12, S14, S22, S24, S30), and 3 are of low quality (S5, S15, S28). Next is benchmark & evaluation improvement (11 studies), with 6 high-quality (S8, S13, S18, S19, S23, S29), 3 Manuscript submitted to ACM

32

Trovato et al. Table 17. Mapping of papers to future direction categories, grouped by their quality scores.

Category

High [3–4]

Medium [2,3)

Low [0,2)

Model improvement & training Benchmark & evaluation improvement Ensembling & hybridization Advancing experimental research Security, Privacy & Trustworthiness Integration into practice Dataset improvement

Count 16 11 7 7 6 5 5

6 (S8, S11, S18, S19, S20, S25) 6 (S8, S13, S18, S19, S23, S29) 4 (S10, S20, S25, S29) 2 (S1, S29) 3 (S11, S23, S27) 4 (S18, S19, S20, S23) 3 (S1, S20, S29)

7 (S7, S9, S12, S14, S22, S24, S30) 3 (S7, S9, S12) 2 (S7, S14) 4 (S7, S12, S22, S30) 1 (S22) 0 1 (S22)

3 (S5, S15, S28) 2 (S15, S17) 1 (S17) 1 (S5) 2 (S15, S16) 1 (S17) 1 (S15)

Not provided

6

1 (S26)

1 (S21)

4 (S2, S3, S4, S6)

medium-quality (S7, S9, S12), and 2 low-quality studies (S15, S17). Further directions include ensembling & hybridization (7 studies: 4 high, 2 medium, 1 low) followed by advancing experimental research (7 studies: 2 high, 4 medium, 1 low) and security, privacy, & trustworthiness (6 studies: 3 high, 1 medium, 2 low). The least represented are dataset improvement (5 studies: 3 high, 1 medium, 1 low) and integration into practice (5 studies: 4 high, 0 medium, 1 low). Finally, 6 studies did not provide explicit future directions (not provided: 1 high, 1 medium, 4 low). Model improvement & training. A more recurring theme across high-quality papers (S11, S19, S20) is domain-specific integration through specialized datasets, code structures, and programming standards. Others stress interpretability and usability, either via transparent synthesis models (S11) or incorporating readable code and better coding styles into training (S8). Improvement with respect to efficiency is also reported, with calls for computational optimization (S11) and finer-grained code completion (S19). More novel directions include self-repairing models (S20) (where an LLM can analyse and repair/make itself more secure), collaborative open-source development of LLMs (S18), advances in prompt design (S25), and merely increasing the size of LLMs for better effectiveness (S18). Medium-quality studies report improvement directions such as efficient fine-tuning: S12 and S9 propose transfer learning, parameter-efficient methods (e.g., prefix-tuning, low-rank adaptation), and targeted fine-tuning. Another direction emphasizes efficiency at the computational level, with S14 and S22 suggest improving efficiency through model optimization and leveraging structural code representations such as abstract syntax trees for cross-language generalization. In addition, several studies explore prompting and context management, including zero-shot prompt optimization (S24) and advanced preprocessing (S7). Finally, concerns about performance stability and interpretability are raised: S30 notes the lack of rigorous evaluation of many proposed improvements, while S9 suggests incorporating explainable techniques to make predictions more transparent. Low-quality papers (S5, S15, S28) note that richer code representations (trees, graphs, data/control-flow features) could better align pre-training with program structure (S5, S15) for model improvement. Generation scope such as file-/project-level and the use of chain-of-thought for complex requirements is also highlighted as possible direction (S15). In line with higher quality papers they suggest domain specialization (S15) and expanded prompt engineering(S28). Finally, they call for interpretability in repair models (S5). Overall, domain-specific integration (specialized datasets, code structures, programming standards) seems to be a repeating future research direction towards model improvement across high-medium-low quality studies. Improving prompting, calls for better interpretability, efficiency, and richer code representations are also commonly mentioned. Finally, less commonly mentioned directions here include, the ability of LLMs to self-repair, open-source collaborative development of LLMs, and training with non-functional constraints. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 33 Benchmark & Evaluation Improvement. Here across all quality studies, there is broad agreement that current evaluations are narrow and fragmented, motivating a comprehensive, multi-dimensional framework to holistically assess LLMs for CGTs (S8, S13, S9, S12, S17, S15). High-quality papers call for frameworks that extend beyond functional correctness to cover code quality, maintainability, performance/efficiency, security, execution stability, and ethics, with cross-language coverage, edge cases, and continuous/living updates (S8, S13). Complementary to this, multiple works advocate metric standardization to enable comparability across studies and tasks (S19, S29, S7). Other directions recommend carefully selected, manually verified test sets and "live" benchmarks to accurately track progress of LLMs (S18). Finally, studies highlight the need for scalable evaluation protocols that balance quantitative metrics with qualitative and human evaluation, to better capture usefulness and robustness in practice (S13). Ensembling & Hybridization. Two trends seem to emerge in this category. First, model ensembling (combining multiple LLMs and/or specialized ML models) aims to exploit model heterogeneity to improve performance (S14, S29, S20; high/medium quality). Typical designs include heterogeneous LLM ensembles and LLM+classifier pipelines; expected benefits are variance reduction and specialization across tasks, but they require careful aggregation, cost–benefit analysis, and standardized evaluation to avoid benchmark overfitting. Second, hybrid techniques integrate LLMs with traditional SE methods such as static/dynamic analysis, APR heuristics, and domain-specific pipelines (S29, S20, S25, S7, S10, S17; mostly high quality). Advancing Experimental Research. In this category, the studies suggest more rigorous and generalizable experimentation. High-quality work suggests expanding study of dataset data quality, focusing on the impact towards efficiency, robustness, and security characteristics of generated code (S1). Another study suggest expanding experiments to include industrial-scale evaluations once data access allows (S29). Medium-quality studies emphasize standardization parameters, datasets, and experimental settings to enable fair comparisons and to diagnose patch overfitting (S7). Incorporation of human studies to assess tool maturity and reliability in practice (S12, S22) is also suggested. Finally, several papers note a coverage bias toward general LLMs and call for targeted evaluations of code-centric models (S30) and for exploratory links across bug classes (S5). Security, Privacy, and Trustworthiness. Studies here seem to agree on the need to treat security, privacy, and trustworthiness as important objectives. Security improvement is particularly emphasized with suggestions to treat generated code with caution if deploying (S16, S27). Privacy risks such as data leakage and model memorization are highlighted as potential deployment risks in industry (S15, S22). Beyond correctness, trustworthiness is highlighted as an important property affecting reliability, interpretability of recommendations, and demonstrable generalization across projects and environments (S23). Ethical issues alongside transparency are suggested to be addressed as well (S11). Integration into Practice. High-quality studies emphasize workflow-centred integration and tool-level integration as the primary routes to integration in practice. Workflow integration entails deployment-ready features and interactive agents, and retrieval-augmented generation (RAG) so that LLM capabilities are integrated within existing development workflows rather than as standalone tools (S18). Tool integration assumes integration into IDEs, code editors, version control, and debugging pipelines (S19, S23). Context-aware integration is also mentioned assuming LLM-based CFTs should be more project/task specific (S20, S17). Dataset Improvement. In this category studies propose several future directions. S1 (high quality) suggests cleaned versions of datasets to be used for training, removing data smells. S29 and S20 (high quality) suggest domain-specific Manuscript submitted to ACM

34

Trovato et al.

datasets to encode programming-domain knowledge in specialized contexts. Medium-quality work emphasizes representativeness: S22 recommends more diverse, real-world datasets to better reflect industrial conditions. S15 (low-quality) proposes manual/automatic cleaning to mitigate data poisoning and suggests adversarial augmentation (perturbed examples) to improve robustness. Answering RQ5: What future directions for LLM-based CGTs are reported in existing secondary studies? Secondary studies most consistently recommend the following areas for future research directions (sorted from most to least mentioned): • Model improvement & training, particularly focusing on domain-aware model improvement; • Benchmark & evaluation improvement with the help of comprehensive/holistic benchmarks and standardized evaluation; • Ensembling & hybridization with either ensembles of ML models (LLMs and other) or hybridization with traditional approaches; • Advancing experimental research including more rigorous experimental methodology and incorporation of human/industrial studies; • Improving security privacy & trustworthiness of LLM-based CGTs; • Integration into practice by merging into existing workflows and tools; • Dataset improvement via higher-quality data and incorporation of domain-specific knowledge.

5

Discussion

5.1

RQ1: Landscape of secondary studies on LLM-based CGTs

The analysis of secondary studies shows their expansion from 2022 to 2024 followed by sustained, if leveling-off, activity in the first half of 2025. There is a marked shift from exploratory studies (surveys/reviews) toward more systematically-performed studies (SLRs particularly): this could suggest a field transitioning from exploratory mapping of studies to more focused synthesis of studies in the area. In evidence-based software engineering, such a shift could be interpreted as a maturation signal: once the topic space has been mapped and terminology stabilized, researchers increasingly look into aggregative inference and interpretation rather than mere mapping [50, 77]. There also seem to be growing expectations by high-ranked venues for more rigorous synthesis: the concentration of publications in Q1 journals (e.g., TOSEM, TSE, CSI) further reinforces this interpretation. At the same time, the total number of primary studies reviewed between 2023 and 2024 has increased only by a small margin (despite a tripling of secondary studies). Two non-exclusive explanations can be plausible here: the speed of primary studies appearing has slowed down somewhat and/or cross-study redundancy arising from similar search strings, sources, and inclusion criteria. From a research-efficiency standpoint, this raises risks of duplicated effort without proportional gain. Reusable transparent datasets (with supplementary protocols) could be used to address this [50, 77]. Practically, this trend implies that future contributions will likely need to differentiate less by volume of included studies and more by scope refinement, methodological innovation (e.g., quantitative meta-analytic models, bias control [51]), or theory-building. It should be noted, however, that the trends above are observed over a relatively short time span (approximately 3.5 years) and should therefore be interpreted as early indicators rather than long-term trajectories. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 35 The dominance of LLM-scoped studies, together with the strong prevalence of SWEBOK’s Software Construction (followed by Maintenance and Security), highlights the application profile of CGTs from an AI perspective: the majority of CGTs are used to construct code and to repair bugs or fix vulnerabilities using LLMs. Finally, venue patterns provide an interesting quality signal. Journal-published studies show the highest average QA scores, consistent with rigorous peer-review process. However, arXiv-hosted studies display only slightly lower QA on average, indicating that preprint dissemination in this fast-moving domain does not necessarily trade rigor for speed (e.g. during the creation of this publication, one paper (S23) previously hosted on arxiv was accepted in TOSEM). Taken together, the landscape depicts a field that is quickly evolving, maturing methodologically and qualitatively, yet still to consolidate around reusable/transparent protocols and datasets, and broader coverage of less represented SWEBOK KAs that require code generating activities (e.g. Software Testing). 5.2

RQ2: HELM measures of LLM-based CGTs

The synthesis of HELM measures reveals several important patterns. First, measure representation is uneven. Accuracy is the most frequently reported outcome, while robustness and efficiency are reported less often. Toxicity and bias remain rarely reported, whereas calibration and fairness are not addressed at all (it should be mentioned that the latter two are less applicable to SE context, though). Although there seems to be a common acknowledgment of accuracy improvements in LLM-based CGTs, higherquality studies consistently remain cautious regarding its reliability (especially when applied to real-world scenarios) due to dataset contamination and benchmark limitations. This aligns with broader critiques in the literature that reported improvements in benchmark-based performance metrics (e.g., pass@k or success rates on curated code-generating benchmarks) may be inflated by data leakage or narrow evaluation setups [12, 19]. Robustness is widely reported as fragile, with performance varying substantially across tasks, languages, and configurations. These findings echo concerns in program repair and software testing studies, for example, where LLM-generated patches often overfit test suites and fail on unseen cases [73]. Efficiency concerns are systematically reported: computational, economic, and energy costs are highlighted across quality levels, consistent with sustainability concerns in broader natural language processing (NLP) research [88]. Toxicity and bias are underexplored but consistently acknowledged as risks, reinforcing conclusions from the AI safety literature that harmful outputs and data biases are persistent challenges [102]. Calibration and fairness are absent, indicating that evaluation practices in SE are not entirely consistent with those in broader NLP, where these measures are more standard. Collectively, the HELM measures suggest that while LLMs achieve measurable accuracy gains on short-form synthesis tasks, these can be fragile and should be cautiously assessed when applying to practice. Efficiency remains a concern, and safety-related risks are insufficiently addressed. Moving towards a more comprehensive benchmark (incorporating several measures) and expanding research into less studied areas could partially address these concerns. 5.3

RQ3: HELM scenarios of LLM-based CGTs reported in secondary studies

The scenario evidence reported by secondary studies is mostly reported in terms of the task dimension, with infrequent reporting of language and very rare reporting of application type. Application domain is not reported at all. This can limit the interpretability of LLM-based CGT studies and aggregation synthesis of results and could also undermine the external validity of such studies. The ‘task’ attribute of the scenario seems to have a large number of synonyms (e.g., overlapping repair labels) and, despite our consolidation of commonly reported CGTs based on prior task lists and Manuscript submitted to ACM

36

Trovato et al.

the LLM-based literature, would benefit from a more widely adopted controlled vocabulary and task normalization to reduce ambiguity and enable meta-analysis and synthesis across studies. At the same time, the broad language coverage reported, spanning general, domain-specific, low-resource, and even end-user languages, suggests that LLM-based CGTs are being evaluated across diverse programming ecosystems. In the future, adopting standardized taxonomies when reporting could improve reproducibility and strengthen claims about generalizability. 5.4

RQ4: Challenges in applying or integrating LLM-based CGTs

The economics related challenges seem to prevail: training, adaptation, and inference costs, evaluation at scale, and the latency/energy constraints of deployment environments jointly limit feasibility and scope. Compounding this, the presence of high-quality evidence around evaluation and benchmark validity indicates that current assessment practices insufficiently represent the heterogeneity of real projects (languages, repositories, dependency graphs, non-functional requirements), increasing risks of overfitting to narrow tasks and cherry-picked data. Tooling & Workflow challenges suggest insufficient integration with existing systems, limited operational controls, and responsiveness/throughput bottlenecks that impede continuous, interactive use. These observations are in line with prior reports of integration and maintenance concerns in ML systems, where the model is the ’easy’ part and the surrounding infrastructure is what determines product viability [85]. Highly aligned with this concern, People & Process suggest that the effective use of LLM-based CGTs assumes communicative alignment between models and developers, appropriate safety training for novice developers, and explicit human controls. Data & Context challenges further highlight that the empirical improvements observed on public benchmarks (e.g. MBPP) may not transfer to real-world codebases without targeted adaptation (prompting, retrieval, or fine-tuning) and expanded context handling. Finally, Security & Safety challenges including vulnerabilities of sorts in generated code, adversarial attacks, and model-level risks (such as memorization, extraction) imply that CGTs must be embedded in guarded pipelines (e.g. organizational policy, static/dynamic analysis, code review) and possibly assessed with threat-aware evaluation protocols [18]. Legal & privacy risks result in emission of licensed or sensitive content and can imply organizations use inhouse/fine-tuned models. Overall, a more successful and widespread integration of LLM-based CGTs into real-world software systems/workflows might require addressing economic concerns, comprehensive multi-metric and multilanguage evaluation, infrastructure that enables controllable, explainable, and low-latency interaction, and addressing security, privacy, and people & process concerns. 5.5

RQ5: Future directions for LLM-based CGTs

Model improvement and training is the most commonly reported future direction among the secondary studies. It can suggest that there is still a decent room for improvement of LLM-based CGTs. Particularly, domain-aware adaptation of the models is repeatedly proposed, reflecting a move towards specialized models tuned to organizational codebases and standards. Efficiency, interpretability, and controllability are mentioned as much as accuracy is. This is consistent with prior studies highlighting ML systems value-increase when models are engineered to align with context, data, and downstream tasks rather than optimized in isolation [85]. Second mostly reported are recommendations for benchmark and evaluation improvement (multi-metric, crosslanguage, and real-world scenario based). This can suggest that current benchmarks insufficiently reflect real software work, aligning with similar concerns in broader LLMs for NLP (where, for example, accuracy alone is not sufficient) [61]. The emphasis on ensembling and hybridization (across LLMs and traditional approaches) and on integration into practice can indicate a pivot from standalone models to systems of models and /or other approaches. This is consistent Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 37 with current existing directions in LLM for SE research, where ensembles, for example, are employed to improve models performance [10]. Finally, directions around security, privacy, and trustworthiness and dataset improvement, though least-reported, are as important. They can suggest rising concerns regarding security vulnerabilities in existing models, privacy violations, and the impact of poorly constructed datasets towards these issues. Overall, there seems to be calls towards domain-specialized modeling with real-world scenario, multi-dimensional evaluation. Additionally, hybrid/ensemble systems seem to be a viable path towards improving models’ performance, whereas security and privacy concerns of LLMs are equally need to be addressed. Notably, comparatively few studies emphasize integration into practice, despite the increasing deployment of LLM-based CGTs in real-world settings. This limited attention suggests that human-in-the-loop concerns remain under-explored relative to model and benchmark-centric directions. 5.6

Application of LLM toward literature review

In this work, AI (GPT4o LLM) was used in backward and forward snowballing to screen large corpora of potentially relevant abstracts and the screening results were subsequently cross-checked by the first author (MC). The results indicate that the AI system consistently captured relevant studies with high precision and perfect observed recall in the validation sample. The agreement rate (0.83) was comparable and even slightly exceeding dual human screening during SLRs [76]. Our strategy and results seem to align with existing literature such as a screening strategies proposed by Waffenschmidt et al. [96], in which one reviewer or model excludes papers while all potential inclusions are retained. Recently, within SE domain, Huotala et al. [45] have shown that GPT-based models can approach human-level performance in screening for systematic reviews, validating the model’s use in SE-specific contexts. This approach is further supported by evidence from other recent research in other domains: for example, Nykvist et al. [70] demonstrated that GPT-4 could achieve perfect recall in screening abstracts for environmental systematic reviews, and Guo et al. [36] found similar performance in the medical domain. 6

Threats to validity

In line with Kitchenham’s guidance for systematic reviews in software engineering and SEGRESS reporting [50, 51], we report on threats to construct, internal, and external validity. Construct validity. Here threats may arise from how data was used and coded. For example, study type labels (e.g., “SLR”, “SMS”, “survey”) are taken as reported by the authors, which may not always align with formal methodological definitions [50]. Second, and similarly, multi-valued non-trivial constructs (e.g., SE scope via SWEBOK v4 KAs [101], scenario attributes, and challenge categories) require normalization of heterogeneous terminology. We mitigated this by establishing extraction and coding rules, cross-validating between the authors, and mapping to standard existing definitions/frameworks (e.g. SWEBOK, HELM framework), where possible. Internal validity. Selection bias and non-inclusion can occur during database searches, screening, and snowballing. We searched multiple bibliographic sources recommended for SLRs [37] and used a PRISMA-like flow [67] with deduplication, yet different index coverage (i.e, variation in which and how venues, publication types, and metadata are indexed across databases) in paper databases and metadata may still omit eligible studies. For Google Scholar (GS), we used fixed blocks of 50 in relevance order and applied a stop rule (two consecutive blocks with < 5% inclusion; a hard cap of 300 results) informed by best-practice guidance on GS precision and practicality [14, 37, 40, 98]. This balances feasibility and sensitivity but may miss low-ranked relevant items. During snowballing we adopted Wohlin’s iterative Manuscript submitted to ACM

38

Trovato et al.

procedure with a stopping rule (< 5% yield) [103]: setting such a threshold risks early termination and omission of relevant studies. To limit single-reviewer bias in title/abstract screening, we audited a balanced random sample of 30 items with multiple raters and obtained almost-perfect agreement (Fleiss’ 𝜅 = 0.86) [29, 57]. For semi-automated screening of 2,769 snowballed records, we validated GPT-4o decisions on a 100-item random sample, observing precision = 0.77 (Wilson 95% CI 0.60–0.89) and recall = 1.00 (Wilson 95% CI 0.86–1.00), and Cohen’s 𝜅 = 0.83 against human judgment[72, 76, 96, 97]. Despite this, the risks still include GS ranking of artifacts, prompt sensitivity in AI screening, and missed citations due to incomplete reference metadata. External validity (generalizability). Our scope targets secondary studies on LLM-based CGTs published since 2017, written in English, and includes arXiv as the sole source of grey literature. As a result, the findings may not generalize to non-English reviews or to grey literature beyond arXiv. More broadly, the results characterize patterns and emphases in the surveyed secondary literature, rather than the full body of primary CGT research. Additionally, because several trends are observed over a relatively short time span (approximately 3.5 years), their generalizability beyond the studied time-frame remains uncertain. 7

Conclusions

To the best of our knowledge, this tertiary study provides the first systematic synthesis of secondary evidence on Large Language Models (LLMs) applied to code-generating tasks (CGTs) in software engineering. To accomplish this, we leveraged Kitchenham/SEGRESS protocol suggested by Kitchenham et al. for conducting/reporting secondary/tertiary studies in software engineering. Particularly, we systematically searched 5 article databases and Google Scholar, subsequently extending this with snowballing to obtain the final set of 30 relevant secondary studies. To address bias risk in extracting/synthesizing/reporting data we systematically employed extractor+checker and multi-rater strategies, achieving almost-perfect agreements. The findings portray a rapidly evolving yet unevenly structured field: in less than three years, even the number of secondary studies has increased substantially (for example, going from 6 in 2023 to 17 in 2024), while also marking a methodological shift from exploratory surveys toward systematic literature reviews and mappings (83% of the latter in 2025). This can suggest maturation of this research area, yet also reveals emerging redundancy and limited coverage of underrepresented software engineering knowledge areas such as testing and management (the majority of studies reviewed seem to focus on LLM-based software construction, n=17). Across HELM measures, LLM-based CGTs demonstrate notable but uneven progress. Accuracy is mostly discussed (n=18) and gains are well documented on standardized benchmarks but questioned by higher-quality studies that highlight questionable applicability to real-world tasks. Robustness (n=16) remains inconsistent across tasks/datasets/languages, and efficiency (n=14) constraints (computational, economic, and energetic) persist across studies. Toxicity and bias, though under-reported (n=8 and n=2, respectively), consistently appear as systemic risks, primarily linked to training data and uncurated code corpora. The scenarios to which these effects apply mostly involve such tasks as code generation, various code repair related generation, code completion and translation. The challenges synthesis underscores the barriers to reliable integration of LLM-based CGTs into development workflows. At a higher level, three dominant challenge clusters emerge: economic feasibility (n=10), evaluation validity (n=7), and socio-technical integration issues combining tooling/workflow and people/process concerns (n=12). In particular, high training and inference costs, energy consumption, and hardware and software constraints represent major obstacles to adoption. High-quality studies further emphasize that current benchmarks are often unrepresentative, Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 39 task definitions ambiguous, and that multi-metric frameworks are required to holistically capture performance, security, and human-in-the-loop factors. The prominence of socio-technical integration challenges—spanning limited IDE integration, poor operational control and responsiveness, trust, and organizational fit—highlights a persistent gap between experimental LLM-based CGTs and deployable software engineering tools. Meanwhile, legal, privacy, and security risks remain insufficiently reported and mitigated. Future research directions converge toward several priorities. The most widely endorsed involve domain-aware model adaptation through specialized datasets and fine-tuning, multi-dimensional and standardized evaluation frameworks, hybrid and ensemble approaches combining LLMs with traditional SE methods, and the inclusion of human-centered and industrial studies to enhance validity. Acknowledgments This work was supported by Science Foundation Ireland grant 13/RC/2094_2. References

[1] Association for Computing Machinery (ACM) [n. d.]. ACM Digital Library. Association for Computing Machinery (ACM). Accessed: 2025-09-12. [2] Schloss Dagstuhl – Leibniz-Zentrum für Informatik [n. d.]. dblp: Computer Science Bibliography. Schloss Dagstuhl – Leibniz-Zentrum für Informatik. Accessed: 2025-09-12. [3] Google [n. d.]. Google Scholar. Google. Accessed: 2025-09-12. [4] Institute of Electrical and Electronics Engineers (IEEE) [n. d.]. IEEE Xplore Digital Library. Institute of Electrical and Electronics Engineers (IEEE). Accessed: 2025-09-12. [5] OpenAI [n. d.]. OpenAI GPT-4. OpenAI. Accessed: 2025-09-12. [6] Elsevier [n. d.]. ScienceDirect. Elsevier. Accessed: 2025-09-12. [7] Elsevier [n. d.]. Scopus. Elsevier. Accessed: 2025-09-12. [8] Clarivate [n. d.]. Web of Science Core Collection. Clarivate. Accessed: 2025-09-12. [9] Areeg Ahmed, Shahira Azab, and Yasser Abdelhamid. 2023. Source-code generation using deep learning: a survey. In EPIA Conference on Artificial Intelligence. Springer, 467–482. [10] Gul Aftab Ahmed, James Vincent Patten, Yuanhua Han, Guoxian Lu, David Gregg, Jim Buckley, and Muslim Chochlov. 2023. Using Ensemble Inference to Improve Recall of Clone Detection. In 2023 IEEE 17th International Workshop on Software Clones (IWSC). IEEE, 15–21. [11] Domenico Amalfitano, Stefano Faralli, Jean Carlo Rossa Hauck, Santiago Matalonga, and Damiano Distante. 2023. Artificial intelligence applied to software testing: A tertiary study. Comput. Surveys 56, 3 (2023), 1–38. [12] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). [13] Abdulrahman Ahmed Bobakr Baqais and Mohammad Alshayeb. 2020. Automatic software refactoring: a systematic literature review. Software Quality Journal 28, 2 (2020), 459–502. [14] Jessica Bates, Paul Best, Janice McQuilkin, and Brian Taylor. 2017. Will web search engines replace bibliographic databases in the systematic identification of research? The Journal of Academic Librarianship 43, 1 (2017), 8–17. [15] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [16] Mohamad Adam Bujang and N Baharum. 2017. Guidelines of the minimum sample size requirements for Kappa agreement test. Epidemiol. Biostat. Public Health 14, 2 (2017). [17] M. C. and Coauthors. 2025. Supplementary Material (private Zenodo link under review). https://zenodo.org/records/17582721?preview=1&token= eyJhbGciOiJIUzUxMiJ9.eyJpZCI6ImZmZmFkNjQ2LTc1ZWQtNGRiYi04ZGVmLWQwOTAwMDU2YzRkNCIsImRhdGEiOnt9LCJyYW5kb20iOiJiMWFhN2Y2MzgwNTQ2YmFiZDRkMGMyM lm0Y2dGigtT7lxiQMjqjvFHn5-zDqC7ep2BB3b9g8sHIjwsD15NjPvPTfjSGP_joNa9BpKRcW5foAk_OV04R1g Private link provided to reviewers; DOI to be added upon publication.. [18] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21). 2633–2650. [19] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [20] Muslim Chochlov, Gul Aftab Ahmed, James Vincent Patten, Guoxian Lu, Wei Hou, David Gregg, and Jim Buckley. 2022. Using a nearest-neighbour, BERT-based approach for scalable clone detection. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 582–591. Manuscript submitted to ACM

40

Trovato et al.

[21] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113. [22] CORE. 2025. CORE Conference Portal. https://www.core.edu.au/conference-portal. Accessed: 2025-09-22. [23] Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2025. The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Available at SSRN 4945566 (2025). [24] Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt, and Wannes Meert. 2022. Code generation using machine learning: A systematic review. Ieee Access 10 (2022), 82434–82455. [25] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. [26] Thomas G Dietterich. 2000. Ensemble methods in machine learning. In International workshop on multiple classifier systems. Springer, 1–15. [27] Liming Dong, Qinghua Lu, and Liming Zhu. 2024. A pilot study in surveying data challenges of automatic software engineering tasks. In Proceedings of the 4th International Workshop on Software Engineering and AI for Data Quality in Cyber-Physical Systems/Internet of Things. 6–11. [28] Alvan R Feinstein and Domenic V Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of clinical epidemiology 43, 6 (1990), 543–549. [29] Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378. [30] International Organization for Standardization. 2023. Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model. ISO. [31] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419. [32] Gordon Fraser and Andrea Arcuri. 2012. Whole test suite generation. IEEE Transactions on Software Engineering 39, 2 (2012), 276–291. [33] Simon Cornelius Gorissen, Stefan Sauer, and Wolf G Beckmann. 2024. A survey of natural language-based editing of low-code applications using large language models. In International Conference on Human-Centred Software Engineering. Springer, 243–254. [34] Muhammet Kürşat Görmez, Murat Yılmaz, and Paul M Clarke. 2024. Large language models for software engineering: A systematic mapping study. In European Conference on Software Process Improvement. Springer, 64–79. [35] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017. Program synthesis. Foundations and Trends® in Programming Languages 4, 1-2 (2017), 1–119. [36] Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Michael Paget, and Christopher Naugler. 2024. Automated paper screening for clinical reviews using large language models: data analysis study. Journal of Medical Internet Research 26 (2024), e48996. [37] Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research synthesis methods 11, 2 (2020), 181–217. [38] Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. Brit. J. Math. Statist. Psych. 61, 1 (2008), 29–48. [39] Kilem L Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. [40] Neal Robert Haddaway, Alexandra Mary Collins, Deborah Coughlin, and Stuart Kirk. 2015. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS one 10, 9 (2015), e0138237. [41] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. [42] Xing Hu, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks. arXiv preprint arXiv:2505.08903 (2025). [43] Kai Huang, Zhengzi Xu, Su Yang, Hongyu Sun, Xuejun Li, Zheng Yan, and Yuqing Zhang. 2023. A survey on automated program repair techniques. arXiv preprint arXiv:2303.18184 (2023). [44] Zhang Huangzhao, Zhang Kechi, Li Zhuo, Li Jia, Li Yongmin, Zhao Yunfei, Zhu Yuqi, Liu Fang, Li Ge, and Jin Zhi. 2024. Deep learning for code generation: A survey. SCIENCE CHINA Information Sciences ISSN (2024). [45] Aleksi Huotala, Miikka Kuutila, Paul Ralph, and Mika Mäntylä. 2024. The promise and challenges of using LLMs to accelerate the screening process of systematic reviews. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 262–271. [46] Rasha Ahmad Husein, Hala Aburajouh, and Cagatay Catal. 2025. Large language models for code completion: A systematic literature review. Computer Standards & Interfaces 92 (2025), 103917. [47] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 (2024). [48] Sathvik Joel, Jie JW Wu, and Fatemeh H Fard. 2024. A survey on llm-based code generation for low-resource and domain-specific programming languages. arXiv preprint arXiv:2410.03981 (2024). [49] Barbara Kitchenham and Pearl Brereton. 2013. A systematic review of systematic review process research in software engineering. Information and software technology 55, 12 (2013), 2049–2075. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 41 [50] Barbara Kitchenham, Stuart Charters, et al. 2007. Guidelines for performing systematic literature reviews in software engineering. (2007). [51] Barbara Kitchenham, Lech Madeyski, and David Budgen. 2022. SEGRESS: Software engineering guidelines for reporting secondary studies. IEEE Transactions on Software Engineering 49, 3 (2022), 1273–1298. [52] Barbara Kitchenham, Rialette Pretorius, David Budgen, O Pearl Brereton, Mark Turner, Mahmood Niazi, and Stephen Linkman. 2010. Systematic literature reviews in software engineering–a tertiary study. Information and software technology 52, 8 (2010), 792–805. [53] Ron Kohavi et al. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, Vol. 14. Montreal, Canada, 1137–1145. [54] Zoe Kotti, Rafaila Galanopoulou, and Diomidis Spinellis. 2023. Machine learning for software engineering: A tertiary study. Comput. Surveys 55, 12 (2023), 1–39. [55] Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. [56] Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511 (2020). [57] J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. [58] Yunseo Lee, John Youngeun Song, Dongsun Kim, Jindae Kim, Mijung Kim, and Jaechang Nam. 2025. Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges. arXiv preprint arXiv:2504.20799 (2025). [59] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019). [60] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092–1097. [61] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022). [62] Jialiang Lin, Yao Yu, Yu Zhou, Zhiyang Zhou, and Xiaodong Shi. 2020. How many preprints have actually been printed and why: a case study of computer science preprints on arXiv. Scientometrics 124, 1 (2020), 555–574. [63] Hsiao-Chuan Liu, Chia-Tung Tsai, and Min-Yuh Day. 2024. A Pilot Study on AI-Assisted Code Generation with Large Language Models for Software Engineering. In International Conference on Technologies and Applications of Artificial Intelligence. Springer, 162–175. [64] Kui Liu, Li Li, Anil Koyuncu, Dongsun Kim, Zhe Liu, Jacques Klein, and Tegawendé F Bissyandé. 2021. A critical review on the evaluation of automated program repair systems. Journal of Systems and Software 171 (2021), 110817. [65] Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282. [66] Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024). [67] David Moher, Larissa Shamseer, Mike Clarke, Davina Ghersi, Alessandro Liberati, Mark Petticrew, Paul Shekelle, Lesley A Stewart, and Prisma-P Group. 2015. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews 4, 1 (2015), 1. [68] Martin Monperrus. 2014. A critical review of" automatic patch generation learned from human-written patches": Essay on the problem statement and the evaluation of automatic software repair. In Proceedings of the 36th International Conference on Software Engineering. 234–242. [69] Robert G Newcombe. 1998. Two-sided confidence intervals for the single proportion: Comparison of seven methods. Statistics in Medicine 17, 8 (1998), 857–872. [70] Björn Nykvist, Biljana Macura, Maria Xylia, and Erik Olsson. 2025. Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis. Environmental Evidence 14, 1 (2025), 7. [71] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730–27744. [72] Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic reviews 4, 1 (2015), 5. [73] Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601 (2021). [74] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 (2023). [75] Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023). [76] Jorge Pérez, Jessica Díaz, Javier Garcia-Martin, and Bernardo Tabuenca. 2020. Systematic literature reviews in software engineering—enhancement of the study selection process using Cohen’s Kappa statistic. Journal of Systems and Software 168 (2020), 110657. [77] Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and software technology 64 (2015), 1–18. Manuscript submitted to ACM

42

Trovato et al.

[78] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67. [79] Leonardo Criollo Ramírez, Xavier Limón, Ángel J Sánchez-García, and Juan Carlos Pérez-Arriaga. 2024. State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review. In 2024 12th International Conference in Software Engineering Research and Innovation (CONISOFT). IEEE, 331–339. [80] Rolando Ramirez-Rueda, Edgard Benítez-Guerrero, Carmen Mezura-Godoy, and Everardo Barcenas. 2024. A Systematic Literature Review of 10 years of Research on Program Synthesis and Natural Language Processing. Programming and Computer Software 50, 8 (2024), 725–741. [81] Stuart Russell and Peter Norvig. 2021. Artificial Intelligence: a modern approach, 4th US ed. aima: site. URL: https://aima. cs. berkeley. edu/(Date: 26.02. 2023) (2021). [82] Yuya Sasaki, Hironori Washizaki, Jialong Li, Dominik Sander, Nobukazu Yoshioka, and Yoshiaki Fukazawa. 2024. Systematic literature review of prompt engineering patterns in software engineering. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 670–675. [83] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering 50, 1 (2023), 85–105. [84] SCImago. 2025. SCImago Journal & Country Rank. https://www.scimagojr.com/. Accessed: 2025-09-22. [85] David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in neural information processing systems 28 (2015). [86] Xinyu She, Yue Liu, Yanjie Zhao, Yiling He, Li Li, Chakkrit Tantithamthavorn, Zhan Qin, and Haoyu Wang. 2023. Pitfalls in language models for code intelligence: A taxonomy and survey. ACM Transactions on Software Engineering and Methodology (2023). [87] Julius Sim and Chris C Wright. 2005. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical therapy 85, 3 (2005), 257–268. [88] Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 13693–13696. [89] Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. 1433–1443. [90] Catherine Tony, Nicolás E Díaz Ferreyra, Markus Mutas, Salem Dhiff, and Riccardo Scandariato. 2024. Prompting techniques for secure code generation: A systematic investigation. arXiv preprint arXiv:2407.07064 (2024). [91] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [92] Mardi Siswo Utomo, Ema Utami, Arief Setyanto, et al. 2024. Machine Learning Innovations in Code Generation: A Systematic Literature Review of Methods, Challenges and Directions. In 2024 International Conference on Information Technology and Computing (ICITCOM). IEEE, 24–29. [93] Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, et al. 2020. ASReview: open source software for efficient and transparent active learning for systematic reviews. (2020). [94] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [95] Antonio Vitale, Rocco Oliveto, and Simone Scalabrino. 2025. A catalog of data smells for coding tasks. ACM Transactions on Software Engineering and Methodology 34, 4 (2025), 1–32. [96] Siw Waffenschmidt, Marco Knelangen, Wiebke Sieben, Stefanie Bühn, and Dawid Pieper. 2019. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC medical research methodology 19, 1 (2019), 132. [97] Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. 2010. Semi-automated screening of biomedical citations for systematic reviews. BMC bioinformatics 11, 1 (2010), 55. [98] William H Walters. 2009. Google Scholar search performance: Comparative recall and precision. portal: Libraries and the Academy 9, 1 (2009), 5–24. [99] Yao Wan, Zhangqian Bi, Yang He, Jianguo Zhang, Hongyu Zhang, Yulei Sui, Guandong Xu, Hai Jin, and Philip Yu. 2024. Deep learning for code intelligence: Survey, benchmark and toolkit. Comput. Surveys 56, 12 (2024), 1–41. [100] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936. [101] Hironori Washizaki, Maria-Isabel Sanchez-Segura, Juan Garbajosa, Steve Tockey, Kenneth E. Nidiffer, and Annette D. Reilly. 2025. Guide to the Software Engineering Body of Knowledge (SWEBOK) Version 4.0a. IEEE Computer Society. [102] Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency. 214–229. [103] Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering. 1–10. Manuscript submitted to ACM

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions 43 [104] Affan Yasin, Rubia Fatima, Lijie Wen, Wasif Afzal, Muhammad Azhar, and Richard Torkar. 2020. On using grey literature and google scholar in systematic literature reviews in software engineering. IEEE access 8 (2020), 36226–36243. [105] Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023. A survey of learning-based automated program repair. ACM Transactions on Software Engineering and Methodology 33, 2 (2023), 1–69. [106] Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A systematic literature review on large language models for automated program repair. arXiv preprint arXiv:2405.01466 (2024). [107] Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2023. A survey on large language models for software engineering. arXiv preprint arXiv:2312.15223 (2023). [108] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 1, 2 (2023). [109] Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. 2023. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372 (2023). [110] Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an understanding of large language models in software engineering tasks. Empirical Software Engineering 30, 2 (2025), 50. [111] Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2025. Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1–31. [112] Fida Zubair, Maryam Al-Hitmi, and Cagatay Catal. 2025. The use of large language models for program repair. Computer Standards & Interfaces 93 (2025), 103951.

Manuscript submitted to ACM

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500