Deep Graph-Language Fusion for Structure-Aware Code Generation Mert Tiftikci∗†
Amir Molzam Sharifloo
Mira Mezini∗†
[email protected] Technische Universität Darmstadt Darmstadt, Germany
[email protected] Technische Universität Darmstadt Darmstadt, Germany
[email protected] Technische Universität Darmstadt Darmstadt, Germany
arXiv:2605.03689v1 [cs.SE] 5 May 2026
Abstract Pre-trained Language Models (PLMs) have the potential to transform software development tasks. However, despite significant advances, current PLMs struggle to capture the structured and relational attributes of code, such as control flow and data dependencies. This limitation is rooted in an architectural mismatch: whereas code structure is best represented by graphs, transformer-based LLMs process input as sequential token patterns and therefore lack explicit structural awareness. While recent research has explored integrating graph-based code representations using techniques like graph feature extraction, retrieval-augmented generation, and prompt engineering, existing approaches suffer from information loss during dense feature extraction or prompt encoding; notably, the potential of deep, token-level fusion of graph features within model internals has not been systematically explored. In this paper, we initiate such an exploration by introducing CGFuse1 , a novel framework that enables token-level integration of graph-derived representations by infusing learned graph features directly into the intermediate layers of pre-trained language models. CGFuse combines a graph neural network (GNN) with a language model to explicitly preserve and exploit fine-grained structural information from code graphs, including abstract syntax trees and data-flow graphs. We systematically evaluate CGFuse across multiple LLMs, demonstrating up to 10–16% BLEU and 6–11% CodeBLEU improvements in code generation performance. These results highlight the potential of deep graph-PLM integration to advance the field toward more robust, capable AI-driven software development.
CCS Concepts • Software and its engineering → Software notations and tools; • Computing methodologies → Neural networks.
Keywords Large Language Models, Code Generation, Graph Neural Network, Code Graphs ACM Reference Format: Mert Tiftikci, Amir Molzam Sharifloo, and Mira Mezini. 2026. Deep GraphLanguage Fusion for Structure-Aware Code Generation. In 2026 IEEE/ACM ∗ Also with The Hessian Center for Artificial Intelligence (hessian.AI). † Also with National Research Center for Applied Cybersecurity ATHENE. 1 The implementation is available at https://github.com/stg-tud/cgfuse.
This work is licensed under a Creative Commons Attribution 4.0 International License. FORGE ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2477-0/2026/04 https://doi.org/10.1145/3793655.3793724
Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3793655.3793724
1
Introduction
Large Language Models (LLMs) are rapidly transforming software engineering practice. State-of-the-art systems such as ChatGPT, Gemini, and CodeLLaMA now deliver significant capabilities in code generation, bug detection, and automated testing. Their integration is advancing the field toward fully automated software development [17, 24]. However, recent studies reveal that, despite their strong performance on many tasks, these models struggle to explicitly capture the structured and relational nature of code, such as control flow and data dependencies [2, 16]. We posit that this limitation stems from a fundamental architectural mismatch: while graph-based representations naturally encode structural relationships, transformer architectures primarily operate on sequential token patterns, providing only implicit structural awareness. To bridge this gap, several approaches have been proposed that integrate graph-based code representations through techniques such as graph-based feature extraction, retrieval-augmented generation (RAG), and prompt engineering. Some approaches operate at the repository level, creating complex hybrid code graphs that incorporate project file structures and sophisticated RAG systems [14, 22]. At the token-and-model level, earlier work such as GraphCodeBERT [10] and UniXcoder [9] incorporate Abstract Syntax Trees (ASTs)[12] and data-flow graphs by flattening them as prompts and adding graph-based auxiliary training objectives for full pertaining. More recent approaches [7, 26] employ soft prompting, wherein GNN-generated features are appended to the input prompt and the model is fine-tuned accordingly. While this approach enables direct learning from graphs, it incurs information loss during dense feature extraction and prompt encoding, limiting the depth of graph-LLM fusion. Pass-Tuning [5] advances this direction by using GNN-based experts within a prefix-tuning setup to achieve deeper fusion at the model level. However, it still relies on dense graph representations, which constrains the model’s capacity to capture rich and fine-grained relational code patterns. Notably, existing approaches have not systematically explored token-level fusion of extracted graph features. This gap raises an important research question: can deeper integration achieved by infusing graph knowledge directly into the internal layers of pretrained language models (PLMs) better preserve and exploit finegrained structural information to enhance model capabilities on coding tasks? Towards addressing this question, we developed CGFuse, a novel framework that seamlessly integrates graph-based representations of code into PLMs to enhance their understanding
FORGE ’26, April 12–13, 2026, Rio de Janeiro, Brazil
Mert Tiftikci, Amir Molzam Sharifloo, and Mira Mezini
/* * * Actually walks the bag to make sure the 3 * count is correct and resets the running total 4 * @return the current total size 5 */
int calcTotalSize () { _total = extractList () . size () ; 3 return _total ; 4 }
1
1
2
2
(b) Example Java code. (a) Example documentation. method_declaration Token Node
block expression_statement
AST Syntax Node
Gold Output
assignment_statement method_invocation
AST edge method_invocation formal_parameters
int
calc
TotalSize
Sub Tokens
(
)
argument_list
{
_
total Sub Tokens
=
extract
List
(
)
Sub Tokens
argument_list
.
size
(
)
DF edge ComingFrom
return_statement
;
Context
return Next Token to be Generated
_
total
;
}
DF edge CalculatedFrom
Sub Tokens
(c) Code graph with AST (solid lines) and DFG edges (dashed/dotted). Syntax nodes 𝑠 are orange, terminal nodes 𝑡 are green.
Figure 1: A sample Java snippet with documentation (a), code (b), and its augmented code graph (c). and generation capabilities. Our approach trains a graph neural network to process code graphs and infuses its learned representations directly into the intermediate layers of PLMs, enabling token-level fusion. Specifically, our main contributions are as follows: Novel Framework for Deep Graph-PLM Integration: We propose CGFuse, the first framework that systematically fuses graph neural networks (GNNs) with pre-trained language models (PLMs) at intermediate layers, enabling PLMs to directly exploit fine-grained structural and relational information from code graphs such as ASTs and control/data-flow graphs at the token level. Systematic Evaluation Across Models and Architectures: We conduct a systematic evaluation across nine selected PLMs – including Encoder and Encoder-Decoder architectures – and three GNN architectures, demonstrating that CGFuse improves code generation performance and effectively leverages structural information.
2
Our Approach
We aim to enhance code generation quality by combining GNNs’ structural understanding with transformer-based PLMs’ contextual knowledge. Our approach consists of three stages: (1) constructing code graphs from source code, (2) pre-training GNNs as graph experts, and (3) fusing GNN representations into a PLM for downstream tasks.
2.1
Constructing Code Graphs
To capture both syntactic and semantic dependencies in code, we represent each snippet as a code graph G𝑐 . This graph is built by augmenting an Abstract Syntax Tree (AST) with data flow edges [10]. The AST encodes the syntactic structure of the program, while data flow edges model how values propagate through variables. Figure 1 illustrates a sample Java snippet and its corresponding graph. Terminal nodes (T , green) represent code tokens, while syntax nodes (S, orange) capture higher-level constructs such as expressions and statements. Edges include AST hierarchy edges (solid lines) and data flow edges (dashed/dotted lines) labeled comingFrom and calculatedBy, representing the origin and computation
of variables, respectively. Formally, the graph is defined as: G𝑐 = (V, E, R),
V = S ∪ T,
R = {𝑝, 𝑐𝑜, 𝑐𝑎},
where 𝑝 denotes AST edges, 𝑐𝑜 denotes comingFrom, and 𝑐𝑎 denotes calculatedBy edges.
2.2
Graph Expert Pre-training
We pre-train a GNN to serve as a graph expert, learning nodelevel representations for both syntax and terminal nodes. The GNN aggregates features from each node’s neighborhood N (𝑣): 𝑎𝑙𝑣 = aggregate𝑙 ({ℎ𝑢𝑙 −1 : 𝑢 ∈ N (𝑣)}), ℎ𝑙𝑣 = combine𝑙 (ℎ𝑙𝑣−1, 𝑎𝑙𝑣 ),
(1)
where ℎ 0𝑣 are initial node embeddings, which include PLM token embeddings for terminal nodes and learned embeddings for syntax nodes. For graphs with multiple relation types (AST, comingFrom, calculatedBy), we optionally use relational aggregation [11]. The GNN is trained on a node classification task, learning to predict properties of each node from its local context, capturing structural and semantic code patterns. We initialize the GNN with PLM embeddings for terminal nodes and random embeddings for syntax nodes, and pre-train on node classification, masking a portion of nodes for validation/testing.
2.3
Fusing GNN and PLM Features
After pre-training, we integrate GNN embeddings into a transformerbased PLM. Figure 2 illustrates this fusion mechanism. At a target layer 𝑙, the transformer produces token representations ℎ𝑙𝜃 , while the GNN outputs node representations ℎ𝑔𝑙 . We fuse them as: ℎ𝑔𝑙 = GNN𝑙 (G𝑐𝑙 ),
ℎ𝑙𝜃 = 𝜃 𝑙 (ℎ𝑙𝑓−1 ),
ℎ𝑙𝑓 = ℎ𝑙𝜃 + 𝜆ℎ𝑔𝑙 ,
(2)
where 𝜆 controls fusion strength. Only nodes in the current context are fed into the GNN; out-of-context edges are masked. As for the implementation, we first attach the pre-trained GNN to target PLM layers and train it alone for several warmup epochs.
Deep Graph-Language Fusion for Structure-Aware Code Generation
FORGE ’26, April 12–13, 2026, Rio de Janeiro, Brazil
Masked edges and nodes ...
_
=
extract
... + ...
_ total = + + + _ total =
total
extract + extract
List
(
)
.
size
(
)
...
assignment_statement
method_invocation
method_invocation argument_list argument_list ... _
total
=
extract
List
(
)
.
size
(
)
...
...
_ total
=
extract
List
(
)
.
size
(
)
...
Table 1: Code Models evaluated in this work. We evaluate two types of architectures (Arch): encoder (E) and encoderdecoder (ED) models. In addition to natural language inputs (NL), models can be trained with code (C) or flattened AST inputs (AST 𝑓 ), where the subscript indicates flattening. For all encoder models converted to encoder-decoder models, the number of parameters is specified for the encoder-decoder versions.
Shared ... + ...
_ total = + + + _ total =
extract + extract
Figure 2: Fusion of GNN features into a PLM layer. Only context nodes are included and edges outside the context are masked. Then, we jointly fine-tune the fused model on the downstream task, optimizing both PLM and GNN parameters.
3
Experiments
In this section, we first present the experimental setup, including the datasets, model configurations, and evaluation metrics used to assess the effectiveness of our approach. We then describe the experiments and report the results to demonstrate the impact of incorporating graph-based representations on model performance.
3.1
Experimental Setup
Baseline Models: We apply our approach to various architectures, trained with natural language, code, and flattened AST inputs. Table 1 provides a comprehensive list of the models used in our experiments. We construct encoder-decoder models for our experiments by adapting encoder-only models. As a result, the parameter counts reported for these models correspond to their encoder-decoder versions, which approximately double the parameters of the original encoder models. GNN experts: We investigated the impact of different GNN architectures and the number of intermediate layers. Specifically, we experimented with the Relational Graph Convolutional Network (R-GCN) [11], the GraphSAGE model (GS) [11], and the Graph Isomorphism Network (GIN) [25]. Dataset: For our experiments, we use the CONCODE dataset2 – an established benchmark for evaluating code generation models. CONCODE comprises 104k Java classes (100k for training and 2k each for validation and testing) collected from GitHub. The code generation task takes a natural-language intent (see Fig. 1) and generates code to fulfill it. Metrics: For evaluation, we use three string-matching-based metrics: Exact Match (EM): Measures the percentage of predictions that exactly match the ground truth. BLEU (B): A widely used metric for evaluating text generation tasks, introduced by Papineni et al. [18]. CodeBLEU (CB): An extension of BLEU tailored for code, which incorporates syntactic and semantic features [21]. It aggregates four complementary signals of n-gram match BLEU, weighted n-gram match that up-weights language keywords, syntax match computed
Models
Param
Arch
Data
BERT [6] RoBERTa [15] BART [13] T5 [19]
255M 285M 143M 226M
E E ED ED
NL NL NL NL
CodeBERT [8] GraphCodeBERT [10] UniXCoder [9] PLBART [1] CodeT5 [23]
277M 277M 280M 139M 222M
E E E ED ED
C C + DFG 𝑓 C + AST 𝑓 C C
over ASTs, and semantic match computed over DFGs, all contribute equally. Implementation details: All experiments were conducted with a single run and the same seed value, using a single-layer GNN expert attached to the decoder’s last layer. All code graphs, except for sub-tokens, are generated before the training. Sub-tokens are generated during model training and testing using the corresponding tokenizer. The GNNs are pretrained for three epochs on these code graphs, which is sufficient to achieve approximately 99% accuracy.
3.2
Results
Our results in Table 2 reveal several key observations: Strong gains from GNN augmentation for natural language models: Models pre-trained solely on natural language benefit substantially from GNN-based code graph infusion. For example, BERT improves +15.1 BLEU (33.4 → 48.5) and +11.9 CodeBLEU (31.2 → 43.1) when combined with a 1-layer R-GCN. This demonstrates that structural embeddings effectively infuse code knowledge into models that were not exposed to code during pre-training. Selected NL models served as the base checkpoints from which the corresponding code-pretrained transformer models are initialized. To assess sample efficiency, we apply CGFuse to RoBERTa and BART. The initialization chains are: RoBERTa → CodeBERT → GraphCodeBERT, (3) RoBERTa → UniXcoder, BART → PLBART Comparing the underlined entries in Table 2 shows that CGFuseenhanced NL models outperform their code-pretrained counterparts, despite using substantially fewer training samples and iterations (see Table 3)3 . Significant improvements for code-pretrained models: PLBART, CodeBERT, GraphCodeBERT, and UniXCoder consistently improve across all metrics when augmented with a GNN. In particular,
2 Dataset available at https://huggingface.co/datasets/AhmedSSoliman/CodeXGLUE-
3 A notable limitation of T5 [20] is its limited vocabulary. This makes CodeBLEU evalu-
CONCODE.
ation significantly unreliable; therefore, the T5 family is omitted from this comparison.
FORGE ’26, April 12–13, 2026, Rio de Janeiro, Brazil
Mert Tiftikci, Amir Molzam Sharifloo, and Mira Mezini
Table 2: Code generation results for baseline and fused models. For each fused model, the GNN architecture is specified with a subscript. 𝜆 represents the fusing strength; see Eq. (2).
Programming Language
Natural Language
Models BERT BERT𝑅−𝐺𝐶𝑁 RoBERTa RoBERTa𝐺𝐼 𝑁 BART BART𝐺𝐼 𝑁 T5 T5𝐺𝐼 𝑁 CodeBERT CodeBERT𝐺𝐼 𝑁 GraphCodeBERT GraphCodeBERT𝐺𝑆 UniXCoder UniXCoder𝐺𝐼 𝑁 PLBART PLBART𝑅−𝐺𝐶𝑁 CodeT5 CodeT5𝐺𝐼 𝑁
𝜆 1 0.5 0.5 0.5
1 1 1 1 1
BLEU
CodeBLEU
EM
33.4 48.5 36.8 52.6 36.9 52.1 33.5 40.0
31.2 43.1 33.1 45.0 31.3 42.7 29.9 31.7
8.0 10.1 11.6 15.6 15.4 20.4 15.2 19.5
36.9 54.1 49.3 56.4 40.5 57.5 42.1 59.3 42.7 60.0
34.4 46.2 35.4 47.3 35.9 49.5 36.6 51.5 36.8 51.5
11.1 19.0 13.7 20.0 14.0 21.2 18.2 27.6 18.2 30.3
PLBART shows greater improvement than the other models, with differences of +17.2, +14.9, and +9.4 on BLEU, CodeBLEU, and EM scores, respectively. Overall, these results demonstrate that GNNbased code graph embeddings can improve code generation accuracy. We further investigated how the number of layers in a GNN affects performance on the code-generation task. We experimented with 1, 2, and 3 intermediate GNN layers to assess the effect of depth. Each additional layer increases the node’s receptive field, allowing aggregation from more distant neighbors, but it also adds parameters and increases training cost. Results (Table 4) show that single-layer GNNs consistently outperform deeper variants across all tested architectures. Adding a second or third layer leads to a substantial drop in scores, suggesting that deeper GNNs may introduce noise or over-smooth node representations, thereby reducing their usefulness for code generation. Among the models, R-GCN achieves the highest scores in BLEU and CodeBLEU, whereas GIN achieves the highest score in EM, both with a single layer. Overall, these results indicate that shallow GNN embeddings provide the most effective structural information for code generation.
4
Conclusion and Future Directions
In this work, we investigated how code PLMs can be enriched with representations generated by GNN experts at the token and layer levels. We evaluated our approach on various code and natural language PLMs in the code generation task, experimenting with nine different models pre-trained on both natural and programming languages, covering different architectures and training
Table 3: The table shows the difference in training time (normalized #Step), the amount of pretraining code data that models are exposed to (#Sample), and the performance gains of NL-based models when CGFuse is applied, compared to their corresponding code model counterparts (e.g., GraphCodeBERT vs. RoBERTa𝐺𝐼 𝑁 ; see Eq. (3)). Models CodeBERT GraphCodeBERT UniXcoder PLBART
×#Step ×137 ×274 ×547 ×293
+#Sample ∼8M ∼12M ∼36M ∼575M
-ΔB 15.7 13.3 12.1 10.0
-ΔCB 10.6 9.6 9.1 6.1
-ΔEM 4.5 1.9 1.6 2.2
Table 4: The average code generation results of CGFuse using different GNN architectures and numbers of layers on 9 PLMs at the last decoder layer (𝜆 = 1). The numbers in the model names indicate the number of GNN layers (L).
Models
1-L
BLEU 2-L 3-L
CodeBLEU 1-L 2-L 3-L
EM 1-L 2-L 3-L
R-GCN 53.3 27.9 24.3 46.0 28.4 27.7 17.5 1.2 1.3 GS 52.4 29.4 23.9 44.6 30.5 28.0 18.0 1.1 1.4 GIN 52.0 14.1 12.1 44.3 20.4 20.0 20.0 1.7 2.0
strategies. In general, we achieved better performance across all models. Moreover, the comparison between the NL models and their code counterparts shows significant sample efficiency in learning. The encouraging results presented in the paper motivate further pursuing the research in several directions. First, developing strategies for efficient GNN training, handling incomplete code during decoding, and mitigating ambiguity in AST generation are promising avenues for further improvement. Training a new GNN for each architectural or vocabulary change is computationally costly and integrating GNN experts into decoders for code generation tasks is challenging, as code graphs must often be constructed from incomplete or ambiguous code, complicating the generation process. Second, applying the presented approach to larger PLMs could test its scalability and benefits for more complex generation tasks. Finally, evaluating performance on benchmarks that measure functional correctness such as HumanEval [4] and MBPP [3] would provide a more rigorous assessment of practical utility.
Acknowledgments We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003) and also by the National Research Center for Applied Cybersecurity ATHENE within the project Athene SecureCoder, and by the LOEWE initiative (Hesse, Germany) [LOEWE/4a//519/05/00.002(0013)/95]. The work has benefited from the early stages of the funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy— EXC-3057.
Deep Graph-Language Fusion for Structure-Aware Code Generation
References [1] Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 2655–2668. doi:10.18653/v1/2021.naacl-main.211 [2] Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini. 2024. A Critical Study of What Code-LLMs (Do Not) Learn. In Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 15869–15889. doi:10.18653/v1/2024. findings-acl.939 [3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs] doi:10.48550/arXiv.2107.03374 [5] Nuo Chen, Qiushi Sun, Jianing Wang, Xiang Li, and Ming Gao. 2023. Pass-Tuning: Towards Structure-Aware Parameter-Efficient Tuning for Code Representation Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 577–591. doi:10. 18653/v1/2023.findings-emnlp.42 [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805 [7] Kounianhua Du, Jizheng Chen, Renting Rui, Huacan Chai, Lingyue Fu, Wei Xia, Yasheng Wang, Ruiming Tang, Yong Yu, and Weinan Zhang. 2025. CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation. arXiv:2405.02355 [cs] doi:10.48550/ arXiv.2405.02355 [8] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020). [9] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 7212–7225. doi:10.18653/v1/2022.acl-long.499 [10] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. In International Conference on Learning Representations. [11] William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 1025–1035. [12] Donald E Knuth. 1968. Semantics of context-free languages. Mathematical systems theory 2, 2 (1968), 127–145. [13] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. CoRR abs/1910.13461 (2019). arXiv:1910.13461 http://arxiv.org/abs/1910.13461 [14] Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. GraphCoder: Enhancing Repository-Level Code Completion via Coarse-to-fine Retrieval Based on Code Context Graph. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24). Association for Computing Machinery, New York, NY, USA, 570–581. doi:10.1145/3691620.3695054 [15] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692 http://arxiv.org/abs/1907.11692
FORGE ’26, April 12–13, 2026, Rio de Janeiro, Brazil
[16] Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, and Yang Liu. 2024. Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities. ACM Trans. Softw. Eng. Methodol. 33, 7, Article 169 (Aug. 2024), 29 pages. doi:10.1145/3664606 [17] Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2025. SWE-GPT: A Process-Centric Language Model for Automated Software Improvement. Proceedings of the ACM on Software Engineering 2, ISSTA (June 2025), 2362–2383. doi:10.1145/3728981 [18] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02. Association for Computational Linguistics, Philadelphia, Pennsylvania, 311. doi:10.3115/1073083.1073135 [19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. [20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551. [21] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: A Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs] [22] Hongyuan Tao, Ying Zhang, Zhenhao Tang, Hongen Peng, Xukun Zhu, Bingchang Liu, Yingguang Yang, Ziyin Zhang, Zhaogui Xu, Haipeng Zhang, Linchao Zhu, Rui Wang, Hang Yu, Jianguo Li, and Peng Di. 2025. Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks. arXiv:2505.16901 [cs] doi:10.48550/arXiv.2505.16901 [23] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 8696–8708. doi:10.18653/v1/ 2021.emnlp-main.685 [24] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering 2, FSE (June 2025), 801–824. doi:10.1145/3715754 [25] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How Powerful Are Graph Neural Networks?. In International Conference on Learning Representations. [26] Ziyin Zhang, Hang Yu, Shijie Li, Peng Di, Jianguo Li, and Rui Wang. 2024. GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding. arXiv:2409.04183 [cs]