arXiv:2605.02256v1 [cs.SE] 4 May 2026
CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation Zirui Wan
Zhaonan Wu
Xinyi Hou
Huazhong University of Science and Technology Wuhan, China [email protected]
Huazhong University of Science and Technology Wuhan, China [email protected]
Huazhong University of Science and Technology Wuhan, China [email protected]
Yanjie Zhao∗
Pengcheng Xia
Haoyu Wang
Huazhong University of Science and Technology Wuhan, China [email protected]
Huazhong University of Science and Technology Wuhan, China [email protected]
Huazhong University of Science and Technology Wuhan, China [email protected]
Abstract
1
High-quality commit messages are critical for maintaining software projects, yet ensuring their consistency and informativeness remains a practical challenge. While the Conventional Commits Specification (CCS) provides a structured format for commit messages, research on CCS-based commit classification and commit message generation (CMG) is limited by the absence of large-scale benchmarks, semantic annotations, and reliable evaluation methods. In this paper, we introduce CommitSuite, a benchmark comprising 63,533 CCS-compliant commits from 243 open-source repositories across seven programming languages. Each commit is labeled with its CCS type and enriched with AST-level code changes, along with LLM-assisted semantic annotations that capture the “what” and “why” behind the change. To evaluate CMG systems, we propose a reference-free framework based on five binary metrics: rationality, comprehensiveness, non-redundancy, authenticity, and logicality, enabling semantic-level assessment without relying on human-written references. Our experiments show that LLMs can effectively support both generation and evaluation, with evaluation achieving 0.849 Cohen’s Kappa agreement against human judgments. CommitSuite offers a unified resource for structured commit understanding and facilitates reproducible research on commit classification and generation.
Commit messages play a fundamental role in software development workflows. In version control systems such as Git, a commit records an atomic change to the codebase, including both the code modifications and a message that conveys the intent and rationale behind the change. Besides, commits form the historical record of a project, allowing developers to trace code evolution, understand past decisions, and collaborate effectively in distributed environments [12]. Well-written commit messages enhance code maintainability, facilitate debugging and impact analysis, and improve team communication [20]. They provide essential context for future readers, aid in fault localization, and document the development process in a reusable form. In large-scale or long-lived projects, commit messages serve as a critical knowledge base that complements the source code. As such, commit message generation (CMG) has become an active research area focused on automatically producing high-quality messages that reflect both the content and the intent of code changes. In practice, commit messages are often inconsistent, lack detail, or are omitted due to time pressure or unclear guidelines [12]. This reduces the usefulness of commit histories and hinders tasks such as code review, changelog generation, and semantic versioning - a standardized approach where commit types (e.g., feat, fix) automatically determine version number increments (major.minor.patch) [2, 4, 30]. Aiming to improve the structure and quality of commit messages, the Conventional Commits Specification (CCS) [30] has been widely adopted in software development. CCS enforces a structured format, typically <type>(<scope>): <description>, where type (e.g., feat, fix) indicates the nature of the change and scope specifies the affected component. This format enhances both human readability and machine interpretability, enabling automated tools for commit classification, changelog generation, and semantic versioning. Currently, numerous high-impact repositories adopt CCS, including puppeteer/puppeteer (91.6k stars), shadcn-ui/ui (92.1k stars), and gatsbyjs/gatsby (55.9k stars). This widespread adoption underscores CCS’s vital role in establishing standardized, machine-interpretable commit practices essential for modern software collaboration.
ACM Reference Format: Zirui Wan, Zhaonan Wu, Xinyi Hou, Yanjie Zhao, Pengcheng Xia, and Haoyu Wang. 2026. CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn. nnnnnnn ∗ Corresponding author ([email protected]).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, Scotland, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Introduction
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Although CCS provides a clear structure for commit messages, research on CCS-based commit classification and CMG still faces several key challenges. One major limitation is the lack of largescale, high-quality benchmarks and tools for fine-grained classification across the full CCS type set [21, 39]. Most existing work focuses on binary or coarse-grained categories, leaving the more nuanced classification tasks underexplored. In addition, many CMG datasets suffer from low standardization and lack semantic annotations that capture both the “what” and “why” of code changes. They also often omit fine-grained contextual information, such as function-level or structural representations, which are essential for generating accurate and informative messages. Finally, evaluation remains a major bottleneck. Current methods rely heavily on reference-based metrics like BLEU and ROUGE-L, which are highly sensitive to inconsistencies in human-written messages. These metrics often fail to reflect semantic correctness, informativeness, or adherence to CCS format [13, 28], limiting the reliability and comparability of CMG systems. To address these limitations, we present CommitSuite, a largescale benchmark for fine-grained CCS-based commit classification and message generation. It comprises 63,533 high-quality commits from 243 open-source repositories, all strictly following the CCS format and covering seven popular programming languages. Each commit is labeled with its CCS type and enriched with functionlevel and AST-level structural changes. We also use large language models (LLMs) to semi-automatically annotate each commit with “what” and “why” semantics to support more informative generation. Beyond providing this comprehensive dataset and structural annotations, we propose a reference-free evaluation framework with binary metrics, which consist of rationality, comprehensiveness, non-redundancy, authenticity and logicality. We show that LLMs can reliably perform these evaluations, enabling scalable and objective assessment of CMG systems. We further utilize CommitSuite to conduct extensive evaluations, benchmarking the performance of state-of-the-art commit message generation tools and leading LLMs on both commit classification and message generation tasks. Our main contributions are as follows:
• A novel, high-quality benchmark dataset. We introduce CommitSuite, comprising 63,533 strictly CCS-compliant commits (243 repos, 7 languages) enriched with AST-level changes and LLM-assisted “what”/“why” annotations. • A reference-free CMG evaluation framework. We propose five binary metrics (rationality, comprehensiveness, non-redundancy, authenticity, logicality) for semantic assessment without human references, validated via LLM automation. • Comprehensive performance benchmarking. Evaluations using CommitSuite show LLMs significantly outperform traditional tools in CMG (especially on semantic metrics), while highlighting challenges in fine-grained classification. CommitSuite’s high quality and CCS alignment enable reliable and reproducible benchmarking. • Open-source and reproducibility. All datasets, evaluation tools, and baseline results are publicly released to support reproducible research at: https://github.com/security-pride/CommitSuite.
Z. Wan, Z. Wu, X. Hou, Y. Zhao, P. Xia, and H. Wang
2 Background and Motivation 2.1 Commit Message 2.1.1 Commit Message Generation (CMG). A commit represents a recorded change to one or more files in a code repository along with a descriptive message, known as commit message. The actual modifications are documented in the diff section, which highlights added, removed, or altered lines of code. CMG refers to the automated generation of commit messages from code diffs and has become an important technique in modern software development. High-quality commit messages are essential for helping developers understand both the content and motivation behind code changes, thereby facilitating effective collaboration and long-term maintenance. Traditional CMG methods, such as retrieval-based and learning-based approaches, often struggle to produce messages that are accurate, complete, and readable [34]. With the rapid development of large language models (LLMs), there is growing interest in applying LLMs to code-related tasks, including CMG. Recent studies [7, 13, 38, 40, 41] have shown that LLMs excel at generating informative and context-aware commit messages, due to their advanced language understanding and generation capabilities. 2.1.2 Conventional Commits Specification (CCS). CCS v1.0.0 was released in 2019 [30, 31], which provides a standardized convention for writing commit messages, aiming to make them easily understood by both humans and machines. CCS defines a specific message format, as shown in Figure 1, which requires each commit message to include fields such as <type>, <scope>, and <description>. Among these, the <type> field describes the nature of the code change, with common values including feat (new feature), fix (bug fix), and docs (documentation update), among others.The primary goal of CCS is to enhance the readability and maintainability of code changes by enforcing a consistent structure for commit messages. In recent years, CCS has gained widespread adoption in the open-source community and has become a widely recognized standard for commit messages. Building on this, Zeng et al. [39] refined and standardized commit classification criteria, as summarized in Table 1. The ten-category classification task can be viewed as a subtask of CMG, highlighting the need to integrate CCS specification checks into the construction of mainstream CMG datasets. Commit Message Format
Example fix: prevent racing of requests
<type>[optional scope]: <description> [optional body] [optional footer(s)]
Introduce a request id and a reference to latest request. Dismiss incoming responses other than from latest request. Remove timeouts which were used to mitigate the racing issue but are obsolete now. Reviewed-by: Z Refs: #123
Figure 1: Commit message format defined by CCS.
2.2
Commit Datasets and Their Limitations
Table 2 compares existing datasets with CommitSuite. CommitChronicle [6] provides scale (10.7M commits, 20 languages) but lacks quality annotations and CCS compliance. CommitBench [22] filters 1.7M messages by license and bot detection yet omits issue links
CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation
Table 1: Ten-category standards proposed by Zeng et al. [39] Type
Description
feat Introduce new features to the codebase. fix Fix bugs or faults in the codebase. perf Improve performance, such as speed or memory usage. style Improve code readability without changing functionality. refactor Restructure code without changing its behavior. docs Modify documentation or comments. test Add or update tests. ci Change CI configuration or scripts (e.g., CI/CD pipelines). build Modify the build system or dependencies. chore Miscellaneous changes not fitting other categories.
and quality evaluation. MCMD [24] covers 2.25M messages from top-starred projects but excludes AST analysis. FIRA [22] enriches 90K Java commits with fine-grained AST edits yet is limited to a single language. CommitBERT [11] selects 345K simple commits with partial CCS filtering but limited context. OMG [13] collects 35K Java commits with issue/PR links and manual/automatic evaluation, yet lacks “what/why” annotations. CMO [14] provides 500 Java commits with issue/PR links, “what/why” labels, and manual quality ratings, but its small scale limits generalizability. None combine strict CCS compliance, multi-language coverage, semantic annotations, and AST-level context—gaps CommitSuite addresses. Pertain to Class OpenIdConnector
2) absence of systematic “what”/“why” annotations, which not only compromises reference-based evaluation but also limits the availability of high-quality training data; 3) most datasets focus solely on commit diffs, which omit critical contextual information such as enclosing class structures and data dependencies. For example, as illustrated in Figure 2, the diff may indicate the modified function but omit critical function-level context such as its enclosing class structure and data dependencies, leading to information loss and reduced semantic understanding by LLMs.
3 Approach 3.1 Overview Figure 3 illustrates the overall architecture of CommitSuite. It includes three parts: data collection (§3.2), dataset construction (§3.3), and evaluation metrics and benchmark suites (§3.4). In the data collection phase, we identified GitHub repositories adhering to CCS via their mentions of the CCS website, then crawled commits from them. At last, we filtered these commits by various rules, resulting in 63,533 commits. In the dataset construction phase, commit messages were systematically divided into two components: the type and the description. Then we manually verified the used what and why information annotated by a LLM. Additionally, we used a static analysis tool to integrate function-level change information into the dataset. For evaluation metrics and benchmark suites, we selected test sets from CommitSuite for the commit classification task and the CMG task. Both tasks were evaluated using traditional metrics as well as newly proposed metrics.
3.2
Figure 2: Missing function-level information in diff.
2.3
Challenges in CMG Evaluation and Data Quality
Limitations of Reference-Based Metrics. Most current CMG tools rely on traditional machine translation metrics such as BLEU, ROUGE-L, and METEOR for evaluation. However, since developerwritten reference messages lack strict standards and vary significantly in quality, these metrics are often unreliable for assessing generation performance [13]. To address this issue, a high-quality evaluation dataset or a reference-free evaluation standard is required. Limitations of Previous Dataset. While previous datasets have advanced the field, they remain insufficient for our focus on fine-grained CCS-based commit classification and semantically rich message generation. Key limitations include: 1) widespread lack of strict CCS compliance, preventing reliable benchmarking of classification tasks and hindering research on semantic versioning workflows;
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Data Collection
3.2.1 Repository Selection. Analysis of top GitHub repositories (ranked by stars) reveals that projects maintaining CCS-compliant commits consistently mention the official CCS website (conventionalcommits.org) in their documentation files like README.md or CONTRIBUTING.md. According to this feature, we identified 243 repositories covering seven common languages: C, C++, Java, Python, Go, JavaScript, and TypeScript. All these repositories were created between 2 and 10 years ago, which use permissive licenses (Apache2.0, MIT, BSD-3-Clause), have ≥10 forks (indicating high community engagement) and have an average daily commit count <10 after 2020 (to avoid massive robot commits or low-quality commits; since CCS v1.0.0 was released in 2019, we only collect data from 2020 onwards). Their median star count is 1, 669, indicating high recognition and popularity in the developer community. Based on the initial search, we performed manual screening. First, we excluded 54 repositories whose commits almost entirely violated the CCS specification (i.e., messages did not follow CCS format or used non-CCS types). Next, we flagged 49 repositories that were mostly CCS-compliant but contained a small fraction of non-conforming commits; all commits from these repositories were retained and explicitly annotated as “partially compliant” during downstream processing. The remaining 140 repositories were confirmed to be strictly CCS-compliant.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Z. Wan, Z. Wu, X. Hou, Y. Zhao, P. Xia, and H. Wang
Table 2: Comparison of different commit message datasets. Feature
CommitChronicle CommitBench
Size Repositories
10,700,000 11,900
Languages
20 languages
1,664,590 72,000 Java, Ruby, JavaScript, Go, PHP, Python
Apache-2.0, MIT, BSD-3-Clause File-type Check × Bot Check ✓ Duplicate Check ✓ Issue/PR × What/Why Information × AST Change Analysis × CCS Check ×
MIT
License Filtering
✓ ✓ ✓ × × × ×
MCMD
FIRA CommitBERT
§ III-B Data Collection
CMO
CommitSuite (ours)
§ III-D Evaluation Metrics and Benchmark Suites
Github repos
LLM
Repo filter Human
Repos with CCS
Commit URL
OMG
2,250,000 90,661 345,759 35,431 500 63,533 500 1,000 52,462 32 32 172 C, C++, Java, Java, Ruby, C++, C#, Python, Go, Java, Python, Java JavaScript, Go, Java Java JavaScript, TypeScript JavaScript PHP, Python Re-distribution Apache Apache Apache-2.0, – – license projects projects MIT, BSD-3-Clause ✓ × ✓ ✓ ✓ ✓ ✓ × × ✓ ✓ ✓ ✓ ✓ × ✓ ✓ ✓ × × × ✓ ✓ ✓ × × × × ✓ ✓ × ✓ × × ✓ ✓ × × × × × ✓
BLEU, ROUGE-L, METEOR
Non-redundancy
Rationality
Authenticity
Comprehensiveness
Logicality
Commit filter Message format File type
Information of commits + Issues/PRs/Comments
Precision, Recall, F1 Score for each of types
CMG tools Ten-category tools
Macro Precision, Macro Recall, Macro F1 Score, Accuracy
Robot commit Multi-type commit
Ten-category-eval dataset
CMG-eval dataset
§ III-C Dataset Construction Description
Type
Split
LLM roles
Message
“What”-analyzer “Why”-analyzer
Unprocessed data Diff
“What” label “Why” label
CommitSuite
Static analysis Before code After code
Function-level info
Figure 3: Overview of CommitSuite. 3.2.2 Data Crawling. In this step, our goal is to collect up-to-date and comprehensive commits. We collected commits up to May 5, 2025 from the selected repositories using PyDriller [23] and GitHub API [32], recording each commit’s hash, message, author, email, date, modifications, associated comments and related issues/PRs. After above steps, a total of 358,921 commits were obtained. 3.2.3 Data Filtering. To ensure the quality of the dataset, we adopted the following rules to filter commits: • We used regular expressions to filter commit messages conforming to the CCS format, removing 124,829 commits (34.78% of the total). • We filtered by file extensions to retain only the seven languages mentioned earlier; commits touching files of multiple languages were discarded. This step removed 152,596 commits (65.19% of the remaining data).
• To avoid spurious patterns, we removed commits with authors or emails containing “bot”, “robert”, or “b0t” (caseinsensitive) and further filtered using known robot lists [3, 9], removing 1,141 commits (1.40% of the remaining data). • A commit was labeled “multi-type” if its message contained two or more CCS type keywords (e.g., “fix:... feat: . . . ”); such commits were excluded. This step removed 2,458 commits (3.06% of the remaining data). Finally, 77,897 unique commits were retained. After inspection, there are no duplicates in the data. 3.2.4 Outlier Removal. To exclude atypical or memory-intensive data, we analyzed distributions of “Diff length”, “Description character count”, “Diff token count”, “Description token count”, and “Number of modified files”. As shown in Figure 4, these metrics did not follow normal distributions, so we used the IQR (Interquartile
CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Figure 4: The distributions of “Diff length”, “Description character count”, “Diff token count”, “Description token count”, and “Number of modified files”. Range) method [29] to remove extreme outliers, eliminating 14,364 commits (18.44% of the remaining data). The final dataset has 63,533 commits from 172 repositories, covering 7 languages.
3.3
Dataset Construction
Firstly, we parse each commit message to extract the type prefix and description text (discarding any scope information) using regular expressions. The distribution of the ten classification types is shown in Figure 5. After that, we assess the entire message content
• 00: Both “what” and “why” absent • 01: “Why” present, “what” absent • 10: “What” present, “why” absent • 11: Both present Three researchers manually verified 182 “good messages” (5% of messages with both “what” and “why”), achieving 93.41% precision. Additionally, on 500 randomly sampled commits, we measured accuracies of 0.854 (what) and 0.966 (why) with an average Cohen’s Kappa of 0.834. Despite this quality annotation, all commits—including those lacking “what” or “why” information—are retained in the dataset to ensure its completeness for the ten-category classification task.
Figure 5: Commit type distribution in CommitSuite. for the presence of “what” and “why” information as defined by Tian et al. [26]. The message quality classifiers proposed in their research have low precision in “good messages” (include both “what” and “why”) and Xue et al. [38] also demonstrated in their research that LLMs exhibit high consistency with humans in this classification task. Therefore, we used the DeepSeek-V3 model to annotate message quality for its cost-effective. With distributions shown in Figure 6, data was divided into four annotation states based on presence flags:
Figure 6: The distribution of commit messages categorized by the presence of “what” and “why” information in CommitSuite. To provide richer CMG context, we used tree-sitter [31, 33] to compute AST-level diffs between pre- and post-commit versions, capturing added, modified, or deleted functions, classes, and other structures (Table 3). By aligning tree-sitter node spans with diff line
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
numbers, we linked every code change to its enclosing function or class, yielding concise, function-level context for each commit. Table 3: Detected data structure changes by programming language. Language Detected Data Structures C/C++ Java Python Go JavaScript
Function, Class, Struct, Enum, Namespace Method, Class, Interface, Enum, Annotation Function, Class, Async Function Function, Struct, Interface Function, Class, Method, Arrow Function, Object Function, Class, Method, Interface, TypeScript Type Alias, Enum, Arrow Function
3.4
Evaluation Metrics and Benchmark Suites
To demonstrate CommitSuite’s utility as a benchmark, we further evaluate both (1) automatically generated commit messages from state-of-the-art CMG tools/LLMs and (2) CCS-based ten-category classification performance, using the evaluation suites defined below. For CCS-compliant messages, the “type” and “description” (content after the colon) should be checked separately. To ensure balanced representation across different languages and commit types for fair evaluation, we constructed two specialized evaluation subsets from CommitSuite: (1) Ten-category-eval Dataset (Dten ): Due to the limited availability of certain types—particularly ci, which had only 116 valid samples after filtering—we selected 116 correctly classified samples for each category (1,160 in total). While preserving developers’ original classification labels, we rigorously excluded commits where the CCS type did not accurately reflect the code changes. This ensures high label reliability while maintaining authentic classification practices. (2) CMG-eval Dataset (Dcmg ): We randomly sampled 1,000 commits whose messages include “what” and “why” from CommitSuite (180 for C/C++, 102 for Java, 180 for Python, 180 for Go, 179 for JavaScript, 179 for TypeScript). These evaluation subsets are designed specifically for controlled benchmarking tasks. As the largest and only fully CCS-compliant commit dataset currently available (63,533 commits with 100% format verification), the full CommitSuite provides unique value for broader research and tool development, enabling studies on realworld CCS adoption patterns, training of CCS-aware versioning and changelog systems, and analysis of structured commit practices across diverse software ecosystems. Since CommitSuite already filters out commits that modify files in multiple programming languages during the data collection phase (see §3.2), all commits in the evaluation datasets modify files exclusively in a single language. This allows evaluation to be performed on a per-language basis as needed. For these two tasks, our evaluation criteria are as follows: 1) Classification Evaluation:
Z. Wan, Z. Wu, X. Hou, Y. Zhao, P. Xia, and H. Wang
Similar to other multi-classification tasks [8, 42], classification performance is measured by precision, recall, and F1-score. 2) Message Quality Evaluation: Traditional text generation tasks often use three metrics [22]: Traditional metrics include BLEU (n-gram similarity with brevity penalty) [19], ROUGE-L (LCS-based content overlap) [16], and METEOR (stem/synonym matching with word order penalties) [1]. These metrics measure similarity to references, but in CMG tasks, reference quality is highly variable due to lax commit message standards. References often contain non-standard content (e.g., URLs, non-English text), compromising metric reliability. This necessitates reference-free evaluation standards. Li et al. [13] pioneered reference-free commit message evaluation using manual metrics scored via a 5-point Likert scale [15]. However, their follow-up study [14] identified two key limitations: 1) Limited automation capability (LLM Macro-F1=0.766); 2) Intrinsic metric conflicts where optimizing one metric degrades another. To overcome these constraints, we propose an enhanced binary metric system defined in Table 4. Table 4: CommitSuite’s commit message quality metrics. Metric
Description
Rationality
Whether it contains “why” information.
Comprehensiveness Whether it contains “what” information and covers all affected files. Non-redundancy
Whether there is no semantic repetition, mergeable details, meaningless content (unrelated to “what” and “why”), or content of little use.
Authenticity
Whether the content in it does not include modifications absent in the actual code changes.
Logicality
Whether the content in it is reasonable and logical.
Our new metrics use binary evaluations. Compared to the evaluation criteria proposed by Li et al. [13], this evaluation method is more suitable for LLM-powered automated assessment. We only need to calculate the proportion of responses to evaluate tool performance. We replaced Conciseness with Non-redundancy to avoid the trade-off between comprehensiveness and conciseness for complex commits. The new Authenticity and Logicality metrics address LLM hallucinations, preventing fictional or illogical content. As for the original Expressiveness indicator, it is indeed difficult to judge it with binary evaluation. However, in Li et al.’s experimental results [14], the consistency between LLMs’ scoring results for this indicator and manual scoring results was relatively high (F1 score of 0.875). Therefore, if evaluation of this indicator is required, we can retain the 5-point Likert scale scoring method and use LLMs or manual methods for scoring. To facilitate later experimental comparisons, we distilled four disjoint or nested subsets from the final 63,533-commit corpus (Dall ); their sizes and origins are summarised in Table 5.
CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation
Table 5: Dataset subsets used in each RQ. RQ
Subset
Size
Derived from
RQ1 RQ2 RQ3 RQ4
Dhuman Dcmg Dten cmg ten Dast , Dast
300 1000 1160 200, 200
Dhuman ⊂ Dcmg Dcmg ⊂ Dall Dten ⊂ Dall cmg ten ⊂ D Dast ⊂ Dcmg , Dast ten
• RQ1 employs Dhuman (300 items sampled from Dcmg ) to quantify human–LLM agreement on our five binary metrics. • RQ2 uses the full CMG-eval Dataset Dcmg (1,000 commits) for large-scale commit-message generation benchmarking. • RQ3 relies on the Ten-category-eval Dataset Dten (1,160 commits) for ten-type classification evaluation. • RQ4 performs two independent ablations: we randomly draw 200 commits from Dcmg to assess the impact of AST-level context on message generation, and another 200 commits from Dten to evaluate its effect on commit-type classification.
4
Evaluation
In this section, we report and analyze the experimental results to address the following research questions (RQs): • RQ1: How consistent is the automated evaluation of the binary evaluation criteria introduced in §3.4 by LLM with human evaluation? • RQ2: How do existing tools and LLMs perform in the commit message generation task on CommitSuite? • RQ3: How do existing tools and LLMs perform in the commit classification task on CommitSuite? • RQ4: What is the impact of function-level contextual information on commit classification tasks and message generation tasks?
4.1
RQ1: Consistency Between Automated LLM Evaluation and Human Evaluation of Binary Metrics
As mentioned in §3.4, we proposed a set of binary evaluation metrics to address the dependency of traditional evaluation metrics on reference texts and shift the evaluation mode from manual to automated assessment. Although this approach appears to save significant workload in evaluating related tools, we first need to validate the effectiveness of LLMs in this binary evaluation task before full-scale application. We randomly draw 300 commits from the CMG experimental results of different tools/LLMs in §4.2 and evaluated them through a rigorous process involving three independent human reviewers (each with 3+ years of software development experience) and DeepSeek-V3. Each reviewer first scored every message on all five metrics according to a strict guideline; disagreements were then resolved in a moderated consensus session, yielding a single groundtruth label set. Inter-rater agreement among the three developers, quantified by pairwise Cohen’s Kappa , averaged 0.817, indicating “almost perfect” concordance. When the final human labels were compared with the fully automated scores produced by DeepSeekV3, Cohen’s Kappa reached 0.849, surpassing the inter-human baseline. These results corroborate the high average Recall, Precision, F1, and Accuracy (>0.9) reported in Table 6 and establish that the
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
LLM-based assessment is not only consistent with, but slightly exceeds, human-level reliability. Table 6: Consistency results of binary metrics between LLM automated evaluation and human evaluation. Field
Recall Precision
F1
Accuracy
Rationality Comprehensiveness Non-redundancy Authenticity Logicality
0.934 0.931 0.960 0.924 0.983
0.942 0.938 0.845 0.959 0.891
0.938 0.934 0.899 0.941 0.935
0.950 0.943 0.857 0.903 0.947
Average
0.946
0.915
0.929
0.920
4.2
RQ2: Performance of Existing Tools and LLMs in CMG
We selected four state-of-the-art (SOTA) CMG tools to cover major CMG paradigms: NNGen (retrieval-based) [17], FIRA (learningbased with AST) [5], CoRec (hybrid retrieval-generation) [35], and KADEL (latest learning-based model achieving top performance) [25]. Additionally, we include six popular LLMs from OpenRouter’s ranking (DeepSeek-V3, DeepSeek-R1, GPT-4.1, GPT-4o-mini, Gemini2.0-flash, Claude-3.7-sonnet) to investigate their performance in CMG tasks. We used the CMG-eval Dataset from §3.4 for evaluation (focusing solely on the five reference-text-independent metrics). For LLM evaluator, we used a zero-shot prompt (core excerpt below; full prompt available in our open-source repository): Core Prompt for LLM Evaluation You are a code reviewer... You are given the following information: - Modified files (with diffs) - Possibly related issues, pull requests, or comments - Possibly AST changes - The generated commit message (CMG_result) You should assess the generated commit tives:...(Rationality/Comprehensiveness/...)
message
from
five
perspec-
The experiment prioritized comparing tool and LLM performance on a new high-quality evaluation dataset. In CMG task, we did not optimize LLM prompts beyond instructing them to generate commit messages using provided diffs, issues, PRs, comments, and AST changes. The final results are presented in Table 7. Among existing tools, FIRA performed best, achieving the highest scores in six of eight metrics. Notably, it outperformed human developers in “Non-redundancy”, “Authenticity”, and “Logicality”, suggesting high-quality training data and strong generalizability. KADEL, however, performed poorly, ranking lowest in five metrics. Its training data contained numerous default-format messages like “Merge pull request from...” and “Update {filename}”, which describe commit actions rather than content [43], leading the tool to generate similarly meaningless outputs. Across eight metrics, LLMs only scored lower than traditional tools in BLEU. Their performance on the five binary metrics was particularly notable: LLMs achieve >0.9 on each binary metric.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Z. Wan, Z. Wu, X. Hou, Y. Zhao, P. Xia, and H. Wang
Table 7: Performance comparison of existing tools, LLMs, and humans in CMG tasks. Traditional Evaluation Metrics
Binary Evaluation Metrics
Category
Name
Existing Tools
NNGen CoRec FIRA KADEL
3.164 4.142 5.374 0.216
11.248 14.887 19.421 6.207
8.647 10.562 12.359 1.511
0.023 0.020 0.013 0.017
0.088 0.084 0.278 0.094
0.802 0.810 0.914 0.544
0.238 0.345 0.693 0.761
0.198 0.231 0.629 0.017
LLMs
DeepSeek-V3 DeepSeek-R1 GPT-4.1 GPT-4o-mini Gemini-2.0-flash Claude-3.7-sonnet
2.066 1.488 1.510 1.848 6.126 2.675
18.440 17.468 16.059 16.169 21.741 20.087
23.289 21.073 21.398 21.571 28.219 23.242
0.920 0.961 0.966 0.959 0.937 0.945
1.000 1.000 1.000 0.997 0.998 1.000
0.970 0.979 0.986 0.971 0.969 0.980
0.997 0.999 1.000 0.985 0.994 0.997
1.000 1.000 1.000 0.998 0.999 1.000
Humans
Human
-
-
-
0.337
0.481
0.492
0.496
0.494
BLEU ROUGE-L METEOR Rationality Comprehensiveness Non-redundancy Authenticity Logicality
Gemini-2.0-flash achieved the highest scores in BLEU, ROUGEL, and METEOR, suggesting its outputs are most similar to highquality human-written messages, while GPT-4.1 led in the remaining five metrics. This distinction arises because different models exhibit distinct characteristics in message generation: traditional metrics prioritize similarity to human-written references, while binary metrics emphasize comprehensiveness and completeness. Figure 7 compares the average scores of humans, LLMs, and tools across the five binary metrics. Notably, a random sample of 1,000 human-written messages from the full dataset performed modestly, partially explaining the tools’ subpar performance, lowquality training data limits tool capabilities, and their performance declines when evaluated on high-quality datasets. Zhang et al.’s experiments [43] showed that retraining and testing tools on cleaned datasets (with noisy commits removed) resulted in decreased evaluation scores, supporting this hypothesis. This indicates that conventional tools may have adapted to prevalent noise patterns in typical commit data. Additionally, some evaluation data exceeded tools’ input length limits, resulting in truncated inputs that caused information loss and degraded performance. Comprehensiveness Human Tool LLM
Non-redundancy
0.0 0.2 0.4 0.6 0.8 1.0 Rationality
research on how to enable LLMs to accurately capture change rationales and refine message content. Meanwhile, BLEU, ROUGEL, and METEOR remain relevant for LLMs, and efforts should focus on enhancing these metrics while maintaining performance on binary indicators to close the gap with human-written messages. This experiment demonstrates that CommitSuite’s binary metrics sharply distinguish tool and LLM behaviors, while its rigorously curated CCS-compliant data lends renewed credibility to traditional scores, enabling a more holistic and trustworthy evaluation of commit message generation.
4.3
RQ3: Performance of Existing Tools and LLMs in CCS-Based Commit Classification
Conventional Commits Specification (CCS) (§2.1) outperforms earlier taxonomies by mandating a machine-readable <type> prefix that cleanly maps commits into ten fine-grained categories (Table 1). Unlike ad-hoc project conventions, CCS is language-agnostic, documented in a single canonical spec, and natively supports downstream automation such as semantic versioning and changelog generation—features that make it the de-facto industry standard for commit classification. We used the Ten-category-eval Dataset from §3.4 for evaluation. Zeng et al. [39] are the first to study CCS-based commit classification, and their work remains the primary reference in this area. In addition to their classification tool (hereafter “the classifier”), we included the six LLMs from §4.2 to evaluate performance on the commit classification task. Results are shown in Table 8. Table 8: Ten-category classification performance of Qunhong Zeng et al.’s method and LLMs.
Authenticity
Name
Logicality
Figure 7: Compare the average scores of humans, LLMs, and tools across the five binary metrics. LLMs clearly hold an advantage in adapting to new scenarios, due to their support for extremely long contexts, which enables handling complex diffs and edge cases. However, “Rationality” and “Non-redundancy” remain areas for improvement, requiring future
Qunhong Zeng et al DeepSeek-V3 DeepSeek-R1 GPT-4.1 GPT-4o-mini Gemini-2.0-flash Claude-3.7-sonnet
Precision Recall 0.580 0.665 0.675 0.641 0.643 0.664 0.668
0.473 0.597 0.638 0.608 0.559 0.631 0.609
F1
Accuracy
0.457 0.588 0.621 0.585 0.554 0.620 0.606
0.473 0.597 0.638 0.608 0.559 0.629 0.609
DeepSeek-R1 performed best among both the classifier and LLMs, achieving the highest scores in all four metrics with an accuracy of
CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation
Qunhong Zeng et al.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
GPT4.1
DeepSeek-r1
Figure 8: Compares the confusion matrices of the classifier, GPT-4.1, and DeepSeek-R1. 0.6379. Figure 8 compares the confusion matrices of the classifier and representative LLMs (GPT-4.1 and DeepSeek-R1), revealing frequent misclassifications among ci, build, and chore. The build category showed the most significant classification discrepancies. Ci and build involve system construction, testing, and deployment, often containing overlapping changes with other categories, leading to confusion. Chore encompasses miscellaneous changes, but classifiers or LLMs unfamiliar with project structures struggle to determine what qualifies as “miscellaneous”, compounded by subjective developer interpretations of the same changes. As Figure 5 previously showed, chore commits comprise a substantial portion of real-world development, highlighting the need for clearer definitions, although standardization remains challenging. Beyond these three categories, perf, style, and fix were frequently misclassified as refactor. Further research is required for these categories prone to misjudgment or confusion. Overall, commit classification tools still have significant room for improvement. Addressing common misclassifications like those noted will be critical for future research. This experiment validate the unique contribution of CommitSuite: by providing the first large-scale, CCS-compliant dataset with manually verified ten-category labels, we enable a controlled examination of classification boundaries that prior benchmarks could not support. The observed inter-category confusion underscores the need for precisely such high-quality data to drive future refinements in both rule-based and LLM-driven classifiers.
4.4
RQ4: Impact of Function-Level Contextual Information on Commit Tasks
To investigate the impact of function-level contextual information on commit classification and message generation, we conducted two comparative experiments: 1) First experiment (CMG task): We drew a random sample of 200 commits from the CMG-eval Dataset (§3.4). Using DeepSeek-V3, we generated messages for each commit with and without AST-level context. Human evaluators then pairwise compared the outputs along two dimensions—“what” (change content) and “why” (change rationale). As shown in Table 9, AST context yielded significantly better “what” descriptions (34 wins vs. 10 losses; Sign-test p = 0.0002). For “why”, AST still provided a slight advantage (9 wins
vs. 2 losses), although the effect is weaker (p = 0.0327). Thus, ASTlevel information markedly improves the summarisation of what changed, while its contribution to explaining why the change was made is more modest. Table 9: Comparison of “what” and “why” information generation with or without AST change in CMG task. Metric AST Better No AST Better Tie Sign-test p-value What Why
34 9
10 2
156 189
0.0002 0.0327
2) Second experiment (commit classification task): We again sampled 200 commits from the Ten-category-eval Dataset (§3.4). Guided by the RQ3 results, we selected the top model (DeepSeekR1), the worst model (DeepSeek-V3), and two median model (GPT4o-mini, Gemini-2.0-flash). All four models were evaluated with and without AST information. Table 10 shows AST data had negligible impact and slightly reduced LLM performance across metrics, suggesting it constitutes redundant information for classification tasks. Function-level contextual information aids “what” content generation in CMG tasks but is unnecessary for commit classification classification, where it may even degrade performance.
5 Discusstion 5.1 Case Study Beyond the differences in metrics, there are other characteristics in the messages generated by humans, LLMs, and tools. Figure 9 compares the messages for the same commit: the human-generated message is precise and reasonable, the tool-generated message omits the change rationale (“for clarity”), while the LLM-generated message provides more comprehensive information, including all affected files and functions. Using current metrics, it is difficult to determine whether human-written messages or LLM-generated messages are better, which may depend on the specific needs of the repository. To address this, we may need to adjust LLMs according to real-world scenarios, requiring them to discard some “less important details” when necessary.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Z. Wan, Z. Wu, X. Hou, Y. Zhao, P. Xia, and H. Wang
Table 10: Ten-category classification performance of LLMs with and without AST change information. Without AST
With AST
Name Precision Recall DeepSeek-V3 DeepSeek-R1 GPT-4o-mini Gemini-2.0-flash
0.6647 0.6752 0.6433 0.6635
F1
0.5974 0.5883 0.6379 0.6214 0.5586 0.5536 0.6308 0.6203
Tools Rename ws_now_ms to ws_timestamp
LLMs Rename ws_now_ms() to ws_timestamp() for clarity
Human refactor: rename ws_now_ms() to ws_timestamp() for clarity
The function ws_now_ms() has been renamed to ws_timestamp() across the codebase to better reflect its purpose of returning a timestamp value. All references to the old function name have been updated in: - common/websockets.c - common/websockets.h - discord-websockets.cpp The change affects websocket operations including identification, event dispatch, and heartbeat functionality.
Figure 9: Compares the messages generated by humans, LLMs, and tools for the same commit.
5.2
Valid Analysis
(1) Internal Valid Analysis. Internal validity is threatened by the fact that approximately 67 % of the collected commits omit explicit “why” information, thereby skewing model learning toward surface change descriptions; LLM-generated “what/why” annotations, though subject to 93.41 % human verification, still propagate residual model bias; and the reliance on small, high-quality subsets for individual RQs limits generalizability to realistic, noise-laden repositories while revealing nonsignificant distinctions among LLMs under the proposed binary metrics, thus necessitating continued recourse to traditional similarity measures such as BLEU, ROUGE-L, and METEOR. Nevertheless, the CommitSuite remains suitable for diverse commit-level tasks. (2) External Valid Analysis. External validity is constrained by evidence drawn exclusively from CCS-compliant repositories written in seven mainstream languages, leaving performance on non-CCS commits, niche languages such as Rust, and low-activity codebases unevaluated; moreover, while LLMs achieve high scores on the five binary metrics, their tendency to misclassify boiler-plate messages (e.g., “Merge pull request . . . ”) as non-redundant suggests that current prompts may overfit to surface patterns rather than genuine content. Additionally, the reliance on the tree-sitter static analyser may introduce systematic deviations in the extraction of function-level information, further limiting the generalisability of our findings.
6 Related Work 6.1 Benchmarks and CMG Tools CMG has evolved from rule-based [2] and statistical models [37] to neural-based approaches [17], with growing focus on semantic richness. Among traditional tools, FIRA [5] achieves strong performance by modeling fine-grained code edits with graph-based representations and incorporating a dual-copy mechanism. Such models often rely on AST-level information [35], yet the integration of structural semantics remains tightly coupled with model-specific
Accuracy Precision Recall 0.5974 0.6379 0.5586 0.6289
0.6636 0.6600 0.6429 0.6654
F1
0.6078 0.6000 0.6241 0.6018 0.5422 0.5335 0.6063 0.5980
Accuracy 0.6078 0.6241 0.5422 0.6070
designs. Regarding benchmarks, CommitBench [22] provides a large-scale CMG dataset but lacks message quality annotations (e.g., “what”/“why”) and standardized formatting checks, limiting its utility for fine-grained evaluation. Other datasets [6, 18] support CMG research but are not aligned with practical labeling standards such as the CCS [30, 39]. Moreover, the absence of reliable CCSlabeled datasets and the semantic overlap between categories (e.g., fix vs. ci) pose persistent challenges for classification consistency. These gaps highlight the need for a high-quality benchmark that not only supports CCS classification with reliable annotations but also enables fine-grained evaluation of commit messages in terms of structure and semantic adequacy.
6.2
LLMs in CMG and Classification
Recent studies have demonstrated that LMs significantly outperform traditional CMG models in terms of generalization and semantic reasoning [10, 36, 38, 41]. For example, Li et al. [13] argue that code diffs alone are insufficient, and that incorporating structured reasoning processes can lead to more accurate message generation. Xue et al. [38] further highlight that LLMs effectively capture both the “what” and “why” of a commit with minimal fine-tuning. In parallel, researchers have begun applying LLMs to conventional commit classification. Zeng et al. [39] introduced the first CCS classification benchmark, while Tong et al. [27] enhanced classification performance using prompt tuning and external knowledge. Despite these advances, challenges such as ambiguous label boundaries and reliance on superficial cues persist. To further improve LLMbased CMG and classification, recent work has explored integrating structural information such as ASTs [5], context-aware inputs [13], and prompting techniques including ICL, RAG, and MCP [38, 40]. However, systematically incorporating AST and history-aware representations into LLM workflows remains an open problem. These findings collectively motivate the design of benchmarks and data protocols that align with LLM strengths, bridging structural code understanding with semantically rich commit reasoning.
7
Conclusion
In this paper, we present CommitSuite, a comprehensive benchmark designed for both commit classification and commit message generation tasks. By integrating CCS (Conventional Commits Specification) and high-quality commit messages, we construct a large-scale, clean dataset covering multiple languages and repositories. We propose a new binary evaluation metric system that enables automated assessment of commit messages without relying on human-written references. Experiments demonstrate that LLMs outperform existing tools in both tasks, especially in message generation.
CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation
References [1] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72. [2] Raymond PL Buse and Westley R Weimer. 2010. Automatically documenting program changes. In Proceedings of the 25th IEEE/ACM international conference on automated software engineering. 33–42. [3] Natarajan Chidambaram, Alexandre Decan, and Tom Mens. 2023. A dataset of bot and human activities in github. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 465–469. [4] Brian De Alwis and Jonathan Sillito. 2009. Why are software projects moving from centralized to decentralized version control systems?. In 2009 ICSE Workshop on Cooperative and Human Aspects on Software Engineering. IEEE, 36–39. [5] Jinhao Dong, Yiling Lou, Qihao Zhu, Zeyu Sun, Zhilin Li, Wenjie Zhang, and Dan Hao. 2022. FIRA: fine-grained graph-based code change representation for automated commit message generation. In Proceedings of the 44th International Conference on Software Engineering. 970–981. [6] Aleksandra Eliseeva, Yaroslav Sokolov, Egor Bogomolov, Yaroslav Golubev, Danny Dig, and Timofey Bryksin. 2023. From commit message generation to history-aware commit message completion. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 723–735. [7] Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, and Shanping Li. 2024. Exploring the capabilities of llms for code change related tasks. ACM Transactions on Software Engineering and Methodology (2024). [8] Nadia Ghamrawi and Andrew McCallum. 2005. Collective multi-label classification. In Proceedings of the 14th ACM international conference on Information and knowledge management. 195–200. [9] Mehdi Golzadeh, Alexandre Decan, Damien Legay, and Tom Mens. 2021. A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments. Journal of Systems and Software 175 (2021), 110911. [10] Aaron Imani, Iftekhar Ahmed, and Mohammad Moshirpour. 2024. Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation. arXiv preprint arXiv:2408.02502 (2024). [11] Tae-Hwan Jung. 2021. Commitbert: Commit message generation using pretrained programming language model. arXiv preprint arXiv:2105.14242 (2021). [12] Jiawei Li and Iftekhar Ahmed. 2023. Commit message matters: Investigating impact and evolution of commit message quality. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 806–817. [13] Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed. 2024. Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model. Proceedings of the ACM on Software Engineering 1, FSE (2024), 745–766. [14] Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed. 2025. Consider What Humans Consider: Optimizing Commit Message Leveraging Contexts Considered By Human. arXiv preprint arXiv:2503.11960 (2025). [15] Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology (1932). [16] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81. [17] Zhongxin Liu, Xin Xia, Ahmed E Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. 2018. Neural-machine-translation-based commit message generation: how far are we?. In Proceedings of the 33rd ACM/IEEE international conference on automated software engineering. 373–384. [18] Andreas Mauczka, Florian Brosch, Christian Schanes, and Thomas Grechenig. 2015. Dataset of developer-labeled commit messages. In 2015 IEEE/ACM 12th working conference on mining software repositories. IEEE, 490–493. [19] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. [20] Eddie Antonio Santos and Abram Hindle. 2016. Judging a commit by its cover: Correlating commit message entropy with build status on travis-ci. (2016). [21] Muhammad Usman Sarwar, Sarim Zafar, Mohamed Wiem Mkaouer, Gursimran Singh Walia, and Muhammad Zubair Malik. 2020. Multi-label classification of commit messages using transfer learning. In 2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 37–42. [22] Maxmilian Schall, Tamara Czinczoll, and Gerard De Melo. 2024. Commitbench: A benchmark for commit message generation. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 728–739. [23] Davide Spadini, Maurício Aniche, and Alberto Bacchelli. 2018. PyDriller: Python framework for mining software repositories. In Proceedings of the 2018 26th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering. 908–911. [24] Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the evaluation of commit message generation models: An experimental study. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 126–136.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
[25] Wei Tao, Yucheng Zhou, Yanlin Wang, Hongyu Zhang, Haofen Wang, and Wenqiang Zhang. 2024. Kadel: Knowledge-aware denoising learning for commit message generation. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–32. [26] Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What makes a good commit message?. In Proceedings of the 44th International Conference on Software Engineering. 2389–2401. [27] Jiajun Tong and Xiaobin Rui. 2025. A Commit Classification Framework Incorporated With Prompt Tuning and External Knowledge. IET Software 2025, 1 (2025), 5566134. [28] Petr Tsvetkov, Aleksandra Eliseeva, Danny Dig, Alexander Bezzubov, Yaroslav Golubev, Timofey Bryksin, and Yaroslav Zharov. 2024. Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings. arXiv preprint arXiv:2410.12046 (2024). [29] John Wilder Tukey et al. 1977. Exploratory data analysis. Vol. 2. Springer. [30] Author unknown. Year unknown. Conventional Commits — conventionalcommits.org. https://www.conventionalcommits.org/en/v1.0.0/. [Accessed 31-052025]. GitHub - conventional[31] Author unknown. Year unknown. commits/conventionalcommits.org: The conventional commits specification — github.com. https://github.com/conventional-commits/conventionalcommits.org. [Accessed 31-05-2025]. [32] Author unknown. Year unknown. GitHub REST API documentation - GitHub Docs — docs.github.com. https://docs.github.com/en/rest. [Accessed 31-05-2025]. [33] Author unknown. Year unknown. Introduction - Tree-sitter — tree-sitter.github.io. https://tree-sitter.github.io/tree-sitter/. [Accessed 31-05-2025]. [34] Bei Wang, Meng Yan, Zhongxin Liu, Ling Xu, Xin Xia, Xiaohong Zhang, and Dan Yang. 2021. Quality assurance for automated commit message generation. In 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 260–271. [35] Haoye Wang, Xin Xia, David Lo, Qiang He, Xinyu Wang, and John Grundy. 2021. Context-aware retrieval-based deep commit message generation. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 4 (2021), 1–30. [36] Yifan Wu, Ying Li, and Siyu Yu. 2024. Commit Message Generation via ChatGPT: How Far Are We?. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering. 124–129. [37] Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu. 2019. Commit message generation for source code changes. In IJCAI. [38] Pengyu Xue, Linhao Wu, Zhongxing Yu, Zhi Jin, Zhen Yang, Xinyi Li, Zhenyu Yang, and Yue Tan. 2024. Automated commit message generation with large language models: An empirical study and beyond. IEEE Transactions on Software Engineering (2024). [39] Qunhong Zeng, Yuxia Zhang, Zhiqing Qiu, and Hui Liu. 2024. A First Look at Conventional Commits Classification. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 127–139. [40] Linghao Zhang, Hongyi Zhang, Chong Wang, and Peng Liang. 2024. RAGEnhanced Commit Message Generation. arXiv preprint arXiv:2406.05514 (2024). [41] Linghao Zhang, Jingshu Zhao, Chong Wang, and Peng Liang. 2024. Using large language models for commit message generation: A preliminary study. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 126–130. [42] Min-Ling Zhang and Zhi-Hua Zhou. 2013. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering 26, 8 (2013), 1819–1837. [43] Yuxia Zhang, Zhiqing Qiu, Klaas-Jan Stol, Wenhui Zhu, Jiaxin Zhu, Yingchen Tian, and Hui Liu. 2024. Automatic commit message generation: A critical review and directions for future work. IEEE Transactions on Software Engineering 50, 4 (2024), 816–835.