CommitLLM: A Fine-Tuned Pipeline for Git Commit Message Generation Md Rafid Haque
Poojan Narendrabhai Patel
Meetkumar Vijaybhai Raychura
University of Illinois at Chicago
University of Illinois at Chicago
University of Illinois at Chicago
arXiv:2607.17532v1 [cs.SE] 20 Jul 2026
Abstract
pute would not significantly hurt model quality. The QLoRA paper [2] suggests this is a safe assumption and our results are consistent with that.
Developers frequently write uninformative git commit messages such as "fix" or "update stuff", degrading the value of version-control history for code review, debugging, and onboarding. We present CommitLLM, a three-stage pipeline that generates concise, Conventional Commits-compliant messages from code diffs using a fine-tuned small language model. The system combines (1) QLoRA fine-tuning of Mistral-7B-Instruct-v0.2 on the CommitPackFT dataset, (2) constrained decoding to enforce brevity, and (3) deterministic post-processing to strip conversational artifacts and enforce format. On a 50-sample evaluation, CommitLLM achieves 98% format compliance (vs. 22% for vanilla Mistral), reduces average output length from 154.8 to 37.9 characters, and improves LLM-as-a-Judge scores from 1.97 to 3.68 out of 5. Notably, the post-processing layers contribute more to quality improvement than the fine-tuning itself, suggesting that for structured-output tasks, treating the LLM as a component in a deterministic pipeline is more effective than optimizing the model alone. The entire system runs on a single consumer GPU (NVIDIA T4, 16 GB VRAM).
• Dataset assumption: CommitPackFT [6] is good enough as-is. We did not try to clean it further beyond what BigCode already did. 1.4 Definitions A few terms come up enough in this paper that we should pin them down: • LoRA (Low-Rank Adaptation) [1]: a parameter-efficient finetuning method that freezes the base model and trains small rank-𝑟 matrices inserted next to the original weights. • QLoRA [2]: LoRA on top of a 4-bit quantized base model. This is what makes 7B fine-tuning fit on a T4. • NF4: a 4-bit quantization data type designed to be optimal for normally-distributed weights, used by QLoRA. • Conventional Commits: an industry convention where commit messages start with a type prefix like feat:, fix:, or docs: [7].
1 Introduction 1.1 Motivation Most developers either skip writing meaningful commit messages or write things like "fix" or "update stuff". A model that reads a diff and produces a clean, Conventional Commits-style message would save time and make commit history actually useful for code review and debugging. This paper asks a practical question: can a small opensource LLM be fine-tuned to write good git commit messages from code diffs, on hardware that any developer has access to?
• BERTScore [3]: a metric for text similarity that compares contextual embeddings rather than exact word overlap, so it understands paraphrases. • LLM-as-a-Judge [4]: using a strong LLM to grade outputs from another model on a fixed rubric. We used Llama-3.3-70B and Llama-3.1-8B served via Groq. 1.5 Summary of Approach We did three things, in this order:
1.2 Goals Our goal was twofold. First, fine-tune Mistral-7B-Instruct-v0.2 on a diff-to-commit dataset using parameter-efficient methods so it could run on a consumer GPU (Google Colab T4). Second, build a complete inference pipeline that reliably produces short, well-formatted commit messages, not just a model that sometimes gets it right.
(1) Fine-tune. We loaded Mistral-7B-Instruct-v0.2 in 4-bit NF4 and applied LoRA (rank 16, 𝛼 = 32) on attention and MLP projection layers. We trained for one epoch on the CommitPackFT data using SFTTrainer from HuggingFace’s TRL library. Total trainable parameters: about 42M out of 3.79B, or roughly 1.1% of the model.
We originally planned to package the model as a VS Code extension, but the engineering effort of local model deployment (shipping weights, hosting endpoints, or assuming a local inference server) proved orthogonal to the core research question. We instead invested in building a more rigorous evaluation harness and discuss the IDE integration as future work (Section 6).
(2) Build an inference pipeline, not just use the model. The fine-tuned model on its own was still too verbose, it would sometimes produce paragraph-length output. So we wrapped it in two extra stages: aggressive constrained decoding (max_new_tokens=20, temperature=0.1, repetition penalty 1.1) and deterministic post-processing (slice after [/INST], take first line only, strip conversational filler before the first colon if it is not a valid Conventional Commits type).
1.3 Assumptions We made a few assumptions and stuck with them throughout:
(3) Evaluate. We compared three systems, vanilla Mistral, raw finetuned Mistral, and the full pipeline, on four metrics: format compliance (regex), commit length, BERTScore semantic similarity to the human ground-truth, and LLM-as-a-Judge scores from Llama 3.3-70B / 3.18B.
• Compute: Google Colab with a single NVIDIA T4 GPU (16 GB VRAM). Everything we did had to fit in this budget. • Language: Python 3 with the HuggingFace ecosystem (transformers, peft, trl, datasets, bitsandbytes, evaluate). • Quantization assumption: 4-bit NF4 quantization with bf16 com-
1
2 Background and Related Work
strong instruction-following (it already understands the [INST] ... [/INST] chat format), it has a permissive license, and it is the right size for a T4 once quantized. Fine-tuning is QLoRA: 4-bit NF4 quantization on the base, LoRA adapters with rank 16 and 𝛼 = 32 on attention (𝑞, 𝑘, 𝑣, 𝑜) and MLP (𝑔𝑎𝑡𝑒, 𝑢 𝑝, 𝑑𝑜𝑤𝑛) projections. One epoch over the CommitPackFT data, AdamW (paged, 32-bit) with a cosine learning-rate schedule starting at 2e-4 and a 3% warmup ratio. Effective batch size 16 (4 per device × 4 gradient accumulation steps). With group_by_length=True the run finished in roughly 40 minutes on a T4.
2.1 Tools and Libraries A few things we tried that did and did not work: • HuggingFace TRL SFTTrainer: worked out of the box once we figured out the new SFTConfig arguments. The group_by_length=True setting alone cut training time noticeably by reducing wasted padding. • bitsandbytes 4-bit quantization: initially failed at inference time with a CUDA-related error; we had to upgrade to bitsandbytes>=0.46.1 to make it work with the Colab T4.
Layer 2: Constrained decoding at inference time. Even after fine-tuning, the model has the chatty habits of its instruction-tuned parent. To suppress them, we generate with very tight constraints: max_new_tokens=20 (a hard cap on length, so the model literally cannot produce a paragraph), temperature=0.1 (almost greedy decoding, no creative drift), and a repetition penalty of 1.1 (so the model does not loop on common tokens from the diff). This is the cheapest possible way to enforce conciseness, no extra weights, no extra compute, just stricter sampling.
• Groq for LLM-as-a-Judge: fast, cheap, generous free tier, but the daily token-per-day limit (100k tokens) forced a model swap from llama-3.3-70b-versatile to llama-3.1-8b-instant for the cleaned-output evaluation. 2.2 Related Work For the modeling side, the two papers that mattered most were the original LoRA paper [1] for the rank/𝛼/target-module choices, and the QLoRA paper [2] for the 4-bit NF4 + double-quantization recipe. The Mistral 7B technical report [5] was useful for understanding why this base model is a good choice for fine-tuning at this scale. For the evaluation side, the BERTScore paper [3] explained why we should expect contextual-embedding similarity to be more forgiving than BLEU/ROUGE for short, paraphrase-heavy outputs like commit messages, and the MT-Bench / LLM-as-a-Judge paper [4] gave us the framing for using an LLM to grade outputs against a fixed rubric.
Layer 3: Deterministic post-processing. Even with constrained decoding, the model sometimes leaks conversational filler (“Here is the commit: ...”). We apply three string-level fixes in sequence: tag slicing (keep only what comes after the final [/INST]), guillotine truncation (take the first line only, splits on the first newline), and conversationalprefix stripping (if there is a colon, look at the word before it; if that word is not in our valid Conventional Commits type set, strip everything up to and including that colon). 3.2 Motivation for the Strategy Why this layered approach instead of just training harder?
Key documentation resources included: • HuggingFace PEFT documentation [8] for the LoRA configuration API and target module selection.
It is much cheaper. Layers 2 and 3 cost nothing at training time and almost nothing at inference time, but they account for a large chunk of the final quality improvement. Our results (Section 5) show that the LLM-as-a-Judge score went from 1.97 (vanilla) to 2.20 (raw fine-tuned) to 3.68 (full pipeline). That is, fine-tuning on its own gave us a +0.23 improvement in judge score; the post-processing pipeline on top of the same weights gave us another +1.48. The pipeline is doing more work than the fine-tuning, for essentially zero additional compute.
• HuggingFace TRL documentation [9] for SFTTrainer and SFTConfig. • The CommitPackFT dataset card [6] for the data structure and licensing. • The Conventional Commits specification [7] for what “compliant” actually means as a target format. • The bitsandbytes documentation BitsAndBytesConfig settings.
[10]
for
the
It is more controllable. Deterministic post-processing is debuggable. If something goes wrong, we can read the log and find out which line of code did the wrong thing. If the same problem were caused by the model’s weights, we would have to retrain. Pushing as much of the “shape” of the output into deterministic code as possible makes the system more predictable.
right
• Groq API docs [11] for endpoint names, rate limits, and the async client. The HuggingFace PEFT and TRL docs were the single most practically useful resources. The QLoRA paper was the most useful academic reference because it specifies exactly which 4-bit format to use and why. The Conventional Commits spec was unexpectedly important on the evaluation side: it gave us a concrete, externally-defined target for what a “good” commit looks like, which is what made the regex-based compliance metric possible.
It separates concerns. The model’s job is to understand the diff and produce a candidate message. The pipeline’s job is to enforce the format. Separating these two jobs means we can swap out the model later (a Llama-3 7B, a Phi-3, a smaller distilled model) without rewriting the format-enforcement layer. Multi-metric evaluation was deliberate. For the evaluation side, we used four different metrics, regex compliance, length, BERTScore, LLM-judge, because we did not trust any single metric to tell the truth. This decision turned out to be more important than we expected: our original compliance regex had two subtle bugs that flattered the baseline and penalized our pipeline (more on this in Section 5.2). If we had only reported compliance, we would have shipped a misleading result. Having BERTScore and LLM-judge as backup metrics meant we could cross-check.
3 Approach 3.1 The Layered Strategy Our overall strategy can be summarized in one sentence: treat CommitLLM as a system, not as a model. Most discussion of LLM finetuning focuses on getting better weights. We started there too, but quickly realized that even after a clean fine-tune, the model would still occasionally produce verbose, conversational, or multi-line output. For a tool that is supposed to drop into a developer’s workflow, that is unacceptable, a commit-message generator that returns a paragraph 30% of the time will get disabled within a day. So we structured the work around three layers, each one constraining the previous one further.
4 Implementation 4.1 Software Design The codebase is organized into seven logical phases: 1. Phase 0 – Setup: load the dataset and install dependencies.
Layer 1: The fine-tuned model. The base model is Mistral-7BInstruct-v0.2. We chose it because it is a 7B parameter model with
2
HuggingFace at runtime; we save and ship only the LoRA adapter delta. • CommitPackFT [6] as the training/eval data. 9,600 train / 1,200 validation / 1,200 test examples, all in the [INST] ... [/INST] format already. • HuggingFace transformers, peft, trl, datasets, evaluate for the modeling stack. • bitsandbytes (≥ 0.46.1) for 4-bit NF4 quantization. • BERTScore via the evaluate library, which uses RoBERTa-large under the hood as the embedding backbone. • Groq API for LLM-as-a-Judge, calling llama-3.3-70b-versatile (raw outputs) and llama-3.1-8b-instant (clean outputs after hitting the daily token limit on the 70B model). • pandas, matplotlib, seaborn for results aggregation and visualization. Figure 1: End-to-end CommitLLM inference architecture. A code diff is wrapped in Mistral’s chat template and passed through three stages: a LoRA fine-tuned base model, constrained decoding that caps output length and suppresses repetition, and deterministic string-level postprocessing. Each stage adds constraints rather than adding parameters, only Stage 1 involves learned weights.
4.3 Experimental Design We compared three systems on the same 50-sample test slice (drawn from the first 50 examples of the CommitPackFT test split): 1. Vanilla Mistral – the base model with the LoRA adapter disabled. This is our baseline. It tells us how much fine-tuning matters. 2. CommitLLM (Raw Fine-Tuned) – the fine-tuned model with default decoding (max_new_tokens=50, temperature=0.2). This isolates the contribution of the LoRA training itself.
2. Phase 1 – Training: load Mistral-7B-Instruct-v0.2 in 4-bit NF4, attach LoRA adapters, run SFTTrainer for one epoch, save the adapter.
3. CommitLLM (Final Pipeline) – the fine-tuned model with constrained decoding (max_new_tokens=20, temperature=0.1, repetition penalty 1.1) plus the deterministic post-processing pass. This is the system as we would ship it.
3. Phase 2 – Inference setup (clean session): reload the base model + adapter on a fresh runtime to free training-time VRAM. 4. Phase 3 – Batch generation: generate predictions for the test slice using both the fine-tuned model and (separately) the vanilla base model with the LoRA adapter disabled.
The evaluation was limited to 50 samples (out of 1,200 available in the test split) due to the Groq free-tier rate limit. Each LLM-as-a-Judge call sends the diff plus the ground truth plus the generated commit, which adds up to several thousand tokens per call. Across three systems and 50 samples, with the 2.1-second sleep between calls to stay under the per-minute rate, evaluation already takes about 6 minutes per pass. The deterministic metrics (compliance, length, BERTScore) would scale to the full 1,200 trivially; we kept the same 50 for consistency across all metrics.
5. Phase 4 – Quantitative evaluation: regex compliance, length distribution, BERTScore. 6. Phase 5 – LLM-as-a-Judge: async calls to Groq’s Llama models with a 1–5 rubric. 7. Phase 6 – Constrained decoding + post-processing: run the same model with the aggressive inference parameters and apply the deterministic string-cleaning pipeline.
We measured four things for every output: 1. Format compliance: a regex-based rule for what a “well-formed” commit message looks like (Section 5.2 for the corrected version).
8. Phase 7 – Final evaluation & visualization: re-run all metrics on the cleaned outputs and build the comparison plots.
2. Length: character count, plus the fraction of outputs at or below 72 characters (the de-facto subject-line limit in git).
The post-processing function is short enough to include verbatim: VALID_TYPES = {"feat", "fix", "docs", "style", " refactor ", "perf", "test", "chore", "build", "ci", " revert "}
3. BERTScore (F1): contextual-embedding similarity to the humanwritten ground-truth commit. Higher = closer to what a human would have written.
def clean( raw_output ): # Tag slicing s = raw_output .split("[/ INST]")[-1] if "[/ INST]" in raw_output else raw_output # Guillotine truncation s = s.strip ().split(’\n’)[0] # Conversational prefix stripping if ":" in s: prefix = s.split(":", 1) [0] words = re. findall (r’[a-zA -Z]+’, prefix ) if words and words [ -1]. lower () not in VALID_TYPES : s = s.split(":", 1) [1] return s.strip ()
4. LLM-as-a-Judge score (1–5): a strong LLM grades the generated commit against the diff and the ground truth, using a fixed rubric.
5 Results 5.1 Headline Numbers Table 1 shows the main result. CommitLLM with the full pipeline beats the vanilla baseline on every metric, often by a large margin, and the post-processing layer accounts for most of the improvement. A few observations from this table.
4.2 External Components We used the following third-party components:
The fine-tuning by itself (vanilla → raw) is a modest improvement: BERTScore goes up by 0.013 and LLM-judge goes up by 0.23 points. The model has learned something about the structure of commit mes-
• Mistral-7B-Instruct-v0.2 [5] as the base model. Loaded from
3
Figure 2: Vanilla Mistral vs. raw fine-tuned model. Fine-tuning narrows the length distribution and shifts BERTScore upward, but the long tail of verbose outputs persists.
Figure 3: Effect of the post-processing pipeline. Length distribution collapses onto a tight peak well below the 72-character ideal.
Figure 4: Three-way LLM-as-a-Judge comparison. Score distribution shifts from the 1–2 range (“garbage / poor”) into the 3–5 range (“acceptable / perfect”). The pipeline on top of the same fine-tuned weights (raw → pipeline) is a much bigger jump: BERTScore goes up by another 0.043, LLM-judge
sages, but it is still verbose, average length only drops from 154.8 to 113.4 characters, which is still well past the 72-char ideal.
4
Figure 5: Head-to-head: vanilla Mistral vs. the final CommitLLM pipeline on all four metrics. Table 1: Comparison across three systems on the 50-sample test slice. “Robust compliance” uses the corrected metric from Section 5.2. Metric Format compliance (robust) Average length % under 72 chars BERTScore (F1) LLM-Judge (avg / 5.0)
Vanilla Mistral
CommitLLM (Raw)
CommitLLM (Pipeline)
22.00% 154.8 chars ∼0% 0.8129 1.97
36.00% 113.4 chars 38.00% 0.8259 2.20
98.00% 37.9 chars 100.00% 0.8688 3.68
by another 1.48 points. This is the clearest evidence that for structured output tasks, treating the LLM as a system with a deterministic postprocessing layer pays off more than treating it purely as a model with weights.
The combined effect is the worst possible kind of measurement error: the metric was biased in favor of the system we were comparing against and against our own system. We replaced it with a stricter four-rule check (is_strictly_compliant):
5.2 The Robust Compliance Metric The format compliance number above (22% / 36% / 98%) uses a corrected metric. Our original metric did not.
1. Reject any output containing a newline character (kills the multi-line false positives). 2. Accept Conventional Commits with optional scope: fix(ui):, etc.
We started with a single regex:
3. Accept ID/date-prefixed [a-zA-Z0-9-]+: .+.
pattern = re. compile ( r"^(( feat|fix|docs|style| refactor |perf|test|chore|build|ci | revert ) (\([^) ]+\))?: .+) |(^[A-Z].+)$" )
commits:
anything
feat:, matching
4. Accept concise single-line commits starting with a capital letter or digit.
The first half of this regex (Conventional Commits prefixes) is fine. The second half, accept anything that starts with a capital letter, has two problems we did not catch until we looked at the data more carefully.
Under this corrected metric, the baseline correctly drops to 22% (because most of vanilla Mistral’s outputs were really paragraphs), and the pipeline correctly rises to 98%. The model weights did not change between the two versions of the metric, only what we were asking the metric to measure changed. We think it is worth flagging this explicitly: a poorly written evaluation metric can hide a system’s real performance, in either direction.
False positives for the baseline. Vanilla Mistral often produces multiline paragraphs like "Here is the commit message for your diff...". The regex sees H as the first character and rubber-stamps the whole paragraph as compliant, even though it is several lines of conversational text and would never pass a real commit hook. So the baseline’s compliance number was artificially inflated.
5.3 The Long Tail of Verbose Outputs One observation that motivated the constrained decoding step: even after fine-tuning, the raw model’s output length distribution still has a long tail past 200 characters. In other words, the average dropped, but a meaningful minority of outputs were still paragraph-length. For a production tool, the average is not what matters, the worst case is. A pre-commit hook that produces a 200-character paragraph one out of every four times is going to get disabled the same day it is installed.
False negatives for the pipeline. Our post-processing pipeline often produces commits like "132: Add command line interface" or "2014-06-17: Fix bug", issue-ID-prefixed and date-prefixed commits that are extremely common in real-world repos and that appear in the training data. The original regex rejected these because they start with digits, not capital letters. So the pipeline’s compliance number was artificially deflated.
5
The pipeline plot (post-processing comparison, Figure 3) shows what the constrained decoding plus post-processing does to that distribution: it collapses onto a tight peak in the 30–50 character range, with no tail past 72. This is exactly what we want for a commit-line generator.
structured-output tasks like commit message generation, engineering the system around the model matters more than engineering the model alone. The entire pipeline runs on a single NVIDIA T4 GPU, making it accessible to individual developers without cloud API dependencies.
5.4 Why CommitLLM Wins on BERTScore BERTScore measures contextual-embedding similarity, so it should be tolerant of paraphrase, “fix login bug” and “resolve authentication error” would both score high against the same ground truth. So why does the vanilla baseline score lower (0.8129) than our pipeline (0.8688), if both models presumably understand the diff?
References [1] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), 2022. arXiv:2106.09685. [2] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.14314.
The answer is verbosity. Vanilla Mistral often buries the correct answer inside a wrapping paragraph (“Here is a commit message describing the change where...”). Even when the right words appear, the surrounding filler dilutes the embedding similarity to the human reference. Our pipeline strips that filler, leaving a short string that aligns much more directly with the ground truth. So the BERTScore improvement is not (or not only) about the model getting better at understanding code, it is about the system getting better at concentrating the semantic signal into a commit-shaped string.
[3] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations (ICLR), 2020. arXiv:1904.09675. [4] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2306.05685.
6 Future Work IDE plugin. The most natural deployment target is a VS Code extension that, on a button click in the source-control panel, reads the current staged diff, sends it through the pipeline, and pre-fills the commit message box. The most viable approach is to assume the user has a local inference server (e.g., Ollama or llama.cpp) running and have the extension talk to that endpoint.
[5] A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. arXiv:2310.06825, 2023.
Mistral 7B.
[6] BigCode. CommitPackFT: A 2GB Filtered Subset of CommitPack with Natural-Language Instruction-Style Commits. HuggingFace dataset card. https://huggingface.co/datasets/bigcode/ commitpackft.
Full-test-set evaluation. We evaluated on 50 samples instead of the full 1,200 in the test split because the LLM-as-a-Judge calls hit the free-tier daily token limit. With paid API access or a self-hosted judge model, scaling to the full test set would be straightforward.
[7] Conventional Commits Contributors. Conventional Commits Specification, v1.0.0. https://www.conventionalcommits.org/.
More training epochs. We trained for one epoch only. Loss curves suggested the model was still improving at the end of training. A 2–3 epoch run with validation-loss-based early stopping might give a meaningfully better raw fine-tuned model and reduce how much work the post-processing layer has to do.
[8] HuggingFace. PEFT: Parameter-Efficient Fine-Tuning – Documentation. https://huggingface.co/docs/peft. [9] HuggingFace. TRL: Transformer Reinforcement Learning – SFTTrainer documentation. https://huggingface.co/docs/trl.
GGUF conversion for local deployment. Once the LoRA adapter exists, the standard recipe for local inference is to merge the adapter into the base, quantize the merged model to GGUF format, and run it under llama.cpp. This is a prerequisite for the IDE plugin path described above.
[10] HuggingFace. bitsandbytes: 4-bit and 8-bit quantization documentation. https://huggingface.co/docs/bitsandbytes. [11] Groq. Groq API documentation. https://console.groq.com/ docs.
Human evaluation. The LLM-as-a-Judge scores are a proxy. The true north-star metric is whether actual developers prefer CommitLLM’s output to either their own or to the vanilla baseline. A user study with developers blind-rating commits on a representative diff set would make the results more robust. Hallucination filtering for fake issue IDs. Because CommitPackFT contains real commits like "132: Add ...", our model sometimes emits commits with plausible but fabricated issue IDs. A regex filter in post-processing could strip these, or alternatively a learned classifier trained on a small set of labeled examples.
7 Conclusion We presented CommitLLM, a three-stage pipeline for generating git commit messages from code diffs. By combining QLoRA finetuning of Mistral-7B with constrained decoding and deterministic postprocessing, the system achieves 98% format compliance, an average output length of 37.9 characters, and an LLM-as-a-Judge score of 3.68/5—substantially outperforming the vanilla baseline on every metric. The key finding is that the deterministic pipeline layers (constrained decoding + post-processing) contribute more to output quality than the fine-tuning itself: the judge score improved by +0.23 from fine-tuning alone, but by +1.48 from the pipeline on top. This suggests that for
6