arXiv:2605.16517v1 [cs.SE] 15 May 2026
Customizing an LLM for Enterprise Software Engineering Aditya Kini
Satish Chandra∗
Milad Hashemi
Saksham Thakur
[email protected] Google
[email protected] Meta
[email protected] Google
[email protected] Google
Aditya Pandey
Vincent Nguyen
Marc Brockschmidt
Franjo Ivančić
[email protected] Google
[email protected] Google
[email protected] Google
[email protected] Google
Danny Tarlow
Parthasarathy Ranganathan
[email protected] Google
[email protected] Google
Petros Maniatis
Ahmed Omran
[email protected] Google
[email protected] Google
Zaheer Abbas
Anita Gergely
Martin Sevenich
Gufeng Zhang
[email protected] Google
[email protected] Google
[email protected] Google
[email protected] Google
Amy Hua
Alexander Frömmgen
[email protected] Google
[email protected] Google
Abstract
1
Enterprise software development is a continuous evolutionary process, characterized by incremental additions, architectural revisions, production deployments and rigorous maintenance. These activities generate valuable data that modern LLMs could be finetuned on, to unlock additional tool possibilities for enterprise software engineering. While frontier LLMs are already very capable, this form of customization offers a compelling path for enterprise-specific optimization. We introduce Gemini for Google (GfG), an adaptation of Gemini specialized for Google’s internal software engineering ecosystem. This paper details the model’s end-to-end development, from curating a trillion-token proprietary dataset to implementing a mid-training strategy that mitigates catastrophic forgetting. In a large-scale blind A/B study across 29,000 developers, Gemini for Google significantly outperformed baselines: reducing the mean number of iterations per turn by 23%, and increasing code survival rates by about 17%. Beyond metrics, we provide a comprehensive blueprint for enterprise model adaptation, covering: (1) the extraction of high-value signals from software engineering data, (2) data preparation strategies, (3) full-stack model tuning (continued pretraining and post-training), and (4) the deployment of downstream applications. We believe this methodology offers a replicable path for other organizations to unlock the full potential of their internal engineering data.
Building, maintaining, and evolving the software solutions that power large organizations requires an ecosystem of engineering activities: from design and documentation, to code generation, code verification, code review, rollouts, capacity management, and production monitoring. Each of these tasks requires specialized, often enterprise-centric and customized human expertise, and often generates artifacts, tools, and documents that are unique to a particular company or a particular codebase. Many recent products and research projects are centered around using large language models (LLMs) to accelerate software development, but often focusing mostly on high-visibility use-cases like code generation. This shift is already operating at a massive scale, as publicly disclosed figures indicate that over 75% of code at Google is now written or drafted by AI [20]. Despite this prevalence, standard evaluation benchmarks, such as HumanEval, primarily assess a model’s ability to generate greenfield solutions for algorithmic challenges. However, this focus on algorithmic generation creates a misalignment with industrial requirements. In practice, the majority of enterprise engineering effort is not dedicated to self-contained or context-independent code generation, but to the maintenance and evolution of existing systems: large-scale refactoring, efficiency optimization, and program comprehension within legacy environments. To effectively assist in these high-value maintenance tasks, LLMs must reason over the specific semantics of an organization’s proprietary ecosystem. At Google, this comprises a monolithic codebase replete with custom coding paradigms, deep integration of proprietary libraries, and high-velocity development patterns that create a continuously shifting context for any AI assistant. General-purpose
Keywords large language models, enterprise solution, automated code review, software quality, empirical study ∗ Work done while at Google.
Introduction
Kini et al.
frontier models possess strong reasoning capabilities but lack exposure to the unique data distributions—internal libraries, architectural patterns, and domain-specific languages—that are absent from public training sets. There are multiple strategies for making this enterprise data visible to, or at least influence, the inference carried out by a foundation LLM. One common approach used by many companies is to leverage retrieval mechanisms and prompt engineering (RAG) on top of generic LLMs without any specialization. The results of retrieval augmentation are quite mixed and depend on the quality of the retrieval and the manner in which the context is populated. Another method is to use an adaptation technique such as LoRA; however (owing to low rank) they lack the representational capacity required for a large-scale domain adaptation of a trillion-token corpus. Finally, organizations may choose to train smaller language models from scratch using enterprise data, a strategy that prioritizes complete control over the model architecture and training lifecycle. In this work, we present a complementary methodology: the direct specialization of foundation models via mid-training intervention and full-parameter fine-tuning. Unlike training from scratch, this approach branches off from a foundational checkpoint, preserving general reasoning capabilities while injecting deep domain knowledge. We propose a comprehensive blueprint for enterprise adaptation, detailing the extraction of high-value signals from software engineering data, data preparation strategies, and full-stack model tuning. While our implementation utilizes the Gemini foundation model, the core contribution of this paper is a replicable framework for data curation and training dynamics. This methodology allows organizations to apply these techniques to any available high-capability models, creating enterprise experts capable of addressing the full spectrum of development tasks. For brevity’s sake, a direct quantitative comparison to retrievalbased or scratch-trained approaches is outside the scope of this article. Training a model from scratch was not pursued due to the prohibitive computational cost and time constraints relative to the efficiency of our intervention strategy. Additionally, the evaluation of open-weight architectures was constrained by incompatibilities with our proprietary distributed training infrastructure and strict data governance protocols regarding the ingestion of internal code. We describe our process for curating the dataset, and provide three case studies on how the resulting model is used as a “firstparty software development assistance specialist”, reducing the human cost of developing and maintaining an enterprise-scale monorepo consisting of billions of lines of code, and improving the efficiency of Google’s planetary-scale data centers. The customized model, GfG, has been in use at Google to serve IDE features [25], to carry out large-scale migrations [18] and performance-improving code refactorings [13], and in other important use cases. Evaluations in these contexts have been presented in the respective articles; we recap these briefly in §6. Further, in this article, we present online evaluations with developer productivity at Google in general as well as offline evaluations with selected coding benchmarks. Contributions. This paper makes the following contributions: □ A Framework for Enterprise Adaptation: We provide a blueprint for building a specialist foundation model from
internal engineering data covering data curation, training, and deployment. □ Mid-Training Intervention Strategy: We employ a midtraining intervention to bridge the gap between general pre-training and post-training via a torso-patch to adapt model weights for Google’s internal software engineering ecosystem. □ Enterprise-Scale Evaluation: We demonstrate the effectiveness of this approach in a blind A/B study with over 29,000 Google developers showing that Gemini for Google (GfG) significantly outperforms the baseline Gemini model by reducing conversational iterations while increasing code acceptance and survival rates. Overview. The rest of the paper is structured as follows. We next provide some background on our developer ecosystem in §2, before we define the data curation methodology that addresses the core target capabilities demanded of a bespoke enterprise model in §3. Next, we detail the mid-training intervention strategy used to specialize the model in §4 and the subsequent supervised finetuning in §5. We then evaluate through online and offline results such as large-scale A/B study and case studies in §6. We conclude with lessons learned and a discussion of future work in §7.
2
Background: Google’s Engineering Ecosystem
Our study is situated within a hyperscale industrial software environment tailored to support global-scale services. Development is centralized within a single monolithic repository (“google3”) comprising billions of lines of code. This environment is characterized by a high degree of reliance on proprietary infrastructure rather than standard open-source equivalents: • Core Libraries and Infrastructure: Extensive use of internal libraries/frameworks and custom distributed storage (colossus [10]) and processing engines (e.g. F1 [22], flume [3], Borg [26], Chubby [2]). • Development Workflow: Engineering activities are conducted via specialized internal tools, including the Cider IDE and Clients in the Cloud (CitC) for workspace management. • Quality Assurance And Review: Code evolution is managed through a rigorous peer-review process via Critique [27, Chapter 19]. Submissions–known internally as Changelists (CLs), similar to PRs or diffs–undergo automated static analysis, linting, and regression testing before and during the review process. Reviewers focus on correctness, maintainability, and test coverage, creating a collaborative dialogue that iteratively refines the codebase. • Observability: A bespoke suite of production monitoring tools provides fine-grained logging, tracing, and efficiency analysis. Crucially, this entire lifecycle—from IDE interactions to code review comments and production logs—generates a massive, structured corpus of software engineering data. Unlike primarily codecentric datasets, this internal data captures the full provenance of software evolution, providing the necessary signal to train models on the nuanced requirements of an enterprise stack [16].
Customizing an LLM for Enterprise Software Engineering
3
Methodology
In order to develop an enterprise expert model, we first define the necessary core capabilities to support Google’s internal development workflows. These requirements then inform our data curation strategy to ensure alignment with the downstream applications.
3.1
Target Capabilities
We focus on high-value areas to provide signficant value to Google’s engineering workflows:
Figure 1: A sample interaction in the code review tool which was resolved by relevant code edits by the code author
• Code Transformations / Automated Program Transformation [25]: We envision an LLM that acts as a “code transformation specialist”, capable of (a) automatically refactoring code, (b) improving performance, (c) migrating code to new APIs, and (d) fixing bugs. Given the size and complexity of Google’s codebase, this capability offers the potential for improving our engineering velocity by automating some code maintenance tasks. • Internal Chatbots/Knowledge Discovery: Navigating Google’s internal documentation and APIs can be challenging given our scale. A Google-aware model has the ability to understand user queries about internal products, which is especially useful given that many internal systems have proprietary names. • Code Completion: While smaller code-generation models were deployed at Google before this work, our intent was to push the boundary in this space using Gemini models: producing a code generation engine that understands larger contexts and anticipates developer intent, writing an ever more substantial fraction of Google’s code. There are several additional use cases that benefit from a custom model, e.g. assistance with debugging, but we focus on the above topics in this paper.
3.2
Data Curation
To satisfy the above requirements, we curated a corpus capturing diverse artifacts generated during software development. We categorize this dataset into six primary domains, each providing distinct supervision signals for the model. (1) Change-Generation: Google’s rigorous code review process provides a broad set of data on code transformations. By including code review data and developer conversations, we provide the LLM with information on imperative code transformation showing how code is improved, refactored, and optimized. One representative source of data in this category is the comment resolution dataset. Building this dataset involves extracting tuples of the initial code state, comment, and final code state: (𝐶 initial, comment, 𝐶 final ) where the difference between 𝐶 final and 𝐶 initial represents the semantic edit required to resolve the reviewer’s comment. As discussed in prior work[6], this data enables the model to learn the nuances of design, complexity, and maintainability by modeling the “critique-and-refine” dialogue inherent to high-quality software engineering.
Figure 2: An example of a question asked by a SWE on Google’s internal Q&A tool. These discussions between developers and domain experts can be synthesized to form good knowledge category training data.
(2) Knowledge: To power an internal chatbot, we need to provide the LLM with a deep understanding of Google’s internal knowledge base. To this end, we aggregate unstructured and semi-structured knowledge artifacts, such as internal technical documentation, API references, design documents and an internal Q&A repository. (3) Issues and Fixes: We leverage Google’s large internal datastore for capabilities like bug fixing, build errors fixing, test failures and other issues. By including data on these issues and their corresponding code fixes, we provide the LLM with valuable data on development context, how developers break down and solve problems, and how we diagnose production events that may arise.
Kini et al.
(6) Activity Timelines: Finally, we incorporate coarse-grained workflow data to capture the broader context of engineering tasks. This includes activity timelines that map tool usage, search queries, and documentation access patterns, providing the model with a holistic view of how engineers navigate the ecosystem to complete complex tasks [16]. Figure 3: A developer’s IDE editing session is visualized as a series of snapshots, numbered sequentially. Fine-grained logging within the IDE captures each editing action. In this example, a build failure occurred at snapshot 2, and the build was successful at snapshot 5. Consequently, the changes made between snapshot 2 and snapshot 5 represent the edits that resolved the build issue.
A key contribution in this domain is the Build Fixing dataset. By utilizing fine-grained IDE instrumentation, we reconstruct developer sessions as a sequential series of file snapshots. We identify discrete intervals [time_initial (𝑡𝑖 ), time_final(𝑡 𝑗 )] where the build state transitions from Failure → Success. The code difference Δ(𝑡𝑖 , 𝑡 𝑗 ), contextualized by the corresponding error at 𝑡𝑖 , provides a source of implicit supervision for automated program repair (APR), allowing us to extract millions of training examples without manual annotation [12]. Other types of data in this category include test failure fixes, bug fixes, fixes for errors discovered by the wide range of static analyzers, and fixes for style guide violations. (4) Code Generation [24]: To support cases like code completion we include code written by Google developers, processed into multiple formats (e.g., FIM [1]) and granularities (single file, multi-file) to improve the model’s ability to generate both short and long-form code completions. (5) Logs and Performance: This category includes a subset of Google’s production metadata and events - focusing on code efficiency and debugging traces. This kind of data provides context to the model to develop efficient and reliable software systems. One notable source of data in this category is our repository of efficiency edits. This is helpful in bridging the gap between static code and dynamic execution. This corpus is derived from a history of optimization changes targeting production services. By identifying commits explicitly linked to latency reduction or CPU/memory optimization—and pairing them with the performance anti-patterns they resolved—we train the model to recognize and suggest efficiency improvements. This aligns the model’s generation objectives with production health metrics [13]. We also incorporate sources like Dapper [23] — a system used for tracing and visualizing computations across Google’s distributed computation stack—to capture the causal and temporal structure of these operations, providing critical visibility into runtime behaviors such as inter-service dependencies and latency bottlenecks. By training on these traces the model learns to reason about complex system interactions and diagnose execution-path inefficiencies that are often invisible to static analysis tools.
3.3
Data Format
While transformer-based, decoder-only architectures inherently rely on autoregressive (causal) modeling, where, for a continuous Î stream of tokens 𝑥 = (𝑥 1, . . . , 𝑥𝑛 ), the likelihood 𝑃 (𝑥) = 𝑖 𝑃 (𝑥𝑖 | 𝑥 <𝑖 ) is maximized for a subsequent token. Applying this objective naively to heterogeneous software engineering tasks yields suboptimal results. To address this, we design our data such that the model can not just learn correct syntax, but also deduce and correctly predict code changes requested by a user through some natural-language instructions, using autoregressive decoding only. When training on the causal language modeling task, we apply masking of the input tokens to limit the loss calculation exclusively to the modified tokens. In the following, we present some of the employed data representations:1 (1) Unstructured natural language and semi-structured operational data (e.g., technical documentation, production logs): This data covers the Knowledge and Operational Semantics categories, where the objective is next-token prediction based on historical context: 𝑃 (Target) = 𝑃 (Token | Prefix, Prefix + Token ∈ Document) (2) Instruction-Guided Code Editing (Code-to-Diff) Here, we provide the model with some context, including some original code and a user instruction on the desired changes (Codeoriginal , Instruction). The model’s target is to predict a code diff Δ that satisfies the instruction: 𝑃 (Target) = 𝑃 (Δ | Codeoriginal, Instruction).
(1)
One example for a commonly used diff format is the unified diff, shown in Fig. 4. A wide range of datasets from the Change Generation category mentioned above fit into this representation. Some examples (not exhaustive) are listed below. • CL Description to Change: Derived from submitted changelists (CLs), where the “Instruction” is the CL description and the target is the submitted patch. • Code Comment Resolution: Derived from peer-review interactions. The input is the code snippet plus the reviewer’s comment; the target is the edit that resolved the comment [6]. This code diff representation also extends well into the issues and fixes category of datasets, where the issue is described using its natural language description or by annotating the code with specific issues like compilation errors or test failures: 𝑃 (Target) = 𝑃 (Δ | Codebroken, Errorsignal ).
(2)
1 The scope of this paper is limited to human-generated trajectories; the analysis of
synthetic or agent-generated data falls outside the current research objectives.
Customizing an LLM for Enterprise Software Engineering
Figure 5: A multi-file example.
which show what actions developers took over shorter periods of time as they write code. To represent this data, we concatenate the text of the different issues or developer actions alongside the generated code in diff format, up to the size of the training window. A prominent example in this category is the Change Generation ReAct dataset. Leveraging the ReAct (Reasoning and Acting) framework [28], we synthetically reconstruct the latent development trajectory for historical changelists. Rather than mapping a description directly to a final diff, we model the plausible sequence of intermediate actions – such as code search queries, file navigation, and reasoning steps – that a developer would undertake to implement the specified change.
Figure 4: A comment resolution example. One example from this category that fits into this representation is listed below. (3) Automated Program Repair (APR): We adapt the diff format for the Issues & Fixes category. Here, the “Instruction” is structured as a compiler error or static analysis warning, and the target is the fix. By modeling compiler errors as “automated review comments,” we unify the representation of human and machine feedback, as described in [12]. (4) Context-Aware Infilling (FIM): This format is used to support cases like code completion within existing files. This “Code Generation” category of data uses FIM-based(Fill in the middle) formats [1], where the code section to be predicted is split into a suffix, middle and prefix with sentinel tokens used to delineate the different sub-sequences. This training method enhances the infilling ability of the models which have strong autoregressive generation abilities. By employing a distinct preprocessing technique, the method divides code into three segments: Prefix, Middle, and Suffix. These segments are then rearranged for training, allowing Middle tokens to rely on both Prefix and Suffix rather than solely Prefix, as typical in causal attention models. We also apply FIM to multi file examples where context across files also aids the model’s ability to generate code. We leverage three types of formats for this dataset: 1. Untransformed: <filepath>\n<content>[eod] 2. PSM: <filepath>\n[pre]<prefix>[suf]<suffix> [mid]<middle>[eod] 3. SPM2: <filepath>\n[pre][suf]<suffix>[mid] <prefix><middle>[eod] The corresponding data with multi-file examples is shown in Fig. 5. (5) Multi-Turn Agentic Trajectories. In order to capture the iterative nature of software development we include multistep journeys datasets from extended periods of time. These data sources could be “bugs”, where developers compile sets of changelists to complete large tasks, or Activity Timelines,
4
Mid-Training Intervention
Using the datasets above, we compile a trillion token-scale data mixture for auto-regressive training, similar to prior work [8, 9, 14, 21]. This is accomplished by intervening towards the end of pretraining, injecting our datasets, and continuing to train the model on the expanded corpus. We train two distinct model variants to address different latency and capability requirements: • Gemini-Flash based: Optimized for low-latency and highthroughput tasks such as real-time code completion. • Gemini-Pro based: Optimized for complex reasoning tasks such as refactoring and architectural Q&A. Our final mixture consists of these categories of data mixed in with some breakdown that works best for our downstream applications. Table 1 showcases a sample distribution for illustrative purposes only. Table 1: Distribution of Software Engineering Tasks Task Category
Percentage (%)
Code Generation SWE Timelines Issues and Fixes CL Generation Knowledge Logs and Performance
25.0 23.0 22.0 20.0 5.0 5.0
Total
100.0
Kini et al.
4.1
Training-Time Evaluation
During the mid-training intervention phase, we continuously assess intermediate checkpoints to validate convergence and ensure alignment with training objectives. These training-time signals facilitate evaluation against real-world enterprise scenarios using a curated dataset derived from product logs—including IDE interactions and developer assistance features. Our evaluation suite encompasses a diverse set of tasks:
The sources used to generate these internal datasets (at the time of this writing) include:2 • Code Transformation [24]: Logs capturing natural language instructions for code modifications within the Cider IDE, paired with model-generated outputs and subsequent user feedback (acceptance, rejection, or rewriting), providing a measure of output quality. This dataset represents many different types of interactions of the developers with the code transformation capabilities in the IDE, including refactorings of the code, implementing methods when the docstrings and signatures have been created, generating tests for already written code or adding logic described in an NL prompt to the existing code (see Figure 6). • Migrations: A carefully compiled collection of high-quality prompts and corresponding model outputs generated through usage of an internal code migration tool, facilitating the transition of the codebase from legacy to contemporary technologies. • NL2SQL: A comprehensive corpus of SQL queries paired with their corresponding natural language descriptions, crowd-sourced from internal software engineers. • Q&A: A refined set of high-quality question-and-answer pairs extracted from the interaction logs of internal chatbot systems and other knowledge retrieval platforms (see Figure 7).
(1) Code Transformation: Natural language-to-code tasks derived from logs of our internal IDE and manually curated for quality assurance. (2) Code Q&A: Conversational code queries featuring expertverified responses. (3) Unit Test Generation: Automated generation of test cases for code diffs. (4) Code Completion: Analysis of model recommendations based on historical developer acceptance logs. (5) Code Migration: Automated refactoring (e.g., int32 to int64 transitions). (6) Efficiency Refactoring: Edits optimized for performance improvement. (7) Build Repair: Automated resolution of build failures. We assess model performance using a multi-dimensional framework: (1) Static Analysis: Code similarity metrics (e.g., fuzzy match, BLEU) against the ground-truth developer action. (2) Model-Based Evaluation: “LLM-as-a-Judge” autoraters that assess semantic correctness and helpfulness. (3) Execution Metrics: For tasks like automated repair, we verify correctness via compilation and test execution. This continuous evaluation protocol provides robust signals, enabling precise validation of model convergence and selection of optimal checkpoints.
4.2
Data Mixture and Ablation Strategy
In order to determine the final mixtures, we run several ablations with models of smaller sizes and different weights assigned to different categories or different datasets in the mixtures. We then observe the impact on evaluations to identify the best mixture weights. Running these ablations with smaller models enables us to run ablations quickly and efficiently. Once we have the final mixtures derived from these ablations, we then scale our training to larger models.
5
Post-training
Following mid-training, the model undergoes a post-training phase to optimize its utility for downstream applications. This phase utilizes Gemini’s post-training methodologies and curated dataset mixtures, augmented with high-quality internal datasets. These proprietary datasets are derived from crowdsourced validation from employee volunteers or from anonymized logs of interactions with internal products and tools, undergoing refinement for quality through expert human review.
These datasets undergo rigorous manual curation to ensure high data quality. A representative subset of examples from each dataset is held out for evaluation purposes. Subsequently, these refined datasets are integrated with Gemini post-training instruction tuning mixtures and this combined mixture is used during the Supervised Fine Tuning phase of our post-training.
6
Results
In the following, we present some lessons learned from applying this kind of model customization in an enterprise setting using Gemini for Google (GfG) within the Google ecosystem. We designed our evaluation to answer the following primary questions, and finally highlight some additional lessons: Q1: Model Proficiency. How does the finetuned GfG model compare to the Gemini baseline on standard public benchmarks? Q2: Industrial Utility. What is the impact of GfG on developer productivity and code adoption rates in a live enterprise environment? Q3: Developer Assistance. To what extent can an enterprisetuned LLM enhance software engineering productivity by automating high-volume maintenance, optimizing resource efficiency, and augmenting interactive developer workflows? Q4: Mitigating Catastrophic Forgetting. What kind of regressions do we see when creating a specialized enterprise LLM? How do we mitigate such regressions, if discovered? Q5: Lessons from Data. What are the insights and drawbacks that we discovered during the curation and scaling of the trillion-token proprietary dataset? 2 The analysis of synthetic or agent-generated trajectory data in SFT or RL falls outside
the scope of this article.
Customizing an LLM for Enterprise Software Engineering
Figure 6: Illustration of how a logged interaction between the developer and the NL based code transformation feature in the IDE can be leveraged to generate training data.
Figure 7: Illustration of how a logged interaction between the developer and the internal developer assistance chatbot is converted to a post-training example.
Model Setup: In the evaluations below we compare the Gemini 2.5 Pro model with the equivalent Gemini for Google version. While newer model iterations (e.g., Gemini 3) exist, we employ Gemini 2.5 Pro to maintain consistency with our extensive internal benchmarking datasets, which were most complete for the 2.5 architecture during the data collection period.
6.1
Offline Evaluation (Q1)
Assessing model proficiency requires a combination of external and internal benchmarks covering both coding tasks and general reasoning. Our internal benchmarks assess the model’s ability to execute tasks representative of a Google Software Engineer’s daily workflow, while the external evals are chosen for their reliability and established use in evaluating frontier models. For example, we relied on SWE-bench Verified [5] for its focus on resolving real-world GitHub issues, Aider [7] for multi-turn editing, and LiveCodeBench [11] for evaluation of broader coding tasks beyond code generation. As seen in Table 2, GfG generally outperforms the baseline Gemini checkpoint on the majority of the coding benchmarks. The model performs marginally better on the Aider benchmarks, notably with a 13% improvement on the Edit Block format that requires “Search
Figure 8: The E2E training process and Replace” blocks. GfG’s proficiency comes from the “critiqueand-refine” examples described in §3.2 where the model is trained to handle the block-like nature of code reviews, where comments are strictly associated with specific code snippets. While GfG shows slight regression on the unified diff (udiff) format likely due to the strict line-prefix required for exact context matching, it remains competitive on whole file and multi-turn editing. Despite the trillion-token injection, Table 3 confirms that cognitive reasoning and multimodal capabilities are preserved. GfG maintains or improves over baseline performance on ARC AGI [4] and GradQA which respectively measure the ability to solve abstract puzzles and graduate-level reasoning. To verify multimodal stability, we used MMMU-Pro [29] which tests reasoning across
Kini et al.
Evaluation Metric
Relative Performance
External Benchmarks SWE-bench Verified Aider Polyglot (Whole File) Polyglot (Edit Block) Polyglot (Unified Diff) Polyglot Multiturn (Edit Block) LiveCodebench Execution Unit Test Internal Benchmarks Google SWE Tasks Google Code Transformations
1.05 1.04 1.13 0.94 1.01 1.01 0.97 1.05 1.04
Table 2: Performance of the GfG checkpoint compared to Gemini 2.5 Pro model on some external and internal coding benchmarks. Values > 1 indicate GfG superiority.
Evaluation Metric
Relative Performance
Hidden Math - Hard Subset avg@4 Humanity’s Last Exam
0.99 0.84
IMO AnswerBench V1 pass@1
0.94
ARC AGI GradQA MMMU - Pro
1.04 1.03 0.96
Table 3: Relative performance of GfG compared to the baseline Gemini on non-coding benchmarks to monitor for regression. Despite massive domain-specific training, both the fundamental intelligence and multimodal reasoning of the base model were preserved throughout the mid-trainign intervention and subsequent post-training.
both visual and textual formats. while exhibiting marginal variances in expert-level domain assessments like Humanity’s Last Exam (HLE) [19] and IMO AnswerBench [15].
6.2
Table 4: Qualitative comparison of GfG vs Gemini 2.5 Pro for key online performance metrics. These results highlight improvement across the board for GfG.
Online Evaluation (Q2)
The quantitative and qualitative impact of the fine-tuning can also be measured through an A/B test. We ran an A/B test comparing baseline Gemini with the fine-tuned GfG model to measure the production impact. The following section lists the details of our observations: 6.2.1 Experimental Setup. To rigorously assess the performance of GfG against the baseline model, we conducted a blind, randomized A/B experiment in September 2025. We evaluated the model’s performance across 29,000 active users by deploying Gemini for Google into the critical development workflows and IDEs that power engineering across Google. Agentic coding development
Dimension
Key Metric
Relative Change
Efficiency
Conversational Iterations Turn Latency
−23.1% −8.9%
Quality
Hunk Acceptance Rate Code Line Acceptance
+4.5% +4.5%
Impact
Code Submission Rate Code Survival Rate
+11–14% +16.8%
in IDE’s was the major use case. The agentic coding development paradigm empowers the model to autonomously plan, edit across multiple files, and iteratively self-correct using test feedback, effectively elevating the developer’s role from manual implementation to high-level orchestration. Within this framework, prompting becomes a critical interface: rather than simple queries, prompts serve as strategic specifications that encode the intent and constraints required for the reasoning engine to navigate complex, multi-step engineering tasks with minimal oversight. The model served as the reasoning engine for an integrated agentic coding workflow, facilitating real-time developer assistance. To eliminate selection bias, users were blindly assigned to either the treatment group (GfG) or the control group (Gemini) for the duration of the experiment. The experiment compared Pro models (based on Gemini 2.5 Pro) and Flash models, with Pro models accounting for 97% of the total usage. The analysis focused on data collected over seven full working days to ensure statistical validity. A power analysis confirmed that this duration provided sufficient statistical power to detect significance across the majority of key metrics. 6.2.2 Quantitative Results. GfG demonstrated statistically significant improvements over the baseline across all critical productivity and quality metrics (see Table 4): • Efficiency and Latency: GfG significantly reduced the cognitive and temporal load on developers. We observed a 23.14% reduction in the mean number of iterations per turn, indicating that Gemini for Google required fewer conversational turns to satisfy user intent. Additionally, the mean end-to-end turn duration decreased by 8.85%. • Code Acceptance: Code quality, measured by user acceptance, showed clear gains. The hunk acceptance rate (the percentage of contiguous model-generated blocks, also called hunks, accepted by the developer) was 4.49% higher for GfG, while the code line acceptance rate experienced a parallel growth of 4.45%. • Production Impact and Code Survival: GfG drove higher engagement and long-term value. There was an 11—14% increase in submitted code changes utilizing the agent in the treatment group. Furthermore, code generated by GfG exhibited better persistence; the survival rate of accepted code in final submissions was 16.80% higher compared to the baseline. In an ecosystem where over 75% of all code is
Customizing an LLM for Enterprise Software Engineering
already written or drafted by AI, submission and survival rates better capture actual engineering value and true utility. GfG’s superior performance on both these fronts shows that it effectively bridges the gap between raw AI generation and enduring, production-grade software. 6.2.3 Qualitative Observations. User feedback revealed a gap between models that highlight the benefits of domain specialization in GfG: • Perceived Quality Drop: Users in the control group reported that the baseline model felt less capable and represented a quality regression. • Domain-Specific Struggles: The baseline model required significantly more turns to correctly implement internal build rules and imports compared to GfG. • User Friction: The performance gap was significant enough that users sought workarounds to bypass the experiment and restore access to GfG.
6.3
Case Studies in Industrial Application (Q3)
We evaluate the model’s deployment across three distinct modes of operation: large-scale automated refactoring (migrations), autonomous code optimization, and automatic paste fixing. (1) Migrations [18] Google successfully employed this model to migrate substantial, decade-old codebases for certain product areas away from older systems. 6.3.1 Case Study A: Infrastructure Modernization (Ads product area). A notable instance was the migration within Ads product area, transitioning from 32-bit numerical IDs for entities like users and campaigns to 64-bit integers to avoid overflow errors. It is estimated that the total time was reduced by 50% compared to manual methods, with AI fully authoring 80% of the code changes in submitted changelists. 6.3.2 Case Study B: Framework Migration (JUnit). This model was used to migrate a substantial set of legacy JUnit3 test files to the modern JUnit4 framework. This automated approach successfully migrated 5,359 files (over 149,000 lines of code) in 3 months, with approximately 87% [24] of the AI-generated code committed without human modification, the main bottleneck being the human review process. (2) Code Efficiency [13] Another application of this model is ECO (Efficient Code Optimizer). ECO detects performance antipatterns in the warehouse-scale codebase and utilizes the model to synthesize optimization patches. • Pipeline: The model acts as the core reasoning engine, taking profiling data and source code as input to generate refactors, which are then verified and submitted for human review. • Scale of Impact: The system has deployed over 6,400 commits (>25,000 lines of changed production code) with a >99.5% production success rate. • Hardware Savings: These optimizations translate to substantial hardware efficiency, saving an estimated 500,000 normalized CPU cores per quarter.
Beyond the described use cases above, this model trained on a wide range of internal enterprise datasets has seen widespread adoption across Google’s internal development infrastructure. Its applications include, but are not limited to: • Automated Unit Test Generation: Reducing the friction of increasing test coverage. • Production Trace Analysis: Identifying bottlenecks by correlating server traces with source code. • Conversational Assistance: An internal chat application for architectural Q&A. (3) In-IDE Assistance [17] “Smart Paste” is a production IDE feature that utilizes the model to intelligently adapt copied clipboard code content to its destination context. This tool addresses the frequent need for manual “fix-up” edits—such as reconciling imports, variable names, and formatting—immediately after a paste action. Utilizing GfG was essential to handle proprietary libraries and meet strict millisecond-latency requirements for an in-flow experience. • Adoption and Impact: The feature has seen widespread adoption, utilized daily by tens of thousands of developers with a 45% acceptance rate. • Efficiency: Each accepted suggestion saves an average of 22 keystrokes (capturing deletions and replacements), and collectively, code generated by Smart Paste now accounts for over 1% of all code produced. • Code Survival: The model’s suggestions demonstrate high utility, with approximately 58% of the generated characters remaining after 30 minutes. • Emergent Workflows: Beyond standard adaptation, the model’s capabilities enabled developers to use copy-paste as a tool for lightweight repetitive refactoring (“chaining”) and context-aware cross-language translation. “Transform Code” is an implementation of inline chat that allows for a natural language instruction to modify a highlighted section of code. GfG provides the basis for contextual awareness of internal code. • Productivity: Early adopters of inline chat showed a 15% increase in throughput over the control. • Acceptance: 35,000 accepted suggestions per week, with a 70% acceptance rate.
6.4
Mitigating Catastrophic Forgetting (Q4)
A core challenge in mid-training intervention is retaining the general reasoning capabilities of the base model while injecting domain knowledge. Problem: Our initial methodology involved starting GfG’s continued pre-training from the 100% Gemini pretrained checkpoint. While the model excelled at internal brownfield tasks, its general reasoning and foundational proficiency on standardized tasks degraded due to the distributional shift between internal data and the general web corpus. Solution: We adopted two strategies: (1) Data Replay: We mixed some of the original pre-training data (parent model data) into the enterprise mixture. This anchors the model’s weights, preserving general capabilities.
Kini et al.
(2) Mid-training intervention: Instead of extending training from the final checkpoint , we branch off from an earlier checkpoint (e.g., before the final convergence phase of the base model).
Model Variant Gemini GfG with continued pretraining GfG mid-training
Data Science Workflows
NL Code Synthesis
Competitive Coding
45.8 11.9
74.6 36.3
16.6 1.3
39.0
70.3
12.0
Table 5: Performance comparison showing the benefits of data replay and mid-training in mitigating catastrophic forgetting on different reasoning and general coding tasks. These benchmarks evaluate proficiency in data science problem-solving, generating code from natural language intent, and solving greenfield programming problems. Results are reported as percentage scores (pass@1).
Additional changes were also needed for adapting the mixture of expert model architecture for specialized use cases. Mixture-ofExperts (MoE) architectures are typically designed as generalists. We observed that standard router load-balancing mechanisms failed during domain-specific fine-tuning. During training, evaluation metrics saturated at approximately 45% of the run. Investigation revealed significant token dropping, as the router disproportionately favored specific experts for the highly homogenous internal data. Hence, we needed to customize the expert routing technique to prevent expert collapse.
7
Lessons Learned (Q5)
Through the development and deployment of our software engineering models, we identified critical challenges in data curation, alignment, and evaluation. We distill our experience into the following three lessons: (1) Data Composition: The maxim “more data is better” does not always hold — pre-training distributions can establish priors that conflict with post-training. Observation: We included a ReAct-style dataset in pre-training intended to improve reasoning. However, this dataset interacted poorly with the post-training phase, degrading the model’s function-calling behaviors. Insight: This highlights negative transfer, where learning a specific formatting style (e.g., verbose reasoning traces) interferes with the precise syntax required for tool use. Careful data ablation is required to balance “learning to code” vs. "learning to use tools." (2) Schema Alignment Minimizing Distributional Shift. Fighting the base model’s data format is computationally expensive and architecturally risky. Observation: Early attempts to restructure data into custom internal formats led to regression issues during posttraining.
Takeaway: Aligning internal data to match the pre-training format of the base model (Gemini) proved essential. It minimizes the "syntax shock" to the model and simplifies the transition between public and proprietary capabilities. (3) The Evaluation Gap Beyond Static Benchmarks. Evaluating generative models for enterprise engineering remains an unsolved problem. Standard “pass@k” metrics on unit tests do not capture the nuance of maintaining legacy code. We found that execution-based metrics often failed to fully capture developer satisfaction. Consequently, we augmented our evaluation strategies with “Autorater” assessments (LLM judges) and live production telemetry (acceptance rates) in addition to static benchmarks.
8
Conclusion
We introduced Gemini for Google, a specialized adaptation of Gemini tailored for Google’s internal software engineering ecosystem. By fine-tuning on a trillion-token proprietary corpus and employing novel strategies like mid-training intervention we successfully mitigated catastrophic forgetting while capturing complex enterprise semantics. A large-scale blind A/B study across 29,000 developers demonstrated that Gemini for Google significantly outperformed baselines. Ultimately, this work provides a replicable blueprint for organizations to leverage their own engineering data for full-stack model adaptation, from data curation to downstream application deployment.
Data Availability Statement The datasets, codebases, and infrastructure utilized in this study— including internal IDE telemetry, code review logs, and proprietary training mixtures—are confidential and proprietary. Due to stringent data governance, corporate security policies, and privacy protocols, these artifacts cannot be made publicly available. We have provided comprehensive methodological details in the paper to ensure the principles of our approach can be understood and replicated by the community on alternative datasets.
Acknowledgements We thank our colleagues at Google, particularly Oriol Vinyals, Niranjan Tulpule, Madhura Dudhgaonkar, Leonard Berrada, Dawn Chen, Hannah Lin, Patrick Musau, Amita Gondi, Arman Hasanzadeh, Dan Zheng, Deniz Altınbüken, Fred Lewis, Hannah Lin, Maxim Tabachnyk, Pat Rondon, Sandeep Katragadda, Sergei Shmulyian, Stephanie Tang, Stoyan Nikolov, Kashmira Phalak, Michael Golahi, Harshit Vadodaria, Sandeep Katragadda, Jerry Peng, Martin Dixon, and David Lo.
References [1] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient Training of Language Models to Fill in the Middle. arXiv:2207.14255 [cs.CL] https://arxiv.org/abs/2207. 14255 [2] Mike Burrows. 2006. The Chubby lock service for loosely-coupled distributed systems. In 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI). [3] Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert Henry, Robert Bradshaw, and Nathan. 2010. FlumeJava: Easy, Efficient Data-Parallel Pipelines. In ACM SIGPLAN Conference on Programming Language Design and
Customizing an LLM for Enterprise Software Engineering
Implementation (PLDI). 2 Penn Plaza, Suite 701 New York, NY 10121-0701, 363– 375. http://dl.acm.org/citation.cfm?id=1806638 [4] Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. 2025. ARC Prize 2024: Technical Report. arXiv:2412.04604 [cs.AI] https://arxiv.org/abs/ 2412.04604 [5] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. https://openai.com/index/ introducing-swe-bench-verified/ [6] Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Dan Zheng, Satish Chandra, and Petros Maniatis. 2024. Resolving Code Review Comments with Machine Learning. In 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). [7] Paul Gauthier. 2025. Aider Polyglot Coding Leaderboard. https://aider.chat/ docs/leaderboards/ [8] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783 (2024). [9] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yun Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [10] Dean Hildebrand and Denis Serenyi. 2021. A peek behind Colossus, Google’s file system. Google Cloud Blog. https://cloud.google.com/blog/products/storagedata-transfer/a-peek-behind-colossus-googles-file-system [11] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https://arxiv.org/abs/2403.07974 [12] Emily Johnston and Stephanie Tang. 2024. Safely repairing broken builds with ML. https://research.google/blog/safely-repairing-broken-builds-with-ml/. Google Research Blog. [13] Hannah Lin, Martin Maas, Maximilian Roquemore, Arman Hasanzadeh, Fred Lewis, Yusuf Simonson, Tzu-Wei Yang, Amir Yazdanbakhsh, Deniz Altinbüken, Florin Papa, Maggie Nolan Edmonds, Aditya Patil, Don Schwarz, Satish Chandra, Chris Kennelly, Milad Hashemi, and Parthasarathy Ranganathan. 2025. ECO: An LLM-Driven Efficient Code Optimizer for Warehouse Scale Computers. arXiv:2503.15669 [cs.SE] https://arxiv.org/abs/2503.15669 [14] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel LamyPoirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173 (2024). [15] Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V. Le, and Junehyuk Jung. 2025. Towards Robust Mathematical Reasoning. arXiv:2511.01846 [cs.CL] https://arxiv.org/abs/ 2511.01846 [16] Petros Maniatis and Daniel Tarlow. 2023. Large sequence models for software development activities. https://research.google/blog/large-sequence-modelsfor-software-development-activities/ [17] Vincent Nguyen, Guilherme Herzog, José Cambronero, Marcus Revaj, Aditya Kini, Alexander Frömmgen, and Maxim Tabachnyk. 2026. Smart Paste: Automatically Fixing Copy/Paste for Google Developers. arXiv:2510.03843 [cs.SE] https://arxiv.org/abs/2510.03843 [18] Stoyan Nikolov, Daniele Codecasa, Anna Sjovall, Maxim Tabachnyk, Satish Chandra, Siddharth Taneja, and Celal Ziftci. 2025. How is Google using AI for internal code migrations? arXiv:2501.06972 [cs.SE] https://arxiv.org/abs/2501. 06972 [19] Long Phan et al. 2025. Humanity’s Last Exam. arXiv:2501.14249 [cs.LG] https: //arxiv.org/abs/2501.14249 [20] Sundar Pichai. 2026. Cloud Next ‘26: Momentum and innovation at Google scale. https://blog.google/innovation-and-ai/infrastructure-and-cloud/googlecloud/cloud-next-2026-sundar-pichai/ [21] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferber, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). [22] Jeff Shute, Radek Vingralek, Bart Samwel, Ben Handy, Chad Whipkey, Eric Rollins, Mircea Oancea, Kyle Littlefield, David Menestrina, Stephan Ellner, John Cieslewicz, Ian Rae, Traian Stancescu, and Himani Apte. 2013. F1: A Distributed
SQL Database That Scales. In VLDB. [23] Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. 2010. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Technical Report. Google, Inc. http://research.google.com/archive/papers/dapper-2010-1.pdf [24] Maxim Tabachnyk and Stoyan Nikolov. 2022. ML-Enhanced Code Completion Improves Developer Productivity. https://research.google/blog/ml-enhancedcode-completion-improves-developer-productivity/. Google Research Blog. [25] Maxim Tabachnyk, Xu Shu, Alexander Frömmgen, Pavel Sychev, Vahid Meimand, Ilia Krets, Stanislav Pyatykh, Abner Araujo, Kristóf Molnár, and Satish Chandra. 2026. Achieving Productivity Gains with AI-based IDE features: A Journey at Google. arXiv:2601.19964 [cs.SE] https://arxiv.org/abs/2601.19964 [26] Abhishek Verma, Luis Pedrosa, Madhukar R. Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. 2015. Large-scale cluster management at Google with Borg. In Proceedings of the European Conference on Computer Systems (EuroSys). Bordeaux, France. [27] Hyrum Wright, Titus Delafayette Winters, and Tom Manshreck. 2020. Software Engineering at Google. [28] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629 [29] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2025. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. arXiv:2409.02813 [cs.CL] https://arxiv.org/abs/2409. 02813