arXiv CSopen access

LLM-based vs. Search-based Merge Conflict Resolution: An Empirical Study of Competing Paradigms

LLM-based vs. Search-based Merge Conflict Resolution: An Empirical Study of Competing Paradigms Heleno de Souza Campos Junior1 and Leonardo Gresta Paulino Murta1

arXiv:2605.16646v1 [cs.SE] 15 May 2026

1

Department of Computer Science, Federal Fluminense University„ Av. Gal. Milton Tavares de Souza, Niterói, RJ 24210-346, Brazil, [email protected], [email protected]

Abstract Context: The resolution of software merge conflicts is being reshaped by two competing and novel paradigms: generative approaches based on Large Language Models (LLMs) and optimization approaches from Search-Based Software Engineering (SBSE). While tools from both paradigms have shown promise, their relative strengths, weaknesses, and fundamental trade-offs are not yet understood. Objective: This paper presents the first in-depth empirical study to directly compare these two paradigms, aiming to identify their respective capabilities and limitations in real-world scenarios. Method: We evaluated MergeGen, a state-of-the-art LLM-based tool, against SBCR — a novel SBSE approach employing a Random Restart Hill Climbing (RRHC) algorithm. The comparison used thousands of real-world conflicts from open-source projects written in Java, C#, JavaScript, and TypeScript. Results: Our findings reveal fundamental trade-offs. The LLM paradigm excels at resolving conflicts with imbalanced content by leveraging learned patterns. However, it struggles with non-English content or large inputs, which can lead to truncated or empty resolutions. Conversely, the SBSE paradigm demonstrates superior generalization across datasets and performs optimally on balanced conflicts, showcasing its potential as a robust, data-independent alternative. Conclusions: Neither paradigm is a silver bullet. Our findings highlight contextdependent strengths and advocate for the development of hybrid systems that intelligently leverage the complementary capabilities of both LLM and SBSE approaches. This will help create more robust and reliable merge conflict resolution tools.

Keywords: Version control systems, software merge, conflict resolution, search-based software engineering, LLM.

1

1

Introduction

In modern collaborative software development, version control systems play a crucial role in managing changes across multiple developers working concurrently. These systems allow developers to work independently, merging their contributions into a shared codebase. However, the merging of code changes often leads to conflicts when concurrent modifications are made to overlapping code regions. Such merge conflicts must be manually resolved, which can be both time-consuming and error-prone, increasing the risk of introducing defects (Grinter, 1996; Shihab et al., 2012). Automated merge conflict resolution techniques aim to alleviate this burden by generating merge resolutions that align closely with developers’ expectations, reducing manual effort and the potential for errors (Mens, 2002; Cavalcanti et al., 2017). A crucial insight from large-scale empirical studies is that the vast majority of merge conflicts do not require developers to write new code. Ghiotto et al. (2020) found that in 87% of cases, the resolution is composed entirely of lines already present in the conflicting versions. These resolutions can be categorized into two groups: trivial and non-trivial. Trivial resolutions follow simple patterns, such as selecting one of the conflicting versions in its entirety or concatenating them. The remaining cases require a non-trivial combination of lines from both versions, which can be seen as a cherry-picking process that interleaves content, eventually adding new code. While writing new code (13% of cases) is also a nontrivial strategy, the combination strategy (9% of cases) is of special interest because of its combinatorial nature. Finding the correct sequence of lines among all possible permutations is the most difficult resolution task that can be solved without adding new code. To tackle this long-standing challenge, the research community has explored various techniques, from structured merging to machine learning. Recently, this has culminated in the emergence of two competing and novel paradigms. The first is a generative approach based on Large Language Models (LLMs), represented by tools like MergeGen (Dong et al., 2023), which leverage vast training data to produce resolutions. The second is an optimizationbased approach from Search-Based Software Engineering (SBSE), represented by the SBCR approach (Campos Junior et al., 2025), which frames conflict resolution as a combinatorial search problem solved by heuristic algorithms. While tools from both paradigms have shown promise, their relative strengths, weaknesses, and fundamental trade-offs are not yet understood. The emergence of these two distinct and modern paradigms—one data-driven and generative, the other heuristic-driven and combinatorial—motivates a critical question: what are their comparative capabilities and ideal use cases? This paper presents the first in-depth empirical study to directly compare these two paradigms. We evaluate MergeGen (Dong et al., 2023), a state-of-the-art LLM-based tool, against SBCR (Campos Junior et al., 2025), which employs a Random Restart Hill Climbing (RRHC) algorithm. Our evaluation focuses specifically on the challenging combination strategy. Scoping our study to these conflicts is deliberate: it allows us to compare the core mechanisms of both paradigms on a complex, non-trivial task that is solvable without generating new content, thus providing a fair and direct basis for comparison. Our study is guided by the following research questions:

2

• RQ1: How does the performance of an LLM-based approach compare to a search-based approach for resolving merge conflicts? • RQ2: Does the performance difference between the LLM and search-based paradigms vary across different programming languages? • RQ3: How do the generalization capabilities of the two paradigms differ when evaluated on previously unseen datasets? • RQ4: What explains the performance differences between the LLM and search-based paradigms? Our experimental results reveal fundamental trade-offs. The LLM paradigm excels at resolving conflicts with imbalanced content between versions, likely benefiting from learned contextual patterns. However, it shows signs of overfitting and struggles with non-English content or large inputs, which can lead to truncated or empty resolutions. Conversely, the SBSE paradigm demonstrates superior generalization across datasets and performs optimally on balanced conflicts, showcasing its potential as a robust, data-independent alternative. We conclude that neither paradigm offers a one-size-fits-all solution. Our findings highlight context-dependent strengths and advocate for the development of hybrid systems that intelligently leverage the complementary capabilities of both LLM and SBSE approaches. This will help create more robust and reliable conflict resolution tools. The remainder of this paper is structured as follows: Section 2 details MergeGen and SBCR. Section 3 outlines the materials and methods used in our comparative experiments. Section 4 presents the experimental results addressing our research questions. Section 5 discusses the key findings, limitations, and implications of our study. Section 6 presents the threats to the validity of our results and our actions to mitigate them. Section 7 covers related work, and Section 8 concludes the paper.

2

Compared Paradigms and Approaches

This section details the two competing paradigms for merge conflict resolution evaluated in this study: the Generative AI (GenAI) paradigm, represented by the state-of-the-art tool MergeGen (Dong et al., 2023), and the Search-Based Software Engineering (SBSE) paradigm, represented by the novel SBCR approach (Campos Junior et al., 2025). For each, we first explain the intuition behind the paradigm’s problem-solving approach before detailing the concrete implementation employed in our study.

2.1

The Generative AI Paradigm represented by MergeGen

The core intuition of the GenAI paradigm is to treat conflict resolution as a sophisticated sequence prediction task. An LLM approaches this problem not by analyzing combinatorial possibilities, but by leveraging patterns learned from observing millions of code changes and resolutions. It views the conflicting versions as a source sequence and aims to generate the most probable target sequence (the resolution) that logically follows from the context. This process is akin to language translation, where the model translates the conflicting inputs into 3

a single, coherent output, with the ability to infer missing information or generate entirely new content if its training data suggests such a pattern is appropriate. MergeGen (Dong et al., 2023) is a state-of-the-art implementation of this paradigm. It is built on two key components that enable this generative process: 1. Structured Conflict Representation: Instead of using raw text, MergeGen first processes the conflict at a token level. It then creates a structured input sequence using special tokens to explicitly delineate the conflicting regions and align the two conflicting versions (V1 and V2 ) with their common ancestor (O). This fine-grained, conflict-aware representation provides the model with a clear context of what has changed, simplifying the resolution task. 2. Generative Model: The second component is an encoder-decoder Transformer, specifically the CodeT5 model, which is pre-trained on a large corpus of code and fine-tuned on language-specific conflict resolution data. This model generates the final resolution tokenby-token in an auto-regressive manner. This mechanism gives MergeGen the flexibility to produce novel code sequences that may not have been present in any of the input versions, making it theoretically capable of handling both Combination and New Code resolution strategies.

2.2

The Search-Based Paradigm represented by SBCR

The intuition of the SBSE paradigm is to frame conflict resolution as a search for the best solution within a vast landscape of possibilities. This approach does not attempt to understand the code’s semantics. Instead, it defines a solution space composed of all possible combinations of lines from the conflicting versions. The goal is to navigate this landscape using heuristic-guided search, iteratively moving towards solutions with a better quality score, until an optimal or near-optimal resolution is found. The intelligence of this paradigm lies not in learned patterns, but in the design of a lightweight evaluation function that effectively guides the search. SBCR (Campos Junior et al., 2025) is a novel implementation of this paradigm, requiring three key components: 1. Problem Representation: A candidate solution is represented as an ordered sequence of lines from the two conflicting versions, V1 and V2 . The search is constrained to respect the partial order of lines within each version, a heuristic that drastically reduces the search space while covering 98.6% of real-world combination-based resolutions (Campos Junior et al., 2024). 2. Evaluation Function: A lightweight proxy for resolution quality is used, avoiding expensive compilation or testing. The function assesses a candidate’s quality by calculating the mean of its textual similarity to both parent versions (V1 and V2 ), based on the finding that this metric is moderately correlated (ρ = 0.64) with the final resolution’s quality (Campos Junior et al., 2025). 3. Manipulation Operators and Search Algorithm: SBCR uses three fundamental line operators (Addition, Removal, Position Exchange) to explore the neighborhood of a solution. It employs a Random Restart Hill Climbing (RRHC) algorithm to navigate the search space, performing local searches to find optima and restarting from random points to avoid getting stuck. 4

2.3

Summary of Paradigmatic Differences

The descriptions of MergeGen and SBCR highlight fundamental differences in their underlying paradigms. MergeGen, representing the GenAI paradigm, is a data-driven system that learns from historical data. Its core mechanism is generative, allowing it to produce resolutions that can fall under both the Combination and New Code categories. It relies on models like CodeT5 and fine-tuning on language-specific datasets of conflict resolutions. In contrast, SBCR, representing the SBSE paradigm, is a heuristic-driven optimization approach. It requires no training data; its search is guided entirely by a problem-specific evaluation function. Its mechanism is purely combinatorial, creating resolutions exclusively by selecting and ordering existing lines. This inherently limits its capability to the Combination strategy and makes it incapable of generating New Code. However, this also makes SBCR language-agnostic, as its logic is independent of code syntax. These core distinctions form the basis of our comparative study.

3

Materials and methods

This section details the experimental design used for the empirical comparison of the GenAI and SBSE paradigms. We first restate our research questions and then describe the scope of our evaluation, the datasets used, the configuration of each approach, and the analysis methods employed.

3.1

Research questions

To guide our research, we formulated the following questions: • RQ1: How does the performance of an LLM-based approach compare to a search-based approach for resolving merge conflicts? The first research question aims to compare the performance of the two paradigms in terms of similarity to the expected resolution and the time required to generate a candidate resolution. We focus this initial comparison on conflicts from Java projects, the most common language in our datasets. • RQ2: Does the performance difference between the LLM and search-based paradigms vary across different programming languages? The second research question investigates whether the performance differences observed in RQ1 hold for conflicts arising in other programming languages, specifically C#, JavaScript, and TypeScript. This assesses the language-dependent behavior of each paradigm. • RQ3: How do the generalization capabilities of the two paradigms differ when evaluated on previously unseen datasets? The third question focuses on assessing the generalizability of both paradigms. We analyze their performance when trained or tuned on one dataset and tested on another, revealing their adaptability to new and unfamiliar contexts. 5

• RQ4: What explains the performance differences between the LLM and search-based paradigms? Lastly, the fourth research question seeks to understand the underlying factors that contribute to the superior performance of one paradigm over the other. This is achieved by qualitatively analyzing extreme cases where there is a significant performance difference between the two approaches.

3.2

Scope of Evaluation: Combination-Based Conflicts

It is important to note that the empirical evaluation in this study is scoped to conflicts that developers resolved exclusively by combining existing lines from the conflicting versions. This methodological choice is deliberate and justified by two primary factors. First, this category of conflicts is highly prevalent and significant in practice. Large-scale empirical studies have shown that the vast majority of real-world merge conflicts are resolved without introducing entirely new code. For instance, Ghiotto et al. (2020) found that in 87% of analyzed conflicts, the final resolution was composed solely of lines already present in one of the conflicting versions. Therefore, focusing on combination-based resolutions allows our study to address a highly representative portion of the overall problem space. Second, this scope enables a direct and fair comparison of the fundamental mechanisms of the GenAI and SBSE paradigms. The SBCR approach, by its combinatorial nature, is designed specifically to find optimal combinations of existing lines and cannot generate new code. While MergeGen is capable of generating novel content, evaluating it on “new code” conflicts against an approach that cannot do so would not yield a meaningful comparison of their core problem-solving strategies. By focusing on a task that is theoretically solvable by both, we create a level playing field. This allows us to rigorously assess their distinct strengths: MergeGen’s learned, context-aware pattern matching versus SBCR’s heuristicdriven optimization on a shared, well-defined problem. While this scope does not test MergeGen’s full generative potential, it provides the necessary focus to conduct a robust comparison of how each paradigm performs in the most common type of complex, non-trivial conflict resolution scenario.

3.3

Datasets

We used two datasets of real-world, combination-based merge conflicts collected from opensource projects. Dataset1 was originally used by Campos Junior et al. (2024) and initially contained 10,177 conflicting chunks from Java projects resolved by combination. After discarding chunks with empty content and reconstructing the base version for each conflict, the dataset comprises 6,269 conflicting chunks from 816 open-source Java projects hosted on GitHub. Dataset2 was originally gathered by Svyatkovskiy et al. (2022) and used to evaluate MergeGen (Dong et al., 2023). It initially contained 151,426 conflicts from projects in four programming languages. As our focus is on combination-based resolutions, we filtered this dataset accordingly, resulting in 47,363 conflicting chunks divided as follows: 20,728 from 2,203 Java projects (Dataset2Java ), 7,017 from 908 C# projects (Dataset2C# ), 14,151 from 6

3,205 JavaScript projects (Dataset2JavaScript ), and 5,467 from 1,083 TypeScript projects (Dataset2T ypeScript ). For all datasets (Dataset1, Dataset2Java , Dataset2C# , Dataset2JavaScript , Dataset2T ypeScript ), we followed the approach adopted by Dong et al. (2023) and Svyatkovskiy et al. (2022) to randomly split them into training, validation, and testing sets, using an approximate ratio of 8:1:1. To answer RQ1, we combine the results obtained from running both approaches on the test partition of Dataset1 and Dataset2Java , since both datasets contain source code in Java. In this combination, we followed a conservative approach and excluded all chunks from both datasets that had the same commit hash to prevent duplicated data. Thus, in RQ1 we use the test partition of DatasetJava , which contains 2,439 valid conflicting chunks.

3.4

Experimental Setup

MergeGen Configuration: MergeGen requires training on language-specific data. For our experiments, we trained a separate model for each language dataset. Due to computational constraints, we set the BPE token limits to 300 for the conflict input and 100 for the resolution output, which is smaller than the 500/200 token limits used in the original MergeGen study. Our experiments were conducted on a machine with an Intel i7-4790K CPU, 16GB of RAM, and an NVIDIA RTX 2080 GPU, a more modest setup than the one used in the original work. These differences may impact the performance and generalization capabilities of MergeGen in our evaluation. SBCR Configuration and Tuning: SBCR is guided by three key parameters: neighbors_per_iteration, max_execution_time, and max_stagnation_iterations. To find the optimal configuration for each dataset, we performed a systematic parameter tuning process on a random sample of 100 conflicts from each training set. We tested multiple values for each parameter (e.g., 1-9 neighbors, 10-40 seconds timeout, 5-20 stagnation iterations). The configuration that provided the best balance between solution similarity and execution time was selected and used for the final evaluation on the test set for that specific dataset. The tuning results are discussed in Section 4.1.

3.5

Evaluation Metrics and Analysis

To compare the two paradigms, we used a combination of quantitative and qualitative analysis methods. Performance Metrics: The primary metric for evaluating the quality of a generated resolution was Similarity, computed using the Gestalt pattern matching measure based on the Longest Common Subsequence (LCS) between the generated candidate and the developer’s actual resolution. We also measured Execution Time. For SBCR, this includes the entire search process. For MergeGen, this measures only the generation time after receiving its specialized input, excluding the necessary pre-processing steps (e.g., tokenization, structural representation construction). We opted for this measurement strategy to ensure a fair comparison focused on the core resolution mechanisms of each paradigm. SBCR operates directly on raw conflict text, while MergeGen requires a separate, and potentially time-consuming, pre-processing stage 7

to create its structured input. By isolating only the resolution generation phase for both approaches, we assess the fundamental efficiency of the search-based algorithm versus the generative model. Consequently, the reported time for MergeGen represents a lower bound of its practical, end-to-end execution time, which would also include this pre-processing overhead. Statistical Analysis: We first used the Shapiro-Wilk test to check if the data distributions were normal. As they were not, we used the non-parametric Wilcoxon Signed-Rank test (Wilcoxon, 1945) for paired comparisons between the approaches on the same set of conflicts. To measure the magnitude of the performance difference, we calculated the Common Language Effect Size (CLES) (McGraw and Wong, 1992), which represents the probability that a random result from one approach is superior to a result from the other. Qualitative Analysis for RQ4: To understand the reasons behind performance differences, we calculated the metric SimSBCR−M ergeGen for each conflict. This metric represents the difference between the similarity score of the candidate generated by SBCR and the one generated by MergeGen (SimSBCR − SimM ergeGen ), allowing us to quantify the performance gap. Positive values indicate that SBCR performed better for a given conflict, while negative values indicate that MergeGen was superior. We then ranked all conflicts by this metric and selected the top 10 cases from each extreme of the distribution (highest positive and lowest negative values) for each of the five datasets, resulting in a total of 100 conflicts for manual analysis. These extreme cases were inspected to identify recurring patterns and characteristics that could explain each paradigm’s strengths and weaknesses.

4

Results

In this section, we present the findings of our study, structured to address the key research questions. We report the results of the parameter tuning process for SBCR, followed by a series of experiments to evaluate and compare the performance of the SBSE and GenAI paradigms. We conclude by discussing potential threats to the validity of our findings.

4.1

SBCR tuning

In this section, we present the tuning process for the SBCR approach, highlighting the selection of the best-performing configurations across various datasets. SBCR relies on three configurable parameters: the number of neighbors per iteration (neighbors_per_ iteration), the maximum execution time (max_execution_time), and the stagnation limit (max_stagnation_iterations). These parameters were tuned to strike a balance between solution quality and efficiency. The configurations for each dataset were tested systematically to evaluate both the similarity of generated solutions to developer-provided resolutions and the required execution time. Each parameter combination’s performance was measured by four key metrics: the Top-1 Similarity Count, Average Ranking, Average Similarity, and Average Time. Table 1 presents the metrics for the best-performing configurations of SBCR across each dataset, ordered by Average Similarity. We highlighted the configurations that were selected

8

Table 1: Best-performing configurations of SBCR from the parameter tuning process across each dataset. Configurations are ordered by Average Similarity, and the highlighted rows were selected for the final experiments based on the optimal trade-off between similarity and execution time. Top-1 Average Average Average Dataset Configuration Similarity Ranking Similarity Time Count Dataset1

7, 10, 30 5, 15, 10 7, 15, 40

46/100 45/100 46/100

9.52 10.33 10.14

0.827 0.827 0.821

12.28 5.96 14.79

Dataset2Java

5, 10, 20 9, 10, 20 9, 15, 40

41/100 39/100 39/100

16.18 11.39 11.66

0.827 0.826 0.825

4.44 18.41 22.93

Dataset2C#

3, 10, 40 5, 10, 20 9, 10, 10

36/100 38/100 38/100

15.15 13.90 11.73

0.791 0.790 0.786

2.32 4.69 12.27

Dataset2JavaScript

9, 5, 40 9, 20, 40 9, 15, 20

34/100 36/100 36/100

16.91 14.01 15.86

0.833 0.824 0.818

14.48 22.54 13.74

Dataset2T ypeScript

9, 15, 10 7, 20, 30 9, 20, 30

20/100 20/100 18/100

14.49 16.77 15.24

0.802 0.801 0.798

8.16 12.11 17.19

to be used in the experiments, based on their balance between average execution time and similarity to the resolution.

4.2

RQ1: How does the performance of an LLM-based approach compare to a search-based approach for resolving merge conflicts?

To answer this research question, we compare the performance of SBCR and MergeGen on the combined Java test partition (DatasetJava ), which contains 2,439 conflicts. The average similarity of candidates generated by MergeGen to the expected resolution is 90.0% (std. dev. 19.3%), while for SBCR it is 79.9% (std. dev. 20.6%) . As shown in Figure 1, MergeGen’s performance is notably strong, achieving a median similarity of 100% and resolving 55% of conflicts with a perfect match. In comparison, SBCR achieves a median similarity of 86.1%, with perfect matches in 19.6% of cases. A Wilcoxon Signed-Rank test (Wilcoxon, 1945) confirmed that this difference is statistically significant (p < 0.001). To measure the effect size, we calculated the Common Language Effect Size (CLES). The computed CLES is 0.294, meaning there is a 29.4% probability that SBCR will generate a more similar resolution than MergeGen for a randomly chosen conflict, while MergeGen has 9

a 70.6% probability of being superior.

Figure 1: Boxplots for the similarities between the resolution adopted by the developers and the conflict resolution candidates generated by SBCR and MergeGen for Java projects. Regarding execution time (Figure 2), MergeGen is demonstrably faster, with a median time of 0.3 seconds, compared to SBCR’s 1.3 seconds. We also performed a hypothesis test to confirm this observation. A Shapiro-Wilk test showed that the time distributions for both approaches are not normal (p < 0.001). Therefore, we used a Wilcoxon SignedRank test, which revealed a statistically significant difference between the two approaches (p < 0.001). The Common Language Effect Size (CLES) was 0.759, indicating that for a randomly selected conflict, there is a 75.9% probability that SBCR’s execution time will be longer than MergeGen’s, confirming the latter’s substantial speed advantage. Finding 1: For Java conflicts, the GenAI paradigm (MergeGen) produces more accurate resolutions than the SBSE paradigm (SBCR), with a higher median similarity (100% vs. 86.1%) and a greater number of perfect matches (55% vs. 19.6%). The difference is statistically significant, with MergeGen generating a more similar resolution in 70.6% of cases (CLES = 0.294). MergeGen is also significantly faster (median 0.3s vs 1.3s, p < 0.001), with a 75.9% probability of being faster on any given conflict (CLES = 0.759).

10

Figure 2: Boxplots for the time in seconds each approach takes to generate a candidate resolution for a conflicting chunk.

4.3

RQ2: Does the performance difference between the paradigms vary across different programming languages?

To answer RQ2, we investigate the performance of both paradigms on conflicts from C#, JavaScript, and TypeScript projects. As shown by the boxplots in Figure 3 and the descriptive statistics in Table 2, the GenAI paradigm (MergeGen) consistently achieves higher similarity scores than the SBSE paradigm (SBCR) across all three languages. MergeGen reaches a median similarity of 1.00 in all cases, while SBCR’s medians are 0.89 for C# and JavaScript, and 0.87 for TypeScript. Table 2: Statistics for the similarity results obtained when generating resolution candidates using each approach for C#, JavaScript, and TypeScript projects. Programming SBCR MergeGen language Mean Std. dev. Median Mean Std. dev. Median C# JavaScript TypeScript

0.80 0.81 0.79

0.24 0.21 0.22

0.89 0.89 0.87

0.89 0.88 0.87

0.23 0.22 0.25

1.00 1.00 1.00

To determine if these observed differences are statistically significant, we first performed a Shapiro-Wilk test to assess the normality of the data distributions for each approach and language. In all cases, the test indicated that the data is not normally distributed 11

Figure 3: Boxplots for the similarities between the resolution adopted by the developers and the conflict resolution candidates generated by SBCR and MergeGen for C#, JavaScript, and TypeScript projects, respectively.

12

(p < 0.001). Consequently, we used the non-parametric Wilcoxon Signed-Rank test for our paired comparisons. The test confirmed that the differences in similarity scores between MergeGen and SBCR are highly significant for all three languages (C#, JavaScript, and TypeScript), with p < 0.001 in each case. To understand the magnitude of this difference, we calculated the Common Language Effect Size (CLES). The results indicate that MergeGen’s advantage is substantial: SBCR has only a 33.4% chance of outperforming MergeGen in a random C# conflict, a 34.3% chance in JavaScript, and a 29.3% chance in TypeScript. While MergeGen’s performance is remarkably consistent, SBCR shows slightly lower performance for TypeScript, suggesting its text-based approach may be more sensitive to language-specific characteristics. In contrast, MergeGen, which leverages a language modelbased approach, appears to generalize more effectively across programming languages. This robustness could be attributed to its richer, conflict-aware representation of the code. Finding 2: The GenAI paradigm (MergeGen) consistently outperforms the SBSE paradigm (SBCR) across C#, JavaScript, and TypeScript, achieving a perfect median similarity of 1.00 in all languages. The differences are statistically significant, with CLES values confirming MergeGen’s advantage in approximately 66-71% of cases, depending on the language.

4.4

RQ3: How do the generalization capabilities of the two paradigms differ?

To assess generalization, we analyze how well SBCR and MergeGen perform when trained or tuned on one dataset and then evaluated on another. We conduct this analysis in two stages: first testing on Dataset1 and then on Dataset2Java . 4.4.1

Evaluation on Dataset1

First, we evaluated both approaches on Dataset1. This stage allows us to compare two distinct scenarios for generalization. The first is a within-dataset evaluation, where the approaches are tuned/trained and tested on different partitions of the same dataset (Dataset1). In this scenario, while the test data is unseen, the models have been exposed to other conflicts from the same set of projects, potentially learning project-specific styles and APIs. The second is a cross-dataset evaluation, where the approaches are tuned/trained on an entirely different set of projects (Dataset2Java ) and tested on Dataset1. This is a more challenging test of generalization, as the models have no prior exposure to the coding styles or context of the test projects. Table 3 presents the descriptive statistics for this stage. Notably, the performance of SBCR is nearly identical in both the within-dataset and cross-dataset scenarios, with mean similarities of 0.809 and 0.810, respectively. This demonstrates its strong ability to generalize to completely unfamiliar projects. The boxplots in Figure 4 visually confirm this stability for SBCR (left subplot) and show a slight variation for MergeGen (right subplot).

13

Table 3: Descriptive statistics for similarities obtained when using each approach tuned or trained on Dataset1 and Dataset2Java and tested on Dataset1 conflicts. SBCR MergeGen Tuning / Training Mean Std. dev. Median Mean Std. dev. Median Dataset1 Dataset2Java

0.809 0.810

0.200 0.201

0.865 0.865

0.923 0.902

0.153 0.194

1.00 1.00

Figure 4: Box plots comparing the similarity of candidates generated by SBCR and MergeGen, both tuned on Dataset1 and Dataset2Java , and tested on Dataset1. The left subplot shows the comparison between SBCR tuned on both datasets, while the right subplot shows the comparison between MergeGen trained on both datasets. The statistical tests, detailed in Table 4, support these observations. For SBCR, a Wilcoxon p-value of 0.953 and a CLES of 0.497 indicate no practical or statistically significant difference. For MergeGen, however, training on a different dataset (Dataset2Java ) resulted in a slight but statistically significant performance drop, as supported by a p-value of 0.022 and a CLES of 0.525, indicating a small advantage when trained on Dataset1. Despite this sensitivity, MergeGen still significantly outperforms SBCR in the crossdataset scenario. Figure 5 compares both approaches when trained/tuned on Dataset2Java and tested on Dataset1. The difference is statistically significant (p < 0.001) with a CLES of 0.322, meaning SBCR is likely to produce a better resolution than MergeGen in 32.2% of cases in this scenario, as also detailed in Table 4. 4.4.2

Evaluation on Dataset2Java

Second, we repeated the evaluation on the Dataset2Java test set. The results, summarized in Table 5, show a similar pattern. SBCR again demonstrates robust generalization, with no statistically significant performance difference when tuned on Dataset1 versus Dataset2Java . The consistency of SBCR is also visible in Figure 6 (left). In contrast, MergeGen again 14

Table 4: Statistical significance tests results for the approaches’ comparisons using different training sets and evaluated on Dataset1. Wilcoxon’s Approach Tuning/Training Approach Tuning/Training CLES p-value SBCR MergeGen SBCR

Dataset1 Dataset1 Dataset2Java

SBCR MergeGen MergeGen

Dataset2Java Dataset2Java Dataset2Java

0.953 0.022 1.617 × 10−25

0.497 0.525 0.322

Table 5: Descriptive statistics for similarities obtained when using each approach tuned on Dataset1 and Dataset2Java and tested on Dataset2Java conflicts. SBCR MergeGen Tuning / Training Mean Std. dev. Median Mean Std. dev. Median Dataset1 Dataset2Java

0.799 0.795

0.206 0.207

0.860 0.858

0.894 0.893

0.190 0.204

1.000 1.000

showed a statistically significant, though minor, sensitivity to the training data, as detailed in Table 6 (p = 0.021 and CLES = 0.490). Table 6: Statistical significance tests results for the approaches’ comparisons using different training sets and evaluated on Dataset2Java . Wilcoxon’s Approach Tuning/Training Approach Tuning/Training CLES p-value SBCR MergeGen SBCR

Dataset1 Dataset1 Dataset1

SBCR MergeGen MergeGen

Dataset2Java Dataset2Java Dataset1

0.230 0.021 2.45 × 10−80

0.506 0.490 0.308

When directly compared on the Dataset2Java test set (with both approaches trained/tuned on Dataset1), MergeGen again significantly outperforms SBCR, as seen in Figure 7. This difference is confirmed to be significant (p < 0.001) with a CLES of 0.308, as shown in Table 6. Finding 3: The SBSE paradigm (SBCR) demonstrates superior generalization, with its performance remaining stable across different datasets. The GenAI paradigm (MergeGen) is slightly sensitive to its training data but consistently achieves higher median similarity scores, outperforming SBCR even in cross-dataset evaluations.

15

Figure 5: Box plot comparing the similarity of candidates generated by SBCR and MergeGen, both trained on Dataset2Java , and tested on Dataset1. This figure highlights the relative performance of both approaches on Dataset1 conflicts, when trained on a different dataset.

Figure 6: Box plots comparing the similarity of candidates generated by SBCR and MergeGen, both tuned on Dataset1 and Dataset2Java , and tested on Dataset2Java . The left subplot shows the comparison between SBCR tuned on both datasets, while the right subplot shows the comparison between MergeGen trained on both datasets.

16

Figure 7: Box plot comparing the similarity of candidates generated by SBCR and MergeGen, both trained on Dataset1, and tested on Dataset2Java . This figure illustrates the direct comparison of both approaches when trained on Dataset1 and evaluated on Dataset2Java .

4.5

RQ4: What explains the performance differences between the two paradigms?

To understand the factors driving the performance differences, we analyzed each conflict using the SimSBCR−M ergeGen metric, which subtracts MergeGen’s similarity score from SBCR’s. This allows us to quantify which paradigm performed better for each specific case. First, we summarize the overall distribution of wins, losses, and ties in Table 7. Table 7: Summary of chunk performance comparison between SBCR and MergeGen for each dataset in terms of similarity to the resolution. Dataset SBCR > MergeGen MergeGen > SBCR SBCR = MergeGen Total Dataset1 Dataset2Java Dataset2C# Dataset2JavaScript Dataset2T ypeScript

131 430 148 331 104

380 1268 318 725 327

117 269 143 239 76

628 1967 609 1295 507

According to Table 7, MergeGen outperformed SBCR in the majority of cases across all datasets. For instance, in Dataset2Java , MergeGen was superior in 1,268 (64.5%) of the conflicts, while SBCR was superior in 430 (21.9%). The histograms in Figure 8 visualize this 17

distribution.

Figure 8: Distribution of Dif fSBCR−M ergeGen values across all datasets. Negative values (gray) indicate better performance by MergeGen, positive values (black) indicate better performance by SBCR, and zero (white bars) indicate ties. The histograms in Figure 8 confirm that while many cases cluster around zero (indicating comparable performance), the distribution is skewed to the left, favoring MergeGen. To understand the reasons behind the performance differences shown in Figure 8, we proceeded with the qualitative analysis detailed in our methodology. This involved a manual inspection of a balanced set of 100 conflicts drawn from the extremes of the performance distribution— 50 cases where SBCR was superior and 50 where MergeGen was superior, spread evenly across all five datasets. Our inspection of these cases revealed clear and recurring patterns that define the contextual strengths and weaknesses of each paradigm. The following sections present the categorized findings from this manual analysis.

18

4.5.1

Qualitative Analysis: When MergeGen Excels

Our manual analysis revealed that MergeGen’s primary strength when compared to SBCR lies in handling conflicts with significant structural imbalance. The specific scenarios where MergeGen was much better than SBCR are categorized in Table 8. Table 8: Situations where MergeGen performed much better than SBCR, categorized by dataset. Situation D1 D2Java D2C# D2JS D2TS Total Use of few lines from the larger version Use of almost all lines from both versions, where one is bigger Whitespace or removed content Few lines from smaller version, many from larger version Use of a few lines from smaller version Use of a few lines from both versions Chunk large compared to resolution Malformed chunk Total

1

2

4

4

2

13

2

4

5

0

1

12

4

2

1

2

0

9

2

0

0

0

4

6

1

2

0

0

0

3

0

0

0

1

2

3

0

0

0

2

1

3

0

0

0

1

0

1

10

10

10

10

10

50

Note: D1 = Dataset1; D2Java = Dataset2Java ; D2C# = Dataset2C# ; D2JS = Dataset2JavaScript ; D2TS = Dataset2T ypeScript .

As categorized in Table 8, this imbalance manifested in several ways. In 50% of the analyzed cases (25 out of 50), the resolution involved selecting either a small subset of lines from a much larger version or nearly all lines from both versions despite a large size disparity. In another 18% of cases, the imbalance was caused by one version removing content that the other modified. In these scenarios, SBCR’s evaluation function, which seeks to balance contributions, was ineffective. In contrast, MergeGen’s learned contextual understanding allowed it to generate the correct, unbalanced resolution, as exemplified in Listing 1. Listing 1 displays a conflicting chunk (file ChronicleMapBuilder.java in merge 9d8e848 from project OpenHFT/Chronicle-Map) from Dataset2Java . This conflict was resolved by the developers by keeping only the line that contains the @Override annotation. MergeGen inferred from the input that one version removed the original content and the other version added a new line to the original content. Thus, it correctly generated the expected resolution. Meanwhile, SBCR generated a candidate containing almost all lines from the bigger version, due to its evaluation function that aims to maximize the similarity with both parents. 19

Listing 1: Example of a conflicting chunk where MergeGen correctly inferred the resolution by keeping only the @Override annotation. SBCR’s balancing strategy led it to preserve most lines from the larger version, resulting in an incorrect resolution. 1 <<<<<<< 2 ======= 3 4 /∗ ∗ 5 ∗ For t e s t i n g 6 ∗/ 7 ChronicleMapBuilder <K, V> f o r c e R e p l i c a t e d I m p l ( ) { 8 t h i s . f o r c e R e p l i c a t e d I m p l = true ; 9 return t h i s ; 10 } 11 12 @Override 13 >>>>>>>

4.5.2

Qualitative Analysis: When SBCR Excels

Conversely, SBCR demonstrated its strengths in scenarios where MergeGen struggled due to its inherent model and data limitations. Table 9 summarizes these situations. Table 9: Situations where SBCR performed much better than MergeGen, categorized by dataset. Situation D1 D2Java D2C# D2JS D2TS Total Empty candidate Non-English content Truncated candidate Other Mismatch with resolution start Unexpected content Unused lines in resolution Malformed chunk Small difference in similarity

0 1 1 2 1 3 0 0 2

3 1 1 2 1 0 2 0 0

3 0 4 0 0 0 0 3 0

3 4 1 2 1 0 0 0 0

7 1 0 0 0 0 1 0 0

16 7 7 6 3 3 3 3 2

Total

10

10

10

10

10

50

Note: D1 = Dataset1; D2Java = Dataset2Java ; D2C# = Dataset2C# ; D2JS = Dataset2JavaScript ; D2TS = Dataset2T ypeScript .

As shown in the table, the most frequent problem for MergeGen was producing an empty candidate (32% of cases), particularly when one of the conflicting versions was also empty. Other significant problems included generating truncated candidates due to token limits (14%) and mishandling non-English content (14%). In these situations, SBCR’s languageagnostic, text-based approach proved to be more robust and capable of generating a complete and more accurate resolution. 20

4.5.3

The Impact of Content Balance on Performance

The relationship between performance and the balance of content between conflicting versions is quantified in Figure 9. Dark bars indicate cases where SBCR is better than MergeGen, gray bars indicate cases where MergeGen is better than SBCR, and white bars indicate ties.

Figure 9: Distribution of the size differences between v1 and v2 when MergeGen is better (gray), when SBCR is better (black), and when they tie (white). Positive values indicate v1 is larger; negative values indicate v2 is larger. The figure illustrates that SBCR performs best on balanced conflicts, while MergeGen excels in imbalanced scenarios. 21

The histograms in Figure 9 show that SBCR’s performance peaks (black bars) when the conflicting versions, v1 and v2 , are of similar size (Diff v1 - v2 sizes ≈ 0). In contrast, MergeGen (gray bars) dominates at the extremes of the distribution (Diff v1 - v2 sizes ≈ ±1), where one version is significantly larger than the other. This analysis also exposed a case of potential overfitting in MergeGen. In Dataset2T ypeScript , the distinct gray bar near the 0.25 mark corresponds to a conflict that was repeated multiple times in the training data. This suggests MergeGen may have memorized this specific resolution rather than learning a generalizable strategy, highlighting a key advantage of SBCR’s search-based approach, which is not susceptible to this type of overfitting. Finding 4: MergeGen excels in scenarios involving unbalanced versions or whitespace content, benefiting from its learned contextual patterns. SBCR, on the other hand, achieves its peak performance with balanced chunks and in situations where MergeGen fails, such as with non-English content or due to model limitations, indicating its strength in generalizing across more diverse situations without overfitting. Each approach has strengths that are context-dependent.

5

Discussion

Our empirical evaluation of MergeGen and SBCR reveals that neither paradigm is universally superior. Instead, they exhibit a series of fundamental trade-offs. In this section, we discuss these trade-offs, the limitations of each approach, and their broader implications for resolving software merge conflicts.

5.1

Potential Overfitting in MergeGen

The results achieved by MergeGen raise concerns about overfitting. Our qualitative analysis in RQ4 identified several cases where the quality of the generated resolution seemed remarkably high, yet it was unclear how the approach could deduce such solutions from the conflict data alone. Upon further examination of the training data, we discovered multiple instances of identical or highly similar conflict cases (e.g., the repetitive chunk in the Azure/autorest.typescript project), which likely contributed to this high performance. This suggests that the model may have memorized specific examples, which calls into question its generalizability in projects where each merge conflict is unique. Future research should focus on testing MergeGen’s capabilities in a live development environment to assess how well it performs on truly unseen conflicts.

5.2

Handling Conflicts of Different Sizes

The evaluation highlights varying strengths in handling conflicts of different sizes. As an LLM-based tool, MergeGen requires truncating both the input and output when conflict chunks exceed a configured token limit. This means that for very large conflicts, MergeGen might not provide complete or useful resolutions, limiting its applicability in complex merge situations. In such cases, SBCR presents a viable alternative, as it does not impose 22

size limitations in the same manner. Conversely, for smaller conflicts, MergeGen typically outperforms SBCR due to its contextual understanding, except in the specific failure cases identified in RQ4.

5.3

Practical Implications of Failure Modes

Beyond the quantitative similarity scores, the nature of the failures from each paradigm has distinct practical implications for developers. Our qualitative analysis in RQ4 provides insights into the different types of risks and cognitive loads associated with each approach. When the GenAI paradigm (MergeGen) fails, it often does so in a way that is obvious and immediately detectable. Producing an empty or clearly truncated candidate is a loud failure. While this forces the developer to resolve the conflict manually from scratch, thus offering no assistance, it carries a low risk of introducing subtle bugs. The developer is fully aware that the automation has failed and that they must take complete ownership of the resolution process. In contrast, the SBSE paradigm’s (SBCR) typical failure mode can be more subtle and potentially more dangerous. When faced with a highly imbalanced conflict, SBCR may produce a resolution that may appear plausible, yet it may be logically incorrect because its evaluation function forced the balance of lines. A developer under time pressure or with a high degree of trust in the tool might approve such a resolution without deep inspection, potentially introducing a latent bug that is much harder to find later during testing or, in the worst case, in production. This reveals a critical trade-off in failure modes: the GenAI paradigm, in these specific cases, tends to fail loudly, whereas the SBSE paradigm can fail silently. Understanding this distinction is crucial for designing user interfaces and review processes for automated merging tools.

5.4

Dependency on the Evaluation Function in SBCR

A key limitation of SBCR, identified in our qualitative analysis, is its dependency on the evaluation function, particularly its sensitivity to the balance of content between conflicting versions. The current function, which averages the similarity to both versions, performs well on balanced conflicts but struggles when one version is significantly larger. This can lead to suboptimal resolutions in highly imbalanced scenarios. Future work should explore alternative evaluation functions that account for this disparity.

5.5

Cost, Training, and Practical Considerations

Finally, a critical distinction between the two paradigms lies in their practical costs. MergeGen, based on a large language model, requires considerable computational resources and financial investment for training and fine-tuning. In our own experiments, resource constraints forced us to truncate inputs, potentially impacting resolution quality. In contrast, SBCR does not require a training phase; its cost is primarily dependent on the configurable search time. This makes SBCR a more cost-effective and accessible solution, particularly for resource-constrained environments. The trade-off is that SBCR’s performance is tied to its 23

search time, whereas a fully-resourced and pre-trained MergeGen can generate resolutions almost instantaneously.

5.6

Towards a Comprehensive Hybrid Approach

Our findings suggest that the GenAI and SBSE paradigms are not mutually exclusive, but rather complementary. The weaknesses of one approach are often the strengths of the other, which points towards the potential of a hybrid or ensemble tool that intelligently leverages both. Such a system could provide a more robust and effective solution than either paradigm in isolation. We envision a practical workflow where a “meta-resolver” first analyzes the characteristics of an incoming merge conflict. Based on this analysis, it would route the conflict to the most suitable engine: • Small to medium-sized, imbalanced conflicts, or those matching common patterns, would be directed to MergeGen. Its speed and learned contextual understanding make it ideal for these scenarios. • Large conflicts that exceed LLM token limits would be routed to SBCR. Its ability to handle inputs of any size (constrained only by time) makes it a crucial fallback for complex cases where MergeGen would otherwise fail by truncating the input. • Conflicts with balanced content would be prime candidates for SBCR, as our results show it performs optimally in these scenarios where its balancing evaluation function is most effective. • Conflicts containing non-English content or other characteristics known to be outside the typical training distribution of LLMs would be handled by SBCR, leveraging its language-agnostic robustness. This intelligent routing mechanism, potentially implemented as a simple rules-based system or a lightweight classifier, would mitigate the primary weaknesses of each paradigm. It would avoid tasking MergeGen with conflicts it is ill-equipped to handle (thus preventing empty or truncated resolutions) while applying SBCR’s more computationally intensive search only when necessary and most effective. This vision of a complementary system aligns with the broader goal of building pragmatic, reliable, and truly automated conflict resolution tools for developers.

6

Threats to Validity

This section discusses the potential threats to our study’s validity and the measures adopted to mitigate them, covering construct, internal, conclusion, and external validity. Construct Validity relates to whether our measurements accurately reflect the intended concepts. A primary threat is the assumption that the developer’s resolution is the optimal ground truth, which, while common practice, is not guaranteed in terms of correctness 24

or maintainability. Additionally, our main metric, textual similarity, may not fully capture qualitative aspects like code readability. We mitigated this by using a well-established Gestalt-based similarity metric, widely used in prior research, but we acknowledge that similarity alone is not a comprehensive measure of resolution quality. Internal Validity concerns confounding factors that could affect the results. First, implementation differences could bias time measurements. We measured only MergeGen’s generation time, excluding its required pre-processing steps; therefore, its reported time is a lower bound. We have noted this distinction throughout our analysis. Second, LLM-based models are susceptible to overfitting. We mitigated this by using distinct training, validation, and testing partitions for all experiments. Finally, the CodeT5 model underlying MergeGen (Wang et al., 2021) may have been pre-trained on code from our test datasets, a potential data leakage issue. However, since CodeT5 was not trained specifically on merge scenarios, we believe the impact of this threat is likely limited. Conclusion Validity relates to the reliability of our findings. To mitigate threats from inappropriate statistical methods or insufficient sample size, we used well-established nonparametric tests (Wilcoxon signed-rank test), the Common Language Effect Size (CLES) metric, and a large dataset comprising thousands of conflicts across multiple programming languages. External Validity concerns the generalizability of our results. Our findings may not generalize beyond the open-source projects in our datasets (Java, C#, JavaScript, and TypeScript), as coding practices may differ in proprietary contexts. To mitigate this, we intentionally selected diverse, multi-language datasets that have been used in previous studies, representing a wide variety of conflict scenarios.

7

Related Work

The challenge of merge conflict resolution has been addressed by a wide range of approaches over the years. To contextualize our comparison of the novel GenAI and SBSE paradigms, this section reviews the main categories of existing work: structured approaches, other optimization techniques, and various learning-based methods.

7.1

Structured and Semi-Structured Merge Approaches

Early attempts to improve upon purely textual merging focused on leveraging the code’s structure through semi-structured and structured approaches. These methods aim to improve conflict resolution effectiveness by using language-specific information, overcoming some limitations of unstructured, line-based merging. Apel et al. (2011) introduce the concept of semi-structured merge as a middle ground between unstructured and structured systems. To balance the lack of semantic awareness in unstructured tools (like Git’s default) and the implementation overhead of fully structured tools, they propose FSTMERGE, which uses annotated grammars to introduce structural awareness. Their evaluation shows a significant reduction in conflicts, though handling some cases like renaming remains challenging. Building on this, Apel et al. (2012) present JDime,

25

a structured merge approach that dynamically switches between structured and unstructured merging to balance precision and performance. Later work by Cavalcanti et al. (2017) and Cavalcanti et al. (2024) further advanced these ideas, demonstrating that leveraging structural information, even partially, significantly reduces false positive conflicts, making them easier for developers to analyze. Compared to these approaches, which are tightly coupled to language-specific parsers, both paradigms in our study offer different forms of flexibility. The SBSE paradigm, represented by SBCR, is inherently language-agnostic due to its textual nature. The GenAI paradigm, represented by MergeGen, while requiring language-specific training data, can be adapted to any language where such data is available, offering a broader potential application than tools with hardcoded grammars.

7.2

Combination and Optimization Approaches

A different line of work has focused on optimization techniques, treating conflict resolution as a problem of finding the best combination of changes. These approaches vary in their reliance on historical data and language-specific features. For instance, Zhu and He (2018) proposed AutoMerge, which uses version space algebra (VSA) and ASTs to efficiently represent and rank a large set of candidate resolutions. Xing and Maruyama (2019) explored Automated Program Repair (APR) techniques, using genetic programming to generate fixes for behavioral conflicts. More aligned with learning, Gonzalez and Fraternali (2022) presented “Almost Rerere”, which uses genetic programming to generate resolution rules from conflicts resolved in the past. While powerful, these approaches often depend on language specificity (AutoMerge) or the availability of high-quality historical examples (Almost Rerere). The SBSE paradigm for conflict resolution, as represented by SBCR in this paper, is built upon two key foundational findings from our prior work. First, Campos Junior et al. (2024) established heuristics for efficiently navigating the solution space, finding that 98.6% of combination-based resolutions maintain the partial order of the original lines. This insight provides a principled way to drastically reduce the search space. Second, Campos Junior et al. (2025) addressed the challenge of evaluating candidates by proposing a lightweight, similarity-based evaluation function. The study found that developer resolutions are, on average, 70% similar to both parent versions and, crucially, that a candidate’s similarity to its parents is strongly correlated (ρ = 0.791) with its similarity to the final resolution (Campos Junior et al., 2025). This provides a feasible proxy for solution quality, avoiding expensive compilation and testing. Taken together, these findings provide the two necessary components for an SBSE approach: a constrained search space and a feasible evaluation function. While sharing the goal of optimization with SBCR, many of the other approaches in this category, such as Almost Rerere, depend on historical data or language-specific features like ASTs. The SBSE paradigm, as implemented in SBCR, distinguishes itself by using these empirically-derived, data-agnostic heuristics, offering a more generalizable optimization strategy for novel scenarios.

26

7.3

Learning-Based Approaches

More recently, the advent of large-scale data and deep learning has given rise to a new category of learning-based approaches, which forms one of the core paradigms of our study. These methods leverage neural networks and large language models (LLMs) to improve accuracy. Early models like DeepMerge (Dinella et al., 2022) used edit-aware embeddings to construct resolutions, while MergeBERT (Svyatkovskiy et al., 2022) framed the task as a tokenlevel classification problem. A significant leap came with generative models. Gmerge (Zhang et al., 2022) applied large pre-trained models like GPT-3 to the task, while MergeGen (Dong et al., 2023) moved away from classification entirely. By treating conflict resolution as a pure generation task, MergeGen can produce novel code and combinations beyond predefined patterns, establishing it as the state-of-the-art. These learning-based techniques represent the paradigm exemplified by MergeGen in our study. They leverage historical data and complex models to understand semantic aspects of conflicts. This stands in direct contrast to the SBSE paradigm (SBCR), which applies a heuristic-based search without requiring historical data or deep learning models. While tools like MergeGen excel in conflicts that require generative capacities, SBCR provides a more computationally feasible and data-independent solution, particularly appealing where training data is unavailable or resources are limited.

7.4

Comparison and Evaluation of Conflict Resolution Approaches

Evaluating the effectiveness of these diverse approaches is a significant challenge in itself. To address this, researchers have developed dedicated benchmarks. Shen and Meng (2024) introduced ConflictBench, a standardized dataset of Java merging scenarios to systematically compare tools on metrics like precision and resolution desirability. In a complementary effort, Schesch et al. (2024) conducted an in-depth evaluation of multiple tools, emphasizing merge correctness through test suite execution. Their work highlights that tool performance is highly scenario-dependent. In contrast to these broad evaluations of multiple tools, such as ConflictBench (Shen and Meng, 2024) and the work by Schesch et al. (2024), our study provides a focused, in-depth comparison of two distinct and novel paradigms for merge conflict resolution: search-based optimization (SBCR) and Generative AI (MergeGen). Our review of the literature reveals a clear evolution in this field, from early structured approaches to more recent learning-based techniques. However, it also highlights persistent gaps, such as limitations in languagespecificity and a heavy dependency on historical data. To our knowledge, no prior work has conducted a direct empirical comparison of the emerging GenAI and SBSE paradigms. Our study addresses this critical gap by analyzing their trade-offs in adaptability, performance, and practical failure modes, aiming to deepen the understanding of how these modern approaches may complement each other.

27

8

Conclusion

This paper presented the first in-depth empirical comparison between two competing and novel paradigms for merge conflict resolution: Generative AI, represented by MergeGen, and Search-Based Software Engineering (SBSE), represented by SBCR. We evaluated both paradigms across multiple datasets consisting of Java, C#, JavaScript, and TypeScript codebases. The experiments included a comprehensive parameter tuning process for SBCR and a systematic performance evaluation of both approaches in terms of resolution similarity, execution time, and generalizability. We also performed a qualitative analysis to understand the specific scenarios where each paradigm excels. Our findings reveal that the SBSE and GenAI paradigms each have distinct strengths and limitations. The SBSE approach (SBCR) achieved its best performance for conflicts with balanced content between versions. In contrast, the GenAI approach (MergeGen) demonstrated superior results in cases where a strong imbalance existed, potentially benefiting from learned behavior in its training data. We also identified key weaknesses: SBCR’s performance is sensitive to its evaluation function in highly unbalanced scenarios, while MergeGen showed clear signs of overfitting. A key conclusion from our study is that, for practical conflict scenarios, the ideal resolution strategy involves a hybrid approach that leverages the complementary strengths of both paradigms. MergeGen performed effectively for smaller and more predictable conflicts, while SBCR showed advantages for larger, more complex scenarios, or those with characteristics (like non-English content) that fall outside the LLM’s typical training data. Thus, a combined approach that routes conflicts based on their characteristics could provide developers with a more robust and effective solution. The implications of these results suggest that relying on a single paradigm for conflict resolution may not be sufficient in real-world software development. A data-dependent approach like MergeGen may struggle with novel conflicts, while a generalized search-based solution like SBCR can complement these limitations. Our findings highlight the need for more adaptive and hybrid solutions that intelligently select the appropriate technique, thus reducing manual intervention and increasing developer productivity. For future work, we plan to explore alternative evaluation functions for SBCR to improve its performance in unbalanced scenarios. We also intend to conduct an experimental evaluation of the proposed hybrid system in a real-world development setting, involving developers to gain insights into the usability, practicality, and perceived effectiveness of each paradigm in daily development tasks. These studies are crucial steps towards moving beyond the comparison of individual paradigms and into the engineering of a new generation of intelligent, context-aware merging tools. By synergistically combining the generative power of LLMs with the robust, heuristic-driven search of SBSE, such tools have the potential to significantly reduce developer burden and fundamentally improve collaborative software development workflows.

28

Declarations Funding We thank the following entities for partially funding this work: INES.IA (National Institute of Science and Technology for Software Engineering Based on and for Artificial Intelligence) www.ines.org.br; Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) for the grants 309300/2023-1, 408817/2024-0, and 155576/2025-9; and Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ) for the grant E-26/204.145/2024.

Conflicts of interest/Competing interests The authors have no relevant financial or non-financial interests to disclose.

Data Availability In the spirit of open science and to ensure the full reproducibility of our study, we have made all data, scripts, and results publicly available in a comprehensive replication package. The main repository, containing the source code for both the SBCR and MergeGen approaches, along with all scripts and detailed instructions to replicate the experiments and analyses, is available on GitHub at: https://github.com/gems-uff/sbcr_study To further facilitate replication and inspection of our results, we also provide two archival datasets on FigShare: • Pre-processed Datasets: For researchers who wish to bypass the initial data preprocessing steps, the final, clean datasets used directly in our experiments are available at: https://figshare.com/s/d196f4ccb3ef34d2e770. • Full Experimental Results: A complete archive of our experimental run, including intermediate files, execution logs, all candidates generated for each conflict, and the models generated by MergeGen, can be found at: https://figshare.com/s/ b3cdd351d077a9b08121. This allows for a deep inspection of all generated artifacts.

Authors’ Contributions All authors contributed to the study conception and design. Material preparation and data collection was performed by Heleno. Data analysis was performed by Heleno and Leonardo. The first draft of the manuscript was written by Heleno and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

29

References Sven Apel, Jörg Liebig, Benjamin Brandl, Christian Lengauer, and Christian Kästner. Semistructured merge: Rethinking merge in revision control systems. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE ’11, pages 190–200, Szeged, Hungary, September 2011. Association for Computing Machinery. ISBN 978-1-4503-0443-6. doi: 10.1145/2025113.2025141. Sven Apel, Olaf Leßenich, and Christian Lengauer. Structured Merge with Auto-Tuning: Balancing Precision and Performance. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pages 120–129, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 978-1-4503-1204-2. doi: 10.1145/2351676.2351694. Heleno de S Campos Junior, Gleiph Ghiotto L de Menezes, Márcio de Oliveira Barros, André van der Hoek, and Leonardo Gresta Paulino Murta. How code composition strategies affect merge conflict resolution? Journal of Software Engineering Research and Development, 12(1):13–1, 2024. doi: 10.5753/jserd.2024.3638. Heleno de S Campos Junior, Gleiph Ghiotto L. de Menezes, Márcio de Oliveira Barros, André van der Hoek, and Leonardo Gresta Paulino Murta. Towards a feasible evaluation function for search-based merge conflict resolution. ACM Transactions on Software Engineering and Methodology, 2025. doi: 10.1145/3748256. Guilherme Cavalcanti, Paulo Borba, and Paola Accioly. Evaluating and improving semistructured merge. Proceedings of the ACM on Programming Languages, 1(OOPSLA):59:1– 59:27, October 2017. doi: 10.1145/3133883. Guilherme Cavalcanti, Paulo Borba, Leonardo dos Anjos, and Jonatas Clementino. Semistructured merge with language-specific syntactic separators. In 2024 39th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2024. doi: 10.1145/3691620.3695483. Elizabeth Dinella, Todd Mytkowicz, Alexey Svyatkovskiy, Christian Bird, Mayur Naik, and Shuvendu Lahiri. Deepmerge: Learning to merge programs. IEEE Transactions on Software Engineering, 49(4):1599–1614, 2022. doi: 10.1109/TSE.2022.3183955. Jinhao Dong, Qihao Zhu, Zeyu Sun, Yiling Lou, and Dan Hao. Merge conflict resolution: Classification or generation? In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1652–1663. IEEE, 2023. doi: 10.1109/ASE56229.2023.00155. Gleiph Ghiotto, Leonardo Murta, Márcio Barros, and André van der Hoek. On the nature of merge conflicts: A study of 2,731 open source java projects hosted by github. IEEE Transactions on Software Engineering, 46(8):892–915, 2020. doi: 10.1109/TSE.2018.2871083.

30

Sergio Luis Herrera Gonzalez and Piero Fraternali. Almost rerere: Learning to resolve conflicts in distributed projects. IEEE Transactions on Software Engineering, 49(4):2255– 2271, 2022. doi: 10.1109/TSE.2022.3215289. Rebecca E Grinter. Supporting articulation work using software configuration management systems. Computer Supported Cooperative Work (CSCW), 5:447–465, 1996. doi: 10.1007/ bf00136714. Kenneth O McGraw and Seok P Wong. A common language effect size statistic. Psychological bulletin, 111(2):361, 1992. doi: 10.1037/0033-2909.111.2.361. Tom Mens. A state-of-the-art survey on software merging. IEEE transactions on software engineering, 28(5):449–462, 2002. doi: 10.1109/TSE.2002.1000449. Benedikt Schesch, Ryan Featherman, Kenneth J Yang, Ben Roberts, and Michael D Ernst. Evaluation of version control merge tools. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 831–83, 2024. doi: 10.1145/3691620.3695075. Bowen Shen and Na Meng. Conflictbench: A benchmark to evaluate software merge tools. Journal of Systems and Software, 214:112084, 2024. doi: 10.1016/j.jss.2024.112084. Emad Shihab, Christian Bird, and Thomas Zimmermann. The effect of branching strategies on software quality. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement, pages 301–310, 2012. doi: 10.1145/2372251. 2372305. Alexey Svyatkovskiy, Sarah Fakhoury, Negar Ghorbani, Todd Mytkowicz, Elizabeth Dinella, Christian Bird, Jinu Jang, Neel Sundaresan, and Shuvendu K Lahiri. Program merge conflict resolution via neural transformers. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 822–833, 2022. doi: 10.1145/3540250.3549163. Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.685. Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6): 80–83, 1945. doi: 10.2307/3001968. Xiaoqian Xing and Katsuhisa Maruyama. Automatic software merging using automated program repair. In 2019 IEEE 1st International Workshop on Intelligent Bug Fixing (IBF), pages 11–16. IEEE, 2019. doi: 10.1109/IBF.2019.8665493. Jialu Zhang, Todd Mytkowicz, Mike Kaufman, Ruzica Piskac, and Shuvendu K Lahiri. Using pre-trained language models to resolve textual and semantic merge conflicts (experience 31

paper). In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 77–88, 2022. doi: 10.1145/3533767.3534396. Fengmin Zhu and Fei He. Conflict resolution for structured merge via version space algebra. Proceedings of the ACM on Programming Languages, 2(OOPSLA):1–25, 2018. doi: 10. 1145/3276536.

32