Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods Julian Frattini Richard Torkar∗ Robert Feldt†
Carlo A. Furia [email protected] USI Università della Svizzera italiana Lugano, Switzerland
arXiv:2604.08200v1 [cs.SE] 9 Apr 2026
{firstname}.{lastname}@chalmers.se Chalmers University of Technology and University of Gothenburg Gothenburg, Sweden
Abstract
Keywords
Controlled experiments are a core research method in software engineering (SE) for validating causal claims. However, recruiting a sample of participants that represents the intended target population is often difficult or expensive, which limits the external validity of experimental results. At the same time, SE researchers often have access to much larger amounts of observational than experimental data (e.g., from repositories, issue trackers, logs, surveys and industrial processes). Transportability methods combine these data from experimental and observational studies to “transport” results from the experimental sample to a broader, more representative sample of the target population. Although the ability to combine observational and experimental data in a principled way could substantially benefit empirical SE research, transportability methods have—to our knowledge—not been adopted in SE. In this vision, we aim to help make that adoption possible. To that end, we introduce transportability methods, their prerequisites, and demonstrate their potential through a simulation. We then outline several SE research scenarios in which these methods could apply, e.g., how to effectively use students as substitutes for developers. Finally, we outline a road map and practical guidelines to support SE researchers in applying them. Adopting transportability methods in SE research can strengthen the external validity of controlled experiments and help the field produce results that are both more reliable and more useful in practice.
Controlled Experiment, Transportability, External Validity, Simulation
CCS Concepts • General and reference → Experimentation; Reliability; • Computing methodologies → Scientific visualization. ∗
Also with The Stellenbosch Institute for Advanced Study. Also with Mid Sweden University.
†
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
ACM Reference Format: Julian Frattini, Richard Torkar, Robert Feldt, and Carlo A. Furia. 2026. Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 7 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Controlled experiments are an essential research method in software engineering (SE)—as in any empirical discipline—for validating claims about causal relationships between variables [34]. The random assignment of study subjects to a treatment or control group eliminates the influence of confounding factors on the relationship of interest [22]. Therefore, the observed effect can be attributed to the treatment rather than to confounding effects. However, this internal validity often comes at the expense of external validity. Experiments are conducted in a contrived setting [27] where a representative sample of subjects must be drawn from a target population [22, 26]. Achieving a broad and representative sample is particularly challenging when an experiment involves human subjects: the intended target population of SE professionals [15] is difficult to reach and expensive to recruit [1]. Consequently, controlled experiments in SE often settle for small samples and participants with a limited experience (e.g., students), thus jeopardizing statistical power and external validity [7]. This hinders transfer of scientific results into practice [24]. Other empirical disciplines face the same challenges. For example, medical researchers aim to predict how well a treatment response observed in a sample will hold in the target population, i.e., all potential recipients of that treatment [19]. To address this, the field of statistical causal inference has developed a formal framework for transportability of statistical relations across populations [18]. Within this broader line of work, “[e]stimation methods to generalize trial findings to a target population of interest” [4] emerged, which we will refer to as transportability methods from here on out. These methods combine experimental results with typically much larger observational data on relevant covariates, allowing results from limited experiments to be transported to a target population without collecting more experimental data [4].
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Frattini et al.
Group Population
Count
100
Figure 1: DAG visualizing causal assumptions of the illustrative, running example
Sample 50
0 10
20
30
40
Experience X (in number of years)
Despite their potential, such methods have—to our knowledge— not been adopted in SE research to date. With this vision paper, our goal is to pave the way for the adoption of transportability methods in SE research through the following contributions: (1) A high-level description of transportability methods and their necessary preconditions (2) A simulation demonstrating the methods’ usefulness (3) A list of valuable cases for application in SE research (4) A road map for enabling adoption in SE research Section 2 explains the process of transporting experimental results from a sample to a target population, including the preconditions that must be met for the methods to work. Section 3 illustrates the application of the method via a simulation to demonstrate its potential. Section 4 then presents several dimensions along which transportability is relevant in SE research, motivating the need for adopting this method in SE. Section 5 derives a road map of future steps for applying transportability methods in SE, before concluding in Section 6 with an outlook onto future work.
Data Availability Statement All figures, scripts, and documentation can be found in our replication package [12].
2
Transportability Methods
Section 2.1 introduces an illustrative example contextualizing the subsequent, methodological descriptions. Then, Section 2.2 lists relevant preconditions that need to be met for the actual transportability methods described in Section 2.3 to work.
2.1
Illustrative Example
At a high level, a controlled experiment estimates the causal effect of a treatment 𝐴 on an outcome 𝑌 (i.e., 𝐴 → 𝑌 ). As a running example, we will consider the effect of using (i.e., the treatment 𝐴 = 1) or not using (i.e., the control 𝐴 = 0) generative AI (GenAI) on the number 𝑌 of successfully identified defects during code reviews [30]. The quantity of interest to estimate from the experiment is the average treatment effect (ATE) 𝜏, i.e., the average difference in detected defects when using GenAI instead of not using it. The level of experience 𝑋 of a subject is an example of a covariate that may affect the outcome 𝑌 directly (𝑋 → 𝑌 ), but may also moderate the ATE [23]: Subjects with less experience may benefit more from using GenAI during code reviews than subjects with more experience. In the absence of experience, suggestions from GenAI may be a decent help, while the same suggestions may be trivial for an experience reviewer. This makes 𝑋 a treatment effect modifier. Figure 1 visualizes these relations as a directed acyclic graph (DAG), commonly used in Pearl’s framework for causal inference [18].
Figure 2: Distributions of the covariate 𝑋 in the target population and experimental sample due to covariate shift
Controlled experiments aim to approximate the ATE 𝜏 in the target population, but can realistically only measure the trial ATE 𝜏1 in the experimental sample. The ATE of interest 𝜏 may differ from the measurable trial ATE 𝜏1 . In our illustrative example, one reason for this difference may stem from the challenge of recruiting subjects. In particular, trial eligibility 𝑆 (i.e., the likelihood of a subject from the target population to be included in the experimental sample) is often affected by a treatment effect modifier such as 𝑋 : We can assume that subjects with more experience 𝑋 are in more senior position because it increases the likelihood of getting promoted [33]. This makes them less accessible and more expensive to recruit as subjects to the experiment. In contrast, subjects with less experience 𝑋 may be more available to participate in the experiment. For this reason, it is common to use university students to represent the target population of software engineers [3]. Figure 2 visualizes this challenge. The black line represents subjects’ trial eligibility (in percentage) which decreases for higher values of 𝑋 . The distribution of the covariate 𝑋 in the experimental sample (teal bars) therefore ends up different from the distribution in the target population (red bars)—a phenomenon known as covariate shift [28]. When 𝑋 is both a treatment effect modifier and experiences a covariate shift, the measurable 𝜏1 can differ from 𝜏. In our example, the experiment likely involves more easy-to-recruit but inexperienced subjects for which the measured effect is particularly strong. As a consequence, the experiment will overestimate 𝜏1 > 𝜏 and suggest that using GenAI is much more effective than it would be in the target population.
2.2
Preconditions
Table 1 lists the preconditions that must hold for the transportability methods (presented in Section 2.3) to work, as elicited by Colnet et al. [4]. Preconditions A1 through A4 are fundamental requirements for a valid controlled experiment. Preconditions A5, A6, and A7 are specific to transportability methods. Thus, we discuss what they mean and what happens if they are violated in the following. A5 requires that at least one covariate 𝑋 affects trial eligibility 𝑆, i.e., the black line in Figure 2 is not just a horizontal line. If A5 does not hold, then the distribution of 𝑋 would be the same in the experimental sample as in the target population. In such a case, the experimental sample perfectly represents the target population (i.e., there is no covariate shift), the trial ATE would perfectly generalize (𝜏 = 𝜏1 ), and there would be no need for transporting. As we argued in the illustrative example in Section 2.1, and as we will further
Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Table 1: Assumptions to apply transportability methods to controlled experiments. An asterisk * marks those that are specific to transportabilty techniques. ID
Name
Definition
Explanation
A1
Consistency
𝑌 = 𝐴𝑌 (1) + (1 − 𝐴)𝑌 (0)
A2 A3
Randomization Ignorability on trial participation Mean exchangeability
{𝑌 (0), 𝑌 (1)} á 𝐴 ⋃︀ 𝑆 = 1, 𝑋 {𝑌 (0), 𝑌 (1)} á 𝑆 ⋃︀ 𝑋
The observed outcome is the potential outcome given the assigned treatment, i.e., we have a connection between treatment and outcome. The treatment is independent of all the potential outcomes and covariates as in a controlled experiment. The outcome 𝑌 is unaffected by trial participation 𝑆 when controlling all relevant covariates 𝑋 .
E (︀𝑌 (𝑎) ⋃︀ 𝑋 = 𝑥, 𝑆 = 1⌋︀ E (︀𝑌 (𝑎) ⋃︀ 𝑋 = 𝑥⌋︀ 𝑌 (1) − 𝑌 (0) á 𝑆 ⋃︀ 𝑋
Instead of requiring that every individual behaves identically in and out of the trial, we only assume that on average the treatment effect is the same between groups with the same observed characteristics. The effect of the treatment is independent of trial eligibility when knowing all covariates 𝑋 .
A4 A5* A6* A7*
Sample ignorability for treatment effects Transportability of the conditional ATE Positivity of trial participation
=
∀𝑥 ∈ 𝑋 ∶ 𝜏1 (𝑥) = 𝜏(𝑥) ∃𝑐∶ P(𝑆 = 1 ⋃︀ 𝑋 ) ≥ 𝑐
In every stratum 𝑥 ∈ 𝑋 , the ATE in the experimental sample 𝜏1 is equal to the ATE in the target population 𝜏. Every subject in the target population must have at least some chance (i.e., non-zero probability) of being included in the experiment.
elaborate in Section 4, in most practical there would be at least some covariate shift, i.e., A5 normally holds. A6 requires that for every stratum of 𝑥 ∈ 𝑋 the ATE in the target population is the same as the trial ATE, i.e., the conditional ATE is the same even if the marginal ATE may not be. In other words, if we stratify by the covariate 𝑋 , the trial ATE generalizes to the target population. This implies that if A6 holds, 𝑋 acts as a treatment effect modifier. A situation where A6 would not hold is if there are other unobserved covariates that moderates the treatment effect. In this case, applying transportabilty methods to only 𝑋 may fail to correct for all of the covariate shift. Finally, A7 requires that every subject from the target population has non-zero probability of being included in the experimental sample, i.e., the black line in Figure 2 is always above 0%. In general, the distribution of 𝑋 in the experimental sample will differ from in the target population (see A5). A7 only requires that the two distributions have the same support. If A7 does not hold—i.e., some stratum of 𝑋 has a 0% probability of being sampled—no statistical method could recover the ATE from the unobserved stratum. Transportability is, hence, constrained to the range of 𝑋 covered in the experimental sample.
from 20 to 21 years. To handle these more complex interactions, a more general approach is needed. Enter transportability methods. Under the conditions in Section 2.2, a transportability method can recover the actual ATE 𝜏 of the target population from (1) the trial ATE 𝜏1 and (2) the distribution the the covariates 𝑋 in the target population, but without requiring further data about 𝐴 or 𝑌 . Colnet et al. discuss two classes of identification formulae [4]:
2.3
Here, 𝑛 is the size of the experimental sample, 𝑚 the size of the larger target population, 𝛼ˆ𝑛,𝑚 (𝑋𝑖 ) represents the trial eligibility 𝑆, and 𝑒 1 (𝑥) the propensity score [4] (i.e., the likelihood of being assigned to a treatment, which is fixed at 50% in most experiments with only one treatment and one control level). Trial eligibility 𝛼ˆ𝑛,𝑚 (𝑋𝑖 ) can be estimated via logistic regression based on the distribution of 𝑋 in the experimental sample and in the target population. Values of 𝑋 that occur often in the sample and in the target population have a high trial eligibility, values of 𝑋 that occur rarely in the sample but more often in the target population have a low trial eligibility. Based on this estimated trial eligibility, the data points from the controlled experiment are re-weighted. Data points from subjects with high trial eligibility contribute less to the ATE than from subjects with low trial eligibility. In the illustrative example, this would mean that the results obtained from one participating senior engineer (high experience 𝑋 , and therefore, low trial eligibility) are weighted more strongly in estimating the ATE than the results obtained from several participating master students (low experience 𝑋 , and therefore, high trial eligibility). This weighting
Formulae and Estimation Methods
One approach to approximate the ATE 𝜏 is to model the treatment effect modification as an interaction effect in a regression formula: 𝑌 ∼ 𝒩 (𝛼 + 𝜏 ⋅ 𝐴 ⋅ 𝑋, 𝜖)
(1)
This formula regresses the outcome 𝑌 (here assumed to be normally distributed 𝒩 with variance 𝜖) on a linear combination of an intercept 𝛼 (the baseline value for 𝑌 ) and the treatment 𝐴, which has an effect of 𝜏 on the outcome but is moderated by 𝑋 . For simplicity of the demonstration, we ignore all marginal effects of 𝐴 and 𝑋 on 𝑌 . However, this approach of estimating 𝜏 only works if the interaction between the continuous 𝑋 and 𝐴 is linear. This would require that every increase in the covariate 𝑋 causes the same proportional increase in the treatment effect moderation. However, not every effect behaves this way, particularly when considering human factors [16]. In the example where the covariate 𝑋 is a continuous measure of experience in number of years, it is possible that an increase of experience from 0 to 1 year has a greater effect than
𝑛 (1) Reweighting: 𝜏 = E [︀ 𝑚×𝛼(𝑋 𝜏 (𝑋 ) ⋃︀ 𝑆 = 1⌉︀ ) 1 (2) Regression: 𝜏 = E (︀𝜇𝐴=1,𝑆=1 (𝑋 ) − 𝜇𝐴=0,𝑆=1 (𝑋 )⌋︀ = E (︀𝜏1 (𝑋 )⌋︀
Based on these formulae, they elaborate several estimation methods for transportability. For brevity, we will only present one from each class and refer the interested reader to Colnet et al. [4]. Transport with reweighting: The inverse probability of sampling weighting (IPSW) is an estimator of 𝜏 based on reweighting. IPSW weighs each data point in the controlled experiment based on trial eligibility: 𝜏ˆIPSW =
1 𝑛 𝑛 𝑌𝑖 𝐴𝑖 1 − 𝐴𝑖 ( − ) ∑ 𝑛 𝑖=1 𝑚 𝛼ˆ𝑛,𝑚 (𝑋𝑖 ) 𝑒 1 (𝑋𝑖 ) 1 − 𝑒 1 (𝑋𝑖 )
(2)
of results by the inverse probability of sampling counteracts the effect of the covariate on the trial eligibility. Transport with regression: The plug-in g-formula is an estimator of 𝜏 based on regression. This estimator approximates 𝜏 by fitting two separate linear models. 1 𝑛+𝑚 (3) 𝜏ˆ𝐺 = ∑ (𝜇ˆ1,1,𝑛 (𝑋𝑖 ) − 𝜇ˆ0,1,𝑛 (𝑋𝑖 )) 𝑚 𝑖=𝑛+1 The two linear models predict the outcome 𝑌 based on 𝑋 , one for the control group (𝜇ˆ0,1 ) and one for the treatment group (𝜇ˆ1,1 ). These regressions 𝑌 ∼ 𝑋 for the two levels of 𝐴 directly model the treatment effect moderation of 𝑋 . The plug-in g-formula estimation then applies the covariate value 𝑋𝑖 of all 𝑚 observational data points to both linear models, averages the results, and calculates the ATE as the difference between the two averages.
3
Simulation
To demonstrate how transportability methods work in practice, we perform a computer simulation [27]. We simulate a target population and draw a sample from it that represents participants of an experimental study We then simulate this experiment with known causal effects among variables. Finally, we estimate the ATE using four methods: mean difference, linear regression with an interaction term, and the two presented transportability methods. We compare the four methods in their ability to recover the simulated causal effects from the data.
3.1
Dataset construction
We simulate the illustrative example described in Section 2.1. The main factor 𝐴 has two levels: control (𝐴 = 0, i.e., not using AI) and treatment (𝐴 = 1, i.e., using AI). We use a normally distributed measure representing defect detection performance instead of the number of identified defects for the outcome 𝑌 ∈ R. Using this normally distributed outcome simplifies interpretation by avoiding link functions required for count data [17], though the transportability methods work as well for counting data. Finally, the covariate 𝑋 ∈ R+ represents experience measured in number of years and follows a negative-binomial (NB) distribution (as in Figure 2). We created a data set by first simulating a target population of 1000 subjects with a random distribution of the covariate 𝑋 ∼ NB(10, 3). The scale parameter 𝜇 = 10 and dispersion parameter 𝛾 = 3 are arbitrary but produce realistic values between 0 and about 50 years of experience with a peak around 𝑋 = 20, as seen in Figure 2. Next, we simulated the trial eligibility 𝑆 ∼ Bernoulli(𝑝), where the likelihood of being included in the experimental sample decreases with 𝑋 (as shown as the black line in Figure 2). Subjects where 𝑆 = 1 are included in the controlled experiment, the remaining subjects where 𝑆 = 0 remain in the observational group. This splits the data set into roughly 𝑛 = 175 experimental subjects and 𝑚 = 825 observational subjects, though the exact numbers vary due to the random distribution. Finally, we randomly divided the experimental subjects into control (𝐴 = 0) and treatment (𝐴 = 1) groups and simulated the outcome 𝑌 which is affected by the treatment 𝐴 but moderated by the covariate 𝑋 . For the ATE, we chose an arbitrary value of 𝜏 = 16.7. The particular value of 𝜏 has no special meaning, but provides us with a simulated ground truth against which we
Frattini et al.
Estimation Method
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Mean difference
ATE = 16.7
Linear model IPSW Estimator Plug−in g−formula 10
20
Estimated ATE
Figure 3: Results from the simulation compared against the simulated effect (red line).
will evaluate all estimation methods in their ability to recover it. For the treatment effect moderation, our simulation decreased the ATE with higher values of 𝑋 in a non-linear way, which models diminishing returns of increasing experience. We did not simulate a marginal effect of 𝑋 → 𝑌 , i.e., the outcome 𝑌 did not change for different values of 𝑋 directly, only through the treatment effect moderation.
3.2
Estimation Setup
In the evaluation, we compare four methods to estimate the ATE: (1) Mean difference (baseline) between the outcome 𝑌 in the control and treatment group (2) Linear regression with interaction effect modeling the treatment effect moderation (Equation (1)) (3) IPSW estimator from the reweighting-class (Equation (2)) (4) Plug-in g-formula from the regression-class (Equation (3)) We run the simulation described above 50 times. For each simulated dataset, we record the ATE estimated by each of the four methods and then plot the distribution of these estimates.
3.3
Results
Figure 3 shows the results of the simulation. The box plots represent the estimated results of each of the four methods over 50 iterations. The red, dashed line shows the simulated ATE (𝜏 = 16.7) that these methods attempted to recover. In a real experiment, this ATE would be unknown, but in the scope of the simulation we can use it as a ground truth to compare the estimations against. The naïve mean difference vastly overestimates the simulated ATE. Since the experimental sample predominantly contained subjects with lower experience 𝑋 and the ATE of the main factor 𝐴 is moderated to be stronger for lower values of 𝑋 than higher, the naïve estimation assumes the ATE to be much stronger than it truly is. The linear model including an interaction effect performs significantly better, but still overestimates the simulated ATE. This is because it assumes the interaction to be linear, while the treatment effect moderation is actually non-linear. The 50%-quantiles of estimations of both transportability methods include the simulated ATE thanks to the covariate distribution 𝑋 in the target population. However, the IPSW estimator shows substantially greater uncertainty around its mean estimate. As Colnet et al. explain, this estimator can be highly unstable, particularly when the trial-eligibility weights become extreme [4]. The plug-in
Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
g-formula performs better on both accounts: it is more accurate and more robust. Using the information about the covariate distribution from the target population, it is able to correct the treatment effect moderation and recover the simulated ATE.
to enable this useful class of methods for SE research by focusing effort on the following steps.
4
Firstly, SE research should develop a clear understanding of which covariates act as moderators on ATEs of interest. These covariates are ultimately responsible for limiting the external validity of results obtained from experiments. Doing so would support a more rigorous and systematic analysis of the threats to external validity, rather than informally referring to common practice [35]. We anticipate that several covariates will be specific to certain SE tasks, while others apply to a broader scope. For example, experience, domain knowledge, or skill probably moderate many causal effects of interest [32], as they are likely to influence almost any SE activity. In contrast, a covariate like programming language proficiency will affect some SE tasks (e.g., source code development and code reviews) [30] more than others (e.g., requirements elicitation). Although identifying all moderators is difficult, causal models (Figure 1) make these assumptions explicit. Rather than aiming for “perfect” knowledge, researchers should use these models for sensitivity analyses that quantify how unobserved moderators could bias the transported ATE [17]. This shifts the focus from exhaustive completeness to the statistical robustness of the external-validity claim. These analyses can also help researchers prioritize the factors that matter most when designing an experiment: They should collect data on key moderating covariates and seek a representative sample that spans the full range of each covariate.
Motivating Examples
Beyond the illustrative example used in Sections 2 and 3, we identify three classes of challenges in empirical SE research for which transportability methods may be worth considering.
4.1
Experiment Participant Experience
A long-running debate in SE research asks whether (undergraduate) students can serve as valid substitutes for SE professionals in controlled experiments [5, 9, 20]. Students are easier to recruit, but they may lack the skills or domain knowledge of professional practitioners [2, 6]. This question has fueled an extensive public discussion, with prominent empirical SE researchers arguing both sides [3, 8, 10]. Yet the debate has relied mostly on hypotheses, assumptions, and anecdotal evidence rather than direct empirical tests. Transportability methods offer a constructive way forward for understanding and addressing the issue of representative subjects in SE experiments.
4.2
System Properties
Not only human participants but also the artifacts used in experiments may fail to represent the target population. Researchers often study software systems built in student projects [13], specifications mocked for the experiment [11, 31], or artificial bugs injected into software [14]. Industry-grade artifacts may be unavailable, unsuitable for time-constrained experiments, or missing properties that the study requires (e.g., ground-truth traceability links [13]). Even when experiments use industry-grade artifacts, they are often restricted to open-source systems because those are accessible [21]. Smaller, simpler, hand-crafted, or open-source artifacts are often more practical experimental objects, but they may not represent the target population of software systems, specifications, or other artifacts. This creates covariate shift in characteristics such as size, complexity, documentation quality, which affects how well results generalize. Framed as a transportability problem, the objects’ representativeness becomes a tangible property and limitations to external validity clear.
4.3
5.2
Understanding Treatment Effect Modifiers
Operationalizing Covariates
Once relevant covariates are identified, SE research must develop appropriate and agreed-upon operationalizations. Since many of the moderating covariates are likely to be latent variables and context factors, their operationalization is critical [25]. For example, experience is often operationalized via the number of years working as a software engineer, which may not adequately reflect the underlying concept: If one software engineer has worked for twice as long as another, there is no guarantee that they are also “twice as experienced.” A proper operationalization underpins the construct validity of these covariates. Without it, the previously introduced transportability methods are not applicable. Therefore, thoroughly assessing the construct validity of operationalizations of covariates moderating an ATE [29] will pave the way towards adjusting for them using transportability methods.
Task Complexity
In addition to human and artifact subjects, experimental tasks themselves are often not fully representative of real-world practice [26]. Researchers often limit the scope of a task to minimize the required time commitment of participants, e.g., code reviews without extensive familiarization with the source code [30]. This sacrifices representativeness of a task, raising the question whether effects observed during the experimental task still hold in reality.
5
5.1
Road Map and Guidelines
Even in medical research, where transportability methods originated, their application is still limited [4]. We see the opportunity
5.3
Collecting Observational Data
With relevant covariates identified and operationalized, the SE research community can steer its efforts towards collecting observational data on these covariates in the target population. While surveying the total target population remains unrealistic, observational studies collecting covariate distributions are likely to involve larger samples of the target distribution compared to interventional studies (e.g., controlled experiments or action research studies), given that they are less obtrusive [27]. For example, if experience is identified as a relevant, ATE-moderating covariate for several SE tasks, surveys collecting the distribution of developer experience in
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
different countries and companies can be conducted to approximate the distribution of that covariate in the general target population.
5.4
Transporting Results
With observational data sets approximating the distribution of relevant covariates, SE researchers can transport the results of controlled experiments from an experimental to an observational sample, where the latter is more representative of the target population. Thanks to the previously presented methods, covariate shift in controlled experiments can be partially addressed when the assumptions hold. For example, controlled experiments can be conducted primarily with students (i.e., subjects with lower experience) as long as there are still a few subjects representing the other end of the spectrum of the covariate (i.e., subjects with higher experience) to meet A7. This also implies the advice that—given an existing sample of students—effort is better spent on recruiting a few senior software engineers instead of a lot more students. Ultimately, when an experiment meets the assumptions in Section 2.2 and observational data on ATE-moderating covariates is available, transportability methods can improve the external validity of results without requiring additional experimental data.
5.5
Presenting Results
Finally, these methods allow contextualizing obtained results in two regards. First, the presence of a treatment effect modifier allows complementing the ATE with the results about the actual moderation. While the ATE represents the average effect aggregated over the full range of the covariate 𝑋 , a stratified view into how the effect changes along 𝑋 provides more detailed insights. In the illustrative example, this would allow the conclusion that the use of GenAI is beneficial for inexperienced subjects but irrelevant for experienced ones. Second, assessing the degree to which precondition A7—the positivity of trial participation—is met allows confining the external validity of the achieved results. If it was impossible to recruit subjects or infeasible to sample objects that cover the full spectrum of a treatment effect modifying covariate, the obtained range should be reported to confine the scope of generalizability. In the illustrative example, the data point with the larges value for 𝑋 (33 in Figure 2) defines the upper end of transportability.
6
Conclusion
Transportability methods have the potential to improve the external validity of results from controlled experiments and increase their practical relevance. If one or more covariates moderate the ATE of a phenomenon of interest and observational data about those covariates from a larger sample is available, results can be transported to this larger sample using transportability methods. Their application could help address several long-standing issues with experimentation in SE. Still, the path to adopt transportability methods in SE requires addressing several challenges in order to meet all preconditions. Targeting this goal will encourage SE researchers to explore and understand relevant covariates, collect data about them, and actively reason about the representativeness of their experimental subjects, objects, and tasks. In future work, we aim to demonstrate the application to real cases of SE research.
Frattini et al.
References [1] Sebastian Baltes and Paul Ralph. 2022. Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering 27, 4 (2022), 94. doi:10.1007/s10664-021-10072-8 [2] Victor R Basili, Scott Green, Oliver Laitenberger, Filippo Lanubile, Forrest Shull, Sivert Sørumgård, and Marvin V Zelkowitz. 1996. The empirical investigation of perspective-based reading. Empirical software engineering 1, 2 (1996), 133–164. doi:10.1007/BF00368702 [3] Jeffrey Carver, Letizia Jaccheri, Sandro Morasca, and Forrest Shull. 2004. Issues in using students in empirical studies in software engineering education. In Proceedings. 5th international workshop on enterprise networking and computing in healthcare industry (IEEE Cat. No. 03EX717). IEEE, 239–249. doi:10.1109/METR IC.2003.1232471 [4] Bénédicte Colnet, Imke Mayer, Guanhua Chen, Awa Dieng, Ruohong Li, Gaël Varoquaux, Jean-Philippe Vert, Julie Josse, and Shu Yang. 2024. Causal inference methods for combining randomized trials and observational studies: a review. Statistical science 39, 1 (2024), 165–191. doi:10.1214/23-STS889 [5] Bill Curtis. 1986. By the way, did anyone study any real programmers?. In Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers. 256–262. doi:10.5555/21842.28899 [6] Oscar Dieste, Natalia Juristo, and Mauro Danilo Martínez. 2013. Software industry experiments: A systematic literature review. In 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI). IEEE, 2–8. doi:10.1109/CESI.2 013.6618462 [7] Tore Dybå, Vigdis By Kampenes, and Dag IK Sjøberg. 2006. A systematic review of statistical power in software engineering experiments. Information and Software Technology 48, 8 (2006), 745–755. doi:10.1016/j.infsof.2005.08.009 [8] Davide Falessi, Natalia Juristo, Claes Wohlin, Burak Turhan, Jürgen Münch, Andreas Jedlitschka, and Markku Oivo. 2018. Empirical software engineering experts on the use of students and professionals in experiments. Empirical Software Engineering 23, 1 (2018), 452–489. doi:10.1007/s10664-017-9523-3 [9] Dror G Feitelson. 2015. Using Students as Experimental Subjects in Software Engineering Research–A Review and Discussion of the Evidence. (2015). doi:10.4 8550/arXiv.1512.08409 [10] Robert Feldt, Thomas Zimmermann, Gunnar R Bergersen, Davide Falessi, Andreas Jedlitschka, Natalia Juristo, Jürgen Münch, Markku Oivo, Per Runeson, Martin Shepperd, et al. 2018. Four commentaries on the use of students and professionals in empirical software engineering experiments. Empirical Software Engineering 23, 6 (2018), 3801–3820. [11] Julian Frattini, Davide Fucci, Richard Torkar, Lloyd Montgomery, Michael Unterkalmsteiner, Jannik Fischbach, and Daniel Mendez. 2025. Applying bayesian data analysis for causal inference about requirements quality: a controlled experiment. Empirical Software Engineering 30, 1 (2025), 29. doi:10.1007/s10664-02410582-1 [12] Julian Frattini, Richard Torkar, Robert Feldt, and Carlo Furia. 2026. Replication Package. https://doi.org/10.5281/zenodo.19451793. Last accessed 2026-04-07. [13] Tobias Hey, Jan Keim, and Sophie Corallo. 2024. Requirements classification for traceability link recovery. In 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 155–167. doi:10.1109/RE59067.2024.00024 [14] René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014, Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (Eds.). ACM, 654–665. doi:10.1145/2635868.2635929 [15] Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings. 26th International Conference on Software Engineering. IEEE, 273–281. doi:10.1109/ICSE.2004.1317449 [16] Johnson Ching-Hong Li. 2018. Curvilinear moderation—a more complete examination of moderation effects in behavioral sciences. Frontiers in Applied Mathematics and Statistics 4 (2018), 7. doi:10.3389/fams.2018.00007 [17] Richard McElreath. 2018. Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman and Hall/CRC. doi:10.1201/9781315372495 [18] Judea Pearl and Elias Bareinboim. 2011. Transportability of causal and statistical relations: A formal approach. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 25. 247–254. [19] Peter M Rothwell. 2005. External validity of randomised controlled trials:“to whom do the results of this trial apply?”. The Lancet 365, 9453 (2005), 82–93. doi:10.1016/S0140-6736(04)17670-8 [20] Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015. Are students representatives of professionals in software engineering experiments?. In 2015 IEEE/ACM 37th IEEE international conference on software engineering, Vol. 1. IEEE, 666–676. doi:10.1109/ICSE.2015.82 [21] Yorick Sens, Henriette Knopp, Sven Peldszus, and Thorsten Berger. 2025. A Large-Scale Study of Model Integration in ML-Enabled Software Systems. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1165–1177. doi:10.1109/ICSE55347.2025.00185
Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods
[22] William R Shadish, Thomas D Cook, and Donald T Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company. doi:10.1086/345281 [23] Janet Siegmund and Jana Schumann. 2015. Confounding parameters on program comprehension: a literature survey. Empirical Software Engineering 20, 4 (2015), 1159–1192. doi:10.1007/s10664-014-9318-8 [24] Dag IK Sjøberg, Bente Anda, Erik Arisholm, Tore Dybå, Magne Jørgensen, Amela Karahasanović, and Marek Vokáč. 2003. Challenges and recommendations when increasing the realism of controlled software engineering experiments. In Empirical Methods and Studies in Software Engineering: Experiences from ESERNET. Springer, 24–38. doi:10.1007/978-3-540-45143-3_3 [25] Dag IK Sjøberg and Gunnar Rye Bergersen. 2022. Construct validity in software engineering. IEEE Transactions on Software Engineering 49, 3 (2022), 1374–1396. doi:10.1109/TSE.2022.3176725 [26] Dag IK Sjøberg, Jo Erskine Hannay, Ove Hansen, Vigdis By Kampenes, Amela Karahasanovic, N-K Liborg, and Anette C Rekdal. 2005. A survey of controlled experiments in software engineering. IEEE transactions on software engineering 31, 9 (2005), 733–753. doi:10.1109/TSE.2005.97 [27] Klaas-Jan Stol and Brian Fitzgerald. 2018. The ABC of software engineering research. ACM Transactions on Software Engineering and Methodology (TOSEM) 27, 3 (2018), 1–51. doi:10.1145/3241743 [28] Masashi Sugiyama and Motoaki Kawanabe. 2012. Machine learning in nonstationary environments: Introduction to covariate shift adaptation. MIT press. doi:10.7551/mitpress/9780262017091.001.0001 [29] Caroline B Terwee, Cecilia AC Prinsen, Alessandro Chiarotto, Marjan J Westerman, Donald L Patrick, Jordi Alonso, Lex M Bouter, Henrica CW De Vet, and Lidwine B Mokkink. 2018. COSMIN methodology for evaluating the content
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
validity of patient-reported outcome measures: a Delphi study. Quality of life research 27, 5 (2018), 1159–1170. doi:10.1007/s11136-018-1829-0 [30] Rosalia Tufano, Alberto Martin-Lopez, Ahmad Tayeb, Ozren Dabic, Sonia Haiduc, and Gabriele Bavota. 2025. Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1640–1652. doi:10.1109/ICSE55347.2025.00060 [31] Andreas Vogelsang, Alexander Korn, Giovanna Broccia, Alessio Ferrari, Jannik Fischbach, and Chetan Arora. 2025. On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability. In 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). 51–55. doi:10.1109/ICSE-NIER66352.2025.00016 [32] Stefan Wagner and Marvin Wyrich. 2021. Code comprehension confounders: A study of intelligence and personality. IEEE Transactions on Software Engineering 48, 12 (2021), 4789–4801. doi:10.1109/TSE.2021.3127131 [33] Michael Waldman. 1984. Worker allocation, hierarchies and the wage distribution. The Review of Economic Studies 51, 1 (1984), 95–109. doi:10.2307/2297707 [34] Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders Wesslén, et al. 2012. Experimentation in software engineering. Vol. 236. Springer. doi:10.1007/978-3-662-69306-3 [35] Marvin Wyrich and Sven Apel. 2024. Evidence Tetris in the Pixelated World of Validity Threats. In Proceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering. 13–16. doi:10.1145/3643664.3648203
Received 23 January 2026; accepted 2 April 2026