arXiv:2604.20777v1 [cs.LG] 22 Apr 2026
Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning Dario Simionato∗
Andrea Tonon
Mingxue Wang
[email protected] Huawei Ireland Research Centre Dublin, Ireland
[email protected] Huawei Ireland Research Centre Dublin, Ireland
[email protected] Huawei Ireland Research Centre Dublin, Ireland
Weiguo Wang
Tong Gui
Xiaoyue Li
[email protected] Huawei Nanjing R&D Center Nanjing, China
[email protected] Huawei Nanjing R&D Center Nanjing, China
[email protected] Huawei Nanjing R&D Center Nanjing, China
Abstract In streaming platforms churn is extremely costly, yet A/B tests are typically evaluated using outcomes observed within a limited experimental horizon. Even when both short- and predicted long-term engagement metrics are considered, they may fail to capture how a treatment affects users’ retention. Consequently, an intervention may appear beneficial in the short term and neutral in the long term while still generating lower total value than the control due to users churn. To address this limitation, we introduce a method that estimates long-term treatment effects (LTE) and residual lifetime value change (Δ𝐸𝑅𝐿𝑉 ) in short multi-cohort A/B tests under user learning. To estimate time-varying treatment effects efficiently, we introduce an inverse-variance weighted estimator that combines estimates from multiple cohorts, reducing variance relative to standard approaches in the literature. The estimated treatment trajectory is then modeled as a parametric decay to recover both the asymptotic treatment effect and the cumulative value generated over time. Our framework enables simultaneous evaluation of steady-state impact and residual user value within a single experiment. Empirical results show improved precision in estimating LTE and Δ𝐸𝑅𝐿𝑉 and identify scenarios in which relying on either short-term or longterm metrics alone would lead to incorrect product decisions.
Keywords A/B testing, user learning, long term effect, lifetime value ACM Reference Format: Dario Simionato, Andrea Tonon, Mingxue Wang, Weiguo Wang, Tong Gui, and Xiaoyue Li. 2026. Efficient Multi-Cohort Inference for Long-Term Effects ∗ Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
and Lifetime Value in A/B Testing with User Learning. In . ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
This is the author’s accepted manuscript of a paper accepted for publication in the Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2026). The final authenticated version will be available in the ACM Digital Library.
1
Introduction
A/B testing is a cornerstone of modern online experimentation, allowing product teams to measure the impact of product changes on user behavior in a controlled setting. In practice, however, several real-world complexities can make interpretation challenging. First, treatment effects often evolve over time, creating a potential mismatch between short-term results and long-term outcomes, with the latter usually being more relevant for business decisions. Second, metrics averaged only over active users in the long term may not correctly capture the total cumulative impact of a treatment, especially when user participation changes across cohorts. The first issue motivated researchers to focus on estimating the long-term treatment effect (LTE) from short-duration experiments where user behavior is significantly influenced by novelty or primacy effects, named as user learning [8, 14]. The cookie-cookie day (CCD) method [13] was the first experimental design specifically proposed to infer long-term metrics using cohort comparisons. More recently, [16] showed that LTE can be estimated in the presence of user learning without requiring ad-hoc experimental designs. Their key idea is to analyze users w.r.t. their entry time into the experiment, rather than calendar time, and to apply a differencein-differences (DiD) estimator across treatment and control cohorts to recover the evolution of the treatment effect. Despite this important advancement, existing approaches remain limited in two aspects. First, they make inefficient use of the available data, leading to high-variance estimations of the treatment effect over time. Second, and more fundamentally, they focus exclusively on the LTE ignoring both the impact of the treatment on user traffic and the cumulative value generated during the learning phase (i.e. when user behavior is temporarily changing).
Conference’17, July 2017, Washington, DC, USA
Dario Simionato et al.
From a business perspective, however, cumulative impact is often as important as steady-state impact. Lifetime value (LTV) frameworks address this objective by measuring the total value generated over time considering also user traffic rather than only asymptotic behaviors. Yet LTV estimation typically requires long historical horizons or detailed user-level modeling, making it poorly suited to short-term online experiments [9, 12, 14]. This paper focuses on an industrial scenario for streaming services, in which a company wants to introduce an advertisement in between each stream. While such increase may initially boost short-term ad clicks as users are not yet accustomed to such feature and are more likely to notice the ad, over time they may develop ad blindness, ignoring the ads entirely [3, 5, 13]. Simultaneously, repeated exposure can lead to advertisement fatigue, frustrating users and increasing churn. Consequently as shown in Figure 1, although clicks number per active user may increase in the short term and may not change in the long term, accounting for user counts and retention can reveal a decline in total engagement and lifetime value. This highlights the need for evaluation methods that consider both long-term metric behavior and the treatment’s effect on user traffic (i.e. how the treatment affects the user churn). We address this problem in two ways. First, we propose a unified framework that jointly estimates the LTE and the incremental expected residual lifetime value (Δ𝐸𝑅𝐿𝑉 ) of users in multi-cohort A/B tests under user learning. Second, we introduce an inverse-variance weighted multi-cohort estimator that combines multiple unbiased estimates of the treatment evolution to improve estimation accuracy. Differently from CCD or DiD approaches, our multi-cohort method fully exploits all staggered cohorts reducing the variance of the estimates. By fitting a parametric decay model to the estimated trajectory, we recover both the asymptotic treatment effect and the cumulative value generated over time. By simultaneously addressing the measurement objective (both LTE and Δ𝐸𝑅𝐿𝑉 ) and the estimation procedure, the proposed method provides a more complete and reliable evaluation of A/B tests in the presence of user learning, capturing both transient dynamics and steady-state effects of product changes accounting for user churns.
Figure 1: Comparison of the number of clicks 𝑚 observed on active users (top) and number of clicks weighted by user count (bottom). In this A/B test, the treatment introduces advertisements in between streams. While the computation of 𝑚 among active users (top) shows a positive short-term effect with no long-term downsides (𝐿𝑇 𝐸 ≈ 0), this view is biased by ignoring how the treatment affects user counts. The second plot (bottom) accounts for user retention, revealing that the treatment significantly reduces the number of active users over time. This leads to a negative overall business impact, as shown by the negative change in Expected Remaining Lifetime Value (Δ𝐸𝑅𝐿𝑉 < 0).
respect to those at their first day in the experiment, and the LTE as 𝐿𝑇 𝐸 = 𝜏 + 𝑙𝑖𝑚𝑡 →+∞𝛿𝑡
2
Problem setup
We consider a randomized A/B test measuring a metric 𝑚 (in our case, the number of ad clicks) and in which users enter the experiment over time. As a consequence, we analyze outcomes using a cohort framework rather than a single fixed observation window. Let 𝑇 be the total duration of the A/B test, users are divided into 2𝑇 cohorts according to their entry time in the experiment 𝑡 0 (0 ≤ 𝑡 0 < 𝑇 ) and whether they receive their treatment (written as 𝑇𝑢 = 1) or they belong to the control group (𝑇𝑢 = 0) throughout all the experiment. In particular, we define 𝐶𝑡𝑡0 and 𝑇𝑡𝑡0 to represent the sample means at time 𝑡 0 ≤ 𝑡 < 𝑇 for the control and treatment cohorts that joined the test at time 𝑡 0 . Additionally, let 𝜎ˆ𝐶2 𝑡 𝑡0
and 𝜎ˆ𝑇2 𝑡 be the sample mean variances of 𝐶𝑡𝑡0 and 𝑇𝑡𝑡0 , respectively. 𝑡0
Following [13, 16], we define the user learning 𝛿𝑘 to measure the difference in treatment effect between users that had been exposed for 𝑘 time steps (and had some time to adapt to the treatment) with
(1)
representing the treatment effect after the user learning stabilized, with 𝜏 representing the treatment effect on the first exposure day. After defining the LTE, one may be tempted to evaluate a treatment solely according to its long-term effect. However, in product experimentation the deployment decision depends on the total value generated during the entire user lifecycle, not only on the steady-state behavior, as illustrated in the bottom part of Figure 1. Moreover, the LTE fails to account for the treatment’s impact on the service user base itself. To bridge this gap, we adopt a framework based on the Expected Remaining Lifetime Value (ERLV). We define the Remaining Lifetime Value for a specific user 𝑢 as the sum of the metric values generated from the current time 𝑡 ∗ until the user churns and stop using the service. Since 𝑢 may churn at any time, we model this using a survival function 𝑆𝑢 (𝑡), which takes the value 1 if the user is active at time 𝑡 and 0 otherwise. The ERLV is the expectation of this quantity over the user population Í 𝐸𝑅𝐿𝑉 = E𝑢 𝑡∞=𝑡 ∗ 𝑚𝑢 (𝑡, 𝑡 0 ) · 𝑆𝑢 (𝑡) where 𝑚𝑢 (𝑡, 𝑡 0 ) is the observed
Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning
metric value for user 𝑢 at time 𝑡, given it entered the experiment at 𝑡 0 . From a decision-making perspective, the relevant quantity is the Incremental Expected Remaining Lifetime Value (Δ𝐸𝑅𝐿𝑉 ). This represents the total change in value created by the treatment, accounting for both the change in user behavior with respect to the metric 𝑚 and the change in retention:
Δ𝐸𝑅𝐿𝑉 = E𝑢
" +∞ ∑︁
# 𝑚𝑢 (𝑡, 𝑡 0 )𝑆𝑢 (𝑡) | 𝑇𝑢 = 1
𝑡 =𝑡 ∗ " +∞ ∑︁
− E𝑢
#
(2)
𝑚𝑢 (𝑡, 𝑡 0 )𝑆𝑢 (𝑡) | 𝑇𝑢 = 0 .
𝑡 =𝑡 ∗
This definition shifts the focus from a single point in time (the steady state after user learning) to the area under the curve of the treatment effect, aligning with traditional lifetime value frameworks [2, 15] which prioritize the total value generated over a lifecycle rather than isolated asymptotic behaviors [1]. Lastly, this formulation naturally captures cases where a treatment might boost 𝑚𝑢 in the short term but decrease 𝑆𝑢 in the long term, resulting in a possibly negative Δ𝐸𝑅𝐿𝑉 despite a potentially neutral or positive LTE.
3 Methods 3.1 A unified formulation for LTE and Δ𝐸𝑅𝐿𝑉 The core of our unified framework is the observation that both 𝐿𝑇 𝐸 and Δ𝐸𝑅𝐿𝑉 can be derived from the same staggered-cohort data by varying the handling of missing values. As established in Section 2, the 𝐿𝑇 𝐸 measures the steady-state behavioral change per active user, while Δ𝐸𝑅𝐿𝑉 measures the cumulative business impact accounting for user participation [1, 2, 15, 18]. Mathematically, we can express the two objectives as follows: • The Long Term Effect (𝐿𝑇 𝐸) is the asymptotic difference between treatment and control when considering only available (non-missing) observations (𝑆𝑢 = 1). This measures the steady-state treatment effect on users assuming they keep engaging in the system. • The Incremental Lifetime Value (Δ𝐸𝑅𝐿𝑉 ) is the cumulative difference between treatment and control weighted by the percentage of active users. This captures the combined effect of user retention and behavioral changes such as novelty or primacy effects [8, 13, 14, 16]. While traditional A/B test analysis often treats missing data as "missing at random" and ignores it, in the context of business value, churn is an informative non-random event. By explicitly modeling user churn, the Δ𝐸𝑅𝐿𝑉 naturally weights the treatment’s behavioral impact by its ability to keep the user active. As illustrated in Figure 1, this distinction is critical: a treatment might have a positive or null effect on 𝑚𝑢 among survivors (𝐿𝑇 𝐸 ≈ 0), but if it simultaneously accelerates churn, the survival rate decrease in the Δ𝐸𝑅𝐿𝑉 sum will correctly reveal a negative business outcome. In the following sections, we demonstrate how our multi-cohort estimator can be applied to A/B test data to recover these distinct but complementary insights.
3.2
Conference’17, July 2017, Washington, DC, USA
Efficient multi-cohort estimation
Methods for long-term effect estimation under user learning first infer the learning component in short A/B tests and then use it to estimate the LTE by fitting some models and analyzing their behavior at 𝑡 → +∞ [13, 16]. Accordingly, we first review how state-of-the-art approaches model user learning, highlighting their limitations. Finally, we introduce our efficient multi-cohort estimation method, which we first apply to the user learning inference problem and we then adapt for LTE and Δ𝐸𝑅𝐿𝑉 estimations. We begin by describing two representative approaches from the literature. The CCD approach [13] estimates the user learning as a difference between treated users exposed at the beginning of the experiment against users that just entered in the experiment, that is 𝛿ˆ𝐶𝐶𝐷 (𝑡) = 𝑇0𝑡 − 𝑇𝑡𝑡 . In [16], the user learning is instead estimated using a DiD method computed as 𝛿ˆ𝐷𝑖𝐷 (𝑡) = (𝑇0𝑡 − 𝑇00 ) − (𝐶 0𝑡 − 𝐶 00 ). Such estimates are then used to fit an exponential model 𝑓 (𝑡) = 𝛾 + 𝛼𝑒 −𝛽𝑡 , and the LTE is obtained by summing 𝛾 and the treatment effect in the first user exposure day. The main limitation of such approaches lies in their inefficient use of data. In the CCD method, each cohort contributes to the estimator only once through the 𝑇𝑡𝑡 term. By contrast, the DiD estimator ignores users who enter in the experiment after it begins. Our approach, instead, overcomes such issue by combining multiple estimates and therefore making full use of the available data. We observe that the user learning at time 𝑡 can be computed as +𝑘 𝛿ˆ0 (𝑡) = 𝑇0𝑡 − 𝑇𝑡𝑡 (as shown in [13]) but also as 𝛿ˆ𝑘 (𝑡) = 𝑇𝑘𝑡 +𝑘 − 𝑇𝑡𝑡+𝑘 (as suggested in [16]) as long as 𝑡 + 𝑘 < 𝑇 . While both estimates are unbiased, combining them with a simple average to obtain a more precise quantity is inefficient due to their different variances. We therefore aggregate such estimates using inverse-variance weighting, which assigns greater weight to more precise estimates 𝛿ˆ𝑘 . This yields to the multi-cohort estimator 𝛿ˆ𝑀𝐶 (𝑡), which combines information across cohorts and by construction has a lower variance than any individual 𝛿ˆ𝑘 [4, 6]. Our approach can then be summarized as follows: (1) Compute the estimates 𝛿ˆ𝑘 (𝑡) = 𝑇 𝑡 +𝑘 − 𝑇 𝑡 +𝑘 and their vari𝑘
𝑡 +𝑘
+𝑘 ) for 0 ≤ 𝑡 < 𝑇 ances 𝜎ˆ 2ˆ (𝑡) = 𝜎ˆ 2 𝑡 +𝑘 + 𝜎ˆ 2 𝑡 +𝑘 −2𝑐𝑜𝑣 (𝑇𝑘𝑡 +𝑘 ,𝑇𝑡𝑡+𝑘 𝛿𝑘
𝑇𝑘
𝑇𝑡 +𝑘
and 0 ≤ 𝑘 < 𝑇 − 𝑡.
1/𝜎ˆ 2ˆ (𝑡 ) 𝛿
(2) Compute the estimates weights 𝑤𝑘 (𝑡) = Í𝑇 −𝑡 𝑘 2 𝑘=0
(3) Compute the weighted mean 𝛿ˆ𝑀𝐶 (𝑡) = and the weighted variance 𝜎ˆ 2ˆ
𝛿 𝑀𝐶
(𝑡) =
1/𝜎ˆ ˆ (𝑡 )
.
𝛿𝑘
Í𝑇 −𝑡
ˆ
𝛿𝑘 (𝑡) 𝑘=0 𝑤 𝑘 (𝑡) ! −1 Í𝑇 −𝑡 1 (as 𝑘=0 𝜎ˆ 2 (𝑡 ) ˆ 𝛿𝑘
shown in [4]). Such approach can also be used for LTE and Δ𝐸𝑅𝐿𝑉 estimation. To estimate the LTE, we first compute 𝜏 + 𝛿𝑡 using our multi-cohort +𝑘 , ignoring estimator applied to 𝑇𝑘𝑡 +𝑘 − 𝐶𝑘𝑡 +𝑘 , rather than 𝑇𝑘𝑡 +𝑘 − 𝑇𝑡𝑡+𝑘 missing values when computing the estimation. Following [16], we then fit an exponential model 𝑓 (𝑡) to the sequence of treatment effect estimates 𝜏ˆ + 𝛿ˆ𝑡 . This parametric model is finally used to extrapolate the total treatment effect beyond the observation window, that is according to Equation 1, 𝐿𝑇 𝐸 = lim𝑡 →+∞ 𝑓 (𝑡) = 𝛾. Estimating Δ𝐸𝑅𝐿𝑉 requires instead modeling treatment and control outcomes, as well as their survival functions, separately over
Conference’17, July 2017, Washington, DC, USA
Dario Simionato et al.
Table 1: Performance comparison on 100 simulations of the evaluated methods reporting mean absolute errors for LTE and Δ𝐸𝑅𝐿𝑉 , and mean confidence interval size. Standard deviations are in parentheses and best results are in boldface. 𝐶𝐼 95 𝑤𝑖𝑑𝑡ℎ
𝑀𝐴𝐸𝐿𝑇 𝐸
𝑀𝐴𝐸 Δ𝐸𝑅𝐿𝑉
CCD [13]
0.76 (0.08)
0.15 (0.22)
0.88 (0.83)
DiD [16]
1.08 (0.12)
0.16 (0.14)
0.88 (0.83)
Multi-cohort approach
0.38 (0.02)
0.13 (0.23)
0.66 (0.50)
time. To this end, we again employ our multi-cohort estimator, now applied directly to 𝑇𝑘𝑡 +𝑘 and 𝐶𝑘𝑡 +𝑘 for estimating the metric values for the two cohorts. We then estimate 𝑆𝑇 (𝑡) and 𝑆𝐶 (𝑡) applying the multi-cohort method to the user presence/absence data, meaning that the metric is 1 if the user is present and 0 otherwise. We then fit two exponential models, 𝑓𝑇 (𝑡) and 𝑓𝐶 (𝑡), to extrapolate the metrics trajectories for the treatment and control cohorts, respectively, and other two exponential models 𝑓𝑆𝑇 (𝑡) and 𝑓𝑆𝐶 (𝑡) to model user churn. Finally, these extrapolated quantities are combined according to Í Equation 2, yielding Δ𝐸𝑅𝐿𝑉 = 𝑡+∞ =0 𝑓𝑇 (𝑡)𝑓𝑆𝑇 (𝑡) − 𝑓𝐶 (𝑡)𝑓𝑆𝐶 (𝑡) .
4
Experimental evaluation
To evaluate the proposed estimators, we conduct a simulation study that reproduces realistic dynamics of user learning and churn. Following established practices [7, 13, 14], we use a 14-day A/A dataset consisting of user-level clicks for 41,768 users. This provides a neutral baseline with no pre-existing systematic differences, allowing us to inject synthetic, time-varying treatment effects and measure how accurately they are recovered. Following the real-world setup of [13], we inject an exponential treatment effect 𝛼𝑒 𝑓 𝑓 𝑒 −𝛽𝑒 𝑓 𝑓 (𝑡 −𝑡0 ) to treated units. This reproduces the scenario validated by their real-world findings where a positive short-term effect decays to a negligible LTE. To simulate staggered entry, we randomly assign each user an entry time drawn uniformly over the experimental window and shift their observed time series accordingly. Preliminary checks on the A/A data confirm that user behavior does not differ systematically across calendar days, ensuring that this resampling does not introduce artificial biases. In addition to learning effects, we simulate users churn in the treatment group such that the number of treated units decreases following 1+𝛼𝑐ℎ𝑢𝑟𝑛 (𝑒 −𝛽𝑐ℎ𝑢𝑟𝑛 (𝑡 −𝑡0 ) −1). Control users instead follow the baseline retention observed in the A/A data. This design induces a scenario in which the treatment initially boosts engagement but also accelerates user churn. Lastly, we simulated different short A/B test durations by truncating the data after 𝑇 days. We first conducted an experiment comparing the three estimation methods. We ran 100 Monte-Carlo simulations, covering scenarios with and without churn. Parameters were drawn as: 𝑇 ∼ 𝑈 (7, 14), 𝛼𝑒 𝑓 𝑓 ∼ 𝑈 (0.05, 0.2), 𝛽𝑒 𝑓 𝑓 ∼ 𝑈 (0.1, 0.5), 𝛼𝑐ℎ𝑢𝑟𝑛 ∼ 𝑈 (0, 0.1), and 𝛽𝑐ℎ𝑢𝑟𝑛 ∼ 𝑈 (0.05, 0.3). We then estimated the treatment effect time series using the CCD method [13], the DiD approach [16] and our proposed multi-cohort estimation. We used such results
to estimate the LTE, the Δ𝐸𝑅𝐿𝑉 1 , and the user learning curve. We compared the LTE, the Δ𝐸𝑅𝐿𝑉 with their ground truth values, and we study the user learning confidence intervals size, with results summarized in Table 1. For CCD and DiD, which do not directly estimate Δ𝐸𝑅𝐿𝑉 , we extended their estimate fitting the models 𝑓𝐶 and 𝑓𝑇 (Section 3.2) on the units that entered at the experiment’s start and remained consistently treated or controlled. Table 1 shows that our multi-cohort estimator achieves substantially lower variance compared to CCD and DiD approaches. The 95% confidence interval width of the estimates time series is reduced by 50% relative to CCD and nearly three times relative to DiD, while the standard deviations of LTE and Δ𝐸𝑅𝐿𝑉 bootstrapped estimates are also 30% smaller. Moreover, our estimator is more accurate resulting in a lower mean absolute error compared to the other methods. We then conducted another Monte-Carlo experiment with 100 simulations reproducing the scenario of Figure 1 with:𝑇 = 14, 𝛼𝑒 𝑓 𝑓 ∼ 𝑁 (0.1, 0.07), 𝛼𝑐ℎ𝑢𝑟𝑛 ∼ 𝑁 (0.2, 0.07), 𝛽𝑒 𝑓 𝑓 = 𝛽𝑐ℎ𝑢𝑟𝑛 = 1/3. At first, we analyzed the data ignoring any user learning and user churn phenomena, i.e. with standard short-term A/B tests on only the first week of data, revealing an average positive short term effect 𝑆𝑇 𝐸 = 0.32 (std 0.01). We then used our multi-cohort approach to estimate both the average LTE of 0.02 (std 0.01) and Δ𝐸𝑅𝐿𝑉 of −0.70 (std 0.18). While standard short-term analysis shows a positive immediate click increase and LTE estimates are near zero, Δ𝐸𝑅𝐿𝑉 captures the treatment’s effect on user churn and highlights its overall negative effect for the business. This demonstrates that computing only LTE or STE could have led to the adoption of a feature that reduces long-term user value, while including Δ𝐸𝑅𝐿𝑉 in the analysis prevents such a mistake. These improvements show that combining staggered cohorts via inverse-variance weighting allows more precise and reliable inference of both LTE and Δ𝐸𝑅𝐿𝑉 , making it particularly suitable for short-duration experiments with user learning and churn.
5
Conclusions
In this paper, we have addressed two critical limitations in modern A/B testing: the high variance inherent in staggered-cohort analysis and the measurement gap between steady-state metrics and cumulative business value. By proposing a unified framework, we demonstrated that the Long-Term Effect (LTE) and the Incremental Expected Remaining Lifetime Value (Δ𝐸𝑅𝐿𝑉 ) can be recovered from the same experimental data by selectively managing missing values. Furthermore, we introduced an inverse-variance weighted multicohort estimator that significantly improves estimation precision. Our simulation study grounded in real-world A/A data confirms the practical variance reduction in the estimates. Beyond the specific case of ad-blindness, this dual-metric approach provides a more detailed overview for other product decisions. For instance, a feature might yield Δ𝐸𝑅𝐿𝑉 ≈ 0, indicating no immediate change in cumulative value, while maintaining a positive 𝐿𝑇 𝐸 > 0. In such cases, the framework reveals a strategic trade-off: the treatment provides a long-term benefit to power users (𝐿𝑇 𝐸 > 0) at the cost of increased churn among less-engaged ones, resulting in a neutral cumulative impact (Δ𝐸𝑅𝐿𝑉 ≈ 0). More 1 As standard practice in such experimental evaluations [11, 18, 19], we computed the
summation up to a maximum time interval 𝑇 ∗ . In our experiments 𝑇 ∗ = 𝑇 days.
Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning
generally, this methodology is broadly applicable to diverse A/B testing scenarios, including search and recommendation systems where user behavior evolves over time or where experimental treatments significantly influence user retention (e.g. slow churn due to recommendation quality worsening). Finally, our formulation bridges the gap between A/B testing and traditional Lifetime Value (LTV) frameworks [11]. Unlike standard LTV models that often incorporate complex multi-year discounting over time [10, 17], our 𝑅𝐿𝑉 is optimized for short- to mediumterm online experiments where the discount factor is constant. This choice is well justified in fast-moving digital products, where the high frequency of subsequent A/B tests on the same features often necessitates a focus on the steady state effect and the near term lifecycle value rather than multiple years discounted financial projections. While traditional LTV literature focuses primarily on monetary units, our analysis operates directly on the A/B test metric 𝑚. This is not a limitation, as such metrics can easily be converted into financial value using unit economics already maintained by the business. For example, in our streaming context, a change in number of clicks can be directly translated into revenue by multiplying it by a Cost-Per-Click coefficient. Ultimately, this framework ensures that product decisions are informed by the total impact on the user lifecycle rather than just steady-state behavior. By quantifying both 𝐿𝑇 𝐸 and Δ𝐸𝑅𝐿𝑉 , practitioners can better navigate the complex trade-offs between longterm engagement gains and user retention.
Presenter Bio Dario Simionato is an Applied Research Scientist at the Huawei Ireland Research Centre, where his work focuses on advancing A/B testing and causal discovery algorithms. His research aims to bridge the gap between theory and practice by addressing the applicability challenges inherent in deploying state-of-the-art methods to realworld environments. He was the recipient of the Best Paper Award at ECML PKDD 2022 and holds a Ph.D. from the University of Padua, Italy.
References [1] Sebastian Ankargren, Mattias Frånberg, and Marten Schultzberg. 2024. It’s About Time: What A/B Test Metrics Estimate. arXiv preprint arXiv:2411.06150 (2024). arXiv:2411.06150 [stat.ME] https://arxiv.org/abs/2411.06150
Conference’17, July 2017, Washington, DC, USA
[2] Eva Ascarza, Peter S. Fader, and Bruce G. S. Hardie. 2017. Marketing Models for the Customer-Centric Firm. In Handbook of Marketing Decision Models, Berend Wierenga and Ralf van der Lans (Eds.). International Series in Operations Research & Management Science, Vol. 254. Springer International Publishing, Cham, Switzerland, 297–329. doi:10.1007/978-3-319-56941-3_10 [3] Jan Panero Benway. 1998. Banner blindness: The irony of attention grabbing on the World Wide Web. In Proceedings of the human factors and ergonomics society annual meeting. SAGE Publications, Los Angeles, CA, 463–467. [4] George Casella and Roger L. Berger. 2024. Statistical Inference (2nd ed.). Chapman and Hall/CRC, Boca Raton, FL. [5] Junghoo Cho and Sourashis Roy. 2004. Impact of Search Engines on Page Popularity. In Proceedings of the 13th International World Wide Web Conference (WWW 2004). ACM, New York, NY, USA, 20–29. doi:10.1145/988672.988676 [6] W. G. Cochran. 1954. The Combination of Estimates from Different Experiments. Biometrics 10, 1 (1954), 101–129. [7] Alex Deng, Jiannan Lu, and Shouyuan Chen. 2016. Continuous monitoring of A/B tests without pain: Optional stopping in Bayesian testing. In 2016 IEEE International conference on data science and advanced analytics (DSAA). IEEE, IEEE, Montreal, QC, Canada, 243–252. [8] Pavel Dmitriev, Somit Gupta, Dong Woo Kim, and Garnet Vaz. 2017. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Halifax, Nova Scotia, Canada, 1427–1436. doi:10.1145/3097983.3098024 [9] Peter S. Fader and Bruce G. S. Hardie. 2007. How to Project Customer Retention. Journal of Interactive Marketing 21, 1 (2007), 76–90. [10] Peter S. Fader and Bruce G. S. Hardie. 2010. Customer-Base Valuation in a Contractual Setting: The Perils of Ignoring Heterogeneity. Marketing Science 29, 1 (2010), 85–93. [11] Rosa Ferrentino, Maria Teresa Cuomo, and Carmine Boniello. 2016. On the customer lifetime value: a mathematical perspective. Computational Management Science 13, 4 (2016), 521–539. doi:10.1007/s10287-016-0266-1 [12] Sunil Gupta, Dominique Hanssens, Bruce Hardie, William Kahn, V Kumar, Nathaniel Lin, Nalini Ravishanker, and S Sriram. 2006. Modeling customer lifetime value. Journal of Service Research 9, 2 (2006), 139–155. [13] H. Hohnhold, D. O’Brien, and D. Tang. 2015. Focusing on the Long-Term: It’s Good for Users and Business. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Sydney, NSW, Australia, 1849–1858. [14] Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (1st ed.). Cambridge University Press, Cambridge, UK. [15] Saharon Rosset, Einat Neumann, Uri Eick, and Nurit Vatnik. 2003. Customer Lifetime Value Models for Decision Support. Data Mining and Knowledge Discovery 7, 3 (2003), 321–339. doi:10.1023/A:1024036305874 [16] S. Sadeghi, S. Gupta, S. Gramatovici, J. Lu, H. Ai, and R. Zhang. 2022. Novelty and Primacy: A Long-Term Estimator for Online Experiments. Technometrics 64, 4 (2022), 524–534. [17] Paul A. Samuelson. 1937. A Note on Measurement of Utility. Review of Economic Studies 4, 2 (1937), 155–161. doi:10.2307/2967612 [18] Georgios Theocharous, Philip S Thomas, and Mohammad Ghavamzadeh. 2015. Ad recommendation systems for life-time value optimization. In Proceedings of the 24th international conference on world wide web. ACM, Florence, Italy, 1305–1310. [19] Ana Alina Tudoran, Charlotte Hjerrild Thomsen, and Sophie Thomasen. 2024. Understanding consumer behavior during and after a Pandemic: Implications for customer lifetime value prediction models. Journal of Business Research 174 (2024), 114527. doi:10.1016/j.jbusres.2024.114527