arXiv CSopen access

Optimizing Social Utility in Sequential Experiments

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Optimizing Social Utility in Sequential Experiments Ander Artola Velasco§ , Stratis Tsirtsis† , and Manuel Gomez-Rodriguez§ §

Max Planck Institute for Software Systems, Kaiserslautern, Germany {avelasco, manuel}@mpi-sws.org

arXiv:2605.06520v1 [cs.GT] 7 May 2026

†

Hasso Plattner Institute, Potsdam, Germany [email protected]

Abstract Regulatory approval of products in high-stakes domains such as drug development requires statistical evidence of safety and efficacy through large-scale randomized controlled trials. However, the high financial cost of these trials may deter developers who lack absolute certainty in their product’s efficacy, ultimately stifling the development of ‘moonshot’ products that could offer high social utility. To address this inefficiency, in this paper, we introduce a statistical protocol for experimentation where the product developer (the agent) conducts a randomized controlled trial sequentially and the regulator (the principal) partially subsidizes its cost. By modeling the protocol using a belief Markov decision process, we show that the agent’s optimal strategy can be found efficiently using dynamic programming. Further, we show that the social utility is a piecewise linear and convex function over the subsidy level the principal selects, and thus the socially optimal subsidy can also be found efficiently using divide-and-conquer. Simulation experiments using publicly available data on antibiotic development and approval demonstrate that our statistical protocol can be used to increase social utility by more than 35% relative to standard, non-sequential protocols.

Introduction

Access to markets in high-stakes domains, such as drug development, is strictly governed by regulatory bodies to ensure that new products meet rigorous safety and efficacy standards [1, 2]. Randomized controlled trials (RCTs) serve as one of the primary mechanisms for access control, requiring developers to gather sufficient statistical evidence to prove that a product is safe and effective [3–5]. However, the high financial costs of RCTs can stifle the development of ‘moonshot’ products that could offer high social utility, as developers may hesitate to proceed if they lack absolute certainty in their product’s efficacy [6–8]. To avoid missing such opportunities, calls have emerged for adaptive RCTs [9–11] and targeted subsidies [12–15], particularly in the context of orphan and rare diseases where small patient populations often discourage private investment. Yet, the effectiveness of such interventions depends on how developers strategically respond to them. In this context, a recent line of work [16–18] has argued that the regulatory approval of products is best modeled as a principal-agent game [19], where the regulator (the principal) designs an approval protocol and a hypothesis testing rule to incentivize the product developer (the agent) to act in a way that aligns with the principal’s interests. In this paper, we extend this line of work to consider a setting in which the RCTs are conducted sequentially and are (partially) subsidized by the principal. Our contributions. We introduce a statistical protocol for experimentation where the product developer (the agent) conducts an RCT sequentially and the regulator (the principal) partially subsidizes its cost. At each step of the protocol, the agent and principal update their beliefs about the product’s effectiveness based on the latest experimental outcome. If there is sufficient statistical evidence to reject the null hypothesis [20], 1

the principal approves the product; otherwise, they request that the agent gather more evidence, and the agent may either proceed with the experiment or terminate without approval. By modeling the protocol using a belief Markov decision process [21], we show that the agent’s optimal strategy can be found efficiently using dynamic programming. Further, we show that the social utility is a piecewise linear and convex function over the subsidy level the principal selects, and thus the socially optimal subsidy can also be found efficiently via divide-and-conquer. To validate our statistical protocol, we conduct simulation experiments using publicly available data on antibiotic development and approval. The results show that our protocol can be used to increase social utility by more than 35% relative to standard, non-sequential protocols.1 Further related work. Our work builds upon further related work on the economic aspects of statistical testing, sequential hypothesis testing, and Bayesian experimental design. A recent and closely related line of work studies the economic and strategic incentives arising in regulatory approval processes [16–18, 22, 23]. Therein, Shi et al. [17] and Hossain et al. [18] focus on determining the optimal hypothesis test that the principal can use to control false positives and false negatives, while Tetenov [22] and Bates et al. [16] study, respectively, the design of approval and payment rules to disincentivize agents who know their product is ineffective from participating in the approval process. Relatedly, McClellan [23] considers a setting in which the principal designs approval rules to encourage agent participation without monetary transfers, e.g., by lowering approval standards if previous experiments were not successful. In contrast to these works, we study how the principal can optimally subsidize the agent to increase social utility in a scenario where both are uncertain about the effectiveness of the product. Within the hypothesis testing literature, our work draws on an active line of research on anytime-valid statistical inference using e-values [20, 24, 25], which has been successfully applied to a wide range of statistical problems [16, 26–35]. Most closely related to ours is the work of Bates et al. [16], who also uses e-values in the context of RCTs. However, their focus is on designing contracts that disincentivize agents with ineffective products from participating in the approval process. We instead use e-values to design an approval protocol that provides sequential error guarantees to the principal. Our work also connects to the broad literature on Bayesian experimental design [36–44]. Among these, the closest works to ours [42–44] model a Bayesian agent’s experiment selection problem using (partially observable) Markov decision processes. However, their objective is to design policies that maximize information gain while trading off experimental costs. In contrast, in the approval setting we study, the agent has direct economic incentives to conduct experiments, as product approval is financially beneficial.

A Protocol for Subsidized Sequential Experimentation

We consider an agent (the product developer) who seeks regulatory approval for a product from a principal (the regulator). The product is characterized by an efficacy parameter θ∗ ∈ [0, 1], unknown both to the agent and the principal, with higher values indicating a more effective product.2 Motivated by multi-stage clinical trials [2, 4], in this section, we introduce a sequential approval process in which the agent conducts a sequence of RCTs to provide sufficient evidence that the product meets the principal’s standards. In turn, the principal commits to subsidize a fraction of the agent’s total experimentation cost and, after each trial, they decide whether to approve the product or require the agent to conduct further experimentation. The agent begins the (sequential) approval process with a prior belief B0 about the efficacy θ∗ of its product, which we model using a Beta distribution B0 = Beta(α0 , β0 ), where α0 , β0 > 0 are given parameters.3 At each time step t ∈ [T ] = {0, 1, . . . , T } of the approval process, the agent’s action is to either (i) conduct a randomized trial with a sample size nt ∈ {1, . . . , nmax } and incur a cost c(nt ), where c : N → R+ is a 1 The code for our experiments is publicly available at https://github.com/Human-Centric-Machine-Learning/strategic-

experiments. 2 In certain settings, the principal may be interested not only in the efficacy of a product but also in other properties of the

product such as its safety. In Appendix C, we discuss how to extend the approval process to such settings. 0 and β0 characterize prior information the agent may have about the product based on, e.g., preliminary tests. In Appendix C, we show that the approval process can be extended to more general settings with arbitrary prior beliefs.

3 The parameters α

non-decreasing cost function, or (ii) opt out and stop the approval process at no additional cost, which we represent as nt = 0 with c(0) = 0. If the agent decides to conduct a trial by selecting nt > 0, the agent observes a random outcome Xt ∼ Bin(nt , θ∗ ), which represents the number of successes in the trial and depends on the unknown efficacy θ∗ , and this outcome is then revealed to the principal.4 Then, based on the outcome Xt , the agent updates their (posterior) belief Bt about the true efficacy θ∗ of the product, i.e., Bt+1 = Beta(αt + Xt , βt + nt − Xt ). | {z } | {z } αt+1

(1)

βt+1

Throughout the process, the agent employs a (possibly randomized) policy π ∈ Π to select the sample sizes nt ∼ π(αt , βt , Ct , t) based on their belief Bt = Beta(αt , βt ) about the efficacy of the product, the total cost Pt−1 Ct = k=0 c(nk ) they have incurred so far, and the time step t of the approval process. Moreover, the principal uses the revealed outcomes Xt as evidence to decide on the approval of the product. Formally, the principal conducts a (sequential) hypothesis test with null and alternative hypotheses given by H0 = θ∗ : θ∗ < θb and H1 = θ∗ : θ∗ ≥ θb , (2) where θb ∈ (0, 1) is a baseline efficacy mandated by the principal and known to the agent (e.g., the efficacy of the current standard-of-care treatment in the context of clinical trials). The null hypothesis H0 therefore corresponds to the product failing to meet the principal’s standard, and we will equivalently refer to the principal approving the product as rejecting H0 . In general, the principal can implement any decision rule to reject H0 . However, in what follows, we draw on the literature on sequential hypothesis testing [20, 25, 45], and consider a principal who aims to control the false positive rate, i.e., the probability of approving a product whose efficacy does not exceed the baseline θb . Concretely, based on the sample size nt and revealed outcome Xt , the principal computes—and shares with the agent—a non-negative quantity E(Xt , nt ) ∈ R+ , referred to as an e-value, quantifying the observed evidence against H0 at time t (i.e., larger e-values correspond to stronger evidence against H0 ).5 Then, the principal aggregates all available evidence multiplicatively via a (stochastic) process M , which we will refer to as the test process, and whose value at time t is defined as: ( 1 t=0 Mt = (3) E(Xt−1 , nt−1 ) · Mt−1 t ≥ 1. Finally, based on the test process M , the principal rejects the null hypothesis H0 (approves the product) as soon as the accumulated evidence exceeds a fixed threshold κ ∈ (0, 1) set in advance, i.e., as soon as Mt+1 ≥ 1/κ for some time step t. Here, κ acts as a tolerance parameter specifying how much evidence must be accumulated before rejecting H0 , where smaller values of κ correspond to a more conservative approval standard set by the principal, and note that, if the process fails to yield sufficient evidence for approval within T time steps, it concludes without approval. In the above sequential test, the false positive rate is bounded by κ as long as the e-values are such that EH0 [E(Xt , nt )] ≤ 1, as shown elsewhere [20], i.e., PH0 ({∃t ∈ [T ]: Mt+1 ≥ 1/κ}) ≤ κ.

(4)

Importantly, the above guarantee on the false positive rate holds throughout the entire approval process—a particularly desirable property known as any-time validity [24]. To ensure that EH0 [E(Xt , nt )] ≤ 1 holds, we construct an e-value that exponentiates a sufficient statistic for the unknown efficacy θ∗ [46, 47] by comparing the empirical success rate Xt /nt against a monotone transformation of the baseline θb :6 4 The specific meaning of the number of successes is application dependent. In the context of clinical trials, it may correspond to

the number of patients who recover after receiving a treatment. 5 Computing E(X , n ) plays a role similar to that of a p-value for rejecting H , while offering stronger guarantees in sequential t t 0

settings. See Appendix B for a brief overview of sequential hypothesis testing with e-values. 6 See Appendix C for an extension to arbitrary e-values; proofs are deferred to Appendix F.

Proposition 1. Given the (unknown) efficacy parameter θ∗ ∈ [0, 1], nt > 0, and a binomial variable Xt ∼ Bin(nt , θ∗ ), the positive random variable E(Xt , nt ) = exp Xt − nt · log(1 + θb (e − 1)) (5) is a valid e-value under H0 = {θ∗ : θ∗ < θb }, that is, EXt ∼Bin(nt ,θ∗ ) [E(Xt , nt )] ≤ 1 for any θ∗ ∈ H0 . Furthermore, given the specific form of e-value in Eq. 5, it is easy to verify that the value of the test process Mt is uniquely determined by the parameters αt and βt characterizing the agent’s belief: Proposition 2. At any time step t ∈ [T ], the value of the test process Mt satisfies: Mt = f (αt , βt )

f (α, β) = exp α − α0 − (α + β − α0 − β0 ) · log(1 + θb (e − 1)) .

where

(6)

If the agent’s product is approved, the agent and the principal obtain benefits ρA , ρS > 0, respectively, and we refer to the latter as the social benefit upon approval. In the context of clinical trials, ρA can be interpreted as the (estimated) economic benefit obtained by the agent—the pharmaceutical company—from drug sales if the drug is approved, while ρS represents the corresponding benefit to society resulting from bringing an effective treatment to market. Since the principal also benefits from a product’s approval, it can be in their interest to incentivize the agent to continue experimenting, particularly in situations where the agent’s benefit ρA is not high enough to compensate for the total experimentation cost required to reject H0 . In the following, we focus our attention on a natural and widely-used mechanism through which the principal can provide such an incentive: subsidizing, conditional on approval, a fraction ε ∈ [0, εmax ] of the total cost incurred by the agent, where εmax ≤ 1 denotes the maximum fraction the principal is willing to subsidize [48, 49]. Given the above benefits, experimentation cost, and subsidies, the agent’s and the principal’s (expected) utilities U A and U S are given by " ! # τ τ X X A A ∗ U (π; ε) = Eπ ρ +ε· c(nt ) · 1{Mτ +1 ≥ 1/κ} − c(nt ) θ , t=0

" S

U (ε; π) = Eπ

ρ −ε·

τ X

t=0

# · 1{Mτ +1 ≥ 1/κ} θ

c(nt )

t=0

∗

(7)

where τ = T ∧ min{t ∈ [T ]: nt = 0 or Mt+1 ≥ 1/κ} is the last step of the approval process, 1{•} is the indicator function, and the expectation Eπ [•|θ∗ ] is taken over the random outcomes Xt ∼ Bin(nt , θ∗ ).7 However, since the agent and the principal do not know the true efficacy θ∗ a priori, they cannot find the policy π and subsidy ε that maximize their respective utilities, as defined in Eq. 7. In the next sections, we investigate how the agent and principal may leverage their evolving beliefs about efficacy to find near-optimal policies and subsidies as the approval process progresses.

Experimental Design Using Belief Markov Decision Processes

If the agent does not know the true efficacy θ∗ , we argue that, as the approval process progresses, they act based on their beliefs regarding θ∗ given the trial outcomes observed [50]. Consequently, we consider an agent who determines their trial sample sizes by planning ahead—not according to the (unknown) actual evolution of the process, but according to how they anticipate their beliefs will evolve depending on their actions and anticipated outcomes. To formalize the agent’s planning strategy, we employ the framework of (belief) Markov decision processes [21, 51]. Specifically, for a fixed subsidy ε specified by the principal, we define the process Mε = (S, A, P, rε , T ), whose components we describe next. 7 Throughout, we adopt the convention min ∅ = +∞ and write x ∧ y = min(x, y) for x, y ∈ R ∪ {+∞}.

The state space S = S in ∪ {S out } includes states S = (α, β, C) ∈ S in , where α and β are the parameters characterizing the agent’s belief and C is the agent’s total running cost, as well as a special absorbing state S out , which indicates that the agent has opted out of the approval process. The action space A = {0, . . . , nmax } consists of all possible sample sizes the agent may select. Further, the transition distribution P characterizes how the agent anticipates their belief and cost will evolve after they conduct a trial of size n. Formally, a transition from a state S = (α, β, C) to a state S ′ follows from the (randomized) assignment   (α + X, β + n − X, C + c(n)) if n > 0 and f (S) < 1/κ ′ if n > 0 and f (S) ≥ 1/κ S = S (8)   out S if n = 0 where X ∼ Bin(n, θ) with θ ∼ Beta(α, β), and the function f (S) = f (α, β) as in Eq. 6 if S ∈ S in and f (S out ) = 0. In words, the upper case captures the Bayesian belief update and the additional cost incurred by the agent for conducting an experiment of size n and observing an outcome X (see Eq. 1), the middle case captures the successful conclusion of the approval process, and the lower case captures the scenario in which the agent decides to opt out. Importantly, note that evaluating the likelihood of such transitions does not depend on the efficacy θ∗ and thus is possible solely based on the agent’s belief. Finally, the reward rε characterizes the agent’s anticipated profit or loss due to conducting a trial under subsidy ε. Formally, the reward for transitioning from state S to state S ′ via action n is given by ( −c(n) + (ρA + ε · (C + c(n))) · 1{f (S ′ ) ≥ 1/κ} if S ̸= S out and f (S) < 1/κ ε ′ r (S, n, S ) = (9) 0 if S = S out or f (S) ≥ 1/κ. In words, as long as the approval process is in progress, the reward is simply equal to the (negative) cost c(n) the agent anticipates to incur by conducting a trial with their chosen sample size n. If the updated state S ′ leads to product approval (i.e., f (S) < 1/κ and f (S ′ ) ≥ 1/κ), the agent anticipates receiving their one-off benefit ρA > 0 and a subsidized fraction of their total incurred cost throughout the process, as determined by the subsidy ε selected by the principal. Now that we have defined all the components of the Markov decision process Mε , we can formalize the agent’s strategy within the process. Starting from the initial state S0 = (α0 , β0 , 0), the agent aims to select a policy π : S × [T ] → ∆(A) maximizing their anticipated utility " T # X A ε r (St , nt , St+1 ) S0 = (α0 , β0 , 0) , Ū (π; ε) = Eπ (10) t=0

where the expectation is taken over the state transitions that the agent anticipates to occur throughout the approval process, given their initial belief. Here, it is important to note that both the reward rε and the transition distribution P are known to the agent, hence they can evaluate the anticipated utility of any given policy π before the approval process starts. Moreover, due to Bellman’s optimality principle in MDPs [51], for any S ∈ S and time step l ∈ [T ], the agent’s optimal policy π ε ∈ argmaxπ Ū A (π; ε) satisfies that " T # X ε ε ε ε π (S, l) ∈ argmax Vπ (S, l) where Vπ (S, l) = Eπ r (St , nt , St+1 ) Sl = S . (11) π

t=l

In the above equation, the function Vπε is often referred as the value function. Moreover, note that Ū A (π; ε) = Vπε (α0 , β0 , 0, 0). In this context, a natural question is how the anticipated utility Ū A (π; ε) relates to the agent’s true utility A U (π; ε) defined in Eq. 7. The following proposition shows that the anticipated utility is equal to true agent utility averaged over the agent’s initial belief about the true efficacy θ∗ of their product at the start of the approval process. 5

Proposition 3. Let ε ∈ [0, εmax ] be any subsidy set by the principal, and π be any agent policy. Then, Ū A (π; ε) = Eθ∗ ∼B0 U A (π; ε) . (12) The above proposition reveals that an agent implementing the decision policy π ε is, in fact, a Bayesian decision maker maximizing their expected utility [39, 41, 52]. Next, we formalize the principal’s strategy within the process. Let Q be a prior distribution characterizing the principal’s prior knowledge about the agent’s initial belief (α0 , β0 ). Then, the principal aims to select a subsidy ε maximizing the anticipated social utility " " T ## X Ū S (ε; π) = E(α0 ,β0 )∼Q Eπ ρS − ε · Ct+1 · 1 {0 < f (St ) < 1/κ ≤ f (St+1 )} S0 = (α0 , β0 , 0) , (13) t=0

where the inner expectation is taken over the state transitions that the principal anticipates to occur throughout the approval process, given the prior knowledge about the agent’s initial belief. In the next sections, we analyze how the agent and the principal maximize their anticipated utilities Ū A (π; ε) and Ū S (ε; π) within a natural Stackelberg setting [53]. In this setting, the principal (the leader) first commits to a subsidy level ε∗ maximizing the social utility Ū S , accounting for the fact that, given any ε, the agent (the follower) will implement the optimal policy π ε that maximizes their utility Ū A . Then, after observing this commitment, the agent implements the optimal policy. Formally, the subsidy level ε∗ is the solution to the following optimization problem: ε∗ = argmax Ū S (ε; π ε )

subject to π ε ∈ argmax Ū A (π; ε).

(14)

π∈Π

ε∈[0,εmax ]

The above Stackelberg setting fits a variety of real-world applications. For example, in clinical trials, funding agencies act as leaders who commit to and announce a funding program; drug developers are the followers who apply to the program and, if successful, run a trial [54, 55].

Optimal Experimental Design under Subsidies

In this section, we characterize the agent’s optimal policy π ε along with its corresponding value function V ε (S, l) = max Vπε (S, l) π∈Π

∀ S ∈ S, l ∈ [T ].

(15)

To this end, we first note that, once the process Mε reaches a state where f (S) ≥ 1/κ, or the state S out , the action n = 0 is always optimal. This is implied by the transition dynamics and the reward definition in Eqs. 8 and 9. Therefore, to find the policy π ε , it suffices to consider states S where 0 < f (S) < 1/κ. Further, we show that, in the process Mε , the number of (unique) reachable states S where f (S) ≥ 1/κ is finite: Proposition 4. For any realization of the process Mε , any reached state S where 0 < f (S) < 1/κ belongs to a finite set S r ⊂ S. Moreover, under linear cost c(n), the set S r has size O((nmax )2 · T 3 ). As an immediate consequence, we can find an optimal (deterministic) policy π ε using standard planning methods. In particular, in Algorithm 2, we provide an adaptation of the classical value iteration algorithm [56], which is guaranteed to find an optimal policy π ε in O((nmax )4 · T 3 ). In the remainder of this section, we derive several key structural insights about the optimal policy π ε and the optimal value function V ε , which will be helpful to efficiently find the principal’s optimal subsidy ε∗ . Our starting point is the observation that, in light of Proposition 2, the value of the test process M at a state S = (α, β, C) increases with α and decreases with β. This suggests that states with larger α and smaller β are more favorable. The following proposition formalizes this intuition by showing that the optimal value function V ε satisfies a monotonicity property in both the belief parameters and the accumulated cost.

Proposition 5. For any time step l ∈ [T ] and pair of states S = (α, β, C) and S ′ = (α′ , β ′ , C ′ ) such that f (S) < 1/κ and f (S ′ ) < 1/κ, the following holds: 1. V ε (α, β, C, l) ≤ V ε (α′ , β, C, l) if α ≤ α′ ; 2. V ε (α, β, C, l) ≥ V ε (α, β ′ , C, l) if β ≤ β ′ ; 3. V ε (α, β, C, l) ≤ V ε (α, β, C ′ , l) if C ≤ C ′ . Leveraging the above proposition, we can characterize the conditions under which the optimal policy π ε opts out of the approval process by selecting the action n = 0. In particular, the following proposition shows that π ε exhibits a threshold structure: at a state with belief (α, β), it selects n = 0 if and only if β exceeds a threshold that depends on α (see Figure 3 in Appendix D for an illustration). Proposition 6. For any time step t ∈ [T ] and total cost Ct , there exists a non-decreasing function β̃ : R+ → R+ such that, for any state St = (αt , βt , Ct ) ∈ S in , the optimal policy π ε opts out of the approval process if βt > β̃(αt ), and it does not opt out if βt < β̃(αt ). Furthermore, we can also characterize how the agent’s anticipated utility Ū A (π ε ; ε) under the optimal policy π ε depends on the subsidy. To this end, we first show that, for any policy π, the value function Vπε is linear in the subsidy ε: Proposition 7. For any policy π, state S ∈ S, and time step l ∈ [T ], the value function Vπε admits a linear decomposition Vπε (S, l) = Vπ0 (S, l) + ε · Aπ (S, l), (16) where Vπ0 (S, l) is the value function in the unsubsidized process M0 , and Aπ (S, l) ≥ 0 denotes the expected total cost incurred conditional on approval, starting from state S at time l (see Eq. 57 in Appendix F.9).

Building upon this result, the agent’s anticipated utility Ū A (π ε ; ε) under the optimal policy π ε admits a concise structural characterization, as formalized by the following proposition: Proposition 8. The agent’s optimal anticipated utility Ū A (π ε ; ε), and its expectation E(α0 ,β0 )∼Q [Ū A (π ε ; ε)] according to the principal’s belief Q, are piecewise linear, continuous, and convex functions of the subsidy ε over a partition P = {ε0 , ε1 , . . . , εL } of the interval [0, εmax ], with 0 = ε0 < ε1 < · · · < εL = εmax . Moreover, for each interval of P, the agent’s optimal policy is constant, i.e., π ε = πi for all ε ∈ [εi , εi+1 ). In the next section, we leverage the above characterization of the agent’s anticipated optimal utility to develop an algorithm that computes the principal’s optimal subsidy ε∗ , as defined in Eq. 14.8

Finding Optimal Subsidies

To find the optimal subsidy ε∗ , our starting point is the observation that, for any policy π, the anticipated social utility Ū S (ε; π) is a decreasing function of ε. This is because, as the subsidy ε increases, the principal covers a higher fraction of the agent’s cost, but the probability that the product receives approval remains unchanged. More formally, we have the following proposition: Proposition 9. For any policy π and subsidy ε ∈ [0, εmax ], the anticipated social utility Ū S (ε; π) admits a linear decomposition Ū S (ε; π) = ρS · E(α0 ,β0 )∼Q [Pπ ({∃t ∈ [T ]: f (St+1 ) ≥ 1/κ} | S0 )] − ε · E(α0 ,β0 )∼Q [Aπ (S0 , 0)] ,

(17)

8 In principle, Eq. 14 may allow for multiple solutions. However, our objective is not to characterize the full set of solutions, but

rather to provide an algorithmic procedure to compute one such solution.

Algorithm 1 Finds the Principal’s Optimal Subsidy 1: Input: MDP solver SolveMDP, maximum subsidy εmax , principal’s belief Q 2: Initialize: I ← ∅, U ← ∅ max 3: (πL , VL0 , AL ) ← SolveMDP(M0 ), (πR , VR0 , AR ) ← SolveMDP(Mε ) ▷ Compute optimal policies for ε = 0 and

ε = εmax using the decomposition in Proposition 7 4: U ← U ∪ {(0, Ū S (0; πL ))} 5: V̄L0 ← E(α0 ,β0 )∼Q [VL0 (α0 , β0 , 0, 0)], ĀL ← E(α0 ,β0 )∼Q [AL (α0 , β0 , 0, 0)] 6: V̄R0 ← E(α0 ,β0 )∼Q [VR0 (α0 , β0 , 0, 0)], ĀR ← E(α0 ,β0 )∼Q [AR (α0 , β0 , 0, 0)] ▷ Compute the average agent value using the principal’s belief. 7: Push (εL , πL , V̄L0 , ĀL , εR , πR , V̄R0 , ĀR ) into I 8: while I is not empty do 9: Pop (εL , πL , V̄L0 , ĀL , εR , πR , V̄R0 , ĀR ) from I 10: if AL ̸= AR then 11: εint ← (V̄L0 − V̄R0 )/(ĀR − ĀL ) ▷ Compute the candidate subsidy to evaluate 0 12: (πint , Vint , Aint ) ← SolveMDP(Mεint ) ▷ Compute the optimal policy for the candidate subsidy 0 0 13: V̄int ← E(α0 ,β0 )∼Q [Vint (α0 , β0 , 0, 0)], Āint ← E(α0 ,β0 )∼Q [Aint (α0 , β0 , 0, 0)] ▷ Compute the average agent value using the principal’s belief. 0 14: if V̄int + εint · Āint ≤ V̄L0 + ·εint ĀL then ▷ Verify if the optimal policy improves over πR and πL 15: U ← U ∪ {(εint , Ū S (εint ; πR ))} ▷ Found a vertex; compute social utility using Eq. 13 and principal’s belief Q 16: else ▷ Split the subsidy interval 0 0 , Āint ), (εint , πint , V̄int , Āint , εR , πR , V̄R0 , ĀR )} into I 17: Push {(εL , πL , V̄L0 , ĀL , εint , πint , V̄int 18: end if 19: end if 20: end while 21: return ε∗ where (ε∗ , u∗ ) = argmax(ε,u)∈U u ▷ Return the optimal subsidy

In the above expression, the first term corresponds to the principal’s anticipated probability of rejecting H0 under policy π, and the second term corresponds to the anticipated total cost borne by the principal through the subsidy. As a consequence, and in light of Proposition 8, in each interval [εi , εi+1 ) of the partition P where a fixed policy πi is optimal, the anticipated social utility Ū S (ε; πi ) is a decreasing (linear) function of the subsidy over [εi , εi+1 ), and therefore it is maximized at the left point εi : max

ε∈[εi ,εi+1 )

Ū S (ε; πi ) = Ū S (εi ; πi ).

(18)

Crucially, the principal can compute each policy πi without knowing the agent’s initial belief (α0 , β0 ), since the optimal policy for the process Mεi does not depend on the initial state. This stands in contrast to many settings in the literature on (Bayesian) Stackelberg games, where the principal typically must anticipate the agent’s best response by averaging over the agent’s private information [57, 58]. Leveraging the above results, we derive an efficient divide-and-conquer procedure to find the optimal subsidy ε∗ . The procedure (i) explicitly constructs the partition P and (ii) applies Proposition 9 to determine the optimal anticipated social utility on each interval [εi , εi+1 ) by evaluating it at the left endpoint εi . Algorithm 1 summarizes the overall procedure, and the following proposition establishes its correctness. Proposition 10. Algorithm 1 is guaranteed to find an optimal subsidy ε∗ in a finite number of iterations. Algorithm 1 maintains a stack I of intervals, alongside their corresponding optimal policies at the endpoints and the linear value function decompositions provided by Proposition 7. We abstract the computation of these policies and value functions into a procedure, SolveMDP, which may be implemented using a value-iteration algorithm (e.g., Algorithm 3). The stack I is initialized to the full interval [0, εmax ] (lines 3 to 7). At each iteration, the algorithm calculates the intersection point εint between the endpoint value functions, V̄L0 + ε · ĀL and V̄R0 + ε · ĀR (line 11), where the bar denotes evaluation at S0 and averaging over the principal’s belief Q. 8

Social utility gain ∗ vs. non-sequential (%) Optimal subsidy, ε

0.21 1600

0.20

1580

0.19 0.18

1560 ∗

ε = 0.108 Partition P

1540 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.17

Probability of opting out

Social utility, U S (ε; π ε ) ($M)

0.22

1620

0.16

0.9

(a) Social utility vs. subsidy (ρS = $2000 M)

0.3 0.2 0.1 0.0 60 50

Non-sequential (no subsidy) Non-sequential (optimal subsidy)

40 30

10 20 30 Social-to-agent approval benefit ratio, ρS /ρA

(b) Optimal subsidy and social utility gain vs. ρS /ρA

Figure 1: Subsidizing antibiotic development. The figure shows the results of the approval process for an antibiotic with true (unknown) efficacy θ∗ = 0.65. Panel (a) shows the result of running Algorithm 1 to compute the optimal subsidy for the principal ε∗ = 0.108 when the social benefit of approval is ρS = $2000 M. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8), and the colors indicate the probability that the agent opts out by selecting n = 0 during the approval process (before the drug is approved). Panel (b) shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy, together with the percentage increase in social utility of the sequential approval protocol relative to a non-sequential approval protocol in which the agent is restricted to a single trial with nmax = 800. The error bars represent 95% bootstrapped confidence intervals. Then, it computes the optimal value function at εint (line 12). Due to the convexity of the optimal value function, if the optimal value function at εint coincides with the value V̄L0 + εint · ĀL = V̄R0 + εint · ĀR , the policy πL is optimal in [εL , εint ), and the policy πR is optimal in [εint , πR ]. In this case, εint is a vertex of P, and the algorithm stores the social utility at εint (line 15). Conversely, if the optimal policy at εint strictly improves over the policy πL , then the algorithm has found a new interval of the partition, and the stack I is updated (line 17). The algorithm then iterates the same steps over all intervals in the stack I.9

Experiments: Subsidizing Antibiotic Development

Antimicrobial resistance is a major global threat, projected to cause 10 million deaths annually by 2050 [59, 60]. Yet, FDA antibiotic approvals have dropped from 13% of all drugs in 1980 to 4% in the 2000s [61], largely for economic reasons: treatments are short, prices must stay low to ensure availability, use is restricted to limit resistance, and competition from existing or generic drugs is intense [62, 63]. Consequently, many major pharmaceutical companies have exited or reduced antibiotic pipelines [64], while small biotech firms often struggle financially [65–67]. To counter this, public and private efforts have focused on incentivizing and subsidizing development [68–70]. In this section, we conduct a series of experiments to demonstrate the effectiveness of our approval protocol in optimally subsidizing antibiotic development. Experimental setup. The principal subsidizes the development of an antibiotic and conducts a hypothesis test with κ = 0.05 (i.e., a false positive rate of at most 0.05) to determine whether the antibiotic’s (unknown) efficacy θ∗ = 0.65 exceeds the benchmark θb = 0.5. The agent can conduct up to four trials (T = 3), each with a maximum sample size of nmax = 200 patients. Although data on the economic cost and sales of antibiotic development are mostly private, recent reports estimate that the average present value of sales is approximately $240 M [71], the per-patient Phase III cost is $66 k [72], and the average fixed cost per trial is $48.9 M [73]. Therefore, we set ρA = $240 M, and c(n) = $48.9 M + $0.066 M · n for any n ∈ {1, . . . , nmax }. Further, we assume the agent has a non-informative (uniform) prior with α0 = 1, β0 = 1, known to the 9We report runtime measurements of Algorithm 1 in Appendix G.

principal, and vary ρS . Refer to Appendix G for additional details regarding our experimental setup, and to Appendix H for results under alternative parameter choices and extensive sensitivity analyses. Results. For an antibiotic with a social benefit upon approval of ρS = $2000 M (a ratio ρS /ρA ≈ 8.3), Panel (a) of Figure 1 shows (i) the social utility U S (ε; π ε ), (ii) the partition P, which consists of 49 intervals, and (iii) the agent’s opt-out probability, over the entire range of subsidy levels. We find that, under the optimal subsidy ε∗ = 0.108, the social utility increases by ∼5.5% and the agent’s opt-out probability decreases by ∼22% compared to a scenario with no subsidies. Interestingly, we also find that the optimal subsidy ε∗ —which maximizes the anticipated social utility Ū S (ε; π ε )—also maximizes the true social utility U S (ε; π ε ). Further, Panel (b) of Figure 1 shows the optimal subsidy ε∗ and the social utility gain compared to two non-sequential baselines for different values of the social-to-agent approval benefit ratio ρS /ρA . We find that the optimal subsidy increases with the social-to-agent approval benefit ratio, reflecting that higher societal utility strengthens the principal’s incentive to subsidize experimentation. We also find that, compared to two non-sequential baselines in which the agent is restricted to conducting at most a single clinical trial (with a larger maximum sample size of nmax = 800), our approval protocol yields substantial gains in social utility. Specifically, relative to a non-sequential protocol without subsidies, our protocol increases social utility by approximately 50%–60%, depending on the ratio ρS /ρA and, relative to a non-sequential protocol with optimal subsidies, our protocol still achieves gains exceeding 35%.

Discussion and Limitations

In this section, we highlight several limitations of our work and discuss avenues for future research. Methodology. In our work, we have considered a Beta–Binomial model, which is particularly natural in the context of RCTs. However, in other application domains, it may be desirable to consider more general models where the agent’s beliefs, the experimental outcomes, and the e-values exhibit greater complexity—for instance, through dependence across trials or through the inclusion of safety characteristics. In Appendix C, we outline how to extend our framework to these more general settings; however, this extension introduces significant computational challenges. Furthermore, although our results do not require the principal to know the agent’s initial belief (i.e., its private information B0 ), observing the agent’s experimental actions over time may provide information about its prior, suggesting that the principal could, in principle, dynamically update its belief about the agent’s prior. It would be interesting to incorporate such learning and elicitation into a sequential approval protocol in future work [74]. In addition, it would also be valuable to extend our protocol to allow for uncertainty in the agent’s benefit ρA , which in practice may vary across product developers and contexts. Finally, real-world regulatory settings may involve additional non-economic factors that are not explicitly captured in our model. For instance, clinical trial duration, patient follow-up requirements, and other operational constraints can affect both the feasibility and optimality of sequential experimentation protocols, and accounting for these factors remains an important direction for future work. Implementation and Evaluation. We have conducted a case study applying our approval protocol to antibiotic development, a well-known setting characterized by underinvestment and market failure, which is particularly suited to studying the effects of subsidies (see Appendix H for further experimental results using different parameter values). However, it would be interesting to extend this analysis to other settings—such as orphan drugs or rare disease treatments—where trial costs and sample sizes may differ substantially from those in standard antibiotic development and may therefore lead to different optimal subsidies. We also empirically evaluated the computational cost of computing the agent’s optimal decision policy and the optimal subsidy. In our experiments, both procedures were efficient and typically completed within a few minutes (see Appendix G for more details). However, as shown theoretically in Section 4, the complexity of computing the optimal agent policy scales polynomially with the number of actions and trials. Consequently, extending our methodology to more general settings—such as those described in Appendix C—might require developing approximate algorithms, as computing (Stackelberg) equilibria in general Bayesian games is known to be computationally intractable in many cases [57]. Broader Impact. Regulatory agencies and product developers are progressively exploring more flexible 10

and data-driven approval methodologies, including Bayesian approaches, to better balance safety and innovation [11]. Our work contributes to this direction by providing a principled framework for designing subsidy mechanisms that improve social utility in approval processes, which may inform future policy discussions and be of interest to both public and private regulators and decision makers.

Conclusions

How can approval protocols be designed to incentivize experimentation? In this work, we have addressed this question by introducing a sequential approval protocol that allows the agent to continuously refine its knowledge about the product and the principal to subsidize a fraction of the agent’s experimental costs—all while maintaining anytime-valid guarantees on the false positive rate. Along the way, we have shown that the agent can efficiently compute the optimal experimental policy, and the principal can find the subsidy that maximizes social utility, even when anticipating that the agent selects its policy strategically. Finally, using real-world data on antibiotic development, we have demonstrated that our sequential, subsidized protocol can substantially improve social utility, yielding gains of up to 60% relative to non-sequential designs without subsidies. More broadly, we hope our work provides insights for designing approval protocols that better align agents’ incentives with social objectives. Acknowledgements. Gomez-Rodriguez acknowledges support from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 945719). Tsirtsis acknowledges supports from the Alexander von Humboldt Foundation in the framework of the Alexander von Humboldt Professorship (Humboldt Professor of Technology and Regulation awarded to Sandra Wachter) endowed by the Federal Ministry of Education and Research via the Hasso Plattner Institute.

References [1] Dale H Gieringer. The safety and efficacy of new drug approval. Cato J., 5:177, 1985. [2] U.S. Food and Drug Administration. Demonstrating substantial evidence of effectiveness for human drug and biological products: Guidance for industry. Draft guidance, U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), December 2019. Draft—Not for Implementation. Clinical/Medical. [3] Alec B O’Connor. Building comparative efficacy and tolerability into the fda approval process. Jama, 303(10):979– 980, 2010. [4] Perrine Janiaud, Telba Irony, Estelle Russek-Cohen, and Steven N Goodman. U.S. food and drug administration reasoning in approval decisions when efficacy evidence is borderline, 2013-2018. Ann. Intern. Med., 174(11):1603– 1611, November 2021. [5] Alberto Farina, Federico Moro, Frederick Fasslrinner, Annahita Sedghi, Miluska Bromley, and Timo Siepmann. Strength of clinical evidence leading to approval of novel cancer medicines in europe: A systematic review and data synthesis. Pharmacology Research & Perspectives, 9(4):e00816, 2021. [6] Allan S Detsky. Are clinical trials a cost-effective investment? Jama, 262(13):1795–1800, 1989. [7] Simon Frantz. Why are clinical costs so high? Nature Reviews Drug Discovery, 2(11), 2003. [8] Linda Martin, Melissa Hutchens, Conrad Hawkins, and Alaina Radnov. How much do clinical trials cost? Nature Reviews Drug Discovery, 16(6):381–382, 2017. [9] C Hendricks Brown, Thomas R Ten Have, Booil Jo, Getachew Dagne, Peter A Wyman, Bengt Muthén, and Robert D Gibbons. Adaptive designs for randomized trials in public health. Annual review of public health, 30(1):1–25, 2009.

[10] Rajiv Mahajan and Kapil Gupta. Adaptive design clinical trials: Methodology, challenges and prospect. Indian journal of pharmacology, 42(4):201–207, 2010. [11] U.S. Food and Drug Administration. Use of bayesian methodology in clinical trials of drug and biological products. Draft guidance, Center for Biologics Evaluation and Research and Center for Drug Evaluation and Research, Food and Drug Administration, March 2026. [12] Clinical trial-specific funding opportunities. https://grants.nih.gov/policy-and-compliance/policytopics/clinical-trials/specific-funding-opportunities. Accessed: 2026-03-31. [13] Clinical trials grants program. https://www.fda.gov/industry/orphan-products-grants-program/clinicaltrials-grants-program. Accessed: 2026-03-31. [14] Clinical trials. https://www.dfg.de/en/research-funding/funding-opportunities/programmes/individual/ clinical-trials. Accessed: 2026-03-31. [15] The european and developing countries clinical trials partnership. https://www.edctp.org/. Accessed: 2026-0331. [16] Stephen Bates, Michael I Jordan, Michael Sklar, and Jake A Soloff. Principal-agent hypothesis testing. arXiv preprint arXiv:2205.06812, 2022. [17] Flora C Shi, Stephen Bates, and Martin J Wainwright. Sharp results for hypothesis testing with risk-sensitive agents. arXiv preprint arXiv:2412.16452, 2024. [18] Safwan Hossain, Yatong Chen, and Yiling Chen. Strategic hypothesis testing. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. [19] Sanford J Grossman and Oliver D Hart. An analysis of the principal-agent problem. In Foundations of insurance economics: Readings in economics and finance, pages 302–340. Springer, 1992. [20] Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference, 2023. [21] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998. [22] Aleksey Tetenov. An economic theory of statistical testing. CeMMAP working papers 50/16, Institute for Fiscal Studies, Sep 2016. [23] Andrew McClellan. Experimentation and approval mechanisms. Econometrica, 90(5):2215–2247, 2022. [24] Peter Grünwald, Rianne de Heide, and Wouter Koolen. Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1091–1128, 03 2024. [25] Aaditya Ramdas and Ruodu Wang. Hypothesis testing with e-values. Foundations and Trends in Statistics, 1(1-2):1–390, 07 2025. [26] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting, 2022. [27] Ziyu Xu and Aaditya Ramdas. Online multiple testing with e-values. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 3997–4005. PMLR, 02–04 May 2024. [28] Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. E-detectors: A nonparametric framework for sequential change detection. The New England Journal of Statistics in Data Science, 2(2):229–260, 2024. [29] Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Trans. Inf. Theor., 70(2):1178–1203, February 2024. [30] Ian Waudby-Smith, Ricardo Sandoval, and Michael I. Jordan. Universal log-optimality for general classes of e-processes and sequential hypothesis tests, 2025.

[31] Ben Chugg, Etienne Gauthier, Michael I Jordan, Aaditya Ramdas, and Ian Waudby-Smith. Post-hoc large-sample statistical inference. arXiv preprint arXiv:2603.08002, 2026. [32] Etienne Gauthier, Francis Bach, and Michael I. Jordan. Backward conformal prediction. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [33] Baihe Huang, Eric Xu, Kannan Ramchandran, Jiantao Jiao, and Michael I Jordan. Towards anytime-valid statistical watermarking. arXiv preprint arXiv:2602.17608, 2026. [34] Ander Artola Velasco, Stratis Tsirtsis, and Manuel Gomez Rodriguez. Auditing pay-per-token in large language models. In The 29th International Conference on Artificial Intelligence and Statistics, 2026. [35] Guneet S. Dhillon, Javier Gonzalez, Teodora Pandeva, and Alicia Curth. E-scores for (in)correctness assessment of generative model outputs. In The 29th International Conference on Artificial Intelligence and Statistics, 2026. [36] Herbert Robbins. Optimal stopping. The American Mathematical Monthly, 77(4):333–343, 1970. [37] Goran Peskir and Albert N Shiryaev. Optimal stopping and free-boundary problems. Lectures in Mathematics. ETH Zürich. Birkhauser Verlag AG, Basel, Switzerland, 2006 edition, August 2006. [38] Warren B Powell and Ilya O Ryzhov. Optimal Learning. Wiley Series in Probability and Statistics. Wiley-Blackwell, Hoboken, NJ, March 2012. [39] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 8(5–6):359–483, November 2015. [40] Namhoon Cho, Seokwon Lee, Hyo-Sang Shin, and Antonios Tsourdos. Bayesian learning approach to model predictive control. arXiv preprint arXiv:2203.02720, 2022. [41] Thomas Kleine Buening, Christos Dimitrakakis, Hannes Eriksson, Divya Grover, and Emilio Jorge. Minimax-bayes reinforcement learning, 2023. [42] Wanggang Shen and Xun Huan. Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning. Computer Methods in Applied Mechanics and Engineering, 416:116304, 2023. [43] Chen Cheng and Xun Huan. Optimal stopping for sequential bayesian experimental design. arXiv preprint arXiv:2509.21734, 2025. [44] Wanggang Shen, Jiayuan Dong, and Xun Huan. Variational sequential optimal experimental design using reinforcement learning. Comput. Methods Appl. Mech. Eng., 444(118068):118068, September 2025. [45] A. Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics, 16(2):117 – 186, 1945. [46] Victor H. de la Peña. A General Class of Exponential Inequalities for Martingales and Ratios. The Annals of Probability, 27(1):537 – 564, 1999. [47] Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales. Probability Surveys, 17(none):257 – 317, 2020. [48] Matthew J Renwick, David M Brogan, and Elias Mossialos. A systematic review and critical assessment of incentive strategies for discovery and development of novel antibiotics. J. Antibiot. (Tokyo), 69(2):73–88, February 2016. [49] Wan-Shu Wu and Kai Zhao. Government r&d subsidies and enterprise r&d activities: theory and evidence. Economic Research-Ekonomska Istraživanja, 35(1):391–408, 2022. [50] John C Harsanyi. Bayesian decision theory, rule utilitarianism, and arrow’s impossibility theorem. Theory and Decision, 11(3):289–317, 1979.

[51] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. [52] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling, 2020. [53] Heinrich von Stackelberg. Market Structure and Equilibrium. Springer, Berlin, Germany, 2011 edition, August 2010. [54] U.S. Congress. 26 U.S.C. 45C — clinical testing expenses for certain drugs for rare diseases or conditions. Internal Revenue Code, 2024. Accessed: 2026-05-03. [55] National Institutes of Health. Small business funding. https://seed.nih.gov/small-business-funding, 2026. Accessed: 2026-05-03. [56] Onésimo Hernández-Lerma and Jean Bernard Lasserre. Discrete-Time Markov Control Processes. Springer New York, 1996. [57] Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy to commit to. In Proceedings of the 7th ACM Conference on Electronic Commerce, EC ’06, page 82–90, New York, NY, USA, 2006. Association for Computing Machinery. [58] Praveen Paruchuri, Jonathan P Pearce, Janusz Marecki, Milind Tambe, Fernando Ordonez, and Sarit Kraus. Playing games for security: An efficient exact algorithm for solving bayesian stackelberg games. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 2, pages 895–902, 2008. [59] GBD 2021 Antimicrobial Resistance Collaborators. Global burden of bacterial antimicrobial resistance 1990-2021: a systematic analysis with forecasts to 2050. Lancet, 404(10459):1199–1226, September 2024. [60] Laura J Shallcross, Simon J Howard, Tom Fowler, and Sally C Davies. Tackling the threat of antimicrobial resistance: from policy to sustainable action. Philos. Trans. R. Soc. Lond. B Biol. Sci., 370(1670):20140082, June 2015. [61] Kevin Outterson, John H Powers, Enrique Seoane-Vazquez, Rosa Rodriguez-Monguio, and Aaron S Kesselheim. Approval and withdrawal of new antibiotics and other antiinfectives in the U.S., 1980-2009. J. Law Med. Ethics, 41(3):688–696, 2013. [62] David M Shlaes. The economic conundrum for antibacterial drugs. Antimicrob. Agents Chemother., 64(1), December 2019. [63] Nupur Gargate, Mark Laws, and Khondaker Miraz Rahman. Current economic and regulatory challenges in developing antibiotics for gram-negative bacteria. NPJ Antimicrob. Resist., 3(1):50, June 2025. [64] Benjamin Plackett. Why big pharma has abandoned antibiotics. Nature, 586(7830):S50–S52, October 2020. [65] Gilles Courtemanche, Rohini Wadanamby, Amritanjali Kiran, Luisa Fernanda Toro-Alzate, Mathew Diggle, Dipanjan Chakraborty, Ariel Blocker, and Maarten van Dongen. Looking for solutions to the pitfalls of developing novel antibacterials in an economically challenging system. Microbiol. Res. (Pavia), 12(1):173–185, March 2021. [66] Laura J V Piddock, Yewande Alimi, James Anderson, Damiano de Felice, Catrin E Moore, John-Arne Røttingen, Henry Skinner, and Peter Beyer. Advancing global antibiotic research, development and access. Nat. Med., 30(9):2432–2443, September 2024. [67] Nadya Wells, Vinh-Kim Nguyen, and Stephan Harbarth. Novel insights from financial analysis of the failure to commercialise plazomicin: Implications for the antibiotic investment ecosystem. Humanit. Soc. Sci. Commun., 11(1), July 2024. [68] Kevin Outterson, John H Rex, Tim Jinks, Peter Jackson, John Hallinan, Steve Karp, Deborah T Hung, Francois Franceschi, Tyler Merkeley, Christopher Houchens, Dennis M Dixon, Michael G Kurilla, Rosemarie Aurigemma, and Joseph Larsen. Accelerating global innovation to address antibacterial resistance: introducing CARB-X. Nat. Rev. Drug Discov., 15(9):589–590, September 2016.

[69] Michael Anderson, Dimitra Panteli, Robin van Kessel, Gunnar Ljungqvist, Francesca Colombo, and Elias Mossialos. Challenges and opportunities for incentivising antibiotic research and development in europe. Lancet Reg. Health Eur., 33(100705):100705, October 2023. [70] United States Congress. H.R. 7352: PASTEUR Act of 2026, 2026. To amend the Public Health Service Act to establish a program to develop innovative antimicrobial drugs. [71] Sakib Rahman, Olof Lindahl, Chantal M Morel, and Aidan Hollis. Market concentration of new antibiotic sales. J. Antibiot. (Tokyo), 74(6):421–423, June 2021. [72] Stella Stergiopoulos, Sara B Calvert, Carrie A Brown, Josephine Awatin, Pamela Tenaerts, Thomas L Holland, Joseph A DiMasi, and Kenneth A Getz. Cost drivers of a hospital-acquired bacterial pneumonia and ventilatorassociated bacterial pneumonia phase 3 clinical trial. Clin. Infect. Dis., 66(1):72–80, January 2018. [73] Thomas J Moore, Hanzhe Zhang, Gerard Anderson, and G Caleb Alexander. Estimated costs of pivotal trials for novel therapeutic agents approved by the US food and drug administration, 2015-2016. JAMA Intern. Med., 178(11):1451–1457, November 2018. [74] Flora C Shi, Martin J Wainwright, and Stephen Bates. Instance-adaptive hypothesis tests with heterogeneous agents. arXiv preprint arXiv:2510.21178, 2025. [75] B. Campbell, N Balakrishnan, and Brani Vidakovic. Encyclopedia of statistical sciences. Methods and Applications of Statistics. John Wiley & Sons, Nashville, TN, 2 edition, December 2005. [76] Joseph P Simmons, Leif D Nelson, and Uri Simonsohn. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci., 22(11):1359–1366, November 2011. [77] Etienne Gauthier, Francis Bach, and Michael I. Jordan. Betting on equilibrium: Monitoring strategic behavior in multi-agent systems, 2026. [78] J. Ville. Étude Critique de la Notion de Collectif. Collection des monographies des probabilités. Gauthier-Villars, 1939. [79] R.T. Rockafellar. Convex Analysis. Princeton landmarks in mathematics and physics. Princeton University Press, 1970.

Summary of Notation

In Table 1 we summarize the key symbols used in the main body of the paper. Table 1: Summary of notation. Symbol

Description

κ θ∗ θb H0 , H1 t l T ε εmax ε∗ Bt αt , βt Q nt nmax π πε Vε Vπε Aπ c Xt Et Mt f Ct Mε rε St τ P ρA ρS UA US Ū A Ū S

Principal’s false positive rate bound True (unknown) product efficacy Baseline efficacy Null and alternative hypotheses Time index Initial time index (if not 0) Maximum number of trials (time horizon of the belief MDP) Principal’s subsidy level Maximum allowable subsidy Principal’s optimal subsidy Agent’s belief at time t Parameters of the agent’s belief at time t Principal’s belief over the agent’s prior Agent’s action (sample size) at time t Maximum sample size Agent’s policy Optimal policy under subsidy ε Optimal value function under subsidy ε Value function of policy π under subsidy ε Expected total cost conditional on approval under policy π Cost function Experimental outcome at time t e-value at time t Principal’s test process at time t Mapping from beliefs to the test process Cumulative experimental cost at time t Belief MDP under subsidy ε Reward function in the belief MDP under subsidy ε State of Mε at time t Stopping time (last non-absorbing state of Mε ) Partition associated with fixed policies Agent’s approval benefit Social approval benefit Agent’s utility under θ∗ Social utility under θ∗ Agent’s anticipated utility computed via the belief MDP Principal’s anticipated social utility computed via the belief MDP

Background on Sequential Hypothesis Testing Using e-values

In this section, we provide a brief overview of sequential hypothesis testing using e-values. For a detailed exposition, we refer the reader to Ramdas et al. [25]. A central object in classical hypothesis testing is the p-value, used to assess a null hypothesis H0 against an alternative H1 . Despite their widespread use across scientific disciplines, p-values suffer from important limitations when used without appropriate precautions. A canonical example is p-hacking (also called sampling to a foregone conclusion): if one repeatedly collects data, computes a p-value, and checks whether it falls below a fixed threshold (e.g., 0.05), rejection of H0 is eventually guaranteed even when H0 is true [75, 76]. Alongside classical methods based on p-values, a rich line of work has developed methods based on a game-theoretic formulation of statistics that is better suited to sequential hypothesis testing [20] and mitigates issues such as p-hacking. The key objects in this framework are e-values, which are nonnegative random variables E that satisfy: EH0 [E] ≤ 1, (19) where the expectation is taken under any distribution in H0 . Typically, as with p-values, e-values are computed from some observed data X that is intended to provide evidence to reject H0 ; in the sequential approval protocol considered in Section 2, we make this dependence explicit by writing E(n, X), where n is the sample size selected by the agent and X the number of successes in the control trial. The intuition behind e-values is straightforward: since E has expectation at most 1 under H0 , it can take large values only with small probability. Hence, observing a large e-value can be interpreted as evidence against H0 —and the larger the e-value, the stronger the evidence. Consequently, one can construct a statistical test that rejects H0 whenever the e-value takes high values, i.e., whenever E ≥ 1/κ for a certain threshold κ ∈ (0, 1). More precisely, by Markov’s inequality, this threshold automatically controls the false positive rate: PH0 (E ≥ 1/κ) ≤ κ · EH0 [E] ≤ κ.

(20)

Proposition 11. If E is an e-value for H0 , i.e., EH0 [E] ≤ 1, then the non-sequential hypothesis test ϕ = 1{E ≥ 1/κ} is a level-κ test for H0 for any κ ∈ (0, 1): its false positive rate is at most κ.

Informally, e-values are as general as p-values in the sense that they exist under essentially the same technical conditions and can be transformed into one another. Their main advantage, however, arises in sequential settings—i.e., when data arrive over time [27–29, 34, 77]—as in clinical trials, which is the primary reason our formalism in Section 2 and 3 is built on e-values. One way to understand this advantage is to note that the defining condition of e-values (Eq. 19) is preserved under a wide range of operations. For example, convex combinations of e-values and multiplications of (independent) e-values remain valid e-values. In contrast, analogous operations do not generally preserve the validity of p-values. To formalize the use of e-values in a sequential setting, let F = (Ft )∞ t=0 be a filtration on a given sample space, where each Ft is a σ-algebra representing the information available at time t. For instance, F may be the filtration generated by the observations X0 , . . . , Xt . Then, given a sequential data stream X0 , X1 , . . . , the goal of sequential hypothesis testing is to maintain and update a running measure of evidence against H0 as new observations arrive. In the e-value framework, this can be achieved by constructing a sequence E0 , E1 , . . . adapted to F —so that each Et depends only on data observed up to time t—and satisfying EH0 [Et | Ft−1 ] ≤ 1.

(21)

The above condition is the sequential analogue of the defining property in Eq. 19. For instance, if the observations Xt are independent, one may construct each Et from Xt alone, in which case the e-values E0 , . . . , Et are mutually independent. In Section 4, we adopt this construction, but we emphasize that more complex e-values can be defined if the experimental protocol requires it. For example, in Eq. 28 we consider a data-dependent construction in which new evidence is collected only if a prior experiment satisfies certain conditions. 17

By interpreting the quantity Et as the new evidence against H0 obtained at time t, a canonical way to construct a sequential test is to define a stochastic test process M = (Mt )t≥0 as ( 1 t=0 Mt = (22) Et · Mt−1 t ≥ 1, which simply corresponds to aggregating the previous e-values multiplicatively.10 By contrast, combining p-values in a sequential or dependent setting typically requires specialized corrections to maintain validity [24]. Whenever the e-values satisfy Eq. 21, the above process M forms a (non-negative) supermartingale under H0 , i.e., EH0 [Mt | Ft−1 ] ≤ Mt−1 . (23) Intuitively, this property means that if H0 is true, the value of Mt does not, in expectation, increase over time. Conversely, sustained growth of the process Mt provides evidence against H0 . Thus, a natural sequential test to reject H0 is to monitor whether Mt ≥ 1/κ for a chosen κ ∈ (0, 1), and to reject H0 as soon as this condition is met. Ville’s inequality [78], a sequential extension of Markov’s inequality, guarantees that this procedure controls the false positive rate uniformly over time:

Theorem 12 (Ville’s inequality). If the process M is a non-negative supermartingale, then, PH0 (∃t ∈ N : Mt ≥ 1/κ) = PH0 sup Mt ≥ 1/κ ≤ κ.

(24)

t≥0

Moreover, validity is preserved if the test is stopped at any stopping time τ adapted to F (but possibly depending on the observed data): PH0 (Mτ ≥ 1/κ) ≤ κ, (25)

which is a property known as any-time validity and provides a principled solution to the problem of p-hacking. Although the above results hold for any choice of e-values satisfying Eq. 21, a natural practical question is which specific form to adopt. We discuss this, and how different choices affect the approval process, in Appendix C.

10 In Eq. 30 of Appendix C we discuss a different method to construct a test process M without multiplying e-values.

Extension to Arbitrary Belief Functions and e-values

The approval protocol introduced in Section 2 can be formulated in a more general setting (at the expense of losing tractability), as we now outline. Generalization of the approval process. In a general setting, the agent may begin the approval process with an arbitrary initial belief B0 ∈ ∆(Θ), where Θ denotes a general parameter space for the unknown parameter θ∗ characterizing the product. At each step of the approval process, the agent selects an action nt ∈ Agen ∪ {0}, where Agen is a (potentially infinite) set describing the design of the next randomized controlled trial (e.g., sample size, participant characteristics, etc.), and nt = 0 again denotes the option to opt out. If nt ̸= 0, an outcome Xt ∼ P (Xt | nt , θ∗ ) is observed and the agent incurs a cost c(nt ), where P (· | nt , θ∗ ) is a likelihood function characterizing the data-collection process specific to each experimental setting and c : Agen → R+ is a cost function. The agent then updates its belief via the Bayesian posterior: Bt+1 (θ) = R

Bt (θ) · P (Xt | nt , θ) . B (θ) · P (Xt | nt , θ) dθ Θ t

To decide on approval, the principal partitions the parameter space as Θ = Θ0 ⊔ Θ1 and defines the following null and alternative hypotheses: ( H0 = {θ∗ ∈ Θ0 } (null) (26) H1 = {θ∗ ∈ Θ1 } (alternative). This formulation allows for richer principal objectives. For instance, the principal may be concerned not only with a drug’s efficacy but also with its safety, in which case H0 may correspond to treatments that are either insufficiently effective or unsafe. To conduct the above hypothesis test sequentially, at each time step, the principal may compute any test process value Mt , subject only to two conditions: (i) the test process M must be predictable with respect to the filtration F = (Ft )∞ t=0 generated by the data and agent actions X0 , n0 , X1 , n1 , . . . , and (ii) the test process M must be a supermartingale under H0 . The principal can then reject H0 whenever Mt ≥ 1/κ at any time step, as described in Appendix B. A particular construction of the test process that generalizes Eq. 3 proceeds multiplicatively as ( 1 t = 0, Mt = (27) Et−1 (Xt−1 , nt−1 ) · Mt−1 t ≥ 1, where each Et−1 is an e-value adapted to F that may depend on all previously observed experimental outcomes and actions, i.e., X0 , n0 , X1 , n1 , . . . , Xt−1 , nt−1 , and satisfies: EH0 [Et (nt , Xt ) | Ft−1 ] ≤ 1. This allows modeling adaptive experimental designs in which future trials depend on past outcomes. Such settings arise naturally in multi-stage clinical trials [2], where progression to a subsequent phase may be contingent on earlier success. This can be represented by defining: ( E(nt , Xt ) if g(n0 , X0 , . . . , nt−1 , Xt−1 ) = 1, Et (nt , Xt ) = (28) 0 if g(n0 , X0 , . . . , nt−1 , Xt−1 ) = 0, where E(·, ·) is a fixed e-value and g is a decision rule determining whether the agent is permitted to conduct the next experiment. If g(n0 , X0 , . . . , nt−1 , Xt−1 ) = 0 at any time t − 1, the test process stops accumulating evidence and approval becomes impossible after time t. Given a test process M , the utilities of both the agent and the principal can be defined analogously to Eq. 7, and a belief Markov decision process Mε can be formulated as in Section 3, with the difference that

the state space now includes any possible belief and any possible value of the test process, i.e.,11 S gen = ∆(Θ) ×

R+ |{z}

total cost C

R+ |{z}

∪

S out ,

test process M

with a state St transitioning to St+1 after taking action nt and observing outcome Xt according to:   (Bt+1 , Ct + c(nt ), Mt+1 (nt , Xt )) if nt ̸= 0 and 0 < f (St ) < 1/κ, if f (St ) ≥ 1/κ, St+1 = S   out S if nt = 0 or St = S out ,

(29)

where for St = (Bt , Ct , Mt ) ̸= S out , Bt+1 (θ) ∝ Bt (θ) · P (Xt | nt , θ). In the above, f is the function that maps a state to the test process value, namely f (St ) = Mt for St = (Bt , Ct , Mt ) ̸= S out and f (S out ) = 0. While this formulation is fully general, it also significantly complicates the subsequent analysis of the optimal agent policy. Alternative e-values and statistical power. For a given choice of test process M , it is important to note that the statistical power of the test—verifying whether Mt ≥ 1/κ at any time step—need not be 1. That is, with nonzero probability it may occur that, even for θ∗ ≥ θb , the approval process is unsuccessful. To quantify the statistical power of sequential tests based on e-values (or supermartingales), there exists a canonical notion called e-power. For a given e-value E for the null H0 , its e-power against an alternative L ∈ H1 is defined as EL [log E]. Under this definition, likelihood ratios are the optimal e-values for simple nulls and alternatives, i.e., when H0 and H1 each correspond to a single probability distribution. In contrast, when H0 and H1 are composite, as in the approval process of Section 4, the choice of e-value becomes more intricate [30]. For concreteness, in Section 4 we therefore adopt a general strategy that constructs e-values by exponentiating random variables [26, 46]. The rationale is as follows: if we wish to test whether an arbitrary random variable X has mean E[X] < θb , we consider the quantity X − θb , which we expect to be large when E[X] > θb and small otherwise. To form an e-value, we consider the positive quantity exp(X − θb ); since this can have expectation exceeding 1 even when E[X] < θb , we obtain a valid e-value satisfying Eq. 19 by shifting the argument by an appropriately chosen constant λ, i.e., exp(X − θb + λ), as given by Proposition 1. This e-value is particularly simple and admits a closed-form expression that simplifies the exposition in Section 3; however, in the context of the approval protocol in Section 2, other choices of e-values may achieve better statistical power for certain values of the efficacy parameter θ∗ . Indeed, for E as defined in Proposition 1 and using the notation therein, the e-power is: Eθ∗ [log Et ] = nt · θ∗ − nt · log(1 + θb (e − 1)).

Thus, Et has positive power Eθ∗ [log Et ] ≥ 0 if and only if θ∗ ≥ log(1 + θb (e − 1)) > θb . This means that if θb < θ∗ < log(1 + θb (e − 1)), it may be that Et ̸→ ∞ even as n → ∞. In other words, the process Mt might never exceed the threshold 1/κ if θ∗ is sufficiently close to—yet strictly above—θb , even for RCTs of arbitrary size. The standard remedy (see Chapter 3 of [25]) is to form a mixture of e-values or supermartingales that generalizes Proposition 1. We now outline how our approval protocol can be extended to use such mixtures, and refer the reader to Appendix H.2.7 for additional experimental results. Our starting point is the simple observation that the e-value E(Xt , nt ) in Proposition 1 can be equivalently written as a likelihood ratio between a Binomial distribution evaluated at θb and one evaluated at a particular alternative: !Xt !nt −Xt θ̃ 1 − θ̃ E(Xt , nt ) = · , θb 1 − θb where θ̃ ≥ θb is the unique efficacy value satisfying: e=

θ̃ · (1 − θb ) . (1 − θ̃) · θb

11 If the e-values depend on all past observations, as in Eq. 28, the state space may need to be augmented to explicitly track the

history of experimental outcomes.

To generalize this construction, one can mix over other values of the alternative parameter. This is achieved by choosing any (smooth) distribution P mix over [θb , 1] and defining the mixture test process as:12 mix Mt+1 =

Z 1 θb

P mix (θ) ·

θ θb

Pti=0 Xi Pti=0 ni −Pti=0 Xi 1−θ · dθ. 1 − θb

(30)

Pt mix It can be shown that, if the true efficacy θ∗ > θb lies in the support of P mix , then Mt+1 → ∞ as i=0 ni → ∞, i.e., the mixture achieves asymptotic power 1 [20]. In this context, Algorithm 3 can be used to compute the optimal agent policy for any subsidy with minimal modifications: it suffices to replace the test process function f with β−β0 α−α0 Z 1 1−θ θ mix mix · dθ, f (α, β) = P (θ) · θb 1 − θb θb and Algorithm 1 can then be used to compute the principal’s optimal subsidy with no further modification.13 100 Uniform mixture Exponential mixture Non-mixture

β − β0

50 α − α0

100

Figure 2: Rejection regions in belief space. The figure shows, for each agent belief with parameters (α, β), whether the condition f (α, β) ≥ 1/κ is satisfied (i.e., whether H0 is rejected; shaded region), under different test processes: the test process defined using the non-mixed e-values in Proposition 1 (orange), a test process defined in Eq. 30 with a uniform mixture P mix = U(θb , 1) (blue), and a test process defined in Eq. 30 with an exponential mixture P mix = Exp(10) restricted to (θb , 1) (green). Here, we set κ = 0.05, θb = 0.5, and α0 = β0 = 1. We have presented the non-mixture process M defined in Section 2 in the main text for two reasons. First, it yields closed-form expressions which simplify the exposition and provide a more transparent (geometric) intuition for the belief Markov decision process (see Figure 3). Second, the multiplicative property in Eq. 3 is exploited extensively in the theoretical proofs in Appendix F, whereas the mixture process M mix does not satisfy this property in non-trivial cases where P mix is not a point mass. However, we conjecture that all our theoretical results in Section 3 and 4 carry over to the mixture setting—in particular, because the belief (αt , βt ) remains a sufficient statistic for Mtmix via the monotone function f mix , and the Bayesian update of the 12 An argument analogous to that of Proposition 1 shows that M mix is a supermartingale under H . 0 t 13 Note that the fact that Algorithm 1 returns an optimal subsidy only relies on the agent’s optimal utility being a piecewise

linear and convex function of the subsidy (Proposition 8), which holds independently of the specific form of f , as can be seen from its proof in Appendix F.10

agent is unchanged—but adapting the proofs would require routing arguments through the Markov property of the belief MDP rather than through the algebraic identity Mt+1 = Et · Mt . We leave a formal treatment of this extension for future work. In Appendix H.2.7, we present experimental results using the above mixture test process with a uniform mixture P mix = U(θb , 1). Further, in Figure 2, we compare the regions of the belief space where f (α, β) ≥ 1/κ, i.e., where H0 is rejected, for different test processes. The test process without mixture (based on Proposition 1) yields a linear rejection region, while the mixture processes yield nonlinear regions whose boundary slope increases for larger values of (α, β).

Geometry of the Belief Markov Decision Process

In Figure 3 we illustrate the geometry of the belief MDP Mε in the (α, β)-plane corresponding to all the possible beliefs for the agent. In blue, we represent the beliefs to which the agent may transition after selecting an action nt > 0 (the figure shows nt = 3 for concreteness). The exact next state depends on the realized outcome Xt of the experiment, as described in Eq. 8. Further, in light of Proposition 2, note that each pair (α, β) is associated with a value f (α, β) for the test process M , with pairs such that f (α, β) ≥ 1/κ corresponding to a state where H0 as been rejected by the principal. The condition f (α, β) ≥ 1/κ ⇐⇒ α − α0 − (α + β − α0 − β0 ) log(1 + θb (e − 1)) ≥ log(1/κ)

(31)

corresponds to a linear region in the (α, β)-plane, represented in green in Figure 3 and labelled “Reject H0 ”. Note that the reward function defined in Eq. 9 only includes the positive term ρA for a transition that crosses the boundary of this region. Similarly, the red region corresponds to beliefs for which—given a fixed total cost C ≥ 0 and initial time step l ∈ [T ]— it is optimal for the agent to opt out of the approval process by choosing n = 0. This region is bounded by the curve α 7→ β̃(α) in Proposition 6. Refer to Appendix H for concrete numerical solutions to the MDP Mε . β − β0

β̃(α) Opt out region

Reject H0

log 1/κ 1−log(1+θ b (e−1))

α − α0

Figure 3: Illustration of the geometry of the state space in the MDP Mε .

Value Iteration in the Belief Markov Decision Process

In Algorithm 2, we present, for completeness, an adaptation of the value-iteration algorithm [51] to exactly compute an optimal policy π ε for the MDP Mε .14 In the proof of Proposition 4 in Appendix F.5, we showed that the set of accessible states in t time steps, which we denoted by S r (t), satisfies: |S r (t)|= O (nmax )2 · t2 . Building on this bound, it is straightforward to verify that the value-iteration method in Algorithm 2 has a time complexity O (nmax )4 · T 3 . Indeed, for a given l, the set |S r (l) \ {S out }|= O (nmax )2 · l2 , and for each such state, the inner loops (line 4 and 5) iterate over all actions n and sums over possible outcomes x. That is, for each l, the algorithm performs O (nmax )4 · l2 iterations. Summing over l, we conclude that max 4 3 the total complexity is O (n ) · T . We refer the reader to Appendix G for further details regarding the implementation of the algorithm.

Algorithm 2 Finds the Optimal Agent Policy π ε for the belief MDP Mε

1: Input Subsidy ε, horizon T , max trials nmax , approval benefit ρA , cost function c(n) = c0 +c1 n, threshold

κ, prior parameters (α0 , β0 ), test process function f

2: Initialize V ε (S, T + 1) ← 0 for all states S ∈ S r 3: for l = T down to 0 do 4: for each S = (α, β, C) ∈ S r (l) \ {S out } such that f (α, β) < 1/κ do

for each action n ∈ {1, . . . , nmaxP } do n Qε (S, n, l) ← −(c0 + c1 n) + x=0 BB(n, α, β)(x) · NextValue(x, n) where ( NextValue(x, n) = ρA + ε(C + c0 + c1 n) if f (α + x, β + n − x) ≥ 1/κ 8: V ε (α + x, β + n − x, C + c0 + c1 n, l + 1) otherwise 9: end for 10: V ε (S, l) ← max 0, maxn∈{1,...,nmax } Qε (S, n, l) 11: if V ε (S, l) > 0 then 12: π ε (S, l) ← arg maxn∈{1,...,nmax } Qε (S, n, l) 13: ▷ Ties broken arbitrarily 14: else 15: π ε (S, l) ← 0 16: end if 17: end for 18: end for 19: return V ε , π ε 5: 6: 7:

Note that the above algorithm can readily be modified to return the linear decomposition of the optimal value function at a subsidy ε in Proposition 7, namely Vπεε = Vπ0ε + ε · Aπε , as shown in Algorithm 3.

14 BB(n, α, β) denotes the Beta–Binomial distribution.

Algorithm 3 Finds the Optimal Agent Policy π ε and Value Function Decomposition 1: Input Subsidy ε, horizon T , max trials nmax , approval benefit ρA , cost c(n) = c0 + c1 n, threshold κ,

prior (α0 , β0 ), test process function f V 0 (S, T + 1) ← 0, A(S, T + 1) ← 0 for all S ∈ S r

2: Initialize V ε (S, T + 1) ← 0, 3: for l = T down to 0 do 4: 5: 6: 7:

9: 10:

for each S = (α, β, C) ∈ S r (l) \ {S out } s.t. f (α, β) < 1/κ do for each action n ∈ {1, . . . , nmax } do cn ← c0 + c1 n, (αx′ , βx′ ) ← (α+x, β+n−x) n X ε Q (S, n, l) ← −cn + BB(n, α, β)(x) · v ε (x, n) Q0 (S, n, l) ← −cn + QA (S, n, l) ←

n X x=0

x=0 n X

x=0

BB(n, α, β)(x) · v 0 (x, n)

BB(n, α, β)(x) · a(x, n)

where (writing approved for f (αx′ , βx′ ) ≥ 1/κ): ( ρA + ε(C + cn ) if approved v ε (x, n) = ε ′ ′ V (αx , βx , C+cn , l+1) otherwise (

ρA if approved 0 ′ ′ V (αx , βx , C+cn , l+1) otherwise ( C + cn if approved a(x, n) = ′ ′ A(αx , βx , C+cn , l+1) otherwise 0

v (x, n) =

end for n∗ ← arg maxn∈{1,...,nmax } Qε (S, n, l) if Qε (S, n∗ , l) > 0 then 14: V ε (S, l) ← Qε (S, n∗ , l), V 0 (S, l) ← Q0 (S, n∗ , l), 15: π ε (S, l) ← n∗ 16: else 17: V ε (S, l) ← 0, V 0 (S, l) ← 0, A(S, l) ← 0 18: π ε (S, l) ← 0 19: end if 20: end for 21: end for 22: return V ε , V 0 , A, π ε

11: 12: 13:

▷ Note: Qε = Q0 + ε · QA by construction ▷ Ties broken arbitrarily A(S, l) ← QA (S, n∗ , l)

Proofs

F.1

Proof of Proposition 1

Fix θ∗ ∈ [0, 1] and nt > 0. Let Xt ∼ Bin(nt , θ∗ ) and define E(Xt , nt ) = exp Xt − nt · log(1 + θb (e − 1)) . We explicitly show that the expectation of the above random variable is upper-bounded by 1 if θ∗ ∈ H0 = {θ∗ : θ∗ < θb }: EXt ∼Bin(nt ,θ∗ ) [E(Xt , nt )] = EXt ∼Bin(nt ,θ∗ ) [exp Xt − nt · log(1 + θb (e − 1)) ] 1 = n · EXt ∼Bin(nt ,θ ∗ ) [exp (Xt )] b (1 + θ (e − 1)) t 1 (∗) ∗ ∗ nt = n · (1 − θ + θ · e) b (1 + θ (e − 1)) t (∗∗)

≤ =1

n (1 + θb (e − 1)) t

· (1 − θb + θb · e)nt

where in (∗) we have used the formula for the moment-generating function of the Binomial distribution, and in (∗∗) we have used that, by definition, if θ∗ ∈ H0 , then θ∗ < θb .

F.2

Proof of Proposition 2

We begin by noting that the definition of the test process in Eq. 3 together with the e-value in Eq. 5 implies that, if the agent has continued the approval process up to time t ≤ T by selecting non-null n0 , . . . , nt , then: Mt+1 =

t Y s=0

exp Xs − ns · log(1 + θb (e − 1))

= exp

t X s=0

Xs −

t X s=0

(32)

(33)

ns · log(1 + θ (e − 1))

Now, unfolding Eq. 1 for t time steps, we can write the parameters αt and βt that characterize the belief of the agent at time t as:  t X   αt+1 = α0 + Xs   s=0

(34)

t t  X X    ns − Xs , βt+1 = β0 + s=0

⇐⇒

s=0

 t X    Xs = αt+1 − α0   s=0

(35)

t t  X X    n = β − β + Xs = αt+1 − α0 + βt+1 − β0 ,  s t+1 0 s=0

s=0

Thus, substituting the above in Eq. 32 we readily obtain: Mt+1 = exp

t X s=0

Xs −

t X s=0

! ns · log(1 + θb (e − 1))

= exp αt+1 − α0 − (αt+1 − α0 + βt+1 − β0 ) · log(1 + θb (e − 1))

F.3

Auxiliary Lemma to Proposition 3

Here, we prove a lemma that will be used later in the proof of Proposition 3. Lemma 1. Let π ∈ Π be a policy, S ∈ S be a state such that S ̸= S out , denote S = (α, β, C) and fix an integrable function G : R3 → R. Further, denote by π(S, t)(•) the density over A given by π at time t, by Beta(α, β)(•) the density of the Beta distribution, by Bin(n, θ)(•) the density of a Binomial distribution with parameters n and θ, and by BB(n, α, β)(•) the density of a Beta-Binomial distribution. Consider any random variables θ∗ , nt and Xt with a joint density P such that nt and θ∗ are independent and:15   P (nt ) = π(S, t)(nt ) P (θ∗ ) = Beta(α, β)(θ∗ )   P (Xt | nt , θ∗ ) = Bin(nt , θ∗ )(Xt ).

(36)

Then, it holds that: Eθ∗ ∼Beta(α,β) Ent ∼π(S,t) EXt ∼Bin(nt ,θ∗ ) [G(nt , θ∗ , Xt )] =

Ent ∼π(S,t) EXt ∼BB(nt ,α,β) Eθ∗ ∼Beta(α+Xt ,β+nt −Xt ) [G(nt , θ∗ , Xt )]. (37)

Proof. We first note that Xt |nt follows a Beta-Binomial distribution with parameters (nt , α, β). Indeed, denoting by B the beta function and by Γ the gamma function: Z 1 P (Xt = k|nt ) = P (Xt = k|nt , θ∗ )P (θ∗ ) dθ∗ 0 Z 1 nt 1 (θ∗ )k (1 − θ∗ )nt −k (θ∗ )α−1 (1 − θ∗ )β−1 dθ∗ = k B(α, β) 0 Z 1 nt k (θ∗ )k+α−1 (1 − θ∗ )nt −k+β−1 dθ∗ B(α, β) 0 nt B(k + α, nt − k + β) = B(α, β) k nt Γ(k + α)Γ(nt − k + β) Γ(α + β) = , k Γ(nt + α + β) Γ(α)Γ(β)

which is precisely the density of a Beta-Binomial distribution with parameters (nt , α, β). Similarly, θ∗ |nt , Xt follows a Beta distribution with parameters (α + Xt , β + nt − Xt ): P (Xt |θ∗ , nt )P (θ∗ ) P (Xt |nt ) i h ih nt 1 ∗ α−1 ∗ β−1 ∗ Xt ∗ nt −Xt (θ ) (1 − θ ) (θ ) (1 − θ ) Xt B(α,β) = nt B(α+Xt ,β+nt −Xt0)

P (θ∗ |Xt , nt ) =

B(α,β)

(θ∗ )Xt (1 − θ∗ )nt −Xt (θ∗ )α−1 (1 − θ∗ )β−1 = B(α + Xt , β + nt − Xt0) ∗ α+Xt −1 (θ ) (1 − θ∗ )β+nt −Xt −1 = . B(α + Xt , β + nt − Xt ) Since nt and θ∗ are independent, the joint distribution can be written as: P (θ∗ , nt , Xt ) = P (nt ) · P (θ∗ ) · P (Xt |θ∗ , nt )

(38)

15We adopt the notation θ ∗ , n , and X to align with the proof of Proposition 3, where this lemma is applied to the efficacy, t t

sample size, and outcomes of the approval process.

Alternatively, we can factor the distribution P as: P (θ∗ , nt , Xt ) = P (nt ) · P (Xt |nt ) · P (θ∗ |nt , Xt ).

(39)

Then, we can conclude using the law of total expectations for the expression E(nt ,θ∗ ,Xt )∼P [G(nt , θ∗ , Xt )] to obtain: Eθ∗ ∼Beta(α,β) Ent ∼π(S,t) EXt ∼Bin(nt ,θ∗ ) [G(nt , θ∗ , Xt )]

= Ent ∼π(S,t) EXt ∼BB(nt ,α,β) Eθ∗ ∼Beta(α+Xt ,β+nt −Xt ) [G(nt , θ∗ , Xt )] (40)

F.4

Proof of Proposition 3

We fix a subsidy ε ∈ [0, εmax ] and a policy π ∈ Π. Since ε is fixed, for clarity, we omit it from the notation for the remainder of the proof, and we will simply use the notation r(•) to denote the (agent’s) reward in Mε . Similarly, Vπ will denote the value function in Mε for policy π. To show that Proposition 3 holds, we will proceed by induction over the time steps of the MDP Mε . We first introduce some additional notation. Preliminaries. For any t ≤ T (representing a duration) and any l ≤ T − t (representing the initial time step), we define the following value function of π in the MDP Mε : " l+t # X Vπ (S, l, t) = Eπ r(Sk , nk , Sk+1 ) Sl = S , (41) k=l

which is the total expected reward obtained by policy π starting from state S at time l and taking t steps. Moreover, given a state S = (α, β, C) ∈ S \ {S out } of the MDP such that f (S) < 1/κ, t ≤ T and l ≤ T − t, consider the approval process described in Section 2 when: (i) the initial time index is l, (ii) the agent has initial belief B0 = Beta(α, β), (iii) the test process M starts at Ml = f (α, β) < 1/κ and (iv) the subsidy A includes an additional total cost C. Then, we define the agent’s utility Ūl,t (π|S) (averaged over its initial belief) for the first t steps as: " " A Ūl,t (π|S) = Eθ∗ ∼Beta(α,β) Eπl



ρA + ε · C +  −

X j=0

c(nj ) · 1{∃j ∈ [τ (S, l, t)]: f (α, β) · Mj+1 ≥ 1/κ}



τ (S,l,t)



τ (S,l,t)

(42)

c(nj ) θ∗ , B0 = Beta(α, β), C0 = C

j=0

where τ (S, l, t) = t ∧ min{j ∈ {0, . . . , t}: nj = 0 or f (α, β) · Mj+1 ≥ 1/κ} is the last step of the approval process using the shifted policy πl defined as πl (•, j) = π(•, j + l). Here, Eπl [•|θ∗ , B0 = Beta(α, β), C0 = C] indicates that: (i) all outcomes Xt are drawn with a fixed efficacy θ∗ , and (ii) the agent has initial belief Beta(α, β) and cumulated cost C. For instance, in the above expectation, the first action at index j = 0 taken by the agent is sampled from the distribution π(α, β, C, l). Alternatively, for a state S such that S = S out or f (S) ≥ 1/κ, we simply define: A Ūl,t (π|S) = 0. (43) Importantly, note that, by definition: A Ū0,T (π | (α0 , β0 , 0)) " "



τ (α0 ,β0 ,0,0,0)

=Eθ∗ ∼Beta(α0 ,β0 ) Eπ0 ρA + ε

X j=0

c(nj ) · 1{∃j ∈ [τ (α0 , β0 , 0, 0, 0)] : Mj+1 ≥ 1/κ}

τ (α0 ,β0 ,0,0,0)

−

## ∗

c(nj ) θ , B0 = Beta(α0 , β0 ), C0 = 0

j=0

= Eθ∗ ∼B0 [U A (π, ε)]. In light of the above, our goal will be to show by induction over t that: A Ūl,t (π | S) = Vπ (S, l, t) for any S ∈ S, t ≤ T, l ≤ T − t.

Then, by particularizing to t = T , l = 0 and S = (α0 , β0 , 0) and using Eq. 10, we will obtain that A Ū A (π; ε) = Vπ (α0 , β0 , 0, 0, T ) = Ū0,T (π| α0 , β0 , 0 ) = Eθ∗ ∼B0 [U A (π, ε)],

which is the statement in Proposition 3. A Base case t = 0. We will show that Ūl,0 (π|S) = Vπ (S, l, 0) holds for any S ∈ S and l ≤ T . We first consider out the case where S = S or f (S) ≥ 1/κ. In this case, note that the transition dynamics of the MDP (Eq. 8) imply that at any time step k ≥ l, Sk = S out or Sk = S. Thus, the expression in Eq. 41 only contains rewards that are null (see Eq. 9), and hence: Vπ (S, l, 0) = 0. A A (π|S) = 0. Thus, Ūl,0 (π|S) = Vπ (S, l, 0). Similarly, by definition (Eq. 43), Ūl,0 out Consider now the non-trivial case where S ̸= S and f (S) < 1/κ, and write S = (α, β, C). In this case, the value function starting at time l for t = 0 can be expanded as (note that t = 0 corresponds to a single step in the MDP):

Vπ (S, l, 0) = Eπ [r(Sl , nl , Sl+1 )|Sl = S] = Eπ −c(nl ) + (ρA + ε · (C + c(nl ))) · 1{f (Sl+1 ) ≥ 1/κ)} Sl = S = Enl ∼π(S,l) EXl ∼BB(nl ,α,β) [ − c(nl )

+ (ρA + ε · (C + c(nl ))) · 1{f (S) · E(Xl , nl ) ≥ 1/κ)}]

In the above, we denote by Xl ∼ BB(nl , α, β) a sample from the Beta-Binomial distribution, corresponding to sampling θl ∼ Beta(α, β), and then Xl ∼ Bin(nl , θl ), and we have used the definition of the test process in Eq. 3. On the other hand, let t = 0 in Eq. 42. Then, τ (S, l, 0) = 0, and we obtain:16 16Whenever we write nested expectations, such as E E [•] for arbitrary random variables Y Y Z

and Z, the inner expectation is understood to be conditional on the outer variable; that is, EY [EZ [• | Y ]]. To simplify notation, we may omit the explicit conditioning when no confusion is likely to arise.

A Ūl,0 (π|S)

= Eθ∗ ∼Beta(α,β) Eπl − c(n0 ) + ρA + ε · (C + c(n0 )) · 1{f (α, β) · M1 ≥ 1/κ} ## ∗

θ , B0 = Beta(α, β), C0 = C "

= Eθ∗ ∼Beta(α,β) Eπl − c(n0 ) + ρA + ε · (C + c(n0 )) · 1{f (S) · M1 ≥ 1/κ} ## θ∗ , B0 = Beta(α, β), C0 = C " = Eθ∗ ∼Beta(α,β) En0 ∼π(S,l) EX0 ∼Bin(n0 ,θ∗ ) − c(n0 ) # A

+ ρ + ε · (C + c(n0 )) · 1{f (S) · M1 ≥ 1/κ} "

(∗)

= En0 ∼π(S,l) Eθ∗ ∼Beta(α,β) EX0 ∼Bin(n0 ,θ∗ ) − c(n0 ) # A

+ ρ + ε · (C + c(n0 )) · 1{f (S) · E(n0 , X0 ) ≥ 1/κ} "

= Enl ∼π(S,l) EXl ∼BB(nl ,α,β) − c(nl ) + ρA + ε · (C + c(nl )) · 1{f (S) · E(nl , Xl ) ≥ 1/κ} = Vπ (S, l, 0), where in (∗) we have used that θ∗ and n0 are independent (since n0 is sampled independently from π(S, l)). This concludes the base case. Inductive step t → t + 1. For the inductive step, assume that given a t < T , for any l ≤ T − t and for any state S ∈ S it holds that: A Ūl,t (π | S) = Vπ (S, l, t).

A We will show that Ūl,t+1 (π|S) = Vπ (S, l, t + 1) for any l ≤ T − (t + 1) and S ∈ S. To this end, fix any such S and l. Firstly, we consider the case where S = S out or f (S) ≥ 1/κ, where the transition dynamics of the MDP (Eq. 8) imply that at any time step k ≥ l, Sk = S out or Sk = S. Then, similarly to the base case, Vπ (S, l, t) = 0 A and Ūl,t (π|S) = 0 for any t ≥ 0, so the equality holds. We focus now on the non-trivial case where S ̸= S out and f (S) < 1/κ, and write S = (α, β, C). We begin

by expanding the value for the first t + 1 time steps in Eq. 41: # "l+t+1 X Vπ (S, l, t + 1) = Eπ r(Sk , nk , Sk+1 ) Sl = S k=l

" = Eπ r(Sl , nl , Sl+1 ) +

l+t+1 X

# r(Sk , nk , Sk+1 ) Sl = S

k=l+1

= Eπ [r(Sl , nl , Sl+1 ) + Vπ (Sl+1 , l + 1, t) | Sl = S] (∗) A = Eπ r(Sl , nl , Sl+1 ) + Ūl+1,t (π|Sl+1 ) Sl = S A (∗∗) = Eπ [r(Sl , nl , Sl+1 ) | Sl = S] + Eπ Ūl+1,t (π|Sl+1 ) · 1{f (Sl+1 ) < 1/κ} Sl = S {z } | | {z } †

(44)

‡

where (∗) follows from the induction hypothesis and (∗∗) because ŪtA (π|Sl+1 ) = 0 if f (Sl+1 ) ≥ 1/κ. Analogously to the base case, the term † above can be written as: † = Eπ [r(Sl , nl , Sl+1 )|Sl = S] = Eπ −c(nl ) + (ρA + ε · (C + c(nl ))) · 1{f (Sl+1 ) ≥ 1/κ} Sl = S h = Enl ∼π(S,l) Eθ∗ ∼Beta(α,β) EXl ∼Bin(nl ,θ∗ ) − c(nl ) i + (ρA + ε · (C + c(nl ))) · 1{f (Sl+1 ) ≥ 1/κ} h = Eθ∗ ∼Beta(α,β) Enl ∼π(S,l) EXl ∼Bin(nl ,θ∗ ) − c(nl ) i + (ρA + ε · (C + c(nl ))) · 1{f (α + Xl , β + nl − Xl ) ≥ 1/κ} h = Eθ∗ ∼Beta(α,β) En0 ∼πl (S,0) EX0 ∼Bin(n0 ,θ∗ ) − c(n0 ) i + (ρA + ε · (C + c(n0 ))) · 1{f (α + X0 , β + n0 − X0 ) ≥ 1/κ} h = Eθ∗ ∼Beta(α,β) En0 ∼πl (S,0) EX0 ∼Bin(n0 ,θ∗ ) − c(n0 ) i + (ρA + ε · (C + c(n0 ))) · 1{f (α, β) · E(n0 , X0 ) ≥ 1/κ} h h = Eθ∗ ∼Beta(α,β) Eπl − c(n0 ) + ρA + ε · (C + c(n0 )) · 1{f (α, β) · M1 ≥ 1/κ} ii θ∗ , B0 = Beta(α, β), C0 = C

(45)

We now focus on the term ‡ in Eq. 44. Given Sl = S, the state Sl+1 is fully determined by the action nl and the value Xl , namely (see Eq. 8), ( (α + Xl , β + nl − Xl , C + c(nl )) if nl > 0 Sl+1 = S out if nl = 0. A Thus, conditional on the policy π(S, l) selecting action nl = 0, we have Sl+1 = S out and Ūl+1,t (π|Sl+1 ) = 0. That is, conditioning on the the event {nl = 0} inside the expectations † and ‡ results in: ( Eπ [r(Sl , nl , Sl+1 ) | Sl = S, nl = 0] = 0 A Eπ Ūl+1,t (π|Sl+1 ) · 1{f (Sl+1 ) < 1/κ} Sl = S, nl = 0 = 0.

Similarly, note that conditional on the policy π(α, β, C, l) selecting action nl = 0, we also have: "





τ (S,l,t+1))

Eθ∗ ∼Beta(α,β) Eπl ρA + ε · C +

c(nj )

j=0

· 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}   ## τ (S,l,t+1) X ∗ − c(nj ) θ , B0 = Beta(α, β), C0 = C, n0 = 0 = 0 j=0

As a consequence, in the following, we assume without loss of generality that the action 0 is not in the support of the distribution π(S, l), and therefore Sl+1 ̸= S out . Then, under the above simplification, we expand the expectation in the term ‡ to average over the A possible values of nl and Xl , and then substitute the expression for the utility Ūl+1,t (π|Sl+1 ) defined in Eq. 42 17 (emphasized below in blue for clarity): A ‡ = Eπ Ūl+1,t (π|Sl+1 ) · 1{f (Sl+1 ) < 1/κ} Sl = S

= En′l ∼π(S,l) EXl′ ∼BB(n′l ,α,β) Eθ∗ ∼Beta(α+Xl′ ,β+n′l −Xl′ ) Eπl+1 1{f (α + Xl′ , β + n′l − Xl′ ) < 1/κ}· 



ρA + ε · C + c(n′l ) +

τ̃ X j=0

 c(nj ) · 1{∃j ∈ [τ̃ ]: f (α + Xl′ , β + n′l − Xl′ ) · Mj+1 ≥ 1/κ}

  τ̃ X − 1{f (α + Xl′ , β + n′l − Xl′ ) < 1/κ} ·  c(nj ) j=0

## ∗

θ , B0 = Beta(α + Xl′ , β + n′l − Xl′ ), C0 = C + c(n′l )

where we have defined τ̃ = τ (S(n′l , Xl′ ), l +1, t), and we use S(n′l , Xl′ ) to denote the state to which S transitions after selecting action n′l and observing the outcome Xl′ , as given by the transition dynamics (Eq. 8). 17We also change notation n → n′ and X → X ′ to avoid confusion with the actions and outcomes that appear when expanding l l l l A Ūl+1,t (π|Sl+1 ).

We now leverage Lemma 1 to reorder the first three expectations as: " " ‡ = Eθ∗ ∼Beta(α,β) En′l ∼π(S,l) EXl′ ∼Bin(n′l ,θ∗ ) Eπl+1 1{f (α + Xl′ , β + n′l − Xl′ ) < 1/κ}· 



ρA + ε · C + c(n′l ) +

τ̃ X j=0

 c(nj ) · 1{∃j ∈ [τ̃ ]: f (α + Xl′ , β + n′l − Xl′ ) · Mj+1 ≥ 1/κ}

  τ̃ X − 1{f (α + Xl′ , β + n′l − Xl′ ) < 1/κ} ·  c(nj ) j=0

## ∗

θ , B0 = Beta(α + Xl′ , β + n′l − Xl′ ), C0 = C + c(n′l ) "

= Eθ∗ ∼Beta(α,β) En0 ∼π(S,l) EX0 ∼Bin(n0 ,θ∗ ) 1{f (α + X0 , β + n0 − X0 ) < 1/κ} · Eπl+1 



ρA + ε · C + c(n0 ) +  −

τ̃ X j=0

 c(ñj ) · 1{∃j ∈ [τ̃ ]: f (α + X0 , β + n0 − X0 ) · M̃j+1 ≥ 1/κ}

# c(ñj )

# ∗

θ , B0 = Beta(α + X0 , β + n0 − X0 ), C0 = C + c(n0 ) , (46)

where, in the last equality, we have renamed the dummy variables appearing in the inner expectation using tildes, i.e., ñj , X̃j , and M̃j .18 This is purely notational at this stage, but will be useful later in the proof when it becomes important to distinguish between different sets of variables. To keep our objective in view, recall that the goal of the inductive step is to show that A Ūl,t+1 (π | S) = † + ‡. A Our goal will now be to expand the term Ūl,t+1 (π | S) and verify that this equality indeed holds. To this end, we will use the identity

1 = 1{f (α, β) · M1 ≥ 1/κ} + 1{f (α, β) · M1 < 1/κ}, which holds for any α, β and M1 since the two events in the indicator functions are complementary, and A substitute it in the definition of Ūl,t+1 (π | S) in Eq. 42: d d d 18 Here, note that τ̃ (S(n , X ), l + 1, t) = τ̃ (S(n′l , Xl′ ), l + 1, t), because n0 = n′l and X0 = Xl′ by definition, and so we kept the 0 0 notation τ̃ to denote τ (S(n0 , X0 ), l + 1, t)

A Ūl,t+1 (π|S)

"





τ (S,l,t+1)

= Eθ∗ ∼Beta(α,β) Eπl ρ + ε · C +

c(nj )

j=0

· 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}   ## τ (S,l,t+1) X ∗ − c(nj ) θ , B0 = Beta(α, β), C0 = C j=0

"





τ (S,l,t+1)

= Eθ∗ ∼Beta(α,β) Eπl ρA + ε · C +

c(nj )

j=0

· 1{f (α, β) · M1 ≥ 1/κ} · 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}   ## τ (S,l,t+1) X ∗ c(nj ) θ , B0 = Beta(α, β), C0 = C − 1{f (α, β) · M1 ≥ 1/κ} ·  j=0

"





τ (S,l,t+1)

+ Eθ∗ ∼Beta(α,β) Eπl ρA + ε · C +

c(nj )

j=0

· 1{f (α, β) · M1 < 1/κ} · 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}   ## τ (S,l,t+1) X ∗ − 1{f (α, β) · M1 < 1/κ} ·  c(nj ) θ , B0 = Beta(α, β), C0 = C j=0

We now observe that if f (α, β) · M1 ≥ 1/κ, then τ (S, l, t + 1) = 0, that is, the approval process stops after the first step, which simplifies the first summand and yields: A Ūl,t+1 (π|S)

= Eθ∗ ∼Beta(α,β) Eπl − c(n0 ) + ρA + ε · (C + c(n0 )) · 1{f (α, β) · M1 ≥ 1/κ} ## ∗

θ , B0 = Beta(α, β), C0 = C "

"



τ (S,l,t+1)

+ Eθ∗ ∼Beta(α,β) Eπl ρA + ε · C +

 c(nj )

j=0

· 1{f (α, β) · M1 < 1/κ} · 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}   ## τ (S,l,t+1) X ∗ − 1{f (α, β) · M1 < 1/κ} ·  c(nj ) θ , B0 = Beta(α, β), C0 = C j=1

Further, we can identify that the first summand in the above expression corresponds to the form of the term † derived in Eq. 45, and replacing it, we obtain: 36

A Ūl,t+1 (π|S)

=† "

"



τ (S,l,t+1)

+Eθ∗ ∼Beta(α,β) Eπl ρ + ε · C +

 c(nj )

j=0

·1{f (α, β) · M1 < 1/κ} · 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}   ## τ (S,l,t+1) X ∗ −1{f (α, β) · M1 < 1/κ} ·  c(nj ) θ , B0 = Beta(α, β), C0 = C j=1

We now focus on the orange term in the expression above, which we denote by □. Our goal will be to show that □ = ‡. We start by factoring out the term 1{f (α, β) · M1 < 1/κ}, and using that by definition M1 = E(n0 , M0 ), we obtain: "

□ = Eθ∗ ∼Beta(α,β) Eπl 1{f (α, β) · E(n0 , X0 ) < 1/κ} · 



τ (S,l,t+1)

ρA + ε · C + c(n0 ) + τ (S,l,t+1)

−

X j=1

 c(nj ) 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ}

! c(nj )

## ∗

θ , B0 = Beta(α, β), C0 = C

j=1

In the above expectation Eπl [•|θ∗ , B0 = Beta(α, β), C0 = C], we can use the law of iterated expectations by conditioning on the first samples n0 and X0 . In particular, note that n0 ∼ π(α, β, C, l) and that

X0 ∼ Bin(n0 , θ∗ ), and thus: "

□ = Eθ∗ ∼Beta(α,β) En0 ∼π(S,l) EX0 ∼Bin(n0 ,θ∗ ) Eπl 1{f (α, β) · E(n0 , X0 ) < 1/κ} · 



τ (S,l,t+1)

ρA + ε · C + c(n0 ) + τ (S,l,t+1)

−

X j=1

 c(nj ) 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ} ##

! c(nj )

∗

θ , B0 = Beta(α, β), C0 = C, n0 , X0

j=1

= Eθ∗ ∼Beta(α,β) En0 ∼π(S,l) EX0 ∼Bin(n0 ,θ∗ ) 1{f (α + X0 , β + n0 + X0 ) < 1/κ} · Eπl 



τ (S,l,t+1)

ρA + ε · C + c(n0 ) + τ (S,l,t+1)

−

X j=1

 c(nj ) 1{∃j ∈ [τ (S, l, t + 1)]: f (α, β) · Mj+1 ≥ 1/κ} #

# c(nj ) θ∗ , B0 = Beta(α, β), C0 = C, n0 , X0

j=1

(47) Here, note that the inner expectation Eπl [•|θ∗ , B0 = Beta(α, β), C0 = C, n0 , X0 ] is conditioned on n0 and X0 . Therefore, the first action n1 is distributed according to π(α + X0 , β + n0 − X0 , C + c(n0 ), l + 1), and any subsequent action nj is distributed according to:  n1 |n0 , X0 ∼ π (α + X0 , β + n0 − X0 , C + c(n0 ), l + 1)      X1 |n0 , X0 ∼ Bin(n1 , θ∗ )       n2 |n0 , X0 ∼ π (α + X0 + X1 , β + n0 + n1 − X0 − X1 , C + c(n0 ) + c(n1 ), l + 2)      X2 |n0 , X0 ∼ Bin(n2 , θ∗ )     .. .     nj |n0 , X0 ∼ π(α + X0 + · · · + Xj−1 , β + n0 + · · · + nj−1 − X0 − · · · − Xj−1 ,      C + c(n0 ) + · · · + c(nj−1 ), l + j)      Xj |n0 , X0 ∼ Bin(nj , θ∗ )       .. . These are equal in distribution to the sequence of actions and experimental outcomes ñ1 , X̃1 , . . . in the inner

expectation of Eq. 46, i.e., we have the following equalities in distribution  d  n1 |n0 , X0 = ñ0      d    X1 |n0 , X0 = X̃0  d    M1 |n0 , X0 = E(n0 , X0 ) · M̃0   d       d E(n1 , X1 )|n0 , X0 = E(ñ0 , X̃0 )    M2 |n0 , X0 = E(n0 , X0 ) · M̃1     .  .    . .. =⇒ . . d   nj |n0 , X0 = ñj−1    d      Mj |n0 , X0 = E(n0 , X0 ) · M̃j−1 d       Xj |n0 , X0 = X̃j−1  .   ..   d   E(nj , Xj )|n0 , X0 = E(ñj−1 , X̃j−1 )       .. .

(48)

Based on the above, we can also conclude the following about the stopping times under the event {f (α, β) · E(n0 , X0 ) < 1/κ}: d

τ (S, l, t + 1)|n0 , X0 = (t + 1) ∧ min{j ∈ {1, . . . , t + 1}: nj = 0 or f (α, β) · Mj+1 ≥ 1/κ} d = (t + 1) ∧ 1 + min{j ∈ {0, . . . , t}: ñj = 0 or f (α, β) · E(n0 , X0 ) · M̃j+1 ≥ 1/κ} d

(49)

= 1 + τ (S(n0 , X0 ), l + 1, t) | {z } τ̃

As a consequence, in Eq. 47, we can change the summation index j → j − 1 and use the equalities in distribution in Eq. 48 and Eq. 49 to finally conclude: "

□ = Eθ∗ ∼Beta(α,β) En0 ∼π(S,l) EX0 ∼Bin(n0 ,θ∗ ) 1{f (α + X0 , β + n0 + X0 ) < 1/κ} · Eπl+1 



ρA + ε · C + c(n0 ) +

τ̃ X j=0

 c(ñj ) · 1{∃j ∈ [τ̃ ]: f (α + X0 , β + n0 − X0 ) · M̃j+1 ≥ 1/κ}

  ## τ̃ X c(ñj ) θ∗ , B0 = Beta(α + X0 , β + n0 − X0 ), C0 = C + c(n0 ) − j=0

=‡ A That is, we have shown that Ūl,t+1 (π|S) = † + ‡ = Vπ (S, l, t + 1). This concludes the induction step and the proof.

F.5

Proof of Proposition 4

Consider the state space S of the MDP Mε . We define the set S r of reachable states from the initial state (α0 , β0 , 0) to be any state that can be reached with non-negative probability by a policy π. More formally, S ∈ S r if an only if S = S out or S = (α, β, C) and there exist a 0 ≤ t ≤ T and a sequence of actions and outcomes n0 , X0 , . . . nt , Xt such that  0 ≤ Xk ≤ nk , for k = 0, . . . , t     t  X    Xk α = α +  0    k=0   t X (50)  β = β + (nk − Xk ) 0     k=0    t  X     c(nt ). C = k=0

In words, the sequence (n0 , X0 , . . . nt , Xt ) allows the initial state to eventually transition to the state S according to the transition dynamics (Eq. 8). Observe that then, any state visited under any realization of the MDP for any policy π is contained in the set S r . Then, S r is finite because, for a fixed t, any sequence (n0 , X0 , . . . nt , Xt ) can only take finitely many values (since |A|= nmax + 1), and the MDP has only finitely many steps, namely, T + 1. We focus now on the case where the cost function c is linear. P Let c(n) = c0 + Pct1 · n, and consider any t state that can be reached at time step t with a total number N = k=0 nt , X = k=0 Xt . Then, t X k=0

c(nt ) = (t + 1) · c0 + N · c1 .

In particular, in light of Eq. 50, the states in S r (except S out ) are in a bijection with the triplets (t, N, X). This is because α and β are uniquely determined by X and N , and if c0 , c1 ̸= 0, then t and N uniquely determine C. Next, observe that if the policy never opts out, given a time step 0 ≤ t ≤ T , the minimum value that N can take is N = t + 1, which corresponds to a sequence of actions n0 = 1, . . . , nt = 1. On the other hand, the maximum value that N can take is N = (t + 1) · nmax , which corresponds to a sequence of actions n0 = nmax , . . . , nt = nmax . Further, for a fixed value of N , the total positive outcomes X can take exactly N + 1 values, i.e., X ∈ {0, . . . , N }. Thus, r

|S | = |{z} 1 + S out

T (t+1)·n X X

max

(N + 1)

t=0 N =t+1

=1+

T X (t + 1) · nmax − t

=1+

T X (nmax )2 − 1

t=0

t=0 max 2

((t + 2) + ((t + 1) · nmax + 1))

3nmax − 1 · (t + 1) + · (t + 1) + 1 2 2

) −1 3nmax − 1 · (T + 1)(T + 2)(2T + 3) + · (T + 1)(T + 2) + (T + 1) 12 4 max 2 3 = O (n ) · T PT where we have used standard summation formulas for the arithmetic progression t=0 (t+1) and the quadratic PT progression t=0 (t + 1)2 . In particular, the above also shows that the set of reachable states in t time steps, which we denote by S r (t), satisfies |S r (t)|= O (nmax )2 · t2 . =1+

F.6

Auxiliary Lemma for Proposition 5

Lemma 2. Consider a state S ∈ S such that S ̸= S out and f (S) < 1/κ, any policy π and 0 ≤ l ≤ T . Write S = (α, β, C). Then, " T # X ε Eπ r (St , nt , St+1 ) Sl = S ≤ ρA + ε · C. (51) t=l

Note that as an immediate consequence of the above, it also holds for the optimal value function (Eq. 52): V ε (S, l) ≤ ρA + ε · C Proof. Fix any such S, l and policy π. Denote by τ the corresponding stopping time when Sl = S, i.e., τ = T ∧ min{t ∈ {l, . . . , T }: St = S out or f (St ) ≥ 1/κ | Sl = S}. Then, rε (St , nt , St+1 ) = 0 for t ≥ τ by definition of the rewards (Eq. 9). Thus, T X

rε (St , nt , St+1 ) =

t=l

τ −1 X

rε (St , nt , St+1 )

t=l

τ −1 X t=l

=−

−c(nt ) + (ρA + ε · (Ct + c(nt ))) · 1{f (St+1 ) ≥ 1/κ}

τ −1 X

c(nt ) +

t=l

τ −1 X t=l

(ρA + ε · (Ct + c(nt ))) · 1{f (St+1 ) ≥ 1/κ}.

Note that by definition of τ , the second term is at most non-null for the summand with t = τ − 1. Thus: T X t=l

rε (St , nt , St+1 ) ≤ − =− =− A

τ −1 X t=l τ −1 X t=l τ −1 X t=l

=ρ −

c(nt ) + (ρA + ε · (Cτ −1 + c(nτ −1 ))) c(nt ) + ρA + ε · (Cτ −1 + c(nτ −1 )) c(nt ) + ρA + ε · Cτ

τ −1 X t=l

c(nt ) + ε ·

C+

τ −1 X

! c(nt )

t=l

≤ ρA + ε · C, where in the last step we have used ε ≤ εmax ≤ 1. This concludes the proof.

F.7

Proof of Proposition 5

We fix any subsidy ε ∈ [0, εmax ]. We will use the Bellman optimality equation [51], which states that the optimal value function V ε for Mε , defined by:  " T #  X  ε  sup E r (St , nt , St+1 ) Sl = S if 0 ≤ l ≤ T π V ε (S, l) = π∈Π (52) t=l    0 if l = T + 1, satisfies the following recursive condition: ( ) X ε ′ ε ′ ε ′ V (S, l) = max P (S |S, n) · (r (S, n, S ) + V (S , l + 1)) , n∈A

S ′ ∈S

where P are the transition dynamics of the MDP, defined implicitly in Eq. 8. We can now particularize to the action space A in Mε . Since the action n = 0 results in a null reward (see Eq. 9), we obtain for l ≤ T and any state S = (α, β, C) such that f (S) < 1/κ (and S ̸= S out ): V ε (S, l) = max 0, max max EX∼BB(n,α,β) [rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)] (53) n∈{1,...,n

}

while V ε (S, l) = 0 if f (S) ≥ 1/κ or S = S out . In the above, to simplify notation, we denote by S(n, X) the state to which the MDP transitions from S after selecting n > 0 and observing outcome X, i.e., S(n, X) = (α + X, β + n − X, C + c(n)). F.7.1

Monotonicity on the belief

We focus on proving the monotonicity property in Proposition 5 of V ε in the parameter α.19 More precisely, fix a state S = (α, β, C) such that f (α, β) < 1/κ and α′ ≥ α such that f (α′ , β) < 1/κ. Denote S ′ = (α′ , β, C). We want to show that V ε (α, β, C, l) ≤ V ε (α′ , β, C, l) for any 0 ≤ l ≤ T . We proceed by induction over l. Base case l = T . Using Eq. 53 we obtain: 19 The monotonicity in the parameter β follows using a completely symmetric argument.

V ε (α′ , β, C, T ) ε ′ ′ = max 0, max EX∼BB(n,α′ ,β) [r (S , n, S (n, X))] n∈{1,...,nmax } h = max 0, max max EX∼BB(n,α′ ,β) − c(n) n∈{1,...,n } i A ′ + (ρ + ε · (C + c(n))) · 1{f (α , β) · E(n, X) ≥ 1/κ} h (∗) ≥ max 0, max max EX∼BB(n,α′ ,β) − c(n) n∈{1,...,n } i + (ρA + ε · (C + c(n))) · 1{f (α, β) · E(n, X) ≥ 1/κ} h (∗∗) ≥ max 0, max max EX∼BB(n,α,β) − c(n) n∈{1,...,n } i + (ρA + ε · (C + c(n))) · 1{f (α, β) · E(n, X) ≥ 1/κ} = V ε (α, β, C, T ) where in (∗) we have used that the function f (•, •) defined in Eq. 3 is non-decreasing in its first component20 . In step (∗∗) we have used that the distribution BB(α′ , β, n) stochastically dominates (in the first-order sense) BB(α, β, n) if α′ ≥ α. Induction step l + 1 → l. Suppose now that V ε (α, β, C, l + 1) ≤ V ε (α′ , β, C, l + 1) holds for a certain l + 1 ≤ T and for all α, β, α′ such that α′ ≥ α, f (α, β) < 1/κ and f (α′ , β) < 1/κ. Using Eq. 53 again, note that:       ε ′ ε ′ ′ ε ′ V (α , β, C, l) = max 0, max max EX∼BB(n,α′ ,β) [r (S , n, S (n, X)) + V (S (n, X), l + 1)]  {z }  n∈{1,...,n } |  †

Consider a given n > 0 and expand the term † above: 20 This is immediate to verify since θ b ∈ (0, 1)

=⇒ log(1 + θb (e − 1)) ∈ (0, 1).

† = EX∼BB(n,α′ ,β) [rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)] h = EX∼BB(n,α′ ,β) 1{f (S ′ (n, X)) ≥ 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)) i + 1{f (S ′ (n, X)) < 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)) h (∗) ≥ EX∼BB(n,α′ ,β) 1{f (S ′ (n, X)) ≥ 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)) i + 1{f (S ′ (n, X)) < 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S(n, X), l + 1)) h (∗∗) = EX∼BB(n,α′ ,β) 1{f (S ′ (n, X)) ≥ 1/κ, f (S(n, X)) ≥ 1/κ} · (rε (S ′ , n, S ′ (n, X)) + V ε (S(n, X), l + 1))

+ 1{f (S ′ (n, X)) ≥ 1/κ, f (S(n, X)) < 1/κ}

· (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1))

+ 1{f (S ′ (n, X)) < 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S(n, X), l + 1))

where in (∗) we have used that if f (S ′ (n, X)) < 1/κ, then f (S(n, X)) < 1/κ and hence the induction hypothesis applies V ε (S ′ (n, X), l + 1) ≥ V ε (S(n, X), l + 1). In (∗∗) we have used that if f (S ′ (n, X)) ≥ 1/κ and f (S(n, X)) ≥ 1/κ, then V ε (S(n, X), l + 1) = V ε (S ′ (n, X), l + 1) = 0. Lastly, in †, consider the case f (S ′ (n, X)) ≥ 1/κ and f (S(n, X)) < 1/κ. Then, rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1) = −c(n) + (ρA + ε · (C + c(n)))

and,

⋄

rε (S, n, S(n, X)) + V ε (S(n, X), l + 1) = −c(n) + V ε (S(n, X), l + 1) ≤ −c(n) + ρA + ε · (C + c(n))),

where ⋄ follows from Lemma 2. Finally, noting that rε (S ′ , n, S ′ (n, X)) ≥ rε (S, n, S(n, X)) and using the first-order stochastic dominance for the Beta-Binomial again, we conclude that: h † ≥ EX∼BB(n,α′ ,β) 1{f (S ′ (n, X)) ≥ 1/κ, f (S(n, X)) ≥ 1/κ} · (rε (S, n, S(n, X)) + V ε (S(n, X), l + 1))

+ 1{f (S ′ (n, X)) ≥ 1/κ, f (S(n, X)) < 1/κ} · (rε (S, n, S(n, X)) + V ε (S(n, X), l + 1))

+ 1{f (S ′ (n, X)) < 1/κ}

· (rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)) That is,

≥ EX∼BB(n,α,β) [rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)].

EX∼BB(n,α′ ,β) [rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)] ≥

EX∼BB(n,α,β) [rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)] , (54)

and taking maximum over the action n and using Eq. 53: V ε (α′ , β, C, l) ≥ V ε (α, β, C, l).

This concludes the induction step and thus the proof of the monotonicity in the belief. 44

F.7.2

Monotonicity on the cost

We now focus on proving the monotonicity property in Proposition 5 of V ε in the cumulated cost C. More precisely, fix a state S = (α, β, C) such that f (α, β) < 1/κ and consider C ′ ≥ C. Denote S ′ = (α, β, C ′ ). We want to show that V ε (α, β, C, l) ≤ V ε (α, β, C ′ , l) for any 0 ≤ l ≤ T . Again, we proceed by induction over l. Base l = T . Using Eq. 53 we obtain:

′

V (α, β, C , T ) = max 0, max EX∼BB(n,α,β) [r (S , n, S (n, X))] n∈{1,...,nmax } h = max 0, max max EX∼BB(n,α,β) − c(n) }

n∈{1,...,n

′

+ (ρ + ε(C + c(n))) · 1{f (α, β) · E(n, X) ≥ 1/κ} h ≥ max 0, max max EX∼BB(n,α,β) − c(n)

}

n∈{1,...,n

+ (ρ + ε(C + c(n))) · 1{f (α, β) · E(n, X) ≥ 1/κ}

= V ε (α, β, C, T ). Induction step l + 1 → l. Suppose now that V ε (α, β, C, l + 1) ≤ V ε (α, β, C ′ , l + 1) holds for a certain l + 1 ≤ T and for all α, β such that f (α, β) < 1/κ and 0 ≤ C ≤ C ′ . Using Eq. 53 again, note that:       ε ′ ε ′ ′ ε ′ V (α, β, C , l) = max 0, max max EX∼BB(n,α,β) [r (S , n, S (n, X)) + V (S (n, X), l + 1)]  {z }  n∈{1,...,n } |  †

Consider a given n > 0 and expand the term † above: † = EX∼BB(n,α,β) [rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)] h = EX∼BB(n,α,β) 1{f (S ′ (n, X)) ≥ 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)) i + 1{f (S ′ (n, X)) < 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)) h (∗) ≥ EX∼BB(n,α,β) 1{f (S ′ (n, X)) ≥ 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)) i + 1{f (S ′ (n, X)) < 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S(n, X), l + 1)) h (∗∗) = EX∼BB(n,α,β) 1{f (S ′ (n, X)) ≥ 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S(n, X), l + 1)) i + 1{f (S ′ (n, X)) < 1/κ} (rε (S ′ , n, S ′ (n, X)) + V ε (S(n, X), l + 1))

where in (∗) we have used that if f (S ′ (n, X)) < 1/κ, then the induction hypothesis applies V ε (S ′ (n, X), l+1) ≥ V ε (S(n, X), l + 1) because the total cost in state S ′ (n, X) is C ′ + c(n) ≥ C + c(n), which equals the total 45

cost in S(n, X). In (∗∗) we have used that if f (S ′ (n, X)) ≥ 1/κ, then f (S(n, X)) ≥ 1/κ and hence V ε (S ′ (n, X), l + 1) = V ε (S(n, X), l + 1) = 0. Finally, noting that rε (S ′ , n, S ′ (n, X)) ≥ rε (S, n, S(n, X)), we conclude that: h † ≥ EX∼BB(n,α,β) 1{f (S ′ (n, X)) ≥ 1/κ} (rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)) i + 1{f (S ′ (n, X)) < 1/κ} (rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)) = EX∼BB(n,α,β) [rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)] That is, EX∼BB(n,α,β) [rε (S ′ , n, S ′ (n, X)) + V ε (S ′ (n, X), l + 1)] ≥ EX∼BB(n,α,β) [rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)] , and taking maximum over the action n and using Eq. 53: V ε (α, β, C, l) ≥ V ε (α, β, C ′ , l). This concludes the induction step and thus the proof of the monotonicity in the cost.

F.8

Proof of Proposition 6

Fix any subsidy level ε ∈ [0, εmax ], total cost C ≥ 0, and an initial time step 0 ≤ l ≤ T . Consider any state S ̸= S out and write S = (α, β, C). Then, the Bellman optimality condition [51] establishes that the action n = 0 is optimal if and only if the value function coincides with the value of action n = 0 (which is 0 by definition of the reward function in Eq. 9), that is, V ε (S, l) = 0, where V ε (S, l) is the optimal value function defined in Eq. 52. Using Eq. 53, this is equivalent to the condition:

max

    

    

0, max EX∼BB(n,α,β) [rε (S, n, S(n, X)) + V ε (S(n, X), l + 1)] = 0   n∈{1,...,nmax }    {z }  |  =H(α,β,C)

⇐⇒ H(α, β, C) ≤ 0. Now, we observe that in proving Proposition 5 in Appendix F.7 (see Eq. 54), we precisely showed by induction that the function H(α, β, C) is non-decreasing in α, and an analogous argument shows that it is non-increasing in β. We leverage this property in what follows. For any α > 0, define the quantity β̃(α) = sup{β : H(α, β, C) > 0 and β > 0}. As a consequence of the monotonicity of H(α, β, C), if α′ ≥ α, then H(α, β, C) > 0 =⇒ H(α′ , β, C) > 0, and thus the following set inclusion holds: {β : H(α, β, C) > 0 and β > 0} ⊆ {β : H(α′ , β, C) > 0 and β > 0}. Taking the supremum, the above implies: sup{β : H(α, β, C) > 0 and β > 0} ≤ sup{β : H(α′ , β, C) > 0 and β > 0}, | {z } | {z } β̃(α′ )

β̃(α)

which shows that β̃(α) is non-decreasing in α. Next, observe that: • If β > β̃(α), then by definition of the supremum we have that H(α, β, C) ≤ 0, and thus the action n = 0 is optimal. • If β < β̃(α), then again by the definition of the supremum, there exists β ′ such that β < β ′ ≤ β̃(α) with H(α, β ′ , C) > 0. Then, using the monotonicity, H(α, β, C) ≥ H(α, β ′ , C) > 0, which means that opting out is strictly sub-optimal: there exists an action n > 0 that leads to a strictly higher expected value if taken at state S and time l. This proves the claim in Proposition 6.

F.9

Proof of Proposition 7

Recall that the value function in the MDP Mε is defined by: " T # X ε ε Vπ (S, l) = Eπ r (St , nt , St+1 ) Sl = S ,

(55)

t=l

with rε defined by: ε

′

r (S, n, S ) =

( −c(n) + (ρA + ε · (C + c(n))) · 1{f (S ′ ) ≥ 1/κ} if 0 < f (S) < 1/κ

if S = S out or f (S) ≥ 1/κ.

(56)

Firstly, note that all rewards in Eq. 55 become 0 as soon as St = S out or f (St ) ≥ 1/κ. This is because, from the transition dynamics in Eq. 8, any such state St is absorbing, i.e., St+1 = St if St = S out or f (St ) ≥ 1/κ, and satisfies rε (St , nt , St+1 ) = 0. Then, the terms in Eq. 55 that are non-zero correspond to time steps t ≤ τ , where τ is the stopping time defined by: τ = T ∧ min{t ∈ {l, . . . , T }: nt = 0 or f (St+1 ) ≥ 1/κ}.

For l ≤ t ≤ τ , the first case of the reward function in Eq. 56 applies: " τ # X ε A Vπ (S, l) = Eπ −c(nt ) + (ρ + εCt+1 )1{f (St+1 ) ≥ 1/κ} Sl = S t=l

= Eπ

" τ X t=l

+ε

(−c(nt ) + ρA 1{f (St+1 ) ≥ 1/κ}) τ X t=l

# Ct+1 1{f (St+1 ) ≥ 1/κ} Sl = S

Note that the first part of the expectation corresponds exactly to the value function under no subsidy, Vπ0 (S, l), and thus, " τ # X ε 0 Vπ (S, l) = Vπ (S, l) + ε · Eπ Ct+1 · 1{f (St+1 ) ≥ 1/κ} Sl = S t=l

Now, define: Aπ (S, l) = Eπ

" τ X t=l

# Ct+1 · 1{f (St+1 ) ≥ 1/κ} Sl = S

= Eπ [Cτ +1 · 1{f (Sτ +1 ) ≥ 1/κ} | Sl = S]

(57)

This term corresponds to the (expected) total cost incurred by the agent conditional on the product being approved, when the MDP starts from state S at time l.21 Indeed, by the definition of the stopping time τ , approval (i.e., f (St+1 ) ≥ 1/κ) can only happen at exactly t = τ . If the agent opts out or the horizon T is reached without approval, the indicator 1{f (St+1 ) ≥ 1/κ} is 0 for all t. We conclude that Vπε (S, l) = Vπ0 (S, l) + ε · Aπ (S, l). 21We use the term “conditional on approval” informally and for didactic purposes, since A

π (S, l) is the quantity that naturally appears in the value decomposition Vπε (S, l) = Vπ0 (S, l) + ε · Aπ (S, l), representing the expected subsidy paid by the principal. Strictly speaking, however, Aπ (S, l) is the expected cost weighted by the indicator of approval, rather than a conditional expectation in the measure-theoretic sense.

F.10

Proof of Proposition 8

Denote by Πr ⊂ Π the set of all deterministic policies that select action n = 0 at S out , any state such that f (S) ≥ 1/κ, or any state such that S ∈ / S r , and observe that Πr is finite. Recall from Proposition 7 that, for r any policy π ∈ Π , Vπε (S, l) = Vπ0 (S, l) + ε · Aπ (S, l). (58) In particular, at the initial state S0 = (α0 , β0 , 0) and time l = 0, we have

Ū A (π; ε) = Vπε (S0 , 0) = Vπ0 (S0 , 0) + ε · Aπ (S0 , 0).

(59)

V ε (S, l) = maxr {Vπ0 (S, l) + ε · Aπ (S, l)}

(60)

Since Π is finite, for every (S, l) ∈ S × [T ] the optimal value function r

π∈Π

is the point-wise maximum of finitely many affine functions in ε. It is well-known that such a point-wise maximum is convex, continuous, and piecewise linear [79]. Here, we particularize to our problem, with the goal of constructing a single partition of [0, εmax ] on which a single optimal policy (for Mε at every state S ∈ S r and time l ∈ [T ] simultaneously) is optimal in each interval of the partition. For any two distinct policies π, π ′ ∈ Πr and any (S, l) ∈ S r × [T ], consider the difference (Vπ0 (S, l) − Vπ0′ (S, l)) + ε(Aπ (S, l) − Aπ′ (S, l)).

If (Vπ0 (S, l), Aπ (S, l)) = (Vπ0′ (S, l), Aπ′ (S, l)) for all (S, l), then the two policies yield identical values for all ε and all (S, l), and we may break ties arbitrarily and retain only one of them. Similarly, if Vπ0 (S, l)+ε·Aπ (S, l) ≥ Vπ0′ (S, l) + ε · Aπ′ (S, l) for all ε ∈ [0, εmax ] and all (S, l), then policy π ′ can be removed without changing the optimal value function at any (S, l). Thus, without loss of generality, we restrict our attention to a subset Π̃r ⊂ Πr such that for any distinct π, π ′ ∈ Π̃r and any (S, l), the corresponding affine functions intersect exactly once in [0, εmax ], and each policy is optimal for some value of ε. Then, let K = {ε ∈ [0, εmax ] : ∃ π ̸= π ′ ∈ Π̃r , ∃ (S, l) ∈ S r × [T ]

such that Vπ0 (S, l) + ε · Aπ (S, l) = Vπ0′ (S, l) + ε · Aπ′ (S, l)}.

Since Π̃r , S r , and [T ] are all finite, and each quadruple (π, π ′ , S, l) contributes at most one point to K, the set K is finite. Ordering its elements and adding the endpoints if necessary, we obtain a partition 0 = ε0 < ε1 < · · · < εL = εmax .

By construction, no two affine functions ε 7→ Vπ0 (S, l)+ε·Aπ (S, l) intersect in any open interval (εi , εi+1 ) at any (S, l). Hence, for every (S, l), the ordering of {Vπ0 (S, l) + ε · Aπ (S, l)}π∈Π̃r is constant on each such interval. It follows that there exists a single policy πi ∈ Π̃r that is optimal for Mε at every (S, l) simultaneously, and such that V ε (S, l) = Vπ0i (S, l) + ε · Aπi (S, l) for all ε ∈ [εi , εi+1 ) and all (S, l). In particular, specializing to (S, l) = (S0 , 0),

Ū A (π ε ; ε) = Vπ0i (S0 , 0) + ε · Aπi (S0 , 0) for all ε ∈ [εi , εi+1 ).

This establishes the result for Ū A (π ε ; ε). Note that E(α0 ,β0 )∼Q [Ū A (π ε ; ε)] is then also convex and continuous because it is an average of convex continuous functions that are uniformly bounded (observe that by Lemma 2, we have 0 ≤ Ū A (π ε ; ε) ≤ ρA uniformly for any initial belief parameters (α0 , β0 ), where the lower bound follows from the fact that the agent can always opt out at no cost); continuity of the expectation then follows from the dominated convergence theorem. Lastly, in each interval [εi , εi+1 ), Ū A (π ε ; ε) is linear, which implies that E(α0 ,β0 )∼Q [Ū A (π ε ; ε)] is also linear: E(α0 ,β0 )∼Q [Ū A (π ε ; ε)] = E(α0 ,β0 )∼Q [Vπ0i (S0 , 0)] + ε · E(α0 ,β0 )∼Q [Aπi (S0 , 0)] for all ε ∈ [εi , εi+1 ).

This concludes the proof.

F.11

Proof of Proposition 9

Recall that the social utility Ū S (ε; π) is defined as (Eq. 13): S

Ū (ε; π) = Eπ,(α0 ,β0 )∼Q

" T X t=0

ρ − ε · Ct+1 · 1 {0 < f (St ) < 1/κ ≤ f (St+1 )}

S0 = (α0 , β0 , 0) .

Let τ be the stopping time defined by: τ = T ∧ min{t ∈ {0, . . . , T }: nt = 0 or f (St+1 ) ≥ 1/κ},

that is, the last time step before reaching an absorbing state—either S out , for which f (S out ) = 0, or any state S such that f (S) ≥ 1/κ. Then, using the linearity of the expectation: Ū S (ε; π) = Eπ,(α0 ,β0 )∼Q

" T X t=0

= ρ · Eπ,(α0 ,β0 )∼Q

# ρ − ε · Ct+1 · 1 {0 < f (St ) < 1/κ ≤ f (St+1 )} S0 = (α0 , β0 , 0)

" T X

− ε · Eπ,(α0 ,β0 )∼Q

# 1 {0 < f (St ) < 1/κ ≤ f (St+1 )} S0 = (α0 , β0 , 0)

t=0 " τ X t=0

# Ct+1 · 1 {0 < f (St ) < 1/κ ≤ f (St+1 )} S0 = (α0 , β0 , 0)

= ρS · E(α0 ,β0 )∼Q Eπ [1 {∃t ∈ [T ]: 0 < f (St ) < 1/κ ≤ f (St+1 )} | S0 = (α0 , β0 , 0)] − ε · E(α0 ,β0 )∼Q Eπ [Cτ +1 · 1 {1/κ ≤ f (Sτ +1 )} | S0 = (α0 , β0 , 0)]

= ρS · E(α0 ,β0 )∼Q [Pπ (∃t ∈ [T ]: 0 < f (St ) < 1/κ ≤ f (St+1 ) | S0 = (α0 , β0 , 0))] − ε · E(α0 ,β0 )∼Q Eπ [Cτ +1 · 1 {1/κ ≤ f (Sτ +1 )} | S0 = (α0 , β0 , 0)]

= ρS · E(α0 ,β0 )∼Q [Pπ (∃t ∈ [T ]: 1/κ ≤ f (St+1 ) | S0 = (α0 , β0 , 0))] − ε · E(α0 ,β0 )∼Q [Aπ (α0 , β0 , 0, 0)]

where we have used the definition of Aπ in Eq. 57, and the fact that the condition 0 < f (St ) < 1/κ ≤ f (St+1 ) can occur at most once at time step t = τ .

F.12

Proof of Proposition 10

In this section, we will show that Algorithm 1 recovers (in a finite number of steps) the partition of the interval [0, εmax ] given by Proposition 8, which we denote by P:22 P = {0 = ε0 < ε1 < · · · < εL = εmax },

where for each interval [εi , εi+1 ), there exists a (deterministic) policy πi that is optimal for any ε ∈ [εi , εi+1 ), that is (writing S0 = (α0 , β0 , 0)), (

Ū A (π ε ; ε) = Vπ0i (S0 , 0) + ε · Aπi (S0 , 0)

E(α0 ,β0 )∼Q [Ū

(π ; ε)] = E(α0 ,β0 )∼Q [Vπ0i (S0 , 0)] + ε · E(α0 ,β0 )∼Q [Aπi (S0 , 0)]

∀ε ∈ [εi , εi+1 )

∀ε ∈ [εi , εi+1 ).

Since the above partition does not depend on the belief Q of the principal (Proposition 8), we assume without loss of generality that the principal knows the agent’s initial belief Beta(α0 , β0 ). Consequently, the expectations E(α0 ,β0 )∼Q [•] simply correspond to evaluating the integrand at the true belief parameters (α0 , β0 ). We begin by proving the following lemma.23 Lemma 3. Let εL < εR be two subsidy levels with, respectively, optimal policies πL and πR , and value functions (evaluated at the initial state (α0 , β0 , 0) and initial time step l = 0, which we omit for notational convenience) VL0 + ε · AL and VR0 + ε · AR (see Proposition 7). The following holds: 1. If AL = AR , then Ū A (π ε ; ε) = VL0 + ε · AL for all ε ∈ [εL , εR ]. V 0 −V 0

2. If AL < AR , let εint = ALR −ARL . Then, Ū A (π ε ; ε) = max(VL0 + ε · AL , VR0 + ε · AR ) for all ε ∈ [εL , εR ] if and only if Ū A (π εint ; εint ) = VL0 + εint · AL . Proof. For part 1., if AL = AR , then since Ū A (π ε ; ε) is convex, its subgradient must be non-decreasing in ε. Thus, for any ε ∈ (εL , εR ), we must have AL ≤ Aπε ≤ AR , which implies Aπε = AL . By continuity and the fact that Ū A (π ε ; ε) is the point-wise maximum of affine functions, it follows that Vπ0ε = VL0 = VR0 , and the value function is a single affine segment on this interval. For part 2., the direct implication follows because if Ū A (π ε ; ε) = max(VL0 + ε · AL , VR0 + ε · AR ) for all ε ∈ [εL , εR ], then evaluating at εint , and since VL0 + εint · AL = VR0 + εint · AR by definition, we obtain Ū A (π εint ; εint ) = VL0 + εint · AL . For the backward direction, we argue by contradiction and suppose Ū A (π εint ; εint ) = VL0 + εint · AL but there exists some ε′ ∈ (εL , εR ) and a policy π ′ such that Vπ0′ + ε′ · Aπ′ > max(VL0 + ε′ · AL , VR0 + ε′ · AR ). Assume without loss of generality ε′ ≤ εint (the argument for the case ε′ ≥ εint is symmetric, with πR in place of πL ). Since πL is optimal at εL , we have VL0 + εL · AL ≥ Vπ0′ + εL · Aπ′ . Combined with the assumption Vπ0′ + ε′ · Aπ′ > VL0 + ε′ · AL and subtracting, we obtain (ε′ − εL )(Aπ′ − AL ) > 0, which implies Aπ′ > AL since ε′ > εL . Then, at εint ≥ ε′ , Vπ0′ + εint · Aπ′ = (Vπ0′ + ε′ · Aπ′ ) + (εint − ε′ ) · Aπ′ > (VL0 + ε′ · AL ) + (εint − ε′ ) · AL = VL0 + εint · AL ,

where the strict inequality uses Vπ0′ + ε′ · Aπ′ > VL0 + ε′ · AL together with Aπ′ > AL and εint ≥ ε′ . This contradicts the assumption that Ū A (π εint ; εint ) = VL0 + εint · AL . To prove Proposition 10, we begin by showing that any point added to the set U in Algorithm 1 corresponds 0 to a point of P. Consider any iteration of the algorithm where Vint + εint · Aint ≤ VL0 + εint · AL and AL ̸= AR . Then, from Lemma 3, it follows that in this case, for any possible subsidy ε ∈ [εL , εR ], the optimal value function is given by Ū A (π ε ; ε) = max(VL0 + ε · AL , VR0 + ε · AR ), with a change in slope at their intersection, 22We denote by P(ε) the interval of P containing ε. 23 For convenience, we use L, R and int as subscripts instead of using π , π and π int . L R

i.e., at εint . That is, εint is a point of the partition P, and the value Ū S (εint ; πR ) = maxε∈P(εint ) Ū S (ε; π ε ) is stored in the set U . 0 Reciprocally, consider any iteration of the algorithm where Vint + εint · Aint > VL0 + εint · AL . Then, Lemma 3 implies that P(εL ) ̸= P(εint ) and P(εR ) ̸= P(εint ). That is, P(εint ) is a new interval in the partition P where the optimal value function is given by the linear component Vπ0int + ε · Aπint . The algorithm pushes the two sub-intervals [εL , εint ] and [εint , εR ] onto the stack (line 17), along with their respective linear components VL0 + ε · AL , Vπ0int + ε · Aπint and VR0 + ε · AR . Since by Proposition 8, there are a finite number of such linear components (or equivalently, P is finite), Algorithm 1 terminates in a finite number of steps. To conclude, we argue that, upon termination, the set U obtained from Algorithm 1 contains every point of P. Let N denote the total number of pop operations performed by the algorithm before I becomes empty (finite by the argument above). For k = 0, 1, . . . , N , denote by Ik the state of the stack after k pops, and by Dk ⊆ P the set of points of P found by the algorithm during the first k iterations, with D0 = {0, εmax }. Observe first that whenever Algorithm 1 pushes two new intervals onto I , the intersection point εint lies strictly interior to the interval P(εint ) (since by Lemma 3, P(εL ) ̸= P(εint ) ̸= P(εR ), and εL < εint < εR ), so εint ∈ / P. Consequently, every endpoint of an interval ever pushed onto Ik is either in Dk or not in P. In light of this, we prove by induction on k the following condition:24 [ P ⊆ Dk ∪ (εL , εR ). (61) [εL ,εR ]∈Ik

Base case k = 0. I0 = {[0, εmax ]} and D0 = {0, εmax }. Every ε ∈ P is either in {0, εmax } = D0 or in (0, εmax ), so the condition in Eq. 61 holds for k = 0. Inductive step k → k + 1. Suppose the condition in Eq. 61 holds and consider the (k + 1)-th pop of some [εL , εR ] ∈ Ik . 0 + εint · Aint ≤ VL0 + εint · AL , then εint is added to Dk+1 , and the popped interval is not replaced. • If Vint By Lemma 3, (εL , εR ) ∩ P ⊆ {εint } ⊆ Dk+1 . Hence any ε ∈ P previously covered by (εL , εR ) is now in Dk+1 . The endpoints εL and εR , if they belong to P, were endpoints of the popped interval and hence are already contained in Dk ⊆ Dk+1 . Thus, the condition in Eq. 61 holds for k + 1. 0 + εint · Aint > VL0 + εint · AL , then [εL , εR ] is replaced by [εL , εint ] and [εint , εR ], while Dk+1 = Dk . • If Vint If εint ∈ Dk , Eq. 61 holds for k +1, and if εint ∈ / P, we have that (εL , εR )∩P = [(εL , εint )∪(εint , εR )]∩P, and the condition in Eq. 61 also holds for k + 1.

Lastly, at the final iteration, IN = ∅ implies P ⊆ DN ; combined with DN ⊆ P by construction, DN = P. Since every ε ∈ DN has its social utility saved in U (at line 7 for the initial endpoints and line 15 for each new point of P found), U contains {(ε, Ū S (ε; π ε )) : ε ∈ P}. The maximizer of Ū S over [0, εmax ] is attained at some left endpoint εi ∈ P, and arg max(ε,u)∈U u returns it.

24With a slight abuse of notation, we identify the elements of I with intervals, meaning that if (ε , π , V̄ 0 , Ā , ε , π , V̄ 0 , Ā ) ∈ L L L R R R L R

I, we write that [εL , εR ] ∈ I.

Additional Experimental Details

The complete code used for our experiments, including the implementation of Algorithm 3 and Algorithm 1, is available as supplementary material. We will publicly release it with the final version of the paper. Hardware setup. Our experiments are executed on a compute server equipped with 2 × Intel Xeon Gold 5317 CPU, 1,024 GB main memory, and 2 × H100 NVIDIA GPU (80 GB, Hopper Architecture). In each experiment, a single Nvidia H100 GPU is used. Software setup. All experiments are implemented in Python 3.13.5 using PyTorch 2.1.1 and NumPy 2.4.4. Computations are performed on an NVIDIA GPU with CUDA 13.0 support. Runtime. For the setting used in Section 6, Algorithm 3 computes the optimal policy of the agent (for any given subsidy) in ∼ 2.4s, and Algorithm 1 computes the optimal subsidy in ∼ 271s, which involves solving 114 times a different belief MDP Mε . In Figure 4, we evaluate the runtime of Algorithm 1 across multiple configurations for the parameters T and nmax , which determine the size of the state and action space (Proposition 4). 250 6.5s

39.3s

239.4s

1043.9s

2402.1s

5875.7s

1.6s

10.6s

62.8s

271.8s

729.5s

1771.0s

2.8s

4.0s

12.6s

43.4s

123.0s

369.6s

5.4s

6.2s

9.0s

12.3s

19.6s

132

207

208

254

114

134

163

100

150

200

250

300

10 0

5.2s

100

150

nmax

5.3s

200

250

300

150 T

102

200 Runtime (s)

103

100

MDPs solved

nmax

Figure 4: Runtime of Algorithm 1. The figure shows, across multiple values of the maximum number of trials T and the maximum sample size per trial nmax , the runtime of Algorithm 1 (left panel) and the number of belief MDPs solved by the algorithm (right panel). All other parameters are fixed as specified in Tables 2 and 3. The experiments are run on an NVIDIA H100 GPU. Parameter details. Tables 2 and 3 report the values of all parameters required to specify the sequential approval protocol used in our fiducial setting (Section 6). Unless otherwise stated, all results are obtained using these values. When any parameter is varied (e.g., in Panel (b) of Figure 1 or in Appendix H), we explicitly indicate it. Table 2: Non-economic parameters T

nmax

θb

θ∗

εmax

(α0 , β0 )

200

0.5

0.05

0.65

0.9

(1,1)

δ(1,1)

Table 3: Economic parameters ρS $2000 M

ρA $240 M

c(n) = c0 + c1 n, n ̸= 0 c0

$48.9 M

$0.066 M

Implementation details. Our Python implementation of Algorithm 3 leverages the bijection between pairs 53

(α, β) and pairs (X, N ), where X is the total number of successes and N the total number of patients (the total sample size), as we detail in Eq. 34. Furthermore, since the cost function is linear, we also use the fact that the total cumulated cost C at a given state (X, N ) at time l can be written as C = l · c0 + N · c1 . As a result, our implementation does not explicitly keep track of the cumulated cost C, which significantly reduces the computational overhead. A similar idea is used in the proof of Proposition 4. Our implementation of Algorithm 1 computes the optimal subsidy ε∗ (and solves each belief MDP Mε ) exactly. Consequently, there is no associated uncertainty in ε∗ or in any of the quantities that we report computed using the belief MDP, including Ū A and Ū S . In contrast, the true quantities under the approval process using θ∗ described in Section 2—such as the true utilities U A and U S (Eq. 7), as well as the probability that the agent opts out during the approval process—are estimated using 100,000 − 200,000 Monte Carlo rollouts, and we report 95% confidence error bars computed via bootstrapping with 1000 resamples.

Additional Experimental Results

H.1

Additional results complementing Section 6

175

50 25

100

125

150

Optimal action n

200 21

200 Optimal value function ($M)

In this section, we provide complementary results to the approval process considered in Section 6, whose parameters are given in Table 2 and Table 3. ∗ In Figure 5, we show the optimal value function and policy for the MDP Mε , with ε∗ = 1.08, and in Figure 6 we show how the belief of the agent evolves in 300 realizations of the approval process, for different values of the true efficacy θ∗ .

α ∗

Figure 5: Optimal value function and policy in the belief MDP Mε for the optimal subsidy ∗ ε∗ = 0.108. The left panel shows the optimal value function in the belief MDP, V ε (α, β, C(α, β), 1), at time step l = 1, where the cost of each state is given by C(α, β) = 1 · c0 + (α + β − α0 − β0 ) · c1 (see Eq. 34). The right panel shows the optimal action n taken by the optimal policy at time step l = 1 for each belief, i.e., ∗ π ε (α, β, C(α, β), 1). The optimal action at time step l = 0 (not shown here) is n = 79.

100

150

100 α

(a) θ∗ = 0.3

150

200

100 α

(b) θ∗ = 0.65

150

200

100 α

150

200

Figure 6: Trajectories of the approval process. Each panel shows 300 realizations of the approval process for different true efficacies θ∗ of the antibiotic. Each blue segment corresponds to the agent conducting a new trial and updating its belief (see Figure 3 for an illustration of the geometry). In the left panel, the agent conducts a first trial and then always opts out; in the middle panel, the agent can conduct multiple trials, and in the right panel, the antibiotic is always approved after the first trial. In Figure 7, we show, for the initial action taken by the agent at time step l = 0 (and belief (α0 , β0 ) = (1, 1)), 55

∗

Expected reward of the action ($M)

the expected cumulative future reward in Mε for the optimal subsidy ε∗ = 0.108, defined for each n > 0 as PT ∗ ε Eπε∗ [ t=0 r (St , nt , St+1 )|S0 , n0 = n]. The sample size maximizing this curve is n = 79, which is the size of the first trial conducted by the agent. Notably, although the expected reward is unimodal, it exhibits small-scale oscillations. These oscillations are not numerical artifacts. In fact, their approximate period is given by 1/log(1 + θb (e − 1)). The reason is that the agent’s actions are discrete, whereas the function f defined in Proposition 2, which determines the approval condition, decreases by exactly log(1 + θb (e − 1)) whenever the agent selects a new action n (recall that αt+1 + βt+1 − αt − βt = nt ). Since this change is not an integer quantity, the expected reward only exhibits a small decrease after n increases by approximately 1/log(1 + θb (e − 1)), which gives rise to the observed oscillations. 35 30 25 20 15 10 5 0 1

101 121 Action, n

141

161

181

0.82

0.22 Probability of approval

Probability of opting out

Figure 7: Expected reward for each sample size. The figure show, for the initial action taken by the ∗ agent at time step l = 0 and state (α0 = 1, β0 = 1, 0), the total expected reward in the MDP Mε under the ∗ optimal subsidy ε∗ = 0.108 when the agent takes action n and then follows the optimal policy π ε .

0.20

0.18

0.81 0.80 0.79 0.78 0.77

0.16 0.0

0.1

0.2

0.3

0.4 0.5 Subsidy, ε

0.6

0.7

0.8

0.9

0.0

(a) Opting out before approval vs. subsidy

0.1

0.2

0.3

0.4 0.5 Subsidy, ε

0.6

0.7

0.8

0.9

(b) Approval probability vs subsidy

Figure 8: Opt out and approval probabilities. The figure shows, for an antibiotic with θ∗ = 0.65, the probability that the agent opts out of the approval process by selecting n = 0 before approval, as well as the probability that the antibiotic is ultimately approved. For each subsidy level, the agent follows the optimal policy. Note that, in principle, the agent may never opt out during the approval process; however, the antibiotic may still fail to be approved if the maximum number of trials is reached. In Figure 9 and Figure 10, we show, respectively, the utility of the agent and the social utility when the agent selects its optimal policy for each possible subsidy, both computed using the belief MDP, and the true 56

50 45 40 35 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

0.22 160 0.20 140 0.18 120

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

realized utilities when the antibiotic has efficacy θ∗ = 0.65 (Eq. 7).

0.16 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

760

755

750

745 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

0.22

1620 1600

0.20

1580 0.18

1560 1540

Probability of opting out

Figure 9: Agent utilities. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.16 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 10: Social utilities. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8). Lastly, in Figure 11 we show that the optimal social utility (that is, the social utility under the optimal subsidy) increases monotonically as the true efficacy θ∗ increases.

0.8

8000 0.7

6000 0.6

θ∗

∗ Optimal social utility, U S (ε∗ ; π ε ) ($M)

0.9

10000

4000 0.5

2000 0.4

0 0

5 10 15 20 25 30 35 40 Social-to-agent approval benefit ratio, ρS /ρA

0.3

Figure 11: Optimal social utility for different antibiotic efficacies. The figure shows how social utility ∗ U S (ε∗ ; π ε )—when the principal chooses the optimal subsidy ε∗ and the agent adopts the corresponding ∗ optimal policy π ε —varies as a function of the ratio ρS /ρA across different levels of efficacy θ∗ . The dashed ∗ line corresponds to the social utility Ū S (ε∗ ; π ε ) computed using the belief MDP, which does not depend on the true efficacy θ∗ .

H.2

Additional results using different parameters

In this section, we present further experimental results for the antibiotic approval process described in Section 6, where we vary selected parameters (see Tables 2 and 3). H.2.1

Approval under increased experimental costs

0 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

150

1.0 0.8

100 0.6 0.4

0.2 0

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

Here, we show the result of an antibiotic approval process with an increased experimental cost. In particular, we take the parameters in Table 2 and Table 3 but increase the fixed cost c0 of a trial to $100 M and the per-patient cost c1 to $0.1 M. We find that the agent opts out at the beginning of the process unless the principal subsidizes a fraction higher than the optimal subsidy, ε∗ = 0.551.

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

600 400 200 0 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1500

1.0 0.8

1000

0.6 0.4

500

0.2

Probability of opting out

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

Figure 12: Agent utilities under increased experimental costs. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex, and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.0

0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 13: Social utilities under increased experimental costs. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

Optimal subsidy, ε∗

In Figure 14 we show that this optimal subsidy is constant as ρS increases, and in Figure 15 that our sequential protocol yields social utility gains > 20% relative to a non-sequential protocol. 0.6 0.4 0.2 0.0 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Social utility gain vs. non-sequential (%)

Figure 14: Optimal subsidy vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

35 30 25

Non-sequential (optimal subsidy) 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 15: Social utility gain vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a nonsequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under the optimal subsidy computed using Algorithm 1 (in the non-sequential protocol without subsidy the agent always opts out, yielding zero social utility).

H.2.2

Approval under increased agent approval benefit

1920 1915 1910 1905 1900 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

1.0

4600

0.8

4580

0.6

4560 4540

0.4

4520

0.2 0.0

4500

0.9

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

Here, we show the result of an antibiotic approval process with an increased approval utility for the agent, ρA = $5000 M (and the rest of the parameters in Table 2 and Table 3 fixed). In this case, the approval utility for the agent covers the expected cost of approval by a large margin, and we find that no subsidy is needed, namely, ε∗ = 0.

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

1860 785 780 775 770 765 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0

1840

0.8

1820

0.6

1800

0.4

1780

0.2

1760

Probability of opting out

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

Figure 16: Agent utilities under increased approval utility. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 17: Social utilities under increased approval agent utility. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

Optimal subsidy, ε∗

0.02 0.01 0.00 −0.01 −0.02

0.00

0.25

0.50 0.75 1.00 1.25 1.50 Social-to-agent approval benefit ratio, ρS /ρA

1.75

2.00

Social utility gain vs. non-sequential (%)

Figure 18: Optimal subsidy vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

Non-sequential (no subsidy)

8.75

Non-sequential (optimal subsidy)

8.50 8.25 0.00

0.25

0.50 0.75 1.00 1.25 1.50 Social-to-agent approval benefit ratio, ρS /ρA

1.75

2.00

Figure 19: Social utility gain vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a nonsequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under (i) the optimal subsidy computed using Algorithm 1 and (ii) no subsidy (ε = 0). In this case, the optimal non-sequential policy is the same for all subsidies, and we apply a small vertical jitter to improve visibility.

H.2.3

Approval under pessimistic prior

12.5 10.0 7.5 5.0 2.5 0.0 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0 150

0.8 0.6

100

0.4 50 0.2 0

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

Here, we show the results of an antibiotic approval process where the agent’s prior is (α0 , β0 ) = (1, 1.5), that is, the agent is slightly pessimistic about its product.25 Note that the mean efficacy of the drug according to its prior is then 0.4 < θ∗ = 0.65. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. In this case, we find that the agent opts out at the beginning of the approval process as long as the subsidy is ε ≲ 0.1. The optimal subsidy is ε∗ ≈ 0.4.

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

400 300 200 100 0 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0

1500

0.8 1000

0.6 0.4

500 0.2

Probability of opting out

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

Figure 20: Agent utilities under a pessimistic prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.0

0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 21: Social utilities under a pessimistic prior. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8). In Figure 22 we show that the optimal subsidy increases with ρS /ρA , and in Figure 23 that our sequential protocol yields gains > 40% in social utility relative to a non-sequential protocol. 25We select β = 1.5 as higher values lead the agent to opt out at the beginning of the process, regardless of the subsidy. 0

Optimal subsidy, ε∗

0.75 0.50 0.25 0.00 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Social utility gain vs. non-sequential (%)

Figure 22: Optimal subsidy vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

Non-sequential (optimal subsidy)

40 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 23: Social utility gain vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a nonsequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under (i) the optimal subsidy computed using Algorithm 1 (in the non-sequential protocol without subsidy the agent always opts out, yielding zero social utility).

H.2.4

Approval under optimistic prior

180 170 160 150 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

200

1.0

180

0.8

160

0.6 0.4

140

0.2 120

Probability of opting out

190

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

Here, we show the results of an antibiotic approval process where the agent’s prior is (α0 , β0 ) = (4, 1), that is, the agent is slightly optimistic about the antibiotic. Note that the mean efficacy of the drug according to its prior is then 0.8 > θ∗ = 0.65. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. In this case, the optimal subsidy is ε∗ = 0, and as can be seen in Figure 25, the optimal subsidy maximizing Ū S (ε; π ε ) does not necessarily maximize the true (unknown) utility U S (ε; π ε ). Nevertheless, Figure 27 shows that the proposed sequential subsidized protocol can still substantially improve social utility relative to a non-sequential protocol, with gains exceeding approximately 18%.

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

1690

1680

1670 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0 1720 0.8 1700 0.6 1680 0.4 1660 0.2 1640

Probability of opting out

1700

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

Figure 24: Agent utilities under an optimist prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 25: Social utilities under an optimist prior. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

Optimal subsidy, ε∗

0.75 0.50 0.25 0.00 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Social utility gain vs. non-sequential (%)

Figure 26: Optimal subsidy vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

60 Non-sequential (no subsidy)

Non-sequential (optimal subsidy) 20 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 27: Social utility gain vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a nonsequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under (i) the optimal subsidy computed using Algorithm 1 and (ii) no subsidy (ε = 0).

H.2.5

Approval under calibrated prior

160

140

120

0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0

200

0.8

180

0.6 0.4

160 0.2

140

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

Here, we show the results of an antibiotic approval process where the agent’s prior is (α0 , β0 ) = (130, 70). This corresponds to a very informative prior that is calibrated to the true efficacy θ∗ = 0.65, since the mean of the prior is precisely 0.65. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. In this case, we find that the calibrated prior allows the agent to increase its utility U A (π ε ; ε) for any possible subsidy, as can be seen by comparing Figure 28 to the fiducial setting in Figure 9. However, and perhaps surprisingly, we also find that it is still optimal for the principal to subsidize a non-negligible fraction ε∗ ≈ 0.234 of the agent’s cost, despite the prior belief supporting that the drug should be approved.

0.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

1585 1580 1575 1570 1565

0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0 1840

0.8 0.6

1830 0.4 1820

0.2

Probability of opting out

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

Figure 28: Agent utilities under a calibrated informative prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 29: Social utilities under a calibrated informative prior. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

Optimal subsidy, ε∗

1.0

0.5

0.0 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 30: Optimal subsidy vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

Social utility gain vs. non-sequential (%)

Lastly, Figure 31 shows that the social utility gain relative to a non-sequential protocol with optimal subsidies is small for high values of ρS /ρA . This is expected because under a calibrated prior, the agent already has an accurate estimate of the product’s efficacy and therefore gathering new information brings little benefit. As a result, the agent can effectively select an optimal sample size and complete the process in a single trial. 15 10

Non-sequential (no subsidy) Non-sequential (optimal subsidy)

5 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 31: Social utility gain vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a nonsequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under (i) the optimal subsidy computed using Algorithm 1 and (ii) no subsidy (ε = 0).

H.2.6

Approval under uncalibrated prior

230 220 210 200 190 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0

200

0.8

180

0.6

160

0.4

140

0.2 120

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

Here, we show the results of an antibiotic approval process where the agent’s prior is (α0 , β0 ) = (130, 30). This corresponds to a very informative and optimistic prior that is uncalibrated to the true efficacy θ∗ = 0.65, since the mean of the prior is precisely 0.8125. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. We find that, across subsidies, the utility U A (π ε ; ε) achieved by the agent slightly decreased compared to an agent with a calibrated prior (Figure 32 vs. Figure 28). However, surprisingly, we also find that the social utility is greater in this case compared to the case where the agent has a calibrated prior (Figure 33 vs. Figure 29).

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

1990 1980 1970 1960 1950

0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

1.0 1860

0.8

1840

0.6

1820

0.4 0.2

1800

Probability of opting out

2000

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

Figure 32: Agent utilities under an uncalibrated informative prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 33: Social utilities under an uncalibrated informative prior. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8). Interestingly, Figure 34 shows that the optimal subsidy remains zero as ρS /ρA increases. At the same time, Figure 35 indicates that our sequential protocol achieves gains exceeding 90% in social utility relative to a 69

Optimal subsidy, ε∗

non-sequential protocol. In other words, the protocol can substantially improve social utility even in the absence of subsidies. 0.02 0.01 0.00 −0.01 −0.02

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Social utility gain vs. non-sequential (%)

Figure 34: Optimal subsidy vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

100 Non-sequential (no subsidy) 95

Non-sequential (optimal subsidy)

90 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 35: Social utility gain vs. ρS /ρA . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a nonsequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under (i) the optimal subsidy computed using Algorithm 1 and (ii) no subsidy (ε = 0).

H.2.7

Approval under a different test process

60 55 50 45 40 0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

0.020

220

0.015

200

0.010 180 0.005 160

Probability of opting out

Agent utility, U A(π ε; ε) ($M)

Agent’s utility, Ū A(π ε; ε) ($M)

In this section, we present additional experimental results for the antibiotic approval process described in Section 6, using an alternative test process to the process M defined in Proposition 1 and Eq. 3, while keeping all other parameters fixed as in Tables 2 and 3. Here, our goal is to illustrate that Algorithm 1 can provide insight into how to optimally subsidize agents under different statistical tests. More concretely, following Appendix C, we consider the mixed process M mix defined in Eq. 30 with a uniform mixture P mix = U(θb , 1). We find that the sequential subsidized protocol yields social utility gains of up to 15% compared to a non-sequential baseline without subsidies. Relative to a non-sequential but optimally subsidized baseline, the gains can also reach up to 15% when the ratio ρS /ρA is low, but diminish rapidly as this ratio increases. The initial optimal action taken by the agent is n ≤ 114 for all subsidies. Lastly, in Figure 40, we confirm that the mixed process M mix has higher power than M for values of θ∗ close to θb , as discussed in Appendix C.

0.000 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Agent utility Ū A (π ε ; ε) computed using Mε

(b) Agent utility U A (π ε ; ε) in the approval process

Social utility, U S(ε; π ε) ($M)

Social utility, Ū S (ε; π ε) ($M)

855

850

845

0.0

0.1

0.2

0.3

0.4 0.5 0.6 Subsidy, ε

0.7

0.8

0.9

0.020 1960 0.015 0.010

1940

0.005

Probability of opting out

Figure 36: Agent utilities under a mixed test process. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex, and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8).

0.000

1920 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Subsidy, ε

(a) Social utility Ū S (ε; π ε ) computed using Mε

(b) Social utility U S (ε; π ε ) in the approval process

Figure 37: Social utilities under a mixed test process. The left panel shows the social utility (Eq. 13) computed using the belief MDP Mε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy and θ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8). 71

Optimal subsidy, ε∗

0.9 0.6 0.3 0.0 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Social utility gain vs. non-sequential (%)

Figure 38: Optimal subsidy vs. ρS /ρA using a mixed test process. The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1.

15 Non-sequential (no subsidy)

Non-sequential (optimal subsidy) 5 0

10 15 20 25 30 Social-to-agent approval benefit ratio, ρS /ρA

Figure 39: Social utility gain vs. ρS /ρA using a mixed test process. The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non-sequential approval protocol in which the agent is restricted to a single trial with nmax = 800, under (i) the optimal subsidy computed using Algorithm 1 and (ii) no subsidy (ε = 0).

Non-mixture Uniform mixture

Probability of approval

1.0 0.8 0.6 0.4 0.2 0.0 0

0.25

θb True efficacy, θ∗

0.75

Figure 40: Probability of approval under the optimal policy and subsidy. The figure shows, across various efficacies θ∗ of the antibiotic, the probability of approval (that is, of rejecting H0 ) when the principal selects the optimal subsidy and the agent its optimal policy. The dashed (orange) curve corresponds to the process M mix defined in Eq. 30 for a uniform mixture (optimal subsidy ε∗ = 0.027), while the solid (green) curve corresponds to the process M defined in Proposition 1 and Eq. 3 (optimal subsidy ε∗ = 0.108).

Record · ID 168344 · SHA-256 00b370b284b0100f
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.