Conceptio › Archive › arXiv CS

arXiv CSopen access

Dynamic Treatment on Networks

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Dynamic Treatment on Networks Bengusu Nar ∗, Jiguang Li †, Veronika Ročková ‡, and Panos Toulis §

arXiv:2605.06564v1 [stat.ML] 7 May 2026

May 8, 2026

Abstract In networks, effective dynamic treatment allocation requires deciding both whom to treat and also when, so as to amplify policy impact through spillovers. An early intervention at a well-connected node can trigger cascades that change which nodes are worth targeting in the next period. Existing treatment strategies under network interference are largely static while dynamic treatment frameworks typically ignore network structure altogether. We integrate these perspectives and propose Q-Ising, a three-stage pipeline that (i) estimates network adoption dynamics via a Bayesian dynamic Ising model from a single observed panel, (ii) augments treatment adoption histories with continuous posterior latent states, and (iii) learns a dynamic policy via offline reinforcement learning. The Bayesian mechanism enables uncertainty quantification over dynamic decisions, yielding posterior ensemble policies with interpretable spillover estimates. We provide a finite-sample regret upper bound that decomposes into standard offline-RL uncertainty, network abstraction error, and first stage error in Ising state estimation. We apply our method to data from Indian village microfinance networks and synthetic stochastic block models under simulated heterogeneous susceptible-infected-susceptible (SIS) dynamics and demonstrate that adaptive targeting outperforms static centrality benchmarks.

Introduction

When a planner makes decisions about a dynamic policy on a network, the central problem is not only whom to treat, but also when. Under network interference, the order in which units receive treatment determines which nodes will spread spillovers first and how those will compound over time. A policy that selects the right nodes to treat but ignores their sequencing can perform strictly worse than one that orders strategically. To see why ordering matters, consider a marketing campaign promoting a product on a social network with a limited budget split across multiple periods. If the planner treats influential users first, the second-period action changes: the planner can now target ∗

Bengusu (Bengüsu) Nar is a doctoral student in Econometrics and Statistics at the Booth School of Business of the University of Chicago † Jiguang Li is a doctoral student in Econometrics and Statistics at the Booth School of Business of the University of Chicago ‡ Veronika Ročková is Bruce Lindsay Professor of Econometrics and Statistics in the Wallman Society of Fellows at the Booth School of Business of the University of Chicago § Panos (Panagiotis) Toulis is a Professor of Econometrics and Statistics, and John E. Jeuck Faculty Fellow at the Booth School of Business of the University of Chicago

neighbors who were exposed but have not yet adopted, or can start fresh elsewhere in the network. The optimal action in period two depends on what period one achieved. Similar sequencing problems arise in viral marketing [Kempe et al., 2003, Domingos and Richardson, 2001], platform engagement campaigns, and public health interventions [Bubar et al., 2021, Buckner et al., 2021]. The core difficulty in this setting is that the planner cannot experiment and is limited to a single observational trajectory under a historical policy. This rules out online influence maximization (IM) [Kempe et al., 2003] algorithms that require oracle access to the diffusion mechanism [Singh et al., 2022]. Also, due to dynamic decision-making aspect, static policies are usually not optimal. To address this problem, the planner needs two things: first, an adaptive model of how the network behaves; second, a way to use that model to choose interventions sequentially. For the first part, we use a dynamic Ising model [Yang, 1992]. This model is designed for estimating the probability of each node’s next state conditional on the current state of itself, its neighbors and the past intervention. Unlike equilibrium Ising models that require intractable partition functions, the dynamic formulation admits tractable nodewise likelihoods. For the second part, we use offline reinforcement learning (RL) which is a method for learning dynamic decision rules from historical data, without needing experimentation [Levine et al., 2020]. As a key contribution, we treat the estimated conditional probabilities as latent states for offline RL. Lastly, to assess the uncertainty of these dynamic decisions, we provide an ensemble framework for uncertainty quantification. This three-stage process provides a unified framework for dynamic policy under network interference. Related work spans multiple fields. Several approaches learn welfare-maximizing policies for single-period interventions [Kitagawa and Tetenov, 2018, Viviano, 2025], and network-aware targeting based on network topologies like centrality and the friendship paradox [Banerjee et al., 2013, Kempe et al., 2003, Christakis and Fowler, 2010, Kim et al., 2015, Chen et al., 2009, Liu et al., 2017]. Dynamic treatment regime methods [Murphy et al., 2001, Chakraborty and Murphy, 2014, Hu et al., 2025, Adusumilli et al., 2019, Kitagawa et al., 2022] provide tools for dynamic decision-making but do not use network structures. Restless multi-armed bandits provide a framework for budget-constrained dynamic policy [Whittle, 1988, Weber and Weiss, 1990, Mate et al., 2020] but assume independently evolving units. Recent extensions embed arms in networks, allowing an intervention to benefit their neighbors [Herlihy and Dickerson, 2023, Ou et al., 2022, Vaswani et al., 2015, Gleich et al., 2025], but these approaches assume known spillover mechanisms or repeated experimentation. Similarly, some works require Nash equilibrium and known dynamics [Kitagawa and Wang, 2023a]. Recent graph neural network (GNN) methods achieve strong empirical performance but also require online experimentation or known dynamics [Manchanda et al., 2020, Sun et al., 2018, Meirom et al., 2021, Feng et al., 2024]. A comparison among some of these methods is provided in Table 1 in Appendix A. We propose Q-Ising, combining Bayesian dynamic Ising inference with offline RL for sequential decision-making under network interference. Methodologically, we transform the intractable problem of policy learning under network interference into a standard offline RL problem by treating estimated network dynamics as latent states. Theoretically, we give a regret upper bound for finite-horizon Q-Ising under pessimistic offline learning and show that the bound decomposes into standard offline RL uncertainty, network abstraction and first stage errors. 2

We employ Q-Ising on susceptible-infected-susceptible (SIS) dynamics [Kermack and McKendrick, 1927, Hethcote, 2000], a widely used model of relapsing contagion [Bass, 1969, Jackson and Rogers, 2007, Bohner and Streipert, 2016] on two regimes. First one is a stochastic block model (SBM) where the most influential nodes form a smaller community that is not identifiable by degree alone, making the setting adversarial for centrality-based methods. In this design, Q-Ising identifies the high-influence group from offline data alone and shifts the treatment budget adaptively as organic spread becomes self-sustaining. Second, we demonstrate Q-Ising’s practical value on real microfinance networks from Karnataka, India [Banerjee et al., 2013], where SIS dynamics are simulated on the empirical adjacency matrices. Across both regimes, Q-Ising matches or improves over the best baseline, provides interpretable coefficient estimates revealing the underlying dynamics, and quantifies uncertainty over recommended actions at each decision point. Neither of these features are available in GNN-based influence maximization approaches [Manchanda et al., 2020, Sun et al., 2018, Meirom et al., 2021, Feng et al., 2024]. The rest of the paper is as follows: First, we formally define the problem in Section 2. Then we introduce our methodology in Section 3. In Section 4, we give the assumptions for deriving the pessimistic suboptimality and present the regret bound. Finally, we conduct our experiments in Section 5.

Framework and Problem Formulation

We study dynamic treatment on a fixed network with binary outcomes. Throughout, yi,t = 1 denotes adoption of a product or behavior by node i in period t; adopted units may later disengage, so the planner must decide not only whom to treat but also when. Setting and Timing. There are N nodes connected by a fixed, undirected, observed network M ∈ {0, 1}N ×N , where Mij = 1 if i and j are linked and Ni = {j : Mij = 1} denotes the neighbors of i. Each node has fixed observed features xi ∈ Rdx . At each period t, node i has a conditional adoption probability li,t ∈ [0, 1] under the realized history and treatment, and realizes yi,t ∼ Bernoulli(li,t ). We write yt = (y1,t , . . . , yN,t )⊤ . We reserve t = 1, . . . , Ttrain for periods in the training panel. For policy evaluation, we re-index the target deployment horizon by h = 1, . . . , H. Before a period-t decision, the planner observes the pre-action history Zt := (y0:t−1 , a1:t−1 , X, M ), with Z1 = (y0 , X, M ). The sigma-algebra generated by Zt is the filtration Ft , but we use Zt directly to keep the notation explicit. Each period allocates one treatment to a node, at ∈ [N ]. We reserve at = ∅ only as an auxiliary no-intervention action used later for counterfactual state construction. Data. The planner observes one trajectory of node-level outcomes and treatments, train D = y0 , (at , yt )Tt=1 , collected under a historical policy. This is the practically important regime of a single long panel from one real network, with M and X fixed and known throughout. We retain nodelevel actions because Stage 1 estimates direct treatment and neighbor-spillover effects. 3

Bin-level policy class. For large N , node-specific targeting is both statistically and practically challenging. Under roughly uniform logging, each node receives only O(Ttrain /N ) treatment observations, which is insufficient for reliable policy learning under interference. Beyond data coverage, treatment rules defined over covariate groups rather than individuals are more interpretable, auditable, and aligned with fairness requirements common in policy applications [Kitagawa and Tetenov, 2018, Kitagawa and Wang, 2023b, Viviano and Bradic, 2024]. We therefore partition nodes into K disjoint bins {B1 , . . . , BK } based on covariates X and/or network structure M , for example communities identified by spectral clustering or demographic strata such as young married households. The choice of K trades off statistical coverage against targeting granularity: each bin accumulates O(Ttrain /K) treatment observations. K should be chosen so that within-bin nodes are reasonably homogeneous with respect to both covariates and network position. We distinguish two levels of action: the planner selects a bin bt ∈ {1, . . . , K} each period, and one node is drawn uniformly at random from the selected bin as the realized treatment: bt = πh (Zt ), at ∼ Unif(Bbt ). (1) Historical node actions are mapped to bin actions when constructing the offline RL transitions. The target deployment rule in (1) imposes uniform within-bin randomization; However, the historical within-bin selector may differ, a distinction accounted for in our regret theory. Welfare Maximization Let ΠZ denote the class of admissible full-history bin policies π = (π1 , . . . , πH ), where πh : Zh → [K] maps the pre-action history Zh to a bin decision. The per-period reward is the network-wide adoption rate, N

rh =

1 X yi,h . N i=1

(2)

For π ∈ ΠZ , its policy value, or cumulative welfare, is the expected reward along the full trajectory induced by deploying π: " H # X WHZ (π; Z1 ) := Eπ rh Z1 , h=1

where the expectation is over the stochastic network evolution and the within-bin randomization generated by the policy. The full-history population target is πZ⋆ ∈ arg max WHZ (π; Z1 ). π∈ΠZ

(3)

This objective is inherently dynamic: treating a node at time t changes subsequent adoption probabilities through persistence and network spillovers, so the value of a current bin decision depends on how it shapes future network states. Thus myopic targeting needs not be optimal. Formally, let a greedy rule select bh ∈ arg maxb∈[K] E[rh | Zh , bh = b] at each stage. Proposition 1 (Greedy Suboptimality). The greedy policy that maximizes the immediate reward is not necessarily optimal. Appendix C.1 proves the proposition by counterexample under basic SIS dynamics. 4

Methodology

Q-Ising proceeds in three stages. First, we estimate a dynamic Ising model from the panel D. Second, we use the fitted model to construct low-dimensional Q-Ising states. Third, we apply offline RL to learn a dynamic bin-level policy over these states.

3.1

Stage 1: Dynamic Ising Inference

Standard Equilibrium Ising model requires computing an intractable partition function and symmetric interactions. We instead use a dynamic conditional model, analogous to logistic pseudo-likelihood estimation for Ising models [Ravikumar et al., 2010], which allows asymmetric influence and temporal dependence. Define the linear predictor for node i in bin Bk at time t: X ηi,t (at ; θi ) = β0,k + β1,k 1at =i + β2,k yi,t−1 + β3,k 1at ∈Ni + γk,mj yj,t−1 . (4) j∈Ni

Here mj represents the bin node j belongs. Although M is undirected, the interaction parameters need not satisfy γk,mj = γk,mj : node j may strongly influence node i without the reverse being true. This asymmetry captures the influencer structure common in social networks, where a well-connected household may drive adoption among neighbors without being equally susceptible to peer influence itself. The standard equilibrium Ising model would not be able to capture this important feature. ⊤ ⊤ The parameter vector θi = [β0,k , β1,k , β2,k , β3,k , γN (i) ] collects all parameters for node i, and θ = [θ1 , . . . , θN ]⊤ . Each parameter has a natural interpretation: β1,k captures the direct effect of treating node in bin Bk ; β2,k captures persistence of past adoption; β3,k captures spillover from treating a neighbor; and γk,mj captures peer influence from neighbors in bin mj on nodes in bin k. Node-level coefficients are a direct extension obtained by replacing bin indices with node indices when the panel is sufficiently long. Let σ(z) = (1+e−z )−1 denote the logistic function. The probability of node i adopting after taking action at is given by P(yi,t = 1 | Zt , at , θi ) = σ(ηi,t (at ; θi )). Given Zt , at , and θ, outcomes are independent Bernoulli draws across nodes. Since the process is also first-order Markov,the likelihood over the panel factorizes as L(θ) =

N Y T Y exp(yi,t · ηi,t (at ; θi ))

1 + exp(ηi,t (at ; θi )) i=1 t=1

(5)

enabling efficient likelihood-based inference. Remark 1 (Connection to standard Ising). Under synchronous updating, which is the relevant case for policy applications where all nodes respond simultaneously, the stationary distribution of the dynamic Ising model cannot in general be expressed as an element-wise Gibbs measure, making the dynamic formulation the appropriate choice for our setting. See Appendix C.2 for proof.

Priors and sparsity. For interaction parameters we impose a continuous spike-andslab prior encouraging sparsity in peer effects [Ročková and George, 2014, George and McCulloch, 1993]: c , γi,j | zij ∼ (1 − zij ) N (0, v0 ) + zij N (0, v1 ), v0 ≪ v1 , zij ∼ Bernoulli |Bm | i

with hyperparameters v0 = 0.01, v1 = 10.0, and c = 1.0. The bin-size dependent inclusion probability encodes the expectation that the number of influential bins scales sublinearly with the average degree of the affected bin. For baseline, treatment, and persistence parameters, βl,k ∼ N (0, τ 2 ) with τ 2 = 10.0. Zero-centered priors ensure that when data are sparse, estimates shrink toward θ̂i ≈ 0, yielding ˆli,t ≈ 0.5: a principled uninformed baseline. Estimation: EMVS and MCMC. We offer two complementary approaches. EMVS [Ročková and George, 2014] alternates between computing posterior inclusion probabilities and solving a weighted penalized logistic regression, requiring only convex optimization per iteration and typically converging in 2–3 steps; it is particularly attractive for large N where speed is essential. When posterior uncertainty is needed, we sample from p(θi | D) via Hamiltonian Monte Carlo Hoffman and Gelman [2014], producing P (p) posterior draws {θi }Pp=1 per node, which serve as the basis for the ensemble policy in Section 3.3. State construction. Given θ̂i , we construct latent states that summarize the network’s adoption landscape at each period. For node i, define the estimated no-intervention adoption probability 0 b b li,t := σ ηi,t (∅; θ) , (6) where at = ∅ sets treatment indicators in (4) to zero. Setting at = ∅ is important because it captures where the network is headed absent the current intervention, serving as a forward-looking baseline. We aggregate both the estimated latent states and the observed outcomes to the bin level: X 1 X 0 b ¯l0 = 1 li,t , ȳt−1,k = yi,t−1 , k ∈ [K]. (7) t,k |Bk | i∈B |Bk | i∈B k

0 0 Let l̄0t = (¯lt,1 , . . . , ¯lt,K ) and ȳt−1 = (ȳt−1,1 , . . . , ȳt−1,K ). The Q-Ising state is

st = l̄0t , ȳt−1 ∈ S := [0, 1]K × [0, 1]K .

(8)

The first component is forward-looking and model-based; the second is the realized binlevel adoption profile before the current decision.

3.2

Stage 2: Offline Q-Learning

train We construct the transitions DRL := {(st , bt , rt , st+1 )}Tt=1 from D, where st is defined in eq. (8), bt ∈ {1, . . . , K} is the bin-level action mapped from at , and rt is defined in eq. (2). Learning over K bin actions is statistically more stable than learning over N node actions, since each bin aggregates many treatments.

For the empirical implementation, we learn a discounted Q-function Q : S × [K] → R with discount ψ ∈ [0, 1), which represents the expected cumulative reward from selecting bin b at state s and following the optimal policy thereafter. The Q-function satisfies the Bellman target [Sutton et al., 1998] Tb Q(st , bt ) = rt + ψ ′ max

b ∈{1,...,K}

Q(st+1 , b′ ).

Since DRL is collected under a historical policy, some state-action pairs are poorly supported. Standard Q-learning can then overestimate unsupported actions. We therefore use the pessimism principle from offline RL: unsupported actions should be penalized rather than optimistically extrapolated. In experiments, we use conservative Q-learning (CQL) [Kumar et al., 2020] via d3rlpy [Seno and Imai, 2022]; Section 4 analyzes an idealized finite-horizon pessimistic variant of the value iteration (PEVI) version of Q-Ising under the same pessimism principle as [Jin et al., 2021]. In particular, the empirical CQL objective is " # h X 2 i ′ + α E(s,b)∼DRL log exp Q(s, b ) − Q(s, b) , LCQL (Q) = E(s,b,r,s′ )∼DRL Q(s, b) − Tb Q̄ {z } | b′ {z } | Bellman error conservative penalty

(9) where α > 0 controls the strength of the conservative penalty. The first term is the standard Bellman error, minimized over transitions observed in DRL . The second term penalizes Q-values on all actions at observed states while pushing up Q-values on the actions actually taken in DRL , mitigating distributional shift in offline policy learning [Levine et al., 2020].

3.3

Stage 3: Ensemble Policy and Uncertainty Quantification via Posterior Sampling

Posterior sampling propagates first-stage uncertainty into the learned policy. Each MCMC 0,(p) 0,(p) draw θ(p) induces node-level estimates b li,t via eq.(6), bin-level aggregates ¯lt,k via eq.(7), (p)

0,(p)

and hence a state st = (l̄t , ȳt−1 ). Note that ȳt−1 is shared across all draws as it depends only on observed data. Training a separate CQL network on each draw produces an ensemble {π̂ (p) }Pp=1 reflecting parameter uncertainty. Effectively, this posterior sampling helps recognizing potential lack of identification in the dynamic Ising model as Bellman optimality provides unique policies for each trained agent. When most draws agree, the planner acts with confidence; when votes are dispersed, the allocation is sensitive to parameter uncertainty and warrants caution. In practice, P = 10–20 posterior draws yield stable ensemble policies.

Theory

We provide a finite-sample regret guarantee for an idealized finite-horizon PEVI version of Q-Ising. The empirical algorithm in Section 3.2 uses CQL as a scalable implementation of the same pessimism principle; the theorem below analyzes the PEVI analogue [Jin et al., 2021].

Algorithm 1 Q-Ising Algorithm: Policy Learning via Dynamic Ising and Offline RL Require: Panel D, adjacency M , bin partition {B1 , . . . , BK }, flag ensemble ∈ {true, false} 1: if ensemble then 2: Draw {θ (p) }Pp=1 via HMC from π(θ | D, M ); set P ← {1, . . . , P } 3: else 4: Fit θ̂ via EMVS on (D, M ); set θ (1) ← θ̂, P ← {1} 5: end if 6: for p ∈ P do 0,(p) (p) 7: Compute b li,t = σ(ηi,t (∅; θi )) for all i, t via (6) (p) 0,(p) 8: Form st = (l̄t , ȳt−1 ) using (7)–(8) (p) (p) −1 9: Construct transitions {(st , bt , rt , st+1 )}Tt=1 10: Train Qp via (9); set π̂ (p) (s) = arg maxk Qp (s, k) 11: end for 12: if ensemble then P 13: return π̂ ens (s) = arg maxk Pp=1 1{π̂ (p) (s) = k} 14: else 15: return π̂(s) = π̂ (1) (s) 16: end if Let θ0 be the population Ising parameter and ϕθ be the Q-Ising state map in (6)–(8). For a pre-action history Zh , write s⋆h = ϕθ0 (Zt ) and sbh = ϕθb(Zh ). For any policy π, define its oracle full history lift by πh◦ (z) = πh (ϕθ0 (z)) Our comparator is the best oracle Q-Ising policy πS⋆ ∈ arg max WHZ (π ◦ ; Z1 ),

ΠS = {π = (π1 , . . . , πH ) : πh : S → [K]}.

π∈ΠS

This comparator is not the unrestricted full-history optimum πZ⋆ in (3), but the optimal Q-Ising policy. Let π b be the finite-horizon pessimistic policy trained on the Q-Ising offline transitions Dh = {(b sτ,h , bτ,h , rτ,h , sbτ,h+1 )}τ ∈Ih ,

h = 1, . . . , H,

where τ indexes the effective training blocks. Let nh = |Ih |, neff = minh nh , and nlog = maxh nh . For a fixed feature map φ : S × [K] → Rdϕ , with ∥φ(s, b)∥2 ≤ 1, define the empirical design matrix X Λh = λIdϕ + φ(b sτ,h , bτ,h )φ(b sτ,h , bτ,h )⊤ . τ ∈Ih

The main theorem analyzes the oracle deployment π bh◦ (z) = π bh (ϕθ0 (z)) with regret RegH (b π ◦ ) := WHZ ((πS⋆ )◦ ; Z1 ) − WHZ (b π ◦ ; Z1 ). The PEVI uncertainty of a policy π under the training data aggregate dynamics is UµD (π) :=

H X

Eπµ

φ(sh , bh )⊤ Λ−1 h φ(sh , bh )

h=1

where Eπµ denotes expectation over the bin-level Q-Ising process induced by following π while using the historical within-bin selector. The proof relies on four regularity conditions, stated formally in Appendix D.2. In words, historical actions must be valid interventions; the single network trajectory must yield effective stage-wise regression blocks; the Q-Ising state must be an approximate binlevel abstraction of the full network process; and the PEVI Bellman regressions must have agg controlled projected misspecification. The abstraction errors εagg measure the loss r , εP from replacing the full network by the oracle Q-Ising state process. The binning errors bin εbin measure the mismatch between the target rule, which randomizes uniformly r , εP within a selected bin, and the historical within-bin selector. The quantities εlin and Lst εθ respectively capture projected Bellman approximation error and first-stage Q-Ising state-estimation error. Finally, Let Eθ denote the first-stage stability event that replacing oracle Q-Ising states s⋆h = ϕθ0 (Zh ) by estimated states sbh = ϕθb(Zh ) induces controllable projected PEVI regression error. We assume P(Eθ ) ≥ 1 − δθ . Theorem 2. Suppose the regularity conditions summarized above and formalized in Appendix D.2 hold. Choose # " s √ H(1 + nlog /λ) + λW , β = Cβ H dϕ log δ where W bounds the linear Bellman coefficients and Cβ is a sufficiently large constant. Then, with probability at least 1 − δ − δθ , RegH (b π ◦ ) ≤ 2β UµD (πS⋆ ) + C [∆abs + Hεlin + HLst εθ ] , 2 agg where C > 0 is a universal constant and ∆abs = ∆agg +∆bin , with ∆agg := Hεagg r +H εP , 2 bin and ∆bin = Hεbin r + H εP .

The first term is the standard pessimistic offline-RL error that shrinks with larger effective sample size under standard coverage conditions. With in ∆abs , ∆agg is the cost of compressing the full network history into bin-level Q-Ising states; ∆bin measures the mismatch between learning under the historical within-bin selector and deploying the target rule that randomizes uniformly within the selected bin. It vanishes to 0 if the historical policy also randomizes uniformly within selected bins. The term Hεlin is the cost of linear approximation for the Bellman equations, which vanishes under an exact linear Bellman model. The term HLst εθ is the first-stage Ising state-estimation cost, which can vanish under consistency of the dynamic Ising estimator. In particular, if the Q-Ising abstraction is sufficient and exact, the historical within-bin selector matches the target uniform selector, and the first-stage and Bellman approximation errors vanish, Theorem 2 reduces to the usual PEVI guarantee. Appendix D gives the formal proof of the theorem.

Experiments

We evaluate Q-Ising across two network regimes: a Stochastic Block Model (SBM) in Subsection 5.1 and microfinance networks from Karnataka, India [Banerjee et al., 2013] in Subsection 5.2. In both cases, we simulate observational data from heteregenous, synthetic SIS dynamics. These dynamics are designed to be adversarial for degree-based 9

methods. The optimal strategy requires treating the community of highly susceptible nodes early which ignites an organic within-group spread, and then reallocating to other communities before saturation erases the marginal benefit of further treatments in the highly-susceptible group. A ranker that sorts nodes by degree will concentrate its budget on central but not necessarily susceptible nodes. Neither whom to treat nor when is recoverable from degree statistics alone and requires adaptive decision making. All experiments run on an M1 chip, with Q-Ising and Plain DQN each requiring roughly 60–65 seconds of wall-clock training time. Ising parameters are estimated from this panel via EMVS; bin-level states l̄t are constructed via (7) and take around 20 − 30 seconds of wall clock time. We compare against five reference policies. Three are topologyonly heuristics: a degree-bin policy that iterates over the bins and selects the highest degree untreated node, a degree centrality policy, and LIR [Liu et al., 2017] which identifies local degree leaders to avoid the rich-club effect. The fourth, Plain DQN, follows our offline RL framework without the Ising augmentation, using the observational dynamics but no structural model. Lastly, we compare against a random bin policy which is also used to generate historical panel data.1

5.1

Experiments in Stochastic Block Model

We simulate a stochastic block model based adjacency matrix with 500 nodes. There are four blocks with varying sizes, split approximately as 187-187-63-63. Within-block edge probability is pin = 0.1 and between-block probability is pout = 0.01. Spread rates are (0.010, 0.012, 0.1, 0.12) and churn rates are (0.4, 0.4, 0.2, 0.2). Small communities get high spread and low churn rates, representing "active" behavior. The planner observes TTrain = 100 train periods from historical random-bin policy. The mean rewards of policies can be seen in Figure 1, where test horizon is H = 25 and 50 independent tests are started from no adoption in the network. Q-Ising Improvement over Degree Bin

Adoption Curves K=4 bins +5% threshold -5% threshold

0.35 % improvement over degree bin

Mean adoption rate

0.30 0.25 0.20 0.15 Random Plain DQN Degree LIR Q-Ising Degree Bin

0.10 0.05 0.00

Period

50%

20% 5% 0% -5% -20% 0

Villages (sorted by improvement)

Figure 1: On the left: The mean period reward from different policies over time averaged over 50 test episodes with standard deviations shaded on the SBM data. On the right: The percentage improvement of Q-Ising over the best topological alternative (degree-bin) in microfinance Villages over 50 test runs. In this set-up, random policy outperforms most of the topological heuristics because it seeds the highly-susceptible blocks by chance occasionally. So, it matches the right target more often than a method deterministically committed to the wrong one. Plain DQN learns a qualitatively similar seed-priority ordering to Q-Ising but has a slow initial ramp. This suggests that Ising augmentation is most valuable in the early campaign 1

Even though NEWM [Viviano, 2025] seems like a natural reference policy, it is not directly scalable to the large networks used in experiments and also does not take panel data as input and would require significant adaptation.

Community Structure and Q-Ising Seeding Village 59 HIGH Q (0.529) N=310 K=8 Q=0.529 imp=-8.1% avg_deg=9.1

Village 71 LOW Q (0.466) N=286 K=8 Q=0.466 imp=+13.5% avg_deg=10.6

Village 59

Comm 7

Comm 6

Comm 5

Comm 4 Comm 3 Comm 0 (n=98, =0.010) Comm 1 (n=42, =0.500) Comm 2 (n=36, =0.050) Comm 3 (n=35, =0.070) Comm 4 (n=32, =0.060) Comm 5 (n=19, =0.020) Comm 6 (n=12, =0.010) Comm 7 (n=12, =0.400)

Comm 2 Comm 1 Comm 0

Step

Community seeded

Comm 0 (n=88, =0.010) Comm 1 (n=56, =0.500) Comm 2 (n=51, =0.050) Comm 3 (n=38, =0.070) Comm 4 (n=26, =0.060) Comm 5 (n=23, =0.020) Comm 6 (n=18, =0.010) Comm 7 (n=10, =0.400)

Q-Ising seeding Comm 7

Q-Ising seeding

Village 71

Comm 4 Comm 3 Comm 2 Comm 1 Comm 0

Step

Figure 2: Trajectory differences across two different networks. The village on the right has a more modular community structure, so identified communities do not spread to other communities as much as the village on the left. The correlation between Q-Ising’s relative performance improvement over Degree-Bin with the village’s modularity is approximately -0.5. window, in addition to providing posterior ensemble policies, interpretable parameters and uncertainty quantification.

5.2

Experiments in Indian Microfinance Villages

We use the empirical adjacency matrices of Indian microfinance villages to provide realistic clustering and degree distributions while simulating SIS dynamics to maintain control over group heterogeneity. To define the bins, we use the edge-betweenness based clustering algorithm. We only consider villages with more than one cluster, 42 of 43 villages satisfy this condition. If an identified cluster has less than 10 nodes, these nodes are considered to be the part of the largest cluster. Each bin gets assigned an unobserved spread and churn rate. The planner observes a single offline panel of Ttrain = 500 periods collected under a uniform random bin policy for each of these networks. The details of the experimental set-up can be found in Appendix E. Evaluation uses 50 independent test runs over test horizon H = 25 initialized from no adoption. Table 2 in Appendix A reports the results. A descriptive figure of performance improvement of Q-Ising over the best non-adaptive policy which is Degree-bin can be found in Figure 1. Q-Ising effectively learns to concentrates its initial treatments on communities with high spread rates and communities that are well connected with others, generating compounding organic spread. Then, it adaptively treats other communities depending on the activity levels. Its performance improvement is more pronounced in villages with low modularity because Q-Ising relies on identifying influential communities and adaptively targeting the remaining ones. In highly modular networks, spillovers compound less, diminishing the advantage of Q-Ising. In addition to the performance improvement, Q-Ising reveals important features about the underlying mechanism through its estimated coefficients. The intercepts β0 are negative for all bins, reflecting a strong baseline non-adoption tendency. Treatment effects β1 are strongly positive for all the bins, recovering the "perfect treatment" structure of the SIS model. Persistence coefficients β2 are positive, capturing adoption stickiness; communities with higher churn rates show weaker persistence. The dominant estimated couplings originate from communities with high spread rates. This interpretable analysis is not possible with other black-box policy methods. Figures about Ising fit quality and posterior distributions can be found in Appendix B. 11

With the ensemble policy approach, Q-Ising can also learn the policy uncertainty. The parameter estimation takes approximately 1 minute of wall clock time with 500 iterations. The training of the 20 agents corresponding to 20 posterior draws takes around 20 minutes. In general, the majority-vote path starts with near-unanimous agreement on communities with high spread rate. After the spreader communities has largely saturated, the majority vote shifts toward multiple alternatives. This dispersion identifies periods near a critical threshold and serves as a measure of strategic uncertainty that a pointestimate policy would silently suppress. Figures showing the ensemble policy trajectory for an example village can be found in Appendix B.

Conclusion

This paper develops Q-Ising, a framework for dynamic treatment allocation under network interference from observational panel data. This is one of the first attempts to offline dynamic policy learning in networks despite its relevance for public health, microfinance, and other settings where experimentation is costly or infeasible. Our approach combines a Bayesian Ising model of network dynamics with conservative Q-learning, yielding a policy that comes with structural parameter estimates and posterior uncertainty while remaining competitive with model-free offline RL. There are many refinements possible. The framework currently seeds a single node per period; multi-node budget extensions are open directions. The method also requires a sufficiently long observational panel for adequate state-action coverage, and its behavior under severe distributional shift between the behavior policy and the target policy requires further study. For state augmentation with network information, higher order Ising interactions or other network representations such as graph neural networks can be used when there is sufficient data. Our experiments demonstrate the framework on SIS dynamics, but the pipeline accommodates to other contagion models that use synchronous updating. Positive applications include public health interventions, microfinance outreach, and information campaigns in low-resource settings. The same framework could be used to exploit social influence for commercial or political ends, and we encourage practitioners to reflect on deployment context accordingly.

References Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24, pages 2312–2320, 2011. Pieter Abbeel and Andrew Y. Ng. Exploration and apprenticeship learning in reinforcement learning. In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, page 1–8, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: 10.1145/1102351.1102352. URL https://doi.org/10.1145/1102351.1102352. Karun Adusumilli, Friedrich Geiecke, and Claudio Schilter. Dynamically optimal treatment allocation using reinforcement learning. arXiv preprint arXiv:1904.01047, 2019.

Abhijit Banerjee, Arun G. Chandrasekhar, Esther Duflo, and Matthew O. Jackson. The diffusion of microfinance. Science, 341(6144):1236498, 2013. Frank M Bass. A new product growth for model consumer durables. Management science, 15(5):215–227, 1969. Martin Bohner and Sabrina H Streipert. The sis-model on time scales. Pliska Stud. Math, 26:11–28, 2016. Kate M Bubar, Kyle Reinholt, Stephen M Kissler, Marc Lipsitch, Sarah Cobey, Yonatan H Grad, and Daniel B Larremore. Model-informed covid-19 vaccine prioritization strategies by age and serostatus. Science, 371(6352):916–921, 2021. Jack H Buckner, Gerardo Chowell, and Michael R Springborn. Dynamic prioritization of covid-19 vaccines when social distancing is limited for essential workers. Proceedings of the National Academy of Sciences, 118(16):e2025786118, 2021. Bibhas Chakraborty and Susan A Murphy. Dynamic treatment regimes. Annual review of statistics and its application, 1(1):447–464, 2014. Wei Chen, Yajun Wang, and Siyu Yang. Efficient influence maximization in social networks. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 199–208, 2009. Nicholas A Christakis and James H Fowler. Social network sensors for early detection of contagious outbreaks. PloS one, 5(9):e12948, 2010. Pedro Domingos and Matt Richardson. Mining the network value of customers. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 57–66, 2001. Yuting Feng, Vincent YF Tan, and Bogdan Cautis. Influence maximization via graph neural bandits. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 771–781, 2024. Edward I George and Robert E McCulloch. Variable selection via gibbs sampling. Journal of the American Statistical Association, 88(423):881–889, 1993. Aidan Gleich, Eric Laber, and Alexander Volfovsky. Scalable policy maximization under network interference. arXiv preprint arXiv:2505.18118, 2025. Christine Herlihy and John P Dickerson. Networked restless bandits with positive externalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11997–12004, 2023. Herbert W Hethcote. The mathematics of infectious diseases. SIAM review, 42(4):599– 653, 2000. Matthew D. Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15 (47):1593–1623, 2014. URL http://jmlr.org/papers/v15/hoffman14a.html.

Yuchen Hu, Shuangning Li, and Stefan Wager. Optimal targeting in dynamic systems. arXiv preprint arXiv:2507.00312, 2025. Matthew O Jackson and Brian W Rogers. Relating network structure to diffusion properties through stochastic dominance. The BE Journal of Theoretical Economics, 7(1): 1–13, 2007. Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International conference on machine learning, pages 5084–5096. PMLR, 2021. David Kempe, Jon Kleinberg, and Éva Tardos. Maximizing the spread of influence through a social network. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 137–146, 2003. William Ogilvy Kermack and Anderson G McKendrick. A contribution to the mathematical theory of epidemics. Proceedings of the royal society of london. Series A, Containing papers of a mathematical and physical character, 115(772):700–721, 1927. David A Kim, Alison R Hwong, Derek Stafford, D Alex Hughes, A James O’Malley, James H Fowler, and Nicholas A Christakis. Social network targeting to maximise population behaviour change: a cluster randomised controlled trial. The Lancet, 386 (9989):145–153, 2015. Toru Kitagawa and Aleksey Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica, 86(2):591–616, 2018. Toru Kitagawa and Guanyi Wang. Individualized treatment allocation in sequential network games. arXiv preprint arXiv:2302.05747, 2023a. Toru Kitagawa and Guanyi Wang. Who should get vaccinated? individualized allocation of vaccines over sir network. Journal of Econometrics, 232(1):109–131, 2023b. Toru Kitagawa, Weining Wang, and Mengshan Xu. Policy choice in time series by empirical welfare maximization. arXiv preprint arXiv:2205.03970, 2022. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1179–1191, 2020. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. Dong Liu, Yun Jing, Jing Zhao, Wenjun Wang, and Guojie Song. A fast and efficient algorithm for mining top-k nodes in complex networks. Scientific reports, 7(1):43330, 2017. Sahil Manchanda, Akash Mittal, Anuj Dhawan, Sourav Medya, Sayan Ranu, and Ambuj Singh. Gcomb: Learning budget-constrained combinatorial algorithms over billionsized graphs. Advances in Neural Information Processing Systems, 33:20000–20011, 2020.

Aditya Mate, Jackson A. Killian, Haifeng Xu, Andrew Perrault, and Milind Tambe. Collapsing bandits and their application to public health interventions. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 15639–15650, 2020. Eli Meirom, Haggai Maron, Shie Mannor, and Gal Chechik. Controlling graph dynamics with reinforcement learning and graph neural networks. In International Conference on Machine Learning, pages 7565–7577. PMLR, 2021. Susan A Murphy, Mark J van der Laan, James M Robins, and Conduct Problems Prevention Research Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):1410–1423, 2001. Han-Ching Ou, Christoph Siebenbrunner, Jackson Killian, Meredith B. Brooks, David Kempe, Yevgeniy Vorobeychik, and Milind Tambe. Networked restless multi-armed bandits for mobile interventions. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’22, page 1001–1009. International Foundation for Autonomous Agents and Multiagent Systems, 2022. ISBN 9781450392136. Pradeep Ravikumar, Martin J. Wainwright, and John D. Lafferty. High-dimensional Ising model selection using ℓ1 -regularized logistic regression. The Annals of Statistics, 38(3):1287 – 1319, 2010. doi: 10.1214/09-AOS691. URL https://doi.org/10.1214/ 09-AOS691. Veronika Ročková and Edward I George. Emvs: The em approach to bayesian variable selection. Journal of the American Statistical Association, 109(506):828–846, 2014. Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315):1–20, 2022. Shashank Sheshar Singh, Divya Srivastva, Madhushi Verma, and Jagendra Singh. Influence maximization frameworks, performance, challenges and directions on social network: A theoretical study. Journal of King Saud University-Computer and Information Sciences, 34(9):7570–7603, 2022. Lichao Sun, Weiran Huang, Philip S Yu, and Wei Chen. Multi-round influence maximization. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2249–2258, 2018. Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. Sharan Vaswani, Laks Lakshmanan, Mark Schmidt, et al. Influence maximization with bandits. arXiv preprint arXiv:1503.00024, 2015. Davide Viviano. Policy targeting under network interference. Review of Economic Studies, 92(2):1257–1292, 2025. Davide Viviano and Jelena Bradic. Fair policy targeting. Journal of the American Statistical Association, 119(545):730–743, 2024.

Richard R Weber and Gideon Weiss. On an index policy for restless bandits. Journal of applied probability, 27(3):637–648, 1990. Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of applied probability, 25(A):287–298, 1988. ZR Yang. Glauber dynamics of the kinetic ising model. Physical Review B, 46(18):11578, 1992.

Tables

Table 1: Comparison of related methods. The check mark the property is satisfied, cross indicates it is not, and – indicates not applicable. Method

Network interference

Dynamic policy

Data is observational

Uncertainty quantification

Empirical welfare maximization Kitagawa and Tetenov [2018] Viviano [2025]

✗ ✓

✗ ✗

✓ ✓

✗ ✗

Topological policies Kempe et al. [2003] Banerjee et al. [2013] Chen et al. [2009] Liu et al. [2017]

✓ ✓ ✓ ✓

✗ ✗ ✗ ✗

– – – –

✗ ✗ ✗ ✗

Dynamic treatment regimes Chakraborty and Murphy [2014] Hu et al. [2025] Kitagawa et al. [2022]

✗ ✗ ✗

✓ ✓ ✓

✗ ✗ ✗

Network bandits Herlihy and Dickerson [2023] Vaswani et al. [2015]

✓ ✓

✗ ✗

GNN / simulation-based Manchanda et al. [2020] Meirom et al. [2021]

✓ ✓

✗ ✓

✗ ✗

Q-Ising

✓

Table 2: Mean reward (mean ± std) for each seeding policy across villages. Bolding indicates policies within 1 std of the best performer. All values rounded to three decimal places.

Village 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

Random

Degree

LIR

Degree-Bin

Plain DQN

Q-Ising

0.040 (0.002) 0.069 (0.002) 0.069 (0.002) 0.074 (0.002) 0.057 (0.002) 0.061 (0.002) 0.015 (0.001) 0.025 (0.001) 0.023 (0.001) 0.026 (0.001) 0.025 (0.001) 0.022 (0.001) 0.050 (0.003) 0.066 (0.003) 0.062 (0.003) 0.088 (0.002) 0.096 (0.003) 0.080 (0.003) 0.018 (0.001) 0.028 (0.001) 0.027 (0.001) 0.028 (0.001) 0.028 (0.001) 0.026 (0.001) 0.042 (0.002) 0.062 (0.002) 0.061 (0.002) 0.069 (0.002) 0.065 (0.002) 0.063 (0.002) 0.020 (0.001) 0.024 (0.001) 0.024 (0.001) 0.033 (0.001) 0.029 (0.001) 0.031 (0.001) 0.018 (0.001) 0.035 (0.001) 0.036 (0.001) 0.031 (0.001) 0.029 (0.001) 0.029 (0.001) 0.052 (0.003) 0.096 (0.003) 0.094 (0.002) 0.084 (0.003) 0.086 (0.003) 0.077 (0.003) 0.017 (0.001) 0.046 (0.001) 0.046 (0.001) 0.037 (0.001) 0.038 (0.001) 0.035 (0.001) 0.028 (0.001) 0.028 (0.001) 0.028 (0.001) 0.044 (0.001) 0.050 (0.001) 0.042 (0.001) 0.037 (0.002) 0.053 (0.002) 0.053 (0.002) 0.069 (0.002) 0.066 (0.002) 0.067 (0.003) 0.029 (0.002) 0.031 (0.001) 0.031 (0.001) 0.054 (0.002) 0.055 (0.002) 0.054 (0.002) 0.023 (0.001) 0.031 (0.001) 0.030 (0.001) 0.039 (0.001) 0.040 (0.001) 0.039 (0.001) 0.023 (0.001) 0.026 (0.001) 0.026 (0.001) 0.044 (0.001) 0.043 (0.002) 0.044 (0.002) 0.030 (0.002) 0.035 (0.001) 0.040 (0.002) 0.051 (0.002) 0.055 (0.001) 0.052 (0.002) 0.038 (0.002) 0.053 (0.002) 0.053 (0.002) 0.060 (0.002) 0.065 (0.002) 0.061 (0.002) 0.042 (0.003) 0.107 (0.003) 0.107 (0.003) 0.084 (0.003) 0.091 (0.003) 0.089 (0.003) 0.043 (0.002) 0.033 (0.001) 0.037 (0.001) 0.061 (0.001) 0.067 (0.002) 0.067 (0.001) 0.023 (0.001) 0.029 (0.001) 0.029 (0.001) 0.040 (0.001) 0.044 (0.001) 0.045 (0.001) 0.027 (0.001) 0.033 (0.001) 0.033 (0.001) 0.041 (0.001) 0.047 (0.001) 0.046 (0.001) 0.055 (0.004) 0.075 (0.003) 0.072 (0.004) 0.082 (0.003) 0.097 (0.003) 0.093 (0.003) 0.023 (0.001) 0.042 (0.001) 0.044 (0.001) 0.038 (0.001) 0.046 (0.002) 0.044 (0.002) 0.026 (0.001) 0.052 (0.002) 0.052 (0.001) 0.044 (0.002) 0.050 (0.002) 0.050 (0.002) 0.051 (0.002) 0.061 (0.002) 0.061 (0.002) 0.068 (0.002) 0.085 (0.002) 0.078 (0.002) 0.037 (0.002) 0.056 (0.002) 0.056 (0.002) 0.056 (0.002) 0.066 (0.002) 0.064 (0.002) 0.040 (0.003) 0.029 (0.001) 0.025 (0.001) 0.056 (0.002) 0.065 (0.002) 0.065 (0.002) 0.021 (0.001) 0.037 (0.001) 0.039 (0.001) 0.036 (0.001) 0.041 (0.001) 0.042 (0.001) 0.025 (0.001) 0.035 (0.001) 0.034 (0.001) 0.036 (0.001) 0.042 (0.001) 0.042 (0.001) 0.054 (0.003) 0.076 (0.002) 0.081 (0.002) 0.082 (0.002) 0.103 (0.003) 0.099 (0.002) 0.035 (0.002) 0.053 (0.001) 0.053 (0.001) 0.049 (0.002) 0.060 (0.002) 0.060 (0.001) 0.043 (0.002) 0.049 (0.001) 0.049 (0.001) 0.063 (0.002) 0.078 (0.002) 0.078 (0.001) 0.035 (0.002) 0.078 (0.002) 0.078 (0.002) 0.054 (0.002) 0.061 (0.002) 0.067 (0.002) 0.034 (0.001) 0.041 (0.001) 0.046 (0.002) 0.055 (0.002) 0.068 (0.001) 0.069 (0.002) 0.057 (0.003) 0.066 (0.002) 0.066 (0.002) 0.078 (0.002) 0.108 (0.002) 0.098 (0.003) 0.040 (0.002) 0.059 (0.002) 0.056 (0.002) 0.055 (0.002) 0.068 (0.002) 0.071 (0.002) 0.045 (0.002) 0.050 (0.001) 0.050 (0.001) 0.063 (0.001) 0.090 (0.002) 0.086 (0.002) 0.046 (0.002) 0.038 (0.001) 0.038 (0.001) 0.063 (0.002) 0.088 (0.002) 0.088 (0.002) 0.069 (0.003) 0.082 (0.002) 0.083 (0.002) 0.079 (0.002) 0.128 (0.002) 0.124 (0.002) 0.035 (0.001) 0.035 (0.001) 0.035 (0.001) 0.046 (0.001) 0.071 (0.002) 0.073 (0.002) 0.052 (0.002) 0.068 (0.002) 0.068 (0.002) 0.067 (0.002) 0.103 (0.002) 0.107 (0.002) 0.052 (0.002) 0.021 (0.001) 0.021 (0.001) 0.047 (0.001) 0.079 (0.001) 0.080 (0.001) 0.023 (0.001) 0.012 (0.000) 0.012 (0.000) 0.026 (0.001) 0.045 (0.001) 0.046 (0.001)

Additional Figures Intercept

3.5

1.50

Comm 0

2.5

1.25

2.0 1.5 1.0 0.5

Comm 1

0.0

0.50

0.10

0.25

0.05

2.5

0.00

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.75

2.5

1.50

2.0

1.25

1.5

1.00 0.75

1.0

0.50

0.5

0.25

0.0

0.00

1.5

2.0

2.5

3.0

1.0 0.5 0

0.0

1.0

3.0

3.5

4.0

0.0

0.5

1.0

1.5

2.0

0.5

2.5

0.4

2.0

0.3

1.5

0.2

1.0

0.1

0.5

2.5

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

2.5

0.6

0.8

1.2

1.4

0.5 2.0

2.0

0.4

1.5

0.3

1.0

0.2

0.5

0.1

0.0

1.5 1.0 0.5

0.0

4.0

7 8

Comm 4

0.5

2.5

1.5

0.0

2.5

3.0

2.5

0.3

4 4

2.0 0.2

3 2

1.0 0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

3.5

EMVS MAP

95% posterior interval

Posterior distributions couplings k, m (village 50, 200 draws)

Neighbor: Comm 0

1.5

0.1

Neighbor: Comm 1

Neighbor: Comm 2

2.0

6 5

2 1

0.0

0.2

1.5

1.0

0.5

0.1

0.0

0.1

0.0

0.2

0.0

0.5

1.0

0, 0

1.5

2.0

3.0

2.0

1.5

1.0

0.2

0.0

0.2

0.4

0.6

1.6

2.5

0, 1

1.0

Neighbor: Comm 4 3.5

3.5

0.5

0.4

Neighbor: Comm 3

4.0 3.0

4 3

3.0

3.5

0.4

Focal: Comm 0

0.0

2.0

0.00

1.4

1.5

2.5

0.15

0.05

1.0

3.0

0.20

0.10

0.5

3.5

0.25

0.5

0.0

4.0

2.00

3.0

Comm 3

0.25 0

0.30

1.0

0.0

0.50

0.35

1.5

0.75

0.00

2.0

1.00

0.15

3.0

1.25

0.25

0.75

0.30 0.20

3.5

0.35

1.50

1.00

0.00

Spillover 1.75

0.40

1.75

3.0

Comm 2

Posterior distributions(village 50, 200 draws) Persistence Direct treatment

2.00

0.0

0.2

0.0

0.2

0, 2

0.4

0.6

0.0

0.8

0.0

0.2

0.4

0, 3

0.6

0.8

1.0

1.2

1.4

0.05

0.10

0.15

0.20

1.0

1.2

1.4

1.6

2.0

2.5

0.4

0.5

0, 4

Focal: Comm 1

Focal: Comm 2

3 2

0.1

0.0

0.1

0.2

0.3

0.4

0.0

0.1

0.2

0.3

1, 0

1.0

0.3

0.4

0.5

0.0

0.6

0.0

0.2

0.4

0.6

0.8

2, 0

0.4

0.5

1.0

1.2

1.4

1.6

2.5 2.0 1.5 1.0

0.0

0.2

0.4

0.6

0.8

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

1.0

0.5

0.0

3, 0

0.5

1.0

4 3

0.1

0.0

0.1

0.2 4, 0

0.3

0.4

3.5

3.0

2.5

2.0

1.5

0.5

1.0

0.0

1.5

0.2

0.3 4, 1

0.4

0.5

0.6

0.0

0.2

0.4

0.6

0.8

0.8 0.6

0.2

0.0

0.2

0.4

0.6

0.4 0.2 0.0

0.2

0.4

0.6

0.8

1.0

0.4 4, 2

1.0

0.6

0.8

1.5

8 6 4

1 0.2

0.5

0.0

3, 4

0.2

0.0

3, 3

0.1

0.05 0.00

2, 4

1.0

0.0

0.10

0.5 0.0

0.1

0.15

1.0

0.0

0.2

0.20

1, 4

3.5

0 4.0

0.4

4.0

3, 2

0.2

1.2

0.0

3, 1

0.2

2, 3

0.5 0.0

0.4

0.5

3.0

0.6

2, 2

3.5

0.8

1, 3

4.0

1 1.0

1.0

2, 1

0.3

0.5

0.5 0.2

0.2

1.0

1.5

0.1

1.5

2.0

0.0

2.0

2.5

0.1

1, 2

3.0

0.2

2.5

3.5

4 3

1, 1

0.4

4.0

Focal: Comm 3

Focal: Comm 4

2 0.2

0.1

0.0 4, 3

0.1

0.2

0.0

0.1

0.2

0.3 4, 4

Figure 3: The posterior distribution of dynamic Ising parameters estimated by MCMC for Village 50. 19

Coupling heatmaps village 50

1.00

0.00

1.00

Comm 1

0.00

0.66

0.00

0.8 0.6

Comm 2

0.00

1.00

0.00

1.00

0.4 Comm 3

0.00

1.00

0.00

Comm 4

0.00

0.94

Comm

Neighbor community m

EMVS MAP

Focal community k

1.0

Comm

Focal community k

0.00

0.2 0.0

MCMC Inclusion Prob P(| | > )

Comm 0

0.00

1.00

0.03

0.04

0.49

Comm 1

0.01

0.89

0.03

0.00

Comm 2

0.02

1.00

0.37

0.07

1.00

Comm 3

0.01

0.06

0.01

1.00

0.93

Comm 4

0.01

0.05

0.10

0.00

0.96

1.0 0.8 0.6 0.4

Comm

Neighbor community m

Comm

0.2 0.0

Posterior Std ( k, m)

k, m

Comm 0

0.23

1.16

-0.02

0.20

1.02

1.0

Comm 0

0.08

0.24

0.13

0.16

0.43

Comm 1

0.09

0.40

0.02

0.03

0.12

0.5

Comm 1

0.10

0.02

0.13

0.17

0.09

Comm 2

0.14

1.30

0.13

0.05

1.11

0.0

Comm 2

0.11

0.17

0.21

0.30

0.16

Comm 3

0.01

-0.01

0.01

0.58

0.13

0.5

Comm 3

0.10

0.24

0.11

0.10

0.57

Comm 4

0.07

0.13

0.06

0.00

0.43

1.0

Comm 4

0.08

0.12

0.17

0.09

0.04

0.5 Focal community k

Focal community k

EMVS Inclusion Prob zk, m Comm 0

0.4 0.3 0.2 0.1

Comm

Neighbor community m

Comm

Neighbor community m

Comm

0.0

Figure 4: Estimated inclusion probabilities for coupling parameters for Village 50 by EMVS and MCMC, MAP estimates for the couplings and posterior standard deviations.

EMVS Group Ising: Macro-Averaged ROC (pooled across all villages and nodes) 1.0

True Positive Rate

0.8 0.6 0.4 0.2

Macro-avg ROC (AUC = 0.762) ±1 std across nodes Random classifier

0.0 0.0

0.2

0.4

0.6

False Positive Rate

0.8

1.0

Figure 5: The AUC of pooled nodes for the microfinance villages.

Ensemble decision tree majority-vote path (T=25, 20 agents, branch threshold > 0 votes) Village 50

[4] Comm 4 (20 nodes)

[3] Comm 3 (27 nodes)

[2] Comm 2 (29 nodes)

0.8

1.0

0.6 Adoption rate

Community

Majority vote (main path) Alternative (> 0 votes)

0.4 [1] Comm 1 (41 nodes)

0.2 14

[0] Comm 0 (106 nodes)

Time step

0.0

Figure 6: MCMC ensemble majority-vote path for Village 50. Bubble area encodes the number of agents voting for each action; orange bubbles mark alternatives receiving more than one vote. Concentrated bubbles in early periods reflect high posterior agreement on treating certain communities, dispersed votes later on signal near-equivalence of strategies once spreader communities saturated.

Proofs

C.1

Proof of Greedy Suboptimality (Proposition 1)

Proof. We provide a counterexample under basic SIS dynamics. Consider a graph G with nodes {A, B, C}. Nodes A and B are connected while C is isolated. The connected nodes {A, B} has spread rate ρ > 0 and a churn rate of 1, meaning that they stay adopted only for one period, while node C has a churn rate of 0, meaning it stays adopted once treated. Let’s assume the planner wants to maximize total adoption over T = 2 periods. A greedy policy would treat one of the connected nodes because the expected immediate reward for treating A is higher than treating C. E[RA,1 ] = 1 + ρ > E[RC,1 ] = 1. However, at t = 2 both of these nodes would churn unless they are re-seeded. The optimal policy treats C first which does not require any maintenance in future periods, and then treats any one of the connected nodes resulting in E[RC,1 + RA,2 ] = 1 + 1 + 1 + ρ = 3 + ρ. The greedy policy can achieve E[RA,1 + RB,2 ] = 2 + 2ρ. In this example, the greedy policy is suboptimal.

C.2

Proof of Dynamic Ising Remark

Remark 2 (Synchronous Updates Are Not Element-wise Representable). Consider the dynamic Ising model on N ≥ 4 nodes under no treatment (β2,i = 0) and symmetric interactions γij = γji . Under synchronous updating, the stationary distribution cannot in gen P P eral be expressed as an element-wise Gibbs measure µ(y) ∝ exp i hi yi + i<j βij yi yj . Proof. Since σ maps R to (0, 1), every transition probability is strictly positive, so the chain admits a unique stationary distribution µ. We give a counterexample to show that this stationary distribution cannot be in general represented as an element-wise Gibbs measure. Take N = 4 on the complete graph with β0,i = 0 and γij = 1 for all i ̸= j. Under synchronous updates, all nodes flip independently given the current state y, so the transition kernel factorizes: K(y → y ′ ) =

4 P yi′ Y 1−yi′ P σ y 1 − σ y j̸=i j j̸=i j i=1

where each factor is the probability that node i updates to yi′ given that it sees field P 4 j̸=i yj . This defines the 16 × 16 matrix K on {0, 1} , and µ is its unique left eigenvector with eigenvalue 1, normalized to a probability distribution. P By permutation symmetry, µ(y) depends only on the weight m = i yi ; write µm for this common value. Any element-wise Gibbs measure reproducing µ must share this symmetry, forcing uniform field h and uniform coupling β, so that µm ∝ exp hm+β m2 . Taking log-ratios, h i µm+1 m = h + β m+1 − = h + βm, ∆m := log 2 2 µm which is affine in m. Solving µK = µ numerically yields ∆0 ≈ 1.860,

∆1 ≈ 2.247,

∆2 ≈ 2.549,

∆3 ≈ 2.765,

with successive differences 0.387, 0.302, 0.216. These are not constant, so ∆m is not affine in m, and no element-wise Gibbs measure can reproduce µ. For N > 4, embed the four-node complete graph and set all remaining couplings to zero. The disconnected nodes evolve independently of the embedded clique, so the marginal of µ on the clique coincides with the four-node stationary distribution above. If µ were element-wise Gibbs on N nodes, the Hamiltonian would split, and marginalizing out the disconnected nodes would yield an element-wise Gibbs measure on the clique.

Regret Analysis

D.1

Notations Setup

We review and define additional notations that are useful for regret analysis. The main text states the guarantee for a compact PEVI analogue of Q-Ising. Here we make the PEVI recursion and the aggregate processes explicit. Oracle and estimated Q-Ising states Let Zτ,h denote the pre-action full-network history at stage h of effective training block τ . The oracle and estimated Q-Ising states are s⋆τ,h = ϕθ0 (Zτ,h ), and sbτ,h = ϕθb(Zτ,h ). For the concentration arguments below and in the lemma section, sbτ,h should be read as a cross-fitted state, and we suppress the fold index in the notation. This is a proof device. In practice, the empirical implementation may use the full first-stage fit. For any state-based policy π = (π1 , . . . , πH ), with πh : S → [K], its oracle and plug-in full-history deployments are πh◦ (z) = πh (ϕθ0 (z)) and πhplug (z) = πh (ϕθb(z)). The theorem in the main text analyzes oracle deployment. Effective sample size For regret analysis, we represent the offline panel as stage-wise transition sets Dh = {(b sτ,h , bτ,h , rτ,h , sbτ,h+1 )}τ ∈Ih ,

h = 1, . . . , H,

where τ indexes effective training blocks from the training panel. Let nh := |Ih | denote the effective number of approximately independent transitions at stage h, after accounting for temporal dependence. We write neff := min nh ,

nlog := max nh .

h∈[H]

The minimum neff controls simplified coverage rates, whereas nlog appears only inside the logarithmic confidence radius in the self-normalized regression bound. Under balanced blocking, nh = neff = nlog for all h. PEVI Regressions Review [Jin et al., 2021] Recall that φ(b sτ,h , bτ,h ) is a normalized feature map with ∥φ(s, b)∥2 ≤ 1. let [x][0,c] = min{max{x, 0}, c}. Setting VbH+1 = 0, PEVI computes, for h = H, . . . , 1, in a backward fashion X X Λh = λIdϕ + φ(b sτ,h , bτ,h )φ(b sτ,h , bτ,h )⊤ , w bh = Λ−1 φ(b sτ,h , bτ,h ){rτ,h + Vbh+1 (b sτ,h+1 )} h τ ∈Ih

τ ∈Ih

q −1 ⊤ bh (s, b) = φ(s, b) w Q bh − β φ(s, b)⊤ Λh φ(s, b)

bh (s, b). Vbh (s) = max Q

[0,H−h+1]

bh (s, b). Its oracle and plug-in fullThe learned PEVI policy is π bh (s) ∈ arg maxb∈[K] Q plug bh (ϕθ0 (z)) and π bh (z) = π bh (ϕθb(z)) respectively. history deployments are π bh◦ (z) = π Aggregate MDP and value-to-go Let MS be the target oracle aggregate MDP over Q-Ising states, where a chosen bin is implemented by uniform within-bin randomization. Let Mµ be the logged oracle aggregate MDP induced by the historical within-bin selector after node actions are mapped to bins. For M ∈ {MS , Mµ }, write rhM and PhM for the reward and transition kernel, and define (ThM V )(s, b) = rhM (s, b) + Es′ ∼PhM (·|s,b) [V (s′ )]. For M ∈ {MS , Mµ } and π ∈ ΠS , define the aggregate value-to-go by " H # X M,π VhM,π (s) := EπM ruM (su , bu ) sh = s , VH+1 = 0. u=h

When M = MS , we write VhS,π ; when M = Mµ , we write Vhµ,π . Finally, we note that the PEVI uncertainty of a policy π under the logged aggregate MDP is q H X µ −1 π φ(sh , bh )⊤ Λh φ(sh , bh ) . UD (π) = EMµ h=1

D.2

Regularity Assumptions

We state the regularity conditions used in the regret proof. In words, Assumption A1 handles causal validity and the effective blocked sample; Assumption A2 handles networkto-state abstraction and logged-versus-target within-bin selection; Assumption A3 handles first-stage Q-Ising estimation; and Assumption A4 handles approximate linear PEVI regression. For each stage h and each value function V appearing in the PEVI backward recursion, define the regression target Yτ,h (V ) := rτ,h + V (b sτ,h+1 ). The assumptions below imply that Yτ,h (V ) admits the decomposition lin θ Yτ,h (V ) = x⊤ τ,h wh (V ) + ξτ,h (V ) + eτ,h (V ) + eτ,h (V ),

(10)

where ξτ,h (V ) is the stochastic Bellman noise, elin τ,h (V ) is the Bellman linear-approximation θ residual, and eτ,h (V ) is the additional residual induced by using estimated Q-Ising states instead of oracle Q-Ising states. Assumption A1 (Causal validity and effective PEVI Samples). Logged actions are valid interventions. For each stage h, ah ⊥ {(Zh+1 (a), rh (a)) : a ∈ [N ]} | Zh ,

(Zh+1 , rh ) = (Zh+1 (ah ), rh (ah )).

The training panel can be represented by effective stage-wise blocks {Ih }H h=1 . For each stage h, after conditioning on the data used to construct the cross-fitted Q-Ising states and the next-stage PEVI value function Vbh+1 , there exists a filtration {Fτ,h }τ ∈Ih such that xτ,h := φ(b sτ,h , bτ,h ) is predictable and, for every value function V appearing in the PEVI recursion, E[ξτ,h (V ) | Fτ −1,h ] = 0, with ξτ,h (V ) conditionally H-sub-Gaussian. Assumption A2 (Abstraction and binning). Let ∥ · ∥TV denote total variation distance. Let PhZ,S (· | z, b) and rhZ,S (z, b) be the full-network transition law and reward under the target rule. Let ϕθ0 # PhZ,S (· | z, b) be the pushforward law of ϕθ0 (Zh+1 ). For all relevant h, z, b, rhZ,S (z, b) − rhS (ϕθ0 (z), b) ≤ εagg r ,

ϕθ0 # PhZ,S (· | z, b) − PhS (· | ϕθ0 (z), b)

≤ εagg P .

Moreover, the target and logged aggregate MDPs are close: sup |rhS (s, b) − rhµ (s, b)| ≤ εbin r ,

sup ∥PhS (· | s, b) − Phµ (· | s, b)∥TV ≤ εbin P .

h,s,b

If the historical policy also randomizes uniformly within the selected bin, then εbin = r bin εP = 0. Assumption A3 (First-stage Q-Ising stability). There exists an event Eθ , with P(Eθ ) ≥ 1 − δθ , such that on Eθ , simultaneously over all stages h, all value functions V appearing in the PEVI recursion, and all (s, b) ∈ S × [K], φ(s, b)⊤ Λ−1 h

xτ,h eθτ,h (V ) ≤ cθ Lst εθ .

(11)

τ ∈Ih

Here Lst is a stability constant controlling the sensitivity of features, PEVI value functions, and Bellman targets to perturbations of the Q-Ising state. Assumption A4 (Approximate linear Bellman regression). For every stage h and every value function V appearing in the PEVI recursion, there exists wh (V ) ∈ Rdϕ , ∥wh (V )∥2 ≤ W , such that sup (Thµ V )(s, b) − φ(s, b)⊤ wh (V ) ≤ εlin . s,b

In addition, the Bellman approximation residual is small after projection onto the empirical ridge geometry: sup φ(s, b)⊤ Λ−1 h s,b

xτ,h elin τ,h (V ) ≤ clin εlin .

τ ∈Ih

Exact linear MDPs satisfy this condition with εlin = 0. 24

(12)

Combining Assumption A3-A4 On the event Eθ , define θ eτ,h (V ) := elin τ,h (V ) + eτ,h (V ).

Combining Assumptions A3 and A4 gives, simultaneously over all stages h, all PEVI value functions V , and all (s, b), φ(s, b)⊤ Λ−1 h

xτ,h eτ,h (V ) ≤ capp (εlin + Lst εθ ) ,

(13)

τ ∈Ih

where capp = clin + cθ . In summary, Assumption A1 is the causal and statistical validity condition needed to treat the logged blocks as usable PEVI regression samples. The cross-fitting convention makes the estimated states sbτ,h predictable with respect to the Bellman noise, and the stage-wise conditioning handles the randomness of Vbh+1 . Assumption A2 elaborates the Q-Ising network-condition that explains the cost of abstractions and binning. Assumption A3 gives the first-stage stability condition and defines such event. Finally, Assumption A4 is a projected misspecification condition, which requires the Bellman residual to be small in the PEVI ridge regression.

D.3

Auxiliary Lemmas

Lemma 3 (Finite-horizon simulation lemma). Consider two finite-horizon MDPs with the same state space, action space, horizon, and initial state x1 : m H Mm = (X , A, {Phm }H h=1 , {rh }h=1 , H),

m ∈ {1, 2},

Further suppose that the rewards are bounded in (0, 1) and for all h, x, a, |rh1 (x, a) − rh2 (x, a)| ≤ εr ,

∥Ph1 (· | x, a) − Ph2 (· | x, a)∥TV ≤ εP

Then, for any Markov policy π = (π1 , . . . , πH ), |V11,π (x1 ) − V12,π (x1 )| ≤ Hεr + H 2 εP . Proof. This is a standard result in offline reinforcement learning (see e.g., Abbeel and Ng [2005]). For completeness, we provide a proof tailored to our MDP structures. For m,π m ∈ {1, 2}, let Vhm,π be the value-to-go of policy π in Mm with VH+1 ≡ 0. Define ∆h := sup |Vh1,π (x) − Vh2,π (x)|. x

Recall that for the total variation distance and for any measurable function f satisfying 0 ≤ f ≤ B, we have |EP [f ] − EQ [f ]| ≤ B∥P − Q∥TV . Since rewards are in [0, 1], 0 ≤ 2,π Vh+1 (x′ ) ≤ H − h. Hence we have ∆h ≤ εr + ∆h+1 + (H − h)εP . Since ∆H+1 = 0, unrolling the expression above from H to 1 gives H X H(H − 1) εP ≤ Hεr + H 2 εP . ∆1 ≤ Hεr + (H − h)εP = Hεr + 2 h=1

Lemma 4 (Q-Ising abstraction transfer). Under Assumption A2, for every policy π ∈ ΠS whose induced trajectory is covered by the local bounds in Assumption A2, WHZ (π ◦ ; Z1 ) − V1S,π (s⋆1 ) ≤ ∆agg , V1S,π (s⋆1 ) − V1µ,π (s⋆1 ) ≤ ∆bin , 2 bin + H 2 εagg and ∆bin := Hεbin where ∆agg := Hεagg r + H εP . Consequently, we have r P

WHZ (π ◦ ; Z1 ) − V1µ,π (s⋆1 ) ≤ ∆abs , where ∆abs := ∆agg + ∆bin . Proof. To see the first statement, recall that for π ∈ ΠS , its oracle full-history lift is πh◦ (z) = πh (ϕθ0 (z)). That is, the full-network policy and the aggregate policy make the same bin decision whenever the aggregate state is s = ϕθ0 (z). Under the target rule, choosing bin b in the full network means drawing ah ∼ Unif(Bb ). On the other hand, the resulting full-network transition is PhZ,S (· | z, b), with the induced next Q-Ising state has law as ϕθ0 # PhZ,S (· | z, b). By Assumption A2, for all relevant h, z, b, we have rhZ,S (z, b) − rhS (ϕθ0 (z), b) ≤ εagg and ϕθ0 # PhZ,S (· | z, b) − PhS (· | ϕθ0 (z), b) ≤ εagg r P . ApTV plying the finite-horizon simulation lemma 3 to the pushed-forward full-network process and the target aggregate MDP yields the first claim. The second statement is more straightforward since both MS and the logged aggregate MDP Mµ share the same state space S, action space [K], and horizon H. The statement follows by the second statement in Assumption A2 and by applying the simulation Lemama 3 again. Lemma 5 (Self-normalized PEVI Noise). For each h ∈ [H], define X X Sh := xτ,h ξτ,h , Λh = λIdϕ + xτ,h x⊤ τ,h . τ ∈Ih

τ ∈Ih

Suppose ∥xτ,h ∥2 ≤ 1, xτ,h is predictable with respect to the stage-h sample filtration, and ξτ,h is conditionally mean-zero and H-sub-Gaussian. Then, with probability at least 1 − δ, simultaneously for all h ∈ [H], s H(1 + nlog /λ) ∥Sh ∥Λ−1 ≤ CH dϕ log , (14) h δ 26

where C > 0 is a universal constant. Consequently, for every x = φ(s, b), s q H(1 + nlog /λ) ⊤ −1 x⊤ Λ−1 x Λh Sh ≤ CH dϕ log h x. δ

(15)

Proof. Fix a stage h. By the standard self-normalized martingale inequality (see Theorem 1 of Abbasi-Yadkori et al. [2011]), we have for any δh ∈ (0, 1), s det(Λh )1/2 ∥Sh ∥Λ−1 ≤ H 2 log h det(λIdϕ )1/2 δh with probability at least 1 − δh . Taking δh = δ/H and applying the union bound gives the same display simultaneously over all stages with probability P at least 1 − δ. It remains to simplify the determinant term. Let Ah := τ ∈Ih xτ,h x⊤ τ,h so that Λh = λIdϕ + Ah . Denote ν1 , . . . , νdϕ as the eigenvalues of Ah . It then follows that Y dϕ det(Λh ) Ah νj 1+ = det Idϕ + = . det(λIdϕ ) λ λ j=1 Since ∥xτ,h ∥2 ≤ 1, we have dϕ X

νj = tr(Ah ) =

∥xτ,h ∥22 ≤ nh ≤ nlog .

τ ∈Ih

j=1

Pn Q aj ≥ ( nj=1 aj )1/n for nonnegative aj . An Recall that the AM-GM inequality gives j=1 n ν application of AM-GM inequality with aj := λj yields dϕ Y j=1

νj 1+ ≤ λ

It follows that

nh 1+ λdϕ

dϕ

d nlog ϕ ≤ 1+ . λdϕ

s ∥Sh ∥Λ−1 ≤ H h

nlog dϕ log 1 + + 2 log(H/δ). λdϕ

Absorbing constants in the log gives s ∥Sh ∥Λ−1 ≤ CH h

H(1 + nlog /λ) dϕ log . δ

This proves the first statement. The second statement follows by Cauchy-Schwarz: q ⊤ −1 ⊤ −1/2 −1/2 |x Λh Sh | = x Λh Λh Sh ≤ x⊤ Λ−1 h x ∥Sh ∥Λ−1 . h

Combining the inequality above with the first statement proves the result.

Lemma 6 (PEVI Bellman confidence). Suppose Assumptions A1, A3, and A4 hold with the residual conditions described in (10)-(13). Define q Γh (s, b) := β φ(s, b)⊤ Λ−1 h φ(s, b). Choose

" s

H(1 + nlog /λ) dϕ log δ

β = Cβ H

√

λW

(16)

for a sufficiently large universal constant Cβ . Then, with probability at least 1 − δ − δθ , for every h ∈ [H] and every (s, b) ∈ S × [K], φ(s, b)⊤ w bh − (Thµ Vbh+1 )(s, b) ≤ Γh (s, b) + ζ,

(17)

ζ = Capp (εlin + Lst εθ )

(18)

where for a universal constant Capp > 0. Proof. Let ESN be the self-normalized concentration event from Lemma 5, and Eθ be the event on which the projected first-stage residual bound (11) holds. We will work on ESN ∩ Eθ . Fix a stage h. P For notational simplicity, write xτ := xτ,h = φ(b sτ,h , bτ,h ), x := φ(s, b), ⊤ and Λh = λIdϕ + τ ∈Ih xτ xτ . Recall from the regression decomposition in (10) that we have lin θ Yτ,h = rτ,h + Vbh+1 (b sτ,h+1 ) = x⊤ τ wh + ξτ,h + eτ,h + eτ,h , θ where we write V = Vbh+1 and wh := wh (Vbh+1 ). Let eτ,h := elin τ,h +eτ,h , then by the ridge reP −1 gression formula, we have w bh = Λh τ ∈Ih xτ Yτ,h . Substituting the target decomposition yields X X −1 −1 w bh − wh = −λΛ−1 w + Λ x ξ + Λ xτ eτ,h . h τ τ,h h h h τ ∈Ih

τ ∈Ih

It follows that ⊤ −1 |x⊤ (w bh − wh )| ≤ λ|x⊤ Λ−1 h w h | + x Λh

xτ ξτ,h + x⊤ Λ−1 h

τ ∈Ih

xτ eτ,h .

τ ∈Ih

We will bound these three terms separately. Specifically, the first term can by bound by Cauchy-Schwarz and the fact that Λh ⪰ λIdϕ : q q ⊤ −1/2 −1/2 ⊤ Λ−1 x w ⊤ Λ−1 w , w | = λ x Λ Λ w ≤ λ (19) λ|x⊤ Λ−1 x h h h h h h h h h −1 2 wh⊤ Λ−1 h wh ≤ λ ∥wh ∥2 ,

which implies λ|x

⊤

Λ−1 h wh | ≤

(20) √

q λW x⊤ Λ−1 h x.

For the second term, we apply Lemma 5 to obtain s q X H(1 + nlog /λ) ⊤ −1 x Λh xτ ξτ,h ≤ CH dϕ log x⊤ Λ−1 h x. δ τ ∈I h

We bound the last term by (12) and (11), which gives x⊤ Λ−1 h

xτ eτ,h ≤ x⊤ Λ−1 h

τ ∈Ih

⊤ −1 xτ elin τ,h + x Λh

τ ∈Ih

xτ eθτ,h

τ ∈Ih

≤ clin εlin + cθ Lst εθ ≤ capp (εlin + Lst εθ ) . Combining the last three displays and choosing Cβ sufficiently large in (16) yields |x⊤ (w bh − wh )| ≤ Γh (s, b) + capp (εlin + Lst εθ ) . By assumption A4, we have x⊤ wh − (Thµ Vbh+1 )(s, b) ≤ εlin . Combining this with the equation above, and absorbing constants into Capp gives φ(s, b)⊤ w bh − (Thµ Vbh+1 )(s, b) ≤ Γh (s, b) + Capp (εlin + Lst εθ ) . Finally, the probability 1 − δ − δθ statement follows by union bound. Remark for Lemma 7 : The following lemma is proved entirely inside the oracle µ H aggregate MDP Mµ = (S, [K], {Phµ }H h=1 , {rh }h=1 , H), in which the nodes does not necessarily have to be selected uniformly from the bin. For a bounded function V : S → [0, H], recall that the Bellman equation gives (Thµ V )(s, b) = rhµ (s, b) + Es′ ∼Phµ (·|s,b) [V (s′ )]. Additionally, the value-to-go in Mµ is given by " H # X µ,π Vhµ,π (s) = EπMµ ruµ (su , bu ) sh = s , VH+1 := 0. u=h µ,π Thus, if b = πh (s), the policy Bellman equation is Vhµ,π (s) = (Thµ Vh+1 )(s, b).

Lemma 7 (PEVI Optimality in Aggregate MDP). Assume the PEVI Bellman confidence event (17) from Lemma 6 holds. Then for every comparator policy π ∈ ΠS , we have V1µ,π (s⋆1 ) − V1µ,bπ (s⋆1 ) ≤ 2β UµD (π) + 2Hζ, (21) q P H µ −1 π bh (s, b) and U (π) = φ(sh , bh )⊤ Λh φ(sh , bh ) . where π bh (s) ∈ arg maxb∈[K] Q D h=1 EMµ Consequently, with ζ = Capp (εlin + Lst εθ ), we have V1µ,π (s⋆1 ) − V1µ,bπ (s⋆1 ) ≤ 2β UµD (π) + CH (εlin + Lst εθ ) . Proof. First observe that to show (21), it is sufficient to prove the following two inequalities: V1µ,π (s⋆1 ) − Vb1 (s⋆1 ) ≤ 2β UµD (π) + Hζ. Vb1 (s⋆ ) − V1µ,bπ (s⋆ ) ≤ Hζ. 1

(22) (23)

First Inequality Fix a comparator policy π ∈ ΠS and define Dhπ (s) := Vhµ,π (s) − Vbh (s). bh (s, a) and V µ,π (s) = (T µ V µ,π )(s, b), we have Dπ (s) ≤ (T µ V µ,π )(s, b)− Since Vbh (s) = maxa∈[K] Q h h h+1 h h h+1 µb b Qh (s, b). Adding and substracting Th Vh+1 )(s, b) on the right hand side gives h i bh (s, b) + Es′ ∼P µ (·|s,b) [Dπ (s′ )]. Dhπ (s) ≤ (Thµ Vbh+1 )(s, b) − Q h+1 h h i bh (s, b) , recall that (1) the Bellman confidence event To upper bound (Thµ Vbh+1 )(s, b) − Q bh (s, b) = from Lemma 6 gives φ(s, b)⊤ w bh − (Thµ Vbh+1 )(s, b) ≤ Γh (s, b) + ζ, and (2) that Q bh (s, b) is not clipped, it is straightforward to see φ(s, b)⊤ w bh − Γh (s, b) . If Q [0,H−h+1]

combining these two facts yields bh (s, b) ≤ 2Γh (s, b) + ζ. (Thµ Vbh+1 )(s, b) − Q bh (s, b) is clipped at 0 or It is also easy to verify the inequality above still holds even if Q at (H − h + 1). It follows that we have π Dhπ (s) ≤ 2Γh (s, πh (s)) + Es′ ∼Phµ (·|s,πh (s)) [Dh+1 (s′ )] + ζ.

(24)

We then unroll (24) along the trajectory generated by policy π in Mµ . Starting at µ,π := 0 and VbH+1 := 0, we have s⋆1 , let bh = πh (sh ), sh+1 ∼ Phµ (· | sh , bh ). Since both VH+1 π DH+1 = 0. Repeatably applying (24) from h = 1 to H gives V1µ,π (s⋆1 ) − Vb1 (s⋆1 ) = D1π (s⋆1 ) ≤ 2

H X

EπMµ [Γh (sh , bh )] + Hζ.

h=1

The first inequality therefore follows by recalling the expression of UµD . Second Inequality To prove the second inequality, it is sufficient to prove that for every h ∈ [H + 1] and every s ∈ S, we have Vbh (s) − Vhµ,bπ (s) ≤ (H − h + 1)ζ. We will prove this via backward induction. It is clear to see the base case H + 1 holds µ,b π because VbH+1 = VH+1 = 0. Suppose this pattern holds at stage h + 1. Fix s ∈ S, and b bh (s, bb) by exact greediness. Again recall the Bellman let b = π bh (s). Note that Vbh (s) = Q confidence event from Lemma 6: φ(s, b)⊤ w bh − (Thµ Vbh+1 )(s, b) ≤ Γh (s, b) + ζ and the fact bh (s, b) = φ(s, b)⊤ w that Q bh − Γh (s, b) . Combining these two facts gives [0,H−h+1]

bh (s, bb) ≤ (T µ Vbh+1 )(s, bb) + ζ. Q h bh (s, bb), expanding the right side with the Bellman equation gives Given that Vbh (s) = Q Vbh (s) ≤ rhµ (s, bb) + Es′ ∼P µ (·|s,bb) [Vbh+1 (s′ )] + ζ. h

µ,b π ′ ≤ rhµ (s, bb) + Es′ ∼P µ (·|s,bb) [Vh+1 (s )] + (H − h)ζ + ζ, h

= Vhµ,bπ (s) + (H − h + 1)ζ. where the second inequality is due to induction hypothesis. Evaluating at h = 1 finishes the proof for the second inequality. 30

D.4

Proof of Theorem 2

r √ H(1+nlog /λ) + λW so that we can work on the PEVI Proof. Let β = Cβ H dϕ log δ Bellman confidence event Econf in Lemma 6, where P(Econf ) ≥ 1 − δ − δθ . We will work on this event throughout the proof. We first apply the first part of Lemma 4 to obtain: ⋆ S,πS

WHZ ((πS⋆ )◦ ; Z1 ) ≤ V1

(s⋆1 ) + ∆agg ,

WHZ (b π ◦ ; Z1 ) ≥ V1S,bπ (s⋆1 ) − ∆agg , which implies that ⋆ S,πS

WHZ ((πS⋆ )◦ ; Z1 ) − WHZ (b π ◦ ; Z1 ) ≤ V1

(s⋆1 ) − V1S,bπ (s⋆1 ) + 2∆agg .

Furthermore, an application of the second part of Lemma 4 yields: ⋆ S,πS

⋆ µ,πS

(s⋆1 ) ≤ V1

(s⋆1 ) + ∆bin ,

V1S,bπ (s⋆1 ) ≥ V1µ,bπ (s⋆1 ) − ∆bin . It follows that ⋆ µ,πS

WHZ ((πS⋆ )◦ ; Z1 ) − WHZ (b π ◦ ; Z1 ) ≤ V1

(s⋆1 ) − V1µ,bπ (s⋆1 ) + 2∆agg + 2∆bin .

Since we work on the event Econf , we can apply Lemma 7 to obtain ⋆ µ,πS

(s⋆1 ) − V1µ,bπ (s⋆1 ) ≤ 2β UµD (πS⋆ ) + CH (εlin + Lst εθ ) .

Recall that ∆abs = ∆agg + ∆bin . Combining the last two inequalities with sufficient large constant C yields WHZ ((πS⋆ )◦ ; Z1 ) − WHZ (b π ◦ ; Z1 ) ≤ 2β UµD (πS⋆ ) + C [∆abs + Hεlin + HLst εθ ] . This finishes the proof.

Experimental Set-up Details

E.1

SIS Dynamics

We model information diffusion on a fixed, undirected graph G = (V, E) with |V | = n nodes. Each node i ∈ V is at every period in one of two states: susceptible (S) or infected/adopted (I). The planner selects a bin at ∈ {0, . . . , K − 1} at each period t, and the environment transitions through three ordered sub-steps: churn, seeding, and spreading. Each currently adopted node i independently reverts to susceptible with a nodespecific probability δi ∈ (0, 1), capturing heterogeneous loss of interest or product abandonment. After churn, one susceptible node is drawn uniformly at random from the chosen bin at and forced to adopt (treatment is “perfect,” i.e. adoption occurs with probability one). Every currently adopted node i (including the newly seeded one) independently attempts to transmit adoption to each susceptible neighbor j ∈ N (i). Transmission from node i to node j succeeds with a node-specific probability βi ∈ (0, 1). Because j may have multiple infected neighbors, the probability that j adopts in this period is Y padopt = 1 − (1 − βi ). (25) j i∈N (j), i adopted

E.2

CQL Design Choices

We use the CQL algorithm implemented by Seno and Imai [2022]. The learning rate for both the encoder and Q-network is 3 × 10−4 . Batch normalisation is applied within the encoder, together with a dropout rate of 0.3. Early stopping monitors the per-epoch TD loss and halts training if improvement is smaller than 10−4 for 10 consecutive epochs (patience = 10, min_delta = 10−4 ). For hyperparameters of the CQL we use [256, 256] hidden layers, the batch size is 64, and the maximum allowed number of steps is 30,000 while, steps per epoch are 1000. For conservative penalty we use α = 0.1 and for discount we use ψ = 0.8 for all the experiments. All the experiments are seeded appropriately for reproducibility.

E.3

Microfinance Villages Experiment Further Details

In this experiment, the planner aims to learn a dynamic policy for selecting which communities (“bins”) within a village to target. These communities can be interpreted as clusters of households (e.g., friend groups) obtained via edge-betweennes based community detection. We use the implementation in igraph library in Python. To avoid very small groups, any identified community with fewer than 10 households is merged into the largest community. 42 of the 43 microfinance villages has at least 2 identifies communities and we experiment in these 42 villages. Some community examples and the distribution of households in all the villages can be seen in Figure 7

Figure 7: Top: Community detection examples from Indian Village dataset. Bottom: Distribution of identified community numbers and average degree per community Each adopted household independently influences each of its neighbors to adopt with a community-specific probability. The spread parameters are set (in order of community size) to [0.01, 0.5, 0.05, 0.07, 0.06, 0.02, 0.01, 0.4, 0.1, 0.3], and the corresponding churn rates (probability that an adopted household stops using the product in a period) are [0.5, 0.9, 0.9, 0.6, 0.5, 0.5, 0.7, 0.6, 0.5, 0.8]. This environment induces substantial heterogeneity across communities, including both supercritical and subcritical diffusion regimes, as well as relatively high churn rates. As a result, learning an effective policy requires balancing expansion into new communities with maintenance of existing adoption. For each village, the planner is provided with 500 periods of historical data generated by a random policy that selects communities uniformly at random. The evaluation (test) phase begins from a zero-adoption state. This can be interpreted as observing past adoption data from a similar product and then deploying a new, related product using a learned policy. The test horizon is 25 periods, and performance is measured by the average adoption rate per period over this horizon. Each test is independently ran 50 times to produce accurate standard deviation estimates. 33

The ensemble alternative for this experiment uses the same hyperparameters as defined in Section 3. 200 MCMC draws are used for posterior parameter estimation, with 300 tune-in iterations. For the ensemble policy 20 agents are trained.

Record · ID 168332 · SHA-256 3adf45781d3ee1bc
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.