Kernel weighted importance sampling for off-policy evaluation in contextual bandits

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Kernel weighted importance sampling for off-policy evaluation in contextual bandits

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

Kernel weighted importance sampling for off-policy evaluation in contextual bandits Joshua Spear1 , Matthieu Komorowski2 , Rebecca Pope1 , Neil J Sebire3,4,5 , and Erica E.M. Moodie6

arXiv:2607.15067v1 [cs.LG] 16 Jul 2026

Institute of Child Health, University College London {joshua.spear.21,r.pope}@ucl.ac.uk 2 Strive Health Ltd. [email protected] 3 National Institute for Health Research 4 University College London, Biomedical Research Council 5 Great Ormond Street Hospital [email protected] 6 Department of Epidemiology, Biostatistics and Occupational Health, McGill University [email protected]

Abstract. This article presents a novel estimator for performing off-policy evaluation using only offline data for contextual bandits. The proposed estimator, Kernel-WIS is demonstrated to be asymptotically consistent and to empirically outperform strong baselines (including vanilla weighted importance sampling), particularly under complex conditions including behaviour policy miss-specification. The benefit of Kernel-WIS is derived from combining the bounded property of vanilla weighted importance sampling with the linearity of vanilla importance sampling. Keywords: Off-policy evaluation, contextual bandits

Introduction

Off-policy evaluation (OPE) for contextual bandits, or more generally time independent causal inference, requires evaluating the performance of a target policy πe using data observed from a (logging) policy πl . If interest is in the expected value of this performance (i.e., the average treatment effect, average treatment effect in the treated etc), then the OPE task requires approximating EPπe (τ ) [R] (or a conditional version) with data generated according to Pπl , where R is the outcome and Pπ is the probability measure associated with the random variable (policy), π : S → A. S defines the set of states or contexts over which the agent acts and A defines the set of actions that the agent can take. S and A define random variables with ranges in S and A, respectively, with associated push through measures implied by the context. Unless otherwise stated, capital letters will refer to random variables (S), capital P subscript will refer to the probability measure associated to the random variable (PS ) and lowercase p will refer to the density (pS ). Throughout the analysis the following data generating density is

Authors Suppressed Due to Excessive Length

assumed: pπ (τ ) = pS (s)pπ (a|s)pR (r|a, s)

(1)

The causal estimand of primary interest for this analysis is the average treatment return in the untreated i.e.,: E[Rdo(A=πe (S)) |A = πβ (S)] (2) where Rdo(A=π(S)) defines the counterfactual reward R when the intervention is set by the policy π. For counterfactual policy optimisation, the selected policy (πe ) can depart quite significantly from the logging policy used to generate the data (πl ), particularly in the time dependent setting. As such, the causal estimand is understanding ”how the new policy would have performed on a set of patients who have not been treated under the new policy”. Throughout, the causal identifiability assumptions of no hidden confounding, consistency and positivity are assumed (Hernán & James (2020)) such that: # " pπe (A|S) do(A=πe (S)) R E[R |A = πβ (S)] = EPπl (τ ) pπβ (A|S) With respect to OPE, the core approaches are outcome and propensity methods, with the addition of composite and doubly robust methods requiring the two aforementioned approaches. Equations 3 and 4 broadly define the propensity and outcome based methods mentioned, where pπβ (ai |si ) and R̂(πe (si ), si ) define the estimated versions of the assumed models for each estimator, respectively. Throughout, the approximate logging policy, pπβ (ai |si ), will be referred to as the ”behaviour policy”. " EPπe [R] = EPπl (τ )

# n 1 X pπe (ai |si ) pπe (A|S) R ≈ ri pπl (A|S) n i=1 pπβ (ai |si )

(3)

EPπe [R] ≈

1X R̂(πe (si ), si ) n i=1

(4)

Note that, the actions fed to the approximate reward function in equation 4 are the actions under the evaluation policy whilst the actions fed to the densities in equation 3 are actions under the logging policy. For clarity, the logging policy actions are used to fit the approximate outcome model, R̂, in equation 4.

1.1

Why importance sampling?

Within the AI community importance sampling estimators have been largely disregarded, for time-dependent OPE. There exists a plethora of research into importance sampling for the time independent setting however, the limitations of these will be discussed later. More generally, within the causal inference community, it is widely understood that robust evaluation of causal effects requires both propensity and outcome

1. INTRODUCTION

based methods: methods incorporating both outcome and propensity models are doubly robust (Wallace & Moodie (2015)); assessing the extent to which propensity and outcome measures agree can provide insight into bias or model misspecification (Wallace et al. (2017)) and; propensity based methods can be used as a diagnostic for bias (Rubin (2001)).

1.2

Types of importance sampling estimators

The estimator in equation 3 defines the vanilla IS (VIS) estimator however, more often than not, the weighted IS (WIS) estimator is used: n

JˆWIS =

1X wi n i=1

!−1

1X ri wi n i=1

(5)

The core challenge in the use of IS methods is the variance of the estimator and the use of the WIS estimator is motived through this variance reduction. The specific form of P the estimator is motivated through the use of control variates. The denominator n 1 i=1 wi converges to 1 in the limit of data (n) and thus the bias of the estimator n vanishes, under positivity and correct identification of the propensity score model.

1.3

Related work

There exists a large amount of research into time independent OPE methods. The focus of the analysis presented is with respect to propensity based methods and as such existing work will focus on these estimators. 1.3.1 Monte Carlo importance sampling Propensity based methods are largely motivated by the Monte Carlo importance sampling (MCIS) literature. MCIS is concerned with evaluating an integral with respect to a target measure, using an alternate base measure (Owen (2013)). This formulation precisely maps to equation 3 and thus much of the concepts are transferable. The core difference between OPE and MCIS is in the assumption of access to data. Many techniques for controlling the variance of IS based methods within the context of MCIS are motivated through selecting an appropriate base measure e.g., selecting a base measure with heavy tails to prevent importance weights from diverging to ∞. In the context of OPE, this would require selecting a heavy tailed logging density p̄πl . However, in the context of OPE, data cannot be observed from this heavy tailed density since data is only available from pπl .

1.3.2 Importance sampling in OPE Aside from the WIS estimator, the most prominent approach for variance reduction in importance sampling is clipping (Alquier et al. (2024)). Alquier et al. (2024) demonstrated that a number of other variance reduction

Authors Suppressed Due to Excessive Length

methods can be viewed under the same theoretical framework, where clipping is a special case. Deriving probably approximately correct (PAC) estimators for conservative OPE measures i.e., asserting P (Jˆ − J < ϵ) ≥ 1 − δ has formed the other large body of research. Alquier et al. (2024) succeeded in this for the vanilla IS estimator however, the only existing PAC based analysis of the WIS estimator for the average treatment return in the untreated (equation 2) is Kuzborskij et al. (2021). Kuzborskij et al. (2021) proposed a conservative PAC bound for off-policy selection, a simpler problem then OPE. A potential reason for the limited WIS PAC research is the strong dependence of random variables within the WIS estimator. Despite the significant progress in understanding vanilla IS based estimators (and associated variance reducing alternatives), the estimator remains challenging to use for OPE due to it being unbounded. Kallus & Uehara (2019) highlight this and include αboundedness as a finite sample condition for a well behaved OPE estimator. The WIS estimator naturally induces boundedness however, the dependence structure means the resulting estimator is challenging to control.

1.4

Contributions

The primary contribution of this work is the Kernel-WIS estimator, a novel estimator which attempts to unify the benefits of both vanilla importance sampling (independence) and weighted importance sampling (boundedness and variance reduction) into a single estimator. Kernel-WIS is shown to be asymptotically consistent under mild conditions (appropriate bandwidth schedule, identification of the propensity model and positivity). The Kernel-WIS estimator is demonstrated to, generally speaking, (statistically) significantly outperform the WIS estimator under non-oracle behaviour policies and, generally speaking, perform (statistically) identically to the WIS estimator under oracle behaviour policies. However, under continuous reward settings the Kernel-WIS estimator (statistically) significantly underperforms in comparison to the WIS estimator. In almost all scenarios assessed, the Kernel-WIS estimator demonstrates smaller or equivalent approximate variance in comparison to the WIS estimator.

Methods

The proceeding introduces the Kernel-WIS estimator for policy evaluation. The estimator is motivated by combining the best of the vanilla and weighted estimators, linearity/independence and boundedness, respectively. Before discussing Kernel-WIS, the State dependent-WIS (State-WIS) estimator is discussed. The State-WIS estimator is also novel to the analysis presented and can be derived from Kernel-WIS in the limit. The results in section 3.2 will demonstrate that State-WIS is overly biased but it is instructive for understanding Kernel-WIS.

2. METHODS

2.1

State-dependent WIS estimator

The State-WIS estimator, JˆState-WIS , is defined as: !−1 n X X 1 1 JˆState-WIS = ri wi wj 1(sj = si ) n i=1 nsi j=1 where nsi =

j=1

(6)

1(sj = si ).

No formal statement is made regarding the Pnconsistency of the State-WIS estimator. Intuitively though, the denominator n1s j=1 wj 1(sj = si ) is clearly an estimai tor for the conditional expectation E[W |S = si ] and, on its own is consistent, assuming positivity and correct identification of the propensity model. Thus the State-WIS estimator can be viewed as using conditional control variates since, E[W |S = si ] = EPπe [1|S = si ] = 1. Additionally, observe that for si ̸= sj , the terms of State-WIS are independent.

2.2

Kernel WIS estimator

The Kernel-WIS estimator, JˆKernel-WIS , is defined as: !−1 Pn X kh (si , sj )wj 1 j=1 Pn ri wi JˆKernel-WIS = n i=1 j=1 kh (si , sj )

(7)

where kh : S × S → R≥0 is a kernel function under assumptions 1 to 8 (below). The parameter h is known as the bandwidth which controls the smoothness of the kernel function. The form of the estimator can be motivated from two points of view. Firstly, the denominator defines a kernel estimate of E[W |S = si ] and thus Kernel-WIS with a given bandwidth could be viewed as an estimator of State-WIS. Alternatively, if h = g(n, λ) is allowed to tend to 0 or ∞, it is hypothesised that bandwidth can be optimised as a hyperparameter (through λ) to control finite sample performance, such that WIS (with h = ∞) or State-WIS (with h = 0) is attained in the limit. The consistency results (theorem 1) are obtained with the limit tending towards State-WIS.

2.3

Theoretical results

The proceeding demonstrates the validity of the Kernel-WIS estimator in terms of asymptotic behaviour. To obtain these results, the Kernel-WIS estimator is viewed as an empirical process of random functions (Vaart (1998)). Before proceeding, the assumptions of the analysis are stated. The implications and relevance of these assumptions are described in table 9 (appendix section A). A kernel, kh (x) is any measurable function which satisfies:

Authors Suppressed Due to Excessive Length

1. ∥kh (x)∥ < ∞ 1 2. For some C > 0 and ν > 0, N (ϵ, K) ≤ Cϵ−ν where K = {kh ((x − ·)/h− d ) : d h > 0, x ∈ R } 3. K is pointwise measurable; 4. The support of kh is [−0.5, 0.5]d ; R k′ (x) 5. kh (x)d(x) = 1 and are of the form: kh (x) = E[kh′ (x)] where limx→∞ kh′ (x) = 0 h and kh′ (x) > 0 for finite x Additional assumptions include: 6. Assume f defines the marginal density of S and, f is uniformly Lipschitz continuous and strictly positive on I ϵ , where I is a compact set in Rd and I ϵ = {max1≤i≤d |si | ≤ ϵ}. 7. There exists wmax such that almost surely, |W |1(S ∈ I ϵ ) ≤ wmax and W ≥ wmin > 0 8. R ∈ [0, 1] 9. Correct identification of the propensity model i.e., p̂πl = pπl Throughout, ∥ · ∥A = supA | · |. Many of the assumptions are provided by Einmahl & Mason (2005), since the results are used extensively. Theorem 2 from Einmahl & Mason (2005) is heavily relied upon and is reproduced in theorem 3. Theorem 1 provides the novel result from this analysis with the proof provided in appendix section A. Theorem 1 (Asymptotic consistency of Kernel-WIS). Under assumptions 1 to 9, for nan 0 < an < h < bn < 1, bn → 0 and log n → ∞, the Kernel-WIS estimator is asymptotically consistent: a.s. JˆKernel-WIS −−→ E[R]

where: 1X JˆKernel-WIS = ri wi n i=1 2.4

j=1 kh (si , sj )wi P n j=1 kh (si , sj )

!−1 .

Bandwidth selection

To perform bandwidth selection, the denominator term,

Pn j=1 kh (si ,sj )wj P , is treated as n j=1 kh (si ,sj )

a Nadaraya–Watson kernel regressor of E[W |S] and cross-validation is performed. With respect to the overall goal of the Kernel-WIS estimator, this criteria is tangential and it is anticipated that optimising a PAC-Bayes lower bound would be more fruitful. Regardless, the proposed cross-validation approach acts as a first step. Consider a dataset, d of n tuples of the form (si , ai , ri ) and randomly split into two subsets dtrain of size ntrain and dtest of size ntest . Dataset dtrain is used for training the

2. METHODS

behaviour policy πβ as well as performing bandwidth selection via cross validation. Recall the denominator of the Kernel-WIS estimator aims to approximate the expectation, E[W |S]. This can be used to define a supervised signal in the cross-validation procedure. The algorithm for performing this is defined in algorithm 2.

Algorithm 1 K-fold cross validation for a fixed bandwidth, h n

train Require: Dataset dtrain = {si , wi }i=1 , number of folds K, a bandwidth h 1: Randomly shuffle dataset dtrain 2: Split dtrain into K disjoint folds {D1 , D2 , . . . , DK } 3: Initialize list of scores S ← [ ] 4: for k = 1 to K do 5: Initialize list of scores Sk ← [ ] 6: d′test ← dS k 7: d′train ← j̸=k dj 8: for i = 1 to nPtest do kb (si,test −sj )wj j∈d 9: ŵi,test = P train kib (si,test −sj ) j∈dtrain

10: Append (ŵi,test − wi )2 to S 11: end for P 12: Append n1test i∈Sk i to S 13: end for P 1 14: return K i∈S i

Algorithm 2 Bandwidth selection with K-fold cross validation n

train Require: Dataset dtrain = {si , wi }i=1 , number of folds K, a set of bandwidth initialisations h = {h1 , ..., hm } and a minimiser g 1: Sbest = 1e32 , hbest = 0 2: for hi ∈ h do 3: S̄, h′i ← g(Algorithm 1, bi , dtrain , K) 4: if S̄ < Sbest then 5: Sbest ← S̄ 6: hbest ← h′i 7: end if 8: end for

For the experiments defined in section 3, the minimiser in algorithm g was the L-BFGSB algorithm, implemented in scipy via the minimise function (Virtanen et al. (2020)). An analytic expression for the gradient (described in proceeding section 2.4.1) was provided assuming the kernel function, kh was a radial basis function (RBF) kernel: kh,RBF = exp(−0.5h−2 sT s′ )

Authors Suppressed Due to Excessive Length

where s and s′ are vectors of dimension l and hi is a real value (≥ 0) or a vector of real values (≥ 0). Experiments were initially run by specifying only a single bandwidth initialisation i.e. h = {h0 } in algorithm 2. It was observed however, that the optimisation by g alone was insufficient as solutions rarely escaped the locality of the initial bandwidth h0 . As such, several bandwidth initialisations h = {h1 , ..., hm } were introduced. Introducing the sweep over different initialisations consistently produced improved improved solutions to algorithm 1 with respect to the bandwidth i.e., improved approximations to E[W |S = s].

2.4.1 Derivative of Nadaraya–Watson kernel regressor with RBF kernel The proceeding describes the analytic gradient for the Nadaraya–Watson kernel regressor with RBF kernel function, with respect to the bandwidth, assuming it is a real value (not a vector). The derivative assuming a vector of unique bandwidths is almost identical and is provided in appendix section B. The gradient of interest is with respect to log h, since the optimisation is performed over log h to circumvent the need for explicit constraints. To begin, observe the following decomposition by repeated application of the chain rule: ∂MSE(h) ∂MSE(h) = ∂ log h ∂h

∂ log h ∂h

!−1

∂MSE(h) ∂ ŷ(si , h) = ∂ ŷ(si , h) ∂h

∂ log h ∂h

!−1

∂MSE(h) ∂u(wi , si , ŷ) ∂ ŷ(si , h) = ∂u(wi , si , ŷ) ∂ ŷ(si , h) ∂h

∂ log h ∂h

for: 1X (yi − ŷi )2 n u(wi , si , ŷ) =(wi − ŷi (si )) n n X −1 X ŷ(si , h) = k(h, si , sj )wj k(h, si , sj ) MSE(h) =

j=1

k(h, si , sj ) = exp(−0.5h−2 ||si − sj ||2 ) X ||si − sj ||2 = (si,k − sj,k )2 k

!−1 (8)

2. METHODS

∂MSE(h) ∂ log h −1 i ,si ,ŷ) Of the terms in equation 8, ∂u(w , ∂u(w are trivial to define ∂ ŷ(si ,h) and ( ∂h ) i ,si ,ŷ) and are listed below:

n n 2X 2X ∂MSE(h) = u(wi , si , ŷ) = (wi − ŷi (si )) ∂u(wi , si , ŷ) n i=1 n i=1

∂u(wi , si , ŷ) =−1 ∂ ŷ(si , h) !−1 ∂ log h =h ∂h

∂ ŷ(si ,h) , however, requires a little more care. First let: ∂h

f (si , h) = g(si , h) =

n X j=1 n X

k(h, si , sj )wj k(h, si , sj )

j=1

and thus: n ∂f (si , h) X −3 h ||si,l − sj,l ||2 k(h, si , sj )wj = ∂h j=1 n ∂g(si , h) X −3 h ||si,l − sj,l ||2 k(h, si , sj ) = ∂h j=1

By the quotient rule:

∂ ŷ(si , h) = ∂h =

∂f (si ,h) i ,h) g(si , h) − ∂g(s f (si , h) ∂h ∂h g(si , h)2 Pn −3 ||si,l − sj,l ||2 k(h, si , sj )wj j=1 h

g(si , h) −3 2 h ||s i,l − sj,l || k(h, si , sj ) j=1

Pn − =

n X j=1

g(si , h)

ŷ(si , h)

g(si , h)−1 h−3 ||si,l − sj,l ||2 k(h, si , sj )(wj − ŷ(si , h))

Authors Suppressed Due to Excessive Length

As such: ∂MSE(h) ∂ log h n n X 2X (wi − ŷi (si ))(−1)h = g(si , h)−1 h−3 ||si,l − sj,l ||2 k(h, si , sj )(wj − ŷ(si , h)) n i=1 j=1 =

n n X −2 X (wi − ŷi (si ))g(si , h)−1 h−2 ||si,l − sj,l ||2 k(h, si , sj )(wj − ŷ(si , h)) n i=1 j=1

(9) Where ||x − x′ ||2 =

′ 2 k (xk − xk )

Regularised derivative If MSE′ (h) = n1

P (yi − ŷi )2 + λh−1 , then:

∂MSE′ (h) ∂MSE′ (h) = ∂ log h ∂h

∂ log h ∂h

!−1

∂MSE(h) ∂λh−1 = + ∂h ∂h

∂ log h ∂h

!−1

∂λh−1 = −λh−2 and therefore: ∂h

∂MSE′ (h) ∂ log h n n X −2 X (wi − ŷi (si ))g(si , h)−1 h−2 ||si,l − sj,l ||2 k(h, si , sj )(wj − ŷ(si , h)) − λh−1 n i=1 j=1

Experiments

The performance of the Kernel-WIS estimator was evaluated under 10 datasets. A semi-simulated setup was used, in the sense that, contextual bandit problem definitions were artificially created such that a ground truth evaluation of policy performance was available, but utilising existing ”real world” datasets. This ensured as much of the simulation was as realistic as possible such as: noise to signal rations and covariate relationships. The proceeding describes the process of generating the contextual bandit setups (section 3.1) and presents the results (section 3.2).

3.1

Dataset generation and description

Let d define an offline dataset where d defines a set of size n of tuples (si , ai )ni=1 . With respect to the original use case of d, all of the datasets considered for this analysis were focused on supervised learning. As such, in the context of supervised

3. EXPERIMENTS

learning, the datasets defined tuples of independent random variables Si ∈ S with an associated ground truth label, Ai ∈ A. All of the datasets were taken from multi-class classification problems and thus, A defined a finite set of sequentially ordered integers, beginning from 0 {0, ..., K − 1}, of size K. As an example, the optdigits dataset includes 64 × 64 pictures of hand drawn numbers with labels defining the digit in the drawing. As such, the set of actions is {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. Let π : S → A define a policy and define a reward function R(ai , π(si )) = 1(ai = π(si )) and thus the supervised learning problem be reformulated as a contextual Pcan n bandit problem where the aim is to maximise n1 i=1 R(ai , π(si )). Given the supervised learning to contextual bandit setup, different policies were specified to assess the Kernel-WIS estimator under different simulated scenarios.

3.1.1 Policy specification Policies were defined in two ways, in line with existing work (Kuzborskij et al. (2021)): as Gibbs measures over the dataset or; learnt by maximising an approximation of the reward. The Gibbs measure policies provided more fine grained control over the policy definition whilst, the learnt policies were deemed to better represent the structure of evaluation policies encountered in real-world applications.

Gibbs measure policies Gibbs measure policies were defined by first specifying two hyperparameters: the temperature τ and the set of faulty actions. The temperature parameter controlled the concentration of the resulting policy around the greedy action |s) ′ i.e., the magnitude of the ratio ppππ(a(amax ′ |s) where amax = argmaxa pπ (a|s) and a is all |s) actions not equal to amax . A lower value of τ corresponded to higher value of ppππ(a(amax ′ |s) i.e., a more concentrated policy or a policy that was more confident regarding the greedy action. The faulty actions, fa, defined the set of actions for which the policy, π was suboptimal. Formally, the deterministic Gibbs measure policy was defined as:

πGibbs (si ) = ai 1(ai ∈ / fa) + (ai + 1

mod K)1(ai ∈ fa)

(10)

where K defines the dimension of the action space. In words, the selected action is defined as equal to the optimal action in the dataset if the optimal action is not a member of the set of the faulty actions. Otherwise, the ai + 1 action is selected except for if ai = K − 1 in which case the greedy action is defined as the first action, 0. This is reason for the modulus function. The density of the Gibbs measure policy, for a given action aj conditional on the state si was defined as: exp(1(aj = πGibbs (si ))τ −1 ) pπGibbs (aj |si ) = P −1 ) k exp(1(ak = πGibbs (si ))τ

Authors Suppressed Due to Excessive Length

The stochastic Gibbs measure policy was defined by randomly sampling an action according to pπGibbs (aj |si ). Learnt policies Learnt policies were exclusively used for evaluation policies. These were derived by maximising (via gradient ascent for 20 epochs) the approximate expected return under the density defined in equation 11, with respect to a set of action wise parameters. The approximate expected return was derived using the vanilla importance sampling estimator or the weighted importance sampling estimator, for the IS and SNIS policies, respectively. exp(βjT si τ −1 ) plearnt (aj |si ) = P T −1 ) k exp(βk si τ

(11)

where τ defines the data independent temperature, fixed a priori.

3.1.2 Dataset description The datasets used for the analysis were downloaded from the OpenML repository (Bischl et al. (2025)). A selection of the datasets used in previous work pertaining to importance sampling estimators for contextual bandits (Kuzborskij et al. (2021)) were selected and are denoted by the Prev. Work column in table 1. In order to push further the evaluation, four additional datasets were included. These were deemed more complex owing to the larger action and state dimensions. Table 1 provides an overview of the datasets used, including the state and action dimensions and number of observations. Logging and behaviour policy definitions The column Faulty actions in table 1 defines the different faulty actions considered for the logging policy on each dataset. For example, under the arrhythmia dataset, an experiment was ran where the logging policy was optimal (fa = {}), and where the logging policy made errors on states with correct action equal to 0 i.e., fa = {0} etc. For each dataset, the set of faulty actions considered was based on the action dimension. Broadly speaking, for datasets with action dimension 15 and under, faulty actions up to and including length 5 were considered (e.g., 0,1,2,3,5 for arrhythmia). For datasets with larger action dimensions, a faulty action of length 6 was additionally included. This was deemed to reasonably trade-off the computation complexity of the experiments with reasonable coverage. Note, under the experimental scenarios used for this analysis, it took up to 24 hours for the suite of experiments to run for some datasets using a 2021 16 inch MacBook Pro with 32GB ram and an Apple M1 Pro CPU. For each faulty action set, an experiment with temperature equal to 0.3 and 0.5 was included. Therefore, for each dataset, ’faulty action sets×{0.3, 0.5}’ different logging policies were defined. Dataset generation and reward noise Under a given experimental setting i.e., logging policy temperature and faulty actions, evaluation policy temperature, dataset and behaviour policy specification, datasets were generated by randomly sampling the logging

3. EXPERIMENTS

Dataset

State dim.

Action Size dim.

arrhythmia

279

soybean

35 (132)

micro-mass

Faulty actions

Prev. Work

1,754

None; 0; 1; 0,1; 0,1,2; 0,1,2,3; 0,1,2,3,4; 0,1,2,3,5

683

None; 0; 1; 0,1; 0,1,2; 0,1,2,4; 0,1,2,4,6; 0,1,2,4,6,8

1,300

571

None; 0; 1; 0,1; 0,1,2; 0,1,2,4; 0,1,2,4,6; 0,1,2,4,6,8

optdigitsDist

5,620

None; 0; 1; 0,1; 0,1,2; 0,1,2,3; 0,1,2,3,4

optdigits

5,620

None; 0; 1; 0,1; 0,1,2; 0,1,2,3; 0,1,2,3,4

Yes

yeast

1,484

None; 0; 1; 0,1; 0,1,2; 0,1,2,3; 0,1,2,3,4

Yes

page-blocks

5,473 None; 0; 1; 0,1; 0,1,2

Yes

pendigits

10,992

None; 0; 1; 0,1; 0,1,2; 0,1,2,3; 0,1,2,3,4

Yes

letter

20,000

None; 0; 1; 0,1; 0,1,2; 0,1,2,4; 0,1,2,4,6; 0,1,2,4,6,8

Yes

kropt

6 (40)

28,056

None; 0; 1; 0,1; 0,1,2; 0,1,2,4; 0,1,2,4,6; 0,1,2,4,6,8

Yes

Table 1: The table displays summary statistics for the datasets used in the evaluation of the KernelWIS estimator. The values in brackets next to state dimensions of the soybean and kropt datasets define the dimensions after one-hot encoding categorical variables.

policy for each state in the dataset and inducing reward noise by the following process:

1(U > 0.1)R(si , πl (si )) + (1 − 1(U > 0.1))(1 − R(si , πl (si ))) where U is a uniform random variable between 0 and 1 and R(·, ·) is the deterministic reward function described at the beginning of section 3.1. In words, 1(U > 0.1) is a Bernoulli random variable with probability 0.9. And thus, with probability 0.1, the reward is flipped.

3.2

Authors Suppressed Due to Excessive Length

Results

The proceeding describes the results of a number of different experimental settings evaluating the point wise predictive performance and tail behaviour of the Kernel-WIS estimator. To assess the relative performance of the Kernel-WIS estimator against the WIS estimator, a two-sided Wald t-test (with unequal variances) was used. Where AD(J) is the absolute difference in performance under estimator J (equation 12), for the KernelWIS estimator the t-test assesses: H0 :E[AD(WIS)] = E[AD(Kernel-WIS)] H1 :E[AD(WIS)] ̸= E[AD(Kernel-WIS)] As such, a p-value of less than or equal to 0.05 suggests that the performance of the WIS estimator is not equal to the Kernel-WIS estimator, at the 95% level. The absolute difference in performance under estimator J is defined as: n

AD(J) =

1X R(πe (si ), si ) − J({si , πβ (si ), E[R(πβ (si ), si )|si ]}ni=1 ) n i=1

(12)

For all results reported, the best performing estimator is highlighted in bold. Estimators that performed within 0.001 of the mean of the best estimator have also been highlighted in bold, where 0.001 was selected as this represents 0.1% of the theoretical maximum reward. The only exception to this is for the p-values in tables 4 and 6 where those values less than or equal to 0.05 have been highlighted.

3.2.1 Natural bias of the Kernel-WIS variants The Kernel-WIS variants naturally bias towards predicting the observed return of the dataset. Consider for a large statespace (i.e., continuous), the probability of observing any identical state is 0. As such, as the bandwidth tends towards 0: n X

kh (si , sj ) →1

j=1 n X

kh (si , sj )wj →wi

j=1 n 1 X wi JˆKernel-WIS → ri n i=1 wi

This is the price being paid for independence between observations but retaining empirical boundedness. Whilst not unreasonable, the observation does alter the interpretation of results i.e., it is both instructive and important with respect to bias (of the interpretation of results), to consider the performance of the Kernel-WIS variants at different divergences in evaluation/logging policy performance.

3. EXPERIMENTS

3.2.2 Overall results Table 2 describes the overall performance of Kernel-WIS estimator against a number of baselines, split by dataset and experimental setting (reward structure and behaviour policy type). The ’Single action’ reward structure aligns to that described in section 3.1 and the definition of ’Continuous’ reward structure is described in the proceeding section 3.2.5. The ’Oracle’ policy setting assumes that the propensity weights are generated under the same logging policy as that which created the dataset whilst the ’Non-Oracle’ setting requires the policy generating the weights to be different i.e., miss-specified. This is described in more detail in the proceeding section 3.2.4. Based on the results in table 2, the relative performance of the Kernel-WIS estimator is mixed. However, this will be discussed in further detail since the results in table 2 are aggregated over a huge array of different scenarios which warrant a more detailed examination. The inference that should be drawn from table 2 is the relative performance of the Kernel-WIS estimator against the VIS and CLPD VIS estimators. In a number of scenarios the VIS and CLPD VIS estimators are optimal however, it is widely understood that in many real-world scenarios, these estimators exhibit unworkable variance. Whilst this does not nullify the results, it does suggest that more challenging benchmarks are required. Table 3 presents the mean 75th and 100th percentile prediction under each of the estimators. These results support the empirical α-boundedness property of the Kernel-WIS estimator but also demonstrate that for scenarios were the vanilla estimator does have large variance (all of the non-oracle settings), the Kernel-WIS estimator performs optimally. The results in table 2 additionally demonstrate the consistent sub-optimality of the State-WIS estimator. This is important to note given the observation in section 3.2.1 i.e., that the Kernel-WIS estimator is naturally biased towards the performance of the logging policy. The sub-optimality of the State-WIS estimator demonstrates that the experimental scenarios used are sufficiently complex such that a naive and strict application of this bias (i.e., just use the logging policy return) is insufficient to obtain good performance and thus the bandwidth selection procedure is truly improving the resulting estimate.

Authors Suppressed Due to Excessive Length

Reward structure

Behaviour policy

Single action Single action Single action Single action Single action Single action Single action Single action Single action

Oracle Oracle Oracle Oracle Oracle Oracle Oracle Oracle Oracle

Dataset

VIS WIS State-WIS CLPD VIS Kernel-WIS

yeast 0.045 0.046 soybean 0.068 0.063 page-blocks 0.07 0.064 pendigits 0.038 0.037 micro-mass 0.061 0.061 arrhythmia 0.074 0.059 optdigits 0.041 0.040 kropt 0.038 0.037 letter 0.033 0.032

0.14 0.166 0.332 0.201 0.156 0.115 0.199 0.146 0.14

0.043 0.056 0.07 0.038 0.055 0.06 0.0410 0.038 0.033

0.043 0.062 0.064 0.038 0.062 0.054 0.04 0.033 0.031

Single action Non-Oracle yeast 0.413 0.142 Single action Non-Oracle soybean 0.308 0.15 Single action Non-Oracle page-blocks 1.638 0.168 Single action Non-Oracle pendigits 0.443 0.134 Single action Non-Oracle arrhythmia 0.515 0.139 Single action Non-Oracle optdigits 0.432 0.137 Single action Non-Oracle micro-mass 0.291 0.15

0.148 0.159 0.312 0.172 0.119 0.174 0.152

0.384 0.251 1.638 0.443 0.367 0.432 0.237

0.117 0.117 0.157 0.114 0.105 0.138 0.136

optdigitsDist 0.144 0.143

0.16

0.144

0.157

Continuous

Oracle

Table 2: The table describes the pointwise finite mean squared error of the Kernel-WIS estimator (with a shared bandwidth) against various baselines, split by dataset, whether an oracle behaviour policy was used and the reward function structure. Aggregations for each row are across logging policy temperature, logging policy faulty action and policy type.

3. EXPERIMENTS

Reward structure

Behaviour policy

Single action Single action Single action Single action Single action Single action Single action Single action Single action

Oracle Oracle Oracle Oracle Oracle Oracle Oracle Oracle Oracle

Dataset

VIS

WIS

State-WIS CLPD VIS Kernel-WIS

75 100 75 100 75 100 75 100 75 yeast 0.52 0.73 0.51 0.72 0.53 0.70 0.52 0.73 0.51 soybean 0.49 0.59 0.49 0.59 0.53 0.59 0.49 0.59 0.47 page-blocks 0.73 0.81 0.75 0.81 0.62 0.80 0.73 0.81 0.75 pendigits 0.73 0.81 0.75 0.81 0.58 0.71 0.73 0.81 0.74 micro-mass 0.48 0.61 0.49 0.59 0.53 0.58 0.48 0.60 0.47 arrhythmia 0.36 0.70 0.34 0.68 0.35 0.68 0.31 0.70 0.32 optdigits 0.71 0.85 0.71 0.84 0.59 0.71 0.71 0.85 0.71 kropt 0.46 0.60 0.47 0.60 0.52 0.60 0.46 0.60 0.46 letter 0.48 0.52 0.48 0.52 0.48 0.53 0.48 0.52 0.47

100 0.72 0.59 0.81 0.81 0.58 0.68 0.84 0.60 0.52

Single action Non-Oracle yeast 0.80 6.41 0.64 0.87 0.53 0.70 0.79 6.24 0.57 Single action Non-Oracle soybean 0.82 2.85 0.57 0.85 0.54 0.60 0.68 2.00 0.51 Single action Non-Oracle page-blocks 1.66 20.67 0.77 0.90 0.62 0.80 1.66 20.67 0.68 Single action Non-Oracle pendigits 1.15 3.54 0.74 0.89 0.58 0.71 1.15 3.54 0.65 Single action Non-Oracle arrhythmia 0.62 11.23 0.43 0.92 0.35 0.68 0.57 5.74 0.35 Single action Non-Oracle optdigits 1.13 3.43 0.73 0.89 0.59 0.71 1.13 3.43 0.62 Single action Non-Oracle micro-mass 0.79 2.40 0.56 0.83 0.53 0.58 0.67 1.84 0.52

0.83 0.80 0.88 0.86 0.89 0.89 0.78

Continuous

Oracle

optdigitsDist 0.64 0.79 0.64 0.75 0.63 0.74 0.64 0.79 0.63 0.75

Table 3: The table describes the 75th and 100th percentile predictions of the Kernel-WIS estimator (with a shared bandwidth) against various baselines, split by dataset, whether an oracle behaviour policy was used and the reward function structure. Aggregations for each row are across logging policy temperature, logging policy faulty action and policy type.

Authors Suppressed Due to Excessive Length

3.2.3 Oracle behaviour policy results The results in table 4 describe the performance of the Kernel-WIS estimator against the vanilla WIS estimator for oracle behaviour policies. Given the bias described in 3.2.1, the results are split by dataset, policy type and the deviation in performance under the evaluation and logging policies. From the perspective of statistical significance, the results in table 4 demonstrate that the WIS and Kernel-WIS estimators perform similarly. On only 11 occasions does the performance of the two estimators differ significantly and on all but one of these occasions the Kernel-WIS estimator outperforms the WIS estimator. Ignoring statistical significance, in terms of the number of scenarios (i.e., rows in the table) where one estimator outperforms the other, the WIS estimator is more optimal. However, in terms of the median performance across datasets (i.e., aggregating the rows for each dataset), the Kernel-WIS estimator is more optimal. These results are shown more clearly in figures 1 and 2, respectively, and furthermore, it is evident from these plots that the relative performance is strongly dependent on sample size (plotted on the x-axis).

Eval/Log Abs. Diff.

Dataset

Policy Type

arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia

Gibbs Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW SNIW

(-0.2, 0.0] 0.037±0.016 0.037±0.016 (0.0, 0.2] 0.046±0.014 0.042±0.018 (0.2, 0.4] 0.055±0.02 0.062±0.017 (0.4, 0.6] 0.064±0.012 0.08±0.036 (0.0, 0.2] 0.087±0.012 0.069±0.028 (0.2, 0.4] 0.086±0.009 0.079±0.004 (0.0, 0.2] 0.093±0.006 0.075±0.011 (0.2, 0.4] 0.073±0.003 0.074±0.002 (0.4, 0.6] 0.111±0.001 0.111±0.001 (0.6, 0.8] 0.092±0.001 0.092±0.001

1.00 0.06 0.27 0.15 0.00 0.17 0.00 0.50 1.00 1.00

16 114 18 12 28 4 24 4 2 2

1.0 0.06 0.27 0.15 0.0 0.17 0.0 0.5 1.0 1.0

kropt kropt kropt kropt kropt kropt kropt kropt

Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.032±0.019 0.032±0.019 (0.0, 0.2] 0.033±0.018 0.033±0.017 (0.2, 0.4] 0.029±0.017 0.03±0.017 (0.0, 0.2] 0.047±0.008 0.046±0.006 (0.2, 0.4] 0.049±0.003 0.03±0.011 (0.0, 0.2] 0.049±0.006 0.026±0.017 (0.2, 0.4] 0.053±0.002 0.049±0.004 (0.4, 0.6] 0.053±0.0 0.046±0.0

1.00 0.89 0.74 0.50 0.00 0.00 0.00 0.00

16 80 56 20 12 20 10 2

1.0 0.89 0.74 0.5 0.0 0.0 0.0 0.0

letter letter letter letter letter letter letter

Gibbs Gibbs Gibbs IW IW SNIW SNIW

(-0.2, 0.0] 0.034±0.026 0.034±0.026 (0.0, 0.2] 0.033±0.025 0.033±0.025 (0.2, 0.4] 0.034±0.024 0.034±0.024 (0.0, 0.2] 0.024±0.004 0.023±0.004 (0.2, 0.4] 0.022±0.003 0.022±0.003 (0.0, 0.2] 0.032±0.007 0.028±0.011 (0.2, 0.4] 0.039±0.014 0.038±0.015

1.00 0.97 0.99 0.41 0.89 0.12 0.95

16 60 76 16 16 24 8

1.0 0.97 0.99 0.41 0.89 0.12 0.95

micro-mass

Gibbs

(-0.2, 0.0] 0.045±0.02

1.00

1.0

WIS

Kernel-WIS P-Value N Obs.

0.045±0.02

Continued on next page

3. EXPERIMENTS

Dataset

Policy Type

Eval/Log Abs. Diff.

micro-mass micro-mass micro-mass micro-mass micro-mass micro-mass

Gibbs Gibbs IW SNIW SNIW SNIW

(0.0, 0.2] (0.2, 0.4] (0.0, 0.2] (0.0, 0.2] (0.2, 0.4] (0.4, 0.6]

0.052±0.015 0.051±0.015 0.05±0.019 0.057±0.018 0.073±0.024 0.071±0.026 0.098±0.004 0.097±0.004 0.095±0.007 0.095±0.007 0.1±0.007 0.1±0.007

0.68 0.04 0.84 0.86 0.99 1.00

68 68 32 14 10 8

0.68 0.04 0.84 0.86 0.99 1.0

optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits

Gibbs Gibbs Gibbs IW IW IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.027±0.013 0.027±0.013 (0.0, 0.2] 0.026±0.011 0.026±0.011 (0.2, 0.4] 0.027±0.011 0.027±0.011 (0.0, 0.2] 0.076±0.011 0.077±0.01 (0.2, 0.4] 0.068±0.006 0.068±0.005 (0.4, 0.6] 0.075±0.004 0.075±0.004 (0.6, 0.8] 0.075±0.0 0.081±0.002 (0.0, 0.2] 0.066±0.007 0.069±0.005 (0.2, 0.4] 0.076±0.006 0.08±0.006 (0.4, 0.6] 0.071±0.006 0.071±0.006

1.00 0.97 0.99 0.94 0.92 0.90 0.10 0.42 0.32 0.94

14 74 44 5 9 12 2 8 5 15

1.0 0.97 0.99 0.94 0.92 0.9 0.1 0.42 0.32 0.94

page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks

Gibbs Gibbs Gibbs Gibbs Gibbs Gibbs IW IW IW IW IW SNIW SNIW SNIW SNIW

(-0.2, 0.0] 0.067±0.024 0.067±0.024 (0.0, 0.2] 0.075±0.017 0.074±0.017 (0.2, 0.4] 0.047±0.022 0.048±0.023 (0.4, 0.6] 0.055±0.026 0.055±0.026 (0.6, 0.8] 0.076±0.011 0.077±0.01 (0.8, 1.0] 0.08±0.014 0.085±0.007 (0.0, 0.2] 0.048±0.013 0.048±0.014 (0.2, 0.4] 0.075±0.003 0.075±0.003 (0.4, 0.6] 0.049±nan 0.049±nan (0.6, 0.8] 0.066±0.012 0.068±0.014 (0.8, 1.0] 0.077±nan 0.078±nan (0.0, 0.2] 0.036±0.005 0.037±0.006 (0.2, 0.4] 0.031±0.002 0.023±0.001 (0.4, 0.6] 0.044±0.002 0.056±0.014 (0.6, 0.8] 0.054±0.014 0.052±0.013

1.00 0.92 0.96 0.98 0.90 0.72 0.98 1.00 NaN 0.63 NaN 0.89 0.05 0.27 0.78

10 36 6 20 18 2 6 2 1 10 1 6 2 3 9

1.0 0.92 0.96 0.98 0.9 0.72 0.98 1.0 nan 0.63 nan 0.89 0.05 0.27 0.78

pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits

Gibbs Gibbs Gibbs IW IW IW IW SNIW SNIW SNIW SNIW

(-0.2, 0.0] 0.028±0.014 0.028±0.014 (0.0, 0.2] 0.026±0.012 0.025±0.012 (0.2, 0.4] 0.027±0.012 0.027±0.012 (0.0, 0.2] 0.071±0.003 0.071±0.003 (0.2, 0.4] 0.062±0.004 0.065±0.002 (0.4, 0.6] 0.065±0.005 0.067±0.004 (0.6, 0.8] 0.062±0.002 0.062±0.002 (0.0, 0.2] 0.05±0.005 0.057±0.009 (0.2, 0.4] 0.045±0.005 0.068±0.022 (0.4, 0.6] 0.064±0.003 0.069±0.009 (0.6, 0.8] 0.064±nan 0.064±nan

1.00 0.85 0.98 0.81 0.10 0.28 1.00 0.10 0.08 0.06 NaN

14 78 40 6 6 14 2 8 5 14 1

1.0 0.85 0.98 0.81 0.1 0.28 1.0 0.1 0.08 0.06 nan

soybean soybean

Gibbs Gibbs

(-0.2, 0.0] 0.044±0.02 0.044±0.02 (0.0, 0.2] 0.049±0.016 0.048±0.019

1.00 0.72

16 68

1.0 0.72

WIS

Kernel-WIS P-Value N Obs.

Continued on next page

Authors Suppressed Due to Excessive Length

Dataset

Policy Type

Eval/Log Abs. Diff.

soybean soybean soybean soybean soybean soybean

Gibbs IW IW SNIW SNIW SNIW

(0.2, 0.4] (0.0, 0.2] (0.2, 0.4] (0.0, 0.2] (0.2, 0.4] (0.4, 0.6]

0.05±0.019 0.051±0.019 0.074±0.008 0.072±0.008 0.072±0.008 0.068±0.007 0.089±0.0 0.082±0.0 0.152±0.001 0.152±0.001 0.149±0.001 0.149±0.001

0.77 0.50 0.35 0.00 1.00 1.00

68 26 6 16 4 12

0.77 0.5 0.35 0.0 1.0 1.0

yeast yeast yeast yeast yeast yeast yeast yeast yeast

Gibbs Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.035±0.019 0.035±0.019 (0.0, 0.2] 0.035±0.016 0.034±0.016 (0.2, 0.4] 0.038±0.015 0.039±0.015 (0.4, 0.6] 0.048±0.006 0.047±0.006 (0.0, 0.2] 0.06±0.014 0.054±0.01 (0.2, 0.4] 0.071±0.006 0.071±0.006 (0.0, 0.2] 0.071±0.01 0.063±0.009 (0.2, 0.4] 0.084±0.004 0.064±0.004 (0.4, 0.6] 0.059±0.003 0.059±0.003

1.00 0.82 0.89 0.92 0.09 1.00 0.03 0.00 1.00

14 80 28 10 24 4 16 8 4

1.0 0.82 0.89 0.92 0.09 1.0 0.03 0.0 1.0

WIS

Kernel-WIS P-Value N Obs.

Table 4: The table describes the pointwise finite mean squared error of the WIS estimator and Kernel-WIS estimator (with a shared bandwidth), split by dataset, policy type and a discretisation of the absolute difference in true performance under the evaluation and logging policy, for oracle behaviour policies. The P-Value column defines the p-value of a two-sided Wald t-test, comparing the finite mean squared error of the WIS and Kernel-WIS estimators. N observations defines the number of experiments aggregated over per row, where the aggregations are across logging policy temperature and logging policy faulty action.

Table 5 again displays the relative performance of the WIS and Kernel-WIS estimators but includes the coverage of the bootstrapped confidence intervals. Generally speaking, neither the WIS or Kernel-WIS estimator appears to outperform the other. The results suggest that the additional bias under the Kernel-WIS estimator is strictly beneficial with respect to the bias-variance trade-off.

Estimate Dataset arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia

Eval/Log Policy Type Abs. Diff. Gibbs Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

WIS

Kernel-WIS

Coverage WIS

Kernel-WIS

(-0.2, 0.0] 0.037±0.016 0.037±0.016 0.618±0.356 0.618±0.356 (0.0, 0.2] 0.046±0.014 0.042±0.018 0.642±0.362 0.628±0.393 (0.2, 0.4] 0.055±0.02 0.062±0.017 0.802±0.234 0.772±0.23 (0.4, 0.6] 0.064±0.012 0.08±0.036 0.667±0.361 0.648±0.348 (0.0, 0.2] 0.087±0.012 0.069±0.028 0.853±0.068 0.877±0.087 (0.2, 0.4] 0.086±0.009 0.079±0.004 0.5±0.111 0.583±0.14 (0.0, 0.2] 0.093±0.006 0.075±0.011 0.87±0.071 0.898±0.098 (0.2, 0.4] 0.073±0.003 0.074±0.002 0.444±0.0 0.444±0.0 (0.4, 0.6] 0.111±0.001 0.111±0.001 0.667±0.0 0.667±0.0 Continued on next page

3. EXPERIMENTS Estimate Dataset

Eval/Log Policy Type Abs. Diff.

WIS

Kernel-WIS

Coverage WIS

Kernel-WIS

0.444±0.0

arrhythmia

SNIW

(0.6, 0.8] 0.092±0.001 0.092±0.001

kropt kropt kropt kropt kropt kropt kropt kropt

Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.032±0.019 0.032±0.019 0.125±0.284 0.125±0.284 (0.0, 0.2] 0.033±0.018 0.033±0.017 0.21±0.35 0.167±0.315 (0.2, 0.4] 0.029±0.017 0.03±0.017 0.288±0.391 0.254±0.373 (0.0, 0.2] 0.047±0.008 0.046±0.006 0.1±0.08 0.1±0.08 (0.2, 0.4] 0.049±0.003 0.03±0.011 0.037±0.055 0.204±0.2 (0.0, 0.2] 0.049±0.006 0.026±0.017 0.061±0.057 0.478±0.237 (0.2, 0.4] 0.053±0.002 0.049±0.004 0.0±0.0 0.0±0.0 (0.4, 0.6] 0.053±0.0 0.046±0.0 0.0±0.0 0.0±0.0

letter letter letter letter letter letter letter

Gibbs Gibbs Gibbs IW IW SNIW SNIW

(-0.2, 0.0] 0.034±0.026 0.034±0.026 0.271±0.349 0.271±0.349 (0.0, 0.2] 0.033±0.025 0.033±0.025 0.359±0.409 0.359±0.409 (0.2, 0.4] 0.034±0.024 0.034±0.024 0.396±0.416 0.395±0.415 (0.0, 0.2] 0.024±0.004 0.023±0.004 0.708±0.151 0.708±0.151 (0.2, 0.4] 0.022±0.003 0.022±0.003 0.715±0.107 0.701±0.113 (0.0, 0.2] 0.032±0.007 0.028±0.011 0.528±0.119 0.634±0.201 (0.2, 0.4] 0.039±0.014 0.038±0.015 0.389±0.341 0.403±0.366

micro-mass micro-mass micro-mass micro-mass micro-mass micro-mass micro-mass

Gibbs Gibbs Gibbs IW SNIW SNIW SNIW

(-0.2, 0.0] 0.045±0.02 0.045±0.02 0.528±0.375 0.528±0.375 (0.0, 0.2] 0.052±0.015 0.051±0.015 0.52±0.316 0.526±0.316 (0.2, 0.4] 0.05±0.019 0.057±0.018 0.554±0.426 0.516±0.397 (0.0, 0.2] 0.073±0.024 0.071±0.026 0.708±0.242 0.719±0.251 (0.0, 0.2] 0.098±0.004 0.097±0.004 0.476±0.166 0.484±0.161 (0.2, 0.4] 0.095±0.007 0.095±0.007 0.344±0.152 0.344±0.152 (0.4, 0.6] 0.1±0.007 0.1±0.007 0.167±0.103 0.167±0.103

optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits

Gibbs Gibbs Gibbs IW IW IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.027±0.013 0.027±0.013 0.238±0.337 0.238±0.337 (0.0, 0.2] 0.026±0.011 0.026±0.011 0.562±0.337 0.569±0.344 (0.2, 0.4] 0.027±0.011 0.027±0.011 0.472±0.399 0.477±0.398 (0.0, 0.2] 0.076±0.011 0.077±0.01 0.178±0.186 0.178±0.186 (0.2, 0.4] 0.068±0.006 0.068±0.005 0.605±0.273 0.617±0.243 (0.4, 0.6] 0.075±0.004 0.075±0.004 0.231±0.146 0.241±0.133 (0.6, 0.8] 0.075±0.0 0.081±0.002 0.556±0.0 0.389±0.079 (0.0, 0.2] 0.066±0.007 0.069±0.005 0.514±0.258 0.444±0.279 (0.2, 0.4] 0.076±0.006 0.08±0.006 0.756±0.05 0.733±0.061 (0.4, 0.6] 0.071±0.006 0.071±0.006 0.393±0.258 0.393±0.258

page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks

Gibbs Gibbs Gibbs Gibbs Gibbs Gibbs IW IW IW

(-0.2, 0.0] 0.067±0.024 0.067±0.024 0.056±0.12 0.056±0.12 (0.0, 0.2] 0.075±0.017 0.074±0.017 0.043±0.146 0.04±0.146 (0.2, 0.4] 0.047±0.022 0.048±0.023 0.019±0.045 0.019±0.045 (0.4, 0.6] 0.055±0.026 0.055±0.026 0.5±0.513 0.5±0.513 (0.6, 0.8] 0.076±0.011 0.077±0.01 0.432±0.459 0.432±0.459 (0.8, 1.0] 0.08±0.014 0.085±0.007 0.5±0.707 0.5±0.707 (0.0, 0.2] 0.048±0.013 0.048±0.014 0.315±0.333 0.296±0.327 (0.2, 0.4] 0.075±0.003 0.075±0.003 0.0±0.0 0.0±0.0 (0.4, 0.6] 0.049±nan 0.049±nan 0.889±nan 0.889±nan Continued on next page

Authors Suppressed Due to Excessive Length Estimate Dataset

Eval/Log Policy Type Abs. Diff.

WIS

Kernel-WIS

Coverage WIS

Kernel-WIS

0.066±0.012 0.068±0.014 0.9±0.161 0.9±0.161 0.077±nan 0.078±nan 1.0±nan 1.0±nan 0.036±0.005 0.037±0.006 0.481±0.335 0.444±0.344 0.031±0.002 0.023±0.001 0.667±0.0 0.556±0.0 0.044±0.002 0.056±0.014 0.963±0.064 0.963±0.064 0.054±0.014 0.052±0.013 0.901±0.103 0.926±0.111

page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks

IW IW SNIW SNIW SNIW SNIW

(0.6, 0.8] (0.8, 1.0] (0.0, 0.2] (0.2, 0.4] (0.4, 0.6] (0.6, 0.8]

pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits

Gibbs Gibbs Gibbs IW IW IW IW SNIW SNIW SNIW SNIW

(-0.2, 0.0] 0.028±0.014 0.028±0.014 0.143±0.297 0.143±0.297 (0.0, 0.2] 0.026±0.012 0.025±0.012 0.41±0.337 0.422±0.353 (0.2, 0.4] 0.027±0.012 0.027±0.012 0.328±0.345 0.319±0.348 (0.0, 0.2] 0.071±0.003 0.071±0.003 0.037±0.057 0.0±0.0 (0.2, 0.4] 0.062±0.004 0.065±0.002 0.574±0.204 0.5±0.153 (0.4, 0.6] 0.065±0.005 0.067±0.004 0.19±0.284 0.19±0.284 (0.6, 0.8] 0.062±0.002 0.062±0.002 0.556±0.0 0.556±0.0 (0.0, 0.2] 0.05±0.005 0.057±0.009 0.458±0.288 0.319±0.338 (0.2, 0.4] 0.045±0.005 0.068±0.022 0.911±0.093 0.6±0.268 (0.4, 0.6] 0.064±0.003 0.069±0.009 0.19±0.252 0.19±0.232 (0.6, 0.8] 0.064±nan 0.064±nan 0.444±nan 0.444±nan

soybean soybean soybean soybean soybean soybean soybean soybean

Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.044±0.02 0.044±0.02 0.535±0.37 0.535±0.37 (0.0, 0.2] 0.049±0.016 0.048±0.019 0.482±0.343 0.453±0.333 (0.2, 0.4] 0.05±0.019 0.051±0.019 0.531±0.445 0.525±0.443 (0.0, 0.2] 0.074±0.008 0.072±0.008 0.667±0.109 0.671±0.115 (0.2, 0.4] 0.072±0.008 0.068±0.007 0.519±0.152 0.593±0.057 (0.0, 0.2] 0.089±0.0 0.082±0.0 0.5±0.057 0.556±0.0 (0.2, 0.4] 0.152±0.001 0.152±0.001 0.778±0.0 0.778±0.0 (0.4, 0.6] 0.149±0.001 0.149±0.001 0.5±0.058 0.5±0.058

yeast yeast yeast yeast yeast yeast yeast yeast yeast

Gibbs Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.035±0.019 0.035±0.019 0.468±0.383 0.468±0.383 (0.0, 0.2] 0.035±0.016 0.034±0.016 0.579±0.401 0.581±0.393 (0.2, 0.4] 0.038±0.015 0.039±0.015 0.647±0.319 0.623±0.33 (0.4, 0.6] 0.048±0.006 0.047±0.006 0.611±0.41 0.633±0.389 (0.0, 0.2] 0.06±0.014 0.054±0.01 0.819±0.097 0.852±0.102 (0.2, 0.4] 0.071±0.006 0.071±0.006 0.444±0.0 0.444±0.0 (0.0, 0.2] 0.071±0.01 0.063±0.009 0.521±0.113 0.583±0.111 (0.2, 0.4] 0.084±0.004 0.064±0.004 0.5±0.059 0.444±0.059 (0.4, 0.6] 0.059±0.003 0.059±0.003 0.556±0.0 0.556±0.0

Table 5: The table describes the point wise mean squared error and the width of the bootstrapped confidence intervals of the WIS estimator and Kernel-WIS estimator (with a shared bandwidth), split by dataset, policy type and a discretisation of the absolute difference in true performance under the evaluation and logging policy, for oracle behaviour policies. Aggregations for each row are across logging policy temperature and logging policy faulty action.

3. EXPERIMENTS

Fig. 1: The figure describes the proportion of scenarios (i.e., unique combinations of policy type and buckets differences in true evaluation and logging policy return) where the Kernel-WIS estimator strictly outperforms the WIS estimator, against dataset size on the x-axis. Performance is defined in terms of the pointwise mean squared error. Data is shown for the single action reward, oracle behaviour policy setting.

Fig. 2: The figure describes the median normalised difference in pointwise mean squared error (AD(WIS) − AD(Kernel-WIS/(AD(WIS) + AD(Kernel-WIS))) between the Kernel-WIS and WIS estimators, against dataset size on the x-axis. Data is shown for the single action reward, oracle behaviour policy setting.

3.2.4 Non-oracle behaviour policy sensitivity A sensitivity analysis of the performance of the relative performance of the WIS and Kernel-WIS estimators was performed, under miss-specification of the behaviour policy. The sensitivity analysis was performed for a subset of datasets due to computational constraints. For each logging

Authors Suppressed Due to Excessive Length

policy, the set of miss-specified behaviour policies evaluated was based on the faulty actions of the logging policy. Specifically, the set of miss-specified behaviour policies included those that were within length 1 of the logging policy faulty actions. For example, under the arrythmia dataset, for logging policy with temperature 0.3 and fa = 1, the following behaviour policies were considered: – Temperature = 0.5, fa = {1}; – Temperature = 0.3, fa = {}; – Temperature = 0.5, fa = {}; – Temperature = 0.3, fa = {0}; – Temperature = 0.5, fa = {0}; – Temperature = 0.3, fa = {0, 1}; – Temperature = 0.5, fa = {0, 1}; Similarly, under the optdigits dataset, for logging policy with temperature equal to 0.5 and fa = {0, 1, 2, 3, 4}, the following behaviour policies were considered: – Temperature = 0.3, fa = {0, 1, 2, 3, 4}; – Temperature = 0.3, fa = {0, 1, 2, 3}; – Temperature = 0.5, fa = {0, 1, 2, 3}; The relative performance of the Kernel-WIS estimator clearly improved, both in terms of the number of significant results and when not considering statistical significance. Similarly to the oracle behaviour setting (section 3.2.3), figures 3 and 4 display aggregated results, across the scenarios displayed in table 6 (i.e., combinations of policy type and bucketed values of the difference between the true evaluation and logging policy performance). Examining the proportion of times the Kernel-WIS estimator strictly outperforms the WIS estimator (figure 3), the results remain mixed however, examining the median performance (figure 4) suggests the Kernel-WIS estimator strongly outperforms the WIS estimator. Similarly to the oracle setting, the relative performance is strongly correlated with sample size.

Eval/Log Abs. Diff.

Dataset

Policy Type

WIS

Kernel-WIS P-Value N Obs.

arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia arrhythmia

Gibbs Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.133±0.101 0.098±0.093 (0.0, 0.2] 0.097±0.074 0.074±0.056 (0.2, 0.4] 0.218±0.148 0.201±0.122 (0.4, 0.6] 0.221±0.15 0.249±0.139 (0.0, 0.2] 0.155±0.115 0.095±0.084 (0.2, 0.4] 0.284±0.117 0.221±0.109 (0.0, 0.2] 0.157±0.11 0.092±0.05 (0.2, 0.4] 0.082±0.045 0.078±0.046 (0.4, 0.6] 0.455±0.21 0.257±0.121

0.01 0.00 0.35 0.23 0.00 0.21 0.00 0.81 0.00

96 684 112 76 181 11 153 15 24

0.01 0.0 0.35 0.23 0.0 0.21 0.0 0.81 0.0

micro-mass micro-mass

Gibbs Gibbs

(-0.2, 0.0] 0.111±0.072 0.076±0.055 (0.0, 0.2] 0.105±0.071 0.088±0.063

0.00 0.00

88 378

0.0 0.0

Continued on next page

3. EXPERIMENTS

Dataset

Policy Type

Eval/Log Abs. Diff.

micro-mass micro-mass micro-mass micro-mass micro-mass

Gibbs IW SNIW SNIW SNIW

(0.2, 0.4] (0.0, 0.2] (0.0, 0.2] (0.2, 0.4] (0.4, 0.6]

0.169±0.104 0.179±0.089 0.151±0.085 0.114±0.076 0.143±0.086 0.115±0.044 0.267±0.138 0.241±0.119 0.328±0.194 0.251±0.124

0.14 0.00 0.01 0.21 0.12

398 176 72 81 23

0.14 0.0 0.01 0.21 0.12

optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits optdigits

Gibbs Gibbs Gibbs IW IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.134±0.087 0.058±0.05 (0.0, 0.2] 0.125±0.085 0.087±0.059 (0.2, 0.4] 0.164±0.104 0.185±0.082 (0.0, 0.2] 0.188±0.158 0.177±0.065 (0.2, 0.4] 0.148±0.108 0.196±0.089 (0.4, 0.6] 0.104±0.094 0.227±0.063 (0.0, 0.2] 0.12±0.046 0.105±0.056 (0.2, 0.4] 0.078±0.069 0.229±0.105 (0.4, 0.6] 0.107±0.088 0.212±0.096

0.00 0.00 0.01 0.58 0.02 0.00 0.13 0.00 0.00

78 420 266 78 51 27 53 43 60

0.0 0.0 0.01 0.58 0.02 0.0 0.13 0.0 0.0

page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks page-blocks

Gibbs Gibbs Gibbs Gibbs Gibbs Gibbs IW IW IW IW SNIW SNIW SNIW SNIW

(-0.2, 0.0] 0.114±0.117 0.085±0.073 (0.0, 0.2] 0.083±0.078 0.078±0.059 (0.2, 0.4] 0.152±0.118 0.132±0.094 (0.4, 0.6] 0.238±0.199 0.247±0.21 (0.6, 0.8] 0.336±0.248 0.366±0.251 (0.8, 1.0] 0.31±0.259 0.313±0.263 (0.0, 0.2] 0.106±0.099 0.053±0.049 (0.2, 0.4] 0.021±0.009 0.067±0.014 (0.4, 0.6] 0.076±0.066 0.065±0.038 (0.6, 0.8] 0.125±0.121 0.124±0.082 (0.0, 0.2] 0.189±0.083 0.066±0.085 (0.2, 0.4] 0.202±0.05 0.027±0.021 (0.4, 0.6] 0.179±0.149 0.198±0.118 (0.6, 0.8] 0.071±0.076 0.062±0.054

0.11 0.48 0.43 0.74 0.38 0.97 0.00 0.00 0.73 0.96 0.00 0.00 0.53 0.60

58 206 36 124 112 12 65 7 6 38 39 9 39 29

0.11 0.48 0.43 0.74 0.38 0.97 0.0 0.0 0.73 0.96 0.0 0.0 0.53 0.6

pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits pendigits

Gibbs Gibbs Gibbs IW IW IW SNIW SNIW SNIW SNIW

(-0.2, 0.0] 0.134±0.087 0.067±0.046 (0.0, 0.2] 0.127±0.087 0.084±0.059 (0.2, 0.4] 0.159±0.104 0.164±0.088 (0.0, 0.2] 0.169±0.129 0.102±0.039 (0.2, 0.4] 0.156±0.14 0.136±0.054 (0.4, 0.6] 0.113±0.121 0.149±0.026 (0.0, 0.2] 0.097±0.035 0.098±0.046 (0.2, 0.4] 0.101±0.084 0.15±0.055 (0.4, 0.6] 0.088±0.069 0.165±0.044 (0.6, 0.8] 0.075±0.002 0.112±0.001

0.00 0.00 0.53 0.00 0.41 0.17 0.90 0.00 0.00 0.00

78 444 242 94 38 24 53 38 63 2

0.0 0.0 0.53 0.0 0.41 0.17 0.9 0.0 0.0 0.0

soybean soybean soybean soybean soybean soybean

Gibbs Gibbs Gibbs IW IW IW

(-0.2, 0.0] 0.102±0.063 0.072±0.05 (0.0, 0.2] 0.104±0.07 0.086±0.06 (0.2, 0.4] 0.165±0.104 0.178±0.092 (0.0, 0.2] 0.139±0.077 0.12±0.067 (0.2, 0.4] 0.241±0.175 0.066±0.063 (0.4, 0.6] 0.493±0.007 0.009±0.002

0.00 0.00 0.07 0.02 0.00 0.00

88 384 392 149 21 6

0.0 0.0 0.07 0.02 0.0 0.0

WIS

Kernel-WIS P-Value N Obs.

Continued on next page

Authors Suppressed Due to Excessive Length Eval/Log Abs. Diff.

Dataset

Policy Type

WIS

Kernel-WIS P-Value N Obs.

soybean soybean soybean

SNIW SNIW SNIW

(0.0, 0.2] 0.141±0.088 0.078±0.027 (0.2, 0.4] 0.32±0.059 0.039±0.034 (0.4, 0.6] 0.336±0.151 0.116±0.055

0.00 0.00 0.00

96 36 44

0.0 0.0 0.0

yeast yeast yeast yeast yeast yeast yeast yeast yeast

Gibbs Gibbs Gibbs Gibbs IW IW SNIW SNIW SNIW

(-0.2, 0.0] 0.112±0.076 0.089±0.069 (0.0, 0.2] 0.116±0.08 0.108±0.078 (0.2, 0.4] 0.143±0.109 0.142±0.088 (0.4, 0.6] 0.211±0.123 0.225±0.116 (0.0, 0.2] 0.147±0.118 0.099±0.085 (0.2, 0.4] 0.188±0.157 0.084±0.062 (0.0, 0.2] 0.148±0.124 0.094±0.07 (0.2, 0.4] 0.22±0.14 0.11±0.067 (0.4, 0.6] 0.378±0.211 0.16±0.071

0.06 0.10 0.95 0.51 0.00 0.00 0.00 0.00 0.00

78 456 166 64 119 37 98 44 14

0.06 0.1 0.95 0.51 0.0 0.0 0.0 0.0 0.0

Table 6: The table describes the pointwise finite mean squared error of the WIS estimator and Kernel-WIS estimator (with a shared bandwidth), split by dataset, policy type and a discretisation of the absolute difference in true performance under the evaluation and logging policy, for nonoracle behaviour policies. The P-Value column defines the p-value of a two-sided Wald t-test, comparing the finite mean squared error of the WIS and Kernel-WIS estimators. N observations defines the number of experiments aggregated over per row, where the aggregations are across logging policy temperature and logging policy faulty action.

Fig. 3: The figure describes the proportion of scenarios (i.e., unique combinations of policy type and buckets differences in true evaluation and logging policy return) where the Kernel-WIS estimator strictly outperforms the WIS estimator, against dataset size on the x-axis. Performance is defined in terms of the pointwise mean squared error. Data is shown for the single action reward, non-oracle behaviour policy setting.

3. EXPERIMENTS

Fig. 4: The figure describes the median normalised difference in pointwise mean squared error (AD(WIS) − AD(Kernel-WIS/(AD(WIS) + AD(Kernel-WIS))) between the Kernel-WIS and WIS estimators, against dataset size on the x-axis. Data is shown for the single action reward, non-oracle behaviour policy setting.

3.2.5 Continuous reward sensitivity All of the results described in sections 3.2.3 and 3.2.4 pertained to contextual bandit definitions where all non-optimal actions achieved a reward of 0 i.e., ∀s ∈ S, ∃!a ∈ A : R(a, s) = 1, ∀a′ ̸= a, R(a, s) = 0. Clearly, this is not representative of all contextual bandit definitions. To try and assess the stability of the results in section 3.2.3 (i.e., under oracle behaviour policies only), a continuous reward analogue of the experiments performed on the optdigits dataset was created. To define the continuous rewards, a variational autoencoder (VAE) (?) was trained on the dataset and the reward of suboptimal actions was defined by the distance to the mean of the suboptimal action under the latent space of the VAE. Concretely, consider a VAE, defined as g · f (s). For a given state si , there exists an optimal action amax = argmaxa′ R(a′ , si ), the associated reward of which is 1, where R is the reward function under the single action reward setup. For all other actions, a′ ̸= amax , define the unnormalised reward as: RCont-U (a′ , si ) = f (si ) −

1 X f (sj ) nH j∈H

where H is the set indices such that a′ is optimal under the single action reward Pof n setting and nH = i=1 1(i ∈ H). The reward function used for the continuous setting, RCont , is derived from RCont-U (a′ , si ) by normalising as: ( 1 if a′ = argmaxa′ R(a′ , si ) ′ RCont (a , si ) = ′ ′′ RCont-U (a , si )/ maxa′′ RCont-U (a , si ) otherwise This approach was inspired by ? who originally proposed the VAE architecture and demonstrated the ability to naturally interpolate between digits. Evidence of

Authors Suppressed Due to Excessive Length

convergence of the model is provided in appendix section C along with other training details. Figure 5 presents a heatmap of the rewards for non-optimal actions under the optdigitDist dataset, which displays some intuitive results. For example, predicting the number 9 when the target is 3 receives a relatively high reward whilst predicting the number 6 receives a low reward. Figure 6 displays two examples of a digit 3, 6 and 9 chosen randomly from the optdigits dataset. Given the similar orientation of 3 and 9 in comparison to 3 and 6, the proposed reward structure is not unreasonable.

Fig. 5: The figure displays a heatmap of the mean reward under sub-optimal actions. The y-axis defines the optimal action and the x-axis defines the sub-optimal action selected as such, the figure should be read from left to right, not up and down.

3. EXPERIMENTS

(a) Random observations depicting 3

(b) Random observations depicting 9

Fig. 6: The figure depicts randomly selected observations from the optdigits dataset.

Clearly, under the continuous reward setting, the relative performance of the KernelWIS estimator significantly degraded. However, understanding the precise reasons for this were left for future work.

Authors Suppressed Due to Excessive Length Dataset

Policy Type Eval/Log Abs. Diff.

WIS

Kernel-WIS P-Value N Observations

optdigits optdigits optdigits optdigits optdigits optdigits optdigits

Gibbs Gibbs Gibbs IW IW SNIW SNIW

[0.0] (0.0, 0.2] (0.2, 0.4] (0.0, 0.2] (0.2, 0.4] (0.0, 0.2] (0.2, 0.4]

0.292±0.131 0.292±0.131 0.307±0.124 0.308±0.123 0.268±0.132 0.269±0.131 0.076±0.017 0.076±0.016 0.072±0.008 0.076±0.007 0.076±0.015 0.082±0.015 0.083±0.005 0.085±0.009

1.00 0.97 0.98 0.94 0.40 0.21 0.62

28 80 24 22 6 21 7

optdigitsDist optdigitsDist optdigitsDist optdigitsDist optdigitsDist optdigitsDist

Gibbs Gibbs IW IW SNIW SNIW

[0.0] (0.0, 0.2] (0.0, 0.2] (0.2, 0.4] (0.0, 0.2] (0.2, 0.4]

0.19±0.093 0.19±0.092 0.195±0.088 0.204±0.089 0.037±0.021 0.063±0.028 0.016±0.026 0.03±0.035 0.017±0.016 0.059±0.031 0.018±0.019 0.049±0.016

0.99 0.48 0.00 0.30 0.00 0.00

28 104 18 10 16 12

Table 7: The table compares the performance of the WIS and Kernel-WIS estimators under a single action reward (optdigits) and continuous reward setting (optdigitsDist), using the optdigits dataset. The table describes the pointwise finite mean squared error of the WIS estimator and Kernel-WIS estimator (with a shared bandwidth), split by dataset, policy type and a discretisation of the absolute difference in true performance under the evaluation and logging policy, for oracle behaviour policies. The P-Value column defines the p-value of a two-sided Wald t-test, comparing the finite mean squared error of the WIS and Kernel-WIS estimators. N observations defines the number of experiments aggregated over per row, where the aggregations are across logging policy temperature and logging policy faulty action.

3.2.6 Bandwidth sensitivity For the results presented in the previous sections, the Kernel-WIS estimator was optimised using a single bandwidth value for every dimension of the feature space i.e., h in equation 7 is a real number and not a vector, herein referred to as Kernel-WIS (Shared). Every experiment was also run by optimising unique bandwidth values for each dimension of the state, herein referred to as Kernel-WIS (Unique). However, this produced worse (or equivalent) results in all but one instance. The aggregate results are provided in table 8. Figure 7 displays boxplots of the standard deviation of bandwidth values over the mean bandwidth value per experiment, calculated for the Kernel-WIS (Unique) estimator. As can be seen on the plot, that variation of bandwidth values under the Kernel-WIS (Unique) estimator was considerable and it is hypothesised that this lead to a kind of ’overfitting’ behaviour. Figure 8 displays the average kernel output i.e., the average value of kh (s, sj ) of the Kernel-WIS (Shared) and Kernel-WIS (Unique) estimators side by side. Clearly, under the Kernel-WIS (Unique) estimator, the distribution of average distances is smaller. This is a result of the ’overfitting’ behaviour causing a single or subset of bandwidth values to be extremely large and all others to be small (demonstrated by ratio of standard deviation to mean on figure 7). In particular, for high dimensional states (i.e., not yeast and page-blocks), the resulting distance was very small, causing the Kernel-WIS (Unique) estimator to behave more like the State-WIS estimator. Both figures 7 and 8 are displayed for the single action reward,

3. EXPERIMENTS

Reward Structure

Behaviour Policy

Single action Single action Single action Single action Single action Single action Single action Single action Single action

Oracle Oracle Oracle Oracle Oracle Oracle Oracle Oracle Oracle

Dataset

WIS

Kernel-WIS Kernel-WIS (Shared) (Unique)

yeast 0.046±0.021 0.043±0.018 0.043±0.032 soybean 0.063±0.032 0.062±0.032 0.1±0.066 page-blocks 0.064±0.021 0.064±0.021 0.07±0.024 pendigits 0.037±0.019 0.038±0.022 0.057±0.045 micro-mass 0.061±0.025 0.062±0.024 0.126±0.1 arrhythmia 0.059±0.024 0.054±0.026 0.096±0.113 optdigits 0.04±0.023 0.04±0.024 0.13±0.119 kropt 0.037±0.017 0.033±0.017 0.034±0.017 letter 0.032±0.021 0.031±0.021 0.031±0.021

Single action Non-Oracle yeast 0.142±0.112 0.117±0.087 0.113±0.086 Single action Non-Oracle soybean 0.15±0.109 0.117±0.083 0.128±0.094 Single action Non-Oracle page-blocks 0.168±0.175 0.157±0.18 0.157±0.179 Single action Non-Oracle pendigits 0.134±0.098 0.114±0.072 0.119±0.075 Single action Non-Oracle arrhythmia 0.139±0.12 0.105±0.095 0.107±0.116 Single action Non-Oracle optdigits 0.137±0.1 0.138±0.091 0.166±0.117 Single action Non-Oracle micro-mass 0.15±0.106 0.136±0.094 0.136±0.104 Continuous

Oracle

optdigitsDist 0.143±0.109 0.157±0.102 0.218±0.086

Table 8: The table describes the pointwise finite mean squared error of the Kernel-WIS estimator with a shared and unique bandwidth (denoted Kernel-WIS (Shared) and Kernel-WIS (Unique)) against the WIS estimator. The results are split by dataset, whether an oracle behaviour policy was used and the reward function structure. Aggregations for each row are across logging policy temperature, logging policy faulty action and policy type.

oracle behaviour policies however, the results transfer. An interesting direction of future research would be to consider whether unique bandwidths could be used with an appropriate regularisation e.g., penalising for a large standard deviation. However, given the bandwidth selection procedure is already tangental to the ultimate estimand of interest, this was not explored any further.

Authors Suppressed Due to Excessive Length

Fig. 7: The figure depicts boxplots of the mean bandwidth value over the standard deviation of bandwidth values for the Kernel-WIS (Unique) estimator, split by datasets and calculated under the single action reward, oracle behaviour policy setting. The values in brackets next to the titles define the proportion of standard deviation value that overflowed (due to already extremely large bandwidth values).

3. EXPERIMENTS

Fig. 8: The figure depicts boxplots of the average kernel value (i.e., the average value of kh (s, sj )), split by Kernel-WIS (Shared) and Kernel-WIS (Unique) and dataset and calculated under the single action reward, oracle behaviour policy setting.

Authors Suppressed Due to Excessive Length

Limitations

The Kernel-WIS estimator does not uniformly outperform the WIS estimator which leaves the analyst with the (almost) impossible task of performing estimator selection. The analysis has provided some relatively clear boundaries: under discrete rewards, the Kernel-WIS estimator can be used to trade-off variance reduction for a (statistically insignificant) drop in performance under oracle behaviour policies and for a (statistically significant) boost in performance under non-oracle behaviour policies. However, again this is not entirely uniform, demonstrated by the results on the Optdigits dataset in particular. Since it is reasonable to assume that behaviour policies are never oracle, always choosing the Kernel-WIS estimator might be a reasonable approach however, improving performance via stronger bandwidth selection algorithms is anticipated to strengthen this recommendation. The main practical limitation with Kernel-WIS is the challenge in optimising the bandwidth. Even with improved bandwidth selection procedures, it is unlikely that the resulting optimisation landscape will be convex, meaning selecting the bandwidth would still require repeated evaluations of the expensive kernel function. This was largely mitigated through the use of analytic derivatives for the cross-validation approach defined in section 2.4 however, this manual derivation limits the generalisability of Kernel-WIS since gradients must be re-derived for any change in the kernel function. Since no research was conducted into the effect of kernel selection, it is unclear how extensive the lack of generalisability will be. The theoretical analysis presented is limited as it assumed fixed evaluation and behaviour policies, and assumes a bandwidth that tends to 0 as n → ∞. This assumption is not unreasonable: practically, the assumption translates to the size of the test set, dtest , tending to infinity (assuming the behaviour policy is trained on the training set) and assumes, given a finite dataset of any size, that the bandwidth will be manually forced to 0 as n → ∞, beginning from the bandwidth selected by the cross validation procedure. It would be beneficial to establish consistency results for a changing behaviour and evaluation policy as well as demonstrate that the sequence of cross validation procedures converges to a bandwidth equal to 0. Intuitively, it is expected that in the limit of infinite data, cross validation will enable an accurate approximation of the random variable E[W |S] and thus, it is not unrealistic for such a proof to exist. However, this should not be a priority of future work since, as discussed in section 5, it is anticipated that better bandwidth selection procedures exist. Finally, whilst the experimental scenarios were non-trivial and expansive in terms of parameters, it would be beneficial to assess the performance of the Kernel-WIS estimator with evaluation and behaviour policy parameterised by common function approximaters such as neural networks. Additionally, it would be beneficial to assess Kernel-WIS considering deterministic, rather than stochastic evaluation policies, since these are common place in off-policy evaluation.

5. CONCLUSION

Conclusion

The analysis presented describes a novel estimator for performing off-policy evaluation in time-independent settings. The estimator was derived under the hypothesis that, whilst the WIS estimator benefits from the α-bounded property, the strong interactions between random variables result in the tail behaviour of the estimator being difficult to control. Unfortunately, there was not sufficient scope to theoretically analyse the tail behaviour of the proposed Kernel-WIS estimator however, the analyses provided has laid the foundations for the future in this direction: – The Kernel-WIS was demonstrated to be asymptotically consistent, almost a necessity for any (frequentist) estimator; – With bandwidth selected by cross-validation, the Kernel-WIS estimator demonstrated almost (statistically) identical performance to the WIS estimator under an oracle behaviour policy and single action reward for a wide number of scenarios however; – The Kernel-WIS estimator demonstrated for the majority of setting assessed, (statistically) superior performance to the WIS estimator under mild miss-specification of the behaviour policy and single action reward; – Generally speaking, under the single action reward setting, the WIS estimator only ever marginally outperformed the Kernel-WIS estimator, demonstrated by figures 1 and 3. However, the Kernel-WIS estimator, particularly under small sample sizes drastically outperformed the WIS estimator (demonstrated by figures 2 and 4 and the statistically significant results); – Additionally, the difference in boostrapped confidence interval coverage did not appear to drastically change between the WIS and Kernel-WIS estimators, despite the additional variance induced in the latter estimator; – Under a continuous reward setting, the performance of the Kernel-WIS estimator degraded significantly. Overall, the analysis demonstrated that there is utility in considering the Kernel-WIS estimator over and above the WIS estimator. Arguably, the Kernel-WIS estimator should always be preferred due to the magnitude of performance improvement that can be attained. It is anticipated that more research into bandwidth selection would provide further performance improvements since the proposed approach via cross-validation is somewhat tangental to the actual goal of the estimator: the estimand is Epπe [R] and not Epπβ [W |S].

5.1

Future work

It is anticipated that the most impactful next step for the Kernel-WIS estimator would be to establish a PAC-Bayes result. This would enable optimisation of the bandwidth hyperparameter to be directly associated with the tail performance of the estimator, rather than tangentially as in the process described in section 2.4. Additional impactful next steps might be to extend the estimator to the full, time dependent, Markov

Authors Suppressed Due to Excessive Length

decision process setting and establish asymptotic normality results. Future work should additionally explore the reasons behind the poor performance of the Kernel-WIS estimator under continuous reward settings.

Bibliography

Alquier, P., Aouali, I., Chopin, N. & Sakhi, O. (2024), Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning, in ‘Advances in Neural Information Processing Systems 37’, Neural Information Processing Systems Foundation, Inc. (NeurIPS), Vancouver, BC, Canada, pp. 80706–80755. URL: http://www.proceedings.com/079017-2566.html Bischl, B., Casalicchio, G., Das, T., Feurer, M., Fischer, S., Gijsbers, P., Mukherjee, S., Müller, A. C., Németh, L., Oala, L., Purucker, L., Ravi, S., van Rijn, J. N., Singh, P., Vanschoren, J., van der Velde, J. & Wever, M. (2025), ‘Openml: Insights from 10 years and more than a thousand papers’, Patterns 6(7), 101317. URL: https://www.cell.com/patterns/fulltext/S2666-3899(25)00165-5 Einmahl, U. & Mason, D. M. (2005), ‘Uniform in bandwidth consistency of kernel-type function estimators’, The Annals of Statistics 33(3). URL: https://projecteuclid.org/journals/annals-of-statistics/volume33/issue-3/Uniform-in-bandwidth-consistency-of-kernel-type-functionestimators/10.1214/009053605000000129.full Francis, W. C. (2023), ‘Variational-Autoencoder-for-MNIST’. URL: https://github.com/williamcfrancis/Variational-Autoencoder-for-MNIST Hernán, M. A. & James, R. M. (2020), Causal Inference: What If. Kallus, N. & Uehara, M. (2019), Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning, in H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox & R. Garnett, eds, ‘Advances in Neural Information Processing Systems’, Vol. 32, Curran Associates, Inc. Kuzborskij, I., Vernade, C., György, A. & Szepesvári, C. (2021), Confident off-policy evaluation and selection through self-normalized importance weighting, in ‘Proceedings of Machine Learning Research’, Vol. 130. Owen, A. B. (2013), Monte Carlo theory, methods and examples, https://artowen.su. domains/mc/. Rubin, D. B. (2001), ‘Using propensity scores to help design observational studies: Application to the tobacco litigation’, Health Services and Outcomes Research Methodology 2. Vaart, A. W. V. D. (1998), Asymptotic Statistics, 1 edn, Cambridge University Press. URL: https://www.cambridge.org/core/product/identifier/9780511802256/type/book Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, İ., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P. & SciPy 1.0 Contributors (2020), ‘SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python’, Nature Methods 17, 261–272.

Authors Suppressed Due to Excessive Length

Wallace, M. P. & Moodie, E. E. M. (2015), ‘Doubly-robust dynamic treatment regimen estimation via weighted least squares’, Biometrics 71(3), 636–644. URL: https://academic.oup.com/biometrics/article/71/3/636-644/7249321 Wallace, M. P., Moodie, E. E. & Stephens, D. A. (2017), ‘Model validation and selection for personalized medicine using dynamic-weighted ordinary least squares’, Statistical Methods in Medical Research 26(4), 1641–1653. URL: https://journals.sagepub.com/doi/10.1177/0962280217708665

Theoretical results

The proceeding section provides supporting analysis for theorem 1. Table 9 describes each of the assumptions required to derive the result and their practical relevance. As can be seen, none of the assumptions are unusual or restrictive (outside of the common causal inference assumptions such as correct identification of the propensity score model). Theorem 1 is also restated and a full proof is given.

Assumption

Implication

∥kh (x)∥ < ∞

Satisfied by many standard kernels e.g., RBF.

For some C > 0 and ν > 0, N (ϵ, K) ≤ Cϵ−ν where 1 K = {kh ((x − ·)/h− d ) : h > 0, x ∈ Rd }

Satisfied by many standard kernels e.g., RBF. Refer to Einmahl & Mason (2005) for a further explanation.

K is pointwise measurable

Standard measurability assumption. Pervasive in almost all statistical analyses.

The support of kh is [−0.5, 0.5]d

This can be achieved through a simple rescaling of the input data, a process which is common in many statistical analysis pipelines.

kh (x)d(x) = 1

Specifies the kinds of kernels that are applicable and notably covers common exponential types e.g., RBF and Matern. Due to the self-normalising property of the estimator, this does not prohibit the use of kernels in their un-normalised form.

Assume f defines the marginal density of S and, f is uniformly Lipschitz continuous and strictly positive on I ϵ , where I is a compact set in Rd and I ϵ = {max1≤i≤d |si | ≤ ϵ}

Impossible to validate but not particularly strict since no specific bound is placed in the Lipschitz constant.

Continued on next page

5. CONCLUSION

Assumption

Implication

There exists M such that almost surely, |W |1(S ∈ I ϵ ) ≤ M and W ≥ wmin > 0

Bounded weights are implied by the positivity assumption of causal inference so the assumption is uncontroversial.

R ∈ [0, 1]

Standard assumption for contextual bandits

Correct identification of the propensity model i.e., p̂πl = pπl

Table 9: The table describes the assumptions required for the theoretical analysis of the KernelWIS estimator, along with the justification for the applicability of the assumption to standard statistical analysis pipelines.

Theorem 2 (Asymptotic consistency of Kernel-WIS (Theorem 1 repeated)). Under nan assumptions 1 to 9, for 0 < an < h < bn < 1, bn → 0 and log n → ∞, the Kernel-WIS estimator is asymptotically consistent: a.s. JˆKernel-WIS −−→ E[R]

where: 1X JˆKernel-WIS = ri wi n i=1

j=1 kh (si , sj )wi P n j=1 kh (si , sj )

!−1 .

Theorem 3 (Theorem 2, Einmahl & Mason (2005)). Assume conditions 1 to 8 are satisfied. Then, for large enough c > 0 and any bn ↓ 0: √ nh∥m̂n,h − fq̄(·,h) ¯(·,h) ∥I <∞ lim sup sup p −1 c log n n→∞ log(h ) ∨ log log n n≤h≤b n

where: Pn

i=1 m̂n,h (s) = P n

Wi kh ((s − Si )h− d ) 1 −d

) i=1 kh ((s − Si )h 1 −d q̄(·, h) =E[W kh ((s − S)h )]/h

1 f¯(·, h) =E[kh ((s − S)h− d )]/h

As the authors state, a clear corollary to this is that, almost surely: lim sup sup n→∞

c log n n≤h≤bn

q̄(·, h) m̂n,h − ¯ f (·, h)

=0 I

Corollary 1 (Corollary 2, Einmahl & Mason (2005)). Under assumption 1 to 8 and assuming p(w, s) is the joint density of (W, S) with the marginal density given by: Z f (s) = p(w, s)dw

Authors Suppressed Due to Excessive Length

nan then, for 0 < an < bn < 1, bn → 0 and log n → ∞:

lim

a.s.

sup

n→∞ an ≤h≤bn

∥m̂n,h − E[W |S = s]∥I −−→ 0

Proof. Let f : S × W × R → R ∈ F define the space of functions of the form: !−1 Pn kh (si , s)wi i=1 fk,n = wr Pn , i=1 kh (si , s) Throughout, the observation that frequently used:

Pn i=1 kh (si ,s)wi P , fn and JˆKernel-WIS are all bounded is n i=1 kh (si ,s)

Pn i=1 kh (si , s)wi wmin ≤ P ≤ wmax n i=1 kh (si , s) 0≤

wr wr wmax ≤ fn ≤ ≤ wmax wmin wmin wmax 0 ≤ JˆKernel-WIS ≤ wmin

Observe that: lim

sup

n→∞ an ≤h≤bn

≤ lim

∥JˆKernel-WIS − Epπe [R]∥I×W×R

sup

n→∞ an ≤h≤bn

n 1X JˆKernel-WIS − wi ri E[W |S = si ]−1 n i=1 I×W×R n

+ lim

sup

n→∞ an ≤h≤bn

1X wi ri E[W |S = si ]−1 − Epπe [R] n i=1 I×W×R

Beginning with: lim

sup

n→∞ an ≤h≤bn

n 1X JˆKernel-WIS − wi ri E[W |S = si ]−1 n i=1 I×W×R

from corollary 1: lim

sup

n→∞ an ≤h≤bn

Pn i=1 kh (si , s)wi P − E[W |S = s] n i=1 kh (si , s)

then, for an ≤ h ≤ bn , uniformly in s ∈ I: Pn i=1 kh (si , s)wi a.s. P −−→ E[W |S = s] n i=1 kh (si , s)

a.s.

−−→ 0 I

(13)

5. CONCLUSION

Consider the transformation: !−1 Pn k (s , s)w h i i i=1 fn (w, r, s; ω) = wr P n i=1 kh (si , s) Since 0 < wmin < wmax < ∞, then the function, fn (w, r, s; ω), is continuous and thus, by the continuous mapping theorem (Vaart (1998)), subject to an ≤ h ≤ bn and uniformly in s ∈ I: a.s.

!−1 ! Pn kh (si , s)wi −1 i=1 Pn − E[W |S = s] i=1 kh (si , s)

≤ sup wr w,r

As such: a.s.

∥fn (w, r, s; ω) − wrE[W |S = s]−1 ∥I×W×R −−→ 0

sup

(14)

an ≤h≤bn

As such, almost surely: lim

sup

n→∞ an ≤h≤bn

n 1X wi ri E[W |S = si ]−1 JˆKernel-WIS − n i=1 I×W×R n

= lim

sup

n→∞ an ≤h≤bn

1 X fn (wi , ri , si ; ω) − wi ri E[W |S = si ]−1 n i=1 I×W×R

1X sup fn (wi , ri , si ; ω) − wi ri E[W |S = si ]−1 =0 n→∞ n I×W×R a ≤h≤b n n i=1

≤ lim

(15) where equation 15 follows from equation 14. For: n

lim

sup

n→∞ an ≤h≤bn

1X wi ri E[W |S = si ]−1 − Epπe [R] n i=1 I×W×R

Observe that: Z E[W |S = s] =

pπe (A|S)) pπ (A|S)da = 1. pπβ (A|S) β

(16)

Authors Suppressed Due to Excessive Length

As such by the strong law of large numbers and under the correct identification of the propensity score: n

lim sup

sup

n→∞ an ≤h≤bn

1X =0 wi ri E[W |S = s]−1 − Epπe [R] n i=1 I×W×R

These two steps, under the decomposition in equation 13, imply: P

lim

sup

n→∞ an ≤h≤bn

∥JˆKernel-WIS − Epπe [R]∥I×W×R > ϵ = 0

and thus, under an ≤ h ≤ bn : a.s. JˆKernel-WIS − Epπe [R] −−→ 0

Derivative of Nadaraya–Watson kernel regressor with RBF kernel

The proceeding derives the gradient for a Nadaraya–Watson kernel regressor with RBF kernel, assuming a vector of bandwidths, rather than just a single value. Similarly to section 2.4.1, the gradient of interest is assumed to be log hl . To begin, observe the following decomposition by repeated application of the chain rule: ∂MSE(h) ∂MSE(h) = ∂ log hl ∂hl

∂ log hl ∂hl

∂MSE(h) ∂ ŷ(si , h) = ∂ ŷ(si , h) ∂hl

!−1

∂ log hl ∂hl

!−1

∂MSE(h) ∂u(wi , si , ŷ) ∂ ŷ(si , h) = ∂u(wi , si , ŷ) ∂ ŷ(si , h) ∂hl

∂ log hl ∂hl

for: 1X (yi − ŷi )2 n u(wi , si , ŷ) =(wi − ŷi (si )) n n X −1 X ŷ(si , h) = k(h, si , sj )wj k(h, si , sj ) MSE(h) =

j=1

k(h, si , sj ) = exp(−0.5h−2 ||si − sj ||2 ) X ||si − sj ||2 = (si,k − sj,k )2 k

!−1 (17)

B. DERIVATIVE OF NADARAYA–WATSON KERNEL REGRESSOR WITH RBF KERNEL ∂MSE(h) ∂ log h −1 i ,si ,ŷ) Of the terms in equation 17, ∂u(w , ∂u(w are trivial to define ∂ ŷ(si ,h) and ( ∂h ) i ,si ,ŷ) and are listed below: n n 2X 2X ∂MSE(h) = u(wi , si , ŷ) = (wi − ŷi (si )) ∂u(wi , si , ŷ) n i=1 n i=1

∂u(wi , si , ŷ) =−1 ∂ ŷ(si , h) !−1 ∂ log hl =hl ∂hl ∂ ŷ(si ,h) , however, requires a little more care. First let: ∂h n X

f (si , h) = g(si , h) =

k(h, si , sj )wj

j=1 n X

k(h, si , sj )

j=1

and thus: n ∂f (si , h) X −3 = h (si,l − sj,l )2 k(h, si , sj )wj ∂hl j=1 n ∂g(si , h) X −3 = h (si,l − sj,l )2 k(h, si , sj ) ∂hl j=1

By the quotient rule: ∂ ŷ(si , h) = ∂hl =

∂g(si ,h) ∂f (si ,h) ∂hl g(si , h) − ∂hl f (si , h) 2 g(si , h) Pn −3 2 h (s i,l − sj,l ) k(h, si , sj )wj j=1

Pn − =

n X

−3

j=1 h

g(si , h) (si,l − sj,l )2 k(h, si , sj ) g(si , h)

ŷ(si , h)

g(si , h)−1 h−3 (si,l − sj,l )2 k(h, si , sj )(wj − ŷ(si , h))

j=1

As such: ∂MSE(h) ∂ log hl n n X 2X = (wi − ŷi (si ))(−1)h g(si , h)−1 h−3 (si,l − sj,l )2 k(h, si , sj )(wj − ŷ(si , h)) n i=1 j=1 =

n n X −2 X (wi − ŷi (si ))g(si , h)−1 h−2 (si,l − sj,l )2 k(h, si , sj )(wj − ŷ(si , h)) n i=1 j=1

Authors Suppressed Due to Excessive Length

OptdigitsDist dataset generation

The code used to construct and train the variational autoencoder (VAE) for generation of the optdigitsDist dataset was directly copied from Francis (2023) except for some inconsequential changes to the metric logging. Figure 9 displays the reconstruction error and Kullback–Leibler (KL) divergence for the trained VAE. The VAE was concluded to reasonably converge due to both elements of the loss function stabilising away from 0.

(a) Training dataset

(b) Testing dataset

Fig. 9: This figure displays the reconstruction error and Kullback–Leibler divergence for the variational autoencoder used to derive the continuous reward optdigits setting (optdigitsDist). The metrics are displayed for both the training set and the testing set.

Record · ID 373386 · SHA-256 b467341019ac9568
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.