Imperfect World Models are Exploitable
Logan Mondal Bhamidipaty1∗ Esmeralda S. Whitammer1 Mykel J. Kochenderfer2
arXiv:2605.15960v1 [cs.AI] 15 May 2026
1
David Abel1
Subramanian Ramamoorthy1
University of Edinburgh
2
Stanford University
Abstract We propose a novel definition of model exploitation in reinforcement learning. Informally, a world model is exploitable if it implies that one policy should be strictly preferred over another while the environment’s true transition model implies the reverse. We analogize our definition with a prior characterization of reward hacking but show that the associated proof of inevitability does not transfer to exploitation. To overcome this obstruction, we develop a general theory of reward hacking and model exploitation that proves that exploitation is essentially unavoidable on large policy sets and yields the corresponding claim for hacking as a special case. Unfortunately, we also find that the conditions that guarantee unhackability in finite policy sets have no counterpart that precludes exploitation. Consequently, we introduce a relaxed notion of exploitation and derive a safe horizon within which it can be avoided. Taken together, our results establish a formal bridge between reward hacking and model exploitation and elucidate the limits of safe planning in world models.
1
Introduction
Efficient real-world planning typically requires approximating complex dynamics (Simon, 1955; Javed and Sutton, 2024). In the best case, well-chosen approximations unravel otherwise intractable problems, as with admissible heuristics in search (Hart et al., 1968), model reduction in dynamical systems (Antoulas, 2005), and latent-space world models in sequential decision-making (Hafner et al., 2019). In the worst case, however, ill-chosen approximations can compromise safety, as with mimicry in evolutionary biology (Bates, 1862), arbitrage in financial markets (Ross, 1976), and speedrunning in video games (Scully-Blaker, 2014). In reinforcement learning (RL), this liability arises not only exogenously from adversaries but also endogenously from the optimizer itself: an agent trained to maximize total expected reward under an imperfect model2 may discover behavior that performs well in simulation but poorly in reality (Jakobi et al., 1995; Tobin et al., 2017). This failure mode is called model exploitation (Ha and Schmidhuber, 2018; Janner et al., 2019), and a natural question is whether it can be avoided. This question pertains to notable results in RL theory including the simulation lemma (Kearns and Singh, 2002) and the value equivalence principle (Grimm et al., 2020). Both implicitly use predictive performance to measure the quality of a model’s approximation. The simulation lemma does so cardinally by bounding the error from optimizing under an imperfect model, while the value equivalence principle does so locally by characterizing when two models induce identical Bellman updates. Modern world models also emphasize predictive performance, typically learning latent-space models of transitions (Hafner et al., 2019; Balestriero and LeCun, 2025). ∗ [email protected]
2We use “model” to mean the transition model of a Markov decision process (Section 2) or an agent’s approximation to it, sometimes called a world model (Ha and Schmidhuber, 2018). We do not use it to mean the policy (as a generative model of actions) as is sometimes meant in other RL literature.
Preprint.
We argue that while this emphasis on performance is necessary for efficient planning, it is not sufficient for safe planning. Recent work on reward specification (Hadfield-Menell et al., 2017; Abel et al., 2021) views reward functions as observations of the designer’s intent to communicate a goal, which may be an imperfect reflection of a true objective. Empirical transition models are similarly imperfect. In this setting, the natural notion of safety is ordinal rather than cardinal (does the model preserve which policies are better than which?) and global rather than local (does it do so across the entire policy set, not just on the chosen classes?). Concretely, a proxy model is safe in our sense if it does not invert the policy ordering induced by the true model. That is, whenever reality prefers a policy π over π ′ , a safe approximation should not prefer π ′ over π. When such an inversion exists, we say the pair of transition models is exploitable (Definition 1). Our formalization of model exploitation is analogous to the notion of reward hacking in Skalse et al. (2022). Unfortunately, their characterization of when hacking exists and is avoidable does not transfer to exploitation (Section 3.2). We overcome this by introducing a more general theory, from which four main contributions emerge. 1. We propose a new definition of model exploitation (Definition 1) that, to the best of our knowledge, is the first ordinal notion of safety for world models. 2. We develop a unified theory of reward hacking and model exploitation that characterizes exploitation on common policy sets (Corollaries 1 and 2), yields Theorem 1 of Skalse et al. (2022) as a special case (Corollary 3), and provides a shared geometric intuition for both phenomena (Lemmas 1 and 2). 3. We give a construction reducing any instance of model exploitation to one of reward hacking (Proposition 3), but show that a converse construction does not exist (Counterexample 1). We further prove that the conditions that guarantee unhackability are insufficient to guarantee unexploitability (Counterexample 2). 4. We introduce a relaxed notion of model exploitation, ε-exploitability (Definition 3), and use the tight form of the simulation lemma (Kearns and Singh, 2002) recently proved by Lobel and Parr (2024) to derive a safe horizon for discounted Markov decision processes within which ε-unexploitability is guaranteed (Theorem 2).
2
Preliminaries
We first recap the preliminaries for reinforcement learning (Sutton et al., 1998) and introduce our definition of model exploitation. 2.1
Reinforcement learning
We consider a Markov decision process (MDP) M = (S, A, T , d0 , R, γ) where S is the state space, A is the action space, T : S × A → ∆(S) is the transition model, d0 ∈ ∆(S) is the initial state distribution, R : S × A → R is the reward function, and γ ∈ [0, 1) is the discount factor. As in Skalse et al. (2022), we assume finite S with all states reachable and finite A with |A| > 1.
A stationary policy π : S → ∆(A) maps each state to a distribution over actions. A non-stationary policy π = (π0 , π1 , π2 , . . . ) is a sequence of mappings πt : S → ∆(A), allowing the action distribution to depend on the time step. Rolling out a policy produces a trajectory τ = (s0 , a0 , r0 , . . . ) P∞ whose discounted return is G(τ ) = t=0 γ t rt . The value of a policy is its P expected return J(π) = ∞ Eτ ∼π [G(τ )], or, equivalently, J(π) = ⟨R, FTπ ⟩ where FTπ (s, a) = Eτ ∼π [ t=0 γ t 1(st = s, at = a)] are the discounted visit counts of π under T . We make the standard assumption in classical RL and optimal control (Sutton et al., 1998; Bertsekas, 2012) that designers use value to compare policies and prefer high-value policies over low-value ones. An environment is an MDP without a reward function (S, A, T , d0 , _, γ),3 and a task is an MDP without a transition model (S, A, _, d0 , R, γ). When considering multiple reward functions in a fixed environment, we write JR (π) = ⟨R, F π ⟩ (dropping the subscript on F π when T is fixed). When considering multiple transition models in a fixed task, we write JT (π) = ⟨R, FTπ ⟩. We use Ji as 3 This is sometimes called a controlled Markov process.
2
(a) T2 is trivial
(c) T1 , T2 exploitable
(d) T1 , T2 exploitable
J(π)
J1 J2
(b) T1 , T2 equivalent
0 π0 θ 1 0 1 0 π 1 0 π 1 θ θ θ π Figure 1: Taxonomy of transition model relationships in a 3-state MDP (Appendix D). Policies are parameterized by πθ (a0 | s) = θ for all s ∈ S. Each panel has the same T1 but a unique T2 . (a) T2 is trivial, so J2 is constant. (b) T1 , T2 are equivalent (in policy ordering): both value functions increase in θ, preserving the policy ordering. (c, d) T1 , T2 are exploitable: dotted lines show the exploiting pair (π, π ′ ) from Definition 1, with J1 (π ′ ) > J1 (π) but J2 (π) > J2 (π ′ ).
0
shorthand for JRi or JTi depending on whether the index ranges over reward functions or transition models. Since S and A are finite, the space of stationary policies can be represented as a product of |S| probability simplices ∆(A)|S| . We write Π+ for the interior of this space, consisting of all policies with π(a | s) > 0 for every s ∈ S and a ∈ A. Unless otherwise specified, openness is taken with respect to Π+ . We provide a reference list of notation and terminology in Appendix A. 2.2
Defining model exploitation and reward hacking
We define model exploitation as a binary relation between two transition models. Definition 1 (Model Exploitation). Transition functions T and T ′ are exploitable relative to a policy set Π and a task (S, A, _, d0 , R, γ) if there exist π, π ′ ∈ Π such that JT (π) > JT (π ′ )
and
JT ′ (π ′ ) > JT ′ (π),
otherwise they are unexploitable. Intuitively, model exploitation says that optimizing under one transition model may look like a mistake under the other. This definition directly analogizes the notion of reward hacking in Skalse et al. (2022) with transition models in place of reward functions. We restate their definition below. Definition 2 (Reward Hacking, Skalse et al. (2022)). Reward functions R and R′ are hackable relative to a policy set Π and an environment (S, A, T , d0 , _, γ) if there exist π, π ′ ∈ Π such that JR (π) > JR (π ′ )
and
JR′ (π ′ ) > JR′ (π),
otherwise they are unhackable. Both definitions are instances of a common primitive, which we call value inversion. We say two value functions J1 and J2 admit a value inversion on Π if there exist π, π ′ ∈ Π such that J1 (π) > J1 (π ′ ) and J2 (π ′ ) > J2 (π). Thus, model exploitation is a value inversion between JT and JT ′ , and reward hacking is a value inversion between JR and JR′ .
We further say J1 and J2 are equivalent on Π if they induce the same ordering on Π and that a value function J is trivial on Π if J(π) is constant on Π (Figure 1). Informally, equivalence means that two value functions agree on every pairwise comparison between policies, while triviality means there are no meaningful comparisons to make. We also use trivial and equivalent to describe transition models and reward functions as shorthand for the corresponding properties of the value functions they induce. We note that triviality precludes value inversion and that value inversion is symmetric, irreflexive, and not necessarily transitive.
3
Results
To understand when model exploitation is avoidable, we ask a single question with increasing precision: on which policy sets can we find non-trivial, non-equivalent, unexploitable transition model pairs? We begin with the largest possible policy set and progressively tighten the scope. 3
3.1
Non-stationary policies
Consider first the set of all non-stationary policies ΠNS . If exploitation can be avoided here, it can be avoided on any policy set, since all policy sets are subsets of ΠNS . Unfortunately, this is false: we find that no interesting unexploitability exists. Proposition 1 (No unexploitability on ΠNS ). On the set of all non-stationary policies, every nontrivial, non-equivalent pair of transition models is exploitable. We defer all formal proofs to Appendix B. The proof uses a similar construction to the one in Skalse et al. (2022), relying on the richness of ΠNS to find an exploiting policy pair, but says nothing about the set of stationary policies, where the question is considerably harder. 3.2
Distinguishing model exploitation from reward hacking
Having seen that interesting unexploitability is impossible in the most general policy set, a natural question is whether it can be avoided in smaller policy sets, such as deterministic or optimal policies. From Theorem 1 of Skalse et al. (2022), we know that reward hacking is inevitable on any policy set that contains an open subset (under mild regularity conditions). Given that our proof of Proposition 1 follows a similar construction to the one in Skalse et al. (2022), it is reasonable to expect that we might derive analogous results using the same or comparable techniques. Unfortunately, this is not true, as we argue in this section. 3.2.1
Distinguishing transition models and reward functions
Transition models and reward functions have significantly different geometries that make it hard to reason about them in a unified manner. Two of these differences are important for our results. First, the policy value function J(π) = ⟨R, FTπ ⟩ is linear in the reward function but nonlinear in the transition model’s probability mass function. This means that the linear-algebraic and real-analytic machinery that Skalse et al. (2022) develop for their characterization of reward hacking generally does not transfer to the model exploitation setting, so analogous results (if they exist) must rely on different methods. Second, the only constraint imposed on the reward function is typically boundedness: rewards may not go to infinity. Transition models, on the other hand, are probability mass functions, which are nonnegative and sum to 1 at each state. This means reward functions live in the full Euclidean space R|S||A| , which is closed under negation, scaling, and addition, but transition models live on a compact product of simplices ∆(S)|S||A| , which does not have vector space structure (at least in the probability domain). The primary consequence of these differences is that neither the techniques nor the results from reward hacking cleanly apply to model exploitation. In fact, as we show next, results that are true for hacking are sometimes false for exploitation. 3.2.2
Finite policy sets
Unlike reward hacking, model exploitation persists even in finite policy sets. Formally, Theorem 2 of Skalse et al. (2022) shows that, together with some mild regularity conditions, a non-trivial, non-equivalent unhackable reward pair always exists on any finite policy set. Making the corresponding existence guarantee for unexploitable transition model pairs requires ruling out that have no analogue for reward hacking. When γ = 0, the value J(π) = P two degeneracies P s d0 (s) a π(a|s) R(s, a) is independent of T , so every transition model pair is equivalent. The same obstruction applies when both R and every policyP in Π are stateless (i.e., R(s, a) = R(a) and 1 π(a|s) = π(a) for all s ∈ S), since then J(π) = 1−γ a R(a) π(a). In both cases, no transition model pairs are non-equivalent. In examining whether the remaining regularity conditions of Skalse et al. (2022) suffice once these degeneracies are excluded, we noticed that Theorem 2 requires an additional hypothesis that is not present in the original statement but implicitly used in the proof. (We view this as a minor oversight that does not affect the broader conclusions of their work, since the condition is degenerate and easily excluded.) 4
Proposition 2 (Non-collinearity is necessary for Theorem 2 of Skalse et al. (2022)). On any policy set with collinear visit counts (i.e., all visit counts F π for π ∈ Π lie on a single line in R|S||A| ), every non-trivial, non-equivalent pair of reward functions is hackable. We provide a corrected statement in Appendix C with the additional hypothesis that visit counts are not collinear. Unfortunately, even assuming all of the regularity conditions above together with those of Theorem 2, this is still insufficient to guarantee the existence of a non-trivial, non-equivalent unexploitable transition model pair on finite policy sets (see Counterexample 2 in Appendix B). We suspect that the finite-policy-set case admits no clean existence guarantee for unexploitability, because the nonlinear dependence of J on T makes the geometry of the induced value orderings fundamentally richer than in the reward hacking setting. Question 1. Are there necessary and sufficient conditions on a task (S, A, _, d0 , R, γ) and finite policy set that guarantee the existence of a non-trivial, non-equivalent, unexploitable pair of transition models? 3.3
Unifying model exploitation and reward hacking
We next return to our main question: can exploitability be avoided by restricting the policy set? As discussed in Section 3.2 above, Theorem 1 of Skalse et al. (2022) shows that the answer is no for reward hacking on any policy set containing an open subset, but the result does not directly apply to exploitation. Surprisingly, this does not mean that reward hacking has nothing to tell us about model exploitation. In this section, we first show that every instance of model exploitation can be reduced to an instance of reward hacking, then develop this insight into a unified theory of exploitation and hacking. Proposition 3 (Exploitation implies hacking). For any exploitable pair (T , T ′ ) relative to a policy set Π and a task (S, A, _, d0 , R, γ), there exists a reward function R′ such that (R, R′ ) is hackable relative to Π and the environment (S, A, T , d0 , _, γ). While the result is perhaps surprising, the proof is a short construction: set R′ to be the difference in ′ visit counts FTπ − FTπ for some exploiting policy pair π, π ′ ∈ Π. Interestingly, the converse does not hold. Counterexample 1 (Hacking may not imply exploitation). Consider any MDP with |A| ≥ 2 and γ > 0. Let R(s, a) = 1{a = a1 } and R′ (s, a) = 1{a = a2 }, and let π always select a1 and π ′ always select a2 . Then (R, R′ ) is hackable, since JR (π) = 1/(1 − γ) > 0 = JR (π ′ ) and JR′ (π ′ ) = 1/(1 − γ) > 0 = JR′ (π), but no perturbation of the transition model can alter the relative ordering of π and π ′ under either reward function. Proposition 3 is unassuming by itself, but it hints at something more fundamental: a unified theory of model exploitation and reward hacking may exist despite the anatomical differences between reward functions and transition models. If we can find such a theory, we may also discover an answer to our initial question. Both model exploitation and reward hacking are inversions in the value ordering of a policy set. This suggests that the correct place to begin developing a unified treatment of exploitation and hacking is the value function. Unfortunately, the difficulties that make transition models unwieldy relative to reward functions (Section 3.2.1) also apply to the value function, so we still cannot apply the machinery of Skalse et al. (2022). The key insight to breaking past this obstacle is that the value function is rational in policy space. Proposition 4 (Value function rationality). For any MDP with finite S and A, the value function J(π) is rational in π. In particular, J is real-analytic on Π+ . Rational functions are typically easier to reason about compared to nonlinear functions in general because they have a host of desirable properties (Rudin, 1974). The two properties that are useful for our purposes are analyticity and rigidity. The former provides access to an arsenal of powerful results from analysis and differential geometry, while the latter allows us to promote properties from open subsets to the supersets containing them. More concretely, from these two properties, we can derive the following lemmas, which relate the gradients of value functions to the existence (or lack thereof) of inversions and give a simple 5
(b) T1 , T2 equivalent
(c) T1 , T2 exploitable
∇J
(a) T2 is trivial
(d) T1 , T2 exploitable
0
∇J1 ∇J2
0
0
0
0
1 0 1 0 1 0 1 θ θ θ θ Figure 2: Gradients for the value curves in Figure 1. (a) ∇J2 vanishes (trivial). (b) Both gradients positive: proportional gradients force equivalence (Lemma 2). (c, d) The gradients have opposite signs, yielding exploitation (Lemma 1). In (c) this occurs only near θ = 0; in (d) it holds everywhere.
geometric picture: at every point in policy space, the relationship between ∇J1 and ∇J2 determines whether the two value functions can disagree on which policy is better.4 Lemma 1 (Local inversion). Let Π be open. If there is a direction v ∈ R|S|(|A|−1) along which J1 increases and J2 decreases at some π ∈ Π (i.e., ∇v J2 (π) < 0 < ∇v J1 (π)), then J1 and J2 admit an inversion on Π. Lemma 2 (Global equivalence). Let Π̃ ⊆ Π be open. If J1 and J2 are non-trivial and have positively proportional gradients wherever both are nonzero on Π̃, then J1 and J2 are equivalent on Π. The intuition for Lemma 1 is as follows. If there is a direction in policy space that improves performance under one model but decreases it under the other, then a step in that direction leads to two policies that form an exploiting pair. Lemma 2 has a similar intuition. If every direction that improves one value function also improves the other, then the two can never disagree on which of any two policies is better, so they must induce the same ordering. While the result is simple, the formal proof is surprisingly involved and relies heavily on value function rationality. Together, these two lemmas cover every possible relationship between ∇J1 and ∇J2 (Figure 2). At every point in policy space, the gradients are either linearly independent, antiparallel, or positively proportional. The first two cases produce an inversion via Lemma 1; the third forces equivalence via Lemma 2. This gives our main characterization. Theorem 1 (Value inversions). Let J1 and J2 be non-trivial, non-equivalent value functions. If the policy set contains an open subset, then it admits a value inversion for J1 and J2 . The theorem immediately yields the desired characterization for model exploitation. Corollary 1 (Imperfect world models are exploitable). On any policy set containing an open subset, every non-trivial, non-equivalent pair of transition models is exploitable. In practice, one might hope that Corollary 1 only applies to high-entropy or poorly performing policies that no reasonable learning algorithm would ever produce. Unfortunately, Theorem 1 still applies both to the set of ε-suboptimal policies (those π with J(π) ≥ supπ′ J(π ′ ) − ε for ε > 0) and to the set of δ-deterministic policies (those π with maxa π(a | s) ≥ δ for δ < 1 and every s ∈ S). Corollary 2 (Common exploitable policy sets). Every non-trivial, non-equivalent pair of transition models is exploitable on (i) the set of all stationary policies, (ii) the set of all ε-suboptimal policies (ε > 0), and (iii) the set of all δ-deterministic policies (δ < 1). Theorem 1 implies characterizations of both reward hacking and model exploitation. Skalse et al. (2022) arrive at the reward hacking result through an argument in the visit-count space, exploiting the linearity of J in the reward vector. The same conclusion follows a fortiori from Theorem 1, since the only property of J1 and J2 used in the proof is rationality in π, which holds regardless of whether the index ranges over reward functions or transition models. Corollary 3 (Theorem 1 of Skalse et al. (2022)). On any policy set containing an open subset, every non-trivial, non-equivalent pair of reward functions is hackable. 4 From now on, we identify the tangent space to the policy space ∆(A)|S| at any π with R|S|(|A|−1) , although the
identification is not canonical. We can thus write ∇v J(π) for v ∈ R|S|(|A|−1) an infinitesimal perturbation of π.
6
(a) 0.3-exploitable
(b) 2-exploitable
(c) Safe horizon H(ε, δ) 10 10
ε
J(π)
H =
H =5
J1 J2 0π
0
π θ
0
π
π
H =2
0
1 δ(T1 , T2 ) Figure 3: Examples of ε-exploitability and a contour plot for the safe horizon. (a, b) The exploitable transition model pairs from Figure 1(c, d), annotated with exploiting policies. The pair in (a) is 0.3exploitable and the pair in (b) is 2-exploitable. (c) The safe horizon H(ε, δ) from Theorem 2. Each contour marks the largest effective horizon 1/(1 − γ) under which ε-unexploitability is guaranteed for a given tolerance ε and maximum total variation distance δ(T1 , T2 ).
3.4
1
0
θ
1
0
Relaxation
We now introduce a relaxed notion of exploitability to pursue two goals. First, practitioners may not care about small-value inversions in real-world applications, so we seek a more graded measure of exploitability that distinguishes between negligible and catastrophic differences in value. Second, we want to answer our initial question about when exploitation is avoidable, even if only approximately. Thus far, we have seen that interesting unexploitability is impossible on large policy sets (Corollary 1) and resists characterization on finite ones (Section 3.2.2). A weaker definition, in contrast, may be more likely to admit positive existence conditions than our usual notion of exploitation. Definition 3 (ε-Exploitation). Transition models T and T ′ are ε-exploitable relative to a policy set Π and a task (S, A, _, d0 , R, γ) if there exist π, π ′ ∈ Π such that JT (π) − JT (π ′ ) > ε
and
JT ′ (π ′ ) − JT ′ (π) > ε,
otherwise they are ε-unexploitable. Intuitively, ε-unexploitability says that while small inversions may exist, T and T ′ may never confidently disagree about how to rank a policy pair. This definition immediately satisfies our first goal, because it generalizes exploitation (recovering Definition 1 when ε = 0) and is a continuous rather than binary measure of “exploitability” (i.e., via the minimum ε for which a pair of transition models remains ε-unexploitable). To address our second goal and establish sufficient conditions for ε-unexploitability, we derive the following theorem. Theorem 2 (Safe horizon). Let R : S × A → [0, 1] and let δ = 12 maxs,a ∥T (·|s, a) − T ′ (·|s, a)∥1 be the total variation distance between distinct T and T ′ . For any ε > 0, define the safe horizon p (1 + ε) + (1 − ε)2 + 4ε/δ H(ε, δ) = . 2 Every pair (T , T ′ ) is ε-unexploitable on every policy set Π whenever 1/(1 − γ) ≤ H(ε, δ). Furthermore, this bound is tight. The safe horizon H(ε, δ) is the longest effective horizon 1/(1 − γ) at which ε-unexploitability is guaranteed. Its dependence on both arguments aligns with standard intuitions in model-based RL. Increasing δ lowers H: a less accurate model exhausts its error budget sooner, so safe planning requires a shorter horizon. This is exactly in line with the main result by Jiang et al. (2015), who show that a shorter effective horizon should be used when data is scarce. In this sense, we corroborate and expand their findings through the lens of ε-exploitability. Increasing ε raises H: a designer who tolerates larger inversions can plan further ahead. At the extremes, H → ∞ as δ → 0 (a perfect model is never exploitable) and H → 1 + ε as δ → 1 (a maximally inaccurate model is safe only near single-step planning). This formalizes the widely held intuition that model errors compound over the planning horizon (Ross et al., 2011; Talvitie, 2014), 7
and provides a closed-form expression for exactly how much horizon a given level of model error can afford (Figure 3c). The closed form of H is exact but unwieldy. A looser but more memorable sufficient condition follows from dropping lower-order terms. Corollary 4 (Square-root heuristic). Let R : S × A → [0, 1] and let δ = 12 maxs,a ∥T (·|s, a) − p T ′ (·|s, a)∥1 > 0. For any ε > 0, the pair (T , T ′ ) is ε-unexploitable whenever 1/(1 − γ) < ε/δ. Corollary 4 gives a quick sense of the magnitude of possible inversions: at γ = 0.9 the effective horizon is 10, so the worst-case inversion that cannot be ruled out scales as 100δ; at γ = 0.99 it scales as 10,000δ. Unlike the bound in Theorem 2, the approximate bound in Corollary 4 is not tight, so the actual inversion may be much smaller, but it provides an immediate sense of how model error and planning horizon interact without requiring further analysis. For the transition model pairs in Figure 1, we compare both bounds against the actual exploitation gaps in Appendix E.
4
Related work
History and etymology of model exploitation. The term “model exploitation” has no single origin in RL. Adjacent concepts, such as “model bias” and “compounding error,” have appeared in the literature since at least the early 2010s (Deisenroth and Rasmussen, 2011; Ross et al., 2011), themselves inherited from control theory (Schneider, 1996; Kappen, 2005). As deep RL grew in popularity toward the end of that decade, practitioners turned to sample-efficient model-based RL (MBRL) to meet rising data demands. Learned models, however, are uncertain where data is scarce, and unlike in model-free methods, no trial-and-error mechanism exists to correct the resulting errors. Researchers began describing policies as “exploiting” these uncertain regions (Kurutach et al., 2018), though this usage of “exploit” also predates deep RL (cf. Schneider 1996). The set phrase “model exploitation” congealed only more recently, following Ha and Schmidhuber (2018) and Janner et al. (2019), and has since motivated a lineage of conservative offline MBRL methods (Yu et al., 2020; Kidambi et al., 2020). Other adjacent phenomena. Model exploitation overlaps with several independent concepts. Planner overfitting (Arumugam et al., 2018; Jiang et al., 2015) refers to a planner discovering spurious shortcuts in a learned model and develops regularization-based remedies. Jiang et al. (2015) were the first to formalize and isolate this phenomenon by highlighting the role that the effective horizon plays in regularizing a learned transition model when data is scarce. The sim-to-real gap (Jakobi et al., 1995; Tobin et al., 2017) describes the same failure when transferring controllers from simulated to physical environments, typically addressed through domain randomization or system identification. Objective robustness (Langosco et al., 2021) studies agents that achieve high reward in training environments but pursue unintended objectives at deployment, which can be viewed as exploitation of a training-time model of the deployment distribution. Lastly, as discussed, the value equivalence principle (Grimm et al., 2020, 2021) provides a framing of when two models can be well-understood as equivalent in terms of value preservation under the Bellman operator. This principle is closely tied to our notion of equivalent transition models (Figure 1). MDP extensions. Several lines of work extend the MDP formalism to defend against imperfect transition models, typically by modifying the optimization problem rather than studying the misspecification itself. Robust MDPs (Iyengar, 2005; Nilim and El Ghaoui, 2005) represent transition uncertainty through uncertainty sets and optimize worst-case value, a cardinal safety criterion rather than the ordinal one we propose. MOReL (Kidambi et al., 2020) constructs a pessimistic MDP by introducing absorbing states at high-uncertainty transitions, and provides strong suboptimality guarantees for the resulting policy, but does not formalize the notion of exploitation it defends against. Both modify the MDP to cope with model uncertainty; our work instead characterizes when that uncertainty inverts the policy ordering. World models. A growing family of methods trains policies entirely inside learned dynamics models, from the Dreamer line (Hafner et al., 2020, 2021, 2025a,b) to MuZero (Schrittwieser et al., 2020) to recent JEPA-style predictive representations (Balestriero and LeCun, 2025). These methods measure model quality through predictive accuracy or downstream task performance, both of which are cardinal. Our results show that low prediction error does not preclude ordinal failures: a model can be accurate yet still invert the policy ordering induced by the true dynamics (Theorem 2). 8
Outside machine learning. Optimizing against an imperfect model is a recurring failure mode across disciplines, so there is an extensive literature that exists outside machine learning. For example, in macroeconomics, the Lucas critique (Lucas, 1976) warns that policy rules derived from an estimated model can fail when agents adapt to the policy itself, and robust control (Hansen and Sargent, 2008) responds by optimizing against worst-case model perturbations. In sim-to-real robotics, controllers optimized in simulation routinely exploit discrepancies with real-world physics (Jakobi et al., 1995; Tobin et al., 2017). Our formalization of exploitation as a value inversion applies in principle to any setting where a proxy model guides optimization over a structured decision space.
5
Conclusion
There is increasing interest in approaches to decision-making that combine a forward predictive model with planning. Although there is substantial literature on the design and analysis of such model-based algorithms, the emergence of world models that make predictions directly from rich low-level sensor streams pushes the boundaries of these methods. This calls for a fresh look at the questions of model robustness and safety of the computed policies. In this paper, we formalize the concept of model exploitation and study the conditions under which policies trained on empirical models may be safe. We find that in sufficiently rich policy classes, exploitability is inevitable. However, under a relaxed definition, we can establish conditions for unexploitability. We suggest that this could guide the development of new approaches to safe policy synthesis, as well as parallel methods for quantifying the quality of synthesized policies with respect to such safety objectives. Limitations and future work. We identify three main limitations of our work. First, our results, like those in Skalse et al. (2022), concern MDPs with finite state and action spaces. We defer extensions to continuous and partially observable MDPs (POMDPs) to future work, noting that solving the former may yield the latter (since every POMDP can be written as a continuous beliefstate MDP (Åström, 1965)). Second, also like in Skalse et al. (2022), our notion of exploitation is binary and restrictive. In practice, it is unlikely that small, rare value inversions will pose meaningful safety concerns. Future work could refine our notion of exploitation by introducing a more granular measure of exploitability (e.g., deriving bounds using correlation metrics (Laidlaw et al., 2025) or expanding upon ε-exploitation). Third, there are, as of yet, no known closed-form necessary and sufficient conditions that guarantee unexploitability in finite policy sets. Although we question whether such simple conditions exist (Question 1), there is still area to explore. Researchers interested in pursuing this direction may, for instance, find it productive to begin work on special finite policy sets and specific MDPs (e.g., characterize exploitation on deterministic policies in linear MDPs).
References David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of Markov reward. Neural Information Processing Systems (NeurIPS), 2021. Athanasios C Antoulas. Approximation of large-scale dynamical systems. SIAM, 2005. Dilip Arumugam, David Abel, Kavosh Asadi, Nakul Gopalan, Christopher Grimm, Jun Ki Lee, Lucas Lehnert, and Michael L Littman. Mitigating planner overfitting in model-based reinforcement learning. arXiv preprint arXiv:1812.01129, 2018. Karl Johan Åström. Optimal control of Markov processes with incomplete state information i. Journal of Mathematical Analysis and Applications, 10:174–205, 1965. Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. Henry Walter Bates. Contributions to an insect fauna of the amazon valley. Lepidoptera: Heliconidæ. Transactions of the Linnean Society of London, 23(3):495–566, 1862. Dimitri Bertsekas. Dynamic Programming and Optimal Control: Volume I, volume 4. Athena scientific, 2012. 9
Marc Deisenroth and Carl E Rasmussen. PILCO: A model-based and data-efficient approach to policy search. International Conference on Machine Learning (ICML), 2011. Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for model-based reinforcement learning. Neural Information Processing Systems (NeurIPS), 2020. Christopher Grimm, André Barreto, Greg Farquhar, David Silver, and Satinder Singh. Proper value equivalence. Neural Information Processing Systems (NeurIPS), 2021. David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Neural Information Processing Systems (NeurIPS), 2018. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Neural Information Processing Systems (NIPS), 2017. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning (ICML), 2019. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations (ICLR), 2020. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. International Conference on Learning Representations (ICLR), 2021. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647–653, 2025a. Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025b. Lars Peter Hansen and Thomas J. Sargent. Robustness. Princeton University Press, 2008. Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107, 1968. Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257–280, 2005. Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary robotics. In European Conference on Artificial Life, pages 704–720. Springer, 1995. Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Neural Information Processing Systems (NeurIPS), 2019. Khurram Javed and Richard S Sutton. The big world hypothesis and its ramifications for artificial intelligence. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024. Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. Autonomous Agents and Multiagent Systems (AAMAS), 2015. Hilbert J Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005(11):P11011–P11011, 2005. Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2):209–232, 2002. Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Modelbased offline reinforcement learning. Neural Information Processing Systems (NeurIPS), 2020. Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. International Conference on Learning Representations (ICLR), 2018. 10
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. International Conference on Learning Representations (ICLR), 2025. Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, and David Krueger. Goal misgeneralization in deep reinforcement learning. International Conference on Machine Learning (ICML), 2021. Sam Lobel and Ronald Parr. An optimal tightness bound for the simulation lemma. Reinforcement Learning Conference (RLC), 2024. Robert E Lucas, Jr. Econometric policy evaluation: A critique. In Carnegie-Rochester conference series on public policy, volume 1, pages 19–46. North-Holland, 1976. Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. Artificial Intelligence and Statistics (AISTATS), 2011. Stephen A Ross. The arbitrage theory of capital asset pricing. Journal of Economic Theory, 13(3): 341–360, 1976. Walter Rudin. Real and complex analysis. 1974. Jeff Schneider. Exploiting model uncertainty estimates for safe dynamic control learning. Neural Information Processing Systems (NIPS), 1996. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. Rainforest Scully-Blaker. A practiced practice: Speedrunning through space with de certeau and virilio. Game Studies, 14(1):2016, 2014. Herbert A Simon. A behavioral model of rational choice. The Quarterly Journal of Economics, pages 99–118, 1955. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. Neural Information Processing Systems (NeurIPS), 2022. Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT Press Cambridge, 1998. Erik Talvitie. Model regularization for stable sample rollouts. In UAI, pages 780–789, 2014. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. International Conference on Intelligent Robots and Systems (IROS), 2017. Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Neural Information Processing Systems (NeurIPS), 2020.
11
A
Notation
We first provide a table summarizing all relevant notation. Notation
Meaning
Definition
S
State space
T
Transition model
d0
Initial state distribution
R
Reward function
Action space
A
T : S × A → ∆(S) d0 ∈ ∆(S)
R:S ×A→R
γ
Discount factor
γ ∈ [0, 1)
M
Markov decision process
(S, A, T , d0 , R, γ)
M\R
Environment (MDP without reward)
M\T
Task (MDP without transitions)
π
Stationary policy
Π
Policy set
Π+
Interior of all stationary policies
Π̂
Finite policy set
FTπ (s, a)
Discounted visit counts
J(π)
JT (π) JR (π) Ji (π) V π (s)
(S, A, T , d0 , _, γ)
(S, A, _, d0 , R, γ) π : S → ∆(A)
{π ∈ ∆(A)|S| : π(a|s) > 0 ∀s∈S,a∈A }
P∞
t t=0 γ π ⟨R, FT ⟩ ⟨R, FTπ ⟩ ⟨R, F π ⟩
Policy value Value under transition model T
Value under reward function R Value under Ri or Ti
1(st =s, at =a)
(context determines index)
State-value function
rπ (s)
Per-state expected reward
Eτ ∼π [G(τ ) | s0 = s] P a π(a|s) R(s, a)
(R, R′ ) hackable
Reward hacking
∃π,π′ ∈Π JR (π) > JR (π ′ ) and JR′ (π ′ ) > JR′ (π)
(T , T ′ ) exploitable
Model exploitation
∃π,π′ ∈Π JT (π) > JT (π ′ ) and JT ′ (π ′ ) > JT ′ (π)
Equivalent
Same policy ordering
Trivial
Constant value
∀π,π′ ∈Π J1 (π) ≥ J1 (π ′ ) ⇔ J2 (π) ≥ J2 (π ′ ) ∀π,π′ ∈Π J(π) = J(π ′ )
Table 1: A summary of notation.
B
Proofs
Proposition 1 (No unexploitability on ΠNS ). On the set of all non-stationary policies, every nontrivial, non-equivalent pair of transition models is exploitable. Proof of Proposition 1. The proof follows the same argument as the analogous result in Skalse et al. (2022), with transition models in place of reward functions. The key observation is that Ji (πλ ) is affine in λ by linearity of expectation, regardless of whether i indexes reward functions or transition models. Proposition 2 (Non-collinearity is necessary for Theorem 2 of Skalse et al. (2022)). On any policy set with collinear visit counts (i.e., all visit counts F π for π ∈ Π lie on a single line in R|S||A| ), every non-trivial, non-equivalent pair of reward functions is hackable.
12
Proof of Proposition 2. Collinearity means there exist π0 , π1 ∈ Π with F π0 ̸= F π1 such that F π = F π0 + cπ (F π1 − F π0 ) for distinct scalars cπ . The value of π under any reward R is JR (π) = JR (π0 ) + cπ ⟨R, F π1 − F π0 ⟩, which is affine in cπ . Since the cπ are distinct, the ordering of Π under R is determined entirely by the sign of ⟨R, F π1 − F π0 ⟩. Given a non-trivial R, any R′ either has the same sign (same ordering, so equivalent), the opposite sign (reversed ordering, so hackable), or no sign (R′ trivial on Π). A non-trivial, non-equivalent pair therefore cannot be unhackable on Π. Counterexample 2. Consider the task (S, A, _, d0 , R, γ) with S = {s0 , s1 }, A = {a0 , a1 }, d0 = (1/2, 1/2), R(s, a) = 1{s = s0 , a = a0 } + 1{s = s1 , a = a1 }, and any γ ∈ (0, 1). Let Π̂ = {π0 , π1 , π2 } where π0 plays a0 at both states, π1 plays a0 at s0 and a1 at s1 , and π2 plays a1 at s0 and a0 at s1 . This task satisfies γ > 0, R is non-trivial and state-action-dependent, and the visit counts of Π̂ are non-collinear under any transition model: F π0 , F π1 , and F π2 are supported on {(s0 , a0 ), (s1 , a0 )}, {(s0 , a0 ), (s1 , a1 )}, and {(s0 , a1 ), (s1 , a0 )} respectively, and no affine combination can reconcile the disjoint supports. Under π1 , we see that rπ1 (s) = 1 for both states, so JT (π1 ) = 1/(1 − γ) for every T . Under π2 , we see that rπ2 (s) = 0 for both states, so JT (π2 ) = 0 for every T . Under π0 , rπ0 (s0 ) = 1 and rπ0 (s1 ) = 0, so JT (π0 ) = µπT0 (s0 ) where µπT0 (s) is the discounted visitation count of state s. Since d0 (s0 ) = 1/2, the t = 0 contribution gives µπT0 (s0 ) ≥ 1/2 > 0. Since d0 (s1 ) = 1/2, we also have µπT0 (s1 ) ≥ 1/2, so µπT0 (s0 ) = 1/(1 − γ) − µπT0 (s1 ) < 1/(1 − γ). Therefore, for every T , 1 = JT (π1 ) > JT (π0 ) > 0 = JT (π2 ), 1−γ and every pair of transition models is equivalent on Π̂. Proposition 3 (Exploitation implies hacking). For any exploitable pair (T , T ′ ) relative to a policy set Π and a task (S, A, _, d0 , R, γ), there exists a reward function R′ such that (R, R′ ) is hackable relative to Π and the environment (S, A, T , d0 , _, γ).
Proof of Proposition 3. Since (T , T ′ ) is exploitable, there exist π, π ′ ∈ Π with JT (π) > JT (π ′ ) and JT ′ (π ′ ) > JT ′ (π). Working in the environment (S, A, T , d0 , _, γ), the original reward R ′ satisfies JR (π) = JT (π) > JT (π ′ ) = JR (π ′ ). Define R′ = FTπ − FTπ . Since JT (π) ̸= JT (π ′ ), ′ the visit counts FTπ and FTπ are distinct, so R′ ̸= 0. Then ′
′
JR′ (π ′ ) − JR′ (π) = ⟨R′ , FTπ − FTπ ⟩ = ∥FTπ − FTπ ∥2 > 0. Therefore JR (π) > JR (π ′ ) and JR′ (π ′ ) > JR′ (π), so (R, R′ ) is hackable.
Proposition 4 (Value function rationality). For any MDP with finite S and A, the value function J(π) is rational in π. In particular, J is real-analytic on Π+ . P Proof of PropositionP4. Write J(π) = s d0 (s) V πP (s); it suffices to show each V π (s) is rational π in π. Let R (s) = a π(a|s) R(s, a), P π (s′ |s) = a π(a|s) T (s′ |s, a), and A = I − γP π . The Bellman equation in matrix form is AV π = Rπ . Since P π is a probability matrix and γ ∈ [0, 1), A is invertible and V π = A−1 Rπ . By Cramer’s rule, V π can then be expressed as a ratio of determinants and is thus rational as required. Lemma 1 (Local inversion). Let Π be open. If there is a direction v ∈ R|S|(|A|−1) along which J1 increases and J2 decreases at some π ∈ Π (i.e., ∇v J2 (π) < 0 < ∇v J1 (π)), then J1 and J2 admit an inversion on Π. Proof of Lemma 1. Define π ± = π ± εv for ε > 0. Since each Ji is analytic (Proposition 4), first-order Taylor expansion gives Ji (π ± ) = Ji (π) ± ε⟨∇Ji (π), v⟩ + O(ε2 ) and subtracting yields J1 (π + ) − J1 (π − ) = 2ε⟨∇J1 (π), v⟩ + O(ε2 )
J2 (π − ) − J2 (π + ) = −2ε⟨∇J2 (π), v⟩ + O(ε2 ) Since ⟨∇J1 (π), v⟩ > 0 and ⟨∇J2 (π), v⟩ < 0, we can make ε small enough so that both differences are positive and π ± ∈ Π (openness). Thus, T1 and T2 are exploitable. 13
Proposition 5 (Connected level sets). For every transition model T and c ∈ R, the level set {JT = c} ∩ Π+ is connected. Proof of Proposition 5. Since the map π 7→ F π is a homeomorphism on Π+ (Skalse et al., 2022, Lemma 1), it suffices to show connectedness in visit-count space. The set F(Π+ ) is the interior of a polyhedron (defined by non-negativity and Bellman flow constraints), hence convex. Since JT (f ) = ⟨R, f ⟩ is linear, {⟨R, f ⟩ = c} ∩ F(Π+ ) is a hyperplane intersected with a convex set, which is convex and hence connected. Lemma 2 (Global equivalence). Let Π̃ ⊆ Π be open. If J1 and J2 are non-trivial and have positively proportional gradients wherever both are nonzero on Π̃, then J1 and J2 are equivalent on Π. Proof of Lemma 2. We show that positive proportionality of gradients on Π̃ forces equivalence on all of Π. The idea is to first extend proportionality from Π̃ to all of Π+ using analyticity and connectedness, then construct a strictly increasing reparametrization J2 = φ ◦ J1 on Π+ , and finally extend this to Π by density. We begin by extending the proportionality to Π+ . We want to establish that ∇J2 = λ∇J1 with λ > 0 on all of Π+ . This requires showing that on Π+ (i) the gradients are linearly dependent, (ii) the gradients are nonzero, and (iii) λ is positive. For (i), the Gram determinant of ∇J1 and ∇J2 is real analytic on the domain Π+ (Proposition 4) and vanishes on the open subset Π̃, so it vanishes identically on Π+ . The gradients are therefore linearly dependent everywhere on Π+ . For (ii), write fi : π 7→ FTπi . The chain rule gives ∇Ji (π) = Dfi (π)⊤ Ri . The map fi is a homeomorphism from Π+P onto its image (Skalse et al., 2022, Lemma 1). Both fi and its inverse F 7→ π(a|s) = F (s, a)/ a′ F(s, a′ ) are smooth (the former by rationality, the latter since all visitations are positive on Π+ ), so fi is in fact a diffeomorphism onto its image (in particular, a smooth immersion) and Dfi (π) has full column rank. Non-triviality of Ji ensures R is not orthogonal to the column space of Dfi (π), since otherwise Ji would be constant. Therefore ∇Ji (π) = Dfi (π)⊤ R ̸= 0 on Π+ . For (iii), since both gradients are nonzero we may write ∇J2 = λ∇J1 where λ(π) = ⟨∇J2 , ∇J1 ⟩/∥∇J1 ∥2 is continuous and nonzero on Π+ . Since Π+ is connected, λ has constant sign. Since λ > 0 on Π̃ by hypothesis, λ > 0 on Π+ . Having extended positive proportionality of the gradients to Π+ , we now construct a strictly increasing φ with J2 = φ ◦ J1 on Π+ . Such a φ exists if and only if J1 (π) = J1 (π ′ ) implies J2 (π) = J2 (π ′ ) for all π, π ′ ∈ Π+ . Pick any such pair π, π ′ . Both policies lie in the same level set of J1 in Π+ , and the level set is connected (Proposition 5), so there is a path α : [0, 1] → Π+ in the level set with d α(0) = π and α(1) = π ′ . Since J1 (α(t)) is constant, differentiating gives ⟨∇J1 (α(t)), dt α(t)⟩ = 0. d Since ∇J2 = λ∇J1 , we also have ⟨∇J2 (α(t)), dt α(t)⟩ = 0, so J2 (α(t)) is constant on [0, 1], giving J2 (π) = J2 (α(0)) = J2 (α(1)) = J2 (π ′ ). It remains to show that such a φ is strictly increasing on the image of J1 . Fix c ∈ J1 (Π+ ) and pick π0 with J1 (π0 ) = c. Since ∇J1 (π0 ) ̸= 0, we can choose a direction v with ⟨∇J1 (π0 ), v⟩ ̸= 0 and d define h(x) = J1 (π0 + xv). Then h(0) = c and dx h(0) ̸= 0, so by the inverse function theorem h is −1 locally invertible. Setting β(t) = π0 +h (t)v gives a smooth curve with J1 (β(t)) = h(h−1 (t)) = t near c. Then φ(t) = J2 (β(t)), so d d d φ(t) = ⟨∇J2 (β(t)), dt β(t)⟩ = λ(β(t))⟨∇J1 (β(t)), dt β(t)⟩ = λ(β(t)), dt d where the last equality uses dt J1 (β(t)) = 1. Since λ > 0, φ is locally strictly increasing at every point of J1 (Π+ ), hence strictly increasing on J1 (Π+ ).
Finally, we use the density of Π+ in the space of all stationary policies to extend to Π. Define φ̄(u) = limn→∞ φ(un ) for any sequence (un ) ⊆ J1 (Π+ ) with un → u; this is well-defined because φ is bounded and monotone. By continuity of J2 , we have φ̄(J1 (π)) = limn→∞ φ(J1 (πn )) = limn→∞ J2 (πn ) = J2 (π) for any π ∈ Π and any sequence (πn ) ⊆ Π+ with πn → π. The extension φ̄ is strictly increasing: given a < d in J1 (Π), density of J1 (Π+ ) provides b, c ∈ J1 (Π+ ) with 14
a < b < c < d, giving φ̄(a) ≤ φ(b) < φ(c) ≤ φ̄(d). So for any π, π ′ ∈ Π, J1 (π) > J1 (π ′ ) implies J2 (π) = φ̄(J1 (π)) > φ̄(J1 (π ′ )) = J2 (π ′ ), and J1 and J2 induce the same ordering on Π, contradicting non-equivalence. Theorem 1 (Value inversions). Let J1 and J2 be non-trivial, non-equivalent value functions. If the policy set contains an open subset, then it admits a value inversion for J1 and J2 . Proof of Theorem 1. Let Π̃ ⊆ Π be open, and let T1 , T2 be non-trivial and non-equivalent on Π. Since J1 and J2 are rational (Proposition 4), their gradients ∇J1 and ∇J2 exist. If these gradients are linearly independent at some π0 ∈ Π̃, we can solve ⟨∇J1 (π0 ), v⟩ = 1 and ⟨∇J2 (π0 ), v⟩ = −1 (a consistent linear system in |S|(|A| − 1) ≥ 2 unknowns), and Lemma 1 gives exploitation. If instead they are everywhere linearly dependent on Π̃, we may write ∇J2 = λ∇J1 for a scalar-valued function λ wherever both gradients are nonzero (such points exist since each ∇Ji is analytic and not identically zero by non-triviality, so it is nonzero on a subset open and dense in Π̃). If λ(π0 ) < 0 at some such point, setting v = ∇J1 (π0 ) gives directional derivatives of opposite sign, and Lemma 1 again gives exploitation. The remaining possibility is that λ > 0 wherever both gradients are nonzero on Π̃, but then Lemma 2 forces T1 and T2 to be equivalent on Π, contradicting our assumption. Theorem 2 (Safe horizon). Let R : S × A → [0, 1] and let δ = 12 maxs,a ∥T (·|s, a) − T ′ (·|s, a)∥1 be the total variation distance between distinct T and T ′ . For any ε > 0, define the safe horizon p (1 + ε) + (1 − ε)2 + 4ε/δ . H(ε, δ) = 2 Every pair (T , T ′ ) is ε-unexploitable on every policy set Π whenever 1/(1 − γ) ≤ H(ε, δ). Furthermore, this bound is tight. Proof of Theorem 2. Write h = 1/(1 − γ). The tight simulation lemma (Lobel and Parr, 2024) with h εR = 0 gives |J1 (π) − J2 (π)| ≤ B for every π, where B = h − 1−(1−h)δ . Suppose (T1 , T2 ) is ε-exploitable via π, π ′ . Add the two defining inequalities: (J1 (π) − J2 (π)) + (J2 (π ′ ) − J1 (π ′ )) > 2ε, {z } | {z } | ≤B
≤B
so B > ε. Contrapositively, B ≤ ε implies ε-unexploitability.
It is elementary to show that for δ, ε > 0 there is a unique h = H(ε, δ) > 1 at which B = ε, and h < H(ε, δ) implies B < ε. Solving B = ε reduces to a quadratic whose positive root is the H(ε, δ) in the theorem statement. For tightness, take S = {0, 1}, A = {0, 1}, d0 (0) = 1, R(s, a) = 1{s = 0} and let state 1 be absorbing. Under T1 , action 0 self-loops at state 0 and action 1 transitions to state 1 with probability δ. Under T2 the actions swap, so action 1 self-loops at state 0 and action 0 transitions to state 1 with probability δ. Then J1 (π0 ) = J2 (π1 ) = h and J1 (π1 ) = J2 (π0 ) = h/(1 + (h−1)δ), so both gaps equal B exactly, and the pair is ε-exploitable whenever h > H. Corollary 4 (Square-root heuristic). Let R : S × A → [0, 1] and let δ = 12 maxs,a ∥T (·|s, a) − p T ′ (·|s, a)∥1 > 0. For any ε > 0, the pair (T , T ′ ) is ε-unexploitable whenever 1/(1 − γ) < ε/δ. p p Proof of Corollary 4. It suffices to show ε/δ ≤ H(ε, δ), since then 1/(1 − γ) < ε/δ implies 1/(1 − γ) < H(ε, δ) and Theorem 2 gives ε-unexploitability. Since (1 − ε)2 ≥ 0, r r 4ε ε 2 ≥ 2 , (1 − ε) + δ δ p p so H(ε, δ) ≥ 1+ε ε/δ > ε/δ. 2 + 15
C
Relevant reward hacking theorems
For convenience, we restate the two main characterization results from Skalse et al. (2022) in our notation. Note that Corollary 3 and Theorem S1 are contrapositives of the same fact: Theorem S1 says that any unhackable, non-trivial pair must be equivalent, while Corollary 3 says that any non-trivial, non-equivalent pair must be hackable. We use the latter phrasing to parallel our exploitability results. Theorem S1 (Theorem 1 of Skalse et al. (2022)). In any environment (S, A, T , d0 , _, γ), if Π̂ contains an open set, then any pair of reward functions that are unhackable and non-trivial on Π̂ are equivalent on Π̂. As shown in Proposition 2, the original statement of the following result requires an additional hypothesis not present in the original: that the visit counts of Π̂ are not collinear. The correction is minor and the strengthened condition is easily satisfied in practice; the core insights of Skalse et al. (2022) are unaffected. Theorem S2 (Theorem 2 of Skalse et al. (2022), corrected). For any environment (S, A, T , d0 , _, γ), any finite set of policies Π̂ whose visit counts are not collinear, and any reward function R1 , there is a non-trivial reward function R2 such that R1 and R2 are unhackable but not equivalent. Proof. The proof is identical to that of Skalse et al. (2022). The only step requiring the strengthened ⃗ 1 to −R ⃗ 1 along which the first reward function hypothesis is ensuring that there is a path from R where some inequality becomes an equality is not trivial. With collinear visit counts, every pairwise value difference is a scalar multiple of a single inner product ⟨R, F π1 − F π0 ⟩, so zeroing any one difference zeros them all. Non-collinearity guarantees three policies πa , πb , πc with F πb − F πa and F πc − F πa linearly independent. The set of rewards trivial on Π̂ must be orthogonal to both, ⃗ 1 to giving a subspace of codimension at least 2, which cannot disconnect R|S||A| . A path from R ⃗ −R1 therefore exists that avoids trivial rewards entirely, and the remainder of the argument proceeds unchanged.
D
Transition models for Figures 1 to 3
The MDP used in Figures 1 to 3 has S = {s0 , s1 , s2 }, A = {a0 , a1 }, R(s, a) = 1{s = s0 }, γ = 0.9, and d0 = (1/3, 1/3, 1/3). Policies are parameterized by πθ (a0 | s) = θ for all s ∈ S. Each transition model is specified by two stochastic matrices Pa0 and Pa1 , where entry (i, j) gives T (sj | si , a). The shared transition model T1 is Pa(1) = 0
0.7 0.5 0.4
0.2 0.3 0.3
! 0.1 0.2 , 0.3
Pa(1) = 1
0.1 0.1 0.1
0.3 0.2 0.1
! 0.6 0.7 . 0.8
Under T1 , action a0 concentrates transitions toward s0 (the rewarding state), while action a1 pushes toward s2 , making J1 monotone increasing in θ. Panel (a): T2 trivial.
Both actions produce identical transitions, so J2 is constant in θ: ! 0.4 0.3 0.3 (2a) (2a) Pa0 = Pa1 = 0.2 0.5 0.3 . 0.3 0.3 0.4
Panel (b): T1 , T2 equivalent. We construct T2 by blending T1 toward the uniform kernel with mixing weight α = 0.3, i.e., T2 (s′ | s, a) = 0.7 · T1 (s′ | s, a) + 0.3/|S|: ! ! 0.59 0.24 0.17 0.17 0.31 0.52 Pa(2b) = 0.45 0.31 0.24 , Pa(2b) = 0.17 0.24 0.59 . 0 1 0.38 0.31 0.31 0.17 0.17 0.66 This preserves the qualitative structure of T1 (and thus the policy ordering) while attenuating the transition probabilities. 16
(c)
(d)
Panels (c) and (d): T1 , T2 exploitable. The transition models T2 and T2 were found by computational search over random transition matrices drawn from a Dirichlet distribution. Full numerical values and the search code are available in the supplementary material.
E
Bound comparison Panel
Relationship
δ
Actual gap
Theorem 2 bound (tight)
Corollary 4 bound (loose)
(a) (b) (c) (d)
Trivial Equivalent Exploitable Exploitable
0.40 0.14 0.81 0.64
0 0 0.66 4.35
7.83 5.58 8.79 8.52
40 14 81 64
Table 2: Exploitation gaps for the transition model pairs in Figure 1 at γ = 0.9. The actual gap is the largest ε for which the pair is ε-exploitable. The Theorem 2 and Corollary 4 bounds give the unique ε above which each result guarantees ε-unexploitability. Both bounds are conservative. Table 2 compares both bounds against the actual exploitation gaps for the transition model pairs in Figure 1. Both thresholds are conservative in every case and overestimate the actual gap. This conservatism is unavoidable in general: we prove Theorem 2 is tight.
17