Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis

Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis Idil Gözel

arXiv:2608.07228v1 [cs.LG] 7 Aug 2026

Department of Physics and Astronomy University College London, London, UK ucapgoz@ucl.ac.uk

Abstract When a reinforcement learning agent cannot observe the full state, we usually blame its policies: it cannot see enough to represent a good one. We show that in a solvable case the bigger problem lies elsewhere. Even when a good policy is available and the agent’s value function is expressive enough to describe it exactly, learning still ends up somewhere far worse. We study a partially observed linear-quadratic problem in which a standard actor-critic learner can be solved in closed form. At our default setting the best policy the agent can represent is already close to optimal, costing 10.4% more than the ideal controller that observes everything. Learning does not find it. The algorithm instead comes to rest at a policy that is 35% worse than the best one available to it, and we can say exactly where and why. The cause is a bias in what the critic learns rather than a limit on what the actor can express. Because the agent cannot attribute what it sees to the part of the state it cannot observe, the critic misreads that unexplained variation as sharp curvature in its own value estimates, and the actor follows that error away from the optimum. We derive closed-form expressions for the resulting policy, for its cost, and for the one design choice that removes the problem, which is how far the learner looks ahead before trusting its own value estimates. Deep reinforcement learning experiments follow these predictions closely. Notably, giving the agent memory of past observations does not help, while changing how far it looks ahead does.

1

Introduction

An agent that cannot see the whole state can fail for two different reasons. It may be unable to represent a good policy, because the actions worth taking depend on what it cannot observe. Or it may be able to represent a good policy and still fail to find it, because the quantities it estimates while learning are biased by the same blindness. The first failure is a property of the policy class. The second is a property of the learning rule. We will call the first the policy-class gap: how much worse the best policy the agent can represent is than the best policy that sees everything. We will call the second the learning gap: how much worse the policy the learner actually settles on is than the best one it could have represented. These two names are used in exactly this sense throughout the paper. The usual reading of partial observability is that the policy-class gap is the problem. Give the agent a richer policy class, or a way to remember the past, and the difficulty should recede. That reading is incomplete, and the reason is easy to state. Learning also has to estimate something, 1

usually a value function, and those estimates are formed from the same impoverished observations. If the estimates are biased, the learner will optimize the wrong objective no matter how good its policy class is. Existing theory has largely bounded this second effect rather than located it, so it has been hard to tell how much of the observed shortfall it accounts for. This paper answers that question exactly in one solvable case. An agent observes and actuates one coordinate of a linear system. A second coordinate, which it never observes, drives the first. Costs are quadratic, so if the agent could see both coordinates the problem would be classical. The learner is a standard actor-critic pair, with a critic at its temporal-difference fixed point and an actor following the policy gradient. Two features of this setup make the two gaps separable, and both are deliberate. First, the policy class is fixed and its optimum can be computed, so the policy-class gap is known exactly rather than estimated. Second, the critic’s features are the correct ones: the true value functions here are quadratic, and the critic uses quadratic features. Nothing is lost to function approximation. What remains is the effect of the features being computed from an incomplete observation, and nothing else. The answer is that the learning gap dominates. At our default setting the best memoryless policy is already close to optimal, costing only 10.4% more than the ideal controller. Nevertheless, actor-critic learning comes to rest somewhere else entirely: at a gain fifteen times larger, whose cost is 35% above the best memoryless policy. Both numbers are exact rather than estimated. Why learning fails even though a good policy exists, and is representable, is the question this paper answers. Why the critic goes wrong. The mechanism is worth stating before any mathematics, because it is simple. Consider what the agent’s observed coordinate looks like in the long run. It fluctuates, and those fluctuations have two sources: the control the agent applies, and the unobserved coordinate pushing it around. In the stationary regime, meaning once the loop has settled into its long-run statistical behaviour, these two exactly balance. On average the unobserved coordinate supplies precisely the motion that the control removes. Now put yourself in the critic’s position. It sees persistent fluctuation in the observation, and it must express the value of a state as a function of that observation alone. It cannot say “this variation comes from a variable I cannot see,” because it has no term for that variable. With quadratic features, the only place left to put unexplained persistence is the curvature of the value function. So the critic reports value that curves far too sharply. At the optimum of the policy class it reports curvature 16.7 against a true value of 1.11. An actor using this critic as its baseline is told that moving to higher gain pays, and it keeps moving until the inflated curvature is finally balanced by control cost, long past the optimum. The key point is that nothing here is a failure of representation. The critic’s feature class can express the true value function exactly. The bias comes from what the features are allowed to see. Contributions. Each item below is stated as one idea. Scope and technical qualifications are given where the result appears. • An exact mechanism (§3). We compute the critic’s fixed point and the actor’s expected update in closed form, and show the resulting equilibrium exists only because time is discrete. • Where learning stops (§4). One elementary formula gives the gain at which learning comes to rest, as a function of a single ratio of exploration to disturbance strength. 2

• What it costs (§5). At that gain the deployed cost has a simple closed form, and it holds for essentially any disturbance process, with no distributional assumption. • Which of these is universal (§6). The cost is distribution-free but the location is not. Five processes with identical autocovariance move the resting point by a factor of 3.4 while leaving its cost nearly unchanged. • The design choice that fixes it (§7). How far the learner looks ahead before trusting its own value estimates, and nothing else, moves the resting point back to the optimum. • Evidence from deep reinforcement learning (§8, §9). Runs follow the closed-form predictions, architecture does not matter, and a pre-registered test rules out the obvious alternative explanation. Two things this paper does not claim. It does not prove that the stochastic iterates converge to the fixed point; every closed-form statement concerns the expected update, and the correspondence with the iterates is verified numerically. It does not claim that adding memory to an architecture is useless in general, only that in this record it changed nothing that the lookahead parameter did not already change. Section 10 states both carefully, together with a reproducible behaviour that the theory does not explain.

2

Setting: a partially observed linear–quadratic problem

2.1

Environment

We study the simplest control problem in which an agent must act on a coordinate it observes while a coordinate it does not observe carries the dynamics’ memory. Simplicity here is the point rather than a limitation. It is what lets both gaps be computed exactly instead of estimated, and two of the three laws we derive turn out to hold far beyond this instance. The environment has a two-dimensional state (v, z): the agent observes and actuates only v, while an unobserved state z drives it, dv = (z + u) dt, dz = (−γz − c v) dt + σ dW, σ 2 = 2Θγc, (1) with u the control input and W a standard Wiener process. Marginalizing the unobserved state is what makes this a partially observed problem of the classical kind. The observed coordinate then obeys a generalized Langevin equation in which z acts through a memory kernel K(τ ) = c e−γτ , together with a stationary noise whose strength σ 2 = 2Θγc makes the stationary variance of z P equal to Θc. In the general N -component version the kernel is K(τ ) = i ci e−γi τ with decay rates {γi }. This projection of an unobserved degree of freedom onto a memory kernel and a matched P stationary noise is classical (Zwanzig, 1961; Mori, 1965). We write K(0) = i ci for the kernel’s zero-lag intensity. Every observation-based policy therefore faces state aliasing (Singh et al., 1994): distinct values of z are indistinguishable from the current observation alone. The agent interacts in discrete time with period h: it observes vn = v(nh) and applies a control held constant over the interval (zero-order hold). The cost rate is qv 2 + ru2 , discounted at rate ρ. Throughout, the default instance is c = 0.3, γ = 1, Θ = 1, q = r = 1, h = 0.05, ρ = 0.05; every claim is stated for general parameters and verified numerically at this instance. Notation. We collect here the symbols used throughout. β := e−ρh is the per-step discount factor. A prime denotes the next sample: v ′ ≡ vn+1 . For a policy with gain k we write κ := kh, 3

q̃ := q + rk 2 , and ν := hse , where se is the exploration variance defined below. The second moments of the stationary closed loop are m2 := E[v 2 ] and x := E[vv ′ ].

2.2

Learning algorithm

The behaviour policy during learning is linear with Gaussian exploration, un = −k vn + ηn ,

ηn ∼ N (0, se ) i.i.d.,

(2)

and deployment sets se = 0. The learner is a standard actor-critic pair (Konda and Tsitsiklis, 2003). The critic is the TD(0)/LSTD fixed point (Sutton, 1988; Bradtke and Barto, 1996) over the observable features (1, v, v 2 ). This is the correct approximation class here: the true value functions of the problem are quadratic, so the choice of features introduces no error. The only limitation is what the features can see. The actor follows the score-function policy gradient with the critic as baseline (Williams, 1992). In the stationary closed loop all variables are jointly Gaussian. By the Isserlis moment factorization (Isserlis, 1918), both the critic’s fixed point and the actor’s expected update therefore reduce to polynomials in m2 and x.

2.3

Two reference values

Two quantities calibrate everything that follows, and both are exact for every instance of (1). The optimal LQG cost J ∗ is attained by Kalman filtering and certainty-equivalent control (Kalman, 1960; Åström, 1965): it is the best any causal controller can do. The best memoryless cost Jml , attained at gain k ∗ , is the best achievable by any deployment policy of the form u = −kv. Policies of this form are called memoryless, or in control terms static output feedback (Fatkhullin and Polyak, 2021). This cost is the optimum of the policy class the learner searches. At the default instance, J ∗ = 0.1989, Jml = 0.2196 at k ∗ = 1.649, (3) a policy-class gap of 10.4%. We verified separately that this gap is not amplified by finite horizons, initial-condition effects, or risk-sensitive objectives. Whatever the learning dynamics do beyond it is therefore not a representation effect. The distinction between Jml and what learning actually attains is the subject of this paper.

2.4

An exact sampling reduction

The zero-order hold makes one step of the sampled dynamics available in closed form, with no approximation. Lemma 1 (Exact one-step reduction). Under the policy (2), for any h > 0 and any realization of the unobserved state, vn+1 = (1 − κ) vn + h z̄n + h ηn ,

1 z̄n := h

Z (n+1)h nh

zt dt.

(4)

Proof. Integrate dv = (z + u)dt over [nh, (n + 1)h) with u held at un = −kvn + ηn . The observed coordinate carries no direct noise, so the integral is exact. Equation (4) says the sampled observation is a linear recursion driven by the interval average of the unobserved state and the exploration noise. Its second-moment consequences require no distributional assumptions at all, a fact we exploit in §5. Gaussianity enters only through the 4

recursion’s interaction with the critic (§6). The stability condition of the recursion, κ ∈ (0, 2), is the sampled-control stability condition. The boundary κ → 2 is worth keeping in mind: it reappears as the edge of every law we derive.

3

The aliased critic and its consequence for the actor

3.1

Stationarity pins the observed mean reversion

The first structural fact is a constraint that stationarity alone places on what any observer of v can see. Write ∆ := v ′ − v for the sampled increment. Lemma 2 (Observed mean reversion). In the stationary closed loop, for any gain k and any environment parameters, E[v∆] = − 21 E[∆2 ]. Evaluated for the model (1), exactly at every h, m2 − x =

νh  κ −1 . 1− 2 2

(5)

The first statement is immediate from E[v ′2 ] = E[v 2 ] and requires no distributional assumptions; the evaluation (5) is proved in Appendix A. Its content is easy to state. In stationarity the unobserved state resupplies, on average, exactly the motion that the control removes. The autocovariance decay of the observed coordinate is therefore set by the exploration noise alone, and it carries almost no trace of either the control effort or the unobserved dynamics. A critic restricted to functions of v must attribute the resulting persistence somewhere, and a quadratic feature class offers only one place to put it: the curvature of the value function.

3.2

The critic’s fixed point

Before computing anything, it is worth predicting what should happen. The critic must assign value using functions of v alone. Lemma 2 says the observed coordinate looks almost like a random walk, because the control and the unobserved state cancel on average. A near-random walk visits large values of v and stays there, which to any observer looks like a state whose value differs sharply from the mean. With features (1, v, v 2 ) the only coefficient that can express such a difference is the one on v 2 . We should therefore expect the fitted curvature to come out far too large, and the following proposition says by how much. Proposition 1 (Critic fixed point, exact at every h). The TD(0)/LSTD fixed point over (1, v, v 2 ) has first-order coefficient zero and curvature coefficient θ2 =

q̃ m2 h q̃ m22 −−−→ , 2 2 h→0 ν + ρ m2 m2 − βx

versus

θ2cond ≈

q̃ , 2k + ρ

(6)

where θ2cond is the curvature of the true conditional value function E[V | v]. At the default instance, at k = k ∗ , the critic reports θ2 = 16.7 against a true conditional curvature of 1.11: a fifteen-fold inflation (Figure 1b), verified against an independent solve to 1.5 × 10−13 . The denominator of (6) is where Lemma 2 acts. The quantity m22 − βx2 is small precisely because stationarity forces x close to m2 . It is kept away from zero only by the exploration term ν and the discount ρ. The key point is that this inflation is not an approximation error. The feature class can represent the true value function exactly. Persistence that the critic cannot attribute to the unobserved state has nowhere else to go but curvature in v. 5

̂ V(v)

u

δ actor u = −kv

15 10 5 0 −1

0

d k*

0

−50 aliased full-state

−100 100

1

observation v

6

keq

101

policy gain k

policy gain k

v

c aliased true

actor update ĝ × 103

z

critic

value curvature

b

a

5 4 3 2 0.0

0.5

1.0

steps (millions)

Figure 1: An aliased critic inflates value curvature, and the actor follows the resulting update field away from the optimum. (a) An unobserved state z drives the observed coordinate v. The critic sees only v, so its temporal-difference signal δ is computed from an aliased state. (b) At the class optimum the critic reports curvature θ2 = 16.7 (red) against a true value of 1.11 (black). (c) The expected update ĝ (red) stays negative until keq = 24.79. A critic given the full state follows the true gradient (black, rescaled), which vanishes at k ∗ . (d) Online actor-critic started at k ∗ , six seeds (red), against the averaged dynamics (black dashed). All six leave the optimum, reaching k = 4.82 ± 0.03 at 1.2M steps. The runs lag the averaged trajectory by ∼20% (§10). Dotted line: k ∗ .

3.3

The actor’s expected update

The actor performs stochastic gradient steps k ← k − α δn ∂k log πk (un | vn ), where δn is the temporal-difference error under the critic of Proposition 1 and ∂k log πk (un | vn ) = −ηn vn /se for the policy (2). Let ĝ(k) denote the expectation of the update direction δn (−ηn vn /se ) in the stationary loop. Proposition 2 (Expected update). Exactly at every h, ĝ(k) = 2hrk m2 − 2β θ2 x g,

(7)

where g is the one-step control response g = 0h ∂vt /∂u dt = h + O(h3 ). Substituting Proposition 1 and passing to the limit h → 0 at fixed k (in which x → m2 and g → h), the stationarity condition ĝ = 0 reduces to rk = β θ2 (k), which has no solution for any k > 0: in continuous time the expected update never changes sign. R 



Two consequences follow. First, the coupled critic-actor dynamics do have a fixed point at any positive sampling period. It exists only through time discreteness, and it is far from the optimum of the policy class: at the default instance ĝ vanishes at keq = 24.79, against k ∗ = 1.649 (Figure 1c). A critic given the full state instead follows the true gradient dJ/dk and is stationary at k ∗ : the displacement is an effect of aliasing, not of discounting or of the gradient estimator. Second, (7) is the expectation of the actual stochastic update, scale included, not merely proportional to it. With the critic held at its fixed point, the sample average of δn (−ηn vn /se ) over 1.5 × 105 steps agrees with ĝ to within 3% at k ∈ {1.65, 3, 6}, the residual being Monte Carlo error at that sample size (Appendix E).

3.4

The stochastic iterates leave the optimum

Proposition 2 concerns the averaged dynamics. To confirm that the stochastic algorithm follows them, we initialize online TD(0) actor-critic at the class optimum k ∗ and run six independent seeds (αc = 5 × 10−3 , αa = 10−4 , critic warm-started at its fixed point). All six leave the optimum 6

monotonically, reaching k = 4.82 ± 0.03 after 1.2M steps (Figure 1d). The trajectory of the averaged dynamics, obtained by integrating (7) at the same step size, reaches 6.02. The iterates track its shape but lag it by roughly 20%, because the online critic trails its instantaneous fixed point as k moves. We return to this gap, the difference between the averaged dynamics and the stochastic iterates, in §10. Nothing in this paper claims the iterates converge to keq . The claims are that the expected update vanishes there and that the iterates leave k ∗ along its field.

4

Fixed points of the expected update: a closed-form law

We now know why the learner moves away from the optimum. The next question is where it stops. Proposition 2 places the fixed point of the expected update at gains that grow as the sampling period shrinks. The natural regime in which to study it is therefore the joint limit h → 0 with κ = kh held fixed. This is high-gain feedback, with the gain measured in units of the sampling rate. In this limit the second moments obey m2 = h2 M (κ) (1 + o(1)) and x = h2 X(κ) (1 + o(1)) with ΘK(0) se ΘK(0) M (κ) = + , X(κ) = (1 − κ) M (κ) + , (8) 2 κ κ(2 − κ) κ and the fixed-point condition ĝ = 0 reduces to M 2 − X 2 = κXM (Appendix A). Everything about the environment now enters through a single ratio, σ :=

se , ΘK(0)

(9)

the exploration power measured against the zero-lag intensity of the memory kernel. Theorem 1 (Closed form for the fixed point). For every σ > 0 the condition M 2 − X 2 = κXM has exactly one root χ ∈ (0, 2). The root is given in closed form by the inverse of σ(χ) =

(2 − χ)

4 + χ2 − χ , 2χ

p

σχ2 (σ + 2 − χ) = (2 − χ)2 .

equivalently

(10)

The root is simple: both factors of σ(χ) are positive and strictly decreasing on (0, 2), so σ(·) is a strictly decreasing bijection of (0, 2) onto (0, ∞). d The monotonicity argument is the entire uniqueness proof: for the second factor, dχ 4 + χ2 − p
2 2 χ = χ/ 4 + χ2 − 1 < 0. The algebraic reduction from M − X = κXM to (10) is elementary and given in Appendix A; we verified the two expressions agree with a direct numerical root-find to 2 × 10−14 across six decades of σ. Three consequences are immediate. First, the fixed-point gain is pinned to the sampling rate, keq = χ(σ)/h: halving the sampling period doubles the gain at which the expected update vanishes. Second, the limits of (10) are explicit,

p

√
χ = 2 − 2 2 + 2 σ + O(σ 2 )

(σ → 0),

χ=

2 + O(σ −2 ) σ

(σ → ∞),

(11)

so as exploration vanishes the fixed point approaches the sampled-control stability boundary κ = 2, while for strong exploration keq ≈ 2ΘK(0)/(se h): the fixed-point gain is inversely proportional to the exploration power. The latter is a quantitative version of the practitioner observation that larger exploration noise mitigates the displacement. Third, the kernel enters (10) only through K(0): two kernels of equal zero-lag intensity but different shape give the same χ, even though 7

b

c 0.35

80 60 40

Jml J(keq )

rΘK(0) J

*

0.30 0.25

0

0.15 0.5

1

2

hidden-mode speed γ/Ω

1.1 1.0 0.9

1.0

10−1

100

0.5

0.20

20

J(keq )/rΘK(0)

2.0 1.5

keq ⋅ h

100

learning damage policy-class gap

cost rate

excess cost (% of J * )

a

0.0 single mode γ=1

split kernel γ = (0.5, 2)

10−1

100

101

se /ΘK(0)

Figure 2: The two gaps, and the law that governs the learning gap. (a) Excess cost over J ∗ as the unobserved state is made faster. The policy-class gap (black) is single-digit and nearly flat. The learning gap (red) is several times larger and grows as memory becomes less important for the policy class. (b) Kernel intensity, not shape, sets the cost at the fixed point. Two kernels with the same K(0) = 0.3 give different optimal and best-memoryless costs but the same cost at the fixed point, at the ceiling rΘK(0) = 0.300 (dashed). (c) The fixed-point law in the joint limit. Markers: twelve instances spanning h, γ, exploration, kernel shape and component count (circles, squares, triangles for N = 1, 2, 3), against the closed form (10), a curve with no fitted parameters. Inset: the same instances’ cost against the ceiling.

they give different optimal LQG and best-memoryless costs. Figure 2c shows twelve instances spanning h, γ, exploration, kernel shape and one to three unobserved components falling onto the curve (10); the curve is the closed form, not a fit. Figure 2a shows the resulting dissociation: as the unobserved state is made faster, the policy-class gap stays nearly flat while the learning gap grows. The two gaps are governed by different properties of the same environment. Rate of convergence. How fast is the limit attained? Consider the variant of (1) in which the feedback of v on z is removed, so that the unobserved state becomes an exogenous stationary disturbance and Lemma 1 closes the loop exactly. For this variant the convergence is first order with an explicit coefficient. Theorem 2 (Finite sampling period). For the exogenous-disturbance variant, with the disturbance P the stationary Gaussian process with autocovariance ΘK(τ ), K(τ ) = i ci e−γi τ , keq h = χ(σ) + c1 h + O(h2 ),

(12)

where c1 depends on the kernel only through the weighted mean of the decay rates, γ̄ = and does so affinely. All constants are explicit (Appendix C).

P

i ci γi /K(0),

Numerically, at σ = 0.3, ρ = 0.05: c1 = −0.891, −1.323 and −1.539 for γ = 1, γ = 1.5 and an equal mixture of γ ∈ {0.5, 3} respectively; the affine dependence predicts the mixture coefficient from the two pure kernels to four digits. For the full model (1) the feedback of v on z contributes a further correction of the same order, quantified in Appendix C. At the default instance it moves keq by −0.4%, an order of magnitude less than the kernel-averaging term. At h = 0.05 the exact fixed point of the full model sits at keq = 24.79 against the limit value χ(0.3)/h = 25.80. Corollary 1 (Multiple unobserved components). For any finite kernel K(τ ) = i ci e−γi τ with P Gaussian stationary components, Theorems 1 and 2 hold with K(0) = i ci ; no other property of the kernel enters the limit. P

The Gaussian qualifier in Corollary 1 is not a technicality. Section 6 shows that the fixed-point location is not invariant to the disturbance’s distribution at fixed autocovariance; only the cost of §5 is. The qualifier marks a real boundary of the law, not a limitation of the proof technique. 8

5

The deployed cost under high-gain feedback

Knowing where the learner stops is only half of what we want. The other half is what stopping there costs. This section prices the fixed point. The result is the most general in the paper. It requires neither Gaussianity nor any parametric structure of the unobserved dynamics, and it says what a controller at high gain is actually doing with its control effort. Consider deployment (se = 0) of a static output-feedback policy at gain k = κ/h on the observed coordinate dv = ζt + u, (13) dt where the disturbance ζ is any stationary, ergodic, mean-square-continuous process with autocovariance C(τ ) = E[ζ0 ζτ ] and C(0) < ∞. The only discrete-time element is the zero-order hold on u. The model of §2 corresponds to the exogenous-disturbance variant with C(τ ) = ΘK(τ ); here nothing else about ζ is assumed. Theorem 3 (Deployed cost). For every fixed κ ∈ (0, 2), in the stationary regime of (13), J(κ/h; h) −−−→ r C(0), h→0

q E[v 2 ] → 0,

E (un + ζ̄n )2 → 0, 



(14)

where ζ̄n is the interval average of the disturbance. The third limit is the interpretive content. At high gain the sampled controller performs disturbance rejection: u converges in L2 to the negative of the disturbance. The deployed cost is exactly the power of the signal being cancelled, at price r. The state is pinned essentially for free; the entire cost is the cancelling control effort. The proof (Appendix B) is short enough to sketch. At deployment Lemma 1 gives the exact recursion vn+1 = (1−κ)vn +hζ̄n , whose stationary solution is a linear filter of {ζ̄n } with geometrically decaying weights. Its second moment is a lattice sum of interval-average autocovariances C̄(m). Cauchy-Schwarz gives |C̄(m)| ≤ C(0), and mean-square continuity gives C̄(m) → C(0) for each fixed lag as h → 0. Dominated convergence then yields m2 = h2 C(0)/κ2 (1 + o(1)). Then E[u2 ] = (κ/h)2 m2 → C(0) while q E[v 2 ] = O(h2 ). No distributional property of ζ beyond its autocovariance ever enters: the second-moment structure of a linear recursion is blind to everything else. Corollary 2 (Cost at the learning fixed point). Wherever the learning dynamics place their fixed point at a gain of order 1/h, that is keq h → χ ∈ (0, 2) as established in §4, the deployed cost at the fixed point satisfies J(keq ) = r C(0) (1 + O(h)), regardless of the distribution of the disturbance and regardless of where in (0, 2) the value χ falls. The corollary decouples two questions that the next section shows are genuinely distinct: where the fixed point sits (which does depend on the distribution) and what it costs (which does not). For exponentially correlated disturbances, C(τ ) = C(0)e−γτ , the finite-h cost has a closed form (Appendix B), with leading correction
J 2γh (1 − κ) γh = 1 − − + O (γh)2 , r C(0) κ(2 − κ) 3

(15)

large and negative at small κ, changing sign near κ = 1. Table 1 tests the closed form against simulations of five disturbance processes sharing the autocovariance C(τ ) = 0.3 e−τ but differing 9

Table 1: Deployed cost at the learning fixed point for five disturbance processes with identical autocovariance C(τ ) = 0.3e−τ (§6), against the closed form of Appendix B evaluated at each process’s own κ = keq h. No fitted parameters. The residual for the two heavy-tailed processes is consistent with slower Monte Carlo averaging at equal sample size (Appendix E). disturbance two-state Markov Ornstein-Uhlenbeck difference of squares squared OU (centred) squared OU, symmetrised

κ = keq h

J (closed form)

J (simulated)

error

0.438 1.231 1.456 1.494 1.494

0.2785 0.2997 0.3041 0.3051 0.3051

0.2798 0.2985 0.3015 0.2975 0.2981

−0.5% +0.4% +0.9% +2.5% +2.3%

in distribution (constructed in §6), each evaluated at its own learning fixed point. The formula has no adjustable parameters, and agreement is 0.5 to 2.5%. The largest deviation from the limiting value rC(0) = 0.300 is the two-state Markov (telegraph) disturbance, measured 6.7% below it. The closed form at its κ = 0.438 predicts 7.2% below, so the measurement agrees with the theory to 0.5%. The truncation (15) is not accurate at this κ, where γh/κ = 0.11 is not small; it gives 9.9%. Finite-h corrections account for the deviation, but the closed form is needed, not its first-order term. The L2 disturbance-rejection statement is also directly measurable. The residual E[(u + ζ̄)2 ] equals E[∆2 ]/h2 by Lemma 1. It comes out at 6 to 11% of C(0) at h = 0.05 across the same five processes, an O(γh/κ) quantity as the proof predicts.

6

What the kernel controls and what the distribution controls

We now have two laws, one for where the learner stops and one for what that costs. They look equally general. They are not, and the difference matters. Theorem 1 says the fixed point depends on the environment only through K(0). That statement was proved for Gaussian unobserved dynamics, and any test run inside the model stays within that family. There is a structural reason why more evidence of the same kind cannot dismiss the qualifier. The critic’s normal equations over (1, v, v 2 ) involve fourth moments of the stationary loop, Var(v 2 ) and Cov(v 2 , v ′2 ). The closed form of Proposition 1 rests on their Isserlis reduction to m2 and x. This is the only place Gaussianity enters the analysis. At high gain the loop keeps v close to a filtered copy of the disturbance, so what feeds the curvature coefficient is the second-order structure of the squared disturbance ζ 2 . For any Gaussian process, however, Cov(ζ02 , ζτ2 ) = 2C(τ )2 : the statistics of ζ 2 are completely determined by the autocovariance. Within a Gaussian family, C(0) and Var(ζ 2 ) cannot be varied independently. No amount of in-family evidence, including the twelve instances of Figure 2c, could therefore distinguish “the fixed point depends on C(0)” from “the fixed point depends on Var(ζ 2 )”. Separating the two requires leaving the Gaussian family while holding the autocovariance fixed.

6.1

Five disturbance processes, one autocovariance

We construct five stationary processes, all with C(τ ) = 0.3 e−τ exactly, spanning Var(ζ 2 ) from zero to 1.26 and including a matched pair that isolates skewness: √ 1. two-state Markov (telegraph): ζ ∈ {± 0.3} with switching rate 1/2. Here ζ 2 is constant: Var(ζ 2 ) = 0. 10

a

b chidiff

keq ⋅ h

1.25

OU

1.025

skewed symmetric

1.00 0.75 0.50

J(keq ) / rC(0)

1.50

0.25

0.975 0.950 0.925

telegraph

0.00

1.000

0.50

0.75

1.00

1.25

0.00

Var(ζ 2 ) at fixed C(τ)

0.25

0.50

0.75

1.00

1.25

Var(ζ 2 ) at fixed C(τ)

Figure 3: The cost is distribution-free; the location is not. Five disturbance processes with identical autocovariance C(τ ) = 0.3 e−τ (§6). (a) The fixed-point gain moves by a factor of 3.4, ordered by Var(ζ 2 ) and saturating above the Gaussian value. Odd moments are irrelevant: the skewed process (red square) and its symmetrised counterpart (open circle) coincide. Grey line: guide to the eye. (b) Cost at each process’s own fixed point against the limit rC(0) (line). Four sit within 0.5%. The telegraph point sits 6.7% below, against 7.2% predicted by the finite-h closed form of Appendix B at its κ = 0.438.

2. Ornstein-Uhlenbeck: the Gaussian reference; Var(ζ 2 ) = 2C(0)2 = 0.18. √ 3. difference of squares: ζ = 0.075 (w12 − w22 ) with w1 , w2 independent unit-variance OU processes of decay rate 1/2. Symmetric, excess kurtosis 6, Var(ζ 2 ) = 0.72. √ 4. squared OU (centred): ζ = 0.15 (w2 − 1), w a unit-variance OU process of decay rate 1/2. √ Skewness 8, Var(ζ 2 ) = 1.26. 5. squared OU, symmetrised: process 4 multiplied by an independent two-state sign process of switching rate 0.01, which removes the skew while changing C(τ ) by at most 2% over lags up to the disturbance correlation time. Same Var(ζ 2 ) = 1.26 as process 4. Processes 4 and 5 are the matched pair: identical autocovariance and identical Var(ζ 2 ), differing only in odd moments. The qualitative outcomes of runs 3 and 5 were recorded in advance of running them (Appendix E). For each process we locate the learning fixed point empirically, as the zero of the sample expected update with the critic fit by least squares on the simulated trajectory. The pipeline reproduces the exact Gaussian value to 0.7% (Appendix E).

6.2

The fixed point moves; the cost does not

Figure 3 shows the result. The fixed-point gain spans keq = 8.75 (telegraph) to 29.88 (both squared-OU processes), a factor of 3.4 at identical autocovariance. It is ordered by Var(ζ 2 ), steeply on the sub-Gaussian side and saturating above the Gaussian value. The matched pair settles the role of odd moments: processes 4 and 5 give the same keq to within the bisection resolution of ±0.4. The mechanism reads directly off the telegraph case. With ζ 2 constant, the fluctuations of v 2 carry far less of the disturbance’s persistence, and the fitted curvature coefficient θ2 at fixed k is roughly one third of its Gaussian value. What the critic misattributes is not the persistence of the observation alone but the persistence of its square. Meanwhile the deployed cost at each process’s own fixed point stays within 0.5 to 6.7% of rC(0) (Table 1), with every deviation accounted for by (15): exactly the split that Corollary 2 predicts. On the Gaussian side, the law extends beyond the finite-component kernels of Corollary 1. A Gaussian disturbance with the power-law autocovariance C(τ ) = 0.3 (1 + τ )−3/2 , which lies outside

11

every finite-component family, gives keq = 24.4 at h = 0.05. This value is inside the finite-h band that Theorem 2 predicts around the limit value (Appendix C).

6.3

Scope of the laws

The results of §4 and §5 therefore split cleanly by what each quantity depends on: • Deployed cost at the fixed point: r C(0), for any stationary ergodic mean-square-continuous disturbance (Theorem 3). Proven, distribution-free. • Fixed-point location: keq h = χ(se /C(0)) for Gaussian disturbances (Theorem 1, Corollary 1); not distribution-invariant in general. The five-process experiment is a counterexample, not an outlier. Conjecture 1. At fixed autocovariance, the fixed-point gain is determined jointly by C(0) and the second-order structure of the squared disturbance ζ 2 , monotonically and with saturation in Var(ζ 2 ), and independently of odd moments. The five processes above are consistent with Conjecture 1 but do not pin it down: across them the correlation time of ζ 2 varies together with Var(ζ 2 ), and a design that separates the two is left open. The practical reading of this section is the paper’s central asymmetry in one sentence: to predict what partial observability will cost a bootstrapped learner, the disturbance’s zero-lag power suffices; to predict where the learner’s update field will vanish, its distribution matters.

7

The bootstrap horizon

So far the picture is entirely negative. This section gives the one design choice that undoes it. Everything so far bootstraps over a single step: the critic of Proposition 1 is the TD(0) fixed point, and the update (7) feeds back its one-step temporal-difference error. Practical implementations interpolate between this extreme and Monte Carlo evaluation using λ-returns (Sutton, 1988), in policy-gradient form generalized advantage estimation (Schulman et al., 2016). The parameter λ sets the bootstrap horizon: the λ-return integrates observed costs over h/(1 − βλ) of physical time, in expectation, before substituting the critic’s estimate for the remainder. This section computes the fixed points of the resulting expected update exactly. The horizon turns out to be the variable that removes the pathology, and the removal is not gradual: it happens across a narrow interval of λ. The critic is now the LSTD(λ) fixed point over the same features (1, v, v 2 ) (Bradtke and Barto, 1996), A(λ) θ = b(λ) with A(λ) =

X

(βλ)j E ϕt−j (ϕt − βϕt+1 )⊤ , 



b(λ) =

j≥0

X

(βλ)j E[ϕt−j ct ],

(16)

j≥0

and the actor’s expected update is ĝλ (k) = E (−ηt vt /se ) 

j j≥0 (βλ) δt+j .

P



Proposition 3 (Expected update at general λ). In the stationary closed loop, every lag moment in (16) and in ĝλ is a Gaussian moment, and the geometric sums evaluate to resolvents in the closed-loop transition matrix Fk . As a result, ĝλ (k) is an explicit algebraic function of (I − βλFk )−1 and (I − βλ Fk ⊗Fk )−1 , exact at every h and every λ ∈ [0, 1]. At λ = 0 it reduces to Proposition 2. The expressions are collected in Appendix D. 12

λ keq J(keq ) excess over Jml

0

0.9

0.95

0.98

0.984

0.986

0.99

0.999

24.79 0.2964 35.0%

20.55 0.2921 33.0%

17.67 0.2888 31.5%

11.40 0.2781 26.6%

8.61 0.2699 22.9%

3.43 0.2365 7.7%

2.20 0.2229 1.5%

1.55 0.2198 0.1%

Table 2: The equilibrium of the expected GAE(λ) update at the default instance. The excess cost drops from a third to one percent inside λ ∈ [0.98, 0.99]; the excess halves at λ1/2 = 0.9854.

At the default instance the root structure is as follows (dense scans in k at each λ; all statements verified numerically). The update has a single root at every λ: there is no fold, and no second fixed point appears or disappears. Instead the one equilibrium migrates continuously to the policy optimum as λ → 1, starting from the aliased location of §4 (the λ = 0 slice of this section, keq h = χ + O(h)). The migration is concentrated in a narrow interval: Three features of Table 2 matter later. First, the excess is essentially flat up to λ = 0.95, a value widely considered conservative in practice, and disappears within the next four hundredths. Second, the λ → 1 endpoint is 1.55, near but not at k ∗ = 1.649: the residual is discounting bias, the mismatch between the discounted learning objective and the average deployed cost. It scales linearly with ρ (reducing ρ five-fold, from 0.05 to 0.01, shrinks the offset from 6.1% to 1.2% of k ∗ ). Third, the location of the transition is set, to first approximation, by the horizon matching the kernel. Across γ ∈ {0.5, 1, 2} and h ∈ {0.025, 0.05, 0.1}, a sixteen-fold variation in γh, the half-excess point λ1/2 sits where the horizon h/(1 − βλ) equals 1.9 to 3.4 correlation times 1/γ of the unobserved state. The matching rule is a heuristic, not an invariant; the exact statement is the root of Proposition 3. The heuristic still has predictive force, because it moves the transition with γ: at λ = 0.97 the model puts the equilibrium excess at 29.6% for γ = 1 but 3.9% for γ = 2. Those are opposite verdicts at the same λ. No single value of λ is safe or unsafe by itself; safety is a property of the horizon relative to the environment’s memory. Figure 4(a) plots J(keq (λ)) as a curve and carries this section’s law into deep reinforcement learning, where §8 tests it.

8

Does this happen in deep reinforcement learning?

The closed-form results concern a linear policy, a quadratic critic, and expected updates. This section asks whether they predict the behaviour of a standard deep implementation. We run proximal policy optimization (Schulman et al., 2017) on the default instance, using a public reference implementation (Huang et al., 2022) at a fixed version, with multilayer-perceptron actor and critic. Section 9 describes the harness, the pre-registration protocol, and the frozen decision rules; here we report the outcomes. Two architectures are compared. Memoryless agents observe vn alone. Frame-stack agents observe the last 32 samples of v, a window spanning 1.6 correlation times of the unobserved state, so the input contains the information needed to estimate z. Every run trains for 107 steps, at five seeds per configuration. We measure two quantities at deployment, with exploration noise removed. The first is the deployed cost J, estimated on one long evaluation episode. The second is the implied gain, defined as the regression coefficient of −u on v under the deployed policy. Figure 4(a) overlays the runs on the law of §7. Three facts stand out. First, the runs follow the memoryless equilibrium curve. At λ = 0.99 the memoryless median is J = 0.2338, against an equilibrium value of 0.2229. At λ = 0.9 the memoryless median is 0.3175, on the aliased side and 13

0.99 0.999

deployed cost J

0.35

memoryless stack 32

0.30

0.25

b

c λ = 0.9

0.40 0.35 0.30 0.25

λ = 0.99

0.20 10

−1

10

0

10

1

λ-return horizon h/(1 − βλ)

0

2

4

6

steps (millions)

8

10

fraction, gain > 5 for ≥ 1M

0.9

deployed cost J

λ=0

a

1.0 0.8 0.6 0.4 0.2 0.0 10−3

10−2

fixed σ

10−1 2

Figure 4: The bootstrap horizon sets the outcome, and the memoryless closed form predicts it. (a) Cost at the equilibrium of the exact memoryless expected update (curve, §7) against the λ-return horizon (top axis: λ). Points: PPO at 107 steps, median of five seeds (whiskers: range), memoryless (open circles) and 32-frame stack (red squares). The two architectures coincide, and no run improves on Jml (dotted). Dashed: the ceiling rΘK(0). Deviations are finite-time effects (§8). (b) The controlled contrast at identical fixed exploration variance σ 2 = 0.09, differing only in λ (five seeds each, 3-sample running median). Every λ = 0.9 seed (red) leaves the low-cost solution; four of five λ = 0.99 seeds (black) hold near Jml . (c) Fraction of seeds sustaining implied gain > 5 for ≥ 106 steps, across fixed σ 2 spanning two decades. Logistic fit with 95% band: slope −0.17 ± 0.58. No threshold exists.

far above Jml ; the seed trajectories overshoot the equilibrium value of 0.2921, a finite-time effect we return to in §10. The implied gains agree as well. The model puts the λ = 0.9 equilibrium at keq = 20.55. The seeds that sustain the high-gain regime for millions of steps end at implied gains with median 20.4. Second, architecture does not enter. The frame-stack medians are 0.3165 at λ = 0.9 and 0.2265 at λ = 0.99, indistinguishable from the memoryless agents at the same λ. Third, no run of either architecture at any λ improves on Jml = 0.2196. The best median in the entire record is 0.2251, at memoryless λ = 1. The two gaps therefore separate empirically. The learning gap opens and closes with the bootstrap horizon, following the closed form. The policy-class gap stays open in every run. The natural reading of frame stacking is that memory should remove the aliasing and with it the pathology. The record says otherwise. Stacked agents leave the optimum at λ = 0.9 exactly as memoryless agents do. They reach the class optimum at λ = 0.99 exactly as memoryless agents do. Input memory is not used memory: the stacked critic behaves like the memoryless critic of the exact model at every λ tested. Memory in the architecture is therefore neither necessary for reaching the class optimum nor sufficient to avoid the aliased fixed point. We note for completeness that an earlier in-house implementation did show stacked agents reaching the class optimum at λ = 0.9 over a shorter training horizon (Appendix E). That effect did not survive the reference implementation at 107 steps. The benefit of frame stacking is implementation-dependent in our record; the λ dependence is not. Two alternative explanations are excluded by construction. The first attributes the failure at λ = 0.9 to the collapse of the learned exploration noise late in training. Panel (b) of Figure 4 shows the controlled contrast: exploration variance held fixed at σ 2 = 0.09 in both arms, with only λ differing. All five λ = 0.9 seeds leave the low-cost solution; four of five λ = 0.99 seeds hold near Jml . Holding the entropy up does not protect the solution. The second explanation posits a threshold in the exploration magnitude itself. Panel (c) shows the fraction of seeds that sustain the high-gain regime across fixed σ 2 spanning two decades. The fraction is flat. A logistic fit of the seed outcomes on log10 σ 2 gives slope −0.17 ± 0.58: no threshold exists. Both arms of this test 14

were pre-registered with frozen decision rules before the runs were launched (§9).

9

Pre-registered experiments

Numerical agreement in a model this small is easy to obtain after the fact. We therefore adopted a pre-registration protocol for the deep experiments. Predictions were written down and committed to version control before each batch was submitted to the cluster. Decision rules were frozen the same way: the analysis constants, the verdict each outcome produces, and the validity controls that must pass before any verdict is read. The harness is a single-file implementation at a fixed version. Each configuration is one text line, and each batch is an array over configurations and seeds. Appendix E lists every frozen constant, and states what the published record does and does not establish about the ordering. The protocol did its job on the experiment that mattered. An earlier draft of this work conjectured, in writing, that the failure of frame-stack agents at λ = 0.9 was caused by the collapse of the learned exploration noise late in training. The committed prediction was that agents with σ 2 held fixed at 0.09 would keep the low-cost solution. The decision rules were frozen before launch as follows. A seed counts as holding the solution if its final deployed cost lies in [0.20, 0.27] with implied gain at most 5. It counts as sustaining the high-gain regime if its implied gain exceeds 5 for at least 106 consecutive steps. It counts as diverged if its final cost exceeds 0.45. A row’s verdict requires agreement of three of five seeds. Divergence routes to a diagnostic arm at one quarter of the learning rate. If that arm holds the solution, the conjecture survives with a step-size caveat; if not, the conjecture is falsified. Two validity controls gate everything. The learned-σ arm must reproduce the failure, with final cost at least 0.29 and gain at least 7. The λ = 0.99 arm must hold the solution. If either control fails, the batch is invalid and no verdict is read. Both controls passed. The learned-σ arm failed as required, with median cost 0.3165 and median gain 13.9. The λ = 0.99 arm held in four of five seeds. The registered prediction then failed completely. Zero of five fixed-σ seeds at λ = 0.9 held the solution; final costs lie between 0.344 and 0.360, above the ceiling rΘK(0) = 0.300. The quarter-learning-rate arm also held in zero of five seeds. Both branches of the frozen rule therefore return the same verdict. The conjecture is falsified. Holding the exploration variance up does not protect the solution, as §8 reports. A second batch swept the fixed variance across two decades, σ 2 ∈ {0.001, 0.003, 0.01, 0.03}, again with predictions committed before launch. Under the threshold hypothesis, the number of seeds sustaining the high-gain regime should decrease in σ 2 . The committed brackets were at least four of five at 0.001, at least three at 0.003, and at most one at 0.03. The observed counts are 2, 3, 2, and 2, with 2 of five at σ 2 = 0.09. The pattern is flat and fails the committed brackets in both directions. The logistic fit of Figure 4(c) makes the conclusion quantitative. No threshold in σ 2 exists. One gap in the frozen rules should be recorded. Three of the five fixed-σ seeds at λ = 0.9 ended in none of the three registered categories. Their gain rose and then fell toward zero. Their cost climbed above the ceiling but stayed below the divergence threshold. The registered prediction fails regardless, because it required the seeds to hold the solution and none did. But the three-way rule was not exhaustive, and a future registration should name this fourth outcome in advance. We return to this behaviour in §10.

15

10

Limitations

Every closed-form statement in this paper concerns the expected update, not the stochastic iterates. The drift experiment of §3 shows the iterates leave the optimum along the expected field, lagging its integrated trajectory by roughly 20%. We do not prove that the iterates converge to keq , and at λ = 0.9 the deep runs settle above the equilibrium cost rather than at it. A two-timescale analysis of the coupled critic and actor iterates would close this gap. We have not done it. One reproducible behaviour is unexplained. It appears in three of five fixed-σ seeds at λ = 0.9 and in four of five seeds of the diagnostic arm. The implied gain rises toward the equilibrium, destabilizes before arriving, and falls to roughly zero. The deployed cost meanwhile rises above the ceiling. The deployed policies in this state are poorly summarized by a linear gain. The averaged theory says nothing about this route, and the frozen decision rules of §9 did not anticipate it. The laws give the locations of equilibria, not the time to reach them. The λ = 0.95 runs illustrate the difference: the equilibrium sits at k = 17.7, but the drift toward it is slow near k ∗ and 107 steps do not resolve it. Any use of Table 2 for finite training budgets needs this caveat. The model is deliberately minimal. The observed coordinate is scalar, the costs are quadratic, the control enters through a zero-order hold, and the policy class is linear. Within the model, the location law is proved for Gaussian disturbances only, and §6 shows that the qualifier is real. Conjecture 1 records what we believe happens beyond it, and the correlation-time structure of ζ 2 remains confounded with Var(ζ 2 ) in our process family. The deep evidence is one environment family, one algorithm, and one reference implementation. The absence of a frame-stacking benefit is a statement about this record, not a theorem: an earlier in-house implementation behaved differently at a shorter horizon (Appendix E). Recurrent architectures are untested. The γ prediction of §7, opposite verdicts at λ = 0.97 for γ = 1 against γ = 2, is stated for future registration rather than claimed.

11

Related work

State aliasing under function approximation is as old as the field’s study of partial observability (Singh et al., 1994). Perkins and Pendrith (2002) proved that fixed points of approximate policy iteration exist under continuous exploration, and gave an ε-greedy counterexample. Notably, their non-convergent agent out-performed their convergent one. That was an early sign that convergence and solution quality separate under aliasing. This paper is in that tradition, with the existence question replaced by exact locations. Modern finite-time analyses bound the effect this paper computes. Cayci et al. (2024) give finite-time guarantees for actor-critic under partial observability in which an aliasing error enters as an upper bound, and Lambrechts et al. (2025) bound the aliasing term for asymmetric actor-critic methods. Bounds of this kind certify that damage is limited; they do not locate the fixed points or price them. The closed forms here do both, for one solvable instance. The full-state critic used as a control in §3 is the idealized asymmetric critic: a critic trained on privileged state while the actor acts on observations (Baisero and Amato, 2022; Cai et al., 2024). In the model, that critic removes the displacement entirely. Recurrent critics that learn state from history are analysed by Cayci and Eryilmaz (2024), and convergence of learning with agent state by Kara and Yüksel (2023). Our frame-stack results suggest that having history and using it are different things, which sharpens the case for such architectures rather than replacing them. On the control side, the policy class u = −kv is static output feedback, whose optimization landscape under true gradients is studied by Fatkhullin and Polyak (2021). The present paper 16

adds the learned critic and shows where its bias moves the stationary points. The optimal LQG benchmark is classical (Kalman, 1960; Åström, 1965), and the reduction of an unobserved coordinate to a memory kernel is the Mori-Zwanzig projection (Zwanzig, 1961; Mori, 1965). Finally, the convergence theory of TD with linear features (Tsitsiklis and Van Roy, 1997; Bradtke and Barto, 1996; Boyan, 2002) assumes the feature process is driven by a Markov state. Under aliasing that premise fails in a specific, computable way. Lemma 2 pins the observed statistics that the critic must fit, and this is what makes the bias exactly calculable rather than merely bounded.

12

Conclusion

Partial observability harms a bootstrapped actor-critic learner through two separate routes. The first is the policy-class gap, which in this model is modest and nearly flat. The second is the learning gap, the displacement of the expected update’s equilibrium away from the class optimum, and it is several times larger. The paper gives the learning gap three closed-form laws. The location law: keq h = χ(σ), with χ an elementary function of the exploration-to-disturbance ratio. The cost law: deployed cost r C(0) at high gain, distribution-free. The horizon law: the equilibrium migrates to the class optimum as the λ-return horizon passes the memory of the environment, with the transition concentrated in a narrow interval of λ. The mechanism is short to state. Stationarity forces the persistence of the unobserved state into the observed statistics that a memoryless critic must fit. A quadratic feature class can put that persistence only into value curvature. The actor then follows an update field whose zero sits at high gain. Everything else in the paper is this sentence made exact. The deep experiments follow the memoryless closed form, architecture does not enter, and no run closes the policy-class gap. A pre-registered test falsified the natural alternative explanation: holding exploration noise up does not protect the solution, and no threshold in exploration magnitude exists. The practical reading is direct. Choose the bootstrap horizon against the memory of the environment, not by convention; do not expect entropy bonuses or input memory alone to prevent the displacement. Open problems remain on both sides: a two-timescale analysis of the stochastic iterates, the non-Gaussian location law of Conjecture 1, recurrent critics, and the registered γ prediction of §7.

Acknowledgements The author acknowledges the use of the UCL Myriad High Performance Computing Facility (Myriad@UCL), and associated support services, in the completion of this work.

References Karl Johan Åström. Optimal control of Markov processes with incomplete state information I. Journal of Mathematical Analysis and Applications, 10:174–205, 1965. Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 44–52, 2022. arXiv:2105.11674.

17

Justin A. Boyan. Technical update: Least-squares temporal difference learning. Machine Learning, 49(2–3):233–246, 2002. Steven J. Bradtke and Andrew G. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22:33–57, 1996. doi: 10.1007/BF00114723. Yang Cai, Xiangyu Liu, Argyris Oikonomou, and Kaiqing Zhang. Provable partially observable reinforcement learning with privileged information. In Advances in Neural Information Processing Systems, volume 37, pages 63790–63857, 2024. doi: 10.52202/079017-2038. Semih Cayci and Atilla Eryilmaz. Convergence of gradient descent for recurrent neural networks: A nonasymptotic analysis. arXiv preprint arXiv:2402.12241, 2024. Semih Cayci, Niao He, and R. Srikant. Finite-time analysis of natural actor-critic for POMDPs. SIAM Journal on Mathematics of Data Science, 6(4):869–896, 2024. doi: 10.1137/23M1587683. Ilyas Fatkhullin and Boris Polyak. Optimizing static linear feedback: Gradient method. SIAM Journal on Control and Optimization, 59(5):3887–3911, 2021. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G. M. Araújo. CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. Leon Isserlis. On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables. Biometrika, 12(1–2):134–139, 1918. Rudolf E. Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME–Journal of Basic Engineering, 82:35–45, 1960. doi: 10.1115/1.3662552. Ali Devran Kara and Serdar Yüksel. Convergence of finite memory Q-learning for POMDPs and near optimality of learned policies under filter stability. Mathematics of Operations Research, 48 (4):2066–2093, 2023. Vijay R. Konda and John N. Tsitsiklis. On actor-critic algorithms. SIAM Journal on Control and Optimization, 42(4):1143–1166, 2003. doi: 10.1137/S0363012901385691. Gaspard Lambrechts, Damien Ernst, and Aditya Mahajan. A theoretical justification for asymmetric actor-critic algorithms. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, 2025. arXiv:2501.19116. Hazime Mori. Transport, collective motion, and Brownian motion. Progress of Theoretical Physics, 33(3):423–455, 1965. doi: 10.1143/PTP.33.423. Theodore J. Perkins and Mark D. Pendrith. On the existence of fixed points for Q-learning and Sarsa in partially observable domains. In Proceedings of the Nineteenth International Conference on Machine Learning (ICML), pages 490–497, 2002. John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR), 2016. arXiv:1506.02438. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 18

Satinder P. Singh, Tommi Jaakkola, and Michael I. Jordan. Learning without state-estimation in partially observable markovian decision processes. In Proceedings of the Eleventh International Conference on Machine Learning (ICML), pages 284–292, 1994. Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9–44, 1988. doi: 10.1007/BF00115009. John N. Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42:674–690, 1997. David V. Widder. The Laplace Transform. Princeton University Press, Princeton, NJ, 1946. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. doi: 10.1007/BF00992696. Robert Zwanzig. Memory effects in irreversible thermodynamics. Physical Review, 124(4):983–992, 1961. doi: 10.1103/PhysRev.124.983.

A

Proofs for Sections 3 and 4

Throughout this appendix all processes are stationary, mean zero and, where stated, jointly Gaussian. We use the Isserlis identities E[abc] = 0 and E[abcd] = E[ab]E[cd]+E[ac]E[bd]+E[ad]E[bc] without further comment. The stationary covariance Σ of the closed loop solves Σ = Fk ΣFk⊤ + se GG⊤ + Σd , where Fk is the closed-loop transition matrix, G the control input vector and Σd the sampled process noise. Recall m2 = Σvv , x = E[vv ′ ], q̃ = q + rk 2 and g = G1 . Every closed form below was checked against the Isserlis moment machinery of theorem2_learning.py. Propositions 1 and 2 agree to 10−13 or better, and the closed form (10) agrees with a direct numerical root of M 2 − X 2 = κXM to 2 × 10−16 over six decades of σ.

A.1

Proof of Lemma 2

The first statement uses stationarity only. Write ∆ = v ′ − v. Then E[v ′2 ] = E[v 2 ] + 2E[v∆] + E[∆2 ], and stationarity gives E[v ′2 ] = E[v 2 ], so E[v∆] = − 12 E[∆2 ]. No distributional assumption enters. For the evaluation, take the (1, 1) entry of the stationary covariance equation. The exploration noise enters v ′ through gη with g 2 se = h2 se + O(h3 ), and the sampled process noise contributes (Σd )11 = O(h3 ), because the observed coordinate carries no direct diffusion. The balance is 2(m2 −x) = g 2 se +(Σd )11 , which gives (5) after substituting ν = hse and collecting the κ dependence of the sampled recursion. In continuous time the same identity reads 2(Σvz − kΣvv ) + se = 0 per unit time. The unobserved state supplies, on average, exactly the motion the control removes, whatever the values of k, γ and c.

A.2

Proof of Proposition 1

That θ1 = 0 follows by parity. Every entry of the v row of the LSTD system is a third Gaussian moment or E[v c]. The only odd term in the cost is −2rkh vη, and E[v · vη] = m2 E[η] = 0 by independence of η. All such entries vanish. For θ2 , centre the ϕ = v 2 equation by subtracting m2 times the ϕ = 1 equation: E[(v 2 − m2 )c] + βθ2 E[v 2 v ′2 ] − m22 − θ2 E[v 4 ] − m22 = 0.

19

Isserlis gives E[v 2 v ′2 ] = m22 + 2x2 and E[v 4 ] = 3m22 . In the cost term write c = h(qv 2 + rk 2 v 2 − 2rk vη + rη 2 ). The vη term contributes E[(v 2 − m2 )vη] = E[v 3 ]E[η] = 0. The η 2 term contributes E[v 2 − m2 ] E[rη 2 ]h = 0. The v 2 terms give hq̃(E[v 4 ] − m22 ) = 2hq̃m22 . Collecting, 2hq̃m22 + 2βθ2 x2 − 2θ2 m22 = 0, which is (6). The denominator is positive: x < m2 strictly by Cauchy-Schwarz, since equality would require v ′ ≡ v, impossible for se > 0.

A.3

Proof of Proposition 2

With ∂k log πk (u | v) = −vη/se we have ĝ = −s−1 e E[vη δ]. Take the three contributions in turn. The cost term keeps only its odd part, E[vη · (−2rkh vη)] = −2rkh m2 se , contributing +2hrk m2 . The bootstrap term gives E[vη · βθ2 v ′2 ] = 2βθ2 E[vv ′ ] E[ηv ′ ] by Isserlis, since the pairing E[vη]E[v ′2 ] vanishes; here E[ηv ′ ] = g se , because η enters v ′ only through Gη and is independent of the state and of the process noise. This contributes −2βθ2 xg. The current-value term E[vη · θ2 v 2 ] is odd and vanishes, and constants pair with E[vη] = 0. Together these give (7). For the continuum statement, substitute (6) and let h → 0 at fixed k, so that x → m2 and g → h. Then ĝ → 2hm2 (rk − βθ2 ), and by (6) the factor rk − βθ2 is bounded away from zero for every k > 0: the expected update does not change sign.

A.4

The scaling limit: derivation of (8)

Set k = κ/h and let h → 0 with κ fixed. To leading order !

Fk ≈

1−κ h , −ch 1 − γh

gη ≈ hη,

Σd ≈ diag O(h3 ), σ 2 h .

The (v, z) stationarity equation gives Σvz κ = hΘc + O(h2 ), using Σzz = Θc + O(h); the loop
2 = perturbs the unobserved state only at O(h). The (v, v) equation reads m 1 − (1 − κ) 2  h2 (Σzz + se ) + 2(1 − κ)hΣvz , that is m2 κ(2 − κ) = h2 se + Θc (2 − κ)/κ . This is M (κ) as stated, and x = (Fk )11 m2 + (Fk )12 Σvz gives X(κ). For N components the computation repeats per component. The cross-covariances Σzi zj are O(h2 ), because the components couple only through v, which is itself O(h) in this regime. Summing gives ΘK(0) in place of Θc, which is Corollary 1. Substituting into the condition ĝ = 0 with g → h, β → 1 and q̃ → rκ2 /h2 , the q term being smaller by O(h2 ), gives M 2 − X 2 = κXM .

A.5

Proof of Theorem 1

Measure everything in units of ΘK(0), so that M = 1/κ2 + σ/(κ(2 − κ)) and X = (1 − κ)M + 1/κ with σ = se /ΘK(0). The derivation is algebraic throughout. From the two expressions, M − X = κM − 1/κ and M + X = (2 − κ)M + 1/κ. Hence (M − X)(M + X) − κXM = κM 2 −

1 (2 − κ)M − 2, κ κ

so the condition M 2 − X 2 = κXM is equivalent to κ3 M 2 − κ(2 − κ)M − 1 = 0.

(17)

Write M = A/ κ2 (2−κ) with A := (2−κ)+σκ, which is exactly the definition of M . Substituting into (17) and clearing denominators gives a quadratic in A,

A2 − A(2 − κ)2 − κ(2 − κ)2 = 0, 20

whose discriminant is (2 − κ)4 + 4κ(2 − κ)2 = (2 − κ)2 (2 − κ)2 + 4κ = (2 − κ)2 (4 + κ2 ). Positivity √   of A selects the upper branch, A = (2 − κ) (2 − κ) + 4 + κ2 /2. Isolating σκ = A − (2 − κ) gives √
σκ = (2 − κ) 4 + κ2 − κ /2, which is (10). For the bijection, both factors of σ(χ) are positive on (0, 2) and strictly decreasing. The first is p p
d 2 − χ. For the second, dχ 4 + χ2 − χ = χ/ 4 + χ2 − 1 < 0. The factor 1/(2χ) is also strictly decreasing and positive. A product of positive strictly decreasing functions is strictly decreasing, so σ(·) is injective. Its limits are σ → ∞ as χ → 0+ and σ → 0 as χ → 2− , so it maps (0, 2) onto (0, ∞). Existence, uniqueness and simplicity of the root follow. 

A.6



Asymptotics (11)

p √ √ Put χ√= 2 − ε. Then 4 + χ2 − χ → 2 2 − 2 and 2χ →p4, so σ = ε( 2 − 1)/2 + O(ε2 ) and ε = (2 2 + 2)σ + O(σ 2 ). For χ → 0+ both 2 − χ → 2 and 4 + χ2 − χ → 2, so σ = 2/χ + O(1), that is χ = 2/σ + O(σ −2 ). At σ = 10−4 the first expansion gives 1.99951716 against 1.99951736 exactly, and at σ = 104 the second gives 2.000 × 10−4 to four digits.

B

Proof of the cost theorem

We prove Theorem 3. Let ζ be stationary, ergodic and mean-square continuous with C(0) < ∞, let se = 0, and let k = κ/h with κ ∈ (0, 2) fixed. Step 1: exact reduction. The zero-order hold freezes u on each interval, and the observed coordinate carries no direct diffusion. Integrating dv/dt = ζ + un over [nh, (n + 1)h) is therefore exact: vn+1 = a vn + h ζ̄n , with a := 1 − κ and ζ̄n the interval average of ζ. The sequence {ζ̄n } is stationary and ergodic, being a measurable factor of ζ. Since |a| < 1 exactly when κ ∈ (0, 2), P the unique stationary solution is vn = h j≥0 aj ζ̄n−1−j , convergent in L2 . Every second-order property of v is now determined by C alone. This is where distribution-freeness enters, structurally rather than by approximation. Step 2: the second moment as a lattice sum. Write R̄(m) := E[ζ̄0 ζ̄m ] = h−2 0h 0h C(mh + P s − s′ ) ds ds′ . Then m2 = h2 j,l≥0 aj+l R̄(j − l). Cauchy-Schwarz gives |R̄(m)| ≤ C(0). Meansquare continuity makes C continuous at the origin, so for each fixed m we have R̄(m) → C(0) as h → 0, uniformly over |m| ≤ M for any fixed M . R R

Step 3: dominated convergence. Collecting by diagonal, j,l aj+l R̄(j − l) = m wm R̄(m) P P with wm = a|m| /(1 − a2 ) and m |wm | = ( j |a|j )2 < ∞ independently of h. Split the sum at P P |m| ≤ M . The head converges to C(0) |m|≤M wm , and the tail is bounded by C(0) |m|>M |wm |, P

which tends to zero as M → ∞ uniformly in h. Hence so m2 = h2 C(0)κ−2 (1 + o(1)).

m wm R̄(m) → C(0)

P

P

P

ja

j 2 = C(0)/κ2 ,

Step 4: the threeR limits. First, E[u2 ] = (κ/h)2 m2 → C(0). Second, within an interval R t −1 2 vt = (1 − κt/h)vn + 0 ζ, so h E[vt ] dt = m2 (1 − κ + κ2 /3) + O(h2 C(0)), which is O(h2 ) and P 2 gives q E[v ] → 0. Therefore J → r C(0). Third, un = −κ j aj ζ̄n−1−j is a weighted average of the disturbance with unit total mass over a window of length O(h/κ). The same lattice-sum argument applied to the cross term gives E[(un + ζ̄n )2 ] → C(0) − 2C(0) + C(0) = 0. The controller converges in L2 to the instantaneous canceller of the disturbance. 21

B.1

Exact finite-h formula for exponential autocovariance

For C(τ ) = C(0)e−γτ the interval-average covariances have closed forms, R̄(0) = C(0)

2(γh − 1 + e−γh ) , (γh)2

R̄(m) = C(0) φ b m (m ≥ 1),

with φ = 2 cosh(γh) − 2 /(γh)2 and b = e−γh . Summing the lattice series in closed form,

"

m2 = h

2

2φ C(0) ab R̄(0) + , 1 − a2 (1 − a2 )(1 − ab) #

"

#

 κ 2 κ2  C(0)h2 m2 . (18) J = q m2 1 − κ + + +r 3 3 h 

Expanding (18) for γh ≪ 1 gives (15). The expansion is reliable only when γh/κ is small, which is why Table 1 is computed from (18) rather than from the truncation. At the two-state Markov process, κ = 0.438 and γh/κ = 0.11: the closed form gives J = 0.2785, which is 7.2% below rC(0) and within 0.5% of the measured 0.2798, while the first-order truncation gives 9.9%. As a check of the limit itself, (18) at κ = 1.2 gives J = 0.29910, 0.29976 and 0.29995 at h = 0.05, 0.01 and 0.002. The tracking residual is measurable without any further theory. By Lemma 1, E[(u + ζ̄)2 ] = E[∆2 ]/h2 exactly. Measured at each process’s own fixed point at h = 0.05, it is 0.034, 0.019 and 0.021 for the two-state Markov, Ornstein-Uhlenbeck and squared-OU processes. Those are 6 to 11% of C(0), consistent with the O(γh/κ) size the proof predicts.

B.2

What the theorem does and does not assume

Mean-square continuity is used once, in Step 2, and it excludes white noise disturbances. Every kernel with finite zero-lag intensity satisfies it. The absence of direct diffusion on v is used in Step 1, and it matters. With a noise floor σv2 on the observed coordinate, the same computation adds terms of order rσv2 /h. That is a separate effect, out of scope here. Setting se = 0 is the definition of deployment, not a restriction. Finally, the theorem prices a gain of order 1/h without explaining why the gain is there. Corollary 2 imports that fact from §4; it does not reprove it.

C

Finite sampling period remainders

This appendix supports Theorem 2. The family is the exogenous-disturbance variant of (1), for which Lemma 1 closes the loop exactly: vn+1 = (1 − κ)vn + hζ̄n + hηn . The disturbance is stationary Gaussian with completely monotone autocovariance. By the Bernstein representation of R −γτ completely monotone functions (Widder, 1946, Theorem 12a), C(τ ) = e µ(dγ) for a positive R measure µ, with C(0) = µ(0, ∞) < ∞. We assume in addition L := |C ′ (0+ )| = γ µ(dγ) < ∞. Gaussianity of the loop makes Propositions 1 and 2 exact at every h, since v is a linear functional of the Gaussian history. No Markov structure is needed. Substituting m2 = h2 M̂ , x = h2 X̂ and k = κ/h into ĝ = 0 gives the exact-h equilibrium condition
qh2 Φh (κ) := κ M̂ 2 − β X̂ 2 − βκ2 M̂ X̂ − β M̂ X̂ = 0, β = e−ρh . (19) r Lemma R. The exploration parts of M̂ and X̂ are exact at every h, because the exploration noise is independent across steps: the term se /(κ(2 − κ)) carries no sampling error. The kernel parts are lattice sums, linear in C, with the closed forms of Appendix B for a single component. 22

Differentiating those closed forms in γh at the origin and bounding the result by min(Dγh, B(κ)) gives, uniformly on compact subsets of (0, 2), |M̂ − M | ≤ h L DM (κ),

|X̂ − X| ≤ h L DX (κ),

with DM and DX elementary. The truncation by B(κ) is what makes the bound integrable against every Bernstein measure of finite L. The constants grow without bound only as κ → 0 or κ → 2, which is why the statement is uniform on compact subsets. Root expansion. By Lemma R, Φh → Φ uniformly on compact subsets of (0, 2), with an error of order h (L + ρC(0) + qhC(0)/r). The limit Φ has a simple root χ(σ), by the strict monotonicity established in Theorem 1. A simple root is stable under uniform perturbation, so for h below an explicit threshold the exact-h root satisfies keq h = χ(σ) + c1 h + O(h2 ) with |c1 | ≤ C(σ) (L/C(0) + ρ). Verification, without Monte Carlo. Every number here comes from the closed forms above. At σ = 0.3 and ρ = 0.05, the ratio (keq h − χ)/h converges as h decreases from 10−2 to 10−3 : kernel

γ̄

h = 10−2

h = 10−3

c1

single, γ = 1 single, γ = 1.5 mixture, γ ∈ {0.5, 3}

1.00 1.50 1.75

−0.8927 −1.3292 −1.5418

−0.8911 −1.3233 −1.5388

−0.891 −1.323 −1.539

The proof structure predicts that c1 is a linear functional of the kernel, since the lattice sums are linear in C and C is linear in µ. The table confirms it. Fitting c1 = −0.027 − 0.864 γ̄ on the two single-component kernels predicts −1.5394 for the mixture, against −1.5388 computed directly. Back-coupling. The full model (1) couples the unobserved state to v through the term −c v, which the reduced family omits. At h = 0.05, γ = 1 and σ = 0.3, all three quantities are available exactly: the limit χ/h = 25.80, the reduced family at keq = 24.90, and the full model at keq = 24.79. Kernel averaging accounts for −3.5% and back-coupling for a further −0.44%. The back-coupling term is of the same order in h but roughly eight times smaller here. Theorem 2 therefore holds for the full model with a modified c1 , whose value we verify numerically but do not derive. A derivation requires the coupled Lyapunov structure and is left open; the assumptions ledger records this status. A final decomposition itemizes the error budget of the empirical pipeline used in §6. The reduced family gives 24.90 exactly, the full model 24.79, and the simulation pipeline 24.62. The last step, −1.1%, is Monte Carlo error together with the substep discretization of the disturbance.

D

Exact LSTD(λ) and GAE(λ) expressions

This appendix supports Proposition 3. Write bλ := βλ and let ϕt = (1, vt , vt2 ). Both the critic system (16) and the expected update require geometric sums of lag moments, and in the stationary Gaussian loop each such sum is a resolvent.

23

The lag covariances are xj = e⊤0 Fkj Σe0 , and the cross moments between the exploration noise and later observations are yj = se e⊤0 Fkj−1 G for j ≥ 1, with y0 = 0. The three families of sums that appear are X j

X j

j≥0

j≥0

bλ xj = e⊤0 (I − bλ Fk )−1 Σe0 ,

bλ x2j = (e0 ⊗e0 )⊤ (I − bλ Fk ⊗Fk )−1 (Σe0 ⊗Σe0 ),

together with the analogous Kronecker resolvent for the products xj yj . Squared lag moments require the Kronecker resolvent because x2j = (e0 ⊗ e0 )⊤ (Fk ⊗ Fk )j (Σe0 ⊗ Σe0 ). No truncation is involved at any point. With these, the centred v 2 row of the LSTD(λ) system gives θ2 (λ) = P

h q̃

P

j 2 j≥0 bλ xj

, P j 2 j 2 j≥0 bλ xj − β j≥0 bλ xj+1

(20)

which reduces to (6) at λ = 0. The expected update combines the one-step term of (7) with the tail
i 1 X jh ĝλ = ĝ0 − bλ 2hq̃ xj yj + θ2 (λ) 2βxj+1 yj+1 − 2xj yj , se j≥1 each sum again evaluated by the resolvents above. Two checks were run. At λ = 0 the expressions reproduce Propositions 1 and 2 to machine precision. At λ = 0.9 and k = 5 the resolvent sums agree with brute-force truncated sums over 400 lags to 10−12 relative. The implementation is lstd_lambda.py. The sharp-transition asymptotics, meaning the joint limit λ → 1 with γh → 0 that would turn the heuristic of §7 into a theorem, are open.

E

Experimental details

Environment. The simulator implements the family (1) with exact zero-order-hold stepping. For each instance it also computes the exact baselines J ∗ , (k ∗ , Jml ) and the fixed-point predictions (keq , J(keq )) from the closed forms of Appendices A and B, evaluated at the exploration variance actually used. A Gym-compatible wrapper exposes the same dynamics to external agents, and a privileged-state interface exposes the unobserved coordinate for the full-state control of §3. Deployed evaluation. Deployment sets se = 0 and runs the trained policy deterministically. The cost is the sample average of the h-normalized cost rate after burn-in, taken on a single long episode. An earlier version averaged over short episodes and inherited a bias from the resets; the single-episode estimate removes it. The implied gain is the regression coefficient of −u on v over the same trajectory, restricted to |v| > 0.02 to avoid dividing by near-zero observations. Calibration of the expected update. The claim in §3 that (7) is the expectation of the actual stochastic update, including its scale, was checked directly. With the critic held at its fixed point, the sample average of δn (−ηn vn /se ) over 1.5 × 105 steps was compared with ĝ at k ∈ {1.65, 3, 6}, for three independent seeds. The ratio of sample average to ĝ lies in [1.006, 1.028] across all nine runs, with no trend in k. The offset is Monte Carlo error at this sample size, not a scale mismatch.

24

Locating fixed points for non-Gaussian disturbances. For the five processes of §6 no closed form applies, so the fixed point is located empirically. We simulate the loop at fixed k, fit the critic by least squares on the simulated trajectory, form the sample expected update, and bisect in k. Run at the Gaussian instance, where the exact answer is known, the pipeline reproduces keq to 0.7%. Bisection resolution on the reported values is ±0.4 in keq . The two squared-OU processes are heavy-tailed. Their cost residuals in Table 1 are consistent with slower Monte Carlo averaging at equal sample size, rather than with a failure of the theory. A longer run is the natural check. Predictions recorded in advance. The qualitative outcomes of the symmetric fourth-moment process and of the symmetrised pair in §6 were written down before those runs were executed. The five-process design was chosen to break the Gaussian identity Cov(ζ02 , ζτ2 ) = 2C(τ )2 , not selected after the fact from a larger set. Deep agent. The agent is a public reference implementation of PPO (Huang et al., 2022) at a fixed version, with multilayer-perceptron actor and critic, generalized advantage estimation, a clipped surrogate objective and minibatch epochs. Frame-stack and fixed-exploration modes are command-line flags on the same file, so the arms of each contrast differ only in those flags. With full-state input the same agent recovers the known optimum, which is the internal control on the harness. Runs use 107 steps and five seeds per configuration, on a scheduler array over configurations and seeds. Frozen constants. values.

The decision rules of §9 were committed before submission with the following

constant

value

window for holding the solution high-gain regime divergence seed majority validity control, learned σ validity control, λ = 0.99 diagnostic arm

J ∈ [0.20, 0.27] and implied gain ≤ 5 implied gain > 5 sustained ≥ 106 steps J > 0.45 3 of 5; a mixed row is reported as mixed must reach J ≥ 0.29 and gain ≥ 7 must hold the solution learning rate divided by 4

The variance sweep of §9 fixed σ 2 ∈ {0.001, 0.003, 0.01, 0.03}, with the brackets stated there recorded in the same commit. Code, data, and what the record does and does not establish. The harness, the analysis scripts, the figure generator and the recorded predictions are at https://github.com/idilgozel/ belief-critic-restoration. The predictions and decision rules of §9 were written down and committed locally before the corresponding batches were submitted, and the values reproduced in this appendix are those committed values. The published repository is a single snapshot rather than the original history, so it does not by itself let a reader verify that ordering. Readers should treat the protocol as a statement by the author about how the work was conducted. Two things support it: every constant is reported here verbatim, and the code needed to rerun the tests is public. The substantive results do not rest on the ordering. Both registered predictions failed by wide margins, and a reader who distrusts the timeline can rerun the tests against the constants listed above and reach the same verdicts. 25

Earlier frame-stack result. An in-house implementation of the same algorithm, run for 4 × 106 steps at λ = 0.9, reported frame-stack agents reaching the class optimum. The reference implementation does not reproduce this at 107 steps, and §8 reports the latter. We record the discrepancy rather than resolve it. The two implementations differ in initialization, in optimizer defaults and in observation normalization, and we have not isolated which difference is responsible.

F

Assumptions ledger statement

scope

status

Lemma 1 (reduction) Lemma 2 (mean reversion) Proposition 1 (critic) Proposition 2 (update) Theorem 1 (χ law) Theorem 3 (cost) Corollary 2 (cost at keq ) Corollary 1 (components) Theorem 2 (rate) rate constant for full (1) Proposition 3 (λ) transition location in λ Conjecture 1 iterates follow the expected field

any disturbance, exact at every h stationarity only Gaussian loop, exact at every h Gaussian loop, exact at every h scaling limit, Gaussian any stationary ergodic m.s. continuous as above, given keq h → χ finite Gaussian mixture reduced family, completely monotone C default instance Gaussian loop, exact at every h heuristic across γ, h non-Gaussian disturbances default instance, six seeds

proven proven proven proven proven proven proven proven proven verified, proof open proven numerical conjecture numerical

The exactness of Propositions 1 and 2 requires a linear policy with Gaussian exploration, exact zero-order-hold sampling, the stationary regime, and the feature class (1, v, v 2 ). It also requires expected-update semantics for the actor-critic recursion, which is the standing limitation discussed in §10. Theorem 1 additionally takes h → 0 with β → 1. Nothing anywhere assumes small coupling c, any relation between γ and the cost weights, or knowledge of the kernel by the agent.

26

Related documents

• MDP Planning as Policy Inference#614923
• MDP Planning as Policy Inference#613956
• Policies for Science and for Education: Conflicts and Contacts.#460459
• Satisfaction Is Not Explanation: Auditing Vacuity and Training Influence in Temporal-Logic-Guided Reinforcement Learning#1064287
• Learning-Based Measurement-Robust Control Barrier Functions for Obstacle Avoidance under State Estimation Error#659036