Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes Zuyuan Zhang1 , Yongshan Chen2 , Mahdi Imani2 , Tian Lan1 1
Department of Electrical and Computer Engineering, The George Washington University 2 Department of Electrical and Computer Engineering, Northeastern University [email protected], [email protected], [email protected], [email protected]
arXiv:2607.27132v1 [cs.LG] 29 Jul 2026
Abstract An agent acting under partial observability must retain a recursively updateable statistic of history that restores the Markov property, but the smallest such statistic is generally unknown. We characterize this minimal Markov sufficient statistic for holonomy-cover decision processes, a structured POMDP class in which the visible dynamics are Markov and every realized visible transition applies a fixed permutation to a hidden mode. In particular, we construct the stable quotient, the coarsest observation-wise abstraction preserving one-step rewards and quotient successors, and prove that the pair of the current observation and stable class forms an exact finite Markov state. When the current class is correctly initialized, exact class tracking requires exactly the minimal memory symbols, in the sense that under reachability and pairwise decision separation at a maximizing observation, no arbitrary finite-memory controller can use fewer. Under resettable diagnostics, nearest-prototype class inference has exponentially decaying error, and a calibrate-then-restart reduction transfers finite-MDP guarantees to the recovered state. The results enable Holonomy Memory Reinforcement Learning. It represents memory by the current stable class, updates it through ordered edge transports, identifies local class coordinates when diagnostics are available, and applies a standard finite-MDP RL backbone after synchronization. Experiments recover an exact compression from raw states to quotient states and achieve perfect paired-order accuracy with three decisiontime memory states, matching the quotient oracle and outperforming the non-oracle baselines.
1
Introduction
Reinforcement learning (RL) relies on a Markov state that makes future rewards and action-conditioned transitions depend only on the present (Sutton, Barto et al. 1998). Under partial observability, the current observation alone may not be sufficient to predict the future. Partially observable Markov decision process (POMDP) theory restores the Markov property through belief states–posterior distributions over latent states conditioned on the interaction history (Åström 1965; Smallwood and Sondik 1973; Monahan 1982; Kaelbling, Littman, and Cassandra 1998). In practice, this belief is often approximated by encoding the action– observation history with sequential or memory-augmented neural architectures (Meuleau et al. 2013). Although these methods can learn useful history representations from a computational perspective, they generally do not characterize
the minimal Markov sufficient statistic–the smallest recursively updateable representation of history that preserves the Markovian property. We study a structured family of POMDPs through the lens of transport theory and identify its minimal Markov sufficient statistic. We model partial observability as hiddenmode transport over the graph of visible transitions. Each latent state consists of an observed component and an unobserved mode. The observed component evolves according to a Markov transition rule that is independent of the hidden mode. Whenever an action leads from one observation to the next, the realized transition applies a fixed permutation to the hidden mode. A trajectory therefore updates the hidden mode by composing these transition-specific transports in temporal order. We call the resulting model a holonomy-cover decision process (HCDP). Its visible dynamics are Markov, but two histories ending at the same observation can induce different hidden modes—and hence different rewards or optimal decisions—because their ordered transports differ. Our approach departs from existing work that primarily designs expressive or efficient history representations rather than characterizing the smallest Markov sufficient statistic. Recurrent, Transformer, and linear-recurrent networks can encode ordered histories in POMDPs, but their learned representations do not generally guarantee that the resulting memory-augmented process is Markov (Hausknecht and Stone 2015; Parisotto et al. 2020; Ni, Eysenbach, and Salakhutdinov 2021; Morad et al. 2024; Zhang, Tang, and Lan; Zhang, Fang, and Lan). Belief compression, predictivestate representations, and world models focus on summarizing posterior or predictive information (Roy, Gordon, and Thrun 2005; Igl et al. 2018; Littman and Sutton 2001; Singh, James, and Rudary 2012; Hafner et al. 2019b,a; Zhang, Imani, and Lan 2024; Zhang et al. 2025). Bisimulation and homomorphism methods provide principled aggregation only after a Markov state is identified (Givan, Dean, and Greig 2003; Ravindran 2003; Ferns, Panangaden, and Precup 2004; Abel, Hershkowitz, and Littman 2016; Gelada et al. 2019; Zhang et al. 2020; Efroni et al. 2022; Lamb et al. 2022), while reward machines and regular decision processes introduce finite automata for history dependence, but do not directly isolate the observation-wise minimal Markov state generated by ordered hidden transport (Bresina 1996; Thiébaux et al. 2006; Icarte et al. 2018; Camacho et al. 2019;
Icarte et al. 2022; Toro Icarte et al. 2019; Brafman, De Giacomo et al. 2019; Ronca and De Giacomo 2021; Cipollone et al. 2023; Deb et al. 2024; Zhang, Joe-Wong, and Lan 2026). Appendix A offers a detailed comparison. We analyze an HCDP by refining the hidden modes separately within each observation fiber. Two modes remain in the same class only if they agree on every immediate action reward and every feasible edge transports them into the same successor classes. The key proof is a monotone finite refinement: the operator stabilizes at a fixed point Π⋆ (Lemma 3.5), the exactness criterion characterizes every reward-and-successor-preserving abstraction (Proposition 3.7), and the resulting factorization shows that every exact observation-wise abstraction refines the stable quotient (Theorem 3.8). Consequently, the pair (o, c) of the current observation o and stable quotient class c = [i]o is an exact value-preserving finite Markov state (Theorem 3.9). Within this HCDP abstraction class, it is therefore shown to be the minimal Markov sufficient statistic. When the current class is correctly initialized, exactly maxo |Co | reusable memory symbols are necessary and sufficient for recursive class tracking (Corollary 3.12). If the initial class is unknown, the generic exact observable information state is instead a posterior over stable classes (Proposition 3.11). The analysis enables Holonomy Memory Reinforcement Learning (HMRL). HMRL represents memory by the current stable class, updates it through ordered edge transports, identifies local class coordinates when diagnostics are available, and applies a standard finite-MDP RL backbone after synchronization. The quotient is structural, but learning it requires additional information. We prove that arbitrary passive logs do not uniformly identify stable classes and transports (Theorem 4.2); under resettable calibrated diagnostics, class error decays exponentially (Theorem 4.6) and transition-anchored estimates eventually recover edge transports up to observation-wise relabeling (Theorem 4.9). Once local class tracking is synchronized, the learned process is isomorphic to the canonical quotient MDP (Theorem 5.2), and a calibrate-then-restart condition transfers the original finite-MDP guarantee to the recovered state (Corollary 5.4). Our main contributions are: • We formulate HCDPs and identify their stable quotient as the coarsest exact observation-wise Markov state, with a matching maxo |Co | memory characterization for exact known-class tracking. • We separate passive nonidentifiability from diagnostic recovery, identify classes and transports up to observationwise relabeling, and give the conditions under which a finite-MDP RL guarantee transfers to the recovered state. • We prove a nonabelian lower bound for count-factorized memories and show that ordered stable-class tracking avoids this loss with finite memory. Experiments support these claims. ChainCover is compressed from 216 raw states to 25 quotient states without reward, transition, or optimal-value inconsistency. On nonabelian LoopGuess, HMRL matches the quotient oracle, achieves 1.000 final success and perfect paired-order accuracy, and uses only three decision-time memory states.
2
Preliminaries
This section separates three objects that are easy to conflate: the hidden Markov state, the observable support path, and the controller’s recursively updated memory. The first is Markov but unavailable, the second is observable but may be insufficient, and the third is implementable but need not be minimal. We fix their notation before imposing the transport structure. Let S, O, and A be finite, and consider P = (S, O, A, P, r, h, γ, ρ0 ) with P (· | s, a) ∈ ∆(S), r : S × A → R bounded, h : S → O, γ ∈ (0, 1), and ρ0 ∈ ∆(S). Here ot = h(st ) and st+1 ∼ P (· | st , at ). For o ∈ O, let So = h−1 ({o}). A length-t observable history is Ht = (o0 , a0 , . . . , at−1 , ot ), and a history-dependent policy is π = (πt )t≥0 , t (· | Ht ) ∈ ∆(A). hπP i Its disπ t counted return is J(π) = E t≥0 γ r(st , at ) . The posterior bt (· | Ht ) is supported on Sot , but different histories ending at ot may induce different posteriors. Thus the visible state need not be Markov; a full belief is sufficient but generally continuous and finer than control requires. The action-labeled observation support graph X = (O, E) has E = n o ′ a P (s | s, a) > 0 for some s ∈ S , ′ o (o − → o ) : s′ ∈ S ′ . For o a path ω = e1 · · · ek , write src(ω) and tgt(ω) for its endpoints and ω2 ◦ ω1 for execution of ω1 followed by ω2 . A support path is latent realizable only when one sequence sj ∈ Soj witnesses all of its edges; edgewise feasibility alone need not provide such a sequence. Supplementary Lemma B.1 relates positive-probability histories to realizable paths. A deterministic finite-memory controller (M, ι, U, πM ) obeys m0 = ι(o0 ), mt+1 = U (mt , ot , at , ot+1 ), πM (· | ot , mt ) ∈ ∆(A). Equivalently, a recursive encoder F satisfies F (Ht · (at , ot+1 )) = U (F (Ht ), ot , at , ot+1 ). Histories merged by F therefore remain merged after every common observable continuation, so finite memory induces a finite right congruence. For evaluation, the analyst may augment the latent state to (st , mt ), which is Markov under a fixed controller; this does not grant the controller access to st , and pointwise Bellman maximization over (s, m) would generally be a latent-state oracle. Unless stated otherwise, realized rewards are training signals rather than within-episode controller observations. Reward-aware histories, stochastic emissions, and the formal augmentation results are in Supplementary Appendix B. We use [n] = {1, . . . , n}, Sn = Sym([n]), ∆(X ) for the simplex on finite X , and 1{·} for an indicator.
3
Holonomy-cover decision processes
This section introduces the transport structure and establishes the minimal exact Markov state of an HCDP. Lemma 3.3 gives every feasible visible path a unique hidden lift, and Lemma 3.5 shows that reward-initialized partition refinement reaches the coarsest stable family in finitely many splits. Proposition 3.7 characterizes exact abstractions, Theorem 3.8 proves that every exact observation-wise abstraction refines the stable quotient, and Theorem 3.9 turns the observation– class pair into a value-preserving finite MDP. Lemma 3.10
identifies the finite permutation group induced by closed directed walks, Proposition 3.11 gives the exact posterior state when the current class is unknown, and Corollary 3.12 proves that known-class tracking uses the minimal alphabet size maxo |Co |. These results separate the environment-side minimal statistic from the controller-side problem of observing or inferring its current value. Definition 3.1 (Holonomy-cover decision process). Fix finite O, A and n ≥ 1. An HCDP has latent state S = O × [n], observation h(o, i) = o, base kernel PO (· | o, a) ∈ ∆(O), an edge permutation σo,a,o′ ∈ Sn whenever PO (o′ | o, a) > 0, mean reward R : O × [n] × A → R, discount γ ∈ (0, 1), and initial law ρ0 ∈ ∆(O × [n]). Its transition is PH ((o′ , i′ ) | (o, i), a) = PO (o′ | o, a)1{i′ = σo,a,o′ (i)}. The visible successor distribution PO (· | o, a) is independent of the layer, while the realized edge deterministically transports that layer. Thus visible dynamics are Markov although rewards may remain path dependent. For statistical statements we observe Yt = R(ot , it , at ) + ξt and impose the bounded conditional sub-Gaussian condition in Supplementary Assumption C.1. Definition 3.2 (Directed path transport). For a feasible edge a e = (o − → o′ ), write σe := σo,a,o′ . For ω = e1 · · · ek , aj−1 where ej = (oj−1 −−−→ oj ), define σω = σek ◦ · · · ◦ σ e1 , σ∅o = id[n] . Hence σω2 ◦ω1 = σω2 ◦ σω1 ; the rightmost map acts first. Lemma 3.3 (Canonical HCDP path lifting). From any (o0 , i0 ), executing the actions of a directed path ω yields that Qk visible path with probability j=1 PO (oj | oj−1 , aj−1 ) > 0, and, conditional on it, the terminal layer is ik = σω (i0 ). Each feasible path therefore has a unique hidden lift from every initial layer. A fundamental-group interpretation additionally requires paired inverse edges, as stated in Supplementary Appendix D. Definition 3.4 (Stable partition operator). A partition family ′ is Π = {Πo }o∈O on [n]; write i ≡Π o j and let Π ⪯ Π mean ′ that every block of Π is contained in a block of Π ; thus Π is T (Π) j ⇐⇒ R(o, i, a) = the finer partition family. Define i ≡o Π R(o, j, a) ∀a, σo,a,o′ (i) ≡o′ σo,a,o′ (j), ∀(a, o′ ) : PO (o′ | o, a) > 0. A family is stable when T (Π) = Π. The reward partition Π(0) first separates layers with different immediate control consequences. The operator then propagates those distinctions backward through every feasible one-step continuation, so a merge survives only when it is recursively closed under future quotient evolution. Iteration stops exactly when no current or future reward distinction requires another split. Lemma 3.5 (Finite stabilization). T is monotone. Starting from the reward partition Π(0) and iterating Π(t+1) = T (Π(t) ) yields Π(t+1) ⪯ Π(t) and stabilizes after at most |O|(n − 1) strict refinements at the coarsest stable family Π⋆ . Every strict iteration increases the total number of blocks, which gives the finite bound. Exactness and fixed-point stability are separated below: an exact refinement may retain
unnecessary distinctions, whereas the fixed point is maximally compressed. Definition 3.6 (Exact class abstraction). Surjections qo : b ·, ·) : Cbo ×A → [n] → Cbo are exact if there are functions R(o, ′ R and, for every feasible (o, a, o ), a map τbo,a,o′ : Cbo → b qo (i), a), qo′ (σo,a,o′ (i)) = Cbo′ such that R(o, i, a) = R(o, ′ τbo,a,o (qo (i)). Proposition 3.7 (Exactness criterion). The quotient maps of a partition family Π define an exact abstraction if and only if Π ⪯ T (Π). Thus every stable family is exact; an exact family is stable precisely when Π = T (Π). Theorem 3.8 (Coarsest exact class abstraction). Let Co = [n]/Π⋆o and qo⋆ (i) = [i]o . Then q ⋆ is exact. For every exact qo : [n] → Cbo , there is a unique surjection κo : Cbo → Co such that qo⋆ = κo ◦ qo . Thus every exact abstraction refines the stable quotient. The factorization allows another exact abstraction to split a stable class but forbids it from merging layers separated by Π⋆ . Theorem 3.9 (Exact Markovization). For every feasible (o, a, o′ ), τo,a,o′ ([i]o ) = [σo,a,o′ (i)]o′ , R̄(o, [i]o , a) = R(o, i, a) are representative independent. Hence S̄ = {(o, c) : c ∈ Co } is an MDP with P̄ ((o′ , c′ ) | ′ ′ (o, P c), a) = PO (o | o, a)1{c = τo,a,o′ (c)}, ρ̄0 (o, c) = ρ (o, i). For every stationary quotient policy π̄, i:[i]o =c 0 ↑ its lift π (a | o, i) = π̄(a | o, [i]o ) satisfies Vπ↑ (o, i) = ⋆ (o, i) = V̄ ⋆ (o, [i]o ). V̄π̄ (o, [i]o ); in particular, Vfull ⋆ Henceforth write ct := qot (it ) = [it ]ot ∈ Cot for the canonical stable class of the realized latent layer. This theorem compresses the fully observed latent HCDP. It does not make the class observable in the original POMDP; the controller must know, infer, or maintain it. For a path ω, let τω be the ordered composition of its edge maps; Supplementary Corollary C.2 gives τω ([i]src(ω) ) = [σω (i)]tgt(ω) . Lemma 3.10 (Directed quotient holonomy). For every closed directed walk ω based at o, τω ∈ Sym(Co ), and Holdir = {τω : ω closed at o} is a finite subgroup of o Sym(Co ). A nonidentity holonomy therefore distinguishes histories that return to the same observation. The quotient solves the structural aggregation problem, but implementability depends on the initial information: a known class propagates deterministically, whereas an unknown class must be represented by a posterior over quotient classes. Proposition 3.11 (Belief reduction over stable classes). When reward samples are not within-episode observations, initialize β0 (·) = P(c0 ∈ · | o0 ) and define, for c ∈ Cot , P βt (c) = P(ct = c | Ht ). It obeys βt+1 (c′ ) = | c:τot ,at ,ot+1 (c)=c′ βt (c), E[R(ot , it , at ) P Ht , at ] = c βt (c)R̄(ot , c, at ). Hence (ot , βt ) is exact; if βt = δct , then ct+1 = τot ,at ,ot+1 (ct ) tracks the class exactly. If the initial class is unknown, the generic exact information state is the posterior βt on Cot . Its ambient simplex is
continuous, although a fixed finite HCDP may generate only a finite reachable orbit of beliefs. If the class is initialized correctly, deterministic transport keeps the posterior degenerate and supplies a finite recursive controller state. The rewardobserved update and the formal known-class memory lemma are in Supplementary Appendix C. Corollary 3.12 (Minimal exact class-tracking memory). Let m⋆exact = maxo∈O |Co |. Among all finite alphabets M for which there exist injective encoders ηo : Co → M and an update U satisfying U (ηo (c), o, a, o′ ) = ηo′ (τo,a,o′ (c)) for every feasible edge and every c ∈ Co , the minimum alphabet cardinality is |M|min = m⋆exact . The same memory symbols may be reused across different observations. This is a known-class representational statement. Pairwise decision separation yields an unconditional controller lower bound in Corollary 6.6.
4
Identifying classes and transports from diagnostics
This section characterizes when the latent stable quotient can be identified. Theorem 4.2 rules out uniform recovery from unrestricted passive logs, whereas Lemma 4.3 shows that every pair of distinct stable classes has a finite separating continuation. Under resettable calibrated diagnostics, Theorem 4.6 gives an exponentially decaying local classification error, and Theorem 4.9 shows that transition-anchored rowwise estimates eventually recover every covered edge map up to observation-wise relabeling. Lemma 4.11 proves exact propagation once the initial label and used transports are correct, while Proposition 4.12 upgrades summable diagnostic errors and eventual map recovery to permanent class synchronization after an almost surely finite time. Thus the quotient is identified only in local gauges, which is sufficient because gauge-equivalent coordinates define isomorphic quotient MDPs. The structural identification problem concerns the local label sets and edge maps. Once the local state is synchronized, the base kernel and mean rewards can be estimated from ordinary transition and reward samples; the initial law depends on the chosen restart distribution. For comparison with the canonical quotient, package a full local-coordinate description as label sets {Cbo }o∈O , a base kernel PbO , rewards b edge maps {b R, τe }e∈E , and an initial law ρb0 . Definition 4.1 (Observation-wise gauge equivalence). A learned description and the canonical quotient are gauge equivalent when PbO = PO and there are bijections λo : b ĉ, a) = R̄(o, λo (ĉ), a), λo′ ◦ τbe = Cbo → Co such that R(o, τe ◦ λo , ρb0 (o, ĉ) = ρ̄0 (o, λo (ĉ)). The commuting transport equation makes the two quotient MDPs isomorphic; no cross-observation agreement between the numerical labels is required. Theorem 4.2 (Passive nonidentifiability). There are two finite HCDPs and a common behavior policy whose stable quotient descriptions are not gauge equivalent but whose complete observable logs (o0 , a0 , Y0 , o1 , . . .) have the same law. Therefore unrestricted passive logs admit no uniformly
consistent estimator of stable classes, transports, and the quotient MDP over all HCDPs and behavior policies. The construction omits a distinguishing action. Accordingly, passive HMRL is conditional, whereas the following result uses resettable diagnostics. Lemma 4.3 (Finite distinguishing witnesses). Let L⋆ = min{L : Π(L) = Π⋆ }. For distinct c, c′ ∈ Co , some path ω of length at most L⋆ and action b satisfy R̄(tgt(ω), τω (c), b) ̸= R̄(tgt(ω), τω (c′ ), b). This supplies finite separating experiments; the statistical interface below adds repeatability and a quantitative margin. Definition 4.4 (Diagnostic fingerprint). Let O+ = {o ∈ O : |Co | ≥ 2}. If O+ = ∅, every quotient fiber is a singleton and no classification probe is required. Otherwise, a bounded-horizon protocol u starting from (o, i) returns a scalar W (o, i; u) with mean G(o, i; u). For each o ∈ O+ , choose Uprobe (o) = {u1 , . . . , udo }. Assume G is constant on stable classes, write Ḡ(o, c; u) = G(o, i; u) for o any i ∈ c, and define ϕo (c) = (Ḡ(o, c; uℓ ))dℓ=1 , ∆ = ′ min o∈O+ ∥ϕo (c) − ϕo (c )∥2 > 0. Singleton fibers use c,c′ ∈Co , c̸=c′
their unique label and require no probes. Assume the interface can restore the same latent checkpoint and generate conditionally independent repetitions with sub-Gaussian variance proxy ν 2 . It supplies a local label set Cbo and prototypes satisfying ∥ϕeo (ĉ) − ϕo (λo (ĉ))∥2 ≤ ∆/8. These assumptions are stated separately in Supplementary Appendix C.2; they constitute calibrated classification, not unsupervised discovery from passive data. Definition 4.5 (Nearest-prototype inference). With b i; uℓ ) = m repetitions per coordinate, set G(o, Pm 1 (s) o b b (o, i; uℓ ), ϕo (i) = (G(o, i; uℓ ))dℓ=1 ,b c ∈ s=1 W m b e arg minĉ∈Cbo ∥ϕo (i) − ϕo (ĉ)∥2 , with deterministic tie breaking. Theorem 4.6 (Local class inference). Under Assumptions C.5, C.7, C.8, C.9, and C.10, for o ∈ O+ , i ∈ [n], and c = [i]o , conditional on the restored start state = (o, i), sstart m∆2 P(λo (b c) ̸= c | sstart = (o, i)) ≤ 2do exp − 128ν 2 do . For o ∈ / O+ , the inferred class is deterministically correct without diagnostic sampling. A logarithmic, summable repetition schedule therefore makes only finitely many classification errors almost surely; see Supplementary Corollaries B.11–B.12. Local class labels alone do not determine how labels propagate between observations. Anchored source–target diagnostics estimate this map without requiring a common numerical naming of classes. Assumption 4.7 (Anchored edge diagnostics). For a sama pled edge e = (o − → o′ ), the interface preserves the exact source checkpoint, executes the edge, and diagnoses the paired source and target checkpoints. Definition 4.8 (Row-wise transport estimator). Let Ne [ĉ, ĉ′ ] count paired inferred labels and set τbe (ĉ) ∈ arg maxĉ′ ∈Cb ′ Ne [ĉ, ĉ′ ], with deterministic tie breaking. o
The true map in local coordinates is τeλ = λ−1 o′ ◦ τ e ◦ λ o ; it need not be injective, so recovery is row-wise. Theorem 4.9 (Eventual transport recovery). Fix the gauges from Assumption C.10. Under Assumptions 4.7 and C.11, if only finitely many source or target classification errors a occur almost surely, then for every feasible e = (o − → o′ ) and ĉ ∈ Cbo , there is an almost surely finite time after which, permanently, τbe (ĉ) = λ−1 o′ (τe (λo (ĉ))). Definition 4.10 (Local-coordinate class tracker). At an exogenous initialization, diagnostics choose b ct0 ∈ Cbot0 . Until (b c ). A reset discards the the next reset, b ct+1 = τb at →ot+1 ) t (ot − propagated label and triggers a new diagnostic initialization. Lemma 4.11 (Exact local tracking). On a reset-free interval, if λot0 (b ct0 ) = ct0 and every used edge satisfies τbe = λ−1 tgt(e) ◦ τe ◦ λsrc(e) , then λot (b ct ) = ct for all t ≥ t0 on that interval. Proposition 4.12 (Eventual exact quotient tracking). If exogenous initialization times are almost surely unbounded, their diagnostic error probabilities are summable, every edge–class transport entry used by the tracker is eventually correct, and the tracker uses the preceding recursion, then for fixed local gauges there is an almost surely finite T such that λot (b ct ) = ct for every t ≥ T . All formal access, coverage, estimator, and exact-tracking conditions are stated in Supplementary Appendix C.2.
5
Quotient-lifted reinforcement learning
This section converts synchronized local class labels into a valid state for an RL backbone. Theorem 5.2 proves that, whenever the current label and all relevant edge maps are correct, the learned observation–label process has the exact quotient transition law, conditional reward mean, and noise control, and is isomorphic to the canonical quotient MDP. Corollary 5.3 combines the diagnostic recovery results with this theorem to obtain eventual ambient correctness, while emphasizing that eventual correctness alone does not automatically justify an arbitrary learning theorem. Corollary 5.4 gives the required transfer condition: restart the finite-MDP backbone at a learned-information stopping time using only post-calibration data, or use a guarantee explicitly robust to the finite corrupted prefix and induced algorithmic state. This distinguishes the proved diagnostic HMRL reduction from passive variants, which remain conditional because of TheF orem 4.2. Let Sb = o ({o} × Cbo ) and zbt = (ot , b ct ). The proof-level coordinate map is Λλ (o, ĉ) = (o, λo (ĉ)). Definition 5.1 (Local quotient MDP). The transport τeλ is the exact canonical map written in local coordinates, whereas bλ indiτbe is its learned estimator. The hats on Pbλ and R cate the local state space, not statistical estimation. Define bλ τeλ = λ−1 tgt(e) ◦ τe ◦ λsrc(e) , R (o, ĉ, a) = R̄(o, λo (ĉ), a), and, for ẑ = (o, ĉ) and ẑ ′ = (o′ , ĉ′ ), Pbλ (ẑ ′ | ẑ, a) = ( λ PO (o′ | o, a)1 ĉ′ = τo,a,o PO (o′ | o, a) > 0, ′ (ĉ) , 0,
PO (o′ | o, a) = 0.
The local initial law is ρbλ0 (o, ĉ) = ρ̄0 (o, λo (ĉ)). Through Λλ , stationary policies and values transfer by π̄(a | o, c) =
bb (o, ĉ) = V̄π̄ (o, λo (ĉ)). Thus the π b(a | o, λ−1 o (c)) and Vπ learned labels may be used directly; the gauges are proof devices, not algorithmic inputs. Theorem 5.2 (Exact reduction after synchronization). Let Gt− be the pre-transition sigma-field generated by the latent and observable histories, the current tracker and transport tables, and the chosen action at , before (Yt , ot+1 ) is observed. Let Fbt− ⊆ Gt− be the corresponding learnedinformation subfiltration, which excludes the latent class. Let At be the event that λot (b ct ) = ct and, for every o′ ′ with PO (o | ot , at ) > 0, τbot ,at ,o′ = λ−1 o′ ◦ τot ,at ,o′ ◦ λot . On b P(b At , for every ẑ ′ ∈ S, zt+1 = ẑ ′ | Gt− ) = Pbλ (ẑ ′ | zbt , at ), bλ (b and E[Yt | Gt− ] = R zt , at ). The centered reward remains conditionally sub-Gaussian with variance proxy σY2 , and Λλ is an MDP isomorphism to the canonical quotient. If an almost surely finite T0 satisfies At for every t ≥ T0 , these laws hold after T0 relative to the ambient filtration (Gt− ). If, in addition, T0 is a stopping time for (Fbt− ), then the same laws hold relative to the post-T0 learned-information filtration. The distribution at T0 need not equal ρbλ0 . The theorem identifies the transition kernel, conditional reward mean, and martingale noise. A stationary rewardoutput kernel additionally requires the full conditional law of Yt to depend only on the quotient state and action. Corollary 5.3 (Diagnostics imply eventual exact reduction). Under Theorem 4.9 and Proposition 4.12, fixed gauges and an almost surely finite T0 satisfy At for every t ≥ T0 , giving eventual ambient correctness in Theorem 5.2. A finiteMDP backbone may be invoked directly only after a certified calibration-and-restart stopping time, as in Corollary E.3, or under a separate theorem formulated for the ambient filtration and robust to a finite corrupted prefix. Corollary 5.4 (Transfer of an MDP-backbone guarantee). Suppose there is an almost surely finite stopping time T for the learned-information filtration such that At holds for every t ≥ T . Any finite-MDP guarantee transfers with its original exploration, sampling, step-size, and approximation conditions if the backbone is restarted at T using only post-T data and its theorem applies to the resulting initial-state law, or uniformly over initial states. The same conclusion also holds when the backbone theorem is explicitly invariant to the finite pre-T prefix and the induced internal algorithmic state. Without one of these conditions, eventual representation correctness alone is insufficient. The finite-sample calibration corollary, diagnostic and passive algorithms, and conditional passive reduction are in Supplementary Appendix E. In the passive regime, exact reduction is therefore conditional on eventual class correctness under fixed local gauges; Theorem 4.2 precludes an unconditional claim from unrestricted logs.
6
Nonabelian loop memory barrier
This section establishes when loop order is intrinsically control relevant. Theorem 6.4 proves that any terminal memory factoring through signed loop counts assigns the same action distribution to equal-count histories and therefore incurs a
fixed continuation-value loss when their transported classes are decision separated; Corollary 6.5 applies this obstruction to a zero-count commutator. Corollary 6.6 removes the count-based architectural restriction and shows that pairwise decision separation forces any exact finite-memory controller to distinguish every reachable stable class. Proposition 6.7 realizes the gap in a three-layer, two-action HCDP, while Proposition 6.8 shows that ordered transport tracking is exact using only the current finite class rather than the complete loop word. Proofs and construction details are given in Appendices I and F. Fix a base observation o. Definition 6.1 (Executable reversible loop alphabet). A fam− r ily of directed closed walks {ℓ+ j , ℓj }j=1 based at o is reversible when τℓ− = τℓ−1 + . The inverse transport need not j
j
reverse physical edges: every permutation has finite order, so its inverse is a positive power and may be executed by repeated positive traversal. These loops induce a representation of the free group Fr = ⟨x1 , . . . , xr ⟩, ρo : Fr → Sym(Co ), ρo (x±1 , j ) = τℓ± j with the algebraic composition convention ρo (uv) = ρo (u)◦ ρo (v), so the loop represented by v is executed first and the rightmost transport acts first. For reachable c0 , write Definition 6.2 (Abelianized loop memory). The abelianization ab : Fr → Zr records signed generator counts. A surrogate S : Fr → M is abelianized when S = f ◦ ab, equivalently ab(w) = ab(w′ ) =⇒ S(w) = S(w′ ).
because both have abelianization zero, and cannot be exact at both resulting classes. In particular, a commutator and the empty word both have zero abelianization, although the commutator may act P nontrivially on stable classes. Any encoder A(w) = j Nj (w)vj followed by an arbitrary readout also factors through ab and inherits the same barrier. The formal countadditive corollary is in Supplementary Appendix F. The preceding result targets count-based encoders. The next corollary removes that architectural restriction: pairwise decision separation forces any exact finite-memory controller to distinguish all reachable classes at the observation. Corollary 6.6 (Arbitrary finite-memory lower bound). Fix o. If every distinct pair in Co is decision separated and every class is reached by a positive-probability history ending at o, then any finite-memory controller attaining V̄ ⋆ (ot , ct ) after every such history satisfies |M| ≥ |Co |. At an observation attaining m⋆exact , this gives |M| ≥ m⋆exact . Proposition 6.7 (Three-layer nonabelian HCDP). With one observation, actions a, b, n = 3, transports σa = (12), σb = (123), and, suppressing the fixed observation in R, reward vectors (R(i, a), R(i, b))3i=1 = ((1, 0), (0, 1), (0, 0)), the stable quotient is discrete and its transports generate S3 . An executable commutator sends class 1 to class 2; for 0 < γ < 1/2, their unique optimal actions are a and b, respectively. Thus the obstruction is finite.
It includes signed-count vectors and arbitrary downstream readouts, but not general order-sensitive recurrent or attention encoders. Let Q̄⋆ be the quotient optimal action value and A⋆ (o, c) = arg maxa Q̄⋆ (o, c, a).
Proposition 6.8 (Ordered stable-class tracking). Starting from known c, exact tracking maps w to ρo (w)(c) and therefore distinguishes any w, w′ with different transported classes, even when ab(w) = ab(w′ ). It stores only the finite current class, not the complete word.
Definition 6.3 (Decision separation). Classes c, c′ ∈ Co are decision separated when A⋆ (o, c) ∩ A⋆ (o, c′ ) = ∅. For p P ∈ ∆(A) and d ∈ {c, c′ }, let ℓ(p, d) = ⋆ V̄ (o, d) − a p(a)Q̄⋆ (o, d, a), and define εdec (o; c, c′ ) = minp∈∆(A) maxd∈{c,c′ } ℓ(p, d) > 0.
The executable construction and count-additive specialization are in Supplementary Appendix F.
The strict positivity follows because ∆(A) is compact, ℓ(·, d) is continuous, and the two disjoint optimal-action faces admit no common zero-loss distribution. Noncommuting raw transports alone do not imply a control obstruction: the commutator must act differently on reachable stable classes, and those classes must demand incompatible decisions. Theorem 6.4 (Abelianization barrier). Suppose ab(w) = ab(w′ ) but cw and cw′ are decision separated. Any controller whose terminal memory has the form m(w) = go (ab(w)), go : Zr → M, uses the same action distribution after both histories; at least one therefore incurs conditional optimality loss at least εdec (o; cw , cw′ ). Such memory cannot be exact on a reachable history set containing both histories. Corollary 6.5 (Commutator barrier). If a reachable c0 is decision separated from ρo ([u, v])(c0 ), then every abelianized memory merges the commutator [u, v] with the empty word
7
Experiments
We evaluate four finite consequences of the theory in controlled HCDPs: exact stable-quotient recovery, concentration of diagnostic identification, failure of commutative count memory under nonabelian transport, and quotient-level control with finite memory. These experiments are intended as falsifiable implementation checks under known model assumptions rather than evidence that arbitrary POMDP benchmarks admit an exact finite quotient. Complete environment definitions, protocols, metrics, robustness sweeps, and hyperparameters are given in Appendix H. Environments and methods. ChainCover is an acyclic depth-D construction in which three reward-relevant classes are revealed only at a terminal query, forcing stable refinement to propagate distinctions backward while merging a nuisance coordinate. LoopGuess contains two visible loop excursions with quotient transports α = (12) and β = (123), which generate the nonabelian group S3 ; in particular, samecount words such as αβ and βα can induce different terminal classes. All control methods use the same tabular Q-learning
Obs-Q Count-Q
concentration reference Local class error Edge-transport error
10−1
Raw-Q oracle
Quotient-Q oracle HMRL-D (ours)
0.9
Query success
Error probability
History-Q
1.0
100
10−2
10−3
0.8 0.7 0.6 0.5 0.4 0.3
10−4
0.2
1
2
4
8
16
32
64
0
500
Diagnostic repetitions m
1000
1500
2000
2500
3000
3500
Online training episodes
Figure 1: Diagnostic recovery and online control. Left: repeated probes reduce class and edge-transport errors at σdiag = 0.5, with zero observed error by m = 8. Right: after calibration and restart, HMRL-D matches the quotient-oracle learning curve, whereas less informative or uncompressed representations remain limited under the same online budget. shading denotes one standard deviation where visible. Quotient-Q oracle
Raw-Q oracle
0.34
Count-Q
0.34 0.50
History-Q
1.00
Raw-Q oracle HMRL-D (ours)
1.00
Quotient-Q oracle
1.00 chance
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Same-count paired-order accuracy
1.0
Final query success
1.0 Obs-Q
Table 1: Primary LoopGuess comparison. Values are mean ± standard deviation. Oracle rows are references; bold marks the best deployable result.
HMRL-D (ours)
0.9 0.8 0.7
History-Q
0.6
Count-Q
0.5 0.4 0.3
Obs-Q
100
101
102
103
Visited decision-time memory states (log scale)
Figure 2: Ordered-memory necessity and memory efficiency. Left: on equal-count histories with different ordered transports, Count-Q remains near the 1/3 chance level, while HMRL-D and the exact oracles attain perfect accuracy. Right: HMRL-D matches quotient-oracle success using three decision-time states, compared with 48 for Raw-Q and more than 103 visited states for History-Q.
backbone and differ only in their state representation: observation only (Obs-Q), commutative loop counts (Count-Q), the complete loop word (History-Q), the privileged raw latent state (Raw-Q), the true quotient class (Quotient-Q), or the class propagated by calibrated transports (HMRL-D). HMRL-D uses the simulator-provided calibrated diagnostic interface, completes transport calibration, and then restarts Q-learning, as required by Corollary 5.4. Structural and diagnostic checks. For ChainCover with D = 6 and nuisance size b = 8, exhaustive refinement compresses 216 raw states to the analytical 25 quotient states and stabilizes at round 7. The recovered quotient has zero reward inconsistency, zero successor-class inconsistency, and zero optimal-value gap, while all 24 one-pair strict coarsenings violate exactness. Rejecting all tested one-pair merges provides a local irreducibility check beyond mere sufficiency and complements the analytical coarseness result. In the noisy-prototype diagnostic experiment, both class and edgetransport errors decrease with repeated probes and are zero in the reported trials by m = 8 at the primary noise level σdiag = 0.5; see Figure 1. These results separately verify quotient construction and the calibrated interface used by HMRL-D. Complete structural and diagnostic measurements are reported in Appendices H.3–H.5.
Method
Final success
Paired-order accuracy
Decision-time states
Obs-Q Count-Q History-Q
0.351 ± 0.055 0.543 ± 0.014 0.686 ± 0.007
0.336 ± 0.005 0.342 ± 0.003 0.502 ± 0.014
1.0 ± 0.0 266.7 ± 6.5 1005.3 ± 16.6
Raw-Q oracle
1.000 ± 0.000
1.000 ± 0.000
48.0 ± 0.0
HMRL-D (ours) Quotient-Q oracle
1.000 ± 0.000 1.000 ± 0.000
1.000 ± 0.000 1.000 ± 0.000
3.0 ± 0.0 3.0 ± 0.0
Control results. The primary LoopGuess setting uses nuisance multiplicity b = 16 and query probability pquery = 0.1. HMRL-D matches Quotient-Q in final success, normalized learning-curve AUC (0.976 ± 0.001), and pairedorder accuracy while using only three decision-time class states. Raw-Q also reaches perfect final success but retains all 48 class–nuisance combinations and has lower AUC (0.966 ± 0.002). Count-Q improves ordinary success over Obs-Q but remains at chance on same-count order pairs, directly exhibiting the information loss predicted by the nonabelian barrier. History-Q retains order in principle, but its visited decision-time memory exceeds 103 states and learns more slowly under the fixed budget. HMRL-D requires 216 calibration interactions, reported separately from online control. Figures 1 and 2, together with Table 1, provide the primary evidence; Appendix H.9 reports complete metrics and robustness sweeps over nuisance multiplicity and expected history length.
8
Conclusion
We presented HMRL and identified, for holonomy-cover decision processes, the stable observation–class quotient as the minimal Markov sufficient statistic among exact observationwise abstractions. This quotient preserves precisely the reward and transition distinctions required for Bellman control while discarding irrelevant latent differences. Under a resettable, locally calibrated diagnostic interface, HMRL identifies local classes and quotient transports up to observationwise relabeling and applies standard finite-MDP RL to the resulting state. We further showed that commutative summaries such as loop counts fail when hidden transports are nonabelian. Experiments verified exact state compression, quotient recovery, and order-sensitive control: ChainCover was reduced from 216 states to 25 exact quotient states, while HMRL achieved 1.000 success and perfect pairedorder accuracy on LoopGuess. ChainCover and LoopGuess thus test structural exactness and order-sensitive control, respectively. Extending quotient discovery to noisy, continuous, and passive-data settings remains an important direction.
References Abel, D.; Hershkowitz, D.; and Littman, M. 2016. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning, 2915–2923. PMLR.
Åström, K. J. 1965. Optimal control of Markov processes with incomplete state information I. Journal of mathematical analysis and applications, 10: 174–205. Brafman, R. I.; De Giacomo, G.; et al. 2019. Regular Decision Processes: A Model for Non-Markovian Domains. In IJCAI, volume 19, 5516–5522. Bresina, J. L. 1996. Heuristic-biased stochastic sampling. In Proceedings of the thirteenth national conference on Artificial intelligence-Volume 1, 271–278. Camacho, A.; Icarte, R. T.; Klassen, T. Q.; Valenzano, R. A.; and McIlraith, S. A. 2019. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In IJCAI, volume 19, 6065–6073. Cipollone, R.; Jonsson, A.; Ronca, A.; and Talebi, M. S. 2023. Provably efficient offline reinforcement learning in regular decision processes. Advances in Neural Information Processing Systems, 36: 39395–39428. Deb, A.; Cipollone, R.; Jonsson, A.; Ronca, A.; and Talebi, M. S. 2024. Tractable offline learning of regular decision processes. arXiv preprint arXiv:2409.02747. Efroni, Y.; Jin, C.; Krishnamurthy, A.; and Miryoosefi, S. 2022. Provable reinforcement learning with a short-term memory. In International Conference on Machine Learning, 5832–5850. PMLR. Fang, Z.; Zhang, Z.; Imani, M.; and Lan, T. 2026. ManifoldConstrained Energy-Based Transition Models for Offline Reinforcement Learning. arXiv preprint arXiv:2602.02900. Ferns, N.; Panangaden, P.; and Precup, D. 2004. Metrics for Finite Markov Decision Processes. In Uai, volume 4, 162–169. Gelada, C.; Kumar, S.; Buckman, J.; Nachum, O.; and Bellemare, M. G. 2019. Deepmdp: Learning continuous latent space models for representation learning. In International conference on machine learning, 2170–2179. PMLR. Givan, R.; Dean, T.; and Greig, M. 2003. Equivalence notions and model minimization in Markov decision processes. Artificial intelligence, 147(1-2): 163–223. Grazzi, R.; Siems, J.; Zela, A.; Franke, J.; Hutter, F.; et al. 2025. Unlocking state-tracking in linear rnns through negative eigenvalues. In International Conference on Learning Representations, volume 2025, 36565–36597. Hafner, D.; Lillicrap, T.; Ba, J.; and Norouzi, M. 2019a. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. Hafner, D.; Lillicrap, T.; Fischer, I.; Villegas, R.; Ha, D.; Lee, H.; and Davidson, J. 2019b. Learning latent dynamics for planning from pixels. In International conference on machine learning, 2555–2565. PMLR. Hausknecht, M. J.; and Stone, P. 2015. Deep Recurrent Q-Learning for Partially Observable MDPs. In AAAI fall symposia, volume 45, 141. Icarte, R. T.; Klassen, T.; Valenzano, R.; and McIlraith, S. 2018. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, 2107–2116. PMLR.
Icarte, R. T.; Klassen, T. Q.; Valenzano, R.; and McIlraith, S. A. 2022. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73: 173–208. Igl, M.; Zintgraf, L.; Le, T. A.; Wood, F.; and Whiteson, S. 2018. Deep variational reinforcement learning for POMDPs. In International conference on machine learning, 2117– 2126. PMLR. Kaelbling, L. P.; Littman, M. L.; and Cassandra, A. R. 1998. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2): 99–134. Lamb, A.; Islam, R.; Efroni, Y.; Didolkar, A.; Misra, D.; Foster, D.; Molu, L.; Chari, R.; Krishnamurthy, A.; and Langford, J. 2022. Guaranteed discovery of control-endogenous latent states with multi-step inverse models. arXiv preprint arXiv:2207.08229. Littman, M.; and Sutton, R. S. 2001. Predictive representations of state. Advances in neural information processing systems, 14. Merrill, W.; Petty, J.; and Sabharwal, A. 2024. The illusion of state in state-space models. arXiv preprint arXiv:2404.08819. Meuleau, N.; Peshkin, L.; Kim, K.-E.; and Kaelbling, L. P. 2013. Learning finite-state controllers for partially observable environments. arXiv preprint arXiv:1301.6721. Monahan, G. E. 1982. State of the art—a survey of partially observable Markov decision processes: theory, models, and algorithms. Management science, 28(1): 1–16. Morad, S.; Lu, C.; Kortvelesy, R.; Liwicki, S.; Foerster, J.; and Prorok, A. 2024. Recurrent reinforcement learning with memoroids. Advances in Neural Information Processing Systems, 37: 14386–14416. Ni, T.; Eysenbach, B.; and Salakhutdinov, R. 2021. Recurrent model-free rl can be a strong baseline for many pomdps. arXiv preprint arXiv:2110.05038. Parisotto, E.; Song, F.; Rae, J.; Pascanu, R.; Gulcehre, C.; Jayakumar, S.; Jaderberg, M.; Kaufman, R. L.; Clark, A.; Noury, S.; et al. 2020. Stabilizing transformers for reinforcement learning. In International conference on machine learning, 7487–7498. PMLR. Ravindran, B. 2003. SMDP homomorphisms: An algebraic approach to abstraction in semi markov decision processes. Ronca, A.; and De Giacomo, G. 2021. Efficient PAC reinforcement learning in regular decision processes. arXiv preprint arXiv:2105.06784. Roy, N.; Gordon, G.; and Thrun, S. 2005. Finding approximate POMDP solutions through belief compression. Journal of artificial intelligence research, 23: 1–40. Singh, S.; James, M.; and Rudary, M. 2012. Predictive state representations: A new theory for modeling dynamical systems. arXiv preprint arXiv:1207.4167. Smallwood, R. D.; and Sondik, E. J. 1973. The optimal control of partially observable Markov processes over a finite horizon. Operations research, 21(5): 1071–1088.
Subramanian, J.; Sinha, A.; Seraj, R.; and Mahajan, A. 2022. Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research, 23(12): 1–83. Sutton, R. S.; Barto, A. G.; et al. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge. Tang, S.; Zhang, Z.; Imani, M.; and Lan, T. 2026. Nonzero: Interaction-guided exploration for multi-agent monte carlo tree search. arXiv preprint arXiv:2605.00751. Thiébaux, S.; Gretton, C.; Slaney, J.; Price, D.; and Kabanza, F. 2006. Decision-theoretic planning with non-Markovian rewards. Journal of Artificial Intelligence Research, 25: 17– 74. Toro Icarte, R.; Waldie, E.; Klassen, T.; Valenzano, R.; Castro, M.; and McIlraith, S. 2019. Learning reward machines for partially observable reinforcement learning. Advances in neural information processing systems, 32. Zhang, A.; McAllister, R.; Calandra, R.; Gal, Y.; and Levine, S. 2020. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742. Zhang, Z.; Fang, Z.; and Lan, T. ???? Geometric Coherence Learning for Structuring Value Functions in Plain MDPs. In Forty-third International Conference on Machine Learning. Zhang, Z.; Imani, M.; and Lan, T. 2024. Modeling other players with bayesian beliefs for games with incomplete information. arXiv preprint arXiv:2405.14122. Zhang, Z.; Joe-Wong, C.; and Lan, T. 2026. Matrix-Space Reinforcement Learning for Reusing Local Transition Geometry. arXiv preprint arXiv:2605.14304. Zhang, Z.; Tang, S.; and Lan, T. ???? HodgeFlow Policy Search by Topologically Dissecting Temporal-Difference Signals in Non-Markovian Environments. In Forty-third International Conference on Machine Learning. Zhang, Z.; Zhou, H.; Imani, M.; Lee, T.; and Lan, T. 2025. Learning to collaborate with unknown agents in the absence of reward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 14502–14511.
A
Positioning relative to prior work
Table 2 compares the scope of formal guarantees rather than empirical expressive power. A checkmark denotes a general guarantee under the method-specific assumptions cited in Section 1; a parenthesized checkmark denotes a guarantee available only for particular variants or additional assumptions; × denotes no general guarantee; and “–” means that the property is not the method’s primary target. Observation aliasing refers to distinct latent states sharing the same current observation while differing in rewards or future behavior.
B
Deferred technical preliminaries
We relate observable histories to latent-realizable support paths and formalize the right congruence induced by finite memory. We then treat reward-aware histories, deterministic and stochastic observation augmentations, and the Bellman equation for a fixed implementable controller, keeping controller evaluation separate from latent-state oracle optimization. Lemma B.1 (Observable histories and realizable support paths). Let π be any history-dependent policy. First, if an observable history Ht = (o0 , a0 , o1 , . . . , at−1 , ot ) has positive probability under π, then its associated visible path a
at−1
0 ω(Ht ) = (o0 −→ o1 ) · · · (ot−1 −−−→ ot )
is a ρ0 -realizable directed support path in X. Conversely, suppose that ω(Ht ) is ρ0 -realizable and that πj (aj | Hj ) > 0,
j = 0, . . . , t − 1,
along the corresponding observable prefixes. Then Ht has positive probability under π. In particular, an arbitrary directed support path in X need not be latent-realizable, because the edge-wise existential witnesses in the definition of E need not be compatible with one another. Finite-memory history quotients. Recall the deterministic history encoder F induced by a finite-memory controller in Section 2. The following result formalizes the recursive indistinguishability imposed by any finite memory state. Lemma B.2 (Finite memory induces a right-congruent history quotient). For histories H, H ′ ending at the same observation o, define H ∼M H ′ if and only if F (H) = F (H ′ ). If H ∼M H ′ , then
Reward-aware histories and transition outputs. main text uses observation–action histories
The
Ht = (o0 , a0 , o1 , . . . , at−1 , ot ) as the controller’s within-episode information. If the reward observed after action at is also allowed to influence the next action, the appropriate history is HtY = (o0 , a0 , Y0 , o1 , . . . , at−1 , Yt−1 , ot ). A reward-aware finite-memory controller therefore uses an update of the form UY : M × O × A × Y × O → M, with mt+1 = UY (mt , ot , at , Yt , ot+1 ), where Y is the reward-observation space. In the deterministic-reward model, Yt = r(st , at ). It is tempting to define an augmented observation by ōt+1 = (ot+1 , Yt ). However, ōt+1 is generally not a deterministic function of st+1 , because Yt depends on the preceding state–action pair (st , at ). Therefore this notation does not, by itself, define another deterministic-observation POMDP on the original latent state space. One exact finite-state reformulation augments the latent state with the preceding state–action pair. Let R := r(S × A) be the finite reward range and let ⊥ be a distinguished initial symbol. Define S̄ = S × {⊥} ∪ (S × A) and Ō = O × {⊥} ∪ R . The initial augmented state is (s0 , ⊥) with distribution ρ̄0 (s, ⊥) = ρ0 (s). For an augmented state (s, η) and action a, sample s′ ∼ P (· | s, a) and set the next augmented state to (s′ , (s, a)). Equivalently,
πM (· | o, F (H)) = πM (· | o, F (H ′ ))
P̄ (s′ , (s, a)) | (s, η), a = P (s′ | s, a),
′
and, for every common observable extension (a, o ), H · (a, o′ ) ∼M H ′ · (a, o′ ).
with zero probability assigned to every other next augmented state. Define the deterministic observation map by
Hence equivalence is preserved under every finite common observable continuation. For each terminal observation, the quotient has at most |M| classes, and jointly over terminal observations it has at most |O||M| classes.
h̄(s, ⊥) = (h(s), ⊥) and
h̄(s′ , (s, a)) = h(s′ ), r(s, a) .
Table 2: Guarantee-level comparison of memory and state representations for control under partial observability. Approach
Obs. Exact Minimal Order Unknown Structural Main requirement aliasing Markov state finite state sensitive initial mode recovery
Belief-state planning
✓
✓
×
✓
✓
×
Recurrent or Transformer memory
✓
×
×
✓
(✓)
×
architecture and training data
Linear-recurrent or SSM memory
✓
×
×
✓
(✓)
×
architecture and training data
Bisimulation or homomorphisms
×
✓
(✓)
–
–
(✓)
an available Markov state
Rich-observation state discovery
×
✓
✓
–
–
✓
decodable observations
PSRs or revealing POMDPs
✓
✓
×
✓
✓
✓
rank or revealing conditions
Reward machines
(✓)
(✓)
×
✓
×
(✓)
event-labeling interface
RDP learning
✓
✓
✓
✓
×
✓
finite regularity and coverage
HMRL (ours)
✓
✓
✓
✓
✓
✓
resettable calibrated diagnostics
known model and initial belief
At decision time t ≥ 1, the augmented observation contains the current visible observation ot and the reward rt−1 produced by the preceding transition. Thus this augmented POMDP generates exactly the same reward-aware information sequence as HtY .
Stochastic emissions. A stochastic-observation POMDP replaces the deterministic map
For stochastic reward observations, it is usually cleaner to use a transition-output kernel. Let G(dy | s, a, s′ ) denote the conditional law of the observed reward output after the transition from s to s′ under action a. One step then consists of st+1 ∼ P (· | st , at ), Yt ∼ G(· | st , at , st+1 ), ot+1 = h(st+1 ). The controller updates its memory from the complete observable transition (ot , at , Yt , ot+1 ).
Z(· | s) ∈ ∆(O)
The distinction between mean-reward abstraction and reward-observation abstraction is important. If reward samples are used only to estimate expected return, an exact quotient need only preserve the conditional mean reward. If reward samples are themselves used as online observations for subsequent control, then two states may be merged only if they induce the same relevant reward-output law. More generally, the quotient must preserve the joint conditional law of (Yt , ot+1 , ct+1 ) given the current quotient state and action. Equality of conditional means alone is insufficient when the realized reward can reveal additional hidden-state information. Latent augmentation for deterministic observations. The next result records the analyst’s Markov augmentation for a fixed finite-memory controller. It is an evaluation construction and does not grant the controller access to the latent state. Lemma B.3 (Latent augmented Markov property). For a fixed controller (M, ι, U, πM ), define an action-conditioned process on S × M by Pe((s′ , m′ ) | (s, m), a) = P (s′ | s, a)1 {m′ = U (m, h(s), a, h(s′ ))} , with reward and initial distribution re((s, m), a) = r(s, a), ρe0 (s, m) = ρ0 (s)1{m = ι(h(s))}. Under π e(a | s, m) = πM (a | h(s), m), the process (st , mt ) is Markov and has exactly the same expected discounted return as the original finitememory controller.
h:S→O by an emission kernel
and is written as P = (S, O, A, P, r, Z, γ, ρ0 ). The deterministic fiber So is replaced by the support fiber SoZ := {s ∈ S : Z(o | s) > 0}. The observation support graph has edge set EZ := {(o, a, o′ ) ∈ O × A × O : ∃ s, s′ ∈ S such thatZ(o | s) > 0, P (s′ | s, a) > 0, Z(o′ | s′ ) > 0}. As in the deterministic case, this graph records local visible support and may contain composable paths that admit no globally consistent latent realization. There is one important change to the latent augmentation. Because the current observation ot is no longer determined by st , the Markov analysis state for a finite-memory controller is (st , ot , mt ), rather than merely (st , mt ). Given (s, o, m) and action a, its action-conditioned transition ker nel is Pb (s′ , o′ , m′ ) | (s, o, m), a = P (s′ | s, a)Z(o′ | s′ )1 {m′ = U (m, o, a, o′ )} . The corresponding initial distribution is ρb0 (s, o, m) = ρ0 (s)Z(o | s)1 {m = ι(o)} . Thus the conceptual separation remains unchanged: the analyst can construct a latent Markov augmentation, while the controller has access only to the observable coordinates (o, m). Bellman evaluation for a fixed finite-memory controller. We now record the Bellman equation corresponding to Lemma B.3. The result evaluates a fixed implementable controller; it does not optimize over policies that observe the latent state. Lemma B.4 (Bellman evaluation for a fixed finite-memory controller). Fix a finite-memory controller (M, ι, U, πM )
in the deterministic-observation POMDP of Section 2. Define the operator TπM ,U on bounded functions V :S ×M→R
Corollary B.9 (Finite-family uniform concentration). Let {µ̂α t : α ∈ It } be a finite family of empirical estimators indexed by a finite set It . Suppose that for each fixed α ∈ It , α Pr(∥µ̂α t − µ ∥∞ ≥ εt ) ≤ δt .
" by (TπM ,U V )(s, m) =
P
a∈A πM (a | h(s), m)
r(s, a) + # P ′ ′ ′ γ s′ ∈S P (s | s, a)V s , U (m, h(s), a, h(s )) . Then TπM ,U is a γ-contraction under the sup norm and has a unique fixed point VπM ,U . The expected discounted return from the original initial distribution is X J(πM , U, ι) = ρ0 (s0 )VπM ,U s0 , ι(h(s0 )) . s0 ∈S
Remark B.5 (Why latent-state Bellman optimality is an oracle problem). A pointwise Bellman optimality operator on S × M would choose arg max Q((s, m), a) a∈A
Then α Pr(∃ α ∈ It : ∥µ̂α t − µ ∥∞ ≥ εt ) ≤ |It |δt . P In particular, if t≥1 |It |δt < ∞, then the corresponding uniform failure event occurs only finitely often almost surely. Lemma B.10 (Summable failure implies eventual correctness). Let (Et )t≥1 be a sequence of events. If X Pr(Et ) < ∞, t≥1
then Pr(Et i.o.) = 0. In particular, if at round t a finite identification procedure P fails with probability at most δt and t δt < ∞, then only finitely many rounds are incorrect almost surely.
B.1
Finite-sample and eventual diagnostic consequences
separately for each latent state s. Such an operator permits different actions at two latent states s and s̃ satisfying h(s) = h(s̃) and having the same memory value m. The resulting policy observes the latent state and is therefore not, in general, implementable by a finite-memory controller whose action must be a function only of (h(s), m). For a fixed memory update U , admissible policy optimization remains coupled across all latent states with the same observable pair (o, m). It is not equivalent to unrestricted statewise Bellman maximization on S×M. The quotient theory in the main text resolves this obstruction by constructing an observation–memory state on which rewards and future class dynamics are independent of the hidden representative. The concentration tools used later to convert finite-sample diagnostic guarantees into eventual almost-sure correctness are recorded next. Definition B.6 (σ 2 -sub-Gaussian random variable). A random variable ξ is called σ 2 -sub-Gaussian if E exp λ(ξ − E[ξ]) ≤ exp(λ2 σ 2 /2) for all λ ∈ R.
The exponential classification bound in Theorem 4.6 yields the following fixed-confidence and almost-sure consequences. Corollary B.11 (Diagnostic sample complexity). For any non-singleton fiber Co and any δ ∈ (0, 1), it suffices that 128ν 2 do 2do m≥ log ∆2 δ to guarantee
Lemma B.7 (Empirical mean concentration). Let ξ1 , . . . , ξm be i.i.d. σ 2 -sub-Gaussian random variables with mean µ. Then for any ε > 0, ! m 1 X mε2 Pr ξi − µ ≥ ε ≤ 2 exp − 2 . m i=1 2σ
Then
Corollary B.8 (Coordinate-wise vector concentration). Let X1 , . . . , Xm ∈ Rd be i.i.d. random vectors with mean µ ∈ Rd , and assume that each coordinate (Xi )j is σ 2 -subGaussian. Then for every ε > 0, ! m 1 X mε2 Pr Xi − µ ≥ ε ≤ 2d exp − 2 . m i=1 2σ ∞
Pr(λo (b c) ̸= [i]o | sstart = (o, i)) ≤ δ. For a singleton fiber, the error probability is zero without diagnostic sampling. Corollary B.12 (Summable schedules imply eventual local correctness). Fix a non-singleton fiber o ∈ O+ . At successive classification events k = 2, 3, . . . for o, suppose that, for some η > 0, mo,k ≥
128ν 2 do (1 + η) log k + log(2do ) . ∆2
Pr(λo (b co,k ) ̸= co,k ) ≤ k −(1+η) . Hence the error probabilities are summable, so the first Borel–Cantelli lemma implies that almost surely only finitely many classification errors occur at o. If every required observation follows such a schedule, then the finiteness of O implies almost surely only finitely many classification errors system-wide.
C
Deferred structural and identification statements
This appendix states the complete technical conditions and secondary results summarized in Sections 3 and 4.
C.1
Stable-quotient details
Assumption C.1 (Bounded mean rewards and reward noise). There exist constants rmax > 0 and σY > 0 such that |R(o, i, a)| ≤ rmax for all (o, i, a) ∈ O × [n] × A. For statistical identification, the observed reward is Yt = R(ot , it , at ) + ξt . Let Gt− be the sigma-algebra generated by the latent and observable histories, every algorithmic state measurable from that history, and the current action, immediately before observing (Yt , ot+1 ). Then, for every λ ∈ R, 2 2 λ σY E[ξt | Gt− ] = 0, E eλξt | Gt− ≤ exp . 2 Corollary C.2 (Path transport descends to stable classes). For a directed path ω = e1 · · · ek from o to o′ , define τω := τek ◦ · · · ◦ τe1 . Then τω ([i]o ) = [σω (i)]o′
Assumption C.8 (Resettable diagnostic repeatability). For every admissible (o, i, u) and m ≥ 1, the interface can restore the same latent start state and produce conditionally independent repetitions W (1) (o, i; u), . . . , W (m) (o, i; u) with the conditional law of W (o, i; u). The value of i need not be revealed. Assumption C.9 (Uniform sub-Gaussian diagnostic noise). There exists ν > 0 such that, for every admissible (o, i, u), repetition s, and η ∈ R, E exp η W (s) (o, i; u) − G(o, i; u) sstart = (o, i) ≤ exp
η2 ν 2 2
Assumption C.10 (Locally calibrated class prototypes). For every o ∈ O+ , there are a finite local label set Cbo , a bijection λo : Cbo → Co , and prototypes satisfying
for every i ∈ [n]. Lemma C.3 (Exact quotient-class memory update). If the current class ct ∈ Cot is known, then the update ct+1 := τot ,at ,ot+1 (ct ) makes (ot , ct ) evolve according to the quotient MDP of Theorem 3.9. Hence a correctly initialized or inferred class is a recursively maintainable finite memory state.
C.2
Diagnostic assumptions and quotient descriptions
Definition C.4 (Stable quotient description). The stable quotient description is Q = {Co }o∈O , PO , R̄, {τe }e∈E , ρ̄0 , where Co = [n]/Π⋆o , PO is the observation kernel, R̄(o, c, a) is the class-level mean reward, τe : Co → Co′ for every a feasible edge e = (o − → o′ ), and X ρ̄0 (o, c) = ρ0 (o, i). i:[i]o =c
Assumption C.5 (Class-stable diagnostic means). For every o ∈ O, u ∈ Uprobe (o), and i, j ∈ [n] with [i]o = [j]o , G(o, i; u) = G(o, j; u). Definition C.6 (Class fingerprint). For Uprobe (o) = {u1 , . . . , udo }, define d o ϕo (c) := Ḡ(o, c; uℓ ) ℓ=1 ∈ Rdo .
.
max ϕeo (ĉ) − ϕo λo (ĉ) ĉ∈Cbo
≤ 2
∆ . 8
For o ∈ / O+ , set Cbo to a singleton and let λo be its unique bijection to Co ; no prototype is required. Assumption C.11 (Coverage of edge–class pairs). For every a feasible e = (o − → o′ ) and c ∈ Co , the protocol produces infinitely many transition-anchored samples with true source class c and realized edge e.
D
Directed transport and topological holonomy
The main theory is formulated using executable directed paths. This is the appropriate control-theoretic object because a formal reverse traversal of an observation edge need not correspond to an action that the environment can execute. A representation of the fundamental group of an undirected graph requires additional inverse structure. This appendix states that structure explicitly and shows how both raw-layer and quotient-layer topological holonomy follow from it. Let X = (O, E) be the directed action-labeled support graph of an HCDP. For each e ∈ E, write src(e) ∈ O, tgt(e) ∈ O for its source and target, and let σe ∈ Sn
Singleton fibers require no diagnostic coordinate; otherwise do ≥ 1.
be its raw layer transport. Assumption D.1 (Paired inverse-consistent directed support). The directed edge set E is equipped with an involution
Assumption C.7 (Fingerprint separability). Each ϕo is injective, and, omitting singleton fibers,
e 7−→ ē
∆ :=
min
o∈O, c,c′ ∈Co c̸=c′
∥ϕo (c) − ϕo (c′ )∥2 > 0.
such that, for every e ∈ E, ē = e,
src(ē) = tgt(e),
tgt(ē) = src(e),
For an oriented loop word
and
σē = σe−1 . The edges e and ē may carry different action labels; the notation ē denotes an inverse edge, not an inverse element of the action set. Under Assumption D.1, define Xund to be the undirected multigraph containing one undirected edge for each pair {e, ē}. An oriented traversal of this undirected edge is represented by either e or ē. Immediate backtracking has identity transport because σē ◦ σe = Id and σe ◦ σē = Id. A based loop in |Xund | may be represented by an oriented edge word w = e1 e2 · · · ek . Reduction of the word removes adjacent pairs of the form eē
or
ēe.
Define its raw transport by σw = σek ◦ · · · ◦ σe1 . Proposition D.2 (Raw topological holonomy). Under Assumption D.1, for every base observation o ∈ O, the assignment Holoraw : π1 (|Xund |, o) → Sn , o
Holoraw o ([w]) = σw ,
is a well-defined group homomorphism. Its image π1 (|Xund |, o) ≤ Sn Graw := Holoraw o o is the raw topological holonomy group at o. We next pass from raw layers to the stable quotient classes Co = [n]/Π⋆o . For every directed edge e : o → o′ , let τe : Co → Co′ be the quotient transport defined in Theorem 3.9. Lemma D.3 (Inverse consistency descends to the stable quotient). Under Assumption D.1, the quotient transports along paired edges satisfy τē ◦ τe = IdCsrc(e) and τe ◦ τē = IdCtgt(e) . Consequently,
τē = τe−1 , and every quotient transport along an edge of Xund is a bijection.
w = e1 e2 · · · ek based at o, define τw = τek ◦ · · · ◦ τe1 : Co → Co . Proposition D.4 (Quotient topological holonomy). Under Assumption D.1, for every base observation o ∈ O, the assignment Holoquot : π1 (|Xund |, o) → o Sym(Co ), Holoquot ([w]) = τ , is a well-defined group w o homomorphism. Its image π1 (|Xund |, o) ≤ Sym(Co ) Gquot := Holoquot o o is the quotient topological holonomy group at o. Proposition D.5 (Change of base observation). Assume Assumption D.1. Let o, o′ ∈ O lie in the same connected component of Xund , and let η be an oriented path from o to o′ . Then raw −1 Graw o′ = ση Go ση and Gquot = τη Gquot τη−1 . o o′ Thus changing the base observation changes the holonomy group only by conjugation through the transport along the chosen reference path. The directed theory in the main text does not require Assumption D.1. Without paired inverse edges, executable directed closed walks still induce the directed quotient holonomy monoid—and, in the finite closed-walk setting of Lemma 3.10, a finite permutation group—but the transport cannot in general be interpreted as a representation of the fundamental group of an undirected graph. The topological formulation is therefore a strengthened special case of the directed control-theoretic formulation, not a prerequisite for the stable quotient or for exact Markovization.
D.1
Optional tree-based gauge fixing
Observation-wise local coordinates are already sufficient for exact control. A spanning-tree synchronization step is useful only when one wants to express several observation fibers relative to one chosen root coordinate system, for example when visualizing transport matrices or comparing loop compositions. Proposition D.6 (Optional gauge fixing on a bijective tree scaffold). Fix a connected component of the underlying undirected observation graph and choose a root observation oroot . Suppose there exists an undirected spanning tree T such that, for each tree adjacency {o, o′ }, one has selected a feasible directed edge in one of the two directions whose quotient transport is bijective. After the corresponding local edge maps have been recovered exactly, an arbitrary labeling of the root class set can be propagated uniquely along the tree, using a recovered bijection when traversing it in its selected direction and its inverse when traversing it in the opposite direction. This produces a root-relative coordinate convention at every observation.
The resulting convention is unique relative to the chosen root labeling, the selected directed tree edges, and the tree T . Every off-tree edge can then be expressed in the root-relative coordinates and yields a cycle-consistency or holonomy diagnostic. Proposition D.6 requires bijective tree transports and therefore may fail when neighboring observations have different numbers of stable classes. This does not obstruct exact quotient control, because Definition 5.1 already provides the required global state space in observation-wise local coordinates, while Lemma E.2 establishes policy and value transfer between local and canonical coordinates. Tree synchronization is an optional gauge choice, not an identifiability requirement.
E
Algorithmic templates and implementation details
This appendix gives concrete wrappers for the two regimes in Section 5. The diagnostic wrapper uses resettable class fingerprints and transition-anchored transport calibration. The passive wrapper uses a history encoder and predictive consistency but carries no unconditional quotient-recovery guarantee. Both algorithms use observation-wise local states (o, b c), rather than unknown canonical labels. Observationwise gauge maps appear only in the correctness proofs.
E.1
Diagnostic quotient-lifted reinforcement learning
The diagnostic interface provides the locally calibrated label sets and prototypes from Assumption C.10. HMRL-D first runs a diagnostic calibration phase that estimates every required edge–class row. It then freezes the recovered maps, reinitializes the RL backbone, diagnoses each episode’s initial class, and propagates that class between resets. This separation implements Corollary 5.4. Algorithm 1 never uses the unknown proof-level gauges λo . Because the displayed wrapper calibrates first and restarts the backbone, its online state stream is isomorphic to the canonical quotient-state stream on the calibration-success event of Corollary E.3. An interleaved implementation is also possible, but then a backbone theorem applies only when it is explicitly robust to a finite corrupted prefix, as stated in Corollary 5.4. The finite pre-identification portion of the RL updates may contain non-Markov state labels. A backbone theorem applies directly only if the RL component is restarted after a certified diagnostic phase or if its convergence theorem is robust to a finite number of incorrect early updates, as stated in Corollary 5.4.
E.2
Passive quotient-lifted reinforcement learning
The passive regime has no latent checkpoint, no calibrated fingerprints, and no oracle class count. Its local labels must therefore be treated as learned representation variables rather than identified quotient classes. Under the main information convention of this paper, the online encoder uses the observation–action history Ht = (o0 , a0 , o1 , . . . , at−1 , ot ).
Reward samples may be used as training targets but are not fed back as within-episode observations. A reward-aware variant may instead use HtY , but then the stronger rewardoutput abstraction described in Appendix B is required. Definition E.1 (Passive local-memory learner). A passive local-memory learner maintains: 1. adaptive local label sets {Cbo }o∈O ; 2. a history encoder qψ (b ct | Ht ),
b ct ∈ Cbot ;
3. a class-conditioned mean-reward model bθ (o, b R c, a); 4. a base observation model PbO,θ (o′ | o, a), which is not conditioned on b c; 5. a local edge-transition model Tbθ (b c′ | o, b c, a, o′ ); 6. a split–merge or compression mechanism for adapting the local label sets. A generic passive objective may be written as L = Lrew + λbase Lbase + λedge Ledge + λord Lord + λcomp Lcomp . Here: • Lrew measures class-conditioned reward-prediction error; • Lbase estimates the class-independent kernel PO ; • Ledge encourages deterministic consistency of inferred labels across each realized edge; • Lord compares ordered compositions of learned edge maps on observed paths and loops; • Lcomp discourages unnecessary local classes. The ordered-composition loss must preserve temporal order. Replacing it by a commutative edge-count loss can erase the nonabelian distinctions studied in Section 6. Algorithm 2 does not call a synchronization routine. Because the observation is part of the state, the local label sets already define one global disjoint-union state space. The algorithm also carries no unconditional identification guarantee. Its exact interpretation is the conditional one in Theorem E.4.
E.3
Auxiliary quotient-lifting results
The following results formalize policy/value invariance under observation-wise gauge changes and the conditional exact-reduction claim used by the passive learner. Lemma E.2 (Policy and value transfer under local gauges). b and define Let π b be a stationary policy on S, π̄(a | o, c) := π b a | o, λ−1 o (c) . b Then, for every (o, ĉ) ∈ S, Vbπb (o, ĉ) = V̄π̄ o, λo (ĉ) ,
Vb ⋆ (o, ĉ) = V̄ ⋆ o, λo (ĉ) .
Hence π b is optimal if and only if its gauge-transformed policy π̄ is optimal.
Corollary E.3 (Finite-sample calibration). Assume O+ ̸= ∅; if every fiber is a singleton, calibration is trivial. Fix δproto , δcls ∈ (0, 1), and suppose the prototype-accuracy condition of Assumption C.10 holds on an event Eproto with P(Eproto ) ≥ 1 − δproto . Let K be the total number of nontrivial diagnostic classifications performed up to the backbone’s horizon, counting each initial-class inference and both members of every transitionanchored source–target pair. Suppose the calibration count tables start at zero. Conditional on Eproto , if every classification uses 128ν 2 d 2dK m≥ log , d := max do , o∈O+ ∆2 δcls repetitions per coordinate, and every feasible edge–class pair receives at least one anchored sample during calibration, then, with probability at least 1 − δproto − δcls , the prototypes and every classification are correct. On that event, every recovered map satisfies τbe = λ−1 tgt(e) ◦ τe ◦ λsrc(e) ,
for every t ≥ T0
Explicit nonabelian construction
This appendix instantiates Proposition 6.7 and makes the executable commutator calculation explicit. The construction uses one visible observation, two executable actions, and three hidden layers. It therefore shows that the nonabelian memory barrier does not require a complicated observation graph. Corollary F.1 (Barrier for count-additive loop encoders). Let Nj (w) := (ab(w))j denote the signed exponent sum of generator xj in w. If an encoder computes A(w) =
r X
Nj (w)vj
j=1
and its readout depends only on A(w), then its memory factors through ab : Fr → Zr . Therefore Theorem 6.4 applies whenever its decision-separation condition holds. Let O = {o}, A = {a, b}, [n] = {1, 2, 3}, with PO (o | o, a) = PO (o | o, b) = 1. Assign
every episode is initialized and tracked exactly, and the synchronization event of Theorem 5.2 holds at every postcalibration time. Consequently, any finite-MDP guarantee for a backbone restarted after calibration holds with its failure probability increased by at most δproto + δcls , provided that guarantee applies to the post-restart initial-state law. Theorem E.4 (Conditional quotient reduction in the passive regime). Suppose a passive learner produces b ct ∈ Cbot , and let zbt = (ot , b ct ). Assume that there exist fixed bijections λo : Cbo → Co and an almost surely finite stopping time T0 for the ambient pre-transition filtration (Gt− )t≥0 such that λot (b ct ) = ct
F
α := σa = (12) and β := σb = (123). The generated transport group is ⟨α, β⟩ = S3 . The inverse transports are executable using the same two actions: α−1 = α, β −1 = β 2 . Thus no additional inverse action is required. Under the function-composition convention of Section 6, the commutator is
almost surely.
[α, β] = αβα−1 β −1 .
′
b and every Then, for every deterministic t ≥ 0, every ẑ ∈ S, η ∈ R, on the event {T0 ≤ t}, P zbt+1 = ẑ ′ | Gt− = Pbλ (ẑ ′ | zbt , at ) , bλ (b E[Yt | Gt− ] = R zt , at ),
A direct calculation gives [α, β] = (123). An executable positive action sequence implementing this transport is obtained by replacing α−1
and 2 2 h i η σY bλ (b . E exp η Yt − R zt , at ) Gt− ≤ exp 2 Moreover, Λλ (o, ĉ) = (o, λo (ĉ)) is an MDP isomorphism from the local-coordinate quotient MDP to the canonical stable-quotient MDP. If T0 is also a stopping time for the learned-information filtration (Fbt− )t≥0 , then the same transition, conditional-reward, and conditional sub-Gaussian conclusions hold relative to Fbt− after T0 . Without these stopping-time and eventual class-correctness conditions, pathwise eventual agreement alone does not imply that the passive learned state is Markov or value preserving.
with
α
and
β −1 with β 2 . Reading temporal execution from right to left under the composition convention, one such chronological action sequence is b, b, a, b, a. Its total transport is α ◦ β ◦ α ◦ β ◦ β = [α, β]. In particular, [α, β](1) = 2.
Choose the reward table layer 1 2 3
a 1 0 0
b 0 1 0
and let
1 . 2 The three immediate reward vectors are distinct: 0<γ<
(1, 0),
(0, 1),
(0, 0).
Therefore the initial reward partition is already the discrete partition, and the stable quotient consists of the three singleton classes {1}, {2}, {3}. At class 1, action a is uniquely optimal. Indeed, both actions send layer 1 to layer 2, so their continuation values are identical while R(o, 1, a) − R(o, 1, b) = 1. At class 2, action b is uniquely optimal for 1 . 2 The immediate advantage of b is 1, while the largest possible discounted continuation-value disadvantage is at most γ < 1. 1−γ γ<
Hence
Q̄⋆ (o, {2}, b) > Q̄⋆ (o, {2}, a). The empty word leaves class 1 unchanged, whereas the commutator sends it to class 2: ρo (e)({1}) = {1}, ρo ([xa , xb ])({1}) = {2}. However, ab(e) = ab([xa , xb ]) = 0. Thus every memory surrogate that factors through abelianization assigns the same memory value to these two histories, even though their unique optimal actions are different. This verifies the hypotheses of Corollary 6.5.
G
Scope remarks on extensions
This appendix records brief scope remarks beyond the finite deterministic HCDP framework studied in the main text. These remarks clarify conceptual reach but do not enlarge the theorem claims of the paper. Remark G.1 (Operator-valued transport). A natural extension replaces edge permutations by stochastic transport kernels. In that setting, path transport is operator-valued and loop effects generate an operator semigroup rather than, in general, a permutation group. The quotient perspective remains meaningful, but both identification and exact recovery become substantially harder because edge recovery is no longer combinatorial.
Remark G.2 (Continuous observation spaces). A second extension replaces the finite observation graph by a finiteresolution skeleton extracted from a continuous observation space. At each fixed resolution one obtains a finite transport-control model to which the structural quotient theory applies. A genuine continuum theory would additionally require discretization-stability, refinement-consistency, and sample-complexity arguments, which are outside the scope of the present paper. Remark G.3 (Conceptual continuity). Both extensions preserve the same conceptual separation as the main text: history acts through path transport, control-relevant memory is obtained by quotienting predictive redundancies, and reinforcement learning becomes standard once the correct augmented state is available.
H
Experimental Details and Additional Results
This appendix gives the complete controlled environments, diagnostic protocol, baselines, metrics, hyperparameters, and measured results for Section 7. The experiments are designed to validate finite consequences of the theory under known HCDP assumptions. They do not claim that generic public POMDP benchmarks possess a finite exact holonomy quotient.
H.1
Experimental Claim Boundary
The experiments test four claims. 1. Exact quotient construction. Stable refinement should recover the known quotient of ChainCover, preserve the exact optimal value, and reject every attempted one-pair strict coarsening. 2. Diagnostic identification. Repeated class-stable probes should reduce local class error, and transition-anchored source–target diagnostics should recover deterministic quotient transports under the simulator’s fixed local class labeling. 3. Nonabelian order dependence. A memory based only on commutative loop counts should fail on equal-count histories whose ordered transport differs. 4. Finite quotient control. Once calibrated, HMRL-D should approach the quotient oracle with a fixed threestate class memory, rather than storing the raw nuisance coordinate or the complete loop word. The experiments do not attempt to prove the theorems. They test implementation-level consequences that would be false if the partition, diagnostic, transport, or quotient-lifting procedures were implemented incorrectly.
H.2
Controlled HCDP Constructions
Both environment families use a raw hidden layer (c, u) ∈ [3] × [b], which is identified with the layer set [3b] through any fixed bijection. The coordinate c ∈ [3] is control relevant, whereas u ∈ [b] is a nuisance coordinate. Rewards depend on c but never on u. Define α = (12), β = (123)
as permutations of [3]. For the nuisance coordinate, define the cyclic permutations ν+ (u) = 1 + (u mod b) ν− (u) = 1 + ((u − 2) mod b). When b = 1, both nuisance transports reduce to the identity. ChainCover For a depth parameter D ≥ 1, ChainCover has observation space Ochain = {o0 , o1 , . . . , oD , q, †} and action space A = {g1 , g2 , g3 }. At every chain observation od , all actions have zero reward and induce the same visible transition. Specifically, d = 0, . . . , D − 1,
and PO (q | oD , a) = 1. The class transport alternates between α and β: α, d even, πd = β, d odd. The full raw transport is σod ,a,od+1 (c, u) =
(πd (c), ν+ (u)), d < D, (πD (c), ν− (u)), d = D,
where the second line denotes the edge from oD to q. At the query observation q, action gj gives reward R(q, (c, u), gj ) = 1{c = j}, and the process transitions to the terminal observation:
Oloop = {h, xα , xβ , q, †} A = {g1 , g2 , g3 }. The observation h is the common loop base. For every action a, PO (q | h, a) = pquery , 1 − pquery PO (xα | h, a) = , 2 and 1 − pquery PO (xβ | h, a) = . 2 All three outgoing transports from h are the identity. The two excursion observations return deterministically to h: PO (h | xα , a) = 1, PO (h | xβ , a) = 1. Their raw transports are σxα ,a,h (c, u) = (α(c), ν+ (u)) and σxβ ,a,h (c, u) = (β(c), ν− (u)). Thus the visible excursions h → xα → h and h → xβ → h are closed walks based at h whose quotient transports are α and β, respectively. At the query observation,
PO († | q, gj ) = 1. The query-to-terminal transport is the identity. At †, every action has zero reward and the process self-loops with identity transport. The stable quotient is known analytically. At every observation o ∈ {o0 , . . . , oD , q}, the stable classes are Co(j) = {(j, u) : u ∈ [b]},
j = 1, 2, 3.
At †, all raw layers form one class. Hence
R(q, (c, u), gj ) = 1{c = j}, and PO († | q, gj ) = 1. All other rewards are zero. The initial distribution is 1 ρ0 (h, (1, u)) = , u ∈ [b]. b At each of h, xα , xβ , q, the stable quotient contains the three classes
|Sraw | = 3b(D + 3)
Co(j) = {(j, u) : u ∈ [b]}.
and |S̄| = 3(D + 2) + 1. The reward partition separates the three classes at q immediately. One additional application of the stability operator propagates this distinction backward by one chain edge. Therefore the ground-truth stabilization depth is L⋆ = D + 1. The primary audit uses
At †, all layers merge. Consequently, |Sraw | = 15b and |S̄| = 13. The primary control setting uses b = 16,
D = 6,
b = 8. |S̄| = 25,
pquery = 0.1.
The expected number of completed loops before a query is
Its exact ground-truth counts are |Sraw | = 216,
LoopGuess has observation space
and action space
and
PO (od+1 | od , a) = 1,
LoopGuess
L⋆ = 7.
1 − pquery = 9. pquery
Table 3: Exact ChainCover structural audit. Measured quantities are obtained by exhaustive enumeration and agree with the analytical construction. Metric
(1 − pquery )48 = 0.948 ≈ 6.36 × 10−3 .
Exact Stable-Quotient Audit
The exact audit enumerates every raw state, action, and feasible visible successor. Starting from the reward partition, it repeatedly applies
until no partition changes. b define For an inferred partition family Π,
Define the successor inconsistency by n o b b Π b = max 1 [σe (i)]Π ϵT (Π) o′ ̸= [σe (j)]o′ . ′ e:o→o , b i≡Π oj
The quotient reward and transition kernel are constructed b and value iteration is run on both the raw latent MDP from Π, and quotient MDP until the sup-norm Bellman residual is below 10−12 . The value-preservation metric is
⋆ ∆V = max Vfull (o, i) − V̄ ⋆ o, [i]Π o b
o,i
.
216 25 8.64× 7 0 0 0 24/24 3.00
1
1
1
1
1
1
1
3
o1
1
1
1
1
1
1
3
3
o2
1
1
1
1
1
3
3
3
2.50
o3
1
1
1
1
3
3
3
3
2.25
o4
1
1
1
3
3
3
3
3
2.00
o5
1
1
3
3
3
3
3
3
1.75
o6
1
3
3
3
3
3
3
3
q
3
3
3
3
3
3
3
3
terminal
1
1
1
1
1
1
1
1
0
1
2
3
4
5
6
7
b = max |R(o, i, a) − R(o, j, a)| . ϵR (Π) o,a, b i≡Π oj
216 25 8.64× 7 0 0 0 24/24
o0
Observation
Π(t+1) = T (Π(t) )
Measured
Raw latent states Stable quotient states Compression ratio Stabilization round Reward inconsistency ϵR Transition inconsistency ϵT Optimal-value gap ∆V Rejected strict coarsenings
The measured primary cap frequency is 0.0058 ± 0.0010, which is consistent with this probability. At pquery = 0.05, the measured cap frequency increases to 0.085±0.010; therefore that sensitivity setting should be interpreted as the configured capped process rather than as an effectively uncapped geometric process.
H.3
Analytical
2.75
1.50
Inferred classes
For numerical safety, an episode is forced to query after at most 48 completed loops. For the primary setting pquery = 0.1, the probability that an uncapped geometric episode would reach this limit is
1.25 1.00
Refinement round
Figure 3: Backward propagation of stable class distinctions in ChainCover. The reward partition separates the three classes at the query observation in round 0. Each subsequent refinement round propagates this distinction backward by one chain position, and the partition stabilizes at round 7, as predicted for depth D = 6. For each trial, the evaluator {1, 2, 4, 8, 16, 32, 64} independent applies nearest-prototype decoding:
averages m ∈ observations and
m
To audit coarseness, for every nonterminal observation and every unordered pair of distinct stable classes, we form the partition obtained by merging exactly that pair while leaving every other block unchanged. A strict coarsening is rejected if ϵR > 0 or ϵT > 0.
H.4
Diagnostic Fingerprints
The standalone recovery experiment assigns the three classes the fixed prototype vectors v1 = (0.85, 0, 0.35), v2 = (−0.42, 0.74, −0.20), v3 = (−0.42, −0.74, −0.20). Their minimum pairwise Euclidean separation is ∆ = min ∥vi − vj ∥2 = 1.48. i̸=j
A diagnostic observation from class c is Y = vc + ξ,
2 ξ ∼ N (0, σdiag I3 ).
b c = arg min j∈[3]
1 X Yℓ − vj m ℓ=1
. 2
The tested noise levels are σdiag ∈ {0.25, 0.50, 1.00}. For every (σdiag , m) configuration, each repeated evaluation uses 500 trials from each of the three classes, giving 1,500 class trials in total. Because the released diagnostic sweep uses a fixed prototype labeling rather than independently permuted local gauges, the reported quantity is direct classification error rather than a separately tested gauge-alignment error.
H.5
Transition-Anchored Transport Recovery
For each generator e ∈ {α, β} and source class c, the target class is τe (c). A transport trial generates m independent noisy observations of the target prototype vτe (c) , averages
them, and applies the same nearest-prototype decoder. The row-wise transport error is Errtransport =
1 6Ne
X
Ne 3 X X
1 {b ce,c,r ̸= τe (c)} ,
Control Baselines
All tabular methods use the same Qlearning update, Qt+1 (zt , at ) = Qt (zt , at ) + ηt [Yt + γ maxa′ Qt (zt+1 , a′ ) − Qt (zt , at )] . Only the state descriptor differs. Obs-Q.
The privileged quotient state is zt = (ot , ct ).
This is the smallest exact class-tracking oracle and supplies the reference control curve.
e∈{α,β} c=1 r=1
where Ne = 250 trials are used for each source-class and generator pair. Thus, each repeated evaluation at a given (σdiag , m) setting contains 6Ne = 1,500 transport trials. At the primary noise level σdiag = 0.5, class and transport errors decrease from approximately 0.11 at m = 1 to below 0.003 at m = 4, and both become zero in the reported trials by m = 8. At the harder noise level σdiag = 1.0, the same monotone pattern is observed, with both errors reaching zero by m = 32.
H.6
Quotient-Q oracle.
The state is the current visible observation:
H.7
Training and Evaluation Protocol
Unless otherwise stated, the primary parameters are: The exploration rate is linearly annealed from 0.90 to 0.03 over the first 1,200 online episodes and remains fixed thereafter. Evaluation uses greedy actions on independently sampled environment trajectories. The standalone recovery sweep in Appendix H.4 and the LoopGuess control calibrator use separate fixed synthetic prototype sets. For LoopGuess control, HMRL-D uses the four-dimensional fingerprints µ1 = (1.25, −0.25, 0.55, −0.90), µ2 = (−0.85, 1.10, −0.20, 0.65), µ3 = (0.15, −0.75, 1.20, 0.35). Each prototype is estimated from 24 repeated observations. Each of the six source rows corresponding to the two generators and three source classes uses 24 independent edge observations. Counting one complete fingerprint observation as one resettable diagnostic interaction gives
zt = ot .
Ncalibration = 3 × 24 + 6 × 24 = 216.
At the query observation, all histories are therefore merged.
After calibration, the Q-table is reinitialized before online training.
Count-Q. The state contains the current observation and the commutative loop-count vector:
The order of completed loops is discarded. This baseline directly instantiates an abelianized loop-memory surrogate. History-Q. word:
H.8
Evaluation Metrics
Query success.
zt = (ot , Nα (t), Nβ (t)) .
Sk = 1 {the selected query action equals the terminal class} . The success rate is
The state contains the complete visible loop zt = (ot , wt ),
Success =
∗
wt ∈ {α, β} .
Because LoopGuess uses a fixed initial quotient class, this representation is exact on the truncated experimental trajectories, but its number of possible states grows exponentially with loop length. Raw-Q oracle.
The privileged state is
This oracle retains the nuisance coordinate and therefore uses up to 3b latent memory values at a fixed nonterminal observation. HMRL-D. HMRL-D first estimates local class prototypes and the two local loop maps τbα , τbβ . After calibration, all Q-tables and optimizer statistics are reinitialized. The initial local class is fixed by the calibrated initial checkpoint, and subsequent classes are propagated as b ct+1 = τbe (b ct ) after each completed loop. The control state is
1 Neval
N eval X
Sk .
k=1
Normalized learning-curve AUC. Let xj be the number of completed online-training episodes at checkpoint j, and let sj be the corresponding evaluation success. The implementation reports the normalized trapezoidal area AUC =
zt = (ot , ct , ut ).
zt = (ot , b ct ).
For an evaluation episode k, let
K−1 X sj + sj+1 1 (xj+1 − xj ) . xK − x0 j=0 2
Because sj ∈ [0, 1], the normalized AUC also lies in [0, 1]. Paired-order accuracy. ( P=
Let
Nα (w) = Nα (w′ ), (w, w ) : Nβ (w) = Nβ (w′ ), ρ(w)(1) ̸= ρ(w′ )(1)
)
′
be a finite evaluation set of same-count, different-transport word pairs. For each word, the environment is rolled forward to the query observation without policy intervention. The method then chooses a query action using the memory produced by that history. Paired-order accuracy is the fraction of individual histories in these pairs on which the correct query action is selected.
Table 4: Diagnostic and transport-recovery errors. Entries are reported as mean ± standard deviation. Each repeated evaluation uses 1,500 class trials and 1,500 transport trials.
Class error
Transport error
m
σ = 0.25
σ = 0.50
σ = 1.00
σ = 0.25
σ = 0.50
σ = 1.00
1 2 4 8 16 32 64
0.0024 ± 0.0010 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000
0.1104 ± 0.0110 0.0287 ± 0.0027 0.0024 ± 0.0010 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000
0.3436 ± 0.0090 0.2198 ± 0.0128 0.1113 ± 0.0052 0.0260 ± 0.0007 0.0018 ± 0.0010 0.0000 ± 0.0000 0.0000 ± 0.0000
0.0029 ± 0.0010 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000
0.1087 ± 0.0131 0.0282 ± 0.0062 0.0018 ± 0.0010 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000
0.3467 ± 0.0127 0.2278 ± 0.0088 0.1127 ± 0.0064 0.0282 ± 0.0034 0.0011 ± 0.0010 0.0000 ± 0.0000 0.0000 ± 0.0000
Table 5: Hyperparameters used for the reported experimental artifacts. Value
ChainCover depth D ChainCover nuisance size LoopGuess nuisance size Query probability Discount factor γ Online training episodes Evaluation interval Evaluation episodes/checkpoint Final evaluation episodes Q-learning rate Initial/final exploration Exploration-decay episodes Maximum completed loops Diagnostic noise Prototype repetitions Edge samples per source row Sweep training episodes Sweep final evaluation episodes
6 8 16 0.10 0.99 3,500 250 500 2,000 0.20 0.90/0.03 1,200 48 0.50 24 24 1,800 1,000
Visited memory states. For each method, we count the number of distinct memory values observed at the query observation during final evaluation. The observation coordinate is held fixed, so this metric measures only the memory alphabet actually used to distinguish hidden conditions at the decision point. Truncation frequency. We report #{episodes forced to query at the loop cap} fcap = . #{all episodes} A non-negligible value means that the reported results characterize the capped process; the cap should be increased only when an uncapped geometric-horizon interpretation is required.
H.9
Complete Control Results
We report two controlled robustness sweeps. The nuisance sweep varies the number of reward-irrelevant latent alternatives while holding the quotient fixed. The query-probability
Raw-Q oracle HMRL-D (ours)
1.0
Final query success
Parameter
Count-Q History-Q
0.9 0.8 0.7 0.6 0.5 0.4 0.3 1
4
16
64
Nuisance multiplicity b
Figure 4: Robustness to nuisance multiplicity. HMRLD preserves perfect final query success while retaining the same three-state decision memory. Raw-Q becomes harder to train at b = 64 because it distinguishes all nuisance values, whereas Count-Q and History-Q remain limited by their respective representation deficiencies.
sweep changes the expected visible-history length before the terminal decision. All entries are reported as mean ± standard deviation. The nuisance sweep confirms that the quotient representation is invariant to an increasing number of control-irrelevant latent alternatives. In contrast, Raw-Q must learn separate values for each nuisance state and falls below perfect success at b = 64 under the fixed sweep budget. The queryprobability sweep provides a direct test of ordered transport: Count-Q remains near chance for all three settings, whereas HMRL-D maintains perfect paired-order accuracy.
I
Proofs
This section collects the proofs in the order in which the corresponding results are used in the main text and the preceding appendices. Each result has a single dedicated proof
Table 6: Final LoopGuess success across nuisance multiplicities. Bold marks the best deployable method; oracle methods are reference rows.
b=1
b=4
b = 16
b = 64
0.352 ± 0.026 0.541 ± 0.016 0.660 ± 0.001 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
0.364 ± 0.026 0.542 ± 0.010 0.656 ± 0.017 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
0.364 ± 0.026 0.542 ± 0.010 0.656 ± 0.017 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
0.364 ± 0.026 0.542 ± 0.010 0.656 ± 0.017 0.967 ± 0.007 1.000 ± 0.000 1.000 ± 0.000
Method Obs-Q Count-Q History-Q Raw-Q oracle HMRL-D (ours) Quotient-Q oracle
Table 7: Paired-order accuracy across query probabilities. A smaller pquery produces longer loop histories on average. Bold marks the best deployable method.
Method
pquery = 0.05
pquery = 0.10
pquery = 0.20
Obs-Q Count-Q History-Q Raw-Q oracle HMRL-D (ours) Quotient-Q oracle
0.325 ± 0.000 0.338 ± 0.006 0.414 ± 0.010 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
0.328 ± 0.005 0.340 ± 0.003 0.448 ± 0.023 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
0.331 ± 0.005 0.342 ± 0.004 0.433 ± 0.008 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
subsection.
I.1
and, similarly,
Proof of Lemma B.2
Proof. Fix an observation o ∈ O, and let H and H ′ be two histories ending at o such that H ∼M H ′ . By the definition of ∼M , this assumption is exactly
F H ′ · (a, o′ ) = U F (H ′ ), o, a, o′ . Substituting F (H) = F (H ′ ) = m into the two identities yields F H · (a, o′ ) = U F (H), o, a, o′
F (H) = F (H ′ ).
= U (m, o, a, o′ )
Let
= U F (H ′ ), o, a, o′ = F H ′ · (a, o′ ) .
m := F (H) = F (H ′ ) ∈ M denote their common memory value. The finite-memory controller selects its action distribution only as a function of the current observation and the current memory state. Therefore, for every action ā ∈ A,
The two extended histories both end at the observation o′ . The defining condition of ∼M therefore applies to them, and the preceding equality implies
πM (ā | o, F (H)) = πM (ā | o, m) = πM (ā | o, F (H ′ )).
H · (a, o′ ) ∼M H ′ · (a, o′ ).
The two distributions consequently assign the same probability to every element of the finite action space A. Hence they are equal as probability distributions in ∆(A), and
The same reasoning shows that equivalence is preserved not only by one observable extension but by every finite common observable continuation. To make this explicit, let ξ = (a0 , o1 ), . . . , (ak−1 , ok )
πM (· | o, F (H)) = πM (· | o, F (H ′ )). Now let (a, o′ ) be any common observable extension of H and H ′ . Because both histories end at the same observation o, the recursive definition of the history encoder gives F H · (a, o′ ) = U F (H), o, a, o′
be a common observable continuation from o0 := o. Define the corresponding history prefixes recursively by H0 := H,
H0′ := H ′ ,
Same-count paired-order accuracy
Obs-Q Count-Q
visited at query
History-Q HMRL-D (ours)
learned Q states
Obs-Q
1.0
Count-Q
0.9 History-Q
0.8 0.7
Raw-Q oracle
0.6
HMRL-D (ours)
0.5
Quotient-Q oracle
0.4
100
101
102
103
Number of states (log scale)
0.3 0.05
0.1
0.2
Query probability pquery
Figure 5: Paired-order robustness across query probabilities. HMRL-D remains exact as the expected visible history becomes longer. Count-Q remains close to the 1/3 chance level because loop counts cannot recover ordering, while History-Q only partially resolves the larger collection of observed words. and, for j = 0, . . . , k − 1, by ′ Hj+1 := Hj′ · (aj , oj+1 ).
Hj+1 := Hj · (aj , oj+1 ),
The histories Hj and Hj′ end at the same observation oj for every j. At the initial prefix, F (H0 ) = F (H) = F (H ′ ) = F (H0′ ). Suppose that, for some j < k, F (Hj ) = F (Hj′ ). Applying the recursive encoder update to the next common extension gives F (Hj+1 ) = U F (Hj ), oj , aj , oj+1 and
′ F (Hj+1 ) = U F (Hj′ ), oj , aj , oj+1 . Since the memory arguments in these two expressions are equal by the induction hypothesis, all four arguments supplied to U are identical. Because U is a deterministic function, F (Hj+1 ) = U F (Hj ), oj , aj , oj+1 = U F (Hj′ ), oj , aj , oj+1 ′ = F (Hj+1 ).
Induction therefore proves that F (Hj ) = F (Hj′ )
for every j = 0, . . . , k.
Thus, after every prefix of the same observable continuation, the two histories remain in the same memory class. In particular, at every corresponding future decision point, they have
Figure 6: Decision-time and learned-state complexity. Filled markers report memory states visited at the query observation, while open markers report the total number of learned Q-states. HMRL-D and Quotient-Q use three states under both measures. Count-Q and especially History-Q create substantially larger learned state spaces. the same terminal observation and the same memory value, so the controller again uses the same action distribution. Finally, for each fixed observation o, the relation ∼M on histories ending at o is an equivalence relation. Reflexivity follows from F (H) = F (H); symmetry follows because F (H) = F (H ′ )
=⇒
F (H ′ ) = F (H);
and transitivity follows because F (H) = F (H ′ ) and F (H ′ ) = F (H ′′ ) =⇒ F (H) = F (H ′′ ). Since every equivalence class is determined by a memory value in the finite set M, there are at most |M| such classes among histories ending at any fixed observation. When histories are indexed jointly by their terminal observation and memory value, the total number of classes is at most |O| |M|. The preservation of these classes under every common observable continuation is exactly the required right-congruence property.
I.2
Proof of Lemma B.3
Proof. Let Zt := (st , mt ) ∈ S × M denote the latent state–memory pair. We first derive its action-conditioned transition law. Fix a time t ≥ 0, a current augmented state (s, m) ∈ S × M, an action a ∈ A, and a candidate next augmented state (s′ , m′ ) ∈ S × M. Conditional on (st , mt ) = (s, m)
and
at = a,
the latent state evolves according to the original transition kernel. Therefore, P (st+1 = s′ | st = s, mt = m, at = a) = P (s′ | s, a).
The presence of mt in the conditioning does not change this probability because, once st and at are fixed, the transition of the latent state is governed by the Markov kernel P (· | st , at ) and is independent of the preceding history and the controller’s internal memory. The memory state does not evolve independently. Since the current and next observations are ot = h(st )
and
ot+1 = h(st+1 ),
We next derive the initial distribution of the augmented process. The original latent state is initialized according to s0 ∼ ρ0 . Its initial observation is o0 = h(s0 ), and the memory state is initialized by m0 = ι(o0 ) = ι(h(s0 )).
respectively, the memory update rule gives mt+1 = U mt , h(st ), at , h(st+1 ) .
Therefore, for every (s, m) ∈ S × M, P(s0 = s, m0 = m) = P s0 = s, ι(h(s0 )) = m
Consequently, on the event
= P(s0 = s)1{m = ι(h(s))} = ρ0 (s)1{m = ι(h(s))}.
{st = s, mt = m, at = a, st+1 = s′ }, the next memory state is uniquely determined by mt+1 = U m, h(s), a, h(s′ ) . Using the conditional product rule, we obtain P (st+1 = s′ , mt+1 = m′ | st = s, mt = m, at = a) = P (st+1 = s′ | st = s, mt = m, at = a) × P (mt+1 = m′ | st = s, mt = m, at = a, st+1 = s′ ) . The first factor is P (s′ | s, a). The second factor is equal to one if m′ = U m, h(s), a, h(s′ ) , and is equal to zero otherwise. Hence P (st+1 = s′ , mt+1 = m′ | st = s, mt = m, at = a) = P (s′ | s, a)1 m′ = U m, h(s), a, h(s′ ) . Therefore, the action-conditioned transition kernel on the e (s′ , m′ ) | (s, m), a = P (s′ | augmented state space is P s, a)1 m′ = U m, h(s), a, h(s′ ) . This expression is a valid probability kernel. It is clearly nonnegative. Moreover, for every fixed (s, m) ∈ S × M and a ∈ A, X X Pe (s′ , m′ ) | (s, m), a s′ ∈S m′ ∈M
=
X X X
P (s′ | s, a)1 m′ = U m, h(s), a, h(s′ )
P (s′ | s, a)
s′ ∈S
X
1 m′ = U m, h(s), a, h(s′ ) .
m′ ∈M
For each fixed s′ , the deterministic update map U produces exactly one memory state U m, h(s), a, h(s′ ) ∈ M. Thus, X
1 m′ = U m, h(s), a, h(s′ ) = 1.
m′ ∈M
It follows that X X s′ ∈S m′ ∈M
Hence the initial distribution of the augmented process is ρe0 (s, m) = ρ0 (s)1{m = ι(h(s))}. This is also a valid probability distribution because X X X X ρe0 (s, m) = ρ0 (s) 1{m = ι(h(s))} s∈S m∈M
s∈S
=
X
m∈M
ρ0 (s)
s∈S
= 1. The memory variable records information used by the controller, but it does not modify the physical reward generated by the latent state and action. Thus, when the augmented state is (s, m) and action a is selected, the augmented reward is re((s, m), a) = r(s, a). Consider now the lifted policy π e(a | s, m) = πM (a | h(s), m). For every (s, m) ∈ S × M, X X π e(a | s, m) = πM (a | h(s), m) a∈A
a∈A
= 1,
s′ ∈S m′ ∈M
=
X Pe (s′ , m′ ) | (s, m), a = P (s′ | s, a) s′ ∈S
= 1.
so π e is a valid stochastic policy on the augmented state space. Although it is written as a function of s, it depends on s only through the observable value h(s), and therefore does not provide the controller with any additional latent-state information. To prove the Markov property, let Ft := σ Z0 , a0 , Z1 , a1 , . . . , at−1 , Zt be the augmented history immediately before the action at time t is selected. Suppose that Zt = (s, m). Under the original finite-memory controller, the conditional distribution of the action is P(at = a | Ft ) = πM (a | h(st ), mt ) = πM (a | h(s), m) =π e(a | s, m).
Thus, the conditional law of at depends on the entire augmented history only through the present augmented state Zt . For any candidate next augmented state (s′ , m′ ), the conditional law of total probability gives P Zt+1 = (s′ , m′ ) | Ft X = P(at = a | Ft )P Zt+1 = (s′ , m′ ) | Ft , at = a . a∈A
Given Zt = (s, m) and at = a, the transition law derived above depends on no earlier variable. Hence P Zt+1 = (s′ , m′ ) | Ft , at = a = Pe (s′ , m′ ) | (s, m), a . Substituting the action law and the action-conditioned transition law yields P Zt+1 = (s′ , m′ ) | Ft X = π e(a | s, m)Pe (s′ , m′ ) | (s, m), a . a∈A
i st , mt )Pe (st+1 , mt+1 ) | (st , mt ), at . Substituting the definitions ρe0 (s0 , m0 ) = ρ0 (s0 )1{m0 = ι(h(s0 ))}, π e(at | st , mt ) = πM (at | h(st ), mt ), and Pe (st+1 , mt+1 ) | (st , mt ), at = P (st+1 | st , at )1 mt+1 = U mt , h(st ), at , h(st+1 ) into the preceding expression gives Paugh (τT ) = QT −1 ρ0 (s0 )1{m0 = ι(h(s0 ))} × π (a | M t t=0 h(st ), mt )P (st+1 | st , at ) × i = Porig (τT ). 1 mt+1 = U mt , h(st ), at , h(st+1 ) Since the trajectory τT was arbitrary, the original finitememory controller and the augmented Markov process induce exactly the same probability distribution on every finite augmented trajectory. The reward functions also agree pointwise along every trajectory:
Define the induced closed-loop kernel by Peπe (s′ , m′ ) | P (s, m) := a∈A π e(a | s, m)Pe (s′ , m′ ) | (s, m), a . Then P Zt+1 = (s′ , m′ ) | Ft = Peπe (s′ , m′ ) | Zt .
It follows that, for every finite horizon T ,
The right-hand side depends on the augmented history only through Zt . Therefore,
"T −1 X
Zt = (st , mt )
re((st , mt ), at ) = r(st , at ).
Eorig
# t
γ r(st , at )
t=0
"T −1 # X is a time-homogeneous Markov process under π e. π e = Eaug γ t re((st , mt ), at ) . It remains to prove that the augmented process has t=0 exactly the same expected discounted return as the original finite-memory controller. Fix an arbitrary finite horiBecause the reward function is bounded, define zon T ≥ 1 and an arbitrary augmented trajectory τT = ((s0 , m0 ), a0 , (s1 , m1 ), a1 , ..., (sT −1 , mT −1 ), aT −1 , (sT , mT )). Rmax := max |r(s, a)| < ∞. Under the original finite-memory controller, the initial state (s,a)∈S×A has probability ρ0 (s0 ), and the initial memory state must For every trajectory and every T , satisfy m0 = ι(h(s0 )). T −1 T −1 X X At every time t, the action is generated according to γ t |r(st , at )| γ t r(st , at ) ≤ t=0 t=0 πM (at | h(st ), mt ), T −1 X the next latent state is generated according to ≤ Rmax γt t=0 P (st+1 | st , at ), 1 − γT = Rmax and the next memory state must satisfy 1−γ mt+1 = U mt , h(st ), at , h(st+1 ) . Rmax ≤ . 1−γ The chain rule therefore gives Porig (τ ) = T QT −1 h ρ0 (s0 )1{m0 = ι(h(s0 ))} × π (at | The same bound holds for the augmented reward sums beM t=0 cause h(st ), mt )P (st+1 | st , at ) × i re((st , mt ), at ) = r(st , at ). 1 mt+1 = U mt , h(st ), at , h(st+1 ) . Under the augmented process initialized from ρe0 and controlled by π e, the probability of the same traQT −1 h jectory is Paug (τT ) = ρe0 (s0 , m0 ) × t=0 π e(at |
Since γ ∈ (0, 1), the truncated discounted returns converge pointwise to the corresponding infinite discounted returns. max The common deterministic bound R1−γ is integrable. There-
fore, by the dominated convergence theorem,
Eorig
"∞ X
layer gives P Ot+1 = o′ | (Ot , It ) = (o, i), At = a n X = PH (o′ , r) | (o, i), a
# γ t r(st , at )
t=0
= lim Eorig T →∞
= lim
T →∞
π e = Eaug
"T −1 X
r=1
# t
γ r(st , at )
=
t=0 π e Eaug
"T −1 X
γ re((st , mt ), at )
t=0
"∞ X
γ t re((st , mt ), at ) .
Thus, the process on S × M has the stated transition kernel, reward function, and initial distribution. Under the lifted policy π e, the process (st , mt ) is Markov and has exactly the same expected discounted return as the original finitememory controller.
Proof of Lemma 3.3
Proof. Throughout the proof, probabilities are taken under the experiment that starts from the fixed latent state (o0 , i0 ) and applies the prescribed action sequence a0 , . . . , ak−1 . For j = 0, . . . , k, define the visible-prefix event Ej := {O1 = o1 , . . . , Oj = oj },
E0 := Ω,
where (Ot , It ) denotes the random latent state and At denotes the random action at time t, while o0 , . . . , ok are the fixed vertices of the prescribed path ω. Under the experiment fixed above, At = at almost surely for t = 0, . . . , k − 1. Also define the prefix transports τ0 := id[n] ,
τj := σej ◦ · · · ◦ σe1
j Y
n X
1{r = σo,a,o′ (i)}
= PO (o′ | o, a). The final equality holds because σo,a,o′ (i) is a single welldefined element of [n], so exactly one term in the sum of indicators is equal to one. In particular, the probability of observing o′ depends only on the current observation o and the applied action a, and does not depend on the current hidden layer i. Moreover, for any candidate next layer r ∈ [n], the assumption PO (o′ | o, a) > 0 allows us to condition on the event {Ot+1 = o′ }. Using the definition of conditional probability and the preceding marginal identity, we obtain P It+1 = r | (Ot , It ) = (o, i), At = a, Ot+1 = o′ P Ot+1 = o′ , It+1 = r | (Ot , It ) = (o, i), At = a = P Ot+1 = o′ | (Ot , It ) = (o, i), At = a PH (o′ , r) | (o, i), a = PO (o′ | o, a) PO (o′ | o, a)1{r = σo,a,o′ (i)} = PO (o′ | o, a) = 1{r = σo,a,o′ (i)} . a
Consequently, once the visible edge o − → o′ has occurred, the next hidden layer is uniquely determined:
for j = 1, . . . , k.
Thus τk = σω by Definition 3.2. We will prove simultaneously, for every j = 0, . . . , k, that P(Ej ) =
= PO (o′ | o, a)
r=1
#
t=0
I.3
PO (o′ | o, a)1{r = σo,a,o′ (i)}
r=1
# t
n X
PO (oℓ | oℓ−1 , aℓ−1 )
ℓ=1
It+1 = σo,a,o′ (i) with conditional probability one. We now establish the two claimed prefix identities by induction on j. For j = 0, the event E0 = Ω has probability P(E0 ) = 1, which agrees with the empty product 0 Y
and P Ij = τj (i0 ) | Ej = 1. The conditional probability in the second display is well defined because the first identity will imply that P(Ej ) > 0. We first derive the relevant one-step consequences of the HCDP transition kernel. Fix a latent state (o, i), an action a, and an observation o′ satisfying
PO (oℓ | oℓ−1 , aℓ−1 ) = 1.
ℓ=1
Furthermore, I0 = i0 = id[n] (i0 ) = τ0 (i0 ) with probability one. Thus both statements hold at j = 0. Suppose that both statements hold for some j − 1, where 1 ≤ j ≤ k. By the induction hypothesis,
PO (o′ | o, a) > 0. Marginalizing the latent transition kernel over the next hidden
P(Ej−1 ) =
j−1 Y ℓ=1
PO (oℓ | oℓ−1 , aℓ−1 ) > 0,
and, conditional on Ej−1 ,
It follows that Ij = τj (i0 )
Ij−1 = τj−1 (i0 ) with probability one. By the definition of Ej−1 , the visible observation at time j − 1 is also equal to oj−1 on this event. Hence, conditional on Ej−1 , the latent state at time j − 1 is almost surely (Oj−1 , Ij−1 ) = oj−1 , τj−1 (i0 ) . The prescribed action at that time is Aj−1 = aj−1 . Applying the one-step visible marginal identity derived above therefore gives P(Ej | Ej−1 ) =P Oj = oj | Ej−1 =P Oj = oj (Oj−1 , Ij−1 ) = oj−1 , τj−1 (i0 ) , Aj−1 = aj−1 =PO (oj | oj−1 , aj−1 ).
with conditional probability one on Ej , or equivalently, P Ij = τj (i0 ) | Ej = 1. This completes the induction. Taking j = k in the probability identity gives P(Ek ) =
Since every ej is an edge of X, every factor in this product is strictly positive, and hence k Y
Finally, taking j = k in the hidden-layer identity and using the definition of directed path transport gives Ik = τk (i0 ) = σek ◦ · · · ◦ σe1 (i0 )
the multiplication rule for probabilities yields
= σω (i0 )
P(Ej ) = P(Ej−1 )P(Ej | Ej−1 ) ! PO (oℓ | oℓ−1 , aℓ−1 ) PO (oj | oj−1 , aj−1 )
ℓ=1
=
j Y
PO (oj | oj−1 , aj−1 ) > 0.
j=1
Ej = Ej−1 ∩ {Oj = oj },
=
PO (oj | oj−1 , aj−1 ).
j=1
Since
j−1 Y
k Y
with conditional probability one on Ek . Therefore, executing a0 , . . . , ak−1 produces the prescribed visible path with the stated positive product probability, and conditional on that visible path, its hidden lift is unique and has terminal layer ik = σω (i0 ).
PO (oℓ | oℓ−1 , aℓ−1 ).
ℓ=1
Because
aj−1
I.4
ej = (oj−1 −−−→ oj ) is an edge of the directed support graph X, the definition of X implies PO (oj | oj−1 , aj−1 ) > 0. Every factor in the product for P(Ej−1 ) is also strictly positive by the induction hypothesis. Therefore, P(Ej ) > 0. It remains to determine the hidden layer after the jth visible edge. Conditional on Ej−1 , the current hidden layer is almost surely τj−1 (i0 ). Conditional further on the next prescribed observation being Oj = oj , the one-step conditional layer identity gives Ij = σoj−1 ,aj−1 ,oj τj−1 (i0 ) = σej τj−1 (i0 ) = σej ◦ τj−1 (i0 ). By the definition of the prefix transports, σej ◦ τj−1 = σej ◦ σej−1 ◦ · · · ◦ σe1 = σej ◦ σej−1 ◦ · · · ◦ σe1 = τj .
Proof of Lemma 3.5
Proof. We first verify that the definition of T indeed produces a partition family. Fix a partition family Π and an observation o ∈ O, and define a binary relation on [n] by declaring that i and j are related precisely when R(o, i, a) = R(o, j, a)
for every a ∈ A
′ ′ and σo,a,o′ (i) ≡Π o′ σo,a,o′ (j) for every (a, o ) such that PO (o | o, a) > 0. This relation is reflexive because equality of rewards is reflexive and each relation ≡Π o′ is reflexive. It is symmetric because equality is symmetric and each ≡Π o′ is symmetric. To verify transitivity, suppose that i is related to j and that j is related to ℓ. Then, for every a ∈ A,
R(o, i, a) = R(o, j, a) = R(o, ℓ, a), so R(o, i, a) = R(o, ℓ, a). Moreover, for every feasible pair (a, o′ ) satisfying PO (o′ | o, a) > 0, we have σo,a,o′ (i) ≡Π o′ σo,a,o′ (j)
and
σo,a,o′ (j) ≡Π o′ σo,a,o′ (ℓ).
The transitivity of ≡Π o′ therefore gives
Applying the monotonicity of T yields T (Π(t) ) ⪯ T (Π(t−1) ).
σo,a,o′ (i) ≡Π o′ σo,a,o′ (ℓ). Thus the relation is an equivalence relation on [n]. Since this holds for every o ∈ O, the family T (Π) = {T (Π)o }o∈O
Using the recursion Π(s+1) = T (Π(s) ), the preceding relation becomes
is well defined as a partition family. We next prove monotonicity. Let Π ⪯ Π′ . By the definition of refinement, for every observation o′ ∈ O and every x, y ∈ [n], ′ =⇒ x ≡Π x ≡Π o′ y. o′ y Fix o ∈ O and suppose that
Π(t+1) ⪯ Π(t) . Together with the base case Π(1) ⪯ Π(0) , induction gives Π(t+1) ⪯ Π(t)
To prove finite stabilization with the stated bound, let
i ≡To (Π) j.
(t) b(t) o := |Πo |
Expanding the definition of T (Π), we obtain R(o, i, a) = R(o, j, a)
denote the number of blocks of the partition at observation o, and define the total block count X B (t) := b(t) o .
for every a ∈ A
and, for every (a, o′ ) with PO (o′ | o, a) > 0,
o∈O
σo,a,o′ (i) ≡Π o′ σo,a,o′ (j).
(t+1)
′
σo,a,o′ (i) ≡Π o′ σo,a,o′ (j) for every such feasible pair (a, o′ ). The reward equalities are unchanged, and hence the defining conditions of T (Π′ ) are satisfied. Therefore ′ i ≡To (Π ) j. We have proved, for every o ∈ O, that every block of T (Π)o is contained in a block of T (Π′ )o . Consequently, T (Π) ⪯ T (Π′ ), which establishes the claimed monotonicity. We now consider the sequence initialized at the reward partition. By definition, (0)
j
⇐⇒
R(o, i, a) = R(o, j, a)
If
(1)
i ≡Π o (1)
(0)
then Π = T (Π of T gives
(t+1)
(t)
refines Πo , every block of Πo Since Πo (t) in exactly one block of Πo . It follows that
Because Π ⪯ Π′ , the latter equivalence implies
i ≡Π o
for every t ≥ 0.
for every a ∈ A.
j,
is contained
b(t+1) ≥ b(t) o o . If the refinement is strict at observation o, then this inequality (t) (t+1) strictly refines Πo but is strict. Indeed, suppose that Πo that b(t+1) = b(t) o o . (t+1)
is contained in a Every block of the finer partition Πo (t) (t) unique block of Πo , and every block of Πo contains at (t+1) , because the finer blocks cover least one block of Πo all of [n]. Since the two finite collections contain the same (t) number of blocks, every block of Πo must contain exactly (t) (t+1) . If C is a block of Πo and D is the one block of Πo (t+1) contained in C, then every element unique block of Πo of C must belong to some finer block contained in C. Since D is the only such finer block, we have C ⊆ D.
), and the first condition in the definition
The refinement relation already gives
R(o, i, a) = R(o, j, a)
for every a ∈ A.
D ⊆ C,
It follows directly from the definition of Π(0) that (0)
i ≡Π o
j.
(1)
(0)
Thus every block of Πo is contained in a block of Πo for every o ∈ O, and therefore
and hence C = D. This holds for every block, so the two partitions are equal, contradicting the assumed strictness. Therefore a strict refinement at observation o necessarily satisfies b(t+1) ≥ b(t) o o + 1. Consequently, whenever the partition family changes,
Π(1) ⪯ Π(0) . Suppose inductively that, for some t ≥ 1, Π(t) ⪯ Π(t−1) .
Π(t+1) ̸= Π(t) , the relation Π(t+1) ⪯ Π(t)
(t+1)
implies that there is at least one observation o at which Πo (t) strictly refines Πo . Hence B (t+1) ≥ B (t) + 1.
Every partition of the nonempty set [n] has at least one block and at most n blocks. Therefore, for every t ≥ 0, 1 ≤ b(t) o ≤n
for every o ∈ O,
By the definition of the reward partition Π(0) , this is equivalent to (0) i ≡Π j. o Since this implication holds for every o ∈ O and every (0) i, j ∈ [n], every block of Θo is contained in a block of Πo . Hence Θ ⪯ Π(0) . We now show by induction that
and summing over o gives
Θ ⪯ Π(t)
|O| ≤ B (0) ≤ B (t) ≤ |O|n. If the sequence underwent N strict refinements, the total block count would increase by at least N , so B (0) + N ≤ |O|n.
for every t ≥ 0.
The statement holds at t = 0 by the preceding argument. Suppose that Θ ⪯ Π(t) for some t ≥ 0. Monotonicity of T gives
It follows that
T (Θ) ⪯ T (Π(t) ).
N ≤ |O|n − B (0) ≤ |O|n − |O| = |O|(n − 1).
Since Θ is stable and Π(t+1) = T (Π(t) ), this becomes
Thus the sequence can undergo at most
Θ = T (Θ) ⪯ T (Π(t) ) = Π(t+1) .
|O|(n − 1) strict refinements. Therefore there exists an index t⋆ , reached after at most that many strict refinements, such that Π
(t⋆ +1)
(t⋆ )
=Π
.
Define
Π⋆ := Π(t⋆ ) . Using the recursive definition of the sequence, we then obtain T (Π⋆ ) = T (Π(t⋆ ) ) = Π(t⋆ +1) = Π(t⋆ ) = Π⋆ . Hence Π⋆ is a fixed point of T . Moreover, once two consecutive iterates are equal, all subsequent iterates remain equal. Indeed, Π(t⋆ +2) = T (Π(t⋆ +1) ) = T (Π⋆ ) = Π⋆ , and repeating the same argument gives Π(t) = Π⋆
The induction is therefore complete, and Θ ⪯ Π(t)
In particular, evaluating this relation at the stabilization index t⋆ yields Θ ⪯ Π(t⋆ ) = Π⋆ . Thus every stable partition family refines Π⋆ . Since Π⋆ is itself stable, it is an element of the collection of stable partition families, and every other stable family lies below it in the refinement order. By the convention in Definition 3.4, a smaller partition family is finer and a larger partition family is coarser. Therefore Π⋆ is the greatest stable family with respect to ⪯, equivalently the coarsest stable partition family.
I.5
qoΠ : [n] −→ [n]/Πo ,
qoΠ (i) = [i]Π o,
define an exact class abstraction in the sense of Definition 3.6. Thus, for every o ∈ O, there exists a class-level mean reward b : {(o, ĉ, a) : o ∈ O, ĉ ∈ [n]/Πo , a ∈ A} −→ R, R and, for every feasible triple (o, a, o′ ) satisfying PO (o′ | o, a) > 0,
T (Θ) = Θ. Fix an observation o ∈ O and layers i, j ∈ [n] satisfying
Proof of Proposition 3.7
Proof. Assume first that the quotient maps
for every t ≥ t⋆ .
It remains to prove the maximality statement among stable families. Let Θ be any stable partition family, so that
for every t ≥ 0.
there exists a deterministic class transport τbo,a,o′ : [n]/Πo −→ [n]/Πo′ ,
i ≡Θ o j.
such that, for every layer i ∈ [n],
By stability, i ≡To (Θ) j. The first condition in the definition of T (Θ) therefore implies R(o, i, a) = R(o, j, a)
for every a ∈ A.
b o, qoΠ (i), a R(o, i, a) = R and qoΠ′ (σo,a,o′ (i)) = τbo,a,o′ qoΠ (i) .
Fix an arbitrary observation o ∈ O and arbitrary layers i, j ∈ [n] satisfying i ≡Π o j. By the definition of the quotient map, two layers have the same quotient label precisely when they belong to the same block of Πo . Hence i ≡Π o j
⇐⇒
Consequently, i ≡Π o j
=⇒
i ≡To (Π) j.
Since o ∈ O and i, j ∈ [n] were arbitrary, every block of Πo is contained in a block of T (Π)o for every o ∈ O. By the definition of the refinement order, this means Π ⪯ T (Π).
qoΠ (i) = qoΠ (j),
Conversely, assume that
and in particular qoΠ (i) = qoΠ (j).
Π ⪯ T (Π).
For any action a ∈ A, the exact reward identity gives b o, q Π (i), a R(o, i, a) = R o
For every observation o ∈ O, define the class-label set by
and
and define the abstraction map to be the canonical quotient map qoΠ : [n] −→ Cbo , qoΠ (i) := [i]Π o. Because [n] is finite, the set [n]/Πo of blocks of Πo is finite. Moreover, qoΠ is surjective: for every class C ∈ [n]/Πo , the block C is nonempty, so one may choose i ∈ C, and then
b o, q Π (j), a . R(o, j, a) = R o Since the quotient labels are equal, these two expressions satisfy b o, q Π (i), a R(o, i, a) = R o b o, qoΠ (j), a =R = R(o, j, a). Because a ∈ A was arbitrary, we have shown that R(o, i, a) = R(o, j, a)
for every a ∈ A.
Now fix any action–successor pair (a, o′ ) satisfying PO (o′ | o, a) > 0. The transport identity in Definition 3.6 gives qoΠ′ (σo,a,o′ (i)) = τbo,a,o′ qoΠ (i) and
qoΠ′ (σo,a,o′ (j)) = τbo,a,o′ qoΠ (j) . Using again the equality qoΠ (i) = qoΠ (j), we obtain qoΠ′ (σo,a,o′ (i)) = τbo,a,o′ qoΠ (i) = τbo,a,o′ qoΠ (j) = qoΠ′ (σo,a,o′ (j)) . Equality of the images under qoΠ′ is equivalent to membership in the same block of Πo′ . Therefore σo,a,o′ (i) ≡Π o′ σo,a,o′ (j). Since the feasible pair (a, o′ ) was arbitrary, this relation holds for every (a, o′ ) such that PO (o′ | o, a) > 0. We have therefore established both conditions R(o, i, a) = R(o, j, a)
for every a ∈ A
′ ′ and σo,a,o′ (i) ≡Π o′ σo,a,o′ (j) for every (a, o ) with PO (o | o, a) > 0. By the definition of the stability operator in Definition 3.4, these two conditions are equivalent to
i ≡To (Π) j.
Cbo := [n]/Πo
qoΠ (i) = C. We define the proposed class-level reward by bΠ o, [i]Π R o , a := R(o, i, a). We also define, for every feasible triple (o, a, o′ ) satisfying PO (o′ | o, a) > 0, the proposed class transport by Π Π Π τbo,a,o := σo,a,o′ (i) o′ . ′ [i]o To obtain an exact class abstraction, it is essential to show that these definitions do not depend on the representative i chosen from the quotient class. Suppose that two representatives i, j ∈ [n] determine the same class at observation o, so that Π [i]Π o = [j]o .
By the definition of quotient classes, this equality is equivalent to i ≡Π o j. The assumption Π ⪯ T (Π) means that every Πo -block is contained in a T (Π)o -block. Therefore i ≡Π =⇒ i ≡To (Π) j, o j and hence i ≡To (Π) j. Expanding the definition of T (Π) gives R(o, i, a) = R(o, j, a)
for every a ∈ A.
It follows that bΠ o, [i]Π , a = R(o, i, a) R o = R(o, j, a) bΠ o, [j]Π =R o ,a .
bΠ is independent of the representaThus the definition of R tive, and the class-level reward is well defined. The same relation
The equivalence just established implies that every stable family defines an exact class abstraction. Exactness alone, however, requires only
i ≡To (Π) j
Π ⪯ T (Π);
also implies, for every feasible (a, o′ ),
it does not require equality. Thus it is possible that
σo,a,o′ (i) ≡Π o′ σo,a,o′ (j).
Π ≺ T (Π),
Equivalence under Πo′ is precisely equality of the corresponding quotient classes. Therefore Π Π σo,a,o′ (i) o′ = σo,a,o′ (j) o′ .
in which case Π distinguishes some layers that T (Π) can safely merge. For completeness, this strict case can be realized by an explicit HCDP. Let
Consequently,
n = 2,
O = {o0 , o1 },
A = {a},
and define the visible transition kernel by Π Π Π τbo,a,o = σo,a,o′ (i) o′ ′ [i]o Π = σo,a,o′ (j) o′ Π Π = τbo,a,o ′ [j]o .
Π Thus τbo,a,o ′ is also independent of the chosen representative and is a well-defined deterministic map Π τbo,a,o ′ : [n]/Πo −→ [n]/Πo′ .
It remains to verify the two exactness identities. For every bΠ gives o ∈ O, i ∈ [n], and a ∈ A, the definition of R bΠ o, qoΠ (i), a = R bΠ o, [i]Π R o ,a
PO (o1 | o0 , a) = 1,
Assign the identity permutation to both feasible edges: σo0 ,a,o1 = id[2] ,
bΠ o, qoΠ (i), a . R(o, i, a) = R
R(o, i, a) = 0
= qoΠ′ (σo,a,o′ (i)) . Equivalently,
T (Π) = Π. Since the refinement order is reflexive,
Πo1 = {1, 2} .
At observation o1 , layers 1 and 2 have equal rewards. Their successors under the only feasible edge are σo1 ,a,o1 (1) = 1,
σo1 ,a,o1 (2) = 2,
and these successors are equivalent under Πo1 because 1 ≡Π o1 2. It follows from the definition of T that 1 ≡To1(Π) 2, and hence T (Π)o1 = {1, 2} . At observation o0 , layers 1 and 2 again have equal rewards. Their successors under the only feasible edge are
Π Π qoΠ′ (σo,a,o′ (i)) = τbo,a,o ′ qo (i) . The quotient maps are surjective, the class label sets are finite, and the class reward and class transports satisfy exactly the two identities in Definition 3.6. Hence {qoΠ }o∈O defines an exact class abstraction. Combining the two implications proves {qoΠ }o∈O defines an exact class abstraction ⇐⇒ Π ⪯ T (Π). Suppose now that Π is stable. By Definition 3.4,
for every o ∈ O and i ∈ [2].
Consider the partition family Πo0 = {1}, {2} ,
′
Similarly, for every feasible (o, a, o ), Π Π Π Π τbo,a,o bo,a,o ′ qo (i) = τ ′ [i]o Π = σo,a,o′ (i) o′
σo1 ,a,o1 = id[2] ,
and let every mean reward be zero:
= R(o, i, a). Equivalently,
PO (o1 | o1 , a) = 1.
σo0 ,a,o1 (1) = 1,
σo0 ,a,o1 (2) = 2.
Both successors lie in the common block {1, 2} of Πo1 , so σo0 ,a,o1 (1) ≡Π o1 σo0 ,a,o1 (2). Therefore
1 ≡To0(Π) 2,
and hence T (Π)o0 = {1, 2} . We have consequently obtained Π ⪯ T (Π),
Π ⪯ Π,
because the two singleton blocks at o0 refine the single block of T (Π)o0 , while the partitions at o1 are equal. However,
Π ⪯ T (Π).
Π ̸= T (Π),
and therefore
For every action a ∈ A, exactness of q gives
because
Πo0 = {1}, {2} ̸= {1, 2} = T (Π)o0 . The established criterion shows that this Π is exact, but it is not stable. Its separation of layers 1 and 2 at o0 is unnecessary: the two layers have identical immediate rewards and are transported, under every feasible continuation, into the same target quotient class. Finally, stability was defined precisely as the fixed-point condition T (Π) = Π. Therefore, for an exact family, as for any partition family, ⇐⇒
Π is stable
b qo (i), a) R(o, i, a) = R(o, b qo (j), a) = R(o, = R(o, j, a). Thus i and j have identical immediate mean rewards for every action. Now fix any (a, o′ ) satisfying PO (o′ | o, a) > 0. Using the exact transport identity and again using qo (i) = qo (j), we obtain qo′ (σo,a,o′ (i)) = τbo,a,o′ (qo (i)) = τbo,a,o′ (qo (j)) = qo′ (σo,a,o′ (j)) .
Π = T (Π).
This proves all assertions of the proposition.
I.6
Proof of Theorem 3.8
Proof. By Lemma 3.5, the family Π⋆ obtained from the refinement iteration is a fixed point of the stability operator, so T (Π⋆ ) = Π⋆ .
By the definition of the kernel partition at o′ , this equality is equivalent to q
σo,a,o′ (i) ≡Π o′ σo,a,o′ (j). Since the feasible pair (a, o′ ) was arbitrary, the two defining conditions of T (Πq ) are satisfied. Therefore
In particular,
q
i ≡To (Π ) j.
Π⋆ ⪯ T (Π⋆ ). We have proved that
For each o ∈ O, the map
q
qo⋆ : [n] −→ [n]/Π⋆o ,
i ≡Π o j
qo⋆ (i) = [i]o ,
is exactly the canonical quotient map associated with the partition Π⋆o . Proposition 3.7, applied with Π = Π⋆ , therefore implies that the family q ⋆ = {qo⋆ }o∈O defines an exact class abstraction. This proves the existence assertion. Let now qo : [n] −→ Cbo , o ∈ O, be any other exact class abstraction. By Definition 3.6, each b qo is surjective, and there exist a class-level mean reward R and, for every feasible triple (o, a, o′ ), a deterministic class transport τbo,a,o′ : Cbo −→ Cbo′ such that, for every i ∈ [n], b qo (i), a) R(o, i, a) = R(o,
=⇒
q
i ≡To (Π ) j
for every o ∈ O and every i, j ∈ [n]. Equivalently, Πq ⪯ T (Πq ). We next compare Πq with the fixed-point sequence from Lemma 3.5. Recall that Π(0) is the reward partition, so (0)
i ≡Π o
j
⇐⇒
R(o, i, a) = R(o, j, a) for every a ∈ A.
q
If i ≡Π o j, the reward calculation above shows that R(o, i, a) = R(o, j, a) Hence
for every a ∈ A.
(0)
i ≡Π j. o This implication holds at every observation, and therefore Πq ⪯ Π(0) .
and qo′ (σo,a,o′ (i)) = τbo,a,o′ (qo (i)) whenever PO (o′ | o, a) > 0. Let Πq = {Πqo }o∈O be the kernel partition family induced by q, namely q
i ≡Π o j
⇐⇒
qo (i) = qo (j).
We first show that this induced partition family is exactcompatible. Fix o ∈ O and suppose that q
We claim that, more generally, Πq ⪯ Π(t)
for every t ≥ 0.
The claim has just been established for t = 0. Suppose that it holds for some t ≥ 0. Monotonicity of T , proved in Lemma 3.5, gives T (Πq ) ⪯ T (Π(t) ).
i ≡Π o j.
Since Πq is exact-compatible and the iteration satisfies Π(t+1) = T (Π(t) ), we have
qo (i) = qo (j).
Πq ⪯ T (Πq ) ⪯ T (Π(t) ) = Π(t+1) .
Then
Thus
Πq ⪯ Π(t+1) , and induction proves the claim for every t ≥ 0. By Lemma 3.5, there exists a finite index t⋆ such that Π(t⋆ ) = Π⋆ . Evaluating the preceding refinement relation at t = t⋆ yields q
satisfies
qo⋆ = λo ◦ qo .
Let ĉ ∈ Cbo . Since qo is surjective, there exists i ∈ [n] with qo (i) = ĉ. Then
λo (ĉ) = λo (qo (i)) = qo⋆ (i) = κo (qo (i)) = κo (ĉ).
⋆
Π ⪯Π . Unpacking this refinement relation, for every o ∈ O and every i, j ∈ [n], q
i ≡Π o j
=⇒
⋆
i ≡Π o j.
Thus λo = κo , proving uniqueness. Conversely, the existence of any factorization qo⋆ = κo ◦ qo immediately implies that
By the definitions of Πq and qo⋆ , this is precisely qo (i) = qo (j)
=⇒
qo (i) = qo (j)
qo⋆ (i) = κo (qo (i)) = κo (qo (j)) = qo⋆ (j),
=⇒
qo⋆ (i) = qo⋆ (j).
This proves that no exact abstraction can merge two layers that are separated by the stable quotient. We now construct the factor map. Fix o ∈ O. For each ĉ ∈ Cbo , surjectivity of qo guarantees that the fiber qo−1 ({ĉ}) = {i ∈ [n] : qo (i) = ĉ} is nonempty. Choose any i ∈ [n] with qo (i) = ĉ and define κo (ĉ) := qo⋆ (i). This definition is independent of the chosen representative. Indeed, if i, j ∈ [n] both satisfy
so the implication and the factorization statements are indeed equivalent. We have shown that the kernel partition Πq of every exact abstraction satisfies Πq ⪯ Π⋆ , while Π⋆ itself defines the exact abstraction q ⋆ . Hence every exact observation-wise abstraction refines the stable quotient. Equivalently, every exact label set admits a surjection onto the corresponding stable label set; in particular, |Co | ≤ |Cbo |
qo (i) = ĉ = qo (j), then the implication proved above gives qo⋆ (i) = qo⋆ (j). Thus every representative of the fiber of ĉ produces the same value, and κo : Cbo −→ Co is a well-defined map. For every i ∈ [n], taking ĉ = qo (i) in the definition of κo gives (κo ◦ qo )(i) = κo (qo (i)) = qo⋆ (i). Therefore qo⋆ = κo ◦ qo . The map κo is surjective. To see this, let c ∈ Co = [n]/Π⋆o be arbitrary. Since c is a nonempty block of the partition Π⋆o , there exists i ∈ [n] such that qo⋆ (i) = c. The factorization identity then gives c = qo⋆ (i) = κo (qo (i)), so c lies in the image of κo . Because c was arbitrary, κo (Cbo ) = Co . Finally, suppose that another map λo : Cbo −→ Co
for every o ∈ O.
Therefore Π⋆ is the coarsest exact observation-wise partition family, and q ⋆ is the coarsest exact class abstraction.
I.7
Proof of Theorem 3.9
Proof. Recall that Π⋆ is stable, and hence T (Π⋆ ) = Π⋆ . Fix an observation o ∈ O and two layers i, j ∈ [n] belonging to the same stable class, so that [i]o = [j]o , or equivalently,
⋆
i ≡Π o j. Since Π⋆ = T (Π⋆ ), this also means ⋆
i ≡To (Π ) j. Expanding the definition of the stability operator gives, for every action a ∈ A, R(o, i, a) = R(o, j, a), and, for every o′ ∈ O satisfying PO (o′ | o, a) > 0, ⋆
σo,a,o′ (i) ≡Π o′ σo,a,o′ (j). The latter equivalence is precisely the equality of the corresponding stable classes: [σo,a,o′ (i)]o′ = [σo,a,o′ (j)]o′ .
It follows that replacing i by any other representative j of the same class [i]o leaves both R(o, i, a) and [σo,a,o′ (i)]o′ unchanged. Therefore the formulas R̄(o, [i]o , a) := R(o, i, a)
because PO (· | o, a) is a probability distribution on O. Thus P̄ (· | (o, c), a) is a probability distribution on S̄. The proposed initial law is also nonnegative. Since the classes in Co = [n]/Π⋆o form a partition of [n], each i ∈ [n] belongs to exactly one class c ∈ Co . Therefore XX X X ρ̄0 (o, c) = ρ0 (o, i) o∈O c∈Co i: [i]o =c
(o,c)∈S̄
=
and τo,a,o′ ([i]o ) := [σo,a,o′ (i)]o′ are independent of the representative and define, respectively, a function R̄ : S̄ × A −→ R and, for every feasible triple (o, a, o′ ), a deterministic map
n XX
ρ0 (o, i)
o∈O i=1
= 1. Hence ρ̄0 ∈ ∆(S̄). Together with the original discount factor γ ∈ (0, 1), these facts show that
τo,a,o′ : Co −→ Co′ .
(S̄, A, P̄ , R̄, γ, ρ̄0 )
For a triple satisfying PO (o′ | o, a) = 0, neither the raw transport nor the quotient transport affects any transition probability. For notational uniformity in sums over all o′ ∈ O, we may extend both σo,a,o′ and τo,a,o′ arbitrarily to such zero-probability triples; these arbitrary values are always multiplied by PO (o′ | o, a) = 0 and therefore do not change any kernel, expectation, or value function. We next verify that the displayed formulas define a finite discounted MDP. The state space
is a well-defined finite discounted MDP. The quotient kernel is exactly the pushforward of the latent HCDP kernel under the map
S̄ = {(o, c) : o ∈ O, c ∈ Co } is finite because O and [n] are finite. The reward R̄ is bounded by the same bound as the latent reward, since for every (o, c) ∈ S̄ one may choose i ∈ [n] with c = [i]o and obtain
q ⋆ (o, i) := (o, [i]o ). Indeed, fix (o, i) ∈ O × [n], put c = [i]o , and fix a ∈ A, o′ ∈ O, and c′ ∈ Co′ . The probability that the latent next state has observation o′ and stable class c′ is X PH (o′ , i′ ) | (o, i), a i′ : [i′ ]o′ =c′
=
X
PO (o′ | o, a)1{i′ = σo,a,o′ (i)}
i′ : [i′ ]o′ =c′
|R̄(o, c, a)| = |R(o, i, a)| ≤ rmax .
= PO (o′ | o, a)
For every (o, c) ∈ S̄ and a ∈ A, the proposed transition probabilities are nonnegative because
′
PO (o′ | o, a) ≥ 0 and the indicator is nonnegative. Their total mass is X X P̄ (o′ , c′ ) | (o, c), a o′ ∈O c′ ∈Co′
=
X X
PO (o′ | o, a)1{c′ = τo,a,o′ (c)}
o′ ∈O c′ ∈Co′
=
X
PO (o′ | o, a)
o′ ∈O
X
1{c′ = τo,a,o′ (c)} .
c′ ∈Co′
For each o′ with PO (o′ | o, a) > 0, the element τo,a,o′ (c) is a single well-defined member of Co′ , so exactly one term in the inner sum is equal to one and all other terms are zero. For o′ with PO (o′ | o, a) = 0, the entire corresponding summand vanishes. Hence X X P̄ (o′ , c′ ) | (o, c), a o′ ∈O c′ ∈Co′
=
X o′ ∈O
= 1,
PO (o′ | o, a)
X
1{i′ = σo,a,o′ (i)}
i′ : [i′ ]o′ =c′
= PO (o | o, a)1{[σo,a,o′ (i)]o′ = c′ } = PO (o′ | o, a)1{τo,a,o′ ([i]o ) = c′ } = P̄ (o′ , c′ ) | (o, [i]o ), a . The fourth equality holds because the deterministic successor layer σo,a,o′ (i) either belongs to the class c′ or does not, and the fifth equality is the definition of the quotient transport. This calculation makes explicit that, once the current stable class is known, the conditional law of the next observation– class pair is independent of the latent representative i. Let now π̄ be an arbitrary stationary policy on the quotient MDP, and define its lift to the fully observed latent state space by π ↑ (a | o, i) := π̄(a | o, [i]o ). This is a valid stationary policy because, for every (o, i), π ↑ (a | o, i) ≥ 0 for all a, and X a∈A
π ↑ (a | o, i) =
X a∈A
= 1.
π̄(a | o, [i]o )
To compare the two value functions, let B(S̄) denote the bounded real-valued functions on S̄, and define the lifting operator
L gives (Tπ↑ Lf )(o, i) "
L : B(S̄) −→ B(O × [n])
=
π̄(a | o, [i]o ) R(o, i, a)
a∈A n X X
+γ
by
X
# PO (o′ | o, a)1{i′ = σo,a,o′ (i)} f (o′ , [i′ ]o′ )
o′ ∈O i′ =1
(Lf )(o, i) := f (o, [i]o ). Define the policy-evaluation operator of the latent " HCDP unP der π ↑ by (Tπ↑ V )(o, i) := a∈A π ↑ (a | o, i) R(o, i, a) + # P Pn ′ ′ ′ ′ γ o′ ∈O i′ =1 PH (o , i ) | (o, i), a V (o , i ) ,
" =
X
π̄(a | o, [i]o ) R(o, i, a)
a∈A
# X
+γ
′
′
PO (o | o, a)f o , [σo,a,o′ (i)]o′
o′ ∈O
" =
X
π̄(a | o, [i]o ) R̄(o, [i]o , a)
a∈A
and
define
the
quotient policy-evaluation operator " P by (T̄π̄ f )(o, c) := | o, c) R̄(o, c, a) + a∈A π̄(a # ′ ′ P P ′ ′ γ o′ ∈O c′ ∈Co′ P̄ (o , c ) | (o, c), a f (o , c ) . Both operators are γ-contractions under the sup norm. For example, for bounded V, W and every (o, i), |(Tπ↑ V )(o, i) − (Tπ↑ W )(o, i)| X ≤γ π ↑ (a | o, i) a∈A n X X
PH (o′ , i′ ) | (o, i), a |V (o′ , i′ ) − W (o′ , i′ )|
+γ
# PO (o | o, a)f o , τo,a,o′ ([i]o ) .
X
′
′
o′ ∈O
For fixed o′ , the definition of P̄ implies X P̄ (o′ , c′ ) | (o, [i]o ), a f (o′ , c′ ) c′ ∈Co′
=
X
PO (o′ | o, a)1{c′ = τo,a,o′ ([i]o )} f (o′ , c′ )
c′ ∈Co′
= PO (o′ | o, a)f o′ , τo,a,o′ ([i]o ) . Substituting this equality into the preceding Bellman expression yields (Tπ↑ Lf )(o, i) = (T̄π̄ f )(o, [i]o )
o′ ∈O i′ =1
≤ γ∥V − W ∥∞
X
π ↑ (a | o, i)
a∈A n X X
PH (o′ , i′ ) | (o, i), a
o′ ∈O i′ =1
= γ∥V − W ∥∞ .
= (LT̄π̄ f )(o, i). Thus, as an operator identity, Tπ↑ L = LT̄π̄ . Since V̄π̄ is the fixed point of T̄π̄ , T̄π̄ V̄π̄ = V̄π̄ . Applying the intertwining identity gives
Taking the supremum over (o, i) gives ∥Tπ↑ V − Tπ↑ W ∥∞ ≤ γ∥V − W ∥∞ . The same calculation with P̄ and π̄ gives ∥T̄π̄ f − T̄π̄ g∥∞ ≤ γ∥f − g∥∞ . Consequently, Tπ↑ and T̄π̄ have unique bounded fixed points, namely Vπ↑ and V̄π̄ . The lifting operator intertwines these two Bellman evaluation operators. To verify this identity, fix f ∈ B(S̄) and (o, i) ∈ O × [n]. Substituting the definitions of π ↑ , PH , and
Tπ↑ (LV̄π̄ ) = L(T̄π̄ V̄π̄ ) = LV̄π̄ . Therefore LV̄π̄ is a fixed point of Tπ↑ . By uniqueness of the fixed point of Tπ↑ , Vπ↑ = LV̄π̄ . Evaluating this identity at (o, i) gives Vπ↑ (o, i) = V̄π̄ (o, [i]o ), which proves the claimed policy-value preservation. It remains to establish the optimality assertion. Let the Bellman optimality operator of the fully " observed ⋆ latent HCDP be (Tfull V )(o, i) := maxa∈A R(o, i, a) +
# γ
P
o′ ∈O
Pn
i′ =1 PH
′
′
′
′
(o , i ) | (o, i), a V (o , i ) , and let
The already proved policy-value identity gives V(π⋆ )↑ (o, i) = V̄π̄⋆ (o, [i]o )
⋆
the quotient " Bellman optimality operator be (T̄ f )(o, c) := P P maxa∈A R̄(o, c, a) + γ o′ ∈O c′ ∈Co′ P̄ (o′ , c′ ) | # (o, c), a f (o′ , c′ ) . Because both state spaces and the action set are finite and γ ∈ (0, 1), these operators are γ-contractions under the sup norm. Their unique fixed points are, respectively, the fully observed latent optimal ⋆ value Vfull and the quotient optimal value V̄ ⋆ . The same calculation used for policy evaluation, now performed before taking the maximum over actions, shows that for every f ∈ B(S̄), ⋆ Tfull Lf = LT̄ ⋆ f.
= V̄ ⋆ (o, [i]o ) ⋆ = Vfull (o, i). Hence (π ⋆ )↑ is optimal for the fully observed latent HCDP. By construction, whenever [i]o = [j]o , (π ⋆ )↑ (· | o, i) = π̄ ⋆ (· | o, [i]o ) = π̄ ⋆ (· | o, [j]o ) = (π ⋆ )↑ (· | o, j). Thus a fully observed latent optimal policy can be chosen constant on stable classes, completing the proof.
I.8
Proof of Corollary C.2
Proof. Write the directed path as where o0 = o and ok = o′ . For each edge ej , use the abbreviations
PO (oj | oj−1 , aj−1 ) > 0.
= max R(o, i, a) a∈A
# PO (o′ | o, a)f o′ , [σo,a,o′ (i)]o′
Theorem 3.9 therefore applies to every ej and shows that τej : Coj−1 −→ Coj
is a well-defined map satisfying, for every r ∈ [n],
o′ ∈O
"
τej ([r]oj−1 ) = τoj−1 ,aj−1 ,oj ([r]oj−1 )
= max R̄(o, [i]o , a) a∈A
= [σoj−1 ,aj−1 ,oj (r)]oj #
+γ
X
PO (o′ | o, a)f o′ , τo,a,o′ ([i]o )
o′ ∈O ⋆
= (T̄ f )(o, [i]o )
= [σej (r)]oj . Thus each raw one-edge transport descends through the stable quotient. Moreover, the codomain Coj of τej is exactly the domain of τej+1 . Consequently, the composition
= (LT̄ ⋆ f )(o, i).
τω = τek ◦ · · · ◦ τe1 : Co0 −→ Cok
⋆
⋆
Since V̄ is the unique fixed point of T̄ , T̄ ⋆ V̄ ⋆ = V̄ ⋆ . Consequently,
is well defined. For j = 0, . . . , k, let ω (j) := e1 · · · ej denote the length-j prefix of ω, with
⋆ Tfull (LV̄ ⋆ ) = L(T̄ ⋆ V̄ ⋆ )
ω (0) := ∅o0 .
⋆
= LV̄ .
Define
Thus LV̄ observed optimal fixed point implies
σω(0) := id[n] , and, for j ≥ 1, define
⋆ Vfull = LV̄ ⋆ ,
σω(j) := σej ◦ · · · ◦ σe1 ,
⋆
τej := τoj−1 ,aj−1 ,oj .
and
Because ej is an edge of the directed support graph, it is feasible, and hence
⋆ (Tfull Lf )(o, i) "
+γ
ej = (oj−1 −−−→ oj ),
σej := σoj−1 ,aj−1 ,oj
Indeed, for every (o, i),
X
aj−1
ω = e1 · · · ek ,
⋆ is a fixed point of Tfull . Uniqueness of the fully
and therefore, for every (o, i), ⋆ Vfull (o, i) = V̄ ⋆ (o, [i]o ).
Finally, because the quotient MDP is finite and discounted, there exists a stationary deterministic quotient policy π̄ ⋆ attaining the maximum in the Bellman optimality equation at every quotient state. Define its lift by (π ⋆ )↑ (a | o, i) := π̄ ⋆ (a | o, [i]o ).
τω(0) := idCo0 , τω(j) := τej ◦ · · · ◦ τe1 .
We prove that, for every j = 0, . . . , k and every i ∈ [n], τω(j) ([i]o0 ) = [σω(j) (i)]oj . For j = 0, both prefix transports are identity maps, and therefore τω(0) ([i]o0 ) = idCo0 ([i]o0 ) = [i]o0 = [id[n] (i)]o0 = [σω(0) (i)]o0 .
Hence the claimed identity holds for the empty prefix. Now fix j ∈ {1, . . . , k} and suppose that the identity holds for the prefix ω (j−1) . Thus, for every i ∈ [n], τω(j−1) ([i]o0 ) = [σω(j−1) (i)]oj−1 . By the definitions of the prefix transports, τω(j) = τej ◦ τω(j−1) and σω(j) = σej ◦ σω(j−1) . Applying the quotient prefix transport to [i]o0 and then using the induction hypothesis gives τω(j) ([i]o0 ) = (τej ◦ τω(j−1) )([i]o0 ) = τej (τω(j−1) ([i]o0 )) = τej [σω(j−1) (i)]oj−1 . The element σω(j−1) (i) belongs to [n], so the one-edge descent identity for ej may be applied with r = σω(j−1) (i). It follows that τej [σω(j−1) (i)]oj−1 = σej (σω(j−1) (i)) o j = (σej ◦ σω(j−1) )(i) oj = [σω(j) (i)]oj . τω(j) ([i]o0 ) = [σω(j) (i)]oj . This completes the induction over all path prefixes. Taking j = k, and using o0 = o,
ok = o ,
ω
(k)
= ω,
together with τω = τek ◦ · · · ◦ τe1 and σω = σek ◦ · · · ◦ σe1 , we obtain, for every i ∈ [n], τω ([i]o ) = [σω (i)]o′ , as required.
I.9
σω ∈ Sn . Let
qo⋆ : [n] −→ Co , qo⋆ (i) = [i]o , be the canonical stable quotient map. Corollary C.2 gives, for every i ∈ [n], τω qo⋆ (i) = τω ([i]o ) = [σω (i)]o = qo⋆ σω (i) . Thus the raw and quotient transports satisfy the commuting relation τω ◦ qo⋆ = qo⋆ ◦ σω . We first show directly that τω is surjective. Let c′ ∈ Co be arbitrary. Since qo⋆ is the quotient map onto Co = [n]/Π⋆o , it is surjective. Hence there exists j ∈ [n] such that qo⋆ (j) = c′ . Because σω is bijective, its inverse σω−1 exists. Define i := σω−1 (j) and let
c := qo⋆ (i) ∈ Co .
By the definition of i, σω (i) = σω σω−1 (j) = j.
Combining the preceding equalities yields
′
Every σej belongs to Sn and is therefore bijective. A composition of bijections is bijective, so
Proof of Lemma 3.10
Proof. Fix an observation o ∈ O and a directed closed walk ω based at o. Since the initial and terminal observations of ω are both o, its raw transport is a self-map
Using the commuting relation, we therefore obtain τω (c) = τω qo⋆ (i) = qo⋆ σω (i) = qo⋆ (j) = c′ . Since c′ ∈ Co was arbitrary, every element of Co has a preimage under τω . Thus τω : Co −→ Co is surjective. The set Co is finite because it is the set of blocks of a partition of the finite set [n]. A surjective self-map of a finite set is injective. Indeed, suppose for contradiction that τω were not injective. Then there would exist two distinct classes c1 , c2 ∈ Co such that τω (c1 ) = τω (c2 ). Consequently, the image of τω would contain at most
σω : [n] −→ [n].
|Co | − 1
By Definition 3.2, σω is a composition of edge permutations. More explicitly, if
distinct elements, contradicting the already established surjectivity τω (Co ) = Co . Therefore τω is injective as well as surjective. Hence it is bijective, and τω ∈ Sym(Co ).
ω = e1 · · · ek , then σω = σek ◦ · · · ◦ σe1 .
Now consider the collection Holdir o = {τω : ω is a directed closed walk based at o} . The empty walk ∅o is a directed closed walk based at o. By Definition 3.2, σ∅o = id[n] . Applying Corollary C.2 to the empty walk gives, for every i ∈ [n], τ∅o ([i]o ) = [σ∅o (i)]o = [id[n] (i)]o = [i]o . Every element of Co is a block of Π⋆o and therefore has the form [i]o for some i ∈ [n]. It follows that τ∅o = idCo .
It remains to verify that the inverse of every element also belongs to Holdir o . Fix an arbitrary element g ∈ Holdir o . By definition, there exists a directed closed walk ω based at o such that g = τω . The first part of the proof shows that g ∈ Sym(Co ). Since Co is finite, the symmetric group Sym(Co ) is finite. Therefore the infinite sequence idCo , g, g 2 , g 3 , . . . cannot consist of pairwise distinct permutations. Hence there exist integers 0≤r<s such that
Consequently, idCo ∈ Holdir o . Let ω1 and ω2 be arbitrary directed closed walks based at o, where ω1 is executed first and ω2 is executed second. Their concatenation ω2 ◦ ω1 is again a directed walk beginning at o and ending at o, and hence is a directed closed walk based at o. By the composition convention in Definition 3.2, σω2 ◦ω1 = σω2 ◦ σω1 . For every i ∈ [n], Corollary C.2 therefore gives τω2 ◦ω1 ([i]o ) = [σω2 ◦ω1 (i)]o = [(σω2 ◦ σω1 )(i)]o = [σω2 (σω1 (i))]o . Applying the same corollary first to ω1 and then to ω2 gives
gr = gs . Because g is bijective, g r is also bijective. We may therefore compose both sides on the left with the inverse permutation (g r )−1 inside Sym(Co ), obtaining (g r )−1 ◦ g r = (g r )−1 ◦ g s , idCo = g s−r . Set m := s − r. Then m ≥ 1 and g m = idCo . It follows that
and likewise
g m−1 ◦ g = g m = idCo .
τω1 ([i]o ) = [σω1 (i)]o Therefore
and
g −1 = g m−1 .
τω2 [σω1 (i)]o = [σω2 (σω1 (i))]o . Combining these identities yields τω2 ◦ω1 ([i]o ) = [σω2 (σω1 (i))]o = τω2 [σω1 (i)]o = τω2 τω1 ([i]o )
g ◦ g m−1 = g m = idCo ,
Because Holdir o contains the identity and is closed under composition, it contains every nonnegative power of g. In particular, g m−1 ∈ Holdir o . More explicitly, for each integer ℓ ≥ 1, let
= (τω2 ◦ τω1 )([i]o ).
ω ⟨ℓ⟩
Since every element of Co has the form [i]o , equality on all such classes implies equality of the two maps. Therefore
denote the directed closed walk obtained by executing ω successively ℓ times, and let
τω2 ◦ω1 = τω2 ◦ τω1 .
ω ⟨0⟩ := ∅o .
Because ω2 ◦ ω1 is again a directed closed walk based at o, it follows that τω2 ◦ τω1 ∈ Holdir o . Thus Holdir o is closed under composition. Together with the presence of the identity, this shows that Holdir o is a submonoid of Sym(Co ).
Repeated application of the composition identity gives τω⟨ℓ⟩ = (τω )ℓ = g ℓ . Consequently, g −1 = g m−1 = τω⟨m−1⟩ ∈ Holdir o .
When m = 1, the exponent m−1 is zero and the inverse is induced by the empty walk. This construction does not require an executable edge-wise reversal of ω; the inverse permutation is realized algebraically by finitely many repetitions of the same directed closed walk. We have shown that Holdir o contains the identity permutation, is closed under composition, and contains the inverse of every one of its elements. Therefore
The HCDP transition law implies that the probability of this visible successor is independent of the current raw layer. Indeed, for every i ∈ [n], P(ot+1 | Ht , at , it = i) n X = PH (ot+1 , i′ ) | (ot , i), at
Holdir o ≤ Sym(Co ).
=
PO (ot+1 | ot , at )1 i′ = σot ,at ,ot+1 (i)
i′ =1
Finally, because Co is finite,
= PO (ot+1 | ot , at )
|Holdir o | ≤ |Sym(Co )| = |Co |! < ∞. Hence Holdir o is a finite subgroup of Sym(Co ), with the empty walk providing the identity.
I.10
i′ =1 n X
i′ =1
= PO (ot+1 | ot , at ). The final equality holds because
Proof of Proposition 3.11
Proof. Fix a time t and a positive-probability realization of the controller history Ht = (o0 , a0 , . . . , at−1 , ot ). All conditional probabilities below are taken under the policy generating the interaction. Let ct := [it ]ot ∈ Cot be the stable quotient class of the current latent layer, so that βt (c) = P(ct = c | Ht ),
σot ,at ,ot+1 (i) ∈ [n] is a single well-defined layer, so exactly one term in the indicator sum is equal to one. Because the right-hand side does not depend on i, the joint probability of the current class and the observed successor can be computed without conditioning on a possibly null class event. For every c ∈ Cot , P(ct = c, ot+1 | Ht , at ) X = P(it = i, ot+1 | Ht , at )
c ∈ Cot .
The action at is selected according to a distribution that is measurable with respect to Ht . Consequently, conditional on Ht , the policy randomization used to choose at does not depend on the hidden layer it or on its class ct . More explicitly, for every action a ∈ A satisfying P(at = a | Ht ) > 0 and every c ∈ Cot , P(ct = c | Ht , at = a) P(ct = c, at = a | Ht ) = P(at = a | Ht ) P(ct = c | Ht )P(at = a | Ht , ct = c) = P(at = a | Ht ) P(ct = c | Ht )P(at = a | Ht ) = P(at = a | Ht ) = P(ct = c | Ht ) = βt (c). The third equality follows from the definition of an admissible history-dependent policy: once Ht is given, the conditional distribution of at is πt (· | Ht ) and cannot depend additionally on the unobserved variable ct . Thus conditioning on the selected action does not change the current class belief. In the notation of the proposition, P(ct = c | Ht , at ) = βt (c) for every action value that can be selected after Ht . Now fix an observed successor ot+1 satisfying PO (ot+1 | ot , at ) > 0.
n X 1 i′ = σot ,at ,ot+1 (i)
i: [i]ot =c
=
X
P(ot+1 | Ht , at , it = i)P(it = i | Ht , at )
i: [i]ot =c
= PO (ot+1 | ot , at )
X
P(it = i | Ht , at )
i: [i]ot =c
= PO (ot+1 | ot , at )P(ct = c | Ht , at ) = PO (ot+1 | ot , at )βt (c). Summing this identity over all current classes gives X P(ot+1 | Ht , at ) = P(ct = d, ot+1 | Ht , at ) d∈Cot
=
X
PO (ot+1 | ot , at )βt (d)
d∈Cot
X
= PO (ot+1 | ot , at )
βt (d)
d∈Cot
= PO (ot+1 | ot , at ), because βt is a probability distribution on Cot . Bayes’ rule now shows that observing ot+1 introduces no class-dependent likelihood correction. For every c ∈ Cot , P(ct = c | Ht , at , ot+1 ) P(ct = c, ot+1 | Ht , at ) = P(ot+1 | Ht , at ) PO (ot+1 | ot , at )βt (c) = PO (ot+1 | ot , at ) = βt (c).
The positivity of PO (ot+1 | ot , at ) guarantees that this conditional probability is well defined. The assumption that reward samples are not used as withinepisode controller observations is used at this point. Under the convention of the proposition, the information available at the next decision time is Ht+1 = Ht · (at , ot+1 ), and therefore no realized reward sample is included in the conditioning information. If a reward sample were included, its class-conditional likelihood could produce an additional Bayes correction, which would require the stronger rewardoutput abstraction discussed in Appendix B. By Theorem 3.9, the stable class transition associated with the observed feasible edge is deterministic. More precisely, ct+1 = [it+1 ]ot+1 = [σot ,at ,ot+1 (it )]ot+1 = τot ,at ,ot+1 ([it ]ot ) = τot ,at ,ot+1 (ct ) almost surely conditional on (Ht , at , ot+1 ). Equivalently, for every c ∈ Cot and every c′ ∈ Cot+1 , P(ct+1 = c′ | Ht , at , ot+1 , ct = c) = 1 c′ = τot ,at ,ot+1 (c) whenever the conditioning event has positive probability. In the zero-probability case, the corresponding term in the lawof-total-probability expansion below vanishes. Using the conditional law of total probability over the finite set Cot , we obtain βt+1 (c′ ) = P(ct+1 = c′ | Ht+1 ) = P(ct+1 = c′ | Ht , at , ot+1 ) X = P(ct+1 = c′ | Ht , at , ot+1 , ct = c) c∈Cot
c∈Cot
X
βt (c).
c∈Cot : τot ,at ,ot+1 (c)=c′
This formula remains valid when the open-edge class transport is not injective. If several current classes are mapped to the same successor class, the posterior masses of all those classes are added. The right-hand side also defines a normalized probability distribution. Indeed, X X X βt+1 (c′ ) = 1 c′ = τot ,at ,ot+1 (c) βt (c) c′ ∈Cot+1
c′ ∈Cot+1 c∈Cot
=
X
βt (c)
X c∈Cot
= 1,
X c′ ∈Cot+1
c∈Cot
=
R(ot , it , at ) = R̄(ot , [it ]ot , at ) = R̄(ot , ct , at ) almost surely. Hence E[R(ot , it , at ) | Ht , at ] = E R̄(ot , ct , at ) | Ht , at X R̄(ot , c, at )P(ct = c | Ht , at ) = c∈Cot
=
X
βt (c)R̄(ot , c, at ).
c∈Cot
This proves the two displayed identities in the proposition. To make the belief-state sufficiency statement explicit, define, for every feasible triple (o, a, o′ ), the deterministic pushforward operator Φo,a,o′ : ∆(Co ) −→ ∆(Co′ ) by X
Φo,a,o′ β (c′ ) :=
β(c).
c∈Co : τo,a,o′ (c)=c′
The update established above can then be written as βt+1 = Φot ,at ,ot+1 βt . Moreover, for every o′ ∈ O, P(ot+1 = o′ | Ht , at = a) = PO (o′ | ot , a). Define the compressed belief-state space by
P(ct = c | Ht , at , ot+1 ) X = 1 c′ = τot ,at ,ot+1 (c) βt (c) =
because τot ,at ,ot+1 (c) is one uniquely defined element of Cot+1 for each c ∈ Cot . The stable quotient also makes the conditional mean reward depend only on the current class. By the representativeindependent reward identity established in Theorem 3.9,
βt (c)
1 c′ = τot ,at ,ot+1 (c)
B := {(o, β) : o ∈ O, β ∈ ∆(Co )} . For any current realization (ot , βt ) and action a, the conditional transition law of the next compressed belief state is therefore P (ot+1 , βt+1 ) = (o′ , β ′ ) | Ht , at = a = PO (o′ | ot , a)1{β ′ = Φot ,a,o′ βt } . The corresponding conditional mean reward is X R̄bel (ot , βt , a) := βt (c)R̄(ot , c, a). c∈Cot
Both the controlled transition law and the conditional mean reward depend on the complete observable history Ht only through the pair (ot , βt ). Since the next belief is obtained recursively from the current belief, the selected action, and the next observation, iteration of this update reproduces exactly the posterior distribution of the stable quotient class after every future observable continuation. Therefore (ot , βt ) is an exact belief state after the raw hidden layers have been compressed to stable classes, for the conditional-mean discounted control objective considered in the paper.
Finally, suppose that the current stable class is known exactly, so that βt = δct . For every c′ ∈ Cot+1 , the belief update gives X δct (c) βt+1 (c′ ) = c∈Cot : τot ,at ,ot+1 (c)=c′
= 1 τot ,at ,ot+1 (ct ) = c′ = δτot ,at ,ot+1 (ct ) (c′ ). Since the two probability distributions agree at every c′ ∈ Cot+1 , we conclude that βt+1 = δτot ,at ,ot+1 (ct ) .
sum equals one. Since the observed successor has positive probability, division by PO (ot+1 | ot , at ) is valid. Therefore P it+1 = j | Ht , it , at , ot+1 PO (ot+1 | ot , at )1 j = σot ,at ,ot+1 (it ) = PO (ot+1 | ot , at ) = 1 j = σot ,at ,ot+1 (it ) . Thus, conditional on the observed transition, it+1 = σot ,at ,ot+1 (it ) almost surely. Taking stable classes and using the representative-independent transport identity from Theorem 3.9, we obtain Ct+1 = [it+1 ]ot+1 = σot ,at ,ot+1 (it ) ot+1 = τot ,at ,ot+1 ([it ]ot )
Thus exact knowledge of the stable class, once available, remains exactly and recursively maintainable.
I.11
Proof of Lemma C.3
Proof. Let Ct := [it ]ot ∈ Cot denote the actual stable class of the latent layer at time t, and let Mt ∈ Cot denote the class label maintained by the controller. Suppose that the maintained label is correct at time t, so that Mt = Ct = [it ]ot . Fix a realized action at and a realized successor observation ot+1 having positive conditional probability. Then PO (ot+1 | ot , at ) > 0, so the triple (ot , at , ot+1 ) is feasible and the transport maps σot ,at ,ot+1 : [n] → [n]
and
τot ,at ,ot+1 : Cot → Cot+1
are well defined. By the HCDP transition kernel, for every candidate layer j ∈ [n], P it+1 = j, ot+1 | Ht , it , at = PH (ot+1 , j) | (ot , it ), at = PO (ot+1 | ot , at )1 j = σot ,at ,ot+1 (it ) .
= τot ,at ,ot+1 (Ct ) = τot ,at ,ot+1 (Mt ) almost surely. Consequently, if the controller performs the update Mt+1 := τot ,at ,ot+1 (Mt ), then Mt+1 = Ct+1 almost surely. Repeating this argument inductively shows that, for any time t0 , correctness at time t0 implies Mt = Ct for every t ≥ t0 almost surely along every realized continuation. Indeed, the equality is true at t = t0 by assumption, and the one-step implication just proved carries correctness from each time t to time t + 1. We now verify that the correctly tracked observation–class pair has exactly the transition law of the quotient MDP. Fix a positive-probability conditioning event on which ot = o, Ct = c, at = a. For any o′ ∈ O and c′ ∈ Co′ , the conditional law of total probability over the possible raw layers contained in the class c gives P ot+1 = o′ , Ct+1 = c′ | Ht , Ct = c, at = a X = P(it = i | Ht , Ct = c, at = a) i: [i]o =c
P(ot+1 | Ht , it , at ) n X = PH (ot+1 , j) | (ot , it ), at j=1 n X = PO (ot+1 | ot , at ) 1 j = σot ,at ,ot+1 (it ) j=1
= PO (ot+1 | ot , at ). The final equality holds because σot ,at ,ot+1 (it ) is a single element of [n], and hence exactly one term in the indicator
X
·
Marginalizing over the candidate next layer gives
PH (o′ , j) | (o, i), a .
j: [j]o′ =c′
For every representative i satisfying [i]o = c, the inner sum is X PH (o′ , j) | (o, i), a j: [j]o′ =c′
=
X
PO (o′ | o, a)1{j = σo,a,o′ (i)}
j: [j]o′ =c′
= PO (o′ | o, a)1{[σo,a,o′ (i)]o′ = c′ } = PO (o′ | o, a)1{τo,a,o′ ([i]o ) = c′ } = PO (o′ | o, a)1{τo,a,o′ (c) = c′ } .
When PO (o′ | o, a) = 0, both sides are zero and the value assigned to the corresponding transport is immaterial. When PO (o′ | o, a) > 0, the fourth equality follows directly from Theorem 3.9. In either case, the final expression is independent of the representative i. Substituting it into the preceding total-probability expansion yields P ot+1 = o′ , Ct+1 = c′ | Ht , Ct = c, at = a = PO (o′ | o, a)1{τo,a,o′ (c) = c′ } X P(it = i | Ht , Ct = c, at = a)
For zero-probability transitions, Uquot may be defined arbitrarily because those transitions are never realized. This recursion uses only the current stored class, the current observation, the selected action, and the newly observed successor. Since correctness is preserved by every feasible update, a class that is correctly initialized or correctly inferred once is a recursively maintainable finite memory state.
I.12
Proof of Corollary 3.12
Proof. For every observation o ∈ O, the stable class set Co = [n]/Π⋆o
i: [i]o =c ′
′
= PO (o | o, a)1{c = τo,a,o′ (c)} = P̄ (o′ , c′ ) | (o, c), a . Thus the conditional distribution of the next observation– class pair depends on the complete past only through the current pair (ot , Ct ) and the selected action, and it coincides exactly with the transition kernel of the quotient MDP. The one-step mean reward also agrees exactly with the quotient reward. Since Ct = [it ]ot , the representative-independent reward identity from Theorem 3.9 gives R(ot , it , at ) = R̄(ot , [it ]ot , at )
is finite and nonempty. Since O is finite, the quantity m⋆exact = max |Co | o∈O
is a well-defined positive integer. We first prove that no alphabet satisfying the requirements of the corollary can contain fewer than m⋆exact symbols. Fix an arbitrary observation o ∈ O. Because ηo : Co −→ M is injective, its restriction to the finite set Co maps distinct classes to distinct memory symbols. Hence |ηo (Co )| = |Co |. Since ηo (Co ) ⊆ M, it follows that
= R̄(ot , Ct , at ) almost surely. Hence both the controlled transition kernel and the mean reward of the correctly tracked process (ot , Mt ) agree with those of the quotient MDP whenever Mt = Ct . If the class is correctly initialized at time zero, so that M0 = C0 = [i0 ]o0 , then its initial joint distribution is also the quotient initial distribution. Indeed, for every (o, c) ∈ S̄, P(o0 = o, M0 = c) = P(o0 = o, C0 = c) X = P(o0 = o, i0 = i) i: [i]o =c
=
X
ρ0 (o, i)
i: [i]o =c
= ρ̄0 (o, c). Combining this initial-law identity with the transition and reward identities proves that the process (ot , Mt ) started from a correct initial class has exactly the controlled law and meanreward structure of the quotient MDP in Theorem 3.9. If the class is instead inferred correctly at a later time t0 , the same argument applies conditionally from time t0 onward, with the inferred class serving as the current quotient state. Finally, the update is a finite-memory recursion. The tagged class alphabet
|Co | = |ηo (Co )| ≤ |M|. This inequality holds for every o ∈ O, and therefore |M| ≥ max |Co | = m⋆exact . o∈O
Equivalently, if |M| < |Co | for some observation o, then the pigeonhole principle would give distinct classes c, d ∈ Co with ηo (c) = ηo (d), contradicting injectivity. Thus, when the visible observation is fixed at o, at least |Co | different memory symbols are necessary to retain exact knowledge of the stable class. This requirement is intrinsic to exact class tracking: by Theorem 3.8, two distinct stable classes cannot be merged by any exact observation-wise abstraction. We now construct an alphabet of exactly m⋆exact symbols satisfying all the required update identities. Let M⋆ := [m⋆exact ] = {1, . . . , m⋆exact }. For each o ∈ O, write ro := |Co |. By the definition of m⋆exact , ro ≤ m⋆exact . Choose an enumeration
Mquot := {(o, c) : o ∈ O, c ∈ Co } = S̄ is finite, and for every feasible observed transition one may define Uquot (o, c), o, a, o′ := o′ , τo,a,o′ (c) .
Co = {co,1 , . . . , co,ro } and define ηo⋆ (co,ℓ ) := ℓ,
ℓ = 1, . . . , ro .
For each fixed o, the labels 1, . . . , ro are distinct, so
The memory update then satisfies mt+1 = U ⋆ (mt , ot , at , ot+1 )
ηo⋆ : Co −→ M⋆ is injective. The same numerical symbol may be used by the encoders associated with different observations; for example, the symbol 1 may encode co,1 at observation o and coe,1 at observation oe. This causes no ambiguity because the maintained controller state is the pair consisting of the current observation and the memory symbol, and the update map receives the current observation o as an explicit argument. We define a total update map ⋆
⋆
⋆
U : M × O × A × O −→ M .
= U ⋆ ηo⋆t (ct ), ot , at , ot+1 = ηo⋆t+1 τot ,at ,ot+1 (ct ) = ηo⋆t+1 (ct+1 ).
Hence a correct encoded class remains correct after one update. Repeating the same argument inductively gives ms = ηo⋆s (cs )
for every later time s.
Therefore an alphabet of size
′
|M⋆ | = m⋆exact
Fix a tuple (m, o, a, o ). If PO (o′ | o, a) > 0
and
m ∈ ηo⋆ (Co ),
then the injectivity of ηo⋆ implies that there is a unique class c ∈ Co such that ηo⋆ (c) = m. For such a tuple, define U ⋆ (m, o, a, o′ ) := ηo⋆′ (τo,a,o′ (c)) .
is sufficient. Combining the lower bound valid for every admissible alphabet with the explicit construction above yields |M|min = m⋆exact = max |Co |. o∈O
The bound is a maximum rather than the sum X |Co |
This value belongs to M⋆ because τo,a,o′ (c) ∈ Co′ and ηo⋆′ maps Co′ into M⋆ . The definition is unambiguous because the class c satisfying ηo⋆ (c) = m is unique. If either PO (o′ | o, a) = 0
o∈O
because class labels need only be injective within each fixed observation fiber. The observation coordinate identifies which encoder ηo⋆ and which interpretation of a reused memory symbol are active, so the same alphabet symbols may be reused across different observations.
I.13
or m∈ / ηo⋆ (Co ), define, for example,
Proof of Theorem 4.2
Proof. Let the common observation and action spaces be O = {o},
U ⋆ (m, o, a, o′ ) := 1. These arbitrary values do not affect exact class tracking: a zero-probability transition is never realized, and a correctly encoded class symbol always lies in ηo⋆ (Co ). Let now (o, a, o′ ) be feasible and let c ∈ Co . Put m := ηo⋆ (c).
A = {a, b},
and let both HCDPs have two raw layers, so that n = 2. We construct two models, denoted by MA and MB . They have the same discount factor γ ∈ (0, 1), the same initial distribution ρ0 (o, 1) = ρ0 (o, 2) =
1 , 2
the same deterministic observation kernel
Then m ∈ ηo⋆ (Co ), and the unique class mapped to m by ηo⋆ is precisely c. Therefore the defining clause for U ⋆ gives U ⋆ ηo⋆ (c), o, a, o′ = ηo⋆′ (τo,a,o′ (c)) .
and the same raw transport on every feasible edge,
Thus U ⋆ satisfies the required compatibility identity for every feasible transition and every source class. To verify explicitly that this compatibility identity recursively maintains the class, suppose that at some time t the true class ct ∈ Cot is known and the memory is initialized as
Thus, under either action, the visible observation remains o and the raw layer remains unchanged. The two models differ only in their mean reward functions. In MA , set
mt = ηo⋆t (ct ). By Lemma C.3, after observing a feasible transition (ot , at , ot+1 ), the true next class is ct+1 = τot ,at ,ot+1 (ct ).
PO (o | o, a) = PO (o | o, b) = 1,
σo,a,o = σo,b,o = id[2] .
RA (o, 1, a) = RA (o, 2, a) = RA (o, 1, b) = RA (o, 2, b) = 0. In MB , set RB (o, 1, a) = RB (o, 2, a) = B B 0, R (o, 1, b) = 0, R (o, 2, b) = 1. Both are finite HCDPs in the sense of Definition 3.1. We may take the reward noise to be identically zero in both models, namely ξt = 0 almost surely for every t. This choice satisfies
Assumption C.1, because it is conditionally mean zero and, for every λ ∈ R, 2 2 λξt λ σY − E e | Gt = 1 ≤ exp . 2 We first compute the two stable quotients. In MA , layers 1 and 2 have the same mean reward under every action. Hence the reward partition at the unique observation is ΠA o = {1, 2} . Because both raw edge transports are the identity, for each u ∈ {a, b} we have σo,u,o (1) = 1,
σo,u,o (2) = 2,
and these two successor layers belong to the same block of ΠA o . Therefore the defining conditions of the stability operator give A 1 ≡To (Π ) 2. There is only one block to begin with, so this implies T (ΠA ) = ΠA . Thus ΠA is stable. Since no partition can be strictly coarser than the one-block partition, it is the coarsest stable family, and consequently ΠA,⋆ = {1, 2} , CoA = {c⋆ }, c⋆ = {1, 2}. o The quotient transports in MA are necessarily
The quotient rewards under action a are R̄B (o, c1 , a) = R̄B (o, c2 , a) = 0, whereas under action b they are R̄B (o, c1 , b) = 0,
The quotient initial distribution is 1 . 2 The two stable quotient descriptions cannot be observationwise gauge equivalent. Indeed, Definition 4.1 would require a bijection between the class sets at the unique observation, whereas |CoA | = 1 and |CoB | = 2. No bijection from a one-element set to a two-element set exists, so the required observation-wise gauge map cannot exist. Now define the common behavior policy by µt (a | Ht ) = 1, µt (b | Ht ) = 0, for every t ≥ 0 and every observable history Ht . Under either model, o0 = o almost surely. Suppose that at some time t the latent state is (o, it ). Since the behavior policy chooses a almost surely, the deterministic base kernel and identity transport imply B ρ̄B 0 (o, c1 ) = ρ̄0 (o, c2 ) =
ot+1 = o,
it+1 = σo,a,o (it ) = it
almost surely.
Moreover, both models assign mean reward zero to action a at both layers, and the reward noise is zero, so under MA , Yt = RA (o, it , a) + ξt = 0,
A A τo,a,o (c⋆ ) = τo,b,o (c⋆ ) = c⋆ ,
and under MB ,
the quotient rewards satisfy R̄A (o, c⋆ , a) = R̄A (o, c⋆ , b) = 0, and the quotient initial distribution satisfies
Yt = RB (o, it , a) + ξt = 0. Induction over t therefore gives, under both models, L∞ = (o, a, 0, o, a, 0, o, a, 0, . . .)
ρ̄A 0 (o, c⋆ ) = 1. In MB , the two layers have different mean rewards under action b, because RB (o, 1, b) = 0 ̸= 1 = RB (o, 2, b). Therefore they cannot belong to the same block of the reward partition, and the reward partition is the discrete partition ΠB o = {1}, {2} . Applying T cannot merge the two singleton blocks, since equality of rewards for every action is a necessary condition in Definition 3.4, and that condition already fails for action b. Hence T (ΠB ) = ΠB . It follows that ΠB,⋆ = {1}, {2} , CoB = o {c1 , c2 }, c1 = {1}, c2 = {2}. Since the raw transports are identities, the quotient transports are identities as well: B B τo,a,o (cj ) = τo,b,o (cj ) = cj ,
R̄B (o, c2 , b) = 1.
j ∈ {1, 2}.
almost surely.
Writing PµA and PµB for the two induced probability laws, we have the stronger identity
LPµA (L∞ ) = δ(o,a,0,o,a,0,...) = LPµB (L∞ ). Thus even the complete infinite passive log has exactly the same distribution in the two models. It remains to derive the statistical impossibility. Let LT b T be any estimator be any finite prefix of L∞ , and let Q measurable with respect to LT . The same argument also covers a randomized estimator by adjoining to LT an auxiliary random seed having the same independent law under both models. Consider the gauge-invariant functional K(Q) := |Co |, which is the number of stable classes at the unique observation, and define b T ). b T := K(Q K For the two constructed models, K(QA ) = 1,
K(QB ) = 2.
Because LT has the same law under PµA and PµB , the estimator output also has the same law. In particular, if b T = 1) = Pµ (K b T = 1) pT := PµA (K B and
b T = 2) = Pµ (K b T = 2), qT := PµA (K B b T = 1} and {K b T = 2} are disjoint, so then the events {K pT + qT ≤ 1. Consequently, n o b T ̸= 1), Pµ (K b T ̸= 2) = max{1 − pT , 1 − qT } max PµA (K B ≥1−
pT + qT 2
1 . 2 This lower bound holds for every sample length T . Hence no estimator can consistently recover the stable class structure at both MA and MB under the common behavior policy µ; a fortiori, no estimator can be uniformly consistent over the full HCDP class and all behavior policies. Finally, the same obstruction applies to quotient transports and to the quotient MDP. A quotient-transport description includes the source and target class sets on which each τe is defined, and therefore determines |Co |. Likewise, by Theorem 3.9, the quotient MDP has state space S̄ = (o′ , c) : o′ ∈ O, c ∈ Co′ . ≥
In the present one-observation construction,
Therefore, if (0)
i ̸≡Π j, o then it is not true that the two layers have equal rewards under every action. Consequently, there exists an action b ∈ A such that R(o, i, b) ̸= R(o, j, b). Choose ω = ∅o , ō = o. Since the empty-path transport is the identity, σ∅o (i) = i, and hence R ō, σ∅o (i), b = R(o, i, b) ̸= R(o, j, b) = R ō, σ∅o (j), b . The empty path has length zero, which is at most t = 0. Thus the desired statement holds at the initial refinement level. Now fix an integer t ≥ 0 and assume that the statement holds for Π(t) . Let o ∈ O and i, j ∈ [n] satisfy (t+1)
i ̸≡Π o
this is equivalent to i ̸≡To (Π
i ≡To (Π
Proof of Lemma 4.3
Proof. We prove a slightly stronger statement at the level of the refinement sequence. For every integer t ≥ 0, every observation o ∈ O, and every pair of layers i, j ∈ [n] satisfying (t)
j,
there exist a directed path ω beginning at o, of length at most t, and terminating at some observation ō, together with an action b ∈ A, such that R ō, σω (i), b ̸= R ō, σω (j), b . The path is allowed to be empty. For the empty path at o, we use the convention σ∅o = id[n] . We establish this statement by induction on t. Consider first t = 0. By the definition of the initial reward partition Π(0) , (0)
i ≡Π o
j
⇐⇒
R(o, i, a) = R(o, j, a)
(t)
)
j.
Definition 3.4 states that
Thus a uniformly consistent estimator of either the quotient transports or the quotient MDP would induce a uniformly consistent estimator of K(Q), contradicting the lower bound above. Therefore unrestricted passive logs do not uniformly identify stable classes, quotient transports, or the quotient MDP.
i ̸≡Π o
j.
Because the refinement sequence is defined recursively by Π(t+1) = T Π(t) ,
|S̄| = |Co |.
I.14
σ∅o (j) = j,
for every a ∈ A.
(t)
)
j
holds if and only if both of the following conditions hold: for every a ∈ A,
R(o, i, a) = R(o, j, a) and
(t)
σo,a,o′ (i) ≡Π o′ ′ for every pair (a, o ) satisfying
σo,a,o′ (j)
PO (o′ | o, a) > 0. Since i and j are not equivalent in T (Π(t) ), at least one of these two requirements fails. Suppose first that the immediate-reward requirement fails. Then there exists an action b ∈ A such that R(o, i, b) ̸= R(o, j, b). As in the base case, choose the empty path ω = ∅o and set ō = o. We then have R ō, σω (i), b = R o, σ∅o (i), b = R(o, i, b) ̸= R(o, j, b) = R o, σ∅o (j), b = R ō, σω (j), b .
The length of this path is zero and therefore is at most t + 1. It remains to consider the case in which the immediate rewards agree under every action, but the successorequivalence requirement fails. In that case there exist an action a ∈ A and an observation o1 ∈ O such that PO (o1 | o, a) > 0 and
(t)
σo,a,o1 (j). σo,a,o1 (i) ̸≡Π o1 The positivity condition means that a
e = (o − → o1 ) is an edge of the directed support graph X. Define the two successor layers i1 := σo,a,o1 (i),
j1 := σo,a,o1 (j).
The preceding failure of successor equivalence is exactly (t)
i1 ̸≡Π o1
⋆
j1 .
The induction hypothesis may therefore be applied at observation o1 to the pair i1 , j1 . It yields a directed path ν from o1 to some observation ō, of length at most t, and an action b ∈ A such that R ō, σν (i1 ), b ̸= R ō, σν (j1 ), b . Let ω be the directed path obtained by first traversing the edge e and then traversing ν. Thus, if ν = e2 · · · ek , we set ω = e e2 · · · ek . This is a directed path from o to ō, because the terminal observation of e is o1 , which is the initial observation of ν. By Definition 3.2, composition follows execution order, and therefore σω = σν ◦ σe ,
The path ω consists of the initial edge e followed by ν. Since ν has length at most t, the length of ω is at most 1 + t = t + 1. Thus the stronger statement holds for Π(t+1) . By induction, it holds for every t ≥ 0. We now apply this statement to the stable partition. Fix an observation o ∈ O and two distinct stable classes c, c′ ∈ Co , c ̸= c′ . Recall that Co = [n]/Π⋆o . Choose representatives i ∈ c, j ∈ c′ . Equivalently, c = [i]o , c′ = [j]o . ′ Because c and c are distinct blocks of the partition Π⋆o , their representatives are not equivalent under that partition:
σe = σo,a,o1 .
Applying this identity to i gives σω (i) = (σν ◦ σe )(i) = σν σe (i) = σν σo,a,o1 (i) = σν (i1 ). Likewise, σω (j) = (σν ◦ σe )(j) = σν σe (j) = σν σo,a,o1 (j)
= σν (j1 ). Substituting these identities into the reward inequality supplied by the induction hypothesis yields R ō, σω (i), b = R ō, σν (i1 ), b ̸= R ō, σν (j1 ), b = R ō, σω (j), b .
i ̸≡Π o j. By the definition of the stabilization depth, n o L⋆ = min L ≥ 0 : Π(L) = Π⋆ , and hence It follows that
Π(L⋆ ) = Π⋆ . (L⋆ )
i ̸≡Π j. o Applying the stronger statement with t = L⋆ , we obtain a directed path ω from o to some observation ō, of length at most L⋆ , and an action b ∈ A such that R ō, σω (i), b ̸= R ō, σω (j), b . It remains only to express this raw-layer distinction in quotient notation. By Corollary C.2, τω ([i]o ) = [σω (i)]ō . Since c = [i]o , this gives τω (c) = [σω (i)]ō . Applying the same corollary to j and using c′ = [j]o gives τω (c′ ) = [σω (j)]ō . Theorem 3.9 defines the quotient reward by R̄(ō, [r]ō , b) = R(ō, r, b) for every r ∈ [n], and proves that this value is independent of the chosen representative of the stable class. Therefore, R̄ ō, τω (c), b = R̄ ō, [σω (i)]ō , b = R ō, σω (i), b , whereas R̄ ō, τω (c′ ), b = R̄ ō, [σω (j)]ō , b = R ō, σω (j), b . Combining these identities with the raw-layer reward inequality gives R̄ ō, τω (c), b ̸= R̄ ō, τω (c′ ), b . The constructed path has length at most L⋆ , and when the distinction is already present in the immediate rewards, the construction correctly permits ω to be the empty path. This proves the lemma.
I.15
Proof of Theorem 4.6
Proof. Fix an observation o ∈ O and a latent state (o, i), and write c = [i]o ∈ Co . If Co contains only one stable class, then Assumption C.10 implies that Cbo also contains only one element, because λo : Cbo → Co is a bijection. In that case the nearest-prototype rule necessarily returns the unique local label, whose image under λo is c, and hence Pr(λo (b c) ̸= c | sstart = (o, i)) = 0. It therefore suffices to consider a non-singleton observation fiber. By Definition C.6, this implies do ≥ 1. Because λo is bijective, there exists a unique local label b b c⋆ := λ−1 o (c) ∈ Co corresponding to the true stable class. For each diagnostic coordinate ℓ ∈ {1, . . . , do }, define Gℓ := G(o, i; uℓ ) and m X b ℓ := G(o, b i; uℓ ) = 1 G W (s) (o, i; uℓ ). m s=1 Assumption C.5 ensures that the diagnostic mean depends on the raw layer i only through its stable class c = [i]o . Therefore Gℓ = Ḡ(o, c; uℓ ) = ϕo (c) ℓ . Consequently, b ℓ − Gℓ = ϕbo (i) − ϕo (c) . G ℓ
We first derive the required concentration inequality for one fixed coordinate. Define the centered repetition noise Zℓ,s := W (s) (o, i; uℓ ) − G(o, i; uℓ ), s ∈ {1, . . . , m}. Conditional on the event sstart = (o, i), Assumption C.8 states that the repetitions are independent, while Assumption C.9 gives, for every η ∈ R, 2 2 η ν E[exp(ηZℓ,s )|sstart = (o, i)] ≤ exp . 2 Since m X b ℓ − Gℓ = 1 G Zℓ,s , m s=1 conditional independence yields, for every η ∈ R, h i b ℓ − Gℓ ) sstart = (o, i) E exp η(G " ! # m η X = E exp Zℓ,s sstart = (o, i) m s=1 m i h η Y Zℓ,s sstart = (o, i) E exp m s=1 2 2 m Y η ν ≤ exp 2m2 s=1 2 2 η ν = exp . 2m
=
Thus, conditionally on the fixed latent start state, the b ℓ − Gℓ is (ν 2 /m)-sub-Gaussian. empirical-mean error G For any ε > 0 and any η > 0, Markov’s inequality gives b ℓ − Gℓ ≥ ε sstart = (o, i) Pr G b ℓ − Gℓ ) ≥ eηε sstart = (o, i) = Pr exp η(G i h b ℓ − Gℓ ) sstart = (o, i) ≤ e−ηε E exp η(G η2 ν 2 ≤ exp −ηε + . 2m The exponent on the right-hand side is minimized at mε η⋆ = 2 . ν Substituting this value gives mε2 b Pr Gℓ − Gℓ ≥ ε sstart = (o, i) ≤ exp − 2 . 2ν b ℓ yields Applying the same calculation to Gℓ − G mε2 b Pr Gℓ − Gℓ ≥ ε sstart = (o, i) ≤ exp − 2 . 2ν Combining the two one-sided inequalities, we obtain mε2 b Pr Gℓ − Gℓ ≥ ε sstart = (o, i) ≤ 2 exp − 2 . 2ν Set
∆ ε := √ . 8 do For each ℓ ∈ {1, . . . , do }, the preceding bound becomes ∆ b Pr Gℓ − Gℓ ≥ √ sstart = (o, i) 8 do m ∆2 ≤ 2 exp − 2 2ν 64do m∆2 = 2 exp − . 128ν 2 do Define the event E := ϕbo (i) − ϕo (c)
∆ < √ ∞ 8 do
.
Since ϕbo (i) − ϕo (c)
∞
b ℓ − Gℓ , = max G 1≤ℓ≤do
the union bound gives Pr(E c |sstart = (o, i)) do X ∆ b ≤ Pr Gℓ − Gℓ ≥ √ sstart = (o, i) 8 do ℓ=1 2 m∆ ≤ 2do exp − . 128ν 2 do
with x = ϕbo (i), y = ϕo (c), ϕo (c′ ), we obtain, on E,
No independence between different diagnostic coordinates is needed for this union bound. On the event E, the Euclidean error of the empirical fingerprint satisfies ϕbo (i) − ϕo (c)
do X
= 2
2
ϕbo (i) − ϕeo (b c′ )
2
≤
w=
≥ ∥ϕo (c) − ϕo (c′ )∥2
!1/2
− ϕbo (i) − ϕo (c)
b ℓ − Gℓ G
′
2
′
− ϕeo (b c ) − ϕo (c )
ℓ=1 do X
z = ϕeo (b c′ ),
2
ϕbo (i) − ϕo (c)
=
do ϕbo (i) − ϕo (c)
<
p
∆ do √ 8 do
=
∆ . 8
2
∆ ∆ >∆− − 8 8 3∆ . = 4
∞
ℓ=1
p
!1/2
∞
Thus, on E, the empirical fingerprint is strictly closer to the correct calibrated prototype than to every incorrect calibrated prototype: ϕbo (i) − ϕeo (b c⋆ )
The calibrated prototype corresponding to the true class satisfies, by Assumption C.10, ϕeo (b c⋆ ) − ϕo (c) = ϕeo (b c⋆ ) − ϕo λo (b c⋆ ) 2
′
< 2
∆ 3∆ < < ϕbo (i) − ϕeo (b c′ ) 4 4 2
⋆
for every b c ̸= b c . Hence the minimizer in Definition 4.5 is unique on E, and the nearest-prototype rule must return
2
b c=b c⋆ .
∆ ≤ , 8
It follows that λo (b c) = λo (b c⋆ ) = c
because λo (b c⋆ ) = c.
on E. Equivalently,
The triangle inequality therefore implies that, on E, ϕbo (i) − ϕeo (b c⋆ )
2
≤ ϕbo (i) − ϕo (c)
2
{λo (b c) ̸= c} ⊆ E c .
+ ϕo (c) − ϕeo (b c⋆ )
∆ ∆ + 8 8 ∆ = . 4 <
Now fix any competing local label b c′ ∈ Cbo ,
I.16
c′ := λo (b c′ ).
c′ = λo (b c′ ) ̸= λo (b c⋆ ) = c.
do ≥ 1.
Fingerprint separability in Assumption C.7 consequently gives ∥ϕo (c) − ϕo (c′ )∥2 ≥ ∆. Moreover, prototype calibration gives
2
Finite-sample and eventual diagnostic consequences
Proof. Fix a non-singleton observation fiber Co , a latent start state (o, i), and a confidence level δ ∈ (0, 1). Since Co is nonsingleton, Definition C.6 gives
Since λo is injective and b c′ ̸= b c⋆ ,
= ϕeo (b c′ ) − ϕo λo (b c′ )
Taking conditional probabilities and using the bound for E c gives Pr(λo (b c) ̸= c|sstart = (o, i)) ≤ Pr(E c |sstart = (o, i)) m∆2 ≤ 2do exp − , 128ν 2 do which is the claimed result.
b c′ ̸= b c⋆ ,
and define its associated canonical stable class by
ϕeo (b c′ ) − ϕo (c′ )
2
≤ 2
∆ . 8
Using the triangle inequality in the form ∥x − z∥2 ≥ ∥y − w∥2 − ∥x − y∥2 − ∥z − w∥2 ,
Moreover, Assumption C.7 gives ∆ > 0, and Assumption C.9 gives ν > 0. Hence ∆2 > 0, 128ν 2 do and every expression appearing below is well defined. Assume that 128ν 2 do 2do m≥ log . ∆2 δ
Multiplying both sides by the strictly positive quantity ∆2 /(128ν 2 do ) preserves the direction of the inequality and yields 2do ∆2 128ν 2 do m∆2 log ≥ · 128ν 2 do 128ν 2 do ∆2 δ 2do = log . δ Multiplying by −1 reverses the inequality, so m∆2 2do − ≤ − log . 128ν 2 do δ Because the exponential function is strictly increasing, exponentiating both sides gives 2do m∆2 ≤ exp − log . exp − 128ν 2 do δ Since do ≥ 1 and δ ∈ (0, 1), the quantity 2do /δ is strictly positive. Therefore, 2do δ exp − log = exp log δ 2do δ = . 2do It follows that m∆2 2do 2do exp − ≤ 2do exp − log 128ν 2 do δ δ = 2do · 2do = δ. Applying Theorem 4.6 to the latent start state (o, i), whose true stable class is [i]o , gives m∆2 . Pr(λo (b c) ̸= [i]o |sstart = (o, i)) ≤ 2do exp − 128ν 2 do Combining this inequality with the preceding bound yields Pr(λo (b c) ̸= [i]o |sstart = (o, i)) ≤ δ, which proves the claimed sample-complexity guarantee. Since the number of diagnostic repetitions must be an integer, it is sufficient in particular to choose 2do 128ν 2 do log . m= ∆2 δ For a singleton fiber, the local label set is also a singleton by the bijection in Assumption C.10, so the inference error is identically zero and no diagnostic repetitions are required.
I.17
Proof of Corollary B.12
Proof. Fix an observation o ∈ O. If the stable fiber Co is a singleton, then Assumption C.10 implies that the local label set Cbo is also a singleton, because λo : Cbo → Co
is a bijection. Hence every classification event at o returns the unique local label, whose image under λo is the unique stable class in Co . Therefore, Pr(λo (b co,k ) ̸= co,k ) = 0 for every classification event k. Thus the conclusion is immediate for a singleton fiber. In the remainder of the proof, assume that Co is non-singleton. By Definition C.6, this implies do ≥ 1, so every quantity appearing in the stated schedule is well defined. For each classification event k ≥ 2, let Io,k ∈ [n] denote the latent layer at the resettable start checkpoint of that event. The corresponding latent start state and true stable class are So,k := (o, Io,k )
co,k := [Io,k ]o .
and
Define the error event Eo,k := {λo (b co,k ) ̸= co,k } . Conditional on the event {Io,k = i}, the diagnostic procedure starts from the fixed latent state (o, i), uses mo,k repetitions for each diagnostic coordinate, and has true stable class [i]o . Therefore, Theorem 4.6 gives, for every i ∈ [n] such that Pr(Io,k = i) > 0, mo,k ∆2 Pr(Eo,k |Io,k = i) ≤ 2do exp − . 128ν 2 do The right-hand side is independent of the particular value of i. Since Io,k takes values in the finite set [n], the law of total probability yields X Pr(Eo,k ) = Pr(Eo,k |Io,k = i) Pr(Io,k = i) i∈[n]: Pr(Io,k =i)>0
X
≤
i∈[n]: Pr(Io,k =i)>0
mo,k ∆2 2do exp − Pr(Io,k = i) 128ν 2 do
mo,k ∆2 = 2do exp − 128ν 2 do
X
Pr(Io,k = i)
i∈[n]: Pr(Io,k =i)>0
mo,k ∆2 = 2do exp − . 128ν 2 do Assume now that, for some η > 0, 128ν 2 do [(1 + η) log k + log(2do )] . ∆2 Since ∆ > 0, ν > 0, and do ≥ 1, the factor mo,k ≥
∆2 128ν 2 do is strictly positive. Multiplying the preceding inequality by this positive factor preserves the direction of the inequality and gives mo,k ∆2 ∆2 128ν 2 do ≥ · [(1 + η) log k + log(2do )] 2 2 128ν do 128ν do ∆2 = (1 + η) log k + log(2do ).
Multiplying by −1 reverses the inequality: −
mo,k ∆2 ≤ −(1 + η) log k − log(2do ). 128ν 2 do
Because the exponential function is increasing, mo,k ∆2 exp − ≤ exp(−(1 + η) log k − log(2do )) . 128ν 2 do
Thus, almost surely, only finitely many local classification errors occur at the fixed observation o. If only finitely many classification events occur at o, then the number of errors at o is trivially finite, so the same conclusion remains valid. Finally, suppose that every required observation follows such a schedule. For each o ∈ O, define Bo := {Eo,k occurs infinitely often} . The preceding argument gives
Using exp(x+y) = exp(x) exp(y), exp(− log x) =
1 x
Pr(Bo ) = 0 for x > 0,
we obtain mo,k ∆2 2do exp − 128ν 2 do ≤ 2do exp(−(1 + η) log k − log(2do )) = 2do exp(−(1 + η) log k) exp(− log(2do )) 1 = 2do k −(1+η) 2do = k −(1+η) .
Combining this inequality with the unconditional error bound derived above gives Pr(λo (b co,k ) ̸= co,k ) = Pr(Eo,k ) ≤ k −(1+η) . Since η > 0, we have 1 + η > 1. Hence the corresponding p-series converges: ∞ X
k −(1+η) < ∞.
k=2
For completeness, this can also be verified by the integral comparison Z ∞ ∞ X k −(1+η) ≤ x−(1+η) dx 1
k=2
−η ∞ x = − η 1 1 = η < ∞. Therefore, ∞ X
Pr(Eo,k ) ≤
k=2
∞ X
k −(1+η) < ∞.
k=2
The first Borel–Cantelli lemma now implies Pr(Eo,k infinitely often) = 0. No independence assumption among the events Eo,k is required for this implication. Equivalently, with probability one, there exists a finite random index Ko such that λo (b co,k ) = co,k
for every k ≥ Ko .
for every o ∈ O. Since the observation space O is finite, ! [ X Pr Bo ≤ Pr(Bo ) = 0. o∈O
o∈O
Hence, with probability one, no observation incurs infinitely many classification errors. On this probability-one event, every observation contributes only finitely many errors, and because O is finite, their total number is finite: ∞ XX
1{λo (b co,k ) ̸= co,k } < ∞
almost surely.
o∈O k=2
Therefore, there are almost surely only finitely many local classification errors system-wide.
I.18
Proof of Theorem 4.9 a
Proof. Fix an arbitrary feasible edge e = (o − → o′ ) and an arbitrary local source label ĉ ∈ Cbo . Set c := λo (ĉ) ∈ Co and define the corresponding local target label by −1 ĉ⋆ := τeλ (ĉ) = λ−1 o′ τe (λo (ĉ)) = λo′ τe (c) . By the prototype-gauge construction, λo and λo′ are bijections, so both c and ĉ⋆ are uniquely determined. Index the transition-anchored samples associated with the fixed edge e by r = 1, 2, . . .. For the r-th such sample, let (o, ir ) be its exact source checkpoint, let (o′ , σe (ir )) be the paired target checkpoint supplied by Assumption 4.7, and write c− r := [ir ]o ,
c+ r := [σe (ir )]o′ .
By the definition of the deterministic quotient transport in Theorem 3.9, c+ r = [σe (ir )]o′ = τe ([ir ]o ) = τe (c− r ). b b Let b c− c+ r ∈ Co and b r ∈ Co′ denote the source and target labels inferred from these two paired checkpoints. The source inference is correct when − λo (b c− r ) = cr ,
and the target inference is correct when + λo′ (b c+ r ) = cr .
By hypothesis, only finitely many source- or target-class inference errors occur almost surely. In particular, after restricting attention to the samples associated with the fixed edge e, there are still almost surely only finitely many indices at which at least one of the two equalities above fails. Therefore, on an event of probability one, there exists a finite random index Rerr ≥ 1 such that, for every r ≥ Rerr , − λo (b c− r ) = cr
and
+ λo′ (b c+ r ) = cr .
We work on this probability-one event throughout the remainder of the proof. Assumption C.11, applied to the fixed edge e and the true source class c = λo (ĉ), states that the protocol produces infinitely many transition-anchored samples satisfying c− r = c.
Consequently, Ne(r) [ĉ, ĉ⋆ ] −→ ∞
We next show that every competing entry in the row indexed by ĉ stops changing after the finite index Rerr − 1. Let dˆ ∈ Cbo′ \ {ĉ⋆ } be an arbitrary competing target label. Suppose that some sample with index r ≥ Rerr increments an entry in the row indexed by ĉ. By the estimator update rule, this means that its inferred source label satisfies b c− r = ĉ. Because the source inference is correct at this index, c− c− r = λo (b r ) = λo (ĉ) = c.
Because only finitely many sample indices precede Rerr , the set I := r ≥ Rerr : c− r =c is also infinite. Fix any r ∈ I. Since the source inference is correct at every index not smaller than Rerr ,
The paired target checkpoint therefore has true quotient class − c+ r = τe (cr ) = τe (c).
− λo (b c− r ) = cr
=c = λo (ĉ).
Target correctness now implies + λo′ (b c+ r ) = cr = τe (c) = λo′ (ĉ⋆ ).
The map λo is injective, and hence b c− r = ĉ. For the same anchored sample, target correctness and the deterministic quotient-transport identity give + λo′ (b c+ r ) = cr
= τe (c− r ) = τe (c) = λo′ λ−1 o′ (τe (c))
Injectivity of λo′ yields ⋆ b c+ r = ĉ .
Hence, after Rerr − 1, every sample that increments the row indexed by ĉ must increment its correct column ĉ⋆ . No sample with index r ≥ Rerr can increment the competing entry ˆ It follows that, for every dˆ ̸= ĉ⋆ and every r ≥ Rerr , Ne [ĉ, d]. ˆ = N (Rerr −1) [ĉ, d]. ˆ Ne(r) [ĉ, d] e
= λo′ (ĉ⋆ ). Since λo′ is injective, it follows that ⋆ b c+ r = ĉ .
Thus every sample indexed by r ∈ I produces the paired inferred labels ⋆ (b c− c+ r ,b r ) = (ĉ, ĉ ) and therefore increments exactly the count Ne [ĉ, ĉ⋆ ]. (r)
For clarity, let Ne denote the count matrix after the first r transition-anchored samples associated with edge e have been incorporated. For every r ≥ Rerr , the update rule in Definition 4.8 gives Ne(r) [ĉ, ĉ⋆ ] ≥ Ne(Rerr −1) [ĉ, ĉ⋆ ] + |I ∩ {Rerr , . . . , r}| . Because I is infinite, |I ∩ {Rerr , . . . , r}| −→ ∞
as r −→ ∞.
as r −→ ∞.
If Cbo′ = {ĉ⋆ }, then ĉ⋆ is the only target label and is therefore the unique maximizer in the row indexed by ĉ at every time, so the result is immediate. Suppose henceforth that Cbo′ \{ĉ⋆ } is nonempty. Since Cbo′ is finite and every count at the finite index Rerr − 1 is finite, the random quantity B :=
max
ˆ Cb ′ \{ĉ⋆ } d∈ o
ˆ Ne(Rerr −1) [ĉ, d]
is finite. Moreover, the preceding stabilization identity implies that, for every r ≥ Rerr and every dˆ ̸= ĉ⋆ , ˆ ≤ B. Ne(r) [ĉ, d] On the other hand, Ne(r) [ĉ, ĉ⋆ ] −→ ∞. Therefore, there exists a finite random index R⋆ ≥ Rerr such that Ne(R⋆ ) [ĉ, ĉ⋆ ] > B.
(r)
The entry Ne [ĉ, ĉ⋆ ] is nondecreasing in r, because every estimator update only adds one to a count and never decreases any count. Thus, for every r ≥ R⋆ , Ne(r) [ĉ, ĉ⋆ ] ≥ Ne(R⋆ ) [ĉ, ĉ⋆ ] > B. Combining this inequality with the bound on every competing column gives, for every r ≥ R⋆ and every dˆ ∈ Cbo′ with dˆ ̸= ĉ⋆ , ˆ Ne(r) [ĉ, ĉ⋆ ] > B ≥ Ne(r) [ĉ, d]. Hence ĉ⋆ is the unique maximizer of the row ˆ dˆ 7−→ Ne(r) [ĉ, d] for every r ≥ R⋆ . In particular, the deterministic tie-breaking convention is no longer relevant after R⋆ , because no tie involving the correct column is possible. Definition 4.8 therefore gives τbe (ĉ) = ĉ⋆
The hypothesis of the lemma states that every edge used on the interval has the correct local-coordinate transport. Applied to the realized edge et , this gives the equality of maps τbet = λ−1 tgt(et ) ◦ τet ◦ λsrc(et ) . Evaluating both sides at b ct and using src(et ) = ot and tgt(et ) = ot+1 , we obtain b ct+1 = τbet (b ct ) = λ−1 ◦ τ ◦ λ (b ct ) e src(e ) t t tgt(et ) = λ−1 ct ))) . ot+1 (τet (λot (b Applying λot+1 to this identity yields h i λot+1 (b ct+1 ) = λot+1 λ−1 ct ))) ot+1 (τet (λot (b = τet (λot (b ct )) .
= τeλ (ĉ) = λ−1 o′ τe (λo (ĉ)) after the R⋆ -th anchored sample of edge e has been processed. The index R⋆ is almost surely finite, so the corresponding anchored sample occurs at an almost surely finite protocol time. Between two updates of Ne , the estimator τbe remains unchanged, and every subsequent update preserves the strict dominance of the correct column established above. The displayed equality therefore holds permanently after that finite time. Since the feasible edge e and the source label ĉ ∈ Cbo were arbitrary, the conclusion holds for every feasible edge and every local source label.
I.19
Proof of Lemma 4.11
λot0 (b ct0 ) = ct0 . Now fix any time t ≥ t0 such that both t and t + 1 belong to the same reset-free interval, and suppose that λot (b ct ) = ct . a
t et := (ot −→ ot+1 ) be the realized edge at time t. Since this edge is realized by the process, it is feasible, and by its definition,
src(et ) = ot
and
λot+1 : Cbot+1 −→ Cot+1 is a bijection. By the induction hypothesis, λot (b ct ) = ct , and therefore λot+1 (b ct+1 ) = τet (ct ). It remains only to identify the right-hand side with the actual successor class. By definition, ct = [it ]ot
and
ct+1 = [it+1 ]ot+1 . a
Proof. Fix an arbitrary realization of the process on the resetfree interval beginning at time t0 . The claim is pathwise and therefore does not require any probabilistic qualification. We prove by induction over the time indices contained in this interval that λot (b ct ) = ct . At the initial time t = t0 , this identity is exactly the assumed initialization condition,
Let
The cancellation in the second equality is valid because
tgt(et ) = ot+1 .
Because no reset occurs between times t and t + 1, Definition 4.10 updates the local label by propagation rather than by a new diagnostic initialization. Hence b ct+1 = τbet (b ct ).
t Along the realized edge et = (ot −→ ot+1 ), the hidden layer evolves according to
it+1 = σet (it ). The quotient transport induced by this raw hidden transition is τet ([i]ot ) = [σet (i)]ot+1 . Substituting i = it gives τet (ct ) = τet ([it ]ot ) = [σet (it )]ot+1 = [it+1 ]ot+1 = ct+1 . Combining the preceding identities, we conclude that λot+1 (b ct+1 ) = τet (ct ) = ct+1 . Thus correctness at time t implies correctness at time t + 1 whenever both times lie in the same reset-free interval. Since correctness holds at the initial time t0 , induction yields λot (b ct ) = ct for every t ≥ t0 belonging to that interval.
I.20
Proof of Proposition 4.12
is almost surely finite. Define
Proof. Let 0 ≤ ζ0 < ζ1 < ζ2 < · · · be the successive exogenous initialization times. By assumption, these times are almost surely unbounded, so the sequence is almost surely infinite and ζj → ∞. For each j ≥ 0, let n o Ejinit := λoζj (b cζj ) ̸= cζj be the event that the diagnostic initialization at time ζj is incorrect. The assumed summability of the initialization error probabilities gives ∞ X
P(Ejinit ) < ∞.
j=0
The first Borel–Cantelli lemma therefore implies P Ejinit occurs infinitely often = 0. Hence, on an event of probability one, there exists a finite random index Jcls such that λoζj (b cζj ) = cζj
for every j ≥ Jcls .
T := ζJ . Then T ≥ Tcls , so the initialization at time T is correct: λoT (b cT ) = cT . Also T ≥ Ttr , so every transport used at or after time T is the true quotient transport expressed in the fixed local gauges. Consider the reset-free interval beginning at ζJ = T . Its initial label is correct, and every transport used on the interval is correct. Lemma 4.11 therefore gives λot (b ct ) = ct throughout that interval. Now let j > J. Since ζj ≥ T ≥ Tcls , the initialization at ζj is also correct; since ζj ≥ T ≥ Ttr , all transports used on the subsequent reset-free interval are correct. Applying Lemma 4.11 again yields exact class tracking on that entire interval. The reset-free intervals beginning at ζJ , ζJ+1 , ζJ+2 , . . . cover every time t ≥ T . Therefore λot (b ct ) = ct
All random times used above are finite on an event of probability one, which proves the proposition.
Set
I.21
Tcls := ζJcls . Then every exogenous initialization occurring at or after Tcls is correct. We next obtain a common time after which every transport entry that can be used by the tracker is correct. Let n o a P := (e, ĉ) : e = (o − → o′ ) ∈ E, ĉ ∈ Cbo .
Proof. Define
The set P is finite because E and every local label set are (t) finite. Write τbe for the transport estimate available to the tracker at time t. By the assumed eventual recovery of every required transport entry, for each (e, ĉ) ∈ P there exists an almost surely finite random time Te,ĉ such that, for every t ≥ Te,ĉ , , τbe(t) (ĉ) = λ−1 tgt(e) τe λsrc(e) (ĉ) and the equality remains true permanently. Since P is finite, the random time Ttr := max Te,ĉ (e,ĉ)∈P
is almost surely finite. Consequently, for every t ≥ Ttr and every feasible edge e, τbe(t) = λ−1 tgt(e) ◦ τe ◦ λsrc(e) as maps on Cbsrc(e) . Let T⋆ := max{Tcls , Ttr }. Because the initialization times are unbounded, the random index J := min{j ≥ 0 : ζj ≥ T⋆ }
for every t ≥ T.
Proof of Lemma E.2 Λλ (o, ĉ) := (o, λo (ĉ)).
Because every λo : Cbo → Co is bijective, Λλ : Sb → S̄ is bijective, with inverse −1 Λ−1 λ (o, c) = (o, λo (c)).
The policy transformation π̄(a | o, c) = π b(a | o, λ−1 o (c)) is therefore well defined and bijective between stationary policies on Sb and stationary policies on S̄. In particular, π̄ a | Λλ (o, ĉ) = π b(a | o, ĉ). By Definition 5.1, rewards are preserved: bλ (o, ĉ, a) = R̄ Λλ (o, ĉ), a . R The transition kernels are preserved as well. For ẑ = (o, ĉ) and ẑ ′ = (o′ , ĉ′ ), if PO (o′ | o, a) = 0, both transition probabilities are zero. If PO (o′ | o, a) > 0, then λ Pbλ (ẑ ′ | ẑ, a) = PO (o′ | o, a)1 ĉ′ = τo,a,o ′ (ĉ) ′ ′ = PO (o | o, a)1 λo′ (ĉ ) = τo,a,o′ λo (ĉ) = P̄ (Λλ (ẑ ′ ) | Λλ (ẑ), a) . For a bounded function f : S̄ → R, define its pullback by (Uλ f )(ẑ) := f (Λλ (ẑ)). Let T̄π̄ and Tbπb be the policy-evaluation Bellman operators of the canonical and local-coordinate quotient MDPs, respectively. Using the policy, reward, and transition identities
above, and then changing variables through the bijection Λλ , gives, for every bounded f ,
Equivalently, 1At 1{b zt+1 = (o′ , ĉ′ )}
Tbπb (Uλ f ) = Uλ (T̄π̄ f ).
= 1At 1{ot+1 = o′ }1 ĉ′ = τoλt ,at ,o′ (b ct ) .
Both Bellman operators are γ-contractions on their finite state spaces. Their unique fixed points therefore satisfy
Taking conditional expectation with respect to Gt− is valid because At ∈ Gt− . The HCDP transition law gives
Vbπb = Uλ V̄π̄ .
P(ot+1 = o′ | Gt− ) = PO (o′ | ot , at ),
b Equivalently, for every (o, ĉ) ∈ S,
so
Vbπb (o, ĉ) = V̄π̄ o, λo (ĉ) .
1At P zbt+1 = (o′ , ĉ′ ) | Gt−
= 1At PO (o′ | ot , at )1 ĉ′ = τoλt ,at ,o′ (b ct )
The same argument applies to the optimality operators. If T̄ and Tb denote the two Bellman optimality operators, then Tb(Uλ f ) = Uλ (T̄ f ) for every bounded f , because the action set is unchanged and the maximum is taken over the same actions on both sides. Uniqueness of the optimal fixed points yields b∗
∗
V = Uλ V̄ . Hence
Vb ∗ (o, ĉ) = V̄ ∗ o, λo (ĉ) .
Finally, the policy transformation is bijective and preserves every statewise value. Thus π b attains Vb ∗ at every local state ∗ if and only if π̄ attains V̄ at every corresponding canonical state. Therefore π b is optimal if and only if π̄ is optimal.
I.22
= 1At Pbλ ((o′ , ĉ′ ) | zbt , at ) . If PO (o′ | ot , at ) = 0, then P(ot+1 = o′ | Gt− ) = 0, and both transition probabilities are zero by the zero-support b branch in Definition 5.1. Thus, for every ẑ ′ ∈ S, P zbt+1 = ẑ ′ | Gt− = Pbλ (ẑ ′ | zbt , at ) almost surely on At . We next verify the reward law. Assumption C.1 gives Yt = R(ot , it , at ) + ξt ,
2 2 η σY − . E exp(ηξt ) | Gt ≤ exp 2 On At , reward consistency of the stable quotient implies R(ot , it , at ) = R̄(ot , ct , at ) = R̄ ot , λot (b ct ), at λ b (b =R zt , at ).
On At , ct = [it ]ot = λot (b ct ). Suppose first that PO (o′ | ot , at ) > 0, and write a
t e = (ot −→ o′ ).
On the event At ∩ {ot+1 = o′ }, the tracker recursion and the definition of At give b ct+1 = τbe (b ct ) = τeλ (b ct ) = λ−1 ct ) o′ τe λot (b = λ−1 o′ (τe (ct )) . By Theorem 3.9, on {ot+1 = o′ }, ct+1 = τe (ct ). ′
Hence, on At ∩ {ot+1 = o }, b ct+1 = λ−1 o′ (ct+1 ).
E[ξt | Gt− ] = 0,
and, for every η ∈ R,
Proof of Theorem 5.2
Proof. Fix a deterministic time t and a candidate successor b By definition, At is determined by the ẑ ′ = (o′ , ĉ′ ) ∈ S. current latent quotient class, the current learned tracker state, the transport tables available at time t, and the selected action at . Therefore At ∈ Gt− .
Therefore bλ (b E[Yt | Gt− ] = R zt , at ) almost surely on At . Moreover, on At , bλ (b Yt − R zt , at ) = ξt , so, for every η ∈ R, 2 2 h i η σY bλ (b E exp η Yt − R zt , at ) Gt− ≤ exp 2 almost surely on At . It remains to prove the model isomorphism. Define Λλ (o, ĉ) := (o, λo (ĉ)). Because each λo is bijective, Λλ is a bijection from Sb to S̄. For every ẑ = (o, ĉ) and action a, bλ (ẑ, a) = R̄ Λλ (ẑ), a . R
For ẑ ′ = (o′ , ĉ′ ), if PO (o′ | o, a) = 0, both kernels are zero. If PO (o′ | o, a) > 0, then λ Pbλ (ẑ ′ | ẑ, a) = PO (o′ | o, a)1 ĉ′ = τo,a,o ′ (ĉ) ′ ′ = PO (o | o, a)1 λo′ (ĉ ) = τo,a,o′ λo (ĉ) = P̄ (Λλ (ẑ ′ ) | Λλ (ẑ), a) . The initial laws satisfy ρbλ0 (ẑ) = ρ̄0 Λλ (ẑ) . Thus Λλ preserves the action set, rewards, transition kernel, discount factor, and canonical initial law, and is therefore an MDP isomorphism. Now suppose that an almost surely finite T0 satisfies At for every t ≥ T0 . For each deterministic t, the transition, reward, and sub-Gaussian identities above are equalities of conditional random variables on the Gt− -measurable event At . Since {T0 ≤ t} ⊆ At , the same identities hold on {T0 ≤ t}. This establishes the post-T0 conclusions relative to the ambient filtration; no stopping-time property of T0 is needed for this implication because the conditional identities were first proved on At itself. Assume additionally that T0 is a stopping time for (Fbt− ). Let Bt := {T0 ≤ t}. Then Bt ∈ Fbt− ⊆ Gt− . Since the right-hand sides of the transition and reward identities are Fbt− -measurable, the tower property gives 1Bt P zbt+1 = ẑ ′ | Fbt− h i = E 1B P zbt+1 = ẑ ′ | Gt− Fbt− t
= 1Bt Pbλ (ẑ ′ | zbt , at ) , and similarly bλ (b zt , at ). 1Bt E[Yt | Fbt− ] = 1Bt R Applying the same tower-property argument to the exponential moment yields h i bλ (b 1Bt E exp η Yt − R zt , at ) Fbt− 2 2 η σY . ≤ 1Bt exp 2 Hence the same controlled-MDP laws hold after T0 relative to the learned information filtration. Finally, ρbλ0 is the pullback of the canonical quotient law at the original time zero, whereas the law of zbT0 is induced by the controlled trajectory and the synchronization time. In general, L(b zT0 ) ̸= ρbλ0 , and the theorem does not require these distributions to coincide.
I.23
Proof of Corollary 5.3
Proof. Under Assumption C.10, for every observation o ∈ O the diagnostic interface provides a finite local label set Cbo together with an unknown bijection λo : Cbo −→ Co . These objects are fixed components of the calibrated diagnostic interface: successive diagnostic calls at observation o return labels in the same set Cbo , and correctness is always evaluated through the same map λo . In particular, the local label names may be arbitrary and may be reused at different observations, but their observation-wise gauges do not vary with time. Since O is finite and every Cbo is finite, the local-coordinate state space G Sb = {o} × Cbo o∈O
is finite. The diagnostic assumptions of Section 4 include the classstability, fingerprint-separation, resettable-repeatability, subGaussian-noise, and calibrated-prototype conditions required by Theorem 4.6, together with the summable diagnostic schedules of Corollary B.12. Therefore, at every required observation, the probability of an incorrect diagnostic classification is summable over successive classification events. Corollary B.12 consequently implies that, almost surely, only finitely many diagnostic calls satisfy λo (b c) ̸= c. Because the observation space is finite, this conclusion holds system-wide: on an event Ωdiag with P(Ωdiag ) = 1, there are only finitely many incorrectly classified initialization checkpoints and only finitely many incorrectly classified source or target checkpoints used by the edge-transport estimator. Under Assumptions 4.7 and C.11, the latter finite-error property is precisely the classification condition required by Theorem 4.9. Hence, for every feasible edge a
e = (o − → o′ ) and every source label ĉ ∈ Cbo , there exists an almost surely finite random time after which τbe (ĉ) = λ−1 τe λo (ĉ) o′ holds permanently. Equivalently, after that time the recovered row agrees with the true quotient transport expressed in the fixed local gauges. Since there are only finitely many feasible edges and finitely many local source labels, these row-wise recovery events may be intersected and their recovery times may be maximized. Thus there is an event Ωtr of probability one and an almost surely finite random time Ttr such that, on Ωtr , every transport required by the tracker satisfies τbe = λ−1 tgt(e) ◦ τe ◦ λsrc(e)
for all times at least Ttr , and every such equality remains valid thereafter. Every exogenous initialization of HMRL-D uses the diagnostic schedule above, and between exogenous resets its label is propagated according to Definition 4.10. The hypotheses of Proposition 4.12 are therefore satisfied. That proposition provides an event Ωsync with P(Ωsync ) = 1 and an almost surely finite random time T such that, on Ωsync , λot (b ct ) = ct for every t ≥ T. By the definition of the canonical quotient state, ct = [it ]ot .
A union bound over the K classifications therefore shows that, conditional on Eproto , all classifications are correct with probability at least 1−δcls . Hence the joint event consisting of prototype accuracy and correct classification has probability at least 1 − δproto − δcls . Condition on this event. Because every calibration table starts at zero and every source and target label in every anchored pair is correct, all samples in the row associated with a source label ĉ have the same target label λ−1 tgt(e) τe (λsrc(e) (ĉ)) . Coverage supplies at least one such sample for every feasible edge–class row. Therefore the row-wise majority rule of Definition 4.8 recovers
Consequently, on the probability-one event
τbe = λ−1 tgt(e) ◦ τe ◦ λsrc(e)
Ω⋆ := Ωdiag ∩ Ωtr ∩ Ωsync , we have λot (b ct ) = ct = [it ]ot
for every t ≥ T.
The intersection still has probability one because it is a finite intersection of probability-one events: P(Ωc⋆ ) = P Ωcdiag ∪ Ωctr ∪ Ωcsync ≤ P(Ωcdiag ) + P(Ωctr ) + P(Ωcsync ) = 0. We now set T0 := max{T, Ttr }. This time is almost surely finite. For every t ≥ T0 , the classtracking identity gives λot (b ct ) = ct , while the definition of Ttr gives τbe = λ−1 tgt(e) ◦ τe ◦ λsrc(e) for every feasible edge whose table may be used at time t. Therefore {T0 ≤ t} ⊆ At for every t. The eventual clause of Theorem 5.2 now applies and yields the local-coordinate quotient transition law, conditional reward mean, and, under Assumption C.1, the conditional sub-Gaussian noise bound. The map Λλ is only a proof-level relabeling: HMRL-D supplies (ot , b ct ) directly to its policy and update rules and never needs a cross-observation synchronization of numerical class names. This proves the corollary.
I.24
Proof of Corollary E.3
Proof. If O+ = ∅, every quotient fiber is a singleton, so no classification or transport-row ambiguity remains and the conclusion is immediate. Assume henceforth that O+ ̸= ∅ and condition on Eproto . For a diagnostic classification performed at observation o, Theorem 4.6 gives m∆2 Pr(classification error) ≤ 2do exp − . 128ν 2 do Since do ≤ d and the function x 7→ x exp(−c/x) is increasing on (0, ∞) for every c > 0, the chosen value of m implies m∆2 δcls Pr(classification error) ≤ 2d exp − ≤ . 2 128ν d K
on every row. Every episode initialization counted among the K classifications is also correct, so Lemma 4.11 gives exact tracking throughout each post-calibration episode. Hence At holds at every post-calibration time. Theorem 5.2 and Corollary 5.4 then give the desired backbone guarantee on the all-correct event. Union bounding its original failure event with the calibration failure event increases the failure probability by at most δ.
I.25
Proof of Corollary 5.4
Proof. Let T be the learned-information stopping time in the corollary. Since At holds for every t ≥ T , Theorem 5.2 gives, relative to (Fbt− ) after T , the transition kernel Pbλ , bλ , and, when required, the condiconditional reward mean R tional sub-Gaussian bound with variance proxy σY2 . Thus the post-T stream satisfies the MDP-side probabilistic conditions assumed by the backbone theorem. This conclusion does not by itself verify exploration, visitation, sampling, step-size, optimization, realizability, concentrability, or function-approximation assumptions; those method-specific conditions remain in force. Suppose first that A is initialized or restarted at a time T ≥ T0 and that all algorithmic information produced before T is discarded. In particular, any parameter values, value tables, replay data, transition counts, eligibility traces, targetnetwork states, optimizer states, or step-size counters that could have been affected by the pre-T representation are either reinitialized or excluded from subsequent updates. For t ≥ T , define the post-restart history z b HT,t := σ zbT , aT , YT , zbT +1 , . . . , at−1 , Yt−1 , zbt . z b Because HT,t ⊆ Ftzb, the tower property and the transition identity above imply z b P zbt+1 = ẑ ′ HT,t , at h i z b = E P zbt+1 = ẑ ′ Ftzb, at HT,t , at h i z b = E Pbλ (ẑ ′ |b zt , at ) HT,t , at
= Pbλ (ẑ ′ |b zt , at ) ,
where the last equality holds because Pbλ (ẑ ′ | zbt , at ) is meaz b surable with respect to σ(b zt , at ) ⊆ σ(HT,t , at ). The same argument for the reward gives i i i h h h z b z b , at , at = E E Yt Ftzb, at HT,t E Yt HT,t h i z b bλ (b =E R , at zt , at ) HT,t
has a failure probability at most δ uniformly over the length and contents of the corrupted prefix, then P(failure) = ≤
= R (b zt , at ) .
Consequently, from the viewpoint of the restarted backbone, the complete data stream beginning at time T is a valid conb λ . Its initial distribution is the actual trolled trajectory of M law of zbT , which need not be the canonical law ρbλ0 . Therefore, if the original theorem for A is uniform over initial distributions, it applies immediately; if it assumes a particular class of initial distributions, the actual post-restart law must belong to that class as part of the retained method-specific assumptions. Since no pre-T data or update remains in the restarted algorithmic state, the hypotheses of the finite-MDP guarantee apply exactly to this post-T execution, and the conclusion b λ. of that guarantee follows for M Suppose next that the backbone is not restarted, but its original theorem is invariant to an arbitrary finite prefix of corrupted states, transitions, or updates. On the probabilityone event on which T0 < ∞, the tuples and updates with indices 0, 1, . . . , T0 − 1 form a finite prefix, while every tuple from index T0 onb λ. ward satisfies the transition and reward conditions of M The prefix may have changed the internal state of the algorithm in an arbitrary way; this is precisely why invariance must cover corrupted updates and not merely the deletion of finitely many observations from a statistical record. By the assumed finite-prefix invariance, replacing the first T0 states, transitions, rewards as assigned to learned states, or updates by arbitrary quantities does not alter the conclusion of the original theorem once the remaining stream is valid and all of its other assumptions hold. Hence the same guarantee again b λ. follows for M The randomness of T0 creates no additional difficulty because time is discrete and {T0 < ∞} =
∞ [
{T0 = m}
m=0
has probability one. On the event {T0 = m}, only the first m indices can be corrupted, and the assumed invariance applies to that finite value of m. For example, if the original theorem
m=0 ∞ X
P (failure|T0 = m) P(T0 = m) δ P(T0 = m)
m=0
bλ
If the conditional sub-Gaussian condition is required, another application of the same tower argument yields h h i i z b bλ (b E exp η Yt − R zt , at ) HT,t , at h h h i i i z b bλ (b = E E exp η Yt − R zt , at ) Ftzb, at HT,t , at 2 2 η σY . ≤ exp 2
∞ X
= δ. An almost-sure conclusion is handled similarly by applying finite-prefix invariance on each event {T0 = m} and then taking their countable union. It remains to transfer the resulting guarantee from local coordinates to the canonical quotient coordinates. Define Λλ (o, ĉ) = o, λo (ĉ) . bλ By Theorem 5.2, this map is an MDP isomorphism from M to M: it is bijective and satisfies bλ (ẑ, a) = R̄ (Λλ (ẑ), a) R and
Pbλ (ẑ ′ |ẑ, a) = P̄ (Λλ (ẑ ′ )|Λλ (ẑ), a) .
b define If the backbone produces a stationary policy π b on S, its gauge-transformed policy on S̄ by π̄(a | z) := π b a Λ−1 λ (z) . b Lemma E.2 gives, for every ẑ ∈ S, Vbπb (ẑ) = V̄π̄ (Λλ (ẑ)) and
Vb ⋆ (ẑ) = V̄ ⋆ (Λλ (ẑ)) .
In particular, Vbπb − Vb ⋆
∞
= max V̄π̄ (Λλ (ẑ)) − V̄ ⋆ (Λλ (ẑ)) ẑ∈Sb
= max V̄π̄ (z) − V̄ ⋆ (z) z∈S̄
= V̄π̄ − V̄ ⋆ ∞ , where the middle equality uses bijectivity of Λλ . More generally, the isomorphism preserves actions, reward observations, transition probabilities, occupancy measures after the corresponding pushforward of the state distribution, and the ordering of policy values. Therefore any guarantee stated in terms of optimality, value error, return, regret, visitation, or sample complexity transfers after the states and policies are relabeled through Λλ , provided that every algorithmspecific assumption appearing in the original theorem is also preserved or separately verified in the local coordinates. This proves the positive transfer claims in both cases (i) and (ii). Finally, eventual correctness alone cannot remove an error already stored permanently in the internal state of a backbone. To see this explicitly, consider the finite discounted MDP with S0 = {s0 , s1 },
A0 = {a⋆ , a− },
On Bt , eventual class correctness holds at both t and t + 1:
deterministic self-loop transitions P0 (sj | sj , a) = 1
b ct = λ−1 ot (ct ),
for every j ∈ {0, 1} and a ∈ A0 ,
Fix a candidate successor
and deterministic rewards R0 (sj , a⋆ ) = 1,
b ẑ ′ = (o′ , ĉ′ ) ∈ S.
R0 (sj , a− ) = 0.
The action a⋆ is uniquely optimal at both states, and its value is ∞ X 1 V ⋆ (sj ) = . γk = 1−γ
If PO (o′ | ot , at ) > 0, then on Bt ∩{ot+1 = o′ }, the quotient transition identity gives b ct+1 = λ−1 o′ (ct+1 ) = λ−1 o′ (τot ,at ,o′ (ct ))
k=0
Define a backbone that selects a⋆ for its first action and then stores one permanent bit. It sets this bit to “good” if its first recorded update has the valid form (sj , a⋆ , 1, sj ) for some j ∈ {0, 1}, and otherwise sets the bit to “bad.” It selects a⋆ forever in the good mode and a− forever in the bad mode. On every uncorrupted trajectory of this MDP, the first recorded update necessarily has the valid form above. The backbone therefore remains in the good mode, selects the unique optimal action forever, and has a correct finite-MDP optimality guarantee on the uncorrupted model. Now let the true quotient state be s0 at every time, but suppose that the learned representation is incorrect only at time zero, with zb0 = s1 ,
zbt = s0
for every t ≥ 1.
Thus one may take T0 = 1, and the representation is exact permanently after that time. The true first action, reward, and successor state are a⋆ , 1, and s0 , respectively, but the first update recorded by the backbone is (s1 , a⋆ , 1, s0 ), which is not a valid self-loop tuple. The backbone enters the bad mode and thereafter selects a− forever, even though every state and transition presented from time one onward is correct. Its post-transient value at the true state is then V (s0 ) =
∞ X
γk 0 = 0 <
k=0
1 = V ⋆ (s0 ). 1−γ
A restart after time one would delete the poisoned bit, and a theorem invariant to finite corrupted updates would exclude this failure, but eventual representation correctness by itself does neither. Hence, without condition (i), condition (ii), or some other assumption that removes or neutralizes all effects of the finite corrupted prefix, the finite-MDP guarantee need not transfer.
I.26
b ct+1 = λ−1 ot+1 (ct+1 ).
Proof of Theorem E.4
Proof. Fix a deterministic time t ≥ 0 and let Bt := {T0 ≤ t}. Because T0 is a stopping time for (Gt− ), Bt ∈ Gt− .
= λ−1 ct ) o′ τot ,at ,o′ λot (b = τoλt ,at ,o′ (b ct ). Therefore 1Bt 1{b zt+1 = (o′ , ĉ′ )} = 1Bt 1{ot+1 = o′ }1 ĉ′ = τoλt ,at ,o′ (b ct ) . Taking conditional expectation with respect to Gt− and using Bt ∈ Gt− and P(ot+1 = o′ | Gt− ) = PO (o′ | ot , at ) yields 1Bt P zbt+1 = (o′ , ĉ′ ) | Gt− = 1B Pbλ ((o′ , ĉ′ ) | zbt , at ) . t
If PO (o′ | ot , at ) = 0, both sides are zero. Thus the transition identity holds on Bt for every candidate successor. Under Assumption C.1, Yt = R(ot , it , at ) + ξt ,
E[ξt | Gt− ] = 0.
On Bt , reward consistency and ct = λot (b ct ) imply bλ (b R(ot , it , at ) = R̄(ot , ct , at ) = R zt , at ). Since Bt ∈ Gt− , bλ (b zt , at ). 1Bt E[Yt | Gt− ] = 1Bt R Moreover, on Bt , bλ (b Yt − R zt , at ) = ξt , so the conditional sub-Gaussian assumption gives, for every η ∈ R, i h bλ (b 1Bt E exp η Yt − R zt , at ) Gt− 2 2 η σY . ≤ 1Bt exp 2 The map Λλ (o, ĉ) = (o, λo (ĉ)) is an MDP isomorphism by the same algebraic argument as in Theorem 5.2: it is bijective and preserves rewards and transition probabilities under the exact local transports τeλ . If T0 is also a stopping time for (Fbt− ), then Bt ∈ Fbt− .
Applying the tower property to the three ambientfiltration identities above yields the corresponding transition, conditional-reward, and conditional sub-Gaussian identities relative to Fbt− on Bt . This argument does not require the passive learner to recover or explicitly store the edge maps; it uses only eventual exact class labels together with the stopping-time condition. Conversely, without eventual exact labels, the learned representation may merge histories from different stable classes. Without the stopping-time condition, pathwise eventual agreement alone does not justify conditioning on the random post-synchronization regime. Therefore the structural quotient results alone do not give an unconditional Markov or value-preservation guarantee for passive learning.
and Lw′ (q) := V̄ ⋆ (o, cw′ ) −
X
q(a)Q̄⋆ (o, cw′ , a).
a∈A
For every quotient class c ∈ Co , the optimal value and optimal state–action value satisfy V̄ ⋆ (o, c) = max Q̄⋆ (o, c, a). a∈A
It follows that, for every q ∈ ∆(A), X X q(a)Q̄⋆ (o, c, a) ≤ q(a)V̄ ⋆ (o, c) a∈A
a∈A
= V̄ ⋆ (o, c)
X
q(a)
a∈A
I.27
= V̄ ⋆ (o, c).
Proof of Theorem 6.4
Proof. Let Hw and Hw′ denote reachable observable histories that start from the fixed reachable quotient class c0 at the base observation o and execute the loop words w and w′ , respectively. Every generator and inverse generator is represented by an executable directed closed walk based at o, so both histories terminate again at the same visible observation o. By the definition of the free-group transport representation and of cw and cw′ , their terminal canonical quotient states are (o, cw ) = o, ρ (w)(c ) and (o, cw′ ) = o 0 o, ρo (w′ )(c0 ) . Since the loop-memory surrogate factors through abelianization, there exists a map f : Zr → M such that S = f ◦ ab. The assumed equality ab(w) = ab(w′ ) therefore gives
Therefore Lw (q) ≥ 0
for every q ∈ ∆(A). We now apply the definition of the common-decision loss to the particular distribution p selected by the controller. Since the minimum over all action distributions cannot exceed the value of the same objective at the particular distribution p, we have max Lw (p), Lw′ (p) ≥ min max Lw (q), Lw′ (q) q∈∆(A) = εdec o; cw , cw′ . Lw (p) ≥ εdec o; cw , cw′
or Lw′ (p) ≥ εdec o; cw , cw′
′
= (f ◦ ab)(w ) = S(w′ ). Both histories end at the same observation o, and hence their terminal memories satisfy m(w) = M o, S(w) = M o, S(w′ ) = m(w′ ). Write this common memory value as m⋆ . Under the finitememory controller convention of Section 2, the action readout depends on a history only through its terminal observation and terminal memory. Consequently, after either history the controller uses the same action distribution a ∈ A.
Because πM (· | o, m⋆ ) is a probability distribution, we have p ∈ ∆(A). For an arbitrary distribution q ∈ ∆(A), define the onedecision optimality gaps at the two terminal quotient classes by X Lw (q) := V̄ ⋆ (o, cw ) − q(a)Q̄⋆ (o, cw , a) a∈A
Lw′ (q) ≥ 0
Hence at least one of the inequalities
S(w) = (f ◦ ab)(w) = f ab(w) = f ab(w′ )
p(a) := πM (a | o, m⋆ ),
and
must hold. For completeness, the lower bound is strictly positive because the two terminal classes are decision separated. The simplex ∆(A) is compact because A is finite, and the function q 7−→ max Lw (q), Lw′ (q) is continuous. Its minimum is therefore attained. Suppose, for contradiction, that εdec o; cw , cw′ = 0. Then there exists q⋆ ∈ ∆(A) such that max Lw (q⋆ ), Lw′ (q⋆ ) = 0. Both losses are nonnegative, so the preceding equality implies Lw (q⋆ ) = 0 and Lw′ (q⋆ ) = 0. For the first equality, we may write 0 = Lw (q⋆ ) = V̄ ⋆ (o, cw ) −
X
q⋆ (a)Q̄⋆ (o, cw , a)
a∈A
=
X a∈A
q⋆ (a) V̄ ⋆ (o, cw ) − Q̄⋆ (o, cw , a) .
For every action a, V̄ ⋆ (o, cw ) − Q̄⋆ (o, cw , a) ≥ 0. Thus the last expression is a sum of nonnegative terms. It can equal zero only if every term having positive weight is zero. Consequently, q⋆ (a) > 0
Q̄⋆ (o, cw , a) = V̄ ⋆ (o, cw ),
=⇒
exceed this quantity. Averaging over its first-action distribution p gives X JM (Hw ) ≤ p(a)Q̄⋆ (o, cw , a). a∈A
Subtracting both sides from V̄ ⋆ (o, cw ) yields X V̄ ⋆ (o, cw ) − JM (Hw ) ≥ V̄ ⋆ (o, cw ) − p(a)Q̄⋆ (o, cw , a) a∈A
and hence q⋆ (a) > 0
=⇒
a ∈ A⋆ (o, cw ).
= Lw (p). The identical argument at the history Hw′ yields
Equivalently, supp(q⋆ ) ⊆ A⋆ (o, cw ). Applying the same reasoning to Lw′ (q⋆ ) = 0 gives supp(q⋆ ) ⊆ A⋆ (o, cw′ ). Therefore supp(q⋆ ) ⊆ A⋆ (o, cw ) ∩ A⋆ (o, cw′ ). The classes cw and cw′ are decision separated, so A⋆ (o, cw ) ∩ A⋆ (o, cw′ ) = ∅. It would follow that supp(q⋆ ) = ∅, which is impossible because q⋆ is a probability distribution and therefore satisfies X q⋆ (a) = 1. a∈A
This contradiction proves that εdec o; cw , cw′ > 0. We next relate the preceding one-decision gaps to the controller’s actual conditional continuation values. Let JM (Hw ) denote the controller’s expected discounted return conditional on reaching the terminal decision point of Hw , with the discount restarted at that decision point. More explicitly, if tw is the terminal time of Hw , then "∞ # X k JM (Hw ) := E γ Rtw +k Hw , k=0
where the future actions are selected by the controller under consideration. Conditional on selecting action a at the terminal state (o, cw ), the controller’s subsequent behavior is one admissible continuation strategy from that quotient state. By the definition of the optimal quotient state–action value, Q̄⋆ (o, cw , a) is the supremal expected discounted return among all admissible continuations that first take action a. Therefore the controller’s conditional return after first selecting a cannot
V̄ ⋆ (o, cw′ ) − JM (Hw′ ) ≥ Lw′ (p). Combining these two inequalities gives n o max V̄ ⋆ (o, cw ) − JM (Hw ), V̄ ⋆ (o, cw′ ) − JM (Hw′ ) ≥ max Lw (p), Lw′ (p) ≥ εdec o; cw , cw′ . It follows that at least one of the two histories satisfies V̄ ⋆ (o, cw ) − JM (Hw ) ≥ εdec o; cw , cw′ or
V̄ ⋆ (o, cw′ ) − JM (Hw′ ) ≥ εdec o; cw , cw′ , respectively. Thus at least one history incurs conditional optimality loss at least εdec o; cw , cw′ . Finally, suppose that a terminal memory factoring through abelianization were exact for quotient-optimal continuation control on a reachable history set containing both Hw and Hw′ . Exactness would require the controller to attain the quotient-optimal continuation value after each of these histories, namely JM (Hw ) = V̄ ⋆ (o, cw ) and
JM (Hw′ ) = V̄ ⋆ (o, cw′ ). Both conditional optimality losses would then be zero, so n o max V̄ ⋆ (o, cw )−JM (Hw ), V̄ ⋆ (o, cw′ )−JM (Hw′ ) = 0. n This contradicts max V̄ ⋆ (o, cw ) − JM (Hw ), V̄ ⋆ (o, cw′ ) − o JM (Hw′ ) ≥ εdec o; cw , cw′ > 0. Therefore no terminal memory of the form m(w) = M (o, S(w)) with S = f ◦ ab can be exact for quotient-optimal continuation control on any reachable history set containing both histories.
I.28
Proof of Corollary 6.5
Proof. Let w = [u, v] = uvu−1 v −1
and
w′ = e.
Because the abelianization map ab : Fr → Zr is a group homomorphism from the multiplicative group Fr to the additive group Zr , it satisfies ab(zz ′ ) = ab(z)+ab(z ′ )
and
ab(z −1 ) = − ab(z)
for every z, z ′ ∈ Fr . Applying these identities successively to the commutator gives ab([u, v]) = ab uvu−1 v −1 = ab(u) + ab(v) + ab(u−1 ) + ab(v −1 ) = ab(u) + ab(v) − ab(u) − ab(v) = 0. Moreover, every group homomorphism maps the identity element of its domain to the identity element of its codomain. Since the identity element of the additive group Zr is 0, we also have ab(e) = 0. Consequently, ab([u, v]) = ab(e). Now let S : Fr → M be an arbitrary abelianized loopmemory surrogate. By Definition 6.2, there exists a map f : Zr → M such that S = f ◦ ab . It follows that S([u, v]) = (f ◦ ab)([u, v]) = f ab([u, v]) = f (0) = f ab(e) = (f ◦ ab)(e) = S(e). Because c0 is reachable and every generator and inverse generator in the executable reversible loop alphabet is realized by an executable directed closed walk based at o, the word [u, v] determines a reachable loop history starting from (o, c0 ). The identity word e determines the corresponding reachable empty-loop history. Both histories terminate at the same visible observation o. Therefore, for any terminal controller memory of the form m(z) = M o, S(z) , we obtain
m([u, v]) = M o, S([u, v]) = M o, S(e) = m(e).
Thus the controller receives exactly the same terminal observation and terminal memory after the commutator history and after the empty-loop history, and hence it must use the same action distribution after both histories. It remains to identify the canonical quotient states reached by these two histories. Since
and therefore
ρo (e)(c0 ) = IdCo (c0 ) = c0 . Accordingly, the terminal quotient classes of the two histories are c[u,v] = ρo ([u, v])(c0 ) and ce = ρo (e)(c0 ) = c0 . The hypothesis of the corollary states precisely that ρo ([u, v])(c0 ) is decision separated from c0 . Equivalently, A⋆ o, ρo ([u, v])(c0 ) ∩ A⋆ (o, c0 ) = ∅. We have therefore verified all hypotheses of Theorem 6.4 with w = [u, v] and w′ = e. That theorem implies that every controller whose terminal memory factors through abelianization must incur conditional optimality loss on at least one of the two histories of at least εdec o; ρo ([u, v])(c0 ), c0 . Because the two terminal quotient classes are decision separated, Definition 6.3 gives εdec o; ρo ([u, v])(c0 ), c0 > 0. If the abelianized memory were exact at both resulting quotient states, then the controller would attain the quotientoptimal continuation value after both histories, so its conditional optimality loss would be zero after each history. This would contradict the strictly positive lower bound above. Hence no abelianized loop-memory surrogate can be exact at both o, ρo ([u, v])(c0 ) and (o, c0 ).
I.29
Proof of Corollary F.1
Proof. Let V denote the additive space in which the vectors v1 , . . . , vr take values. Since the abelianization map records the signed exponent sums of the free generators, for every word w ∈ Fr we have ab(w) = N1 (w), . . . , Nr (w) ∈ Zr . Define a map Φ : Zr −→ V by r X Φ(n1 , . . . , nr ) := n j vj . j=1
This map is well defined. Moreover, if n = (n1 , . . . , nr ) and n′ = (n′1 , . . . , n′r ) are arbitrary elements of Zr , then Φ(n + n′ ) = Φ(n1 + n′1 , . . . , nr + n′r ) r X = (nj + n′j )vj
ρo : Fr → Sym(Co ) is a group homomorphism, it maps the identity word e to the identity permutation on Co . Hence ρo (e) = IdCo ,
=
j=1 r X j=1
n j vj +
r X j=1
= Φ(n) + Φ(n′ ).
n′j vj
Thus Φ is a homomorphism from the additive group Zr into the additive group underlying V. Injectivity of Φ is neither assumed nor needed. Indeed, if two distinct vectors in Zr are mapped to the same element of V, then the count-additive statistic discards even more information than abelianization; such additional collisions cannot recover the ordering information that abelianization has already removed. For every w ∈ Fr , substituting the coordinates of ab(w) into the definition of Φ gives (Φ ◦ ab)(w) = Φ ab(w) = Φ N1 (w), . . . , Nr (w) r X = Nj (w)vj
Because the readout depends only on the value of the countadditive statistic, this equality further implies S(w) = ψ A(w) = ψ A(w′ ) = S(w′ ). Thus no readout applied only after A(w) can distinguish two loop histories having the same abelianization. Now fix a reachable class c0 ∈ Co and an executable reversible loop alphabet, and suppose that w, w′ ∈ Fr satisfy ab(w) = ab(w′ ), cw = ρo (w)(c0 ), cw′ = ρo (w′ )(c0 ), where cw and cw′ are decision separated. Since the loop-memory surrogate has the form
j=1
S = f ◦ ab,
= A(w). Therefore A = Φ ◦ ab . By assumption, the readout of the encoder depends on the loop word w only through the value of A(w). Consequently, there exists a map ψ : V −→ M such that the resulting loop-memory surrogate satisfies S(w) = ψ A(w) for every w ∈ Fr . Combining this representation with the identity A = Φ ◦ ab yields S(w) = ψ A(w) = ψ (Φ ◦ ab)(w) = (ψ ◦ Φ) ab(w) . Define f := ψ ◦ Φ : Zr −→ M. Then, for every w ∈ Fr , S(w) = f ab(w) , and hence S = f ◦ ab . It follows directly from Definition 6.2 that S is an abelianized loop-memory surrogate. The same conclusion can also be verified directly from equality of signed generator counts. Let w, w′ ∈ Fr satisfy ab(w) = ab(w′ ). Since equality in Zr is coordinatewise ab(w) = N1(w), . . . , Nr (w) , ab(w′ ) N1 (w′ ), . . . , Nr (w′ ) , we obtain Nj (w) = Nj (w′ )
for every j ∈ {1, . . . , r}.
Therefore A(w) =
=
r X j=1 r X
Nj (w)vj
and =
all of the hypotheses of Theorem 6.4 concerning the memory representation are satisfied. More explicitly, for any terminal controller memory of the form m(z) = M o, S(z) , the equality of the two abelianizations gives m(w) = M o, S(w) = M o, f (ab(w)) = M o, f (ab(w′ )) = M o, S(w′ ) = m(w′ ). Both loop histories terminate at the same base observation o, and the controller therefore receives the same terminal observation and the same terminal memory after both histories. It must consequently use the same action distribution after the terminal quotient states (o, cw ) = o, ρo (w)(c0 ) and
(o, cw′ ) = o, ρo (w′ )(c0 ) . Since these quotient classes are decision separated, Theorem 6.4 implies that at least one of the two histories incurs conditional optimality loss at least εdec o; cw , cw′ . By Definition 6.3, decision separation further implies εdec o; cw , cw′ > 0. If the count-additive encoder were exact for quotient-optimal continuation control on a reachable history set containing both histories, then the conditional optimality loss after each history would be zero. This would contradict the strictly positive lower bound above. Hence every encoder of the form r X A(w) = Nj (w)vj j=1
Nj (w′ )vj
j=1
= A(w′ ).
whose readout depends only on A(w) is subject to the barrier of Theorem 6.4 whenever the theorem’s equal-abelianization and decision-separation conditions hold.
I.30
Proof of Corollary 6.6
We next verify that this partition is already stable. If
Proof. For each c ∈ Co , choose a positive-probability history Hc that ends at observation o with quotient class c, and let F (Hc ) ∈ M be the memory state induced by the controller. Suppose, toward a contradiction, that two distinct classes c, c′ satisfy F (Hc ) = F (Hc′ ). Because both histories end at the same observation, the controller uses the same action distribution
T (Π(0) )
i ≡o quires
j, then the first condition in Definition 3.4 re-
R(o, i, a) = R(o, j, a)
T (Π(0) ) = Π(0) . Thus Π(0) is a fixed point of the stability operator. Since the refinement sequence defining the stable quotient begins at Π(0) , its fixed point is Π⋆o = Π(0) o = {1}, {2}, {3} . Consequently, the stable quotient has exactly the three singleton classes c1 := {1},
a∈A
and analogously Vctrl (Hc′ ) ≤
X
p(a)Q̄⋆ (o, c′ , a).
a∈A
By Definition 6.3, pairwise decision separation imP ⋆ plies max{ V̄ (o, c) − p(a) Q̄⋆ (o, c, a), V̄ ⋆ (o, c′ ) − a P ⋆ ′ ′ a p(a)Q̄ (o, c , a)} ≥ εdec (o; c, c ) > 0. Therefore the controller’s continuation value is strictly below the optimum after at least one of Hc and Hc′ , contradicting the assumed exactness after every positive-probability history. Thus the memory states {F (Hc ) : c ∈ Co } are pairwise distinct, and |M| ≥ |Co |. If the hypotheses hold at an observation o⋆ with |Co⋆ | = m⋆exact , the same inequality gives |M| ≥ m⋆exact .
I.31
and
c2 := {2},
c3 := {3}.
Because every stable class is a singleton, the quotient transports induced by actions a and b are identified with the corresponding raw permutations. More precisely, Theorem 3.9 gives τa (ci ) = cα(i)
and
τb (ci ) = cβ(i)
for every i ∈ {1, 2, 3}. Hence, after identifying ci with i, τa = (12) = α
and
τb = (123) = β.
Let
G := ⟨τa , τb ⟩ ≤ Sym({c1 , c2 , c3 }) ∼ = S3 . The element τa has order 2, while τb has order 3. By Lagrange’s theorem, both 2 and 3 divide |G|, and therefore 6 divides |G|. On the other hand, G is a subgroup of S3 , so |G| divides |S3 | = 6. Hence
Proof of Proposition 6.7
|G| = 6,
Proof. Because there is only one observation and PO (o | o, a) = PO (o | o, b) = 1, every finite action sequence is an executable directed closed walk based at o. Let α := σa = (12)
R(o, i, b) = R(o, j, b).
The pairwise distinctness of the three reward vectors then implies i = j. Conversely, every layer is equivalent to itself under T (Π(0) ), because equality of rewards and equality of successor blocks are reflexive. Therefore
p(a) := πM (a | o, F (Hc )) = πM (a | o, F (Hc′ )) after both histories. Let Vctrl (Hc ) denote the controller’s conditional continuation value after Hc . Conditional on its first action, the controller’s subsequent behavior is only one admissible continuation policy, whereas Q̄⋆ (o, c, a) uses an optimal continuation. Hence X Vctrl (Hc ) ≤ p(a)Q̄⋆ (o, c, a),
and
β := σb = (123).
The reward vector of a layer is the vector of its immediate rewards under the two available actions. For the three layers these vectors are R(o, 1, a), R(o, 1, b) = (1, 0), R(o, 2, a), R(o, 2, b) = (0, 1), and R(o, 3, a), R(o, 3, b) = (0, 0). They are pairwise distinct. Hence two layers i, j ∈ {1, 2, 3} have equal rewards under every action if and only if i = j. It follows that the reward partition appearing in Lemma 3.5 is Π(0) o = {1}, {2}, {3} .
and therefore G = S3 . This proves that the loop transports of the stable quotient generate the nonabelian group S3 . We now construct an executable reversible loop alphabet − in the sense of Definition 6.1. Let ℓ+ 1 and ℓ1 both be the one-edge closed walk labeled by action a, and let ℓ+ 2 be the one-edge closed walk labeled by action b, while ℓ− 2 is the two-edge closed walk whose consecutive action labels are b, b. Since α−1 = α and β −1 = β 2 = β ◦ β, the corresponding quotient transports satisfy −1 τℓ− = α = α−1 = τℓ+ 1
and
1
τℓ− = β ◦ β = β 2 = β −1 = τℓ+ 2
2
−1
.
− + − Thus {ℓ+ 1 , ℓ1 , ℓ2 , ℓ2 } is an executable reversible loop alphabet. Let F2 = ⟨x1 , x2 ⟩ be the associated free group, so that ρo (x1 ) = α, ρo (x2 ) = β, ρo (x−1 1 ) = −1 −1 −1 α , ρo (x2 ) = β . For the commutator −1 [x1 , x2 ] = x1 x2 x−1 1 x2 ,
{o} × {1, 2, 3}, there exists at least one layer i0 ∈ {1, 2, 3} such that ρ0 (o, i0 ) > 0. The permutation β = (123) cycles through all three layers. Hence there exists k ∈ {0, 1, 2} such that
the homomorphism convention ρo (uv) = ρo (u)◦ρo (v) gives
β k (i0 ) = 1.
ρo ([x1 , x2 ]) = α ◦ β ◦ α−1 ◦ β −1 .
Executing action b exactly k times has visible probability
Set κ := α ◦ β ◦ α
−1
◦β
−1
k Y
.
Using α−1 = α = (12) and β −1 = (132), and remembering that the rightmost permutation acts first, we obtain κ(1) = α β α−1 β −1 (1) = α β α−1 (3) = α(β(3)) = α(1) = 2, κ(2) = α β α−1 β −1 (2) = α β α−1 (1) = α(β(2)) = α(3) = 3, and
κ(3) = α β α−1 β −1 (3) = α β α−1 (2) = α(β(1)) = α(2) = 1.
Therefore
PO (o | o, b) = 1
t=1
and transports layer i0 to layer 1. It follows that the latent state (o, 1), and therefore the stable quotient class c1 , is reachable with positive probability. Starting from this reachable class, the action sequence implementing the commutator is also executable with probability one. It remains to prove the asserted decision separation. Write V̄i⋆ := V̄ ⋆ (o, ci ),
i ∈ {1, 2, 3}.
Every one-step reward in this HCDP lies in the interval [0, 1]. Therefore, under every admissible policy and from every state, the discounted return lies between 0 and P∞ quotient t t=0 γ . Taking the supremum over policies gives 0 ≤ V̄i⋆ ≤
∞ X
γt =
t=0
1 1−γ
for every i ∈ {1, 2, 3}.
At class c1 , both actions have the same successor class, because α(1) = 2 and β(1) = 2. Consequently, the optimal quotient state–action values satisfy Q̄⋆ (o, c1 , a) = 1 + γ V̄2⋆ and Q̄⋆ (o, c1 , b) = 0 + γ V̄2⋆ .
ρo ([x1 , x2 ]) = κ = (123) = β. In particular, ρo ([x1 , x2 ])(c1 ) = c2 . This commutator is executable using only the original actions. Indeed, the inverse letter x−1 1 is implemented by one use of action a, and the inverse letter x−1 2 is implemented by two consecutive uses of action b. Because the rightmost factor in the commutator transport acts first, the chronological action sequence b, b, a, b, a implements [x1 , x2 ]. If ωcom denotes this directed closed walk, then Definition 3.2 gives τωcom = α ◦ β ◦ α ◦ β ◦ β = α ◦ β ◦ α−1 ◦ β −1 = ρo ([x1 , x2 ]). We also verify that c1 is reachable, as required by Corollary 6.5. Since ρ0 is a probability distribution on
Subtracting the second equality from the first yields Q̄⋆ (o, c1 , a) − Q̄⋆ (o, c1 , b) = 1 + γ V̄2⋆ − γ V̄2⋆ =1 > 0. Thus action a is uniquely optimal at c1 , and hence A⋆ (o, c1 ) = {a}. At class c2 , action a transports the class to c1 , whereas action b transports it to c3 , because α(2) = 1
and
β(2) = 3.
The corresponding optimal quotient state–action values are therefore Q̄⋆ (o, c2 , a) = 0 + γ V̄1⋆ and Q̄⋆ (o, c2 , b) = 1 + γ V̄3⋆ .
Their difference satisfies Q̄⋆ (o, c2 , b) − Q̄⋆ (o, c2 , a) = 1 + γ V̄3⋆ − γ V̄1⋆ ≥ 1 − γ V̄1⋆ γ ≥1− 1−γ 1 − 2γ = . 1−γ The assumption 0 < γ < 1/2 implies 1 − 2γ > 0 so
and
d(2) = τe2 (d(1) )
1 − γ > 0,
= τe2 τe1 (d)
1 − 2γ > 0. 1−γ
= (τe2 ◦ τe1 )(d). Continuing recursively through all m edges yields
It follows that Q̄⋆ (o, c2 , b) > Q̄⋆ (o, c2 , a). Thus action b is uniquely optimal at c2 , and hence A⋆ (o, c2 ) = {b}. Therefore A⋆ (o, c1 ) ∩ A⋆ (o, c2 ) = {a} ∩ {b} = ∅, which means, by Definition 6.3, that c1 and c2 are decision separated. We have exhibited an executable reversible loop alphabet, a reachable class c0 = c1 , and the words u = x1
and
v = x2
d(m) = τem (d(m−1) ) = τem ◦ τem−1 ◦ · · · ◦ τe1 (d) = τℓ(z) (d). The defining assignment of the free-group transport representation gives τℓ(z) = ρo (z). Indeed, when z = xj , τℓ(z) = τℓ+ = ρo (xj ), j
whereas when z = x−1 j , τℓ(z) = τℓ− = ρo (x−1 j ). j
such that ρo ([u, v])(c0 ) = ρo ([x1 , x2 ])(c1 ) = c2 , while c2 is decision separated from c1 . These are exactly the hypotheses of Corollary 6.5. Hence the commutator and empty-loop histories have the same abelianized memory but terminate at decision-separated quotient states, completing the proof.
I.32
directed closed walk ℓ− j . Denote the corresponding closed walk by ℓ(z) = e1 e2 · · · em , where every eq is a feasible directed edge and the walk begins and ends at o. Starting from an arbitrary known class d ∈ Co , repeated application of the exact one-edge update gives, after the first edge, d(1) = τe1 (d), and, after the second edge,
Proof of Proposition 6.8
Proof. Fix a base observation o ∈ O, an executable re− r versible loop alphabet {ℓ+ j , ℓj }j=1 based at o, and a known initial stable class c ∈ Co . Exact quotient tracking stores the current stable class and, after every realized feasible observation transition (ot , at , ot+1 ), applies the deterministic update ct+1 = τot ,at ,ot+1 (ct ). By Lemma C.3, if the stored class is correct before a feasible transition, then the class produced by this update is the true stable class after that transition. Consequently, once the tracker is initialized with the correct class, its correctness is preserved along every finite executable directed path. Consider first one letter ±1 z ∈ {x±1 1 , . . . , xr }.
If z = xj , its executable realization is the directed closed −1 walk ℓ+ j , whereas if z = xj , its executable realization is the
Therefore, after completing the executable loop associated with a single letter z, exact quotient tracking transforms every known class d ∈ Co according to d 7−→ ρo (z)(d). Now let w ∈ Fr . If w = e is the empty word, no loop is executed, and therefore exact tracking leaves the initial class unchanged. Since ρo is a group homomorphism, it maps the identity of Fr to the identity permutation of Co , so ρo (e) = IdCo . Consequently, the terminal tracked class for the empty word is c = IdCo (c) = ρo (e)(c). Thus the asserted identity holds when w = e. Suppose next that w has a reduced expression w = z1 z2 · · · zk ,
±1 zq ∈ {x±1 1 , . . . , xr },
k ≥ 1.
The convention in Section 6 is ρo (uv) = ρo (u) ◦ ρo (v), so the rightmost factor acts first. Accordingly, the executable realization of the written word w = z1 · · · zk performs the loop associated with zk first, then the loop associated with zk−1 , and continues in this order until the loop associated with z1 is performed last. Every one of these loops begins
and ends at o, so their concatenation is executable, and the tracked class again belongs to Co after each completed letter. Let d0 := c, and, for every t ∈ {1, . . . , k}, define dt := ρo zk−t+1 (dt−1 ). We show that dt is exactly the stable class stored after the first t loops in this chronological execution. For t = 0, the statement follows from the initialization d0 = c. Suppose that the statement holds after t − 1 completed loops. The next executed letter is
and similarly −1 ρo (x−1 j ) ◦ ρo (xj ) = ρo (xj xj )
= ρo (e) = IdCo . Thus free reduction does not alter the recursively tracked terminal class. Now let w, w′ ∈ Fr satisfy ρo (w)(c) ̸= ρo (w′ )(c). Applying the identity proved above separately to the two executable loop histories gives cw = ρo (w)(c)
zk−t+1 .
and
By the one-letter calculation above, completing the executable loop associated with this letter transforms the correctly stored class dt−1 into ρo zk−t+1 (dt−1 ) = dt .
Hence
Lemma C.3 guarantees that this recursively produced class remains the true stable class throughout the constituent edges of the loop. Hence the assertion holds after t loops. By induction, it holds for every t ∈ {0, . . . , k}. At the terminal index, repeated expansion of the recursion gives dk = ρo (z1 )(dk−1 ) = ρo (z1 ) ρo (z2 )(dk−2 ) = ρo (z1 ) ρo (z2 ) · · · ρo (zk )(c) · · · = ρo (z1 ) ◦ ρo (z2 ) ◦ · · · ◦ ρo (zk ) (c). Repeated use of the homomorphism identity gives ρo (w) = ρo (z1 z2 · · · zk ) = ρo (z1 ) ◦ ρo (z2 · · · zk ) = ρo (z1 ) ◦ ρo (z2 ) ◦ ρo (z3 · · · zk ) = ··· = ρo (z1 ) ◦ ρo (z2 ) ◦ · · · ◦ ρo (zk ). Combining the preceding two displays yields dk = ρo (w)(c). Therefore exact quotient tracking maps the loop word w to the terminal stable class cw := ρo (w)(c). This terminal class is well defined as a function of the free-group element w, rather than of a particular spelling of that element. To see this explicitly, insertion or deletion of an adjacent inverse pair does not change the terminal transport, because −1 ρo (xj ) ◦ ρo (x−1 j ) = ρo (xj xj ) = ρo (e) = IdCo ,
cw′ = ρo (w′ )(c).
cw ̸= cw′ . Since the canonical exact quotient tracker stores the stable class itself, its two terminal memory states are different. More generally, if the stable classes are represented by an injective class encoder ηo : Co → M, then injectivity implies ηo (cw ) ̸= ηo (cw′ ). Therefore the two loop histories remain distinguished by the exact quotient memory. This conclusion is independent of their abelianized summaries and, in particular, remains valid when ab(w) = ab(w′ ). The reason is that exact quotient tracking recursively composes the quotient transports in their prescribed order, whereas abelianization retains only the signed number of occurrences of each generator and discards their order. Finally, let u, v ∈ Fr and suppose that the commutator [u, v] = uvu−1 v −1 acts nontrivially on the known class c, so that ρo ([u, v])(c) ̸= c. The empty word leaves the class unchanged: ce = ρo (e)(c) = IdCo (c) = c. By contrast, exact quotient tracking after the commutator history produces c[u,v] = ρo ([u, v])(c). The assumed nontriviality therefore gives c[u,v] ̸= ce . Nevertheless, because abelianization is a homomorphism from the multiplicative group Fr to the additive abelian group Zr , we have ab(z −1 ) = − ab(z)
for every z ∈ Fr , and consequently ab([u, v]) = ab(uvu−1 v −1 ) = ab(u) + ab(v) + ab(u−1 ) + ab(v −1 ) = ab(u) + ab(v) − ab(u) − ab(v) = 0. Moreover,
such that 0 < ρ0 (s⋆0 )1{h(s⋆0 ) = o0 } t−1 i Yh × πj (aj | Hj )P (s⋆j+1 | s⋆j , aj )1{h(s⋆j+1 ) = oj+1 } . j=0
Every factor in this product is nonnegative. Strict positivity of the product therefore implies
ab(e) = 0,
ρ0 (s⋆0 ) > 0,
ab([u, v]) = ab(e). Thus the commutator and empty-loop histories have identical abelianized summaries, but exact ordered stable-class tracking assigns them different terminal classes whenever the commutator acts nontrivially on c. Therefore exact quotient tracking preserves every nontrivial commutator effect on the known stable class while storing only the current finite stable class rather than the complete loop word.
forces all observation-consistency indicators to be equal to one, and gives
I.33
For every j ∈ {0, . . . , t − 1}, the states
and hence
Proof of Lemma B.1
P (s⋆j+1 | s⋆j , aj ) > 0
The indicator equalities are precisely so
Proof. For every j ∈ {0, . . . , t}, let Hj = (o0 , a0 , o1 , . . . , aj−1 , oj ) denote the length-j observable prefix of Ht . To distinguish random variables from their realized values, write Sj , Oj , and Aj for the latent state, observation, and action at time j, and let Ht := {O0 = o0 , A0 = a0 , O1 = o1 , . . . , At−1 = at−1 , Ot = ot } be the event that the realized observable history equals Ht . Thus saying that Ht has positive probability means that Pπ (Ht ) > 0. Fix an arbitrary latent-state sequence (s0 , s1 , . . . , st ) ∈ S t+1 . Because the observation map is deterministic, the chain rule and the definition of a history-dependent policy give Pπ S0 = s0 , S1 = s1h, . . . , St = st , Ht = Qt−1 ρ0 (s0 )1{h(s0 ) = o0 } j=0 πj (aj | Hj )P (sj+1 | i sj , aj )1{h(sj+1 ) = oj+1 } . When t = 0, the product is the empty product and is equal to one. The events obtained by varying (s0 , . . . , st ) ∈ S t+1 are pairwise disjoint and their union is Ht . Summing over all latent-state sequences therefore gives X
ρ0 (s0 )1{h(s0 ) = o0 }
(s0 ,...,st )∈S t+1
×
t−1 Yh
i πj (aj | Hj )P (sj+1 | sj , aj )1{h(sj+1 ) = oj+1 } .
j=0
The sum is finite because S is finite, and every summand is nonnegative. Suppose first that π
P (Ht ) > 0. A finite sum of nonnegative numbers is strictly positive only if at least one summand is strictly positive. Hence there exists (s⋆0 , s⋆1 , . . . , s⋆t ) ∈ S t+1
h(s⋆j ) = oj
for every j ∈ {0, . . . , t},
s⋆j ∈ Soj
for every j ∈ {0, . . . , t}.
s⋆j ∈ Soj
and
s⋆j+1 ∈ Soj+1
together with
P (s⋆j+1 | s⋆j , aj ) > 0 witness, by the definition of E, that (oj , aj , oj+1 ) ∈ E. Thus at−1
a
0 ω(Ht ) = (o0 −→ o1 ) · · · (ot−1 −−−→ ot )
is a directed support path in X. More importantly, the same sequence (s⋆0 , . . . , s⋆t ) simultaneously realizes all of its edges, rather than providing unrelated witnesses edge by edge, and its initial state satisfies ρ0 (s⋆0 ) > 0. Therefore ω(Ht ) is ρ0 realizable. When t = 0, the argument states simply that there exists s⋆0 ∈ So0 with ρ0 (s⋆0 ) > 0, which is exactly ρ0 -realizability of the empty path based at o0 . Conversely, suppose that ω(Ht ) is ρ0 -realizable. By definition, there exist latent states (s⋆0 , s⋆1 , . . . , s⋆t ) such that s⋆j ∈ Soj
Pπ (Ht ) =
for every j ∈ {0, . . . , t − 1}.
for every j ∈ {0, . . . , t}, ρ0 (s⋆0 ) > 0,
and P (s⋆j+1 | s⋆j , aj ) > 0
for every j ∈ {0, . . . , t − 1}.
Since
Soj = h−1 ({oj }), the membership s⋆j ∈ Soj implies h(s⋆j ) = oj
for every j ∈ {0, . . . , t}.
Hence every observation-consistency indicator in the trajectory-probability formula is equal to one. The additional assumption of the lemma gives πj (aj | Hj ) > 0
for every j ∈ {0, . . . , t − 1}.
It follows that the summand associated with the particular compatible latent sequence (s⋆0 , . . . , s⋆t ) is ρ0 (s⋆0 )1{h(s⋆0 ) = o0 } t−1 i Yh πj (aj | Hj )P (s⋆j+1 | s⋆j , aj )1{h(s⋆j+1 ) = oj+1 } j=0 = ρ0 (s⋆0 )
t−1 Yh
πj (aj | Hj )P (s⋆j+1 | s⋆j , aj )
j=0
Since Pπ (Ht ) is the sum of this strictly positive term and other nonnegative terms, we conclude that
Proof. Let B(S × M) := {V : S × M → R : ∥V ∥∞ < ∞} , ∥V ∥∞ := max(s,m)∈S×M |V (s, m)|. Because S and M are finite, every real-valued function on S × M is bounded, and B(S × M), equipped with the sup norm, is complete. Since the reward function is bounded, define Rmax :=
This also covers t = 0, because in that case the displayed quantity reduces to ρ0 (s⋆0 ) > 0. It remains to justify why membership of every edge in E does not by itself imply latent realizability of the whole path. The definition of E contains an existential quantifier separately for each edge, so the terminal latent-state witness chosen for one edge need not coincide with the initial latentstate witness chosen for the next edge. Consider, for example,
Fix V ∈ B(S × M) and (s, m) ∈ S × M. Using the triangle inequality, nonnegativity of the action and transition probabilities, and the fact that both probability distributions sum to one, we obtain
S = {s0 , p, q, s2 },
a∈A
h(p) = h(q) = o1 ,
P (s′ | s, a)V s′ , U (m, h(s), a, h(s′ )) ]
s′ ∈S
≤
X
πM (a | h(s), m) |r(s, a)|
a∈A
X
+γ
P (s′ | s, a) V s′ , U (m, h(s), a, h(s′ ))
h(s2 ) = o2 .
(o0 , a, o1 ) ∈ E, with witnesses s0 ∈ So0 and p ∈ So1 , while the third transition shows that (o1 , b, o2 ) ∈ E, with witnesses q ∈ So1 and s2 ∈ So2 . Therefore
" ≤
X
=
X
#
πM (a | h(s), m) Rmax + γ
is a directed support path in X. Nevertheless, any latent realization of the first edge must arrive at p, because s0 is the only latent state over o0 and P (p | s0 , a) = 1. From p, action b returns to p with probability one, since P (p | p, b) = 1. It therefore has zero probability of reaching any latent state in So2 . The state q that witnesses the second edge cannot be used as the terminal state of the first edge, so the two edgewise witnesses cannot be joined into a single latent-state sequence. Hence this directed support path is not latentrealizable, proving the final assertion.
X
P (s′ | s, a)∥V ∥∞
s′ ∈S
πM (a | h(s), m) [Rmax + γ∥V ∥∞ ]
a∈A
= Rmax + γ∥V ∥∞ . Taking the maximum over (s, m) gives ∥TπM ,U V ∥∞ ≤ Rmax + γ∥V ∥∞ < ∞, so the operator maps bounded functions to bounded functions. Now fix V, W ∈ B(S × M). The reward terms cancel after subtraction, and therefore, for every (s, m) ∈ S × M, (TπM ,U V )(s, m) − (TπM ,U W )(s, m) X πM (a | h(s), m) =γ a∈A
b
(o0 − → o1 )(o1 → − o2 )
s′ ∈S
a∈A
Choose a transition kernel satisfying P (p | s0 , a) = 1, P (p | p, b) = 1, P (s2 | q, b) = 1, and let every remaining unspecified state–action pair be a deterministic self-loop. The first transition shows that
a
X
+γ
with deterministic observation map h(s0 ) = o0 ,
|r(s, a)| < ∞.
max (s,a)∈S×A
(TπM ,U V )(s, m) X = πM (a | h(s), m)[r(s, a)
Pπ (Ht ) > 0.
A = {a, b},
Proof of Lemma B.4
i
> 0.
O = {o0 , o1 , o2 },
I.34
X
P (s′ | s, a)[V s′ , U (m, h(s), a, h(s′ ))
s′ ∈S
− W s′ , U (m, h(s), a, h(s′ )) ] X ≤γ πM (a | h(s), m) a∈A
X
P (s′ | s, a)|V s′ , U (m, h(s), a, h(s′ ))
s′ ∈S
− W s′ , U (m, h(s), a, h(s′ )) | X X ≤γ πM (a | h(s), m) P (s′ | s, a)∥V − W ∥∞ a∈A
= γ∥V − W ∥∞ .
s′ ∈S
States s′ with P (s′ | s, a) = 0 contribute zero to the following sum. Hence the law of total expectation gives
Taking the maximum over (s, m) yields ∥TπM ,U V − TπM ,U W ∥∞ ≤ γ∥V − W ∥∞ . Since 0 < γ < 1, the operator is a γ-contraction. The Banach fixed-point theorem therefore gives a unique function VπM ,U ∈ B(S × M)
VN +1 (s, m) " =
X
πM (a | h(s), m) r(s, a)
a∈A
such that VπM ,U = TπM ,U VπM ,U . The fixed-point identity and the bound already proved further imply ∥VπM ,U ∥∞ = ∥TπM ,U VπM ,U ∥∞ ≤ Rmax + γ∥VπM ,U ∥∞ , so Rmax ∥VπM ,U ∥∞ ≤ . 1−γ We next identify this fixed point with the discounted return of the given controller. For an arbitrary pair (s, m) ∈ S ×M, initialize the augmented process at
+γ
X
′
′
′
P (s | s, a)VN s , U (m, h(s), a, h(s ))
s′ ∈S
= (TπM ,U VN )(s, m). Therefore VN +1 = TπM ,U VN
for every N ≥ 0.
Since V0 = 0, induction gives VN = TπNM ,U 0
for every N ≥ 0.
The contraction property and the fixed-point identity imply ≤ γ N ∥VπM ,U ∥∞
and generate it according to At ∼ πM (· | h(St ), Mt ),
St+1 ∼ P (· | St , At ), Mt+1 = U Mt , h(St ), At , h(St+1 ) .
≤
By Lemma B.3, (St , Mt ) is a time-homogeneous Markov process under this fixed controller. For every integer N ≥ 0, define "N −1 # X t VN (s, m) := Es,m γ r(St , At ) , t=0
Rmax γ N . 1−γ
Hence VN converges uniformly to VπM ,U . On the other hand, boundedness of the reward implies that the infinite discounted series converges absolutely on every trajectory. Define "∞ # X t V∞ (s, m) := Es,m γ r(St , At ) . t=0
where the sum is empty when N = 0. Thus
For every (s, m) and every N ≥ 0,
for every (s, m) ∈ S × M.
Fix N ≥ 0 and (s, m) ∈ S × M. Separating the reward at time zero and reindexing the remaining terms gives " # N X VN +1 (s, m) = Es,m r(S0 , A0 ) + γ t r(St , At )
|V∞ (s, m) − VN (s, m)| = Es,m ≤ Es,m
= Es,m r(s, A0 ) + γ
N −1 X
# t
γ r(St , At ) #
γ t |r(St , At )|
t=N
# γ q r(Sq+1 , Aq+1 ) .
q=0
For any action a ∈ A and any next state s′ ∈ S satisfying P (s′ | s, a) > 0, M1 = U (m, h(s), a, h(s′ )).
γt
t=N
Rmax γ N . = 1−γ
V∞ = VπM ,U .
By the time-homogeneous Markov property of the augmented process, conditional on A0 = a,
≤ Rmax
∞ X
Thus VN also converges uniformly to V∞ . A sequence in a normed space has at most one limit, so
the deterministic update rule gives
M0 = m,
" ∞ X
t=N " ∞ X
t=1
"
S0 = s,
#
∥VN − VπM ,U ∥∞ = TπNM ,U 0 − TπNM ,U VπM ,U ∞
(S0 , M0 ) = (s, m)
V0 (s, m) = 0
S1 = s′ ,
the expected discounted reward accumulated over the next N stages, with discount restarted at time 1, is VN s′ , U (m, h(s), a, h(s′ )) .
Equivalently, for every augmented initial state (s, m), "∞ # X t VπM ,U (s, m) = Es,m γ r(St , At ) . t=0
Finally, under the original initialization, the latent state satisfies S0 ∼ ρ0 ,
the initial observation is h(S0 ), and the memory is initialized deterministically as M0 = ι(h(S0 )). Lemma B.3 states that the original finite-memory controller and the augmented process have the same expected discounted return, and that the augmented initial distribution is ρe0 (s, m) = ρ0 (s)1{m = ι(h(s))}. Using the value interpretation just proved and averaging over this initial distribution, we obtain X X J(πM , U, ι) = ρe0 (s, m)VπM ,U (s, m)
P Exponentiating this identity and using exp( i xi ) = Q i exp(xi ), we obtain ! m λ X exp λ(ξ m − µ) = exp (ξi − µ) m i=1 m Y λ (ξi − µ) . = exp m i=1
The variables ξ1 , . . . , ξm are independent, and each factor in the last product is a measurable function of only the corresponding variable ξi . Therefore,
s∈S m∈M
=
X X
ρ0 (s)1{m = ι(h(s))}VπM ,U (s, m)
s∈S m∈M
=
X
ρ0 (s)VπM ,U s, ι(h(s)) .
s∈S
Renaming the dummy variable s as s0 gives X J(πM , U, ι) = ρ0 (s0 )VπM ,U s0 , ι(h(s0 )) , s0 ∈S
which completes the proof.
I.35
Proof of Lemma B.7
Proof. Set m
1 X ξ m := ξi . m i=1 Since the random variables are identically distributed and each of them has mean µ, linearity of expectation gives " # m 1 X ξi E[ξ m ] = E m i=1 m
=
1 X E[ξi ] m i=1 m
1 X µ m i=1 mµ = m = µ. =
We first derive a sub-Gaussian moment-generating-function bound for the empirical mean. Fix an arbitrary λ ∈ R. By the definition of ξ m , ! m 1 X λ(ξ m − µ) = λ ξi − µ m i=1 ! m m 1 X 1 X =λ ξi − µ m i=1 m i=1 m
=
λ X (ξi − µ). m i=1
"m # Y λ E exp λ(ξ m − µ) = E exp (ξi − µ) m i=1 m Y λ = (ξi − µ) . E exp m i=1 By the definition of a σ 2 -sub-Gaussian random variable, applied to ξi with the real parameter λ/m, we have 2 ! λ 1 λ (ξi − µ) ≤ exp E exp σ2 m 2 m 2 2 λ σ = exp 2m2 for every i ∈ {1, . . . , m}. Multiplying these m inequalities gives 2 2 m Y λ σ E exp λ(ξ m − µ) ≤ exp 2m2 i=1 ! m X λ2 σ 2 = exp 2m2 i=1 mλ2 σ 2 = exp 2m2 2 2 λ σ . = exp 2m Thus the centered empirical mean satisfies λ2 (σ 2 /m) E exp λ(ξ m − µ) ≤ exp for every λ ∈ 2 R, so ξ m is (σ 2 /m)-sub-Gaussian. We next bound the upper tail. Fix ε > 0 and an arbitrary λ > 0. Because the exponential function is strictly increasing, ξ m − µ ≥ ε = exp λ(ξ m − µ) ≥ exp(λε) . The random variable exp(λ(ξ m − µ)) is nonnegative. Hence
Markov’s inequality implies Pr ξ m − µ ≥ ε = Pr exp λ(ξ m − µ) ≥ exp(λε) E exp λ(ξ m − µ) ≤ exp(λε) = exp(−λε)E exp λ(ξ m − µ) 2 2 λ σ ≤ exp(−λε) exp 2m 2 2 λ σ = exp −λε + . 2m This bound holds for every λ > 0. Consider the exponent λ2 σ 2 . q(λ) := −λε + 2m Its derivative is λσ 2 q ′ (λ) = −ε + , m and its second derivative is σ2 > 0. q ′′ (λ) = m Therefore q is strictly convex, and its unique minimizer is determined by q ′ (λ) = 0. Solving this equation gives λσ 2 −ε + = 0, m and hence mε λ⋆ = 2 . σ Because ε > 0, this choice satisfies λ⋆ > 0. Substituting it into the exponent gives mε σ 2 mε 2 ε+ q(λ⋆ ) = − 2 σ 2m σ 2 2 2 mε σ m2 ε2 =− 2 + σ 2m σ 4 2 mε mε2 =− 2 + σ 2σ 2 2 mε =− 2. 2σ Consequently, mε2 Pr ξ m − µ ≥ ε ≤ exp − 2 . 2σ We now bound the lower tail. Again fix an arbitrary λ > 0. Since µ − ξ m = −(ξ m − µ), the moment-generating-function estimate established above, applied with the parameter −λ, yields E exp λ(µ − ξ m ) = E exp −λ(ξ m − µ) (−λ)2 σ 2 ≤ exp 2m 2 2 λ σ = exp . 2m
Moreover, ξ m − µ ≤ −ε = µ − ξ m ≥ ε . Using monotonicity of the exponential function followed by Markov’s inequality, we obtain Pr ξ m − µ ≤ −ε = Pr µ − ξ m ≥ ε = Pr exp λ(µ − ξ m ) ≥ exp(λε) ≤ exp(−λε)E exp λ(µ − ξ m ) λ2 σ 2 ≤ exp −λε + . 2m The exponent is the same function q(λ) considered for the upper tail. Choosing mε λ = λ⋆ = 2 σ therefore gives mε2 Pr ξ m − µ ≤ −ε ≤ exp − 2 . 2σ Finally, the two-sided deviation event satisfies |ξ m − µ| ≥ ε = ξ m − µ ≥ ε ∪ ξ m − µ ≤ −ε . Applying the union bound together with the two one-sided estimates yields Pr |ξ m − µ| ≥ ε ≤ Pr ξ m − µ ≥ ε + Pr ξ m − µ ≤ −ε mε2 mε2 ≤ exp − 2 + exp − 2 2σ 2σ mε2 = 2 exp − 2 . 2σ Recalling that m 1 X ξm = ξi m i=1 proves ! m 1 X mε2 Pr ξi − µ ≥ ε ≤ 2 exp − 2 , m i=1 2σ as required.
I.36
Proof of Corollary B.8
Proof. Define the empirical mean vector m 1 X X m := Xi . m i=1 For each coordinate j ∈ {1, . . . , d}, write X m,j := (X m )j
and
µj := (µ)j .
Because vector addition and scalar multiplication in Rd are defined coordinatewise, we have ! m 1 X X m,j = Xi m i=1 j
m
1 X (Xi )j . = m i=1
Moreover, since E[Xi ] = µ, equality of vectors implies equality of their coordinates, and therefore E[(Xi )j ] = (E[Xi ])j = (µ)j = µj for every i ∈ {1, . . . , m} and every j ∈ {1, . . . , d}. Fix an arbitrary coordinate j ∈ {1, . . . , d}. Since the random vectors X1 , . . . , Xm are independent, applying the measurable coordinate projection pj : Rd → R, pj (x) = xj , shows that pj (X1 ), . . . , pj (Xm ) are independent. Since the vectors are also identically distributed, their coordinate projections are identically distributed. Thus (X1 )j , . . . , (Xm )j are independent and identically distributed scalar random variables. Each of them has mean µj , and, by the assumption of the corollary, each of them is σ 2 -sub-Gaussian. Hence all assumptions of Lemma B.7 are satisfied by the scalar sequence ξi := (Xi )j ,
Taking probabilities of both sides gives d [ X m,j − µj ≥ ε . Pr X m − µ ∞ ≥ ε = Pr j=1
Applying the union bound, which does not require the coordinate events to be independent, yields Pr
j=1
Substituting the coordinatewise estimate established above into every term of the sum, we obtain d X mε2 Pr X m − µ ∞ ≥ ε ≤ 2 exp − 2 2σ j=1 d X mε2 =2 1 exp − 2 2σ j=1 mε2 = 2d exp − 2 . 2σ Finally, recalling that
i = 1, . . . , m.
m
Xm =
Applying that lemma gives Pr
d X Xm − µ ∞ ≥ ε ≤ Pr X m,j − µj ≥ ε .
! m 1 X mε2 (Xi )j − µj ≥ ε ≤ 2 exp − 2 . m i=1 2σ
Using the coordinate identity for X m , this can equivalently be written as mε2 Pr X m,j − µj ≥ ε ≤ 2 exp − 2 . 2σ Since the coordinate j was arbitrary, the same estimate holds for every j ∈ {1, . . . , d}. No independence between different coordinates is required here. For each fixed coordinate, independence across the sample index i follows from independence of the random vectors, and the coordinate events will be combined below only by the union bound. By the definition of the ℓ∞ norm on Rd , X m − µ ∞ = max (X m − µ)j 1≤j≤d
= max (X m )j − (µ)j 1≤j≤d
= max X m,j − µj . 1≤j≤d
Because this maximum is taken over the finite set {1, . . . , d}, it is at least ε if and only if at least one of its coordinate terms is at least ε. Therefore, X m − µ ∞ ≥ ε = max X m,j − µj ≥ ε 1≤j≤d
=
d [ j=1
X m,j − µj ≥ ε .
1 X Xi , m i=1
the preceding inequality becomes ! m 1 X mε2 Pr Xi − µ ≥ ε ≤ 2d exp − 2 , m i=1 2σ ∞
which is the desired result.
I.37
Proof of Corollary B.9
Proof. For every t ≥ 1 and every α ∈ It , define the individual failure event α Etα := {∥µ̂α t − µ ∥∞ ≥ εt } .
Also define the uniform failure event at round t by α Et := {∃ α ∈ It : ∥µ̂α t − µ ∥∞ ≥ εt } .
By the meaning of the existential quantifier over the finite index set It , this event is exactly the union of the individual failure events: [ Et = Etα . α∈It
If It = ∅, then the union is empty, and therefore Pr(Et ) = 0 = |It |δt . Thus the desired inequality is immediate in this case. Suppose henceforth that It is nonempty. Since It is finite, it can be written as It = {α1 , . . . , αNt },
Nt := |It |.
Using the representation of Et as a finite union and applying finite subadditivity of probability, we obtain ! Nt [ αk Pr(Et ) = Pr Et k=1
≤
Nt X
where the final two equalities follow from De Morgan’s laws. Since Pr(Et i.o.) = 0, we have ∞ \ [ Pr Etc = 1.
Pr(Etαk ) .
k=1
For every k ∈ {1, . . . , Nt }, the assumed pointwise concentration inequality gives αk k Pr(Etαk ) = Pr(∥µ̂α t − µ ∥∞ ≥ εt ) ≤ δt .
Substituting these bounds into the preceding finite sum yields Pr(Et ) ≤
Nt X
T =1 t≥T
Thus, with probability one, there exists a finite random index T such that Etc for every t ≥ T. For each fixed t, the complement of the uniform failure event can be expanded as !c [ c α Et = Et α∈It
δt
=
k=1
=
Recalling the definition of Et , we have therefore proved α Pr(∃ α ∈ It : ∥µ̂α t − µ ∥∞ ≥ εt ) ≤ |It |δt .
Now suppose that |It |δt < ∞.
t≥1
The bound established above implies, for every t ≥ 1, that 0 ≤ Pr(Et ) ≤ |It |δt . Since all terms are nonnegative, comparison of the corresponding series gives X X Pr(Et ) ≤ |It |δt t≥1
(Etα )c
α∈It
= Nt δt = |It |δt .
X
\ \
α {∥µ̂α t − µ ∥∞ < εt } .
α∈It
It follows that, with probability one, there exists a finite random index T such that, simultaneously for every t ≥ T and every α ∈ It , α ∥µ̂α t − µ ∥∞ < εt . Equivalently, the corresponding uniform failure event occurs only finitely often almost surely. Neither the finite union bound nor the first Borel–Cantelli implication requires independence among the estimators indexed by α or among the failure events corresponding to different rounds t.
I.38
Proof of Lemma B.10
Proof. For every integer N ≥ 1, define the tail-union event
t≥1
BN :=
< ∞. Lemma B.10, equivalently the first Borel–Cantelli lemma, therefore implies Pr(Et i.o.) = 0. The event that the uniform failure events occur infinitely often is the limsup event {Et i.o.} =
∞ [ \ T =1 t≥T
Its complement is consequently c ∞ [ \ {Et i.o.}c = Et T =1 t≥T
=
∞ [
[
T =1
=
c
Et
t≥T
∞ \ [ T =1 t≥T
Etc ,
Et .
t=N
An outcome belongs to BN precisely when at least one of the events Et occurs at an index t ≥ N . Consequently, an outcome belongs to infinitely many of the events Et if and only if it belongs to BN for every N ≥ 1. In event notation, {Et i.o.} =
Et .
∞ [
∞ \
BN =
N =1
∞ [ ∞ \
Et .
N =1 t=N
Indeed, suppose that an outcome belongs to infinitely many of the events Et . Then, for every N ≥ 1, there exists an index t ≥ N for which that outcome belongs to Et . Hence the outcome T belongs to BN for every N , and therefore it belongs to N ≥1 BN . Conversely, suppose that an outcome belongs to only finitely many of the events Et . Then there exists an integer N0 ≥ 1 such that none of the events Et with t ≥ N0 occurs for that outcome. It follows that the outcome does not belong to BN0 =
∞ [ t=N0
Et ,
T and hence it cannot belong to N ≥1 BN . This proves the displayed identity. By countable subadditivity of probability, for every N ≥ 1, ! ∞ [ Pr(BN ) = Pr Et
The preceding conclusion can equivalently be expressed as an eventual-correctness statement. By De Morgan’s laws, !c ∞ [ ∞ \ c {Et i.o.} = Et N =1 t=N
t=N
≤
∞ X
=
Pr(Et ).
t=N
=
By assumption, ∞ X
Pr(Et ) < ∞.
N X
and let S :=
Pr(Et )
for N ≥ 1,
∞ X
Pr(Et ).
The convergence of the series means that SN −→ S
as N −→ ∞.
For every N ≥ 1, the corresponding tail can be written as ∞ X
we have
∞ \ ∞ [
Pr
! Etc
= 1.
Thus, with probability one, there exists a finite random index N such that Etc for every t ≥ N. Equivalently, with probability one, the random set {t ≥ 1 : Et occurs}
t=1
Pr(Et ) −
N −1 X t=1
t=1
is finite. For the final assertion, let Ft denote the event that the finite identification procedure is incorrect at round t. The assumed per-round failure guarantee states that Pr(Ft ) ≤ δt
Pr(Et )
= S − SN −1 . Since SN −1 → S, it follows that ∞ X
Etc .
N =1 t=N
t=1
t=N
Et
N =1 t=N ∞ \ ∞ [
Pr(Et i.o.) = 0,
SN :=
Pr(Et ) =
!c
Since
Because this is a convergent series of nonnegative real numbers, its tails converge to zero. To write this explicitly, define
∞ X
∞ [
N =1 t=N
t=1
S0 := 0,
∞ [
for every t ≥ 1.
Since all terms are nonnegative, comparison of series gives ∞ X
Pr(Ft ) ≤
t=1
∞ X
δt
t=1
< ∞. Pr(Et ) −→ 0
as N −→ ∞.
t=N
Applying the result just proved to the sequence of failure events (Ft )t≥1 gives
Moreover, for every fixed N ≥ 1, {Et i.o.} =
∞ \
BK ⊆ BN .
K=1
Monotonicity of probability therefore gives 0 ≤ Pr(Et i.o.) ≤ Pr(BN ) ∞ X ≤ Pr(Et ). t=N
This inequality holds for every N ≥ 1. Letting N → ∞ and using the convergence of the tail sums yields 0 ≤ Pr(Et i.o.) ≤ 0. Consequently, Pr(Et i.o.) = 0.
Pr(Ft i.o.) = 0. Taking complements and expanding the infinitely-often event as above, we obtain ! ∞ \ ∞ [ c Pr Ft = 1. T =1 t=T
Therefore, with probability one, there exists a finite random round T such that Ftc
for every t ≥ T.
Since Ftc is exactly the event that the identification procedure is correct at round t, this means that Pr(∃ T < ∞ such that the identification procedure is correct for every t ≥ T ) = 1. Hence only finitely many rounds are incorrect almost surely. No independence assumption among the events Et , or among the identification failure events Ft , is required.
I.39
Proof of Proposition D.2
σw = σek ◦ · · · ◦ σe1
does not change the raw transport. For the geometric realization of a graph, based edge-path homotopy is generated by precisely these insertions and deletions of immediate backtracks, together with insertion or deletion of constant vertex pauses, which do not introduce an edge and hence have identity transport. Equivalently, two based oriented edge loops represent the same element of π1 (|Xund |, o) if and only if one can pass from one edge word to the other by a finite sequence of these elementary moves. Let w and w′ represent the same based homotopy class. Then there exist oriented edge words
is again an element of Sn , while the empty word has transport
w = w(0) , w(1) , . . . , w(L) = w′
σ∅o = Id[n] .
such that, for every ℓ ∈ {0, . . . , L − 1}, the word w(ℓ+1) is obtained from w(ℓ) by one elementary insertion or deletion. The calculation above gives
Proof. Fix a base observation o ∈ O. Every based edge loop in |Xund | can be represented by an oriented edge word w = e1 e2 · · · ek , where src(e1 ) = o, tgt(ej ) = src(ej+1 ) for j = 1, . . . , k − 1, tgt(ek ) = o. The empty word ∅o represents the constant loop at o. Since every σe is an element of Sn , the composition
We first prove that σw is unchanged by the elementary reductions defining edge-path homotopy in the geometric realization of a graph. Suppose that an oriented edge word contains an adjacent backtracking pair and can be written as w = u e ē v, where u and v are possibly empty oriented edge words and all displayed concatenations are composable. Traversal occurs from left to right, so the transport of the complete word is obtained by composing the transports in the reverse written order. Hence σw = σv ◦ σē ◦ σe ◦ σu . Assumption D.1 gives σē = σe−1 , σē ◦ σe = σe−1 ◦ σe = Id[n] . Substituting this identity into the preceding expression yields σw = σv ◦ σē ◦ σe ◦ σu = σv ◦ Id[n] ◦ σu = σv ◦ σu . The word obtained by deleting the adjacent pair eē is uv, and its transport is exactly σuv = σv ◦ σu . Thus σueēv = σuv . The other possible immediate backtrack is handled in the same way. If w = u ē e v, then σw = σv ◦ σe ◦ σē ◦ σu = σv ◦ σe ◦ σe−1 ◦ σu = σv ◦ Id[n] ◦ σu = σuv . Consequently, insertion or deletion of either adjacent pair or
for every ℓ ∈ {0, . . . , L − 1}.
Applying transitivity of equality along this finite sequence gives σw = σw(0) = σw(1) = ··· = σw(L) = σw′ . Therefore the value assigned to [w] depends only on the based homotopy class and not on the chosen oriented edge-word representative. Hence Holoraw o ([w]) := σw
and therefore
eē
σw(ℓ) = σw(ℓ+1)
ēe
defines a well-defined map Holoraw : π1 (|Xund |, o) −→ Sn . o We next verify compatibility with the group operations. Let w1 and w2 be based oriented edge loops at o, where w1 is traversed first and w2 is traversed second. Under the pathcomposition convention used throughout the paper, their product in the fundamental group is represented by w2 ◦ w1 . If w1 = e1 · · · ek and w2 = f1 · · · fℓ , then the chronological edge word for w2 ◦ w1 is e1 · · · ek f1 · · · fℓ . Its raw transport is therefore σw2 ◦w1 = σfℓ ◦ · · · ◦ σf1 ◦ σek ◦ · · · ◦ σe1 = σfℓ ◦ · · · ◦ σf1 ◦ σek ◦ · · · ◦ σe1 = σw2 ◦ σw1 . It follows that Holoraw [w2 ] [w1 ] = Holoraw [w2 ◦ w1 ] o o = σw2 ◦w1 = σw2 ◦ σw1 raw = Holoraw o ([w2 ]) ◦ Holoo ([w1 ]).
Thus Holoraw preserves multiplication. It also preserves the o identity, because the identity element of the fundamental group is represented by the constant loop and Holoraw o ([∅o ]) = σ∅o
I.40
Proof of Lemma D.3
Proof. Fix an arbitrary directed edge e ∈ E, and write o := src(e)
o′ := tgt(e).
and
By Assumption D.1, the paired edge ē also belongs to E and satisfies
= Id[n] , which is the identity element of Sn . For completeness, the inverse operation is also represented correctly. If w = e1 e2 · · · ek ,
src(ē) = o′ ,
σē = σe−1 .
tgt(ē) = o,
Since both e and ē are edges of the directed support graph, both are feasible. Theorem 3.9 therefore gives well-defined quotient transports τe : Co −→ Co′
then the reversed loop is represented by
τē : Co′ −→ Co
and
satisfying
w−1 = ēk ēk−1 · · · ē1 .
τe ([i]o ) = [σe (i)]o′
Its transport is
for every i ∈ [n]
and
σw−1 = σē1 ◦ σē2 ◦ · · · ◦ σēk = σe−1 ◦ σe−1 ◦ · · · ◦ σe−1 1 2 k = σek ◦ σek−1 ◦ · · · ◦ σe1
−1
−1 = σw .
τē ([j]o′ ) = [σē (j)]o for every j ∈ [n]. In particular, the codomain of τe is the domain of τē , and the codomain of τē is the domain of τe , so both compositions appearing in the statement are well defined. Let c ∈ Co be arbitrary. Because Co = [n]/Π⋆o ,
Therefore
there exists a layer i ∈ [n] such that −1 −1 Holoraw ) = Holoraw ]) o ([w] o ([w = σw−1 −1 = σw
c = [i]o . Using first the quotient-transport formula for e and then the corresponding formula for ē, we obtain
−1 = Holoraw . o ([w])
(τē ◦ τe )(c) = (τē ◦ τe )([i]o ) = τē τe ([i]o ) = τē [σe (i)]o′
Hence Holoraw is a group homomorphism. o Finally, define
= [σē (σe (i))]o .
Graw := Holoraw π1 (|Xund |, o) . o o Assumption D.1 gives This set is a subgroup of Sn . Indeed, raw Id[n] = Holoraw o ([∅o ]) ∈ Go .
If g1 , g2 ∈ Graw o , then there exist based loop classes [w1 ] and [w2 ] such that g1 = Holoraw o ([w1 ])
and
g2 = Holoraw o ([w2 ]).
Using the homomorphism property, raw g2 ◦ g1 = Holoraw o ([w2 ]) ◦ Holoo ([w1 ]) = Holoraw o ([w2 ] [w1 ]) ∈ Graw . o
Likewise, −1 g1−1 = Holoraw o ([w1 ]) −1 = Holoraw ) o ([w1 ] raw ∈ Go .
Thus Graw ≤ Sn , and by definition it is the raw topological o holonomy group at the base observation o.
σē = σe−1 , and hence
σē (σe (i)) = σe−1 (σe (i)) = (σe−1 ◦ σe )(i) = Id[n] (i)
= i. Substituting this identity into the preceding quotient-class calculation yields (τē ◦ τe )(c) = [σē (σe (i))]o = [i]o = c. Since c ∈ Co was arbitrary, the two maps agree at every element of their common domain, and therefore τē ◦ τe = IdCo = IdCsrc(e) . We prove the reverse composition in the same explicit manner. Let c′ ∈ Co′ be arbitrary. Since Co′ = [n]/Π⋆o′ ,
there exists j ∈ [n] such that c′ = [j]o′ . Applying the quotient-transport formulas first to ē and then to e gives (τe ◦ τē )(c′ ) = (τe ◦ τē )([j]o′ ) = τe τē ([j]o′ ) = τe [σē (j)]o = [σe (σē (j))]o′ . Again using σē = σe−1 , we have σe (σē (j)) = σe (σe−1 (j)) = (σe ◦ σe−1 )(j) = Id[n] (j)
I.41
Proof. Fix an arbitrary base observation o ∈ O. Let w = e 1 e 2 · · · ek be an oriented edge loop in Xund based at o. Thus there exist observations o0 , o1 , . . . , ok with o0 = ok = o, such that, for every j ∈ {1, . . . , k}, src(ej ) = oj−1
(τe ◦ τē )(c′ ) = [σe (σē (j))]o′ = [j]o′ = c′ . ′
Because c ∈ C was arbitrary, it follows that τe ◦ τē = IdCo′ = IdCtgt(e) . The two identities show that τē is simultaneously a left inverse and a right inverse of τe . We now verify explicitly that this makes τe bijective. Suppose that c1 , c2 ∈ Co satisfy τe (c1 ) = τe (c2 ). Applying τē to both sides and using the first composition identity gives c1 = (τē ◦ τe )(c1 ) = τē (τe (c1 )) = τē (τe (c2 )) = (τē ◦ τe )(c2 ) = c2 . Thus τe is injective. Next, let c′ ∈ Co′ be arbitrary. The second composition identity gives c′ = (τe ◦ τē )(c′ ) = τe τē (c′ ) .
τw = τek ◦ τek−1 ◦ · · · ◦ τe1 : Co −→ Co . By Lemma D.3, every edge transport τej : Coj−1 −→ Coj is a bijection, with inverse τe−1 = τēj . j
o′
Therefore c′ is the image under τe of the class τē (c′ ) ∈ Co , so τe is surjective. Hence τe is a bijection. Since τē is both its left and right inverse, uniqueness of the inverse map yields τē = τe−1 . The same argument, with e and ē interchanged, shows that τē is also a bijection and that τe = τē−1 . Every oriented traversal of an edge of Xund is represented by one member of a paired set {e, ē}. Since the quotient transport in either orientation is bijective, every quotient transport along an edge of Xund is a bijection.
tgt(ej ) = oj .
and
The associated quotient transport is
= j. Consequently,
Proof of Proposition D.4
A finite composition of bijections is a bijection. Hence τw is a bijection from Co to itself, and therefore τw ∈ Sym(Co ). For the empty loop ∅o , we use the empty-composition convention τ∅o = IdCo . Thus the proposed assignment takes values in Sym(Co ) for every based oriented edge loop, including the empty loop. We next prove that the transport is invariant under the elementary edge-word reductions representing based homotopy in the geometric realization of the graph. Suppose first that a based oriented edge loop contains an immediate backtrack and can be written as w = u e ē v. Let p := src(e)
and
q := tgt(e).
Then u is a path from o to p, the edge e is traversed from p to q, the paired edge ē is traversed from q back to p, and v is a path from p to o. Consequently, the corresponding quotient transports have the compatible types τu : Co → Cp , τe : Cp → Cq , τē : Cq → Cp , τv : Cp → Co . Because the edges are executed in the written order while maps compose from right to left, the transport of w is τw = τv ◦ τē ◦ τe ◦ τu . Lemma D.3 gives τē ◦ τe = IdCp .
Substituting this identity into the preceding expression yields τw = τv ◦ τē ◦ τe ◦ τu = τv ◦ IdCp ◦ τu = τv ◦ τu . Deleting the adjacent pair eē produces the based loop uv. Since u is executed first and v is executed second, its transport is τuv = τv ◦ τu . Therefore τueēv = τuv . The other orientation of immediate backtracking is handled similarly. Suppose
Thus the value τw depends only on the based homotopy class [w], not on the chosen oriented edge-word representative. Consequently, Holoquot ([w]) := τw o defines a well-defined map Holoquot : π1 (|Xund |, o) −→ Sym(Co ). o We now verify that this map preserves the group operation. Let w1 and w2 be arbitrary oriented edge loops based at o, with w1 executed first and w2 executed second. Write w1 = e1 e2 · · · ek
It follows that τw = τv ◦ τe ◦ τē ◦ τu = τv ◦ τu = τuv . Hence insertion or deletion of either adjacent pair ēe
does not change the quotient transport. Insertion or deletion of a constant pause at a vertex also does not change the transport, because the empty path at an observation p acts by τ∅p = IdCp . For the geometric realization of a graph, based edge-path homotopy is generated by these elementary insertions and deletions of immediate backtracks, together with constant vertex pauses. Equivalently, two based oriented edge loops represent the same element of π1 (|Xund |, o) precisely when their edge words are connected by a finite sequence of such elementary moves. Let w and w′ represent the same based homotopy class. Then there exist based oriented edge loops w=w
(0)
,w
(1)
,...,w
(L)
=w
′
such that each w(ℓ+1) is obtained from w(ℓ) by one elementary insertion or deletion. The calculations above imply τw(ℓ) = τw(ℓ+1) Therefore
and its chronological edge word is e1 e2 · · · ek f1 f2 · · · fℓ . Applying the definition of quotient path transport gives
= τw2 ◦ τw1 .
= τv ◦ IdCq ◦ τu
or
w2 ◦ w1 ,
τw2 ◦w1 = τfℓ ◦ · · · ◦ τf1 ◦ τek ◦ · · · ◦ τe1 = τfℓ ◦ · · · ◦ τf1 ◦ τek ◦ · · · ◦ τe1
τe ◦ τē = IdCq .
eē
w2 = f1 f2 · · · fℓ .
Under the path-composition convention used in Definition 3.2, the concatenated loop is denoted by
w = u ē e v. In this case, u ends at q = tgt(e), the edge ē moves from q to p = src(e), the edge e returns from p to q, and v begins at q. Thus τw = τv ◦ τe ◦ τē ◦ τu . The second identity in Lemma D.3 gives
and
for every ℓ ∈ {0, . . . , L − 1}.
The multiplication in the fundamental group follows the same convention, so the class product of [w1 ] followed by [w2 ] is represented by w2 ◦ w1 . Hence Holoquot [w2 ][w1 ] = Holoquot [w2 ◦ w1 ] o o = τw2 ◦w1 = τw2 ◦ τw1 ([w1 ]). ([w2 ]) ◦ Holoquot = Holoquot o o Thus Holoquot preserves multiplication. It also preserves the o identity element, because the identity of π1 (|Xund |, o) is represented by the constant loop ∅o , and ([∅o ]) = τ∅o Holoquot o = IdCo . For completeness, we also verify explicitly that loop reversal is sent to the inverse permutation. Let w = e1 e2 · · · ek . The reversed loop is represented by w−1 = ēk ēk−1 · · · ē1 . By the definition of transport along the reversed edge word, τw−1 = τē1 ◦ τē2 ◦ · · · ◦ τēk .
τw = τw(0) = τw(1) = ··· = τw(L) = τw′ .
Using Lemma D.3 at every edge gives τw−1 = τe−1 ◦ τe−1 ◦ · · · ◦ τe−1 1 2 k = τek ◦ τek−1 ◦ · · · ◦ τe1 = τw−1 .
−1
Therefore
The transport along the reversed path is Holoquot ([w]−1 ) = Holoquot ([w−1 ]) o o
ση−1 = σē1 ◦ σē2 ◦ · · · ◦ σēk
= τw−1
= σe−1 ◦ σe−1 ◦ · · · ◦ σe−1 1 2 k
= τw−1
= σek ◦ σek−1 ◦ · · · ◦ σe1
= Holoquot ([w])−1 . o This is consistent with, and also follows from, the multiplication and identity identities above. Hence
= ση−1 , where the second equality follows from σēj = σe−1 j
Holoquot o is a group homomorphism. Finally, define Gquot := Holoquot π1 (|Xund |, o) . o o Every element of this image lies in Sym(Co ). Moreover,
for every j ∈ {1, . . . , k}. Thus the transport along the reversed reference path is exactly the inverse of the transport along the reference path. The same conclusion holds on the stable quotient. By Lemma D.3, every edge transport
IdCo = Holoquot ([∅o ]) ∈ Gquot . o o Let g1 , g2 ∈ Gquot . By the definition of the image, there exist o based loop classes [w1 ] and [w2 ] such that g1 = Holoquot ([w1 ]) o
and
τej : Coj−1 −→ Coj is a bijection and satisfies τēj = τe−1 . j
g2 = Holoquot ([w2 ]). o Consequently,
The homomorphism property gives g2 ◦ g1 = Holoquot ([w2 ]) ◦ Holoquot ([w1 ]) o o quot = Holoo [w2 ][w1 ]
τη = τek ◦ τek−1 ◦ · · · ◦ τe1 : Co −→ Co′ is a bijection. Moreover,
∈ Gquot . o Likewise,
τη−1 = τē1 ◦ τē2 ◦ · · · ◦ τēk = τe−1 ◦ τe−1 ◦ · · · ◦ τe−1 1 2 k
([w1 ])−1 g1−1 = Holoquot o
= τek ◦ τek−1 ◦ · · · ◦ τe1
([w1 ]−1 ) = Holoquot o ∈ Gquot . o Thus the image contains the identity and is closed under composition and inverses. Therefore Gquot ≤ Sym(Co ), o and, by definition, it is the quotient topological holonomy group at the base observation o.
I.42
Proof. Fix an oriented path η from o to o′ . Write η = e 1 e 2 · · · ek ,
= τη−1 . In particular, conjugation by τη maps permutations of Co to permutations of Co′ . We first prove the raw-layer identity. We use the notation −1 −1 raw ση Graw . o ση := ση ◦ g ◦ ση : g ∈ Go Let be arbitrary. By the definition of the raw topological holonomy group, there exists a based loop class [w] ∈ π1 (|Xund |, o)
where there are observations o = o0 , o1 , . . . , ok = o′ such that and
−1
g ∈ Graw o
Proof of Proposition D.5
src(ej ) = oj−1
−1
tgt(ej ) = oj
for every j ∈ {1, . . . , k}. Under Assumption D.1, the reversed oriented path is η −1 = ēk ēk−1 · · · ē1 , which begins at o′ and ends at o. By the definition of raw path transport, ση = σek ◦ σek−1 ◦ · · · ◦ σe1 .
with an oriented edge-loop representative w such that g = Holoraw o ([w]) = σw . Consider the oriented path λ := η ◦ w ◦ η −1 . Under the path-composition convention of Definition 3.2, the rightmost path is executed first. Hence λ first follows η −1 from o′ to o, then traverses the loop w based at o, and finally follows η from o back to o′ . Therefore λ is a loop based at o′ , and [λ] ∈ π1 (|Xund |, o′ ).
Repeated use of the path-composition identity gives σλ = ση◦w◦η−1 = ση ◦ σw◦η−1 = ση ◦ σw ◦ ση−1 = ση ◦ g ◦ ση−1 . Since λ is a loop based at o′ , its transport belongs to the raw holonomy group at o′ . More explicitly, ση ◦ g ◦ ση−1 = σλ = Holoraw o′ ([λ]) . ∈ Graw ′ o
We now prove the corresponding quotient-layer identity. Define τη Gquot τη−1 := τη ◦ q ◦ τη−1 : q ∈ Gquot . o o The domains and codomains in this composition are compatible. Indeed, τη−1 : Co′ −→ Co , every q ∈ Gquot o is a permutation from Co to itself, and τη : Co −→ Co′ . Thus
Since g ∈ Graw was arbitrary, this proves o
τη ◦ q ◦ τη−1
−1 raw ση Graw o ση ⊆ Go′ .
is a permutation of Co′ . Let
For the reverse inclusion, let
q ∈ Gquot o
g ′ ∈ Graw o′
be arbitrary. By the definition of the quotient topological holonomy group, there exists a based loop class
be arbitrary. There exists a based loop class [w′ ] ∈ π1 (|Xund |, o′ )
[w] ∈ π1 (|Xund |, o) ′
with an oriented edge-loop representative w satisfying g
′
with an oriented edge-loop representative w such that
′ = Holoraw o′ ([w ]) = σw′ .
Define
κ := η −1 ◦ w′ ◦ η. This path first follows η from o to o′ , then traverses w′ at o′ , and finally follows η −1 from o′ back to o. Hence κ is a loop based at o. Its raw transport is
q = Holoquot ([w]) = τw . o Using the loop λ = η ◦ w ◦ η −1 based at o′ , quotient path composition gives
σκ = ση−1 ◦w′ ◦η = ση−1 ◦ σw′ ◦ ση
τλ = τη◦w◦η−1 = τη ◦ τw◦η−1 = τη ◦ τw ◦ τη−1
= ση−1 ◦ g ′ ◦ ση .
= τη ◦ q ◦ τη−1 .
Because κ is a loop based at o,
Since λ is a loop based at o′ ,
σκ ∈ Graw o . Set
τη ◦ q ◦ τη−1 = τλ = Holoquot o′ ([λ])
g := σκ = ση−1 ◦ g ′ ◦ ση .
Then g ∈ Graw o . Conjugating this identity by ση gives ση ◦ g ◦ ση−1 = ση ◦ ση−1 ◦ g ′ ◦ ση ◦ ση−1 = ση ◦ ση−1 ◦ g ′ ◦ ση ◦ ση−1 = Id[n] ◦ g ′ ◦ Id[n]
∈ Gquot o′ . As q ∈ Gquot was arbitrary, this proves o τη Gquot τη−1 ⊆ Gquot o o′ . Conversely, let q ′ ∈ Gquot o′
′
=g.
be arbitrary. There exists a based loop class
Therefore g
′
−1 ∈ ση Graw o ση .
Since g ′ ∈ Graw o′ was arbitrary, it follows that
[w′ ] ∈ π1 (|Xund |, o′ ) with an oriented edge-loop representative w′ such that
raw −1 Graw o′ ⊆ ση Go ση .
Combining the two inclusions yields raw −1 Graw o′ = ση Go ση .
′ q ′ = Holoquot o′ ([w ]) = τw′ .
For the loop κ = η −1 ◦ w′ ◦ η
based at o, quotient path composition gives τκ = τη−1 ◦w′ ◦η = τη−1 ◦ τw′ ◦ τη = τη−1 ◦ q ′ ◦ τη .
and similarly −1 −1 λ−1 q ◦ τ e ◦ λ p ◦ λp ◦ τ e ◦ λ q −1 −1 = λ−1 ◦ τe ◦ λq q ◦ τ e ◦ λp ◦ λ p −1 = λ−1 q ◦ τe ◦ τe ◦ λq
Since κ is a loop based at o,
= λ−1 q ◦ IdCq ◦ λq
τκ ∈ Gquot . o Define
q := τκ = τη−1 ◦ q ′ ◦ τη .
Then q ∈ Gquot , and o τη ◦ q ◦ τη−1 = τη ◦ τη−1 ◦ q ′ ◦ τη ◦ τη−1 = τη ◦ τη−1 ◦ q ′ ◦ τη ◦ τη−1 = IdCo′ ◦ q ′ ◦ IdCo′ = q′ . Therefore
q ′ ∈ τη Gquot τη−1 . o
was arbitrary, we obtain Since q ′ ∈ Gquot o′ Gquot ⊆ τη Gquot τη−1 . o o′ Combining the two inclusions gives Gquot = τη Gquot τη−1 . o o′ Therefore both the raw and stable-quotient topological holonomy groups are transported from one base observation to another by conjugation through the corresponding referencepath transport. In particular, changing the base observation preserves the holonomy structure up to the relabeling induced by the chosen path η.
I.43
Proof of Proposition D.6
= IdCbq . Consequently, the inverse of the recovered map is well defined and is given by −1 τbe−1 = λ−1 p ◦ τe ◦ λ q .
The synchronization construction below uses only the recovered maps and their inverses; the unknown proof-level gauges λo are not required by the procedure itself. For each undirected tree adjacency {p, q} ∈ E(T ), let e{p,q} denote the directed edge selected in the hypothesis. Define the transport used for an ordered traversal from p to q by ( τbe{p,q} , if src(e{p,q} ) = p and tgt(e{p,q} ) = q, Ψp→q := , if src(e{p,q} ) = q and tgt(e{p,q} ) = p. τbe−1 {p,q} In either case, Ψp→q : Cbp −→ Cbq is a bijection. Reversing the ordered traversal exchanges the two cases in the definition, and therefore Ψq→p = Ψ−1 p→q . In particular, adjacent observations in T have class sets of equal cardinality. Since T is connected, repeated application along a tree path shows that
Proof. Work in the observation-wise local class coordinates after exact recovery of the selected tree-edge maps. Thus, for each observation o in the chosen connected component, the recovered class set is Cbo , and, for every selected feasible directed tree edge a e = (p − → q), the recovered map satisfies
Fix an arbitrary observation o ∈ V (T ). Because T is a tree, there exists a unique simple path from the root to o. Write its ordered vertex sequence as
b b τbe = τeλ = λ−1 q ◦ τe ◦ λp : Cp −→ Cq .
Define the corresponding root-to-o synchronization map by
The maps λp and λq are bijections, and the quotient transport τe is bijective by the hypothesis on the tree scaffold. Hence τbe is also bijective. Indeed, the map
Ψo := Ψok−1 →ok ◦Ψok−2 →ok−1 ◦· · ·◦Ψo0 →o1 : Cboroot −→ Cbo .
|Cbo | = |Cboroot |
o0 = oroot ,
−1 = λ−1 p ◦ τe ◦ τe ◦ λp
= λ−1 p ◦ IdCp ◦ λp = IdCbp ,
o1 ,
...,
ok = o.
For the root itself, define
−1 b b λ−1 p ◦ τe ◦ λq : Cq −→ Cp
is a two-sided inverse, because −1 −1 λ−1 p ◦ τe ◦ λq ◦ λq ◦ τe ◦ λp −1 −1 = λ−1 ◦ τe ◦ λ p p ◦ τe ◦ λq ◦ λq
for every o ∈ V (T ).
Ψoroot := IdCbo
.
root
Every factor in the composition defining Ψo is a bijection, so Ψo is a bijection. Its inverse is obtained by traversing the unique tree path in the reverse direction: Ψ−1 o = Ψo1 →o0 ◦ Ψo2 →o1 ◦ · · · ◦ Ψok →ok−1 . Indeed, using Ψoj →oj−1 = Ψ−1 oj−1 →oj
and successively cancelling adjacent inverse pairs gives
Equivalently, for every ĉ ∈ Cbp ,
Ψ−1 o ◦ Ψo = Ψo1 →o0 ◦ · · · ◦ Ψok →ok−1 ◦ Ψok−1 →ok ◦ · · · ◦ Ψo0 →o1 = Ψo1 →o0 ◦ · · · ◦ Ψok−1 →ok−2 ◦ IdCbo
k−1
◦ Ψok−2 →ok−1 ◦ · · · ◦ Ψo0 →o1
= ··· = Ψo1 →o0 ◦ Ψo0 →o1 = IdCbo . root
Consequently, if the selected directed edge points from p to q, labels are propagated through the recovered bijection τbe{p,q} ; if it points from q to p, labels are propagated from . This is exactly the propagation rule p to q through τbe−1 {p,q} stated in the proposition. We now prove uniqueness. Suppose that n o ℓeo : Cbo → L o∈V (T )
The reverse composition satisfies Ψo ◦ Ψ−1 o = IdCbo by the same cancellation argument. The uniqueness of the simple root-to-o path is precisely what makes Ψo unambiguous; no choice among multiple tree paths is required. Let L be any finite label set satisfying |L| = |Cbo |, root
and choose an arbitrary root labeling, that is, an arbitrary bijection ℓoroot : Cboroot −→ L. For every o ∈ V (T ), define ℓo := ℓo ◦ Ψ−1 : Cbo −→ L. root
ℓq Ψp→q (ĉ) = ℓp (ĉ).
o
Since both factors are bijections, each ℓo is a bijection. This family is the desired root-relative coordinate convention. Explicitly, if a root class ĉroot is transported along the unique tree path to ĉo = Ψo (ĉroot ), then its label at o is ℓo (ĉo ) = ℓoroot Ψ−1 o (ĉo ) = ℓoroot Ψ−1 o (Ψo (ĉroot )) = ℓoroot IdCbo (ĉroot ) root
= ℓoroot (ĉroot ). Thus the root label is preserved when it is propagated along the tree. The same statement can be written locally on each rooted tree edge. Let p be the parent of q in the tree rooted at oroot . The unique root-to-q path is obtained by appending the traversal p → q to the unique root-to-p path, and hence Ψq = Ψp→q ◦ Ψp . Taking inverses gives Ψ−1 q = Ψp→q ◦ Ψp
−1
is another family of coordinate maps satisfying the same root labeling ℓeoroot = ℓoroot and the same tree-edge compatibility equations ℓeq ◦ Ψp→q = ℓep for every parent–child pair p, q in the rooted tree. We show that ℓeo = ℓo for every o ∈ V (T ). The equality holds at the root by assumption. Suppose it holds at a vertex p, and let q be any child of p. Since Ψp→q is bijective, the compatibility equation uniquely determines the coordinate map at q: ℓeq = ℓep ◦ Ψ−1 p→q = ℓp ◦ Ψ−1 p→q . On the other hand, the previously established relation ℓq ◦ Ψp→q = ℓp implies Consequently, ℓeq = ℓq . Induction on the graph distance from the root therefore gives ℓeo = ℓo for every vertex o of the tree. Hence the root-relative coordinate convention is unique once the root labeling, the spanning tree, and the selected directed tree edges are fixed. Finally, let a f = (p − → q) be any feasible directed edge not used as a tree edge, and let τbf : Cbp −→ Cbq
−1 = Ψ−1 p ◦ Ψp→q .
Therefore ℓq ◦ Ψp→q = ℓoroot ◦ Ψ−1 q ◦ Ψp→q
denote its recovered local edge map. Its expression in the common root-relative label set is the map Df := ℓq ◦ τbf ◦ ℓ−1 p : L −→ L.
−1 = ℓoroot ◦ Ψ−1 p ◦ Ψp→q ◦ Ψp→q
= ℓoroot ◦ Ψ−1 p ◦ IdCbp = ℓoroot ◦ Ψ−1 p = ℓp .
ℓq = ℓp ◦ Ψ−1 p→q .
Using and
ℓq = ℓoroot ◦ Ψ−1 q −1 ℓ−1 p = Ψp ◦ ℓoroot ,
we obtain Df = ℓoroot ◦ Ψ−1 bf ◦ Ψp ◦ ℓ−1 q ◦τ oroot . The middle map Ψ−1 bf ◦ Ψp : Cboroot −→ Cboroot q ◦τ is the transport obtained by moving from the root to p along the unique tree path, applying the off-tree edge f , and returning from q to the root along the reverse of the unique tree path. Thus it is the residual transport around the closed walk determined by f and the tree scaffold, and Df is precisely that residual transport written in the chosen root labels. The off-tree edge is perfectly consistent with the identification induced by the tree if and only if this residual map is the identity. Indeed, Df = IdL ⇐⇒ ℓq ◦ τbf ◦ ℓ−1 p = IdL ⇐⇒ τbf = ℓ−1 q ◦ ℓp −1 ⇐⇒ τbf = Ψq ◦ ℓ−1 oroot ◦ ℓoroot ◦ Ψp
⇐⇒ τbf = Ψq ◦ IdCbo
root
◦ Ψ−1 p
⇐⇒ τbf = Ψq ◦ Ψ−1 p . The map on the right is exactly the transport from p to q predicted by the synchronized tree coordinates. Hence Df provides a cycle-consistency diagnostic. More generally, a nonidentity Df records the residual effect around the corresponding cycle rather than being erased by the gauge choice. If τbf is itself bijective, then Df ∈ Sym(L), and the displayed conjugacy shows that it is the holonomy permutation of the associated closed walk, expressed in rootrelative coordinates. To make the dependence on the arbitrary root labeling explicit, suppose that the root labels are changed by a permutation χ ∈ Sym(L),
Algorithm 1: HMRL-D: diagnostic quotient-lifted RL in local coordinates 1: Input: local label sets {Cbo }o∈O ; calibrated prototypes {ϕeo }o∈O ; diagnostic protocols {Uprobe (o)}o∈O ; transition-anchored diagnostic interface; edge-count updater; RL backbone (RLSelect, RLUpdate). 2: Initialize local edge-count matrices {Ne }e∈E and local edge maps {b τe }e∈E . 3: Run transition-anchored diagnostics until every required edge–class row satisfies the chosen calibration criterion. 4: Freeze {b τe }e∈E and reinitialize the RL backbone, including its parameters, replay data, counters, and optimizer state. 5: Reset the environment and obtain a resettable checkpoint with initial observation o0 . 6: Diagnose the checkpoint and infer b c0 ∈ Cbo0 . 7: for t = 0, 1, 2, . . . do 8: Form the local-coordinate RL state
zbt = (ot , b ct ). 9:
at ← RLSelect(b zt ). 10: 11:
= χ ◦ ℓoroot ◦ Ψ−1 o = χ ◦ ℓo . The corresponding off-tree diagnostic becomes Df′ = ℓ′q ◦ τbf ◦ (ℓ′p )−1 = χ ◦ ℓq ◦ τbf ◦ (χ ◦ ℓp )−1 −1 = χ ◦ ℓq ◦ τbf ◦ ℓ−1 p ◦χ
= χ ◦ Df ◦ χ−1 . Thus all root-relative coordinates and off-tree diagnostics are completely determined by the chosen root labeling, the selected directed tree edges, and the tree T . Changing only the arbitrary root names produces the expected common relabeling, or conjugation, and introduces no additional ambiguity.
Preserve or clone the source checkpoint when an edgecalibration sample is scheduled. Execute at and observe (Yt , ot+1 , done).
12: 13:
ℓ′oroot := χ ◦ ℓoroot .
The propagated coordinate map at an arbitrary observation o is then ℓ′o = ℓ′oroot ◦ Ψ−1 o
Select
14: 15: 16:
if a post-calibration diagnostic audit is scheduled then Diagnose the preserved source checkpoint and the resulting target checkpoint and record whether the frozen map is consistent; do not update the RL state or the frozen transport table. end if Propagate b ct+1 = τbet (b ct ). Form zbt+1 = (ot+1 , b ct+1 ).
17:
Run RLUpdate(b zt , at , Yt , zbt+1 , done).
18: 19: 20:
if done then Reset the environment and obtain a new resettable checkpoint. Diagnose the new initial checkpoint to obtain (ot+1 , b ct+1 ).
21: end if 22: end for
Algorithm 2: A-HMRL: passive quotient-lifted RL from standard tuples bθ , PbO,θ , Tbθ ); label1: Input: history encoder qψ ; models (R refinement routine RefineLabels; replay buffer B; RL backbone (RLSelect, RLUpdate). 2: Initialize one or more local labels for each encountered observation and initialize all model and RL parameters. 3: Reset the environment, observe o0 , and set H0 = (o0 ). 4: for t = 0, 1, 2, . . . do 5: Infer
qψ (· | Ht )
on Cbot ,
6:
and select or sample a current local label b ct . Form zbt = (ot , b ct ).
7:
Select at ← RLSelect(b zt ).
8:
Execute at and observe (Yt , ot+1 , done).
9:
Update the observation–action history Ht+1 = Ht · (at , ot+1 ).
10:
Infer a provisional next label b ct+1 ∼ qψ (· | Ht+1 ).
11:
Store (Ht , b ct , at , Yt , ot+1 , b ct+1 )
12:
13:
14:
and posterior information in B. Update the representation models by minimizing Lrew + λbase Lbase + λedge Ledge + λord Lord + λcomp Lcomp . Periodically run RefineLabels to split persistently incompatible labels and merge predictively equivalent labels. Form zbt+1 = (ot+1 , b ct+1 ) and run RLUpdate(b zt , at , Yt , zbt+1 , done).
if done then Reset the environment, observe a new initial observation, and reinitialize the history. 17: end if 18: end for 15: 16: