NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games

NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games

arXiv:2609.01549v1 [cs.LG] 1 Sep 2026

Tomáš Holeček Viliam Lisý Artificial Intelligence Center, Department of Computer Science Faculty of Electrical Engineering, Czech Technical University in Prague holecto6@fel.cvut.cz, viliam.lisy@agents.fel.cvut.cz

Abstract Model-based reinforcement learning (MBRL) has achieved remarkable results in single-agent domains, yet its extension to competitive imperfect information games (IIGs) remains underexplored. In multi-agent settings, opponent-induced non-stationarity complicates the learning process, and decentralized model learning faces severe identifiability barriers, which we argue make centralized model learning a mathematical necessity. Building on this analysis, we propose NashDreamer, a principled MBRL framework for two-player zero-sum IIGs. It introduces a centralized Multi-Agent Recurrent State-Space Model (MARSSM) that decouples environment dynamics from the effect of players’ strategies on their individual observations. NashDreamer is designed to use arbitrary policy gradient algorithms and inherits their convergence guarantees towards Nash equilibria under an idealized model. Empirical evaluations across four benchmark games demonstrate that NashDreamer substantially improves sample efficiency over model-free baselines early in the training. Finally, we theoretically analyze the architecture’s optimization landscape, identifying the vulnerability of the Dreamer family of algorithms to posterior collapse in stochastic environments. We leave it as an open challenge.

1

Introduction

Model-based reinforcement learning (MBRL) has emerged as a powerful paradigm for sampleefficient learning. MuZero [34] and its extensions demonstrate that learned latent models can serve as effective planning abstractions, while the Dreamer series [12–15] shows that world models can generate imagination trajectories that reduce sample requirements. These approaches have matched or surpassed the state of the art across diverse single-agent domains, including robotics [41], autonomous driving [11, 18], and nuclear fusion control [7]. Complex imperfect information adversarial domains currently force a trade between fidelity and tractability. Automated cyber defense is studied on emulators made fast by simplifying the attacker, the defender or the network itself [19], and dogfight simulators, such as the one in Lu and Chen [25] hand each aircraft a complete frame rather than only what lies within its own sensor range. Sample-efficient generative MBRL offers the opposite trade, recovering speed from a small budget of interactions with the unsimplified environment. To lay the foundation, we investigate the Dreamerstyle paradigm for multi-player imperfect information games (IIGs), leveraging generative world models purely to synthesize large artificial datasets for scalable, sample-based policy optimization. Adapting MBRL to IIGs, however, exposes fundamental identifiability barriers. In a standard decentralized setting, a learning agent cannot distinguish between the objective stochastic dynamics of the environment and the shifting policy of its opponent [1, 28]. We argue that this “policy-physics entanglement” dictates that Centralized Training with Decentralized Execution (CTDE) is not merely an architectural convenience, but a mathematical necessity to learn a sound, stationary world model. Preprint.

We further provide the following contributions: (1) We propose NashDreamer, a principled CTDE framework for two-player zero-sum IIGs. It introduces a centralized Multi-Agent Recurrent StateSpace Model (MARSSM), which can be combined with game-theoretically sound policy-gradient methods, to facilitate convergence towards the Nash Equilibrium. (2) We show that DreamerV3 “KL-balancing” can cause posterior collapse, highlighting the challenge of mitigating latent space nonstationarity while maintaining a theoretically sound framework. (3) We provide thorough empirical evaluations showing that NashDreamer improves sample efficiency compared to model-free baselines across various domains. NashDreamer is not meant to replace well-tuned model-free policy-gradient methods, which reach excellent policies given abundant simulator access [33]. It is a sample-efficiency layer agnostic to the optimizer running inside imagination. The appropriate comparison is thus a policy-gradient method X against NashDreamer[X] at an equal budget of real environment interactions. This is the regime of our motivating applications, such as automated cyber defense or expensive simulations, where a real interaction, rather than a gradient step, is the dominant cost.

2

Background

We formalize our target domains as two-player zero-sum (2p0s) simultaneous-move games. We use a custom variant of Partially Observable Stochastic Games (POSGs) inspired by [20]. We denote N = {1, 2} the player set with c for the chance player; S the state space with initial state s0 ; A = A1 × A2 the joint action space, whose elements we write as a = (a1 , a2 ); O = O1 × O2 is the joint observation space with per-player observations. T : S × A → ∆S is the (partial) transition function ; O : S → O is the (deterministic) observation function1 ; R : S × A → R is the zero-sum reward function (R1 = −R2 ); P : S → 2N identifies the acting players at each state; and Li : S → 2Ai specifies legal actions for player i . States where P (s) = ∅ and the transition function is defined are chance nodes 2 ; states with undefined transitions are terminal and other nodes are called decision nodes, where we assume P (s) = {1, 2}. Turn-based games, such as Leduc Poker, are embedded by assigning a noop action to the inactive player (Appendix F). A history h = (s0 , a0 , . . . , st ) is the full sequence of states and joint actions from the initial state, the ground-truth object no player observes directly; s(h) is its last state, through which we lift the state-indexed functions, writing T (h, a) := T (s(h), a) and likewise for O and Li . Player i observes only their action-observation history (AOH) τi (h) = (o0i , a0i , . . . , oti ), and we assume perfect recall. The information set (infoset) of player i at h is Ii (h) = {h′ : τi (h′ ) = τi (h)}, with Ii the set of all of them. A policy πi : Ii → ∆(Ai ) conditions on τi alone and never on h, legal-action masks entering only as an implementation-level restriction of its support. We use π for optimized policies and reserve µ for the joint sampling policy that collects data. We assume collective observability: the joint action-observation history identifies the true underlying history, i.e., I1 (h) ∩ I2 (h) = {h}. 3 . It is a technical convenience rather than a restriction, since chance outcome eventually observed by some player can be deferred to the point where it is first observed [40]. It is also the assumption that makes public belief states well defined [5], and all our benchmarks satisfy it. A Nash equilibrium (NE) [27] is a joint policy π ∗ where no player can improve their utility by unilateral deviation. We measure distance to NE using NashConv [23]: NashConv(π) = u(BR(π2 ), π2 ) − u(π1 , BR(π1 )), where BR(πi ) is the best response of the player i’s opponent. 2.1

DreamerV3

DreamerV3 [14] learns a latent world model, the Recurrent State-Space Model (RSSM), for acting in POMDPs. The model state mt = [ĥt , zt ] concatenates a deterministic recurrent state ĥt produced by a sequence model ĥt = seq(mt−1 , at−1 ) and a stochastic categorical state zt sampled from either 1 A stochastic observation function is no restriction either: Kovařík et al. [20, fn. 3] note that the variant with O : S → ∆O is equivalent to the deterministic one, since stochastic observations can be emulated by adding a hidden variable to the world state. Recording the drawn observation in that variable additionally makes O a function of the state alone, which is the form we use. 2 Chance is the only source of stochasticity in our environments, so the transition function is deterministic everywhere except at chance nodes. The explicit chance player is a formal convenience, and the chance node can equivalently be absorbed by turning the previous transition stochastic. This is how the world model treats chance during learning. 3We note that the players identify h jointly, not individually, so this is not a perfect-information assumption

2

the posterior encoder qtθ = enc(ĥt , ot ) (during training with real observations) or the prior dynamics predictor pθt = dyn(ĥt ) (during imagination). The RSSM is trained end-to-end by optimizing: Lmodel =

T X

dyn dyn β enc Lenc Lt + β pred Lpred t , t +β

(1)

t

The prediction loss Lpred trains decoders to reconstruct rewards, continuation flags, and observations t from the model state. The encoder and dynamics losses Lenc = KL(qtθ ∥ sg(pθt )) and Ldyn = t t θ θ KL(sg(qt ) ∥ pt ) are both KL divergences between the posterior and prior (sg denotes stop-gradient). These asymmetric losses perform “KL balancing”: Ldyn t trains the dynamics to match the encoder, while Lenc t serves as an encoder regularizer to remain predictable by the dynamics. From trained model states, Dreamer generates imagination trajectories using the dynamics predictor, effectively amplifying the training data. For policy optimization, the actor uses REINFORCE with an entropy bonus, and the critic uses TD(λ). 2.2

Game-Theoretic Policy Optimization

Policy-gradient methods developed for single-agent problems are often not sound in imperfectinformation games: the online-learning dynamics they descend from are known to cycle rather than converge to an NE, in matrix games [26] and in sequential IIGs [31] alike. Regularizing toward a reference policy has proven the successful remedy, recovering last-iterate convergence. Regularized Nash Dynamics (RNaD) [32] regularizes in reward space, subtracting a KL penalty against a reference policy π reg from the payoff, which preserves the zero-sum structure. Periodically resetting π reg ← π then solves a sequence of regularized games whose solutions converge to an NE of the original one. Magnetic Mirror Descent (MMD) [36] instead regularizes in policy space, pairing a KL term toward a fixed magnet policy with a trust-region term on consecutive iterates. The difference in guarantees matters for the comparisons that follow: RNaD converges to an NE, whereas MMD under the fixed uniform magnet [33] that we adopt has convergence guarantees only to a quantal response equilibrium, so its asymptotic exploitability is bounded away from zero. Appendix A details both algorithms, including the off-policy simultaneous-move variant of RNaD [21] that we use.

3

World Models in Games: The Necessity of Centralized Training

Learning a world model without observing the opponent’s actions faces the following fundamental barriers preventing the use of contemporary decentralized approaches in adversarial environments (see Appendix D). Action identifiability. For a given state s and ego action a1 , different opponent actions a2 ̸= a′2 must produce distinct next state transition distributions: T (s, (a1 , a2 )) ̸= T (s, (a1 , a′2 )). When violated, action aliasing makes the data insufficient to disambiguate the causes of state changes [1, 24]. While this aliasing is benign for a purely forward-predictive model (as the resulting state is identical), it poses a fundamental identifiability barrier for any decentralized model attempting to reverse-engineer the objective environment dynamics independent of the opponent’s policy. Policy-physics entanglement. Without observing a2 , a learner cannot distinguish true environment dynamics T from a modified dynamic T ′ where the opponent’s strategy artificially mimics the effects of T [1]. A decentralized model is therefore a joint-system model rather than an environment model. It captures how the world works against that specific opponent, failing to capture the objective environment dynamics [28]. Adversarial exploitation. In strictly competitive environments, a strong opponent will actively exploit this identifiability gap, deliberately selecting actions that maximize the mismatch between true physical reality and the ego-agent’s corrupted internal representation. Centralized training bypasses these barriers entirely. By conditioning the world model on the joint action (a1 , a2 ) during training, the objective environment physics is perfectly decoupled from the opponent’s shifting strategy. We intentionally structure the world model so that it can be used with only the ego-agent partial observations at test time. 3

The barrier is directly observable: in perturbed Rock-Paper-Scissors, a decentralized model trained with the identical objective and optimizer learns the components of the game that do not depend on the opponent’s action, driving its error on observations, terminal flags and legal-action masks towards zero, while its reward error grows over training instead of falling, since without a2 the reward is not a function of its inputs. Appendix L reports the comparison. Centralization also amplifies the usual benefits of a world model. Disentangling the stochasticity of the opponent’s policy from the environment dynamics reduces the real samples needed to learn the model: in Rock-Paper-Scissors, a tabular world model would require only 9 transitions, after which policy training can proceed entirely in imagination. A world model further decouples data collection from policy optimization, so agents may collect with highly exploratory policies while training purely on-policy inside imagination, avoiding the off-policy corrections causing prohibitive variance in IIGs.

4

NashDreamer

As established in Section 3, applying decentralized world models to adversarial environments inevitably results in policy-physics entanglement. To resolve this, we introduce the centralized MultiAgent Recurrent State-Space Model (MARSSM) to learn the objective environment dynamics, while a separate recurrent network handles information aggregation for each player in a decentralized manner. Crucially, to preserve the causal chain of strategic deductions required for accurate information aggregation, we unroll complete trajectories starting at the initial state t = 0 for the world model training. We detail possible mechanisms allowing training on shorter, mid-trajectory chunks and their respective pitfalls in Appendix E.2 Definition 1 A centralized MARSSM is a tuple (seq, enc, dyn, iset), where: ĥt = t−1 θ t t seq(mt−1 , (at−1 , a )) is the sequence model, receiving the joint action; q = enc( ĥ , (o t t 1 , o2 )) 1 2 is the posterior encoder, inferring the stochastic state zt ∼ qtθ from the joint current observations; pθt = dyn(ĥt ) is the prior dynamics predictor, predicting the stochastic state zt ∼ pθt without access , at−1 , oti ) is the infoset model for player i, recurrently to the current observations; Iˆit = iset(Ît−1 i i building the latent infoset embedding from their own AOH τi (Section 2). The central model state mt = [ĥt , zt ] aggregates the private observations of both players and handles the environment dynamics. The infoset model embeds each player’s AOH τi into a fixed-size vector Iˆit , serving as our latent infoset representation. 4.1

World Model Objective

Our predictor loss extends DreamerV3 with legal action prediction and per-player observation reconstruction. Given the centralized model state mt = [ĥt , zt ], the predictors decode the reward R1t for Player 1, a termination flag dt , and, for each player i, the legal action mask ℓti and observation oti : X   t BCE Lmpred = Lrew (dt | mt ) + LBCE (ℓti | mt ) + LMSE (oti | mt ) , (2) t t (R1 | mt ) + Lt t t i∈{1,2} rew

where L follows the DreamerV3 categorical parameterization, BCE denotes binary cross-entropy (per element for the legal action mask), and MSE is mean squared error. The latent infoset embeddings Iˆit are trained by two further losses. The unifier loss requires the joint embeddings (Iˆ1t , Iˆ2t ) to reconstruct the centralized model state mt , a target well defined precisely because of collective observability (Section 2); the individual loss requires each embedding to reconstruct its own player’s observation oti and previous action at−1 , so that it captures local context: i i X h t−1 ˆt MSE ˆt ˆt Liset (sg(ĥt ) | Iˆ1t , Iˆ2t ) + LCE + LMSE (oti | Iˆit ) + LCE | Ii ) t = Lt t (sg(zt ) | I1 , I2 ) t t (ai | {z } {z } i∈{1,2} | Unifier Loss

Individual Loss

(3) where CE is categorical cross-entropy. 4

dyn The total model loss combines the standard DreamerV3 KL-balancing losses Lenc (Sect and Lt tion 2.1) with our prediction and infoset losses:

Lmodel =

T X

dyn dyn β enc Lenc Lt + β pred Lmpred + β iset Liset t t +β t ,

(4)

t

where we set β dyn = β pred = β iset = 1 and β enc = 0.1. Following the DreamerV3 convention, all task losses carry weight 1 while the encoder regularizer is down-weighted, so the posterior is shaped primarily by the prediction targets and only mildly pulled toward the prior. We deliberately do not tune these coefficients per environment, with the exception of Leduc, where we showcase failure modes of the KL balancing loss in stochastic environments (Appendix J). Figure 1 visualizes two steps of world-model training. 4.2

Imagination

Imagination supplies the synthetic data on which the actor-critic is trained. As in DreamerV3, we start one imagined trajectory for every non-chance node of trajectories collected from the real environment, terminal nodes included. Goofspiel-5, for instance, yields five imagined trajectories per real one. The imagined volume per gradient step therefore scales with the length of the game rather than with the batch size alone. We deviate from DreamerV3 in where a rollout begins: instead of branching from the encountered node, every rollout restarts from the root state t = 0 and runs for the maximum trajectory length of the underlying game, for the reasons detailed in Appendix E.2. Starting from a model state m0 obtained during world model training, each imagined step reconstructs the per-player legal actions, the reward and the termination flag from mt ; samples each actor’s action from its policy conditioned on that player’s latent infoset and legal actions 4 ; advances the sequence model on the joint action; samples the next stochastic state from the prior to form mt+1 ; and updates each player’s latent infoset from the observations reconstructed at mt+1 (see Appendix C.1). That last reconstruction is a key difference from standard Dreamer, where the decoder is merely a training signal. Here the infoset model consumes observations, so the decoder becomes an essential generative component of the imagination pipeline. We acknowledge the risks introduced by explicit reconstruction of high-dimensional observations and discuss their mitigation in Appendix E.1. Following the CTDE paradigm, the decentralized actor for player i is conditioned on Iˆit alone, ensuring it cannot access the opponent’s private observations. The centralized critic, used exclusively during training, operates on the joint representation (Iˆ1t , Iˆ2t ), grounding its value estimates in the full game history. All MARSSM parameters are frozen during the imagination phase.Figure 2 visualizes two imagined steps; Appendix C.1 states the step in full. 4.3

Actor-Critic

We choose not to use the DreamerV3 standard REINFORCE, as its fragility towards hyperparameter tuning is unsuitable for MBRL (Appendix N). Among the game-theoretically sound alternatives, such as MMD [36], we suggest RNaD because: (i) its last-iterate convergence guarantee is what our model-convergence result composes with (Corollary 2); (ii) it reportedly needs less domain-specific tuning, which matters here because the game it optimizes against is a learned, non-stationary object; and (iii) its off-policy simultaneous-move variant [21] is a natural fit for the replay/exploratory model learning. The choice is nevertheless modular as we demonstrate by also testing with MMD. We train the actor-critic networks concurrently on both real and imagined trajectories. While the MARSSM imagination pipeline theoretically permits purely on-policy optimization, we deliberately employ the off-policy formulation in our empirical setup to ensure exact parity to our (necessarily) off-policy model-free RNaD with replay buffer and to permit exploratory policies for the model learning. 4We explicitly mask out the policy output with the legal actions, rather than forcing the network to learn which actions are

illegal, which is inefficient

5

Real Data (t=1)

Real Data (t=2)

(o11 , o21 , d1 , r1 , a1d , a2d )

(o12 , o22 , d2 , r2 , a11 , a21 )

a1d , o11

Legend Player 1 Player 2 Det. latent Prior Loss

a11 , o12 Î11

x̂ = latent

Î21 a2d , o21

a21 , o22

Î12

Î22

a1d , a2d

o11 , o21

a11 , a21

Unifier Loss

Unifier Loss

Central WM

Central WM

ĥ1

ĥ2 Posterior

Posterior

ẑ1

Predictions

ô11 , ô21 , r̂1 , dˆ1 , ℓ̂11 , ℓ̂21

o12 , o22

ẑ2

Predictions

ô12 , ô22 , r̂2 , dˆ2 , ℓ̂12 , ℓ̂22

Figure 1: Two steps of NashDreamer world-model training on a real trajectory. The centralized RSSM is unrolled on the joint action and joint observation, the infoset model separately for each player. All stochastic states are posterior. The KL losses between posterior and prior are omitted; ad denotes the initial dummy action.

4.4

Model Convergence

To understand how NashDreamer captures environment dynamics, we analyzed the global minimum of the world model objective (all formal definitions and proofs are deferred to Appendix B). We define an infoset-isomorphic model (M∗ ) as one that establishes a one-to-one mapping between latent states and true histories, and clusters latent states into latent infosets if and only if corresponding true histories belong to the same true infoset. Our first result is that this structure-preserving model is strictly better, in expected loss, than models with two structural deviations: a “branch duplication”, which maps one history to several hallucinated latent branches with a distribution over them, and a model that violates the information partitioning of the game5 . The same result fixes the value of the optimum, Eµ [Lmodel (M∗ )] = (β enc + β dyn ) Eh,a∼µ [H(T (h, a))], so the optimal loss of the “perfect” model scales linearly with environment stochasticity (Theorem 2). Because M∗ captures the POSG structure, which unrolls into an EFG [20], and because branch duplication is indistinguishable from it from the actor-critic’s view, RNaD’s convergence result composes with it: an infoset-isomorphic model held fixed, together with exactly realized RNaD dynamics, yields convergence to a Nash equilibrium of the underlying game (Corollary 2). That guarantee is idealized in a way worth stating explicitly, since the practical algorithm approximates every one of its assumptions: neural function approximation, sampled trajectories, off-policy corrections, and a learned model updated concurrently with the policy. 5 Either by aliasing distinct true infosets or by subdividing a single true infoset into multiple latent ones.

6

Legend Player 1 Player 2

Start (WM training)

Î11

Î21

Î12

Î22

Det. latent Actors Prior x̂ = latent

â11 â21

Actors â11 , â21

â11 , â21 Central WM

Central WM

ĥ1

ĥ2 Posterior

Prior

ẑ1

ẑ2

ô12 ô22 Decoded ô12 , ô22

Figure 2: Two imagined steps. Imagination is grounded in a posterior model state taken from world-model training and proceeds with prior stochastic states. The observation decoder is required during imagination, since the infoset model consumes observations.

However, our second result exposes a profound vulnerability of two-way KL-balancing objective in a stochastic environment : Theorem 1 (Vulnerability to Posterior Collapse) Let Mcollapse be a model, where KL(enc(ĥt , ot ) ∥ dyn(ĥt )) = 0 for all reachable ĥt , and ot where p(ot |ĥt ) > 0. There exists an environment, sampling policy µ, and loss coefficients, such that Eµ [L(Mcollapse )] < Eµ [L(M∗ )]. Specifically, this can happen in environments with high stochasticity and small magnitude changes for the predictor, turning disambiguation of the chance outcomes into a suboptimal solution. We note that this theorem also holds for a single-agent environment and the standard DreamerV3 architecture. The dependence of the isomorphism loss bound on the (typically) non-stationary µ renders static coefficient tuning unsound. However, this degenerate minimum can be mitigated by using a “one-way” KL loss (applying a stop-gradient to the posterior), which drives convergence to either M∗ or Mb if our sampling policy is sufficiently exploratory (Appendix B). Unfortunately, removing this posterior regularization allows for representation drift (Section 2.1). Consequently, Dreamer-style MBRL faces a trade-off: the regularizer serving to mitigate representation non-stationarity and unpredictability can drive the optimization towards a degenerate solution. Finally, this artifact does not conflict with DreamerV3’s empirical success on stochastic domains such as Minecraft, Atari and BSuite. What makes collapse harmful is not stochasticity as such, but chance that is discrete, many-outcome, and strategically decisive. Minecraft’s procedural generation is highly stochastic, yet the next frames are largely predictable from those already seen, the initial-only randomness is closer to epistemic uncertainty than aleatoric, and pixel-level error is strategically benign where collapse does occur. The randomness injected into Atari and BSuite is genuinely 7

aleatoric but either few-outcome (sticky actions) or noise-like. Chance in games is the opposite on every count, which is why the same artifact becomes damaging, as we demonstrate in Leduc.

5

Empirical Evaluation

To avoid imagination in a highly inaccurate model, a world model warm-up precedes imagination: 1000 gradient steps for Imperfect-Information Goofspiel-5 (GS5) and Leduc Poker (LP), and 2000 for the larger environments: Stochastic Imperfect Information Goofspiel-13 (GS13), Battleship 5 × 5 (BS5) and Phantom Tic-Tac-Toe (PTTT). Appendix R ablates this choice. Goofspiel and Battleship are natively simultaneous-move; Leduc Poker and Phantom Tic-Tac-Toe are turn-based, embedded by the dummy-action construction of Section 2. The world model has no turn-based mode. We refer the reader to Appendix F for detailed environment rules, G.3 for extended hyperparameters, and M for comparison with DreamerV3 in a single-agent domain. All the experiments were carried out with 10 distinct training seeds. Evaluation protocol. All reported NashConv values are computed exactly against the true game by best-response dynamic programming over the full game tree, not estimated by sampling. For Phantom Tic-Tac-Toe, we utilize the exp-a-spiel library [33]. Appendix H details the protocol, including the handling of chance nodes and simultaneous moves. Baselines and positioning. We compare against model-free RNaD (with and without replay) and against MMD [36], the strongest-performing model-free policy-gradient method in the evaluation of Rudolph et al. [33]. MMD serves both as a baseline and, run inside imagination as NashDreamer[MMD], as the demonstration that the policy optimizer is interchangeable (Section 4.3). Its hyperparameters are the generic setting of Rudolph et al. [33] and are held fixed across all domains, detailed in Appendix G.5. We do not compare directly against published multi-agent Dreamer systems, because each is structurally inapplicable to 2p0s IIGs : cooperative communication-based methods lose their central mechanism in a zero-sum game, opponent-embedding models define no fixed joint policy. Appendix D argues each case in detail. However, we do perform the comparison against a decentralized model and show its unidentifiability in Appendix L. 5.1

Sample Efficiency: NashDreamer vs. Model-Free Baselines

We compare against the identical model-free versions of our policy optimization algorithms and compare in terms of real environment steps seen. To compensate for the additional steps seen in imagination, we also evaluate against RNaD augmented with a replay buffer. Our environments directly provide infoset representation, used as an input for our model-free baseline. To ensure exact input parity for NashDreamer, we use these in place of observations. We validate the capabilities of NashDreamer to learn directly from partial observations via an ablation in Imperfect Information Goofspiel 5 (Appendix O) Imperfect Information Goofspiel 5: While both methods achieved comparable asymptotic performance, NashDreamer reaches it utilizing substantially fewer environment interactions (Figure 3). The off-policy replay failed to improve RNaD’s sample efficiency. As theorized in Appendix E.2, we attribute this failure to the variance injected by unbiasing the off-policy data. Leduc Poker: While NashDreamer initially outpaced the model-free baseline, it exhibited late-stage performance deterioration (Figure 3). Investigating the learned dynamics reveals a structural error present throughout training: the model hallucinated physically impossible events at the deeper public chance node, such as dealing a public card already present in a player’s hand. We also tested the one-way KL loss of Theorem 3, which sets β enc = 0, removing the posterior regularization and signal towards collapse, it fails to prevent the structural error and deteriorates furthest of all variants we ran, because of representation drift (Appendix J), confirming out tradeoff claim from Section 4.4.Appendix K reports quantitative world-model diagnostics evidencing this failure. Phantom Tic-Tac-Toe. Phantom TTT serves as a benchmark of a large. yet tractable game. In this game, both optimizers improve when wrapped in NashDreamer, with NashDreamer[RNaD] attains model-free RNaD’s final exploitability using roughly a quarter of the gradient steps (Figure 3). Replay improves substantially on plain RNaD here, as it also does in Leduc, but it stays above NashDreamer[RNaD] throughout training. 8

Goofspiel-5

Leduc

100

Phantom TTT

100

NashConv

100

10 1 10 1

0.0

0.2

0.4

0.6

Environment steps

0.8

1.0 1e7

NashDreamerRNaD RNaD

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 1e7 Environment steps

RNaD with replay NashDreamerMMD

MMD Uniform Policy

10 1

0.0

0.5

1.0

1.5

2.0

2.5

Environment steps

3.0

3.5 1e7

WM warm-up end

Figure 3: NashConv against real environment interaction on the three domains where it is exactly computable. In Goofspiel-5 NashDreamer[RNaD] reaches a given NashConv with fewer environment steps than either RNaD variant. In Leduc it converges faster initially and then deteriorates, the world model retaining a structural error at the public chance node (Appendix I). In Phantom Tic-Tac-Toe both optimizers improve inside imagination, and NashDreamer[RNaD] continues to improve while the two MMD variants settle together. Bold curves are means over the 10 training seeds, faint curves the individual seeds; the dotted line marks the end of the world-model warm-up.

MMD as a model-free baseline. On small games, model-free MMD converges faster than RNaD. Wrapping MMD in NashDreamer changes little, as MMD already converges within the world-model warm-up phase. This phenomenon is observed in Goofspiel 5 and Leduc, however in Phantom Tic-Tac-Toe the model already brings visible acceleration. Appendix R tests this explanation directly by shortening the warm-up in Leduc, which recovers the acceleration even in Leduc. 5.2

Approximate NashConv on Larger Domains

Exact NashConv is intractable in stochastic Goofspiel-13 and Battleship, so we approximate it by budgeted best-response search. We freeze a trained checkpoint’s policy and, from each seat in turn, train 10 PPO seeds against it for 3,000 gradient steps, evaluating each over 100,000 games. The largest best-response value obtained by any PPO seed is taken as that seat’s estimate, and the two seats are summed. Appendix H gives the full protocol. We have trained each of the algorithms for 30k gradient steps, giving a world-model warm-up of 2k gradient steps for the NashDreamer variants and we evaluate the approximate NashConv each 10k steps. Since a fixed search budget need not recover the true best response it is a budgeted lower bound on NashConv: values are comparable across methods evaluated at the same budget, but not against the exact NashConv of Section 5.1 and not against results obtained at any other budget. Intervals are ±2σ over the 10 training seeds, not over PPO seeds. Goofspiel (13 cards) Method

10k

20k

Battleship (5×5) 30k

10k

20k

30k

MMD 0.827 ± 0.076 0.741 ± 0.093 0.732 ± 0.075 1.306 ± 0.070 1.161 ± 0.095 1.022 ± 0.150 ND[MMD] 0.853 ± 0.074 0.791 ± 0.085 0.763 ± 0.075 1.209 ± 0.126 0.969 ± 0.097 0.817 ± 0.161 RNaD 0.742 ± 0.388 0.448 ± 0.247 0.356 ± 0.291 1.416 ± 0.066 1.326 ± 0.132 1.155 ± 0.183 RNaD w/ replay 0.756 ± 0.074 0.531 ± 0.061 0.499 ± 0.074 1.438 ± 0.086 1.395 ± 0.122 1.223 ± 0.120 ND[RNaD] 0.469 ± 0.138 0.250 ± 0.145 0.168 ± 0.139 1.199 ± 0.131 1.184 ± 0.168 1.169 ± 0.256

Table 1: Budgeted approximate NashConv (lower is better) at three checkpoints, as mean ±2σ over the 10 training seeds. Values are a lower bound on NashConv at a fixed best-response budget and are not comparable with exact NashConv or with other budgets. ND denotes NashDreamer with the given actor-critic inside imagination; best value per domain in bold. In Goofspiel-13 NashDreamer[RNaD] reaches lower approximate NashConv after 20k gradient steps than model-free RNaD reaches only at 30k, and it leads at every checkpoint. In Battleship the same 9

holds for the other optimizer, NashDreamer[MMD] at 20k surpassing MMD at 30k. Which optimizer is the stronger one is domain-dependent, but our model nevertheless accelerates it further (Table 1).

6

Related Work

Model-based RL. MuZero [34] and Stochastic MuZero [2] learn latent models for planning in perfect-information games. The Dreamer series [12–15] trains world models for imagination-based policy learning in single-agent settings. NashDreamer specifically builds upon the online architecture of DreamerV3, as adapting the offline learning paradigm of DreamerV4 to adversarial multi-agent settings introduces overfitting to a particular opponent set, as well as prohibitive requirements of dataset size. LAMIR Kubicek and Lisý [21] extends MuZero-style look-ahead reasoning to IIGs, but focuses on search-time planning rather than pure RL and assumes deterministic games. Multi-agent Dreamer. MAMBA [9] and DreamerCF [6] extend Dreamer to cooperative settings with inter-agent communication. In competitive ones, Lu and Chen [25] condition the world model on ego and opponent embeddings, the latter randomly initialized at test time and so requiring zero-shot generalization, and Orsula [29] use single-player modeling with population-based self-play. All use policy-gradient actor-critics without game-theoretic convergence guarantees. Game-theoretic RL. CFR [42] and its variants are the gold standard for solving IIGs but require explicit game tree access. DREAM [39] removes this requirement via Monte Carlo sampling, enabling model-free deep CFR. RNaD/DeepNash [32] achieves human-level Stratego play using model-free RL with regularized reward dynamics. PSRO [22] maintains a population of best-response policies. MMD [36] achieves superhuman performance in Stratego [37]. All mentioned model-free approaches are potential alternatives to RNaD as the actor-critic component within NashDreamer’s learned world model, and we substantiate this for MMD in Section 5.1 and Appendix Q.

7

Conclusion

We introduced NashDreamer, a principled MBRL framework for two-player zero-sum imperfect information games. By pairing a centralized latent world model (MARSSM) with a decentralized, game-theoretic actor-critic (such as RNaD), NashDreamer overcomes the identifiability barriers inherent to multi-agent learning. Empirically, it significantly improves convergence in early-to-mid stages of training. However, as formalized theoretically and validated in Leduc Poker, environments with high chance stochasticity remain challenging due to the tradeoff between stability and collapse. Resolving this tension without inducing degenerate minima is the most critical direction for future work. Additional limitations to overcome include relaxing the t = 0 unrolling constraint and validating the architecture in massive environments, closer to the desired real-world deployment.

8

Broader impacts

We envision the primary application of our framework to be a building block for automated cyber defense. We acknowledge the dual-use nature of this research, as the same methodology could be leveraged to train sophisticated adversarial agents. We maintain, however, that the most effective countermeasure is making the algorithm widely accessible, allowing the community to train robust defenders with symmetric capabilities, rather than relying on the inherently fragile paradigm of security through obscurity.

Acknowledgments and Disclosure of Funding We thank the anonymous reviewers, whose thorough and constructive feedback prompted several additional experiments and analyses that substantially improved both the empirical evaluation and the presentation of this work. This research is supported by Czech Science Foundation (GA25-18353S). Computational resources were supplied by (e-INFRA CZ LM2018140) supported by the Ministry of Education, Youth and Sports of the Czech Republic. 10

References [1] Eyal Amir and Allen Chang. Learning partially observable deterministic action models. In Journal of Artificial Intelligence Research, volume 33, pages 349–402, 2008. [2] Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Planning in stochastic environments with a learned model. In International Conference on Learning Representations, 2022. [3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [4] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: Composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. [5] Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games. In Advances in Neural Information Processing Systems, volume 33, pages 17057–17069, 2020. [6] Jiajun Chai, Yuqian Fu, Dongbin Zhao, and Yuanheng Zhu. Aligning credit for multi-agent cooperation via model-based counterfactual imagination. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 281–289, 2024. [7] Ian Char, Joseph Abbate, László Bardóczi, Mark Boyer, Youngseog Chung, Rory Conlin, Keith Erickson, Viraj Mehta, Nathan Richner, Egemen Kolemen, et al. Offline model-based reinforcement learning for tokamak control. In Learning for Dynamics and Control Conference, pages 1357–1372. PMLR, 2023. [8] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724–1734, 2014. [9] Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 381–390, 2022. [10] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407–1416. PMLR, 2018. [11] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In Advances in Neural Information Processing Systems, 2024. [12] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020. [13] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. [14] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647–653, 2025. [15] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. In Advances in Neural Information Processing Systems, 2025. [16] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for jax. 2020. URL http://github.com/google/flax. 11

[17] Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar Duéñez-Guzmán, and Karl Tuyls. Neural replicator dynamics: Multiagent learning via hedging policy gradients. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, 2020. [18] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. [19] Jaromír Janisch, Tomáš Pevný, and Viliam Lisý. Nasimemu: Network attack simulator & emulator for training agents generalizing to novel scenarios, 2023. URL https://arxiv. org/abs/2305.17246. [20] Vojtěch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, and Viliam Lisý. Rethinking formal models of partially observable multiagent decision making. Artificial Intelligence, 303: 103645, 2022. [21] Ondrej Kubicek and Viliam Lisý. Look-ahead reasoning with a learned model in imperfect information games. In International Conference on Learning Representations, volume 2026, pages 53533–53547, 2026. [22] Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017. [23] Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453, 2019. [24] Lennart Ljung. System Identification: Theory for the User. Prentice Hall, 2nd edition, 1999. [25] Tianyu Lu and Bing Chen. Empowering aerial maneuver games through model-based constrained reinforcement learning. 2025. [26] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2703–2717. SIAM, 2018. [27] John F Nash Jr. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences, 36(1):48–49, 1950. [28] Frans A Oliehoek. Decentralized pomdps. In Reinforcement Learning: State-of-the-Art, pages 471–503. Springer, 2012. [29] Andrej Orsula. Learning to play air hockey with model-based deep reinforcement learning. arXiv preprint arXiv:2406.00518, 2024. [30] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv e-prints, pages arXiv– 1906, 2019. [31] Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning, pages 8525–8535. PMLR, 2021. [32] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990–996, 2022. 12

[33] Max Rudolph, Nathan Lichtle, Sobhan Mohammadpour, Alexandre M Bayen, J Zico Kolter, Amy Zhang, Gabriele Farina, Eugene Vinitsky, and Samuel Sokota. Reevaluating policy gradient methods for imperfect-information games. In The Fourteenth International Conference on Learning Representations, 2026. [34] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. [35] Shai Shalev-Shwartz and Yoram Singer. A primal-dual perspective of online learning algorithms. Machine Learning, 69(2–3):115–142, 2007. [36] Samuel Sokota, Ryan D’Orazio, J Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. In The Eleventh International Conference on Learning Representations. [37] Samuel Sokota, Eugene Vinitsky, Hengyuan Hu, J Zico Kolter, and Gabriele Farina. Superhuman ai for stratego using self-play reinforcement learning and test-time search. arXiv preprint arXiv:2511.07312, 2025. [38] Finnegan Southey, Michael P Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes’ bluff: Opponent modelling in poker. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pages 550–558. AUAI Press, 2005. [39] Eric Steinberger, Adam Lerer, and Noam Brown. DREAM: Deep regret minimization with advantage baselines and model-free learning. In International Conference on Learning Representations, 2020. [40] Frederick B Thompson. Equivalence of games in extensive form. In Harold W Kuhn, editor, Classics in Game Theory, pages 36–45. Princeton University Press, 1997. Originally RAND Research Memorandum RM-759, 1952. [41] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226–2240. PMLR, 2023. [42] Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in Neural Information Processing Systems, 2007. [43] Liu Ziyin, Zhikang T Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam. In International Conference on Machine Learning, 2020.

13

A

Regularized Policy Optimization in Imperfect-Information Games

This appendix expands Section 2.2 with the full statement of the two actor-critic algorithms that NashDreamer runs inside imagination. Their implementations and hyperparameters are given separately in Appendices G.4 and G.5. A.1

Why Policy Gradients Cycle

Follow the Regularized Leader (FoReL) [35] is a general framework from online learning. When the regularizer is entropy, FoReL yields Replicator Dynamics, which is closely connected to policygradient methods via Neural Replicator Dynamics (NeuRD) [17]. However, FoReL has been proven to cycle and fail to converge to a Nash Equilibrium in both matrix games [26] and sequential IIGs [31]. A.2

Regularized Nash Dynamics

Regularized Nash Dynamics (RNaD) [32] overcomes this by introducing policy-dependent reward regularization: π1 (a1 | I1 (h)) π2 (a2 | I2 (h)) Rreg (h, a, π, π reg ) = R(h, a) − η log reg + η log reg , (5) π1 (a1 | I1 (h)) π2 (a2 | I2 (h)) where η > 0 is the regularization coefficient. This formulation naturally preserves the zero-sum property of the game. By periodically updating the regularization policy π reg ← π, RNaD provably converges to an NE. The practical deep reinforcement learning instantiation, DeepNash [32], combines RNaD with a V-trace estimator [10] for the actor-critic. We adapt the simultaneous-move variant of RNaD [21], which introduces counterfactual importance weighting to enable off-policy learning. A.3

Magnetic Mirror Descent

Magnetic Mirror Descent (MMD) [36] is the second optimizer we employ. It is a mirror-descent method whose update carries two regularizers, one anchoring the policy to a magnet policy π mag and one to the previous iterate: n o πt+1 = arg min −⟨Qπt , π⟩ + α KL(π ∥ π mag ) + η1 KL(π ∥ πt ) . (6) π

Both methods thus regularize toward a reference policy, but through different mechanisms: RNaD transforms the reward (Eq. 5), which preserves the zero-sum property of the game, whereas MMD constrains the policy update itself. We use the instantiation of Rudolph et al. [33], in which the magnet is held fixed and uniform, so that α KL(π ∥ π mag ) reduces to an entropy bonus and the proximal term is realized by PPO-style clipped updates. One consequence matters for the comparisons that follow: with a fixed magnet and α > 0, MMD converges to a quantal response equilibrium rather than to an NE, so its asymptotic exploitability is bounded away from zero. Unlike RNaD, there are, to the best of our knowledge no convergence results for MMD when switching the magnet policy or annealing the magnet regularization.

B

Proofs and Theoretical Analysis

B.1

Definitions and Formal Setup

We first formalize the concepts necessary to state and prove our theoretical results. We make use of the fact that a POSG can be unrolled into an EFG structure [20]. Because games of this class form a tree, every non-root history has a unique predecessor. As in Section 2, we lift the state-indexed functions of the game (transition, observation, legal actions) to histories through the last state s(h) of the history, writing e.g. T (h, a) := T (s(h), a). Accordingly, H(T (h, a)) below denotes the entropy of the chance outcome at h. Definition 2 (Predecessor Function) Let h be a non-root true history in the game tree. We define the predecessor function Prev(h) = (hprev , aprev , rprev ), which returns the unique immediately preceding history hprev , the joint action aprev taken at hprev that led to h, and the immediate environment reward rprev generated by this transition. 14

Definition 3 (Latent Trajectory) A latent trajectory is a sequence τ̂ = (m0 , a0 , ô0 , Iˆ0 , m1 , a1 , ô1 , Iˆ1 , . . . , mt , ôt ) produced by unrolling the model, where the latent model state ml = (ĥl , zl ) encapsulates both the deterministic recurrent state ĥl and the stochastic categorical state zl , al = (a1,l , a2,l ) is the joint action, ôl is the joint observation reconstruction of ml , and Iˆl = (Iˆ1,l , Iˆ2,l ) are the latent infoset embeddings. Latent trajectories are the objects over which tree-structure isomorphism is stated (Definition 7). Definition 4 (Latent Reach Probability) Let τ̂ be a latent trajectory as in Definition 3. We define the reach probability of a specific latent model state mt under a joint sampling policy µ as: θ

ν̂µ (mt ) = p (zt | ĥt )

t−1 Y

θ

p (zl | ĥl )

2 Y

µi (ai,l | Iˆi,l )

i=1

l=0

Similarly, the reach probability of the deterministic state ĥt , which strictly isolates the probability prior to the final chance node resolution, is: ν̂µ (ĥt ) =

t−1 Y

pθ (zl | ĥl )

2 Y

µi (ai,l | Iˆi,l )

i=1

l=0

Definition 5 (Valid State and History Sets) Let M be the set of all possible latent model states generated by the architecture. We define M∗ ⊆ M as the subset of latent states that are reachable (ν̂µ (m) > 0) under a sufficiently exploratory sampling policy µ with full support over all legal actions. Furthermore, let H be the set of all valid histories in the true game tree. We define Hnc ⊆ H as the subset of true histories where the last node is not a chance node. We restrict our mapping to Hnc because our architecture inherently models environmental stochasticity as latent transitions rather than explicit nodes. ˆ denote the predicted reward and terminaDefinition 6 (ϵ-Distance Matching) Let r̂(m) and d(m) tion flag from the latent state m. Let ˆli (m) and ôi (m) be the predicted legal actions and reconstructed observation for player i. A reachable latent model state m ∈ M∗ is considered within ϵ-distance of a true history h ∈ Hnc (where Prev(h) = (hprev , aprev , rprev )) if its predictions match the deterministic true environment emissions within an arbitrarily small error bound ϵ, where ∥ · ∥ denotes the max-norm over vector components: 1. ∥r̂(m) − rprev ∥ < ϵ ˆ 2. ∥d(m) − d(h)∥ < ϵ 3. For i ∈ {1, 2}: ∥ˆli (m) − Li (h)∥ < ϵ 4. For i ∈ {1, 2}: ∥ôi (m) − Oi (h)∥ < ϵ where d(h) = 1 if the history is terminal and 0 otherwise. To make condition 4 concrete, consider Leduc Poker, where each Oi (h) is a binary vector whose blocks encode the player’s private card, the public card (if already dealt), and the current bets. Take a history h in which player 1 holds the jack and no public card has been dealt, so the private-card block of O1 (h) is one-hot on the jack and the public-card block is all zeros. A latent state m whose decoder outputs ô1 (m) with 0.99 on the jack, at most 0.01 on every other private card and at most 0.01 on every public-card entry is within ϵ-distance for ϵ = 0.02: the reconstruction is unambiguous both about which private card was dealt and about no public card being present. A latent state that instead splits mass between two private cards, say 0.5 and 0.5, violates condition 4 for every ϵ < 0.5. Such a state has aliased two distinct true histories into one latent state, which is exactly the structural failure excluded from M∗ , and which we observe empirically at the round-2 public chance node in Leduc (Appendix J). 15

Definition 7 (Tree-Structure Isomorphism) A generative world model is tree-structure isomorphic if there exists a bijective mapping fϵ : Hnc → M∗ , parameterized by the tolerance ϵ of Definition 6 and evaluated at a history h, that satisfies the following conditions for all h ∈ Hnc : 1. fϵ (h) is within ϵ-distance of h. 2. fϵ (h) is the ending state of a latent trajectory τ̂ (Definition 3) that contains the exact same joint action sequence as the true history h, and ôl is within ϵ-distance of ol for each timestep l. Definition 8 (Infoset Isomorphism) Let Iˆi (m) denote the latent infoset embedding of player i for latent state m. Assume a world model is tree-structure isomorphic with bijective mapping fϵ : Hnc → M∗ . The model is infoset-isomorphic if, for any player i ∈ {1, 2} and any pair of valid true histories h, h′ ∈ Hnc , their corresponding mapped model states m = fϵ (h) and m′ = fϵ (h′ ) satisfy the following equivalence relation: ∥Iˆi (m) − Iˆi (m′ )∥ < ϵ ⇐⇒ Ii (h) = Ii (h′ ) where Ii (h) denotes the true infoset of player i at history h. A world model exhibits branch duplication if the encoder introduces artificial stochasticity, mapping a single true history to multiple distinct latent model states. Semi-formally, instead of a bijection fϵ , the model induces a mapping fϵ′ : Hnc → P(M∗ ) where a true history h ∈ Hnc maps to a set of reachable model states Mh ⊆ M∗ . We assume these sets are disjoint for distinct histories and cover the entire reachable model space (Mh ∩ Mh′ = ∅ for h ̸= h′ ), all states within Mh are within ϵ-distance for all predictor targets. Crucially, all states within Mh yield the exact same joint latent infoset representation (Iˆ1 , Iˆ2 ) (this condition will be satisfied directly, since all the model states were created by the same AOH and all satisfy the ϵ-distance accuracy). Definition 9 (Information Partition Violation) A world model violates the game’s information partitioning if it is tree-structure isomorphic but not infoset-isomorphic. This means the model correctly captures the objective physical dynamics, but its mapped latent infosets either alias distinct true infosets together, or improperly separate histories belonging to the same true infoset. B.2

Proofs

We will now analyze the loss for isomorphic model, leading up to a proof for Theorem 2. We assume that the KL losses are unclipped. Lemma 1 Let ĥ be a deterministic recurrent state induced by the true history and joint action (hprev , aprev ). For any model that perfectly reconstructs deterministic environment emissions (including both infoset-isomorphic models and models exhibiting branch duplication), the expected dynamics loss over the chance outcome is exactly equal to the scaled inherent Shannon entropy of the true environment transition: H(T (hprev , aprev )). Proof 1 The dynamics loss is defined as the expected KL divergence between the posterior encoder q θ (z | o, ĥ) and the prior dynamics predictor pθ (z | ĥ) across all possible observations o sampled from the true environment distribution p(o | ĥ). Assuming an optimal prior that perfectly matches the aggregated marginal posterior, we have pθ (z | ĥ) = Eo∼p [q θ (z | o, ĥ)]. We define the induced true marginal distribution of the latent state as p(z | ĥ) = pθ (z | ĥ), and the joint distribution over observations and latent chance states as p(o, z | ĥ) = p(o | ĥ)q θ (z | o, ĥ). 16

Using this joint distribution, we expand the expected KL divergence: h  i X X q θ (z | o, ĥ) Eo∼p KL q θ (z | o, ĥ) ∥ pθ (z | ĥ) = p(o | ĥ) q θ (z | o, ĥ) log pθ (z | ĥ) o z X p(o, z | ĥ)/p(o | ĥ) = p(o, z | ĥ) log p(z | ĥ) o,z =

X o,z

p(o, z | ĥ) log

p(o, z | ĥ) p(o | ĥ)p(z | ĥ)

Because the final expression is the expected log-ratio of the joint distribution to the product of its marginals, it is exactly the formal definition of Conditional Mutual Information (CMI), I(z, o | ĥ). Applying standard information-theoretic identities, we rewrite this CMI as the difference between conditional entropies: I(z, o | ĥ) = H(o | ĥ) − H(o | z, ĥ) Because both an isomorphic model and a branch-duplicated model map true histories to states within an ϵ-distance bound for predictor targets, the combination of the deterministic history representation ĥ and the chosen latent chance state z must perfectly reconstruct the true environment observation o. Since o is fully and deterministically defined by the tuple (ĥ, z), there is zero remaining uncertainty about the observation. Therefore, the conditional entropy H(o | z, ĥ) = 0. Consequently, the expected dynamics loss simplifies exactly to H(o | ĥ). Because the recurrent state ĥ uniquely identifies the predecessor history and joint action (hprev , aprev ), the entropy of the subsequent observation o is exactly the inherent aleatoric entropy of the true environment chance node: h  i Eo∼p KL q θ (z | o, ĥ) ∥ pθ (z | ĥ) = H(T (hprev , aprev )) Because this equality holds as long as H(o | z, ĥ) = 0 (which is strictly required to maintain the ϵ-distance bounds for valid reconstruction), it is invariant to how the encoder maps the observation into the latent space. Thus, branch duplication cannot decrease the expected KL penalty below the true chance entropy. With the dynamics loss lower-bounded by the inherent environment stochasticity, we now establish the convergence properties of the remaining objective terms under the assumption of unbounded network capacity. Corollary 1 (Deterministic Posterior) By the definition of tree-structure isomorphism, the bijective mapping fϵ (h) requires the model state to be induced by the exact same AOH as the true history. Because the recurrent state ĥ contains all historical context up to the current observation, the stochastic state z must necessarily encode the chance-induced observation. To satisfy H(o | z, ĥ) = 0 (as established in Lemma 1), the posterior encoder q θ (z | o, ĥ) must collapse into a deterministic (Dirac delta/one-hot) distribution, mapping each distinct chance outcome of T (hprev , aprev ) to a unique z. Lemma 2 (Predictor and Unifier Loss Minimization) Let M∗ be an infoset-isomorphic model. As ϵ → 0 in the ϵ-distance bound, the expected loss for all deterministic predictors (reward, continuation, legal actions, observation reconstruction) and all latent infoset targets decreases to zero. Proof 2 By the definition of ϵ-distance matching, all predictions generated by a mapped state m ∈ M∗ match the true environment emissions within an arbitrary bound ϵ. As ϵ → 0, the Mean Squared Error (MSE) and Cross-Entropy losses for these predictive targets trivially evaluate to zero. Furthermore, the decentralized infoset networks process the true action-observation history (AOH) to generate Iˆ1 and Iˆ2 . Given unbounded capacity, the auto-encoding targets (decoding the current 17

observation and previous action from the infoset embedding) are perfectly reconstructed, achieving zero loss. Crucially, under our assumptions of perfect recall and immediately observable chance outcomes, the intersection of the true infosets of all players uniquely identifies the true underlying history (collective observability, Section 2): I1 (h) ∩ I2 (h) = {h}. Because the model is infoset-isomorphic, its latent infosets perfectly preserve this partitioning (∥Iˆi (m) − Iˆi (m′ )∥ < ϵ ⇐⇒ Ii (h) = Ii (h′ )). Therefore, the joint latent infoset embedding (Iˆ1 , Iˆ2 ) uniquely identifies the mapped latent state m. Because this mapping is uniquely resolvable, the centralized unifier network can perfectly reconstruct m from (Iˆ1 , Iˆ2 ), allowing the unifier loss to minimize to zero. Using these, we can prove the first result of Section 4.4. Theorem 2 Let µ be an arbitrary joint sampling policy. Let M∗ be an infoset-isomorphic world model, Mb be a world model exhibiting branch duplication, and Ml be a model violating the game’s information partitioning. It holds that Eµ [Lmodel (M∗ )] < Eµ [Lmodel (Mb )] and Eµ [Lmodel (M∗ )] < Eµ [Lmodel (Ml )]. Furthermore, under any µ, M∗ achieves an expected loss: Eµ [Lmodel (M∗ )] = (β enc + β dyn )Eh,a∼µ [H(T (h, a))], where H denotes Shannon entropy. Proof 3 (Proof of Theorem 2) We first establish the exact expected loss for the infoset-isomorphic model, M∗ , under the sampling policy µ. By Lemma 2, because M∗ is both tree-structure isomorphic and infoset-isomorphic, its ϵ-distance bounds approach zero, allowing all deterministic predictor losses (reward, continuation, legal actions, observation reconstruction) and all unifier/infoset losses to minimize to zero. The only remaining non-zero terms in the objective are the KL divergence losses. As established in Lemma 1, the expected KL divergence at a given recurrent state ĥ induced by the true predecessor (h, a) evaluates exactly to the chance entropy H(T (h, a)). Because the objective utilizes two separate KL penalties—one for updating the dynamics prior (scaled by β dyn ) and one for updating the posterior encoder (scaled by β enc )—the total expected loss at this specific step is: LKL (ĥ) = (β enc + β dyn )H(T (h, a)) To compute the expected loss of the entire model under the joint sampling policy µ, we aggregate this per-step loss over all reachable latent recurrent states. We denote the set of all true chance nodes as Hc = H \ Hnc and the set of all reachable recurrent states as Ĥ∗ . Because the bijective mapping fϵ : Hnc → M∗ guarantees a one-to-one correspondence between true non-chance histories and latent states, if we reparametrize deterministic transitions to be represented via a trivial chance node with a single outcome, each latent recurrent state ĥt corresponds perfectly to a true chance node transition from some (h, a), i.e. fϵ induces also a bijective mapping g : Ĥ∗ → Hc . Furthermore, because the optimal prior exactly captures the true marginal chance distribution, the latent reach probability of any recurrent state, ν̂µ (ĥ), exactly equals the true environment reach probability of its corresponding chance node, νµ (h, a). Consequently, we can translate the latent summation directly into an expectation over the true game space: X Eµ [L(M∗ )] = ν̂µ (ĥ)LKL (ĥ) ĥ∈Ĥ∗

=

X X h∈Hc

= (β

enc

νµ (h, a) (β enc + β dyn )H(T (h, a))

a

+ β dyn )Eh,a∼µ [H(T (h, a))]

Next, we establish the strict suboptimality of the model exhibiting branch duplication, Mb . By Lemma 1, Mb achieves the same KL loss . However, by definition, Mb maps a single true history h ∈ Hnc to a set of distinct reachable latent states Mh ⊆ M∗ . Crucially, all duplicated states m ∈ Mh yield the exact same joint latent infoset (Iˆ1 , Iˆ2 ). 18

The centralized unifier network is a deterministic function trained to predict the central model state m = (ĥ, z) from the joint latent infosets. In Mb , the input (Iˆ1 , Iˆ2 ) remains constant across the set Mh , but the target state m varies as the encoder distributes the observation across different stochastic outcomes z. Because a deterministic neural network mapping a constant input to multiple varying targets incurs an irreducible variance penalty, it must yield a strictly positive unifier loss. Therefore, Eµ [L(Mb )] > Eµ [L(M∗ )]. Finally, we establish the strict suboptimality of the model violating the game’s information partitioning, Ml . By definition, Ml is tree-structure isomorphic but fails the infoset isomorphism condition. This means there exists at least one player i and two valid true histories h, h′ ∈ Hnc mapped to latent states m, m′ such that the equivalence relation ∥Iˆi (m) − Iˆi (m′ )∥ < ϵ ⇐⇒ Ii (h) = Ii (h′ ) is broken. This violation must manifest in one of two cases: Case 1; Ii (h) = Ii (h′ ) but ∥Iˆi (m) − Iˆi (m′ )∥ ≥ ϵ. In games with perfect recall, histories in the same true infoset share the exact same Action-Observation History (AOH) for player i. Because the decentralized infoset network processes this AOH to generate Iˆi , it acts as a deterministic function. It is impossible for a deterministic function to output distinct embeddings for identical inputs. Thus, this case cannot practically occur. Case 2: Ii (h) ̸= Ii (h′ ) but ∥Iˆi (m) − Iˆi (m′ )∥ < ϵ. Because the true infosets are distinct, perfect recall dictates that the underlying AOH for player i must differ in at least one previous action or observation. However, the model aliases them to the same latent infoset embedding. The architecture utilizes decentralized auto-encoding targets, forcing the network to decode the current observation and previous action from Iˆi . Because a deterministic decoder cannot map a single, aliased embedding to two distinct true AOH targets, it necessarily incurs a strictly positive reconstruction error. We note that forcing this individual infoset reconstruction loss is a theoretical necessity, not merely an empirical stabilization trick. In an asymmetric game where Player 1 possesses perfect information, Player 1’s infoset alone could uniquely identify the history h. Without individual decentralized targets, the centralized unifier loss could perfectly minimize to zero relying exclusively on Player 1, entirely failing to constrain Player 2’s latent partition. Because Case 1 is structurally impossible, the violation in Ml must manifest as Case 2, incurring a strictly positive individual infoset loss. Therefore, Eµ [L(Ml )] > Eµ [L(M∗ )]. Remark 1 (Optimization and Benign Branch Duplication) While Theorem 2 establishes that the branch-duplicated model Mb has a strictly higher global minimum than the perfectly isomorphic model M∗ , standard one-way optimization (where the centralized unifier target is treated as a constant with stop-gradients) does not provide a direct gradient signal to the encoder to actively merge these duplicated branches. Consequently, in practice, the model may converge to Mb rather than M∗ . However, because all duplicated states within Mh accurately reconstruct the true environment emissions and preserve the true marginal chance probabilities, this branch duplication is a benign structural artifact that does not degrade downstream policy optimization, unlike information partition violations or posterior collapse. Because M∗ accurately captures the underlying POSG structure, which can be “unrolled” into an EFG [20], and because Mb is indistinguishable from it from the actor-critic’s view, the RNaD convergence result applies directly. Corollary 2 (Nash Equilibrium Convergence) Assume that (i) the world model has converged to the infoset-isomorphic model M∗ (or to Mb ) and is held fixed during policy optimization, and (ii) the RNaD dynamics are realized exactly, i.e., with exact policy evaluation and without functionapproximation or sampling error. Then NashDreamer converges to a Nash Equilibrium of the underlying game. Proof 4 (Proof of Theorem 1) To prove the existence of this degenerate global minimum, we construct a specific counterexample environment and evaluate the expected losses of the isomorphic model M∗ and the collapsed model Mcollapse . Assume the standard Dreamer loss scaling coefficients: β enc = 0.1, and all other coefficients (dynamics, reward, observation, and infoset) are 1.0. 19

Counterexample Environment Construction: Consider an environment that begins with a single chance node with 101 uniformly distributed outcomes. The remainder of the environment is deterministic, and a deterministic joint policy µ guarantees reaching the terminal state. The initial chance node outcome has two distinct effects: 1. It dictates a continuous component of a subsequent observation for Player 1, ot1 , selecting a value such that its transformed representation in symlog space falls uniformly in the set {0, 0.01, 0.02, . . . , 1}. 2. It penalizes the final terminal reward, subtracting a value such that the final reward in symlog space falls uniformly in the same set {0, 0.01, 0.02, . . . , 1}. Expected Loss of the Isomorphic Model (M∗ ): By Theorem 2, the isomorphic model incurs zero prediction and unifier losses, leaving only the KL divergence penalty. The inherent entropy of the 101-outcome uniform chance node is log(101) ≈ 4.605. Applying the loss coefficients, the expected loss is exactly: Eµ [L(M∗ )] = (β enc + β dyn )H(T ) = (0.1 + 1.0) log(100) ≈ 5.06 Expected Loss of the Collapsed Model (Mcollapse ): Now consider a degenerate model where the posterior completely ignores the observation and collapses to the prior. Because q θ (z | ot , ĥt ) = pθ (z | ĥt ), the KL divergence loss is exactly 0. The encoder maps all 101 chance outcomes to a single model state. Consequently, the model must rely on this static representation to predict the varying observations and rewards. For the observation reconstruction (optimized via Mean Squared Error), the target symlog values span {0, . . . , 1}. A sufficiently trained predictor will output the mean value of 0.5, the maximum possible squared error for any outcome is (1 − 0.5)2 = 0.25. Therefore, the expected observation loss is strictly bounded: Lobs ≤ 0.25. For the reward prediction, we utilize the standard two-hot cross-entropy loss over discretized bins. The true targets in symlog space fall uniformly in {0, . . . , 1}. In the two-hot encoding scheme, a target y ∈ [0, 1] is represented as a linear mixture: (1 − y) probability mass on the bin for 0, and y probability mass on the bin for 1. A sufficiently trained predictor thus outputs a uniform distribution over the two adjacent bins (predicting 0.5 for bin 0 and 0.5 for bin 1), the cross-entropy loss for any target y is: − ((1 − y) log(0.5) + y log(0.5)) = − log(0.5) = log(2) ≈ 0.69 Thus, the expected reward loss is bounded: Lreward ≤ 0.7. The individual infoset losses can all minimize to zero. For Player 2, all true histories fall into the same infoset, requiring only a constant embedding. For Player 1, the decentralized network can simply produce a distinct latent infoset embedding for each unique observation ot1 . Finally, because the encoder collapses all outcomes into a single latent state, the centralized unifier network only needs to map these varying joint latent infosets to a single, constant target state. Learning this trivial surjective mapping allows the unifier loss to reach 0. We explicitly note that this degenerate mapping does not contradict our previous result (Theorem 2) regarding the strict suboptimality of information partition violations. That penalty fundamentally relied on the model maintaining tree-structure isomorphism (perfect deterministic reconstruction), whereas this collapsed model deliberately abandons structural isomorphism to save on KL divergence. Summing the bounded predictor penalties, the total expected loss of the collapsed model is: Eµ [Lmodel (Mcollapse )] ≤ 0.25 + 0.69 = 0.94 Because Eµ [Lmodel (Mcollapse )] ≤ 0.94 and Eµ [Lmodel (M∗ )] ≈ 5.06, the degenerate collapsed model achieves a strictly lower objective loss than the perfectly structure-preserving isomorphic model. This proves that under standard static KL-balancing, the global minimum is vulnerable to posterior collapse in environments with high chance entropy and low-magnitude predictor variance. We note that, in our counterexample environment, we let only one player’s observations and reward be determined by the chance outcome. Since our world model is also centralized, an analogous counterexample single-agent environment could be constructed, proving this property also for DreamerV3. 20

Finally, we discuss that with the encoder/representation loss disabled, our model will, in theory, converge to an infoset-isomorphic model or a model with duplicated branches, both of which enable sound actor-critic learning. Theorem 3 (Convergence of One-Way Optimization) Let Lone−way denote the optimization objective where the posterior encoder receives no gradient from the KL divergence penalty (i.e. setting β enc = 0 ). Under the assumptions of unbounded network capacity and a sufficiently exploratory sampling policy µ, the global minimum of Lone−way is achieved strictly by the set of models that are either perfectly infoset-isomorphic (M∗ ) or exhibit benign branch duplication (Mb ). Proof 5 Under Lone-way , the parameters of the posterior encoder q θ (z | ot , ĥt ) are updated exclusively by the deterministic predictor losses (reward, continuation, observation, legal actions). To achieve a loss of 0 on the deterministic environment predictors, the encoder must map true histories to latent states that maintain ϵ-distance matching. This enforces either tree-structure isomorphic model, or duplicated branches. Even though Theorem 2 shows that the isomorphic model achieves a lower total loss, as stated in Remark 1 the model lacks a gradient signal to force this convergence. However, as branch duplication is a benign artifact and all the branches are structurally identical and have the same joint infoset embedding, we can merge them for the purpose of this proof and consider only the infoset isomorphic model M∗ . With the encoder forced to maintain tree-structure isomorphism to satisfy the predictors, we consider the remaining structural losses. By Theorem 2, any model that maintains tree-structure isomorphism but violates the game’s true information partitioning (Ml ) will incur a strictly positive, reducible penalty in the infoset individual loss Concurrently, the dynamics prior pθ (z | ĥt ) is updated to minimize the KL divergence against this fixed, structurally sound posterior target. As established in Lemma 1, the optimal prior will perfectly match the aggregated marginal posterior, achieving a KL divergence exactly equal to the true chance entropy. M∗ and Mb perfectly minimize all reconstruction losses to 0, reduce the dynamics loss to the real environment entropy (the lowest obtainable non-degenerate value) and achieve a global/local minimum of the infoset loss, respectively. So, M∗ forms the global learnable minimum under Lone−way (but not necessarily the global minimum in general, as demonstrated in Theorem 1) and Mb forms a local minimum. Thus, given sufficient data and compute, the architecture is mathematically guaranteed to converge to an environmentally and game-theoretically sound representation.

C

Training Algorithm

Algorithm 1 states the complete NashDreamer training loop, collecting in one place the world-model objective of Section 4.1, the imagination procedure of Section 4.2 and the actor-critic of Section 4.3. Three properties of the loop are worth making explicit, as they are easy to miss in the prose. Real environment data is collected at every iteration. The world model is updated at every iteration and is not frozen during the warm-up period. The warm-up period gates the imagination phase alone: during it the actor-critic trains on real trajectories only, and the world model trains exactly as it does afterwards. All hyperparameter values are listed in Table 2. The centralized critic v is evaluated on the joint representation (Iˆ1t , Iˆ2t ) and is used only during training; each actor πi conditions on Iˆit alone, so the policies remain executable from one player’s action-observation history at test time. Note that phase (iii) generates one full rollout from the root per non-chance node of the real minibatch, terminal nodes included, so the number of imagined trajectories per gradient step grows with the length of the game rather than being a fixed multiple of |B|: Goofspiel-5 yields five imagined trajectories per real trajectory, Goofspiel-13 thirteen, and Battleship 5 × 5 twenty-eight. The resulting counts for the head-to-head domains are given in Appendix Q. On the warm-up period. The warm-up of phase (iii) is a stabilizing mechanism rather than a theoretical requirement, and nothing about it is specific to imperfect information. Policy-gradient actor-critics, including the REINFORCE actor of DreamerV3, are not designed for a non-stationary optimization target, so a warm-up would be equally defensible in a single-agent POMDP. What the 21

Algorithm 1 NashDreamer[RNaD]. N warm is the world-model warm-up length, N reg the regularization-policy update period, |B| the batch size and T max the maximum trajectory length of the game (Table 2). Phase (ii) corresponds to Figure 1 and phase (iii) to Figure 2. Require: game G; world model parameters θ comprising the MARSSM (seq, enc, dyn, iset) and the predictors; decentralized actors π1 , π2 ; centralized critic v reg 1: πi ← πi for i ∈ {1, 2} 2: for n = 1, . . . , N steps do 3: (i) Real environment interaction — at every iteration 4: collect a minibatch B of |B| complete trajectories from G under the joint sampling policy µ, each unrolled from s0 to termination 5: (ii) World model update — at every iteration; θ is never frozen 6: for all trajectories in B, for t = 0, . . . , T do t−1 7: ĥt ← seq(mt−1 , (at−1 1 , a2 )) {joint action} θ t t 8: qt ← enc(ĥt , (o1 , o2 )), pθt ← dyn(ĥt ), zt ∼ qtθ {posterior} 9: mt ← [ĥt , zt ] 10: Iˆit ← iset(Iˆit−1 , at−1 , oti ) for i ∈ {1, 2} {own AOH only} i 11: end for 12: take a gradient step on θ with Lmodel of Eq. 4 13: (iii) Imagination — gated by the warm-up; θ frozen throughout 14: B̂ ← ∅ 15: if n > N warm then 16: for all non-chance nodes of each trajectory in B, terminals included do 17: m0 , Iˆ10 , Iˆ20 ← posterior states at t = 0 computed in (ii) {all rollouts start at the root} 18: for t = 0, . . . , T max − 1 do ˆlt , ˆlt , r̂t , b̂t ← predictors(mt ) 19: 1 2 20: ati ∼ πi (· | Iˆit , ˆlit ) for i ∈ {1, 2} {decentralized} 21: ĥt+1 ← seq(mt , (at1 , at2 )) 22: pθt+1 ← dyn(ĥt+1 ), zt+1 ∼ pθt+1 {prior} 23: mt+1 ← [ĥt+1 , zt+1 ] 24: ôt+1 ← observation_decoder(mt+1 ) for i ∈ {1, 2} {decoder is generative here} i 25: Iˆit+1 ← iset(Iˆit , ati , ôt+1 ) for i ∈ {1, 2} i 26: end for 27: append the rollout to B̂ 28: end for 29: end if 30: (iv) Actor-critic — one RNaD step on real and imagined data jointly 31: update π1 , π2 , v by off-policy simultaneous-move RNaD on B ∪ B̂, with rewards regularized toward π reg as in Eq. 5 32: if n mod N reg = 0 then 33: πireg ← πi for i ∈ {1, 2} 34: end if 35: end for

imperfect-information setting adds is that the non-stationarity arrives from several directions at once, and each family of methods available here pays for it somewhere: through careful hyperparameter tuning (MMD, PPO-style methods), through a regularization mechanism that is itself a source of non-stationarity (the periodic π reg update of RNaD, Section 2.2), or by requiring explicit access to the true game/estimating the true distributions via importance sampling (CFR and its variants). Imagination from a world model that is still changing rapidly adds a further source on top of these, and the warm-up removes it from the phase of training where it is most harmful. Appendix R sweeps its length in Leduc Poker and finds the method not to be sensitive to it. The warm-up should not be read as realizing assumption (i) of Corollary 2, which requires a world model that has converged and is held fixed during policy optimization. As Algorithm 1 makes explicit, the model is updated for the entire run; the warm-up narrows the gap to that assumption without closing it, and it is one of the approximations discussed in Section 4.4. 22

C.1

The Imagination Step

Section 4.2 summarizes the imagined step in prose. Stated in full, and starting from a model state m0 obtained during world model training, each step proceeds as follows. 1. Reconstruct current per-player legal actions ˆlit , the reward rˆt and the termination flag bˆt from mt 2. Each actor samples an action ati from its policy conditioned on the player’s latent infoset and legal actions: ati ∼ πi (· | Iˆit , ˆlit ). The legal actions are used to explicitly mask out the policy output, rather then letting the actor learn the illegal actions. 3. The sequence model advances the recurrent state: ĥt+1 = seq(mt , (at1 , at2 )). 4. The dynamics predictor produces the prior distribution pθt+1 = dyn(ĥt+1 ), from which a stochastic state zt+1 is sampled, yielding the next model state mt+1 = [ĥt+1 , zt+1 ]. 5. We obtain per-player observation reconstructions ôt+1 from mt+1 using the predictor. i 6. The infoset model updates each player’s latent infoset: Iˆt+1 = iset(Iˆt , at , ôt+1 ). i

i

i

i

Step 4 is the key difference from standard Dreamer, where the decoder is merely a training signal. Here, the infoset model requires observation input (step 5), so the decoder becomes an essential generative component of the imagination pipeline.

D

Contemporary approaches

So far, Dreamer family algorithms in multi-agent setting employ standard MARL techniques for non-stationarity mitigation [30] and decentralized models. We argue that these fail in adversarial settings. Explicit communication [9, 6] is incompatible with zero-sum games, as rational players will learn to ignore it. Alternatively, population self-play or explicit opponent modeling [25, 29] suffers from the aforementioned policy-physics entanglement, which can have catastrophic results upon encountering a novel opponent. This is also why we do not report these systems as empirical baselines (Section 5): each is structurally inapplicable to 2p0s IIGs rather than merely inconvenient to run. • MAMBA [9] and DreamerCF [6] are cooperative methods built around inter-agent communication. In a zero-sum game, a communication channel carries no equilibrium value, as rational opponents ignore or exploit it, so porting these methods amounts to deleting their central mechanism. • The opponent-embedding approach of Lu and Chen [25] conditions the world model on an embedding of the opponent’s identity that must be randomly initialized at test time against an unseen opponent. There is then no fixed joint policy to evaluate, and NashConv, which requires one, is not well defined for such a model. • Population self-play with single-agent models [29] instantiates exactly the policy-physics entanglement described above: what is learned is a joint-system model of the environment together with the historical opponent population, not a model of the game. Our decentralized MARSSM ablation is a controlled instance of this family, with the same optimizer and capacity and only the centralization removed, which we consider a fairer comparison than re-implementing any single such system together with its confounding design choices. • LAMIR [21] extends MuZero-style search to IIGs, but assumes deterministic transitions, which excludes both Leduc and Goofspiel-13, and targets search-time planning rather than model learning for sample efficiency, so the comparison would be ill-posed along the axis we measure. As a motivating example, consider the Battleship game. Since a decentralized model is, from a particular player point of view, effectively a POMDP marginalized over past opponents, the player might hallucinate the opponent’s fleet placement based purely on historical correlations with the ego-agent’s own layout. Observing a hit or miss that contradicts this entrenched hallucination breaks the internal belief representation, rendering all subsequent latent planning nonsensical. 23

E

Architectural design

E.1

Mitigating Compounding Errors in High-Dimensional Spaces

As noted in Section 4.2, our current MARSSM formulation reconstructs the explicit observation ôi,t at each imagination step to update the individual latent infosets Iˆi,t . Because our empirical evaluations isolate structural learning by using compact infoset representation vectors, this explicit decoding is computationally inexpensive and highly interpretable. However, scaling this architecture to high-dimensional raw observations (such as images) presents a vulnerability. Standard Dreamer architectures transition strictly within the latent space to avoid the compounding auto-regressive errors and computational overhead of decoding raw pixels at every step of imagination. To resolve this when scaling MARSSM, the architecture can seamlessly adopt a latent bottleneck approach. Rather than decoding the raw observation directly, the centralized world model would predict a low-dimensional observation embedding ei,t (this approach is already employed by the encoder, before predicting the posterior latent stochastic state). The decentralized infoset networks would then aggregate these compact embeddings instead of raw observations. The actual highdimensional observation reconstruction would be decoupled from the autoregressive imagination loop, acting solely as an auxiliary loss on ei,t during training. This two-stage approach preserves the causal information chain required for IIGs while maintaining the computational efficiency and robustness to compounding errors characteristic of the broader Dreamer family.

E.2

Replay Buffer Design

Our replay buffer stores complete trajectories and unrolls sequences strictly from the initial state t = 0. We store fixed-size trajectories and pad with invalid data if needed. This differs fundamentally from standard MBRL practice of sampling random mid-episode chunks. In IIGs, games often have rigid starting configurations, and uncertainty is generated iteratively by hidden opponent actions. Initializing a recurrent state at t > 0 severs the causal chain of strategic deductions, making it impossible for the infoset networks to reconstruct accurate beliefs. While this presents a scaling limitation for very long-horizon games, it ensures rigorous adherence to the game’s informationtheoretic properties. While it is theoretically possible to bypass the t = 0 requirement by storing the latent model states and infoset vectors directly into the buffer [14], doing so in an IIG introduces two fatal issues. First, due to representation drift, latent states stored in the buffer are generated by stale network parameters and lose their semantic meaning as the networks evolve. Second, and more fundamentally, because a player’s policy conditions solely on their infoset, optimizing that policy requires the empirical distribution of the underlying histories within that infoset to perfectly match their true reach probabilities under the current joint policy. Sampling mid-episode histories from an off-policy replay buffer intrinsically violates this property, as the sampled histories reflect stale opponent strategies. Correcting this mismatch requires importance sampling or counterfactual unbiasing, which injects prohibitively high variance into the learning process. How costly that correction is for a replay buffer of full trajectories, which is the design we do use, is a game-dependent matter rather than a uniform verdict on replay. The importance ratio accumulates along a trajectory, so the variance grows with the length of the game, and it grows with how far the policies travel between the moment a trajectory enters the buffer and the moment it is replayed. We consequently clip the counterfactual importance-sampling ratio (Table 2); in Battleship the gradients diverge without it. The empirical picture follows suit: replay does not improve on plain RNaD in Goofspiel-5, whereas in Leduc and in Phantom Tic-Tac-Toe it does, and in the two largest domains it is the weaker of the two RNaD baselines under both head-to-head play (Appendix Q) and approximate exploitability (Table 1). We therefore offer the variance argument as an explanation of the cases where replay does not help, not as a prediction that it never will. 24

F

Environments

Point Card Matching (PCM). A single-agent environment inspired by Goofspiel. The agent holds M cards (1, . . . , M ); point cards are revealed in descending order. Matching the point card scores a point. Reward is sparse (total score at termination). Features dynamic action masking and strict sequentiality. Imperfect Information Goofspiel. A two-player simultaneous-move card game. Players simultaneously play cards; the higher card wins the revealed point card (ties discard it). We use an imperfect-information variant where players observe round outcomes (win/loss/tie) but not the opponent’s specific card. We evaluate on 3-card, 5-card (descending point order), and 13-card (random point order) variants. Leduc Poker [38]. A condensed Texas Hold’em with a 6-card deck (J, Q, K in two suits). Features private cards, a public community card, and two betting rounds. As this is a turn-based game, it is adapted to our framework via dummy actions. Trajectories reach up to 8 steps with multiple chance nodes. Battleship. A two-player board game with placement and battle phases. Players simultaneously and secretly place ships on their grid (ships may not overlap or be adjacent), then simultaneously target opponent tiles each round, receiving hit/miss feedback. We evaluate on a 5×5 board, with two size-2 ships and one size-3 ship for head-to-head comparison. Phantom Tic-Tac-Toe. A turn-based, imperfect-information variant of Tic-Tac-Toe on a 3 × 3 board in which neither player observes the opponent’s marks. A player attempting to claim a tile already held by the opponent is informed of the collision and replays their move, repeating until they place on a free tile, so a turn can consist of several attempts. Following the OpenSpiel [23] and exp-a-spiel [33] implementations, we additionally forbid a player from attempting a tile they already know the opponent occupies. This keeps the game length bounded, and it is without loss of information under our perfect-recall assumption (Section 2), since a player remembers every collision they have encountered. This is a deterministic turn-based game, with trajectories reaching up to 17 steps. It is also adapted via dummy actions. Perturbed Rock-Paper-Scissors. The canonical one-shot matrix game, modified so that the rewards for playing Scissors are multiplied by two, which moves the equilibrium off the uniform strategy; the uniform policy has a NashConv of 2/3. It is the smallest simultaneous-move game we use, and serves as an isolated testbed for the identifiability barriers of Section 3 (Appendix L).

G

Implementation Details

All models and environments were implemented in Python 3.12 using the JAX [4] ecosystem and the newest version of Flax [16] (NNX). To keep consistent with the DreamerV3 architecture [14], we use a single LaProp optimizer [43] with a constant learning rate of α = 3×10−4 for all network parameters. Training utilizes a batch size of 64 complete trajectories per gradient step. All experiments were conducted on a cluster, utilizing 2 cores of AMD server processors with up to 30 GB of system RAM for the PPO budgeted approximate NashConv across 10 stored model seeds and 12 GB for other experiments (experiment dependent) and a single NVIDIA RTX A4000 GPU, running on Debian 12 with CUDA 13.0. To upper-bound the complexity, our most time-demanding experiments (Battleship 5x5) required a maximum of 24 hours of total compute time to train for 30,000 minibatches across 10 seeds. The estimated compute time for all the experiments (including preliminaries not reported in the paper) is around 4 weeks of CPU/GPU time. G.1

Network Architecture

As our work focuses on algorithmic adaptations rather than novel network structures, we use the standard DreamerV3 network architecture throughout. Each predictor component is implemented as a Multi-Layer Perceptron (MLP) utilizing layer normalization [3] and SiLU activations. The sequential network embeds input components separately, creates a skip connection with the recurrent state, concatenates the features, passes them through an MLP, and uses a GRU cell [8] as the gate unit. The posterior encoder consists of two primary pathways: one embedding the joint observation, 25

and one predicting the stochastic latent distribution conditioned on both the recurrent state and this observation embedding. G.2

Capacity Constraints

MLP Dimensions: Every MLP predictor across the architecture consists of an input layer, a single hidden layer of 256 units, and an output layer (with layer normalization and SiLU applied after the input and hidden layers). The dimensions of the input and output layers are determined based on the specific input and target sizes of the respective environment. To test the model’s ability to capture dynamics under heavily constrained capacity, we strictly the size of latent state representations. Deterministic States: We constrain the dimensionalities of the deterministic latent vectors to correspond directly to the true information content of the environment. Specifically, the central recurrent state ĥt matches the dimensionality of the true joint infoset representation, and each decentralized latent infoset Iˆi,t matches the true individual player’s infoset dimensionality. The specific per-environment infoset representation sizes are: • Point Card Matching with 3 cards (PCM 3): 16 • Goofspiel: 35 (3 cards), 87 (5 cards), 535 (13 cards) • Leduc Poker: 39 • Phantom Tic-Tac-Toe: 104 • Battleship: 716 (for a 5x5 board with 3 ships each) Stochastic Latent Space: Rather than relying on the excessively large stochastic capacity of standard DreamerV3 (e.g., 32 × 32 classes), we deliberately restrict the categorical distribution to the minimal capacity required to uniquely identify chance outcomes without forcing the network to “temporally multiplex” representations. Temporal multiplexing occurs when the latent capacity is so constrained that the network must assign the exact same latent code zt to completely distinct physical chance outcomes, relying entirely on the recurrent state ĥt to disambiguate them (e.g., class 1 representing a private card at step 1, but a public card at step 3). This places immense optimization strain on the dynamics MLP. To prevent this, we allocate exactly enough capacity to assign a distinct latent configuration to each unique class of chance node. For Imperfect Information stochastic Goofspiel-13, we use a single distribution of 13 categories. While a single distribution of 34 categories should be sufficient for Leduc, we find that under our capacity constraints, the model cannot learn the underlying environment, due to gradient dilution for a single, large distribution. So, our main results use a factored stochastic state of 2 categorical distributions with 6 categories. We present a comparison in Appendix J. We explicitly note that utilizing domain knowledge to size the latent space is an experimental choice designed to evaluate the framework under minimal capacity conditions. It is not a structural requirement of MARSSM. For environments with unknown or intractably large chance dynamics, the framework natively supports scaling back up to latent spaces with many distinct outcomes (e.g., 32 × 32 classes) typical of the Dreamer family. In deterministic environments, the free-nats KL clipping threshold is set to 0, while in stochastic environments it is maintained at 0.2. G.3

Hyperparameters

Unless explicitly stated otherwise in the environment descriptions, all NashDreamer experiments share the hyperparameters listed in Table 2. The architecture is trained entirely online for 30, 000 gradient steps, where, at each step, a new minibatch is collected from the real environment, used to update the world model, which then generates imagination trajectories minibatch. Then, the actor-critic performs a step on the concatenation of the imagined minibatch with the real minibatch. For the significantly larger state spaces of Goofspiel-13, Battleship 5 × 5 and Phantom Tic-Tac-Toe, the initial world model warm-up phase is extended from the default 1, 000 to 2, 000 gradient steps to ensure stable structural representations before policy learning begins. 26

To ensure numerical stability during training, we apply several targeted regularizations following [14]. First, we add a 0.01 uniform mixture to both the prior and posterior categorical distributions to prevent KL divergence spikes to infinity. Second, during imagination, we apply a state sampling threshold of 0.05 (thresholding individual category probabilities and renormalizing before sampling the stochastic state zt ) to prevent the actor from exploring highly improbable hallucinated branches. The only exception to this threshold is Leduc under the flat stochastic space, where this threshold is 1 set to 0.01 (as the initial chance outcomes have probability 30 < 0.05). Finally, the target predictors for terminal states and legal actions are thresholded at 0.5 to yield binary masks during execution. For the off-policy RNaD optimization, we also clip the counterfactual importance sampling weights to prevent exploding gradients. We further adopt the DreamerV3 practice of zero-initializing the output layers of the reward and critic predictors, so that both heads predict exactly zero at the start of training; this keeps large, arbitrary early predictions from dominating the world-model loss and the imagined returns during the first iterations, and complements the warm-up period discussed in Appendix C. Table 2: Shared NashDreamer hyperparameters used across all experiments.

G.4

Hyperparameter

Value

General Training Total gradient steps World model warm-up period Batch size

30, 000 1, 000 (2, 000 for large envs) 64

Optimizer (Shared) Learning rate Learning rate warmup steps Betas (β1 , β2 ) Epsilon (ϵ) Adaptive Gradient Clipping (AGC)

3 × 10−4 1, 000 (0.9, 0.999) 1 × 10−8 0.3

World Model Targets & Regularization Reward symlog bins (one-sided) Categorical uniform mixture State sampling threshold Terminal / Legal prediction threshold Dynamics KL scale (β dyn ) Encoder KL scale (β enc ) Prediction loss scales Infoset loss scale

20 0.01 0.05 0.5 1.0 0.1 1.0 1.0

Actor-Critic & RNaD Critic symlog bins (one-sided) RNaD entropy scale (η) V-trace parameters (ρ̄, c̄, γ, λ) Target network EMA rate (τ ) Counterfactual IS clipping threshold NeuRD advantage absolute clip threshold NeuRD advantage relative clip threshold Imagination uniform policy mixture (ϵ) Loss weighting (Real : Imagined)

20 0.2 (∞, 1.0, 1.0, 1.0) 0.001 100.0 10, 000 2.0 0.2 1.0 : 1.0

Standalone RNaD Baseline Configuration

To ensure a rigorous and fair empirical comparison, the model-free standalone RNaD baseline utilizes the exact same Actor-Critic hyperparameters and optimizer configurations as NashDreamer (listed in Table 2). Because the standalone baseline relies on offline experience replay rather than online model-based imagination, we maintain a per-trajectory replay buffer with a capacity of 10, 000 trajectories. The baseline is trained using an environment-dependent replay ratio (the number of trajectories sampled 27

from the buffer per newly collected online trajectory) to compensate for the additional computational overhead of the model-based variant. For Goofspiel-13, this ratio is set to 26, and for Battleship 5 × 5, it is set to 40. We train the baseline for 30, 000 gradient steps, matching NashDreamer. Each gradient step entails the collection of a new minibatch from the environment, concatenation with replayed trajectories (if used) and performing the step on this larger minibatch. G.5

MMD Baseline Configuration

The MMD baseline [36] shares the network architecture, optimizer and batch size of the RNaD baseline; the two differ only in the policy-optimization rule, with MMD replacing RNaD’s policydependent reward regularization by magnet-policy regularization. This parity is what allows the NashDreamer[MMD] and NashDreamer[RNaD] comparison of Appendix Q to isolate the effect of the world model rather than of the surrounding implementation. Concretely, we follow the OpenSpiel PPO implementation, with an entropy exploration bonus and the additional inverse-KL penalty described by Rudolph et al. [33], and we adopt their “generic” hyperparameter setting. All network architectures and layer sizes are identical to RNaD (Appendix G). The MMD-specific values are listed in Table 3. Table 3: MMD-specific hyperparameters, shared by the model-free MMD baseline and NashDreamer[MMD], and held fixed across all domains. Hyperparameter

Value

Role

PPO epochs PPO clipping coefficient Inverse-KL coefficient Magnet coefficient (α) Advantage-normalization ϵ Discount / bootstrapping (γ, λ)

7 0.1 0.05 0.2 1 × 10−8 (1.0, 1.0)

optimization epochs per batch realizes the proximal term of Eq. 6 loss multiplicand of the inverse KL penalty fixed uniform magnet, i.e. an entropy bonus numerical guard, not a tuned quantity as for RNaD (Table 2)

The magnet is fixed and uniform, so the magnet term contributes an entropy bonus rather than an anchor that moves with the policy; as noted in Section 2.2, this means MMD targets a quantal response equilibrium rather than an NE. Advantages are normalized in the standard PPO fashion over the valid (non-padded) entries of the batch, A − E[A] Ã = p , Var[A] + ϵ with γ and λ set to 1.0 to match the RNaD configuration rather than as independent choices. We hold these values fixed across domains and do not tune them per environment. The primary reason is that tuning would not affect the claim under test: the model-free MMD baseline and NashDreamer[MMD] use the same MMD hyperparameters, so whichever values are chosen, the comparison isolates the effect of the world model on a given optimizer. An untuned MMD is thus a conservative baseline for the modularity claim of Section 4.3 and a fair one for the sample-efficiency comparison. The values are moreover not arbitrary: they are the generic configuration reported by Rudolph et al. [33] for exactly this family of games. Finally, the domains that motivate this work are those in which environment interaction is the dominant cost, which raises the value of an optimizer that performs acceptably without per-domain tuning; we note as a secondary point that tuning against a world model that is itself still changing is of questionable soundness. We acknowledge that per-domain tuning would likely improve MMD’s absolute numbers, and we make no claim that the values used are optimal for it.

H

Exploitability Evaluation Protocols

H.1

Exact NashConv

All NashConv values reported in Section 5 are computed exactly against the true game. They are not computed against the learned world model, and they are not estimated by sampling. 28

At evaluation time we first extract a behavioral policy over the true game. We traverse every infoset of the true game and query the trained actor network at it; for NashDreamer this means replaying that infoset’s AOH τi through the infoset model to obtain the latent embedding Iˆi and reading off πi (· | Iˆi ), with the true legal-action mask applied. The evaluated object is therefore a well-defined joint policy over the true game’s infosets, which is what makes NashConv meaningful as an equilibrium metric. Given this joint policy, each player’s best response is computed by dynamic programming over the full true game tree, following the OpenSpiel exploitability routine [23]; for Phantom Tic-Tac-Toe we instead use the exp-a-spiel library of Rudolph et al. [33], whose implementation of the game matches the variant described in Appendix F. Chance nodes are enumerated exactly with their true probabilities rather than sampled. Simultaneous-move nodes are expanded in the standard turn-based form, in which the second player’s infoset does not contain the first player’s action at that node, so the best-response computation respects the original information structure. The reported values are exact for the extracted policies; the only approximation in the pipeline is the extraction itself, i.e., the network’s own numerical output at each infoset. NashConv curves report values per training seed, evaluated at fixed intervals of environment interaction. H.2

Best-Response Learner

Section 5.2 states the budgeted best-response protocol and reports its results; this section documents the learner that protocol runs. Taking the maximum over the ten PPO seeds rather than their mean makes the estimate the tightest lower bound the search budget supports, which is the conservative choice for a metric on which we claim an improvement. The best-response learner is plain PPO: the same implementation and hyperparameters as the MMD baseline of Appendix G.5, with the inverse-KL penalty and the entropy exploration bonus removed. It is run identically in both games, for 3,000 gradient steps with batch size 64, γ = 1.0, λ = 1.0, clipping coefficient 0.1 and 7 inner epochs. The budget of 3,000 steps is a tenth of that of the largest evaluated checkpoint, which suffices because computing a best response against a fixed opponent is a single-agent problem and materially easier than equilibrium finding. A budget that is too small manifests as a best response failing to reach the value the evaluated profile achieves against itself, which indicates that the search fell short rather than that the checkpoint is unexploitable. We mark any such cell with †. No cell of Table 1 is affected, so the budget is inside the useful range everywhere the main text draws a conclusion from it. H.3

Aggregation over PPO Seeds

Table 4 reports the same experiment aggregated by averaging over the 10 PPO seeds instead of taking the worst case. The values are uniformly smaller, as expected of a looser bound, and the least exploitable method is the same one under both aggregations at every checkpoint of both games. The full ranking is identical throughout Goofspiel-13; in Battleship a single adjacent pair exchanges places at each checkpoint, the two RNaD baselines at 10k, MMD and NashDreamer[RNaD] at 20k, and NashDreamer[RNaD] and model-free RNaD at 30k. Each of those exchanges is between values lying well inside one another’s intervals, and every claim made in Section 5.2 holds under either aggregation. One cell is flagged under this looser bound, NashDreamer[RNaD] at 30k in Goofspiel-13, on two of the ten training seeds. Goofspiel (13 cards) Method MMD ND[MMD] RNaD RNaD w/ replay ND[RNaD]

Battleship (5×5)

10k

20k

30k

10k

20k

30k

0.789 ± 0.076 0.805 ± 0.069 0.675 ± 0.368 0.701 ± 0.075 0.408 ± 0.133

0.695 ± 0.098 0.744 ± 0.085 0.381 ± 0.258 0.476 ± 0.073 0.184 ± 0.143

0.684 ± 0.067 0.708 ± 0.082 0.279 ± 0.276 0.434 ± 0.081 0.093 ± 0.134†

1.202 ± 0.058 1.069 ± 0.110 1.312 ± 0.067 1.306 ± 0.057 0.979 ± 0.105

0.986 ± 0.100 0.781 ± 0.099 1.135 ± 0.172 1.235 ± 0.089 0.951 ± 0.140

0.831 ± 0.147 0.612 ± 0.127 0.937 ± 0.163 1.007 ± 0.152 0.912 ± 0.235

Table 4: Budgeted approximate exploitability averaged over the 10 PPO seeds, rather than taking the worst case as in Table 1. Mean ±2σ over the 10 training seeds. † marks the one cell in which the best response failed to reach the profile’s own value, on two of the ten training seeds, i.e. where the budget was too small.

29

Figure 5: NashDreamer LE-Tree for round 1 of Leduc Poker under a flat latent space (1 × 34). The single categorical distribution suffers from severe mode collapse, failing to learn the true environment structure.

Figure 4: NashDreamer LE-Tree for round 1 of Leduc Poker (private cards 1 and 2 dealt). Utilizing the factored latent space, the model accurately learns the underlying game dynamics.

I

Latent Error Trees

To qualitatively assess the structural fidelity of the learned world model, we introduce Latent Error Trees (LE-Trees). Starting from the initial state, we recursively unroll the game tree following the agent’s current joint policy, pruning any branch where a player’s action probability falls below a threshold of 0.05. In these visualizations, blue diamond nodes represent the ground-truth game states, while the adjacent rectangular nodes represent their corresponding latent reconstructions. These latent nodes are color-coded based on the maximum prediction error across all deterministic decoder heads (reward, continuation, legal actions, and individual infoset reconstruction): green indicates low error (error < 0.2), orange indicates moderate error (0.2 ≤ error < 0.5), and red indicates severe error (error ≥ 0.5). Any non-green node is explicitly annotated with the specific predictor component responsible for the maximum error. Finally, all transitions (edges) are visually weighted according to their corresponding chance or policy probabilities.

J

Leduc Failure Modes and Latent Space Capacity

As described in Section 5, while the world model accurately matches the dynamics for round 1 of Leduc Poker (Figure 4), it mistakenly predicts the wrong public card in round 2 (the LE-Tree for round 2 is contained in the Supplementary Material due to size constraints). Our Leduc implementation utilizes binary observation and infoset tensors, alongside rewards divided by the maximal bet (13), effectively rescaling the reward signal into the [−1, 1] range. This bounded structure meets the criteria of low prediction magnitude changes combined with high stochasticity, which inherently exacerbates the tension between posterior collapse and representation drift in the late stages of the game. Figure 6 compares the three configurations of NashDreamer[RNaD] we ran on this domain. Replacing the factored latent with a minimal flat one, a single categorical with 34 categories, costs surprisingly little in NashConv, but the world model then fails to capture the environment’s topological structure (Figure 5), which Appendix K.4 quantifies as posterior collapse at the initial deal. The actorcritic evidently compensates for that inaccuracy; since our objective is a structurally accurate and interpretable model, the factored latent remains the better choice. The one-way variant sets β enc = 0, removing the posterior regularizer as Theorem 3 prescribes, and is the worst of the three in the later stages of training. It does not prevent the structural error at the public chance node, as its prior instead spreads over far more latent configurations than the node has outcomes (Appendices K.4 and K.5), which is what representation drift looks like once nothing pulls the posterior toward the prior. The variant our theory endorses is therefore the one that can perform the worst in practice, which is the tension stated in Section 4.4. 30

NashConv

100

NashDreamerRNaD factored NashDreamerRNaD one_way NashDreamerRNaD flat Uniform Policy WM warm-up end

10 1

0.00

0.25

0.50

0.75

1.00

Environment steps

1.25

1.50

1.75 1e7

Figure 6: NashConv in Leduc Poker for the three latent and loss configurations of NashDreamer[RNaD]: the factored latent, the flat single-categorical latent, and the one-way variant with β enc = 0. All three deteriorate after the initial descent, the one-way variant furthest and the factored latent least. Bold curves are means over the 10 training seeds, faint curves the individual seeds; the dotted line marks the end of the world-model warm-up.

K

Quantitative World-Model Diagnostics

The Latent Error Trees of Appendix I are qualitative. To attribute the policy results of Section 5 to world-model quality rather than to actor-critic dynamics alone, we measure the learned model directly under three protocols, defined in Appendices K.1, K.2 and K.3: rollout validity, the per-step emission error of a trajectory generated entirely by the model; the posterior-collapse indicator, the expected KL(q θ ∥ pθ ) at chance nodes; and chance-outcome calibration, the agreement between the prior’s distribution over successors and the true one. Results are reported in Appendices K.4 to K.7. All three protocols filter the model’s categorical stochastic state before use, applying the thresholding used during imagination (Appendix G.3), so that the model is scored on the states it generates at run time. The filter acts on each categorical distribution independently: within a distribution, categories below the threshold are set to zero and the remainder renormalized, so a factored state whose two categoricals read [0.9, 0.0, 0.01] and [0.01, 0.0, 0.9] becomes [1, 0, 0] ⊗ [0, 0, 1] at a threshold of 0.05. The threshold must lie below the smallest outcome probability the game presents, or real outcomes are removed before they can be scored; we use 0.05 for the factored variants and 0.01 for the flat one, whose single categorical has to resolve the 1/30 of Leduc’s initial deal. In Goofspiel-5, which has no chance nodes, every emission converges: terminal flags and legal actions within 1,000 training steps, observations about an order of magnitude later, and the reward last of all. Appendix K.7 reports the four curves and the ordering in full. In Leduc Poker, the picture is qualitatively different and matches the deterioration visible in Figure 3. Terminal flags and legal actions are again identified without error within 1,000 steps, and deterministic transitions are reconstructed exactly by the factored and one-way variants, but the observation and reward errors never reach zero and the chance transitions stay miscalibrated for the whole of training. Because the three tested model variants in Leduc fail in visibly different ways, we report each diagnostic separately for all three rather than summarizing them here: Appendix K.4 for the posteriorcollapse indicator, Appendix K.5 for chance-outcome calibration and Appendix K.6 for the prediction accuracies under rollout. These diagnostics confirm the mechanism our theory predicts. NashDreamer learns models of deterministic game components without major issues, whereas in the presence of chance stochasticity the world model can settle on an incorrect representation, or even degrade with further training, and the degradation is localized in the chance transitions rather than in the deterministic emissions. This is the empirical counterpart of Theorem 1: the objective itself rewards a model that declines to disambiguate chance outcomes. 31

K.1

Rollout Validity

This protocol measures whether a trajectory generated entirely by the model corresponds to a realizable trajectory of the true game. Unlike a single-step emission score, errors compound along the rollout, as they do when the actor-critic trains on imagined data. The model rollout and the true game are advanced with the same joint actions, sampled from the model. The recurrent state is initialized at h0 and the latent infoset embeddings at their initial values; the initial stochastic state is sampled from the prior pθ . Each step proceeds as follows. 1. At a chance node of the true game, the outcome is not sampled from the true chance distribution but chosen to minimize the L∞ distance between the true joint observation and the one reconstructed from the current model state. Divergence later in the rollout is then attributable to the learned dynamics rather than to the two trajectories having drawn different chance outcomes. 2. Each player’s action is sampled as ati ∼ πi (· | Iˆit ) from the latent infoset embedding, or from the infoset reconstructed from the model state when the model was trained on observations alone. 3. (at1 , at2 ) advances the recurrent state through the centralized sequence model and advances the true game; zt+1 is then sampled from the prior at the new recurrent state. 4. The observations reconstructed from mt+1 update each Iˆt+1 . i

The rollout terminates when the true game reaches a terminal state. Error statistics. At each step the model’s emissions are compared with the true ones and the discrepancy accumulated: L1 for the reward, L∞ for the joint observation, and 1 per boolean mismatch for the terminal flag and the legal-action mask. Each statistic is the accumulated magnitude divided by the number of valid steps, so all four are mean per-step magnitudes, on different scales. K.2

Posterior Collapse at Chance Nodes

By Theorem 2, an infoset-isomorphic model attains an expected Eo KL(q θ ∥ pθ ) at a chance node equal to the entropy of that node, whereas posterior collapse drives the same quantity to zero. The model is conditioned on the true action sequence (teacher forcing). At each non-chance node the stochastic state is taken from the posterior, selecting the outcome whose reconstructed joint observation is closest to the true one in L∞ norm, which keeps the model state aligned with the true history. At a chance node we enumerate the outcomes, evaluate the posterior q θ (· | o) induced by each, and record their expected divergence from the single prior pθ at that node, X
C= p(o) KL q θ (· | o) pθ . (7) o

Chance nodes are grouped by their number of outcomes K, and C is averaged, unweighted, over all nodes with the same K. Every chance node in our environments is uniform, so an isomorphic model attains C = log K. Deviation from log K has two distinct causes. C < log K indicates posterior collapse: the posterior depends only weakly on the chance outcome. C > log K indicates that the prior assigns less mass to the realized posterior mode than a calibrated prior over K equiprobable outcomes would, i.e. that it is spread over more latent configurations than the node has outcomes. A modest excess is expected when the latent space is larger than K and distinct histories reaching the same outcome occupy distinct configurations; a large excess indicates that the prior does not track the posterior. K.3

Chance-Outcome Calibration

This protocol measures the prior’s distribution over successors: what fraction of its mass corresponds to realizable outcomes, and whether those outcomes carry the correct probabilities. Teacher forcing is applied as in Appendix K.2, but the stochastic states are drawn from the prior pθ . At each node the model’s stochastic states are enumerated and assigned to true successors: a state 32

is assigned to the successor whose joint observation is nearest its reconstruction in L∞ norm, or to a separate unmatched bucket if that distance exceeds 0.3 for every successor. Summing model probabilities within each bucket gives a marginal over the true successors, plus the unmatched mass. Deterministic nodes are included, their true successor distribution being a Dirac. Two statistics are reported: the L1 distance between the bucketed, unnormalized marginal (so the model marginal need not sum up to 1) and the true successor distribution and the unmatched mass, both bounded by 1. Both are averaged over all nodes of a kind, with chance and deterministic nodes reported separately, and are not grouped by K. The two isolate different failures: unmatched mass measures prior mass on latent states with no corresponding true successor, whereas a large L1 distance with little unmatched mass indicates successors that are represented but mis-weighted. K.4

Posterior Collapse in Leduc Poker

Leduc has chance nodes of two sizes: the initial private deal, with K = 30 outcomes, and the second-round public card, with K = 4. We measure the statistic of Appendix K.2 at both over 10 seeds and 30,000 gradient steps, for the three configurations the paper uses: the flat latent, a single categorical with 34 categories (Appendix J); the factored latent, with 36 configurations; and the one-way variant, which keeps the factored latent but sets β enc = 0 as Theorem 3 prescribes. At the initial node (K = 30) the flat latent collapses and the other two do not (Figure 7). It settles near 1.7, almost exactly half of the log 30 = 3.401 baseline, so it cannot distinguish all thirty outcomes; since it also struggles at deterministic nodes (Appendix K.5), the difficulty lies with the flat stochastic state rather than with this node. The factored latent recovers most of the gap at 2.9, leaving a residual collapse, and the one-way variant sits just above the baseline at 3.45, which follows from its 36 configurations: a prior spread over all of them exceeds log 30. At the public node (K = 4) the ordering inverts (Figure 8). The flat latent sits on the log 4 = 1.386 baseline, the factored latent just above at 1.45, and the one-way variant far above at 2.4, with a visibly wider spread across seeds. Its prior is therefore spread over many more configurations than the node has outcomes, which we attribute to the prior being unable to track the moving posterior — the mechanism Section 5.1 holds responsible for the one-way variant’s poor performance. No configuration is faithful at both nodes: the flat latent is calibrated at small K and collapses at large, the factored latent is close at both and exact at neither, and dropping the posterior regularizer trades collapse for drift. This is the tension of Section 7, and why the structural error at the deeper chance node survives every variant we ran. Posterior collapse: chance nodes with 30 outcomes

Posterior collapse: chance nodes with 4 outcomes

3.0

Expected KL(posterior || prior)

Expected KL(posterior || prior)

3.5

2.5 2.0 1.5 1.0

one_way factored flat Uniform baseline: log(30) = 3.401

0.5 0.0

0

5000

10000

15000

Training step

20000

25000

2.0

1.5

1.0

one_way factored flat Uniform baseline: log(4) = 1.386

0.5

30000

0

Figure 7: Expected KL(q θ ∥ pθ ) at the K = 30 private-deal chance node of Leduc Poker, averaged over nodes of that size. Bold curves are means over 10 seeds, faint curves the individual seeds. The flat latent collapses to roughly half the uniform baseline. K.5

2.5

5000

10000

15000

Training step

20000

25000

30000

Figure 8: The same statistic at the K = 4 publiccard chance nodes of Leduc Poker. Here the flat latent is calibrated and the one-way variant overshoots the baseline by a wide margin, indicating a prior that is not tracking the posterior.

Chance-Outcome Calibration in Leduc Poker

The calibration statistics of Appendix K.3 localize the Leduc failure more sharply than the collapse statistic can, because they separate the nodes at which the model must represent a distribution from 33

those at which it must represent a single successor. We measure them for the same three configurations over 10 seeds and 30,000 gradient steps; both statistics are bounded by 1. At deterministic nodes the factored and one-way variants drive both statistics to zero within roughly 10,000 steps, while the flat latent settles near 0.11 on both (Figures 11 and 12). At chance nodes the three separate: the factored variant reaches an L1 distance of 0.16 with 0.07 of its mass unmatched, the flat latent 0.45 and 0.44, and the one-way variant 0.63 and 0.63 (Figures 9 and 10). For the one-way and flat variants the two statistics coincide, so no successor is over-weighted and their whole discrepancy is deficit. They reach that signature for different reasons, which the collapse statistic separates: at K = 30 the one-way posterior still resolves every deal whereas the flat one sits at about half of log K (Appendix K.4). The one-way variant therefore represents every outcome and only mis-weights it, diverting two thirds of its prior mass to configurations the game cannot produce, while the flat variant’s deficit includes outcomes it no longer distinguishes at all. The flat latent is also the only configuration that fails to resolve a deterministic successor. We attribute this to a single one-hot supplying the sequence model with less usable structure than a pair of them, so the factored latent earns its place by the form in which it presents the outcomes rather than by how many it can represent. The factored variant is the only one whose L1 distance exceeds its unmatched mass, so beyond a small leakage it misallocates probability among the outcomes it does represent. It is nevertheless the best configuration at chance nodes and exact at deterministic ones: the structural error of Section 5.1 is, for this configuration, a misallocation rather than a failure to represent. Chance nodes: l1 distance to true distribution

Chance nodes: unmatched probability mass (error) one_way factored flat

one_way factored flat

1.0

Unmatched probability mass (error)

L1 distance to true distribution

1.0

0.8

0.6

0.4

0.2

0.8

0.6

0.4

0.2

0.0 0

5000

10000

15000

Training step

20000

25000

30000

0

Figure 9: L1 distance between the bucketed marginal of the prior and the true outcome distribution at the chance nodes of Leduc Poker. Bold curves are means over 10 seeds, faint curves the individual seeds.

Unmatched probability mass (error)

L1 distance to true distribution

0.6

0.4

0.2

0.0 10000

15000

Training step

20000

25000

20000

25000

30000

one_way factored flat

1.0

0.8

5000

15000

Training step

Deterministic nodes: unmatched probability mass (error)

one_way factored flat

0

10000

Figure 10: Unmatched probability mass at the chance nodes of Leduc Poker: the share of the prior that decodes to no real outcome. The oneway variant leaves roughly 0.63 unmatched, the flat latent 0.44 and the factored latent 0.07.

Deterministic nodes: l1 distance to true distribution 1.0

5000

0.8

0.6

0.4

0.2

0.0

30000

0

Figure 11: The same L1 statistic at the deterministic nodes of Leduc Poker, where the true distribution is a Dirac on the single successor. The factored and one-way variants reach zero; the flat latent does not.

5000

10000

15000

Training step

20000

25000

30000

Figure 12: Unmatched probability mass at the deterministic nodes of Leduc Poker. It vanishes for the factored and one-way variants but plateaus near 0.11 for the flat latent.

34

K.6

Rollout Validity in Leduc Poker

Rollout validity (Appendix K.1) removes the teacher forcing of the two statistics above, so errors compound as they do during imagination. We report it for the same three configurations over 10 seeds and 30,000 gradient steps. Leduc rewards are divided by the maximal bet of 13 into [−1, 1] (Appendix J), so reward errors are on that normalized scale. The deterministic emissions survive the rollout (Figures 15 and 16): terminal flags and legal-action masks reach zero within roughly 1,000 gradient steps for all three configurations and stay there. The Leduc failure is not a loss of track of when the game ends or which actions are available. Neither observations nor rewards reach zero (Figures 13 and 14), and they disagree on which configuration is best. On observations the one-way variant leads at 0.36, ahead of the factored at 0.54 and the flat at 0.83, all down from an initial 4.1; on rewards the first two reverse, at 0.024 and 0.029, with the flat latent again worst at 0.037. The flat latent is the only one worst on both, consistent with its failure to resolve even deterministic successors (Appendix K.5). That the one-way variant leads here despite leaving 0.63 of its chance-node mass unmatched follows from what each protocol measures: rollout validity scores whether a generated trajectory is realizable, not the probability with which it is generated, and the one-way defect is in the probabilities. The two protocols are therefore complementary, since a model can score well here while visiting realizable states at incorrect frequencies. That the factored variant nonetheless trails it on observations, despite being far better calibrated at chance nodes, reflects two different notions of a latent state supporting an outcome. Bucketing (Appendix K.3) credits an outcome whenever some configuration in the prior’s support decodes close to it, whereas a rollout uses the configuration actually sampled. The size of the gap suggests the factored variant’s best supporter for an outcome is considerably better than its typical sampled one. Rollout validity: obs error

Rollout validity: reward error

one_way factored flat

4.0

one_way factored flat

0.06

3.5 0.05

Reward avg error

Obs avg error

3.0 2.5 2.0 1.5

0.03 0.02

1.0 0.5

0.01 0

5000

10000

15000

Training step

20000

25000

30000

0

Figure 13: Rollout observation error in Leduc Poker, as the mean per-step L∞ error of the joint observation. Bold curves are means over 10 seeds, faint curves the individual seeds. No configuration reaches zero. K.7

0.04

5000

10000

15000

Training step

20000

25000

30000

Figure 14: Rollout reward error in Leduc Poker, as the mean per-step L1 error on rewards normalized to [−1, 1]. The ordering of the factored and one-way variants is the reverse of the one on observations.

Rollout Validity in Goofspiel-5

Goofspiel-5 reveals its point cards in descending order (Appendix F) and therefore contains no chance nodes, so the failure mode of Theorem 1 cannot arise. It serves as the positive control for the Leduc results: it establishes that the architecture can recover the rules of a game exactly. We report rollout validity for NashDreamer[RNaD] over 10 seeds and 30,000 gradient steps. Every statistic converges. Terminal flags and legal-action masks reach zero error within 1,000 gradient steps and remain there (Figures 19 and 20). The observation error falls from roughly 4.4 to 0.25 over the same 1,000 steps and to near zero by 10,000 (Figure 17). The reward converges last (Figure 18): it reaches roughly 0.04 within 1,000 steps, then descends slowly with individual seeds spiking to 0.1, and approaches zero only in the final third of training. The reward is also the slowest statistic in Leduc (Appendix K.6), which suggests a property of the reward signal rather than of either game. We attribute it to the reward being terminal: it is zero at 35

Rollout validity: terminal error 0.7

one_way factored flat

1.0

0.8

0.5

Legal avg error

Terminal avg error

Rollout validity: legal error

one_way factored flat

0.6

0.4 0.3 0.2

0.6

0.4

0.2

0.1 0.0 0

5000

10000

15000

Training step

20000

25000

0.0

30000

0

Figure 15: Rollout terminal-flag error in Leduc Poker. All three configurations reach zero within roughly 1,000 gradient steps and the curves coincide thereafter.

5000

10000

15000

Training step

20000

25000

30000

Figure 16: Rollout legal-action-mask error in Leduc Poker, which likewise reaches zero for all three configurations and stays there.

every step except the one ending the game, and small in magnitude, so it contributes little to the prediction loss and is resolved last. Rollout validity: obs error

Rollout validity: reward error

NashDreamerRNaD 5

0.150

Reward avg error

4

Obs avg error

NashDreamerRNaD

0.175

3

2

1

0.125 0.100 0.075 0.050 0.025

0 0

5000

10000

15000

Training step

20000

25000

0.000

30000

0

Figure 17: Rollout observation error in Goofspiel-5, as the mean per-step L∞ error of the joint observation. Bold curve is the mean over 10 seeds, faint curves the individual seeds.

15000

Training step

20000

25000

30000

Rollout validity: legal error NashDreamerRNaD

1.0

NashDreamerRNaD

0.7

0.8

Legal avg error

0.6

Terminal avg error

10000

Figure 18: Rollout reward error in Goofspiel-5, the slowest of the four to converge, with individual seeds spiking before falling back.

Rollout validity: terminal error 0.8

5000

0.5 0.4 0.3

0.6

0.4

0.2

0.2 0.1

0.0 0.0

0 0

5000

10000

15000

Training step

20000

25000

30000

5000

10000

15000

Training step

20000

25000

30000

Figure 20: Rollout legal-action-mask error in Goofspiel-5, likewise at zero within 1,000 gradient steps.

Figure 19: Rollout terminal-flag error in Goofspiel-5, at zero within 1,000 gradient steps.

36

L

Centralized versus Decentralized World Models

Section 3 argues that a decentralized world model faces identifiability barriers rather than merely an accuracy penalty. This appendix tests that claim directly by ablating the one property under dispute: we train a decentralized MARSSM in which each player’s world model conditions only on that player’s own actions and observations, leaving the architecture, the objective and the optimizer untouched. Setup. The domain is perturbed Rock-Paper-Scissors (Appendix F) a matrix game with non-uniform equilibrium. Both variants use RNaD as the actor-critic and run for 1,000 gradient steps without switching the regularization policy, over 10 seeds. At each evaluated checkpoint we collect 4 batches of 2,058 rollouts and score them by the protocol of Appendix K.1. The decentralized model learns everything except the reward. Figures 21–24 report the four error statistics. On the three quantities that do not depend on the opponent’s action, the two variants are essentially the same model: terminal flags and legal-action masks reach zero error within 100 gradient steps for both, and the joint observation error falls to zero for both, the decentralized variant simply taking longer to get there, around 600 steps against roughly 200 for the centralized one. The reward separates them completely. The centralized model error in predicting the reward actually increases with training, due to it being unidentifiable for the ego-centric model. Rollout validity: legal error

Rollout validity: terminal error 0.6

0.8

Legal avg error

Terminal avg error

0.5 0.4 0.3 0.2

0.6

0.4

0.2

0.1

0.0

0.0 0

200

400

Training step

600

800

0

1000

200

400

Training step

600

800

1000

Figure 22: Legal-action-mask error in perturbed Rock-Paper-Scissors, on the same convention. The two variants again coincide, both reaching zero within 100 gradient steps.

Figure 21: Terminal-flag error over rollouts in perturbed Rock-Paper-Scissors. Bold curves are means over the 10 seeds, faint curves the individual seeds. The two variants coincide.

Rollout validity: reward error

Rollout validity: obs error

central decentral

1.0

central decentral

1.0

0.8

Reward avg error

0.8

Obs avg error

central decentral

1.0

central decentral

0.6

0.4

0.6

0.4

0.2 0.2 0.0 0.0

0 0

200

400

Training step

600

800

1000

200

400

Training step

600

800

1000

Figure 24: Reward error in perturbed RockPaper-Scissors. The centralized model converges to zero, whereas the decentralized model diverges to approximately 1.0: without observing a2 the reward is unidentifiable from its inputs.

Figure 23: Joint-observation error in perturbed Rock-Paper-Scissors. Both variants converge to zero; the decentralized model is slower but not obstructed.

37

M

Single-Agent Validation

We validated NashDreamer against the reference DreamerV3 implementation on PCM with 3 cards. To ensure a fair comparison, we adapted the reference baseline to accommodate the specific constraints of the environment. Standard DreamerV3 utilizes a burn-in period to approximate the recurrent state from sampled trajectory chunks. However, in a strictly sequential game like PCM, a short burn-in period discards critical historical context, causing structure learning to fail. To resolve this, we configured the reference implementation to sample full trajectories starting from the root node, exactly mirroring the NashDreamer setup. Furthermore, rather than relying on standard reward penalties and episode truncation for invalid moves, we explicitly masked illegal actions during reference sampling. Under these matched conditions, both models successfully learned the optimal policy within 1000 gradient steps with a minibatch size of 32. However, deeper analysis via Latent Error Trees (LE-Trees; see Appendix I for the visualization methodology and full figures) revealed a NashDreamer structural advantage. NashDreamer’s LE-Tree was an exact match with the true reachable dynamics (Figure 26), while DreamerV3 retained artificial stochasticity in its transitions (Figure 25) (exactly the aforementioned branch duplication). We attribute this structural precision to two factors: (1) the additional legal action prediction signal, and (2) the centralized unifier loss, which penalizes unpredictable variance in the stochastic component. Crucially, as established in our theoretical analysis, without the unifier loss acting as a structural “tiebreaker,” the perfectly isomorphic model and a model exhibiting benign branch duplication would incur the exact same dynamics loss.

P: 0.104 P: 0.895 d0

d1

A: 2, P: 0.991 A: 2, P: 0.998 d0a2 P: 1.000 d0a2d0 A: 1, P: 1.000 d0a2d0a1 P: 0.562 P: 0.093 P: 0.346 d0a2d0a1d0

d0a2d0a1d1

d0a2d0a1d2

d1a2 P: 1.000 d1a2d0 A: 1, P: 1.000 d1a2d0a1 P: 0.559 d1a2d0a1d0

P: 0.093

d1a2d0a1d1

P: 0.350 d1a2d0a1d2

Figure 25: DreamerV3 LE-Tree on PCM 3 (1000 gradient steps). Predictions are accurate along the optimal path, but the model fails to learn that the environment is deterministic.

N

Figure 26: NashDreamer LE-Tree on PCM 3 (1000 gradient steps). The learned model is an exact match of the true reachable dynamics within the 0.2 accuracy threshold.

REINFORCE vs. RNaD

Since REINFORCE has no convergence guarantees in two-player zero-sum IIGs and statically tuning it is not sound in the context of changing world model, we opted not to use it as our policy optimization algorithm. Nevertheless, we test in in Rock-Paper-Scissors (RPS) as a standalone algorithm, and in Goofspiel 3 descending wrapped in NashDreamer to validate our choice. N.1

REINFORCE in RPS

Since every decision node in a POSG 2 is effectively a matrix game, we tested model-free REINFORCE without neural network function approximation in the canonical matrix game Rockpaper-scissors (RPS). We evaluated a low entropy bonus of 3e − 4, characteristic of single-agent environments and a higher entropy bonus of 0.2, typical of multi-agent environments. We tested two distinct initializations: a uniform initialization, which initializes the algorithm in the equilibrium of the game, and a pure (Rock, Rock) initialization. The results are presented for 10 distinct seeds, with a 1-σ bar interval. The algorithm always diverged away from equilibrium, even with the higher entropy bonus, which only slowed the divergence (Figure 27). Under the deterministic strategy initialization, low entropy bonus causes the algorithm to remain trapped in that determinism (Figure 28). 38

0.6

eta=0.0003 eta=0.2

2.00

eta=0.0003 eta=0.2

1.75 1.50

0.4

NashConv

NashConv

0.5

0.3 0.2 0.1

1.25 1.00 0.75 0.50 0.25

0.0 0

2000

4000

6000

Iterations

8000

10000

0.00

Figure 27: REINFORCE in RPS with uniform initialization. Even at equilibrium, low entropy bonus causes divergence; high bonus slows but does not prevent it.

0

2000

4000

6000

Iterations

8000

10000

Figure 28: REINFORCE in RPS with pure initialization. Low entropy bonus is trapped in determinism.

NashDreamerRNaD NashDreamerREINFORCE Uniform Policy WM warm-up end

NashConv

100

10 1

0

25000 50000 75000 100000 125000 150000 175000 200000

Environment steps

Figure 29: NashConv on Goofspiel 3. RNaD converges to the pure NE; REINFORCE with η = 0.2 entropy bonus cannot.

N.2

Goofspiel 3

To evaluate REINFORCE as a component of the full NashDreamer architecture, we perform experiments in Goofspiel 3 (which has a pure equilibrium). We test a reimplementation of the DreamerV3 version of REINFORCE [14], with identical hyperparameters, except the entropy bonus, which we set to 0.2 to match the RNaD regularization strength. RNaD successfully converged to the pure NE, while REINFORCE, burdened by the entropy exploration bonus, was unable to collapse into the required pure strategies (Figure 29). Analysis of the LE-Trees (Figure 30 for RNaD; REINFORCE is contained in the supplementary material due to size) confirmed that REINFORCE spread probability mass across all actions rather than concentrating on the equilibrium.

O

Learning from partial observations

In our primary experiments (Section 5), we provide both NashDreamer and the standalone RNaD baseline with pre-aggregated infoset representation vectors. This design choice was made to isolate the primary experimental variable: the sample efficiency of the centralized world model versus an off-policy replay buffer and prevent confounding the difference by differing input representations. However, the MARSSM architecture is designed to handle aggregating sequences of raw, partial observations via the decentralized recurrent infoset model. To empirically validate this capability, we conducted an ablation on the Imperfect Information Goofspiel-5 environment. 39

Figure 30: NashDreamer with RNaD LE-Tree on Goofspiel 3 (1000 gradient steps). The model learns both the pure equilibrium and accurate predictions along the equilibrium path. In this experiment, we tested NashDreamer both with pre-aggregated infoset input and with only the observation in the current state (e.g., the current point card and win/loss/tie information). As demonstrated in Figure 31, the difference in performance is marginal, validating the capability of NashDreamer to aggregate per-player information and model the environment dynamics simultaneously. NashDreamer infoset NashDreamer observation Uniform Policy WM warm-up end

NashConv

100

10 1

0.0

0.2

0.4

0.6

Environment steps

0.8

1.0 1e7

Figure 31: Comparison of NashConv in Imperfect Information Goofspiel 5 when using pre-aggregated infoset representation versus latent infoset modelling from partial observations. The model demonstrates the capability to model accurate latent infosets.

P

NashConv Against Total Actor-Critic Steps

The NashConv curves of Section 5.1 are plotted against real environment interaction, because that is the quantity our sample-efficiency claim concerns. For completeness, we report here the same runs plotted against the total number of transitions consumed by the actor-critic, counting real, imagined and replayed transitions identically. Figure 32 gives this view for Imperfect Information Goofspiel-5, Leduc Poker and Phantom Tic-Tac-Toe. In every panel, the horizontal axis is logarithmic, evaluated checkpoints are marked on the curves, and the plotted range begins at the end of the world-model warm-up, before which no imagined data exists. The Leduc runs use the configuration that performed best empirically: the factored latent stochastic state of Appendix J with β dyn = 0.1 and β enc = 0.01. Two observations are worth drawing from these plots. First, the model-based runs consume roughly an order of magnitude more total transitions than model-free RNaD to reach a comparable NashConv. This is expected, and it is precisely why the main text reports real interaction: imagined transitions are cheap in the regime we target, so counting them equally with real ones measures a different quantity than the one we claim to improve. Second, the comparison that is meaningful on this axis 40

is against RNaD with replay, which was deliberately configured to consume more total transitions than NashDreamer generates by imagination (Appendix Q). In Goofspiel-5 and in Phantom Tic-TacToe alike, NashDreamer[RNaD] reaches a lower NashConv than the replay baseline at comparable total step counts, so the improvement is not attributable to step volume alone. In Leduc the NashDreamer[RNaD] curve additionally exhibits the late-training instability reported in Section 5.1 and localized in Appendix K. How to read these plots. Every run is evaluated at 30 checkpoints spaced 1,000 gradient steps apart, and each evaluated checkpoint is marked on its curve. Because the horizontal axis counts transitions consumed rather than gradient steps, an equal number of gradient steps maps to a different horizontal position for each method, so the curves neither begin nor end together. This is most visible for the replay-augmented baseline, whose first checkpoint lies well to the right of the others because it already consumes replayed transitions during the warm-up period; the model-based runs separate from their model-free counterparts in the same manner once imagination begins. At the first checkpoint, by contrast, NashDreamer[RNaD] and model-free RNaD coincide, as they must: up to the end of the warm-up the two consume exactly the same real transitions, and the same holds for the MMD pair. Goofspiel-5

Leduc

100

Phantom TTT

100

NashConv

100

10 1 10 1

106

107

106

NashDreamerRNaD RNaD

MMD NashDreamerMMD

Total steps seen by actor-critic

107

Total steps seen by actor-critic

108

RNaD with replay

10 1

106

107

108

Total steps seen by actor-critic

Uniform Policy

Figure 32: NashConv against the total number of transitions consumed by the actor-critic, counting real, imagined and replayed transitions identically, in Imperfect Information Goofspiel-5 (left), Leduc Poker (middle) and Phantom Tic-Tac-Toe (right). Markers indicate evaluated checkpoints. Bold curves are means over the 10 training seeds, faint curves the individual seeds. The Leduc runs use the factored latent stochastic state with β dyn = 0.1 and β enc = 0.01.

Q

Head-to-Head Evaluation

We also report statistics of head-to-head play on larger games, using the same checkpoints as reported in Section 5.2, which were obtained by playing 1M games per seed pairing, in each seat (so 20M games per ego-opponent pair in general). The results are averaged across the seats. Battleship 5 × 5. NashDreamer[RNaD] wins against every other method at all three checkpoints (Table 5), but its margin shrinks monotonically as interactions accumulate: against model-free RNaD it falls from 66.7% at 10K to 62.6% at 20K and 56.7% at 30K, and against RNaD with replay from 71.9% to 66.6% to 61.5%. NashDreamer[MMD] shows the same pattern against model-free MMD, 61.5% falling to 56.5%. This is the early-to-mid-training scope we claim throughout: the world model buys an advantage that the model-free baseline gradually closes given enough environment interaction. Between the two RNaD baselines, the one with replay is the weaker at every checkpoint, losing to plain RNaD in all three matrices, which is consistent with the variance argument of Appendix E.2. Stochastic Goofspiel (13 cards). Here head-to-head play does not separate a NashDreamer variant from its model-free counterpart at all (Table 6). NashDreamer[RNaD] and RNaD are at parity across all three checkpoints, the model-based side winning 50.3%, 49.9% and 49.7% of matches, and NashDreamer[MMD] and MMD likewise (49.0%, 48.6%, 48.9%). RNaD consistently beats MMD across the tested checkpoints, and RNaD with replay is sligthly weaker than without it. In Goofspiel-13, head-to-head play attributes no benefit at all to the world model: NashDreamer[RNaD] and RNaD are indistinguishable in direct play, and each fares the same against the rest of the pool. Under best-response search the two are far apart, 0.168 against 0.356 at 30K. 41

Win rate (%) vs. Method

MMD

ND[MMD] ND[RNaD]

RNaD

RNaD w/ replay

41.4 ± 2.7 52.7 ± 2.2 66.7 ± 2.1 – 43.1 ± 1.7

48.3 ± 2.3 58.8 ± 1.1 71.9 ± 1.6 56.9 ± 1.7 –

42.0 ± 2.0 51.0 ± 2.2 62.6 ± 1.2 – 45.0 ± 1.2

45.1 ± 1.8 53.4 ± 1.4 66.6 ± 0.9 55.0 ± 1.2 –

39.3 ± 1.9 46.3 ± 2.5 56.7 ± 2.3 – 45.1 ± 1.9

43.2 ± 1.7 49.9 ± 1.9 61.5 ± 0.8 54.9 ± 1.9 –

10K gradient steps MMD ND[MMD] ND[RNaD] RNaD RNaD w/ replay

– 61.5 ± 2.6 74.4 ± 2.5 58.6 ± 2.7 51.7 ± 2.3

38.5 ± 2.6 – 64.3 ± 2.2 47.3 ± 2.2 41.2 ± 1.1

25.6 ± 2.5 35.7 ± 2.2 – 33.3 ± 2.1 28.1 ± 1.6

20K gradient steps MMD ND[MMD] ND[RNaD] RNaD RNaD w/ replay

– 59.3 ± 1.8 71.0 ± 1.6 58.0 ± 2.0 54.9 ± 1.8

40.7 ± 1.8 – 62.7 ± 1.5 49.0 ± 2.2 46.6 ± 1.4

29.0 ± 1.6 37.3 ± 1.5 – 37.4 ± 1.2 33.4 ± 0.9

30K gradient steps MMD ND[MMD] ND[RNaD] RNaD RNaD w/ replay

– 56.5 ± 2.0 67.8 ± 1.7 60.7 ± 1.9 56.8 ± 1.7

43.5 ± 2.0 – 61.5 ± 1.8 53.7 ± 2.5 50.1 ± 1.9

32.2 ± 1.7 38.5 ± 1.8 – 43.3 ± 2.3 38.5 ± 0.8

Table 5: Head-to-head play of 1M games for each seed pairing and seat in Battleship 5 × 5: win rate (%) of the row method against the column method, mean ±2σ over training seeds. ND denotes NashDreamer with the given actor-critic inside imagination.

In Battleship the disagreement runs the other way. NashDreamer[RNaD] wins the cross-play table outright, yet at 30K it is the second most exploitable of the five methods (1.169), whereas MMD, which loses to every other method in direct play, is harder to exploit (1.022), and NashDreamer[MMD] is the hardest of all (0.817). Neither observation contradicts the results of Section 5.2, since head-to-head measures performance across a particular opponent pool, rather than how hard the policy is to exploit overall. We therefore report exact NashConv wherever it is tractable (Section 5.1), budgeted approximate exploitability where it is not (Section 5.2), and treat the tables above as evidence about pool performance alone.

42

Win rate (%) vs. Method

MMD

ND[MMD] ND[RNaD]

RNaD

RNaD w/ replay

36.6 ± 1.9 35.2 ± 1.5 50.3 ± 1.6 – 43.6 ± 2.2

42.5 ± 1.6 41.3 ± 1.7 56.4 ± 1.0 56.4 ± 2.2 –

37.2 ± 2.4 36.0 ± 1.9 49.9 ± 1.4 – 44.3 ± 2.4

41.6 ± 1.6 40.4 ± 2.4 54.9 ± 1.3 55.7 ± 2.4 –

37.1 ± 1.6 36.1 ± 1.9 49.7 ± 1.3 – 44.2 ± 1.4

41.5 ± 1.5 40.6 ± 1.7 54.8 ± 1.4 55.8 ± 1.4 –

10K gradient steps MMD ND[MMD] ND[RNaD] RNaD RNaD w/ replay

– 49.0 ± 1.8 62.6 ± 1.8 63.4 ± 1.9 57.5 ± 1.6

51.0 ± 1.8 – 63.5 ± 2.0 64.8 ± 1.5 58.7 ± 1.7

37.4 ± 1.8 36.5 ± 2.0 – 49.7 ± 1.6 43.6 ± 1.0

20K gradient steps MMD ND[MMD] ND[RNaD] RNaD RNaD w/ replay

– 48.6 ± 3.9 61.6 ± 1.7 62.8 ± 2.4 58.4 ± 1.6

51.4 ± 3.9 – 62.6 ± 2.5 64.0 ± 1.9 59.6 ± 2.4

38.4 ± 1.7 37.4 ± 2.5 – 50.1 ± 1.4 45.1 ± 1.3

30K gradient steps MMD ND[MMD] ND[RNaD] RNaD RNaD w/ replay

– 48.9 ± 1.4 62.1 ± 2.3 62.9 ± 1.6 58.5 ± 1.5

51.1 ± 1.4 – 62.9 ± 1.8 63.9 ± 1.9 59.4 ± 1.7

37.9 ± 2.3 37.1 ± 1.8 – 50.3 ± 1.3 45.2 ± 1.4

Table 6: Head-to-head play of 1M games for each seed pairing and seat in stochastic 13-card Imperfect Information Goofspiel: win rate (%) of the row method against the column method, mean ±2σ over training seeds. ND denotes NashDreamer with the given actor-critic inside imagination.

43

R

World-Model Warm-Up Ablation

The warm-up is a stabilizing mechanism rather than a theoretical requirement (Appendix C). It supplements the DreamerV3 practice of zero-initializing the reward and critic heads (Appendix G.3), compensating for the additional non-stationarity that the opponent introduces in the multi-agent setting. We sweep its length over {0, 250, 500, 1000, 2000} gradient steps in Leduc Poker, 1000 being the value used in every other Leduc experiment, running both actor-critics inside imagination at each setting with model-free RNaD and MMD for reference (Figure 33). Runs are 10,000 gradient steps, since the parameter acts only early in training; the remaining configuration is unchanged. NashDreamer[RNaD] is invariant to the setting: no value, zero included, separates from the others by more than the spread across training seeds. Nor does a longer warm-up improve the asymptote, as expected given that the Leduc model remains badly learned at every setting (Appendix K). The warm-up is therefore a stability parameter, not a lever on final policy quality. For MMD, warm-ups of 0 and 250 steps accelerate early convergence relative both to the longer settings and to model-free MMD, while 500 and above are indistinguishable from the model-free baseline. This supports the explanation in Section 5.1 for NashDreamer[MMD]’s lack of gain on the small games: MMD converges inside the warm-up window, leaving imagination nothing to accelerate.

100

NashConv

NashDreamerMMD_0 NashDreamerMMD_250 NashDreamerMMD_500 NashDreamerMMD_1000 NashDreamerMMD_2000 NashDreamerRNaD_0 NashDreamerRNaD_250 NashDreamerRNaD_500 NashDreamerRNaD_1000 NashDreamerRNaD_2000 RNaD MMD Uniform Policy

10 1

0

1

2

3

4

Environment steps

5

6 1e6

Figure 33: World-model warm-up sweep in Leduc Poker over {0, 250, 500, 1000, 2000} gradient steps, applied to both actor-critics inside imagination, with model-free RNaD and MMD for reference. Legend entries are labelled by the warm-up length in gradient steps. Bold curves are means over training seeds, faint curves the individual seeds. NashDreamer[RNaD] is largely unaffected by the setting, whereas NashDreamer[MMD] separates from model-free MMD in the early phase only at the two shortest warm-ups.

44

Related documents

• Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation#151738
• When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient#141473
• Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies#1143008
• Optimization methods for reinforcement learning: theory and applications#211648
• Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning#478231