StratFormer: Adaptive Opponent Modeling and Exploitation in Imperfect-Information Games
arXiv:2604.25796v1 [cs.AI] 28 Apr 2026
Andy Caen[0009-0009-7799-3913] , Mark H.M. Winands[0000-0002-0125-0824] , and Dennis J.N.J. Soemers[0000-0003-3241-8957] Department of Advanced Computing Sciences, Maastricht University, Maastricht, The Netherlands [email protected] {m.winands, dennis.soemers}@maastrichtuniversity.nl
Abstract. We present StratFormer, a transformer-based meta-agent that learns to simultaneously model and exploit opponents in imperfectinformation games through a two-phase curriculum. The first phase trains an opponent modeling head to identify behavioral patterns from action histories while the agent plays a game-theoretic optimal (GTO) policy. The second phase progressively shifts the policy toward bestresponse (BR) exploitation, guided by a per-opponent regularization schedule tied to exploitability. Our architecture introduces dual-turn tokens—feature vectors constructed at both agent and opponent decision points—coupled with bucket-rate features that encode opponent tendencies across five strategic contexts. On Leduc Hold’em, a small poker variant with six cards and two betting rounds, we test against six opponent archetypes at two strength levels each, with exploitability ranging from 0.15 to 1.26 Big Blinds (BB) per hand. StratFormer achieves an average exploitation gain of +0.106 BB per hand over GTO, with peak gains of +0.821 against highly exploitable opponents, while maintaining near-equilibrium safety. Keywords: Opponent modeling · Imperfect-information games · Transformer · Curriculum learning · Poker · Exploitation
1
Introduction
A fundamental tension in imperfect-information games lies between safety and exploitation. Game-theoretic optimal (GTO) strategies, grounded in Nash equilibrium [20], guarantee worst-case performance but leave value on the table against suboptimal opponents. Best-response (BR) strategies maximally exploit a known opponent but are themselves highly exploitable if the opponent model is wrong [8]. Human experts navigate this tradeoff by gradually modeling opponents during play and adjusting their strategy in real time [19]—a capability that has proven difficult to replicate in learned agents. Prior work has approached opponent modeling through search-based methods [12], explicit Bayesian inference [19], policy reconstruction networks [9], or
2
A. Caen et al.
theory-of-mind architectures [18]. However, these approaches typically treat opponent modeling and strategic adaptation as separate modules, creating a gap between understanding the opponent and acting on that understanding. Reinforcement learning methods such as LOLA [7] and neural fictitious self-play [10] learn adaptive strategies but do not provide interpretable opponent representations. Recent advances in sequence modeling have shown that transformer architectures can capture long-range dependencies through self-attention mechanisms [21]. Unlike fixed-state or hand-engineered opponent models, attention allows the model to dynamically weight past observations when forming predictions. In imperfect-information games, where opponent behavior must be inferred from partial and temporally distributed signals, this ability to selectively attend to relevant past actions provides a natural foundation for adaptive opponent modeling and strategic adjustment. We propose StratFormer, a transformer-based meta-agent that unifies opponent modeling and exploitation in a single architecture through a two-phase curriculum. In the first modeling phase, the agent plays a GTO policy while training an opponent modeling head to predict opponent actions from behavioral features. In the second exploitation phase, the policy head learns to deviate from GTO toward BR play, with the degree of deviation controlled by opponent exploitability: highly exploitable opponents receive aggressive exploitation while near-GTO opponents retain a higher safety tether. The architecture processes dual-turn tokens at every decision point in the game—both agent and opponent turns—with causal masking ensuring each token only attends to past information. Five bucket-rate features encode running statistics of opponent behavior across strategic contexts, enabling the transformer to condition its policy on a causally-correct opponent model. While our evaluation focuses on Leduc Hold’em as a tractable testbed where exact best-responses and exploitabilities can be computed, the architectural and training contributions are domain-general: dual-turn tokens, the two-phase curriculum, and the exploitability-tied regularization λ(ε) apply to any sequential imperfect-information game with observable opponent actions and a tractable equilibrium baseline. We evaluate on Leduc Hold’em [19] against six opponent archetypes at two strength levels each, spanning a wide range of exploitability. StratFormer achieves positive exploitation gains against the majority of opponents while maintaining near-equilibrium safety against GTO play. We present ablation studies identifying the key design decisions that enable this balance.
2
Background
Imperfect-information games require players to act without full knowledge of the underlying game state, typically because some variables (such as an opponent’s private cards in poker) are hidden. In two-player zero-sum settings, a Nash equilibrium guarantees worst-case performance: no opponent can obtain positive expected value by deviating. GTO play therefore provides safety, but it is not
StratFormer: Adaptive Opponent Modeling and Exploitation
3
tailored to a specific opponent and may leave value unexploited against systematically suboptimal behavior. In contrast, a best response maximizes expected payoff against a fixed opponent strategy but can itself become highly exploitable if the opponent model is incorrect. To quantify this safety–exploitation tradeoff, we measure exploitability formally. Let vi (σi , σ−i ) denote the expected payoff to player i under strategy profile (σi , σ−i ), and let vi∗ denote the equilibrium value. The exploitability of a strategy σi is defined as ∗ ε(σi ) = v−i BR−i (σi ), σi − v−i , which measures how much additional payoff the opponent can obtain, relative to equilibrium, by optimally countering σi . OpenSpiel [14] reports exploitability via NashConv, the sum of both players’ exploitabilities under the symmetric profile (σ, σ). In two-player zero-sum games, NashConv = 2 ε(σ), so we report per-player exploitability as NashConv/2. Leduc Hold’em [19] is a two-player poker game with a six-card deck consisting of two copies each of J, Q, and K. Each player antes 1 chip and receives one private card. Two betting rounds follow: preflop (bet size 2) and postflop after a public card is revealed (bet size 4). Each round permits up to two bets (an initial bet and one raise), with three legal actions at each decision point: fold, call (or check), and raise (or bet). The game contains 936 information sets and admits an analytically computable Nash equilibrium, making it a standard testbed where exact best responses and exploitability can be computed. We compute a tabular Nash equilibrium using counterfactual regret minimization (CFR) [20] as implemented in OpenSpiel [14], running for 106 iterations until exploitability falls below 10−6 , which is well below the deviations of even the closest-to-equilibrium evaluation opponent (ε = 0.15). The resulting strategy is a lookup table mapping each information state to a probability distribution over legal actions. This tabular GTO serves three roles in our pipeline: as a pretraining target, as a baseline agent for paired-seed evaluation, and as the foundation for constructing opponent archetypes.
3
Related Work
The problem of adapting to opponents in games has a long history. Early work by Iida et al. [12] and Donkers [5] introduced and built on opponent-model search, which integrates predictions of opponent behavior directly into game-tree search. In the poker domain, Southey et al. [19] proposed Bayesian opponent modeling using Dirichlet priors over opponent strategies, and Bard et al. [1] extended this with online implicit agent modeling. He et al. [9] trained neural networks to predict opponent actions and used these predictions to guide MCTS rollouts, while Ganzfried and Sandholm [8] provided theoretical bounds on how much an agent can safely deviate from equilibrium when exploiting.
4
A. Caen et al.
In parallel, equilibrium-finding methods have advanced rapidly. CFR [20] and its deep variants [3,17] achieve superhuman play by converging to Nash equilibrium, but these agents play a fixed strategy and do not adapt to specific opponents. A common thread across both lines of work is that opponent modeling and strategic adaptation are treated as separate problems: one module estimates what the opponent is doing, and another decides how to respond. On the meta-learning side, MAML [6] trains agents that adapt at deployment, PSRO [13] and LOLA [7] explore adaptation in multi-agent settings, and Decision Transformer [4] cast offline RL as sequence modeling. ToMnet [18] learns a theory of mind but uses it separately from its own policy. Our approach differs in that adaptation is implicit in the transformer’s attention over behavioral history, requiring no gradient steps or separate inference modules at test time.
4
Architecture
StratFormer is built on a causal transformer encoder with two task-specific output heads (Figure 1). We create a 25-dimensional feature vector at every decision point—both agent and opponent turns—approximately doubling training tokens per episode and ensuring causal alignment: each token’s features reflect the game state immediately before the labeled action. Importantly, both agentturn and opponent-turn tokens are constructed exclusively from the agent’s observable information (its own cards, the public card, pot size, and the history of actions). Opponent-turn tokens simply capture this observable state at the moment the opponent acts, rather than revealing any hidden information. A binary feature distinguishes the two token types.
Agent turns only
Opp turns only
Policy Head πθ (a|ht )
Opp Modeling qθ (a|ht )
Causal Transformer Encoder (4 layers, d=512, 8 heads) Agent Token
Opp Token
25-dim
25-dim
Agent Token
Opp Token
···
Fig. 1. StratFormer architecture. Dual-turn tokens from agent and opponent decision points are processed by a shared causal transformer. The policy head outputs the agent’s action distribution; the opponent modeling head is trained via supervised learning to predict the opponent action.
StratFormer: Adaptive Opponent Modeling and Exploitation
4.1
5
Feature Engineering
Each token encodes 25 features organized into three groups. Nine base features capture card information (agent rank, public rank, normalized), pot geometry (agent investment, total pot, normalized), action context (agent’s last action, last round action), position flag, round flag, and observed opponent card (populated after showdown, −0.5 sentinel otherwise). A single turn-type indicator distinguishes agent from opponent decision points. The remaining fifteen dimensions are running action frequencies of the opponent across five strategic contexts, with three values (fold/call/raise rate) per context: global rates across all decisions, round one and round two rates capturing per-round tendencies, facing raise rates for situations where the opponent must call or fold to aggression, and not facing raise rates for voluntary betting positions. These bucket rates are maintained by a running opponent statistics tracker that updates incrementally across hands. At test time, the bucket rates accumulate from the first hand, providing the transformer with increasingly precise opponent statistics to condition its policy on. 4.2
Model Architecture and Loss Functions
The shared encoder [21] consists of 4 layers with dmodel = 512, 8 attention heads, feedforward dimension 512, and dropout 0.15. Causal masking restricts each token to attend only to preceding positions. Input tokens are linearly projected to 512 dimensions, and sinusoidal positional encodings support sequences up to length 3000 (approximately 375 hands of context). Two task-specific output heads branch from the shared encoder. The policy head is a linear projection R512 → R3 (fold, call, raise), applied only to agentturn representations. The opponent modeling head is a two-layer multi-layer perceptron (MLP) (512 → 512 → 3) with GELU activation [11], applied only to opponent-turn representations. Separating the heads reduces gradient interference between the exploitation objective (adjusting the agent’s policy) and the opponent-modeling objective (predicting opponent actions). The policy head is trained using cross-entropy against a mixture of two targets: a GTO imitation target and a BR target. A parameter λ ∈ [0, 1] controls the balance between these targets, where λ = 1 corresponds to pure GTO imitation and λ = 0 to pure best-response. The policy loss is: N
h 1 X [λi · CE(πGTO , πθ ) + (1 − λi ) · CE(1aBR , πθ )] , Lpolicy = Nh i=1
(1)
P where CE(p, q) = − a p(a) log q(a) denotes cross-entropy with legal-action masking. Both terms use light label smoothing (ϵs = 0.01), preventing target probabilities from collapsing to zero. The GTO term uses cross-entropy rather than Kullback–Leibler (KL) divergence: cross-entropy only penalizes insufficient coverage of GTO-supported actions, leaving the model free to allocate additional probability mass to exploitative moves, while KL(πθ ∥ πGTO ) would penalize any
6
A. Caen et al.
mass on non-GTO actions, suppressing the very deviations the BR loss is trying to learn (we ablate this in Section 6). The opponent modeling head is trained via supervised cross-entropy to predict the opponent’s chosen action: N
Lopp =
o 1 X CE(1a(j) , qθ ). opp No j=1
The total loss is L = Lpolicy + α Lopp , where α controls the relative importance of opponent modeling. The scheduling of λ and α during curriculum training is described in Section 5. The encoder is initialized from a GTO imitation model trained for 15,000 epochs via cross-entropy against the tabular equilibrium, achieving 96% action agreement. This pretraining stage uses only the 9 base features (without bucket rates), providing a strong equilibrium baseline so curriculum training focuses on learning controlled deviations. The input projection is expanded from 9 to 25 dimensions by zero-padding new columns, preserving pretrained weights while initializing bucket-rate projections to zero.
5
Training Methodology
5.1
Opponent Archetypes
We construct opponents by systematically perturbing the tabular GTO strategy. At each information set where the equilibrium mixes between actions (i.e., assigns non-zero probability to multiple actions), we convert the action probabilities to log-odds space, apply a directional bias, and convert back. Concretely, for a modifier function f that maps action indices to bias magnitudes, the perturbed log-odds are ℓ′a = ℓa + w · f (a) + ηa , where w ∈ [0, 1] controls deviation strength and ηa ∼ N (0, σ 2 ) adds independent Gaussian noise. Working in log-odds space ensures that the resulting probabilities remain valid and that small perturbations produce smooth, realistic deviations rather than degenerate strategies. Information states where GTO plays a pure strategy (probability 1 on a single action) are left unmodified, since there is no mixing to perturb. Six modifier functions define characteristic behavioral directions (Table 1). For example, the over-caller modifier increases the log-odds of the call action and decreases fold, while the maniac modifier boosts raise at the expense of call and fold. Each archetype is instantiated at two strength levels (“mid” w = 0.35, “high” w = 0.70), selected to produce a broad and roughly linear distribution of exploitability values across opponent archetypes. Exploitability values range from ε = 0.15 (passive_mid, nearly indistinguishable from GTO) to ε = 1.26 (maniac_high, extremely exploitable).
StratFormer: Adaptive Opponent Modeling and Exploitation
7
Table 1. Evaluation opponent suite (mid: w=0.35, high: w=0.70). Exploitability via exact best-response in OpenSpiel [14]. All held out from training. Archetype
Level
Over-caller Nit Maniac Passive Loose aggressive Tight passive
mid / high mid / high mid / high mid / high mid / high mid / high
5.2
ε 0.23 / 0.53 0.34 / 0.48 0.46 / 1.26 0.15 / 0.33 0.25 / 0.60 0.33 / 0.53
Behavioral bias Excess calling, never folds Excess folding, plays tight Excess raising, hyper-aggressive Never raises, passive play Wide range, frequent raises Narrow range, check-calls
Two-Phase Curriculum
Training uses a diverse population of 50 opponents generated with the log-odds perturbation method. For training, the deviation weight w is sampled uniformly from the full range [0, 1], producing a continuous spectrum of opponent strengths rather than restricting to the two fixed mid and high levels used for evaluation. After generating a large candidate set, we compute exploitability for each opponent, sort them by ε, and select evenly spaced representatives. This procedure ensures approximately uniform coverage across the exploitability range from ε = 0.15 to 1.26. The evaluation opponents (Table 1) are generated with distinct random seeds and held out entirely from training. Algorithm 1 summarizes the two-phase curriculum. Each epoch generates a buffer of 250 hands played sequentially against the same opponent, so that bucket-rate features accumulate across hands within the buffer. The modeling phase trains the opponent modeling head while the agent plays GTO (λ = 1.0), skipping best-response computation for approximately 10× faster data generation. The opponent modeling loss receives heavy weighting (α = 2.0), and the agent plays a uniformly random legal action with probability ϵ = 0.15 to ensure the modeling head observes opponent responses to nonequilibrium actions. Ten percent of buffers use GTO as the opponent. The phase transitions when opponent modeling cross-entropy drops below 0.65 for three consecutive checks, or after a maximum of 3,000 epochs. Once modeling converges, the agent transitions to exploitation training. For each opponent with exploitability ε, the GTO regularization weight is computed as: ε λ(ε) = λmax · max 0, 1 − (2) εmax with λmax = 0.35 and εmax = 0.40. This gives λ = 0 (pure best-response) for opponents with ε ≥ 0.40 and a linear taper to λ = 0.35 for GTO opponents. Hyperparameters (α, λmax , εmax ) were chosen via preliminary sweeps on held-out training opponents to balance exploitation gain and stability. In Phase 2, the agent always plays the model’s own policy during data generation. Additionally, 10% of training buffers use GTO as the opponent. Since
8
A. Caen et al.
Algorithm 1 StratFormer Two-Phase Curriculum Training Input: Tabular GTO πGTO , opponent population O, pretrained model θ0 1: θ ← θ0 // from GTO imitation pretraining 2: // Phase 1: Modeling 3: for epoch = 1 to max_phase1_epochs do 4: Sample opponent o ∼ O; generate buffer of N hands with agent = πGTO (ϵgreedy, ϵ=0.15) 5: Record dual-turn tokens with bucket-rate features from opponent tracker 6: Update θ on L = α · Lopp (α=2.0) 7: if Opp CE < 0.65 for 3 consecutive checks then 8: break // modeling converged 9: end if 10: end for 11: // Phase 2: Exploitation 12: for epoch = 1 to remaining_epochs do 13: Sample opponent o with exploitability ε 14: Compute λ = λmax · max(0, 1 − ε/εmax ) 15: Compute BR action aBR via tabular best-response to o 16: Generate buffer of N hands: agent = πθ ; opponent = o (prob 0.9) or πGTO (prob 0.1) 17: Lpol = λ · CE(πGTO , πθ ) + (1−λ) · CE(1aBR , πθ ) 18: Update θ on L = Lpol + α · Lopp (α=0.5) 19: end for Output: Trained model θ
BR(GTO) = GTO, the BR loss for these buffers provides implicit equilibrium regularization. Several design choices ensure causal correctness. Training tokens store independent copies of feature vectors that are never backfilled with the opponent’s card after showdown. A separate inference context (used for on-policy action selection) receives showdown information normally, preventing causal leakage into the training data. These safeguards ensure that the model’s adaptation arises strictly from information available at decision time.
6
Experiments
We evaluate whether StratFormer successfully balances opponent-specific exploitation with near-equilibrium safety. Our experiments measure (i) exploitation gain relative to a tabular GTO baseline and (ii) performance degradation when facing GTO itself. We further analyze adaptation dynamics and ablate key design choices. 6.1
Setup
We evaluate against the six opponent archetypes described in Section 5.1, each at two strength levels (12 opponents total), plus GTO itself; all evaluation opponents are held out from the training population. We report expected value (EV)
StratFormer: Adaptive Opponent Modeling and Exploitation
9
Table 2. Held-out evaluation (3,000 hands × 3 trials, paired seeds). Gain = Model − GTO on identical deals. Bold: 95% CI excludes zero (ḡ ± 1.96 · SE across trial means). Opponent (ε)
Model EV
GTO EV
Gain
±95%
GTO (0.00)
−0.086
−0.037
−0.050
—
passive_mid (0.15) over_caller_mid (0.23) loose_agg_mid (0.25) tight_passive_mid (0.33) passive_high (0.33) nit_mid (0.34) maniac_mid (0.46) nit_high (0.48) over_caller_high (0.53) tight_passive_high (0.53) loose_agg_high (0.60) maniac_high (1.26)
−0.016 −0.165 +0.008 +0.085 +0.117 −0.050 +0.131 +0.082 +0.019 −0.000 +0.220 +0.839
+0.041 +0.006 +0.028 +0.014 −0.022 −0.030 −0.042 −0.002 +0.006 −0.047 +0.027 +0.018
−0.058 −0.171 −0.020 +0.071 +0.139 −0.021 +0.173 +0.084 +0.013 +0.046 +0.193 +0.821
0.050 0.025 0.050 0.026 0.011 0.025 0.031 0.045 0.102 0.038 0.030 0.074
Average (excl. GTO)
+0.106
per hand in units of Big Blinds (BB) throughout. To reduce evaluation variance, we use paired-seed evaluation: a fixed sequence of per-hand random seeds controls card deals, so both the model and GTO baseline face identical deals and the measured gain reflects purely strategic differences—analogous to common random numbers in Monte Carlo simulation [15]. We train with AdamW [16] (learning rate 3 × 10−5 , weight decay 0.05, cosine schedule), gradient accumulation of 8 in Phase 1 and 16 in Phase 2, for 20,000 epochs (1,468 gradient updates total) on a single Apple M-series GPU in approximately 72 hours. Final evaluation uses 3,000 hands × 3 independent trials per opponent. 6.2
Results
Table 2 presents held-out evaluation results using 3,000 hands × 3 independent trials per opponent with paired-seed variance reduction. StratFormer achieves positive exploitation gain against 8 of 12 opponents, with an average gain of +0.106 BB per hand. Performance against GTO itself averages −0.050, indicating that the model maintains approximate equilibrium safety despite significant exploitative deviations against weak opponents. The strongest exploitation occurs against highly exploitable opponents: +0.821 against maniac_high (ε = 1.26), +0.193 against loose_aggressive_high (ε = 0.60), and +0.173 against maniac_mid (ε = 0.46). Across opponents, six of twelve gains are statistically significant at 95% confidence, with the largest effects concentrated among the most exploitable archetypes. Four archetypes show negative or near-zero gains: passive_mid (−0.058, near-GTO with ε = 0.15 and little value to exploit), loose_agg_mid (−0.020) and nit_mid (−0.021, both
10
A. Caen et al.
Table 3. Ablation study. Full model uses held-out evaluation (Table 2); ablated variants use training-time evaluation (1,500 hands, single trial). Relative comparisons remain valid. Configuration StratFormer (full) No phasing (λ=0) No phasing (λ=0.5) Single-turn tokens KL instead of CE Low opp wt. (α=0.2) High λmax =0.70
Avg Gain
GTO EV
+0.106 +0.102 −0.042 +0.051 −0.004 +0.061 +0.009
−0.050 −0.101 +0.010 −0.085 −0.008 −0.283 −0.166
Note Two-phase curriculum Pure BR, GTO collapses λ too high, no exploit Agent-only tokens KL suppresses exploitation Lost modeling, unstable Excessive GTO pull
within statistical noise), and over_caller_mid (−0.171). The over_caller_mid case is the most informative failure mode: the model loses substantially relative to GTO, suggesting it over-bluffs against opponents who rarely fold. The modeling phase transitions at epoch 3,499 when opponent modeling cross-entropy drops below 0.65 (from an initial ln 3 ≈ 1.099). After transition, BR cross-entropy decreases from 0.70 to 0.42, indicating the policy head is learning exploitative actions, while GTO cross-entropy rises from 0.40 to 0.79, reflecting the expected drift from equilibrium. The opponent modeling cross-entropy continues improving, stabilizing around 0.59 by training end. The GTO cross-entropy climbs to 0.79 by training end, yet the model achieves approximately −0.05 against GTO in held-out evaluation—a modest loss rather than a collapse, suggesting that near-equilibrium behavior partially emerges through the best-response pathway (BR(GTO) = GTO) rather than explicit imitation alone. We compare StratFormer against several ablated configurations, evaluated at the same total training budget (Table 3). Without separate phases, the λ=0 variant achieves moderate exploitation (+0.111) but sacrifices GTO safety (−0.101); the fixed λ=0.5 variant retains GTO performance but achieves negative average gain. Switching the GTO loss to KL(πθ ∥ πGTO ) eliminates all exploitation entirely. Reducing α from 0.5 to 0.2 in Phase 2 causes exploitation to drop to +0.061 while GTO performance collapses to −0.283, confirming that sustained opponent-modeling gradients are necessary. Finally, with λmax = 0.70 and εmax = 0.70, the average lambda rises to 0.33, producing near-zero exploitation.
7
Conclusion
We presented StratFormer, a transformer-based meta-agent that learns adaptive opponent exploitation through a two-phase curriculum. On Leduc Hold’em, it achieves +0.106 BB per hand average gain over a tabular GTO baseline, with peak gains of +0.821 against the most exploitable opponent. Against GTO itself,
StratFormer: Adaptive Opponent Modeling and Exploitation
11
the model incurs a −0.05 BB per hand difference, which is small relative to variance at this scale and does not indicate systematic collapse. Overall, the agent demonstrates that substantial opponent-specific gains can be achieved while retaining near-equilibrium behavior. The main contributors to this performance are: (i) curriculum phasing that stabilizes opponent modeling before introducing exploitative gradients, (ii) dualturn tokens that align supervision precisely to decision points, and (iii) a mixed objective that interpolates between GTO imitation and opponent-specific bestresponse targets via λ(ε). Remaining failure modes highlight the limits of the current objective. Performance degradation against near-GTO opponents and losses against call-heavy opponents suggest that accurate opponent modeling alone is insufficient. The policy must learn how to translate opponent models into effective counterstrategies, while accounting for how exploitation incentives change with opponent tendencies and call/fold behavior. Future Work Scaling beyond Leduc requires approximate best-response targets or searchbased targets in larger games such as Texas Hold’em, where exact tabular BR is infeasible. Richer abstractions and learned value models will likely be necessary to preserve exploitative precision in high-dimensional settings. Two components in the current pipeline are poker-specific: the bucket-rate features (fold/call/raise frequencies across betting contexts) and the tabular GTO oracle. Adapting StratFormer to other domains thus requires reengineering the per-game behavioral features and substituting an approximate equilibrium baseline (e.g., Deep CFR [2]) for the tabular Nash. The sharedencoder architecture, dual-turn tokenization, two-phase curriculum, and λ(ε) schedule themselves can be reused across environments. A promising alternative to the current mixed GTO/BR objective is a more direct reinforcement learning formulation. Rather than interpolating between two supervised targets—which can produce partially conflicting gradient signals— future work could optimize expected reward directly while regularizing by exploitability or distance-to-equilibrium constraints. Such an approach would allow the agent to discover exploitative deviations through reward signals, while maintaining safety through explicit penalty terms or trust-region style bounds. Additional directions include integrating formal deviation guarantees (e.g., safe exploitation bounds [8]), adapting to non-stationary opponents who shift strategies mid-match, and evaluating against human players rather than synthetic archetypes. Together, these extensions would test whether the learned opponent representations generalize beyond controlled experimental settings and support reliable real-time adaptation in more complex environments.
12
A. Caen et al.
References 1. Bard, N., Johanson, M., Burch, N., Bowling, M.: Online implicit agent modelling. In: AAMAS, pp. 255–262 (2013) 2. Brown, N., Lerer, A., Gross, S., Sandholm, T.: Deep counterfactual regret minimization. In: ICML, pp. 793–802 (2019) 3. Brown, N., Sandholm, T.: Superhuman AI for multiplayer poker. Science 365(6456), 885–890 (2019) 4. Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., Mordatch, I.: Decision Transformer: Reinforcement learning via sequence modeling. In: NeurIPS, pp. 15084–15097 (2021) 5. H. H. L. M. Donkers: NOSCE HOSTEM: Searching with Opponent Models. PhD thesis, Department of Computer Science, Universiteit Maastricht, Maastricht, the Netherlands, 2003. 6. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: ICML, pp. 1126–1135 (2017) 7. Foerster, J., Chen, R.Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., Mordatch, I.: Learning with opponent-learning awareness. In: AAMAS, pp. 122–130 (2018) 8. Ganzfried, S., Sandholm, T.: Safe opponent exploitation. ACM Trans. Econ. Comput. 3(2), 1–28 (2015) 9. He, H., Boyd-Graber, J., Kwok, K., Daumé III, H.: Opponent modeling in deep reinforcement learning. In: ICML, pp. 1804–1813 (2016) 10. Heinrich, J., Silver, D.: Deep reinforcement learning from self-play in imperfectinformation games. arXiv:1603.01121 (2016) 11. Hendrycks, D., Gimpel, K.: Gaussian Error Linear Units (GELUs). arXiv:1606.08415 (2016) 12. Iida, H., Uiterwijk, J., van den Herik, H., Herschberg, B.: Potential applications of opponent-model search. Part 1: The domain of applicability. ICCA Journal 16(4), 201–208 (1993) 13. Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Pérolat, J., Silver, D., Graepel, T.: A unified game-theoretic approach to multiagent reinforcement learning. In: NeurIPS, pp. 4190–4203 (2017) 14. Lanctot, M., et al.: OpenSpiel: A framework for reinforcement learning in games. arXiv:1908.09453 (2019) 15. Law, A.M.: Simulation Modeling and Analysis, 5th edn. McGraw-Hill (2015) 16. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019) 17. Moravčík, M., et al.: DeepStack: Expert-level artificial intelligence in heads-up no-limit poker. Science 356(6337), 508–513 (2017) 18. Rabinowitz, N.C., Perbet, F., Song, H.F., Zhang, C., Eslami, S.M.A., Botvinick, M.: Machine theory of mind. In: ICML, pp. 4218–4227 (2018) 19. Southey, F., Bowling, M., Larson, B., Piccione, C., Burch, N., Billings, D., Rayner, C.: Bayes’ bluff: Opponent modelling in poker. In: UAI, pp. 550–558 (2005) 20. Zinkevich, M., Johanson, M., Bowling, M., Piccione, C.: Regret minimization in games with incomplete information. In: NeurIPS, pp. 1729–1736 (2007) 21. Vaswani, A., et al.: Attention is all you need. In: NeurIPS, pp. 5998–6008 (2017)