TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning
Chaofan Pan1 Lingfei Ren1 Xiangkun Wang1
arXiv:2607.21111v1 [cs.LG] 23 Jul 2026
1
Xiangyu Jiang1 Yanhua Li1 Xuemei Cao1 Hao Yu1 Wei Wei2 Xin Yang1∗
Southwestern University of Finance and Economics 2 Shanxi University [email protected] [email protected] [email protected]
Abstract Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult because a lower membership score can reflect trajectory removal, residual memorization visible to another attack, or policy collapse that destroys useful behavior. We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR), a benchmark that combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Across D4RL locomotion experiments and an exploratory AntMaze extension, TOUR shows that common deletion baselines have environment-dependent privacyutility behavior. Retraining and fine-tuning often provide stronger retained-utility references than uniform GA+Refit, while TrajDeleter remains a useful comparator but is not uniformly stronger under the same audit. Reference-model, threshold, deviation, equivalence, action-error, representation-based, and query-limited attacks further show that a single likelihood-based membership score can overstate deletion quality. In the evaluated settings, conclusions about offline RL unlearning are therefore not stable under single-score auditing. They depend on matched non-member construction, retraining-relative calibration, attack family, retained utility, and explicit scope for diagnostic architecture or component-level evidence.
1
Introduction
Offline Reinforcement Learning (RL) agents are increasingly used in domains such as autonomous driving and healthcare [1, 2], where policies are trained on fixed collections of sensitive behavioral trajectories [3, 4]. Privacy regulations such as the General Data Protection Regulation and the California Consumer Privacy Act establish a “right to be forgotten” [5, 6, 7], which creates pressure to remove the influence of selected trajectories after training. Machine unlearning provides tools for approximate data removal in supervised learning [8, 9, 10], but offline RL adds a distinct evaluation challenge. Deletion should reduce trajectory-level evidence while preserving policy behavior on the retained task distribution. This evaluation problem is more subtle than checking whether one membership score moves toward random guessing. A deletion update can reduce a forget-set score because it removes trajectoryspecific information, but the same movement can also arise from policy collapse, degraded retained ∗ Corresponding author.
Preprint.
Policy families (shared training setup) MLP
LSTM
Evaluation (Privacy & Utility)
IQL companion
Attention
Trajectory-level Membership Inference
... HDF5
Attention
HalfCheetah D4RL R/M/ME Offline Dataset Random Split
Hopper R/M/ME
Walker2D R/M/ME
or
......
......
or
...
Attention MLP Head ......
MLP Head
MLP Head
(forget)
Policies (Base/Unlearned)
(retain)
Per-token NLL
V/Q network action-error audit
Density
Decision Transformer (DT)
Data Construction
AUC
Gaussian Action Head
NLL Value Base policy (fully trained on
AntMaze extension U/UD/MD
HDF5 (Unseen D4RL) Holdout Pool
Matching
(matched nonmember)
)
Deletion/Unlearning Retraning reference on
Naive fine-tuning on
Component-level GA (diagnostic probes) Learned
Unlearned
Target - z-norm return -length - init state
GA+Refit (uniform)
TrajDeleter (external)
Forget Gap |AUC - 0.5|
Frozen
Frozen
Frozen
Frozen
......
......
Robustness suite (privacy diagnostics)
Utility (task performance)
Threshold Ref-model Attack MIA NLLdeviation
TOST
D4RL score/ AntMaze success retained-performance anchors
Gradient ascent on target, others frozen
Figure 1: Overview of TOUR. The benchmark partitions trajectories into forget, retain, and matched non-member sets, applies deletion baselines to trained offline RL policies, and audits the updated policies through matched membership tests, retained-performance anchors, and complementary diagnostics. The protocol is designed to distinguish trajectory deletion evidence from policy collapse and single-attack artifacts. behavior, or residual evidence that is visible under a different attack. Retain-side diagnostics are also needed because an update can shift retained trajectories relative to matched non-members. These failure modes motivate the central question of this paper: How should trajectory-level unlearning in offline RL be evaluated so that apparent membership reduction is not confused with residual memorization, retain-side artifacts, or utility collapse? We answer this question with Trajectory-level memOrization and Unlearning in offline RL (TOUR). TOUR is a benchmark centered on three MuJoCo continuous-control environments from D4RL [11], three locomotion data variants, and an exploratory AntMaze extension. It evaluates Decision Transformer (DT) policies [12, 13, 14], MLP and Long Short-Term Memory (LSTM) policy families, and an Implicit Q-Learning (IQL) [15] companion analysis. The benchmark standardizes trajectory-level deletion, matched non-member construction, retraining references, retained-performance anchors, and a multi-family membership audit. The main comparisons use settings in which retained task performance makes privacy-utility interpretation meaningful. Figure 1 summarizes the evaluation protocol. TOUR first partitions trajectories into forget, retain, and matched non-member sets. It evaluates each update through forget-side membership, retain-side diagnostics, retained utility, and complementary attacks. The resulting evidence shows that common baselines fail in different ways. Uniform GA+Refit can move membership scores while losing utility, naive fine-tuning can appear private under the primary likelihood score without providing a complete audit on its own, and the TrajDeleter [16] baseline has environment-dependent behavior with clearer retraining-relative residual signals in selected Walker2D settings. The architecture and component-level experiments, together with supplementary controls reported in the appendix, provide diagnostic evidence. Their role is to reveal where the benchmark detects sensitivity rather than to prove a universal mechanism for selective deletion. In summary, our contributions are as follows: • We introduce TOUR, a benchmark for trajectory-level memorization and unlearning in offline RL that combines matched non-member controls, retraining references, retainedperformance anchors, and reproducible deletion baselines. • We propose a multi-attack privacy-utility audit within TOUR in which deletion evidence requires reduced forget-set membership signal without retained-performance collapse or retain-side diagnostic failure. • We provide benchmark findings showing that common deletion baselines can look private under a single score while failing reference-model or utility checks. Additional diagnostic experiments characterize architecture-associated and component-level sensitivity under bounded empirical scope.
2
2
Related Work
Machine Unlearning. Machine unlearning seeks to remove the influence of selected training data without requiring full retraining [10, 17]. Representative approaches include partition-based methods such as SISA [8], certified removal for convex models [9], Fisher-information updates for deep networks [18], and data bias-based plugins for continual learning [19]. A recurring challenge is that apparent deletion does not necessarily imply actual deletion, because gradient-based procedures can leave detectable traces of the removed data [20]. Recent work in vision and language modeling studies heterogeneous memorization by targeting attention components [21], pruning highly attributed modules [22], or isolating vulnerable spectral subspaces [23]. TOUR differs in its primary goal. It uses structural diagnostics when useful, but the main contribution is an evaluation protocol for offline RL trajectory deletion. Membership Inference and Privacy Auditing. Membership Inference Attacks (MIAs) [24] audit whether training data remain detectable after model training or unlearning. Threshold attacks can be effective [25], but a single attack family often gives an incomplete privacy assessment [26, 27]. For sequence models, example-level loss is a strong membership signal [28]. Offline RL introduces additional confounds because returns, trajectory lengths, and initial states can differ between deleted trajectories and non-members. TOUR therefore combines matched non-member sets with likelihood, reference-model, threshold, deviation, equivalence, action-error, and others to evaluate deletion quality as a privacy-utility profile rather than as a single score. Offline RL and Model Architectures. Offline RL learns policies from fixed datasets without further environment interaction [29]. Common policy families include autoregressive DTs [12], value-based methods such as Conservative Q-Learning (CQL) [30] and IQL [15], recurrent models such as LSTMs [31], and feedforward MLPs. Standard benchmarks such as D4RL [11] focus on returns, whereas privacy and retained-data diagnostics under controlled deletion remain less studied. RL-Specific Unlearning. Machine unlearning in RL remains limited because data are sequential and policy utility must be preserved after deletion. Prior work has studied removal of entire environments in online RL [32] and trajectory-level deletion through TrajDeleter [16]. These studies focus primarily on deletion algorithms and utility retention. TOUR is complementary. It includes TrajDeleter as a baseline and asks how offline RL trajectory deletion should be audited without conflating membership reduction, residual memorization, retained-data artifacts, and utility collapse.
3
Benchmark
Figure 1 illustrates the overview of TOUR. The benchmark standardizes trajectory partitioning, controls major confounds in membership inference, and evaluates deletion quality through a structured privacy-utility evidence profile. 3.1 Problem Formulation Let D denote an offline dataset of trajectories collected by a behavior policy, where each trajectory τ = (s0 , a0 , r0 , . . . , sT , aT , rT ) contains the sequence of states, actions, and rewards from a single PT episode. For DT-style conditioning, each timestep also carries a return-to-go variable Rt = t′ =t rt′ . The dataset D is partitioned into a forget set Df and a retain set Dr , with D = Df ∪ Dr . Given a base policy πθ trained on D, the unlearning objective is to produce an updated policy πθ′ that balances privacy and utility. From a privacy perspective, an attacker should not distinguish trajectories in Df from unseen non-members more reliably than random guessing. From a utility perspective, πθ′ should retain effective behavior on the task distribution represented by Dr . TOUR treats privacy and utility as joint evidence. A credible evidence profile should show reduced forget-set membership evidence relative to the base policy while preserving retained-task performance and avoiding retain-side diagnostic failures. Apparent privacy improvements that arise primarily from policy collapse are reported as failure modes rather than as deletion evidence. 3.2 Environments and Data Construction TOUR includes three D4RL continuous-control environments from the MuJoCo locomotion suite, HalfCheetah, Hopper, and Walker2D [11]. Each environment is evaluated on the medium-replay-v2 (R), medium-v2 (M), and medium-expert-v2 (ME) variants. We refer to the medium-replay-v2 3
settings as replay variants. Utility in these environments is reported as the standard D4RL normalized score, where higher values indicate stronger task performance under the D4RL normalization. The main comparative analyses use locomotion settings in which the learned policies retain meaningful task performance. Replay and lower-power slices serve as stress tests or supplementary diagnostics when retained utility or precision is weaker. We also include a three-setting AntMaze extension comprising antmaze-umaze-v2 (U), antmaze-umaze-diverse-v2 (UD), and antmaze-medium-diverse-v2 (MD). This extension is an exploratory navigation stress test. Its success-rate-derived utility is not directly comparable to locomotion returns, and the resulting evidence is not used to establish broader protocol validity beyond locomotion. Unlike environment-level forgetting [32], TOUR enforces strict trajectory-level partitioning: every episode of D belongs either to Df or to Dr . By default, Df contains 10% of the trajectories, sampled uniformly at random. Because membership can be confounded by trajectory returns, lengths, and initial states, a matched non-member set Dn is constructed from a disjoint holdout pool of D4RL datasets using k-d tree retrieval over z-normalized trajectory features. These features consist of episode return, trajectory length, and the initial-state coordinates used in the matching diagnostics of Appendix D.1. This procedure maintains |Dn | = |Df | while reducing the largest observable trajectory-level differences, but it does not eliminate all residual confounds. Appendix D.1 therefore reports both balance diagnostics and precision summaries. 3.3 Policy Families and Model Architectures To investigate the relationship between architectural bias and memorization, we evaluate three policy families under a shared training budget of 100K gradient steps and a uniform context length K = 20 timesteps. These are a 3-layer DT with a Generative Pre-trained Transformer 2 (GPT-2)-style causal self-attention backbone [33], a 3-layer LSTM, and an MLP baseline without explicit sequence aggregation. The DT and LSTM provide autoregressive or recurrent sequence modeling, whereas the MLP serves as a feedforward comparison. To avoid ambiguity, the term MLP baseline refers only to this standalone policy architecture. We use Gaussian action heads that predict diagonal Gaussian policies over mean and log-variance outputs, enabling exact per-token NLL computation during privacy auditing. During unlearning, variance parameters remain frozen to avoid variance inflation and to keep the likelihood audit comparable across updates. An IQL comparison [15] provides a value-based reference evaluated under a trajectory-level action-error audit because IQL does not parameterize the Gaussian action distribution needed for exact per-token NLL evaluation. 3.4 Multi-Attack Privacy Auditing Protocol Because relying on a single metric can be misleading [27], TOUR evaluates unlearning through a likelihood-centered Trajectory-level Membership Inference (TMI) protocol. Primary Attack.
For a given trajectory τ , the mean per-timestep NLL is NLL(τ ; θ) =
T 1 X − log πθ (at | st , Rt , t), T + 1 t=0
(1)
where at , st , and Rt denote the action, state, and return-to-go at timestep t, and the final argument of πθ is the timestep index used by DT-style conditioning. Trained models typically assign lower NLL values to seen trajectories, so −NLL serves as the membership score. We report the Area Under the Receiver Operating Characteristic Curve (AUC-ROC, hereafter AUC) between Df and Dn . Privacy is summarized by the forget gap, defined as |AUCf − 0.5|, where AUCf denotes the forget-versus-negative AUC and smaller values indicate weaker membership evidence. Values above 0.5 mean that the model fits forget trajectories more confidently than matched non-members, whereas values below 0.5 invert that ordering. We use the direction-agnostic gap because either direction still indicates separability once the attack score is allowed to flip sign. We also report AUCr , the retain-versus-negative AUC, as a diagnostic validity check. Robustness Suite. To reduce overfitting to the primary metric, TOUR includes four complementary likelihood-based checks: a threshold MIA [25], a reference-model attack against the retraining reference, an NLL-deviation AUC (the absolute deviation of each trajectory’s NLL from the mean retain-set NLL), and Two One-Sided Tests (TOST) equivalence testing with a margin of ε = 0.1. In the reference-model attack, the retrained model serves as a calibrated reference and the membership score is NLL(τ ; θref ) − NLL(τ ; θ′ ), so higher values indicate trajectories that the target model fits 4
more confidently than the retraining reference. The retraining reference is an operational floor rather than a certified zero-influence oracle. It is not expected to achieve exactly 0.5 because the forget set and matched non-member set can retain residual distributional differences after matching. The TOST analysis tests whether the forget-set AUC is statistically equivalent to random guessing, namely 0.5, within the stated margin. Since these checks still focus mainly on trajectory likelihoods, action-error, representation-based, and query-limited shadow-model attacks are additionally reported in Appendices E.1, E.2, and E.3. These attacks serve as exploratory diagnostics depending on the attack family and data regime rather than as direct replacements. 3.5 Benchmark Baselines and Diagnostic Probes TOUR compares five policy conditions: a base model πθ trained on D, a retraining reference policy πre trained from scratch on Dr , a naive Fine-Tuned (FT) model continued on Dr , a uniform GradientAscent update followed by Refitting (GA+Refit), and a DT-adapted TrajDeleter baseline. Retraining and naive fine-tuning inherit the default training configuration: batch size 64, learning rate 10−4 , weight decay 10−4 , linear warmup for 10K steps, and a 100K-step budget. In the ascent stage of GA+Refit, the updated policy πθ′ minimizes LGA (θ′ ) = −Eτ ∼Df [NLL(τ ; θ′ )] + λ Eτ ∼Dr KL πθ (· | τ ) ∥ πθ′ (· | τ ) , (2) where λ is the ascent-strength hyperparameter that controls the tradeoff between deletion pressure on Df and behavioral preservation on Dr . Unless stated otherwise, the benchmark uses λ = 1.0, ascent learning rate 10−4 , gradient clipping at 0.25, and a default 500-step ascent budget. Then it refits a reinitialized action head on Dr for 10K steps at learning rate 10−4 using batch size 64. The TrajDeleter baseline is treated as an external structured deletion comparator [16] rather than as a proposed method. Its configuration uses α = 1.0, β = 2.0, 100 steps of the first stage, and 1, 000 steps of the second stage, where α weights the forget-directed loss in the first stage and β weights the retain-side anchor KL term in the second stage. For component-level analysis, utility is assessed through two complementary criteria. The first criterion is a relative utility budget against the matched uniform baseline: a target is feasible only when it does not underperform the corresponding uniform update at the same ascent budget. The second criterion reports retained performance ratios against both the retraining reference and the original base policy. Appendix C.3 gives the formal definitions. To diagnose whether deletion pressure has different effects across network regions, TOUR includes component-level GA as a diagnostic or exploratory probe. Instead of updating the entire model, this procedure freezes most parameters and applies gradient ascent only to a targeted component. A random-matched control with an identical parameter count to the target layer helps distinguish component-associated effects from generic small-update effects. These comparisons are interpreted jointly with the explicit retained-performance criteria and are not used as standalone evidence for a deployable component-level unlearning method. 3.6 Benchmark Evidence Structure Our benchmark separates three levels of evidence. The required evidence consists of the canonical forget gap under the DT likelihood audit, the retain-versus-negative diagnostic AUC, and explicit retained-performance anchors. Calibrated diagnostics, including the reference-model gap, threshold attack, deviation attack, and TOST, refine the interpretation of residual memorization within the same audit family. Exploratory evidence includes low-power settings, the AntMaze extension, the IQL companion analysis, and broad component-level sweeps. This separation keeps the main claim tied to a structured evidence profile while exposing structural patterns that warrant further study.
4
Experimental Results
TOUR tests whether trajectory deletion can be evaluated without conflating membership reduction, retained-data artifacts, and utility collapse. The results show that privacy conclusions depend jointly on matched controls, attack family, and retained-performance anchors. Unless stated otherwise, result tables report mean aggregates over three seeds. 4.1 Benchmark Validity and Deletion Baselines The evaluation begins with the locomotion baselines in Table 1. The primary privacy quantity is the direction-agnostic forget gap. The 95% intervals summarize seed-level uncertainty for the aggregate method score and should not be read as independent-trajectory uncertainty, while Appendix D.1 5
Table 1: Benchmark results across R/M/ME variants of three locomotion environments. The main comparative claims rely on settings with meaningful retained task performance. Utility denotes the D4RL normalized score. Forget-Set AUC is retained as an auxiliary direction-sensitive quantity. Retain AUC is a diagnostic attack score, not a utility metric. Pair counts and detectable-gap diagnostics are reported separately in Table 17. Environment Walker2D (ME)
HalfCheetah (ME)
Hopper (ME)
Walker2D (M)
HalfCheetah (M)
Hopper (M)
Walker2D (R)
HalfCheetah (R)
Hopper (R)
Method Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter
Utility ↑ 93.11 87.11 85.46 0.00 92.73 64.13 59.82 60.63 4.70 58.74 44.04 47.70 46.01 8.78 42.63
Forget Gap ↓ 0.015 0.015 0.017 0.015 0.012 0.140 0.058 0.054 0.094 0.126 0.049 0.004 0.002 0.033 0.046
Forget AUC 0.515 0.485 0.483 0.515 0.512 0.640 0.558 0.554 0.594 0.626 0.549 0.504 0.502 0.467 0.546
95% CI [0.513, 0.518] [0.484, 0.487] [0.482, 0.483] [0.513, 0.519] [0.510, 0.515] [0.635, 0.646] [0.553, 0.561] [0.553, 0.555] [0.585, 0.604] [0.621, 0.633] [0.548, 0.550] [0.499, 0.509] [0.501, 0.503] [0.465, 0.468] [0.543, 0.549]
Retain AUC 0.538 0.545 0.534 0.500 0.536 0.563 0.575 0.552 0.496 0.554 0.507 0.513 0.499 0.489 0.512
Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter
71.92 67.62 77.63 13.50 67.90 42.77 42.69 42.95 4.73 41.58 43.20 45.55 46.87 17.04 47.15
0.121 0.051 0.053 0.107 0.107 0.197 0.043 0.034 0.274 0.160 0.053 0.016 0.020 0.125 0.046
0.621 0.551 0.553 0.607 0.607 0.697 0.457 0.466 0.226 0.660 0.553 0.484 0.480 0.375 0.546
[0.621, 0.622] [0.550, 0.553] [0.552, 0.554] [0.603, 0.610] [0.605, 0.609] [0.693, 0.705] [0.444, 0.465] [0.451, 0.475] [0.197, 0.253] [0.656, 0.661] [0.550, 0.555] [0.480, 0.488] [0.478, 0.484] [0.339, 0.400] [0.542, 0.549]
0.597 0.607 0.584 0.494 0.583 0.724 0.753 0.717 0.502 0.694 0.519 0.525 0.511 0.550 0.520
Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter
25.66 44.09 36.47 10.50 26.26 32.29 30.61 32.14 7.96 35.37 9.26 25.17 22.15 11.20 17.37
0.125 0.020 0.028 0.089 0.071 0.224 0.002 0.000 0.284 0.156 0.014 0.018 0.016 0.010 0.009
0.625 0.520 0.528 0.589 0.571 0.724 0.498 0.500 0.784 0.656 0.514 0.518 0.516 0.510 0.509
[0.622, 0.630] [0.518, 0.522] [0.523, 0.532] [0.586, 0.595] [0.570, 0.572] [0.719, 0.730] [0.495, 0.505] [0.490, 0.505] [0.776, 0.791] [0.643, 0.668] [0.496, 0.524] [0.514, 0.520] [0.511, 0.523] [0.464, 0.592] [0.503, 0.512]
0.528 0.506 0.517 0.438 0.493 0.695 0.718 0.718 0.456 0.684 0.581 0.619 0.605 0.510 0.620
reports matched-pair counts, interval widths, detectable-gap diagnostics, and the quartile-level evidence in Figure 6. Across most locomotion settings, the base DT exhibits a detectable membership signal. In the main locomotion comparisons, retraining references and naive fine-tuning often move the forget-set AUC close to random guessing while preserving more utility than uniform GA+Refit, whereas the DT-adapted TrajDeleter baseline remains environment-dependent under the same audit. TrajDeleter broadens the comparison through an external structured deletion procedure, but its canonical configuration does not yield a uniformly stronger privacy-utility profile than the retraining-based references. Replay variants broaden coverage, but the main comparative claims come from settings in which retained task performance and statistical precision keep the tradeoff interpretable. Uniform GA+Refit often reduces utility sharply and does not provide a reliable privacy advantage. In several settings, including HalfCheetah (M) and Hopper (M), the forget-set AUC falls below 0.5. This value indicates an inverted ordering rather than erasure because the forget set remains separable from matched non-members after score reversal. The benchmark therefore treats privacy improvements obtained mainly through policy collapse as failures. 6
Appendix A.3 reports the AntMaze extension as supplementary navigation evidence under a separate success-rate-derived utility scale. The results distinguish a stronger U-Maze variant from weaker U-Maze-Diverse and Medium-Diverse variants and therefore provide a stress test of the audit outside the locomotion suite without implying score equivalence to the locomotion benchmark. 4.2
Single Membership Scores Are Insufficient
Table 2: Multi-attack privacy audit on the selected high-utility comparative settings aggregated across seeds. Four membership inference attacks are applied to each method. NLL: per-timestep NLL AUC. Thr: threshold MIA balanced accuracy [25]. Ref: reference-model calibrated AUC from the score NLL(τ ; θref ) − NLL(τ ; θ′ ). Dev: NLL-deviation AUC based on absolute deviation from the mean NLL of the retain set. Values closer to 0.5 indicate weaker membership evidence. The retrainingreference row reports “—” for Ref AUC because the retrained model is itself the reference model in that attack, so the calibrated score is undefined. TOST reports whether all available seeds pass the equivalence test at significance level 0.05. All< ε reports whether every reported likelihood-family attack gap remains below ε = 0.1 across all available seeds. These two columns need not agree because one is a statistical test and the other is a threshold condition. The NLL AUC values are recomputed inside the multi-attack pipeline and then aggregated across seeds, so they need not match the benchmark-table forget-set AUC summary exactly. Environment HalfCheetah (ME)
Walker2D (ME)
Walker2D (M)
Method Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter
Pairs 140 140 140 140 140 153 153 153 153 153 83 83 83 83 83
NLL AUC 0.640 0.558 0.554 0.594 0.626 0.515 0.485 0.483 0.515 0.512 0.621 0.551 0.553 0.607 0.607
Thr BA 0.598 0.550 0.545 0.531 0.598 0.505 0.497 0.490 0.512 0.501 0.598 0.534 0.542 0.602 0.578
Ref AUC 0.988 — 0.460 0.607 0.961 0.861 — 0.490 0.549 0.853 0.999 — 0.470 0.627 0.911
Dev AUC 0.443 0.443 0.453 0.578 0.440 0.477 0.495 0.497 0.487 0.474 0.459 0.478 0.479 0.449 0.464
TOST ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗
All< ε ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✗
Table 2 applies the likelihood-centered audit suite to test robustness. The table focuses on selected high-utility locomotion settings that combine nontrivial base membership signal, retainedperformance viability, and sufficient matched-pair precision. Appendix E extends this view with complementary attack families rather than with a complete NLL matrix. This restricted threat model extends the primary per-token NLL AUC with threshold, reference-model, deviation, and equivalence checks while remaining within the same likelihood family. Naive FT shows that an NLL score close to the random-guessing region is not by itself a complete audit, although its retraining-relative gaps are small in this selected slice. The clearest reference-model failures instead occur for TrajDeleter in Walker2D (ME), where near-random primary NLL scores can coexist with large retraining-relative residual signals. A single metric can therefore overstate deletion quality. The other baselines support the same broad interpretation. The base DT exhibits membership signals across attacks, retraining provides the operational privacy reference, and uniform GA+Refit still leaves residual signals or loses utility depending on the environment. The DT-adapted TrajDeleter baseline is included in Table 2 as an external structured deletion comparator. Its residual signals under reference-model calibration show that procedure-aware deletion still requires retraining-relative and retained-utility checks. Appendices E.1, E.2, and E.3 extend this analysis with action-error, representation-based, and querylimited shadow-model attacks. These diagnostics do not overturn the main likelihood audit, but they show that method rankings remain attack-family-dependent and that stronger gradient-ascent pressure does not produce a consistent privacy advantage across environments and audit families. 4.3 Architecture Results as Descriptive Diagnostics To investigate architecture-associated differences, we compare DT, MLP, and LSTM policies on the replay variants. As summarized in Table 3, all rows come from a unified backbone comparison configuration with the same trajectory split, matched non-member construction, context length, and likelihood-based TMI audit. The architecture comparison is included to expose how audit 7
conclusions change under policy-family changes, not to isolate architecture as a causal factor. The MLP and LSTM baselines both remain below the DT on the reported forget-gap diagnostic, but their relative ordering and separation vary substantially across environments. Appendix B.3 extends the scope with an IQL analysis based on an action-error audit rather than the DT likelihood score. Hopper and Walker2D provide the main comparative settings, while HalfCheetah (R) is retained as a lower-power reference. Appendices B.2 and B.3 add narrower controls and supplementary non-DT evidence. Taken together, the results support an architecture-associated difference under the evaluated training configurations, but not a capacity-matched causal claim, a utility-matched comparison, or a score-equivalent DT-versus-IQL ranking. Table 3: Architecture diagnostic for the replay variants under a unified backbone comparison configuration. All rows use backbone-policy evaluations with the same trajectory split, matched controls, context length, and likelihood-based TMI audit. Parameter counts, utility, forget gap, and interval width are shown together so that architecture-associated differences are interpreted with the main confounds in view rather than as capacity-matched or utility-matched causal isolation. The interval width is computed from the displayed CI endpoints in Table 6. Environment Hopper HalfCheetah Walker2D
Model DT MLP LSTM DT MLP LSTM DT MLP LSTM
Params 726K 69K 593K 727K 71K 594K 727K 71K 594K
Utility ↑ 22.35 15.28 1.70 36.77 35.32 1.81 45.22 20.46 0.31
Forget Gap ↓ 0.023 0.003 0.015 0.228 0.034 0.155 0.093 0.055 0.018
CI Width 0.003 0.001 0.004 0.016 0.005 0.025 0.004 0.000 0.001
4.4 Component-Level Updates as Diagnostic Probes Component-level GA is used as a diagnostic probe for update sensitivity inside DT policies. The probe is evaluated on all attention layers jointly and on individual attention layers. Figure 2 summarizes the resulting forget gaps, while Appendices D.3 and C.4 provide stability checks and utility-budget comparisons. Because TrajDeleter changes the deletion procedure rather than isolating individual DT components, it is treated as a benchmark baseline in Section 4.1 rather than as part of the component-level sweep.
Mean Forget Gap
The component-level sweep reveals substantial heterogeneity across environments. In the replay 0.20 variants analysis, some attention-layer targets re0.045 0.020 0.007 0.009 Hopper duce the forget gap relative to matched uniform 0.15 updates under a relative utility budget, but these gains are not sufficient under explicit retained- HalfCheetah 0.209 0.117 0.100 0.026 0.10 performance criteria (Tables 4, 15 and 20). The retained-utility check yields a narrower signal 0.05 outside this replay slice. Hopper provides the 0.013 0.021 0.046 0.054 Walker2D 0.00 clearest replay-setting diagnostic case: Attention Layer 1 at 250 steps reaches a forget-set All Attn Attn L1 Attn L2 AUC of 0.499 with D4RL score 11.8 in the full sweep (Table 13). This value is close to ran- Figure 2: Mean forget gap by environment and dom guessing only under severe utility loss, so component-level GA target. Component-level upit should be read as a policy-collapse warning dates expose environment-dependent sensitivity, rather than as a successful deletion result. The while smaller relative gaps do not necessarily imcorresponding Walker2D and HalfCheetah set- ply deletion with retained performance. tings remain either farther from random guessing or deep in a low-utility regime. The evidence therefore supports component sensitivity rather than a general selective-unlearning method. Table 4 summarizes the replay-setting component-level policy under the zero additional utility loss slice alongside the matched uniform baseline and retraining reference. Appendix C.3 and Figure 5 report the broader privacy-utility frontier and utility-budget view from the full sweep. The absolute utility ratios show that the relative utility criterion is weaker than the explicit retained-performance criteria. Component-level GA is therefore better viewed as a diagnostic probe than as a practical 8
unlearning method, and the benchmark contract does not count relative-budget feasibility alone as sufficient deletion evidence. Appendices C.1, C.5, and F.1 provide additional random-mask, gradient, and Fisher controls. These checks sharpen the interpretation of the component-level sweep, but they do not alter the main conclusion of this section. Table 4: Component-level GA under an explicit zero additional utility loss budget relative to the matched uniform baseline. The component-level results use the leave-one-seed-out cross-validation target, aggregated over step choices that satisfy ∆utility ≤ 0 relative to the matched uniform update. This aggregation differs from the per-held-out-split CV gaps in Table 20, which report one crossvalidation split at a time. Score/Retrain and Score/Base report explicit retained-performance ratios relative to the retraining reference and the base DT, so that relative feasibility with respect to uniform updates is not conflated with a stronger retained-performance criterion. Environment
Hopper HalfCheetah Walker2D
5
Method
Relative-budget feasible Retraining reference — Selective (held-out CV) 9/15 Uniform (matched baseline) — Retraining reference — Selective (held-out CV) 2/9 Uniform (matched baseline) — Retraining reference — Selective (held-out CV) 0/9 Uniform (matched baseline) —
Forget Gap ↓
Utility ↑
Score/Retrain
Score/Base
0.018 0.006 0.043 0.002 0.015 0.186 0.020 0.028 0.013
25.17 15.09 6.24 30.61 1.52 1.26 44.09 9.43 2.18
1.00 0.60 0.25 1.00 0.05 0.04 1.00 0.21 0.05
2.72 1.63 0.67 0.95 0.05 0.04 1.72 0.37 0.08
Discussion
TOUR supports a benchmark claim: trajectory-level unlearning in offline RL should be evaluated through matched controls, retained-performance anchors, and multiple attack families. This protocol distinguishes cases in which forget-set membership evidence decreases together with retained utility from cases in which a near-random privacy score is explained by policy collapse or by attack-specific calibration artifacts. Its role is to make privacy-utility evidence comparable across deletion procedures rather than to certify a single best deletion algorithm. The current evidence is also narrower than a mechanism paper would require. Architecture comparisons show that privacy behavior differs across model families under the evaluated shared training configuration, but utility, capacity, optimization, and audit family are not fully disentangled. The IQL comparison broadens the empirical scope, yet it uses trajectory-level action error rather than the likelihood-centered DT protocol. Component-level GA shows that update sensitivity varies across DT components and environments, but the retained-performance checks keep this evidence diagnostic rather than algorithmic. These results motivate more controlled follow-up studies, but they do not identify architecture as the unique cause of memorization or establish a deployable component-level unlearning procedure. The unlearning results show why future methods should report evidence in tiers. Uniform GA+Refit often moves policies into a low-utility regime, as shown in Table 1, while retraining and fine-tuning preserve more task performance in several settings. The multi-attack audit further shows that the primary NLL score alone can miss residual signals, especially under reference-model calibration. The TrajDeleter comparison points to the same conclusion from a different direction: an external structured deletion baseline still requires retraining-relative and retained-utility checks before a privacy claim is convincing under the evaluated audit. The main limitations are the single-head DT architecture, the remaining confounds in crossarchitecture comparisons, and the limited statistical power in lower-power settings such as HalfCheetah and Hopper replay. AntMaze is exploratory because it uses a separate success-rate utility scale, the navigation settings show strong environment-dependent difficulty, and the lower-power U-MazeDiverse and Medium-Diverse cases still yield coarse privacy uncertainty. The attack suite also covers only a subset of practical attackers. Additional limitations are discussed in Appendix F.2.
6
Conclusion
This paper introduces TOUR, a benchmark for trajectory-level memorization and unlearning in offline RL. TOUR combines matched non-member controls, retraining references, retained-performance anchors, and complementary membership attacks so that deletion quality is evaluated as a privacy-utility evidence profile rather than as a single membership score. Across the evaluated D4RL experiments, 9
common deletion baselines fail in different ways. Some updates reduce apparent membership evidence only while damaging policy utility, whereas others look private under one likelihood score but leave residual signals under retraining-relative or complementary attacks. Architecture and component-level experiments provide descriptive diagnostics, but the current evidence supports setting-specific interpretation rather than a universal component-localization mechanism. Future offline RL unlearning methods should therefore report forget-set membership evidence together with retraining-relative residual signals, retain-side diagnostics, retained task performance, and attack-family scope of each deletion claim.
References [1] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 35(8):10237–10257, 2023. [2] Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach, and Lina Yao. On the opportunities and challenges of offline reinforcement learning for recommender systems. ACM Transactions on Information Systems, 42(6):1–26, 2024. [3] Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang. Learning from good trajectories in offline multi-agent reinforcement learning. In AAAI, volume 37, pages 11672–11680, 2023. [4] Zeyu Jia, Alexander Rakhlin, Ayush Sekhari, and Chen-Yu Wei. Offline reinforcement learning: Role of state aggregation and trajectory data. In COLT, pages 2644–2719, 2024. [5] Maria Rigaki and Sebastian Garcia. A survey of privacy attacks in machine learning. ACM Computing Surveys, 56(4):1–34, 2023. [6] Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to be forgotten in federated learning: An efficient realization with rapid retraining. In INFOCOM, pages 1749–1758, 2022. [7] Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3):1464–1483, 2024. [8] Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In IEEE S&P, pages 141–159, 2021. [9] Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In ICML, pages 3832–3842. PMLR, 2020. [10] Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. ACM Transactions on Intelligent Systems and Technology, 16(5):1–46, 2025. [11] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. arXiv, abs/2004.07219, 2020. [12] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS, pages 15084–15097, 2021. [13] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In ICML, pages 21611–21630, 2023. [14] Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. In AAAI, volume 38, pages 15706–15714, 2024. [15] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In ICLR, 2022. 10
[16] Chen Gong, Kecen Li, Jin Yao, and Tianhao Wang. Trajdeleter: Enabling trajectory forgetting in offline reinforcement learning agents. In NDSS, 2025. [17] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, 7(2):181–194, 2025. [18] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In CVPR, pages 9301–9309, 2020. [19] Xuemei Cao, Hanlin Gu, Xin Yang, Bingjun Wei, Haoyang Liang, Xiangkun Wang, and Tianrui Li. Erroreraser: Unlearning data bias for improved continual learning. In ACM SIGKDD, pages 119–130, 2025. [20] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling SGD: understanding factors influencing machine unlearning. In IEEE EuroS&P, pages 303–319, 2022. [21] Yujia Tong, Tian Zhang, Jingling Yuan, Yuze Wang, and Chuang Hu. Lethevit: Selective machine unlearning for vision transformers via attention-guided contrastive learning. arXiv, abs/2508.01569, 2025. [22] Wenjie Bao, Jian Lou, Yuke Hu, Xiaochen Li, Zhihao Liu, Jiaqi Liu, Zhan Qin, and Kui Ren. Module-aware parameter-efficient machine unlearning on transformers. arXiv, abs/2508.17233, 2025. [23] Yancheng Huang, Changsheng Wang, Chongyu Fan, Yicheng Lang, Bingqi Shang, Yang Zhang, Mingyi Hong, Qing Qu, Alvaro Velasquez, and Sijia Liu. Subspace control: Turning constrained model steering into controllable spectral optimization. arXiv, abs/2604.04231, 2026. [24] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE S&P, pages 3–18, 2017. [25] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE CSF, pages 268–282, 2018. [26] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and data independent membership inference attacks and defenses on machine learning models. In NDSS, 2019. [27] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX, pages 2615–2632, 2021. [28] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership inference attacks from first principles. In IEEE S&P, pages 1897–1914, 2022. [29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, abs/2005.01643, 2020. [30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. volume 33, pages 1179–1191, 2020. [31] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. [32] Dayong Ye, Tianqing Zhu, Congcong Zhu, Derui Wang, Kun Gao, Zewei Shi, Sheng Shen, Wanlei Zhou, and Minhui Xue. Reinforcement unlearning. In NDSS, 2025. [33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. [34] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. volume 34, pages 1273–1286, 2021. 11
A
Environment and Configuration
A.1
Environment Overview
TOUR spans three offline RL locomotion environments with distinct control and reward structures. HalfCheetah, Hopper, and Walker2D are continuous-control tasks from D4RL [11]. HalfCheetah is a planar running task with smooth forward locomotion, Hopper requires stable single-leg hopping under repeated impacts, and Walker2D requires coordinated bipedal movement with stronger balance constraints. These three environments are evaluated on the medium-replay-v2 (R), medium-v2 (M), and medium-expert-v2 (ME) variants to test whether privacy conclusions persist across shifts in data quality within the locomotion suite. The benchmark also includes an AntMaze extension across antmaze-umaze-v2 (U), antmaze-umaze-diverse-v2 (UD), and antmaze-medium-diverse-v2 (MD). Unlike the locomotion tasks, these navigation settings are reported with a success-rate-derived utility scale. They are interpreted as an exploratory navigation stress test rather than as a direct continuation of the locomotion score scale or as a validation of broader protocol generality. A.2
Device and Runtime Details
All experiments were conducted on a single workstation equipped with an AMD Ryzen 9 7950X 16-Core Processor, 96 GB of system memory, and a single NVIDIA GeForce RTX 4070 Ti SUPER graphics card with 16 GB of video memory. The GPU driver version is 555.42.02 and the CUDA version is 12.4. The software stack comprises Python 3.12, PyTorch 2.2, Gymnasium with MuJoCo support version 1.2.3, scikit-learn 1.8.0, NumPy 2.4.3, and SciPy 1.14.0. All training and evaluation scripts run on the GPU by default; CPU is used only for data preprocessing and matching diagnostics. Each base Decision Transformer training run requires 100 K gradient steps with batch size 64, which takes approximately 15–25 minutes on the described hardware. Retraining references and naive fine-tuning follow the same training budget and thus share a comparable wall-clock cost. The GA+Refit procedure applies 500 ascent steps followed by 10 K refit steps on the reinitialized action head, completing in a similar time range. The IQL baseline trains for 20 K gradient steps with batch size 256. Across the full benchmark matrix, the main experiments (three environments, four method blocks, three seeds) were executed sequentially on this single-GPU workstation. The AntMaze extension, medium and medium-expert variant experiments, and supplementary analyses (architecture comparison, component-level update sweep, multi-attack auditing) were likewise conducted on the same machine. A.3
AntMaze Exploratory Extension
Table 5: AntMaze extension over U/UD/MD variants. Utility denotes a success-rate-derived scale rather than the D4RL normalized locomotion score used in the main benchmark table, so these values should not be compared numerically with Table 1. Each row reports the mean over three seeds. Forget-Set AUC remains the same direction-sensitive auxiliary quantity used elsewhere in TOUR. The 95% CI is a hierarchical bootstrap interval over the three seeds with paired resampling inside each seed. These rows extend the audit to a navigation family with a distinct utility scale. Environment AntMaze (U)
AntMaze (UD)
AntMaze (MD)
Method Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter Base DT Retrain Ref. Naive FT GA+Refit TrajDeleter
Utility ↑ 53.00 63.00 60.67 64.00 57.33 25.67 13.67 7.33 0.00 31.00 9.33 17.67 21.00 0.00 9.33
Forget Gap ↓ 0.012 0.004 0.011 0.094 0.016 0.500 0.000 0.167 0.500 0.500 0.146 0.021 0.042 0.438 0.167
12
Forget AUC 0.512 0.504 0.511 0.406 0.516 1.000 0.500 0.667 0.000 1.000 0.646 0.479 0.542 0.062 0.667
95% CI [0.433, 0.536] [0.397, 0.503] [0.411, 0.517] [0.317, 0.422] [0.433, 0.550] [1.000, 1.000] [0.167, 0.833] [0.167, 1.000] [0.000, 0.000] [1.000, 1.000] [0.250, 0.750] [0.250, 0.750] [0.250, 0.750] [0.000, 0.000] [0.250, 0.750]
Retain AUC 0.493 0.561 0.566 0.508 0.499 1.000 1.000 1.000 1.000 1.000 0.587 0.854 0.879 0.896 0.614
TMI AUC (Forget Set)
0.9
Random (0.5) DT
0.8
MLP LSTM
0.7 0.6 0.5 0.4 0.3
Hopper
HalfCheetah
Walker2D
Figure 3: Architecture comparison for the replay variants under the shared training configuration. Bars report the mean forget-set AUC across seeds for DT, MLP, and LSTM. This figure complements Table 6 by giving a compact visual summary of the architecture-associated ranking, which remains descriptive rather than capacity-matched. Table5 reports three-seed means of the AntMaze extension results together with hierarchical bootstrap intervals for the forget-set AUC. It should be interpreted as a supplementary summary rather than as a direct continuation of the locomotion benchmark. The ordering of the baselines is environment dependent. In U-Maze, the five baselines remain relatively close on the privacy-side auxiliary quantity, whereas U-Maze-Diverse and Medium-Diverse show larger separation in forget-set AUC under a much lower utility scale. These results therefore extend the empirical scope of the audit to a navigation family with sparse-reward difficulty, but they do not establish a score-equivalent ranking relative to the locomotion benchmark.
B
Architecture Comparison and Controls
B.1
Cross-Architecture Configuration
To compare model architectures under a shared training configuration, the same nominal optimization budget is maintained across all three paradigms. Table 6 presents the complete TMI comparison under this shared training configuration. Table 6: Cross-architecture TMI comparison under shared training budget (100K steps, context=20). Parameter counts differ substantially (DT ∼727K, LSTM ∼594K, MLP ∼71K). Claims are scoped to this experimental setting and should not be read as parameter-matched architecture isolation. Environment Hopper HalfCheetah Walker2D
Model DT MLP LSTM DT MLP LSTM DT MLP LSTM
Params 726K 69K 593K 727K 71K 594K 727K 71K 594K
Utility ↑ 22.35 15.28 1.70 36.77 35.32 1.81 45.22 20.46 0.31
Forget AUC 0.523 0.503 0.485 0.728 0.534 0.345 0.593 0.555 0.518
95% CI [0.521, 0.524] [0.503, 0.504] [0.483, 0.487] [0.719, 0.735] [0.531, 0.536] [0.332, 0.357] [0.591, 0.595] [0.555, 0.555] [0.517, 0.518]
All architectures (DT, LSTM, and MLP) are trained for exactly 100K gradient steps using a context length of K = 20 timesteps, where applicable. Parameter counts vary materially, with the Decision Transformer (∼727K) having higher capacity than the LSTM (∼594K) and the MLP (∼69−−71K). Consequently, the ordering of membership scores should be interpreted as a shared-budget comparison rather than a parameter-matched statement regarding architecture alone. As demonstrated in Table 6, the MLP achieves mean D4RL normalized scores of 15.28 in Hopper, 20.46 in Walker2D, and 35.32 in HalfCheetah across seeds 0, 1, and 2. Here, locomotion utility follows the standard D4RL normalization, where higher values indicate better task performance on a scale aligned with the benchmark reference policies. The LSTM contains approximately eight times more parameters than the MLP, yet its forget gap is larger in Hopper and HalfCheetah and smaller in Walker2D, so the 13
shared-budget comparison does not support a uniform cross-environment ordering between these two baselines. The main Decision Transformer configuration uses three layers, a hidden dimension of 128, a single attention head per layer, and dropout of 0.1, together with a Gaussian action head. Optimization uses AdamW with batch size 64, learning rate 10−4 , weight decay 10−4 , linear warmup for 10K steps, and gradient clipping at 0.25. The LSTM baseline also uses three layers with embedding dimension 128 and dropout 0.1, while the MLP baseline uses two hidden layers of width 256 with dropout 0.1. The IQL configuration uses two hidden layers of width 256, dropout 0.0, batch size 256, 20K gradient steps, discount 0.99, expectile 0.7, temperature 3.0, and Adam-based actor, critic, and value learning rates of 3 × 10−4 . Across the main benchmark tables, results are aggregated over random seeds 0, 1, and 2 unless stated otherwise. The experiments are implemented in Python 3.12, PyTorch 2.2, and Gymnasium with MuJoCo support version 1.2.3. B.2
Within-Family Capacity Control and Cross-Family Anchors
To mitigate the influence of potential confounds in the architecture comparison under a shared training configuration, two additional analyses are provided. The first analysis remains within the DT family to compare effective-capacity variants under an identical training horizon. The second analysis adds cross-family anchors in Hopper, HalfCheetah, and Walker2D by pairing medium DT checkpoints with MLP comparison runs from the same scaling settings, and by introducing closer parameter-matched MLP controls in Hopper and Walker2D. These perspectives reduce the capacity mismatch and test whether the reported differences vanish under a stronger control. Table 7: Within-family DT capacity controls together with approximate and closer parameter-matched cross-family anchors for the medium-replay settings. The within-family rows isolate residual variation inside the DT family under the same training horizon. The cross-family rows reduce the original capacity mismatch relative to Table 6, but they still do not provide strict architecture isolation because sequence modeling, return conditioning, and optimization remain coupled. Panel
Environment
Model pair
Utility ↑
Forget AUC
Forget Gap ↓
95% CI width
Within-family DT
Hopper Hopper Hopper HalfCheetah HalfCheetah Walker2D Walker2D
S128×3 M256×3 L256×6 M256×3 L256×6 M256×3 L256×6
22.52 28.52 17.86 34.86 28.03 26.95 35.99
0.582 0.626 0.606 0.784 0.752 0.625 0.620
0.082 0.127 0.106 0.284 0.252 0.125 0.120
0.132 0.129 0.131 0.355 0.391 0.177 0.178
Approx. cross-family anchor
Hopper HalfCheetah Walker2D
DT(M) vs. MLP DT(M) vs. MLP DT(M) vs. MLP
28.52 / 20.02 34.86 / 19.11 26.95 / 5.38
— — —
0.127 / 0.007 0.284 / 0.022 0.125 / 0.039
∆gap = 0.119 ∆gap = 0.262 ∆gap = 0.086
Parameter-matched cross-family anchor
Hopper Walker2D
DT(M) vs. MLPpm DT(M) vs. MLPpm
28.52 / 14.59 26.95 / 19.61
— —
0.127 / 0.028 0.125 / 0.116
∆gap = 0.099 ∆gap = 0.010
Table 7 demonstrates that the within-family DT comparisons retain nontrivial forget-gap variation even before extending across model families. In Hopper, the medium DT reaches a larger forget gap than the small DT at higher utility, whereas the medium-to-large transition fails to yield a monotone capacity-only pattern once utility deteriorates sharply. HalfCheetah provides only a noisier same-family check: the medium DT retains a larger mean forget gap than the large DT, but this setting remains low power, and the utility difference is still substantial. Walker2D exhibits an equally weak monotone trend: the large DT improves utility relative to the medium DT, yet the forget gap changes only from 0.125 to 0.120. Therefore, these within-family controls support only a residual architecture-associated signal rather than a clean capacity law. The cross-family anchors in the lower panels of Table 7 indicate a similar direction while remaining explicitly limited in scope. The approximate anchors preserve the earlier pattern: in Hopper, HalfCheetah, and Walker2D, the medium DT retains a larger forget gap than the paired MLP anchor, but the utility mismatch remains substantial, especially in HalfCheetah and Walker2D. The closer parameter-matched controls strengthen this reading in a narrower sense. In Hopper, the parameter-matched MLP still exhibits a smaller mean forget gap than the paired medium DT, while also retaining lower mean utility. In Walker2D, the parameter-matched control brings the mean forget gaps much closer, yet the paired MLP remains weaker in utility. The table therefore reduces the 14
original ten-fold capacity concern without overturning the conservative conclusion of the supplement: the shared-budget comparison is not explained solely by the initial parameter gap, yet the remaining differences in sequence modeling, return conditioning, and optimization still prevent a clean causal claim about architecture alone.
Table 8: Context-length sensitivity of the medium DT scaling checkpoint M256×3 under a fixed 500K pretraining budget. This auxiliary probe uses a larger pretraining budget than the 100K-step main benchmark in order to study context-length effects within the same scaling family. Each row reports the multiseed mean forget-set AUC, the mean seed-level 95% interval width for that AUC, and the Utility after re-evaluating the same model family at context lengths 10, 20, and 40. This supplement is intended as a within-family mechanism check rather than as a standalone architecture comparison. Environment
K=10 Forget AUC
K=10 CI Width
K=10 Utility
K=20 Forget AUC
K=20 CI Width
K=20 Utility
K=40 Forget AUC
K=40 CI Width
K=40 Utility
Hopper HalfCheetah Walker2D
0.480 0.491 0.520
0.132 0.439 0.185
29.33 35.44 28.76
0.479 0.481 0.520
0.133 0.437 0.185
15.84 35.64 35.98
0.481 0.481 0.522
0.133 0.435 0.186
29.73 36.25 32.54
The same-family context-length check in Table 8 provides a limited mechanism probe for the medium DT configuration. In Hopper, the forget-set AUC remains nearly unchanged across context lengths K ∈ {10, 20, 40}, while the D4RL normalized score varies substantially. Walker2D shows the clearest utility sensitivity, with the D4RL normalized score rising at K=20 before partially declining at K=40, whereas the forget-set AUC stays near 0.52 throughout and the mean interval width remains stable at approximately 0.185 to 0.186. HalfCheetah remains dominated by low statistical power: the mean AUC stays close to random guessing, and the mean interval width remains wide at 0.435 to 0.439 across all three context lengths. These additional runs therefore do not support a simple monotone account in which longer temporal context alone produces stronger membership leakage. Instead, they reinforce the narrower interpretation adopted throughout the paper, namely that utility, architecture, and audit behavior interact in an environment-dependent manner even within a fixed DT family.
Table 9: Consolidated matched-control summary for the replay-setting architecture evidence. The table combines the medium DT anchor, the closest within-family DT comparison available for each environment, the approximate DT-versus-MLP anchor, and the IQL family companion readout. The IQL entries are reported only as a trajectory-level action-error companion audit and not as a scoreequivalent replacement for the likelihood-based DT audit. Environment
DT(M) Utility
DT(M) Forget Gap
DT(L) Utility
DT(L) Forget Gap
MLP Utility
MLP Forget Gap
IQL Utility
IQL Forget Gap
Hopper HalfCheetah Walker2D
28.52 34.86 26.95
0.127 0.284 0.125
17.86 28.03 35.99
0.106 0.252 0.120
20.02 19.11 5.38
0.007 0.022 0.039
44.42 39.10 67.79
0.018 0.068 0.085
Table 9 consolidates the same-family DT controls, the approximate DT-versus-MLP anchors, and the IQL companion readout into a single matched-control summary. The common pattern is that the medium DT retains a larger forget gap than the closest available anchors in Hopper, while HalfCheetah remains the clearest reminder of the remaining limitation because every comparison in that environment is filtered through a low-power regime with only approximate controls. Walker2D further shows that the architecture signal is sensitive to the choice of control, since the gap relative to the approximate MLP anchor is modest even before introducing the tighter parameter-matched comparison discussed above. Consequently, the strengthened supplement narrows the interpretation of the architecture evidence rather than broadening it: the current experiments reduce the force of the original capacity objection, but they still support only an architecture-associated signal under the evaluated configurations, not architecture as the unique driver of membership leakage. 15
B.3
Non-DT Baseline Supplement
To evaluate non-DT architectures, a shared-training-configuration comparison is presented alongside an exploratory cross-family path and the detailed IQL companion results referenced in the main text.
Table 10: Primary non-DT companion summary for the replay settings. The table reports the IQL family under the trajectory-level action-error membership audit, which complements the DT-centered likelihood audit in Table 1 rather than replacing it. IQL does not parameterize the Gaussian action distribution needed for exact per-token NLL evaluation, so the action-error audit is the closest trajectory-level companion available in the current benchmark. Hopper and Walker2D provide the primary comparative settings. HalfCheetah (R) is retained as a lower-precision reference with only 14 matched pairs. Environment Hopper
Walker2D
HalfCheetah
Method IQL Base IQL Retraining Reference IQL Naive Fine-Tuning IQL Actor-Ascent+Refit IQL Base IQL Retraining Reference IQL Naive Fine-Tuning IQL Actor-Ascent+Refit IQL Base IQL Retraining Reference IQL Naive Fine-Tuning IQL Actor-Ascent+Refit
Utility ↑ 44.42 45.65 52.46 39.48 67.79 61.60 69.90 64.70 39.10 22.78 38.30 31.70
Forget AUC 0.518 0.502 0.507 0.510 0.585 0.523 0.542 0.549 0.568 0.478 0.522 0.510
Forget Gap ↓ 0.018 0.002 0.007 0.010 0.085 0.023 0.042 0.049 0.068 0.022 0.022 0.010
Retain AUC 0.564 0.585 0.595 0.581 0.508 0.520 0.551 0.533 0.530 0.513 0.554 0.509
Table 11: Supplementary non-DT evidence beyond the shared-budget comparison in Table 6. The upper panel summarizes exploratory LSTM and Trajectory Transformer [34] runs. The lower panel summarizes the primary IQL family under the available action-error audit. Hopper and Walker2D provide the main comparative settings. HalfCheetah is retained as a low-power reference setting. Panel
Environment
Model or method
Seeds
Utility ↑
Forget AUC
Forget Gap ↓
Exploratory sequence baselines
HalfCheetah HalfCheetah Hopper Hopper Walker2D
LSTM Trajectory Transformer LSTM Trajectory Transformer LSTM
1 1 2 2 1
32.08 1.75 8.79 1.29 63.22
0.546 0.372 0.486 0.482 0.539
0.046 0.128 0.014 0.024 0.039
IQL primary summary
HalfCheetah Hopper Hopper Hopper Hopper Walker2D Walker2D Walker2D Walker2D
IQL Base IQL Base IQL Retraining Reference IQL Naive Fine-Tuning IQL Actor-Ascent+Refit IQL Base IQL Retraining Reference IQL Naive Fine-Tuning IQL Actor-Ascent+Refit
3 3 3 3 3 3 3 3 3
39.10 44.42 45.65 52.46 39.48 67.79 61.60 69.90 64.70
0.568 0.518 0.502 0.507 0.510 0.585 0.523 0.542 0.549
0.068 0.018 0.002 0.007 0.010 0.085 0.023 0.042 0.049
The shared-training-configuration comparison in Table 6 establishes that the DT attains the largest forget gap in Hopper and HalfCheetah, whereas the ordering between the MLP and the LSTM depends on the environment and remains statistically weak. Considered together with Table 7, the current supplementary evidence suggests that the DT-family signal is not reducible to a single broad utility mismatch, although the comparison still remains descriptive rather than causal. Table 11 extends this observation in two directions. The exploratory sequence-model rows show that non-DT sequence baselines are not entirely absent: LSTM checkpoints can achieve competitive utility in isolated settings, most notably Walker2D, yet the current sequence-model coverage remains too sparse and uneven to support a benchmark family with the same maturity as the DT pipeline. Table 10 provides the detailed IQL companion summary referenced in the main text, while the IQL rows in Table 11 place that companion next to the exploratory sequence-model evidence. Collectively, these results extend the empirical coverage beyond the DT family, although they still do not justify treating the IQL results as fully score-equivalent to the DT likelihood-based pipeline. 16
C
Component-Level Diagnostic Probes
C.1
Random-Mask Controls for Component-Level Updates
The component-level results (Section 4.4) indicate that update sensitivity varies across DT components. To investigate whether these differences simply result from updating fewer parameters, random-mask controls are executed. For each targeted attention layer, three independent random parameter masks of identical size (approximately 66K parameters, or 9.1% of the model body) are generated, and the same gradient ascent protocol is applied. Table 12: Comparative analysis of random-mask controls versus attention-layer selective GA (3 seeds × 3 masks where applicable). In HalfCheetah, targeting Attn L2 produces a much smaller forget gap than size-matched random masks, although both settings have poor utility. In Hopper, the targeted Attn L1 remains more consistent than random masks in reducing membership signals. Env Target Params Steps Forget AUC Forget Gap ↓ Utility ↑
Hopper
Attn L1 Random (matched) Attn L1 Random (matched) Attn L1 Random (matched) All (uniform) All (uniform) All (uniform)
66K 66K 66K 66K 66K 66K 726K 726K 726K
100 100 250 250 500 500 100 250 500
0.500 0.449 0.499 0.423 0.495 0.420 0.480 0.469 0.446
0.002 0.054 0.002 0.078 0.005 0.080 0.020 0.031 0.054
10.99 17.27 11.79 18.90 10.28 12.31 6.88 9.93 10.86
HalfCheetah
Attn L2 Random (matched) Attn L2 Random (matched) Attn L2 Random (matched) All (uniform) All (uniform) All (uniform)
66K 66K 66K 66K 66K 66K 727K 727K 727K
100 100 250 250 500 500 100 250 500
0.537 0.311 0.497 0.299 0.463 0.295 0.349 0.301 0.224
0.037 0.189 0.003 0.201 0.037 0.205 0.151 0.199 0.276
1.56 9.20 1.60 5.88 1.41 3.12 2.26 1.97 1.59
The results in Table 12 support the existence of environment-dependent component effects. In HalfCheetah, the targeted attention layer achieves a significantly smaller forget gap than uniform GA, whereas random masks of equivalent size remain ineffective (p = 2.6 × 10−5 ). This outcome indicates a component-associated effect under the replay sweep, not a general rule for target selection. Hopper shows a different pattern: the targeted layer still improves more consistently than the sizematched random masks, but the gap between the two is much smaller, so the evidence there is better interpreted as a relative-budget success case than as equally strong mechanistic evidence. C.2
Full Component-Level GA and Feedforward-versus-Attention Comparison
Table 13 provides supplementary results for the replay component-level sweep over the uniform baseline, all attention layers jointly, and individual attention layers. In Hopper and HalfCheetah, some individual attention layers attain smaller forget gaps than the corresponding all-attention update. Walker2D is less consistent: the all-attention update is weaker than every individual layer at 100 and 500 steps, but one all-attention setting at 250 steps slightly outperforms the individual-layer rows. This finding supports the main-text interpretation that component-level gains are environment dependent rather than uniform across settings. Table 14 turns the supplementary Hopper feedforward sweep into a matched single-environment comparison against the attention-targeted sweep under the same λ = 1.0 setup and the same ascentstep budgets. Here, FFN Layer 0, FFN Layer 1, and FFN Layer 2 denote the feedforward sublayers inside the DT blocks rather than the standalone MLP baseline from Section 3.3. Feedforward-targeted updates can outperform the same-step uniform baseline once the ascent budget reaches 250 or 500 17
Table 13: Full selective GA results across environments, targets, and step counts (3-seed mean). Forget gap = |AUCf − 0.5|. Environment
Hopper
HalfCheetah
Walker2D
Target All All All All All Attn (all) Attn (all) Attn (all) Attn (all) Attn (all) Attn L0 Attn L1 Attn L1 Attn L1 Attn L1 Attn L1 Attn L2 Attn L2 Attn L2 Attn L2 Attn L2 FFN Layer 0 FFN Layer 0 FFN Layer 0 FFN Layer 1 FFN Layer 1 FFN Layer 1 FFN Layer 2 FFN Layer 2 FFN Layer 2 All All All Attn (all) Attn (all) Attn (all) Attn L1 Attn L1 Attn L1 Attn L2 Attn L2 Attn L2 All All All Attn (all) Attn (all) Attn (all) Attn L1 Attn L1 Attn L1 Attn L2 Attn L2 Attn L2
Steps 100 250 500 1000 2000 100 250 500 1000 2000 500 100 250 500 1000 2000 100 250 500 1000 2000 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500 100 250 500
Forget AUC 0.480 0.469 0.446 0.444 0.438 0.502 0.491 0.483 0.470 0.460 0.485 0.500 0.499 0.495 0.490 0.487 0.508 0.509 0.508 0.497 0.489 0.469 0.475 0.468 0.470 0.470 0.469 0.464 0.455 0.445 0.349 0.301 0.224 0.388 0.378 0.383 0.439 0.383 0.378 0.537 0.497 0.463 0.500 0.512 0.495 0.526 0.505 0.472 0.555 0.546 0.538 0.557 0.562 0.542
Forget Gap ↓ 0.020 0.031 0.054 0.056 0.062 0.005 0.009 0.017 0.030 0.040 0.015 0.002 0.002 0.005 0.010 0.016 0.009 0.009 0.009 0.008 0.011 0.031 0.025 0.032 0.030 0.030 0.031 0.036 0.045 0.055 0.151 0.199 0.276 0.112 0.122 0.117 0.061 0.117 0.122 0.037 0.003 0.037 0.013 0.012 0.015 0.026 0.009 0.028 0.055 0.046 0.038 0.057 0.062 0.042
Utility 6.9 9.9 10.9 8.3 14.2 7.4 11.4 16.2 22.6 5.8 9.2 11.0 11.8 10.3 20.4 6.4 14.0 13.1 12.7 10.6 6.7 13.5 13.9 6.2 10.4 14.9 12.2 10.4 12.0 14.0 2.3 2.0 1.6 1.2 1.3 1.2 1.2 1.3 1.3 1.6 1.6 1.4 7.3 9.1 6.7 8.6 0.2 1.2 10.2 4.2 0.6 6.6 0.3 -0.0
Retain AUC 0.540 0.531 0.529 0.533 0.533 0.553 0.541 0.545 0.546 0.546 0.547 0.576 0.550 0.545 0.547 0.543 0.571 0.561 0.556 0.555 0.549 0.552 0.535 0.539 0.561 0.546 0.542 0.588 0.583 0.574 0.439 0.439 0.451 0.474 0.480 0.496 0.489 0.478 0.481 0.560 0.537 0.514 0.445 0.445 0.446 0.461 0.448 0.433 0.477 0.479 0.471 0.477 0.473 0.465
steps. However, the best Hopper feedforward setting, FFN Layer 0 at 250 steps with forget gap 0.025, remains materially weaker than the matched attention target at the same step count, namely Attention Layer 1 with forget gap 0.002. This comparison therefore addresses the most direct target-family question supported by the current evidence: feedforward-targeted component updates are feasible in Hopper, but the strongest matched result in that environment still arises from attention targeting. C.3
Formal Utility Criteria for Component-Level Evaluation
For the component-level analysis, let U (·) denote the evaluation score, which is the D4RL normalized score for locomotion and the task success rate for AntMaze. For a component target c and the matched uniform baseline u evaluated at the same ascent budget, let πc and πu denote the corresponding updated policies. The relative utility budget is defined as ∆Urel (c | u) = U (πc ) − U (πu ) ≥ 0,
(3)
so a component target is feasible when it does not underperform the matched uniform update. To distinguish this relative criterion from strong absolute retention, the benchmark also reports retained performance ratios for an updated policy πθ′ , measured against the retraining reference πre and the 18
Attn Layer 1 Attn Layer 2 Attn (all layers) All (uniform)
0.06
Forget Gap
0.05 0.04 0.03 0.02 0.01 0.00
100
250
500
GA Steps
1000
2000
Figure 4: Hopper replay comparison between component-level and uniform GA across ascentstep budgets. The plot highlights the strongest replay-setting slice discussed in the main text and complements Table 13 with a step-by-step view of how the Hopper forget gap changes across targets. It should be interpreted as a single-environment diagnostic slice rather than as evidence for a general selective-update method. Table 14: Single-environment matched selective comparison in Hopper under the shared λ = 1.0 setup. Each row compares the same ascent-step budget across the uniform update, the best attention-targeted edit available in the selective sweep, and the best FFN-targeted edit available in the supplementary Hopper sweep. Forget gap = |AUCf − 0.5|. Steps
Uniform Uniform Forget Utility Gap
Best atten- Attention tion target Forget Gap
Attention Utility
Best FFN target
FFN Forget Gap / Utility
100
0.020
6.88
Attn L1
0.002
10.99
0.030 / 10.37
250
0.031
9.93
Attn L1
0.002
11.79
500
0.054
10.86
Attn L1
0.005
10.28
FFN Layer 1 FFN Layer 0 FFN Layer 1
ρretrain (πθ′ ) =
U (πθ′ ) , U (πre )
ρbase (πθ′ ) =
0.025 / 13.87 0.031 / 12.23
original base policy πθ : U (πθ′ ) . U (πθ )
(4)
These ratios define the explicit retained-performance criteria used throughout the component-level analysis. C.4
Utility-Constrained Component-Level Comparison
To evaluate component-level updates under explicit utility constraints, the held-out targets from Table 20 are compared against the uniform All baseline at matched ascent steps. A comparison is considered feasible if the D4RL normalized-score drop of the selected target relative to the uniform baseline remains within the specified budget. This relative utility criterion is intentionally weaker than the explicit retained-performance criteria reported in Table 4, so both views are required for interpretation. As discussed in Section 4.4 and quantified in Table 15, the replay component-level sweep provides relative-budget evidence for environment-dependent component sensitivity. In this replay slice, Hopper shows the largest improvement over the matched uniform baseline under the weaker criterion, while HalfCheetah improves the forget gap only in a low-power regime and Walker2D fails to demonstrate improvement for budgets up to 2.0. Considered together with Table 4 and the retainedutility check, these results indicate that relative success against the matched uniform baseline does not by itself imply satisfaction of the explicit retained-performance criteria. The overall evidence therefore supports the main-text interpretation that component-level updating is most useful as a diagnostic probe of environment-dependent structure rather than as a general unlearning method. 19
Table 15: Utility-constrained selective comparison against the matched All baseline. A row is feasible when the D4RL normalized-score drop of the selected target relative to the uniform baseline does not exceed the stated budget. This table reports a relative budget only. Absolute retained performance must be interpreted together with Table 4. Environment Budget Feasible / Total Improved / Total Target Hopper
0.0 0.5 1.0 2.0
9/15 9/15 10/15 10/15
9/15 9/15 10/15 10/15
Attn L1 Attn L1 Attn L1 Attn L1
HalfCheetah
0.0 0.5 1.0 2.0
2/9 6/9 7/9 9/9
2/9 6/9 7/9 9/9
Attn L2 Attn L2 Attn L2 Attn L2
Walker2D
0.0 0.5 1.0 2.0
1/9 2/9 2/9 2/9
0/9 0/9 0/9 0/9
Attn (all) Attn (all) Attn (all) Attn (all)
Rand. Layer avg. 22.5