$\text{DT}^2$: Decision-Targeted Digital Twins

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

$\text{DT}^2$: Decision-Targeted Digital Twins

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

DT2 : Decision-Targeted Digital Twins

Harry Amad 1 Mihaela van der Schaar 1

arXiv:2606.25923v1 [cs.LG] 24 Jun 2026

Abstract

with a user deliberating over DT-generated simulations of potential policies (Tao et al., 2018; Corral-Acero et al., 2020) to select the best one, or establish a preference ordering. Much of the DT literature highlights this critical use case, asserting that a useful DT is one that “aids decision making” (Wagg et al., 2020), and “[informs] decisions that realize value” (National Academy of Engineering and National Academies of Sciences, Engineering, and Medicine, 2024). To enable such decision support, DTs must generate realistic simulations that allow for user interrogation and verification, while ensuring the learned dynamics lead to correct rankings of relevant policies.

A digital twin (DT) is a virtual model of a realworld system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce DT2 , a decision-targeted DT training paradigm. Firstly, DT2 uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures. DT2 consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity.

In machine learning (ML)-based DTs, however, this goal of decision support is often overlooked. Typically, ML-based DTs involve a model that is trained to minimise a measure of one-step transition error, such as mean squared error (MSE) or negative log likelihood (NLL), across all variables and time points in a dataset of observed state-action trajectories (San et al., 2023; Kuang et al., 2024; Holt et al., 2024; Canzaniello et al., 2024; Makarov et al., 2025; Amad et al., 2025). While this can lead to high fidelity simulators, such direction-, variable-, and timepoint-agnostic objectives do not explicitly focus on correctly ranking policies, and they can misalign DTs with their primary downstream purpose by under-emphasising variables or temporal slices that are especially crucial for discriminating between policies. For example, consider a medical DT, used to help clinicians treat septic shock. Despite large data availabilities in ICU settings, clinicians typically focus on only a few key indicators to make timely decisions (Pickering et al., 2013), like blood pressure and lactate levels for sepsis. A DT trained with a typical, variable-agnostic, simulation loss considers non-critical dimensions as equivalent to such key indicator variables. By wasting model capacity on largely irrelevant features, such a DT could provide suboptimal decision support, jeopardising patient health. For a visual example of how simulation training can lead to poor decisions due to being direction-agnostic, see Figure 1.

1. Introduction A digital twin (DT) is a virtual model of a real-world system that can simulate its dynamics. DTs are appealing in many domains, such as finance (Slepneva et al., 2021), climate science (Voosen, 2020), manufacturing (Rosen et al., 2015), energy (Ismail et al., 2024), agriculture (Purcell & Neubauer, 2023), robotics (Mazumder et al., 2023), and medicine (Katsoulakis et al., 2024), due to their ability to guide decision-making processes. This typically manifests 1

Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom. Correspondence to: Harry Amad <[email protected]>.

In this work, we emphasise that, for human-in-the-loop decision support, a DT need not be a perfect replica of previously observed reality, and we contend that DT construction should move away from such context-agnostic optimisation, towards what is best for decision support.

Decision-Targeted Digital Twins True rollouts

decision-targeted training (§3).

DT rollouts

Glucose

Optimal glucose

Time

Glucose

Φ̂1

2. We propose DT2 , an architecture-agnostic decisiontargeted DT training paradigm that aligns simulations with proxy policy rankings (§4). We use a smooth approximation of Kendall’s rank correlation coefficient (Kendall, 1938) (§4.1) to enable supervision with pairwise preference targets derived from OPE policy value estimates (§4.2).

Φ̂2

3. We empirically demonstrate the alignment gap predicted by our theory (§6.1), and show that this persists even with highly expressive hypothesis spaces (§6.2). Across a suite of continuous control tasks, DT2 reduces decision regret by more than 50% at an MSE cost of only 17%, demonstrating capacity for better decision support. Furthermore, we show that this advantage extends to out-of-distribution settings, achieving better preference orderings amongst policies never seen during training (§6.3).

Time

Figure 1. We visualise two diabetes DTs, Φ̂1 (top) and Φ̂2 (bottom), under treatment plans with high, low, and adaptive insulin levels, and we consider the ‘value’ of a treatment plan as the time it spends in the optimal glucose region. Φ̂1 has lower MSE from the true rollouts, yet it leads to an incorrect treatment plan preference ordering (high ≈ low ≻ adaptive). In contrast, Φ̂2 leads to a correct preference ordering (adaptive ≻ high ≈ low), despite its higher MSE, as its rollouts traverse the correct glucose regions.

2. Formalism We now formalise the problem of decision support that we aim to address with DTs.

For this purpose, we introduce DT2 : a decision-targeted DT training paradigm that balances between seeking fidelity in general, and in those parts of the transition distribution that are most relevant for ranking candidate policies according to a known reward function. Since ground-truth rankings amongst candidate policies are not generally known a priori, we firstly estimate policy values using off-policy evaluation (OPE) methods, which can estimate scalar policy values from offline data (Fu et al., 2021) but, distinct from DTs, offer little in the way of interpretability, limiting their application in high-stakes domains (Rudin, 2019). We use these OPE estimates to construct proxy ground-truth pairwise rankings amongst candidate policies, and introduce a differentiable ranking loss function that operates alongside standard simulation objectives to encourage DTs to generate simulations that preserve these proxy rankings. By doing so, we distil the ranking ability of OPE into the interpretable structure of a DT. A tunable hyperparameter, λ, modulates the trade-off between raw simulation accuracy and reconstruction of OPE policy rankings. See code here https://github. com/harrya32/decision-targeted-dt and here https://github.com/vanderschaarlab/ decision-targeted-dt. In summary, our contributions are:

2.1. The Decision-Making Goal Let a real-world system be modelled as a Markov Decision Process (MDP) denoted by M := (X , A, Φ, r, γ). Here, X ⊆ RdX is a dX -dimensional state space, A ⊆ RdA is a dA -dimensional action space, Φ : X × A → P(X ) is the transition distribution, where P(X ) denotes the set of probability measures over X , r : X × A × X → R is a bounded reward function, and γ ∈ [0, 1) is a discount factor. At time t, given the current state xt ∈ X and action at ∈ A, the next state is sampled according to xt+1 ∼ Φ(·|xt , at ). A policy π : X → P(A) can interact with M in an initial state x0 , yielding a trajectory τ = (x0 , a0 , x1 , a1 , . . .), where at ∼ π(xt ), and we denote such τ ∼ (Φ, π). We define the value of a policy π in M as the expected sum of discounted rewards: "∞ # X t J(π) := Eτ ∼(Φ,π) γ r(xt , at , xt+1 ) . (1) t=0

In decision-making processes, a user may want to interrogate trajectories induced by each π in some candidate policy set Π, as well as establish a preference ordering over Π. This is defined by the relation ≻, where for any pair πi , πj ∈ Π, πi ≻ πj ⇐⇒ J(πi ) > J(πj ). Alternatively, a reduced goal may be to identify the optimal policy π ∗ :

1. We prove that standard DT training can be theoretically suboptimal for decision-making when the DT hypothesis space is limited, establishing motivation for

π ∗ := arg max J(π). π∈Π

(2)

Decision-Targeted Digital Twins

Generating such trajectories and determining such preference orderings with repeated experimentation on the realworld system is generally infeasible, especially in highstakes domains such as medicine and public policy.

Φ̂θ∗

2.2. Digital Twins as Practical Surrogates A DT can serve as a practical surrogate for the real system, involving a parametrised model Φ̂θ that is designed to approximate Φ. Given a policy π, the DT can simulate a trajectory from x0 , denoted τ̂ = (x̂0 , a0 , x̂1 , a1 , ...) ∼ (Φ̂θ , π). We denote the DT-estimated value of π as: "∞ # X t ˆ J(π; θ) := E γ r(x̂t , at , x̂t+1 ) , (3)

Φ̂θ′ Figure 2. When Φ ∈ / F, the Lsim -optimal DT, Φ̂θ∗ , may land in a region of low decision quality for some candidate policy set Π and reward function r (grey areas). With decision-targeted DT training, we wish to find Φ̂θ′ that resides in a better decision region (green areas), at a small simulation fidelity cost.

τ̂ ∼(Φ̂θ ,π)

t=0

ˆ which leads to a DT preference ordering defined by ≻, ˆ i ; θ) > J(π ˆ j ; θ), and a DTˆ πj ⇐⇒ J(π where πi ≻ optimal policy π̂ ∗ : ˆ θ). π̂ ∗ := arg max J(π;

Proof. See Appendix A.1. From Theorem 3.1 we can see that, when a DT cannot exactly model its target system, optimising Lsim will not generally prioritise decision-making. Given that collecting large data for certain real-world systems can be difficult, and that DTs should constantly update alongside their counterpart system (Amad et al., 2025), using lightweight model classes, such as equation-based DTs, or small neural DTs, is common, and here Theorem 3.1 can directly apply. Furthermore, even when expressive model classes are used, the complex dynamics or partial observability of real-world systems may still result in non-covering hypothesis spaces. Moreover, under a further assumption on F, we can see that better models than the Lsim -optimal DT exist for decision-making.

(4)

π∈Π

3. On the Misalignment of Conventional Digital Twin Training Conventional ML-based DT training paradigms involve a training dataset D = {(x, a, x′ )i }ni=1 of n observed transitions collected from the real-world system, or related systems. The optimal DT parameters, θ∗ , are typically found by minimising a measure of simulation error, Lsim (θ), over D. There are a few possible instantiations of Lsim (θ), with the most notable being NLL for probabilistic models: h i ′ ′ LNLL (θ) := E − log Φ̂ (x | x, a) , (5) θ (x,a,x )∼D sim

Definition 3.2. Let (x̄, ā) ∈ supp(D) and G ⊆ X be such that Φ(G | x̄, ā) ̸= Φ̂θ∗ (G | x̄, ā). Define

or MSE when focusing on the expected trajectory or assuming constant variance: ′ 2 LMSE (6) sim (θ) := E(x,a,x′ )∼D ∥x − µ̂θ (x, a)∥ ,

p := Φ(G | x̄, ā), p̂ := Φ̂θ∗ (G | x̄, ā).

Assume, w.l.o.g., that p > p̂. Then, F allows local improvement at θ∗ if there exists θ′ ∈ Θ such that

where µ̂θ (x, a) = Ex′ ∼Φ̂θ (·|x,a) [x′ ]. Such loss functions are a poor proxy for the ultimate goal of establishing preference orderings over policies. Theorem 3.1. Let Φ be the transition distribution of a realworld system, D = {(x, a, x′ )i }ni=1 be a training dataset collected from Φ, and F = {Φ̂θ | θ ∈ Θ} be a DT hypothesis class such that Φ ∈ / F and h i min E(x,a)∼D DKL (Φ(·|x, a)∥Φ̂θ (·|x, a)) > 0. (7)

p̂′ := Φ̂θ′ (G | x̄, ā) > p̂.

(10)

This definition applies to F when it contains an alternative to the Lsim -optimal model whose predictions on a local subset of transitions are closer to the true transition distribution. Theorem 3.3. Assume the conditions of Theorem 3.1 and that F allows local improvement at θ∗ . Then there exists a reward function r, a set of policies Π, and θ′ ∈ Θ \ {θ∗ } such that Φ̂θ′ induces a strictly better preference ordering on Π according to r than Φ̂θ∗ .

θ∈Θ

Let Φ̂θ∗ be the Lsim -optimal DT, i.e.

θ∗ ∈ arg min Lsim (θ).

(9)

(8)

θ∈Θ

Proof. See Appendix A.2.

Then, there exists a reward function r and a set of policies Π such that Φ̂θ∗ induces a suboptimal preference ordering on Π according to r.

Theorem 3.3 outlines our theoretical motivation to search for a better training paradigm for DTs than optimising Lsim , 3

Decision-Targeted Digital Twins

A key benefit of this formulation is that the tanh(∆ij /α) term acts as an implicit weighting mechanism. When the reference evaluator is indifferent between two policies (∆ij ≈ 0), the term tanh(∆ij /α) approaches zero, causing the gradient contribution for that pair to vanish. Consequently, the model is encouraged to ignore noise in policy separation, allocating capacity toward distinguishing between high-regret pairs. We empirically compare this formulation to alternative ranking losses in Appendix E.

that will better converge to a model that induces a good preference ordering on Π according to r (Figure 2). We elaborate on the scenarios when these theorems directly apply, and empirically validate them, in §6.1. Moreover, we hypothesise that, even when Φ ∈ F, optimising Lsim will poorly converge to an optimal decision-making model, and we show empirical evidence for this in §6.2 and §6.3.

4. DT2 : Decision-Targeted Digital Twin Training

4.2. Off-Policy Evaluation for Proxy Preference Targets

To train a decision-targeted DT we must address some practical hurdles. First, comparing a ground-truth preferˆ with standard rankence ordering ≻ to a DT-estimated ≻ ing metrics involves indicator functions, e.g. checking ˆ πj ) ∀πi , πj ∈ Π, which are diffiI(πi ≻ πj ) = I(πi ≻ cult to optimise, because they have zero gradient almost everywhere. We address this by using a smoothed ranking loss for decision-targeted training (§4.1). Second, the ground-truth ≻ is almost never available, as this is, in part, the reason for constructing a DT in the first place (§2.2). So, we derive proxy preference orderings using OPE policy value estimates to target during training (§4.2). Finally, to avoid the need to generate and backpropagate through long ˆ during training, we employ DT rollouts when optimising ≻ a value-bootstrapped scheme (§4.3). This forms DT2 , our method for DT training that focuses on modelling dynamics that are critical for decision support.

Since ground-truth policy values J(π) are generally inaccessible in the offline setting we operate in, we derive proxy ∆ij targets using principles from OPE (§5.4). Specifically, we use fitted Q-evaluation (FQE) (Le et al., 2019) to estimate the action-value function Qπ (x, a) for each π ∈ Π. Qπ (x, a) denotes the expected long-term return of executing action a in state x and following π thereafter: "∞ # X π t Q (x, a) := Eτ ∼(Φ,π) γ rt x0 = x, a0 = a . (13) t=0

FQE approximates this with a parameterised function Qπψ trained on D to minimise the expected Bellman error: 2 π π ′ ′ LFQE (ψ) := ED Qψ (x, a) − r + γQψ̄ (x , π(x )) (14) where Qπψ̄ is a target network. The parameters ψ̄ are frozen copies of ψ that are updated periodically to stabilise the bootstrapping target. Once converged, the scalar value of π can be estimated by averaging Qπψ over the initial states,

4.1. Smoothed Ranking Loss We define a ranking objective based on a smooth approximation of Kendall’s rank correlation coefficient (Kendall, 1938). Let ∆ij := J(πi ) − J(πj ) be the ground-truth ˆ ij (θ) := value difference between two policies, and ∆ ˆ ˆ J(πi ; θ) − J(πj ; θ) be the corresponding DT-estimated value difference. Kendall’s correlation measures the concordance of signs between these differences: 1 X ˆ ij (θ) , sign (∆ij ) · sign ∆ (11) |C|

JFQE (π) ≈ Ex0 ∼D, a∼π(x0 ) [Qπψ (x0 , a)].

(15)

These estimates allow us to construct the proxy pairwise rankings targets ∆FQE ij = JFQE (πi ) − JFQE (πj ) that are used in place of ∆ij in Eq. 12. We select FQE amongst the plethora of OPE methods given its simplicity and strong performance on OPE benchmarks (Fu et al., 2021; Voloshin et al., 2021). Furthermore, while FQE can be prone to systematic bias, particularly for outof-distribution policies, these errors will tend to cancel out during pairwise comparisons. This makes FQE-based rankings potentially more robust than its scalar value estimates, as relative orderings can be preserved even when absolute value magnitudes are biased.

(i,j)∈C

where C is the set of pairs in a candidate policy set Π. To make this suitable for gradient-based optimisation, we replace the sign functions with hyperbolic tangents, and set the complement as our ranking loss to be minimised: ! ˆ ij (θ) 1 X ∆ij ∆ Lrank (θ) := 1 − tanh · tanh |C| α α (i,j)∈C

4.3. Bootstrapping Digital Twin Value Estimates

(12) where the temperature α governs the trade-off between faithfulness to the discrete Kendall correlation and gradient smoothness. Smaller α produces sharper, more sign-like curves, while larger α yields smoother gradients that can lead to more stable optimisation.

ˆ ij (θ) for Eq. 12 requires Computing the DT-estimated ∆ simulating trajectories τ̂ ∼ (Φ̂θ , π) (Eq. 3). However, backpropagating through long rollouts to optimise Lrank is computationally expensive and prone to vanishing or exploding gradients (Bengio et al., 1994; Pascanu et al., 2013). 4

Decision-Targeted Digital Twins

5.2. Model-Based Reinforcement Learning

To mitigate this, we employ a bootstrapping scheme that truncates DT rollouts at a fixed horizon H and approximates the remaining return by re-using a frozen Qπψ trained with FQE. The bootstrapped DT-value estimate for policy π is defined as: "H−1 # X t H π ˆ JH (π; θ) := E γ rt + γ Qψ (x̂H , aH ) .

A DT is similar to a world model in model-based RL (MBRL) (Sutton, 1991), which is used to learn new policies. Some MBRL research has a similar motivation to ours, recognising that global model accuracy is unnecessary, and that optimising for this can be suboptimal for policy learning (Lambert et al., 2020; Grimm et al., 2020). Proposed solutions include policy-aware methods (Eysenbach et al., 2022; Wang et al., 2023; Ma et al., 2023) that favour areas of the state-action space that are traversed by the current policy, and value-aware methods (Farahmand et al., 2017; Farahmand, 2018; Grimm et al., 2020; Farquhar et al., 2021; Voelcker et al., 2022), which are more similar to DT2 , focusing model capacity in areas important to determine the current policy value. Nevertheless, these works operate in the online setting, where the true environment can be queried during training, whereas we focus on offline DT construction.

τ̂ ∼(Φ̂θ ,π)

t=0

(16) ˆ ij (θ), we reduce the cost of By using JˆH to compute ∆ calculating Lrank , and improve its gradient stability.

To ensure DT2 still permits generation of high fidelity trajectories, which is important for the interpretability aspects of human-in-the-loop decision support, the ultimate training objective combines Lrank with Lsim : LDT2 (θ) := (1 − λ)Lsim (θ) + λLrank (θ).

(17)

The hyperparameter λ ∈ [0, 1] modulates the trade-off between optimising for raw simulation fidelity and decisiontargeting. λ → 0 recovers standard DT training, while λ → 1 yields a purely decision-targeted model. We investigate its effects in §6.4. By shaping the DT loss landscape 2 with ∆FQE ij , DT distils the decision-making power of ‘blackbox’ OPE methods into the interpretable structure of a DT.

There are also similar works in offline MBRL which forego global model accuracy, typically to encode so-called ‘pessimistic’ dynamics to discourage the learned policy from traversing parts of the state-action space not covered by the offline dataset (Kidambi et al., 2020; Yu et al., 2020; Qiao et al., 2026). While useful for policy learning, such alterations to the learned model do not improve policy ranking abilities to the same extent as DT2 (§6.2). Furthermore, since the models in MBRL are a means to learning a performant policy, maintaining simulation fidelity, for interpretability purposes, is not a priority, whereas this is essential for human-in-the-loop decision support with DTs.

5. Related Works 5.1. Digital Twins 1960s aerospace engineering pioneered the use of DTs (Allen, 2021), where physical laws can fully describe system dynamics, but manually-defining such DTs requires significant domain expertise. Methods like SINDy (Brunton et al., 2016) and PySR (Cranmer, 2023) aim to automate the discovery of such mechanistic models, but they are often outperformed by more expressive deep learning approaches.

5.3. Decision-Focused Learning Our work is conceptually related to the field of decisionfocused learning (DFL), which studies training predictive models whose outputs are used in downstream optimisation processes, with the objective of directly improving decision quality. Areas such as stochastic programming (Donti et al., 2017) and combinatorial optimisation (Wilder et al., 2019) have received attention in DFL, and extensions to learning models in MDPs have also been proposed (Futoma et al., 2020; Wang et al., 2021; Nikishin et al., 2022; Sharma et al., 2024). Generally, DFL requires some way to differentiate through the downstream optimisation process, which can limit the complexity of the models or policies considered in these works. Our method has minimal restrictions for the policies that can be considered in Π, and the DT model class, and we demonstrate good performance in a variety of complex, high-dimensional environments (§6.2). Furthermore, these works are largely designed to learn a model such that the optimal policy within it has high value, whereas we uniquely aim to learn a model such that policies from a candidate set are well ranked by it.

Deep learning DTs approximate system dynamics from longitudinal data using a variety of model architectures, including standard MLP approaches (San et al., 2023; Holt et al., 2024), as well as tailored sequential models like RNNs (Elman, 1990; Canzaniello et al., 2024), transformers (Vaswani et al., 2017), and pre-trained (Amad et al., 2025) or finetuned large language models (Makarov et al., 2025). Neural ODEs (Chen et al., 2018; Dupont et al., 2019; Alvarez et al., 2020) further extend deep learning capabilities to modelling continuous-time dynamics. Hybrid approaches (Raissi et al., 2019; Qian et al., 2021; Takeishi & Kalousis, 2021; Kuang et al., 2024) combine deep learning with mechanistic components, to improve sample efficiency and generalisation. Most existing DT frameworks seek to minimise transition errors across a training set, and, to our knowledge, no previous works optimise explicitly for decision-making over candidate policies. 5

Decision-Targeted Digital Twins

5.4. Off-Policy Evaluation

one-dimensional representation prevents any model from simultaneously representing the independent evolution of xd,t and xc,t , guaranteeing Φ ∈ / F.

OPE methods are used to estimate J(πtarget ) from a dataset collected from some behavioural policy πbehavioural . Popular approaches include direct methods (Lagoudakis & Parr, 2003; Ernst et al., 2005; Munos & Szepesvári, 2008; Le et al., 2019), which learn and use Qπtarget to estimate J(πtarget ), importance sampling methods (Precup et al., 2000; Hallak & Mannor, 2017; Liu et al., 2018; Xie et al., 2019; Nachum et al., 2019; Uehara et al., 2020; Zhang et al., 2020a;b), which use importance weights to correct for the difference in densities between πbehavioural and πtarget to estimate J(πtarget ), and doubly robust methods (Jiang & Li, 2016; Thomas & Brunskill, 2016), which combine direct and importance sampling methods.

We consider Π with two constant-action policies, πhigh = 1.0 and πlow = 0.9, and a reward r(xt , at , xt+1 ) = xc,t such that πhigh ≻ πlow . Across 20 seeds, we generate D by drawing at ∼ U [−1, 1], and train DTs with LMSE sim and LDT2 . Because xd,t dominates the variance in D, the LMSE sim trained DT allocates it significant capacity, and consequently achieves only a 20% success rate across seeds in identifying πhigh as the optimal policy via model-estimated policy values. In contrast, LDT2 encourages the bottlenecked model to identify that xc drives value differences, despite its small magnitude, resulting in 100% ranking accuracy.

While OPE methods can be directly used for decision support, a primary limitation is their lack of interpretability and the difficulty in verifying their value estimates. DTs offer several advantages over such methods because they generate full simulations of each policy under consideration. Inspecting these simulations allows for the consideration of subjective criteria in decision-making, that may not be perfectly captured by a formally defined reward function. Furthermore, verification of DT value estimates is easier than for OPE methods, as if a simulation violates known constraints a user can choose to discard it.

6.1.2. M ISSPECIFIED M ECHANISTIC M ODEL We now consider a mechanistic setting, where the assumed functional form of the dynamics is incorrect. The true onedimensional system has cubic action-dependent dynamics, xt+1 = xt + at − a3t + ξ, where ξ ∼ N (0, 0.1), and we set F as the family of linear models, xt+1 = xt + βat , such that Φ ∈ / F. We collect D by drawing at ∼ U [−1.5, 1.5]. We consider Π containing three constant-action policies π1 = −0.58, π2 = 0, π3 = +0.58, and a reward r(xt , at , xt+1 ) = xt such that π3 ≻ π2 ≻ π1 . We show the mechanistic models learnt by optimising LMSE sim and LDT2 , and the true dynamics in Figure 3. For the action range in D, the cubic term −a3t tends to dominate, creating a broadly negative correlation between action and state change δ = xt+1 −xt , and the LMSE sim -trained DT learns this negative ˆ 2 ≻π ˆ 3 . DT2 , on the other slope, consequently ordering π1 ≻π hand, recognises that, in the local decision-critical region for Π, the linear term dominates, sacrificing global fit to learn a positive slope, recovering the true preference ordering.

6. Empirical Investigation We now empirically validate the efficacy of DT2 . We cover the limited model capacity setting that relates to Theorems 3.1 and 3.3 in §6.1, before showing that their takeaways hold with expressive models as well in §6.2 and §6.3. Finally, we investigate the effects of λ in LDT2 in §6.4. We provide detailed experimental set-ups in Appendix B. 6.1. Restricted Hypothesis Spaces

To empirically demonstrate Theorems 3.1 and 3.3, we construct three toy scenarios which satisfy the required assumptions, where Φ ∈ / F and F allows local improvement at θ∗ . N.B. in the following, we assume access to ground-truth policy values during DT2 training, for simplicity. In the more realistic settings, from §6.2 onward, we demonstrate the full DT2 pipeline, with FQE-derived proxy rankings.

True: δ = a − a3 DT: δ̂ = −0.36a DT2 : δ̂ = 0.80a

1 0 −1 −2 −2

6.1.1. L IMITED N EURAL C APACITY

−1

Figure 3. State delta across action values, as determined by mechanistic models learnt via LMSE sim and LDT2 , compared to the true Φ. Black dots represent constant-action policies in Π.

Consider a two-dimensional system xt = [xd,t , xc,t ]⊤ that decomposes into independent ‘decoy’ and ‘critical’ components, with an associated scalar action at . xt evolves according to xd,t+1 = M sin(xd,t ) and xc,t+1 = xc,t + ϵat , and we set M and ϵ such that Var(xd,t ) >> Var(xc,t ). We set F as the family of three layer MLPs with a singleneuron bottleneck as the middle layer. Forcing an internal

6.1.3. PARTIALLY O BSERVED DYNAMICS Finally, we investigate a system with partially observable dynamics, where an unobserved latent variable zt dictates 6

Decision-Targeted Digital Twins Pendulum

Lunarlander

6 5

1.00

0.75 0.50 0.25

Hopper 3

Regret

1.25

Regret

1.50

3 2

0.00

0.6 0.4 0.2 0.0

0.8

Spearman

1.0 0.8

0.6 0.4 0.2 0.0

Walker

Cheetah

4 2

0.8

0.6 0.4 0.2

DT DT2

Transformer

DT DT2

GRU

DT DT2

MLP

DT DT2

ResNet

DT DT2

NODE

Spearman

0.0

Spearman

5.0 2.5

0.0

7.5

Regret

Ant

10.0

0.5

0.0

−0.5

DT DT2

Transformer

DT DT2

GRU

DT DT2

MLP

DT DT2

ResNet

DT DT2

NODE

0.5

0.0 −0.5

DT DT2

Transformer

DT DT2

GRU

DT DT2

MLP

DT DT2

ResNet

DT DT2

NODE

Figure 4. Regret and Spearman’s correlation for preference orderings from LNLL sim (hashed) and LDT2 (solid) DTs across base architectures in six continuous control environments. We report averages over 10 seeds, with 95% CIs.

the transition distribution, ensuring Φ ∈ / F even with an expressive hypothesis space of three-layer MLPs. We consider a one-dimensional system governed by xt+1 = xt + at + zt , where zt ∼ N (0, 25) dominates, and set Π with two constant-action policies, πhigh = 0.01 and πlow = −0.01, and a reward r(xt , at , xt+1 ) = xt such that πhigh ≻ πlow . Across 20 seeds, we collect D by drawing at ∼ U [−1, 1].

then generate D by deploying each π ∈ Π in Φ for a fixed number of steps. To determine the ground-truth preference ordering over Π, we empirically estimate J(π) for each π ∈ Π using Monte Carlo rollouts in Φ. Firstly, we compare LNLL sim -trained DTs, represents the prevailing context-agnostic approach, with training via DT2 . For DT2 , we first learn a Qπψ for each π ∈ Π using FQE on D, and then train the DT with λ = 0.1 and α = 1. To show the generality of our method, we use a number of expressive architectures to form the basis of the DTs, training them to output the mean and variance of a multivariate Gaussian distribution for the next state. These include a transformer (Vaswani et al., 2017), gated recurrent unit (GRU) (Cho et al., 2014), multi-layer perceptron (MLP), residual network (ResNet) (He et al., 2016), and neural ODE (NODE) (Chen et al., 2018).

Minimising LMSE sim encourages learning the conditional expectation E[xt+1 | xt , at ] by averaging over the unobserved latent zt . While at and zt are independent in the generative process, the high variance of zt can cause, with finite data, spurious correlations to emerge where actions coincide with opposing latent values. Consequently, the empirical conditional mean can exhibit a slope flip, causing the LMSE sim model to learn a negative relationship between at and xt+1 to fit the noise in D. As such, it derives the correct preference ordering on Π only 65% of the time. In contrast, DT2 utilises the ranking loss to distinguish πhigh from πlow , better conserving the true positive slope of the action, achieving 100% ranking accuracy.

ˆ we To determine the DT-derived preference orderings, ≻, also use Monte Carlo rollouts. To evaluate these against the ground-truth ≻, we report the Spearman’s rank correlation (Spearman, 1904), to measure global ranking alignment, and the decision regret when selecting the optimal policy, to measure the value gap between π ∗ and π̂ ∗ , in Figure 4. LDT2 DTs outperform their LNLL sim counterpart in 83.3% of the architecture-environment-metric couplings, and there is no environment or architecture that DT2 does not outperform in, on average. The average regret, across all environments and architectures, is 4.08 using LNLL sim , and 1.88 using LDT2 , corresponding to a 54% reduction. Similarly, for Spearman’s correlation, LNLL sim DTs average 0.45 while LDT2 DTs average 0.66, corresponding to a 47% improvement.

6.2. Expressive Hypothesis Spaces We now wish to test our hypothesis that the takeaways from Theorems 3.1 and 3.3 persist even when F is expressive and could contain Φ. We use six Mujoco (Todorov et al., 2012) and Gymnasium (Towers et al., 2025) continuous control environments (Pendulum, Lunarlander, Hopper, Walker, Cheetah, Ant) to act as ‘real-world systems’ Φ that span a range of complexities, from 4- to 35-dimensional state-action spaces. For each environment, we define a candidate set Π of six policies taken from evenly-spaced checkpoints of a PPO (Schulman et al., 2017) training run. We

In terms of simulation fidelity, we do see that there is a 7

Decision-Targeted Digital Twins Table 1. Regret and Spearman’s correlation for preference orderings from different model learning paradigms in six continuous control environments. For LDT2 and LNLL sim we report the best-performing architectures from Figure 4 for each environment. We report averages over 10 seeds, with standard errors. Red indicates best performing, blue indicates second best. Pendulum

Lunarlander

Hopper

Walker

Cheetah

Ant

Method

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

LNLL sim MOReL MOPO ROMI VaGraM HDTwin FQE LDT2

0.08 (0.07) 0.14 (0.05) 0.36 (0.26) 0.32 (0.27) 1.18 (0.79) 13.85 (6.55) 0.02 (0.01) 0.03 (0.02)

0.79 (0.11) 0.69 (0.09) 0.47 (0.06) 0.49 (0.12) 0.50 (0.13) −0.10 (0.21) 0.76 (0.03) 0.72 (0.06)

1.41 (0.72) 21.06 (9.77) 1.45 (0.52) 8.50 (1.34) 25.47 (15.28) 49.10 (39.01) 2.51 (1.85) 0.45 (0.45)

0.98 (0.01) 0.73 (0.03) 0.62 (0.09) 0.80 (0.02) 0.84 (0.06) 0.54 (0.18) 0.83 (0.04) 0.97 (0.02)

0.59 (0.45) 0.51 (0.17) 8.50 (4.19) 0.64 (0.11) 0.65 (0.15) 0.94 (0.49) 0.54 (0.27) 0.60 (0.16)

0.64 (0.06) 0.57 (0.09) −0.27 (0.12) 0.60 (0.10) 0.46 (0.08) 0.40 (0.11) 0.78 (0.05) 0.66 (0.04)

0.55 (0.30) 0.35 (0.06) 8.13 (2.31) 0.25 (0.07) 4.63 (2.28) 7.50 (1.15) 1.18 (0.66) 0.21 (0.08)

0.83 (0.01) 0.87 (0.02) 0.21 (0.15) 0.84 (0.03) 0.43 (0.25) −0.13 (0.16) 0.78 (0.07) 0.91 (0.01)

2.77 (1.28) 5.08 (1.20) 9.57 (1.28) 13.45 (0.34) 6.62 (2.00) 5.32 (2.04) 1.29 (0.30) 0.55 (0.16)

0.48 (0.16) −0.29 (0.12) −0.51 (0.12) −0.87 (0.05) −0.26 (0.32) 0.16 (0.23) 0.84 (0.03) 0.90 (0.02)

7.10 (1.56) 5.12 (1.23) 11.98 (1.38) 6.14 (2.45) 12.50 (0.89) 6.18 (2.07) 0 (0) 1.64 (0.68)

−0.09 (0.16) 0.22 (0.08) 0.05 (0.13) 0.33 (0.16) −0.38 (0.11) 0.24 (0.15) 0.94 (0) 0.84 (0.04)

necessary trade-off to achieve this increased preference ordering performance. The average test set transition MSE using LNLL sim is 1.21, while for LDT2 it is 1.41, performing 17% worse in this regard. These results validate our core thesis: models optimised solely for simulation fidelity often misallocate capacity to dynamics irrelevant for policy ranking, even when using expressive hypothesis classes. Through explicit decision-targeted training, DT2 recovers significantly better preference orderings, at a modest MSE cost.

ponents), to find the architecture that minimises validation MSE. These results are reported in Table 1. We see that DT2 tends to outperform these baselines, placing amongst the top two methods, in terms of both regret and Spearman’s, in the majority of environments. FQE is the most competitive of the baselines, which validates the motivation for our method—to use the ranking abilities of OPE to supervise DT training—but, interestingly, we even see that DT2 improves upon its ranking supervisor in some environments.

Secondly, we compare the preference orderings from the best performing LDT2 and LNLL sim models from Figure 4 with a handful of baseline models, including MBRL baselines, a recent DT baseline, and pure FQE rankings. The MBRL methods include MOReL (Kidambi et al., 2020) and MOPO (Yu et al., 2020), which both train neural ensembles of dynamics models and construct a ‘pessimistic MDP’ that penalises rewards in transitions where there is high model uncertainty, i.e. they redefine the reward as r̃(xt , at , xt+1 ) = r(xt , at , xt+1 ) − λ u(xt , at )

While the MBRL methods go beyond optimising for raw simulation fidelity in ways that are effective for policy learning, we see that, generally, the preference orderings from these methods are about as competitive as the LNLL sim DTs. Clearly, bespoke alterations, such as via our DT2 method, are required to optimise dynamics models for the task of policy selection. 6.3. Case Study – Cancer Treatment Digital Twin

(18)

We now investigate a medical case study, where Π is comprised of human-defined treatment plans. We use a Cancer environment from DTR-Bench (Luo et al., 2024) that simulates the progression of cancer with metastasis under chemotherapy and radiotherapy, based on the mathematical model of (Ghaffari et al., 2016), with a 9-dimensional state-action space and a reward function that balances tumour reduction against treatment toxicity. We define five distinct treatment plans to act as the candidate policy set: 1) no treatment, 2) fractionated radiotherapy, 3) metronomic chemotherapy, 4) adaptive combined therapy, and 5) aggressive combined therapy. We pursue a similar training and evaluation pipeline as in §6.2, using the ResNet architecture as the backbone for the LNLL sim and LDT2 DTs, since it tended to perform well in the previous environments.

for some measure of model uncertainty u. We also compare with a more recent offline MBRL work, ROMI (Qiao et al., 2026), which alternates between training an ensemble of neural dynamics models and a SAC policy (Haarnoja et al., 2018), weighting the dynamics training using a value-aware approach to incorporate pessimism adversarially, rather than based on uncertainty. Finally, we adapt VaGraM (Voelcker et al., 2022), an online value-aware MBRL method, to be suitable for our offline setting, training using an MSE loss weighted by the average squared gradient norm of all Qπψ functions: " # 1 X ′ 2 π ′ ′ 2 ED ∥x − µθ (x, a)∥ · ∥∇x′ Qψ (x , π(x ))∥ . |Π| π∈Π (19) This weights the importance of transitions by how sensitive candidate policy returns are to errors in those regions.

We report the results in Table 2. Once again, using LDT2 leads to a significantly better regret and Spearman’s rank correlation than LNLL sim , indicating substantially better potential for clinical decision support. This is heightened by the fact that the MSE cost is only 2% here, meaning that decision-targeted training would minimally impact the result

For the DT baseline, we compare with HDTwin (Holt et al., 2024), which performs an LLM-guided search process over hybrid DT structures (involving mechanistic and neural com8

Decision-Targeted Digital Twins

LNLL sim LDT2

59.56 (0.83) 26.96 (11.04)

Spearman (↑) 0.10 (0.08) 0.64 (0.10)

MSE (↓)

95.88 (18.23) 49.27 (17.63)

0.40 (0.04) 0.50 (0.06)

LDT2

0.26 0.24

0.6

0.22

0.4

0.20

0.2 0.0

0.18 0

0.1

0.25

0.5

0.8

0.90

0.006

0.6

0.85

0.004

0.4

0.75

0.0275

0.80

0.002

0.2

0.75

0.000

0.0

0.75

1.00

0.70

0.75

0.36

0.65

0.50

0.34

0.60

0.25

0.32

0.55

0.00

0.0250 0.0225 0.0200

Cheetah

1.0

0.8

0.8 0.6 0.4 0.2 0.0

0.0175 0.0150 0.0125

Ant

0.30

0.50 −0.25 0

0.1

0.25

0.5

0.75

0.45 −0.50

0.28 0

0.1

0.25

0.5

0.75

Figure 5. Spearman’s correlation and test MSE for LDT2 DTs in six continuous control environments across λ levels. Averaged over 5 seeds, with 95% CIs.

7. Discussion This work challenges the prevailing practice of maximising simulation fidelity in DT construction. We introduced DT2 , a training paradigm that aligns DTs with their primary downstream purpose: decision support. By distilling FQE-derived policy rankings into DT models, DT2 induces significantly better preference orderings over candidate policies at a modest simulation fidelity cost. We demonstrate that these benefits hold across diverse environments, architectures and simulation losses (see further results using LMSE sim as the simulation loss in Appendix D), and that the trade-off between decision-making and simulation fidelity can be navigated via λ.

Table 3. Regret and Spearman’s correlation for preference orderings from LNLL sim and LDT2 DTs in Cancer environment across unseen policies. Averaged over 10 seeds with standard errors.

Spearman (↑)

Walker

1.0

Furthermore, we test preference orderings across 11 policies that are unseen during training. These include six PPO checkpoints, obtained as in §6.2, and five further manuallydefined policies: 1) pulsed chemotherapy, 2) hypofractionated radiotherapy, 3) aggressive combined therapy followed by maintenance chemo, 4) alternating modality treatment, and 5) dose-escalating combined treatment. In Table 3 we see that DT2 performs best again, indicating that decisiontargeting is not overfitting to the policies observed during training, but encourages learning dynamics that assist in general decision-making according to r. For some further analysis on unseen policy results across more environments, see Appendix C.

Regret (↓)

10−6

0.0

of any human interrogation of DT-generated simulations in this case.

LNLL sim

1.0

0.008

−5

0.4 0.2

0.329 (0.01) 0.334 (0.01)

Loss

0.6

Hopper

0.010

0.95

MSE

Spearman

Regret (↓)

Lunarlander

1.00 10−4

0.8

Spearman

Loss

Pendulum

1.0

MSE

Table 2. Regret and Spearman’s correlation for preference orderings from LNLL sim and LDT2 DTs in Cancer environment. Averaged over 5 seeds with standard errors.

7.1. Limitations DT2 does have several limitations that are worth noting, and that can form the basis for various future works. The effect that DT2 training has on the ranking ability of the resulting DT naturally depends on the quality of the proxy ranking targets used. Biased or high-variance FQE value estimates can occur if the state-visitation distributions of the candidate policies are poorly covered by D, and these may be replicated by the trained DT. To address this, future work could focus on improving the general efficacy or sample-efficiency of OPE methods, or investigating some uncertainty-aware training extensions to DT2 , to neglect the signal from Lrank when the proxy values are likely poor. Additionally, DT2 introduces computational overhead compared to standard training, requiring pre-trained Q-functions and H-step model rollouts during training. Potential strategies to mitigate this could involve using only a small representative subset of policies in Π during training, or using adaptive bootstrapping horizons.

6.4. The Effect of λ in LDT 2 A key choice in LDT2 is setting λ, which dictates where the model is placed on the trade-off between preserving the FQE-derived policy ranks and optimising for simulation fidelity. We investigate the performance of LDT2 DTs across a range of values λ ∈ [0, 0.75] in the six continuous control environments from §6.2, using the same set-ups and, again, using the ResNet architecture for each DT. Figure 5 plots the Spearman’s correlation and test MSE obtained across these λ settings (N.B. that the λ = 0 setting is equivalent to standard training with LNLL sim only). As expected, we observe a consistent increase in ranking performance and decrease in simulation fidelity as λ increases. Practitioners can therefore effectively use λ to navigate to their desired point along this trade-off during training. However, we do see that ranking performance experiences diminishing returns at larger λ values, and we experienced training instability with λ ≈ 1, so we generally suggest setting a small λ > 0 for most use cases.

Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal 9

Decision-Targeted Digital Twins

consequences of our work, none of which we feel must be specifically highlighted here.

Cho, K., Van Merriënboer, B., Gulçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1724–1734, 2014.

Acknowledgements Harry Amad’s studentship is funded by Canon Inc. This work was supported by Azure sponsorship credits granted by Microsoft’s AI for Good Research Lab.

References Allen, B. D. Digital twins and living models at NASA. In Digital Twin Summit, 2021. URL https://ntrs. nasa.gov/citations/20210023699. Alvarez, V. M. M., Roşca, R., and Fălcuţescu, C. G. Dynode: Neural ordinary differential equations for dynamics modeling in continuous control. arXiv preprint arXiv:2009.04278, 2020. URL https://arxiv. org/abs/2009.04278. Amad, H., Astorga, N., and van der Schaar, M. Continuously updating digital twins using large language models. In International Conference on Machine Learning. PMLR, 2025. URL https://openreview.net/ pdf?id=jk3TjZAHem.

Corral-Acero, J., Margara, F., Marciniak, M., Rodero, C., Loncaric, F., Feng, Y., Gilbert, A., Fernandes, J. F., Bukhari, H. A., Wajdan, A., Martinez, M. V., Santos, M. S., Shamohammdi, M., Luo, H., Westphal, P., Leeson, P., DiAchille, P., Gurev, V., Mayr, M., Geris, L., Pathmanathan, P., Morrison, T., Cornelussen, R., Prinzen, F., Delhaas, T., Doltra, A., Sitges, M., Vigmond, E. J., Zacur, E., Grau, V., Rodriguez, B., Remme, E. W., Niederer, S., Mortier, P., McLeod, K., Potse, M., Pueyo, E., Bueno-Orovio, A., and Lamata, P. The ‘digital twin’ to enable the vision of precision cardiology. European Heart Journal, 41(48):4556–4564, 03 2020. ISSN 0195668X. doi: 10.1093/eurheartj/ehaa159. URL https: //doi.org/10.1093/eurheartj/ehaa159. Cranmer, M. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582, 2023.

Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994.

Donti, P., Amos, B., and Kolter, J. Z. Task-based end-to-end model learning in stochastic optimization. Advances in neural information processing systems, 30, 2017.

Brunton, S. L., Proctor, J. L., and Kutz, J. N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016. doi: 10.1073/pnas. 1517384113. URL https://www.pnas.org/doi/ abs/10.1073/pnas.1517384113.

Dupont, E., Doucet, A., and Teh, Y. W. Augmented neural odes. Advances in neural information processing systems, 32, 2019. URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 21be9a4bd4f81549a9d1d241981cec3c-Paper. pdf. Elman, J. L. Finding structure in time. Cognitive Science, 14(2):179–211, 1990. ISSN 0364-0213. doi: https://doi.org/10.1016/0364-0213(90)90002-E. URL https://www.sciencedirect.com/ science/article/pii/036402139090002E.

Canzaniello, M., Amitrano, S., Prezioso, E., Giampaolo, F., Cuomo, S., and Piccialli, F. Leveraging digital twins and generative ai for effective urban mobility management. In 2024 IEEE Cyber Science and Technology Congress (CyberSciTech), pp. 146–153. IEEE, 2024.

Ernst, D., Geurts, P., and Wehenkel, L. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.

Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., and Li, H. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129–136, 2007.

Eysenbach, B., Khazatsky, A., Levine, S., and SalakhutChen, R. T., Rubanova, Y., Bettencourt, J., and Duvedinov, R. R. Mismatched no more: Joint model-policy naud, D. K. Neural ordinary differential equations. optimization for model-based rl. Advances in Neural Advances in neural information processing systems, 31, Information Processing Systems, 35:23230–23243, 2022. 2018. URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ Farahmand, A.-m. Iterative value-aware model learning. 69386f6bb1dfed68692a24c8686939b9-Paper. Advances in Neural Information Processing Systems, 31, pdf. 2018. 10

Decision-Targeted Digital Twins

Farahmand, A.-m., Barreto, A., and Nikovski, D. Valueaware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics, pp. 1486– 1494. PMLR, 2017.

Jiang, N. and Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In International conference on machine learning, pp. 652–661. PMLR, 2016. Katsoulakis, E., Wang, Q., Wu, H., Shahriyari, L., Fletcher, R., Liu, J., Achenie, L., Liu, H., Jackson, P., Xiao, Y., et al. Digital twins for health: a scoping review. NPJ Digital Medicine, 7(1):77, 2024. URL https://doi. org/10.1038/s41746-024-01073-0.

Farquhar, G., Baumli, K., Marinho, Z., Filos, A., Hessel, M., van Hasselt, H. P., and Silver, D. Self-consistent models and values. Advances in Neural Information Processing Systems, 34:1111–1125, 2021.

Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1-2):81–93, 1938.

Fu, J., Norouzi, M., Nachum, O., Tucker, G., Novikov, A., Yang, M., Zhang, M. R., Chen, Y., Kumar, A., Paduraru, C., et al. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.

Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33: 21810–21823, 2020.

Futoma, J., Hughes, M., and Doshi-Velez, F. Popcorn: Partially observed prediction constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 3578–3588. PMLR, 2020.

Kuang, K., Dean, F., Jedlicki, J. B., Ouyang, D., Philippakis, A., Sontag, D., and Alaa, A. Med-real2sim: Noninvasive medical digital twins using physics-informed self-supervised learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/pdf? id=xCUXJqQySD.

Ghaffari, A., Bahmaie, B., and Nazari, M. A mixed radiotherapy and chemotherapy model for treatment of cancer with metastasis. Mathematical methods in the applied sciences, 39(15):4603–4617, 2016.

Lagoudakis, M. G. and Parr, R. Least-squares policy iteration. Journal of machine learning research, 4(Dec): 1107–1149, 2003.

Grimm, C., Barreto, A., Singh, S., and Silver, D. The value equivalence principle for model-based reinforcement learning. Advances in neural information processing systems, 33:5541–5552, 2020.

Lambert, N., Amos, B., Yadan, O., and Calandra, R. Objective mismatch in model-based reinforcement learning. Proceedings of Machine Learning Research vol, 120:1– 15, 2020.

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. Pmlr, 2018.

Le, H., Voloshin, C., and Yue, Y. Batch policy learning under constraints. In International Conference on Machine Learning, pp. 3703–3712. PMLR, 2019.

Hallak, A. and Mannor, S. Consistent on-line off-policy evaluation. In International Conference on Machine Learning, pp. 1372–1383. PMLR, 2017.

Liu, Q., Li, L., Tang, Z., and Zhou, D. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in neural information processing systems, 31, 2018.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.

Luo, Z., Zhu, M., Liu, F., Li, J., Pan, Y., Zhou, J., and Zhu, T. Dtr-bench: An in silico environment and benchmark platform for reinforcement learning based dynamic treatment regime. CoRR, 2024.

Holt, S., Liu, T., and van der Schaar, M. Automatically learning hybrid digital twins of dynamical systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.net/pdf?id=SOsiObSdU2.

Ma, Y. J., Sivakumar, K., Yan, J., Bastani, O., and Jayaraman, D. Learning policy-aware models for model-based reinforcement learning via transition occupancy matching. In Learning for Dynamics and Control Conference, pp. 259–271. PMLR, 2023.

Ismail, F. B., Al-Faiz, H., Hasini, H., Al-Bazi, A., and Kazem, H. A. A comprehensive review of the dynamic applications of the digital twin technology across diverse energy sectors. Energy Strategy Reviews, 52:101334, 2024.

Makarov, N., Bordukova, M., Quengdaeng, P., Garger, D., Rodriguez-Esteban, R., Schmich, F., and Menden, M. P. Large language models forecast patient health trajectories enabling digital twins. npj Digital Medicine, 8(1):588, 2025. 11

Decision-Targeted Digital Twins

Mazumder, A., Sahed, M. F., Tasneem, Z., Das, P., Badal, F. R., Ali, M. F., Ahamed, M. H., Abhi, S. H., Sarker, S. K., Das, S. K., et al. Towards next generation digital twin in robotics: Trends, scopes, challenges, and future. Heliyon, 9(2), 2023.

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL http: //jmlr.org/papers/v22/20-1364.html.

Munos, R. and Szepesvári, C. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9 (5), 2008.

Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physicsinformed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019. URL https: //doi.org/10.1016/j.jcp.2018.10.045.

Nachum, O., Chow, Y., Dai, B., and Li, L. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.

Rosen, R., von Wichert, G., Lo, G., and Bettenhausen, K. D. About the importance of autonomy and digital twins for the future of manufacturing. IFACPapersOnLine, 48(3):567–572, 2015. ISSN 2405-8963. doi: https://doi.org/10.1016/j.ifacol.2015.06.141. URL https://www.sciencedirect.com/ science/article/pii/S2405896315003808. 15th IFAC Symposium onInformation Control Problems inManufacturing.

National Academy of Engineering and National Academies of Sciences, Engineering, and Medicine. Foundational Research Gaps and Future Directions for Digital Twins. The National Academies Press, Washington, DC, 2024. ISBN 978-0-309-70042-9. doi: 10.17226/26894. URL https://doi.org/10.17226/26894. Nikishin, E., Abachi, R., Agarwal, R., and Bacon, P.L. Control-oriented model-based reinforcement learning with implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7886–7894, 2022.

Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206– 215, 2019.

Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310–1318. Pmlr, 2013.

San, O., Pawar, S., and Rasheed, A. Decentralized digital twins of complex dynamical systems. Scientific Reports, 13(1):20087, 2023.

Pickering, B. W., Gajic, O., Ahmed, A., Herasevich, V., and Keegan, M. T. Data utilization for medical decision making at the time of patient admission to icu. Critical care medicine, 41(6):1502–1510, 2013.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Precup, D., Sutton, R. S., and Singh, S. P. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 759–766, 2000.

Sharma, A., Parbhoo, S., Gottesman, O., and Doshi-Velez, F. Decision-focused model-based reinforcement learning for reward transfer. In Machine Learning for Healthcare Conference. PMLR, 2024.

Purcell, W. and Neubauer, T. Digital twins in agriculture: A state-of-the-art review. Smart Agricultural Technology, 3: 100094, 2023.

Slepneva, T., Chernysheva, M., Zaitseva, K., et al. Impact of digital twin technology on the financial performance of corporations. European Proceedings of Social and Behavioural Sciences, 2021. URL http://dx.doi. org/10.15405/epsbs.2021.04.02.145.

Qian, Z., Zame, W., Fleuren, L., Elbers, P., and van der Schaar, M. Integrating expert odes into neural odes: pharmacology and disease progression. Advances in Neural Information Processing Systems, 34:11364–11383, 2021. URL https://openreview.net/pdf? id=tDqef76wFaO.

Spearman, C. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101, 1904.

Qiao, Z., Lyu, J., Lyu, B., Shu, Y., Gao, S., and Qiu, S. Model-based offline rl via robust value-aware model learning with implicitly differentiable adaptive weighting. In The Fourteenth International Conference on Learning Representations, 2026.

Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160– 163, 1991. 12

Decision-Targeted Digital Twins

Takeishi, N. and Kalousis, A. Physics-integrated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems, 34:14809–14821, 2021. URL https: //openreview.net/pdf?id=0p0gt1Pn2Gv.

Wagg, D., Worden, K., Barthorpe, R., and Gardner, P. Digital twins: state-of-the-art and future directions for modeling and simulation in engineering dynamics applications. ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B: Mechanical Engineering, 6(3): 030901, 2020.

Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., and Sui, F. Digital twin-driven product design, manufacturing and service with big data. The International Journal of Advanced Manufacturing Technology, 94:3563– 3576, 2018. URL https://doi.org/10.1007/ s00170-017-0233-1.

Wang, K., Shah, S., Chen, H., Perrault, A., Doshi-Velez, F., and Tambe, M. Learning mdps from features: Predictthen-optimize for sequential decision making by reinforcement learning. Advances in Neural Information Processing Systems, 34:8795–8806, 2021. Wang, X., Wongkamjan, W., Jia, R., and Huang, F. Live in the moment: Learning dynamics model adapted to evolving policy. In International Conference on Machine Learning, pp. 36470–36493. PMLR, 2023.

Thomas, P. and Brunskill, E. Data-efficient off-policy policy evaluation for reinforcement learning. In International conference on machine learning, pp. 2139–2148. PMLR, 2016.

Wilder, B., Dilkina, B., and Tambe, M. Melding the datadecisions pipeline: Decision-focused learning for combinatorial optimization. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 1658–1665, 2019.

Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026–5033. IEEE, 2012.

Xie, T., Ma, Y., and Wang, Y.-X. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems, 32, 2019.

Towers, M., Kwiatkowski, A., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Andreas, K., Krimmel, M., KG, A., Perez-Vicente, R. D. L., et al. Gymnasium: A standard interface for reinforcement learning environments. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.

Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. Advances in neural information processing systems, 33:14129–14142, 2020.

Uehara, M., Huang, J., and Jiang, N. Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning, pp. 9659–9668. PMLR, 2020.

Zhang, R., Dai, B., Li, L., and Schuurmans, D. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020a.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.

Zhang, S., Liu, B., and Whiteson, S. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning, pp. 11194–11203. PMLR, 2020b.

Voelcker, C. A., Liao, V., Garg, A., and Farahmand, A.m. Value gradient weighted model-based reinforcement learning. In International Conference on Learning Representations, 2022. Voloshin, C., Le, H. M., Jiang, N., and Yue, Y. Empirical study of off-policy policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. Voosen, P. Europe is building a ‘digital twin’of earth to revolutionize climate forecasts. Science, 2020. URL https: //doi.org/10.1126/science.abf0687. 13

Decision-Targeted Digital Twins

A. Theorem Proofs Herein we provide the proofs for Theorem 3.1 and Theorem 3.3. A.1. Proof of Theorem 3.1 Proof. By the theorem assumption, E(x,a)∼D [DKL (Φ∥Φ̂θ∗ )] > 0. This implies the existence of a reachable state-action pair (x0 , a0 ) ∈ supp(D) and a measurable set G ⊆ X where the transition probabilities differ: Φ(G | x0 , a0 ) ̸= Φ̂θ∗ (G | x0 , a0 ).

(20)

Let p := Φ(G | x0 , a0 ) and p̂ := Φ̂θ∗ (G | x0 , a0 ). Without loss of generality, assume p > p̂. Consider a decision scenario with horizon H = 1 (or, alternatively, where rt = 0 for t > 0). Assume |A| ≥ 2, such that there be a second action a1 ∈ A available at x0 . We define a bounded reward function r : X × A × X → [0, 1]: r(x, a, x′ ) := I{x = x0 } (I{a = a0 }I{x′ ∈ G} + I{a = a1 } c) ,

(21)

where c ∈ (0, 1) is a constant chosen such that p̂ < c < p. Let Π = {π0 , π1 } be a set of deterministic policies where π0 (x0 ) = a0 and π1 (x0 ) = a1 . The policy values under the true system Φ are: J(π0 ) = p, J(π1 ) = c. (22) Since p > c, the true preference ordering is π0 ≻ π1 . However, under Φ̂θ∗ the estimated policy values are: ˆ 0 ; θ∗ ) = p̂, J(π

ˆ 1 ; θ∗ ) = c. J(π

(23)

ˆ 0 . Thus, Φ̂θ∗ yields an incorrect preference ordering. Since p̂ < c, the model induces preference ordering π1 ≻π A.2. Proof of Theorem 3.3 To prove the existence of a better model in F, we rely on the assumption of local improvement: that F contains an alternative model whose predictions on a local subset of transitions move in the correct direction relative to Φ̂θ∗ . Proof. From the proof of Theorem 3.1, we have p > p̂ and we consider the same one-step decision task with reward r(x, a, x′ ) := I{x = x0 } (I{a = a0 }I{x′ ∈ G} + I{a = a1 } c) , and policy set Π = {π0 , π1 } where π0 (x0 ) = a0 and π1 (x0 ) = a1 . By assuming Definition 3.2, p̂′ > p̂. Choose any constant

c ∈ (p̂, min{p, p̂′ }).

(24)

This interval is non-empty because p̂′ > p̂ and p > p̂. True ordering.

As before, J(π0 ) = p and J(π1 ) = c, hence π0 ≻ π1 since p > c by (24).

Ordering under θ∗ .

ˆ 0 ; θ∗ ) = p̂ and J(π ˆ 1 ; θ∗ ) = c, hence π1 ≻ ˆ θ∗ π0 since c > p̂. Under Φ̂θ∗ , J(π

Ordering under θ′ .

ˆ 0 ; θ′ ) = p̂′ and J(π ˆ 1 ; θ′ ) = c, hence π0 ≻ ˆ θ′ π1 since p̂′ > c by (24). Under Φ̂θ′ , J(π

Therefore, on the same decision problem (r, Π), Φ̂θ∗ induces an incorrect preference ordering, whereas Φ̂θ′ induces the correct preference ordering. That is, Φ̂θ′ yields a strictly better preference ordering on Π than Φ̂θ∗ .

Decision-Targeted Digital Twins

B. Experimental Details We provide detailed set-ups for all experiments run here. Furthermore, we will release code upon acceptance. B.1. Limited Neural Capacity Experiment System Dynamics. The environment consists of a 2-dimensional state space xt = [xd,t , xc,t ]⊤ and a 1-dimensional action space at ∈ R. The transitions are defined as: xd,t+1 = M sin(xd,t )

(25)

xc,t+1 = xc,t + ϵat

(26)

where M = 500, ϵ = 0.01. The horizon is H = 20. The reward function is r(xt , at , xt+1 ) = xc,t . Data Generation. We collected a dataset D of 3,000 episodes using a behavior policy that samples actions uniformly a ∼ U(−1, 1). Model Architecture.

We utilise an MLP with an explicit information bottleneck.

• Input: State (2) + Action (1). • Encoder: Linear(3 → 32), Tanh, Linear(32 → 1). • Bottleneck: The 1-dimensional output of the encoder represents the compressed state representation z. • Decoder: Linear(2 → 32) (taking z and a), Tanh, Linear(32 → 2). Training. We trained for 30 epochs with a learning rate of 1 × 10−3 using the Adam optimiser and LMSE sim as the simulation loss. For DT2 , we used λ = 0.99 and α = 1.0. The ranking targets were computed using the ground truth value difference between πhigh and πlow . B.2. Misspecified Mechanistic Model Experiment System Dynamics.

The true system is a 1D scalar system governed by cubic control inputs: xt+1 = xt + (at − a3t ) + ξt ,

Hypothesis Space.

ξt ∼ N (0, 0.1)

(27)

The model is constrained to be linear in actions: x̂t+1 = xt + βat

(28)

Here, β is the only learnable parameter. Training Data. Policies.

The dataset consists of 5,000 transitions where actions are sampled uniformly from: a ∼ U (−1.5, 1.5).

We evaluate three constant policies:

1. π1 : a = −0.58 2. π2 : a = 0.0 3. π3 : a = +0.58 The optimal policy is π3 . Training. We trained for 2000 epochs with a learning rate of 0.01 using the Adam optimiser and LMSE sim as the simulation loss. For DT2 , we used λ = 0.9 and α = 1.0. The ranking targets were computed using the ground truth policy values. 15

Decision-Targeted Digital Twins

B.3. Partially Observed Dynamics Experiment System Dynamics. The true system is a 1-dimensional linear system governed by an unobserved latent variable zt with high variance. The transitions are defined as: xt+1 = xt + at + zt ,

zt ∼ N (0, 25)

(29)

where the state xt ∼ N (0, 1) and the action at ∈ [−1, 1]. The large standard deviation of the latent variable (σz = 5) relative to the action magnitude mimics a scenario where unobserved factors dominate the transition dynamics. Data Generation. To emphasise data scarcity, we collected a small dataset D of N = 100 transitions. Actions in the dataset were sampled uniformly: a ∼ U (−1, 1). Model Architecture. We utilise an MLP with a single hidden layer to approximate the dynamics xt+1 ≈ Φ̂θ (xt , at ). • Input: State xt and Action at (Dimension: 2). • Hidden: Linear(2 → 16), ReLU activation. • Output: Linear(16 → 1) representing the predicted next state x̂t+1 . Policies. We evaluate the model’s ability to rank two constant-action policies: 1. πhigh = 0.01 2. πlow = −0.01 Training. We trained for 200 epochs with a learning rate of 1 × 10−2 using the Adam optimiser. We used LMSE sim as the simulation loss. For DT2 , we used λ = 0.9 and a temperature α = 0.1. The ranking targets were computed using the known ground truth policy values. B.4. Continuous Control Experiments B.4.1. E NVIRONMENTS We use six continuous control environments from the Gymnasium library (Towers et al., 2025) (https://github.com/ Farama-Foundation/Gymnasium, MIT license), utilising the MuJoCo physics engine (Todorov et al., 2012). These environments were selected to cover a diverse range of complexities, spanning state-action space dimensionalities from 4 to 35. The specific details for each environment are as follows: • Pendulum-v1 A classic control problem where the goal is to swing up and balance a pendulum in an inverted position from a random start. – X : dX = 3, including the cosine and sine of the pendulum’s angle, and its angular velocity. – A: dA = 1, the scalar joint torque applied to the pendulum. • LunarLanderContinuous-v3: A rocket trajectory optimisation task where the agent must safely land a spacecraft on a designated pad. – X : dX = 8, including coordinates, linear velocities, angle, angular velocity, and boolean flags for leg ground contact. – A: dA = 2, controls the main engine throttle and the side engine thrusters. • Hopper-v4: A two-dimensional one-legged robot that aims to hop forward as fast as possible. – X : dX = 11, comprising positional values and velocities of the body parts. – A: dA = 3, controls the torques applied to the thigh, leg, and foot joints. • Walker2d-v5: A two-dimensional bipedal robot that aims to walk forward. 16

Decision-Targeted Digital Twins

– X : dX = 17, including joint positions and velocities for both legs and the torso. – A: dA = 6, controls the torques for the joints. • HalfCheetah-v5: A two-dimensional robot resembling a cheetah that aims to run forward. – X : dX = 17, including positions and velocities of the torso and limbs. – A: dA = 6, controls the torques on the forward and back thighs, shins, and feet. • Ant-v5: A complex three-dimensional quadruped robot that aims to run forward. – X : dX = 27, including the position, and orientation of the torso, joint angles, and velocities for all four legs. – A: dA = 8, controls the torques on the hip and ankle joints of the four legs. N.B. we run this environment with contact forces disabled, which would add an extra 78 state dimensions. B.4.2. P OLICIES To construct the set of candidate policies Π, we trained agents using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017), as implemented in the Stable Baselines3 library (Raffin et al., 2021) (https: //github.com/DLR-RM/stable-baselines3, MIT license). We use the standard MlpPolicy architecture, and train for T = 1 × 106 timesteps. The hyperparameters used during PPO training, across all environments, are mostly the default options within the Stable Baselines3 library, detailed in Table 4. Table 4. PPO Hyperparameters used for training candidate policies.

Hyperparameter Total Timesteps (T ) Number of Environments (Nenvs ) Steps per Environment (Nsteps ) Batch Size Learning Rate Discount Factor (γ) GAE Lambda (λ) Clip Range (ϵ) Entropy Coefficient Value Function Coefficient Optimizer

Value 1,000,000 (5,000,000 for Pendulum-v1) 8 2048 1024 3 × 10−4 0.97 (0.95 for Pendulum-v1) 0.95 0.2 0.01 0.5 Adam

We constructed the candidate policy set Π by saving checkpoints at evenly spaced intervals during training. Specifically, we selected policies at {0, 0.2T, 0.4T, 0.6T, 0.8T, T } steps. The ground-truth preference ordering over Π, denoted by ≻, was established by estimating the expected discounted return J(π) for each π ∈ Π. This was calculated via Monte Carlo estimation using 500 rollouts in the true environment. To collect the DT training dataset, D, we unroll each π ∈ Π in the true environments for a N steps. For Pendulum we set N = 5000, for Lunarlander we set N = 1000, and for the remaining environments we set N = 10000. B.4.3. F ITTED Q E VALUATION To obtain the proxy ground-truth rankings for decision-targeted training, we estimated the action-value function Qπ (s, a) for each policy π ∈ Π using Fitted Q Evaluation (FQE) (Le et al., 2019). Qπψ was parametrised as an MLP with Layer Normalization and SiLU activations. The network takes the concatenated state and action vectors (s, a) as input and outputs a scalar Q-value. To handle varying reward scales across environments and improve training stability, we standardised the rewards in the offline dataset to have zero mean and unit variance. Qπψ was trained to predict the cumulative return of these standardised rewards. During inference, predictions were rescaled to the original magnitude. 17

Decision-Targeted Digital Twins

Qπψ were trained by minimising the expected Bellman error over the offline dataset D. We employed a target network Qπψ̄ with an identical architecture, whose weights were updated via a hard update every 5 epochs. We trained a separate Qπψ for each policy in Π using the AdamW optimizer, with gradient norm clipping. The specific hyperparameters are summarised in Table 5. Table 5. Hyperparameters for Fitted Q Evaluation.

Hyperparameter

Value

Hidden Dimensions Activation Function Normalization Training Epochs Batch Size Learning Rate Target Update Frequency Target Action Samples (K) Optimizer Gradient Clipping Norm

[256, 256] SiLU LayerNorm 200 1, 024 3 × 10−4 Every 5 epochs 32 AdamW 10.0

B.4.4. D IGITAL T WIN T RAINING We evaluated our method using five distinct backbone architectures to demonstrate the architecture-agnostic nature of DT2 . All dynamics models were implemented as probabilistic ensembles (though we treat them as single stochastic models for the purpose of this work) that output the mean µθ (x, a) and diagonal log-variance log Σθ (x, a) of a Gaussian distribution N (µθ , Σθ ). To improve training stability, we standardised inputs (state and action) to zero mean and unit variance using statistics computed from the offline dataset. The models were trained to predict the normalised state difference ∆x = xt+1 − xt . The final prediction is obtained by denormalising the output and adding it to the current state. To ensure numerical stability, the predicted log-variance is clamped to the range [−10.0, 2.0]. Additionally, during trajectory generation, the predicted next states are explicitly clipped to the specific physical bounds of the environment (e.g., joint limits) to ensure the simulation remains valid. Base Architectures. We evaluated our method using five distinct feature extraction backbones. The output of each backbone is projected by two separate linear layers to produce the mean and log-variance vectors. The backbone details are as follows: • MLP: A standard feed-forward network. It projects the input to the hidden dimension, followed by a SiLU activation, a second linear layer, and another SiLU activation. • ResNet: A residual network consisting of an input projection followed by 4 residual blocks. Each block applies: Linear → LayerNorm → SiLU → Dropout(0.1) → Linear → LayerNorm, with a skip connection adding the block input to the output. A final SiLU activation is applied after the blocks. • Neural ODE (NODE): This backbone models the state evolution as an Ordinary Differential Equation (Chen et al., 2018). The hidden state transformation is defined as z(T ) = ODESolve(z(0), f, 0, T ), where f is a neural network (Linear → Tanh → Linear) representing the derivative dz/dt. We used an explicit Runge-Kutta 4 (RK4) solver with a fixed step size (N = 1 step). • Transformer: Utilizes a standard TransformerEncoder (Vaswani et al., 2017) with 2 layers, 4 attention heads, and a feed-forward dimension of twice the hidden size. To capture temporal dependencies, the model receives a context window of the T = 8 most recent transitions. Learned positional embeddings are added to the input sequence. • GRU: A Gated Recurrent Unit network (Cho et al., 2014) with 2 stacked layers. Similar to the Transformer, it processes a history sequence of length T = 8 to predict the next state. 18

Decision-Targeted Digital Twins

Training Hyperparameters. All models were trained using the AdamW optimizer with gradient clipping. We employed early stopping based on the validation loss to prevent overfitting. The full set of hyperparameters is listed in Table 6. Table 6. Hyperparameters for Digital Twin Training.

Standard DT

DT2

Optimizer Learning Rate Batch Size Hidden Dimension Gradient Clipping Norm Max Epochs Early Stopping Patience

AdamW 3 × 10−4 1024 64 10.0 2000 20 epochs

DT2 Specific Parameters Ranking Weight (λ) Rank Loss Temperature (α) Rollout Horizon (H) Rollout Episodes (M ) Bootstrapping

– – – – –

0.1 1.0 20 128 Yes (using FQE)

Hyperparameter

N.B. in the Ant environment, we set H = 50. For DT2 and standard DTs, we used LNLL sim as the simulation loss for the results displayed in §6.2, and results using LMSE are in Appendix D. sim VaGraM Baseline. We adapted the VaGraM method (Voelcker et al., 2022) to our offline ranking setting. Originally designed for online MBRL, alongside learning a single policy, we adapt it to train on batched data, and to weight transition errors based on the sensitivity of the value functions of all candidate policies in Π. The model was trained to minimise a weighted Mean Squared Error (MSE) objective: LVaGraM (θ) = E(x,a,x′ )∼D ||x′ − µθ (x, a)||2 · w(x′ ) , where the weight w(x′ ) is the average squared norm of the value gradients at the next state across all policies: 1 X ||∇x′ Qπψ (x′ , π(x′ ))||22 . w(x′ ) = |Π|

(30)

(31)

π∈Π

The gradients ∇x′ Q were computed via backpropagation through the pre-trained FQE networks (which were frozen during this process). We trained the VaGraM model using the MLP backbone and the same settings as the standard DTs. MOReL Baseline We implement the MOReL (Kidambi et al., 2020) dynamics ensemble following the parameterisation described in the original work. Each ensemble member is a Gaussian mean model of the form xt − µs at − µa x̂t+1 = xt + σδ · MLP , , (32) σs σa where µs , σs , µa , σa are the empirical mean and standard deviation of states and actions in D, and σδ is the empirical standard deviation of one-step state deltas. Each model is trained by minimising the MSE on normalised deltas, equivalent to Gaussian MLE under fixed diagonal covariance. The specific architecture and training hyperparameters are given in Table 7. After training, we compute the pairwise ℓ2 disagreement between all ensemble members on the training transitions and set the Unknown State-Action Detector (USAD) threshold as τ = µd + β · σd ,

(33)

where µd and σd are the mean and standard deviation of the per-transition disagreement scores, and β = 5 following the paper default. Transitions flagged as unknown by USAD receive the halt reward rhalt = rmin − δr , where rmin is the minimum reward observed in D and δr is an environment-specific offset. For policy evaluation, we roll out each candidate policy π ∈ Π in the resulting pessimistic MDP. 19

Decision-Targeted Digital Twins Table 7. Hyperparameters for the MOReL dynamics ensemble.

Hyperparameter

Value

Ensemble Size Hidden Dimension Hidden Layers Activation Max Epochs Early Stopping Patience Batch Size Learning Rate Optimizer Bootstrap Sampling Validation Fraction Threshold β

4 200 4 ReLU 2,000 20 epochs 256 5 × 10−4 Adam Yes 0.1 5.0

MOPO Baseline We implement the MOPO (Yu et al., 2020) dynamics ensemble following the architecture described in the original work. Each ensemble member is a probabilistic model that outputs the mean and diagonal log-variance of a Gaussian distribution over normalised state deltas: pθ (∆x | xt , at ) = N µθ (xt , at ), diag(σθ2 (xt , at )) , (34) where ∆x = xt+1 − xt . Inputs are normalised to zero mean and unit variance using statistics computed from D, and outputs are denormalised prior to rollout. Spectral normalisation is applied to all linear layers except the log-variance head, 2 2 following the original implementation. Log-variances are soft-clamped to the range [log σmin , log σmax ] via learnable bound parameters. Each member is trained by minimising the Gaussian NLL with a small regularisation term penalising the spread of the log-variance bounds. We train an ensemble of 7 members and retain the 5 with the lowest holdout NLL as the elite subset. Policy evaluation uses only the elite models. The per-transition uncertainty is defined as the maximum across elite members of the ℓ2 norm of the predicted standard deviation: (e) u(xt , at ) = max σθ (xt , at ) , (35) e∈E

where E denotes the elite set. The penalised reward used for policy evaluation is then r̃(xt , at ) = r(xt , at ) − λ · u(xt , at ), where λ is the penalty coefficient. The specific architecture and training hyperparameters are given in Table 8. Table 8. Hyperparameters for the MOPO dynamics ensemble.

Hyperparameter

Value

Ensemble Size Elite Size Hidden Dimension Hidden Layers Activation Spectral Normalisation Max Epochs Early Stopping Patience Elite Holdout Size Batch Size Learning Rate Optimizer Bootstrap Sampling

7 5 200 4 SiLU Yes (trunk & mean head) 2,000 20 epochs 1,000 transitions 256 3 × 10−4 Adam Yes

Decision-Targeted Digital Twins

ROMI Baseline We implement ROMI (Qiao et al., 2026), which jointly learns a dynamics ensemble, an adaptive sample-reweighting network, and a SAC policy. Each ensemble member is a probabilistic model with the same Gaussian parameterisation over normalised state deltas as MOPO, but without spectral normalisation. The ensemble is first pre-trained for 50 epochs by minimising the standard Gaussian NLL on bootstrapped subsets of D. Following the pre-training phase, ROMI alternates between (i) SAC policy updates on mixed real and model data, and (ii) a bilevel optimisation that jointly refines the dynamics and an adaptive weighting network wν (x, a, x′ ). The weighting network is an MLP that outputs per-transition weights in the range [wmin , wmax ] via a tanh output activation. The inner (e) level of the bilevel objective trains dynamics member Φθ with a weighted NLL: Linner = E(x,a,x′ )∼D [wν (x, a, x′ ) · ℓNLL (x, a, x′ ; θ)] ,

(36)

where wν is held fixed. The outer level then updates ν to minimise a robust value-aware loss. Specifically, given a one-step look-ahead from the updated dynamics x̂′ = µθ+ (x, a), and a pessimistic value target computed by perturbing x′ with Gaussian noise of scale σu and taking the minimum Q-value over K samples: K

V − (x′ ) = min min(Q1 (x′ + ϵk , π(x′ + ϵk )), Q2 (x′ + ϵk , π(x′ + ϵk ))) , k=1

ϵk ∼ N (0, σu2 I),

(37)

the outer loss encourages the dynamics to predict next states whose value matches this robust target: Louter = V (x̂′ ) − V − (x′ )

(38)

The SAC policy is trained on batches mixing real transitions (ratio ρ = 0.5) with model rollouts of horizon H = 5 stored in a replay buffer. Following the ROMI paper, we use σu = 0.1 across all environments, except Hopper (σu = 1.0) and Walker (σu = 0.01). The full set of hyperparameters is given in Table 9. HDTwin Baseline We implement HDTwin (Holt et al., 2024) using the official repository here: https://github. com/samholt/HDTwinGen (MIT License). This method prompts an LLM to propose a Python class that represents the functional form of a DT, which is then trained on the training data. Per-variable validation MSE is then calculated, and reported back to the LLM, which is prompted to suggest changes to improve this validation MSE. The functional form is then iteratively improved in this manner. We use GPT-5 via the Azure API for the LLM, allow 3 cycles of iterative improvement, and train each model for up to 2000 epochs with early stopping and patience of 20. All other hyperparameters are kept at default. B.5. Cancer Experiment B.5.1. E NVIRONMENT D ETAILS We used the GhaffariCancerEnv from DTR-Bench/DTRGym (Luo et al., 2024) (https://github.com/ GilesLuo/DTR-Bench, https://github.com/GilesLuo/DTRGym, MIT License), which simulates the treatment of cancer with metastasis under combined radiotherapy and chemotherapy. The environment is based on the mathematical model by Ghaffari et al. (2016). • X : dX = 7, log-transformed cell counts and drug concentrations. • A: dA = 2, radiotherapy dose (D ∈ [0, 10] Gy), chemotherapy concentration (vM ∈ [0, 8]). • We use ‘Setting 5’ of the environment, representing a challenging medical scenario. This includes: – State transition noise (σstate = 0.5). – Observation noise (σobs = 0.2). – Pharmacokinetic/Pharmacodynamic parameter noise (σpkpd = 0.1). – A 50% probability of missing observations at any given timestep. 21

Decision-Targeted Digital Twins Table 9. Hyperparameters for ROMI.

Hyperparameter

Value

Dynamics Ensemble Ensemble Size Hidden Dimension Hidden Layers Activation Pre-train Epochs Dynamics Learning Rate Bootstrap Sampling

2 SiLU 50 3 × 10−4 Yes

Weighting Network Hidden Dimension Hidden Layers Weight Range [wmin , wmax ] Weight Learning Rate Bilevel Inner Learning Rate Adversarial Train Steps Uncertainty Samples (K)

64 2 [0.5, 2.0] 1 × 10−4 3 × 10−4 1,000 20

SAC Policy Actor/Critic Hidden Dimension Actor Learning Rate Critic Learning Rate Soft Update (τ ) Training Schedule Epochs Batch Size Real Data Ratio (ρ) Policy Updates per Epoch Rollout Horizon (H) Rollout Batch Size

256 1 × 10−4 3 × 10−4 5 × 10−3 500 256 0.5 200 5 512

B.5.2. T RAINING P OLICIES For this experiment, we defined 5 interpretable clinical policies to construct Πtrain : 1. No Treatment: Baseline policy with zero intervention (D = 0, vM = 0). 2. Fractionated Radiotherapy: 2 Gy/day radiation, 5 days/week, with weekends off. No chemotherapy. 3. Metronomic Chemotherapy: Continuous low-dose chemotherapy (vM = 2.0) administered daily. No radiotherapy. 4. Adaptive Combined Therapy: A reactive strategy that monitors tumour burden. If the log-size exceeds a threshold (50% of initial burden), it applies aggressive combination therapy (D = 2.0, vM = 4.0). Otherwise, no treatment is given. 5. Aggressive Combined Therapy: High dose of both modalities (D = 4.0, vM = 6.0) administered daily. B.5.3. E VALUATION P OLICIES To test the generalisation capabilities of DT2 , we evaluated the models on a set of 11 unseen policies, including 6 PPO checkpoints along a training run of 200,000 steps, and the 5 following manually-defined policies: • Pulsed Chemotherapy: High dose chemotherapy (vM = 5.0) administered once every 21 days. 22

Decision-Targeted Digital Twins

• Hypofractionated Radiotherapy: High radiation dose (D = 8.0) administered every 3 days. • Induction-Maintenance: Aggressive combination therapy (D = 2.0, vM = 4.0) for the first 30 days, followed by low-dose maintenance chemotherapy (vM = 1.5). • Alternating Modality: A 14-day cycle consisting of 7 days of daily radiotherapy (D = 2.0) followed by 7 days of daily chemotherapy (vM = 3.0). • Dose Escalation: Doses for both modalities linearly ramp up from 0 to a maximum (D = 3.0, vM = 5.0) over the first 100 days. B.5.4. D IGITAL T WIN T RAINING The training pipeline followed the same structure as the continuous control experiments, with the following environmentspecific adjustments: • Dataset: We collected 1,000 steps from each of the 5 training expert policies. • Hyperparameters: We used a batch size of 512 and a discount factor γ = 0.99. The Q-networks were trained for 500 epochs. B.6. Lambda Ablation For this experiment, we use the ResNet backbone in all models. All settings are kept the same as in the continuous control experiments, except we vary λ ∈ {0, 0.1, 0.25, 0.5, 0.75}.

Decision-Targeted Digital Twins

C. Unseen Policy Rankings in Continuous Control Environments We now report some further results to assess how DT2 performs in ranking policies that are unseen during training. Here, we compare LNLL sim and LDT2 in ranking unseen policies across the three smaller continuous control environments from §6.2 (Pendulum, Lunarlander, Hopper). Again, we use the ResNet architecture, for consistency with §6.3. In this case, since it is not so clear how ‘expert defined’ policies might look, we investigate rankings of five simple policies that are not seen during training, that involve taking 1) random actions, 2) constant 0 action, 3) constant minimum action, 4) constant median action, and 5) constant maximum action. These policies are generally much less sophisticated, and worse-performing, than the PPO policies used to collect the datasets and train DT2 in §6.2, so they can be considered quite out of distribution. 2 In Table 10 we report the ranking performance of LNLL sim and LDT2 DTs across these simple policies. We see that DT training still leads to an increase in ranking performance, although, as expected, to a lesser extent than when ranking policies that are seen during training.

Table 10. Regret and Spearman’s correlation on preference orderings of five unseen constant-action policies in continuous control environments. We report averages over 5 seeds, with standard errors.

Pendulum

LunarLander

Hopper

Loss

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

Regret (↓)

Spearman (↑)

LNLL sim

8.70 (6.03) 5.37 (4.35)

0.340 (0.201) 0.580 (0.169)

211.78 (83.92) 102.15 (63.56)

0.760 (0.058) 0.780 (0.061)

1.83 (0.91) 0.61 (0.60)

0.811 (0.042) 0.911 (0.042)

LDT2

Decision-Targeted Digital Twins

D. Comparisons with MSE Simulation Loss NLL We display results for the continuous control experiments using LMSE sim for the simulation loss, rather than Lsim in Figure 6. 2 We see similar takeaways from these results as those in §6.2. Namely, DT outperforms its standard counterpart in 76.7% of the environment-architecture-metric couplings. In general, the DTs using LMSE sim as their simulation loss seem to perform worse than when using LNLL sim . Pendulum

Lunarlander

5.0

Regret

40 7.5

3 2 1

1.00

0.8

0.6

0.75

0.4 0.2

DT DT2

Transformer

GRU

DT DT2

MLP

ResNet

NODE

Spearman

0.0 0.8

Spearman

2.5

0.0

Hopper

10.0

0.50 0.25 0.00

Walker

DT DT2

Transformer

GRU

DT DT2

MLP

ResNet

NODE

0.6 0.4 0.2 0.0

DT DT2

Transformer

GRU

MLP

ResNet

NODE

Cheetah

Ant 15

10.0

5.0

Regret

8 7.5

6 4

2.5

0.0

0 1.0

0.5

Spearman

0.50 0.25 0.00 −0.25

Spearman

0.75

Spearman

0.0 −0.5

DT DT2

Transformer

GRU

MLP

ResNet

NODE

Transformer

GRU

MLP

ResNet

NODE

Transformer

GRU

MLP

ResNet

NODE

Figure 6. Regret and Spearman’s correlation for preference orderings from LMSE sim (hashed) and LDT2 (solid) DTs across base architectures

in six continuous control environments. We report averages over 10 seeds, with 95% CIs. For visual purposes, the top end of some error bars are cropped out.

Decision-Targeted Digital Twins

E. Ranking Loss Ablation We now consider some variants to the differentiable ranking loss used for DT2 . Let y ∈ RM be the vector of target proxy values (derived from FQE) for the M policies in a batch, and ŷ ∈ RM be the corresponding vector of cumulative returns predicted by the DT rollouts. Let C = {(i, j) | 1 ≤ i < j ≤ M } denote the set of all unique pairs in the batch. Pairwise Hinge Loss. This variant enforces that the DT’s predicted value difference between two policies preserves the sign of the ground-truth difference, with a sufficient margin. To ensure the margin scales appropriately across different environments with varying reward magnitudes, we implemented an adaptive margin ϵ based on the mean absolute value of the targets: M 1 X ϵ = 0.1 · |yk |. (39) M k=1

The loss is defined as: LHinge (θ) =

1 X max (0, ϵ − sign(yi − yj )(ŷi − ŷj )) . |C|

(40)

(i,j)∈C

This penalises the model if the predicted ordering is incorrect or if the separation between the pair is smaller than ϵ. ListNet Loss. ListNet (Cao et al., 2007) treats the ranking problem as a probability distribution matching problem. It maps the vector of values to a probability distribution using a softmax function. Let Py and Pŷ be the softmax distributions of the target and predicted values, respectively, controlled by a temperature parameter α: exp(yi /α) exp(ŷi /α) Py (πi ) = PM , Pŷ (πi ) = PM . (41) k=1 exp(yk /α) k=1 exp(ŷk /α) The loss is calculated as the Cross-Entropy between these distributions: LListNet (θ) = −

M X

Py (πi ) log (Pŷ (πi )) .

(42)

i=1

Unlike the pairwise approaches, ListNet considers the entire list of policies simultaneously. E.1. Continuous Control Results We display results for the continuous control experiments with these variants in Figure 7, and report their average performances in Table 11. Across all environment-architecture-metric couplings, our smoothed Kendall formulation has the highest win-rate. While the other ranking formulations can improve upon LNLL sim , our proposed method tends to work best. In Table 11, we can see that our smoothed Kendall formulation achieves the lowest average regret, and the highest average Spearman’s correlation. Interestingly, in this case, while the Hinge and ListNet formulations improve on LNLL sim in terms of average Spearman’s, they are worse in terms of average regret. As we can see from Figure 7, they each have some cases where they have a very poor regrets (e.g. in the Pendulum and Lunarlander environments), which drives up their averages for this metric.

LDT2

Table 11. Average Spearman’s correlation and regret for different DT variants across all architectures and continuous control task.

Loss

Regret (↓)

Spearman (↑)

LNLL sim Kendall Hinge ListNet

4.08 1.88 4.57 11.51

0.45 0.66 0.59 0.53

Decision-Targeted Digital Twins

Lunarlander

Pendulum

1.00

Regret

Hopper 4

1.25

0.75 0.50

Regret

1.50

0.0

1.0

0.8

Spearman

0.5

0.0

0.4

0.0

Walker

Ant

Cheetah

Regret

0.6

0.2

−0.5

2 1

0.25 0.00 1.0

6 4

2 2 0

0.4

Spearman

0.6

Spearman

0.8 0.5

0.0

0.5

0.0

0.2 0.0

−0.5

Architecture Transformer

GRU

MLP

ResNet

NODE

Method DT

Kendall

Hinge

ListNet

Figure 7. Regret and Spearman’s correlation for preference orderings from DTs trained via LNLL sim DTs (hashed) and LDT2 with ranking losses of our smoothed Kendall’s (solid), a hinge loss (circles), and a ListNet loss (dotted) across different base architectures. We report averages over 10 seeds, with 95% CIs. For visual purposes, the top end of some bars are cropped out.

Record · ID 307021 · SHA-256 89d0c02d55998d74
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.