Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

arXiv:2610.03662v1 [cs.LG] 2 Oct 2026

Angel Wang Amazon wangange@amazon.com

Dominique Perrault-Joncas Amazon joncas@amazon.com

Dean Foster Amazon foster@amazon.com

Alvaro Maggiar Amazon maggiara@amazon.com

Carson Eisenach Amazon ceisen@amazon.com

Abstract Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy’s actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2–3.1 percentage points in Study 1 and 12.5–18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.

1

Introduction

Decision policies in real-world systems are increasingly developed offline and evaluated in simulation before deployment [1, 2, 3]. This raises the classical simulation-to-reality (Sim2Real) question of whether behavior learned or evaluated in simulation transfers to the real system [4, 5]. A related problem arises when deployment also relies on predictive models whose targets depend on the policy’s actions – queue lengths under scheduling, traffic conditions under routing, power demand under load control, or future inventory under replenishment. Historical observations reflect previously operated policies, while deploying a new one can induce a different distribution of states, actions, and resulting system responses. Prediction models whose targets depend on these responses therefore face a cold-start problem: real observations under the target policy do not yet exist. Simulation can address this gap by rolling out the target policy under counterfactual scenarios and generating on-policy trajectories before corresponding real observations are available. This shifts the Sim2Real question from whether a simulator-trained policy transfers to whether a prediction model learned from simulated target-policy trajectories transfers to real behavior. We Preprint.

evaluate this process from three angles: simulator fidelity, zero-shot predictive transfer, and adaptation as real target-policy observations become available. We study this setting in large-scale inventory control using an exogenous interactive decision process (Exo-IDP) simulator [1, 2] and two historical deployments of different inventory-ordering policies at a major retailer. Product-level replenishment decisions jointly determine aggregate inbound volume, which must be coordinated with shared inbound processing capacity. Capacity planning therefore requires forecasting how aggregate inbound will respond to system-level coordination signals. We use the simulator to roll out the deployed policy over a range of counterfactual coordination signals and train an aggregate forecaster of the resulting inbound utilization [6]. We then backtest the simulator-trained forecaster against realized deployment outcomes and evaluate whether it can be further refined as real target-policy observations become available. Across both deployments, the simulator-trained forecaster achieves lower point-estimate error than the same architecture trained on historical real data, while online calibration further reduces error.

2

Related Work

Sim2Real transfer in reinforcement learning. Sim2Real transfer has been widely studied in robotics and reinforcement learning (RL), where the central challenge is bridging the “reality gap” between simulated and physical environments [7, 4, 8, 9]. Prior work studies, among other settings, training policies under randomized simulation [4, 8], adapting simulation-trained controllers or dynamics models with real data [9, 10], and measuring whether simulated performance predicts real performance [5, 11]. Our setting differs in the object being transferred: the decision policy is fixed, and we ask whether a prediction model learned from simulator-generated trajectories under that policy transfers to realized system behavior. Synthetic training data and backtest evaluation. Synthetic data have been used before to train forecasting models for broad generalization, including pretraining on simulated time series [12, 13]. Our approach extends this idea by training the forecaster on policy-conditioned simulated data, which is then validated against realized outcomes from the corresponding deployment. Our evaluation is retrospective [14, 15]: the forecaster is trained using information available before the corresponding deployment outcomes and evaluated against subsequently realized outcomes. Because those outcomes are observed under the deployed policy, this evaluation does not require the counterfactual corrections used in off-policy evaluation [16, 17].

3

Backtesting Sim2Real Transfer of Simulator-Trained Forecasts

3.1

Exo-IDP Simulator and Counterfactual Rollouts

Exo-IDP simulator. Let M denote the real decision process and M̂ an approximate simulator of that process. Our setting follows the Exogenous Interactive Decision Process (Exo-IDP) formulation [1, 2]. For each agent i, the local state separates into an exogenous process ξti , which evolves independently of the agent’s actions, and an endogenous state yti , which depends on past actions: xit = (yti , ξti ),

ait ∼ π(· | xit ),

i yt+1 ∼ P (· | yti , ait , ξti ).

An Exo-IDP simulator replays historical realizations of the exogenous process while evolving the endogenous state using approximate dynamics P̂ . The same exogenous trajectory can therefore be used to simulate the system response under counterfactual actions. Inventory instantiation. In our inventory-control application, each agent is a product managed by a replenishment policy, and ait is its order quantity. Demand Dti , prices, and vendor lead times vti are exogenous, while inventory, open orders, and inbound receipts are endogenous. The endogenous dynamics are governed by the following inventory-flow relationships: orders placed in previous periods arrive according to their realized lead times, X aik 1{vki =t−k} , (1) Jti = 0≤k<t

2

where Jti denotes inbound receipts for product i. Inventory evolves as i Iti− = It−1 + Jti ,

sit = min(Dti , Iti− ),

Iti = (Iti− − Dti )+ ,

(2)

where sit denotes realized sales. Equations (1) and (2) produce the counterfactual rollouts used below to study how the fixed replenishment policy responds to alternative capacity-control inputs. Full simulator construction details are given in Appendix A. 3.2

Learning an Aggregate Response Forecaster

Shared capacity and coordination signal. In the capacity-constrained inventory-control problem, the shared resource is inbound capacity: aggregate inbound utilization X Jt = Jti i

must satisfy E[Jt ] ≤ Gt for the available capacity Gt . This aggregate utilization is the quantity we forecast. Following the Lagrangian coordination formulation of Eisenach et al. [2], the dual variable λt ≥ 0 acts as a broadcast per-unit cost on inbound utilization: higher costs discourage ordering and thereby reduce future inbound. The replenishment policy therefore responds to both its local state and the shared cost signal, ait ∼ π(· | xit , λt ). Through the inventory dynamics above, these product-level policy responses jointly determine aggregate inbound Jt . Capacity planning requires evaluating how candidate cost signals change this aggregate utilization relative to the desired capacity plan. We therefore learn a response forecaster that predicts aggregate inbound under a proposed cost trajectory. Response forecaster. Let Xt denote the forecast-origin representation constructed from local-state and action trajectories over an H-week lookback window,   Xt = Φ {xiτ , aiτ }t−1 i,τ =t−H . Given Xt and a proposed cost trajectory λt:t+L , the response forecaster predicts aggregate inbound over horizon L, Jˆt:t+L = ψθ (Xt , λt:t+L ). (3) This forward map from proposed control inputs to aggregate utilization corresponds to the primal coordination interface of Wang et al. [6]. Training on rollouts. We train ψθ by supervised learning on policy-conditioned simulated rollouts. We generate capacity-control scenarios represented by cost trajectories λ0:T and apply each trajectory to the target policy in M̂. The simulator evolves the resulting actions and endogenous states and records aggregate inbound, yielding supervised examples (Xt , λt:t+L , Jt:t+L ). We use the aggregate multi-horizon forecasting architecture described in Appendix B, which also provides feature and rollout-generation details. 3.3

Backtesting on Historical Deployments

Backtest protocol. The central question is whether a response learned from an imperfect simulator transfers to the real system. Completed deployments make this directly testable: realized target-policy outcomes provide ground truth for a forecaster trained on counterfactual data. For a deployed policy π π, let DM̂ denote the distribution of forecasting examples (X, λ, J) generated by simulator rollouts π under that policy, and let DM denote the distribution realized under π in the corresponding real π deployment. The simulator-trained forecaster is ψ̂sim = Learn(DM̂ ), and its zero-shot Sim2Real π [ℓ(ψ̂sim (X, λ), J)]. risk is the expected loss E(X,λ,J)∼DM

3

Cold-start benchmark. We train the same forecasting architecture on historical real observations π0 collected under earlier policies π0 , ψ̂hist = Learn(DM ), and evaluate it on target-policy deployment outcomes. Online calibration. After deployment, real target-policy observations become available sequentially. We evaluate whether these observations can refine the simulator-trained forecast using an online calibration (an expanding-window linear regression). At forecast origin t, let Jˆτsim = ψ̂sim (Xτ , λτ :τ +L ) denote an earlier simulator-trained forecast. A single pair (αt , βt ), shared across horizons, is estimated from all horizon-level forecast–outcome pairs observed before t, 2 X X  sim (α̂t , β̂t ) = arg min Jτ +h − α − β Jˆτ,h , (4) α,β

τ <t h∈Hτ,t

where Hτ,t indexes the horizons of origin τ whose outcomes are observed by t; calibration begins after the first four deployment weeks. We then apply the fitted calibration to the subsequent forecast, Jˆtcal = α̂t + β̂t Jˆtsim .

4

Results

We evaluate on two completed production deployments of inventory-ordering policies, run in different years under different policies: Study 1 is a year-long randomized deployment of an RL ordering policy covering approximately 20K products, and Study 2 is a later three-month deployment of a different policy covering approximately 100K products. Both are real deployments whose realized outcomes are observed, enabling retrospective evaluation. Forecast accuracy is evaluated at the aggregate level using mean absolute percentage error (MAPE) at one-, two-, and three-week horizons, with weeks 4–10 grouped as a longer planning horizon; reported values average the absolute percentage error of the aggregate forecast across weekly forecast origins. 4.1

Fidelity of Simulator-Generated Training Data

The simulator reproduces the forecasting target to within roughly 12–13%: rolling out the deployed policy one week at a time from realized states, while replaying each evaluation window’s realized exogenous sequences, yields aggregate-inbound MAPE of 13.2 ± 2.4% in Study 1 and 11.8 ± 2.6% in Study 2 against realized outcomes. Because this diagnostic conditions on realized information unavailable at forecast time, it isolates the simulator’s endogenous dynamics and serves as a reference point rather than a deployable forecast. We additionally examine fidelity at the product–week level. For each simulated action or state variable Y , we regress the corresponding realized value on its simulated counterpart without intercept, Yti = βY Ŷti + εit . A slope βY = 1 indicates calibration in scale, while R2 measures cross-sectional agreement. Order quantity and inbound exhibit larger calibration gaps, whereas inventory and sales drain are more closely calibrated (Table 1). Calibration is also stronger for products that actually operated under the deployed policy than for those under the baseline, consistent with measured fidelity being policy-conditional (Appendix C). Table 1: Component-level trajectory calibration (Study 1): realized values regressed on simulated values at the product–week level, with slope 1 indicating perfect calibration. Order and inbound exhibit larger calibration gaps; inventory and sales drain are better calibrated. Component

Calibration slope (95% CI)

R2

Order quantity Inbound Inventory Sales drain

[0.70, 0.71] [0.67, 0.67] [0.90, 0.91] [0.99, 1.00]

0.24 0.47 0.81 0.92

4

Table 2: Zero-shot transfer: MAPE (mean ± 95% CI) on realized deployment outcomes for the same forecaster architecture trained on simulator rollouts under the deployed policy versus historical real data from prior policies (off-policy), by forecast horizon. CIs are 95% normal-approximation intervals across weekly forecast origins, with one error observation per origin and horizon. Study

Training source

1 week

2 weeks

3 weeks

4–10 weeks

Study 1 Study 1

Simulator rollouts Historical (off-policy)

13.3 ± 2.7 15.0 ± 2.9

11.7 ± 2.8 14.8 ± 3.1

13.2 ± 3.1 15.2 ± 3.1

14.8 ± 2.8 16.0 ± 3.1

Study 2 Study 2

Simulator rollouts Historical (off-policy)

12.7 ± 3.6 25.6 ± 5.2

15.3 ± 7.4 27.8 ± 6.3

9.4 ± 3.8 28.1 ± 6.4

11.5 ± 3.4 25.2 ± 5.7

Table 3: Calibration with early real observations: MAPE (mean ± 95% CI) for the simulator-trained forecaster, zero-shot versus after regression calibration on early deployment observations, by forecast horizon. Zero-shot and calibrated variants are evaluated on the same forecast origins as Table 2, whose interval definition applies. Calibrated historical-data baselines are reported in Appendix D.

4.2

Study

Variant

1 week

2 weeks

3 weeks

4–10 weeks

Study 1 Study 1

Zero-shot +Calibration

13.3 ± 2.7 11.8 ± 2.6

11.7 ± 2.8 11.2 ± 3.1

13.2 ± 3.1 13.1 ± 3.6

14.8 ± 2.8 14.8 ± 3.2

Study 2 Study 2

Zero-shot +Calibration

12.7 ± 3.6 12.6 ± 3.6

15.3 ± 7.4 13.0 ± 7.1

9.4 ± 3.8 6.9 ± 3.1

11.5 ± 3.4 9.7 ± 2.9

Zero-Shot Transfer of Simulator-Trained Forecasts

Before testing transfer to reality, we verify that the simulated cost-response relationship is learnable: over the same evaluation windows, the architecture achieves 9.1–11.9% MAPE on held-out simulated inbound, comparable to models trained and evaluated on real data (Appendix D), confirming that the simulated trajectories provide a usable supervised training signal. We then evaluate whether this learned response transfers to realized deployment outcomes. For each study, the forecaster is trained only on simulator rollouts under the target policy (Section 3.2). The historical-data baseline uses the same forecasting architecture trained on real observations collected under earlier policies. Simulator-trained forecasters transfer: they achieve lower point-estimate MAPE than the off-policy historical baseline at every reported horizon in both studies (Table 2), with reductions of 1.2–3.1 percentage points in Study 1 and 12.5–18.7 points in Study 2 (8–21% and 45–67% in relative terms). Notably, the forecaster attains accuracy comparable to the direct-fidelity diagnostic of Section 4.1 while using only information available at the forecast origin. This is consistent with counterfactual rollouts providing training data that reflect the target policy’s response dynamics across the control range, whereas historical observations are limited to previously realized policies and controls; the larger gains in Study 2 are consistent with a larger shift between the prior operating regime and the deployed policy.

4.3

Adaptation with Real Observations

As real on-policy observations accumulate, we can also update the simulator-trained forecaster using the online calibration of Section 3.3: the regression is refit at each forecast origin using all outcomes available up to that time and applied to the subsequent forecast. Online calibration further reduces error: point-estimate MAPE improves at multiple horizons, with the largest reduction of 2.5 percentage points (9.4% to 6.9%) in Study 2 (Table 3); gains are concentrated at horizons of three weeks or less and are not uniform across settings. Calibration thus complements rather than replaces the simulator-based cold start: rollouts provide the pre-deployment forecaster, and early real observations refine it. 5

5

Discussion and Limitations

We investigate whether simulator-generated counterfactual data can address the forecasting cold start created by deployment of a new decision policy. Across two large-scale supply-chain deployments, the simulator exhibits measurable discrepancies from realized behavior, yet forecasters trained on target-policy simulator rollouts transfer to real outcomes and achieve lower point-estimate error than the same architecture trained on historical prior-policy data. As real target-policy observations become available, lightweight calibration further reduces error, especially at shorter horizons. Useful downstream transfer thus does not require perfect direct simulator fidelity; fidelity metrics alone are an incomplete measure of a simulator’s value as a training-data generator, and characterizing which fidelity properties govern transfer remains open. More broadly, our results support simulation as a source of target-policy training data before real observations exist: the simulator can expose the target policy to a broad range of control scenarios and generate the corresponding system responses – coverage that historical data, limited to the combinations that actually occurred, cannot provide. We validated the approach on two deployments of one supply-chain simulator; the evidence so far is application-specific, and extending the evaluation to additional simulators and domains is a natural next step.

6

References [1] M ADEKA , D., T ORKKOLA , K., E ISENACH , C., L UO , A., F OSTER , D. and K AKADE , S. (2022). Deep inventory management. arXiv:2210.03137. [2] E ISENACH , C., G HAI , U., M ADEKA , D., T ORKKOLA , K., F OSTER , D. and K AKADE , S. (2024). Neural coordination and capacity control for inventory management. arXiv:2410.02817. [3] D ULAC -A RNOLD , G., L EVINE , N., M ANKOWITZ , D. J., L I , J., PADURARU , C., G OWAL , S. and H ESTER , T. (2021). Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning 110 2419–2468. [4] T OBIN , J., F ONG , R., R AY, A., S CHNEIDER , J., Z AREMBA , W. and A BBEEL , P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). [5] K ADIAN , A., T RUONG , J., G OKASLAN , A., C LEGG , A., W IJMANS , E., L EE , S., S AVVA , M., C HERNOVA , S. and BATRA , D. (2020). Sim2Real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters 5 6670–6677. [6] WANG , A., P ERRAULT-J ONCAS , D., M AGGIAR , A., F OSTER , D. and E ISENACH , C. (2026). Ready from day 1: Population-aware coordination for large-scale constrained multi-agent systems. ArXiv:2605.13900. [7] JAKOBI , N., H USBANDS , P. and H ARVEY, I. (1995). Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive behavior 6 325–368. [8] P ENG , X. B., A NDRYCHOWICZ , M., Z AREMBA , W. and A BBEEL , P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA). [9] TAN , J., Z HANG , T., C OUMANS , E., I SCEN , A., BAI , Y., H AFNER , D., B OHEZ , S. and VANHOUCKE , V. (2018). Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems. [10] NAGABANDI , A., K AHN , G., F EARING , R. S. and L EVINE , S. (2018). Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA). [11] M URATORE , F., G IENGER , M. and P ETERS , J. (2021). Assessing transferability from simulation to reality for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 43 1172–1183. [12] D OOLEY, S., K HURANA , G. S., M OHAPATRA , C., NAIDU , S. and W HITE , C. (2023). ForecastPFN: Synthetically-trained zero-shot forecasting. In Advances in Neural Information Processing Systems, vol. 36. [13] A NSARI , A. F., S TELLA , L., T URKMEN , C., Z HANG , X., M ERCADO , P., S HEN , H., S HCHUR , O., R ANGAPURAM , S. S., A RANGO , S. P., K APOOR , S. ET AL . (2024). Chronos: Learning the language of time series. Transactions on Machine Learning Research . [14] M IZRAHI , L., NANDAN , S. and W IEMER , S. (2021). Embracing data incompleteness for better earthquake forecasting. Journal of Geophysical Research: Solid Earth 126. [15] C ERQUEIRA , V., T ORGO , L. and M OZETI Č , I. (2020). Evaluating time series forecasting models: an empirical study on performance estimation methods. Machine Learning 109 1997–2028. [16] L EVINE , S., K UMAR , A., T UCKER , G. and F U , J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv:2005.01643. [17] J IANG , N. and L I , L. (2016). Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning. 7

A

Exo-IDP Simulator and Historical Replay

A.1

Interactive Decision Process Formulation

This appendix instantiates the Exo-IDP setting of Section 3 in the supply-chain simulator used to generate counterfactual rollout data. We consider discrete time steps t = 0, . . . , T , where St is the active product population and each agent i ∈ St corresponds to a product governed by a local replenishment policy. The shared resource in this instantiation is inbound processing capacity, so the aggregate inbound utilization Jt of the main text is realized as aggregate inbound volume – the terms are used interchangeably in the appendices – and λt is the per-period inbound cost signal. Exogenous processes.

For each product i and week t, the exogenous state series is ξti = (Dti , pit , cit , vti ),

where the exogenous variables are demand, sales price, ordering cost, and vendor lead time. Importantly, these variables are outside the control of the ordering actions; thus exogenous to the decision process. To be precise, Dti denotes the unconstrained demand – demand not affected by historical i stock availability – and vti is the lead time incurred when an order is placed. Because ξ0:T evolves independently of the policy, its historical realizations can be replayed under counterfactual actions; this property enables counterfactual rollout generation under alternative actions and coordination signals. Endogenous states. The endogenous state yti collects the inventory variables – on-hand inventory, open orders, and arrivals – which depend on past ordering actions. Unlike the exogenous processes, these cannot be replayed: under a counterfactual policy they must be simulated forward from the dynamics below. The full product state is xit = (yti , ξti ). Control process. At each time t, each product observes its local state and the broadcast cost signal λt and selects an order quantity according to a local replenishment policy: ait ∼ π(· | xit , λt ). Transition dynamics.

The endogenous state evolves as i yt+1 ∼ P (· | yti , ait , ξti ).

Orders placed in previous periods arrive during period t according to their realized lead times: X Jti = aik 1{vki =t−k} , 0≤k<t

where vki is the realized vendor lead time for the order placed at time k. Beginning-of-period inventory after receipts is

i i It− = It−1 + Jti . End-of-period inventory after demand fulfillment is i Iti = (It− − Dti )+ ,

and realized sales are

i sit = min(Dti , It− ). Aggregate inbound utilization across the active population is X Jt = Jti . i∈St

Reward.

The per-agent utility Uti (xit , ait ) is instantiated as a revenue-minus-ordering-cost reward: Rti = pit sit − cit ait .

The unconstrained inventory-control objective is # " T XX i max E Rt . π

t=0 i∈St

8

Cost signal. The system broadcasts a nonnegative per-unit inbound cost λt , and the fixed local policy responds as ait ∼ π(· | xit , λt ): higher costs reduce ordering, lower costs relax it. The response forecaster therefore treats the proposed cost trajectory λt:t+L as an input. The deployment-time optimization of λt is outside the scope of this paper; Appendix B.3 describes how capacity scenarios are mapped to cost trajectories for training-data generation. A.2

Simulator Construction

Given historical data D collected under a historical operating policy πD , we construct the simulator of the main text as M̂ = (ξD , P̂ , R̂), where ξ D are replayed historical exogenous sequences, and P̂ , R̂ are learned approximations of the endogenous transition dynamics P and per-agent reward Rti defined in Section A.1. Conditional on ξ D , endogenous states are simulated through P̂ , enabling counterfactual rollouts under alternative capacity plans and coordination signals. Following Madeka et al. [1], Exo-IDP simulators constructed from historical data admit counterfactual policy-evaluation guarantees under conditions including replayable exogenous trajectories and sufficiently calibrated endogenous dynamics. For the setting here, this requires the simulator to reproduce the response induced by the deployed policy with enough fidelity to provide a useful training signal; Sections 4.1 and C evaluate this fidelity empirically under the deployment conditions.

B

Response Forecaster and Training-Data Generation

B.1

Base Architectures

The main text writes the response forecaster compactly as Jˆt:t+L = ψθ (Xt , λt:t+L ) (Eq. 3), with Xt constructed from local-state and action trajectories over the lookback window. In this appendix, we expand Xt into per-product histories over the active product set St (defined in Appendix A) and aggregate/global histories; note that the implemented Xt additionally includes known future covariates over the forecast horizon (e.g., holiday distances and forecasted demand, listed below). We then specify the encoder and decoder for the two forecaster classes evaluated in Appendix D. Input.

We use H=64 historical steps and L=26 forecast horizon. i

• xit−H:t ∈ RH×d : per-product historical features for product i ∈ St (g)

• xt−H:t ∈ RH×dg : global historical features (aggregated across products) (f )

• xt:t+L ∈ RL×df : future known features (e.g., holidays, forecasted demand) • Capacity signal: λt:t+L ∈ RL ≥0 , the proposed cost trajectory Encoder. Each historical sequence is mapped to a fixed-size embedding by a stacked dilated causal CNN: eit = Encθ (xit−H:t ), dilation rates [1, 2, 4, 8, 16, 32]. (g)

(g)

Applied to the global history xt−H:t this yields et ; applied to each product history xit−H:t this yields eit ∈ Rde , where de =32 is the encoder output dimension. Decoder. For response prediction, the decoder maps the relevant state representation and proposed cost trajectory to a multi-horizon utilization forecast. Global Aggregate uses aggregate features, while Bottom-Up sums independent per-agent predictions over St :  (g) (f )  Global Aggregate, Decθ (et , xt:t+L , λt:t+L ), X ˆ (f ) Jt:t+L = Deca (eit , xt:t+L , λt:t+L ), Agent Bottom-Up.   i∈St

Concretely, each decoder processes the cost trajectory and future features through another dilated CNN, concatenates the result with the remaining inputs, and applies a two-layer ELU MLP to produce its horizon-level output. 9

B.2

Input Features

The forecasters consume the following features. Historical features span [t−H, t); future features span [t, t+L). • Historical state (aggregate, [t−H, t)): order quantities P P corrected demand i D̂i , inbound i J i .

i i a , inventory

P

i i I , availability-

P

• Historical state (per-product i ∈ St , [t−H, t)): order quantity ai , inventory I i , demand D̂i , inbound J i , unit economics (price, cost). P i • Forecasted aggregate ([t, t+L), L weeks): mean demand D̂t:t+L = i D̂t:t+L ; projected P i Pt+L−1 i
inventory after expected drain i It − τ =t D̂τ . • Time-series (global): distance to public holidays. • Capacity cost (global): proposed cost trajectory λt:t+L . B.3

Training-Data Generation

The response forecaster is trained on cost trajectories and the aggregate utilization they induce in closed-loop simulator rollouts, so training requires a distribution over cost trajectories λ0:T that covers the capacity-control problems the deployment will face. We use a constraint-space sampling (n) strategy. First, target capacity trajectories G0:T are generated using a truncated Haar wavelet basis, following [2]: random Gaussian coefficients produce a diverse family of piecewise-constant capacity paths with controlled temporal variation, capturing realistic on/off capacity shifts (Fig. 1). Each sampled path is rescaled to the aggregate utilization scale of the target population; in our supply-chain setting, this corresponds to aggregate inbound volume. Second, each sampled plan is mapped to (n) a broadcast cost trajectory λ0:T by a trained dual coordinator from the coordination system this forecaster belongs to [6], which focuses the training distribution on cost signals relevant to feasible capacity-control problems rather than arbitrary perturbations in λ-space.

Figure 1: Example capacity target trajectories generated by the wavelet sampler using a truncated Haar wavelet basis. (n)

The simulator is then rolled out under each λ0:T , applying the broadcast costs to the fixed local (n) policies and recording the induced aggregate utilization J0:T . This yields supervised training pairs (n)

(n)

(n)

(Xt , λt:t+L ) 7→ Jt:t+L , (n)

(n)

where at each training step t the window λt:t+L and the corresponding target Jt:t+L are extracted from the rollout. 10

B.4

Training Procedure

At each training epoch, a capacity-target path G0:T ∼ PG is sampled from the wavelet distribution and mapped to a cost trajectory as above. The response forecaster ψθ is trained by offline supervised regression on the resulting rollouts, without backpropagating through the local policies: Jˆt:t+L = ψθ (Xt , λt:t+L ),

L(θ) =

X

Jˆt:t+L − Jt:t+L

t

2

. 2

The local policies are frozen throughout; only ψθ is updated. B.5

Training Hyperparameters

Table 4 lists the training hyperparameters for the response forecaster. Table 4: Training hyperparameters for the response forecaster ψθ . Hyperparameter

Value

Optimization Optimizer Learning rate Batch size Cohort resampling Training epochs Gradient clipping

Adam 10−3 3,000 every 10 epochs 2,000 0.01 (ℓ∞ -norm)

Sequence lengths History window H Forecast horizon L Training episode length Evaluation horizon

64 26 156 52

Weeks of lookback Weeks ahead Weeks per episode; start randomized Weeks per evaluation rollout; held out from training

Architecture CNN channel width Future CNN channel width Decoder hidden size History dilation rates Future dilation rates Encoder embedding de

32 16 32 [1, 2, 4, 8, 16, 32] [1, 2, 4, 8, 11] 32

Both history and future encoders

C

Notes

Fixed throughout Products sampled per rollout; 5 cohorts per epoch Uniform over the study population

Two-layer ELU MLP Stacked causal CNN Reversed causal CNN Per-product encoder output

Additional Simulator Diagnostics

The analyses in this appendix support the component-level fidelity results of Section 4.1 with scatter diagnostics, an exogeneity check, a transition-logic validation, rollout-protocol sensitivity, and a policy-sensitivity analysis. The diagnostics are measured on Study 1, comparing simulated quantities against realized observations at the product–week level under the deployed policy π, using the calibration regression of Section 4.1. The regressions are fit by ordinary least squares without intercept over all product–week pairs of the Study 1 cohort in the evaluation window; reported intervals are the 95% OLS intervals for the slope, and R2 is the uncentered coefficient of determination of the no-intercept fit. C.1

Component-Level Transition Fidelity

Figure 2 shows the product–week scatter underlying the calibration slopes reported in Table 1, measured under the one-step rollout protocol (Appendix C.4). One plausible explanation for the order and inbound gaps is that production execution constraints modify orders but are not represented in the simulator; the diagnostic does not identify the cause. Because aggregate inbound is the forecasting target, product-level inbound calibration provides a finer-grained diagnostic of the simulator-generated target. 11

C.2

Exogeneity Check

The replay argument of Section 3.1 assumes that exogenous processes evolve independently of the ordering actions. Study 1’s randomized design permits a direct test: for each exogenous variable, we regress its observed value on the treatment indicator; under exogeneity, the treatment coefficient should be indistinguishable from zero. Table 5 reports the estimates. Price and cost are consistent with exogeneity, while demand and vendor lead time show statistically significant but small differences (both with very low R2 ), indicating that the assumption holds approximately but not perfectly and is worth further investigation. Table 5: Exogeneity check (Study 1): regression of representative exogenous variables on the treatment indicator. Under exogeneity, coefficients should be indistinguishable from zero.

C.3

Exogenous variable

Coef. (95% CI)

R2

Demand Price Cost Vendor lead time

[1.37, 2.44] [−1.68, 0.18] [−0.26, 0.19] [0.35, 0.37]

0.02 0.08 0.05 0.01

Transition-Logic Validation

To confirm that the calibration gaps in Table 1 arise from the learned model components rather than implementation errors, we validate the simulator’s deterministic inventory-update logic: feeding realized orders, receipts, and sales into the inventory transition equations of Section 3.1 reconstructs realized inventory with regression coefficient [0.95, 0.96] and R2 = 0.948. State propagation is therefore correctly implemented, and the residual fidelity gaps are attributable to the learned order and inbound components. C.4

Rollout-Protocol Sensitivity

The diagnostics above use a one-step rollout strategy that reinitializes the simulator from realized states at each evaluation week. Table 6 compares this with a continuous rollout that initializes once and simulates forward without resetting. Calibration slopes are broadly similar across the two rollout protocols, and inventory and sales drain remain well calibrated under both; pointwise agreement for inbound, however, decreases under continuous rollouts as stochastic variation accumulates. Averaging over multiple independent rollouts reduces this variation for flow quantities. Table 6: Rollout-protocol sensitivity (Study 1): product–week calibration under one-step versus continuous rollouts. The one-step column corresponds to the diagnostics of Table 1.

C.5

Component

One-step Coef. (95% CI)

R2

Continuous Coef. (95% CI) R2

Order quantity Inbound Inventory Sales drain

[0.70, 0.71] [0.67, 0.67] [0.90, 0.91] [0.99, 1.00]

0.24 0.47 0.81 0.92

[0.65, 0.66] [0.65, 0.66] [1.04, 1.04] [0.94, 0.94]

0.25 0.12 0.87 0.87

Policy Sensitivity

A simulator built from historical data must be assessed under the policy for which it will be used. We roll out the deployed policy for all products – including those that operated under the baseline policy in reality – and compare calibration on the two groups. For baseline products, the comparison is between the simulated deployed policy and realized baseline-policy outcomes, so part of the divergence is expected from the policy difference itself rather than simulator error. As shown in Figure 3, calibration is stronger for products actually operated under the deployed policy (on-policy) than for those operated under the baseline (off-policy). This divergence is evidence that measured fidelity is policy-conditional. It does not by itself identify the mechanism causing the off-policy discrepancy, but it directly supports the premise of the main text: data collected under one policy need not characterize behavior under another, whether that data is real or simulated. 12

Figure 2: Trajectory fidelity under the deployed policy: each point represents a product–week, with the x-axis showing simulated values and the y-axis showing corresponding realized observations; both axes of each panel are scaled to [0, 1] by a common constant, which leaves the regression slope and R2 unchanged. A linear regression fit reports slope and R2 , with the dotted line indicating perfect calibration.

Figure 3: Policy sensitivity: the x-axis shows simulated values under the deployed policy, the y-axis realized observations; both axes of each panel are scaled to [0, 1] by a common constant. Separate regressions for products operated under the deployed policy (on-policy) and under the baseline (off-policy); the dotted line indicates perfect calibration.

13

D

Additional Transfer Results

This appendix provides full detail for the backtest results summarized in Section 4: learnability of simulated dynamics, Sim2Real transfer, and calibration with early real observations. Backtest data cutoffs. The simulator’s transition and reward models and the forecaster ψ̂sim are fit before each deployment’s evaluation window, whose realized outcomes are held out. Responsemodel training uses simulator rollouts from the pre-evaluation period. For the direct simulatorfidelity diagnostic, realized exogenous sequences from the evaluation window are replayed, so the measured error reflects the simulator’s endogenous dynamics rather than exogenous forecast error. Adaptation follows the expanding-window protocol of Section 3.3: calibration begins after the first four deployment weeks, and at each forecast origin uses only target-policy outcomes observed by that time, so no future labels enter the calibration. Zero-shot and calibrated variants are evaluated on the same forecast origins. Confidence intervals in the tables below are 95% normal-approximation intervals across weekly forecast origins with one error observation per origin and horizon. Section 2 discusses how this backtest design relates to hindcast evaluation and off-policy evaluation. The main text uses the Global Aggregate forecaster; as a robustness check, the tables below repeat available analyses with a Bottom-Up forecaster that sums per-product predictions. D.1

Learnability of Simulated Dynamics: Sanity Check

Before the transfer evaluation, we check that simulator rollouts provide a stable supervised training signal at all. Models are trained on simulator rollouts and evaluated on temporally held-out simulator data from the same study; real-data references use the same model classes and training procedures. This is a Sim→Sim diagnostic, not a transfer result. As shown in Table 7, simulator-trained models achieve stable errors on held-out simulator trajectories, with error ranges broadly comparable to those of models trained and evaluated on historical real data, for both model classes. Table 7: Learnability of simulator-generated dynamics. Models are trained on simulator rollouts and evaluated on held-out simulator data from the same study. Historical real-data baselines are trained on pre-study historical observations and evaluated on held-out real data from the Study 2 evaluation period, using the same model classes and training procedures. Reported values are MAPE (mean ± 95% CI). Training Source

Eval Source

Model

1 week

2 weeks

3 weeks

4–10 weeks

Simulator (Study 1) Simulator (Study 1)

Simulator (Study 1) Simulator (Study 1)

Global Aggregate Bottom-Up

9.2 ± 1.8 12.2 ± 3.1

9.2 ± 1.6 11.5 ± 2.7

10.8 ± 2.0 10.7 ± 2.5

11.1 ± 2.0 9.6 ± 0.9

Simulator (Study 2) Simulator (Study 2)

Simulator (Study 2) Simulator (Study 2)

Global Aggregate Bottom-Up

10.0 ± 1.4 10.0 ± 2.6

10.5 ± 1.5 12.9 ± 2.8

11.9 ± 1.6 14.0 ± 3.0

9.1 ± 1.3 12.2 ± 1.7

Historical Real Data Historical Real Data

Held-out Real Data Held-out Real Data

Global Aggregate Bottom-Up

11.7 ± 1.5 8.9 ± 2.2

11.9 ± 1.5 9.9 ± 2.3

11.8 ± 1.4 10.5 ± 2.4

11.5 ± 1.3 9.7 ± 1.2

D.2

Zero-Shot Transfer: Full Results

Table 8 reports the available horizon-resolved zero-shot transfer results summarized in Section 4.2. The source results do not include an off-policy Bottom-Up row for Study 2, so that comparison is not reported. Benchmark construction. In Study 1 the historical benchmark shares the simulator-trained forecaster’s architecture, is fit to realized observations of the same product cohort from the weeks preceding the deployment, and is scored on the same series at the same forecast origins. Study 2 follows a similar construction. D.3

Sim2Real Transfer with Real-Data Adaptation

We apply lightweight calibration using observations collected early in the real-world evaluation period. For each model, we fit a simple regression adjustment from model forecasts to realized inbound outcomes, and then evaluate the calibrated forecasts on the remaining evaluation window. 14

Table 8: Zero-shot transfer, full results. Models are trained on pre-evaluation simulator rollouts under the deployed policy or on historical real data collected under prior operating policies (off-policy), and evaluated on realized aggregate utilization during the matched deployment windows. Reported values are MAPE (mean ± 95% CI) by forecast horizon. Training Source

Eval Source

Model

1 week

2 weeks

3 weeks

4–10 weeks

Simulator (Study 1) Simulator (Study 1)

Real (Study 1) Real (Study 1)

Global Aggregate Bottom-Up

13.3 ± 2.7 13.4 ± 3.0

11.7 ± 2.8 13.3 ± 3.0

13.2 ± 3.1 13.4 ± 2.8

14.8 ± 2.8 13.4 ± 1.1

Simulator (Study 2) Simulator (Study 2)

Real (Study 2) Real (Study 2)

Global Aggregate Bottom-Up

12.7 ± 3.6 16.4 ± 6.2

15.3 ± 7.4 17.7 ± 7.4

9.4 ± 3.8 20.1 ± 8.0

11.5 ± 3.4 19.4 ± 3.9

Off-policy Real Data Off-policy Real Data Off-policy Real Data

Real (Study 1) Real (Study 1) Real (Study 2)

Global Aggregate Bottom-Up Global Aggregate

15.0 ± 2.9 16.7 ± 2.9 25.6 ± 5.2

14.8 ± 3.1 16.8 ± 3.0 27.8 ± 6.3

15.2 ± 3.1 17.7 ± 3.0 28.1 ± 6.4

16.0 ± 3.1 19.5 ± 1.2 25.2 ± 5.7

Table 9 reports the available forecasting results after calibration. The source results do not include a calibrated off-policy Global Aggregate row for Study 2; conversely, its calibrated off-policy BottomUp row has no retained zero-shot counterpart. We therefore draw before–after conclusions only where both variants are available. Calibration lowers point-estimate error for the simulator-trained Global Aggregate model in both studies, most at horizons of three weeks or less, and for the BottomUp model at longer horizons, but effects are not uniform across models or studies. These results characterize the observed correction behavior but do not identify its cause. Table 9: Available forecasting results after regression-based calibration using early real observations. Simulatortrained models are corrected using data from the beginning of the evaluation period. Off-policy historical baselines use the same calibration procedure. The retained source results do not contain every model–study combination. Values are MAPE (mean ± 95% CI). Training Source

Eval Source

Model

1 week

2 weeks

3 weeks

4–10 weeks

Simulator (Study 1) Simulator (Study 1)

Real (Study 1) Real (Study 1)

Global Aggregate + Calib. Bottom-Up + Calib.

11.8 ± 2.6 13.5 ± 6.0

11.2 ± 3.1 12.4 ± 5.7

13.1 ± 3.6 12.3 ± 5.4

14.8 ± 3.2 11.2 ± 1.8

Simulator (Study 2) Simulator (Study 2)

Real (Study 2) Real (Study 2)

Global Aggregate + Calib. Bottom-Up + Calib.

12.6 ± 3.6 19.6 ± 6.8

13.0 ± 7.1 21.3 ± 6.3

6.9 ± 3.1 18.2 ± 9.1

9.7 ± 2.9 14.4 ± 4.7

Off-policy Real Data Off-policy Real Data Off-policy Real Data

Real (Study 1) Real (Study 1) Real (Study 2)

Global Aggregate + Calib. Bottom-Up + Calib. Bottom-Up + Calib.

15.5 ± 3.0 18.4 ± 3.0 29.9 ± 9.6

15.9 ± 3.3 15.4 ± 2.6 29.6 ± 11.7

15.8 ± 3.6 15.2 ± 2.3 28.4 ± 12.9

18.0 ± 3.4 14.1 ± 1.8 20.5 ± 5.5

Summary. Taken together, these results show that the simulator has measurable aggregate discrepancy yet generates learnable trajectories and response forecasters whose point-estimate errors on realized outcomes are lower than the corresponding off-policy historical baselines in most reported comparisons. Lightweight calibration reduces residual error in some settings but not uniformly. These conclusions are empirical, conditional on the two deployments and the simulator used here, and do not identify data source or policy alignment as the causal mechanism.

15

Related documents

• Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation#1188373
• TimeRSI: Towards Self-Improved Agentic Deployment for Time-Series Foundation Models#1356170
• Goal-Oriented Reactive Simulation for Closed-Loop Trajectory Prediction#1035438
• ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation#1326496
• Time-series Foundation Models for Predictive Control: The Role of Excitation#1215550