When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game

When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker–Trader Game Siu Tung Wong1

Carlo Campajola1,2

1 Institute of Finance and Technology

1 Institute of Finance and Technology

arXiv:2610.03598v1 [q-fin.TR] 2 Oct 2026

University College London London, United Kingdom ucesstw@ucl.ac.uk

University College London London, United Kingdom 2 UZH Blockchain Center Zürich, Switzerland c.campajola@ucl.ac.uk

Accepted for publication at ICAIF 2026.

Abstract

1

Introduction

Reinforcement learning (RL) is increasingly used for financial optimal-control problems when complex dynamics make analytical strategies difficult to obtain. There are financial mathematics literactures which provides many solved models whose equations and controls could evaluate and guide learning; we ask whether RL can exploit these results. We place a proximal policy optimisation (PPO) agent in an analytically solved continuous-time broker–trader game. PPO replaces the broker and chooses its trading speed while interacting with an informed trader and stochastic uninformed order flow. We derive a finite-step reward from the broker’s continuous-time payoff and verify its discrete implementation through grid refinement and an exact one-step identity. With zero uninformed flow, a validationselected PPO–FFNN approaches the reference action. With stochastic uninformed flow, the tested PPO–FFNN and PPO–LSTM remain inaccurate, although supervised learning confirms that their actors can represent the action. Monte Carlo diagnostics show that their critics do not reliably rank nearby actions; potential-based reward shaping also gives no reliable improvement. Under partial information, a causal certainty-equivalent controller based on the broker’s observable history remains close to the reference, while PPO has larger errors and lower payoffs. Finally, we freeze the analytical policy and train PPO to adjust it after the execution cost changes. Halving the cost yields a repeatable improvement that closes 2.22% of the gap to the changed-cost reference. The analytical solution therefore provides both a benchmark for diagnosing RL and a useful starting policy for adaptation.

Optimal execution balances execution costs against inventory exposure and adverse price movements. Reinforcement learning (RL) is attractive when richer market dynamics make analytical strategies difficult to obtain. We ask whether RL can recover an analytically known broker strategy from its performance criterion and use that strategy after the environment changes. Our environment follows the continuous-time game of Cartea, Jaimungal, and Sánchez-Betancourt [3], who model the interaction between a broker, an informed trader, and an uninformed trader. The broker internalises client orders and hedges in the lit market. Informed flow contains private information, while rapid trading is costly and slow trading leaves inventory exposed. Their analytical equilibrium gives an action at every state, allowing same-state action comparisons together with realised payoffs and inventory trajectories. Prior studies use RL to adjust an Almgren–Chriss schedule, learn liquidation policies against analytical benchmarks, or structure actor–critic learning with closed-form policy and value functions [8, 15, 24]. They do not test whether a finite-step reward recovers the known state-dependent strategy in this broker–trader game. Analytical-policy initialisation and adaptation have also been proposed [21, 24]. We separate discretisation, policy learning, and hidden-state estimation. We study three research questions: • RQ1: Full-information control. Can PPO–FFNN and PPO–LSTM recover the broker’s analytical trading strategy when every relevant state variable is observed? • RQ2: Partial-information control. How closely can PPO– FFNN and PPO–LSTM approach the certainty-equivalent benchmark [16] when the informed trader’s inventory, signal, and current trading rate are hidden? • RQ3: Adapting an analytical policy. When the execution cost differs from the value used to calculate the analytical policy, can PPO learn an adjustment that improves the unchanged policy?

CCS Concepts • Computing methodologies → Reinforcement learning; • Applied computing → Economics.

Keywords optimal execution, stochastic control, proximal policy optimization, partial observability, potential-based reward shaping, residual reinforcement learning, model misspecification, analytical policy initialization, analytical benchmarks

Section 2 reviews related work, Section 3 introduces the environment, Section 4 answers RQ1–RQ2, and Section 5 answers RQ3. 1

Wong and Campajola

2

applies Equation (3) at every state generated by the broker’s actions. Therefore even if the PPO agent takes suboptimal-actions, the informed trader continues to use the analytical feedback rule and does not operate in subgame nash equilibrium. We refer to it as the analytical-feedback trader. The environment uses Gymnasium 1.3.0 and PPO–FFNN is implemented with Stable-Baselines3 2.9.0 and PPO–LSTM with SB3Contrib 2.9.0. The main parameters are 𝑇 = 1, Δ𝑡 = 2 × 10−4 , 𝑎 = 0.0012, 𝑏 = 𝑐 = ℎ = 0.001, 𝑝 = 𝑟 𝐵 = 𝑟 𝐼 = 0, 𝜑 = 𝜓 = 1, 𝜅𝛼 = 5, 𝜎𝛼 = 𝜎𝑆 = 1, and 𝜅𝜉 = 15, with 𝜎𝜉 ∈ {0, 100}. Initially 𝑆 0 = 100 and all other states are zero. Here 𝑎 is the broker’s lit-market executioncost coefficient, while 𝑏 and 𝑐 determine the transaction prices for informed and uninformed client flow; ℎ, 𝑝 are impact and resilience; 𝑟 𝐵 , 𝑟 𝐼 are running inventory penalties; and 𝜑,𝜓 are terminal inventory penalties. Thus 𝑝 = 0 gives zero resilience and 𝑟 𝐵 = 0 removes the broker’s running, but not terminal, inventory penalty. Let 𝑋𝑡𝐵 be the broker’s cash. The broker maximises   ∫ 𝑇 𝐽 𝐵 (𝜈) = E 𝑋𝑇𝐵 + 𝑄𝑇𝐵 𝑆𝑇 − 𝜑 (𝑄𝑇𝐵 ) 2 − 𝑟 𝐵 (𝑄𝑡𝐵 ) 2 d𝑡 . (4)

Related Work

Analytical execution and broker games: Analytical execution models relate trading actions to inventory, impact, order flow, and predictive signals [2, 4]. Broker–trader games add strategic interaction because hedging affects prices while client orders change the broker’s inventory and information [3, 25]. We use the known broker action as the RL target. RL for optimal execution: RL has learned liquidation schedules through value-based methods, PPO, and continuous-time actor–critic [12, 14, 17, 19, 21]. Hybrid methods adjust or initialise from an analytical schedule, or embed closed-form structure [8, 21, 24]. The broker–trader equilibrium instead permits same-state action comparisons. Although policy gradients have convergence results for linear–quadratic control [6], PPO remains sensitive to implementation and evaluation choices [1, 5, 7]. Analytical guidance and reward shaping: Analytical equations can guide learning and provide a benchmark. Prior work adds a Hamilton–Jacobi–Bellman loss or closed-form policy and value structure [18, 24]. Residual RL adds a learned action to an existing controller [11]; in Section 5, we use it to adjust the frozen broker action after the execution cost changes. We also test potential-based shaping, which preserves policy ordering [20, 23]. RL under partial information: With hidden state, an agent can summarise observations recurrently or estimate the hidden variables explicitly [10, 13]. We compare feedforward and recurrent PPO with a certainty-equivalent controller that updates such an estimate from observable history.

0

After substituting the cash and price dynamics, this equals an action∫𝑇 independent constant plus E 0 𝑔𝑡𝐵 d𝑡, with 𝑔𝑡𝐵 = − 𝑎𝜈𝑡2 + 𝑏𝜂𝑡2 + 𝑐𝜉𝑡2   + 𝑄𝑡𝐵 𝛼𝑡 + ℎ𝜈𝑡 − 𝑝𝑌𝑡 − 2𝜑 (𝜈𝑡 − 𝜂𝑡 − 𝜉𝑡 ) − 𝑟 𝐵 𝑄𝑡𝐵 .

Thus 𝐽 𝐵 is the broker’s total expected payoff, while 𝑔𝑡𝐵 is its actiondependent payoff rate during trading.

3.2

3 Environment Design 3.1 Broker Execution Game

d𝑄𝑡𝐼 = 𝜂𝑡 d𝑡,

d𝑌𝑡 = (ℎ𝜈𝑡 − 𝑝𝑌𝑡 )d𝑡,

d𝛼𝑡 = −𝜅𝛼 𝛼𝑡 d𝑡 + 𝜎𝛼 d𝑊𝑡𝛼 ,

d𝜉𝑡 = −𝜅𝜉 𝜉𝑡 d𝑡 + 𝜎𝜉 d𝑊𝑡 , 𝜉

The broker’s trade changes inventory, incurs execution cost, and affects the price at which its remaining inventory is valued. The analytical solution gives the feedback controls (2)

𝜂𝑡★ = 𝑃21 (𝑡)𝑄𝑡𝐵 + 𝑃 22 (𝑡)𝑄𝑡𝐼 + 𝑃 23 (𝑡)𝑌𝑡 + ℎ 1 (𝑡)𝛼𝑡 + ℎ 2 (𝑡)𝜉𝑡 .

(3)

𝐵 𝑄𝑘+1 = 𝑄𝑘𝐵 + 𝑢𝑘 Δ𝑡,

𝐼 𝑄𝑘+1 = 𝑄𝑘𝐼 + 𝜂𝑘 Δ𝑡,

𝑌𝑘+1 = 𝑌𝑘 + (ℎ𝜈𝑘 − 𝑝𝑌𝑘 )Δ𝑡,

𝛼𝑘+1 = 𝜌𝛼 𝛼𝑘 + 𝜎𝛼 Δ𝑊𝑘𝛼

𝜉 𝜉𝑘+1 = 𝜌 𝜉 𝜉𝑘 + 𝜎𝜉 Δ𝑊𝑘 ,

𝑆𝑘+1 = 𝑆𝑘 + 𝜇𝑘 Δ𝑡 + 𝜎𝑆 Δ𝑊𝑘𝑆 , .

(6)

The three Δ𝑊 terms are independent Gaussian increments with variance Δ𝑡. We define 𝜏𝑘 = (𝑇 −𝑡𝑘 )/𝑇 as the fraction of the trading horizon remaining. Full-information PPO receives 𝑆𝑘 , 𝑄𝑘𝐵 , 𝑄𝑘𝐼 , 𝑌𝑘 , 𝛼𝑘 , 𝜉𝑘 , and 𝜏𝑘 . Partial-information PPO receives 𝑆𝑘 , 𝑄𝑘𝐵 , 𝑌𝑘 , 𝜉𝑘 , 𝜏𝑘 , and 𝜈𝑘 −1 , while 𝑄𝑘𝐼 and 𝛼𝑘 are hidden. Before entering either network, 𝑆 is centred at 100, 𝑌 is multiplied by 1,000, and 𝜉 and the previous action are divided by 100; we found these rescaling provides better results as they place the network inputs on comparable numerical scales; the dynamics and reported payoffs remain in the model’s original units. 𝐵 and After the action has been executed, the broker observes 𝑄𝑘+1 can reconstruct the informed flow from the inventory transition:

d𝑆𝑡 = (𝛼𝑡 + ℎ𝜈𝑡 − 𝑝𝑌𝑡 )d𝑡 + 𝜎𝑆 d𝑊𝑡𝑆 . (1)

𝜈𝑡★ = 𝑃11 (𝑡)𝑄𝑡𝐵 + 𝑃 12 (𝑡)𝑄𝑡𝐼 + 𝑃 13 (𝑡)𝑌𝑡 + 𝑔1 (𝑡)𝛼𝑡 + 𝑔2 (𝑡)𝜉𝑡 ,

Discrete PPO Environment

We divide [0,𝑇 ] into 𝑁 = 𝑇 /Δ𝑡 intervals and use left-endpoint Euler–Maruyama transitions [9]. Define 𝑢𝑘 = 𝜈𝑘 − 𝜂𝑘 − 𝜉𝑘 , 𝜇𝑘 = 𝛼𝑘 + ℎ𝜈𝑘 − 𝑝𝑌𝑘 , 𝜌𝛼 = 1 − 𝜅𝛼 Δ𝑡, and 𝜌 𝜉 = 1 − 𝜅𝜉 Δ𝑡. The finite-grid dynamics are

Our baseline is the continuous-time broker–trader model of Cartea, Jaimungal, and Sánchez-Betancourt [3]. An informed trader sends orders to the broker at rate 𝜂𝑡 , uninformed clients generate flow 𝜉𝑡 , and the broker hedges the resulting inventory in a lit market at rate 𝜈𝑡 . The informed trader responds to a predictive signal process 𝛼 = (𝛼𝑡 )𝑡 ∈ [0,𝑇 ] . PPO chooses the broker’s trading rate 𝜈𝑡 , while the informed trader follows its analytical feedback rule and the uninformed flow remains exogenous. Let 𝑄𝑡𝐵 and 𝑄𝑡𝐼 denote broker and informed-trader inventories, 𝑌𝑡 the price-impact state, and 𝑆𝑡 the midprice. Their dynamics are d𝑄𝑡𝐵 = (𝜈𝑡 − 𝜂𝑡 − 𝜉𝑡 )d𝑡,

(5)

𝐵 − 𝑄𝐵 𝑄𝑘+1 𝑘

. (7) Δ𝑡 The broker therefore cannot use 𝜂𝑘 when choosing 𝜈𝑘 , but can use the reconstructed value when forming its estimates for decision 𝑘 + 1. 𝜂𝑘 = 𝜈 𝑘 − 𝜉 𝑘 −

The time-dependent coefficients 𝑃𝑖 𝑗 (𝑡), 𝑔𝑖 (𝑡) and ℎ𝑖 (𝑡) follow from the Riccati equations in [3]. In the simulator, the informed trader 2

When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker–Trader Game

1. Information before decision 𝑘

2. Broker policy

3. Execution and feedback

Broker and market state 𝑆𝑘 , 𝑄𝑘𝐵 , 𝑌𝑘 , 𝜉𝑘 , 𝜏𝑘 are observed; 𝜈𝑘 −1 also enters the partial policy

Observation vector full: 𝑜𝑘𝐹 = (𝑆, 𝑄 𝐵 , 𝑄 𝐼 , 𝑌 , 𝛼, 𝜉, 𝜏)𝑘 partial: 𝑜𝑘𝑃 = (𝑆, 𝑄 𝐵 , 𝑌 , 𝜉, 𝜏, 𝜈𝑘 −1 )𝑘

Lit-market execution 𝜈𝑘 creates price impact ℎ𝜈𝑘 ; client flows update 𝑄 𝐵

Broker policy PPO–FFNN / PPO–LSTM / analytical policy + PPO adjustment

Environment transition 𝐵 ,𝑌 produces 𝑆𝑘+1, 𝑄𝑘+1 𝑘+1 and reward 𝑟𝑘

Informed trader and signal 𝑄𝑘𝐼 , 𝛼𝑘 determine 𝜂𝑘★; full-state only

full state

Uninformed clients current flow 𝜉𝑘 is observed

Reference 𝜈𝑘ref

Executed 𝜈𝑘 clipped actor action

updated state determines the ★ next informed response 𝜂𝑘+1

evaluation only next decision; past 𝜂𝑘 is now reconstructible

Figure 1: Information and feedback for one broker decision. The policy maps the full- or partial-information observation to a trading rate, which is clipped to [−100, 100] before the environment updates the state and reward. Red variables are hidden under partial information; past informed flow 𝜂𝑘 is reconstructible at decision 𝑘 + 1. The reference action (𝜈𝑘ref ) is evaluated at every PPO state. Table 1: Networks and training settings. The reported zero-flow FFNN uses an initial log standard deviation of 3; the other FFNN and LSTM experiments use 0. Checkpoints are selected by mean payoff on 50 separate validation paths. Setting

PPO–FFNN

Actor / critic

Actor and critic each use two fully connected 64-unit Actor and critic each use 24-unit LSTM and a fully ReLU hidden layers connected 64-unit Tanh hidden layer Block B: 𝜉 = 0, log std 3; 𝜎𝜉 = 100, log std 0. Block C: Blocks B and D: log std 0. Block C: supervised, 𝜉 = 0 supervised, 𝜉 = 0. Block D: 𝜎𝜉 = 100, log std 0 Adam; rate 10 −4 ; 𝛾 = 1; GAE 𝜆 = 0.99; clip 0.2; batch 250; ten epochs; value/entropy coefficients 0.5/0; gradient norm 0.5; normalised advantages; state-dependent exploration sampled every four steps

PPO Section 4 use

PPO–LSTM

Analytical policy with PPO adjustment

Training / selection

4 × 106 transitions; five training seeds; activation and checkpoints selected by finite-grid payoff on

Supervised diagnostic

ReLU FFNN and Tanh LSTM with the same actor shapes as PPO; Adam at 10 −3 , 100 epochs; 40/20/40 analytical train/validation/test episodes

3.3

separate validation paths

Let Δ𝑀𝑘𝑆 = 𝜎𝑆 Δ𝑊𝑘𝑆 . The transitions used in the reward calculation are

Δ𝑄𝑘𝐵 = 𝑢𝑘 Δ𝑡,

(8)

Applying the exact discrete product rule gives Δ(𝑄 𝐵 𝑆)𝑘 = 𝑄𝑘𝐵 Δ𝑆𝑘 + 𝑆𝑘 Δ𝑄𝑘𝐵 + Δ𝑄𝑘𝐵 Δ𝑆𝑘 𝐵 = (𝑄𝑘𝐵 𝜇𝑘 + 𝑆𝑘 𝑢𝑘 )Δ𝑡 + 𝑢𝑘 𝜇𝑘 Δ𝑡 2 + 𝑄𝑘+1 Δ𝑀𝑘𝑆 ,

(9)

while the terminal-inventory term satisfies 𝐵 (𝑄𝑘+1 ) 2 − (𝑄𝑘𝐵 ) 2 = 2𝑄𝑘𝐵 𝑢𝑘 Δ𝑡 + 𝑢𝑘2 Δ𝑡 2 .

(10)

3.4

Adding the cash increment cancels −𝑆𝑘 𝑢𝑘 Δ𝑡 + 𝑆𝑘 𝑢𝑘 Δ𝑡. Therefore, define the accumulated finite-grid objective and its one-step increment by ∑︁ V𝑘 = 𝑋𝑘𝐵 + 𝑄𝑘𝐵 𝑆𝑘 − 𝜑 (𝑄𝑘𝐵 ) 2 − 𝑟 𝐵 (𝑄 𝐵𝑗 ) 2 Δ𝑡,

Environment Step

We use PPO as our main algorithm [22] and compare FFNN and LSTM actors, each trained for four million environment transitions. At time 𝑡𝑘 , the policy samples a raw action 𝜈𝑘𝜋 (“Raw” denotes the network output before clipping), while the environment executes 𝜈𝑘 = clip(𝜈𝑘𝜋 , −100, 100), as shown in Figure 1. PPO evaluates the likelihood of the raw action during training. We record how often

𝑗 <𝑘

𝐵 V𝑘+1 − V𝑘 = 𝑔𝑘𝐵 Δ𝑡 + 𝑢𝑘 𝜇𝑘 − 𝜑𝑢𝑘2 Δ𝑡 2 + 𝑄𝑘+1 Δ𝑀𝑘𝑆 .

100 critic-only warm-up and 800 joint updates; four environments × 1,024 steps; actor/critic rates 2×10 −5 /2×10 −4 ; 𝛾 = 1, advantage-estimation 𝜆 = 0.95, clip 0.1, batch 1,024, ten epochs Five training seeds; checkpoints at updates 50, 100, 200, 400, and 800; the highest mean payoff on 50 common validation paths is selected; final evaluation on 100 previously unused paths Not used

The Δ𝑡 2 terms are the cross-products generated by the discrete product rule and squared-inventory increment. The remaining term, 𝐵 Δ𝑀 𝑆 , is the change in inventory value caused by the unpre𝑄𝑘+1 𝑘 dictable price innovation. Its conditional expectation is zero, so the reward supplied to PPO is
𝑟𝑘 = 𝑔𝑘𝐵 Δ𝑡 + 𝑢𝑘 𝜇𝑘 − 𝜑𝑢𝑘2 Δ𝑡 2 . (12) Í𝑁 −1 𝐵 We write 𝐽Δ𝑡 (𝜋) = E𝜋 [ 𝑘=0 𝑟𝑘 ]. Summing Equation (11) shows that this expected return equals the expected cash-and-inventory objective in Equation (4), up to the fixed initial value. On one realised path, the two accumulated quantities differ by the omitÍ 𝐵 ted price innovations 𝑘 𝑄𝑘+1 Δ𝑀𝑘𝑆 . The analytical payoff is undiscounted, so we set 𝛾 RL = 1.

Finite-step reward

Δ𝑆𝑘 = 𝜇𝑘 Δ𝑡 + Δ𝑀𝑘𝑆 ,
Δ𝑋𝑘𝐵 = −𝑆𝑘 𝑢𝑘 − 𝑎𝜈𝑘2 + 𝑏𝜂𝑘2 + 𝑐𝜉𝑘2 Δ𝑡 .

Fixed analytical action; two-layer, width-64 Tanh correction network; two-layer, width-128 Tanh critic Not used in Section 4

(11) 3

Wong and Campajola exec is the action taken where 𝑀 is the number of test paths, and 𝜈𝑖,𝑘 by the agent. We refer to this quantity as the LQ loss. We therefore replay the continuous analytical feedback and the finite-grid LQ feedback on 250 common market paths at three time resolutions. Table 2(a) shows that the action RMSE and LQ loss both decrease as the grid is refined. At the grid used for training, Δ𝑡 = 2 × 10−4 , they are 0.003389 and 1.383 × 10−8 . The linear dynamics and Gaussian innovations make every unconstrained reference action Gaussian. If its mean and standard deviation at decision 𝑘 are 𝑚𝑘 and 𝑠𝑘 , the union bound gives      ∑︁   100 − 𝑚𝑘 −100 − 𝑚𝑘 LQ +1−Φ . Pr max |𝜈𝑘 | > 100 ≤ Φ 𝑠𝑘 𝑠𝑘 𝑘 𝑘 (16) For the zero-flow reference this bound is 1.07×10−5 , and no crossing occurs on 500 additional simulated paths. A separate numerical check verifies Equation (11) to floatingpoint precision. Together, the grid, action-bound, and reward checks show that these numerical effects are much smaller than the PPO errors reported below.

clipping occurs and the size of the adjustment between the raw and executed actions. The informed trader calculates 𝜂𝑘★ from the state at time 𝑘 before either action changes the state.

4

Experimental Results

We now answer the research questions introduced in Section 1. Block A verifies the reference action 𝜈𝑘ref used throughout the experiments. Blocks B–C answer RQ1 by testing full-information PPO, actor capacity under supervised learning, the PPO critic’s ability to rank nearby actions, and potential-based reward shaping. Block D answers RQ2 by comparing partial-information PPO with a certainty-equivalent (CE) controller. Section 5 answers RQ3. Table 1 gives the training settings; code and fixed experiment configurations will be released after peer review. Each PPO configuration uses five training seeds and four million transitions per seed. Because PPO performance can deteriorate late in training, we evaluate checkpoints every 0.4 million transitions on 50 validation paths and select the checkpoint with the highest average broker payoff. After checkpoint selection, every trained policy is evaluated on the same 250 test paths, which are not used for training or checkpoint selection. The reported 95% Student-𝑡 intervals are calculated across five independently trained policies, all evaluated on the same test paths.

4.1

4.2

Block A: Continuous and finite-grid alignment

Block A tests whether the comparison with PPO is affected by discretising the continuous-time game. The simulator applies Equation (2) at every step of the Euler–Maruyama simulator. Let 𝑥𝑘 = (𝑄𝑘𝐵 , 𝑄𝑘𝐼 , 𝑌𝑘 , 𝛼𝑘 , 𝜉𝑘 ) T . The finite-grid linear–quadratic (LQ) action is 𝐾𝑘 𝑥𝑘 , where 𝐾𝑘 is the time-dependent feedback vector obtained from the backward recursion below. We use the reference action as shown in Figure 1, 𝜈𝑘ref = clip(𝐾𝑘 𝑥𝑘 , −100, 100). This gives a feasible reference in the PPO action range; the unconstrained LQ action is 𝐾𝑘 𝑥𝑘 . To solve the finite-grid problem, write the Euler transition and reward as

4.3

𝑥𝑘+1 = 𝐴𝑘 𝑥𝑘 + 𝐵𝑘 𝜈𝑘 + 𝜀𝑘+1, 𝑟𝑘 (𝑥𝑘 , 𝜈𝑘 ) = 𝑥𝑘T Q𝑘 𝑥𝑘 + 2𝜈𝑘 𝑠𝑘T𝑥𝑘 + 𝑅𝑘 𝜈𝑘2 ,

(13)

𝛽𝑘 = 𝑠𝑘 + 𝐴𝑘T 𝐹𝑘+1 𝐵𝑘 ,

𝜒𝑘 = 𝑅𝑘 + 𝐵𝑘T 𝐹𝑘+1 𝐵𝑘 ,

𝐾𝑘 = −𝛽𝑘T /𝜒𝑘 ,

𝐹𝑘 = 𝐻𝑘 − 𝛽𝑘 𝛽𝑘T /𝜒𝑘 .

(14)

The complete value is 𝑉𝑘★ (𝑥) = 𝑥 T 𝐹𝑘 𝑥 + 𝑐𝑘 , where 𝐹 𝑁 = 𝑐 𝑁 = 0. For noise covariance Σ𝑘 , 𝑐𝑘 = 𝑐𝑘+1 + tr(𝐹𝑘+1 Σ𝑘 ). The constant does

not change the optimal action but is used in the reward-shaping LQ experiment below. Since 𝜒𝑘 < 0, 𝜈𝑘 = 𝐾𝑘 𝑥𝑘 is the unique uncon𝐵,★ 𝐵 (𝜋) = strained maximiser. Completing the square gives 𝐽Δ𝑡 − 𝐽Δ𝑡 Í LQ 2 E𝜋 𝑘 (−𝜒𝑘 ) (𝜈𝑘 − 𝜈𝑘 (𝑥𝑘 )) . We estimate it by 𝐿LQ =

𝑀 𝑁 −1 h i2 1 ∑︁ ∑︁ LQ exec (−𝜒𝑘 ) 𝜈𝑖,𝑘 − 𝜈𝑘 (𝑥𝑖,𝑘 ) , 𝑀 𝑖=1

Block C: Actor capacity and learning diagnostics

Block C tests whether the FFNN and LSTM can reproduce 𝜈𝑘ref when that action is supplied directly. With 𝜉 = 0, we train a ReLU FFNN and a Tanh LSTM through supervised learning, using 𝜈𝑘ref as labels and the supervised learning has the same architectures as the PPO actors. We evaluate them on analytical test trajectories and then allow each actor to control the simulator. Table 2(c) shows that supervised learning substantially reduces the test error of both architectures. The supervised FFNN also remains close to the reference when it controls the simulator, whereas the LSTM result varies considerably across training seeds. Supervised learning therefore shows that both networks can approximate 𝜈𝑘ref during testing. Figure 2 shows the performance of the models. Reward and critic diagnostics: Starting from the full-information baseline (𝜎𝜉 = 100, ReLU FFNN), we change one component at a time to probe the sensitivity of PPO. The analytical broker action and corrected reward do not depend on the midprice 𝑆𝑘 . We therefore remove 𝑆𝑘 to test whether this input

where 𝑥𝑘 = (𝑄𝑘𝐵 , 𝑄𝑘𝐼 , 𝑌𝑘 , 𝛼𝑘 , 𝜉𝑘 ) T . If the next-step value has quadratic coefficient 𝐹𝑘+1 , moving one step backwards gives 𝐻𝑘 = Q𝑘 + 𝐴𝑘T 𝐹𝑘+1𝐴𝑘 ,

Block B: Full-information PPO recovery

We first set 𝜉 = 0, the zero-uninformed-flow model also considered in [3]. We then set 𝜎𝜉 = 100 to introduce stochastic uninformed flow. Five independently initialised policies per architecture observe the full state and train for four million transitions. We compare each action by the policy with 𝜈𝑘ref . For the zero-flow FFNN, we use an initial log standard deviation of 3. Table 2(b) reports an LQ loss of 0.045 ± 0.031 and an action RMSE of 5.494 ± 2.163 across five independently trained roots. Figure 2 shows that the selected FFNN approaches the zeroflow reference, whereas the LSTM remains inaccurate. Table 2(d) shows that the PPO agents tested with 𝜎𝜉 = 100 also remain inaccurate. Thus, for RQ1, PPO–FFNN can approach the full-information strategy when 𝜉 = 0, but we do not obtain similar performance after introducing stochastic uninformed flow.

(15)

𝑘=0

4

When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker–Trader Game

Table 2: Summary of Blocks A–D. Action RMSE is the root mean squared difference between the broker action and the stated reference action (𝜈𝑘ref ). (a) compares the continuous analytical action with 𝜈𝑘ref at three time steps. (b) compares full-information PPO with 𝜈𝑘ref when 𝜉 = 0. (c) evaluates the supervised FFNN and LSTM on analytical test states and then when their actions determine subsequent states. (d) reports payoff, payoff shortfall from the clipped reference, and action RMSE when 𝜎𝜉 = 100. In (b) and (d), the reference is recalculated at each state reached by the controller. Brackets are 95% intervals across five training seeds; the LSTM entry 11.85–339.70 is the five-seed range. Higher payoff is better; lower LQ loss, payoff shortfall, and RMSE are better. (b) Block B: Full information, 𝜉 = 0

(a) Block A: Grid alignment Δ𝑡

Steps 𝑁

LQ loss

Action RMSE

10 −3 5 × 10 −4 2 × 10 −4

1,000 2,000 5,000

3.462 × 10 −7 8.681 × 10 −8 1.383 × 10 −8

0.016835 0.008468 0.003389

Policy

Unconstrained LQ diagnostic

Action RMSE

FFNN–ReLU LSTM–Tanh

0.045 ± 0.031 0.688 ± 0.687

5.494 ± 2.163 16.304 ± 2.921

(d) 𝜎𝜉 = 100: feasible comparisons

(c) Block C: Direct action supervision

FFNN–ReLU LSTM–Tanh

2.31 [1.99,2.63] 2.62 [2.44,2.81]

2.06 [1.27,2.85] 11.85–339.70

5 10 40

0.0

0.2

0.4

0.6

Time

Finite-grid reference PPO--FFNN PPO--LSTM Supervised FFNN

0.0

0.2

0.4

0.6

0.8

Time

Analytical policy + PPO

0.3150

0

20

40

60

80

Training progress (% of each budget)

0.2

0.4

0.6

Time

100

PPO--FFNN PPO--LSTM Analytical policy + PPO

0.000 0.002 0.004 0.006

RMSE

0 7.191 39.324 0.00061 21.810 24.238

0 20.449 31.284 0.732 24.819 26.585

0.1297

−21.680 −24.108

(d) a = 0.0006: broker inventory

Changed-cost reference Frozen original policy Analytical policy + PPO

0.0

0.002

Shortfall

(c) a = 0.0006: trading speed

actor starts

0 5 PPO--LSTM 10 0.3175

75 50 25 0 25 50 75

𝐵 𝐽Δ𝑡

0.1303

−7.060 −39.194

0.8

(f) PPO actor loss Clipped-surrogate loss

Mean J Bt (separate scales)

(e) Validation broker payoff

0 PPO--FFNN 1 2

0.3200

Clipped reference FFNN–ReLU LSTM–Tanh CE estimate FFNN–ReLU LSTM–Tanh

(b) = 0: broker inventory

1 0.5 0 0.5 1 5

0.8

Full Full Full Partial Partial Partial

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Environment transitions (millions)

4 2 0 2 4 6 8

Changed-cost reference Frozen original policy Analytical policy + PPO

0.0

0.2

0.4

0.6

Time

0.8

(g) Critic value loss 101

Value loss (log scale)

0

5

Inventory QkB

Finite-grid reference PPO--FFNN PPO--LSTM Supervised FFNN

Controller

Trading speed k

(a) = 0: trading speed

50 20 10 5

Info.

Inventory QkB

Supervised closed-loop RMSE

Trading speed k

Actor

Supervised test RMSE

100 10 1

PPO--FFNN PPO--LSTM Analytical policy + PPO

10 2 10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Environment transitions (millions)

Figure 2: Trajectories and PPO diagnostics. Panels (a)–(b) show the zero-flow experiment and (c)–(d) the change from 𝑎 = 0.0012 to 0.0006. Lines are means and bands span the 5th–95th percentiles; (a)–(b) use signed symmetric-log vertical scales. Panel (e) uses separate payoff scales and normalises progress by each training budget; markers denote checkpoints. Panels (f)–(g) show losses smoothed over 25 updates and averaged across five training seeds. PPO–FFNN/LSTM are from Block B and “Analytical policy + PPO” from Section 5. The dotted line marks the end of its 100-update critic warm-up. makes PPO learning more difficult, and varify the effects of the Δ𝑡 2 correction by removing it or multiplying the reward by 100. Every condition uses five seeds, four million training transitions per seed, and the same numbers of validation and test paths. Table 3 reports the resulting payoff, action RMSE, and critic explained variance. Removing the Δ𝑡 2 correction or the price input 𝑆𝑘 does not produce

a consistent payoff change across seeds. Multiplying the reward by 100 lowers the payoff for every seed, showing that PPO is sensitive to reward scale.

5

Wong and Campajola

Table 3: Full-information ReLU FFNN ablations at 𝜎𝜉 = 100. 𝐵 is evaluated using Equation (12), and action RMSE is calcu𝐽Δ𝑡 𝑟𝑒 𝑓 lated using 𝜈𝑘 . Explained Variance (EV) is the critic’s value predictions at the selected checkpoint. Relative to the corrected baseline, paired payoff changes (95% intervals) are 1.48[−7.34, 10.31] without the Δ𝑡 2 term, −29.24[−33.93, −24.54] after multiplying rewards by 100, and −1.09[−8.56, 6.37] without 𝑆𝑘 . Higher payoff and EV, and lower RMSE, are better. Training reward / input Corrected / with 𝑆𝑘 First order / with 𝑆𝑘 Corrected ×100 / with 𝑆𝑘 Corrected / without 𝑆𝑘

𝐵 𝐽Δ𝑡

Action RMSE

EV

−5.75±4.82 −4.27±5.59 −34.99±.78 −6.84±7.79

19.76±4.15 17.95±4.63 31.01±.19 19.13±5.66

.93±.07 .79±.15 .42±.35 .84±.15

trained with shape

★ = 𝑟𝑘 + 𝑉𝑘+1 (𝑥𝑘+1 ) − 𝑉𝑘★ (𝑥𝑘 ),
2 shape E[𝑟𝑘 | 𝑥𝑘 , 𝜈𝑘 ] = 𝜒𝑘 𝜈𝑘 − 𝐾𝑘 𝑥𝑘 ≤ 0.

𝑟𝑘

episode returns differ by the same constant. Reward shaping therefore changes the learning signal without changing the ordering of policies [20]. Since 𝐾𝑘 𝑥𝑘 is calculated from the unconstrained finite-grid LQ problem it can produce trading speeds that lie outside the action range, the squared term measures deviation from the unconstrained LQ action. We train ten FFNN policies, five with the original reward in Equation (12) and 5 with the shaped reward in Equation (18). Each pair has the same initial weights, market paths, and training settings. Shaping increases mean payoff by 1.57 (95% interval [-5.00,8.14]) and reduces action RMSE by 1.76 (95% interval [-4.50,8.02]); both changes are favourable in three of five seeds. It also increases the per-step reward standard deviation by approximately 75%. Raw training actions cross the feasible range in approximately 0.1% of decisions under either reward. Under this protocol, shaping changes the learning signal but does not reliably recover 𝜈𝑘ref .

PPO learns a state-value function 𝑉𝜃 (𝑠). Since 𝛾 RL = 1, the Bellman relation gives   𝜋 𝑄𝑘𝜋 (𝑠, 𝑎) = E 𝑟𝑘 (𝑠, 𝑎) + 𝑉𝑘+1 (𝑠𝑘+1 ) | 𝑠𝑘 = 𝑠, 𝜈𝑘 = 𝑎 . We test whether the critic correctly ranks two actions near the trained policy. At a test state 𝑠 that is not used to train or select the critic, let 𝜈¯𝜋 (𝑠) be the deterministic policy action and set 𝑎 ± = clip(𝜈¯𝜋 (𝑠) ± 10, −100, 100). Clipping is inactive in these 120 comparisons, so the two actions are 20 units apart. For Monte Carlo sample ℓ, 𝑟𝑘ℓ (𝑠, 𝑎) is the immediate reward after taking action 𝑎, and

4.4

H𝑘𝐵 = 𝜎 (𝑜 0𝑃 , 𝜈 0, . . . , 𝜈𝑘 −1, 𝑜𝑘𝑃 )

# " 𝐿     ∑︁ (𝑎 ,ℓ ) (𝑎 ,ℓ ) b𝜃,𝑘 = 1 Δ𝑄 𝑟𝑘ℓ (𝑠, 𝑎 + ) + 𝑉𝜃 (𝑠𝑘+1+ ) − 𝑟𝑘ℓ (𝑠, 𝑎 − ) + 𝑉𝜃 (𝑠𝑘+1− ) , 𝐿 ℓ =1 {z } | {z } | bℓ (𝑠,𝑎 + ) 𝑄 𝜃,𝑘

Δ𝑄𝑘MC =

𝐿 ℓ =1

 𝜋 𝜋 (𝑎 − ) , (𝑎 + ) − G𝑘,ℓ G𝑘,ℓ

Block D: Partial-information control

Block D asks how closely PPO can approach the causal CE controller when both use the same information available before each decision. PPO observes (𝑆, 𝑄 𝐵 , 𝑌, 𝜉, 𝜏, 𝜈𝑘 −1 ), while 𝑄 𝐼 , 𝛼, and the current informed-trader rate are hidden before acting. The CE controller uses the game dynamics and the broker’s causal history

(𝑎,ℓ ) 𝑠𝑘+1 is the resulting next state. Replacing 𝑉 𝜋 with 𝑉𝜃 and averaging over 𝐿 simulated transitions gives

𝐿 1 ∑︁ 

(18)

Because 𝑉𝑁★ = 0 and the initial state is fixed, the shaped and original

to estimate the hidden components 𝑄𝑘𝐼 and 𝛼𝑘 . It then substitutes these estimates into the finite-grid feedback rule: 𝜈𝑘CE = Π A (𝐾𝑘 𝑥b𝑘 ) .

bℓ (𝑠,𝑎 − ) 𝑄 𝜃,𝑘

Specifically, the CE controller forms

(17)

b𝐼 − = E[𝑄 𝐼 | H 𝐵 ], 𝑄 𝑘 𝑘 𝑘 |𝑘

  𝜋 (𝑎) = 𝑟 ℓ (𝑠, 𝑎) + ℓ 𝑎,ℓ 𝜋 where G𝑘,ℓ 𝑗=𝑘+1 𝑟 𝑗 𝑠 𝑗 , 𝜈 𝑗 , is the sum of the re𝑘 maining rewards from Monte Carlo sampling after selecting action b𝜃,𝑘 is the action preference predicted by the critic 𝑎 at decision 𝑘. Δ𝑄 using the immediate reward and next-state value, while Δ𝑄𝑘MC is the action preference estimated from the complete simulated returns b𝜃,𝑘 and Δ𝑄 MC have the same sign, it indicates that the critic (if Δ𝑄 𝑘 correctly ranks the two actions). The two candidate actions use the same market innovations and future policy noise, and 𝐿 = 32 Monte Carlo simulations are averaged for each comparison. We also compare 𝑉𝜃 (𝑠) with the Monte Carlo return of the current policy at 24 test states per seed. The critics can distinguish states with generally higher or lower expected remaining returns, but Figure 3 shows that they cannot reliably rank two nearby actions taken from the same state. Inaccurate local action ranking is therefore a plausible source of weak PPO performance in 𝜉 = 100. Potential-based reward shaping: We next test whether the long-horizon broker payoff provides insufficient information about the quality of each action. The potential 𝑉𝑘★ (𝑥) = 𝑥 T 𝐹𝑘 𝑥 + 𝑐𝑘 is computed from the finite-grid LQ solution, which is used to shape our reward from Equation (12). The PPO is Í𝑁 −1

𝛼b𝑘 |𝑘 − = E[𝛼𝑘 | H𝑘𝐵 ].

𝐵 , the controller reconstructs the informed flow After observing 𝑄𝑘+1 that has already occurred and updates the informed trader’s inventory: 𝑄 𝐵 − 𝑄𝑘𝐵 𝜂𝑘rec = 𝜈𝑘 − 𝜉𝑘 − 𝑘+1 , Δ𝑡 (19) b𝐼 b𝐼 − + 𝜂 rec Δ𝑡 . 𝑄 = 𝑄 − 𝑘+1|𝑘+1 𝑘 |𝑘 𝑘

Writing 𝑃𝑖 𝑗,𝑘 = 𝑃𝑖 𝑗 (𝑡𝑘 ) and ℎ𝑖,𝑘 = ℎ𝑖 (𝑡𝑘 ), we rearrange the informed trader’s feedback equation to estimate the signal: b𝐼 − 𝛼b𝑘 |𝑘 = 𝜂𝑘rec − 𝑃 21,𝑘 𝑄𝑘𝐵 − 𝑃 22,𝑘 𝑄 𝑘 |𝑘
− 𝑃 23,𝑘 𝑌𝑘 − ℎ 2,𝑘 𝜉𝑘 /ℎ 1,𝑘 ,

(20)

𝛼b𝑘+1| (𝑘+1) − = 𝜌𝛼 𝛼b𝑘 |𝑘 . The estimated informed-trader inventory and predictive signal are combined with the observed state variables to calculate the CE CE at the next decision. On 250 test paths, the reconstructed action 𝜈𝑘+1 informed-trader inventory has negligible error, while the predicted signal and pre-action informed flow have RMSE 0.0141 and 0.873. The remaining error is caused by the new signal innovation, which is unavailable before the broker acts. 6

When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker–Trader Game

Pearson=0.216 Spearman=0.053 Sign accuracy=51.7% Sign accuracy (MC CI excludes 0)=49.3%

1.00

Same action

0.25 0.00

Different actions

0.25 0.50

Different actions

1 2

50

0.50

1 0

1e 2

0.75

Payoff shortfall J B, ref J B,

Critic-predicted payoff difference

1e 2 2

0.75

Training seed Same action

2

2101 2102 2103

1 0 1 Monte Carlo: payoff(a + ) payoff(a )

2

2104 2105 Ideal

1.00

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

1e 3

2101

2103

2104

2105

40 30 20 10 0 Full FFNN

1e 2

Training seed

2102

Full LSTM

Partial FFNN

Partial LSTM

Figure 3: Learning diagnostics. Left and centre: each point compares 𝑎 + and 𝑎 − at one of 120 test states. The horizontal axis is their difference in cumulative reward estimated by Monte Carlo; the vertical axis is the difference predicted using the b𝜃,𝑘 ), where both differences subtract the payoff of immediate reward and critic value of the next state. Each point is (Δ𝑄𝑘MC, Δ𝑄 𝑎 − from 𝑎 + . In the top-right region, both estimates prefer 𝑎 + ; in the bottom-left region, both prefer 𝑎 − . In the top-left region, Monte Carlo prefers 𝑎 − but the critic predicts 𝑎 + , while in the bottom-right region, Monte Carlo prefers 𝑎 + but the critic predicts 𝑎 − . Pearson correlation measures whether the critic difference increases with the Monte Carlo difference, while Spearman correlation measures whether the critic ranks the action comparisons in the same order as Monte Carlo. The pooled Pearson and Spearman correlations are 0.216 and 0.053, and sign accuracy is 51.7%. Horizontal bars are 95% Monte Carlo intervals; open points indicate decisions where 𝑎 + and 𝑎 − appear equally good within this uncertainty. Without including open points the remaining 71 points show a sign accuracy is 49.3%. Right: each seed’s payoff shortfall from the clipped reference, with 𝜎𝜉 = 100; diamonds and bars show the mean and 95% Student-𝑡 interval. Correlations closer to one and higher sign accuracy are better; lower payoff shortfall is better. Table 2(d) shows that the CE controller remains close to 𝜈𝑘ref , while both partial-information PPO agents have substantially lower payoffs and larger action errors. The broker’s observable history therefore contains enough information for accurate control in this environment, but the PPO agents do not learn to use that history as effectively. This answers RQ2.

5

PPO adjustment after a cost change: In Enew = E (𝑎 true ), the analytical action calculated in E0 remains fixed and PPO learns an additive correction: 𝜇 (𝑜𝑘 ) = 𝜈𝑘E0 + 𝛿𝜈 (𝑜𝑘 ).

(22)

Here, 𝜈𝑘E0 is the action generated by the frozen analytical policy and 𝛿𝜈 (𝑜𝑘 ) is the residual adjustment learned by PPO. The residual is a two-layer, width-64 Tanh FFNN of the full-information observation in Enew (we also tested the ReLU setup from Block B, the wider exploration improves learning when 𝜉 = 0, but it lowers payoff for all five residual policies; thus we found this Tanh setup to be the best for our misspecification problem). Its output layer is initialised at zero, so the initial policy mean equals 𝜈𝑘E0 . PPO then trains the residual without an additional adjustment bound or penalty. Gaussian exploration is centred on the combined mean, and the environment clips sampled actions to [−100, 100]. Table 1 reports the remaining training settings. All three cost experiments use full information and 𝜎𝜉 = 100. We test 𝑎 true ∈ {0.0006, 0.0012, 0.0018} in Enew . The informed trader’s coefficients are recomputed at the true cost, while the broker retains the coefficients from E0 and does not observe 𝑎 true as an input. A separate PPO adjustment is trained for each cost. For evaluation, 𝜈𝑘ref is recomputed at the true cost. For each seed, we evaluate five checkpoints on 50 validation paths and, as in Section 4, select the checkpoint with the highest mean finite-grid broker payoff. The selected policy is then evaluated on 100 common test paths. When the execution cost is halved, all five PPO adjustments improve the test payoff. The mean increase is 0.00519, with a 95% interval of [0.00296, 0.00743] across the five independently trained policies, and closes 2.22% of the gap between the unchanged policy and the changed-cost reference. The largest residual action on the

Adapting the Analytical Policy with PPO

We now address RQ3 by testing the direction we introduced in Section 1 and proposed in [21, 24]: using an analytical policy as the starting point for RL. Our propsed method would be to freeze the analytical broker action and train PPO to correct execution-cost misspecification through a residual adjustment [11]. Let E (𝑎) denote the broker environment with execution-cost coefficient 𝑎. We calculate the analytical policy in the original environment E0 = E (𝑎 0 ), where 𝑎 0 = 0.0012, and train the PPO adjustment in Enew = E (𝑎 true ). Frozen analytical policy: In E0 , we evaluate and store the analytical coefficients 𝑃11, 𝑃12, 𝑃13 and 𝑔1, 𝑔2 from [3], with the optimal action 0 0 0 (𝑡𝑘 )𝑄𝑘𝐵 +𝑃12 (𝑡𝑘 )𝑄𝑘𝐼 +𝑃13 (𝑡𝑘 )𝑌𝑘 +𝑔10 (𝑡𝑘 )𝛼𝑘 +𝑔20 (𝑡𝑘 )𝜉𝑘 . (21) 𝜈𝑘E0 = 𝑃11

Equation (21) has the same broker-feedback form as Equation (2), and the superscript 0 identifies coefficients calculated in E0 . On 100 test paths generated at 𝑎 0 , the frozen and replayed analytical trajectories have trading-speed and inventory RMSEs of 6.36 × 10−7 and 3.29 × 10−9 , respectively. 7

Wong and Campajola

Table 4: Finite-grid payoff after changing execution cost. “Reference” is recalculated in Enew , while “frozen” uses E0 . Brackets are 95% Student-𝑡 intervals over five training seeds after averaging 100 common paths. Higher payoff and change are better. 𝑎 true

Reference

Frozen

PPO adjusted

Change (95% CI)

0.0006 0.0012 0.0018

0.53010 0.06882 −0.15783

0.29632 0.06882 −0.16164

0.30151 0.06901 −0.16145

0.00519 [0.00296, 0.00743] 0.00019 [ −0.00041, 0.00079] 0.00019 [ −0.00034, 0.00071]

by introducing a filter into the actor, and practical applications for the partial information setting would include having a 𝜉 + 𝜂 flow.

References [1] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. 2020. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study. arXiv:2006.05990 [cs.LG] doi:10.48550/arXiv.2006.05990 [2] Álvaro Cartea and Sebastian Jaimungal. 2016. Incorporating Order-Flow into Optimal Execution. Mathematics and Financial Economics 10, 3 (2016), 339–364. doi:10.1007/s11579-016-0162-z [3] Álvaro Cartea, Sebastian Jaimungal, and Leandro Sánchez-Betancourt. 2026. Nash Equilibrium between Brokers and Traders. Finance and Stochastics (2026). doi:10.1007/s00780-026-00595-7 Published online 27 May 2026. [4] Philippe Casgrain and Sebastian Jaimungal. 2019. Trading Algorithms with Learning in Latent Alpha Models. Mathematical Finance 29, 3 (2019), 735–772. doi:10.1111/mafi.12194 [5] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation Matters in Deep RL: A Case Study on PPO and TRPO. In International Conference on Learning Representations. https://openreview.net/forum?id=r1etN1rtPB [6] Maryam Fazel, Rong Ge, Sham M. Kakade, and Mehran Mesbahi. 2018. Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 1467–1476. https://proceedings. mlr.press/v80/fazel18a.html [7] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep Reinforcement Learning That Matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, Vol. 32. 3207–3214. doi:10.1609/aaai.v32i1.11694 [8] Dieter Hendricks and Diane Wilcox. 2014. A Reinforcement Learning Extension to the Almgren–Chriss Framework for Optimal Trade Execution. In 2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr). IEEE, 457–464. doi:10.1109/CIFEr.2014.6924109 [9] Desmond J. Higham. 2001. An Algorithmic Introduction to Numerical Simulation of Stochastic Differential Equations. SIAM Rev. 43, 3 (2001), 525–546. doi:10.1137/ S0036144500378302 [10] Maximilian Igl, Luisa M. Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. 2018. Deep Variational Reinforcement Learning for POMDPs. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 2117–2126. https://proceedings. mlr.press/v80/igl18a.html [11] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. 2019. Residual Reinforcement Learning for Robot Control. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 6023–6029. doi:10.1109/ICRA. 2019.8794127 [12] Christos Spyridon Koulouris and Carlo Campajola. 2026. Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Execution. arXiv:2605.20348 [q-fin.CP] doi:10.48550/arXiv.2605. 20348 [13] Gaspard Lambrechts, Adrien Bolland, and Damien Ernst. 2022. Recurrent Networks, Hidden States and Beliefs in Partially Observable Environments. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id= dkHfV3wB2l [14] Siyu Lin and Peter A. Beling. 2020. An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization. In Proceedings of the TwentyNinth International Joint Conference on Artificial Intelligence, IJCAI-20, Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4548–4554. doi:10.24963/ijcai.2020/627 Special Track on AI in FinTech. [15] Andrea Macrì and Fabrizio Lillo. 2024. Reinforcement Learning for Optimal Execution when Liquidity is Time-Varying. arXiv:2402.12049 [q-fin.TR] doi:10. 48550/arXiv.2402.12049 [16] Horia Mania, Stephen Tu, and Benjamin Recht. 2019. Certainty Equivalence Is Efficient for Linear Quadratic Control. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc. https://proceedings.neurips. cc/paper/2019/hash/5dbc8390f17e019d300d5a162c3ce3bc-Abstract.html [17] Ciamac C. Moallemi and Muye Wang. 2022. A Reinforcement Learning Approach to Optimal Execution. Working paper, current revision February 1, 2022. [18] Amartya Mukherjee and Jun Liu. 2023. Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton–Jacobi–Bellman Proximal Policy Optimization (HJBPPO). In ICML 2023 Workshop on New Frontiers in Learning, Control, and Dynamical Systems. arXiv:2302.00237 [cs.LG] https: //arxiv.org/abs/2302.00237

lower-cost test paths is 1.41, and final-action clipping occurs on 0.029% of decisions. PPO therefore provides a small, repeatable improvement only in the lower-cost case tested here.

6

Discussion and Conclusion

We studied whether PPO can recover and subsequently adapt an analytically known broker strategy. For RQ1, we have shown that PPO–FFNN approaches 𝜈𝑘ref under full information when 𝜉 = 0, however when 𝜉 = 100, the policies remain inaccurate. The finitestep reward and grid-refinement results show that these errors are not caused by numerical issues, while supervised learning shows that the FFNN can approximate and execute the reference action on states generated by the analytical policy. The critic diagnostic shows inaccurate local action ranking can contribute to the difficulty of learning from rewards. For RQ2, the certainty-equivalent controller remains close to the reference action 𝜈𝑘ref when the informed trader’s inventory and signal are hidden. Its performance shows that the available history supports accurate control in this environment, while PPO–FFNN and PPO–LSTM do not learn to use that information as effectively. For RQ3, the frozen analytical policy provides an accurate starting strategy after the execution cost changes. Residual PPO produces a small and repeatable payoff improvement when the cost is halved. The analytical solution is therefore useful both as a benchmark for diagnosing learned policies and as a prior for subsequent adaptation, thus providing some evidence of analytical policy initialisation being useful [21]. The numerical checks and supervised learning make discretisation, reward implementation, and insufficient FFNN capacity unlikely to be the primary limitations. A more plausible explanation is the interaction between exploration and reward-based value estimation. Wider initial exploration improves PPO when 𝜉 = 0, but performance remains poor after stochastic uninformed flow is introduced, where the critic also shows weak local action ranking. Partial information additionally requires relevant information to be recovered from past observations. When an analytical policy is already available, wider exploration can instead move the adjustment away from a useful starting point, while conservative adjustments produce a small improvement under the lower-cost misspecification. Future work can use analytical value or action-value information to improve the local learning signal, study alternative action and policy parameterisations, and develop structured residual policies for nonlinear costs or uncertain market impact. Under partial information, it would be interesting to test the performance of the model 8

When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker–Trader Game

[19] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. 2006. Reinforcement Learning for Optimized Trade Execution. In Proceedings of the 23rd International Conference on Machine Learning. Association for Computing Machinery, 673–680. doi:10. 1145/1143844.1143929 [20] Andrew Y. Ng, Daishi Harada, and Stuart Russell. 1999. Policy Invariance under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann, 278–287. https://ai.stanford.edu/~ang/papers/icml99-shaping.pdf [21] Brian Ning, Franco Ho Ting Lin, and Sebastian Jaimungal. 2021. Double Deep Q-Learning for Optimal Execution. Applied Mathematical Finance 28, 4 (2021), 361–380. doi:10.1080/1350486X.2022.2077783

[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https: //arxiv.org/abs/1707.06347 [23] Henry Sowerby, Zhiyuan Zhou, and Michael L. Littman. 2022. Designing Rewards for Fast Learning. arXiv:2205.15400 [cs.LG] doi:10.48550/arXiv.2205.15400 [24] Boyu Wang, Xuefeng Gao, and Lingfei Li. 2026. Reinforcement Learning for Continuous-Time Optimal Execution: Actor-Critic Algorithm and Error Analysis. Finance and Stochastics 30 (2026), 597–655. doi:10.1007/s00780-026-00589-5 [25] Xuchen Wu and Sebastian Jaimungal. 2024. Broker-Trader Partial Information Nash-Equilibria. arXiv:2412.17712 [q-fin.MF] doi:10.48550/arXiv.2412.17712

9

Related documents

• When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game#1188309
• Plan Before You Trade: Inference-Time Optimization for RL Trading Agents#1356044
• Mathematics of accounting and finance#653197
• Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning#997027
• Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QA#1122674