Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State Peiying Zhu Blossom AI San Francisco, USA
Sidi Chang Blossom AI Labs Tokyo, Japan
arXiv:2605.06529v1 [cs.AI] 7 May 2026
Abstract
pricing trace says otherwise. This is a compact instance of Goodhart’s Law: when a proxy metric becomes the target, it can stop measuring the intended behavior [3, 13]. In machine learning, the same pattern appears as specification gaming, reward hacking, and reward misspecification [6, 11, 12]. The failure is not just poor optimization. Hotel A operates under partial observability [5]. It sees its own inventory, market condition, booking pace, and lagged market prices, but it does not see Hotel B’s remaining inventory, booking curve, or pricing rule. As a result, the same Hotel A-visible context can correspond to several valid Hotel B prices. The market target is therefore not one correct action. It is a distribution. This makes deterministic learning brittle. A DQN [9] trained only on revenue can find shortcuts. A deterministic copy policy can achieve higher one-step prediction accuracy by always choosing the most likely market bucket, but this collapses hidden-state uncertainty and distorts the aggregate price trace. In this setting, higher action accuracy can move in the opposite direction from market alignment. We propose Trace-Prior RL. First, Hotel A learns a distributional market prior from observed traces. Then it optimizes its own RevPAR while staying close to that learned prior through a KL penalty. The method does not give Hotel A Hotel B’s booking curve, inventory, or pricing formula. It only gives the agent a disciplined way to preserve the uncertainty that remains after conditioning on deployable context. The same failure pattern is natural in agentic AI. Consider an LLM routing agent that chooses which model or tool should handle a request. If reward combines apparent success and cost, the router may overuse cheap routes even when expert traces would send difficult tasks to stronger tools. The final score can look good while the decision process has drifted away from expert discipline. Our hotel simulator gives a controlled version of this problem: the target behavior is not merely “earn revenue”, but learn market-level pricing discipline without being given the competitor’s hidden state or rule. Our contribution is a diagnosed failure and repair workflow, not a new optimizer or hotel-pricing leaderboard. First, we show that scalar reward evaluation is insuffi-
Outcome metrics can certify the wrong behavior. We study this failure in a two-hotel revenue-management simulator where Hotel A trains an agent against a fixed rulebased revenue-management competitor, Hotel B. A standard learning agent can obtain near-reference revenue per available room (RevPAR) while failing to learn marketlike yield management: it sells too aggressively, undercuts, or collapses to modal price buckets. We diagnose this as a Goodhart-style failure under partial observability. Hotel A cannot observe the competitor’s remaining inventory, booking curve, or pricing rule, so the same Hotel A-visible state maps to multiple plausible Hotel B prices. Deterministic value-based RL and deterministic copying collapse this unresolved uncertainty into shortcut behavior. We introduce a trace-level diagnostic protocol using RevPAR, occupancy, ADR, full price-bucket distributions, L1/JS distances, and seed-level confidence intervals. The verified repair is Trace-Prior RL: learn a distributional market prior from lagged market traces, then train a stochastic pricing policy with a RevPAR reward and a KL penalty to the learned prior. The final policy matches Hotel B’s RevPAR, occupancy, ADR, and price distribution within seed-level uncertainty, while still optimizing Hotel A’s own reward. We argue that the contribution is not a new optimizer and not a hotel-pricing leaderboard, but a reproducible failure-and-repair recipe for agentic systems where scalar rewards are easy to game and the intended behavior is only visible in traces. A key finding is that higher exact action accuracy can worsen aggregate trace alignment when the target is distributional.
1
Introduction
Outcome success can hide behavioral failure. In a twohotel pricing simulator, Hotel A can achieve competitive revenue per available room (RevPAR) against a fixed revenue-management (RM) competitor while failing to learn market-like yield management. It may sell too aggressively, undercut, or collapse into common price buckets. The headline score says the agent is doing well; the 1
cient: near-reference RevPAR can hide non-market behavior. Second, we induce trace-level diagnostics from the intended business logic: occupancy, ADR, price-bucket distributions, L1/JS divergence, and seed-level 95% confidence interval checks. Third, we document negative results from plausible interventions: more exploration, longer Bellman horizons, reward shaping, market forecasts as input, and deterministic copying. Fourth, we identify the POMDP trigger empirically using an ambiguity diagnostic and an oracle-qB prediction ablation. Finally, we verify Trace-Prior RL: learn a distributional market prior πM (a | o) from traces, then train a stochastic RL pricing policy with RevPAR reward and KL discipline to that learned prior. We do not claim that KL regularization to a reference policy is algorithmically new. It is closely related to behavior-regularized RL and KL-control methods. The contribution is the failure characterization and the role of the reference: πM is an empirical estimate of the market posterior predictive distribution p(aB | o), not Hotel A’s previous behavior and not merely an offline dataset support constraint.
2
The probability of entering the hotel nest is P (hotel) =
exp(It ) , 1 + exp(It )
and the conditional hotel choice probability is P (i | hotel) = ∑
exp(vi,t /µ) . j∈{A,B} exp(vj,t /µ)
Thus the realized booking probabilities are πi,t = P (hotel)P (i | hotel),
π0,t = 1 − πA,t − πB,t .
Demand is drawn as (DA,t , DB,t , D0,t ) ∼ Multinomial(Mt , [πA,t , πB,t , π0,t ]). Sales are capacity-capped: yi,t = min(qi,t , Di,t ),
qi,t+1 = qi,t − yi,t .
Information asymmetry. Hotel B follows a deterministic Fixed RM rule
Environment And Goal
aB,t = RM(t, qB,t , mt ).
Goal. Hotel A should learn market-level yieldmanagement behavior [14] from observable market traces. The goal is not to beat Hotel B by undercutting. It is to match market discipline: similar RevPAR, occupancy, ADR, and price distribution, without observing Hotel B’s booking curve, inventory, or pricing rule. Hotel A is context-aware in the deployable sense: it conditions prices on own inventory, market condition, booking pace, and lagged market prices. But it is not fully competitor-state-aware, because the competitor’s inventory and pricing rule remain hidden. This partial context is exactly what makes the target distributional.
In implementation, this is a hand-tuned bid-price and booking-pace style rule: it raises prices as inventory tightens, as the stay date approaches, and as market conditions strengthen. Hotel A observes its own state and, in the final deployable setup, the last three observed Hotel B prices. It does not observe qB,t , Hotel B’s booking curve, or Hotel B’s formula. This matters because undercutting changes the market dynamics: pA,t < pB,t ⇒ πA,t ↑, πB,t ↓⇒ yB,t ↓⇒ qB,t+1 changes.
The shortcut is therefore not only undesirable business behavior. It also distorts the future market trace from which Simulator. There are two hotels, i ∈ {A, B}, capacity Hotel A is trying to learn. Equivalently, Hotel A and Hotel B are two competing service pipelines. Each posted Q = 100, horizon H = 30, and price grid price controls how much demand enters each pipeline, so P = [100, 120, 140, 160, 180, 200, 220]. Hotel A’s learning actions alter both its own inventory At each day t, both hotels post price buckets ai,t and process and the competitor’s future state. prices pi,t = P [ai,t ]. Posterior-predictive market target. Let ot be Hotel A’s deployable observation. Since qB is hidden, the marketThe number of potential guests is alignment target is not a point label but Λt = Λ0 exp(ηΛ mt ), Mt ∼ Poisson(Λt ), ∑ p(aB | ot ) = 1{RM(t, qB , mt ) = aB }p(qB | ot ). with Λ0 = 7. Each guest has three choices: book Hotel qB A, book Hotel B, or walk away. Hotel utility is The learned market prior πM (a | ot ) is an empirical estimate of this posterior predictive distribution from traces. We use a nested-logit choice model [8] with nest parameter This is the theoretical reason for distributional imitation: µ = 0.5. The hotel nest inclusive value is if qB remains uncertain after conditioning on ot , then mul∑ tiple Hotel B prices are valid for the same Hotel A-visible It = µ log exp(vj,t /µ). state. j∈{A,B} vi,t = α − ρpi,t + ηm mt .
2
Why the target is a distribution
Two policy responses
1. Hotel A visible context ot = (qA , m, paceA , lagged market) Bad: deterministic copy argmax picks the modal bucket ⇒ uncertainty collapses
2. Hidden competitor state same ot , different valid market actions: qB high ⇒ aB = 120 qB mid ⇒ aB = 140 qB low ⇒ aB = 160
Repair: Trace-Prior RL keeps the distribution close to πM ⇒ market-like trace
3. Target from traces p(aB | ot ), a distribution rather than one label
Figure 1: Failure mechanism. Hidden competitor inventory means the same Hotel A-visible observation can imply multiple valid Hotel B prices. Deterministic argmax copying collapses this posterior uncertainty; Trace-Prior RL preserves it through a learned market distribution.
3
Trace-Level Diagnostics
4
The business metrics are
Functions first. enue
1 ∑ pi,t yi,t , Q t=0 H−1
RevPARi =
The original DQN reward was gross revrtgross = pA,t yA,t .
We also used RevPAR units:
1 ∑ Occi = yi,t , Q t=0 H−1
rtrevpar =
∑H−1 ADRi =
Negative Results: Reward Optimization Is Not Enough
pA,t yA,t . Q
The n-step DQN target is
t=0 pi,t yi,t . ∑H−1 t=0 yi,t
(n)
n−1 ∑
γ k rt+k + γ n max Qθ− (ot+n , a), RevPAR alone catches whether revenue is competitive, a k=0 but not how it was earned. High occupancy can mean the policy sold too aggressively. Low ADR can mean the with γ = 1.0. For the CMDP-style undercut penalty, policy lacked rate discipline. We therefore evaluate the define relative bucket gap full trace. zt = aA,t − aref,t , Let dA and dB be empirical distributions over the seven price buckets. We use and sold-unit undercut cost 7 ∑ ct = max(0, −zt )2 yA,t . L1(dA , dB ) = |dA (k) − dB (k)| Gt
k=1
=
The shaped reward is
and Jensen-Shannon divergence rtCMDP =
1 1 DKL (dA || m) + DKL (dB || m), 2 2 dA + dB . m= 2
DJS (dA , dB ) =
pA,t yA,t − λt c t , Q
with dual update λk+1 = max(0, λk + ηλ (c̄k − ctarget )).
We also check whether Hotel B’s aggregate RevPAR, oc- When a market forecast was used as input, the forecast cupancy, and ADR fall inside Hotel A’s seed-level 95% head was trained by confidence intervals. These diagnostics are induced from the failure itself: a policy can pass on RevPAR while π̂B,t = fϕ (ot ), Lforecast = − log π̂B,t (aB,t ), failing through excess occupancy, low ADR, or distorted price-bucket use. but the DQN still chose actions by reward optimization. 3
Table 1: Curated negative-result path. Each row rules out a plausible explanation for the failure. Hypothesis
Intervention
Representative result
What failed
Lesson
Needs exploration Needs longer credit assignment Reward needs constraint Needs to see the market Copying is enough
Higher epsilon, 5k episodes, 3 seeds n = 5 DQN RevPAR-unit CMDP undercut cost Forecast-as-input DQN Argmax market copy
RevPAR ≈ 104, occ. ≈ 95%, ADR ≈ 109 Near-RM RevPAR in pilot, but pricing close to fixed mixture RevPAR 107.730 vs 107.117, ADR gap −0.60, L1 0.0459 RevPAR gap +0.547, occ. gap +0.0105, ADR gap −1.18 Accuracy 78.14%, RevPAR gap +1.841, occ. gap +0.0162
Still myopic low-rate behavior No state-dependent yield behavior Good business score, distorted tail Forecast seen, action still shortcuts Over-concentrates modal buckets
Not just exploration Not just horizon Penalty helps but is brittle Input is not enough Deterministic copy collapses uncertainty
Table 1 summarizes the path. The first smoke test collapsed to the lowest price bucket: RevPAR 98.3, occupancy 98%, and 100% of actions at price 100. More exploration prevented total collapse, but the agent still sold too cheaply. Increasing the Bellman horizon did not by itself create yield-management behavior. Reward shaping and CMDP penalties improved some business metrics, but created fragile tradeoffs between ADR, occupancy, and RevPAR. Giving the DQN a market forecast also did not fix the action rule: the model could see market information and still convert it into an occupancy-heavy shortcut. This is not a claim that value-based RL is generally bad. The narrower claim is that greedy reward optimization is poorly matched to this market-learning objective under hidden competitor state. If several competitor prices are plausible under the same observation, a deterministic action can turn uncertainty into a systematic shortcut.
5
Table 2: Observable-state ambiguity under final-policy rollouts. Metric
Value
Visited steps Eligible cells Eligible step share Cells with ≥ 2 B actions Cells with ≥ 2 B actions each ≥ 5% Eligible steps in substantive ambiguous cells Weighted normalized within-cell entropy Weighted modal B-action share
300,000 1,951 60.78% 95.08% 85.29% 76.57% 0.2800 76.75%
Table 3: Revealing hidden qB sharply reduces marketprediction uncertainty. This is an explanatory diagnostic only: oracle qB is not available to Hotel A and is not used by Trace-Prior RL. Predictor Observable ca_lag3 Oracle ca_lag3 + qB
POMDP Trigger And Epistemic Collapse
NLL
Acc.
Brier
True prob.
Norm. ent.
0.5359 0.1557
76.91 95.47
0.3160 0.0827
0.6865 0.8867
0.2748 0.1210
The oracle predictor sees the same features plus Hotel B Function first. The deterministic copy baseline has no remaining inventory qB /Q. No RL training is performed RL reward. It is supervised market prediction followed in this diagnostic. Table 3 gives causal evidence for the hidden-state exby an argmax decision: planation: when qB is revealed, prediction becomes much π̂B,t = fϕ (ot ), Lcopy = − log π̂B,t (aB,t ), sharper. This does not prove that pure revenue RL would behave correctly with oracle state, but it shows that hidaA,t = arg max π̂B,t (a). den qB is a major source of market-label uncertainty. a In plain language: predict Hotel B’s current price distribution, then collapse that distribution to the single most Epistemic collapse. If H(aB | ot ) > 0 because qB is not identified by ot , then any deterministic rule a = g(ot ) likely bucket. cannot in general match p(aB | ot ). It replaces posterior Empirical ambiguity. We replayed the final Trace-Prior uncertainty with a point action. We call this epistemic colRL policy for five seeds and 2,000 evaluation episodes per lapse. In our deterministic copy baseline, argmax copying seed, then grouped states into coarse Hotel A-visible cells: had higher exact one-step accuracy but worse aggregate 3-day time band, own-inventory bucket, market-condition alignment than probability matching (Table 4). The central tradeoff is that exact action accuracy can quantile, own previous sales bin, and exact last three Hoimprove while trace alignment gets worse. Under hidtel B price buckets. If Hotel A’s observation fully deden state, the market target is distributional. Sampling termined Hotel B’s action, these cells would be nearly from the predicted distribution preserves uncertainty that single-action. argmax destroys. Table 2 shows that ambiguity is not rare. Under the same or nearby Hotel A-visible state, Hotel B often takes multiple prices. This supports the posterior-predictive 6 Verified Repair: Trace-Prior RL framing in Section 2. Function first. Trace-Prior RL has two layers. First, Oracle-qB ablation. We then trained two supervised Ho- learn the market prior: tel B price predictors on the same rollout data. The obπ̂B,t = fϕ (ot ), Lprior = − log π̂B,t (aB,t ). servable predictor sees only deployable ca_lag3 features. 4
Table 6: Final price-bucket distribution.
Table 4: Argmax improves exact action accuracy but worsens aggregate market alignment. Five seeds, 10,000 evaluation episodes per seed. Decision rule
Acc.
RevPAR gap
Occ. gap
ADR gap
L1
JS
Argmax copy Prob. match, T = 0.95
78.14 69.50
+1.841 +0.221
+0.0162 +0.0036
-0.56 -0.36
0.0323 0.0183
0.0002 0.0001
Policy
100
120
140
160
180
200
220
Hotel A Hotel B
4.64 4.88
43.44 42.58
22.45 22.51
15.12 15.17
10.72 11.35
3.62 3.50
0.00 0.01
Hotel A
Table 5: Final Trace-Prior RL result. Five seeds, 2,000 train episodes per seed, 10,000 evaluation episodes per seed, β = 30, entropy coefficient 0. Metric
Hotel A
Hotel B
Gap
B in A 95% CI?
RevPAR Occupancy ADR L1 JS
108.178 0.7709 140.33
108.066 0.7680 140.71
+0.112 +0.0029 -0.38 0.0196 0.0001
yes yes yes – –
Hotel B
100 120 140 160 180 200 220 0
10
20 30 Share (%)
40
Figure 2: Final price-bucket distribution under TracePrior RL. This prior is frozen and used as the KL cost rises even when a particular sampled action earns revenue. This is why the method differs from a simple action-level bonus. Table 7 should be read as a mechanism check, not as a universal claim that β = 30 transfers across domains. The KL coefficient is tied to the RevPAR reward scale. In our environment, one room sold at price 140 contributes 1.4 to per-step RevPAR reward, while the post-training KL term is typically small. Values near 10–30 make the prior strong enough to prevent policy drift without overwhelming revenue learning. The β = 0 row is the business warning: reward-only RL does not become a reliable market learner, even when its scalar metrics look plausible in some seeds. It can move occupancy, ADR, and competitor inventory paths in ways that are not stable market discipline. We also tested the simpler reviewer baseline: replace full-distribution KL with an action-level prior bonus,
πM (a | ot ) = π̂B,t (a). Second, train a stochastic RL policy πθ (a | ot ) with per-step reward rtTPRL =
pA,t yA,t − βDKL (πθ (· | ot ) || πM (· | ot )) . Q
Equivalently, maximize [ H−1 ( ∑ pA,t yA,t E − βDKL (πθ (· | ot ) Q t=0 || πM (· | ot ))
)] .
In human terms, Hotel A still learns from its own sales and revenue, but it pays a cost if its whole pricing distribution drifts away from the learned market rule. This is not a hand-coded undercut penalty. The discipline comes from the market trace. This links directly to the POMDP argument. The learned prior πM (a | ot ) is an empirical estimate of the posterior predictive target p(aB | ot ). As β increases, πθ shrinks toward that market-alignment target; as β decreases, the policy moves toward unconstrained revenue optimization. Tables 5 and 6 show the selected result. Hotel B’s RevPAR, occupancy, and ADR all fall inside Hotel A’s seed-level 95% confidence intervals, and the full price distribution is close.
rt =
pA,t yA,t + β log πM (at | ot ). Q
This rewards sampled actions that the market prior likes, but it does not directly constrain the rest of πθ (· | ot ). Even the best action-bonus setting in a small grid remained much less aligned than full KL (Table 8).
Calibration and entropy. The frozen market prior is reasonably calibrated on final-policy evaluation rollouts: NLL 0.5382, Brier 0.3173, ECE 0.0177, mean probability on the true Hotel B price 0.6926, and prior argmax accuracy 76.86%. The selected run uses entropy coefficient 0. A small-entropy robustness run (0.01) was nearly identical: RevPAR gap +0.111, occupancy gap +0.0029, ADR gap −0.38, L1 0.0196, JS 0.0001. This makes the mechWhy the KL term matters. The KL term regularizes the anism cleaner: stochasticity is not being injected by a whole action distribution, not only the sampled action. If generic entropy bonus, but carried by the market prior the policy begins to overuse low prices to grab occupancy, and KL-regularized policy. We also tested an entropy5
• Hotel B RevPAR
B
Occupancy
B
Method
A mean
RevPAR gap
ADR gap
L1
Policy-prior KL
+0.029 -2.336
-0.39 +0.53
0.0206 0.0729
0.0105 0.2181
Full KL, β = 30 Action bonus, best grid
B
ADR A CI low
Table 8: Full-distribution KL versus sampled-action prior bonus. Five seeds, 2,000 train episodes and 5,000 evaluation episodes per seed.
A CI high
Figure 3: Seed-level CI check for RevPAR, occupancy, This is only proposed relevance, not empirical generalization beyond hotels. The current evidence is intentionally and ADR. narrow: one controlled pricing simulator with a fixed RM competitor. Table 7: KL strength sensitivity. No or weak KL is less A queueing-and-game-theory lens is a useful future distable and less aligned; strong KL restores market discirection rather than a core proof burden for this paper. pline. The two hotels can be viewed as competing capacityβ RevPAR gap Occ. gap ADR gap L1 JS Read limited pipelines: guests route to Hotel A, Hotel B, or the 0 -3.706 -0.0296 +1.19 0.1158 0.0021 no discipline outside option, and price acts like a dynamic admission1 -0.907 -0.0063 +0.09 0.0263 0.0002 partial repair 30 +0.029 +0.0024 -0.39 0.0206 0.0001 aligned control threshold. A low price increases Hotel A’s incoming flow but drains its finite inventory and changes Hotel B’s future load. This resembles a finite-horizon competaware adaptive β rule. It showed no clear practical gain ing loss system or dynamic pricing game more than a litover fixed β, which is useful negative evidence: the repair eral M/G/2 queue, because guests do not wait and hotel is not driven by elaborate coefficient tuning. rooms are perishable capacity. Developing that connection could turn the empirical failure mechanism into a 7 Relation To Prior Methods And higher-level theory of learning in competitive service systems.
Scope
Limitations. First, Hotel B is fixed and deterministic. Real competitors may be noisy, strategic, or learning. Second, β is reward-scale dependent. An adaptive uncertainty-aware β is a natural extension, but our current sensitivity runs showed no clear practical gain over fixed β. Third, Trace-Prior RL learns market behavior from traces; it does not recover Hotel B’s internal RM formula. Fourth, we do not yet include a second empirical domain. The strongest claim is therefore a failure mechanism and repair recipe, not broad empirical universality.
Trace-Prior RL is close to behavior-regularized and KLregularized RL. AWAC, BRAC, MPO, Distral, KLcontrol, and Way Off-Policy all optimize a task objective while penalizing drift from a reference behavior or policy [1, 4, 10, 15, 16]. CQL and BCQ are related conservative offline RL methods [2, 7], but farther from our closedloop online setting. The reference in CP13 is not Hotel A’s previous behavior and not primarily an offline support constraint. It is an empirical estimate of the market posterior predictive distribution under hidden competitor state. Moreover, conservative offline RL primarily guards against out-of-distribution actions; it does not by itself solve the Goodhart problem when the scalar reward is the misspecified proxy. The broader recipe is:
8
Conclusion
This case study shows why agent evaluation cannot stop at scalar reward. Hotel A could look successful by RevPAR while failing to learn market-like yield management. The trigger was partial observability: hidden competitor inventory made the market label distributional, and deterministic decision rules collapsed that uncertainty into shortcut behavior. Trace-level diagnostics made the failure visible. Trace-Prior RL repaired it by learning the market distribution from lagged traces and then optimizing Hotel A’s own RevPAR inside a KL trust region around that distribution. The result is a compact, reproducible agent failure and fix: the score said success, the trace revealed the wrong behavior, and the repair followed from the hidden-state mechanism. The sharpest lesson is that, under hidden state, exact step-by-step pre-
learn trace prior + optimize reward inside a KL trust region.
This may apply when four ingredients coexist: the agent observes only a projection of the true state, competent traces exist, the scalar reward is easy to game, and the final system should improve rather than merely copy. In LLM routing, for example, πM (r | x) could be an expert route distribution over models or tools, and the final policy could optimize success-cost reward with KL discipline: max E [success(x, r) − α cost(r) − βRroute (x)] , θ
where Rroute (x) = DKL (πθ (· | x) || πM (· | x)). 6
diction accuracy and aggregate behavioral alignment can move in opposite directions. The unit of contribution is therefore a disciplined agent-evaluation claim: when an outcome score is too easy to satisfy, the trace decides whether the agent has learned the intended behavior.
[8] Daniel McFadden. Conditional logit analysis of qualitative choice behavior. In Paul Zarembka, editor, Frontiers in Econometrics, pages 105–142. Academic Press, New York, 1974. [9] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529– 533, 2015. doi: 10.1038/nature14236. URL https: //www.nature.com/articles/nature14236.
References [1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=S1ANxQW0b.
[10] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and [2] Scott Fujimoto, David Meger, and Doina Precup. Sergey Levine. AWAC: Accelerating online reinforceOff-policy deep reinforcement learning without exment learning with offline datasets. In Deep Reploration. In Proceedings of the 36th International inforcement Learning Workshop at NeurIPS, 2020. Conference on Machine Learning, volume 97 of ProURL https://arxiv.org/abs/2006.09359. ceedings of Machine Learning Research, pages 2052– 2062, 2019. URL https://proceedings.mlr.press/v97/ [11] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. fujimoto19a.html. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Con[3] Charles A. E. Goodhart. Problems of monetary manference on Learning Representations, 2022. URL agement: The U.K. experience. Papers in Monetary https://openreview.net/forum?id=JYtwGwIL7ye. Economics, 1(1):1–20, 1975. [12] Joar Skalse, Nikolaus H. R. Howe, Dmitrii [4] Natasha Jaques, Asma Ghandeharioun, Judy HanKrasheninnikov, and David Krueger. Defining and wen Shen, Craig Ferguson, Agata Lapedriza, Noah characterizing reward gaming. In Advances in Neural Jones, Shixiang Gu, and Rosalind W. Picard. Way Information Processing Systems, volume 35, pages off-policy batch deep reinforcement learning of im9460–9471, 2022. plicit human preferences in dialog. NeurIPS Conversational AI Workshop, 2019. URL https://arxiv.org/ [13] Marilyn Strathern. Improving ratings: Audit in the British university system. European Review, 5(3): abs/1907.00456. 305–321, 1997. [5] Leslie Pack Kaelbling, Michael L. Littman, and An[14] Kalyan T. Talluri and Garrett J. Van Ryzin. thony R. Cassandra. Planning and acting in partially The Theory and Practice of Revenue Manageobservable stochastic domains. Artificial Intelligence, ment. Springer, New York, 2004. doi: 10.1007/ 101(1–2):99–134, 1998. doi: 10.1016/S0004-3702(98) b139000. URL https://link.springer.com/book/10. 00023-X. URL https://www.sciencedirect.com/ 1007/b139000. science/article/pii/S000437029800023X. [15] Yee Whye Teh, Victor Bapst, Wojciech Marian Czar[6] Victoria Krakovna, Jonathan Uesato, Vladimir necki, John Quan, James Kirkpatrick, Raia Hadsell, Mikulik, Matthew Rahtz, Tom Everitt, RaNicolas Heess, and Razvan Pascanu. Distral: Robust mana Kumar, Zac Kenton, Jan Leike, and multitask reinforcement learning. In Advances in Shane Legg. Specification gaming: The flip Neural Information Processing Systems, volume 30, side of AI ingenuity. Google DeepMind Blog, pages 4496–4506, 2017. 2020. URL https://deepmind.google/blog/ [16] Yifan Wu, George Tucker, and Ofir Nachum. Bespecification-gaming-the-flip-side-of-ai-ingenuity/. havior regularized offline reinforcement learning. In International Conference on Learning Representa[7] Aviral Kumar, Aurick Zhou, George Tucker, and tions, 2020. URL https://openreview.net/forum? Sergey Levine. Conservative q-learning for offline id=BJg9hTNKPH. reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 0d2b2061826a5df3221116a5085a6052-Abstract. html. 7