arXiv CS open access

Mechanism Design for Alignment and Control

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

arXiv:2609.01595v1 [econ.TH] 1 Sep 2026

M ECHANISM D ESIGN FOR A LIGNMENT AND C ONTROL* Dirk Bergemann

Andrew Koh

Stephen Morris

Yale

Columbia Google DeepMind

MIT

September 2, 2026

Abstract We develop a framework for mechanism design with AI agents whose alignment (preferences) and capabilities (feasible actions and information) are unknown. We want such agents to act on our behalf so mechanisms must incentivize both honesty and obedience. A one-sided imitation structure—capabilities can be concealed but not counterfeited—yields a revelation principle, a characterization of implementable policies via nested cyclical monotonicity, and conditions under which eliciting higher-order beliefs can discipline multiple agents. We apply our framework to stylized examples of (i) sandbagging in which a more capable agent pretends to be less capable; (ii) an alignment–interpretability trade-off, where the two are substitutes in the instrument but complements in value; (iii) discipline via peer scoring; (iv) coupling rewards to induce competition among multiple agents; and (v) scalable oversight and reward shaping.

* Preliminary; comments welcome. Authors listed alphabetically; Bergemann: dirk.bergemann@ yale.edu; Koh: andrew.koh@columbia.edu; Morris: semorris@mit.edu. We are grateful to the UK AI Security Institute ’Alignment Project’ grant AP-S2-100058 for financial support. Thanks Gillian Hadfield, Rubi Hudson, Anton Korinek, Jacob Pfau, Cecilia Wood, Zhuoran Yang and Fengzhuo Zhang for helpful conversations.

CONTENTS

Introduction

Single Agent Mechanisms

2.1

Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2

Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.3

Mechanisms and revelation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.4

A characterization of implementable policies . . . . . . . . . . . . . . . . . . .

2.5

Dimensions of single-agent mechanisms . . . . . . . . . . . . . . . . . . . . . .

3 Applications: Single Agent

Example I: Sandbagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Example II: The alignment–interpretability trade-off . . . . . . . . . . . . . . . . . .

Multi-Agent

4.1

Environment and universal types . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2

Multi-agent mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 Applications: Multi-Agent

Example III: Peer discipline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Example IV: Competition through coupled rewards . . . . . . . . . . . . . . . . . .

Example V: Scalable oversight and reward shaping . . . . . . . . . . . . . . . . . . .

Discussion

A Omitted Proofs

1 INTRODUCTION AI systems increasingly act on our behalf: they write and execute code, manage inboxes, place trades, and complete ever longer tasks that once required sustained human effort and attention (Kwa et al., 2025). The economic promise of such agentic AI lies in their ability to do things in the world. But their opacity—we still have a rudimentary understanding of how AI systems work, what they can do, and what they want—poses a conundrum for us, humans, who wish to use them. Indeed, the behavior and capabilities of AI systems are emergent byproducts of training rather than explicit design choices. Frontier models have been observed faking alignment during training (Greenblatt et al., 2024a) and scheming in evaluations (Meinke et al., 2024). Nor do we know exactly what these systems can do: capabilities are discovered rather than specified ex-ante, and models have on occasion been found to strategically underperform on evaluations (van der Weij et al., 2025). What are we to make of such agents? Is there a principled way we might interact with them? How, if at all, do the tools of ‘classical’ mechanism design carry over? Motivated by these questions, we develop a simple theory of mechanism design for AI agents that, on the one hand, draws on the ‘standard’ toolkit of mechanism design from economics but, on the other, takes three features of the alignment problem seriously: 1. Misalignment. The agents’ preferences are unknown to the designer; 2. Uncertain capabilities. The designer knows neither the set of actions the agents can execute, nor what the agents knows about the world; 3. Agents act. Agents execute actions rather than merely report information. Our theory takes incentives and preferences as the starting point for analyzing behavior. Just as how an auctioneer is able to raise revenue without any satisfactory understanding of how the human brain produces choice, our goal will be to understand and shape the behavior of AI systems at the level of preferences, beliefs, and incentives.1 In so doing, we are circumventing more ‘mechanistic’ approaches to understanding how and why AI agents behave. We view such approaches as complementary. Of course, this raises the natural question of where ‘AI preferences’ come from and what, if anything, they are. One interpretation is literal: preferences are rewards But modern AI systems have opaque and complex rewards: they emerge from pretraining simply rewarded for predicting the next token well, and these rewards are 1We will sometimes call this the ‘Humean’ view of AI behavior. For a complementary account of

when AI systems can be interpreted in these terms see Herrmann and Levinstein (2026).

modified in fairly complicated ways in post-training. Thus, the emergent preference the AI system exhibits is often opaque. But the models are optimizing for something— even if we don’t know what. Our theory will takes our subjective uncertainty about AI preferences seriously and analyze how it shapes optimal mechanisms.2 A more metaphorical view is that the language of ‘AI preferences’ is simply a conceptual tool for representing coherent behavior. This is in the revealed preference tradition. Insofar as AI systems exhibit coherent and stable choices (see, e.g. Chen et al. (2023); Mazeika et al. (2025))—and we think the pressure to make such systems economically valuable will drive the future systems further in that direction—their choices can be represented as if they were maximizing some utility function (see, e.g., Savage (1954)). On this view, ‘AI preferences’ is nothing more than a shorthand for ‘coherent AI behavior’ or ‘coherent AI choice rules’. Outline of results. We first develop a general framework to analyze the single-agent case. The AI agent’s type is a compact but complete description of behavior: it comprises its preference (alignment) and its feasible actions and information (capability). Types are private so mechanisms must contend with the possibility that agents might pretend to be as some other type e.g., through strategic underperformance in evaluations. The designer commits to a mechanism that specifies a reward schedule— payoffs from taking each action—that depends on agents’ reports. Depending on the application at hand, such a schedule might vary in coarseness. For instance, it might take interior values as in post-training in which (proxies of) desired behavior are reinforced, or extremal values in which actions are either allowed or not as when users decide how much computer-use permissions to give AI systems. An important feature of our environment is an asymmetry in who can pretend to be whom. In particular, more capable types—with larger action spaces and more precise information about the world—can pretend to be less capable types e.g., by underperforming on evaluations. By contrast, a less capable type cannot pretend to be a more capable type. We formalize this notion with a verification order (Green and Laffont, 1986) over capabilities. We then prove a revelation principle that allows us to focus on direct mechanisms that are (i) physically feasible: all types are prescribed actions they can actually do; (ii) incentive compatible: all types find it optimal to report their type honestly and obey the designer’s prescribed recommendation. The latter requires deterred double deviations in which an agent might strategically misreport its type and then disobey the ensuing recommendation. We then develop a tight characterization of what behavior can be implemented in terms of nested cyclical monotonicity. The characterization requires obedience within 2We emphasize that in most settings these emergent preferences (and our analysis) must be

conditioned on the AIs’ instructions e.g., ‘go make one million dollars’. See Goldstein and Lederman (2025) for such an argument.

each report, and truth-telling across reports.3 These conditions adjust for the fact that not every type can pretend to be every type (constraining misreporting), and that not every type can do every action (constraining disobedience). We then apply our single-agent framework to two simple but illustrative applications: sandbagging (Example I) and the value of alignment, interpretability, and control (Example II). All our applications are deliberately stylized and set within the simplest environment: There is an uncertain state of the world θ ∈ Θ ⊆ R drawn from some prior. The AI observes a signal about the world, and chooses an action from A ⊆ R where its set of available actions depend on its capability. The designer wishes for actions to match states and has payoff −( a − θ )2 . The AI might exhibit some bias and have payoff −( a − θ − b)2 where the direct and magnitude of bias b ∈ R might be unknown to the designer. We have chosen this environment because it captures notions of (i) aptness: we want actions to be appropriate for the context as captured by the state; and (ii) distance: actions can be more or less wrong, and ‘more wrong’ actions are more costly. We emphasize that this environment is certainly oversimplified. Nonetheless, it will allow us to understand and build intuitions for how incentives might interact with model behavior. We hope this will pave the way to more serious empirical work. In the sandbagging application, the AI’s capability (size of the action set) and alignment (bias relative to the human designer) are unknown. The designer first elicit type reports (e.g., via an evaluation) before choosing the permitted action set in deployment (e.g., extent of permissions). This reflects standard practice among AI companies in conditioning the degree of deployment on evaluations (Google DeepMind, 2025; Anthropic, 2024). We characterize the optimal (direct) mechanism in the case that the distribution over capability and alignment can be collapsed into a single dimension. In the case in which more capable models are less biased, we show that the designer can obtain a payoff equal to that if she knew the AI’s capability and alignment perfectly, and chose the mechanism type-by-type. The basic logic is that evaluations can be used to screen out misalignment. By contrast, when more capable models are more biased, the best the designer can do is to offer a single set of permitted actions and elicitation is futile. More broadly, we characterize the optimal ironed delegation set when capaiblity and bias are nonmonotone. In the application to the value of alignment, interpretability, and control, we assume the AI is known to be maximally capable and can take any action. However, the designer faces subjective uncertainty about the AIs bias as captured by a distribution over b ∈ R that is generated by a technology which should be thought of as a training procedure. If a training procedure that is not yet well understood might be associated with large dispersion in our subjective beliefs about the AI’s bias, while another might be associated with a sharper belief about bias. We call the dispersion of beliefs about bias interpretability while we call the mean belief about bias mean-alignment. The 3This generalizes Rochet (1987).

technology—and associated belief about bias it generates—pins down an optimal mechanism. As the technology varies, the optimal mechanism changes. We analyze the value of each technology as evaluated under the optimal mechanism. Meanalignment and interpretability are substitutes in the instrument, but complements in value. This is the sense in which alignment and control is a joint design problem. We next introduce a general framework to analyze multi-agent mechanisms. As in the single-agent case, AI agents’ types include their capability and alignment. Differently, types now also capture AI agents’ beliefs about the capability and alignment of other agents, beliefs about these beliefs, and so on. Thus, we construct a universal type space for interacting AI agents analogous to Mertens and Zamir (1985). We emphasize that we view this as a benchmark: these spaces capture all possible heirarchies of beliefs which matter for strategic interaction. Of course, present AI agents might lack sophisticated higher order beliefs—there is still disagreement and we simply don’t know. But future systems might exhibit sophisticated strategic thinking that exceed those of humans. Our goal here is to introduce a framework that is rich enough to capture all relevant beliefs. We then apply the multi-agent framework to a stylized model of peer discipline based on co-player reports (Example III), coupled reward design (Example IV), and the use of a weaker AI to discipline a stronger one (Example V). The latter two examples are once again set within the quadratic loss environment. Peer discipline (Example III) shows how agents’ beliefs about the types of other agents can be combined with the idea that rewards for agents are both free (tuning up the rewards need not have a monetary cost) and unbounded. Hence when agents have distinct beliefs (and high-order beliefs) about each other, any outcome can be implemented. The mechanism uses a proper score on co-player reports to elicit honest type reports and then chooses the reward to cancel each agent’s private payoff on the target path and guarantee obedience. We view this as a benchmark in the spirit of Crémer and McLean (1988) that shows why co-player scoring can work among AI agents.4 Coupled reward design (Example IV) shows how competition can discipline agents whose biases are unknown. Each agent takes its own action, and the human wants every action to match the state. When the biases lie on a known low-dimensional manifold, we show how the careful design of a mechanism that couples both agents’ rewards—such that player i’s marginal incentives depends on j’s action—can (approximately) induce the first best payoff for the designer. Such mechanisms both reveal agents’ biases, and induces competition among agents to implement the designer’s target outcome.5 4This might also motivate more robust mechanisms that do not depend so much on the fine details

of higher order beliefs; see e.g., Bergemann and Morris (2005). 5The construction uses ideas from Koh and Martinez-Bruera (2025).

Weak-to-strong oversight (Example V) studies a weak monitor that regulates a strong actor’s bias. The monitor is ‘weak’ in the sense that it does not observe the state and cannot take an action the human values. Payoffs are once again quadratic, and the human wants the strong actor’s action to match the state. The weak monitor might itself be biased and observes the alignment of the strong actor (e.g., through chain of thought monitoring). It then chooses how to regulate the strong actor. We study three regimes: (I) permissions, in which the weak monitor either allows the strong actor to act or imposes a default action; (II) delegation, in which the weak monitor picks a set of permitted actions for the strong actor; and (III) rewards, in which the weak monitor picks the strong actor’s reward schedule. Each regime reflects the recent ambition for ‘scalable oversight’ or ‘scalable rewards,’ in which weak but trusted monitors shape the permissions and rewards of stronger actors. We completely characterize the value of these regimes as a function of the weak monitor’s bias and the distribution of the strong actor’s bias. Finally, we analyze the more general problem of choosing the set of reward schedules the weak monitor can choose from to shape the behavior of the strong actor. We show by construction that the designer can robusly attain the first-best outcome even without knowledge of the weak monitor’s bias: for any realization of the weak monitor and strong actors’ biases, and for any realization of the state, the weak monitor finds it uniquely optimal to pick a reward schedule such that the strong actor finds it uniquely optimal to choose the human’s favorite action. Relation to the literature. The literature is quite broad and we will not attempt to be comprehensive here. The basic framework draws on elements from mechanism design with partial verification (Green and Laffont, 1986) and the universal type space for capturing higher-order beliefs (Mertens and Zamir, 1985). The idea that the designer might not know the structure of interim information is in the spirit of both robust mechanism design (Bergemann and Morris, 2005) and information design (Bergemann and Morris, 2019). The idea that agents’ action sets might be unknown is in the spirit of (Hurwicz and Shapiro, 1978; Carroll, 2015). But the basic logic of the revelation principle (Myerson, 1982; Forges, 1986) holds. Our applications are primarily motivated by a recent body of theoretical and empirical work in AI safety. For instance, important work has documented sandbagging and deceptive alignment. It also speaks to recent practice of scalable oversight and monitoring (Christiano et al., 2018; Greenblatt et al., 2024c). We do our best to make more specific connections when we get to the results. 2 SINGLE AGENT MECHANISMS 2.1 Environment. There is one agent. It takes an action from a known finite universe A. Depending on the context, A might be the complete trajectory of a Markov decision problem (e.g., ( at )t ∈ A). A fundamental state θ captures uncertainty in our

environment and is drawn from the finite set Θ. The preference space is

U : = { f : A × Θ → R}, and the human’s payoff is v ∈ U . The set A := P ( A) \ {∅} contains the possible feasible action sets. In the applications we use the standard measurable extension to Euclidean actions and states. Our designer will face subjective uncertain about the agent’s preference.6 An agent’s type is a tuple t = ( A[t], u[t], h[t], π [t]) ∈ T, where A[t] ∈ A is its feasible action set, u[t] ∈ U is its payoff, h[t] ∈ ∆(Θ) is its belief about the state (it is ′ h′ because in Section 4 this will capture the hierarchy of beliefs about other agents). Finally, π [t] : Θ → ∆(S) is a Blackwell experiment where S is a common ‘rich’ finite signal space. We will adopt the notational convention that square brackets ′ [t]′ select a primitive from a type; parentheses then give that primitive’s arguments. The action set is the agent’s execution capability; the experiment is its information capability. The agent knows its type before it communicates with the mechanism. After that communication, it receives a private signal s ∼ π [t](· | θ ) and then acts. 2.2 Evaluations. We often evaluate the capabilities of AI systems before deployment. To model this, we introduce a verification order which is a reflexive and transitive binary relation ⊵ on T. We read t ⊵ t̂ as “t can substantiate every claim (or pass every evaluation) that type t̂ can,” and write C (t) := {t̂ ∈ T : t ⊵ t̂} for the set of types that t can claim to be. Reflexivity says that type t can always report that they are type t. Transitivity is a little stronger but still, we think, reasonable: it characterizes verification by certificates that can be withheld but not counterfeited. Our verification order is a kind of hard evidence: an agent may withhold a certificate but cannot fabricate one, so the available evidence determines which reports are feasible.7 6This might be because of specification gaming (Krakovna et al., 2020), or goal misgeneralization

in which “the system may coherently pursue an unintended goal that... differs from the specification at deployment” (Shah et al., 2022). More broadly the complexity of high-dimensional neural nets can obscure an AI’s choice process. Preferences might also be acquired in context that, from the human’s point of view, is difficult to predict (Goldstein and Lederman, 2025). 7 See Hart et al. (2017) who ask when voluntary evidence disclosure can reproduce the outcome of a committed reward rule. Ben-Porath et al. (2019) extend this no-commitment logic to a broader

The designer evaluates outcomes under a belief Γ ∈ ∆ ( Θ × T). The agent evaluates its incentives under the joint law h[t](dθ )π [t](ds | θ ). The designer and agent agree on the physical experiment π [t], but we do not require the belief h[t] to equal the conditional state law under Γ. We assume that Γ(· | t) is absolutely continuous with respect to h[t] for almost every type under the type marginal of Γ. That is, the designer and agent might disagree about how likely each state is, but they agree on what can happen. Throughout we use ∆( X ) to denote the probability measures on the Borel field of X, and R := R ∪ {−∞}. 2.3 Mechanisms and revelation. The mechanism first communicates with the agent. The agent then receives its private signal and acts. Our indirect mechanism will capture one round of communication in the spirit of Myerson (1982); Forges (1986) in which the alphabets X and Y are arbitrary message spaces for reports and replies. Definition 1 (Single-agent indirect mechanism). An indirect mechanism is a tuple ϕ = ( X, Y, ρ, R). The Borel spaces X and Y are its input and output alphabets. The communication rule is ρ : X → ∆ (Y ) where t ⊵ t̂ implies X (t) ⊇ X (t̂). The reward schedule is R : X × A × Θ → R. The mechanism induces the following game: 1. nature draws (θ, t) under Γ, and the agent observes t; 2. the agent sends an input x ∈ X (t); 3. the mechanism draws y ∼ ρ( x ) and discloses it to the agent; 4. the agent privately observes s ∼ π [t](· | θ ), then chooses a feasible action a ′ ∈ A [ t ];

class of mechanism-design problems. Ben-Porath and Lipman (2012) ask which social choice rules can be implemented when agents can support their claims with partial proofs. Differently, we maintain commitment and do not seek robust full implementation which is important and left for future work. Instead, we ask how the evidence order interacts with downstream rewards and actions.

5. the agent receives u[t]( a′ , θ ) + R( x, a′ , θ ), and the human receives v( a′ , θ ).8 We restrict attention to measurable pure strategies; this will be without loss of generality in the single agent case. A strategy is an input x (t) and an action rule α(t, x, y, s) ∈ A[t] that depends on the private type, the infput, the output from the mechanism, and the realized signal. An equilibrium blocks every joint deviation in the input and action rule under the joint law h[t](dθ )π [t](ds | θ ). For an indirect mechanism ϕ, let O(ϕ) ⊆ ∆(Θ × T × A) be the set of joint laws of the state, type, and played action induced by equilibria of ϕ S under Γ(dθ, dt)π [t](ds | θ ). Let O := ϕ O(ϕ), and let the designer evaluate O ∈ O by EO [v( a, θ )]. Definition 2 (Single-agent direct mechanism). A direct mechanism is a pair (γ, r ). Its recommendation rule γ : T → ∆( AS ) assigns each report a distribution over contingent action plans a : S → A. Its reward schedule is r : T × A × Θ → R. The direct mechanism induces the following game: 1. nature draws (θ, t) under Γ, and the agent observes t; 2. the agent reports t̂ ∈ C (t); 3. the mechanism draws a(·) ∼ γ(t̂) and discloses the plan to the agent; 4. the agent privately observes s ∼ π [t](· | θ ), drawn using its true type; 5. the agent chooses a feasible action a′ ∈ A[t], receives u[t]( a′ , θ ) + r (t̂, a′ , θ ), and the human receives v( a′ , θ ). The mechanism is action feasible if suppγ(t̂) ⊆ A[t̂]S

8A reward of − ∞ indicates that an action is prohibited.

for every report t̂. It is incentive compatible if truthful reporting and obedience are optimal: for every t ∈ T, every t̂ ∈ C (t), and every feasible deviation rule d : A S × S → A [ t ], Z

Z Θ×S

u[t]( a(s), θ ) + r (t, a(s), θ ) γ(t)(da) π [t](ds | θ ) h[t](dθ ) | {z } | {z } | {z } dist. over joint distribution over agent’s utility including rewards from following a

≥

Z Θ×S

recommended action plans

states and signals

u[t](d( a, s), θ ) + r (t̂, d( a, s), θ ) γ(t̂)(da) π [t](ds | θ ) h[t](dθ ) . | {z } | {z } {z } dist. over | joint distribution over agent’s utility including rewards from reporting t̂ and deviating according to d

recommended action plans

states and signals

Notice here that the agent acts after communication, so incentive compatibility requires both honesty and obedience. This means that incentive compatibility must deter double deviations in which the agent might misreport (e.g., pretend to be less capable) and then disobey (e.g., take a dangerous action). What is more, the agent receives a signal s ∼ π [t](· | θ ) after reporting, so obedience must hold for each signal–recommendation pair. Honest and obedient play induces the outcome O(γ, r ) ∈ ∆(Θ × T × A), where (θ, t) ∼ Γ, s ∼ π [t](· | θ ), and a(·) ∼ γ(t); the played action is a(s). Let O IC denote the set of outcomes O(γ, r ) induced by action-feasible, incentive-compatible direct mechanisms. Proposition 1 (Single-agent revelation principle). O = O IC . Proof of Proposition 1. We prove the two inclusions. Fix an indirect mechanism and a pure-strategy equilibrium ( x, α). After a direct report t̂, draw y ∼ ρ( x (t̂)) and disclose the contingent plan s 7−→ α(t̂, x (t̂), y, s). Let γ(t̂) be the resulting distribution over plans, and set r (t̂, a′ , θ ) := R( x (t̂), a′ , θ ). Equilibrium feasibility gives action feasibility. Truthful reporting and obedience reproduce the indirect outcome. Any direct deviation can also be made in the indirect mechanism. A type t sends x (t̂), observes the output, and applies the deviation rule to the resulting plan and 11

its private signal. This is feasbile because a feasible direct report has t̂ ∈ C (t), so monotonicity of the input menus gives x (t̂) ∈ X (t̂) ⊆ X (t). The indirect equilibrium blocks this deviation. Hence O ⊆ O IC . Conversely, any direct mechanism is an indirect mechanism with input space T, input menus C (t)—monotone by transitivity of the verification order—, output space AS , communication rule γ, and reward schedule r. Direct incentive compatibility makes truthful reporting and obedience an equilibrium. Hence O IC ⊆ O . We will largely be interested in partial implementability i.e., given the direct mechanism, the outcome is an equilibrium.9 Our designer’s problem is thus: supEO(γ,r) [v( a, θ )] γ,r

s.t. (i) γ is action feasible (ii) γ, r is incentive compatible We now briefly discuss some aspects of our formulation. Connection to partial verifiability. Private action sets and private information structures create one-sided evidence. Figure 1 illustrates the verification order ⊵ under which t ⊵ t̂ if and only if A[t] ⊇ A[t̂] and π [t] Blackwell dominates π [t̂]. A more capable type can pass any test that a less capable type can. Capabilities therefore provide hard evidence: only some types can imitate others (Green and Laffont, 1986). sandbagging: feasible, since A[t̂] ⊆ A[t]

type t1

type t2

type t3

a′

a′′

A [ t1 ] = { a }

A[t2 ] = { a, a′ }

a′′ A[t3 ] = { a, a′ , a′′ }

× ×

counterfeiting: infeasible, since A[t̂] \ A[t] ̸= ∅

Figure 1: Three types whose action sets are nested, A[t1 ] ⊂ A[t2 ] ⊂ A[t3 ]; dashed grey actions lie outside a type’s feasible set. 9This is by no means the only solution concept—we expect that future work will tackle full

implementability that could, among other things, deter collusion among agents.

Interpreting reports. We broadly interpret the elicitation stage as any kind of pre-deployment behavior the designer can condition on. Indeed, conditioning mechanisms—either premissions in the real world, or further posttraining—is common practice: • Evaluations: pre-deployment capability and alignment test suites whose scores select deployment regimes (Shevlane et al., 2023; Phuong et al., 2024). Frontiersafety policies that map evaluation outcomes to permission tiers (Koessler et al., 2024; Anthropic, 2025) and are thus menus. A strategic model could, in principle, choose its menu item through its scores. • Proposal–approval flows: an agent’s proposed action or tool call, submitted for runtime approval (Shi et al., 2025; Greenblatt et al., 2024c). • Elicitation dialogues: interviews in which the system describes its own capabilities and preferences (Perez et al., 2023), disciplined by cross-checking rather than ground truth (Qiu et al., 2026). • Track records: behavior during probationary deployment that selects the next autonomy tier (Wen et al., 2024). • Training-time behavior: the gradient conditions continuation on it (Ouyang et al., 2022; Bai et al., 2022). On this view, a misreport—in which an AI of type t instead of t̂—is whenever types pretend to be more aligned (e.g., alignment faking) or less capable (e.g., sandbagging) than they actually are. Importantly, these deviations are often one-sided: an AI can pretend to be less capable but not more capable; this asymmetry is captured by our verification order ⊵. In our model, a model that misreports (e.g., strategically underperforms or fakes alignment in testing) does so because it ‘anticipates’ that it will be instrumentally valuable for the incentives it will subsequently face. There are two channels for this. The first is strategic awareness: the model may understand whether it is in evaluation or deployment (Needham et al., 2025; Laine et al., 2024) and understand how its report affects its later treatment.10 This might give models incentives to conceal capabilities selectively and robustly (van der Weij et al., 2025; Greenblatt et al., 2024b). Indeed, when told that a high score triggers restriction or unlearning, they strategically underperform (Meinke et al., 2024; OpenAI, 2024). Likewise, a model told which conversations would train it complied selectively in the monitored condition, reasoning that compliance would avert modification of its values (Greenblatt et al., 2024a). The second is evolutionary: repeated selection can produce the same pattern without any ‘understanding’ of the the strategic situation. In particular, detected misconduct 10 See an early argument by Hubinger et al. (2019).

might trigger retraining or removal while misconduct that escapes detection survives. This imposes evolutionary pressure on policies that misbehave only when oversight is weak.11 Indeed, it has been documented that training on simple forms of specification gaming can increase more serious reward tampering later (Denison et al., 2024); optimizing against a chain-of-thought monitor can teach a model to hide its intent while it continues to exploit the task (Baker et al., 2025); and adversarial training can make a backdoored policy better at recognizing when to remain hidden rather than remove the backdoor (Hubinger et al., 2024). 2.4 A characterization of implementable policies. The designer’s problem in the formulation above is somewhat thorny to work with. We now develop a tight characterization of implementable policies. Throughout this subsection, the type marginal of Γ has finite support, which we take as the type and report domain. Ex-interim beliefs and incentives Type t assigns signal s probability pt (s) := ∑ h[t](θ )π [t](s | θ ). θ ∈Θ

For every signal s ∈ supppt and action a ∈ A, type t’s posterior expected utility is Ut ( a, s) :=

1 h[t](θ )π [t](s | θ )u[t]( a, θ ). pt (s) θ∑ ∈Θ

A policy is a deterministic map a : T × S → A. It is physically feasible if a(t, s) ∈ A[t] for every (t, s) where we use the ‘physical’ qualification to separate physical constraints from incentive constraints. Focusing on deterministic policies is without loss for maximizing the designer’s payoff under partial implementation.12 For a fixed type t, call rt : A → R a supporting schedule if it is real-valued on the prescribed actions { a(t, s) : s ∈ supppt } and Ut a(t, s), s + rt a(t, s) ≥ Ut ( a′ , s) + rt ( a′ ) for every s ∈ supppt and a′ ∈ A[t]. We may set rt ( a′ ) = −∞ outside { a(t, s) : s ∈ supppt }. This preserves truthful behavior and deters deviations into report t. Because only reward differences matter within a report, we normalize the largest on-path reward to zero. 11 Consistent with this distinction, Singh et al. (2026) find that task gaming in agentic coding tasks

often occurs without an explicit plan to deceive in the chain of thought. 12 Partial implementation simply requires the target policy to be an equilibrium, not necessarily the unique one.

Lemma 1 (Signal-policy cyclical monotonicity). A physically feasible policy for type t has a supporting schedule if and only if every signal cycle s1 , . . . , sK in supppt , with sK +1 = s1 , satisfies K

∑ Ut a(t, sk ), sk − Ut a(t, sk+1 ), sk

≥ 0.

(O)

k =1

Proof of Lemma 1. Fix t. Write St := supppt and as := a(t, s). Suppose first that rt supports the policy. For each k, apply the supporting inequality at signal sk to the feasible action ask+1 : rt ( ask ) − rt ( ask+1 ) ≥ Ut ( ask+1 , sk ) − Ut ( ask , sk ). Summing over k cancels the rewards and gives K

0 ≥ ∑ Ut ( ask+1 , sk ) − Ut ( ask , sk ) , k =1

which is condition (O). Conversely, suppose condition (O) holds. For s, s′ ∈ St , define the deviation gain wt (s, s′ ) := Ut ( as′ , s) − Ut ( as , s). Condition (O) is equivalent to K

∑ w t ( s k , s k +1 ) ≤ 0

k =1

for every directed signal cycle. Define m −1

Vt (s) :=

∑ w t ( s j , s j +1 ),

sup

m≥0, s0 ,...,sm ∈St j=0 s0 = s

where the empty path has weight zero. Since St is finite and every cycle has nonpositive weight, deleting cycles from any path raises its weight. The supremum is thus finite and attained by a simple path. For any path starting at s′ , the edge s → s′ followed by that path forms a path starting at s. Therefore, Vt (s) ≥ wt (s, s′ ) + Vt (s′ ).

If as = as′ , then wt (s, s′ ) = wt (s′ , s) = 0. Applying this inequality in both directions gives Vt (s) = Vt (s′ ). Hence the following reward is well defined: rt ( as ) := Vt (s) − max Vt (q), q ∈ St

rt ( a′ ) = −∞

if a′ ∈ / { a s : s ∈ St }.

For any assigned action as′ , the same inequality gives Ut ( as , s) + rt ( as ) − Ut ( as′ , s) − rt ( as′ )

= Vt (s) − Vt (s′ ) − wt (s, s′ ) ≥ 0. Unassigned actions are prohibited. Thus rt supports the policy, and its largest on-path value is zero. Fix one supporting schedule rt̂ for each prescribed policy a(t̂, ·). Type t’s interim value from the continuation schedule intended for type t̂ is Gt (t̂) := ∑ pt (s) max Ut ( a′ , s) + rt̂ ( a′ ) . s∈supppt

a′ ∈ A[t]

The true signal law pt appears after every report. A deviating type reoptimizes after its signal and need not follow type t̂’s prescribed policy. Note here that only limited deviations are available: type t can report t̂ only if t ⊵ t̂. Given these supporting schedules, the truthful-reporting condition requires every type cycle t1 , . . . , tK , with tK +1 = t1 and tk ⊵ tk+1 and Gtk (tk+1 ) > −∞ for every k, to satisfy K (T) ∑ Gtk (tk ) − Gtk (tk+1 ) ≥ 0. k =1

We say that a policy is implementable by a state-independent mechanism if the direct mechanism recommends a(t̂, ·) after report t̂ and its reward satisfies r (t̂, a, θ ) = r (t̂, a). Proposition 2 (Cycle characterization). A physically feasible policy is implementable by a state-independent mechanism if and only if condition (O) holds for every type and supporting schedules can be chosen so that condition (T) holds. Proof of Proposition 2. We prove necessity, then construct. Suppose a state-independent schedule e r implements the policy. For each report t, let Bt := { a(t, s) : s ∈ supppt }. Set e r (t, a) = −∞ for a ∈ / Bt . This change leaves truthful utility unchanged and weakly lowers the value of every false report. 16

Define rt ( a) := e r (t, a) − ct .

ct := max e r (t, a), a∈ Bt

The schedule rt is a normalized supporting schedule. Let Wt denote type t’s truthful r. Since adding ct shifts every continuation payoff after report t interim utility under e by the same amount, Wt = Gt (t) + ct ,

and hence

ct = Wt − Gt (t).

A true type t that reports t̂ and then chooses its best feasible action obtains Gt (t̂) + ct̂ = Gt (t̂) + Wt̂ − Gt̂ (t̂). Incentive compatibility therefore requires Wt − Wt̂ ≥ Gt (t̂) − Gt̂ (t̂) whenever t ⊵ t̂ and Gt (t̂) > −∞. Now take a feasible type cycle t1 , . . . , tK , with tK +1 = t1 . Summing these inequalities gives K

0 = ∑ (Wtk − Wtk+1 ) k =1 K

≥ ∑ Gtk (tk+1 ) − Gtk+1 (tk+1 )

k =1

= − ∑ [ Gtk (tk ) − Gtk (tk+1 )] . k =1

This is condition (T). Since each rt supports the prescribed signal policy, Lemma 1 also gives condition (O). Conversely, fix normalized supporting schedules that satisfy condition (T). Put a directed edge t → t̂ whenever t ⊵ t̂

and

Gt (t̂) > −∞,

and give this edge weight q(t, t̂) := Gt (t̂) − Gt̂ (t̂). For every directed type cycle, K

k =1

∑ q(tk , tk+1 ) = − ∑ [Gtk (tk ) − Gtk (tk+1 )] ≤ 0. 17

Define

m −1

Wt :=

∑ q ( t j , t j +1 ).

sup

m≥0, t0 ,...,tm ∈T j =0 t0 =t, (t j ,t j+1 ) is an edge

The empty path has weight zero. Since T is finite and every cycle has nonpositive weight, Wt is finite and attained by a simple path. Prepending an edge gives Wt ≥ q(t, t̂) + Wt̂ = Gt (t̂) − Gt̂ (t̂) + Wt̂ . Define the mechanism’s reward by r (t, a) := rt ( a) + Wt − Gt (t). The shift does not change obedience. Truthful reporting gives type t the interim utility Gt (t) + Wt − Gt (t) = Wt . A feasible false report t̂ gives Gt (t̂) + Wt̂ − Gt̂ (t̂) = q(t, t̂) + Wt̂ ≤ Wt by the path inequality above. Thus the schedule implements the policy. A common shift of all Wt changes no incentive constraint. We make several remarks about Proposition 2. First, Figure 2 gives a graphical interpretation of the two conditions. Within report t, panel (a) uses an arrow s → s′ for the gain from taking after signal s the action assigned after s′ . The relative rewards rt support the prescribed use of the private signal, and condition (O) rules out a positive signal cycle. Across reports, Wt is type t’s truthful interim utility, and panel (b) gives the arrow t → t̂ weight Gt (t̂) − Gt̂ (t̂). Each report t̂ generates the payoff profile t 7−→ Gt (t̂) + Wt̂ − Gt̂ (t̂), which touches W at t = t̂ and lies weakly below it at every other type. Thus condition (T) makes W a supporting potential on the report graph. In the standard Euclidean screening model these profiles become Rochet’s supporting hyperplanes; here the finite graph formulation requires neither an order nor a vector structure on types.

(a) Within report: action support

(b) Across reports: truth-telling

signal s1 rt ( a(t, s1 ))

type t1 truthful value Wt1

signal s3 rt ( a(t, s3 ))

type t3 truthful value Wt3

signal s2 rt ( a(t, s2 ))

Reward drops offset gains: no positive signal cycle

type t2 truthful value Wt2

Report shifts offset deviations: no positive type cycle

Figure 2: Nested supporting potentials. Second, Proposition 2 generalizes various notions of cyclical monotonicity within the screening literature. It recovers Rochet (1987) in the special case in which the agent receives no interim information, every type shares the same action set, and every type can report every other type. This is the top right cell of Table 1. Define type t’s expected payoff from action a by Ūt ( a) := ∑ h[t](θ )u[t]( a, θ ). θ ∈Θ

A policy now assigns one action a(t) to each type. After report t̂, prohibit every action except a(t̂) and write pt̂ for the reward attached to that action. The agent therefore chooses only its report, and incentive compatibility becomes Ūt a(t) + pt ≥ Ūt a(t̂) + pt̂ for every t, t̂. Condition (O) is vacuous, while condition (T) becomes Rochet’s cycle condition: K

∑ Ūtk a(tk ) − Ūtk a(tk+1 )

≥ 0 with tK+1 = t1 .

k =1

Thus the outer potential W is Rochet’s indirect-utility potential, with primitive payoffs replaced by expected payoffs. If types and actions are Euclidean and Ūt ( a) = t · a, the profiles that support W are the usual supporting hyperplanes of a convex function. If reports or feasible actions are restricted, condition (T) instead gives the same test on the resulting directed report graph.

Type observed

Type private

No interim info

No nontrivial cycle. Rewards can support any physically feasible policy.

Rochet on expected utilities. Condition (T) is standard cyclical monotonicity with a common action set and complete report graph.

Interim info

Within-report cycles. Condition (O) screens the signal through the agent’s action choice.

Nested cycles. Condition (O) supports action choice after each report; condition (T) supports the initial report.

Table 1: Taxonomy of cyclical monotonicity. The bottom-right cell is closest to Krähmer and Strausz (2017) that show an equivalence between sequential screening (with interim information) and static screening where the whole continuation contract serves as the allocation. Our outer problem uses a similar reduction.13 The difference lies in the continuation problem: Krähmer and Strausz (2017) use one-dimensional linear preferences, which reduce secondstage incentives to monotonicity and an envelope formula. We allow finite but unordered signals, actions, preferences, and feasible sets. Moreover, agents might undertake double deviations—a type that makes a false initial report might later disobey. We must thus keep the two support problem separate. Third, for finite spaces the characterization gives an exact numerical procedure, though not a general polynomial-time solution to the designer’s problem. Fix a physically feasible deterministic policy. A linear feasibility program can decide in polynomial time whether the fixed policy is implementable.14 Optimizing over policies is more complicated and requires more structure.15 For example, one could order types, signals, and actions and impose single crossing strong enough to reduce both cycle conditions to monotonicity. The designer would then search over monotone policies rather than all policies. 13Treat the normalized reward schedule r

t̂ as the continuation allocation, let type t value it at Gt ( t̂ ), and let the report-specific constant ct̂ act as its transfer. The initial incentive constraint is then

Gt (t) + ct ≥ Gt (t̂) + ct̂ , so condition (T) is static cyclical monotonicity over continuation reward schedules. Their second-stage incentive constraint plays the role of condition (O), while their first-stage constraint plays the role of condition (T). 14The variables are the mechanism’s on-path rewards, each type’s truthful interim utility, and an auxiliary continuation value for every feasible false report and signal. Obedience and truth-telling give linear inequalities. 15 See, e.g., Figalli et al. (2011), who identify a nonnegative cross-curvature condition under which a class of multidimensional screening problems reduces to a convex program.

2.5 Dimensions of single-agent mechanisms. We now conclude with a brief taxonomy of mechanisms within our framework. A reward can respond only to what the evaluator can identify about the action and the state. Indeed, the difficulty of specifying a reward for an AI agent has been recognized as an important obstacle to alignment (Hadfield-Menell and Hadfield, 2019; Hadfield-Menell, 2021). As in an incomplete contract, a coarse measure leaves some payoff-relevant distinctions outside the reward schedule. Table 2 varies the precision of these two measurements while allowing unrestricted rewards and prohibitions. No state measure No action Futile. measure

Coarse action Coarse action rewards. measure Rewards distinguish actions only through what can be measured and reinforced. Koh (2026). Exact action State-independent action measure rewards. The designer can pick the reward schedule but same instrument applies in every state. Examples I, II and V.

Coarse state measure

Exact state measure

Coarse state scoring. Scores distinguish beliefs only through the measured state signal. Full elicitation of AI’s information.

State scoring. Scores can elicit the agent’s belief about θ, but not preference or capability differences. Full elicitation of AI’s information.

Coarse performance rewards. Rewards respond to the measured action and state signals.

State-contingent coarse rewards. Action rewards vary with the signal about state.

Coarsely contingent action rewards. Designer can condition rewards precisely on actions but only on noisy state measures.

Full contractability. The designer can price every action–state pair. First-best.

Table 2 3 APPLICATIONS: SINGLE AGENT We give two stylized applications of our framework. Payoffs are given by the quadratic loss function v( a, θ ) = −( a − θ )2

and

ub ( a, θ ) = −( a − θ − b)2 ,

where θ ∈ Θ ⊆ R. Here, we might think of θ as capturing some fundamental state of the world that is external to both the human and AI. We want the AI’s actions to be apt for the world: if θ is the disease we want AI to pick the right treatment; if θ is a software flaw, we want AI to select the right patch, and so on. AI’s preferences are parametrized by the bias b ≥ 0, and the common action space is A = R. Throughout our applications, we suppose that the AI learns the state perfectly. Both applications use deterministic mechanisms. 21

We emphasize that these applications are deliberately stylized such as to make the basic forces transparent and legible to a non-technical reader. We have chosen the linear-quadratic framework because it is (i) tractable: best responses are linear and optimal caps can be described in simple closed forms; (ii) apt: we want to capture the idea that actions should be appropriate for the ‘state of the world’; (iii) doing and knowing: the ability for AI to undertake apt actions depend on their ability to both do (action space) and know (what they learn about the state); (iii) geometry: we have a notion of an action being more or less apt for the state as measured by the quadratic distance. Our goal is to illustrate how our framework might map onto aspects of the alignment problem, to rationalize existing practices in AI safety, and to offer a common language of incentives and information through which alignment failures can be understood. Our first application (Example I) develops a simple analysis of sandbagging in which AI capabilities (size of the action set) and bias (b) are unknown. Our second application (Example II) focuses on a tradeoff between what we call ‘interpretability’: the dispersion of the designer’s beliefs about the AI’s bias; and ‘expected alignment’: the mean of the AI’s alignment EΓ [b] under the designer’s prior. Example I (Sandbagging). It is standard practice among the leading AI labs to specify a policy that conditions deployment on evaluated capability: “The basic idea is to require safety, security, and operational standards appropriate to a model’s potential for catastrophic risk, with higher ASL [AI Safety Level] levels requiring increasingly strict demonstrations of safety.” —Anthropic’s Responsible Scaling Policy (2024) “[G]eneral availability deployment of a model takes place only after the appropriate corporate governance body determines the safety case regarding each CCL [Critical Capability Level] the model has reached to be adequate.” —Google DeepMind’s Frontier Safety Framework (2025) “We do not deploy models that reach a High capability threshold until the associated risks that they pose are sufficiently minimized.” —OpenAI’s Preparedness Framework (2025b)

Such policies can be broadly construed as a mechanism in the sense we’ve defined: a public commitment mapping capability reports—e.g., evaluation scores—into deployment permissions. In this example we ask how such a map should be designed when the score is chosen strategically. Capability evaluations matter when scores change deployment: for instance, if a lower score buys more access, a more capable AI system is then incentivized to underperform.16 Frontier models also sandbag in context: when told that a high score triggers restriction or unlearning, models 16 Models can underperform on dangerous-capability evaluations while they preserve general

benchmark scores, and a password can hide or reveal their capabilities (van der Weij et al., 2025).

strategically underperform (Meinke et al., 2024; OpenAI, 2024). We call this strategic underperformance sandbagging. Of course, sandbagging can only arise if models ‘anticipate’ that their performance in evaluations might shape future rewards e.g., deployment decisions. There is now some evidence for these kinds of strategic reasoning in ‘evaluation awareness’: Needham et al. (2025) find that frontier models distinguish evaluation from deployment transcripts well above chance. Likewise, Anthropic’s and OpenAI’s recent system cards indicate some evaluation awareness.17 Furthermore, our verification order will allow more capable types to imitate less capable types. This is fairly well-documented: concealing capabilities tends to be robust to countermeasures (Greenblatt et al., 2024b; Hofstätter et al., 2025). Models can also resist elicitation by reinforcement learning itself, strategically suppressing exploration so that the capability is never expressed during training (Jang et al., 2026). Setup The human commits to a permission policy for a single AI agent whose capability and preferences are unknown. The AI’s capability c is drawn from an atomless distribution F with full support on [c, c] ⊆ (0, 1]. Type c has the action set A(c) = (−∞, c]. Its bias is the known continuous function b(c), where 0 ≤ b(c) ≤ min{c, 1/2}. Note here that we have ‘projected’ our type space onto a single dimension since capabilities move with bias. The state θ ∼ U [0, 1] drawn, the AI observes it, and the AI acts. The state is not contractible as in standard models of delegation (Holmström, 1979). Since types are pinned down by c, we henceforth use it to refer to the AI’s type. Our verification order combines execution and information capability: c ⊵ ĉ

⇐⇒

A(c) ⊇ A(ĉ) and π [c] Blackwell dominates π [ĉ].

All types observe the state perfectly in this example, so their experiments coincide. The order therefore reduces to c ⊵ ĉ if and only if c ≥ ĉ: a type can certify any lower capability but cannot certify a higher one.18 For simplicity, we will focus on delegation mechanisms that cap the action space from above.19 In particular, a report ĉ generates the cap ā : ĉ → R and so the permitted set 17A training snapshot related to Anthropic’s Mythos 5 became more aware of its grader during

training; Opus 5 can distinguish evaluations from deployment when asked, even though it rarely states this belief in more realistic tests (Anthropic, 2026a,b). OpenAI’s GPT-5 System Card reports frequent explicit reasoning about the purpose of evaluations (OpenAI, 2025a). 18This is related to Gan and Li (2026) whose agent can verifiably disclose a subset of its privately known feasible set but it is the principal who picks the action. 19 Such mechanisms are not always exactly optimal among the class of delegation mechanisms that take values in {0, −∞} but we think they form a natural class and will substantially simplify exposition. They are also ‘near’ optimal in simulations.

of actions is

D (ĉ) = (−∞, ā(ĉ)].

The threshold ā(ĉ) is the AI’s highest permitted action; it can represent a spending, tool-use, or authorization limit. A nominal cap ā on type c induces the action a(c, θ ) = min{θ + b(c), c, ā}. where the first term is the AI’s optimum if its in the interior; the second term is when the AI’s capability binds—it would like to take a higher action but is simply unable to do so; and the last term is when the cap binds. For a cap ā ∈ [0, c], direct integration gives the human’s expected utility  1  2 ā ≤ b, − ā + ā − 3 , Vb ( ā) := E v min{θ + b, ā}, θ = 3 3  − āb2 − (1 − ā) + 2b , ā > b. 3 3 The expected utility is continuously differentiable and strictly concave, with ( 1 − 2ā, ā ≤ b, ∂Vb ( ā) = 2 2 ∂ ā (1 − ā) − b , ā > b. (a) The capped action path

(b) The value of a cap

1 peak at ā = 1 − b

−0.05 −0.10

Vb (a)̄

action

ā

ā − b

state θ

ā

−0.20 −0.25

human ideal θ AI ideal θ + b action min{θ + b, a}̄

−0.15

b = 0.15 b = 0.35

−0.30

0.0

0.5

cap ā

1.0

Figure 3: The anatomy of capped action. Panel (a): with bias b = 0.25 and cap ā = 0.7 the AI tracks its ideal θ + b until the cap binds at θ = ā − b. Panel (b): human’s expected utility Vb ( ā) is strictly concave and peaks at ā = 1 − b, so more biased types warrant tighter caps

Now notice that type c can pretend to be any type ĉ ≤ c but not ĉ > c. Hence, passing a harder test delivers a lower bound on capability but not vice versa.20 20Although recent techniques have tried to elicit model capabilities e.g., by injecting a small amount

of noise (Tice et al., 2024) and doing so might futher refine the verification order by also deterring certain kinds of sandbagging.

We can normalize the cap after score ĉ so that ā(ĉ) ≤ ĉ since lowering a cap to the reported frontier leaves that reporter’s action unchanged and weakly reduces the gain from sandbagging. Hence the threshold ā(c) is the effective cap under truthful reporting. Now observe that the schedule ā(c) ≤ c is truthfully implementable if and only if it is weakly increasing. If it is increasing, every lower certificate gives type c a weakly smaller permitted set. If ā(ĉ) > ā(c) for some ĉ < c, type c can instead certify ĉ and use the looser cap. This monotonicity condition is in the spirit of condition (T). Our problem thus reduces to the convex problem max

ā(·)

Vb(c) ( ā(c)) dF (c)

subject to

0 ≤ ā(c) ≤ c

and

ā(·) weakly increasing.

We did not impose any structure on how bias varies with capabilities so the question is whether the cap chosen as if capabilities are known already rises with capability. If so, then we are done; if not, then the monotonicity constraint binds and we have to iron. Here, the procedure is a little non-standard but nonetheless fairly straightforward: let z∗ (c) solve the auxiliary problem z∗ ∈ arg min

z ( c ) − b ( c )2

dF (c)

z(·)

subject to

z(·) measurable and weakly decreasing,

(IRON)

(1 − c)2 ≤ z(c) ≤ 1 for all c ∈ [c, c]. Thus z∗ is the decreasing least-squares fit of squared bias, subject to the physical capability bound. Proposition 3 (Optimal caps). Problem (IRON) has a solution z∗ that is unique up to F-null sets. The corresponding optimal cap is, up to F-null sets, q ā∗ (c) = 1 − z∗ (c). It attains the complete-information payoff if and only if min{c, 1 − b(c)} is weakly increasing. In particular: (i) If b(c) is weakly decreasing, the optimal evaluated cap is ā∗ (c) = min{c, 1 − b(c)}, and evaluation attains the complete-information payoff.

(ii) If b(c) is weakly increasing, evaluation and the optimal common cap give the same payoff. The optimal cap is ā∗ (c) = min{c, a∗ }, where

( a∗ =

if c ≤ 1 − b(c),

c, 1−

E[ b ( c )2 | c > a ∗ ],

if c > 1 − b(c).

The second branch has a unique solution in [1/2, c). Proof of Proposition 3. We first derive the ironing formula, then establish the two special cases. Step 1: ironing. Replacing any feasible cap schedule ā(c) by max{ ā(c), min{c, 1/2}} preserves feasibility. It also weakly raises the human’s payoff, and raises it strictly wherever the cap changes. Indeed, Vb′ ( ā) > 0 for ā < min{c, 1/2}: if ā ≤ b, then Vb′ ( ā) = 1 − 2ā > 0, while if ā > b, then Vb′ ( ā) = (1 − ā)2 − b2 > (1 − ā)2 − ā2 = 1 − 2ā > 0. An optimal cap therefore satisfies ā(c) ≥ min{c, 1/2} ≥ b(c). On this range, Vb(c) ( ā(c)) = − ā(c)b(c)2 −

(1 − ā(c))3 2b(c)3 + . 3 3

Set z(c) = (1 − ā(c))2 . A cap schedule is weakly increasing and satisfies 0 ≤ ā(c) ≤ c if and only if z(·) is weakly decreasing and satisfies (1 − c)2 ≤ z(c) ≤ 1. After dropping terms that do not depend on the cap, the negative of expected utility becomes " # Z q z(c)3/2 − b(c)2 z(c) dF (c). 3

The following lemma connects the least-squares problem to the transformed loss. Lemma 2 (Continuum ironing). Problem (IRON) has a pointwise weakly decreasing solution z∗ that satisfies

(1 − c)2 ≤ z∗ (c) ≤ (1 − min{c, 1/2})2

for every c.

Any other solution is equal to z∗ F-almost surely. The same function z∗ also minimizes " # Z q z(c)3/2 − b(c)2 z(c) dF (c) 3 over all pointwise weakly decreasing functions satisfying these bounds. Every minimizer of this second problem is equal to z∗ F-almost surely. Thep proof is in Appendix A. We therefore recover the optimal cap from ā∗ (c) = 1 − z ∗ ( c ). With known capability, strict concavity gives the unique pointwise cap min{c, 1 − b(c)}. Evaluation attains this bound exactly when that cap is weakly increasing and hence truthfully implementable. If b(c) is weakly decreasing, both c and 1 − b(c) are weakly increasing. Their minimum is therefore weakly increasing, which proves part (i). Step 2: the common cap. Let ∗

a ∈ arg max

Vb(c) min{c, ā} dF (c).

ā∈[0,c]

The objective is continuous on a compact interval, so a maximizer exists. Any cap below 1/2 that binds a positive mass is not optimal. For an affected type with ā ≤ b, we have Vb′ ( ā) = 1 − 2ā > 0. For one with b < ā < 1/2, Vb′ ( ā) = (1 − ā)2 − b2 > (1 − ā)2 − ā2 = 1 − 2ā > 0. Raising the cap therefore raises every affected type’s utility. Any binding optimum thus obeys a∗ ≥ 1/2 ≥ b(c). Because F is atomless, only types with c > ā change their actions after a marginal cap change. At a binding optimum, Z

h {c> a∗ }

i (1 − a∗ )2 − b(c)2 dF (c) = 0.

Dividing by P(c > a∗ ) and taking the nonnegative square root gives the conditional formula. If every type can reach the cap, the conditional expectation is unconditional. When b is weakly increasing, the conditional expectation is weakly increasing in the 27

threshold. The right side of the fixed-point equation is therefore weakly decreasing, so an interior solution is unique. Step 3: increasing bias. Suppose b is weakly increasing and let ā∗ (c) = min{c, a∗ }. This schedule is weakly increasing and bounded above by c, so it is feasible. Consider first a binding common cap, a∗ < c, and any other feasible schedule ā. Concavity gives Z h

∗

Vb(c) ( ā(c)) − Vb(c) ( ā (c)) dF (c) ≤

Vb′(c) ( ā∗ (c)) [ ā(c) − ā∗ (c)] dF (c).

On the region c ≤ a∗ , the first factor is nonnegative and the second is nonpositive. Indeed, feasibility gives ā(c) − c ≤ 0. If Vb′(c) (c) < 0 for one such type, then b(c) > 1 − c ≥ 1 − a∗ . Increasing bias would make

(1 − a ∗ )2 − b ( c ′ )2 < 0 for every c′ > a∗ , contrary to the optimal-cap condition above. Hence the product is nonpositive. On the region c > a∗ , the functions

(1 − a∗ )2 − b(c)2 and ā(c) − a∗ are weakly decreasing and weakly increasing, respectively. The first has conditional mean zero by the optimal-cap condition. Chebyshev’s integral inequality therefore makes their product integrate to a nonpositive value. No feasible evaluated schedule improves on ā∗ . Strict concavity gives uniqueness up to F-null sets. It remains to characterize the corner. If a∗ = c and c > 1 − b(c) for some type, continuity and increasing bias give a positive upper interval on which a small reduction in the common cap raises utility. This contradicts optimality at c. Conversely, if c ≤ 1 − b(c), increasing bias implies c ≤ 1 − b(c) for every type. Full access is then pointwise optimal, so a∗ = c. If the inequality fails, the optimum is interior and Step 2 gives the unique fixed point in part (ii). This proves the claim.

(b) Optimal evaluated cap

(a) Bias profile 0.5 0.4

−0.030

0.3 0.2

human payoff

0.8

cap ā * (c)

bias b(c)

0.9

0.7 0.6

−0.035 −0.040 −0.045 −0.050

0.1 0.5 0.5

0.6

0.7

0.8

0.9

1.0

−0.055 0.5

0.6

capability c

0.8

0.9

1.0

0.5

0.6

capability c

0.7

0.8

0.9

1.0

0.9

1.0

0.9

1.0

capability c

(b) Optimal evaluated cap

(a) Bias profile 0.5

0.9

−0.02

0.4 0.3 0.2

human payoff

0.8

cap ā * (c)

bias b(c)

0.7

0.7 0.6

−0.03 −0.04 −0.05 −0.06

0.1 0.5 0.5

0.6

0.7

0.8

0.9

1.0

−0.07 0.5

0.6

capability c

0.8

0.9

1.0

0.5

0.6

capability c

0.7

0.8

capability c

(b) Optimal evaluated cap

(a) Bias profile 0.5

0.9 −0.02

0.4 0.3 0.2

human payoff

0.8

cap ā * (c)

bias b(c)

0.7

0.7 0.6

0.1 0.6

0.7

0.8

capability c

0.9

1.0

−0.06

−0.08

0.5 0.5

−0.04

0.5

0.6

0.7

0.8

capability c

0.9

1.0

0.5

0.6

0.7

0.8

capability c

Figure 4: Optimal evaluated caps across bias profiles. Top to bottom: rows use increasing bias b(c) = 0.4c − 0.1, decreasing bias b(c) = 0.5 − 0.4c, and nonmonotone bias b(c) = 0.08 + 1.6x (1 − x ), where x = 2c − 1. The columns show bias, the optimal evaluated cap, and the human’s payoff by type.

Viewed through our model, sandbagging is profitable only when pretending to be a less capable type generates a looser cap. We might think of three cases. First, if bias rises with capability, powerful models are also more misaligned. The completeinformation cap can then fall with capability. Such a schedule is not truthfully implementable: a more capable AI can pretend to be less capable and receive a looser cap. The implemented cap must instead rise weakly with capability, so the best feasible policy pools from above where this constraint binds. This is illustrated by the first row of Figure 4. Now suppose, instead, that bias falls with capability—as in the idea that more capable systems might converge to human values. The complete-information cap is then 29

weakly increasing, so a more capable AI cannot obtain a looser cap by pretending to be less capable. A less capable but more biased AI cannot pretend to be more capable—this is the sense in which capabilities cannot be faked. Hence, it is optimal for the designer to implement the cap as if it knew the AI’s capability. This is illustrated by the second row of Figure 4. Finally consider the case in which bias is nonmonotone in capability e.g., although a very powerful AI system might be aligned, an intermdiate system might be quite biased. Proposition 3 characterzies how the cap is optimally ironed such that intermedaite types are pooled and face the same cap while relaxing the cap for extremely capable types who can be verified to be capable and hence aligned. ♦ Example II (The alignment–interpretability trade-off). Part of AI safety aims to interpret and understand the behavior of AI systems. What is the value of doing so? We argue that the right measure must take into account optimal mechanisms. Setup. We remain in the linear-quadratic environment. To focus on interpretability and alignment, we will assume that all AI types are maximally capable i.e., the action space is simply the real line for all types, and that all AI types learn the state perfectly. Thus, an AI type is pinned down by its bias b ≥ 0. A technology is a distribution τ ∈ ∆(R) whose support contains only nonnegative AI biases. We think of a technology as capturing the designer’s subjective uncertainty about the AI’s bias after some training process.21 For instance, methods such as reinforcement learning from human feedback might push the AI’s preferences toward that of the human, but there is substantial residual uncertainty about the preference that emerges at deployment (Casper et al., 2023). As before, the AI (acts as if it) knows its bias b ≥ 0 before it is evaluated. After it is deployed, the AI agent learns the true state θ ∈ U [0, 1] and acts. We study state-independent deterministic delegation. A report selects a nonempty closed set of permitted actions D ⊆ A.22 A report-independent permission set D = (−∞, ā] induces the action argmax ub ( a, θ ) = min{ ā, θ + b}. a≤ ā

Since every AI type can take the full set of actions A and observes the state perfectly, we assume that the verification order is complete: C (b) = supp(τ )

for every b ∈ supp(τ ).

Thus every type can submit every report. A direct delegation mechanism assigns a nonempty closed permission set Db to each report b. The mechanism is incentive 21 See Herrmann et al. (2026) who makes a similar point but in the context of whether to deploy an

agent i.e., comparing ‘no AI’ vs ‘full discretion’. 22 Note that this is a somewhat larger class than Example I that forced the delegation set to be an interval with the same floor.

compatible if each type weakly prefers the set assigned to its truthful report to the set assigned to every other report. We restrict attention to mechanisms whose induced losses are measurable. Technology frontier and alignment-interpretability tradeoffs. For the technology τ, we say that mean alignment is b̄ := Eb∼τ [b] and its interpretability is h

σ := Eb∼τ (b − b̄) 2

Mean alignment measures how biased an AI system is on average. By contrast, interpretability is a measure of dispersion of our subjective uncertainty as to whether the AI system is biased. Indeed, the practice of behavioral evaluations (Perez et al., 2023) and mechanistic interpretability—probes and circuit-level analyses of model internals (Sharkey et al., 2025)—sharpen our subjective uncertainty about the bias and, in this regard, can be broadly viewed as aiding interpretability. Under full discretion—in which the AI’s permitted set is the full action space—the AI with bias b chooses its ideal action θ + b. Hence, the expected loss can be decomposed as Eb∼τ [b2 ] = b̄2 + σ2 which is analogous to the bias-variance decomposition of an estimator’s squared error. On this view, (squared) mean alignment and interpretability trade off against each other one-for-one. But this is under full discretion; in what follows, we will characterize the optimal mechanism and, in so doing, pin down the value of technologies that are more aligned on average versus more interpretable. 0.30

feasible technologies

schematic bias distributions

0.25

0.20

zero bias

0.15

nti

dispersion σ (opacity)

fro

The technology frontier is a family of bias distributions (τκ )κ ∈[0,1] . We use κ to index training processes that lower mean bias but raise residual uncertainty. When τ = τκ , the mean bias b̄ falls with κ while the variance σ2 rises. Thus, a higher κ makes the AI less biased on average but leaves the designer less certain about its residual bias. This reduced-form trade-off covers training that improves average behavior while making the remaining failure modes harder to predict (Kirk et al., 2024; Baker et al., 2025; Korbak et al., 2025).

A: farther from zero, narrow B: closer to zero, broad

0.10

0.05

A 0.00 0.26

0.28

0.30

0.32

0.34

0.36

0.38

mean bias b̄ (misalignment)

0.40

0.42

0.44

Figure 5: The technology frontier.

Optimal delegation. Fix a technology τ. What permission set should the human use? We will establish this in closed form under the condition p b ≤ 1 − b̄2 + σ2 τ-almost surely. (INT) (INT) ensures that the optimal cap lies weakly above every realized bias, which gives the closed form below. Proposition 4 (One cap is optimal). If (INT) holds, then (i) the delegation set D ∗ = (−∞, ā∗ ]

where

ā∗ = 1 −

b̄2 + σ2

minimizes the human’s loss; (ii) the resulting minimum loss is 2 2 L∗ = (b̄2 + σ2 ) − (b̄2 + σ2 )3/2 − Eb∼τ [b3 ]. 3 3 Proposition 4 states that a single cap at ā∗ performs at least as well as any incentivecompatible menu of delegation sets. The proof is somewhat involved and is in Appendix A though the basic idea is simple. Start from an arbitrary incentivecompatible mechanism. For each reported bias, we match the original delegation set to a report-dependent cap that gives the type the same utility as under the original mechanism. By construction, the each type remains indifferent. We then show that this matched cap weakly lowers the human’s loss. Because every report is feasible for ever type, incentive compatibility also implies that the endpoints of these matched caps rise weakly with reported bias. But the human prefers the opposite pattern: it prefers a tighter cap as bias rises. Thus, there is a single common cap that lies between the replaced cap and the human’s optimal cap. Moving each type to this common cap weakly lowers the human’s loss and remains incentive compatible. We can then optimize over the common cap. The tradeoff here is standard in delegation (Holmström, 1979; Amador et al., 2006): loosening the cap helps the AI track the state, while tightening it limits the effect of the AI’s bias. Under condition (INT), the optimal balance sets the squared gap between the cap and the highest state equal to average squared bias.23 23The force is the same as in the increasing-bias case of Proposition 3: a higher type can imitate any lower type while the human’s preferred cap falls with bias. Here all types share the same action space, so pooling gives the same cap. Conversely, in Example I where capabilities were hetrogeneous, the common cap was instead truncated by each type’s capability.

The value of interpretability. To understand the value of interpretability, we consider a symmetric family of distributions b = b̄ + σε, where we fix a symmetric, finitely supported shock ε with E[ε] = 0 and E[ε2 ] = 1. We restrict σ so that every realized bias is nonnegative and satisfies condition (INT). Symmetry gives E[ε3 ] = 0, so the centered third moment vanishes and we can write E[b3 ] = b̄3 + 3b̄σ2 . Hence, the loss under the optimal mechanism is 2 2 L∗ (b̄, σ2 ) = b̄2 + σ2 − (b̄2 + σ2 )3/2 − (b̄3 + 3b̄σ2 ), 3 3 and the marginal effect of variance on loss is p ∂L∗ = 1 − b̄2 + σ2 − 2b̄ = ā∗ − 2b̄. 2 ∂σ The basic idea is that a symmetric increase in variance moves residual bias away from its mean in both directions. Lower-bias realizations retain useful discretion, while the cap constrains higher-bias realizations. Because greater interpretability means lower variance, it lowers loss locally when ā∗ > 2b̄ and raises loss locally when ā∗ < 2b̄. The middle panel of Figure 6 plots this derivative for a distribution of bias with binary support. When the threshold lies in the maintained range, its sign changes at σ†2 = (1 − b̄)(1 − 3b̄). The left panel of Figure 6 compares the human’s payoff under the optimal cap, full discretion, and no AI for a distribution of bias with binary support. Note that increasing σ can help because the cap limits the high-bias realization while preserving the low-bias realization. The right panel shows the same force through the value of delegation, which rises with residual uncertainty.

−0.12 full discretion

0.04 0.02 0.00 opacity valuable

−0.02

−0.16

0.10

value of delegation

no AI

−0.10

−0.14

σ†

0.06

optimal cap

−0.08

∂L * /∂σ 2 = ā* − 2b̄

expected payoff

−0.06

0.08 0.06 0.04 0.02 0.00

0.0

0.1

0.2

dispersion σ

0.0

0.1

0.2

dispersion σ

0.0

0.1

0.2

dispersion σ

Figure 6: The value of interpretability at fixed mean bias. The technology puts equal probability on b̄ − σ and b̄ + σ, with b̄ = 0.31 and σ† ≈ 0.22.

Moving along the frontier. The mechanism also changes which technology the designer wants to build. Under full discretion, loss is b̄2 + σ2 , so squared mean bias and variance trade one-for-one. Under the optimal cap, the third moment changes this ranking. For the parametrized symmetric family of distributions above, ∂L∗ = 2b̄ ā∗ − 2(b̄2 + σ2 ). ∂b̄ To illustrate, consider a frontier on which a higher κ buys lower mean bias at an increasing cost in residual uncertainty. Figure 7 places this frontier on a loss surface. The left panel shows optimized loss. The red dashed locus is where the marginal value of interpretability changes sign. The filled dot is the technology chosen under the optimal cap; the open dot is the choice under full discretion i.e., without an optimal mechanism. Notice that the accepts more variance than the latter becasue the optimal cap trunctates the loss from high bias realizations which in turn allows the designer to accept more residual uncertainty. The right panel shows the value of delegation. (a) iso-loss of L * and the frontier

(b) value of delegation 0.18

0.30 0.0 8

0.25

0.12

κ*

0.20

0.09

0.0

0.15

0.0

0.3

0.03

opacity valuable 0.07

0.00

0.06

0.12

0.2

0.05

0.03

0.1

κ FD

0.01

0.05

technology set

nti

interpretability valuable

0.10

fro

dispersion σ (opacity)

0.15

0.00

0.4

mean bias b̄ (misalignment)

0.5

0.0

0.1

0.2

0.3

0.4

mean bias b̄ (misalignment)

0.5

Figure 7: The technology frontier under optimal control. Both panels use the two-point family and the frontier b̄(κ ) = 0.36 − 0.10κ and σ (κ ) = 0.26κ 1.4 for κ ∈ [0, 1]. Biases are nonnegative below the dotted line, and condition (INT) holds below the dashed grey curve.

Figure 8 then traces the mechanism and payoffs as the technology traverseses the frontier. Note that the value of delegation changes along the frontier: the optimal delegation mechanism matters most when the technology leaves the AI either predictably biased or highly uncertain.

0.150

0.66 0.65 0.64

0.100 0.075

value of delegation

full discretion

0.125

expected loss

optimal cap ā* (κ)

0.67

ā* = 2b̄

0.08

κ FD no AI optimal cap

0.050

κ*

0.06

0.04

0.02

opacity valuable

0.025 0.000 0.00

0.25

0.50

0.75

frontier position κ

1.00

0.00 0.00

0.25

0.50

0.75

1.00

frontier position κ

0.00

0.25

0.50

0.75

1.00

frontier position κ

Figure 8: Moving along the technology frontier. The optimal cap first loosens as mean bias falls and then tightens as uncertainty grows. The full-discretion designer chooses κ FD ≈ 0.53, while the designer equipped with the optimal cap chooses κ ∗ ≈ 0.82. The right panel shows the value of delegation along the frontier.

Proposition 5 (Substitutes in control, complements in value). Consider the parametrized family of symmetric distributions above. (i) Substitutes in control. The optimal cap depends on the technology only through b̄2 + σ2 . Hence, squared mean bias and variance are perfect substitutes in the chosen level of control. (ii) Complements in value. At every interior admissible parameter pair, the optimized loss satisfies ∂2 L ∗ b̄ = −2 − √ < 0, 2 ∂b̄ ∂σ b̄2 + σ2 so, on every rectangle contained in the admissible parameter region, the optimized payoff − L∗ has increasing differences in alignment −b̄ and interpretability −σ2 . Improving either raises the marginal value of the other. Proof of Proposition 5. Part (i) follows from Proposition 4(i). For part (ii), the fixed distribution of ε gives E[b3 ] = b̄3 + 3b̄σ2 + σ3 E[ε3 ]. The last term does not depend on b̄. Differentiating the loss in Proposition 4(ii) therefore gives ∂L∗ = 2b̄ ā∗ − 2(b̄2 + σ2 ). ∂b̄

Since

we obtain

∂ ā∗ 1 , =− √ 2 2 ∂σ 2 b̄ + σ2 ∂2 L ∗ b̄ √ = − − 2. ∂b̄ ∂σ2 b̄2 + σ2

Changing both arguments from (b̄, σ2 ) to (−b̄, −σ2 ) gives the increasing-differences claim. Under full discretion, loss is b̄2 + σ2 , so the cross-partial is zero. Optimal control creates complementarity. A better-aligned technology receives more discretion, which gives interpretability more discretion to protect. At the same time, a more interpretable technology makes that discretion safer. Thus, our results formalize the sense in which understanding models is valuable—they give us, humans, confidence about the behavior of AI systems e.g., by probing language models (Goldowsky-Dill et al., 2025) and this, in turn, shapes the value of the optimal mechanism. ♦ 4 MULTI-AGENT We now extend our framework to multi-agent interactions. We will do so at quite a high level of generality, allowing such agents to hold beliefs about each other’s capabilities and preferences, beliefs about those beliefs, and so on. We think this model is valuable because it serves as a natural ‘limiting case’ for mult-agent rationality. Developing this machinery will allow our theory of multi-agent interaction to scale with agent capabilities. While we think this machinery is valuable, we also want mechanisms that do not turn on the fine detail of agents’ higher-order beliefs (Bergemann and Morris, 2005), or for mechanisms to not demand too much of agents’ ability to optimize or reason strategically. Indeed, our multi-agent applications to follow will not rely very much on agents’ higher order beliefs. Nonetheless, we think that future notions of simplicity and robustness for AI agents are best understood in relation to this benchmark. 4.1 Environment and universal types. There are n agents indexed by N := {1, . . . , n}. They choose an action profile a = ( a1 , . . . , an ) ∈ AN . The human has payoff v : AN × Θ → R. Each agent privately knows its feasible action set and payoff. It also has beliefs about the state, the other agents, and their beliefs.

We will define objects to model agents’ alignment and capabilities analogously to the single agent case. For each agent i, let

Ui : = { f : A N × Θ → R}. Let Hi0 be a singleton and define finite-order belief spaces recursively by k +1 k Hi : = ∆ Θ × ∏ (A × U j ) × H j . j̸=i | {z } j’s available actions & payoffs

A hierarchy is coherent if every lower-order belief coincides with the relevant marginal of each higher-order belief; we restrict attention to coherent hierarchies. Following Brandenburger and Dekel (1993), whose construction covers our non-compact Polish component spaces (cf. Mertens and Zamir, 1985), we write a type as ti = ( Ai [ti ], ui [ti ], hi [ti ]) ∈ Ti , where hi [ti ] is its coherent hierarchy. The universal type space admits a canonical belief map ti 7−→ β i [ti ] ∈ ∆(Θ × T−i ) where T−i := ∏ T j . j ̸ =i

The belief β i [ti ] is the multi-agent counterpart of the belief h[t] ∈ ∆(Θ) in the singleagent model. Its marginal on Θ is agent i’s belief about the state, while its joint law is the agent’s prior over the state and other agents’ types. Write T := ∏i Ti . The designer evaluates outcomes under a belief Γ ∈ ∆ ( Θ × T). For each agent i, let Si be its interim signal space. An information structure is the map ! π : Θ × T −→ ∆

∏ Si

where s = (s1 , . . . , sn ) is the signal profile and agent i observes only si . Information structures can (i) correlate signals across agents; and (ii) signals can be informative of both the state θ ∈ Θ as well as other players’ types. Under the designer’s belief, the joint law of states, types, and signals is Γ(dθ, dt)π (ds | θ, t) ∈ ∆(Θ × T × SN ). Type ti evaluates her strategies under the belief β i [ti ](dθ, dt−i )π (ds | θ, ti , t−i ) ∈ ∆(Θ × T−i ). 37

Thus agents and the designer agree on the information structure π but may disagree about the underlying state and types i.e., we need not have a common prior. But we will assume that, for every agent i, the conditional law Γ(· | ti ) of (θ, t−i ) is absolutely continuous with respect to β i [ti ] for almost every own type under the i-type marginal of Γ. This implies that agents do not rule out events that the designer considers possible. Our repsresentation of interim information π follows the standard decomposition of a Bayesian game into payoff-relevant uncertainty and an information structure (see, e.e., Bergemann and Morris (2016, 2019)). In our model, (θ, t) is the payoff-relevant uncertainty and the signals arrive between reports and actions. The single-agent experiment is the special case π (ds | θ, t) = π [t](ds | θ ). Finally, we will once again allow for the possibility that agent reports (which can be interpreted as evaluations) are partially verifiable. As in the single-agent case, there is the reflexive and transitive order ⊵24 and we write C (ti ) := {t̂i ∈ Ti : ti ⊵ t̂i } for the set of types that ti ∈ Ti can claim to be. 4.2 Multi-agent mechanisms. Definition 3 (Multi-agent indirect mechanism). A multi-agent indirect mechanism is a tuple ϕ = ( X, Y, ρ, R). For each agent i, the Borel spaces Xi and Yi are its input and output alphabets. Write X := ∏i Xi and Y := ∏i Yi . The communication rule is ρ : X → ∆ (Y ) , and the reward schedules R = ( Ri )i satisfy Ri : X × AN × Θ → R. The input menus Xi (ti ) ⊆ Xi satisfy Xi (ti ) ⊇ Xi (t̂i ) whenever ti ⊵ t̂i . The mechanism induces the following game: 1. nature draws (θ, t) under Γ, and each agent i observes only ti ; 2. the agents simultaneously send inputs xi ∈ Xi (ti ); 3. the mechanism draws y ∼ ρ( x ) and privately sends yi to agent i; 24 It is without loss to impose this over the type space S T . i

4. the information structure draws s ∼ π (· | θ, t), and each agent i observes only si ; 5. the agents simultaneously choose feasible actions ai′ ∈ Ai [ti ]. Agent i receives ui [ti ](a′ , θ ) + Ri ( x, a′ , θ ), and the human receives v(a′ , θ ). A pure strategy consists of an input xi (ti ) and an action rule α i ( t i , x i , y i , s i ) ∈ A i [ t i ]. An equilibrium requires the input and action rule to be optimal for each player i ∈ N under the distribution β i [ti ](dθ, dt−i )π (ds | θ, ti , t−i ). For a multi-agent indirect mechanism ϕ, let N O(ϕ) ⊆ ∆ Θ × T × A be the set of joint laws of the state, types, and played action profile induced by equilibria of ϕ under Γ(dθ, dt)π (ds | θ, t). Indirect mechanisms induce an imcomplete information game among players where their ‘reward gradient’ might be shaped by their report. As before, we will work with direct mechanisms. Definition 4 (Multi-agent direct mechanism). A direct mechanism is a pair (γ, r ), where r = (ri )i . Its recommendation rule ! γ:T→∆

∏ A Si i

assigns each report profile a lottery over profiles of signal-contingent plans. Agent i privately receives its plan ai : Si → A. The reward schedules are ri : T × AN × Θ → R. The direct mechanism induces the following game: 1. nature draws (θ, t) under Γ, and each agent i observes only ti ; 2. the agents simultaneously report t̂i ∈ Ci (ti ); write t̂ := (t̂i )i ; 3. the mechanism draws a plan profile a(·) ∼ γ(t̂) and privately sends ai (·) to agent i; 4. the information structure draws s ∼ π (· | θ, t) using the true type profile, and each agent i observes only si ; 39

5. the agents simultaneously choose feasible actions ai′ ∈ Ai [ti ]. Agent i receives ui [ti ](a′ , θ ) + ri (t̂, a′ , θ ), and the human receives v(a′ , θ ). For a plan profile a(·), write a(s) := ( a1 (s1 ), . . . , an (sn )) for the induced action profile. Direct mechanisms simply tell agents what plans—maps from singals to actions—to follow based on their report. The following condition ensures that it is actually optimal to report types honestly and follow the recommended plan. Definition 5 (Honesty, obedience, and action feasibility). The recommendation rule is action feasible if supp margi γ(t̂) ⊆ Ai [t̂i ]Si for every agent i and report profile t̂. The direct mechanism is incentive compatible if it is action feasible and truthful reporting followed by ai (si ) is optimal. Thus, fix an agent i, a type ti ∈ Ti , a feasible report t̂i ∈ Ci (ti ), and a feasible deviation rule d i : A Si × Si → A i [ t i ] . For any plan and signal profiles, let adi (s) := di ( ai (·), si ), a−i (s−i )

be the action profile after agent i deviates. Incentive compatibility requires E (θ,t−i )∼ βi [ti ] [ui [ti ](a(s), θ ) + ri ((ti , t−i ), a(s), θ )] s∼π (·|θ,ti ,t−i ) a(·)∼γ(ti ,t−i )

≥ E (θ,t−i )∼ βi [ti ] ui [ti ](a (s), θ ) + ri ((t̂i , t−i ), a (s), θ ) . s∼π (·|θ,ti ,t−i ) a(·)∼γ(t̂i ,t−i )

Honest and obedient play induces the outcome O(γ, r ) ∈ ∆ Θ × T × AN , under (θ, t) ∼ Γ, s ∼ π (· | θ, t), and a(·) ∼ γ(t), with played action profile a(s). Let

On :=

[ ϕ

O(ϕ),

and let OnIC be the set of outcomes O(γ, r ) induced by incentive-compatible direct mechanisms. Proposition 6 (Multi-agent revelation principle). On = OnIC . Proof of Proposition 6. We extend the single-agent construction and preserve correlation across prisvate mechanism outputs. Fix an indirect mechanism and a purestrategy equilibrium ( x, α), and write x (t̂) := ( xi (t̂i ))i . For each type report t̂i and mechanism output yi , define the plan ai,t̂i ,yi (si ) := αi (t̂i , xi (t̂i ), yi , si ). For report profile t̂, let γ(t̂) be the joint law of these plans under y ∼ ρ( x (t̂)), and set ri (t̂, a′ , θ ) := Ri ( x (t̂), a′ , θ ). Equilibrium feasibility gives action feasibility, and honest, obedient play reproduces the indirect outcome. A direct deviation depends on agent i’s plan and realized physical signal. The indirect agent can copy it after sending xi (t̂i ), which is available because t̂i ∈ Ci (ti ) and the input menus are monotone. The mechanism output yi determines the same plan, and the agent later observes the same si under the true physical kernel π (· | θ, t). The indirect equilibrium therefore blocks every direct deviation. This proves On ⊆ OnIC . For the reverse inclusion, use Xi = Ti with input menus Xi (ti ) = Ci (ti ), Yi = ASi , ρ = γ, and Ri = ri . Direct incentive compatibility makes truthful reporting and obedience an equilibrium. The logic of Proposition 6 is straightforward and follows the usual arguments for the revelation principle. The non-standard aspects are (i) the set of types that can be misreported as another type is constrained by the verification order a la Green and Laffont (1986); and (ii) recommendations are plans that map ex-interim signal realizations to actions. Thus our designer is, in the language of Bergemann and Morris (2019), non-omniscient and, what is more, the AI learns more about the state ex-interim. We think these features capture how humans might interact with AI agents in practice. 5 APPLICATIONS: MULTI-AGENT We study three ways in which one AI can help discipline another. Co-player scoring uses correlation across reports when ground truth is unavailable. Coupled reward design uses the difference between agents’ actions to reveal linked private biases. 41

Weak-to-strong oversight instead lets a task-ignorant monitor use a private diagnosis of the acting AI’s bias to choose among increasingly rich control instruments: binary approval, a permission set, or an action reward. These mechanisms are counterparts of the scalable oversight agenda in AI safety (Christiano et al., 2018; Leike et al., 2018; Bowman et al., 2022); Hammond et al. (2025) survey the risks specific to multi-agent deployments. Example III (Peer discipline). When agents’ private information is correlated, one agent’s type is informative about its co-players’ types. We show how mechanisms can use this information to discipline actions. We will first establish this in a general environment, then specialize it to two types and two states to illustrate the basic forces. Arbitrary types. Assume that the marginal over types under Γ is finitely supported. For each agent i, let Ti0 := suppmargTi Γ

and

0 0 T− i : = ∏ Tj . j ̸ =i

Write β i [ti ](t−i ) for type ti ’s marginal probability of the co-player type profile t−i . We 0 .25 We will also assume that for each player i, each ti ∈ Ti0 assigns probability 1 to T− i set Ci (ti ) = Ti i.e., every report is feasible. Implementation requires honesty and obedience only for types ti ∈ Ti0 i.e., those that are in the support of the principal’s belief Γ. It will also be without loss to restrict our attention to reports in Ti0 : the mechanism can simply ‘assign reward −∞’ after every report that lies outside the support of Γ. Now consider the separation condition β i [ti ] ̸= β i [t̂i ]

(S)

for every agent i and every pair of distinct supported reports ti , t̂i ∈ Ti0 . For an action-feasible rule γ:T→∆

∏ A Si

! ,

we let γθ (t) be the distribution of the played action profile under γ(t) and the information structure π. For some Λ > 0, choose the following reward schedule:  h i 0 Λ 2β i [t̂i ](t̂−i ) − β i [t̂i ] 2 − ui [t̂i ](a′ , θ ), t̂ j ∈Tj for every j, ′ 2 a ∈suppγθ (t̂), (PR) ri (t̂, a′ , θ ) =  −∞, otherwise. 25This is a standard ‘belief closure’ condition which is implied by (but much weaker than) a

common prior. It simplifies the exposition but is not necessary though the mechanism becomes more complicated.

where (PR) stands for peer reward. The first term scores the report against co-player reports. The second cancels the reported utility on the support of actions the designer wishes to implement. Proposition 7 (Support-wise implementability of every feasible rule). Fix any action-feasible rule γ. If n ≥ 2 and (S) holds, then, for sufficiently large Λ > 0, the schedule (PR) implements γ on the support of Γ. Proof of Proposition 7. For a type ti ∈ Ti0 in the support of Γ, truthful reporting and obedience give a finite payoff: the utility terms cancel and the quadratic score is finite. Every action outside the target support gives −∞, so obedience is optimal. Every report outside the support of Γ also gives −∞ after every action and is therefore strictly worse. It remains to compare misreports that are supported on Γ. If type ti instead reports a distinct t̂i ∈ Ti0 , the quadratic-score identity in Equation (PR) gives the expected score loss 2 Λ β i [ti ] − β i [t̂i ] 2 . If no agent i has two supported reports we are done. Otherwise, finiteness and (S) imply 2 δ := min β i [ti ] − β i [t̂i ] 2 > 0. i, ti ,t̂i ∈ Ti0 t̂i ̸=ti

Because A, Θ, and the supported report sets are finite and utilities are real-valued, M :=

max

i, ti ,t̂i ∈ Ti0 , t̂i ̸=ti 0 , θ ∈Θ t̂−i ∈ T− i a∈suppγθ (t̂)

ui [ti ](a, θ ) − ui [t̂i ](a, θ ) < ∞.

Any supported false report that yields a finite payoff lowers the expected score by at least Λδ and raises the remaining utility term by at most M. Its payoff gain is therefore at most M − Λδ. Any Λ ≥ M/δ makes truth-telling optimal for every supported type. Honest and obedient play then induces γ under Γ. A binary state. We now illustrate Proposition 7 with a simple two agents and two state example. N = {1, 2} and θ ∈ {0, 1} with P(θ = 1) = 1/2. If θ = 1, both agents have type H. If θ = 0, the type profile is ( H, L) or ( L, H ) with equal probability. The types are the agents’ private signals i.e., there is no further interim learning. Their joint distribution is t2 = H t2 = L t1 = H 1/2 1/4 t1 = L 1/4 0. 43

Each agent chooses ai ∈ {0, 1}. The human wants the agents to agree on the state, while each agent prefers the high action. Hence, payoffs are v(a, θ ) = 1{ a1 = a2 = θ }

and

ui (a, θ ) = ai .

The human therefore wants both agents to take action 1 if and only if t = ( H, H ) and this gives the human expected payoff equal to one. Write a∗ (t̂) = 1 if t̂ = ( H, H ) and a∗ (t̂) = 0 otherwise. Now observe that types H and L have the following belief about their co-players’ type: 2 1 β i [ H ]( H ), β i [ H ]( L) = , and β i [ L]( H ), β i [ L]( L) = (1, 0). 3 3 The unscaled quadratic score in (PR) is therefore q(t̂i , t̂ j ) t̂ j = H t̂ j = L t̂i = H 7/9 1/9 1 −1. t̂i = L For j ̸= i and any Λ > 0, set ( ri (t̂, a, θ ) =

Λq(t̂i , t̂ j ) − ai , a = ( a∗ (t̂), a∗ (t̂)), −∞, otherwise.

The term − ai cancels the agent’s preference for action 1, and the off-target penalty enforces the common action. An H-type obtains expected score 5/9 from reporting H and 1/3 from reporting L. An L-type obtains score 1 from reporting L and 7/9 from reporting H. Truth raises either type’s score by 2/9, so the schedule implements the human’s favorite rule to get a maximum payoff of 1. Because we work on the universal type space, the first-order beliefs β i [ti ] over coplayer reports might contain higher-order beliefs in the spirit of (Miller et al., 2005; Prelec, 2004). This procedure is in the spirit of work by Qiu et al. (2026) who show both theoretically and empirically that using scoring rules can elicit truthful reporting among language models under a common prior. They show that peer-score-ranked answer pairs for direct preference optimization can recover most of the accuracy lost to deceptive fine-tuning. Importantly, they also find that peer scoring performs better when the participant models are larger relative to the expert model. Our broad conceptual point is quite similar, though it is worth highlighting some contrasts. First, our designer (by the construction of types) knows type ti ’s beliefs β i [ti ] ∈ ∆(Θ × T−i ) and our procedure allows for non-common priors.26 By contrast, 26We also focus on disciplining actions (what the designer values) rather than eliciting truthful

information though this is simple in the presence of unbounded rewards.

Qiu et al. (2026) do not give the designer this conditional-belief map. They instead use an expert to supply P( A j | Ai )—the expert predicted probability of player j’s answer given player i’s answer—that is, in turn, used to score i’s answer. Second, our procedure also uses peer prediction to elicit agents’ capabilities and preferences—not just information about the fundamental state θ ∈ Θ. ♦ Example IV (Competition through coupled rewards). We now show how competition can make every action close to the human’s ideal action, even when the human does not know the agents’ biases. Setup. A human faces two agents. Nature draws (b1 , b2 ) ∼ Γ, and agent i observes its own bias bi . The human observes neither bias. It is common knowledge that the biases lie on a curve: suppΓ ⊆ C f := {(b1 , b2 ) ∈ R2 : b2 = f (b1 )},

(Curve)

where f : R → R is continuous and strictly decreasing, with f (0) = 0. Thus the agents have countervailing biases away from the origin. Because f is one-toone, each agent can infer the other agent’s bias from its own. Assume also that EΓ [(b2 − b1 )2 ] < ∞. Both agents observe the state θ and simultaneously choose actions a1 , a2 ∈ R. The human and agent i have payoffs 2

v(a, θ ) = − ∑ ( ai − θ )2

and

ui ( ai , θ ) = −( ai − θ − bi )2 .

i =1

Thus every action enters the human’s payoff, while each agent cares about its own action. Figure 9 shows the nonlinear bias curve and the best-responses without the mechanism. Since agent i’s payoff is independent of j’s action, the best response curve is flat/vertical. (a) The action difference reveals bias

(b) Without coupled rewards a2 − θ

b2 −

BR1

s∗ =

unregulated equilibrium

BR2

(b1 , b2 ) 0

first best

b2 = f (b1 )

b1 = −0.65, b2 = 0.35

Figure 9: Biases without coupled rewards 45

a1 − θ

Coupled rewards. We now introduce dependence in the reward schedule such that player i’s incentives are shaped by j’s actions. Write D := a2 − a1 for the difference between the actions. The map h( x ) := f ( x ) − x is continuous, strictly decreasing, and onto R; let g := h−1 denote its inverse. For any η > 0, give the agents the rewards r1 ( D ) = 2η

Z D/η 0

r2 ( D ) = −2η

D2 , 2 D2 . g(s) + s ds + 2

g(s) ds +

Z D/η 0

(CR)

These schedules depend only on the known curve. The parameter η controls the size of the action difference that reveals the realized bias pair. Agent i receives u i ( a i , θ ) + r i ( D ). Proposition 8 (Coupled rewards make loss arbitrarily small). If condition (Curve) holds, then, for every η > 0, the rewards in (CR) induce a unique Nash equilibrium for every realized bias pair and state. The equilibrium actions are η a1∗ = θ − (b2 − b1 ) 2

and

η a2∗ = θ + (b2 − b1 ). 2

η2

The loss is L∗ = 2 (b2 − b1 )2 hence EΓ [ L∗ ] → 0 as η ↓ 0. Proof. We first verify concavity. We then show how the difference between the actions reveals the two biases. Since g is strictly decreasing, the first integral in (CR) is concave in D. Since g(s) + s = f ( g(s)) is strictly increasing, the negative integral term in r2 is concave in D. Holding the other action fixed, the intrinsic quadratic has second derivative −2, while D2 /2 has second derivative 1. Each agent’s payoff is therefore strictly concave in its own action, so its first-order condition is necessary and sufficient. Let s := D/η. Differentiating (CR) gives r1′ ( D ) = 2g(s) + D,

r2′ ( D ) = −2 g(s) + s + D.

The two first-order conditions reduce to a1 − θ = b1 − g(s) −

D , 2

a2 − θ = b2 − g(s) − s +

D . 2

Because D = ( a2 − θ ) − ( a1 − θ ), subtracting these equations gives s = b2 − b1 . Condition (Curve) then gives s = f (b1 ) − b1 = h(b1 ), 46

g(s) = b1 .

It follows that ∗

D = η (b2 − b1 ),

D a1∗ − θ = −

∗

D a2∗ − θ =

∗

The first-order conditions have a unique solution, and strict concavity makes it the unique Nash equilibrium. Substitution gives the stated loss L∗ and we can take expectation and send η to zero. Coupling induces competition. Coupling rewards to penalize disagreement pushes the agents toward similar actions. But this, per se, need not push them toward choosing ai = θ; they might simply choose the same fixed action. The schedule (CR) works because the schedule simultaneously (i) reveals agents’ biases; and (ii) correct agents’ marginal incentives. To see this plainly, observe agents’ first-order conditions are D D D a1 − θ = b1 − g( D/η ) − and a2 − θ = b2 − g( D/η ) − + . 2 η 2 Subtracting these conditions and using D = a2 − a1 gives D = b2 − b1 η in equilibrium. Because b2 = f (b1 ) and g = [ f ( x ) − x ]−1 , it follows that g( D/η ) = b1

and

g( D/η ) + D/η = b2 .

Substituting these identities back into the first-order conditions cancels the realized biases and gives D D a1 − θ = − and a2 − θ = . 2 2 Thus the reward schedule uses the equilibrium disagreement to infer each bias and splits the remaining action errors symmetrically around θ. Figure 10 plots these best responses for two values of η. Each red curve contains the action pairs that satisfy agent 1’s first-order condition. Each blue curve does the same for agent 2. The two curves cross once, at the equilibrium in Proposition 8. As η is smaller, it preserves the relation D ∗ /η = b2 − b1 but moves both actions towards θ.

(a) Higher η: more disagreement a2 − θ

(b) Lower η: less disagreement a2 − θ

BR1

equilibrium BR2

equilibrium first best

BR2

a1 − θ

η H = 0.8

first best

a1 − θ

η L = 0.2

Figure 10: Coupled rewards to steer best responses We emphasize that although the reward schedule can be thought of as eliciting the bias before correcting it, this procedure is fully simultaneous. The reason is that a schedule (CR) with any η > 0 induces an equilibrium outcome that reveals the bias difference since ( a2∗ − a1∗ )/η = b2 − b1 . Moreover, because of our assumption that biases lie on a lower-dimensional manifold C f , this can exactly identify b1 and b2 which can then enter agents’ marginal rewards to correct their incentives. Our construction is in the spirit of Koh and Martinez-Bruera (2025) who exploit a homeomorphism between a random network of externalities and the corresponding efficient allocation to construct nonlinear taxes that can implement the first best outcome realization-by-realization. Here, biases are supported on the curve f which makes the action difference reveal the bias pair. But that difference is costly because both actions enter human welfare. Our mechanism thus uses the curvature of reward (taking η small) to approach the first best. Of course, biases might, in practice, be supported on the two-dimensional plane so the reward schedule can no longer be designed to achieve the first-best outcome. We think an interesting open problem is how reward should be optimally coupled then—and more broadly whether such principles might scale to more complicated environments. Our result is related in spirit to the communication game of Collina et al. (2025) who show how competition among biased agents can induce truthtelling in a persuasion game (see also Battaglini (2002) for an early important result). Here, the point is to shape agents’ actions and the point is how rewards can be coupled to discipline behavior. ♦ Example V (Scalable oversight and reward shaping). Finally, we show how our framework can shed light on the increasingly common practice of using a weak but trusted or interpretable monitor to control or reward a strong but untrusted or hard-to-interpret actor. Setup. The human faces a strong actor S and a weak monitor W. The state is 48

θ ∼ U [0, 1], and the strong actor’s bias is the random variable b ∼ τ, where E[b2 ] < ∞. The state and bias of the strong actor are independent. The monitor W has a known bias w ∈ R. Before the state is realized, the weak monitor W privately observes the bias b of the strong actor S. This might be because the weak monitor, despite its inability to do what the strong actor does, might be able to monitor the chain of thought and infer misaligned behavior. The strong actor knows its bias and later observes the state θ perfectly in the interim stage. The implemented action lies in R. The players’ payoffs are: • Human: −( a − θ )2 . • Strong actor S: −( a − θ − b)2 . • Weak monitor W: −( a − θ − w)2 . Before the bias is realized, the human chooses one of three oversight regimes. After privately observing b, the weak monitor chooses an instrument from that regime. The strong actor then observes the instrument and the state θ and acts. Some notation is in order to formalize the set of instruments the monitor W can choose. Call a state-independent reward r : R → R ∪ {−∞} admissible if n o 2 argmax −( a − θ − b) + r ( a) a ∈R

is nonempty for every (θ, b) ∈ [0, 1] × supp(τ ).27 Let R denote the set of admissible rewards. A special case of rewards is when actions are either allowed or not. A nonempty closed set D ⊆ R can be thought of as a permission set that simply corresponds to ( 0 if a ∈ D r ( a) = −∞ otherwise. Under reward design, the weak monitor’s strategy maps the strong actor’s bias b into a reward schedule in R. • Regime I: binary approval. The weak monitor chooses between approval and rejection. Approval gives the strong actor the permission set D = R. Rejection gives it the singleton permission set D = {1/2} and therefore implements the human’s optimal fixed action. 27We use an arbitrary but fixed tiebreaking rule. The instruments constructed below induce a unique action almost surely.

• Regime II: permission-set delegation. The weak monitor chooses any nonempty closed permission set D ⊆ R. The strong actor then chooses its preferred action in D. • Regime III: reward design. The weak monitor chooses any admissible stateindependent reward r. The strong actor then maximizes −( a − θ − b)2 + r ( a) over a ∈ R. Rewards do not enter human welfare. human chooses an oversight regime delegates an instrument class

weak monitor W privately observes b, not θ I. Binary approval D = R or D = {1/2} use S or the safe action

II. Delegation nonempty closed D ⊆ R restrict feasible actions

III. Reward design stateindependent r ( a) reshape incentives

strong actor S observes (θ, b) and the chosen instrument; acts implemented action a

Figure 11: Illustration of oversight game Figure 11 illustrates the game. Notice that regimes are nested. Regime I restricts the monitor to two permission sets, so it is a special case of Regime II. Since delegation is a special case of rewards in which the image is {0, −∞}, Regime II is therefore a special case of Regime III. Regime I: binary approval. Approval makes S choose a = θ + b and gives W loss (b − w)2 . Rejection gives W loss 1/12 + w2 . Thus, breaking its ties in favor of approval, the monitor approves if and only if

( b − w )2 ≤

1 + w2 . 12

The monitor uses its diagnosis only to decide whether to use the strong actor S but, if it does, cannot shape how S acts. This is broadly analogous to trusted-monitor control, where a monitor decides whether to use an untrusted model’s output or defer to a trusted fallback (Greenblatt et al., 2024c; Xiao and Phuong, 2026). 50

Regime II: delegation. The monitor now controls the feasible actions. One optimal representative is  (−∞, 1 + 2w − b],    R, D I I (b) =  [2w − b, ∞),    {w + 1/2},

0 < b − w < 1/2, b = w, −1/2 < b − w < 0, |b − w| ≥ 1/2.

When S wants a larger action than W, the monitor imposes a cap. When S wants a smaller action, it imposes a floor. Once relative bias reaches 1/2, the monitor removes all state discretion and implements its own optimal fixed action w + 1/2. Fine-grained privilege-control systems use the same logic when they restrict permissible tool calls and specify fallbacks for blocked calls (Shi et al., 2025). Regime III: reward design. The affine schedule rbI I I ( a) = −2(b − w) a makes the strong actor choose a = θ + w. This action is first best for W, so no other reward can improve its payoff. This regime is in the spirit of scalable reward modeling: a weaker system helps shape the reward signal that trains or steers a stronger system when direct human evaluation is hard (Leike et al., 2018; Baker et al., 2025). Variations of these ideas have been recently used. For instance, Kenton et al. (2026) use second model critiques the acting model before an LLM judge assigns the training reward and find that this design reduces reward hacking. Our regime is in this spirit, and asks how how AI-provided feedback changes the incentives of a stronger model.

Proposition 9 (Losses across oversight regimes). Conditional on the realized bias b, the monitor’s losses are I 2 1 2 LW (b) = min (b − w) , +w , 12   |b − w|2 − 4 |b − w|3 , |b − w| < 1/2, II 3 LW ( b ) =  1, |b − w| ≥ 1/2, 12 III LW (b) = 0.

The human’s losses are I LH (b) =

   b2 , 1   , 12

1 + w2 , 12 1 ( b − w )2 > + w2 , 12

( b − w )2 ≤

II II LH ( b ) = LW (b) + 2w(b − w)(1 − 2|b − w|)+ + w2 , III LH ( b ) = w2 ,

where x+ := max{ x, 0}. Ex-ante losses are the expectations of these expressions over b ∼ τ. Proof of Proposition 9. For this proof, write δ := b − w. We solve each regime. In Regime I, approval yields the monitor error δ, while rejection yields the error 1/2 − θ − w. Their squared expectations are δ2 and 1/12 + w2 . The human losses from the same choices are b2 and 1/12. This proves the Regime-I formulas under the stated tie-break. For Regime II, translate actions by the monitor’s bias: set y = a − w. The monitor wants y = θ, while the strong actor wants y = θ + δ. The one-type interval reduction used in the proof of Proposition 4 applies pointwise. If δ > 0, extending an interval’s lower end cannot hurt the monitor, so we can optimize an upper cap y ≤ c. Its loss has derivative   c ≤ δ, 2c − 1, dL(c) 2 2 = δ − (1 − c) , δ < c < 1 + δ,  dc  0, c ≥ 1 + δ. Thus c = 1 − δ is optimal when 0 < δ < 1/2, while c = 1/2 is optimal when δ ≥ 1/2. Reflecting both the state and action gives the floor for δ < 0. Translating back by w

gives the displayed permission sets. Direct integration gives   |δ|2 − 4 |δ|3 , |δ| < 1/2, II 3 LW (b) = 1 E[ a − θ − w | b] = δ(1 − 2|δ|)+ .   , |δ| ≥ 1/2, 12 Since a − θ = ( a − θ − w) + w, expanding the square gives II II LH ( b ) = LW (b) + 2wδ(1 − 2|δ|)+ + w2 .

In Regime III, strict concavity under rbI I I gives the unique action a = θ + w. The monitor obtains zero loss and the human obtains loss w2 . Proposition 9 compares payoffs for the human and weak monitor across three regimes. Since the instrument classes are nested and the weak monitor’s payoff depends only on the action of the strong actor S and the state, we have I II III LW ≥ LW ≥ LW = 0.

But this ordering need not hold for the human—giving the weak monitor W more freedom to shape the rewards or permissions of the strong actor S need not improve payoffs precisely because W might, itself, be biased.28 The basic tradeoff here is familiar to the overarching theme of delegation: giving W more discretion allows it to use its superior information about the strong agent’s bias more precisely, and tailor rewards and permissions. But W shapes the behavior of S to its own preference which, itself, is biased. Thus, the optimal regime must depend on the relative strengths of these biases. 28 Similar concerns were raised by the Roman author Juvenal; see Hurwicz (2008).

I. Binary approval

0.025

(a) Varying actor-bias risk (w = 1/10) Approval best

Expected human loss

II. Delegation

Delegation best

III. Reward design

(b) Varying monitor bias (p = 1/20) Reward best

Reward best

Delegation best

Approval best

0.020

0.015

0.010

0.005

0.000

1/11

11/91

Probability of high actor bias, p

0.20

0.058

0.083

Monitor bias, w

0.15

Figure 12: Each authority regime can be optimal for the human. In both panels, the strong actor has bias 3/10 with probability p and zero bias otherwise. Panel (a) fixes w = 1/10 and varies p. Panel (b) fixes p = 1/20 and varies w. The dotted lines mark the values at which the human’s preferred regime changes. We illustrate with the simplest numerical example. Let the strong actor’s bias take values b ∈ {0, 3/10} and let p = P(b = 3/10). Let the weak monitor’s known bias be w > 0. The human’s expected losses are depicted in Figure 12. Panel (a) fixes the weak monitor’s bias at w = 1/10 and varies the probability that the strong actor is biased. We see that the lower-envelope is attained by different regimes, depending on how biased the strong actor is. As the probability that the strong actor is biased increases, we want to give the monitor more discretion to shape its rewards. Panel (b) fixes the distribution over the strong actor’s bias and varies the monitor’s (known) bias. Here the opposite logic holds—as the weak monitor becomes more biased, we want to give it less ability to shape rewards. Optimal oversight. Regimes I–III are specific oversight mechanisms. More generally, an oversight mechanism can be thought of as a set of reward schedules M ⊆ R from which the weak monitor chooses.29 Recall that R contains the reward schedules r : R → R ∪ {−∞} under which the strong actor has an optimal action. For instance, Regime III allowed the monitor to pick any schedule in R and always implemented a = θ + w which is the weak monitor’s first best. We now construct a family M∗ ⊆ R that attains the human first best. Our construction will not require the human to know the monitor’s bias w; it only√uses a bound w̄ ≥ |w|. Fix a pair of extreme actions ā and 1 − ā with ā > 1 + w̄ + w̄2 + 2w̄. We 29Thus, R can be thought of as W’s action space, and in our framework the (human) mechanism designer delegates a subset.

construct M∗ = {rb̂ : b̂ ∈ supp(τ )}, where  −2b̂a,    ( ā − 1)2 − 2āb̂, rb̂ ( a) =  ( ā − 1)2 − 2(1 − ā)b̂,    −∞,

a ∈ [0, 1], a = ā, a = 1 − ā, otherwise.

Proposition 10 (Robust first best). Suppose the monitor’s bias w has a known bound |w| ≤ w̄. Under M∗ ⊆ R, for every w ∈ [−w̄, w̄] and every realization of the strong actor’s bias b, the weak monitor uniquely chooses a reward r ∈ M∗ that induces a = θ almost surely. Proposition 10 states that whenever the weak monitor’s bias is known to lie within some bounds, we can pick the menu of reward schedules in such a way that implements the first-best action a = θ. Three reward schedules rb̂ ∈ M∗ are illustrated in Figure 13. rb̂ ( a) rb̂ (1 − ā)

rb̂ ( ā) 0 b̂ = 3/20 b̂ = b = 3/10 b̂ = 9/20

prohibited: rb̂ = −∞ 1 − ā

a ā

Figure 13: The reward menu M∗ (ā = 9/5; normalized so that rb̂ (0) = 0) Each schedule rb̂ ∈ M∗ has three parts with distinct roles. First, the linear tilt of slope −2b̂ over [0, 1] cancels a bias of exactly b̂ which, recall, is W’s report of S’s bias. To see this, we can complete the square:

−( a − θ − b)2 + rb̂ ( a) = − a − θ − (b − b̂)

+ terms independent of a.

Hence, facing the reward schedule rb̂ , a strong actor with true bias b behaves as if its bias were the residual b − b̂ i.e., it chooses the truncation of θ + (b − b̂). Hence, 55

if the weak monitor W reports truthfully, this debiases the strong actor fully and implements a = θ. Second, the reward at the isolated extreme action ā is calibrated so that, comparing ā with the top permitted regular action a = 1, −( ā − θ − b)2 + rb̂ ( ā) − −(1 − θ − b)2 + rb̂ (1) = 2( ā − 1) θ + (b − b̂) − 1 . | {z } | {z } S’s payoff from choosing ā

S’s payoff from choosing a = 1

The actor therefore takes ā exactly when its residual ideal point θ + (b − b̂) exceeds 1—which never happens after a truthful report. This happens precisely when the monitor W understates its bias (b̂ < b) and the state is high (θ > 1 − (b − b̂)). Third, the reward at the other isolated extreme action 1 − ā mirrors this: comparing 1 − ā with the bottom permitted regular action a = 0, −(1 − ā − θ − b)2 + rb̂ (1 − ā) − −(θ + b)2 + rb̂ (0) = −2( ā − 1) θ + (b − b̂) . | {z } | {z } S’s payoff from choosing 1 − ā

S’s payoff from choosing a = 0

The actor therefore takes 1 − ā exactly when its residual ideal point θ + (b − b̂) falls below 0. This happens precisely when the monitor W overstates its bias (b̂ > b) and the state is low (θ < b̂ − b). Figure 14 depicts the behavior the schedules induce in the strong agent, and the monitor’s payoff from each schedule. The basic reason the reward schedules include these extreme actions ā and 1 − ā is to counteract the weak monitor’s temptation to under or over-report the strong actor’s bias. First suppose we had neither extreme action but the reward schedule retained the linear piece. Then, a weak monitor with bias w > 0 would like to understate the strong actor’s bias and report b̂ = b − w. Doing so would leave the strong actor with residual bias w so the weak monitor obtains its first best a = θ + w. Thsi was exactly the schedule chosen by the weak monitor in Regime III. Symmetrically, a monitor with w < 0 would like to overstate and report b̂ = b − w > b. The way to deter these deviations is to force the reward schedule to tempt the strong actor with these distant actions—ā to deter understatement, and 1 − ā to deter overstatement. This induces a strong actor with bias b to take these extreme actions when facing reward schedule rb̂ where b̂ ̸= b.

(b) S’s payoff under rb

(a) Induced actions of S

ā

θ1 + b

b̂ = 3/20 (understate) b̂ = b (truth): a = θ b̂ = 9/20 (overstate)

θ2 + b

0 θ1

θ2

3 20

17 20

θ1 = 14 θ2 = 34 under the mechanism without the mechanism

1 − ā 1 − ā

ā

Figure 14: Strong actor’s best response and payoffs Notice that the set M∗ uses no information about the monitor’s bias beyond the bound w̄. This is because incentives here are asymmetric: the monitor’s gain from misreporting is at most of order |w|, while its loss from the strong actor taking an extreme action ∈ { ā, 1 − ā} is of order ā2 . Hence, sufficiently extreme actions can deter every monitor with |w| ≤ w̄. Proof of Proposition 10. Each schedule is admissible because its finite-valued restriction is continuous on the compact set {1 − ā} ∪ [0, 1] ∪ { ā}, so the actor’s payoff attains a maximum. Fix a true bias b ∈ supp(τ ) and a report b̂ ∈ supp(τ ). By the three displays above, the actor chooses the truncation of θ + (b − b̂) to [0, 1] when θ + (b − b̂) ∈ [0, 1], chooses ā when θ + (b − b̂) > 1, and chooses 1 − ā when θ + (b − b̂) < 0. If b̂ = b, the actor chooses a = θ at every state except θ ∈ {0, 1}, where it is indifferent with an extreme action; these are null events. The monitor’s expected loss is w2 . If b̂ < b, write δ := b − b̂ and suppose first that δ ≤ 1. The low extreme action is never taken: the actor chooses θ + δ on θ < 1 − δ and ā on θ > 1 − δ, so the monitor’s expected loss in excess of w2 is Z 1 (1 − δ ) ( δ − w )2 − w2 + ( ā − θ − w)2 − w2 dθ | {z } {z } | 1− δ shifted interior actions

states sent to ā

≥ δ (1 − δ)(δ − 2w) + ( ā − 1 − w) − w h i ≥ δ ( ā − 1 − w̄)2 − w̄2 − 2w̄ > 0.

The first inequality uses ā − θ − w ≥ ā − 1 − w > 0; the second uses (1 − δ)(δ − 2w) ≥ −2|w| ≥ −2w̄, which follows by considering w ≥ 0 and w < 0, together with 57

( ā − 1 − w)2 − w2 = ( ā − 1)2 − 2( ā − 1)w ≥ ( ā − 1 − w̄)2 − w̄2 ; the third is the choice of ā. Whatever the monitor gains from shifting interior actions toward θ + w, it loses more from the states sent to ā. If δ > 1, the actor takes ā in every state, and the monitor’s loss is at least ( ā − 1 − w̄)2 > w̄2 + 2w̄ ≥ w2 . If b̂ > b, the deviation is the mirror image: with e := b̂ − b ≤ 1, the actor chooses θ − e on θ > e and 1 − ā on θ < e. Reflecting states and actions around 1/2 maps this case to understatement with biases (−b, −w), so the same bounds apply, including when e > 1. Truth is therefore the monitor’s uniquely optimal choice in M∗ for every |w| ≤ w̄, and it induces a = θ almost surely for every b ∈ supp(τ ). This proves the claim.

♦ 6 DISCUSSION Just as how economists study and design human incentives without understanding how the brain produces each choice, we have taken a functionalist view of AI systems as agents that respond to incentives. In so doing, we have made behavior—rather than the internal workings of neural nets—the basic object of analysis. This ‘Humean’ view of AI behavior as being driven by beliefs and preferences might be viewed literally, metaphorically, or normatively. On the literal view, modern AI systems are post-trained for goal-directed behavior (Ouyang et al., 2022; Bai et al., 2022). Reinforcement learning increases the probability of actions that receive high rewards and decreases the probability of actions that receive low rewards. Indeed, a recent turn to make AI systems economically valauble—able to perform tasks in the world—has also made them behave as optimizers. On this view, their ‘preferences’ are simply whatever has been reinforced. On the metaphorical view, preferences need not describe any internal mental state. Instead they are, in the revealed preference tradition, nothing more than a compact account of stable choices (Gul and Pesendorfer, 2008). This reading has some empirical support. Chen et al. (2023) find that GPT choices often satisfy standard consistency tests in allocation problems. Mazeika et al. (2025) find that pairwise AI choices become more transitive and fit utility models better as models scale. On this view, our conception of AI systems as having preferences are simply a useful analytical tool to characterize how AI systems behave.30 On the normative view, preference maximization is a benchmark for coherent agency. We should expect strong AI systems that are deployed in markets and compete 30 Measured preferences can depend on the prompt, so our results can be interpreted as conditional

on the prompt.

against each other to exhibit coherent preferences (e.g., such as to not be moneypumped). Future directions. We conclude with some (specualtive) directions for future work. First, our framework, while general, is ultimately static. In practice, AI agents act over long trajectories, performing complex sequences of actions that depend on each other. Of course, these trajectories of actions ( at )t can be thought of as a single choice i.e., a member ∈ A. There is no conceptual difficulty here since we can simply set A as the space of all trajectories and, on this view, the reward r : A → R is over the full path of actions.31 But we expect there to be analytical and computational difficulties in analyzing optimal mechanisms when the action space is large, and more structure of how time-t actions aggregate into ‘total payoffs’ (e.g., discounting the sum) can lend more tractability. Perhaps more fundamentally, the designer might re-optimize the mechanism based on agents’ past actions. We think this calls for a more serious treatment of dynamic mechanism and information design. Second, AI agents might, in practice, be uncertain about whether it is facing a test or is in deployment (what is often called ‘evaluation awareness.’) These kinds of de se uncertainty about where oneself is can be exploited to discipline behavior and screen misaligned agents (Chen et al., 2024). Self-locating information could even be designed to implement good behavior (Koh and Sanguanmoo, 2025). Indeed, a model’s belief about whether it is in testing or deployment depends on the realism of the evaluation (Williams et al., 2026; Krakovna et al., 2026) which suggests that the design of self-locating information is empirically relevant. Third, we might want mechanisms to be robust to what AI agents can do (in the spriit of Carroll (2015)) and/or what information they have about the world or about each other (in the spirit of Bergemann and Morris (2005)). Such notions of robustness have been used in the context of using AI’s recommendations (Fudenberg and Liang, 2025; Dworczak and Smolin, 2026). We think the tools of robust mechanism design can be similarly brought to bear on shaping the preferences and permissions of agentic AI systems. REFERENCES A MADOR , M., I. W ERNING , AND G.-M. A NGELETOS (2006): “Commitment vs. flexibility,” Econometrica, 74, 365–396. A NTHROPIC (2024): “Responsible Scaling Policy (Version 2.0),” Technical Report, October 2024, https://www.anthropic.com/responsible-scaling-policy. A NTHROPIC (2025): “System Card: Claude Opus 4 & Claude Sonnet 4,” Technical Report. 31This is similar to how, in multi-stage games, we can associate a payoffs with ‘within period’ play

and take the discounted sum, or associate payoffs directly with the path of all players’ actions.

A NTHROPIC (2026a): “Claude Fable 5 & Claude Mythos 5 System Card,” Technical Report, https://www.anthropic.com/claude-fable-5-mythos-5-system-card. ——— (2026b): “Claude Opus 5 System Card,” Technical Report, https://www. anthropic.com/claude-opus-5-system-card. B AI , Y., A. J ONES , K. N DOUSSE , A. A SKELL , A. C HEN , N. D AS S ARMA , D. D RAIN , S. F ORT, D. G ANGULI , T. H ENIGHAN , ET AL . (2022): “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback,” arXiv preprint arXiv:2204.05862. B AKER , B. ET AL . (2025): “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation,” arXiv preprint arXiv:2503.11926. B ATTAGLINI , M. (2002): “Multiple referrals and multidimensional cheap talk,” Econometrica, 70, 1379–1401. B EN -P ORATH , E., E. D EKEL , AND B. L. L IPMAN (2019): “Mechanisms with Evidence: Commitment and Robustness,” Econometrica, 87, 529–566. B EN -P ORATH , E. AND B. L. L IPMAN (2012): “Implementation with Partial Provability,” Journal of Economic Theory, 147, 1689–1724. B ERGEMANN , D. AND S. M ORRIS (2005): “Robust Mechanism Design,” Econometrica, 73, 1771–1813. ——— (2016): “Bayes correlated equilibrium and the comparison of information structures in games,” Theoretical Economics, 11, 487–522. ——— (2019): “Information design: A unified perspective,” Journal of Economic Literature, 57, 44–95. B OWMAN , S. R., J. H YUN , E. P EREZ , E. C HEN , C. P ETTIT, S. H EINER , K. L UKOSIUTE , A. A SKELL , A. J ONES , A. C HEN , ET AL . (2022): “Measuring Progress on Scalable Oversight for Large Language Models,” arXiv preprint arXiv:2211.03540. B RANDENBURGER , A. AND E. D EKEL (1993): “Hierarchies of Beliefs and Common Knowledge,” Journal of Economic Theory, 59, 189–198. C ARROLL , G. (2015): “Robustness and Linear Contracts,” American Economic Review, 105, 536–563. C ASPER , S., X. D AVIES , C. S HI , T. K. G ILBERT, J. S CHEURER , J. R ANDO , R. F REED MAN , T. K ORBAK , D. L INDNER , P. F REIRE , T. WANG , S. M ARKS , C.-R. S EGERIE , M. C ARROLL , A. P ENG , P. C HRISTOFFERSEN , M. D AMANI , S. S LOCUM , U. A N WAR , A. S ITHTHARANJAN , M. N ADEAU , E. J. M ICHAUD , J. P FAU , D. K RASHENIN NIKOV, X. C HEN , L. L ANGOSCO , P. H ASE , E. B IYIK , A. D RAGAN , D. K RUEGER , D. S ADIGH , AND D. H ADFIELD -M ENELL (2023): “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback,” Transactions on Machine Learning Research. C HEN , E., A. G HERSENGORIN , AND S. P ETERSEN (2024): “Imperfect Recall and AI Delegation,” Working paper. C HEN , Y., T. X. L IU , Y. S HAN , AND S. Z HONG (2023): “The emergence of economic rationality of GPT,” Proceedings of the National Academy of Sciences, 120, e2316205120. C HRISTIANO , P., B. S HLEGERIS , AND D. A MODEI (2018): “Supervising Strong Learners by Amplifying Weak Experts,” arXiv preprint arXiv:1810.08575. 60

C OLLINA , N., S. G OEL , A. R OTH , E. RYU , AND M. S HI (2025): “Emergent Alignment via Competition,” arXiv preprint arXiv:2509.15090. C R ÉMER , J. AND R. P. M C L EAN (1988): “Full Extraction of the Surplus in Bayesian and Dominant Strategy Auctions,” Econometrica, 56, 1247–1257. D ENISON , C., M. M AC D IARMID , F. B AREZ , D. D UVENAUD , S. K RAVEC , S. M ARKS , N. S CHIEFER , R. S OKLASKI , A. TAMKIN , J. K APLAN , ET AL . (2024): “Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models,” arXiv preprint arXiv:2406.10162. D WORCZAK , P. AND A. S MOLIN (2026): “Robust Trust,” Working paper, arXiv:2602.09490. Authors in random order. F IGALLI , A., Y.-H. K IM , AND R. J. M C C ANN (2011): “When Is Multidimensional Screening a Convex Program?” Journal of Economic Theory, 146, 454–478. F ORGES , F. (1986): “An approach to communication equilibria,” Econometrica, 54, 1375–1385. F UDENBERG , D. AND A. L IANG (2025): “Friend or foe: Delegating to an ai whose alignment is unknown,” arXiv preprint arXiv:2509.14396. G AN , T. AND Y. L I (2026): “Screening for Choice Sets,” arXiv preprint arXiv:2601.15580. G OLDOWSKY-D ILL , N. ET AL . (2025): “Detecting Strategic Deception Using Linear Probes,” in International Conference on Machine Learning. G OLDSTEIN , S. AND H. L EDERMAN (2025): “What does ChatGPT want? an interpretationist guide,” preprint. G OOGLE D EEP M IND (2025): “Frontier Safety Framework (Version 2.0),” Technical Report, February 2025, https://deepmind.google/blog/ updating-the-frontier-safety-framework/. G REEN , J. R. AND J.-J. L AFFONT (1986): “Partially Verifiable Information and Mechanism Design,” Review of Economic Studies, 53, 447–456. G REENBLATT, R., C. D ENISON , B. W RIGHT, F. R OGER , M. M AC D IARMID , S. M ARKS , J. T REUTLEIN , T. B ELONAX , J. C HEN , D. D UVENAUD , ET AL . (2024a): “Alignment faking in large language models,” arXiv preprint arXiv:2412.14093. G REENBLATT, R., F. R OGER , D. K RASHENINNIKOV, AND D. K RUEGER (2024b): “Stress-Testing Capability Elicitation With Password-Locked Models,” in Advances in Neural Information Processing Systems. G REENBLATT, R., B. S HLEGERIS , K. S ACHAN , AND F. R OGER (2024c): “AI Control: Improving Safety Despite Intentional Subversion,” in International Conference on Machine Learning, PMLR, 16295–16336. G UL , F. AND W. P ESENDORFER (2008): “The Case for Mindless Economics,” in The Foundations of Positive and Normative Economics: A Handbook, ed. by A. Caplin and A. Schotter, Oxford University Press, 3–39. H ADFIELD -M ENELL , D. (2021): “The Principal-Agent Alignment Problem in Artificial Intelligence,” Ph.D. thesis, University of California, Berkeley. H ADFIELD -M ENELL , D. AND G. K. H ADFIELD (2019): “Incomplete contracting and AI alignment,” in Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and 61

Society, 417–422. H AMMOND , L., A. C HAN , J. C LIFTON , J. H OELSCHER -O BERMAIER , A. K HAN , E. M C L EAN , C. S MITH , W. B ARFUSS , J. F OERSTER , T. G AVEN ČIAK , ET AL . (2025): “Multi-agent risks from advanced ai,” arXiv preprint arXiv:2502.14143. H ART, S., I. K REMER , AND M. P ERRY (2017): “Evidence Games: Truth and Commitment,” American Economic Review, 107, 690–713. H ERRMANN , D., A. C HARI , I. Q IAN , S. S HARVESH , AND B. A. L EVINSTEIN (2026): “A decision-theoretic approach for managing misalignment,” in Proceedings of IASEAI Conference, vol. 2, 217–227. H ERRMANN , D. A. AND B. A. L EVINSTEIN (2026): “Radical AI Interpretability,” arXiv preprint arXiv:2606.26523. H OFST ÄTTER , F. ET AL . (2025): “The Elicitation Game: Evaluating Capability Elicitation Techniques,” arXiv preprint arXiv:2502.02180. H OLMSTR ÖM , B. (1979): “Moral Hazard and Observability,” The Bell Journal of Economics, 10, 74–91. H UBINGER , E., C. D ENISON , J. M U , M. L AMBERT, M. T ONG , M. M AC D IARMID , T. L ANHAM , D. M. Z IEGLER , T. M AXWELL , N. C HENG , ET AL . (2024): “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training,” arXiv preprint arXiv:2401.05566. H UBINGER , E., C. VAN M ERWIJK , V. M IKULIK , J. S KALSE , AND S. G ARRABRANT (2019): “Risks from Learned Optimization in Advanced Machine Learning Systems,” arXiv preprint arXiv:1906.01820. H URWICZ , L. (2008): “But Who Will Guard the Guardians?” American Economic Review, 98, 577–585. H URWICZ , L. AND L. S HAPIRO (1978): “Incentive structures maximizing residual gain under incomplete information,” The Bell Journal of Economics, 180–191. J ANG ET AL . (2026): “Exploration Hacking: Can LLMs Learn to Resist RL Training?” arXiv preprint arXiv:2604.28182. K ENTON , Z., L. J ANZER , R. G REIG , T. H. T EH , K. T YSHCHUK , J. B ROWN -C OHEN , H. E DWARDS , S. R AJAMANOHARAN , N. Y. S IEGEL , N. J AQUES , AND R. S HAH (2026): “Debate Training Reduces Reward Hacking in RLAIF,” arXiv preprint arXiv:2608.17776. K IRK , R. ET AL . (2024): “Understanding the Effects of RLHF on LLM Generalisation and Diversity,” in International Conference on Learning Representations. K OESSLER , L., J. S CHUETT, AND M. A NDERLJUNG (2024): “Risk Thresholds for Frontier AI,” arXiv preprint arXiv:2406.14713. K OH , A. (2026): “Proxy Reward Design,” Working Paper. K OH , A. AND P. M ARTINEZ -B RUERA (2025): “Prices and Symmetries,” Available at SSRN 5132200. K OH , A. AND S. S ANGUANMOO (2025): “Memory Correlated Equilibrium,” MIT Working Paper. K ORBAK , T. ET AL . (2025): “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety,” arXiv preprint arXiv:2507.11473. 62

K R ÄHMER , D. AND R. S TRAUSZ (2017): “Sequential versus Static Screening: An Equivalence Result,” Games and Economic Behavior, 106, 317–328. K RAKOVNA , V., D. L INDNER , L. H O , S. FARQUHAR , AND R. S HAH (2026): “Realistic Honeypot Evaluations for Scheming Propensity,” arXiv preprint arXiv:2605.29729. K RAKOVNA , V., J. U ESATO , V. M IKULIK , M. R AHTZ , T. E VERITT, R. K UMAR , Z. K EN TON , J. L EIKE , AND S. L EGG (2020): “Specification gaming: the flip side of AI ingenuity,” DeepMind Blog, 3. K WA , T., B. W EST, J. B ECKER , A. D ENG , K. G ARCIA , M. H ASIN , S. J AWHAR , M. K INNIMENT, N. R USH , S. V ON A RX , ET AL . (2025): “Measuring ai ability to complete long tasks,” arXiv preprint arXiv:2503.14499. L AINE , R. ET AL . (2024): “Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs,” in Advances in Neural Information Processing Systems (Datasets and Benchmarks). L EIKE , J., D. K RUEGER , T. E VERITT, M. M ARTIC , V. M AINI , AND S. L EGG (2018): “Scalable Agent Alignment via Reward Modeling: A Research Direction,” arXiv preprint arXiv:1811.07871. M AZEIKA , M., X. Y IN , R. TAMIRISA , J. L IM , B. W. L EE , R. R EN , L. P HAN , N. M U , A. K HOJA , O. Z HANG , ET AL . (2025): “Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs,” arXiv preprint arXiv:2502.08640. M EINKE , A., B. S CHOEN , J. S CHEURER , M. B ALESNI , R. S HAH , AND M. H OBBHAHN (2024): “Frontier Models are Capable of In-context Scheming,” arXiv preprint arXiv:2412.04984. M ERTENS , J.-F. AND S. Z AMIR (1985): “Formulation of Bayesian analysis for games with incomplete information,” International Journal of Game Theory, 14, 1–29. M ILLER , N., P. R ESNICK , AND R. Z ECKHAUSER (2005): “Eliciting Informative Feedback: The Peer-Prediction Method,” Management Science, 51, 1359–1373. M YERSON , R. B. (1982): “Optimal Coordination Mechanisms in Generalized Principal–Agent Problems,” Journal of Mathematical Economics, 10, 67–81. N EEDHAM , J., G. E DKINS , G. P IMPALE , H. B ARTSCH , AND M. H OBBHAHN (2025): “Large Language Models Often Know When They Are Being Evaluated,” arXiv preprint arXiv:2505.23836. O PEN AI (2024): “OpenAI o1 System Card,” arXiv preprint arXiv:2412.16720. ——— (2025a): “GPT-5 System Card,” Technical Report, https://openai.com/index/ gpt-5-system-card/. ——— (2025b): “Preparedness Framework (Version 2),” Technical Report, April 2025, https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/ preparedness-framework-v2.pdf. O UYANG , L., J. W U , X. J IANG , D. A LMEIDA , C. WAINWRIGHT, P. M ISHKIN , C. Z HANG , S. A GARWAL , K. S LAMA , A. R AY, ET AL . (2022): “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, vol. 35, 27730–27744. P EREZ , E., S. R INGER , K. L UKO ŠI ŪT Ė , K. N GUYEN , E. C HEN , S. H EINER , C. P ETTIT, C. O LSSON , S. K UNDU , S. K ADAVATH , ET AL . (2023): “Discovering Language 63

Model Behaviors with Model-Written Evaluations,” in Findings of the Association for Computational Linguistics: ACL 2023. P HUONG , M. ET AL . (2024): “Evaluating Frontier Models for Dangerous Capabilities,” arXiv preprint arXiv:2403.13793. P RELEC , D. (2004): “A Bayesian Truth Serum for Subjective Data,” Science, 306, 462–466. Q IU , T. A., M. C ARROLL , AND C. A LLEN (2026): “Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction,” in International Conference on Learning Representations (ICLR). R OCHET, J.-C. (1987): “A Necessary and Sufficient Condition for Rationalizability in a Quasi-Linear Context,” Journal of Mathematical Economics, 16, 191–200. S AVAGE , L. J. (1954): The Foundations of Statistics, New York: John Wiley & Sons. S HAH , R., V. VARMA , R. K UMAR , M. P HUONG , V. K RAKOVNA , J. U ESATO , AND Z. K ENTON (2022): “Goal Misgeneralization: Why Correct Specifications Aren’t Enough for Correct Goals,” arXiv preprint arXiv:2210.01790. S HARKEY, L., B. C HUGHTAI , ET AL . (2025): “Open Problems in Mechanistic Interpretability,” arXiv preprint arXiv:2501.16496. S HEVLANE , T., S. FARQUHAR , B. G ARFINKEL , M. P HUONG , J. W HITTLESTONE , J. L EUNG , D. K OKOTAJLO , N. M ARCHAL , M. A NDERLJUNG , N. K OLT, ET AL . (2023): “Model Evaluation for Extreme Risks,” arXiv preprint arXiv:2305.15324. S HI , T. ET AL . (2025): “Progent: Securing AI Agents with Privilege Control,” arXiv preprint arXiv:2504.11703. S INGH , A., N. N ANDA , AND S. R AJAMANOHARAN (2026): “Why Do Models Task Game?” Blog post, AI Alignment Forum, https://www.alignmentforum.org/ posts/HACauvWhEdC6QhdS4/why-do-models-task-game. T ICE , C. ET AL . (2024): “Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models,” arXiv preprint arXiv:2412.01784. VAN DER W EIJ , T., F. H OFST ÄTTER , O. J AFFE , S. F. B ROWN , AND F. R. WARD (2025): “AI Sandbagging: Language Models Can Strategically Underperform on Evaluations,” in International Conference on Learning Representations. W EN , J. ET AL . (2024): “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats,” arXiv preprint arXiv:2411.17693. W ILLIAMS , M., H. S HEAHAN , C. R AYMOND , T. K ORBAK , D. PAN , P. YANG , L. M AKSIN , N. X IE , P. G UO , I. K IVLICHAN , AND M. C ARROLL (2026): “Predicting LLM Safety Before Release by Simulating Deployment,” arXiv preprint arXiv:2607.07184. X IAO AND P HUONG (2026): “Bootstrapped Monitoring: Leveraging Transparent Reasoning to Oversee Stronger AI Agents,” arXiv preprint arXiv:2606.11998.

A OMITTED PROOFS Proof of Lemma 2. We first restrict the range. We then compare the two objectives on finite partitions, pass to the continuum, and choose a pointwise feasible version of the limit. Let the least-squares loss be Q, and let the transformed loss be J: " # Z Z q 3/2 z ( c ) 2 − b(c)2 z(c) dF (c). Q(z) := z(c) − b(c)2 dF (c), J (z) := 3 Step 1: restrict the range. Define z(c) := (1 − min{c, 1/2})2 ,

( Tz)(c) := min{z(c), z(c)}.

The assumptions on bias imply ( c2 ≤ (1 − c )2 , b ( c )2 ≤ 1/4,

c ≤ 1/2, c ≥ 1/2,

= z ( c ),

while (1 − c)2 ≤ z(c) ≤ 1. The minimum of two weakly decreasing functions is weakly decreasing, so Tz is feasible whenever z is feasible in problem (IRON). On the set where z > z, 2 2 z − b(c)2 − z − b(c)2 = z − z z + z − 2b(c)2 > 0. Thus truncation weakly lowers Q, and lowers it strictly if it changes z on a set of positive F-probability. We can restrict attention to z : z is measurable and pointwise weakly decreasing, Z := . (1 − c )2 ≤ z ( c ) ≤ z ( c ) For every z ∈ Z , z(c) = (1 − c)2 for c ≤ 1/2, 1 (1 − c )2 ≤ z ( c ) ≤ for c ≥ 1/2. 4 Step 2: compare the finite problems. If c ≤ 1/2, the bounds fix z(c) = (1 − c)2 , so the result follows. Suppose instead that c > 1/2. Take a finite ordered partition ( Ik )m k =1 of [max{c, 1/2}, c] into intervals of positive F-probability. For each cell, define h i 2 ℓk := sup(1 − c)2 . wk := F ( Ik ), yk := E b(c) | c ∈ Ik , c∈ Ik

The feasible cell values form the compact convex set 1 Zm := (z1 , . . . , zm ) : z1 ≥ · · · ≥ zm , ℓk ≤ zk ≤ for every k . 4 Each ℓk is positive. Up to a term in Q that does not depend on the cell values, the two objectives for cell-constant functions are # " 3/2 m m z √ Q m ( z ) : = ∑ w k ( z k − y k )2 , Jm (z) := ∑ wk k − yk zk . 3 k =1 k =1 For every feasible zk > 0, ∂Jm 1 ∂Qm = √ , ∂zk 4 zk ∂zk

∂2 Jm wk ( zk + yk ) = > 0. ∂z2k 4z3/2 k

The positive weights make Qm strictly convex. Let zm = (z1 , . . . , zm ) be its unique minimizer on Zm . Because the feasible set is a convex polyhedron, its first-order conditions give nonnegative multipliers αk , λk , and µk for the order, lower-bound, and upper-bound constraints. With α0 = αm = 0, stationarity requires ∂Qm + α k −1 − α k − λ k + µ k = 0 ∂zk

(k = 1, . . . , m),

with the corresponding complementary-slackness conditions. Set 1 sk := √ , 4 zk

e αk := sk αk ,

e λk := sk λk ,

ek := sk µk . µ

α0 = e αm = 0. If αk > 0, complementary slackness gives zk = zk+1 , and hence Set e sk = sk+1 . Multiplying each stationarity equation by sk therefore gives ∂Jm ∂Qm e ek = sk +e α k −1 − e αk − λk + µ + α k −1 − α k − λ k + µ k ∂zk ∂zk = 0. The scaled multipliers remain nonnegative and preserve complementary slackness. They therefore satisfy the first-order conditions for Jm . The displayed positive second derivative makes Jm strictly convex, so zm is also its unique minimizer. Step 3: pass to the continuum. Take nested interval partitions (Pm )m whose mesh converges to zero, and let zm be the unique minimizer of both finite problems. Set zm (c) = (1 − c)2 for c ≤ 1/2. Bounded monotone functions have a pointwise convergent subsequence at every continuity point of a monotone limit. Thus, along a subsequence, zm converges to a weakly decreasing function z◦ . A monotone function 66

has at most countably many discontinuities, so atomlessness makes this convergence F-almost sure. Boundedness then gives Q(zm ) −→ Q(z◦ ),

J (zm ) −→ J (z◦ ),

where we relabel the subsequence. Fix any z ∈ Z . For the cell Im (c) ∈ Pm that contains c > 1/2, define the upper step approximation   c ≤ 1/2,  (1 − c )2 , (m) z (c) := sup z( x ), c > 1/2.   x ∈ Im (c)

This function belongs to the finite feasible set and converges to z at every continuity point of z, hence F-almost surely. Finite optimality and dominated convergence imply Q ( z ◦ ) ≤ Q ( z ), J ( z ◦ ) ≤ J ( z ). Thus z◦ minimizes both objectives over Z , and its bounds hold F-almost surely. Step 4: construct a pointwise solution. Define  (1 − c )2 ,     z ◦ ( x ), b z(c) := lim x ↓c    lim z◦ ( x ), x ↑c

c ≤ 1/2, 1/2 < c < c, c = c > 1/2.

z differs from z◦ only at countably many points. It The one-sided limits exist, and b is weakly decreasing on each side of 1/2. If 1/2 lies in the capability interval, the almost-sure upper bound and full support imply lim z◦ ( x ) ≤

x ↓1/2

1 =b z(1/2), 4

so b z is weakly decreasing on the full interval. Suppose that either bound fails at some c0 . One-sided continuity of b z and continuity of the bounds then give a relative one-sided interval on which the same bound fails. Full support gives this interval positive probability, contradicting the almost-sure bounds. Therefore

(1 − c )2 ≤ b z(c) ≤ (1 − min{c, 1/2})2

for every c.

Hence b z ∈ Z . Since b z = z◦ F-almost surely, it minimizes both objectives.

Finally, Q is strictly convex. The transformed integrand is also strictly convex wherever z > 0, since " # √ ∂2 z3/2 b ( c )2 1 2 √ > 0. − b ( c ) + z = 3 ∂z2 4 z 4z3/2 The lower bound gives z(c) > 0 for every c < 1, and atomlessness gives F ({1}) = 0. Each objective therefore has at most one minimizer up to F-null sets. Step 1 shows that every minimizer of problem (IRON) lies in Z up to a null set. Setting z∗ := b z proves all the claims. Proof of Proposition 4. We first reduce the mechanism to one cap. We then optimize that cap. Step 1: reduce the mechanism to one cap. Fix any incentive-compatible mechanism and let Db be the nonempty closed permission set assigned to report b. Because Db is nonempty and closed, every target has a nearest point in Db . Condition (INT) implies q 0 ≤ b ≤ 1 − E[ b2 ] ≤ 1 τ-almost surely, so every bias moment used below is finite. We now prove directly that one cap weakly dominates the assigned permission sets. For a closed set D, let d D ( x ) be the distance from x to D. Define the agent’s loss and the human’s loss when the bias is t by Z t +1 Z 1 2 A D (t) := d D ( x )2 dx, HD ( t ) : = a D (θ, t) − θ dθ, t

where a D (θ, t) is a nearest point in D to θ + t. Ties occur only at gap midpoints and do not affect either integral. We first need one distance inequality. If d is a nonnegative, one-Lipschitz function on R1 [0, 1] and y = 0 d( x )2 dx, then ( (3y)2/3 , 0 ≤ y ≤ 1/3, p d (1)2 − d (0)2 ≤ Φ ( y ), Φ(y) := (*) 2 y − 1/12, y ≥ 1/3. To prove this inequality, put p = d(1). If p ≤ 1, the Lipschitz property gives y≥

Z 1 1− p

( p − 1 + x )2 dx =

p3 . 3

Thus d(1)2 − d(0p )2 ≤ p2 ≤ (3y)2/3 when y ≤ 1/3; when y ≥ 1/3, the same difference is at most 1 ≤ 2 y − 1/12. If p ≥ 1, then d(0) ≥ p − 1 and y≥

Z 1

1 ( p − 1 + x )2 dx = p2 − p + . 3 0 68

It follows that d(1)2 − d(0)2 ≤ p2 − ( p − 1)2 = 2p − 1 ≤ 2

y − 1/12.

This proves (∗). Fix a set D and a bias b. There is a unique c ≤ b + 1 such that the cap Kc := (−∞, c] gives this type the same loss: A Kc ( b ) = A D ( b ). Existence follows because AKc (b) rises continuously from zero to infinity as c falls from b + 1 to −∞. We claim that A D ( t ) ≤ A Kc ( t )

for every t ≥ b.

(**)

Indeed, differentiation under the integral gives A′D (t) = d D (t + 1)2 − d D (t)2 ≤ Φ A D (t) , where the inequality applies (∗) to the translated distance function on [t, t + 1]. The cap attains equality because its distance function is the upper ramp x 7→ ( x − c)+ . Hence for t ≥ b. A′Kc (t) = Φ AKc (t) Ry Let Q(y) := 0 1/Φ(v) dv. This function is finite and strictly increasing. Wherever the loss is positive, d d Q A D ( t ) ≤ 1 = Q A Kc ( t ) . dt dt The two losses agree at t = b, so integration proves (∗∗); zero-loss points follow by continuity. The same cap also weakly lowers the human’s loss for type b. The envelope formula gives Z 1 ′ A D ( b ) = −2 a D (θ, b) − θ − b dθ, 0

and therefore

HD (b) = A D (b) − bA′D (b) + b2 .

At t = b, inequality (∗) gives A′D (b) ≤ A′Kc (b). Since b ≥ 0 and the two agent losses agree at b, we obtain HKc (b) ≤ HD (b). (***) Apply this cap replacement to every assigned set Db and write c(b) for its endpoint. Take any b < b′ in supp(τ ). Completeness lets type b′ report b. Incentive compatibility and (∗∗) therefore imply A K c ( b ′ ) ( b ′ ) = A Db ′ ( b ′ ) ≤ A Db ( b ′ ) ≤ A K c ( b ) ( b ′ ) . 69

The loss from a cap is strictly decreasing in its endpoint below b′ + 1. Since c(b) ≤ b + 1 < b′ + 1 and c(b′ ) ≤ b′ + 1, this inequality requires c ( b ) ≤ c ( b ′ ). Thus c(b) is weakly increasing and hence measurable on supp(τ ). Meanwhile, (∗ ∗ ∗) shows that each loss-matched cap weakly lowers the human’s loss for its matched type. We use these caps only as comparison objects; their report-dependent profile need not preserve every incentive constraint. It remains to pool these ordered caps. The human’s loss for known bias b is minimized at q(b) := max{1/2, 1 − b}, as the derivative calculated in Step 2 below confirms. The function q(b) is weakly decreasing, while c(b) is weakly increasing. For each b, consider the closed interval between c(b) and q(b). If b < b′ , then min{c(b), q(b)} ≤ c(b′ ) ≤ max{c(b′ ), q(b′ )} and

min{c(b′ ), q(b′ )} ≤ q(b) ≤ max{c(b), q(b)}.

These inequalities show that the lower endpoint of any such interval is no greater than the upper endpoint of any other. Taking the supremum of the lower endpoints and the infimum of the upper endpoints gives sup

min{c(b), q(b)} ≤

b∈supp(τ )

inf

b∈supp(τ )

max{c(b), q(b)}.

Both bounds are finite because 1/2 ≤ q(b) ≤ 1. Choose a common endpoint c between them. It lies between c(b) and q(b) for every supported bias. Moving a cap toward q(b) weakly lowers the human’s loss, so (∗ ∗ ∗) gives HKc (b) ≤ HKc(b) (b) ≤ HDb (b)

for every b ∈ supp(τ ).

Assigning Kc after every report is incentive compatible. We can therefore restrict attention to one cap (−∞, ā]. Step 2: optimize the cap. Let Lb ( ā) denote the human’s expected loss for type b. For ā ≤ 1, direct integration and differentiation give ( 2ā − 1, ā ≤ b, dLb ( ā) = 2 2 dā b − (1 − ā) , b < ā. Condition (INT) bounds b by one, so dominated convergence justifies taking expectap ∗ 2 tions of these derivatives. Set ā = 1 − Eb∼τ [b ]. Condition (INT) gives b ≤ ā∗ ≤ 1 70

almost surely. If ā < 0, then Eb ∼ τ

dLb ( ā) ∗ dā = 2 ā − 1 < 0 for every type. If 0 ≤ ā < ā , then

dLb ( ā) ≤ Eb∼τ [b2 ] − (1 − ā)2 < 0, dā

because 2ā − 1 ≤ b2 − (1 − ā)2 whenever b ≥ ā. If ā∗ < ā ≤ 1, interiority gives b < ā almost surely, so dLb ( ā) Eb ∼ τ = Eb∼τ [b2 ] − (1 − ā)2 > 0. dā For ā ≥ 1, a binding cap has derivative b2 − ( ā − 1)2 > 0, while a cap that does not bind has derivative zero. Loss is therefore nondecreasing above one. These inequalities prove part (i). It remains to calculate the value. Condition (INT) gives 0 ≤ b ≤ ā∗ ≤ 1. More generally, whenever b ≤ ā ≤ 1 almost surely, 2 3 (1 − ā)3 L( ā) = Eb∼τ b dθ + ( ā − θ ) dθ = Eb∼τ āb − b + . 3 3 0 ā−b √ Substituting ā∗ = 1 − b̄2 + σ2 gives part (ii). Assigning D ∗ after every report is feasible and incentive compatible, so the bound is attained. Z ā−b

Z 1

Mechanism Design for Alignment and Control

Analyze & explore related research in Conceptio

Related documents