Opal: An Operator Algebra View of RLHF

We present Opal, an operator view of reinforcement learning from human feedback (RLHF). Objectives are expressed as ladders of two primitives on a base utility: additive penalties and multiplicative pairwise weights. We show a simple reduction law with if-and-only-if conditions: such ladders collapse to a normal form on pairwise margins when the reference is fixed, penalties are additive, and weights are independent of intermediate margins. When these assumptions do not hold (reference shift, non-additive gates, score-dependent weights), finite examples witness non-reducibility. Building on this view, we introduce GKPO (Generalized Kernel Preference Object), a canonical schema that many RLHF methods can be expressed in and, when reducible, mapped back from. GKPO provides a standard JSON serialization, canonicalization and hashing rules, and explicit flags with finite witnesses when assumptions fail. As illustrations, we show worked GKPO examples for DPO, RRHF, and ORPO, cross-method conversions via GKPO (where assumptions permit), and minimal stress tests (SHIFT/GATE/SCORE) that surface non-reducibility. We also release a small Python reference library implementing the GKPO schema, canonical hashing, and adapters for DPO and RRHF.