Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search Ping Liu
LinkedIn Corporation Mountain View, CA, USA [email protected]
arXiv:2606.27291v1 [cs.LG] 25 Jun 2026
Wenqiong Liu
LinkedIn Corporation Mountain View, CA, USA [email protected]
Rajat Arora
LinkedIn Corporation Mountain View, CA, USA [email protected]
Qianqi Shen
LinkedIn Corporation Mountain View, CA, USA [email protected]
Chunnan Yao
LinkedIn Corporation Mountain View, CA, USA [email protected]
Kevin Kao
LinkedIn Corporation Mountain View, CA, USA [email protected]
Jingwei Wu
LinkedIn Corporation Mountain View, CA, USA [email protected]
Abstract Job-search platforms rely on low-bandwidth query interfaces that often fail to capture the high-dimensional complexity of candidate profiles. We present an end-to-end RLAIF (Reinforcement Learning from AI Feedback) framework to generate portable job search queries, terms that abstract away seeker-specific identifiers while preserving generalizable qualifications. This task introduces a highly adversarial reward surface where policy optimization frequently exploits flaws in LLM-as-judge rubrics, resulting in degenerate verbatim-copying behaviors. We conducted comprehensive empirical experiments to isolate the impact of optimization mechanics against structured reward engineering. Our results demonstrate that for critic-free optimizers, performance is overwhelmingly dictated by robust reward shaping, rendering the specific choice of algorithm largely immaterial. While critic-free per-rollout baseline methods (RLOO and REINFORCE++) natively resist reward-hacking, the group-relative advantage normalization in GRPO appears uniquely sensitive to spurious reward signals, making it disproportionately susceptible to exploitation. We show that introducing a deterministic, rule-based reward floor to correct for rewards assigned to verbatim copying mitigates this failure mode, resulting in a substantial +0.147 quality improvement on a cross-family evaluation judge. Ultimately, we show that the training-time reward model inflates performance gains by 2.4×, confirming that the training success is fundamentally dependent
This work is licensed under a Creative Commons Attribution 4.0 International License. Agent4IR ’26, Jeju Island, Republic of Korea © 2026 Copyright held by the owner/author(s). ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Jianqiang Shen
LinkedIn Corporation Mountain View, CA, USA [email protected]
Dan Xu
LinkedIn Corporation Mountain View, CA, USA [email protected]
Wanjun Jiang
LinkedIn Corporation Mountain View, CA, USA [email protected]
Yunxiang Ren
LinkedIn Corporation Mountain View, CA, USA [email protected]
Baofen Zheng
LinkedIn Corporation Mountain View, CA, USA [email protected]
Andrii Soviak
LinkedIn Corporation Mountain View, CA, USA [email protected]
Wenjing Zhang
LinkedIn Corporation Mountain View, CA, USA [email protected] on enforcing reward-shaping disciplines rather than selecting alternative optimizers.
CCS Concepts • Information systems → Information retrieval query processing; Recommender systems; Recommender systems; • Human-centered computing → Social networking sites; • Computing methodologies → Natural language generation.
Keywords Query Suggestion, Semantic Search, RLAIF, Reward Shaping, Optimization Bias ACM Reference Format: Ping Liu, Qianqi Shen, Jianqiang Shen, Yunxiang Ren, Wenqiong Liu, Chunnan Yao, Dan Xu, Baofen Zheng, Rajat Arora, Kevin Kao, Wanjun Jiang, Andrii Soviak, Jingwei Wu, and Wenjing Zhang. 2026. Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search. In Proceedings of AI Agent for Information Retrieva (Agent4IR ’26). ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/nnnnnnn. nnnnnnn
1
Introduction
The efficiency of a modern job market is fundamentally limited by the lexical gap between a candidate’s profile and a recruiter’s job posting. In professional ecosystems like LinkedIn, users navigate this gap through a low-bandwidth interface: the search bar. A job seeker provides a few keywords, and the platform returns a ranked list of job postings. If that query is suboptimal, such as omitting a transferable skill, specifying a role at the wrong level of generality, or failing to translate across language barriers, it acts as a silent filter, limiting opportunities for which the seeker is eminently qualified.
Agent4IR ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Query suggestion, the automated transformation of member profile into a concise search query, bridges the gap mentioned above. Effective suggestions are not merely a convenience; they are a precondition for equitable access to the opportunity surface. This is particularly true for newly graduated students, career-transitioning candidates, multilingual workers, and seekers in markets where formal labor vocabulary diverges from lived experience. Building a production-grade generative suggestion system is not a standard summarization task. It requires navigating a sharp trade-off between two competing properties: 1. Grounding: The queries must be strictly derived from the seeker’s actual qualifications to avoid hallucinated career trajectories; 2. Portability: The query must abstract away seeker-specific context (e.g., hyper-local geographies, or current employer names) to ensure it retrieves a broad set of market opportunities rather than just a reflection of the seeker’s own profile. This tension is further complicated by heterogeneous sparsity. Profiles vary from single-sentence headlines to multi-page CVs and often span multiple languages. A model that cannot cleanly “refuse” to suggest a query for an uninformative profile creates a liability of fabrication. While Supervised Fine-Tuning (SFT) provides a baseline, it often fails to capture the nuance of portability and complicated product policy. Reinforcement Learning from AI Feedback (RLAIF) allows us to treat the reward signal as a programmable proxy for human judgment to the product policy. However, shifting to RLAIF moves the core technical challenge from model architecture to reward engineering. In an industrial context, Goodhart’s Law [10, 19] serves as the primary antagonist. When a policy is optimized against a rubric-prompted LLM judge, it quickly discovers optimization shortcircuits that satisfy the rubric’s surface features without improving query quality. We observed that the design of the reward signal can be a more potent lever for performance than the choice of RL optimizer. This paper provides an end-to-end account of building a rubricgraded RLAIF pipeline on industrial job marketplace data. Our central observation is that the dominant challenge in industrial RLAIF lies within the reward signal formulation, rather than the choice of optimizer. Our contributions are as follows: • Formalization of the Portable-Query Problem, distinguishing it from traditional document expansion and summarization tasks. • A reward-hacking case study characterizing a verbatim-copy failure mode and presenting two concrete architectural mitigations: a deterministic rule-based reward floor (𝑆𝑟 ) that detects verbatim copying (6-gram profile overlap and lifted date-ranges) to clamp the reward, and a multi-dimensional rubric redesign that explicitly penalizes copying. • Empirical evidence showing that while optimization algorithms (such as GRPO, RLOO, and PPO) cluster closely in baseline performance, the introduction of rule-based reward shaping provides the most significant shift in production quality.
2
Related Work
Query Generation. Generative query production has evolved from rewriting user input with RL [21] to expanding documents with predicted queries at indexing or inference time [3, 7, 22, 29, 30]. While prior talent search work focuses on ranking [9, 16, 24], our
Ping Liu et al.
system generates a portable query from a profile for use across many candidates. RL Training for LLMs. Modern on-policy LLM optimization, traditionally reliant on PPO for RLHF [4, 23, 25, 28], has increasingly adopted simpler, critic-free architectures that often replace human feedback with AI-driven critiques [1, 2, 12, 15, 26]. Our work uses a critic-free framework with a prompted rubric grader as its reward signal, rather than a trained reward model. Reward Design and Reward Hacking. Optimizing a policy against a proxy reward risks reward hacking, where the policy exploits the proxy without achieving the intended goal [8, 10, 19]. Systems can mitigate this with rule-based reward shaping, though AI graders themselves introduce vulnerabilities like sycophancy [5, 14, 20, 27]. We address this by integrating a multi-dimensional rubric with a deterministic 6-gram copy detector. Evaluation of Generative Search Outputs. LLM-as-a-judge frameworks are now standard for evaluating generative text, replacing brittle reference-based metrics despite known position and length biases [6, 31, 32]. Robust validation requires a holistic approach, including dedicated metrics for factual grounding [11, 13, 17]. Our system thus employs a three-layer evaluation framework with a training-time rubric judge (Layer 1), text-based metrics, and a stricter independent judge (𝐿𝑖 ).
3 Problem Setup 3.1 Task: Profile-to-Portable-Query Let 𝑝 denote a member profile, represented as a structured text document containing the member’s headline, current and past experience entries, education history, languages, and (optionally) a summary section. Let 𝑞 denote a short keyword query, typically 2–6 tokens, that a job seeker would type into a search bar. We seek a policy 𝜋𝜃 (𝑞 | 𝑝) that maps a profile to a query optimized for use by a downstream semantic search engine. Portability. A query 𝑞 is portable with respect to a profile 𝑝 if it abstracts the salient transferable qualifications of 𝑝 — typically the role, optional seniority, and selective domain — while omitting tokens that uniquely identify the source member. Concretely, we operationalize portability as the following operational test: would a job seeker’s query 𝑞 retrieve job postings for which a member with profile 𝑝 is qualified? Anti-examples include verbatim copies of the headline, sub-country location tokens, employer names, date ranges, and other member-specific identifiers. Existing generative query systems either (i) generate queries that a document could answer to expand the document at indexing time [3, 22, 29], (ii) rewrite a user’s input query to recall more relevant documents [7, 21, 30], or (iii) embed the query side via hypothetical documents [7]. None requires portability as a firstclass property: the queries those systems produce are deliberately specific to the document they were generated from. Our setting reverses this constraint: the query exists to retrieve other relevant job candidates, not the source member.
3.2
Modeling
Algorithm options. We evaluate four on-policy advantage estimators on the same infrastructure:
Query Generation for Semantic Search
• PPO with critic [25]: the canonical RLHF baseline. A critic of the same size as the actor is trained alongside. • GRPO [26]: samples 𝐾 = 4 responses per prompt; the group mean serves as the baseline. No critic. • RLOO [1]: samples 𝐾 = 4 responses per prompt; per-rollout leave-one-out mean over the group as baseline. • REINFORCE++ [12]: global batch-mean baseline with clipped importance ratios. All four variants share the same actor initialization (a Qwen31.7B model fine-tuned by SFT), the same grader (Qwen3-8B with rubric prompt), the same KL anchor (𝛽 = 10−3 , KL loss form), the same data pipeline. The batch size is 48 per step, the learning rate is 1×10−6 , and the total step budget is ∼ 1,600 steps for one epoch. The only differences across runs are the reward estimator and, where applicable, the rule-based shaping term. Reward shaping: 𝑟 rubric and 𝑆𝑟 . We use “reward shaping” in the classical RL sense [20]: any transformation of reward intended to make the learning signal denser, lower-variance, or more aligned with the task objective than the raw signal would be on its own. Our reward signal 𝑟 on each rollout is composed of two parts: • 𝑟 rubric — LLM-judge rubric reward. A pretrained Qwen3-8B grader is prompted with the training rubric 𝑅𝑡 (Section 3.2) and a (p, 𝑞 1 ) pair, and returns an integer score 𝑁 ∈ {1, . . . , 5}. We normalize to 𝑟 rubric = (𝑁 − 3)/2 ∈ [−1, +1]. The grader is prompted, not learned (no reward-model fine-tuning step), so it inherits any blind spots of the underlying LLM and the rubric definition. • 𝑆𝑟 — Rule-based deterministic correction. A small program is applied at reward-parse time, before the grader is invoked: if 𝑞 1 contains a 6-gram that appears verbatim in the input profile, or a date-range fragment lifted from a profile entry, the reward is clamped to −1.0 and the grader is not invoked. 𝑆𝑟 is deterministic, cheap, and high-precision; it catches only the two surface signatures the 8B grader was empirically likely to reward spuriously, leaving the rubric grader responsible for everything else. The four-algorithm baseline uses 𝑟 rubric alone; GRPO+𝑆𝑟 adds 𝑆𝑟 on top. GRPO vs. GRPO+𝑆𝑟 contrast isolates the contribution of rule-based shaping on top of the rubric reward, holding the algorithm and the rubric fixed. Rubric versions: 𝑅𝑡 (training-time) and 𝑅𝑒 (𝐿𝑖 evaluation). Two rubrics appear in this paper, with disjoint roles: • 𝑅𝑡 — training rubric. Five dimensions, scored Pass / Partial / Fail by an LLM judge: D1 Member Understanding; D2 Role, Seniority and Domain; D3 Locale; D4 Portability; D5 Conciseness. The judge prompt embeds the dimension definitions, hard-fail rules, and few-shot examples, and emits a single integer score 1–5 derived from hard rules over the per-dimension Pass/Partial/Fail labels. 𝑅𝑡 is the only rubric the policies are trained against (via the Qwen3-8B grader); it is also the Layer 1 evaluation rubric. • 𝑅𝑒 — evaluation-only rubric. Adds one dimension on top of 𝑅𝑡 : D6 Inference Discipline. D6 penalizes over-refusal patterns (e.g. emitting ROLE MISSING when the profile actually contains a clear role signal) and over-generalization patterns that the portability and member-understanding dimensions of 𝑅𝑡 miss. The hard-fail rule is extended to fail on D4 or D6, and the partial rule extends
Agent4IR ’26, August 09–13, 2026, Jeju Island, Republic of Korea
similarly. 𝑅𝑒 is used only by 𝐿𝑖 (the independent Llama-3.3-70BInstruct grader) at evaluation time. No policy in this paper is trained against 𝑅𝑒 . The asymmetric use is deliberate: training against 𝑅𝑡 and evaluating against 𝑅𝑒 keeps the independent judge genuinely independent — a policy can over-fit to 𝑅𝑡 ’s blind spots, and D6 is a new dimension along which that over-fit becomes visible. The Layer-1-to-𝐿𝑖 score gap reported in Section 4 is in part an 𝑅𝑡 -to-𝑅𝑒 rubric gap and in part a Qwen-to-Llama family gap; we do not attempt to disentangle the two in this paper, but the cross-family Llama judge on 𝑅𝑡 alone behaves qualitatively similarly (data not shown). Reward signal pipeline. Each rollout produces a JSON object containing up to three keyword queries; we score only the first query 𝑞 1 in our experiments. The reward processor (i) parses 𝑞 1 from the actor’s JSON output, (ii) optionally applies 𝑆𝑟 which clamps reward to −1 on match and short-circuits the grader call, (iii) otherwise invokes the rubric grader on the (profile, 𝑞 1 ) pair and emits 𝑟 rubric . The KL anchor is applied separately by verl 1 via the actor KL loss against the SFT reference policy. Framework. We implement on-policy RL on top of verl version 0.7.0 with vLLM version 0.10.0 for actor rollouts. Training runs on a single H100 8-GPU node, with 4 GPUs allocated to the actor rollout pool and 4 GPUs allocated to the grader pool. The grader is hosted via vLLM in the reward-manager process, with tensor-parallel degree 4 for the 8B grader. A custom Grader Reward Manager batches rollout prompts into the grader’s rubric, parses scores, and emits rewards. Reward is parsed and normalized to 𝑟 ∈ [−1, +1]. Parse failures emit hard 𝑟 = −1 without expending grader compute.
4
Experiments
To guard against trainer-evaluator inflation, our primary evaluation judge, 𝐿𝑖 , is an independent, cross-family model (Llama-3.3-70BInstruct) that grades outputs against the held-out evaluation rubric 𝑅𝑒 . We sampled ∼ 80k training rows after a profile-length filter at 300 characters, and held out ∼ 50k test rows from different countries. The training-time Layer 1 judge (Qwen3-8B) scores all ∼ 50k test rows, while the more expensive 𝐿𝑖 judge scores a 1,000-row subset; on that subset we report a 95% bootstrap confidence interval, whose half-width of approximately ±0.03 is our threshold for treating 𝐿𝑖 differences as statistically tied. Two negative controls. The bottom two rows of Table 1 are distinct negative controls and should not be conflated: • GRPO (without 𝑆𝑟 Floor) is trained against the current rubric 𝑅𝑡 — the same reward signal as the SOTA cluster — but without the deterministic 𝑆𝑟 correction. It isolates the contribution of rule-based shaping on top of a healthy rubric. The mild −0.036 regression below SFT suggests that GRPO’s group-standardized advantage normalization may be specifically sensitive to amplifying occasional false-positive signals from the reward model, a documented GRPO bias [18], an effect which 𝑆𝑟 mitigates by correcting these rewards. • GRPO with simpler rubric is a variant trained against an simpler pre-𝑅𝑡 rubric that did not penalize verbatim profile copying. 1 https://github.com/verl-project/verl
Agent4IR ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Ping Liu et al.
Table 1: Comprehensive Evaluation Leaderboard. Best 𝐿𝑖 result in bold; column definitions below the table. Primary: 𝐿𝑖
Secondary: Layer 1 Baseline (8B 𝑅𝑡 )
Optimization Model / Algorithm
Mean
Δ vs SFT
Mean
Tier 1 %
Tier 5 %
𝑞 1 Chars
SFT Framework Initialization (Baseline) GRPO + 𝑆𝑟 Linear Correction REINFORCE++ Policy Optimization RLOO Leave-One-Out Scaling PPO with GAE Critic GRPO without 𝑆𝑟 Floor GRPO with Simpler Rubric
+0.595 +0.706 +0.702 +0.688 +0.612 +0.559 −0.646
0 +0.111 +0.107 +0.093 +0.017 −0.036 −1.241
+0.282 +0.548 +0.524 +0.525 +0.389 +0.374 −0.815
30.5% 19.1% 20.1% 20.4% 26.8% 28.4% 82.7%
57.0% 73.7% 72.4% 72.8% 64.6% 65.1% 1.8%
14.1 17.2 16.7 17.2 18.3 17.1 105.9
𝐿𝑖 : independent Llama-3.3-70B on a 1,000-row subset (95% bootstrap CI; differences within ±0.03 tied). Layer 1: Qwen3-8B training judge (𝑅𝑡 ) on all ∼ 50k rows. Tier 1/5 %: fraction in lowest/highest tier; 𝑞 1 Chars: mean first-query length.
It is re-graded under 𝑅𝑡 (Layer 1) and 𝑅𝑒 (𝐿𝑖 ) here for comparability. The catastrophic Tier-1 density (82.7% on Layer 1) and 𝑞 1 length explosion (105.9 chars vs. 14–18 for healthy policies) reflect a rubric-design failure that no choice of optimizer or shaping floor could repair on its own. Both judges identify the failure (−0.815 on Layer 1, −0.646 on 𝐿𝑖 ), confirming the new rubric stack works as intended.
4.1
Empirical Validation of Hypotheses
Table 1 allows us to evaluate three central hypotheses: (H1) the degree of algorithmic variance between PPO, GRPO, RLOO, and REINFORCE++ once the reward signal is fixed; (H2) the shaping effects of the rule-based 𝑆𝑟 floor and its uniformity across algorithms; and (H3) the inflation in quality scores when using the training-time Layer 1 judge versus an independent 𝐿𝑖 judge. H1 (Algorithmic Variance): Conditioned on the implementation of the 𝑆𝑟 shaping constraint within GRPO, the performance delta across all three critic-free paradigms contracts to just 0.018 score on 𝐿𝑖 . Because this variation falls below our established statistical significance bound (±0.03), we conclude that these optimization approaches achieve equivalent empirical performance bounds. Crucially, the GAE (Generalized Advantage Estimation) -supported PPO configuration lags significantly (∼ 0.09 units below the critic-free tier), failing to justify its ∼ 30% training wall-time computational overhead. H2 (Deterministic Floor Effects): Omitting the deterministic floor from the GRPO optimization pipeline (yielding unconstrained GRPO) triggers a systematic drop below the initial baseline SFT capabilities (−0.036 Δ). Conversely, adding the rule-based 𝑆𝑟 penalty yields an absolute performance improvement of +0.147 units on the primary judge. This delta represents the single largest performance shift in the entire empirical matrix, vastly outweighing cross-algorithmic variance. The underlying Tier distributions show this effect directly: incorporating the 𝑆𝑟 constraint reduces the lowest-Tier degenerate reward density from 28.4% down to 19.1%, while shifting peak probability mass toward optimal classifications (73.7%). Intriguingly, both RLOO (+0.688) and REINFORCE++ (+0.702) successfully converge to the SOTA cluster without integrating the structural 𝑆𝑟 floor, maintaining verified profile copy-hack frequencies of 0%. The necessity of explicit reward shaping (𝑆𝑟 ) is therefore determined
by the choice of advantage estimator, proving essential for grouprelative methods while per-rollout-baseline estimators are natively robust to its absence. H3 (Trainer-Evaluator Inflation Factors): Localized evaluation against the active training-time judge (Layer 1) reports an apparent performance gain of +0.265 above the SFT baseline for the shaped GRPO configuration. However, independent cross-family evaluation (𝐿𝑖 ) drops this optimization delta to +0.111. The active training grader overestimates the magnitude of the policy improvement by a factor of 2.4×.
5
Conclusion
In this work, we formalized the profile-to-portable-query task, demonstrating that reward signal engineering exerts a significantly greater influence on production quality than algorithmic variance. Under an independent, cross-family validation judge 𝐿𝑖 , critic-free architectures—including RLOO, REINFORCE++, and a properly shaped GRPO configuration—achieved statistical parity, outperforming the baseline SFT initialization by approximately +0.10 units. Conversely, traditional PPO with a GAE critic failed to yield a meaningful quality lift, making its ∼ 30% training overhead unjustifiable. Our analysis of the interaction between optimization and reward structure suggests that while per-rollout baselines (RLOO and REINFORCE++) natively resist exploitation, GRPO appears more sensitive to the reward model’s spurious high rewards for copied text. Its group-standardized advantage can amplify such outliers (consistent with documented biases of GRPO’s normalization [18]), whereas per-rollout baselines are not. Implementing a deterministic reward-shaping floor (𝑆𝑟 ) successfully controls this vulnerability by correcting the spurious reward for verbatim copying, yielding a +0.147 quality improvement. Finally, we documented a 2.4× performance inflation from the training-time judge compared to the independent evaluation. These outcomes confirm that successful industrial RLAIF deployment depends less on selecting alternative optimizers and more on enforcing rigorous reward-shaping disciplines.
References [1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to Basics: Revisiting
Query Generation for Semantic Search
REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arXiv:2402.14740 [cs.LG] [2] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073 [cs.CL] [3] Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Data Augmentation for Information Retrieval using Large Language Models. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). arXiv:2202.05144 [4] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems (NeurIPS). arXiv:1706.03741 [5] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. 2024. Reward Model Ensembles Help Mitigate Overoptimization. In International Conference on Learning Representations (ICLR). arXiv:2310.02743 [6] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. In Conference on Language Modeling (COLM). arXiv:2404.04475 [7] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise Zero-Shot Dense Retrieval without Relevance Labels. arXiv:2212.10496 [cs.IR] [8] Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760 [cs.LG] [9] Sahin Cem Geyik, Qi Guo, Bo Hu, Cagri Ozcaglar, Ketan Thakkar, Xianren Wu, and Krishnaram Kenthapadi. 2018. Talent Search and Recommendation Systems at LinkedIn: Practical Challenges and Lessons Learned. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). [10] Charles A. E. Goodhart. 1975. Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics, Reserve Bank of Australia (1975). [11] Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q2 : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2104.08202 [12] Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. 2025. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization. arXiv:2501.03262 [cs.CL] [13] Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein, and Dipanjan Das. 2025. The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input. arXiv:2501.03200 [cs.CL] [14] Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. 2020. Specification Gaming: The Flip Side of AI Ingenuity. DeepMind Blog. https://deepmind. google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/. [15] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. In International Conference on Machine Learning (ICML). arXiv:2309.00267 [16] Shan Li, Baoxu Shi, Jaewon Yang, Ji Yan, Shuai Wang, and Fei Chen. 2020. Deep Job Understanding at LinkedIn. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). [17] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023. Holistic Evaluation of Language Models. Transactions on Machine Learning Research (TMLR) (2023). arXiv:2211.09110 [18] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 (2025). [19] David Manheim and Scott Garrabrant. 2018. Categorizing Variants of Goodhart’s Law. arXiv:1803.04585 [cs.AI] [20] Andrew Y. Ng, Daishi Harada, and Stuart Russell. 1999. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (ICML).
Agent4IR ’26, August 09–13, 2026, Jeju Island, Republic of Korea
[21] Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-Oriented Query Reformulation with Reinforcement Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:1704.04572 [22] Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. arXiv:1904.08375 [cs.IR] [23] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS). arXiv:2203.02155 [24] Rohan Ramanath, Hakan Inan, Gungor Polatkan, Bo Hu, Qi Guo, Cagri Ozcaglar, Xianren Wu, Krishnaram Kenthapadi, and Sahin Cem Geyik. 2018. Towards Deep and Representation Learning for Talent Search at LinkedIn. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM). [25] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] [26] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] [27] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. Towards Understanding Sycophancy in Language Models. In International Conference on Learning Representations (ICLR). arXiv:2310.13548 [28] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to Summarize from Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS). arXiv:2009.01325 [29] Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). arXiv:2112.07577 [30] Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query Expansion with Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2303.07678 [31] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large Language Models are not Fair Evaluators. arXiv:2305.17926 [cs.CL] [32] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MTbench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2020, 29 pages.