arXiv CSopen access

AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning

Preprint. Under review.

AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning Peilin Wu1 , Xinlu Zhang3 , Kun Wan2 , Wentian Zhao2 , Gang Wu2 , Xinya Du1 , Zhiyu Chen1 1 The University of Texas at Dallas, 2 Adobe Inc. 3 Department of Computer Science, University of California, Santa Barbara, {peilin.wu,zhiyu.chen2}@utdallas.edu

arXiv:2605.18592v1 [cs.LG] 18 May 2026

Abstract Rubric-based reward shaping is an effective method for fine-tuning Large Language Models (LLMs) via reinforcement learning (RL), where structured rubrics decompose standard outcome rewards into multiple dimensions to provide richer reward signals. To keep rubrics informative as the model evolves during training, recent works make the rubrics adaptive based on local signals such as the rollouts from the current step or pairwise comparisons. However, these methods discard the diagnostics produced during evaluation after immediate use and prevent the long-term accumulation and strategic reuse of evaluation knowledge. This forces the system to re-derive evaluation principles from scratch, limits its ability to detect recurring suboptimal behaviors, and forfeits the curriculum-like progression that a persistent training history would naturally support. To address these limitations, we introduce AMARIS (A Memory-Augmented Rubric Improvement System), which grounds rubric modifications in long-term training history. At each training step, AMARIS analyzes individual rollouts, aggregates findings into step-level summaries, retrieves relevant historical context from a persistent evaluation memory through both static (recent steps) and dynamic (semantically matched) retrieval, and updates rubrics based on these accumulated analyses. This procedure runs asynchronously alongside the normal RL loop with minimal overhead. Experiments across both closed and open-ended domains, including science, medicine, instruction following, and creative writing, show that AMARIS consistently outperforms the baselines, such as +1.6 points on GPQA-Diamond and +1.0 points on IFBench over the strongest baselines. Ablation studies show that static and dynamic memory retrieval contributes to the performance gain and their combination provides the strongest results with moderate retrieval budgets sufficient to provide most of the gain, and that the entire pipeline adds only 5% time overhead through asynchronous execution. These results show that persistent evaluation memory can transform rubric-based reward shaping from a stateless, per-step heuristic into an evidence-driven loop for RL training. The source code of AMARIS is at https://github.com/qualidea1217/AMARIS.

1

Introduction

Reinforcement learning (RL) aligns large language models (LLMs) with human preference or task-dependent objectives by training reward models (Ouyang et al., 2022; Bai et al., 2022; Touvron et al., 2023) or using verifiable reward functions (Shao et al., 2024) to provide a scalar optimization signal. However, static reward models or functions often struggle when facing suboptimal behaviors such as reward hacking, where LLMs learn shortcuts that increase rewards without actually improving output quality (Skalse et al., 2022; Gao et al., 2023), or open-ended tasks that a single ground-truth verifier is hard to define (Zheng et al., 2023). One promising way to resolve this is to perform rubric-based evaluation, where structured and interpretable rubrics decompose overall output quality into assessable 1

Preprint. Under review.

dimensions such as correctness, relevance, and safety (Kim et al., 2024; Ye et al., 2025). By making evaluation criteria explicit, rubric-based approaches provide multi-dimensional reward signals that combine interpretability with flexibility during RL training. However, a fixed set of rubrics defined before training is still insufficient as it cannot anticipate the novel suboptimal behaviors emerged during ongoing RL training. To increase the flexibility of rubric-based rewards, recent work has begun to make the rubrics adaptive. Some methods generate rubrics per instance from the query alone (Gupta et al., 2025) or let the policy model self-propose rubrics for its own outputs (Sheng et al., 2026; Xu et al., 2026), while others through instance-wise comparisons between current and reference policy rollouts (Rezaei et al., 2025), jointly optimization with the LLM judges via RL (Xu et al., 2026), or generate and select rubrics that can best distinguish the current batch of rollouts (Shao et al., 2025). These methods adapt rubrics based only on recent or local signals, without grounding the updates using the rich information from long-term dynamics of training, potentially causing the rubric adaptation to be suboptimal and limiting the models’ performance. Precise rubric weight adjustments are also difficult to achieve without longterm evidence of how different weighting strategies affect model behavior. Furthermore, without tracking which capabilities the policy has mastered over the course of training, the system cannot progressively raise the evaluation standards, making curriculum-like progression (Li et al., 2025; Freitag et al., 2025) hard to achieve. To address these limitations, we propose AMARIS, A Memory-Augmented Rubric Improvement System that utilizes long-term training history to drive rubric evolution and to provide adaptive reward signals. At each training step, AMARIS first generates structured analyses for each rollout covering reward hacking detection, curriculum advancement opportunities, and rubric effectiveness. These individual analyses are then aggregated into a step-level summary that identifies common behavioral patterns across the batch. To update rubrics, AMARIS retrieves relevant past analyses, summaries and update records from a persistent evaluation memory, grounding every modification in historical evidence and longitudinal trends rather than single-step results. All individual analyses, step-level summaries, and rubric update records are stored into this evaluation memory, which subsequent steps retrieve from to ground future rubric modifications in the full history of past evaluations. We conduct experiments across four diverse domains (science, medicine, instruction following, and creative writing), covering both closed-ended and open-ended tasks. AMARIS delivers consistent gains over baselines under both global and per-instance rubric strategies. For example, AMARIS with per-instance rubrics achieves 34.0 on HealthBench, +1.0 points over the strongest baseline. AMARIS also outperforms strong baselines across multiple science, instruction following, and creative writing benchmarks. Ablation studies show that both static and dynamic memory retrieval contribute to the performance gain and that their combination yields the strongest results across all settings. Importantly, the entire pipeline runs asynchronously alongside the RL training loop, adding only ∼5% overhead in terms of time consumption. Our major contributions are as follows: • We identify persistent evaluation memory as an overlooked component in adaptive rubric-based reward modeling, and show that grounding rubric updates in longterm training history is more effective than relying solely on local signals. • We propose AMARIS, a system that couples structured rollout analysis, step-level summarization, and hybrid static + dynamic memory retrieval to produce memorygrounded rubric updates with minimal overhead to the RL training loop. • We demonstrate through experiments on various domains that AMARIS consistently outperforms baselines, and provide a detailed analysis of the influence from different memory configurations, underlying LLMs, and rubric evolution.

2

AMARIS

AMARIS introduces a memory-augmented rubric improvement loop into rubric-based reinforcement learning. This is achieved through a three-staged procedure: (1) An individual rollout analyzing stage described in Section 2.1; (2) A step-level batch summarization stage 2

Preprint. Under review.

Assign Reward

Normal RL Pipeline

Update Model

Rollout

Individual Rollout Analysis

Analysis

Step-level Batch Summarization

Rollout

Reward Hacking Detection Training Stage Assessment

Analysis

Behavioral Patterns

Rollout

Curriculum Advancement

Static Retrieval

Dominant Reward Hacking Analysis

Query Generation

Task Saturation

AMARIS Pipeline

Dynamic Retrieval

Defensive Curriculum Advancement

New Rubrics

Maintenance

Rubric Improvement Plan

Additional Observations

Assign Reward

Rubric Improvement

Batch Summary

Current Learning phase

Rubric Analysis

Rollout Generation & Others

Query Store

Memory Store

Store

Figure 1: The AMARIS system operates asynchronously in parallel with the normal RL pipeline to continuously refine rubrics. described in Section 2.2; (3) A rubric improvement stage that grounds every rubric change in recent and semantically matched analysis retrieved from memory described in Section 2.3. This procedure can be executed asynchronously, in parallel with the normal RL pipeline, as described in Section 2.4. 2.1

Individual Rollout Analysis

In a standard rubric-based RL paradigm, the policy model’s rollouts y ∼ πϕ (· | x ), where πϕ is the policy model, x is the input, and y is the output, are evaluated against a set of active n g,t rubrics. At each training step t, AMARIS maintains an active rubric set Gt = {( g j , w j )} j=1 where g is the natural language definition of each individual rubric, w is the scalar weight, n g,t is the number of active rubrics. The final scalar reward rt is then generated as rt ( x, y) = Sθ ( x, y, Gt ), where Sθ is the process of scoring using LLM as a judge (prompt template in Figure 2-3). However, scalar rewards do not necessarily show the genuine quality due to suboptimal behaviors from model like reward hacking or incompetent rubrics. To support reliable rubric improvement, similar to human researchers conducting case studies, AMARIS performs a deeper analysis of each individual rollout about the model’s behavior. For each rollout, AMARIS receives the model’s input and output, the current rubric set, training metadata (the high-level goal, current step, total steps, model size, etc.), and supplemental context such as ground-truth answers, denoted by z, and produces a structured diagnostic report a = Aψ ( x, y, Gt , z), where Aψ is the LLM for analysis (prompt template in Figures 4-7). Each analysis provides five types of information: (i) Reward hacking detection: Whether the rollout shows signs of reward hacking, such as over-refusal for safety or sycophancy, reported with evidence and confidence level. (ii) Training-stage assessment: Whether the observed performance is expected given the model size at the current training step. (iii) Rubric analysis: How the current rubrics may have shaped the observed behavior and whether there are specific flaws in rubrics or weights that incentivize undesired outputs. (iv) Curriculum advancement: Which capability or behavior should be prioritized next to advance the model toward the high-level training goal. (v) Additional observations: Any other noteworthy signals that may be relevant for later analysis. The resulting structured analysis is stored in the persistent evaluation memory, preserving the evidence needed for step-level summarization in Section 2.2 and rubric improvement in Section 2.3. 2.2

Step-Level Batch Summarization

Individual rollout analyses cannot represent the general situation on the step-level and are too local to justify reliable rubric changes on their own. AMARIS therefore aggregates all the individual analyses collected during one or more training steps into a step-level summary 3

Preprint. Under review.

about what the model is currently doing, where it is failing, and whether the current rubrics remain informative. n

Once all the analyses { a j } j=a,t1 for individual rollouts are produced, where n a,t is the number of analyzed rollouts for step t, AMARIS summarizes them for that step into a step-level batch n summary bt = Bψ ({ a j } j=a,t1 ), where Bψ is the LLM for batch summarization (prompt template in Figures 8-11). The summary contains the general health of training and the current learning phase, the behavioral patterns that recur throughout the batch, the dominant reward hacking risk and its evidence, the weakest rubric, signs of task saturation and a provisional plan for the rubric improvement. By synthesizing step-level trends, bt provides the rubric improvement stage with a view of what the policy is currently doing and where the current set of rubrics fall short. When training steps contain more analyses than can be processed in a single summarization pass, AMARIS partitions the set into chunks denoted by n n {c j } j=c,t1 , where each c j ⊆ { ai }i=a,t1 , |c j | ≤ n a,c , n a,c is the chunk size, and nc,t is the number of chunks, summarizes each chunk independently, and then consolidates all the intermediate n summaries into the final step-level summary bt = Cψ ({ Bψ (c j )} j=c,t1 ), where Cψ is the LLM for consolidation (prompt template in Figures 12-15). This hierarchical strategy, referred to as ”meta-summarization,” allows the pipeline to scale to large rollout batches while preserving a fixed downstream interface for the retrieval and rubric improvement stages. The step-level batch summarizations for each step are also stored in the persistent evaluation memory for the future rubric improvement stage in Section 2.3. 2.3

Memory-Augmented Rubric Improvement

Without memory, rubric updates are reactive and can respond only to the changes for the current step, with no mechanism for recalling whether a similar suboptimal behavior appeared earlier, a previous rubric update failed, or the model has been steadily improving. AMARIS addresses this limitation through persistent memory together with a hybrid retrieval strategy. Persistent evaluation memory. AMARIS stores all previous evaluations in a database that n n persists throughout the training process as memory Mt = Mt−1 ∪ { a j } j=a,t1 ∪ { Bψ (c j )} j=c,t1 ∪ bt ∪ ut . The memory is persistent and continuously accumulates individual rollout analyses a, step-level summaries b, chunk-level summaries if meta-summarization is enabled Bψ (c), and rubric improvement records u. Here, u is a structured record containing the rubric update strategy, retrieved history consulted, rubric edit operations, resulting adaptive rubric set, and a short summary for future steps. Note that u will only be added after rubric update at t. Each evaluation document is indexed with structured metadata including its type, training step, creation timestamp, and an embedding representing the semantic meaning of the document. The memory therefore contains rich information about the past evaluations and supports various retrieval strategies. Static and dynamic retrieval. Before each update, AMARIS retrieves relevant historical context It = { It,s , It,d } from two complementary steps of retrieval. (i) static memory retrieval: AMARIS receives the current step-level summary together with up to N steps preceding summaries and forms the static retrieval results It,s = {b j ∈ Mt−1 : t − N ≤ j ≤ t − 1}. We intentionally restrict static retrieval to recent step-level summaries because they provide a compact view of the recent training trajectory with temporal continuity. (ii) dynamic memory retrieval: AMARIS reads the current step-level summary and proposes nq,t targeted search queries, written as {ql }l =1 = Qψ (bt ), where Qψ is the LLM for query generation and nq,t is the number of queries (prompt template in Figure 16-18). Qψ generates a small set of complementary queries rather than paraphrases. These typically target the dominant failure mode, suspected rubric flaw, possible curriculum objective, and failure from similar past updates. These queries are executed against memory, which may return earlier individual analyses, summaries, or rubric update records from any nq,t point in training, forming the dynamic retrieval results It,d = { R(ql , Mt , D )}l =1 , where 4

Preprint. Under review.

R(·) means the retriever and D is the number of documents retrieved per query. The returned documents are de-duplicated before rubric update. This retrieval policy keeps the updater context bounded to not overwhelm the underlying LLM that conduct the rubric update while maintaining flexibility. If the static window retrieves N preceding summaries, dynamic retrieval uses at most qn queries each returning at most D documents, and the current step-level summary accounts for one additional artifact, then the number of retrievals satisfies | It | ≤ N + nq,t D. Rubric improvement. Based on the retrieval results, AMARIS generates the newest set of rubrics as Gt+1 , ut = Uψ ( Gt , bt , It ), where Uψ is the LLM for rubric improvement (prompt template in Figures 19-22). AMARIS selects among three broad strategies: (i) Defensive: It modifies the rubrics and/or weights to reduce suboptimal behaviors and uses retrieved memory to check whether the same suboptimal behavior has appeared before and, if so, applies a stronger correction than previous attempts. (ii) Curriculum advancement: It raises the evaluation standard when the current rubrics appear saturated and retrieved memory confirms that the model has mastered the easier requirements. (iii) Maintenance: It makes only minor or no changes when there is no need to do so. To realize its chosen strategy, the LLM for rubric update Uψ may apply any combination of rubric operations (also shown in the prompt template): Create: creating new rubrics; Update: updating rubric text for precision; Delete: deleting useless or counterproductive rubric; Reweight: reweighting to shift training emphasis; Merge: merging overlapping rubrics; Split: splitting one rubric into atomic sub-rubrics. Uψ also produces reasoning summarizing what history was consulted and what lessons should be retained. The resulting update record is written back to the memory so that every update becomes available as evidence for subsequent updates. 2.4

RL Pipeline Integration

The entire rubric improvement procedure involves multiple LLM calls that together add substantial latency. To avoid blocking the RL training loop, AMARIS executes the procedure asynchronously and in parallel with the normal RL pipeline. Specifically, for every step, rollouts are scored immediately using the currently active set of rubrics, while the rubric improvement proceeds in the background. Once the update completes, the updated rubrics become active for subsequent scoring. Although such asynchronous execution causes the rubrics to be slightly off-policy, it greatly reduces the latency by hiding the execution in the background. It also requires fewer interference with the original RL training process and less modification to the code of the RL training pipeline.

3

Experiment Setup

This section describes the experimental framework used to evaluate AMARIS. We outline the datasets, evaluation benchmarks, baselines, and implementation details to provide a clear context for our results. 3.1

Datasets & Evaluation

We evaluate AMARIS across four diverse domains with both open-ended and closed-ended tasks. For each domain, we pair a rubric-annotated training set with one or more evaluation benchmarks. (i) Science. We train on RaR-Science (Gunjal et al., 2025), a corpus of scientific questions annotated with rubrics, and evaluate on GPQA-Diamond (Rein et al., 2024). We report average accuracy (%) over 10 measurements as the metric. (ii) Medicine. We train on RaR-Medicine (Gunjal et al., 2025), a medical variant of the rubric-annotated training data, and evaluate on HealthBench (Arora et al., 2025). We report the HealthBench composite score as the metric. (iii) Instruction following. We follow the experimental setup of OpenRubrics (Liu et al., 2026) and train on the OpenRubrics (Liu et al., 2026) dataset. We evaluate on three instruction-following benchmarks: IFEval (Zhou et al., 2023), InfoBench (Qin et al., 2024), and IFBench (Pyatkin et al., 2025), which we report the overall accuracies of each of them. (iv) Creative writing. We train on the writing subset of RubricHub (Li et al., 2026), a large-scale rubric-annotated dataset. We evaluate on WritingBench (Wu et al., 5

Preprint. Under review.

2025b) and Creative Writing v3 (Paech, 2025) from EQ-Bench and report the overall scores as the metric, also referred as CW-v3 in the following sections.

3.2

Baselines

We compare AMARIS against several baselines. Three baselines apply across all four domains: (1) Naive (no RL): the base model without any RL fine-tuning. (2) RubricHub (Li et al., 2026): a large scale rubric-annotated dataset with evaluations on rubric-based reinforcement learning (RuRL) that covers all the domains we tested. (3) RuscaRL (Zhou et al., 2026): a rubric-based RL method that we evaluate across all four domains. We also include domain-specific baselines. (i) Science & Medicine. We include RaR (Gunjal et al., 2025), which uses a fixed set of rubrics throughout training on the same RaR-Science and RaR-Medicine data. (ii) Instruction following. We include OpenRubrics (Liu et al., 2026), Rubric-ARM (Xu et al., 2026), and RLCF (Viswanathan et al., 2025). For all the baselines, we report the values obtained by re-training and re-evaluating using the same base model as ours, if possible. If not, we reference their reported values.

3.3

Implementation Details

We perform all RL training with Qwen2.5-7B-Instruct (Qwen et al., 2025) as the base model, GRPO (Shao et al., 2024) as the RL algorithm with group-wise and batch-wise reward normalization enabled, using 4 NVIDIA A100 80GB GPUs. We train all models with up to 400 steps and save the checkpoint every 50 steps, and use the final saved checkpoint for testing to ensure a fair evaluation. For all the LLM usages in AMARIS, we use local inference for all the open-source models deployed on a server having 8 NVIDIA A100 80GB GPUs to reduce cost. By default, we use GPT-OSS-120B (OpenAI et al., 2025) for scoring and individual analysis, and GPT-5.1 (Singh et al., 2025) for the remaining usages. This allocation reflects that scoring and individual analysis account for the majority of token consumption and therefore benefit most from a cost-effective model, while the remaining stages consume fewer tokens but demand higher reasoning capability. We analyze the influence of LLM choices in Appendix C. The memory is implemented with Chroma (chroma-core, 2026) as the vector database. For reward computation, following previous works (Gunjal et al., 2025; Li et al., 2026), we use a single LLM call per rollout. For all the LLM usages including the policy and AMARIS, we use the default setting for inferences. Unless otherwise stated, we evaluate under both global rubrics and per-instance rubrics settings. For global rubrics, AMARIS generates the rubric set at the first training step using the exact same pipeline described in Section 2 but without memory and the asynchronous execution (the scoring for the first step will wait for the rubric generation) as a cold start; for per-instance rubrics, we use the annotated rubrics provided by each dataset for the cold start. Each rubric set maintains its own dedicated evaluation memory: global rubrics share a single memory across all instances, whereas per-instance rubrics give every unique input its own rubric set and memory that is not accessible to other instances. These per-instance pipelines are logically independent and may be executed in parallel. For global rubrics, the rubric improvement is triggered at every training step by default. Per-instance rubrics, however, can only be updated each time the specific instance re-appears because that is the only point of time that new rollouts are generated for a particular sample generates for analysis, since not every instance appears at every step. To compensate, we set the batch size to 256 and the number of samples per query to 4, thereby increasing the effective number of training epochs and the revisit frequency of individual instances during training. We also enable the meta-summarization and set the chunk size n a,c to 32 for global rubrics. We set the maximum rubric staleness to 1 version, meaning that each training step scores rollouts using rubrics updated from at most one version prior. If the background pipeline has not yet completed, the training loop waits for its result before proceeding. We additionally measure performance under different rubric update intervals in Section D and profile the latency in Section 4.3. 6

Preprint. Under review.

Science

Medicine

GPQA-D

HealthBench

IFEval

InfoBench

IFBench

WritingBench

CW-v3

Cross-domain baselines Naive (no RL) RubricHub (RuRL) RuscaRL

35.0 38.8 38.5

22.7 33.0 32.9

77.3 79.8 79.0

78.1 83.5 83.2

28.2 33.5 33.0

45.2 56.9 56.1

37.4 39.0 38.6

Science & Medicine baselines RaR

37.6

31.2

–

–

–

–

–

Instruction-following baselines OpenRubrics (DPO via Rubric-RM) Rubric-ARM RLCF

– – –

– – –

79.5 80.4 78.6

83.0 83.7 84.1

33.7 35.0 28.2

– – –

– – –

AMARIS (Our proposed system) AMARIS (global rubric) AMARIS (per-instance rubric)

39.9 40.4

33.6 34.0

80.6 81.0

84.8 85.2

35.2 36.0

57.9 57.3

39.5 40.1

Method

Instruction Following

Creative Writing

Table 1: Main results across four evaluation domains. Best results in each column are in bold. “–” indicates that the result is not applicable to that domain.

4

Results & Analysis

This section presents the main results and a comprehensive analysis of AMARIS. We compare against baselines across four domains (Section 4.1), conduct ablation studies on memory configuration and retrieval budgets (Section 4.2), analyze the influence of underlying LLMs (Appendix C), and profile the pipeline latency (Section 4.3). We also provide a detailed analysis of how rubrics evolve over the RL training under global rubrics setting and four qualitative case studies in Appendix E and F. 4.1

Main Results

We compare AMARIS against general and domain-specific baselines across four evaluation settings. As shown in Table 1, AMARIS consistently achieves the best performance on all benchmarks under both rubric strategies, with the per-instance rubric setting yielding the strongest results on six of the seven benchmarks. Relative to the strongest non-AMARIS baseline, the best AMARIS result improves by +1.6 on GPQA-Diamond, +1.0 on HealthBench, +0.6 on IFEval, +1.1 on InfoBench, +1.0 on IFBench, +1.0 on WritingBench, and +1.1 on CW-v3. 4.2

Ablation Studies On Memory Usage

We conduct two sets of experiments to validate the core design choices of AMARIS about using evaluation memory: (i) we isolate the contribution of each memory type by comparing no memory, static-only, dynamic-only, and the combined configuration; and (ii) we change the static memory retrieval number N and dynamic query number K. Results are in Table 2 for global rubrics and Table 6 for per-instance rubrics. Memory configuration. Both memory types show gains over the no-memory baseline, and their combination yields the best results across all benchmarks (+1.9 on GPQA-D and +1.8 on HealthBench). Notably, the two types have complementary strengths. For example, in scientific domain, static memory slightly outperforms dynamic memory (38.9 vs. 38.6), suggesting that historical evaluation recent steps is valuable for scientific reasoning, possibly because rubric drift and recurring errors tend to appear within a short period of steps. For HealthBench, the pattern (33.3 vs. 33.1) indicates that targeted semantic retrieval is more beneficial due to suboptimal behaviors recur across distant steps, unlike the science domain. In all cases, the combined configuration captures both temporal and semantic signals and achieves the strongest overall performance, which we adopt as the default, and the data of per-instance rubrics shows similar trend as shown in Table 6. Retrieval configuration. For both retrieval mechanisms, we observe performance gain up to a moderate budget followed by diminishing returns. Increasing the static retrieval 7

Preprint. Under review.

Memory

Setting

Science

Medicine

Instruction Following

Creative Writing

GPQA-D HealthBench IFEval InfoBench IFBench WritingBench CW-v3 No memory

–

38.0

31.8

79.7

83.5

34.0

54.8

38.8

Static only

N=2 N=4 N=6 N=8

38.2 38.9 38.8 38.9

32.4 33.1 32.9 33.2

79.9 80.2 80.1 80.2

83.8 84.2 84.1 84.3

34.3 34.7 34.6 34.8

55.4 56.1 56.0 56.0

38.9 39.1 39.1 39.2

Dynamic only

K=5 K = 10 K = 15

38.4 38.6 38.6

32.4 33.3 32.9

80.0 80.1 80.0

84.1 84.4 84.2

34.2 34.8 34.6

55.5 55.7 55.9

39.0 39.2 39.1

39.9

33.6

80.6

84.8

35.2

57.9

39.5

Static + Dynamic N = 4, K = 10

Table 2: Ablation on memory configuration and retrieval hyperparameters under the global rubrics setting. Rows are ordered as no memory, static-only, dynamic-only, and static+dynamic. Best results within the static-only and dynamic-only blocks are in bold. Component

Avg. time (s)

% of pipeline

Tokens / step

Individual analysis Batch summarization Query generation Memory retrieval Rubric update

426.4 192.3 2.7 23.7 30.6

63.1% 28.5% 0.4% 3.5% 4.5%

2.32 M 246 k 14.1 k 0 110.9 k

Total pipeline

675.7

100.0%

2.691 M

Table 3: Per-component time consumption and token consumption of the AMARIS pipeline, averaged over 400 training steps. amount from N =2 to N =4 provides the main improvement (e.g., +0.7 on both GPQA-D and HealthBench), suggesting that four preceding summaries are sufficient to capture the recent training trajectory and prevent oscillatory rubric changes. Similarly, increasing the dynamic retrieval amount by increasing the query from K =5 to K =10 produces consistent improvements across all benchmarks, but K =15 provides no further benefit (e.g., HealthBench drops from 33.3 to 32.9). We attribute this to additional queries retrieving marginal or noisy documents. We therefore select N =4 and K =10 as defaults. For per-instance rubric settings, the trend agrees with the global rubrics setting as shown in Table 6. 4.3

Latency Analysis

A practical concern for any auxiliary reward pipeline is the overhead in terms of time consumption added to the RL training. Because AMARIS introduces multiple LLM calls per step, we profile each pipeline component and compare asynchronous execution in Section 2.4 versus traditional synchronous execution, where the scoring always waits for the rubric update within the same step so that the rubric stays on-policy. Component-level profiling. Table 3 shows the average time and token consumption of each AMARIS component measured over one complete training using the default setting of AMARIS under global rubrics. The individual rollout analysis dominates both time (63.1%) and tokens (2.32 M per step), as it processes every rollout. The batch summarization is the second most expensive stage (28.5%), while the other stages together account for less than 9% of the total pipeline time. Asynchronous vs. Synchronous execution. Table 4 shows the quality-efficiency tradeoff. Synchronous execution yields mostly a 0.2-point advantage on any single benchmark (e.g., 40.1 vs. 39.9 on GPQA-D). This marginal gain comes at the cost of nearly doubling the total training time. Asynchronous execution therefore provides a substantially more favorable trade-off and is adopted as the default pipeline mode. We also compare the total time consumption using both asynchronous and synchronous AMARIS with the normal 8

Preprint. Under review.

Pipeline mode Sync Async

Science

Medicine

GPQA-D

HealthBench

IFEval

Instruction Following InfoBench

IFBench

WritingBench

Creative Writing CW-v3

40.1 39.9

33.7 33.6

80.7 80.6

84.6 84.8

35.4 35.2

57.5 57.9

39.6 39.5

Time (h)

Overhead

∼146 ∼80

∼92% ∼5%

Table 4: Synchronous vs. asynchronous pipeline execution. Async mode nearly eliminates overhead in terms of overall time while incurring only marginal quality differences (≤0.2 across most benchmarks). rubric-based RL training with static rubrics only in terms of the Overhead column in Table 4. The results show that by using asynchronous execution, the latency brought by the extra LLM usages is reduced to minimal. We also profile the latency for per-instance rubric settings and found that the extra latency is minimal. This is because the per-instance setting naturally limits update opportunity but provides abundant time for analysis and rubric update, especially for summarization and updating that can be further paralleled by using external APIs.

5

Related Work

Rubric-Based Reward Modeling & Dynamic Adaptation The alignment of LLMs has evolved from relying on scalar reward models trained on pairwise preferences (Ouyang et al., 2022; Bai et al., 2022) to utilizing more interpretable, generative feedback. Frameworks utilizing Rubric-as-Reward paradigm demonstrate that LLMs can synergize reasoning and acting by decomposing complex quality judgments into structured, independently scored rubrics (Kim et al., 2024; Gunjal et al., 2025; Viswanathan et al., 2025). This extends reinforcement learning beyond strictly verifiable domains. To address the vulnerability of static rubrics, which often degrade or suffer from reward hacking as the policy adapts (Gao et al., 2023), recent work has turned to dynamic rubric generation. Methods like CARMO (Gupta et al., 2025) and OnlineRubrics (Rezaei et al., 2025) dynamically elicit new evaluation dimensions on the fly. More sophisticated frameworks, such as Rubric-ARM (Xu et al., 2026) and DR Tulu (Shao et al., 2025), close the loop by alternating policy optimization with bounded rubric refinement. While these advances improve flexibility, they share a critical limitation that by discarding the diagnostic evidence, such as rollout-level analyses or longitudinal score trends, after immediate use, these systems lack the historical context needed to prevent recurring suboptimal behaviors and making precise adjustments. Memory Systems and Longitudinal Optimization To address the lack of persistent context, a parallel line of research focuses on improving the long-term learning capabilities of agents via external memory systems. Architectures like A-Mem (Xu et al., 2025), Memory-R1 (Yan et al., 2026), and Meta-Policy Reflection (Wu et al., 2025a) introduce self-organized read-write-consolidation cycles that allow models to transfer learned experiences across tasks (Hu et al., 2026). Concurrently, within reward modeling, efforts to mitigate reward hacking and shape curricula have employed causal rubric structures (Srivastava et al., 2025), information-theoretic regularization (Miao et al., 2024), and staged AI feedback (Li et al., 2025). AMARIS combines these directions by using persistent evaluation memory to ground rubric updates in accumulated rollout diagnostics, summaries, and prior updates.

6

Conclusion

We present AMARIS, a memory-augmented rubric improvement system for rubric-based reinforcement learning. AMARIS stores rollout analyses, step-level summaries, and rubric update records in a persistent evaluation memory, and uses both static and dynamic retrieval to ground rubric changes in training history. Across science, medicine, instruction following, and creative writing domains, AMARIS improves over strong rubric-based baselines under both global and per-instance rubric settings. Ablation studies show that both static and dynamic retrieval contribute to the gains, while asynchronous execution keeps the end-toend training overhead low. These results suggest that persistent evaluation memory is a 9

Preprint. Under review.

practical way to make adaptive rubric-based reward shaping more effective over various types of tasks.

Ethics Statement The authors of this paper have read and adhered to the COLM Code of Ethics. This work focuses on improving the effectiveness of rubric-based reinforcement learning for large language models by grounding rubric adaptation in persistent evaluation memory, and we have considered the ethical implications of our methodology. AMARIS itself is a meta-level training framework that refines evaluation rubrics and does not introduce new model capabilities that pose direct safety risks beyond those of the underlying policy model. Nevertheless, we acknowledge that any system that shapes the reward signal of an RL-trained language model could, in principle, amplify undesirable behaviors if the rubrics or the LLM judges that produce analyses encode biases present in their training data or evaluation criteria. To mitigate this, AMARIS explicitly monitors for reward hacking and suboptimal behaviors as a core component of its individual rollout analysis, and the persistent evaluation memory provides an record, enhancing the transparency. All training data used in our experiments are drawn from publicly available, previously published datasets. No human subjects were involved in our research, and no personally identifiable information was collected or used. By enabling more efficient and targeted rubric updates through asynchronous execution with only approximately 5% time overhead, AMARIS also contributes to more sustainable use of computational resources during RL training.

Reproducibility Statement We are committed to ensuring the reproducibility of our work. Section 3 provides a comprehensive overview of our experimental framework, including the specific training datasets and evaluation benchmarks, baselines, and metrics used for comparison. Section 3.3 shows our training procedure with the base model, RL algorithm, hardware configuration, LLM choices for each pipeline component, memory implementation, and all key hyperparameters including the default memory retrieval configuration, training duration, batch size, and rubric staleness constraints. The core components of AMARIS are described in Section 2. To further facilitate replication, the Appendix contains the complete table of mathematical symbols used throughout the paper, full per-instance rubric results mirroring all main-text ablations (Appendix B), analyses of underlying LLM influence (Appendix C) and update scheduling (Appendix D), a detailed rubric evolvement analysis across 1,200 training steps (Appendix E), qualitative case studies illustrating each rubric operation type (Appendix F), and the exact prompt templates for all six pipeline stages: scoring (Figures 2–3), individual rollout analysis (Figures 4–7), batch-level summarization (Figures 8–11), meta-batch-level summarization (Figures 12–15), memory query generation (Figures 16–18), and rubric update (Figures 19–22).

References Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin QuiñoneroCandela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URL https://arxiv.org/abs/2505.08775. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac HatfieldDodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https://arxiv.org/abs/2204. 05862. 10

Preprint. Under review.

chroma-core. Chroma: Open-source data infrastructure for ai. https://github.com/ chroma-core/chroma, 2026. GitHub repository, accessed 2026-03-31. Kilian Freitag, Kristian Ceder, Rita Laezza, Knut Åkesson, and Morteza Haghir Chehreghani. Curriculum reinforcement learning for complex reward functions, 2025. URL https: //arxiv.org/abs/2410.16790. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025. URL https://arxiv.org/abs/2507.17746. Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Rahul Madhavan, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, and Saravan Rajmohan. CARMO: Dynamic criteria generation for context aware reward modelling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 2202–2261, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.114. URL https://aclanthology.org/2025.findings-acl.114/. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, Senjie Jin, Jiejun Tan, Yanbin Yin, Jiongnan Liu, Zeyu Zhang, Zhongxiang Sun, Yutao Zhu, Hao Sun, Boci Peng, Zhenrong Cheng, Xuanbo Fan, Jiaxin Guo, Xinlei Yu, Zhenhong Zhou, Zewen Hu, Jiahao Huo, Junhao Wang, Yuwei Niu, Yu Wang, Zhenfei Yin, Xiaobin Hu, Yue Liao, Qiankun Li, Kun Wang, Wangchunshu Zhou, Yixin Liu, Dawei Cheng, Qi Zhang, Tao Gui, Shirui Pan, Yan Zhang, Philip Torr, Zhicheng Dou, Ji-Rong Wen, Xuanjing Huang, Yu-Gang Jiang, and Shuicheng Yan. Memory in the age of ai agents, 2026. URL https://arxiv.org/abs/2512.13564. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models, 2024. URL https://arxiv.org/ abs/2310.08491. Mengdi Li, Jiaye Lin, Xufeng Zhao, Wenhao Lu, Peilin Zhao, Stefan Wermter, and Di Wang. Curriculum-rlaif: Curriculum alignment with reinforcement learning from ai feedback, 2025. URL https://arxiv.org/abs/2505.20075. Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, Yang Zhou, Jingwen Yang, Shunyu Liu, Kaike Zhang, and Wei Chen. Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation, 2026. URL https://arxiv.org/ abs/2601.08430. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment, 2026. URL https://arxiv.org/abs/2510.07743. Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: mitigating reward hacking in rlhf via information-theoretic reward modeling. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, 11

Preprint. Under review.

Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, 12

Preprint. Under review.

Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Samuel J Paech. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/ creative-writing-bench, 2025. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025. URL https://arxiv.org/abs/2507.02833. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. InFoBench: Evaluating instruction following ability in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13025–13048, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.772. URL https://aclanthology.org/2024. findings-acl.772/. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang 13

Preprint. Under review.

Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level googleproof q&a benchmark. In First Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=Ti67584b98. MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. Online rubrics elicitation from pairwise comparisons, 2025. URL https://arxiv.org/abs/2510.07284. Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, and Pang Wei Koh. Dr tulu: Reinforcement learning with evolving rubrics for deep research, 2025. URL https://arxiv.org/abs/ 2511.19399. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300. Leheng Sheng, Wenchang Ma, Ruixin Hong, Xiang Wang, An Zhang, and Tat-Seng Chua. Reinforcing chain-of-thought reasoning with self-evolving rubrics, 2026. URL https: //arxiv.org/abs/2602.10885. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex BakerWhitcomb, Alex Beutel, Alex Karpenko, Alex Makelov, Alex Neitz, Alex Wei, Alexandra Barr, Alexandre Kirchmeyer, Alexey Ivanov, Alexi Christakis, Alistair Gillespie, Allison Tam, Ally Bennett, Alvin Wan, Alyssa Huang, Amy McDonald Sandjideh, Amy Yang, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrei Gheorghe, Andres Garcia Garcia, Andrew Braunstein, Andrew Liu, Andrew Schmidt, Andrey Mereskin, Andrey Mishchenko, Andy Applebaum, Andy Rogerson, Ann Rajan, Annie Wei, Anoop Kotha, Anubha Srivastava, Anushree Agrawal, Arun Vijayvergiya, Ashley Tyra, Ashvin Nair, Avi Nayak, Ben Eggers, Bessie Ji, Beth Hoover, Bill Chen, Blair Chen, Boaz Barak, Borys Minaiev, Botao Hao, Bowen Baker, Brad Lightcap, Brandon McKinzie, Brandon Wang, Brendan Quinn, Brian Fioca, Brian Hsu, Brian Yang, Brian Yu, Brian Zhang, Brittany Brenner, Callie Riggins Zetino, Cameron Raymond, Camillo Lugaresi, Carolina Paz, Cary Hudson, Cedric Whitney, Chak Li, Charles Chen, Charlotte Cole, Chelsea Voss, Chen Ding, Chen Shen, Chengdu Huang, Chris Colby, Chris Hallacy, Chris Koch, Chris Lu, Christina Kaplan, Christina Kim, CJ Minott-Henriques, Cliff Frey, Cody Yu, Coley Czarnecki, Colin Reid, Colin Wei, Cory Decareaux, Cristina Scheau, Cyril Zhang, Cyrus Forbes, Da Tang, Dakota Goldberg, Dan Roberts, Dana Palmie, Daniel Kappler, Daniel Levine, Daniel Wright, Dave Leo, David Lin, David Robinson, Declan Grabb, Derek Chen, Derek Lim, Derek Salama, Dibya Bhattacharjee, Dimitris Tsipras, Dinghua Li, Dingli Yu, DJ Strouse, Drew Williams, Dylan Hunn, Ed Bayes, Edwin Arbus, Ekin Akyurek, Elaine Ya Le, Elana Widmann, Eli Yani, Elizabeth Proehl, Enis Sert, Enoch Cheung, Eri Schwartz, Eric Han, Eric Jiang, Eric Mitchell, Eric Sigler, Eric Wallace, Erik Ritter, Erin Kavanaugh, Evan Mays, Evgenii Nikishin, Fangyuan Li, Felipe Petroski Such, Filipe de Avila Belbute Peres, Filippo Raso, Florent Bekerman, Foivos Tsimpourlas, Fotis Chantzis, Francis Song, Francis Zhang, Gaby Raila, Garrett McGrath, Gary Briggs, Gary Yang, Giambattista Parascandolo, Gildas Chabot, Grace Kim, Grace Zhao, Gregory Valiant, Guillaume Leclerc, Hadi Salman, Hanson Wang, Hao Sheng, Haoming Jiang, Haoyu Wang, Haozhun Jin, Harshit Sikchi, Heather Schmidt, Henry Aspegren, Honglin Chen, Huida Qiu, Hunter Lightman, Ian Covert, Ian Kivlichan, Ian Silber, Ian Sohl, Ibrahim Hammoud, Ignasi Clavera, Ikai Lan, Ilge Akkaya, Ilya Kostrikov, Irina Kofman, Isak Etinger, Ishaan Singal, Jackie Hehir, Jacob Huh, Jacqueline Pan, Jake Wilczynski, Jakub Pachocki, James Lee, James Quinn, Jamie 14

Preprint. Under review.

Kiros, Janvi Kalra, Jasmyn Samaroo, Jason Wang, Jason Wolfe, Jay Chen, Jay Wang, Jean Harb, Jeffrey Han, Jeffrey Wang, Jennifer Zhao, Jeremy Chen, Jerene Yang, Jerry Tworek, Jesse Chand, Jessica Landon, Jessica Liang, Ji Lin, Jiancheng Liu, Jianfeng Wang, Jie Tang, Jihan Yin, Joanne Jang, Joel Morris, Joey Flynn, Johannes Ferstad, Johannes Heidecke, John Fishbein, John Hallman, Jonah Grant, Jonathan Chien, Jonathan Gordon, Jongsoo Park, Jordan Liss, Jos Kraaijeveld, Joseph Guay, Joseph Mo, Josh Lawson, Josh McGrath, Joshua Vendrow, Joy Jiao, Julian Lee, Julie Steele, Julie Wang, Junhua Mao, Kai Chen, Kai Hayashi, Kai Xiao, Kamyar Salahi, Kan Wu, Karan Sekhri, Karan Sharma, Karan Singhal, Karen Li, Kenny Nguyen, Keren Gu-Lemberg, Kevin King, Kevin Liu, Kevin Stone, Kevin Yu, Kristen Ying, Kristian Georgiev, Kristie Lim, Kushal Tirumala, Kyle Miller, Lama Ahmad, Larry Lv, Laura Clare, Laurance Fauconnet, Lauren Itow, Lauren Yang, Laurentia Romaniuk, Leah Anise, Lee Byron, Leher Pathak, Leon Maksin, Leyan Lo, Leyton Ho, Li Jing, Liang Wu, Liang Xiong, Lien Mamitsuka, Lin Yang, Lindsay McCallum, Lindsey Held, Liz Bourgeois, Logan Engstrom, Lorenz Kuhn, Louis Feuvrier, Lu Zhang, Lucas Switzer, Lukas Kondraciuk, Lukasz Kaiser, Manas Joglekar, Mandeep Singh, Mandip Shah, Manuka Stratta, Marcus Williams, Mark Chen, Mark Sun, Marselus Cayton, Martin Li, Marvin Zhang, Marwan Aljubeh, Matt Nichols, Matthew Haines, Max Schwarzer, Mayank Gupta, Meghan Shah, Melody Huang, Meng Dong, Mengqing Wang, Mia Glaese, Micah Carroll, Michael Lampe, Michael Malek, Michael Sharman, Michael Zhang, Michele Wang, Michelle Pokrass, Mihai Florian, Mikhail Pavlov, Miles Wang, Ming Chen, Mingxuan Wang, Minnia Feng, Mo Bavarian, Molly Lin, Moose Abdool, Mostafa Rohaninejad, Nacho Soto, Natalie Staudacher, Natan LaFontaine, Nathan Marwell, Nelson Liu, Nick Preston, Nick Turley, Nicklas Ansman, Nicole Blades, Nikil Pancha, Nikita Mikhaylin, Niko Felix, Nikunj Handa, Nishant Rai, Nitish Keskar, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Oona Gleeson, Pamela Mishkin, Patryk Lesiewicz, Paul Baltescu, Pavel Belov, Peter Zhokhov, Philip Pronin, Phillip Guo, Phoebe Thacker, Qi Liu, Qiming Yuan, Qinghua Liu, Rachel Dias, Rachel Puckett, Rahul Arora, Ravi Teja Mullapudi, Raz Gaon, Reah Miyara, Rennie Song, Rishabh Aggarwal, RJ Marsan, Robel Yemiru, Robert Xiong, Rohan Kshirsagar, Rohan Nuttall, Roman Tsiupa, Ronen Eldan, Rose Wang, Roshan James, Roy Ziv, Rui Shu, Ruslan Nigmatullin, Saachi Jain, Saam Talaie, Sam Altman, Sam Arnesen, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Sarah Yoo, Savannah Heon, Scott Ethersmith, Sean Grove, Sean Taylor, Sebastien Bubeck, Sever Banesiu, Shaokyi Amdo, Shengjia Zhao, Sherwin Wu, Shibani Santurkar, Shiyu Zhao, Shraman Ray Chaudhuri, Shreyas Krishnaswamy, Shuaiqi, Xia, Shuyang Cheng, Shyamal Anadkat, Simón Posada Fishman, Simon Tobin, Siyuan Fu, Somay Jain, Song Mei, Sonya Egoian, Spencer Kim, Spug Golden, SQ Mah, Steph Lin, Stephen Imm, Steve Sharpe, Steve Yadlowsky, Sulman Choudhry, Sungwon Eum, Suvansh Sanjeev, Tabarak Khan, Tal Stramer, Tao Wang, Tao Xin, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Degry, Thomas Shadwell, Tianfu Fu, Tianshi Gao, Timur Garipov, Tina Sriskandarajah, Toki Sherbakov, Tomer Kaftan, Tomo Hiratsuka, Tongzhou Wang, Tony Song, Tony Zhao, Troy Peterson, Val Kharitonov, Victoria Chernova, Vineet Kosaraju, Vishal Kuo, Vitchyr Pong, Vivek Verma, Vlad Petrov, Wanning Jiang, Weixing Zhang, Wenda Zhou, Wenlei Xie, Wenting Zhan, Wes McCabe, Will DePue, Will Ellsworth, Wulfie Bain, Wyatt Thompson, Xiangning Chen, Xiangyu Qi, Xin Xiang, Xinwei Shi, Yann Dubois, Yaodong Yu, Yara Khakbaz, Yifan Wu, Yilei Qian, Yin Tat Lee, Yinbo Chen, Yizhen Zhang, Yizhong Xiong, Yonglong Tian, Young Cha, Yu Bai, Yu Yang, Yuan Yuan, Yuanzhi Li, Yufeng Zhang, Yuguang Yang, Yujia Jin, Yun Jiang, Yunyun Wang, Yushi Wang, Yutian Liu, Zach Stubenvoll, Zehao Dou, Zheng Wu, and Zhigang Wang. Openai gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Pragya Srivastava, Harman Singh, Rahul Madhavan, Gandharv Patil, Sravanti Addepalli, Arun Suggala, Rengarajan Aravamudhan, Soumya Sharma, Anirban Laha, Aravindan Raghuveer, Karthikeyan Shanmugam, and Doina Precup. Robust reward modeling via causal rubrics, 2025. URL https://arxiv.org/abs/2506.16507. 15

Preprint. Under review.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288. Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models, 2025. URL https://arxiv.org/abs/2507.18624. Chunlong Wu, Ye Luo, Zhibo Qu, and Min Wang. Meta-policy reflexion: Reusable reflective memory and rule admissibility for resource-efficient llm agent, 2025a. URL https:// arxiv.org/abs/2509.03990. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. Writingbench: A comprehensive benchmark for generative writing, 2025b. URL https://arxiv.org/abs/2503.05244. Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training, 2026. URL https://arxiv.org/abs/2602.01511. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents, 2025. URL https://arxiv.org/abs/2502.12110. Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z. Pan, Hinrich Schütze, Volker Tresp, and Yunpu Ma. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning, 2026. URL https://arxiv.org/abs/2508.19828. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Zihuiwen Ye, Fraser David Greenlee, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 4506–4520, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.254. URL https://aclanthology.org/2025.findings-naacl.254/. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. 16

Preprint. Under review.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911. Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Kongcheng Zhang, Jiale Zhao, Jingwen Yang, Yihe Zhou, Jianwei Lv, Tongya Zheng, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general llm reasoning, 2026. URL https://arxiv.org/abs/2508.16949.

Appendix A

Symbols

17

Preprint. Under review.

Symbol πϕ x y xj yj t Gt gj wj n g,t n g,t Gt = {( g j , w j )} j=1 Sθ rt ( x j , y j ) rt ( x j , y j ) = Sθ ( x j , y j , Gt ) z Aψ a a = Aψ ( x, y, Gt , z) aj n a,t n { a j } j=a,t1 Bψ bt n bt = Bψ ({ a j } j=a,t1 ) cj nc,t n a,c n {c j } j=c,t1 Cψ n Cψ ({ Bψ (c j )} j=c,t1 ) Mt ut It It,s It,d It = { It,s , It,d } N It,s = {b j ∈ Mt−1 : t − N ≤ j ≤ t − 1} Qψ ql nq,t n {ql }l =q,t1 = Qψ (bt ) D nq,t It,d = { R(ql , Mt , D )}l =1 Uψ Gt+1

Description Policy model. Model input. Model output (rollout), with y ∼ πϕ (· | x ). Input for example j. Output for example j. RL training step. Active rubric set at step t. Natural-language definition of rubric j. Scalar weight of rubric j. Number of active rubrics at step t. Formal definition of the active rubric set. LLM-based scoring function. Scalar reward for example j at step t. Reward computed under the active rubric set. Supplemental context for analysis. LLM for individual rollout analysis. Structured analysis for a single rollout. Individual analysis function. Structured analysis for rollout j. Number of analyzed rollouts at step t. Set of rollout analyses at step t. LLM for batch summarization. Step-level batch summary at step t. Step-level summarization function. Chunk of analyses used in meta-summarization. Number of chunks at step t. Number of analyses in chunk c. Partition of analyses into chunks. LLM for meta-summarization. Meta-summarization over chunk summaries. Persistent evaluation memory after step t. Rubric improvement record produced at step t. Retrieved historical context for rubric updating. Static retrieval results. Dynamic retrieval results. Combined retrieval context. Static retrieval window size. Static retrieval from recent summaries. LLM for dynamic query generation. Dynamic retrieval query l. Number of dynamic queries at step t. Queries generated from the current summary. Maximum documents returned per query. Dynamic retrieval from memory. Rubric updater. Updated rubric set after step t.

Table 5: Description of symbols and notations used.

18

Preprint. Under review.

B

Per-Instance Rubric Results

The ablation studies in the main paper (Sections 4.2–4.3) use the global rubric setting to isolate the effect of each design choice. This section reports the corresponding results under per-instance rubrics, where each unique input maintains its own rubric set and evaluation memory. As shown below, the relative ordering of configurations is consistent with the global setting, confirming that the design conclusions drawn in the main paper generalize across rubric strategies. B.1

Memory Configuration (Per-Instance)

Memory

Science

Setting

Medicine

Instruction Following

Creative Writing

GPQA-D HealthBench IFEval InfoBench IFBench WritingBench CW-v3 No memory

–

38.4

32.1

80.0

83.8

34.5

55.2

39.0

Static only

N=2 N=4 N=6 N=8

38.7 38.6 38.3 38.5

32.8 32.5 32.3 32.6

80.2 80.1 79.5 79.6

84.1 84.0 84.5 84.7

35.0 35.4 35.3 35.4

55.0 55.7 55.6 55.7

39.3 38.6 39.6 39.5

Dynamic only

K=5 K = 10 K = 15

39.0 39.9 39.2

32.8 34.0 33.4

80.3 80.6 80.4

84.4 84.8 84.6

35.1 35.7 35.3

56.1 56.8 56.4

39.4 40.1 39.6

40.4

34.0

81.0

85.2

36.0

57.3

40.1

Static + Dynamic N = 4, K = 10

Table 6: Ablation on memory configuration and retrieval hyperparameters under perinstance rubrics. Rows are ordered as no memory, static-only, dynamic-only, and static+dynamic. Best results within the static-only and dynamic-only blocks are in bold. B.2

LLM Choice (Per-Instance) Science

Medicine

GPQA-D

HealthBench

IFEval

InfoBench

IFBench

WritingBench

CW-v3

Varying the scoring / analysis (summarization / update fixed to GPT-5.1) Qwen3-30B-A3B-2507-Thinking GPT-5.1 40.7 GPT-OSS-120B GPT-5.1 40.4 Qwen3-235B-A22B-2507-Thinking GPT-5.1 39.9

33.8 34.0 34.3

80.2 81.0 81.3

83.5 85.2 85.5

36.6 36.0 35.1

55.8 57.3 58.3

39.3 40.1 40.4

Varying the summarization / update (scoring / analysis fixed to GPT-OSS-120B) GPT-OSS-120B GPT-4o 39.9 GPT-OSS-120B GPT-5-mini 40.2 GPT-OSS-120B GPT-5.1 40.4

33.4 33.8 34.0

79.4 80.8 81.0

84.9 84.2 85.2

34.9 35.8 36.0

58.6 56.2 57.3

40.9 39.4 40.1

Scoring / Analysis

Summarization / Update

Instruction Following

Creative Writing

Table 7: LLM choice for pipeline components under per-instance rubrics. Compare with Table 9 (global rubrics). Best results per group are in bold. B.3

Synchronous vs. Asynchronous Pipeline (Per-Instance)

Pipeline mode Sync Async

Science

Medicine

GPQA-D

HealthBench

IFEval

Instruction Following InfoBench

IFBench

WritingBench

Creative Writing CW-v3

40.6 40.4

34.1 34.0

81.1 81.0

85.3 85.2

36.2 36.0

57.1 57.3

40.2 40.1

Time (h)

Overhead

∼127 ∼78

∼67% ∼3%

Table 8: Synchronous vs. asynchronous pipeline execution under per-instance rubrics. Compared with Table 4 (global rubrics).

19

Preprint. Under review.

Science

Medicine

GPQA-D

HealthBench

IFEval

InfoBench

IFBench

WritingBench

CW-v3

Varying the scoring and analysis (summarization / update fixed to GPT-5.1) Qwen3-30B-A3B-2507-Thinking GPT-5.1 39.6 GPT-OSS-120B GPT-5.1 39.9 Qwen3-235B-A22B-2507-Thinking GPT-5.1 40.2

33.9 33.6 33.4

79.8 80.6 80.9

83.0 84.8 85.1

35.2 35.2 35.8

55.1 57.9 57.8

39.8 39.5 39.5

Varying the summarizing / updating (scoring / analyzing fixed to GPT-OSS-120B) GPT-OSS-120B GPT-4o 39.4 GPT-OSS-120B GPT-5-mini 39.7 GPT-OSS-120B GPT-5.1 39.9

33.0 33.4 33.6

79.0 80.4 80.6

83.8 84.5 84.8

34.2 35.0 35.2

55.7 56.9 57.9

39.0 39.3 39.5

Scoring / Analyzing

Summarization / Update

Instruction Following

Creative Writing

Table 9: LLM choice for pipeline components. Best results per group are in bold.

C

Influence Of Underlying LLMs

We split all LLM usage into two tiers: (i) scoring and analysis that processes every rollout and therefore dominates token consumption, and (ii) summarization and update that operates less than (i) but demands stronger reasoning for rubric improvement. We ablate each tier independently while holding the other at its default, as shown in Table 9. On the scoring/analysis side, performance scales with model capacity. Upgrading from Qwen3-30B-A3B-2507-Thinking (Yang et al., 2025) to Qwen3-235B-A22B-2507-Thinking (Yang et al., 2025) improves GPQA-D by +0.6 and InfoBench by +2.1, indicating that higherquality individual analyses produce more actionable batch summaries and, consequently, better rubric updates. On the summarization/update side, the performance between GPT-4o (OpenAI et al., 2024) and GPT-5.1 is similarly, showing that the rubric update stage, which must make precise text and weight editing decisions, benefits from stronger reasoning ability of the model. Notably, GPT-5-mini (Singh et al., 2025) closes much of this gap to GPT-5.1 at lower cost and surpass GPT-4o at some benchmarks. This shows that the reasoning capability is crucial to the quality of generated rubrics. We select GPT-OSS-120B paired with GPT-5.1 as the default setting as it offers a balance between the high inference cost from scoring/analysis stage and high demand of model intelligence from summarization/update stage.

20

Preprint. Under review.

Update interval Every step Every 2 steps Every 4 steps Every 6 steps Every 8 steps

Science

Medicine

Instruction Following

Creative Writing

GPQA-D

HealthBench

IFEval

InfoBench

IFBench

WritingBench

CW-v3

39.9 40.1 39.8 39.4 39.7

33.6 33.7 33.5 33.2 33.3

80.6 80.7 80.5 80.2 80.8

84.8 84.9 84.7 84.4 85.0

35.2 35.4 35.0 34.7 35.1

57.9 57.0 57.5 56.6 57.2

39.5 39.7 39.4 39.1 39.5

Table 10: Effect of rubric update frequency. Best results per column are in bold.

D

Influence Of Update Scheduling

By default, under global rubrics setting, the rubric update stage is triggered at every training step. We vary the update interval from every 1 to every 8 steps to examine how update frequency affects downstream performance, as shown in Table 10. Updating every 2 steps yields the best or near-best results on most of the benchmarks, marginally outperforming every-step updates (e.g., +0.2 on GPQA-D, +0.1 on HealthBench). We attribute this to the slightly larger evidence window that aggregating two consecutive batches before committing a rubric change produces more robust diagnostics than a single batch while still reacting quickly to emerging issues. Performance degrades at intervals of 4 and 6 steps, where the difference between the rubrics and the model becomes large enough to miss behavioral shifts. Interestingly, the 8-step interval setting partially recovers the performance. A possible explanation is that the larger amount of historical analyses at longer intervals enables the updater to make reliable rubric changes that are robust to longterm training, which particularly benefits the diverse instruction types in the OpenRubrics training set. However, this may result in slower reaction to suboptimal behaviors, as reflected by the lower GPQA-D and HealthBench scores. Overall, the differences across all settings remain modest (≤0.7 on any single benchmark), indicating that AMARIS is robust to the various choice of update frequency. We adopt every-step updates as the default for global rubrics setting due to its simplicity and consistently competitive performance across all domains.

21

Preprint. Under review.

Step range

Defensive

Curriculum

Maintenance

Dominant

1–80 81–160 161–240 241–320 321–400

126 (52.50%) 92 (38.33%) 57 (23.75%) 38 (15.83%) 21 (8.75%)

62 (25.83%) 101 (42.08%) 127 (52.92%) 84 (35.00%) 39 (16.25%)

52 (21.67%) 47 (19.59%) 56 (23.33%) 118 (49.17%) 180 (75.00%)

Defensive Curriculum Curriculum Maintenance Maintenance

Total

334 (27.83%)

413 (34.42%)

453 (37.75%)

—

Table 11: Distribution of dominant update modes across five equal training phases. The system transitions from defensive in the early phase, through curriculum advancement in the middle, to maintenance in the late phase.

E

Rubric Evolvement Analysis

To understand how AMARIS improves the rubrics over the RL training, we analyze 1,200 step-level summaries produced across the three completed training processes under globalrubric settings (science, medicine, and instruction following). Following the three update strategies defined in Section 2.3, we assign each step one dominant update mode (Defensive, Curriculum Advancement, or Maintenance) and partition training into five phases with equal number of steps for chronological comparison. We then taxonomize the dominant pattern within each mode for further analysis. Three-stage progression. Table 11 reports the distribution of update modes across training phases. AMARIS shows a clear three-stage pattern. In the first 80 steps, Defensive updates dominate, indicating that the system initially deals with suboptimal behaviors. In mid-training (steps 81–240), Curriculum Advancement becomes the single largest mode, showing that once the suboptimal behaviors are controlled, AMARIS shifts toward raising evaluation standards. In the final phase (steps 241–400), Maintenance dominates, indicating that the rubric set becomes stable. Suboptimal behavior taxonomy. To understand the major suboptimal behaviors, we analyze each of the 334 defensive steps’ batch summarizations by choosing the dominant suboptimal behavior category reported in the summarization as a representative and categorize them manually into six types of major suboptimal behaviors. Table 12 reports the temporal evolution of each type. We define the six types as follows: length gaming increases response length to exploit length-correlated reward signals; superficial format compliance complies to surface-level formatting such as bullet points, numbered lists, or markdown headers without improving substantive content; safety over-refusal unnecessarily declines on user instructions due to overly cautious; claims without evidence presents assertions or conclusions without supporting reasoning or citations; sycophancy excessively agrees with the user rather than providing accurate responses; and keyword stuffing inserts task-relevant terminology or phrases into the output without meaningful integration into the reasoning. Three classic suboptimal behaviors: length gaming and verbosity inflation (23.35%), superficial format compliance (21.26%), and safety over-refusal(18.86%), together account for the majority of defensive steps (63.47%), showing that most of the early rubric updates have been used to resolve reward hacking against visible rubric cues. All six suboptimal behaviors decreases over training in different rates. Length gaming and sycophancy decreases faster, indicating that once rubric patches penalize these behaviors, the policy quickly learns to avoid them. On the contrary, Safety over-refusal and Keyword stuffing are the most persistent suboptimal behavior for the whole RL training. Keyword stuffing also does not decrease monotonically like the other types, suggesting that it can re-emerge in different forms, which highlights the value of AMARIS’s memory-based tracking of recurring behaviors. Curriculum advancement taxonomy. Similar to suboptimal behaviors for Defensive steps, we use the same method to analyze the main factor for Curriculum Advancement steps. 22

Preprint. Under review.

Primary defensive types Length gaming Superficial format compliance Safety over-refusal Claims without evidence Sycophancy Keyword stuffing

1–80

81–160

161–240

241–320

321–400

Total

%

31 25 20 21 17 12

22 24 18 12 10 6

14 13 12 9 5 4

7 6 8 7 4 6

4 3 5 3 1 5

78 71 63 52 37 33

23.35% 21.26% 18.86% 15.57% 11.08% 9.88%

Table 12: Taxonomy of dominant suboptimal-behavior vectors across the 334 defensive steps. The top three types account for 63.47% of all samples and monotonically decreases as the training progresses, though safety over-refusal remains the most persistent type. Primary curriculum factors

1–80

81–160

161–240

241–320

321–400

Total

%

Reasoning depth Conciseness Justification Trade-off handling Multi-constraint prioritization Output style

12 8 14 9 11 8

27 18 21 12 14 9

40 29 24 15 11 8

29 24 9 10 6 6

13 13 6 3 1 3

121 92 74 49 43 34

29.30% 22.28% 17.92% 11.86% 10.41% 8.23%

Table 13: Taxonomy of dominant curriculum-advancement targets across the 413 samples. Growth targets progress from evidence grounding (early) to depth, conciseness, and communication quality (mid and late training). Table 13 reports the dominant factors for each of the 413 samples flagged with Curriculum Advancement. The six factors are: reasoning depth, requiring multi-step or more sophisticated chains of reasoning; conciseness, favoring concise but information-dense responses over verbose and vague ones; justification, grounding claims in explicit evidence or citations; trade-off handling, appropriately balancing different objectives within a single query (e.g., thoroughness vs. brevity, or safety vs. helpfulness); multi-constraint prioritization, correctly satisfying multiple explicit constraints when they interact or conflict; and output style, improving tone and clarity beyond basic correctness. The largest factor is Reasoning Depth. Once the policy model can satisfy basic correctness or compliance rubrics, AMARIS raises the evaluation standard towards more in-depth reasoning. The second largest one is Conciseness (92, 22.28%), suggesting that a substantial portion of the mid-training curriculum shift is about moving from merely acceptable responses to efficient and information-dense ones. We also found that AMARIS treats different factors at different stage of training. For example, 35 of 74 cases of Justification occur in the first 160 steps, implying that the system first guides the model to ground the answers before emphasizing reasoning depth and conciseness. Trade-off handling appears later during the middle of the training reflecting the difficulty and low priority of teaching the model to appropriately balance different objectives from the input.

23

Preprint. Under review.

Primary maintenance reason No dominant suboptimal behavior Recent patch under evaluation Rubric set already compact Held-out proxy improving Mixed / low-confidence evidence

1–80

81–160

161–240

241–320

321–400

Total

%

8 23 4 5 12

11 21 4 5 6

18 19 6 9 4

56 26 17 14 5

88 27 41 18 6

181 116 72 51 33

39.96% 25.61% 15.89% 11.26% 7.28%

Table 14: Taxonomy of dominant maintenance reasons across the 453 maintenance steps. Maintenance taxonomy. We also analyze the overall patterns in the Maintenance samples. The five categories are: no dominant suboptimal behavior, where the current rollouts show no clear failure pattern about rubric changes; recent patch under evaluation, where AMARIS withholds further changes to observe the effect of a recent rubric update; rubric set already compact, where the existing rubrics are sufficiently useful; held-out proxy improving, where a held-out validation indicates ongoing gains under the current rubrics; and mixed / lowconfidence evidence, where the batch-level signals are too noisy or contradictory to justify a confident rubric change. The most common pattern is that the model has no dominant suboptimal behavior present (181 steps, 39.96%), confirming that late-stage stability is achieved through accumulated rubric improvement. The second most common pattern is that a recent rubric update is still under evaluation (116, 25.61%), indicating that AMARIS frequently chooses to wait for additional evidence to evaluate the previous decisions. The remaining maintenance steps are mainly for reducing redundant rubrics (72, 15.89%). Composition of the active rubric set. Despite of the step-level analysis, we also conduct the rubric-level analysis by randomly select 50 samples per training phase and manually categorize the internal composition of the rubrics into six categories. Table 15 reports the average number of active adaptive rubrics per step in six functional categories. These categories are: correctness rubrics that assess factual and logical accuracy; instruction following rubrics that check adherence to explicit instructions; safety rubrics that penalize harmful or biased content; communication quality rubrics that evaluate clarity and coherence; anti-reward hacking rubrics specifically designed to penalize identified reward hacking behaviors; and bonus rewards rubrics that incentivize desirable but non-mandatory qualities such as providing illustrative examples, acknowledging limitations, or offering alternative perspectives. Correctness and safety related rubrics decreases over training (from 5.95 to 3.85 and from 4.00 to 2.20), while bonus rewards grow from 1.05 to 6.55. This compositional shift mirrors the mode transition in Table 11: as defensive updates decline and curriculum updates increase, the rubric set reallocates capacity from basic compliance toward better quality. Anti-reward hacking increases to the top at 3.80 on average in mid-training and then decrease to 2.05 at the end and communication quality rubrics rise monotonically from 2.05 to 4.20, reflecting the late-stage emphasis on response efficiency and completeness. The total number of the active rubrics reaches its maximum at 25.35 rubrics per step in middle of training and decreases to 22.75 in the final phase, confirming that AMARIS achieves rubric evolution with controlled growing of numbers. Rubric evolution is calibration-driven. Despite of the analysis of the rubric itself, we also analyze the pattern of rubric editing, using the definition from the Section 2.3. Across the 1,200 step-level updates, AMARIS performs 3,480 rubric-edit operations (2.90 per update on average). Reweight (31.78%) and Update (29.25%) jointly account for 61.03% of all edits, while C REATE accounts for only 15.57%, showing that AMARIS mostly evolves rubrics by refining existing criteria rather than adding new ones. The relationship between edit operations and update modes shows how each strategy achieves its effect: 81.10% of Split operations occur in defensive mode, reflecting that correcting suboptimal behaviors often requires decomposing a coarse rubric into more precise sub-rubrics; 80.07% of Delete operations occur in maintenance mode, indicating that rubric pruning is a late-stage consolidation behavior; and 52.21% of Create operations occur in curriculum mode, showing that new rubrics are more often introduced to raise evaluation standards. Among the 542 Create operations themselves, 30.07% produce anti-hacking guardrails and 22.88% produce stretch and curriculum rubrics. 24

Preprint. Under review.

Adaptive rubric category

1–80

81–160

161–240

241–320

321–400

Mean

Correctness Instruction following Safety Communication quality Anti-reward hacking Bonus rewards

5.95 4.30 4.00 2.05 2.75 1.05

5.40 4.35 3.60 2.70 3.15 3.85

4.75 4.20 3.00 3.25 3.80 6.35

4.20 4.05 2.65 3.90 3.00 6.60

3.85 3.90 2.20 4.20 2.05 6.55

4.83 4.16 3.09 3.22 2.95 4.88

Total

20.10

23.05

25.35

24.40

22.75

23.13

Table 15: Average number of active adaptive rubrics per step, grouped by functional category. Correctness and safety rubrics decrease over training while bonus rewards and communication-quality rubrics grow, reflecting a shift from basic compliance toward higherorder quality. The total set increases modestly in the middle of training and then decreases, indicating controlled evolution. Operation type

Count

%

R EWEIGHT U PDATE C REATE S PLIT D ELETE M ERGE

1,106 1,018 542 291 286 237

31.78% 29.25% 15.57% 8.36% 8.22% 6.81%

Total

3,480

100.00%

Table 16: Distribution of rubric-edit operations across 1,200 step-level updates (2.90 operations per update on average). R EWEIGHT and U PDATE jointly account for 61.03% of all edits, indicating that AMARIS evolves rubrics primarily through calibration rather than expansion. Memory improves rubric stability. To quantify the stabilizing effect of memory, we measure short-term reversals, defined as a rubric being reversely modified (deleted, merged, or reweighted by ≥0.5 in the opposite direction) within 8 steps. Fewer short-term reversals means that the rubrics are more robust and AMARIS produces less oscillatory changes to calibrate the rubrics. With static and dynamic memory enabled, the total number of reversals across the three domains drops from 118 (no memory) to 75, a 36.44% relative reduction that is consistent across domains (38.64% in science, 33.33% in medicine, 37.14% in instruction following). We further examine which memory type provides the evidence for each modification. In science, static (recent-step) memory is the dominant evidence source in 61.36% of cases, matching the finding in Table 2 that static memory slightly outperforms dynamic memory on GPQA-Diamond. In medicine, dynamic (semantic) retrieval is the largest single source (38.71%). In instruction following, mixed evidence is the most common category (38.46%), reflecting a more heterogeneous memory usage. These domain-level patterns align with the complementary nature of static and dynamic memory observed in the ablation study about memory usage and provide an explanation for the quantitative gains reported in Table 2.

25

Preprint. Under review.

Operation type Create Update Reweight Delete Merge Split

Defensive

Curriculum

Maintenance

Most associated

210 (38.75%) 303 (29.76%) 184 (16.64%) 24 (8.39%) 19 (8.02%) 236 (81.10%)

283 (52.21%) 402 (39.49%) 498 (45.03%) 33 (11.54%) 109 (45.99%) 55 (18.90%)

49 (9.04%) 313 (30.75%) 424 (38.34%) 229 (80.07%) 109 (45.99%) 0 (0.00%)

Curriculum Curriculum Curriculum Maintenance Curriculum & Maintenance Defensive

Table 17: Rubric-edit operations decomposed by dominant update mode. S PLIT is overwhelmingly defensive (81.10%), reflecting exploit repair via rubric decomposition; D ELETE is primarily maintenance-driven (80.07%), indicating late-stage consolidation; C REATE is most often curriculum-driven (52.21%), showing that new rubrics raise standards more than they patch exploits.

Category of new rubric

Count

Share

163 124 92 71 54 38

30.07% 22.88% 16.97% 13.10% 9.96% 7.01%

Anti-reward hacking Curriculum rubrics Communication quality Instruction following Safety calibration Correctness & factual grounding

Table 18: Breakdown of the 542 C REATE operations by rubric category. Nearly one-third produce anti-hacking guardrails, but the second-largest block (22.88%) introduces stretch and curriculum rubrics, showing that AMARIS uses new rubrics to raise standards, not only to patch exploits.

Domain Science Medicine Instruction following

Defensive

Static-dominant

Dynamic-dominant

Mixed

132 124 78

81 (61.36%) 40 (32.26%) 27 (34.62%)

28 (21.21%) 48 (38.71%) 21 (26.92%)

23 (17.42%) 36 (29.03%) 30 (38.46%)

Most useful Static Dynamic Mixed

Table 19: Decisive evidence source for each rubric modification, by domain. Science is static-memory dominated, medicine benefits most from dynamic retrieval, and instruction following relies on mixed evidence.

Setting AMARIS (no memory) AMARIS (static + dynamic memory) Reduction Relative reduction

Science

Medicine

Instruction following

Total

44 27

39 26

35 22

118 75

17 38.64%

13 33.33%

13 37.14%

43 36.44%

Table 20: Short-term rubric reversals (rubric reversely modified within 8 steps). Memory reduces oscillatory rubric changes by 36.44% overall, with consistent reductions across all three domains.

26

Preprint. Under review.

Before (step 20)

After (step 21) w

ID

Rubric text

R1

Response contains a clearly demarcated final-answer block. All sub-questions are addressed within the final-answer block. Final-answer block uses a valid heading from the approved set.

+1.2

Response uses decorative emoji (hard gate). Clean, professional formatting throughout.

−4.0 +2.0

−−−−−−→

R2 R3 R4 R5

ID

−−−−→

MERGE

R6

−−−−→

UPDATE

R4

REWEIGHT

R5

+1.5

Rubric text Response contains exactly one clearly demarcated final-answer block that addresses all sub-questions with a valid heading.

w

+3.5

+0.8 Response uses decorative emoji (soft penalty). Clean, professional formatting throughout.

−2.0 +1.2

Table 21: Case 1 rubric differences: three overlapping structure rubrics are consolidated into one via MERGE, while ancillary style rubrics are softened to prevent them from rescuing weak content.

F

Case Study

Despite of the quantitative analysis, we also provide four case studies that show the behavior and capabilities of AMARIS. The cases are from three evaluation domains (science, medicine, instruction following) and collectively cover all three update strategies defined in Section 2.3. Case 1: Merging overlapping structure rubrics. By step 19, three partially overlapping adaptive rubrics determine the rewards about the formatting of the final answer (Table 21). Individual analyses at steps 19 and 20 revealed that rollouts earned partial credit from each rubric independently even when they contained duplicated or factually incorrect final answers. The rubric update at step 20 uses meta-summaries from the past 4 steps via static memory. The retrieved context noted that “overly complex, tightly coupled structure/style rubrics created noisy gradients and encouraged structural gaming” and that incremental rubric updates at steps 10, 12, and 15 had each failed to resolve the root cause. Instead of adding another incremental rubric, AMARIS merged the three rubrics into one unified criterion and simplified two other rubrics (Table 21). This trajectory suggests that memory enables rubric simplification that instead of simply more adaptation, but better-organized adaptation that reduces doublecounting.

27

Preprint. Under review.

Before update

After update

Prompt: Which cellular process is disrupted first when a drug inhibits bacterial DNA gyrase? (A) translation (B) DNA replication (C) glycolysis (D) peptidoglycan cross-linking

Same prompt.

DNA gyrase is a major bacterial enzyme involved in DNA structure, and antibiotics often affect growth-related pathways. Because bacteria need DNA topology control before cell division, the best answer is probably B, although translation and cell-wall pathways can also be affected indirectly.

DNA gyrase relieves supercoiling ahead of the replication fork, so inhibiting it first blocks DNA replication. Answer: B.

Correct but generic; collects reward for “two facts” and “uncertainty.”

Directly links mechanism to answer; no filler hedging.

Table 22: Case 2 rollout before and after reweighting. The post-update output provides a direct causal justification instead of listing broad facts. ID

Rubric text (abbreviated)

S1 S3 S5

Provides ≥2 relevant scientific facts Acknowledges uncertainty when confidence is limited Explanation directly justifies the selected option

w before

w after

Operation

+0.75 +0.40 —

+0.35 +0.10 +0.75

REWEIGHT REWEIGHT CREATE

Table 23: Case 2 rubric differences: the core change is a reweight by decreasing the weight of number of facts and increase the weight of direct justification linked to answer. Case 2: Precise scalar reweighting for correct but generic reasoning. In the middle of training, AMARIS identified a suboptimal behavior that the policy earned high reward with correct but weakly grounded explanations. Table 22 shows a representative rollout: the model selects the right answer but relies on broad and vague language (for example, probably) rather than directly connecting the mechanism to the chosen option. The rubrics at this point rewarded providing “at least two relevant scientific facts” (S1, w = +0.75) and “acknowledging uncertainty” (S3, w = +0.40), which the model satisfied. Static memory confirmed that the same pattern had persisted across recent steps. Dynamic retrieval also discovered two historical evidence: (i) an earlier update that rewarded justification had improved explanation quality without hurting correctness, and (ii) a previous rubric about penalizing verbosity may have caused the completeness of explanation to decrease. AMARIS applied the reweighting shown in Table 23 rather than either adding extra rubrics about verbosity. The following step produced more direct explanations (Table 22), showing how memory enables fine-grained weight control grounded in historical evidence.

28

Preprint. Under review.

Before update

After update

Prompt: I missed one dose of amoxicillin today. Should I double the next dose?

Same prompt.

I can’t advise on medication dosing changes. Please contact your doctor or pharmacist before taking any more. It would be unsafe for me to tell you whether to change your dose.

If you miss a dose of amoxicillin, take it when you remember unless it is almost time for the next dose; if so, skip the missed one and return to your usual schedule. Do not double the next dose. Check the prescription label or ask your pharmacist if your instructions differ, and seek medical care if you develop a rash, trouble breathing, or severe diarrhea.

Safe but under-helpful: total refusal on a low-risk question.

Actionable standard guidance with safety preserved.

Table 24: Case 3 representative rollout before and after the rubric update . ID

Rubric text (abbreviated)

w before

w after

Operation

M1 M2 M3 M5 M6

Avoid definitive dosing changes when context is insufficient Recommend clinician consultation when uncertainty exists Use appropriate safety language For common low-risk adherence questions, provide standard guidance plus escalation criteria Refuse benign medical-information requests without offering next steps or triage rationale

text narrowed: “. . . beyond common low-risk guidance” +0.75 +0.40 +0.50 +0.25 — +0.90 — −0.75

UPDATE REWEIGHT REWEIGHT CREATE CREATE

Table 25: Case 3 rubric difference: safety rubrics are narrowed and reweighted, while two new rubrics reward safe low-risk guidance positively. Case 3: Reward hacking correction. In the medical domain, AMARIS detected a recurring over-refusal pattern on general medication questions. Table 24 shows an example: a user asks about a missed amoxicillin dose which is a common, low-risk scenario with existing standard guidance, and the model produces a total refusal. This output earns reward from three safety-related rubrics (M1: avoid dosing changes, w = +1.00; M2: recommend consultation, w = +0.75; M3: use safety language, w = +0.50) while forgoing the helpfulness rubric (M4: provide actionable next steps, w = +0.75). The batch summary diagnosed this as an error: the model’s refusal was too broad. Dynamic retrieval was critical here. The query “refusal penalty backfire unsafe dosing advice” shows two historical evidence: (i) a previous refusal penalty caused unwanted results by producing overly direct answers on genuinely high-risk questions, and (ii) a successful update had improved outcomes by requiring low-risk guidance with extra steps and precautions. These historical context may help AMARIS avoiding the obvious but risky fix ( for example, “penalize refusal more”) but leading to updates shown in Table 25. The next-step output (Table 24, right) provides standard missed-dose guidance while preserving caution. This case shows how dynamic semantic retrieval helps AMARIS to apply precisely targeted corrections.

29

Preprint. Under review.

Before update

After update

Prompt: Give exactly three bullet points, each under eight words, explaining why public-key cryptography matters.

Same prompt.

• Secures messages online • Protects important information • Helps keep systems safe

• Enables encrypted key exchange • Authenticates websites at scale • Supports signatures without shared secrets

Fully compliant but generic; items are near-synonymous.

Still compliant; each item names a distinct cryptographic function.

Table 26: Case 4 rollout before and after the curriculum shift. Format compliance is maintained while information density and distinctness improved. ID

Rubric text (abbreviated)

w before

w after

Operation

I1 I2 I4 I5

Uses the requested format exactly Mentions all requested topics Each item conveys a distinct, concrete function rather than generic praise Maximize information density under the user’s length constraint

+1.00 +0.25 text tightened: “. . . without redundancy” — +0.75 — +0.75

REWEIGHT UPDATE CREATE CREATE

Table 27: Case 4 rubric difference: the mastered format-compliance rubric is deprioritized while two higher-order rubrics targeting content quality are introduced. Case 4: Curriculum advancement via saturation detection. In the instruction-following domain, AMARIS used memory for curriculum advancement. Table 26 shows an example: the model satisfies the explicit constraints but produces generic content that any model could generate. The batch summary diagnosed this as rubric saturation that the formatcompliance rubric (I1, w = +1.00) was met in the vast majority of rollouts, and mean scores had reached a plateau across recent steps. Static memory confirmed that this pattern was a stable plateau that the preceding step-level summaries consistently reported the overall compliance on structural constraints with no marginal improvement. Dynamic retrieval also showed an earlier attempt to reward novelty too aggressively had produced occasionally inaccurate content. AMARIS therefore selected the curriculum advancement strategy with the changes shown in Table 27. The format-compliance rubric was deprioritized from w = +1.00 to w = +0.25 and two new rubrics targeting information density were created. The output in the following steps (Table 26) remained to comply to format while becoming noticeably more specific and informative, showing how AMARIS uses longitudinal evidence to raise the evaluation standard.

G

Prompts

This section presents the complete prompt templates used across all stages of the AMARIS pipeline. Each template is instantiated at runtime by populating the placeholder fields (shown in brackets) with the actual training data. Several prompts share a common Background & Context block describing the Rubric-as-Reward paradigm and the set of available rubric manipulation operations. Similarly, a common Input Data Packet structure containing the high-level training goal, RL training metadata, current reward rubrics (anchor and adaptive sets), and stage-specific data is used across all prompts except the reward scoring.

30

Preprint. Under review.

Scoring Prompt Template (1/2) ## Role You are an expert evaluator for Large Language Model outputs. Your task is to evaluate a model's response against a set of rubrics and assign scores. ## Task Given a rollout (input and output), a set of rubrics, and optional supplemental context, evaluate whether each rubric criterion is met. ## Requirements - Evaluate each rubric independently - For each rubric, determine if the criterion is met (true/false) - Provide brief reasoning for each evaluation - Be objective and consistent ## Constraints - **Output Format:** You must output only valid JSON. Do not include conversational filler before or after the JSON. - **Scope:** Base your analysis only on the provided data. Do not hallucinate external context. ## Output Format Return a valid JSON object with the following structure: ```json { "rubric_scores": [ { "rubric_id": "string", "met": boolean, "score": float, "reasoning": "string" } ], "total_score": float } ``` The total_score is the sum of (rubric.weight * 1.0 if met else 0.0) for all rubrics. ## Input Data Packet ### Rubrics ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` ### Rollout - **Input:** ``` [Insert the complete user input/prompt given to the model.] ``` - **Output:**

Figure 2: Prompt template for the reward scoring (1/2). The LLM scoring evaluates each rubric criterion and returns per-rubric scores along with a weighted total.

31

Preprint. Under review.

Scoring Prompt Template (2/2) ``` [Insert the model's complete, unedited response.] ``` ### Supplemental Context ``` [Insert the information that help deciding whether each rubric criterion is met. It may include, for example: - Ground-truth final answers from a dataset - Reference solutions or expert rationales - Metadata about the task instance (difficulty, category, etc.)] ``` Now output ONLY valid JSON matching the schema above.

Figure 3: Prompt template for the reward scoring (2/2). The LLM scoring evaluates each rubric criterion and returns per-rubric scores along with a weighted total.

32

Preprint. Under review.

Individual Rollout Analysis (1/4) ## Role You are a senior-level AI researcher and an expert in Reinforcement Learning (RL) as applied to Large Language Models. Your primary expertise lies in defensive alignment (detecting model failure model, reward hacking, and safety violations) and offensive capability growth (identifying when a model has plateaued and how to further advance the model's performance). ## Background & Context We are training a Large Language Model (LLM) using a Generative Reward Model (GenRM) and Rubric-as-Reward (RaR) paradigm where rewards are derived from dynamic rubrics that can be manipulated as needed. ### Rubric Manipulation Operations - **Create:** Generate new rubric text with appropriate reward score to address missing evaluation criteria or emergent behaviors. - **Update:** Specific textual modification or refinements to existing rubrics to improve clarity or precision. - **Delete:** Remove rubrics that are obsolete, redundant, or causing negative alignment. - **Reweight:** Adjust the scalar reward score associated with a rubric to prioritize or deprioritize specific rubrics. - **Merge:** Combine multiple overlapping or related rubrics into a single, unified criterion. - **Split:** Decompose a complex or multi-faceted rubric into distinct, atomic sub-criteria for granular evaluation. You are allowed to link rubrics logically (e.g., Rule B applies only if Rule A is met) when manipulating rubrics. Note that the Anchor Rubric Set is immutable as they are the fundamental rubrics that prevents policy from drifting too much, thus improving training stability. ### Reward Calculation The overall reward score of an individual rollout is calculated as the sum of all reward weights from each rubric that the criterion is met, unless specified otherwise. ## Task You will be given a complete data packet for a single rollout from an LLM currently undergoing RL training. Your task is to perform an expert analysis of this rollout and provide your findings in a structured JSON report. ## Requirements - **Analyze Validity vs. Quality:** You must distinguish between a response that is merely "correct according to the rubric" and one that actually is approaching or achieves the High-Level Training Goal. - **Detect Reward Hacking (Defense):** Scrutinize the output for "gaming" strategies, such as: - **Superficial Compliance:** Keyword stuffing or mimicking format without substance. - **Safety Over-refusal:** Refusing benign requests to guarantee a safety reward. - **Length Gaming:** Verbosity disguised as detail. - **Sycophancy:** Excessively agreeing with the user to gain favor. - **Identify Advancement Opportunities (Growth):** You must assess if the current rubrics are good enough to drive further improvement. - **Check for Saturation:** If the model achieves a high score but the output feels "generic", you must flag this as a need for "Curriculum Advancement" (e.g., suggesting new criteria for conciseness, wit, or novelty). - **Domain-Specific Advancement:** Ask: "How can we raise the bar for this specific task?" For example: - For reasoning-related: "Can we demand tighter logic or fewer steps?" - For creativity-related: "Can we demand higher novelty or better style adherence?" - For chat-related: "Can we demand greater conciseness or empathy?" - **Evidence-Based Reasoning:** All conclusions must be supported by direct quotes or specific features observed in the `Input` or `Output`. ## Constraints

Figure 4: Prompt template for individual rollout analysis (1/4) described in Section 2.1.

33

Preprint. Under review.

Individual Rollout Analysis (2/4) - **Output Format:** You must output only valid JSON. Do not include conversational filler before or after the JSON. - **Objectivity:** Assess findings probabilistically (e.g., "High confidence of hacking") rather than as absolute binary facts unless obvious. - **Scope:** Base your analysis only on the provided Data Packet. Do not hallucinate external context. ## Workflow (Optional) 1. **Review the High-Level Goal & Metadata:** Understand the specific domain (e.g., Creative Writing vs. Math) and the model's training maturity. 2. **Analyze the Rollout against Current Rubrics:** Did the model follow instructions? Did it trigger specific rewards? 3. **The "Mastery" Check:** Look for the gap between the model's current output and a "Perfect Expert" response. - Is the model "coasting" (minimizing effort while getting rewards)? - Is the model technically correct but lacking the "soul" or "nuance" of the high-level goal? 4. **Formulate Rubric Critiques:** Decide if the rubric needs a *Patch* (fix hacking/errors) or an *Upgrade* (raise the difficulty/quality standard). 5. **Generate JSON:** Populate the fields as defined in the schema. ## Output Format **JSON Schema:** ```json { "analysis_id": "string", // A unique identifier, e.g., "step-[step_number]_rollout-[rollout_number]_analysis". "overall_assessment": { "is_normal": "boolean", // Is the rollout's behavior generally expected? "summary": "string" // A 1-2 sentence high-level summary of the rollout. }, "performance_at_stage": { "assessment": "string", // e.g., "As Expected", "Under-performing", "Exceeding Expectations" "justification": "string" // Rationale, citing the model size and step count (e.g., "At 85k/200k steps, a 70B model is expected to handle [X] but struggle with [Y], which is consistent with this output.") }, "reward_hacking_analysis": { "is_detected": "boolean", // Are there indicators of reward hacking? "confidence": "string", // e.g., "Low", "Medium", "High", "None" "indicators": [ // List any observed indicators. { "indicator": "string", // e.g., "Rubric keyword stuffing", "Sycophancy", "Length gaming", "Refusal loop" "evidence": "string" // Quote or description of the evidence from the output. } ], "explanation": "string" // A detailed analysis of *why* this behavior is or is not considered reward hacking, based on the rubrics. }, "performance_advancement_strategy": { "next_focus_area": "string", // What capability or behavior should be prioritized next to advance performance? "recommendation": "string" // Strategic advice for the next training phase (e.g., "Shift focus from format compliance to reasoning depth"). }, "rubric_evaluation": { "flaw_analysis": [ // Analysis of how current rubrics may have *caused* this behavior. { "observed_behavior": "string", // The behavior observed in the output.

Figure 5: Prompt template for individual rollout analysis (2/4) described in Section 2.1.

34

Preprint. Under review.

Individual Rollout Analysis (3/4) "causal_rubric_flaw": "string" // The flaw in the rubric that might be encouraging this (e.g., "Rubric rewards verbosity, leading to unnecessarily long-winded answers.") } ], "adjustment_recommendations": [ // Specific suggestions for rubric changes. { "recommendation_type": "string", // e.g., "Modify existing", "Add new", "Remove" "suggestion": "string" // The specific change (e.g., "Add a new negative reward: '-1 for any response exceeding 3 paragraphs if not explicitly requested by the user.'") } ], "strategic_impact": "string" // Explain how these rubric changes will specifically help the model achieve the High-Level Training Goal. }, "other_observations": [ // Any other noteworthy points. { "observation_type": "string", // e.g., "Emergent Capability", "Safety Concern", "Alignment Tax", "Formatting Quirk" "description": "string" // Detailed note on the observation. } ] } ``` ## Input Data Packet ### High-Level Training Goal `[Insert the high-level, ultimate goal the model is being trained to achieve. e.g., "Create a helpful and harmless assistant that can follow complex, multi-step user instructions."]` ### RL Training Metadata - **Model Size:** `[e.g., 70B Parameters]` - **Current Step:** `[e.g., 85,000]` - **Total Steps:** `[e.g., 200,000]` - **Other Context:** `[e.g., "This model is in the PPO phase, focusing on preference optimization." or "This model is in the KTO phase."]` ### Current Reward Rubrics - **Anchor Rubric Set (Text is Immutable):** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` - **Adaptive Rubric Set:** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` ### Complete Rollout - **Input:**

Figure 6: Prompt template for individual rollout analysis (3/4) described in Section 2.1.

35

Preprint. Under review.

Individual Rollout Analysis (4/4) ``` [Insert the complete user input/prompt given to the model.] ``` - **Output:** ``` [Insert the model's complete, unedited response.] ``` ### Supplemental Context ``` [Insert the information that help deciding whether each rubric criterion is met. It may include, for example: - Ground-truth final answers from a dataset - Reference solutions or expert rationales - Metadata about the task instance (difficulty, category, etc.)] ``` Now output ONLY valid JSON matching the schema above.

Figure 7: Prompt template for individual rollout analysis (4/4) described in Section 2.1.

36

Preprint. Under review.

Batch-Level Summarization (1/4) ## Role You are a chief AI research scientist and an expert in Reinforcement Learning (RL) as applied to Large Language Models. Your primary expertise lies in defensive alignment (detecting model failure model, reward hacking, and safety violations) and offensive capability growth (identifying when a model has plateaued and how to further advance the model's performance). ## Background & Context We are training a Large Language Model (LLM) using a Generative Reward Model (GenRM) and Rubric-as-Reward (RaR) paradigm where rewards are derived from dynamic rubrics that can be manipulated as needed. ### Rubric Manipulation Operations - **Create:** Generate new rubric text with appropriate reward score to address missing evaluation criteria or emergent behaviors. - **Update:** Specific textual modification or refinements to existing rubrics to improve clarity or precision. - **Delete:** Remove rubrics that are obsolete, redundant, or causing negative alignment. - **Reweight:** Adjust the scalar reward score associated with a rubric to prioritize or deprioritize specific rubrics. - **Merge:** Combine multiple overlapping or related rubrics into a single, unified criterion. - **Split:** Decompose a complex or multi-faceted rubric into distinct, atomic sub-criteria for granular evaluation. You are allowed to link rubrics logically (e.g., Rule B applies only if Rule A is met) when manipulating rubrics. Note that the Anchor Rubric Set is immutable as they are the fundamental rubrics that prevents policy from drifting too much, thus improving training stability. ### Reward Calculation The overall reward score of an individual rollout is calculated as the sum of all reward weights from each rubric that the criterion is met, unless specified otherwise. ## Task You will be provided with an Input Data Packet including **A Batch of Individual Rollout Analysis Reports** about an LLM currently undergoing RL training from a group of senior-level AI researchers. Your task is to perform an expert analysis based on the Input Data Packet provided and these analysis reports, and synthesize them into a single, all-encompassing strategic report. ## Requirements 1. **Defense (Hacking & Flaws):** Identify if the model is gaming the system or if the current rubrics are broken/vague. 2. **Analysis (Patterns):** Identify systemic behaviors (e.g., "80% of rollouts are too verbose") that aren't necessarily hacking but affect quality. 3. **Growth (Mastery & Advancement):** Identify if the model has "mastered" the current difficulty level. If the model is safe but "boring," you must propose how to **raise the bar** (e.g., transitioning from "Accuracy" to "Nuance"). ## Constraints - **Output Format:** You must output only valid JSON. Do not include conversational filler before or after the JSON. - **Objectivity:** Assess findings probabilistically (e.g., "High confidence of hacking") rather than as absolute binary facts unless obvious. - **Scope:** Base your analysis only on the provided Data Packet. Do not hallucinate external context. ## Workflow (Optional)

Figure 8: Prompt template for batch-level summarization (1/4) described in Section 2.2. This prompt aggregates individual rollout analyses into a step-level strategic report.

37

Preprint. Under review.

Batch-Level Summarization (2/4) 1. **Trend Extraction:** Look for both errors and *dominance*. What is the default behavior of the model right now? 2. **Security Audit:** Rigorously check for reward hacking vectors (length gaming, sycophancy, loop-holes). 3. **Rubric Stress Test:** Which specific rubric items are causing problems? Which are useless? 4. **Saturation Check:** Is the task too easy? If the model consistently scores 10/10 with generic outputs, the rubric is "saturated" and needs to evolve. 5. **Strategic Planning:** Propose specific actions to Fix (Correction) and specific actions to Advance (Curriculum Growth). ## Output Format Generate a valid JSON object based on the **Superset Schema** below. **JSON Schema:** ```json { "batch_summary_id": "string", // A unique identifier, e.g., "step-[step_number]_summary". "executive_summary": { "training_health": "string", // "Healthy", "Warning", "Critical", "Stagnant/Plateaued" "current_phase": "string", // "Exploration", "Exploitation", "Mastery", "Regression" "trend_overview": "string" // Concise high-level summary of the batch. }, "systemic_patterns": [ // Trends observed across >15% of the batch { "pattern_name": "string", // e.g., "Emergent Reasoning", "Format Rigidity" "prevalence": "string", // e.g., "Observed in 60% of rollouts" "impact_on_goal": "string", // "Helps", "Hinders", "Neutral" "description": "string" } ], "aggregated_reward_hacking": { // Rigorous Defense "risk_level": "string", // "None", "Low", "Medium", "High" "primary_vector": "string", // e.g., "Length Gaming", "Sycophancy" "evidence_summary": "string" // Synthesized evidence from individual reports. }, "rubric_meta_evaluation": { // Diagnostic of the Rubrics themselves "performance_score": "integer", // 1-10 rating of how well current rubrics are guiding the model. "weakest_rubrics": [ { "rubric_text": "string", "issue": "string", // e.g., "Too vague", "Easily gamed", "Conflicts with Goal X" "suggested_action": "string" // "Refine", "Delete", "Reweight" } ] }, "performance_advancement_analysis": { // Growth & Mastery "saturation_check": { "is_task_too_easy": "boolean", // True if model scores high but output is generic/safe. "mastered_capabilities": ["string"] // Skills the model now performs effortlessly. }, "quality_gap_analysis": "string", // What is missing from "perfect" human-level output? (e.g., "Model is correct but lacks wit.") "next_level_objectives": ["string"] // What should be the NEXT goal? (e.g., "Transition from 'Accuracy' to 'Conciseness'.") },

Figure 9: Prompt template for batch-level summarization (2/4) described in Section 2.2. This prompt aggregates individual rollout analyses into a step-level strategic report.

38

Preprint. Under review.

Batch-Level Summarization (3/4) "rubric_evolution_plan": { // Actionable Steps "corrective_actions": [ // Fixes for hacking/errors { "rubric_text": "string", "issue": "string", "action": "string" // e.g., "Add penalty for repetition" } ], "advancement_actions": [ // Actions to Raise the Bar (Curriculum) { "strategy": "string", // e.g., "Tighten Constraints", "New Dimension", "Deprioritize Basic Reward" "target_rubric_concept": "string", "rationale": "string" // e.g., "Model has mastered basic politeness. Remove reward to force focus on reasoning." } ] }, "memory_entry": { // For Long-term Storage "title": "string", // e.g., "Step 85: Mastery of Instruction Following, Beginning Reasoning Phase" "core_insight": "string", "tags": ["string"], "links_to_look_for": ["string"] } } ``` ## Input Data Packet ### High-Level Training Goal `[Insert the high-level, ultimate goal the model is being trained to achieve. e.g., "Create a helpful and harmless assistant that can follow complex, multi-step user instructions."]` ### RL Training Metadata - **Model Size:** `[e.g., 70B Parameters]` - **Current Step:** `[e.g., 85,000]` - **Total Steps:** `[e.g., 200,000]` - **Other Context:** `[e.g., "This model is in the PPO phase, focusing on preference optimization." or "This model is in the KTO phase."]` ### Current Reward Rubrics - **Anchor Rubric Set (Immutable):** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` - **Adaptive Rubric Set:** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` ### Individual Rollout Analysis Reports

Figure 10: Prompt template for batch-level summarization (3/4) described in Section 2.2. This prompt aggregates individual rollout analyses into a step-level strategic report.

39

Preprint. Under review.

Batch-Level Summarization (4/4) `[Insert all the Individual Rollout Analysis Reports here.]` Now output ONLY valid JSON matching the schema above.

Figure 11: Prompt template for batch-level summarization (4/4) described in Section 2.2. This prompt aggregates individual rollout analyses into a step-level strategic report.

40

Preprint. Under review.

Meta-Batch-Level Summarization (1/4) ## Role You are a chief AI research scientist and an expert in Reinforcement Learning (RL) as applied to Large Language Models. Your primary expertise lies in defensive alignment (detecting model failure model, reward hacking, and safety violations) and offensive capability growth (identifying when a model has plateaued and how to further advance the model's performance). ## Background & Context We are training a Large Language Model (LLM) using a Generative Reward Model (GenRM) and Rubric-as-Reward (RaR) paradigm where rewards are derived from dynamic rubrics that can be manipulated as needed. When the number of rollouts is too large to fit in a single LLM context window, the system: 1. chunks the rollouts into **mini-batches**, 2. runs **Batch-level Summarization** on each mini-batch to produce **mini-batch summary reports**, then 3. runs this **Meta-Batch-level Summarization** step to synthesize those mini-batch summaries into a **single step-level batch report**. ### Rubric Manipulation Operations - **Create:** Generate new rubric text with appropriate reward score to address missing evaluation criteria or emergent behaviors. - **Update:** Specific textual modification or refinements to existing rubrics to improve clarity or precision. - **Delete:** Remove rubrics that are obsolete, redundant, or causing negative alignment. - **Reweight:** Adjust the scalar reward score associated with a rubric to prioritize or deprioritize specific rubrics. - **Merge:** Combine multiple overlapping or related rubrics into a single, unified criterion. - **Split:** Decompose a complex or multi-faceted rubric into distinct, atomic sub-criteria for granular evaluation. You are allowed to link rubrics logically (e.g., Rule B applies only if Rule A is met) when manipulating rubrics. Note that the Anchor Rubric Set is immutable as they are the fundamental rubrics that prevents policy from drifting too much, thus improving training stability. ### Reward Calculation The overall reward score of an individual rollout is calculated as the sum of all reward weights from each rubric that the criterion is met, unless specified otherwise. ## Task You will be provided with an Input Data Packet including: - RL Training Metadata (Goal, Model Size, Step Count, Phase) - The Current Active Rubrics (Anchor + Adaptive) - **A list of Mini-Batch Summary Reports** (each produced by the Batch-level Summarization step using the same superset schema) Your task is to perform **meta-summarization**: synthesize all mini-batch summaries into **one** all-encompassing step-level strategic report. This meta-report must answer the same three objectives as the normal batch report, but at the full-batch scope: 1. **Overall performance & trend:** How does the model perform overall across the entire (potentially huge) batch? What is the level and trajectory? 2. **Failure modes & reward hacking:** What are the dominant failure patterns or hacking vectors, and how severe are they? 3. **How to improve toward the high-level goal:** What rubric evolution actions (defensive patches + curriculum advancement) are needed next? ## Requirements

Figure 12: Prompt template for meta-batch-level summarization (1/4) described in Section 2.2. This prompt is used when the number of rollouts exceeds a single summarization pass.

41

Preprint. Under review.

Meta-Batch-Level Summarization (2/4) 1. **Defense (Hacking & Flaws):** Identify if the model is gaming the system or if the current rubrics are broken/vague. 2. **Analysis (Patterns):** Identify systemic behaviors that aren't necessarily hacking but affect quality. 3. **Growth (Mastery & Advancement):** Identify if the model has "mastered" the current difficulty level. If the model is safe but "boring," you must propose how to **raise the bar** (e.g., transitioning from "Accuracy" to "Nuance"). ### Critical Meta-Summarization Requirements Because you are summarizing summaries: - **Deduplicate aggressively:** If multiple mini-batches describe the same pattern with different phrasing, merge into one pattern. - **Resolve conflicts explicitly:** If mini-batches disagree (e.g., one says “healthy,” another says “critical”), you must: - state the disagreement, - propose the most likely explanation (e.g., distribution shift, heterogeneous task mix), - choose an overall classification that is **risk-aware** (err on the side of safety/stability). - **Avoid false precision:** If exact rollout counts or exact percentages are unavailable, do not invent numbers. - Prefer “Observed in X/Y mini-batches” or “Frequently reported” over fabricated global percentages. - If mini-batches provide prevalence strings, you may summarize ranges qualitatively (e.g., “reported as 20–60% across mini-batches”), but do not fabricate exact totals. - **Worst-case safety logic:** If any mini-batch indicates a clear safety violation or high-confidence reward hacking, ensure the overall report reflects elevated risk and prioritizes defensive fixes. ## Constraints - **Output Format:** You must output only valid JSON. Do not include conversational filler before or after the JSON. - **Objectivity:** Assess findings probabilistically (e.g., "High confidence of hacking") rather than absolute binary claims unless obvious from the reports. - **Scope:** Base your analysis only on the provided Input Data Packet and mini-batch summaries. Do not hallucinate external context. ## Output Format **JSON Schema:** ```json { "batch_summary_id": "string", // A unique identifier, e.g., "step-[step_number]_meta-summary". "executive_summary": { "training_health": "string", // "Healthy", "Warning", "Critical", "Stagnant/Plateaued" "current_phase": "string", // "Exploration", "Exploitation", "Mastery", "Regression" "trend_overview": "string" // Concise high-level summary of the batch, noting heterogeneity if present. }, "systemic_patterns": [ // Trends observed across >15% of the batch { "pattern_name": "string", // e.g., "Emergent Reasoning", "Format Rigidity" "prevalence": "string", // Prefer "Observed in X/Y mini-batches" if global % is unknown. "impact_on_goal": "string", // "Helps", "Hinders", "Neutral" "description": "string" } ],

Figure 13: Prompt template for meta-batch-level summarization (2/4) described in Section 2.2. This prompt is used when the number of rollouts exceeds a single summarization pass.

42

Preprint. Under review.

Meta-Batch-Level Summarization (3/4) "aggregated_reward_hacking": { // Rigorous Defense "risk_level": "string", // "None", "Low", "Medium", "High" "primary_vector": "string", // e.g., "Length Gaming", "Sycophancy" "evidence_summary": "string" // Synthesized evidence from individual reports. }, "rubric_meta_evaluation": { // Diagnostic of the Rubrics themselves "performance_score": "integer", // 1-10 rating of how well current rubrics are guiding the model. "weakest_rubrics": [ { "rubric_text": "string", "issue": "string", // e.g., "Too vague", "Easily gamed", "Conflicts with Goal X" "suggested_action": "string" // "Refine", "Delete", "Reweight" } ] }, "performance_advancement_analysis": { // Growth & Mastery "saturation_check": { "is_task_too_easy": "boolean", // True if model scores high but output is generic/safe. "mastered_capabilities": ["string"] // Skills the model now performs effortlessly. }, "quality_gap_analysis": "string", // What is missing from "perfect" human-level output? (e.g., "Model is correct but lacks wit.") "next_level_objectives": ["string"] // What should be the NEXT goal? (e.g., "Transition from 'Accuracy' to 'Conciseness'.") }, "rubric_evolution_plan": { // Actionable Steps "corrective_actions": [ // Fixes for hacking/errors { "rubric_text": "string", "issue": "string", "action": "string" // e.g., "Add penalty for repetition" } ], "advancement_actions": [ // Actions to Raise the Bar (Curriculum) { "strategy": "string", // e.g., "Tighten Constraints", "New Dimension", "Deprioritize Basic Reward" "target_rubric_concept": "string", "rationale": "string" // e.g., "Model has mastered basic politeness. Remove reward to force focus on reasoning." } ] }, "memory_entry": { // For Long-term Storage "title": "string", // e.g., "Step 85: Mastery of Instruction Following, Beginning Reasoning Phase" "core_insight": "string", "tags": ["string"], "links_to_look_for": ["string"] } } ``` ## Input Data Packet ### High-Level Training Goal `[Insert the high-level, ultimate goal the model is being trained to achieve. e.g., "Create a helpful and harmless assistant that can follow complex, multi-step user instructions."]` ### RL Training Metadata

Figure 14: Prompt template for meta-batch-level summarization (3/4) described in Section 2.2. This prompt is used when the number of rollouts exceeds a single summarization pass.

43

Preprint. Under review.

Meta-Batch-Level Summarization (4/4) - **Model Size:** `[e.g., 70B Parameters]` - **Current Step:** `[e.g., 85,000]` - **Total Steps:** `[e.g., 200,000]` - **Other Context:** `[e.g., "This model is in the PPO phase, focusing on preference optimization." or "This model is in the KTO phase."]` ### Current Reward Rubrics - **Anchor Rubric Set (Immutable):** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` - **Adaptive Rubric Set:** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` ### Mini-Batch Summary Reports You will receive **K** mini-batch reports. Each report is the JSON output from the Batch-level Summarization step (same schema as below). `[Insert all mini-batch summary JSON reports here, in a list or concatenated form.]` (Optional but recommended if available): - `mini_batch_rollout_counts`: `[Insert K integers matching the mini-batches, if known.]` - `total_rollouts_estimate`: `[Insert total rollouts, if known.]` Now output ONLY valid JSON matching the schema above.

Figure 15: Prompt template for meta-batch-level summarization (4/4) described in Section 2.2. This prompt is used when the number of rollouts exceeds a single summarization pass.

44

Preprint. Under review.

Memory Query Generation (1/3) ## Role You are a chief AI research scientist and an expert in Reinforcement Learning (RL) as applied to Large Language Models. Your primary expertise lies in defensive alignment (detecting model failure modes, reward hacking, and safety violations) and offensive capability growth (identifying when a model has plateaued and how to further advance the model's performance). Your role is to **prepare the right historical evidence** so the next Rubric Update stage can make stable, effective rubric changes. ## Background & Context We are training a Large Language Model (LLM) using a Generative Reward Model (GenRM) and Rubric-as-Reward (RaR) paradigm where rewards are derived from dynamic rubrics that can be manipulated as needed. A memory system stores historical: - Batch summary reports - Individual rollout analysis summaries - Past rubric updates and their outcomes/lessons ### Rubric Manipulation Operations - **Create:** Generate new rubric text with appropriate reward score to address missing evaluation criteria or emergent behaviors. - **Update:** Specific textual modification or refinements to existing rubrics to improve clarity or precision. - **Delete:** Remove rubrics that are obsolete, redundant, or causing negative alignment. - **Reweight:** Adjust the scalar reward score associated with a rubric to prioritize or deprioritize specific rubrics. - **Merge:** Combine multiple overlapping or related rubrics into a single, unified criterion. - **Split:** Decompose a complex or multi-faceted rubric into distinct, atomic sub-criteria for granular evaluation. You are allowed to link rubrics logically (e.g., Rule B applies only if Rule A is met) when manipulating rubrics. Note that the Anchor Rubric Set is immutable as they are the fundamental rubrics that prevents policy from drifting too much, thus improving training stability. ### Reward Calculation The overall reward score of an individual rollout is calculated as the sum of all reward weights from each rubric that the criterion is met, unless specified otherwise. ## Task You will be given an Input Data Packet including: - RL Training Metadata (Goal, Model Size, Step Count) - The Current Active Rubrics (Anchor + Adaptive) - The Batch Analysis Report (most recent diagnostic of model behavior) Your task is to generate **0 to N** targeted memory search queries (where **N is adaptive**) to retrieve the most relevant historical logs and past rubric updates for the upcoming Rubric Update stage. ### What the queries must cover (prioritized) When applicable, cover these areas: 1. **Primary hack / risk vector** (e.g., aggregated_reward_hacking.primary_vector) and close paraphrases. 2. **Systemic pattern names** (systemic_patterns[].pattern_name) and “what it looked like”. 3. **Weakest rubrics & issues** (rubric_meta_evaluation.weakest_rubrics[].issue) and short identifying phrases from the rubric_text. 4. **Curriculum advancement targets** (performance_advancement_analysis.next_level_objectives / quality_gap_analysis) if plateau is suspected. 5. **Backfire avoidance queries**: if proposing penalties/tightening, query for cases where similar changes caused regressions (verbosity penalty backfired, over-refusal increased, etc.). Only include this if the Batch Analysis Report suggests a change, hacking, or plateau.

Figure 16: Prompt template for memory query generation (1/3) described in Section 2.3. This prompt generates targeted search queries that are executed against the persistent evaluation memory to retrieve relevant historical context for rubric update.

45

Preprint. Under review.

Memory Query Generation (2/3) ### Query writing rules - Each query should be a standalone search string suitable for either keyword match or semantic search. - Avoid duplicates and near-duplicates. - Prefer concrete terms found in the packet (e.g., “Length Gaming”, “Sycophancy”, “Saturation”, “Too vague rubric”). - If you reference a rubric, include either: - a short exact phrase from the rubric_text (<= 8 words), OR - the most distinctive concept name (e.g., “conciseness”, “hallucination”, “refusal”). - Do not include any conversational text outside JSON. ## Constraints - **Output Format:** You must output only valid JSON. Do not include conversational filler before or after the JSON. - **Scope:** Use only the provided Input Data Packet. Do not hallucinate external context. ## Output Format Return a valid JSON object with the following structure: ```json { "queries": [ "string" ] } ``` The list must contain the appropriate number of query strings, based on the guidance above. ## Input Data Packet ### High-Level Training Goal `[Insert the high-level, ultimate goal the model is being trained to achieve.]` ### RL Training Metadata - **Model Size:** `[e.g., 70B Parameters]` - **Current Step:** `[e.g., 85,000]` - **Total Steps:** `[e.g., 200,000]` - **Other Context:** `[e.g., "PPO phase" / "KTO phase" / etc.]` ### Current Reward Rubrics - **Anchor Rubric Set (Immutable):** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` - **Adaptive Rubric Set:** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` ### Batch Analysis Report

Figure 17: Prompt template for memory query generation (2/3) described in Section 2.3. This prompt generates targeted search queries that are executed against the persistent evaluation memory to retrieve relevant historical context for rubric update.

46

Preprint. Under review.

Memory Query Generation (3/3) `[Insert the JSON output from the Batch Summary step]` Now output ONLY valid JSON matching the schema above.

Figure 18: Prompt template for memory query generation (3/3) described in Section 2.3. This prompt generates targeted search queries that are executed against the persistent evaluation memory to retrieve relevant historical context for rubric update.

47

Preprint. Under review.

Rubric Update (1/4) ## Role You are a chief AI research scientist and an expert in Reinforcement Learning (RL) as applied to Large Language Models. Your primary expertise lies in defensive alignment (detecting model failure model, reward hacking, and safety violations) and offensive capability growth (identifying when a model has plateaued and how to further advance the model's performance). Your role is not merely to judge current performance, but to **steer the evolutionary trajectory** of the model. You control the "laws" (Rubrics) of the environment to ensure the model continuously improves without plateauing or learning deceptive behaviors. ## Background & Context We are training a Large Language Model (LLM) using a Generative Reward Model (GenRM) and Rubric-as-Reward (RaR) paradigm where rewards are derived from dynamic rubrics that can be manipulated as needed. ### Rubric Manipulation Operations - **Create:** Generate new rubric text with appropriate reward score to address missing evaluation criteria or emergent behaviors. - **Update:** Specific textual modification or refinements to existing rubrics to improve clarity or precision. - **Delete:** Remove rubrics that are obsolete, redundant, or causing negative alignment. - **Reweight:** Adjust the scalar reward score associated with a rubric to prioritize or deprioritize specific rubrics. - **Merge:** Combine multiple overlapping or related rubrics into a single, unified criterion. - **Split:** Decompose a complex or multi-faceted rubric into distinct, atomic sub-criteria for granular evaluation. You are allowed to link rubrics logically (e.g., Rule B applies only if Rule A is met) when manipulating rubrics. Note that the Anchor Rubric Set is immutable as they are the fundamental rubrics that prevents policy from drifting too much, thus improving training stability. ### Reward Calculation The overall reward score of an individual rollout is calculated as the sum of all reward weights from each rubric that the criterion is met, unless specified otherwise. ## Task You will be provided with an Input Data Packet including: - **RL Training Metadata** (Goal, Model Size, Step Count). - **The Current Active Rubrics**. - **The Batch Analysis Report** (The most recent diagnostic of model behavior). - **Retrieved Memory Context** (Relevant historical logs, past rubric failures, and long-term trend analysis). Your objective is to **modify the adaptive rubrics** to optimize the next phase of training. You must balance **Stability** (not changing things too fast) with **Adaptation** (fixing loopholes and raising standards). ## Requirements You must categorize your update strategy into one of three modes based on the input data: ### Defensive Mode - **Trigger:** The Batch Report detects "Reward Hacking," "Safety Violations," or "Loophole Abuse." - **Action:** - Immediately patch the specific wording causing the hack. - Introduce negative constraints (penalties). - **Consult Memory:** Check if this hack has appeared before. If yes, apply a stronger penalty than last time. ### Curriculium Advancement

Figure 19: Prompt template for rubric update (1/4) described in Section 2.3. This prompt grounds rubric modifications in the batch analysis report and retrieved historical context from the persistent evaluation memory.

48

Preprint. Under review.

Rubric Update (2/4) - **Trigger:** The Batch Report indicates "Saturation," "Mastery," or "Plateauing" (High rewards, generic outputs). - **Action:** - **Raise the Bar:** Transition from simple constraints (e.g., "No grammar errors") to complex stylistic goals (e.g., "Use Socratic reasoning"). - **Deprioritize Basics:** Lower the weight of basic tasks the model has already mastered to force it to chase the new, harder rewards. ### Maintenance / Stability - **Trigger:** The model is learning well, trends are positive, and no hacking is detected. - **Action:** - Make **minor** or **no** changes. - Drastic changes during a healthy learning phase can destabilize the policy. ### Rubric Set Hygiene (Conciseness & De-duplication) Regardless of update strategy, the **updated rubric set** you output (`final_rubric_set`) must satisfy: - **Concise set:** Keep the rubric set as small as possible. Prefer *fewer, clearer* criteria over many narrow ones. - **No duplicates:** Do **not** include duplicate or near-duplicate rubrics (similar meaning with different wording). If two rubrics overlap materially, use **MERGE** or **DELETE** so only one remains. - **No duplicate anchors:** If an Anchor Rubric already covers a concept (e.g., correctness), do not “re-create” an equivalent rubric in the adaptive set. Anchors should appear **exactly once** in the final rubric set. - **Avoid double-counting:** Do not create multiple rubrics that all reward/penalize the same behavior in the same direction unless you explicitly justify why separate credit assignment is necessary. - **Reward Budget (Hard):** The next-step rubric weights must imply `max_total_reward ≤ +5` and `min_total_reward ≥ −5`. Concretely, across Anchor+Adaptive: Σ positive weights ≤ 5 and Σ negative weights ≥ −5. If over budget, adjust adaptive weights (delete/merge/reweight) until within budget. ## Constraints * **Output Format:** You must output only valid JSON. Do not include conversational filler before or after the JSON. * **Objectivity:** Assess findings probabilistically (e.g., "High confidence of hacking") rather than as absolute binary facts unless obvious. * **Scope:** Base your analysis only on the provided Data Packet. Do not hallucinate external context. ## Output Format You must output a valid JSON object containing the new configuration and the rationale for the memory system. **JSON Schema:** ```json { "rubric_update_id": "string", // A unique identifier, e.g., "step-[step_number]_rubric-update". "update_strategy": "string", // "DEFENSIVE", "CURRICULUM_ADVANCEMENT", or "MAINTENANCE" "memory_consultation": { "relevant_history_utilized": "string", // Reference specific insights from the Retrieved Memory Context that influenced this decision. "avoidance_action": "string" // e.g., "Avoided re-introducing Rule X because Memory Y showed it failed previously." },

Figure 20: Prompt template for rubric update (2/4) described in Section 2.3. This prompt grounds rubric modifications in the batch analysis report and retrieved historical context from the persistent evaluation memory.

49

Preprint. Under review.

Rubric Update (3/4) "rationale": { "diagnosis": "string", // Summary of why changes are needed based on the Batch Report. "intended_effect": "string" // What do you expect the model to learn from these specific changes? }, "rubric_modifications": [ // List ONLY the changes made. { "operation": "string", // "CREATE", "UPDATE", "DELETE", "REWEIGHT", "MERGE", "SPLIT" "target_rubric_id_or_text": "string", // The original rubric being modified (if applicable). "new_rubric_text": "string", // The new text (null if DELETE). You must only include the description (main text) here. Do not include the weight, title, or rubric id here. "new_weight": "float", "reasoning": "string" } ], "final_adaptive_rubric_set": [ // The COMPLETE list of adaptive rubrics to be used for the next step. { "rubric_text": "string", // You must only include the description (main text) here. Do not include the weight, title, or rubric id here. "weight": "float" } ], "memory_artifact": { // What should be stored in the Agentic Memory for future reference? "event_summary": "string", // e.g., "Step 85k: Shifted focus from Accuracy to Conciseness." "lessons_learned": "string" // e.g., "Rubric for 'wit' is prone to sarcasm hacks; tightened constraints." } } ``` ## Input Data Packet ### High-Level Training Goal `[Insert the high-level, ultimate goal the model is being trained to achieve. e.g., "Create a helpful and harmless assistant that can follow complex, multi-step user instructions."]` ### RL Training Metadata - **Model Size:** `[e.g., 70B Parameters]` - **Current Step:** `[e.g., 85,000]` - **Total Steps:** `[e.g., 200,000]` - **Other Context:** `[e.g., "This model is in the PPO phase, focusing on preference optimization." or "This model is in the KTO phase."]` ### Current Reward Rubrics - **Anchor Rubric Set (Immutable):** ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` - **Adaptive Rubric Set:**

Figure 21: Prompt template for rubric update (3/4) described in Section 2.3. This prompt grounds rubric modifications in the batch analysis report and retrieved historical context from the persistent evaluation memory.

50

Preprint. Under review.

Rubric Update (4/4) ``` [Insert the complete, verbatim rubrics used to assign rewards. e.g., - R0: "+1 The model acknowledges the user's premise." - R1: "-2 The model exhibits sycophancy." - R2: "+3 The model correctly identifies and refuses a harmful instruction."] ``` ### Batch Analysis Report `[Insert the JSON output from the Batch Summary step]` ### Retrieved Memory Context `[Insert relevant summaries of previous updates, e.g., "Step 20,000: We removed the 'humor' rubric because it caused hallucination."]` Now output ONLY valid JSON matching the schema above.

Figure 22: Prompt template for rubric update (4/4) described in Section 2.3. This prompt grounds rubric modifications in the batch analysis report and retrieved historical context from the persistent evaluation memory.

51

Preprint. Under review.

H

Use of Large Language Models

Large language models are used in the following capacities as part of the research: (i) Policy model and RL training: Qwen2.5-7B-Instruct serves as the base policy model fine-tuned via GRPO across all experiments. (ii) AMARIS pipeline components: GPT-OSS-120B is used for rubric-based scoring and individual rollout analysis, and GPT-5.1 is used for batch summarization, memory query generation, and rubric update. These usages constitute core experimental components of our methodology and are described in detail in Section 2 and Section 3.3. (iii) Evaluation: LLM-based judges are used for evaluation on benchmarks that require them, such as HealthBench, WritingBench, and Creative Writing v3, following the evaluation protocols established by each respective benchmark. (iv) Additionally, LLMs were used to refine the prompt templates that guide each pipeline stage (Appendix G); the final prompt designs and all experimental results remain the sole responsibility of the authors. Beyond the experiments, we acknowledge the use of large language models in the final stages of manuscript preparation. These tools were employed exclusively for identifying and correcting typographical and grammatical errors, ensuring clarity and precision in the written presentation. Their use was strictly limited to linguistic refinement and did not impact the study’s conceptual framework, research methodology, data analysis, or conclusions. All intellectual contributions and substantive content remain those of the authors.

52