arXiv:2606.27136v1 [cs.AI] 25 Jun 2026
Joint Learning of Experiential Rules and Policies for Large Language Model Agents Shicheng Ye
Chao Yu*
Sun Yat-sen University [email protected]
Sun Yat-sen University [email protected]
These two uses of experience have complementary limitations. Retrieved rules can quickly address recurring local errors by reminding the agent of task preconditions or correcting persistent action biases [8]–[12]. Yet rules do not execute themselves: they help only if the current model can interpret and apply them correctly in context. They can also become stale as the policy changes during training, and rules that were once useful may gradually lose value or even become misleading [13]. Parameter updates can improve behavior more globally, but feedback in complex interactive tasks is often sparse. As a result, some local but consequential mistakes may not be exposed early enough to receive timely and targeted correction [7], [14]. We therefore study whether the same interaction data can support both explicit rule updates and policy optimization. We propose Joint Learning of Experiential Rules and Policies for LLM Agents (JERP), which maintains a dynamically updated long-term experiential-rule pool and uses it together with the interaction history as decision context during each episode. Once an episode ends, the resulting trajectories are used twice: to optimize model parameters and to update the rule pool by comparing current behavior with reference successful trajectories. This allows the rule pool to evolve with the policy while gradually absorbing stable and repeatedly validated behaviors into the model itself.
Abstract—For LLM agents in multi-step interactive environments, a key challenge is to make effective use of accumulated interaction experience. Existing work has typically separated two uses of such experience: keeping it outside the model as naturallanguage rules for later prompting, or using trajectories and feedback to update the model parameters. The former is easy to interpret but can fall out of sync with the evolving policy; the latter improves the policy more broadly but provides only limited correction for local mistakes in sparse-reward settings. We present Joint Learning of Experiential Rules and Policies for LLM Agents (JERP), which updates a long-term experientialrule pool and the policy from the same interaction trajectories. At decision time, JERP retrieves task-relevant rules and conditions the agent on them together with the interaction history. After each episode, it uses the collected trajectories both to optimize the policy and to revise the rule pool by comparing current rollouts with reference successful trajectories. This coupling keeps the rule pool aligned with the evolving policy while allowing stable and effective behaviors to be gradually absorbed into the model itself. Experiments on AlfWorld and WebShop show that JERP yields consistent gains in decision performance for complex interactive tasks. Index Terms—large language model agents, learning from experience, reinforcement learning
I. I NTRODUCTION Large language models (LLMs) have become a natural foundation for general-purpose agents because they combine broad world knowledge with strong in-context learning and language reasoning abilities [1]. In settings such as web navigation [2], embodied interaction [3], [4], and tool use [5], however, solving a task rarely amounts to producing a single correct response. Agents often have to explore, recover from errors, and adjust their strategy over multiple steps before they succeed. In this setting, an important question is how an agent can accumulate experience during interaction and use it to make better decisions on future tasks [6], [7]. Most existing approaches exploit interaction experience in one of two ways. One line of work converts past experience into reusable natural-language knowledge, including reflective texts [8], guiding rules [9], [10], and operation manuals [11]. In these methods, experience is externalized into a form that can be stored, retrieved, and revised, then fed back to the agent as additional context. The other line of work uses trajectories and environmental feedback to update model parameters directly, with the goal of improving the agent’s policy on later tasks [7].
II. R ELATED W ORK A. Experience Reuse through Prompt Engineering A common way to reuse interaction experience is to store it outside the model and feed it back as context in later episodes. Some methods preserve feedback or intermediate summaries after trial and error. For example, Reflexion [8] writes natural-language reflections after failures into episodic memory to guide future attempts. Generative Agents [15], MemoryBank [16], and MemGPT [17] study how such memories can be retained, maintained, and retrieved over longer horizons. Other studies distill interaction experience into reusable skills, rules, or manuals. For example, Voyager [18] accumulates experience and transfers it across tasks through an expanding library of code skills. ExpeL [9], AutoGuide [10], and AutoManual [11] distill multi-task trial-and-error experience into reusable natural-language instructions for contextual guidance in subsequent tasks.
1
Parameter Parameter Parameter Update 交互轨迹组 Update 交互轨迹组 Update 交互轨迹组 Compute gradients Compute gradients Compute gradients based on the lossloss based on the on lossthe based
Parameter Parameter Rule Rule Parameter Rule 交互轨迹组 交互轨迹组 Update Update 交互轨迹组 交互轨迹组 Update Update 交互轨迹组 交互轨迹组 Update Update Compute gradients rules Compute gradients rules and Compute gradientsUpdateUpdate Update rulesand and based on the loss optimize prompts based on the loss prompts based on the loss optimize optimize prompts
RuleRule Rule 交互轨迹组 Update 交互轨迹组 Update 交互轨迹组 Update Update and Update rulesrules and Update rules and optimize prompts optimize prompts optimize prompts
Reward Reward Reward
Strategic Strategic Strategic Insights Insights Insights
Reward Functions Reward Functions Reward Functions Interaction Interaction Interaction
Learned Learned Learned Reward Model Reward Model Reward Model
Generate Generate Generate Trajectories Trajectories Trajectories
Environment Environment Environment Reward Reward Reward Custom Custom Custom Reward Reward Reward
Trajectory Trajectory Trajectory 交互轨迹组 交互轨迹组 交互轨迹组 Group Group Group
{𝜏!{𝜏 , 𝜏!",{𝜏 ,𝜏𝜏"!#,,,𝜏𝜏⋯ ,𝜏𝜏#$, }⋯ 𝜏$,}𝜏$ } #",, ⋯
(a) Prompt optimization paradigm
Interaction Interaction Interaction
Generate Generate Generate Trajectories Trajectories Trajectories
Manuals Manuals or oror Manuals Guidelines Guidelines Guidelines Executable Executable Executable SkillsSkills Skills Reflection Reflection Reflection Analysis Analysis Analysis
Reward Reward Reward
Strategic Strategic Strategic InsightsInsights Insights
Reward Functions Reward Functions Reward Functions Learned Learned Learned Reward Reward ModelModel Reward Model Environment Environment Environment Reward Reward Reward Custom Custom Custom Reward Reward Reward
Interaction Interaction Interaction
Executable Executable Executable Skills Skills Skills Generate Generate Generate Trajectories Trajectories Trajectories Trajectory Trajectory Trajectory 交互轨迹组 交互轨迹组 交互轨迹组 GroupGroup Group
Trajectory Trajectory Trajectory 交互轨迹组 交互轨迹组 交互轨迹组 Group Group Group
{𝜏! , {𝜏 𝜏"!,{𝜏 }, 𝜏$, 𝜏}$ } ,𝜏𝜏#!",, ,⋯ 𝜏𝜏"#,, ,𝜏𝜏⋯ #$, ⋯
Manuals or Manuals or Manuals or Guidelines Guidelines Guidelines
Reflection Reflection Reflection Analysis Analysis Analysis
{𝜏! , 𝜏"{𝜏 ,{𝜏 𝜏!#!, ,𝜏,⋯ }⋯, ,𝜏𝜏$$}} 𝜏"", ,𝜏, 𝜏𝜏#$ #, ,⋯
(b) Model-parameter optimization paradigm
(c) JERP paradigm
Fig. 1. Comparison between JERP and two basic paradigms.
improve the policy through parameter updates, but they do not maintain an explicit rule pool that can be inspected, revised, and reused during later interaction.
Beyond reflection-style memory, prior work has also induced tool-use patterns [19], workflows [20], and reusable skills [21] from historical interactions, and has explored long-term use of experiential knowledge through knowledge bases [22], reflective memory [23], and experience replay [12]. These studies establish that stored experience can improve later decisions when it is organized in a reusable form. Most of them, however, treat rule reuse and parameter learning as separate processes, leaving open how a rule pool should be maintained when the policy using it is also changing.
III. P ROBLEM D EFINITION AND BASIC PARADIGMS We consider interactive tasks in which an LLM agent must choose actions from partial observations over multiple steps. Besides the policy parameters that produce actions, the agent also uses external experiential rules retrieved into the prompt. This section defines the resulting rule-augmented interaction process and introduces the two basic paradigms adopted by the two parts of JERP, namely prompt-based rule learning and group-relative policy optimization.
B. Policy Optimization for LLMs via Reinforcement Learning Fine-Tuning Another line of work uses interaction experience to update the policy directly, turning trajectories, feedback, or preferences into optimization signals. Through reinforcement learning fine-tuning and closely related preference-optimization methods, these signals are used to update the model so that improved behavior is encoded in its parameters [7]. Early representative work such as InstructGPT [24] adopts a threestage pipeline consisting of supervised fine-tuning, rewardmodel training, and PPO optimization, demonstrating that reinforcement learning from human feedback can substantially improve instruction following and output quality. Subsequent studies have explored how to perform reinforcement learning fine-tuning for LLMs more stably and efficiently [25]. In addition to PPO-based methods, GRPO [26] estimates advantages by comparing samples within the same group, reducing the need for a separate value model and the associated memory overhead. DPO [27] rewrites reinforcement learning from human feedback into a simpler direct preference-optimization objective. DMPO [28] extends preference optimization to multi-turn agent tasks. For long-horizon interactive agent tasks, GiGPO [14] proposes a hierarchical group policy optimization method that combines episode-level and step-level relative advantage estimation to mitigate sparse rewards and fine-grained credit assignment. These methods
A. Partially Observable Markov Decision Process Let D denote the task-instance space, with each instance d ∈ D drawn from distribution p (d). For a fixed task instance, the agent interacts with the environment until the task terminates or the step budget is exhausted. Since the agent receives observations rather than full environment states, we model the interaction as a partially observable Markov decision process (POMDP), written as E = (S, A, O, P, Ω, R, γ, T ) ,
(1)
where S, A, and O are the state, action, and observation spaces; P (st+1 | st , at ) and Ω (ot+1 | st+1 , at ) are the transition and observation functions; R (st , at ) is the reward function; γ ∈ [0, 1) is the discount factor; and T is the maximum number of interaction steps. At step t, st is the hidden environment state, ot is the observation available to the agent, and at is the executed action. Since the true state is not directly observed, the policy must condition on the interaction history rather than on st . We write the history at step t as ht = (o0 , a0 , o1 , . . . , at−1 , ot ) , where o0 contains the initial task description.
2
(2)
JERP differs from a standard POMDP policy in that the prompt also contains reusable experience. For task instance d, we maintain a long-term experiential-rule pool K (d), and denote the collection of such pools by K = {K (d)}. Before an episode starts, a size-controlled subset is selected from K (d) e (d). Thus, the policy does not act as the working rule set K from the history alone. Its decision context combines the task, the observed interaction history, and the currently retrieved rules. The input prompt at step t is e (d) . xt = d, ht , K (3)
C. Group-Relative Policy Optimization The other paradigm updates the policy from interaction feedback. For a given task instance d, group-relative methods sample N complete trajectories {τ1 , τ2 , . . . , τN } under the old policy πθold and assign each trajectory a scalar reward R (τi ). Instead of relying on an additional value model, they construct an advantage signal by comparing rewards within the sampled group. A (τi ) = GroupComputation {R (τj )}N (8) j=1 . For example, GRPO [26] normalizes each trajectory reward by the mean and standard deviation of rewards in the same group. This style of optimization is suitable for our setting because it uses complete task outcomes and avoids training a separate value-function estimator. In JERP, the same sampled trajectory group is used not only for this policy update but also for updating the experiential-rule pool, which is the main departure from a pure parameter-optimization paradigm.
The parameterized policy then samples an action by at ∼ πθ (· | xt ) .
(4)
After executing at , the environment returns reward rt = R (st , at ), next observation ot+1 , and termination indicator zt+1 ∈ {0, 1}. For a complete trajectory τ , the discounted return is T −1 X G (τ ) = γ t rt . (5)
IV. J OINT L EARNING OF E XPERIENTIAL RULES AND P OLICIES We propose Joint Learning of Experiential Rules and Policies for LLM Agents (JERP), which updates policy parameters and the experiential-rule pool in the same training loop. In each episode, the current policy acts with a working rule set selected from the long-term rule pool. The sampled trajectories are then used for both a group-relative policy update and an experiential-rule-pool update. Fig. 1 compares promptengineering-based experiential-rule learning, group-based reinforcement learning for policy optimization, and the overall structure of JERP.
t=0
The learning problem is therefore not only to find better policy parameters. The rule pools also affect the trajectory distribution because they enter the prompt before each action. We define the training objective over both θ and K as h i (6) J (θ, K) = Ed∼p(d) Eτ ∼pθ (τ |d,K) G (τ ) , where pθ (τ | d, K) is the trajectory distribution induced by πθ when working rules are selected from the current rule pools. This notation makes the coupling explicit. Changing θ changes how the agent interprets the prompt, while changing K changes the prompt from which the same policy acts.
A. Overall Framework This work maintains experiential rules for each task. For a given task instance d, its corresponding long-term experientialrule pool is |K(d)| K (d) = {(zi , si )}i=1 , (9)
B. Prompt-Based Experiential-Rule Learning One way to use experience is to keep it outside the model. Given trajectory samples {e1 , e2 , . . . , eN } from related tasks, prior prompt-based methods ask an LLM to summarize reusable behavioral guidance and store the result as naturallanguage rules. Abstractly, K = ExtractRules {ei }N (7) i=1 .
where zi is the textual content of the i-th experiential rule, and si is its utility score. The rule text records guidance information in natural language, while the utility score measures the rule’s reference value for future decisions. At the beginning of each episode, the system selects a subset from the current long-term experiential-rule pool K (d) as the e (d), and provides it to the policy model working rule set K together with the task description and current interaction history. The LLM agent then interacts with the environment on task instance d and generates a trajectory group Td . The same trajectory group is used to update model parameters according to relative trajectory performance and to update the long-term experiential-rule pool by contrasting current trajectories with reference successful trajectories. The training process can be summarized as e (d) , Td ∼ πθ · | d, K (10)
This paradigm is useful because the stored experience remains inspectable and editable, and can be revised without changing model parameters. Its limitation is equally important for our setting. The rule set is only effective through the current policy’s ability to read and apply it. If the policy evolves during training while the rules remain static, the two may become poorly matched. This motivates treating rule maintenance as part of the training process rather than as a separate preprocessing step.
3
θ′ = Updateθ (θ, Td ) , ′
K (d) = UpdateK K (d) , Td , T
(11) +
(d) .
construct group-relative advantages. Let the mean and standard deviation of trajectory rewards in the group be v u N N u1 X 2 1 X R (τj ) , σd = t R (τj ) − µd . µd = N j=1 N j=1 (17) The group-relative advantage of the i-th trajectory is then defined as R (τi ) − µd , (18) Ai = σd + δ
(12)
where parameter updates are driven by the trajectory group Td , while rule updates combine Td with the reference successful trajectory set T + (d). The overall framework is shown in Fig. 2. B. Trajectory Sampling Because the rule pool K (d) for task d expands during training, we do not feed all rules into the model in each episode. Instead, we select a size-controlled subset of rules for the current decision process. Since the working rules are already organized by task, the current implementation does not perform further instance-level relevance retrieval. The system sorts rules in K (d) by utility score in descending order and selects the top k rules as the input for the current episode, e (d) = topk K (d) , k . K (13)
where δ is a numerical stability constant that prevents division by zero. Following the advantage modeling strategy of GiGPO [14] for multi-step tasks, all time steps in the same trajectory share the same advantage, Ai,t ≡ Ai . (19) On this basis, the parameter-update objective is defined as N
|τi |−1
1 X 1 X ℓi,t (θ) , LGRPO (θ) = − N i=1 |τi | t=0
If the rule pool contains fewer than k rules, all rules are used. This score-based selection is an implementation choice; we do not claim it is generally superior to finer-grained instancelevel retrieval. Within a single episode, the working rule set e (d) remains fixed, avoiding context shifts caused by timeK step-level rule reselection. Rules in the working set are sorted by score and organized into the input prompt using a fixed template. Together with the task description and interaction history, they form the decision basis at the current time step. At time step t, the LLM policy defined in (4) generates the action as
where the per-step surrogate term is ℓi,t (θ) = min ρi,t (θ) Ai,t , ρ̄i,t (θ) Ai,t − βDKL πθ (· | xi,t ) ∥ πref (· | xi,t ) ,
(20)
(21)
and ρ̄i,t (θ) = clip ρi,t (θ) , 1 − ϵ, 1 + ϵ . with ρi,t (θ) =
at ∼ πθ (· | xt ) , (14) e (d) denotes the input prompt at the where xt = d, ht , K current time step.
πθ (ai,t | xi,t ) . πθold (ai,t | xi,t )
(22) (23)
Here, ai,t denotes the action generated at time step t of trajectory τi ; xi,t is the corresponding input prompt; |τi | is the number of environment interaction steps in τi ; ϵ is the clipping coefficient; β is the coefficient of KL regularization; and πref denotes the reference policy. This stage updates model parameters according to the relative performance of trajectories under the given task instance and working rule set.
C. JERP Optimization Stage I: Model-Parameter Update For a fixed task instance d, we sample a trajectory group {τi }N i=1 from the old policy πθold . The i-th trajectory τi denotes a complete task-solving interaction episode. At time step t, its corresponding input prompt is e (d) . xi,t = d, hi,t , K (15)
D. JERP Optimization Stage II: Experiential-Rule Updating via Contrastive Reflection In the experiential-rule updating stage, the long-term experiential-rule pool is maintained rather than only expanded. Given the current long-term experiential-rule pool, the trajectory group sampled in the current episode, and reference successful trajectories for the same task, the update needs to extract reusable experience, revise inaccurate or weak rules, merge semantically similar and functionally overlapping rules, and keep the rule pool within its capacity limit. Because the content of the rule pool is represented in natural language, rule addition and revision are difficult to formulate as a standard numerical optimization problem. We adopt a structured update method based on contrastive reflection. The system generates rule-level edit operations from
The agent generates action ai,t conditioned on this input and receives environment feedback until the trajectory terminates. In our setting, task rewards are returned only when a trajectory terminates. Therefore, for trajectory τi , its trajectory-level reward is (i) R (τi ) = r|τi | , (16) (i)
where r|τi | is the environment reward obtained by τi at termination. For the same task instance d, we standardize the trajectorylevel rewards R (τi ) within the sampled group {τi }N i=1 and
4
Prompt Input Long-Term Experiential-Rule Pool 1. When performing tasks that require inspecting objects, it is necessary to use a light source, such as a desk lamp, to ensure that the environment is not dark. 2. After observing an object in a location, if interaction fails…
Retrieval 1. When performing tasks that require inspecting objects, it is necessary to use a light source, such as a desk lamp, to ensure that the environment is not dark.
Trajectory Group
{𝜏! , 𝜏" , 𝜏# , ⋯ , 𝜏$ }
Compute Advantages Estimate advantages from trajectory rewards
{𝐴! , 𝐴" , 𝐴# , ⋯ , 𝐴$ }
Task & Current Interaction History
Working Rule Set
+
Task: look at bowl under the desklamp. Observation 0: You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1…
Trajectory Group Model Parameter Updating Compute the GRPO loss: ℒ!"#$
Input Prompt Policy Update the trajectory with new observations
Env
Input
{𝜏! , 𝜏" , 𝜏# , ⋯ , 𝜏$ } Input Long-Term Experiential-Rule Pool 1. When performing tasks that require inspecting objects, it is necessary to use a light source, such as a desk lamp, to ensure Update that the environment is not dark. 2. After observing an object in a location, if interaction fails…
Update parameters via gradient ascent:
𝜃 ← 𝜃 + ∇% ℒ&'()
Action
Experiential Rule Analysis Input: Successful & Failed Trajectories + Long-term Experiential-Rule Pool Output: Experiential Rule Editing Operations Rule-level edit operation Experiential-Rule Updating ADD/UPDATE/UPVOTE/ DOWNVOTE/MERGE
Fig. 2. Overview of the JERP framework.
the current rule pool, current trajectory group, and reference successful trajectories, and then obtains the updated longterm experiential-rule pool. This idea is close to ExpeL [9], which maintains natural-language rules through experiential reflection; both approaches prompt an LLM to generate ruleupdate operations. We further introduce a merge operation to combine semantically similar and functionally overlapping rules when the long-term experiential-rule pool expands during online training. For task instance d, let its long-term experiential-rule pool be K (d) and let the trajectory group sampled in the current episode be Td = {τi }N (24) i=1 .
a structured list of rule-edit operations from the current rule pool, current trajectory group, and reference successful trajectories. The output is constrained to a fixed edit format so that it can be parsed and applied automatically. To balance result parseability and online maintenance cost, we use a small and fixed structured edit syntax to represent rule-pool updates. The system generates five types of primitive operations, ADD(z), EDIT(q, z), UPVOTE(q), DOWNVOTE(q), and MERGE(Q, z). Here, q denotes the identifier of a target rule, Q denotes a set of rule identifiers to be merged, and z denotes newly generated rule text. In ADD and EDIT, z corresponds to the newly added rule text and the revised rule text, respectively. In MERGE, z corresponds to the merged rule text. These five operations correspond to supplementation, revision, reinforcement, weakening, and compression in rulepool maintenance. Specifically, ADD introduces supplementary experience, EDIT modifies existing rules, UPVOTE and DOWNVOTE increase or decrease the utility score of a target rule, and MERGE combines semantically close and functionally overlapping rules into one rule. After generating structured edit operations, the system first edits rule texts according to the operation list generated in the current episode, K (d) ← Apply K (d) , O (d) . (27)
The reference successful trajectory set for the same task instance is + T + (d) = {τ1+ , τ2+ , . . . , τM }, (25) where T + (d) denotes the successful trajectories accumulated for the same task during training, and M = |T + (d)| denotes the number of reference successful trajectories for task instance d. We do not use human demonstrations or successful trajectories from the test stage. If no successful trajectory has been accumulated for the current task instance, we set T + (d) = ∅; in this case, rule-pool updating relies only on the current rule pool and current trajectory group. Based on these inputs, the system generates a set of rulelevel edit operations, O (d) = ReflectAndEdit K (d) , Td , T + (d) , (26)
The system then updates the utility score of each rule to reflect its future retention value. We adopt a fixed-step update rule. For ADD, the new rule is assigned an initial score s0 > 0. For UPVOTE and DOWNVOTE, the target rule score is adjusted by fixed increments δ+ and δ− , respectively. For EDIT, the edited rule retains its original score. For MERGE, the score
where ReflectAndEdit(·) denotes the rule-update operator based on contrastive reflection. In our implementation, this operator prompts an LLM with frozen parameters to generate
5
Algorithm 1 JERP Training Procedure Input: Training task set D, initial policy parameters θ, working rule set size k, trajectory group size N , maximum interaction steps T , training steps E. 1: for each task d ∈ D do 2: Initialize long-term experiential-rule pool K(d) ← ∅ 3: Initialize reference successful trajectory set T + (d) ← ∅ 4: end for 5: for e = 1, 2, . . . , E do 6: Sample task instance d ∈ D 7: Set θold ← θ e 8: Construct working rule set K(d) ← topk(K(d), k) 9: Initialize trajectory group Td ← ∅ 10: for i = 1, 2, . . . , N do 11: Initialize τi ← ∅ and hi,0 ← {oi,0 } 12: for t = 0, 1, . . . , T − 1 do e 13: Set xi,t ← (d, hi,t , K(d)) e 14: Sample action ai,t ∼ πθold (· | d, hi,t , K(d)) 15: Execute ai,t and update hi,t+1 16: τi ← τi ∪ {(xi,t , ai,t , hi,t+1 )} 17: if τi terminates then 18: break 19: end if 20: end for 21: Compute trajectory reward R(τi ) 22: Td ← Td ∪ {τi } 23: end for 24: Compute group-relative advantages from trajectory rewards 25: Update θ using the GRPO objective 26: O(d) ← ReflectAndEdit(K(d), Td , T + (d)) 27: Apply rule updates K(d) ← Apply(K(d), O(d)) 28: Update rule scores and prune K(d) under the capacity limit 29: T + (d) ← T + (d) ∪ {τi ∈ Td | R(τi ) > 0} 30: end for 31: return θ and {K(d)}d∈D
We then examine the effect of continual experiential-rule-pool updating through an ablation experiment. A. Environments and Evaluation Metrics We evaluate JERP on two interactive environments, AlfWorld [29] and WebShop [30]. AlfWorld is a text-based household manipulation environment. We use its six task types, pick and place, pick clean then place, pick heat then place, pick cool then place, look at obj, and pick two obj, abbreviated as Pick, Clean, Heat, Cool, Look, and Pick2. The environment is partially observable and requires multi-step interaction. WebShop is an interactive environment for online shopping tasks. Given a user’s product requirements, the agent performs search, browsing, comparison, and purchase actions through Web-page interactions. We use environment-specific evaluation metrics. For AlfWorld, we report task success rate (%). For WebShop, we report average score and task success rate (%). B. Baseline Methods We compare JERP with five baselines, Vanilla LLM, ReAct [31], Reflexion [8], RLOO [32], and GRPO [26], covering direct prompting, online reasoning, reflection-style memory reuse, and group-relative policy optimization. Vanilla LLM directly prompts the base model to act without reasoning scaffolding, cross-episode memory, or parameter updates. ReAct alternates between reasoning and action during interaction. It relies on observations and context from the current episode, without cross-episode memory or parameter updates. Reflexion writes natural-language reflections from prior attempts into memory to guide later decisions. Compared with ReAct, it reuses cross-episode experience, but this experience remains in external memory rather than in model parameters. RLOO updates policy parameters with a REINFORCE-style objective, using a leave-one-out baseline computed from the other trajectories in the same sampled group to estimate the advantage. GRPO updates policy parameters through grouprelative trajectory comparison without relying on an independent value model. Unlike ReAct and Reflexion, RLOO and GRPO improve later decisions through training rather than through test-time reasoning or memory alone.
of the merged rule is set to the maximum score among the original rules. If a rule’s updated score is below a predefined threshold, it is removed from the rule pool. Through these edit and score updates, the long-term experiential-rule pool incorporates useful experience from the current trajectory group while removing or merging rules with lower retention value. Algorithm 1 summarizes the full training process, including trajectory sampling, parameter updating, and rule-pool updating.
TABLE I H YPERPARAMETER CONFIGURATIONS OF JERP IN DIFFERENT BENCHMARK ENVIRONMENTS . Hyperparameter Learning rate Group sampling size Reward discount factor Training epochs Batch size LoRA rank LoRA scaling factor Maximum steps per episode
V. E XPERIMENTS This section presents the experimental evaluation on two interactive benchmarks and analyzes where the gains of JERP come from. We first compare JERP with baselines that use online reasoning, reflection-style memory, or policy optimization.
6
AlfWorld
WebShop
3e-6 8 0.95 200 256 64 64 50
3e-6 8 0.95 300 64 64 64 15
TABLE II P ERFORMANCE COMPARISON BETWEEN JERP AND BASELINE METHODS ON TWO BENCHMARK ENVIRONMENTS .
Method Vanilla LLM ReAct Reflexion RLOO(+LoRA) GRPO(+LoRA) JERP(+LoRA) a b
Pick
Look
AlfWorlda Clean Heat Cool
5.9 17.4 35.3 71.5 78.5 72.2
5.5 20.5 22.2 68.3 73.3 69.8
3.3 15.7 21.7 61.2 50.7 65.4
9.7 6.2 13.6 34.4 62.7 67.4
4.2 7.7 19.4 41.0 51.7 60.1
Pick2
All
WebShopb Score Success Rate
0.0 2.0 3.7 19.9 33.9 42.5
4.1 12.8 21.8 48.7 57.8 61.5
23.1 40.1 55.8 71.9 78.1 79.0
5.2 11.3 21.9 57.8 56.2 64.1
AlfWorld uses task success rate (%) as the evaluation metric. WebShop uses average score and task success rate (%) as evaluation metrics.
rules can preserve effective operation patterns observed in earlier episodes. These results suggest that rule-pool updating is most helpful when the task exposes recurring intermediate mistakes. Parameter updates still provide the main policy improvement, while the rule pool keeps task-specific corrections available during later trajectory sampling.
C. Implementation Details All experiments are conducted on a server running Ubuntu 22.04 LTS with two Intel Xeon Gold 6226R processors, approximately 512 GB of memory, and four NVIDIA A30 GPUs. For GRPO and JERP, which require reinforcement learning training, we use LoRA [33] for parameter-efficient fine-tuning. Unless otherwise specified, all reported results are averaged over three independent runs on the same platform. The key hyperparameters are shown in Table I.
E. Ablation on the Key Mechanism To analyze the role of dynamic experiential-rule-pool updating, we construct an ablated variant that pauses rule-pool updates. This variant uses the same training pipeline as JERP, but updates the rule pool only once at the initial training stage. The rule-pool content is then fixed and no longer supplemented or revised according to later interaction trajectories. The results are shown in Fig. 3. The complete JERP maintains a higher task success rate throughout training. The ablated variant still benefits from rules formed at the initial stage, but its improvement slows after the rule pool is fixed. As the policy changes, later trajectories expose different errors and useful behavior patterns. A fixed rule pool cannot absorb these later observations, whereas JERP continues to revise the rule pool during training.
D. Comparison with Baselines Table II and Fig. 4 show the performance differences between JERP and the baselines. Vanilla LLM performs poorly on both environments (4.1% success rate on AlfWorld and 5.2% on WebShop), confirming that direct prompting of the base model is insufficient for these multi-step tasks and that the gains of the other methods come from how they use interaction experience. On AlfWorld, JERP obtains the best overall success rate at 61.5%, above GRPO at 57.8% and RLOO at 48.7%. The gains over GRPO are concentrated on Clean, Heat, Cool, and Pick2, while GRPO remains higher on Pick and Look. On WebShop, JERP also performs best, with the highest average score (79.0) and a success rate of 64.1%, clearly above both RLOO (57.8%) and GRPO (56.2%). The comparison also separates the roles of different forms of experience use. ReAct mainly relies on context within the current episode and does not reuse cross-episode experience. Reflexion adds language-based memory, but the stored reflections remain outside parameter learning. RLOO and GRPO improve the policy through reinforcement learning and are much stronger than ReAct and Reflexion, but they use trajectories only for parameter updates. JERP adds a second use of the same trajectories by maintaining the experientialrule pool during training, which is consistent with its stronger results on the more constraint-heavy AlfWorld categories and on WebShop success rate. The training curves in Fig. 4 show a similar pattern. JERP and GRPO are close at the early stage, but JERP improves more clearly in the middle and late stages on Heat, Cool, Clean, and Pick2. These tasks usually require longer operation sequences and more intermediate constraints, where reusable
Ablated Varitant of JERP Alfworld
JERP
0.7
Success Rate
0.6 0.5 0.4 0.3 0.2 0.1 0.0
0
25
50
75
100
125
150
175
200
Training Steps
Fig. 3. Success-rate curves of JERP and its ablated variant on AlfWorld over training steps.
7
GRPO