Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments
Yuxin Chen1,2,∗ , Xiaodong Cai2,3,∗ , Junfeng Fang1 , Zhuowen Han2,4 , Yu Wang2,5 , Yaorui Shi2,5 , Yi Zhang2,5 , Qi Gu2,† , Xunliang Cai2 , Xiang Wang5 , An Zhang5,† , Tat-Seng Chua1 National University of Singapore, 2 Meituan, 3 Tsinghua University, 4 Tianjin University, 5 University of Science and Technology of China ∗ Equal contribution. † Corresponding authors: [email protected], [email protected]
arXiv:2605.27209v1 [cs.AI] 26 May 2026
1
Abstract Recent advances in large language models (LLMs) have facilitated the widespread deployment of LLMs as interactive agents capable of reasoning, planning, and tool use. Despite strong performance on existing benchmarks, such agents often exhibit notable degradation when deployed in real-world settings, where environments are inherently stochastic and imperfect. We argue that this discrepancy arises from a fundamental mismatch between idealized training settings and real-world interaction dynamics, where current paradigms rely on carefully curated task instructions and stable, well-controlled environments. To address this gap, we propose NoisyAgent, an agentic training framework that explicitly incorporates environmental imperfections into the agent learning process. We identify two major sources of interaction noise in real-world scenarios: user noise, which captures ambiguity and variability in user interaction, and tool noise, which reflects failures and anomalies in tool execution. We introduce such perturbations into the training pipeline by modifying user interaction patterns and simulating tool execution results within the training environment. To stabilize training while encouraging agents to handle increasingly challenging imperfection, noise is applied to only a subset of rollouts and progressively increased in difficulty as the model adapts to the current noise level. Extensive experiments demonstrate that our approach consistently improves agent robustness under noisy and dynamic environments. Our analysis reveals that training under noise condition also yields performance gains on idealized benchmarks, suggesting that controlled exposure to environmental noise promotes more generalizable reasoning and decision-making behaviors. Our findings highlight the importance of modeling interaction imperfections for bridging the gap between agent training and real-world deployment.
1
Introduction
Recent advances in large language models (LLMs) have transformed them from passive text generators into interactive agents capable of reasoning, planning, and tool use [1–3], enabling their widespread deployment in real-world applications. As these capabilities continue to improve [4–6], LLM agents have achieved strong performance across a wide range of benchmarks [7–9]. However, this success does not consistently transfer to more realistic settings: when confronted with complex and dynamic environments, many agents exhibit notable performance degradation [10–12]. Preprint.
Figure 1: Overview of NoisyAgent. We inject structured perturbations into both user instructions and tool responses to simulate real-world imperfections. Training is conducted via hybrid rollouts that combine clean and noisy trajectories, together with an adaptive scheduler that increases noise difficulty based on the performance gap ∆. Policy optimization is performed with group-wise normalization to stabilize learning under heterogeneous interaction conditions.
We argue that current agent learning paradigms exhibit a fundamental gap between training conditions and real-world deployment. A common characteristic shared by existing agent training paradigms is their reliance on idealized assumptions, where agents are trained with carefully curated instructions and interact with stable, well-controlled environments [13–15]. In contrast, real-world environments are inherently stochastic and imperfect. Users often exhibit diverse interaction styles and unpredictable behaviors [16–18], while external tools may return noisy, incomplete, or even failed outputs due to various uncontrollable factors [19, 20]. This discrepancy between training conditions and deployment environments limits the robustness of current agents, often leading to degraded performance in practical applications [21–23]. Inspired by the success of stochastic perturbations in reinforcement learning [24–26], we argue that agent robustness emerges from exposure to diverse imperfections in learning process. Rather than relying on idealized training settings and expecting agents to adapt post hoc, we explicitly incorporate environmental noise and uncertainty into the agentic training process. However, how to model and introduce such noise in agentic training remains underexplored, and naively injecting noise into the training environment can easily destabilize training dynamics, making it a non-trivial challenge. Toward this goal, we propose NoisyAgent, an agentic RL method for training under noisy environments. We begin by identifying representative forms of real-world noise and developing an automated pipeline to incorporate such imperfections into the training process. Concretely, we consider two major sources of interaction noise in real-world agent scenarios: user noise, which captures ambiguity and variability in user interactions, and tool noise, which simulates execution anomalies from external tools. These perturbations are introduced by modifying user instructions and simulating tool execution results within the training environment, with perturbations applied to only a subset of rollouts for each task. Training follows a curriculum schedule. Starting from mild perturbations, we progressively increase the difficulty and ratio of noise as the model exhibits sufficient robustness at each stage. Robustness is quantified by the performance gap between idealized and perturbed environments on the same tasks. This adaptive process ensures that training remains informative rather than overwhelming, while avoiding inefficient exploration of excessively noisy regimes. Benefiting from our noise-aware training, agents achieve improved performance on benchmarks augmented with real-world noise, indicating enhanced robustness under imperfect and dynamic environments. Interestingly, we also observe consistent gains on standard, idealized benchmarks. 2
We hypothesize that appropriately designed noise introduces controlled instability into the training environment and promotes more generalizable reasoning and decision-making. In particular, exposure to noisy and uncertain interactions encourages agents to recover from errors, resolve ambiguities, and adapt to unexpected outcomes. From this perspective, noise serves as a form of implicit difficulty augmentation, enriching the training distribution and improving robustness beyond idealized settings. Overall, our contributions can be concluded as follows: • We identify a fundamental gap between idealized agent training and real-world deployment, highlighting the importance of modeling environmental uncertainty for robust agent learning. • We develop a noise-aware training framework that systematically incorporates instruction and tool perturbations into the training environment. • Extensive experiments demonstrate that our approach consistently improves agent robustness under noisy and dynamic environments, while also yielding performance gains on standard benchmarks.
2
Preliminary
2.1
Agentic Reinforcement Learning
In representative agentic training paradigm, each taks can be formalized as a Partially Observable Markov Decision Process (POMDP) [27]: M = (S, A, O, T , R).
(1)
At each step t, the agent maintains a state st = (senv t , ht , q) ∈ S, which captures the environment state senv , the interaction history h , and the task prompt q. Based on the current observation ot ∈ O, t t the agent selects an action at ∈ A, where the action space A = Auser ∪ Atool includes both user interaction and tool calling invocations. Correspondingly, the observation space O = Ouser ∪ Otool consists of user-side feedback and tool execution results. Upon taking action at , the environment states evolves according to the transition function T : S ×A → S ×O, producing the next observation ot+1 . The hP training i objective is to learn a policy πθ that maximizes the expected cumulative reward T Eτ ∼πθ r t=0 t over trajectories τ = (o0 , a0 , o1 , a1 , . . . , oT ). A widely adopted training paradigm is Reinforcement Learning with Verifiable Rewards (RLVR) [28, 29], where a verifier evaluates whether the final environment state senv T or the full trajectory τ satisfies the task instruction given rubrics, providing a scalar reward at the trajectory level. To optimize the policy, a representative approach is Group Relative Policy Optimization (GRPO) [30], which extends PPO [31] by computing advantages relative to a group of sampled rollouts. Concretely, given a task prompt q and G sampled trajectories {τ1 , . . . , τG }, the advantage of each trajectory is computed as Âi = (ri − µ)/σ, where µ and σ are the mean and standard deviation of the group rewards. The objective can be written as: " # Li G 1 X 1 X JGRPO (θ) = Eq min ρi,t Âi , clip(ρi,t , 1±ϵ) Âi . (2) G i=1 Li t=1 π (a
|h
)
i,t where ρi,t = πoldθ (ai,t and Li is the length of trajectory τi . Building on this standard optimization i,t |hi,t ) paradigm, effective agentic training relies on access to a diverse set of interactive environments that support both user-agent interaction and tool-grounded execution [32, 33].
2.2
Scaling Environment for Agentic Training
Constructing interactive environments manually for agentic training is costly and difficult to scale. Recent work addresses this challenge by synthesizing executable environments from high-level domain specifications in a fully automated environment scaling pipeline [34]. Given a domain definition, the pipeline initializes a domain-specific tool set together with a unified database schema, forming a structured domain graph D that serves as the foundation for executable environment generation. By sampling from this graph, each training environment can be instantiated as consisting of two tightly coupled components: a user-side construction that specifies task objectives and interaction patterns, and a tool-side construction that defines environment dynamics. 3
On the user side, tasks are synthesized by sampling tool chains from the domain graph and generating corresponding task queries together with interaction patterns, resulting in compositional objectives that specify both what to solve and how the user agent interacts within the environment. Formally, the user-side construction can be expressed as: (q, uint ) = fuser (D),
(3)
where q is the task prompt and πint denotes the interaction pattern governing user-agent interactions. fuser denote simplified abstractions of user-side construction processes. On the tool side, complete executable environments are constructed by implementing structured tool APIs and underlying environment databases based on the domain graph. The sampled tool chains are instantiated as reference executions, and the tool set is further expanded along the domain graph while ensuring both correctness and verifiability of the execution process. Formally, the tool-side construction can be written as: E = ftool (D, q, uint ), (4) where E defines the executable environment grounded in the task specification, including tool APIs, valid state transitions, and verifiable execution paths. ftool denote simplified abstractions of tool-side construction processes. While this design enables scalable and reliable task construction, it assumes that both components are well-specified: user interactions are restricted to be clear and helpful, while tool behaviors are stable. As a result, the resulting training environments are often idealized, leading to a mismatch between training and deployment, where real-world environments are inherently imperfect.
3
Methodology
To bridge the gap between idealized training and noisy deployment, we propose NoisyAgent, an agentic training framework that explicitly incorporates environmental imperfections into learning. We first introduce an automatic noise injection pipeline (Section 3.1) that augments training with user- and tool-side perturbations, and then present an adaptive training strategy (Section 3.2) that progressively adjusts noise difficulty to ensure stable and effective learning. 3.1
Automatic Noise Injection
We systematically analyze common real-world noise and design an automated pipeline to explicitly incorporate these imperfections into any synthesized agentic training environment. Concretely, we consider two major sources of interaction noise in real-world agentic scenarios: user-side noise, which captures ambiguity and variability in user interaction patterns, and tool-side noise, which reflects failures and anomalies in external tool execution. To model such imperfections, we introduce a noise generator πnoise that stochastically perturbs the agent–environment interaction at each step to simulate imperfect observation from the real world. User-side Injection. On the user side, noise is injected before the task starts by modifying the interaction patterns specified by the user. We simulate representative non-ideal interaction patterns observed in real-world scenarios, including: (1) Ambiguous, where user intent is underspecified; (2) Inconsistent, where user needs change or conflict over time; and (3) Redundant, where irrelevant or unnecessary information is included. Formally, given the interaction pattern uint defined by any environment scaling pipeline, the injection of user-side noise can be expressed as: ũint = πnoise (uint ),
(5)
where ũint denotes the perturbed counterpart. This transformation introduces additional variability and ambiguity into user–agent interactions. To avoid inducing unreliable or misleading reward signals, we preserve the underlying task objective q, ensuring that the injected perturbations do not invalidate task solvability, but instead increase the difficulty and stochasticity of the interaction process. Tool-side Injection. Tool-side noise is injected during agent rollouts by randomly perturbing a subset of tool execution results to simulate stochasticity in real-world environments. Specifically, we model common execution anomalies in real-world systems, including: (1) Failures, where tool requests return errors; (2) Incomplete, where outputs are truncated; (3) Misleading, where responses 4
contain incorrect or inconsistent information; and (4) Redundant, where outputs include unnecessary details. Formally, the injection of tool-side noise can be formulated as: õt = πnoise (ot ),
(6)
where ot denotes the original tool response and õt is the perturbed output. This process simulates imperfect tool behaviors while maintaining executable interaction dynamics. 3.2
Adaptive Noise Training
Hybrid Training. The proposed automatic noise injection pipeline enables the incorporation of imperfections into agent training process. However, agent learning is highly sensitive to both task instructions and environment feedback, naively injecting uncontrolled noise can destabilize training dynamics. To preserve training stability while improving robustness, we adopt a hybrid training scheme that combines idealized and perturbed environments. Concretely, under the GRPO training paradigm, given a task set Q, we sample a task q ∈ Q and perform N independent rollouts in parallel environments. Among these, a subset of Nnoise rollouts are perturbed by injecting user-side or tool-side noise with a controllable difficulty level, while the remaining N − Nnoise rollouts are conducted in clean, idealized environments. Formally, let Tclean and Tnoise denote the sets of clean and perturbed trajectories for a given task q, respectively. In our setting, rollouts are partitioned into these two groups, and we modify the standard GRPO objective by computing advantages separately within each group while optimizing over their union. The overall objective is defined as: Lj Li X 1 X X 1 X 1 clean noise J (θ) = Eq Li,t (Âi ) + Lj,t (Âj ) , (7) G Li t=1 Lj t=1 i∈Tclean
j∈Tnoise
where Lk,t (Â) = min ρk,t Â, clip(ρk,t , 1 ± ϵ)Â ,
ρk,t =
πθ (ak,t | hk,t ) . πold (ak,t | hk,t )
(8)
The advantages are computed separately within each group: Âclean = i
ri − µclean , σclean
Ânoise = j
rj − µnoise , σnoise
(9)
where µclean , σclean and µnoise , σnoise denote the mean and standard deviation of rewards computed within each group. This group-wise normalization prevents the dominance of either clean or noisy rollouts during optimization, and stabilizes training under heterogeneous interaction conditions. Noise Scheduling. To adaptively introduce noise while maintaining training stability, we first quantify the model’s robustness to different noise types and adjust the noise level accordingly. We measure the model’s robustness to a specific noise type via the performance gap between clean and perturbed rollouts on the same task: ∆ = Eτ ∼Tclean [1(r(τ ) = 1)] − Eτ ∼Tnoise [1(r(τ ) = 1)],
(10)
where r(τ ) = 1 indicates successful task completion. This gap reflects the extent to which current noise degrades task performance. Based on this measure, we adopt a progressive noise scheduling strategy. Training is initialized in fully idealized environments, with noise gradually introduced as the model adapts. At each stage, we control two factors: (i) the noise scale, defined as the proportion of perturbed rollouts ρ = Nnoise /N ; and (ii) the noise difficulty, characterized by the frequency of tool-side perturbations and the severity of user-side interaction anomalies. When ∆ < θ, with θ denoting a predefined threshold, the model is considered to have adapted to the current noise level, and we increase both the difficulty and the proportion of that noise type. This yields a curriculum over noise, progressively increasing interaction complexity while maintaining training stability. 5
Table 1: Main results under the noisy setting on AgentNoiseBench. We report Avg@4 and Pass@4 averaged across four runs. Best results are in bold, and second-best are underlined. AgentNoiseBench-τ 2 Method
Retail
Airline
AgentNoiseBench-Vita Telecom
Delivery
In-Store
OTA
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Qwen3-8B + GRPO + DAPO + GSPO + Ours
24.12 30.48 29.39 31.80 36.40
44.74 50.88 53.51 54.39 61.40
23.00 33.50 31.00 32.50 38.00
42.00 54.00 50.00 52.00 56.00
21.05 31.58 34.21 34.43 38.38
41.23 53.51 57.89 56.14 64.91
11.75 15.25 15.75 16.00 21.50
18.00 24.00 25.00 26.00 34.00
8.50 14.25 12.75 15.00 16.25
12.00 23.00 19.00 22.00 25.00
0.75 2.50 2.25 2.75 4.75
2.00 4.00 4.00 5.00 8.00
Qwen3-32B + GRPO + DAPO + GSPO + Ours
31.14 38.16 36.18 37.72 43.20
52.63 61.40 57.89 60.53 65.79
31.50 37.00 39.50 39.00 46.00
56.00 62.00 66.00 64.00 70.00
26.54 36.84 38.16 39.25 43.42
45.61 62.28 66.67 65.79 70.18
19.50 23.25 24.00 23.75 28.75
30.00 35.00 36.00 36.00 42.00
14.75 19.50 16.75 17.50 22.00
21.00 28.00 24.00 25.00 31.00
5.50 7.25 7.50 7.50 9.50
9.00 11.00 11.00 12.00 14.00
4
Experiments
4.1
Experiment Settings
Training Environment. Our training environment follows the environment scaling pipeline of [34]. Within the synthesis pipeline, we leverage a diverse suite of high-performance LLMs for different roles. Specifically, GPT-4.1 is used for environment construction due to its favorable trade-off between cost and efficiency, while Claude-Sonnet-4.5 serves as a verifier given its strong evaluation capability. GLM-4.6 is employed to synthesize diverse instructions, forming the basis of our RL task set. Building on the synthesized tasks, we use Qwen2.5-72B-Instruct as a noise injector to introduce controlled perturbations into the interaction process. During training, Qwen2.5-72B-Instruct also acts as the user simulator to generate natural language feedback, while a Qwen3-32B model is trained as an evaluator to assign rewards based on the synthesized rubrics. Evaluation. We evaluate the robustness of the model on AgentNoiseBench [35], a benchmark designed to assess agent performance under real-world noise. We select two representative subsets, AgentNoiseBench-τ 2 and AgentNoiseBench-Vita for evaluation. To assess performance in idealized environments, we evaluate on representative standard agent benchmarks: (i) τ 2 -Bench, a dual-control conversational benchmark where both the user and the agent can invoke tools in customer-service domains such as retail, airline, and telecom; (ii) Vita-Bench, a multi-tool agent benchmark covering real-world scenarios including food delivery, in-store services, and travel. Across all benchmarks, GPT-4.1 is used as the user simulator, and Claude-Sonnet-4.5 is used as the evaluator. Each experiment is repeated four times. We report Avg@4 and Pass@4 metrics averaged across tasks. Implementation Details and Baselines. We adopt Qwen3-8B and Qwen3-32B as backbone models. On these backbones, we compare several representative training methods, including GRPO, DAPO, and GSPO, where our method is based on GSPO. The training batch size is set to 32, with 64 rollouts per sample. The proportion of noisy trajectories is capped at 50% of the total rollouts. We set the scheduling threshold ∆ to 0.05. The maximum prompt length is 8,192 tokens, and the maximum response length is 32,768 tokens. Detailed training configurations are provided in Appendix A. 4.2
Main Results
Table 1 and Table 2 present the evaluation results under noisy and ideal settings, respectively. We have the following observations. Noise-aware training significantly improves robustness under imperfect environments. Across all domains and both model scales, NoisyAgent consistently achieves the best performance on AgentNoiseBench, outperforming strong baselines such as GSPO and DAPO by a clear margin in both Avg@4 and Pass@4. In contrast, while standard RL methods improve performance under clean settings, their gains diminish substantially in the presence of noise, often exhibiting notable relative degradation across domains compared with their gains in idealized settings. This suggests that existing 6
Table 2: Main results under the ideal setting on standard agent benchmarks. We report Avg@4 and Pass@4 averaged across four runs. Best results are in bold, and second-best are underlined. τ 2 -Bench Method
Retail
Airline
VitaBench Telecom
Delivery
In-Store
OTA
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Avg@4
Pass@4
Qwen3-8B + GRPO + DAPO + GSPO + Ours
35.31 46.05 44.52 46.49 47.59
59.65 73.68 71.05 74.56 77.19
27.00 36.50 38.00 37.50 40.00
52.00 62.00 66.00 64.00 68.00
22.59 37.28 39.47 39.04 40.79
42.98 57.89 63.16 61.40 64.91
13.75 21.00 21.50 21.25 22.25
22.00 33.00 34.00 33.00 35.00
15.50 22.75 23.25 23.00 24.00
24.00 35.00 36.00 35.00 37.00
1.75 4.25 4.00 4.50 5.00
4.00 7.00 7.00 8.00 9.00
Qwen3-32B + GRPO + DAPO + GSPO + Ours
49.12 58.11 56.58 58.55 60.31
72.81 83.33 80.70 84.21 86.84
38.00 45.00 47.50 46.50 49.50
66.00 72.00 76.00 74.00 78.00
28.95 41.67 43.42 43.86 45.39
49.12 68.42 71.93 70.18 78.07
23.00 27.00 27.75 27.25 29.00
35.00 40.00 41.00 40.00 43.00
26.00 30.25 29.50 30.50 32.25
38.00 43.00 42.00 44.00 46.00
7.00 8.75 9.25 9.00 9.75
12.00 14.00 15.00 14.00 15.00
Table 3: Ablation study of key components on Delivery domain of both AgentNoiseBench-Vita and VitaBench with Qwen3-8B. We report Avg@4 and Pass@4 averaged across four runs. Method Ours w/o controlled injection w/o scheduling w/o noise w/o training
AgentNoiseBench-Vita
VitaBench
Avg@4
Pass@4
Avg@4 Pass@4
21.50 13.25 20.00 16.00 11.75
34.00 21.00 31.00 26.00 18.00
22.25 14.75 21.50 21.25 13.75
35.00 24.00 33.00 33.00 22.00
training paradigms are less effective when facing ambiguous user instructions and imperfect tool feedback. By incorporating structured perturbations during training, our method enables the agent to better handle uncertainty, recover from intermediate failures, and maintain consistent progress toward task completion under noisy conditions. Training with noise leads to consistent gains even in idealized settings. Despite being designed for noisy environments, NoisyAgent also achieves consistent improvements on standard benchmarks without noise. Across both τ 2 -Bench and VitaBench, our method outperforms all baselines across domains and metrics. This indicates that training with noise does not harm performance in ideal settings, and can improve overall agent capability. We attribute this to the fact that exposure to diverse and imperfect interaction patterns encourages the agent to learn more robust and effective decision-making strategies, rather than relying on brittle interaction assumptions. 4.3
Analysis
Ablation Study. To isolate the effect of each component, we perform ablations by removing individual elements from our framework. w/o controlled injection removes the hybrid training scheme, applying noise to all rollouts instead of mixing clean and noisy trajectories. w/o scheduling removes the curriculum over noise training, using perturbations of fixed complexity throughout training. w/o noise reduces training to an idealized setting without any perturbations. w/o training evaluates the base model without RL optimization. Overall, removing any component leads to performance degradation, indicating that each part contributes to the final performance. In particular, uncontrolled noise injection (w/o controlled injection) causes the largest drop, suggesting that naively introducing perturbations can destabilize training. In contrast, incorporating a proper scheduling strategy further improves performance, showing that progressively adjusting noise leads to more effective and stable learning. Training Dynamics. Figure 2 compares the training dynamics of NoisyAgent and the baseline trained without noise under both ideal and noisy evaluations. In the early stage of training, the two methods exhibit comparable performance, as optimization is largely conducted on clean trajectories serving as a warm-up phase. The initial introduction of moderate noise may even lead to a slight 7
(a) Idealized setting
(b) Noisy setting
Figure 2: Training dynamics on Vita-Bench Delivery (Qwen3-8B). We compare NoisyAgent with a baseline trained without noise under both ideal (no-noise) and noisy evaluations.
degradation in performance, reflecting the increased difficulty of the perturbed trajectories. As training progresses, the model gradually adapts to noisy conditions, and the curriculum introduces increasingly challenging perturbations, raising the requirements for successful task completion. While the baseline continues to improve, its gains remain moderate. In contrast, NoisyAgent achieves more substantial improvements, particularly under the noisy evaluation, where the performance gap becomes increasingly pronounced. This trend indicates that learning in noisy environments provides informative training signals, enabling the agent to develop stronger robustness and improved performance under challenging conditions. Interaction Pattern. Beyond aggregate performance, we analyze how curriculum training alters the agent’s interaction behavior compared to the base model and GSPO, along three dimensions: tool usage, response verbosity, and reasoning overhead. As shown in Table 4, under the noisy setting, NoisyAgent reduces tool usage from 13.9 to 11.4 calls per episode (18%), while GSPO yields only marginal change. In contrast, under the ideal setting, all methods exhibit similar tool usage (6.7–7.4 calls), with negligible differences. This indicates that the reduction in tool calls is not due to a general degradation of capability, but arises specifically in noisy environments, where NoisyAgent avoids excessive or redundant interactions. In parallel, NoisyAgent produces substantially longer responses, with output tokens increasing from 2,014 to 4,248 under noise (2.1×), and a consistent trend observed in the ideal setting. This suggests a shift toward more explicit and detailed interaction, potentially reducing the need for additional clarification through further tool calls. Taken together, these results show that curriculum training primarily improves the efficiency and clarity of interaction—reducing unnecessary tool usage while producing more informative responses. We provide a case study in Appendix B.1.
5
Related Work
5.1
LLM as Agent
With the rapid improvement in reasoning and instruction-following capabilities, LLMs have evolved from passive text generators into agents capable of tool use, multi-step planning, and interaction with dynamic environments [36–39]. Early approaches primarily rely on hand-crafted pipelines, where reasoning–action patterns, tool schemas, and memory mechanisms are manually designed on top of frozen models [36, 38, 40–42]. While effective, such prompt-level designs are brittle and do not fundamentally improve the underlying policy. More recent work instead trains agent behaviors directly via reinforcement learning with verifiable rewards (RLVR) [43–45]. One line of research focuses on stabilizing long-horizon training and improving credit assignment, with a variety of algorithmic advances [45–50]. In parallel, another line of work explores scalable environment design and task construction, enabling RL training over increasingly diverse and realistic agent scenarios, including tool use and retrieval [51–54], software engineering [55–57], and web or GUI interaction [58–60]. Despite these advances, existing work is largely conducted under idealized settings, leaving a gap between training conditions and real-world noisy environments. 8
Table 4: Interaction patterns on Retail domain with Qwen3-8B. Method Tool Calls Output Tokens Reasoning Tokens
5.2
Noisy setting: Base 13.9 GSPO 13.7 Ours 11.4
2,014 2,180 4,248
10,897 11,012 10,964
Ideal setting: Base 7.1 GSPO 7.4 Ours 6.7
1,931 1,982 3,923
7,091 7,265 7,534
Robustness of Agent
As LLM-based agents are increasingly deployed in complex real-world settings, robustness has emerged as a critical concern alongside raw capability [61–63]. A growing body of work shows that agent performance degrades substantially under distributional shifts in environment dynamics [64– 67, 12, 68]. On the user side, prior work investigates how perturbations in prompts, clarifications, and multi-turn dialog interactions affect agent behavior [69–75]. These studies suggest that realistic user interactions are often noisy, under-specified, and evolving, exposing agents to a broader and more dynamic input distribution than curated settings. On the execution side, reliance on external tools introduces an additional source of instability, as tools may return incomplete, outdated, or erroneous outputs [76, 77, 19, 78, 20, 79, 80]. Such local errors frequently propagate along the interaction trajectory, leading to cascading failures in downstream decisions [75, 81, 82]. To systematically characterize these effects, recent work proposes robustness benchmarks and diagnostic protocols [83, 84, 68, 85, 86]. AgentNoiseBench [35] further introduces a unified taxonomy of user-side and toolside noise with controllable perturbations, revealing consistent performance degradation across a wide range of models under realistic noise. However, existing approaches primarily focus on evaluation, leaving the problem of learning robust agent behaviors under realistic noise largely underexplored.
6
Limitation
While our framework demonstrates consistent improvements in robustness, we note several aspects that could be further explored in future work. First, our primary goal is to investigate whether incorporating real-world interaction noise can improve the robustness of agent policies. To this end, we focus on two representative sources of noise—user-side and tool-side perturbations—and model a set of common failure patterns observed in practice. While this design captures a broad range of realistic imperfections, it does not aim to exhaustively cover all possible forms of uncertainty. In real-world environments, noise can be more complex, compositional, and dynamically evolving. Extending the framework to model richer and more diverse interaction patterns is an important direction for future work. Also, our experiments are primarily conducted in synthesized environments that approximate real-world interaction dynamics. In principle, the proposed framework is general and can be applied to any agentic environment by augmenting it with structured noise. However, due to the high cost of agentic training and the need to systematically evaluate robustness under out-of-distribution conditions, we focus on controlled settings rather than extensively benchmarking across multiple in-domain training and testing datasets. We believe that applying our framework to broader real-world and in-domain benchmarks is an important direction for future work. We leave these directions for future work.
7
Conclusion
In this work, we investigate the fundamental gap between idealized agentic training and real-world deployment, and identify the lack of environmental imperfections during training as a key factor limiting agent robustness. To address this issue, we propose a noise-aware training framework that explicitly incorporates stochasticity and imperfections into the agent learning process. By systematically modeling instruction noise and tool noise, and introducing them through an automatic noise injection pipeline, our approach exposes agents to more realistic interaction dynamics. To ensure 9
stable optimization, we further design an adaptive training strategy that combines clean and perturbed rollouts while progressively increasing noise difficulty based on the model’s robustness. Extensive experiments demonstrate that our method consistently improves agent performance under noisy and dynamic environments, validating its effectiveness in enhancing robustness. Notably, we also observe consistent gains on standard, idealized benchmarks, suggesting that controlled exposure to environmental noise promotes more generalizable reasoning and decision-making behaviors. Overall, this work highlights the importance of aligning training conditions with real-world interaction characteristics, and provides a practical framework for improving the robustness of LLM-based agents in realistic deployment settings.
References [1] OpenAI. Introducing gpt-5.2. introducing-gpt-5-2/.
2025.
URL https://openai.com/index/
[2] Google. Gemini 3 pro model card. https://storage.googleapis.com/deepmind-media/ModelCards/Gemini-3-Pro-Model-Card.pdf, 2025. [3] Meituan LongCat Team, Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang, Chengcheng Han, Chenhui Yang, Chi Zhang, et al. Introducing longcat-flash-thinking: A technical report. arXiv preprint arXiv:2509.18883, 2025. [4] Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. [5] Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471, 2025. [6] Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025. [7] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. CoRR, abs/2406.12045, 2024. doi: 10.48550/ARXIV.2406.12045. URL https://doi.org/10.48550/arXiv.2406.12045. [8] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. [9] Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, et al. Vitabench: Benchmarking llm agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490, 2025. [10] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023. [11] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. [12] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025. [13] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, 2024. 10
[14] Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024. [15] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. [16] Cindy Gallois, Tania Ogay, and Howard Giles. Communication accommodation theory. Theorizing about intercultural communication, pages 121–148, 2005. [17] Johanne R Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, and Luke Gallagher. What do users really ask large language models? an initial log analysis of google bard interactions in the wild. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2703–2707, 2024. [18] Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. Understanding user experience in large language model interactions. arXiv preprint arXiv:2401.08329, 2024. [19] Sri Vatsa Vuddanti, Aarav Shah, Satwik Kumar Chittiprolu, Tony Song, Sunishchal Dev, Kevin Zhu, and Maheep Chaudhary. Paladin: Self-correcting language model agents to cure tool-failure cases. arXiv preprint arXiv:2509.25238, 2025. [20] Qian Xiong, Yuekai Huang, Ziyou Jiang, Zhiyuan Chang, Yujia Zheng, Tianhao Li, and Mingyang Li. Butterfly effects in toolchains: A comprehensive analysis of failed parameter filling in llm tool-agent systems. arXiv preprint arXiv:2507.15296, 2025. [21] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024. [22] Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. Gui-xplore: Empowering generalizable gui agents with one exploration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19477–19486, 2025. [23] Youssef Shoeb, Azarm Nowzad, and Hanno Gottschalk. Out-of-distribution segmentation in autonomous driving: Problems and state of the art. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4310–4320, 2025. [24] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017. [25] Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. In Robotics: Science and Systems (RSS), 2017. [26] Minghao Zhao, Wenhan Xiong, Lei Zhang, et al. Robust reinforcement learning as a stackelberg game. In International Conference on Machine Learning (ICML), 2021. [27] Hao Zhao et al. The landscape of agentic reinforcement learning for LLMs: A survey. arXiv preprint arXiv:2509.02547, 2025. [28] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [29] Qiushi Jiang et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055, 2025. [30] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 11
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [32] Meituan LongCat Team. Longcat-flash-thinking-2601 technical report. CoRR, abs/2601.16725, 2026. [33] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. [34] Dunwei Tu, Hongyan Hao, Hansi Yang, Yihao Chen, Yi-Kai Zhang, Zhikang Xia, Yu Yang, Yueqing Sun, Xingchen Liu, Furao Shen, Qi Gu, Hui Su, and Xunliang Cai. Scaleenv: Scaling environment synthesis from scratch for generalist interactive tool-use agent training. arXiv preprint arXiv:2602.06820, 2026. [35] Ruipeng Wang, Yuxin Chen, Yukai Wang, Chang Wu, Junfeng Fang, Xiaodong Cai, Qi Gu, Hui Su, An Zhang, Xiang Wang, Xunliang Cai, and Tat-Seng Chua. AgentNoiseBench: Benchmarking robustness of tool-using LLM agents under noisy condition. arXiv preprint arXiv:2602.11348, 2026. [36] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [37] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539–68551, 2023. [38] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [39] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. [40] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. [41] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations (ICLR), 2024. [42] Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In ACM Symposium on User Interface Software and Technology (UIST), 2023. [43] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [44] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. [45] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 12
[46] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. [47] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. [48] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. [49] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Yan, et al. VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025. [50] Zhiyuan Yao, Yi-Kai Zhang, Yuxin Chen, Yueqing Sun, Zishan Xu, Yu Yang, Tianhao Hu, Qi Gu, Hui Su, and Xunliang Cai. Coba-rl: Capability-oriented budget allocation for reinforcement learning in llms. arXiv preprint arXiv:2602.03048, 2026. [51] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. ReTool: Reinforcement learning for strategic tool use in LLMs. arXiv preprint arXiv:2504.11536, 2025. [52] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. [53] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-Searcher: Incentivizing the search capability in LLMs via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025. [54] Yaorui Shi, Yuxin Chen, Siyuan Wang, Sihang Li, Hengxing Cai, Qi Gu, Xiang Wang, and An Zhang. Look back to reason forward: Revisitable memory for long-context llm agents. arXiv preprint arXiv:2509.23040, 2025. [55] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. [56] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139, 2024. [57] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025. [58] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), 2024. [59] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [60] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024. 13
[61] Zike Li, Mingwei Liu, An Li, Kaifeng He, Yanlin Wang, Xin Peng, and Zibin Zheng. Enhancing the robustness of LLM-generated code: Empirical study and framework. arXiv preprint arXiv:2503.20197, 2025. [62] Natan Levy, Adiel Ashrov, and Guy Katz. Towards robust LLMs: An adversarial robustness measurement framework. arXiv preprint arXiv:2504.17723, 2025. [63] Aryan Agrawal, Lisa Alazraki, Shahin Honarvar, and Marek Rei. Enhancing LLM robustness to perturbed instructions: An empirical study. arXiv preprint arXiv:2504.02733, 2025. [64] Catalin Anghel, Andreea A. Anghel, Emilia Pecheanu, Adina Cocu, Adrian Istrate, and Constantin A. Andrei. Diagnosing bias and instability in LLM evaluation: A scalable pairwise meta-evaluator. Information, 16(8):652, 2025. [65] Borui Wan, Gaohong Liu, Zhe Song, Jiarui Wang, Yukang Zhang, Guangming Sheng, Shuguang Wang, Hui Wei, Chao Wang, Wen Lou, et al. Robust LLM training infrastructure at ByteDance. pages 186–203, 2025. [66] David Herrera-Poyatos, Carlos Peláez-González, Cristina Zuheros, Andrés Herrera-Poyatos, Virilo Tejedor, Francisco Herrera, and Rosana Montes. An overview of model uncertainty and variability in LLM-based sentiment analysis: Challenges, mitigation strategies, and the role of explainability. Frontiers in Artificial Intelligence, 8:1609097, 2025. [67] Hongchen Wang, Kangming Li, Scott Ramsay, Yvonne Fehlis, Edward Kim, and Jason HattrickSimpers. Evaluating the performance and robustness of LLMs in materials science Q&A and property predictions. Digital Discovery, 2025. [68] Tong Yu, Yongcheng Jing, Xikun Zhang, Wentao Jiang, Wenjie Wu, Yingjie Wang, Wenbin Hu, Bo Du, and Dacheng Tao. Benchmarking reasoning robustness in large language models. arXiv preprint arXiv:2503.04550, 2025. [69] Jinnan Li, Jinzhe Li, Yue Wang, Yi Chang, and Yuan Wu. StructFlowBench: A structured flow benchmark for multi-turn instruction following. arXiv preprint arXiv:2502.14494, 2025. [70] Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18632–18702, 2025. [71] Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. Agentif: Benchmarking instruction following of large language models in agentic scenarios. arXiv preprint arXiv:2505.16944, 2025. [72] Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. Understanding user experience in large language model interactions. arXiv preprint arXiv:2401.08329, 2024. [73] Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio. CLARQ-LLM: A benchmark for models clarifying and requesting information in task-oriented dialog. arXiv preprint arXiv:2409.06097, 2024. [74] Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models. arXiv preprint arXiv:2405.12063, 2024. [75] Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kästner, and Tongshuang Wu. What prompts don’t say: Understanding and managing underspecification in LLM prompts. arXiv preprint arXiv:2505.13360, 2025. [76] Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, and Kai Yu. Reducing tool hallucination via reliability alignment. arXiv preprint arXiv:2412.04141, 2024. 14
[77] Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, et al. Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models. arXiv preprint arXiv:2406.20015, 2024. [78] Shirley Kokane, Ming Zhu, Tulika Manoj Awalgaonkar, Jianguo Zhang, Akshara Prabhakar, Thai Quoc Hoang, Zuxin Liu, Rithesh RN, Liangwei Yang, Weiran Yao, et al. Toolscan: A benchmark for characterizing errors in tool-use llms. In ICLR 2025 Workshop on Building Trust in Language Models and Applications. [79] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691, 2024. [80] Ruian Zhang, Hao Wang, Jiaxin Wang, Min Li, Yu Huang, Dawei Wang, and Qi Wang. From allies to adversaries: Manipulating LLM tool-calling through adversarial injection. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2009– 2028, 2025. [81] X. Zhu et al. Compounding errors in tool-augmented agents. arXiv preprint, 2025. [82] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for LLM agents. arXiv preprint arXiv:2403.02502, 2024. [83] Grigor Nalbandyan, Rima Shahbazyan, and Evelina Bakhturina. Score: Systematic consistency and robustness evaluation for large language models. arXiv preprint arXiv:2503.00137, 2025. [84] Z. Wen, Z. Liu, Z. Tian, S. Pan, Z. Huang, D. Li, and M. Huang. Scenario-independent uncertainty estimation for LLM-based question answering via factor analysis. In Proceedings of the ACM on Web Conference, pages 2378–2390, 2025. [85] Riccardo Lunardi, Vincenzo Della Mea, Stefano Mizzaro, and Kevin Roitero. On robustness and reliability of benchmark-based evaluation of LLMs. arXiv preprint arXiv:2509.04013, 2025. [86] C. Siska, K. Marazopoulou, M. Ailem, and J. Bono. Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 10406–10421, 2024.
15
A
Training Configuration Details
This section provides the complete training configurations for reproducing our experiments. A.1
Model and Infrastructure
We use Qwen3-8B and Qwen3-32B as backbone models, both trained in BF16 precision with vLLM (v0.8.5) for efficient rollout generation. All models use RoPE with θ = 106 and RMSNorm with ϵ = 10−6 . A.2
Optimization Hyperparameters
Table 5 summarizes the optimization hyperparameters shared across all methods. Table 5: Optimization hyperparameters. Hyperparameter
Value
Optimizer β1 , β 2 Adam ϵ Learning rate LR schedule Weight decay Gradient clipping KL coefficient Discount factor γ GAE λ PPO epochs per step Data reuse epochs Gradient accumulation steps Total training steps
A.3
Adam 0.9, 0.95 10−8 1 × 10−6 Constant 0.01 1.0 0.0 1.0 1.0 1 2 2 100
Rollout and Generation Configuration
Table 6 details the rollout generation settings. Table 6: Rollout and generation configuration. Parameter
Value
Training batch size Rollouts per sample Micro batch size Max prompt length Max response length Max sequence length Sampling temperature (rollout) Sampling temperature (eval) Top-p Max interaction turns
A.4
16 32 1 8,192 tokens 32,768 tokens 40,960 tokens 1.0 0.0 1.0 100
Method-Specific Configurations
Table 7 compares the loss configurations across different training methods. For GRPO, we follow the original formulation with fixed clip range and batch-level advantage normalization. DAPO and GSPO employ an asymmetric clip range [0.2, 0.28] with dynamic temperature scaling and buffer-level advantage normalization, following their respective original implementations. Our method inherits the GSPO loss configuration and adds the noise-aware curriculum on top. 16
Table 7: Method-specific loss configurations.
A.5
Method
Loss Type
Clip Range
Clip Ratio c
Adv. Norm
Loss Agg.
GRPO DAPO GSPO Ours
grpo dapo gspo gspo
[0.2, 0.2] [0.2, 0.28] [0.2, 0.28] [0.2, 0.28]
10.0 3.0 3.0 3.0
batch buffer buffer buffer
token-mean seq-mean-token-mean seq-mean-token-mean seq-mean-token-mean
Noise-Aware Training Configuration
Our noise-aware training strategy consists of two components: controlled injection and noise scheduling, corresponding to the two factors described in Section 3.2. Controlled Injection. This component controls the noise scale ρ = Nnoise /N , i.e., the proportion of perturbed rollouts within each task’s rollout group. For each task, N rollouts are generated in parallel, among which Nnoise rollouts are executed in noisy environments and the remaining N − Nnoise in clean environments. In our experiments, we fix the maximum noise proportion at 50% of total rollouts (i.e., ρ ≤ 0.5). Training starts with ρ = 0, and the noise proportion is increased by a fixed step size each time the model’s performance plateaus, as determined by the scheduling mechanism below. Noise Scheduling. This component controls the noise difficulty and determines when to increase the noise scale. We measure the model’s robustness via the performance gap ∆ between clean and perturbed rollouts (as defined in Section 3.2): when ∆ falls below a predefined threshold θ, the model is considered to have adapted to the current noise level. Upon adaptation, we increase both the noise difficulty (characterized by the frequency of tool-side perturbations and the severity of user-side anomalies). This yields a progressive curriculum that gradually increases interaction complexity while maintaining training stability. Noise Types. We define noise along two axes: user-side noise (ambiguous, inconsistent, redundant, out-of-scope) and tool-side noise (failures, incomplete, misleading, redundant), as detailed in Section 3.1. A.6
Training Data Configuration
For the multi-domain training setup (Table 1 and Table 2), we train on tasks from two benchmarks simultaneously: • τ 2 -Bench: Airline, Retail, and Telecom domains, each filtered to medium-to-low pass rate difficulty. • VitaBench: Delivery, In-Store, and OTA domains. All baseline methods (GRPO, DAPO, GSPO) are trained exclusively on clean environments without noise injection. In our method, noisy trajectories are progressively introduced via the controlled injection mechanism described above. Groups where all rollouts receive identical rewards (all-pass or all-fail) are filtered out to ensure meaningful gradient signal. A.7
Evaluation Protocol
Throughout both training and final evaluation, we use GPT-4.1 as the user simulator and ClaudeSonnet-4.5 as the evaluator. During training, we evaluate every 5 steps with 4 rollouts per task, assessing both the ideal (noise-free) and noisy settings. For the final evaluation reported in our main results, each experiment is repeated 4 times. We report Avg@4 (average score across 4 runs) and Pass@4 (fraction of tasks solved in at least one of the 4 runs). 17
Task: User requests return of gaming items from orders #W5490111 and #W7387996. Base model (failed, reward = 0.0): [Turns 1–11] Agent correctly verifies the user’s identity, retrieves both orders, and identifies the gaming items. User confirms: “Yes, please return the Mechanical Keyboard and the Gaming Mouse.” [Turn 15–30] Instead of calling the return API, the agent starts recommending desk lamps and discussing student discounts. The conversation ends without any return being processed. Our model (success, reward = 1.0): [Turns 1–18] Agent verifies identity, retrieves orders, and summarizes the items with refund details. User confirms the same request. [Turn 22] Agent immediately executes: > return_delivered_order_items(#W5490111, [keyboard]) > return_delivered_order_items(#W7387996, [mouse]) Both returns processed successfully in a single turn.
Figure 3: Case study from τ 2 -Bench Retail (noisy setting). Both agents complete the informationgathering phase correctly, but the base model fails to execute the final action after encountering API noise, while our model stays on task.
A.8
Computational Resources
For Qwen3-8B training, we use 32 NVIDIA H800 GPUs. For Qwen3-32B training, we use 64 NVIDIA H800 GPUs. Each training run of 100 steps takes approximately 3–5 days depending on the model scale and domain complexity.
B
Discussion
B.1
Case Study
We analyze a representative example from the τ 2 -Bench Retail domain to illustrate how noise affects agent behavior. In this task, a user requests the return of two gaming-related items (a mechanical keyboard and a gaming mouse) from two separate orders, with refunds issued to the original payment method. During the interaction, the environment injects intermittent API failures (e.g., Error 429) and corrupted fields. As shown in Figure 3, both the base model and NoisyAgent successfully complete the informationgathering phase, including identity verification, order retrieval, and item identification. However, their behaviors diverge in the execution phase. The base model fails to invoke the return API and instead shifts to unrelated recommendations, resulting in task failure. In contrast, NoisyAgent directly executes the required return operations once sufficient information is obtained, completing both returns successfully. This example highlights a key behavioral difference under noise: while both models are capable of correctly understanding user intent, only NoisyAgent reliably translates this understanding into effective action. Under noisy conditions, the base model fails to maintain focus on the primary objective and does not complete the required API calls. In contrast, NoisyAgent remains aligned with the task objective and executes the necessary actions without deviation. This phenomenon generalizes beyond the illustrated example. Among the 23 tasks in this domain where NoisyAgent succeeds but the base model fails under noise, 78% exhibit the same pattern: the base model correctly gathers all required information but fails to execute the critical action. This suggests that noise primarily affects the transition from understanding to action, rather than the understanding itself. Overall, the results indicate that curriculum training improves the agent’s ability to maintain goaldirected behavior and reliably execute actions under noisy conditions, even when intermediate observations are corrupted or inconsistent. 18
B.2
Code of Ethics
This work complies with the NeurIPS Code of Ethics. Our research focuses on improving the robustness of LLM-based agents under realistic, imperfect environments. All data used in this work are either synthetically generated by large language models or constructed through controlled pipelines, without involving any real user data or sensitive personal information. To ensure responsible development, we adopt a strict data construction and validation process. Synthetic environments, interaction patterns, and noise perturbations are carefully designed to reflect realistic scenarios while avoiding harmful, unsafe, or misleading content. All generated data are further reviewed and refined to ensure consistency, correctness, and safety. In addition, our experiments are conducted in controlled simulation environments, and the proposed framework does not directly interact with real users or external systems. Therefore, no human subjects are involved, and no ethical risks related to data privacy or user consent arise in this work. B.3
Broader Impacts
This work aims to improve the robustness of LLM-based agents under noisy and imperfect environments, which has important implications for real-world deployment. By exposing agents to diverse interaction uncertainties during training, our approach can lead to more reliable and adaptive systems in applications such as customer service, recommendation, and task automation. However, improving agent robustness may also introduce potential risks. More capable and adaptive agents could be misused in scenarios requiring manipulation or exploitation of uncertain environments. In addition, if deployed without proper safeguards, robust agents may still propagate biases or make incorrect decisions under ambiguous inputs, potentially leading to negative user experiences or unintended consequences. To mitigate these risks, our work focuses on controlled training settings and emphasizes the importance of structured evaluation and validation. The proposed framework is designed to improve generalization and reduce over-reliance on brittle patterns, which may contribute to safer and more reliable deployment. We encourage future work to further investigate fairness, safety, and alignment aspects when applying robust agent training methods in real-world systems. B.4
Safeguards
We implement several safeguards to ensure responsible use and construction of data and models in this work. First, all environments, interaction data, and noise perturbations are generated through a controlled synthesis pipeline. The generation process is designed to avoid unsafe or harmful content, and all synthesized data are subject to validation and refinement to ensure correctness and consistency. Second, our framework does not rely on real user data. All interaction patterns and task environments are synthetic, eliminating risks related to privacy leakage or misuse of personal information. Finally, we adopt a modular design for noise injection, allowing controlled manipulation of user-side and tool-side perturbations. This prevents the introduction of uncontrolled or unrealistic behaviors that could compromise evaluation validity.
19