IEEE JOURNAL MANUSCRIPT
1
arXiv:2609.05133v1 [cs.RO] 4 Sep 2026
A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning 1st Chongwen Dong Department of Mechanical Engineering Northern Arizona University 15600 S. McConnell Dr., Flagstaff, AZ 86011 2nd Mithun Paul Saint-Germain and Pinjari Asif School of Informatics, Computing, and Cyber Systems Northern Arizona University 1295 Knoles Dr., Flagstaff, AZ 86011 3rd Carlo daCunha Helen and John C. Hartmann Department of Electrical Engineering and School of Applied Engineering and Technology New Jersey Institute of Technology 323 Martin Luther King Jr. Blvd., Newark, NJ 07102 [email protected]
Abstract—This paper addresses navigation by composite heterogeneous robots in a decentralized system when policy reasoning and local control operate at different update levels. In a NetLogo–Python implementation, three robots share motion dynamics but use different LLM backends. Each robot independently combines a large language model (LLM) policy agent, an Upper Confidence Bound (UCB) bandit, and a Double Deep QNetwork (Double DQN) controller; no central LLM generates team actions. LLM inference is confined to round-level policy generation and refinement rather than tick-level action selection. The robots perform cross-LLM communication through a shared round summary containing policies, outcomes, and learning feedback. UCB performs refinement-mode selection, and the policyconditioned Double DQN performs tick-level action selection from navigation variables, active policy parameters, and the LLM action prior. Each of the four configurations was evaluated over 30 rounds. In the fixed simulation, the complete configuration reached the goal in all 90 correlated robot–round records and achieved the lowest median completion time (42 ticks) and P90 (73.2 ticks); its median was 25.0–39.1% lower than those of the other configurations. These observations provide descriptive, configuration-level evidence from the evaluated configurations. Index Terms—Large language models, cross-LLM communication, decentralized multi-robot systems, reinforcement learning.
I. I NTRODUCTION
H
ETEROGENEOUS multi-robot systems combine agents with different physical or decision capabilities to perform shared tasks. Large language models (LLMs) extend these systems with language-mediated task decomposition, communication, and policy adaptation [1], [2]. For continual navigation, however, policy reasoning and motion control
operate at different update levels. An LLM can generate or refine a policy at a task boundary, whereas a local controller must select an action at every environment tick. Calling an LLM for each action would place policy reasoning inside the local control loop. The resulting design problem is to retain LLM reasoning at the policy level while providing a separate mechanism for tick-level execution. Existing LLM-based multi-robot frameworks have investigated dialogue-based coordination, capability-aware task allocation, decentralized planning, and feedback-driven policy revision [3]–[7]. Hierarchical robotic systems have also separated high-level language reasoning from lower-level execution [9], [10]. This current work is motivated by these aforementioned works, but instead asks a more specific system-design question: how can persistent robot-specific LLM policies be refined from shared completed-round information while local robot learning supplies every motion action? This question requires an explicit interface among policy ownership, crossLLM communication, round-level policy refinement, and ticklevel action selection. This paper introduces a solution for this problem: a decentralized two-tier architecture one with robot-local policy ownership, and the second one guided by LLMs. Specifically, in the first tier, a local policy guidance is combined with a Deep Q-Network (DQN). This tier recieves input from second tier which has an LLM shared board which is in-turn guided by Upper Confidence Bound (UCB) algorithm. Specifically the contributions of this work are as follows: 1) A one-LLM-per-robot architecture for composite heterogeneous robots that keeps policy ownership, UCB-
IEEE JOURNAL MANUSCRIPT
guided refinement, and Double DQN control local to each robot; 2) A temporally decoupled LLM policy-generation and refinement mechanism that confines LLM inference to round boundaries, while robot-local controllers perform tick-level action selection without per-action LLM invocation; 3) A shared-board-mediated cross-LLM communication mechanism that exchanges completed-round policies, outcomes, rationales, DQN feedback, and UCB advice while preserving robot-local policy ownership; II. R ELATED W ORK LLM-based multi-agent and multi-robot systems organize decision authority in centralized, decentralized, or hybrid forms. COHERENT uses a centralized task assigner to decompose long-horizon tasks for heterogeneous robot executors [6]. In contrast, MHRC places LLM reasoning at individual heterogeneous robots and coordinates them through local observations and inter-robot messages [5]. Chen et al. compare centralized, decentralized, and hybrid communication frameworks, showing that topology changes both coordination performance and token use [2]. RoCo adopts a hybrid organization: robot-specific LLM agents deliberate over subtask and waypoint plans, while a centralized multi-arm motion planner produces executable trajectories [1]. SMART-LLM addresses capability-aware task allocation [4]. Modular embodied-agent frameworks use LLMs to construct cooperative agent systems [3], whereas EMOS incorporates embodiment descriptions into heterogeneous multi-robot planning [11]. Collectively, these studies show that shared communication does not by itself determine whether a system is decentralized; the location of decision authority and persistent state is also decisive. In the architecture studied here, the shared board transports completed-round information, while each robot retains its own policy and learning states. A second distinction is the temporal level at which language reasoning enters control. SAS-Prompt treats an LLM as a numerical policy optimizer that revises robot behavior from trajectory traces [7], [8]. GenSwarm generates deployable code policies for multi-robot tasks [9]. ELLMER separates highlevel LLM reasoning and generated code from sensorimotor feedback during execution [10]. YOLO-MARL makes the separation between language reasoning and local execution more explicit: it queries an LLM before multi-agent reinforcement learning and subsequently executes decentralized neural policies without repeated LLM inference [12]. These methods range from language-generated task or motion specifications to persistent policies executed by non-LLM controllers. Relative to these approaches, the proposed system uses dedicated LLMs to generate and refine robot-specific policies at round boundaries, while robot-local controllers perform tick-level action selection. Reinforcement learning provides several interfaces between language-level guidance and executable multi-agent behavior. Language-conditioned offline RL embeds natural-language commands into decentralized multi-robot navigation policies
2
[13]. YOLO-MARL uses LLM-generated planning functions to guide subsequent MARL policy learning [12]. LAMARL instead uses LLM-generated prior policies and reward functions to guide cooperative multi-robot training [14]. Yoshida and Sueoka combine LLM-based high-level policy selection with MARL-based swarm control [15], while hierarchical analyses model an LLM planner above a lower-level RL actor [16]. In contrast to approaches where RL learns the principal policy from language-provided specifications, RL is auxiliary to the LLM policy process in this work. The LLM generates and refines the round-level policy. DQN and Double Q-learning provide the algorithmic foundations for the robot-local Double DQN, which selects tick-level actions and supplies execution feedback [17], [18]. UCB1 contributes the exploration– exploitation rule used only to select the refinement mode for the next LLM update [19]. The resulting interface combines persistent one-LLM-per-robot policy ownership, completedround cross-LLM communication, and robot-local learning assistance without assigning team-level policy generation or robot action selection to a central LLM. III. F RAMEWORK A. System Model and Overall Framework This paper introduces a solution for this problem: a decentralized two-tier architecture one with robot-local policy ownership, and the second one guided by LLMs. Specifically, in the first tier, a local policy guidance is combined with a Deep Q-Network (DQN). This tier recieves input from second tier which has an LLM shared board which is inturn guided by Upper Confidence Bound (UCB) algorithm. In detail, a dedicated LLM generates and refines the robotspecific policy at round boundaries, while the local controller selects tick-level actions based on inputs from the LLMs. That is, policy ownership and learning states remain local to each robot; no central LLM generates a team policy or controls robot actions. Further, round-level communication and policy updates follow the event-driven schedule (detailed in Section III-B). Further, we introduce a shared board that aggregates completed-round policies, outcomes, rationales, DQN feedback, and UCB advice. Each LLM reads the same board but retains its robot’s policy history and generates only that robot’s next policy. UCB selects a refinement mode, and the policy-conditioned Double DQN selects tick-level actions and returns execution feedback. In this paper, heterogeneous robots denotes model- and decision-layer heterogeneity within this composite architecture; robot embodiment and motion dynamics remain fixed. Consider N mobile robots R = {r1 , . . . , rN } that repeat the same goal-reaching task for K rounds, with at most Tmax control ticks per round. The notation indexes robot-local quantities by N ; the implementation and experiments use N = 3, and no claim is made about team-size scalability. At tick t of round k, NetLogo maintains each turtle’s position, heading, terminal flags, and finish tick. These variables determine the current goal distance and signed heading error. Success requires the turtle’s current patch to equal the goal patch. A control tick is one pass through the local action phase for every active robot. NetLogo increments the global counter
IEEE JOURNAL MANUSCRIPT
after this phase. An arrival or timeout detected after an executed action is therefore recorded as ticks + 1, whereas a turtle already on the goal patch retains the current counter because it executes no new action. Let yik ∈ {0, 1} denote success and Tik the recorded completion tick. A timeout gives yik = 0 and Tik = Tmax ; d0i denotes the fixed start-to-goal distance used by the round-level credit. Each robot is implemented as a four-component composite agent, ri = ⟨Ti , Mi , Ui , Di ⟩ , (1) where Ti is the NetLogo turtle, Mi is its dedicated LLM policy agent, Ui is its UCB1 bandit, and Di is its Double DQN controller. The three robots are assigned Llama-3.3-70BInstruct-quantized, Phi-4, and Meta-Llama-3.1-70B-Instructquantized, respectively. Each robot retains its own LLM policy history, UCB values and counts, online and target networks, replay buffer, exploration state, and action statistics. The evaluated heterogeneity is at the model and decision layers. The turtles share embodiment, motion rules, and action categories, but use different LLM backends and develop robot-specific policies, UCB histories, and DQN trajectories. Fixed initial poses are experimental conditions rather than morphological heterogeneity. The system is decentralized with respect to decision authority and persistent control state: no central LLM creates a team policy or selects robot actions. The shared text board, NetLogo world, and Python runtime are simulation infrastructure; they exchange or execute robotlocal information without replacing the individual controllers. The code executes in two coupled loops. During setup, the Python layer initializes the NetLogo task and the robot-local UCB and Double DQN states, then generates the three initial LLM policies concurrently. During a rollout, each active turtle provides its local state, its Double DQN selects one action category, NetLogo applies the active policy parameters, and the resulting reward and next state are stored for robot-local training. When all turtles are terminal, the Python layer records the round, credits any UCB arm associated with the evaluated policy, selects the next refinement mode, and updates bestso-far records. At an eligible round boundary, it constructs the shared board, queries the dedicated LLMs concurrently, sanitizes and loads the returned robot-specific policies, and resets the task for the next rollout. Section III-B defines the exact LLM-call schedule and failure handling. Fig. 1 separates the round-level policy path from the robotlocal tick-level loop. Cross-robot information is limited to completed-round records published through B k . Fig. 2 shows this boundary for a representative robot; the same local topology is instantiated for all three robots. B. Round-Level Policy Refinement and Cross-LLM Communication The implementation uses an event-driven two-level schedule. Each dedicated LLM generates one initial policy before round 1. After every completed nonfinal round k < K, UCB evaluates the rollout and selects a refinement mode, after which that robot’s LLM is called once to generate the policy
3
for round k+1. Evaluating K rounds therefore requires exactly K LLM calls per robot: one initialization call and K − 1 refinement calls. No LLM is called within the tick-level control loop or after the final round. The two levels denote roundlevel policy adaptation and tick-level local control, rather than three independent numerical update rates. Fig. 3 summarizes this schedule. At each eligible boundary, the implementation records rollout histories, credits the UCB arm associated with the evaluated policy, selects the next advice, updates best-sofar records, constructs B k , and installs the sanitized LLM outputs before resetting the turtles. Network parameters, replay memory, ϵ, UCB state, policy histories, and lifetime statistics persist across rounds; only round-specific DQN action counts and loss statistics are cleared. The round-level LLM output is a structured policy. Policy θ ki contains a numerical motion vector and four regional action preferences, θ ki = mki , ρki , (2) with
k mki = [δik , δi,near , ωik , αik , dki,near ],
ρki = [aki,FM , aki,FA , aki,NM , aki,NA ].
(3)
Here, δ and δnear are the standard and near-goal forward steps, ω is the maximum turn angle, α is the alignment threshold, and dnear is the near-goal distance threshold. The subscripts FM, FA, NM, and NA denote far/misaligned, far/aligned, near/misaligned, and near/aligned regions. Each regional preference belongs to the five-action set A = {FORWARD, FORWARD_SMALL, TURN_ONLY, TURN_AND_FORWARD,
(4)
TURN_AND_FORWARD_SMALL}. For distance d and signed goal-angle error e, the LLMrecommended action is ai,FM , d > di,near , |e| > αi , a d > di,near , |e| ≤ αi , i,FA , (5) aLLM (d, e) = i a , d ≤ d , |e| > α , i,NM i,near i ai,NA , d ≤ di,near , |e| ≤ αi . This recommendation is a soft prior for the DQN, not a compulsory action. The DQN-selected category is converted to motion using the active policy: FORWARD and FORWARD_SMALL translate by δ and δnear ; TURN_ONLY rotates toward the goal by min(ω, |e|); and the two turn-and-forward actions apply the same bounded rotation before the corresponding translation. The generated numerical fields are sanitized to the implementation ranges 0.5 ≤ δ ≤ 3.0,
0.01 ≤ δnear ≤ 1.0,
3 ≤ ω ≤ 60,
1 ≤ α ≤ 45,
(6)
1 ≤ dnear ≤ 10. Here ω and α are measured in degrees. The parser first attempts to recover either a JSON object or a Python dictionary from the LLM response. Once a dictionary is recovered, policy sanitization starts from DEFAULT_POLICY and overwrites only the supplied fields that can be validated; missing or
IEEE JOURNAL MANUSCRIPT
4
Composite robot ri
Dedicated LLM Mi initial and roundlevel policy generation
1
Local UCB1 Ui bounded success/finishtime credit and fivearm refinement advice
2
3
NetLogo turtle Ti sanitized policy storage, local sensing, motion execution, and reward calculation 4
5
Local Double DQN Di 15-D input, five action categories, replay, and online/target learning
Round-level path: 1 next UCB advice; 2 sanitized policy (five numerical fields and four regional actions); 3 completed-rollout success, finish tick, and start distance. Tick-level loop: 4 15-D state, reward, and next state; 5 selected action category. Fig. 1. Information flow within composite robot ri . Numbered arrows keep the round-level policy path visually separate from the tick-level Double DQN loop; all four components and their persistent states are owned by the same robot.
Shared text board Bk for completed-round exchange all robots’ completed-round summaries and policy-refinement records publish local round summary
read the same shared board
Representative composite robot ri , i = 1, . . . , N dedicated LLM and turtle; local policy/history, UCB1, and Double DQN
replacing an established controller while guaranteeing an executable policy before the first rollout. At initialization, each LLM receives the environment description, its robot’s start state, the goal, the action vocabulary, and the output schema. For subsequent policy refinement, B k aggregates each robot’s outcome and final state, current policy fields, mission and rationale, expected DQN improvement, cross-LLM communication summary, current and next UCB advice, and DQN round feedback. The subsequent policy refinement is
local state, action, reward, and next state at each tick Shared NetLogo task environment identical motion dynamics; each turtle senses and acts through its own local control path Fig. 2. Representative communication topology. The board is common to all robots, whereas policy generation, UCB1, Double DQN, and turtle execution remain robot-local. Solid arrows denote round-level information exchange; the dashed arrow denotes the robot-local tick-level environment loop.
invalid numerical and action fields therefore retain their fieldspecific defaults. Failure to recover a dictionary invokes a stage-dependent fallback. An initialization failure loads the complete default policy, whereas a failure during a later roundlevel refinement retains the robot’s previously active control policy. This distinction prevents malformed LLM output from
h i θ k+1 = S FMi Hik , θ ki , B k , zik+1 , i
(7)
where FMi denotes generation by robot i’s dedicated model, Hik is its complete local history, zik+1 is its local UCB advice, and S is the sanitization operator. Every model receives the same board but only its own current policy, complete history, and next advice; the three model calls run concurrently with three worker threads, and each returned policy is loaded only by its corresponding robot. The output also records a mission statement, rationale, expected DQN improvement, and crossLLM learning summary for the next board. These textual fields do not enter the tick-level action loop.
IEEE JOURNAL MANUSCRIPT
5
Completedround processing record histories → k<K update the previously selected UCB arm, if any → select zik+1 → update best-so-far records
Tick-level local loop encode the 15-D state → select one Double DQN action → execute it → compute reward, store the transition, and train
Round-level policy boundary for every k < K: build Bk → query the dedicated LLMs in parallel → sanitize and load θ k+1 i
Prepare round k + 1 increment the round → reset poses, ticks, and round-specific DQN statistics; retain networks, replay, ϵ, UCB state, and histories
k=K start the next local tick loop Final round: show final results and stop
Fig. 3. Implemented control and update cycle. Once all turtles are terminal, tick-level Double DQN learning is followed by completed-round history and UCB updates. Shared-board construction and parallel LLM regeneration occur after every non-final round. The final round terminates before board construction or another LLM call.
C. UCB-Guided Policy Refinement UCB1 supplies a robot-local exploration–exploitation rule over five interpretable LLM refinement modes [19], Z = {KEEP_CURRENT, ADOPT_BEST_SHARED, SMALL_MUTATION, EXPLORE_NEW,
(8)
HYBRID_WITH_BEST}. The modes ask the LLM, respectively, to retain the current policy, adopt useful components from the best shared policy, apply a small numerical mutation, explore a substantially different policy, or combine stable local components with useful shared components. An arm is natural-language refinement advice; it neither selects a motion action nor chooses an LLM backend. Untried arms are selected first in the fixed order listed above. If arm zik generated the policy evaluated in round k, its count and empirical value are updated as ni,zik ← ni,zik + 1, Ri,zik ← Ri,zik +
Rik − Ri,zik ni,zik
.
(9)
Here, ni,z is the number of credited rollouts and Ri,z is the online empirical mean. The initial LLM policy is not associated with a UCB arm and is therefore not credited. For a successful rollout, the implementation defines the optimistic movement lower bound 0 di Li = max 1, , δmax = 3.0, (10) δmax where d0i is the fixed start-to-goal distance and δmax is the largest sanitized forward step. This bound counts translation ticks without adding turns. The UCB credit is yik = 0, 0, , (11) Rik = round6 Li min 1, , yik = 1, k max(Ti , 1) where round6 is the code’s six-decimal rounding operation. Thus, Rik ∈ [0, 1]: it is zero for failure, equals one at the optimistic lower bound, and decreases with successful completion time. Accumulated tick-level DQN reward is excluded because
the bandit credits the round-level refinement mode rather than individual actions. The denominator clamp reproduces the code’s protection against a zero finish-tick value. After all arms have been observed, the next mode is selected by s " # ln(ni + 1) k+1 , c = 1.4, zi = arg max Ri,z + c z∈Z ni,z (12) P where ni = z∈Z ni,z . The evaluated implementation fixes c = 1.4 for every robot and does not tune it by LLM backend, configuration, or round. UCB therefore selects only the next refinement mode; numerical and categorical policy synthesis remains the responsibility of the robot’s LLM. D. Policy-Conditioned Local Double DQN Control Each robot implements a Double DQN controller based on DQN and Double Q-learning [17], [18]. It owns an online − network Qi (·; ϕi ) and a target network Q− i (·; ϕi ) with architecture R15 → 64 ReLU → 64 ReLU → R5 . (13) The two hidden layers each contain 64 ReLU units, and the five outputs are the Q-values for the five action categories in A. The 15-dimensional state concatenates navigation, activepolicy, and LLM-prior components: i h policy prior ski,t = snav , si,t . (14) i,t , si,t Each block contributes five entries, yielding the 15dimensional input used by the implemented network. With dt and et denoting the current distance and signed angle error, the navigation component is dt dt et , , 1[d ≤ d ], 1[|e | ≤ α], . (15) snav = t near t i,t 100 180 100 The fifth entry reserves a distance-history channel within the fixed 15-dimensional interface. When the post-action next state is encoded, rl-prev-distance still stores the pre-action distance; the replayed transition therefore contains both the post-action distance in the first entry and its preceding value in the fifth entry, providing an explicit one-step progress signal for the Double-DQN update. The tracker is synchronized
IEEE JOURNAL MANUSCRIPT
6
after the transition is stored, so the two entries coincide when the following online action is selected. Retaining this channel keeps the feature layout consistent across online action selection, replay storage, and minibatch training. The policy component is δ δnear ω α dnear spolicy = , , , , , (16) i,t 3.0 1.0 60.0 45.0 10.0 and sprior = pi,t ∈ {0, 1}5 is the one-hot encoding of i,t LLM ai (dt , et ). Conditioning on the policy parameters prevents the same categorical action from appearing identical to the DQN after an LLM update changes its physical step or turn magnitude. During training, the behavior policy is ( Uniform(A), u < ϵi , ai,t = (17) arg max [Qi (si,t , a; ϕi ) + βpi,t,a ] , u ≥ ϵi , a∈A
where u ∼ Uniform(0, 1) and ϵi is the robot’s current exploration probability. The evaluated C4 controller fixes β = 0.05; the vanilla DQN controllers in C1 and C2 do not apply this additive prior bias. In C4, the bias guides exploitation without preventing learned Q-value differences from selecting another action. For replay transition (si,t , ai,t , ri,t , si,t+1 , ηi,t ), where ηi,t = 1 for a terminal transition, Double DQN uses the online network, including the same LLM prior, to select the next action: a∗i,t+1 = arg max [Qi (si,t+1 , a; ϕi ) + βpi,t+1,a ] . a∈A
(18)
The target network evaluates that action, − ∗ ybi,t = ri,t + γ(1 − ηi,t )Q− i (si,t+1 , ai,t+1 ; ϕi ),
γ = 0.99. (19) The prior bias therefore affects online-network action selection in the preceding equation, but it is not added to the targetnetwork value gathered in this equation. For mini-batch DB , the online network minimizes Smooth L1 loss, 1 X ℓH (Qi (sj , aj ; ϕi ) − ybj ) , (20) Li = |DB | j∈DB
where
( ℓH (x) =
1 2 2x ,
|x| − 12 ,
|x| < 1, |x| ≥ 1.
(21)
Thus, x = Qi (sj , aj ; ϕi ) − ybj is the temporal- difference residual, and the implementation averages this Smooth L1 loss over the sampled mini-batch. For transition (st , at , rt , st+1 ), let (dt , et ) and (dt+1 , et+1 ) denote the pre-action and post-action distance and signed angle error. The implemented reward is the sum of independently activated components,
TABLE I I MPLEMENTED T ICK -L EVEL R EWARD C OMPONENTS Component Base Away Near-away Corrective turn
Activation condition Value Every executed transition −0.01 dt+1 > dt + εd −0.02 dt ≤ dnear and away −0.05 at = TURN_ONLY, |et | > α, +0.005 |et+1 | < |et | − εe Other turn-only TURN_ONLY and corrective condition false −0.01 Near large motion dt ≤ dnear , σt+1 = 0, −0.01 at ∈ {FORWARD, TURN_AND_FORWARD} Success σt+1 = 1 +1.0 Timeout φt+1 = 1 −1.0
Algorithm 1 Two-Level LLM–UCB–DQN Procedure 1: Initialize one policy with each dedicated LLM in parallel. 2: Initialize independent UCB1 and Double DQN states for all robots. 3: for k = 1, . . . , K do 4: Reset turtle poses and round-specific DQN statistics. 5: while at least one robot is active and t < Tmax do 6: for each active robot ri do 7: Encode ski,t and select aki,t . 8: Execute aki,t using θ ki . k and observe sk 9: Compute ri,t i,t+1 . 10: Store the transition and train local Double DQN. 11: end for 12: Advance the NetLogo tick. 13: end while 14: Record yik , Tik , local history, and DQN feedback. 15: Update the previously selected UCB arm, if one exists. 16: Select zik+1 for every robot; update each successful robot’s best-tick and best-policy records. 17: if k < K then 18: Construct Bk from all robot summaries. 19: Generate and sanitize all θ k+1 in parallel. i 20: end if 21: end for 22: Return finish-tick trajectories and persistent learning statistics.
Because the away and near-away tests are independent, both penalties apply when a robot moves away from the goal inside the near-goal region. A nonterminal corrective turn with no other activated penalty has net reward −0.005 after the base cost. Success and timeout are also added independently. If exact-goal arrival and timeout occur on the same transition, their terminal contributions cancel, while the base and any other activated components remain. Training uses Adam with learning rate 10−3 , batch size 32, replay capacity 5000, and target-network synchronization every 100 successful training updates. After each mini-batch update, ϵi ← max(0.05, 0.997 ϵi ),
ϵ0i = 1.0.
(23)
Thus, exploration starts at one, decays by a factor of 0.997 after each successful mini-batch update, and is lower-bounded by 0.05. Learning begins when the replay buffer contains at least 32 transitions. Algorithm 1 gives the end-to-end procedure for the evaluaway near-away large turn success timeout ated configuration. rt = −0.01+rt +rt +rt +rt +rt +rt . (22) IV. E XPERIMENTAL D ESIGN Table I gives every component used by the code, with εd = εe = 10−6 . Here, σt+1 is the post-action exact-goal indicator A. Evaluation Objectives and φt+1 = 1[t + 1 ≥ Tmax ] is the timeout flag passed to the The evaluation addresses four questions. RQ1 examines the joint effect of cross-LLM communication and UCB DQN.
IEEE JOURNAL MANUSCRIPT
7
TABLE II C ONTROLLER C ONFIGURATIONS Configuration Shared board UCB Tick-level control Input dim. Prior bias C1: Local LLM–DQN No No Vanilla DQN 5 No C2: Cross-LLM–UCB + DQN Yes Yes Vanilla DQN 5 No C3: Cross-LLM–UCB, no DQN Yes Yes Regional rule – – C4: Complete configuration Yes Yes Double DQN 15 0.05
refinement-mode selection. RQ2 compares learned tick-level local control with direct execution of LLM-generated regional rules under the same round-level components. RQ3 compares the policy-conditioned Double DQN in C4 with the state-only vanilla DQN in C2. RQ4 characterizes round-level adaptation and robot-specific outcomes in C1 and C4. Because each LLM backend remains assigned to the same robot, robot-wise results describe the instantiated composite heterogeneous robots and do not rank the backends. B. Controller Configurations Table II defines the four configurations. Each retains one dedicated LLM per robot and follows the round-level schedule in Section III-B. The world, start and goal assignments, action vocabulary, and motion interface are fixed; the controller and learning differences are stated below. C1 uses robot-local history for subsequent policy refinement and disables the shared board and UCB; its vanilla DQN receives the five-dimensional navigation observation. C2 enables cross-LLM communication and UCB refinement-mode selection while retaining the same vanilla DQN. C3 keeps the round-level LLM–UCB components and executes the current policy’s regional action preference directly, without DQN. The complete configuration C4 uses the 15-dimensional policyconditioned Double DQN described in Section III. C1–C2 changes cross-LLM communication and UCB jointly, so it cannot separate their effects. C2–C3 removes the learned tick-level controller. C1 and C2 use a five-dimensional vanilla DQN with γ = 0.95, MSE loss, an ϵ-decay factor of 0.9995, and the distance-progress reward implemented in those configurations. C4 uses the 15-dimensional Double DQN with γ = 0.99, Smooth L1 loss, an ϵ-decay factor of 0.997, the reward in Table I, and the additive LLM-prior bias β = 0.05. C2–C4 is therefore a bundled configuration comparison rather than an isolated test of Double DQN. C. Common Simulation Setup The NetLogo world spans [−50, 50] × [−50, 50], and the common goal is located at (10, −5). The three robots start at (49, 40), (−29, 40), and (−29, −50) and initially face the goal. They are respectively paired with Llama-3.3-70B-Instructquantized, Phi-4, and Meta-Llama-3.1-70B-Instruct-quantized. These assignments, the five-action motion interface, and all physical transition rules are fixed across configurations. Each configuration is evaluated for 30 rounds, with at most 1000 ticks per round. This produces 90 robot–round records (three robots over 30 rounds). Because policy and learning states persist across rounds, these records are analyzed as repeated measurements rather than independent replicates. The
Fig. 4. Representative NetLogo rollout with three composite heterogeneous robots and the common goal patch. Text labels show the active runtime action and local best-tick record for that displayed state.Each robot is connected to an LLM, specifically, LLama3.1, Phi4, Llama3.3 respectively.
round-level LLM calling schedule follows Section III-B. A robot’s round ends when it reaches the goal patch or exhausts the tick budget. D. LLM Backends and Simulation View The three dedicated policy agents use Llama-3.3-70BInstruct, Phi-4, and Llama-3.1-70B-Instruct. Llama 3.1 and 3.3 belong to Meta’s instruction-tuned Llama 3 family, which supports multilingual text, coding, reasoning, and tool-use tasks [20]. The Llama-3.3 model card specifies a 70B text model with a 128k-token context window, optimized for multilingual dialogue [21]. Phi-4 is a 14B model whose training emphasizes high-quality data, synthetic reasoning data, and post-training; its technical report also identifies strict instruction following as a limitation [22]. These published characteristics motivate using distinct backends for robotspecific policy generation, but they do not predict navigation performance directly. The evaluated endpoints include servicespecific quantized Llama variants, and the exact serving stack and quantization parameters are not exposed by the current interface. Figure 4 shows a representative NetLogo rollout. The colored turtles are paired with their dedicated LLM agents, and the green patch is the common goal. On-screen action and best-tick labels are runtime status indicators, not aggregate experimental statistics. Because each model is permanently paired with one start pose and one local learner, later agentlevel differences are interpreted as properties of the complete instantiated robot–controller combinations rather than strengths or weaknesses of the isolated LLMs. E. Evaluation Protocol and Metrics At the start of every round, robot positions, headings, and round-local counters are reset to the common initial conditions. The enabled UCB statistics, DQN learner state, policy history, and best-policy record persist across rounds.
IEEE JOURNAL MANUSCRIPT
8
The policy generated at the round boundary remains fixed during the ensuing rollout, while the tick controller defined in Table II produces the physical actions. For each robot–round observation, the simulator records goal attainment and the successful completion tick. The primary metrics are success rate and the distribution of successful completion ticks, summarized by the median, interquartile range (IQR), mean, and 90th percentile (P90). Round-wise traces and early-to-late medians describe adaptation over the fixed horizon. Two complementary descriptive metrics combine success with tick efficiency and retain slow-tail behavior. Episode reward, exploration rate, action counts, and training loss are retained only as DQN diagnostics. LLM computation cost is outside the scope of this draft. F. Primary Metrics and Analysis k ∈ {0, 1} indicate whether robot i in configuration c Let yi,c reaches the goal in round k. With N = 3 robots and K = 30 rounds, the task-success rate is N K 1 XX k yi,c . SRc = N K i=1
(24)
k=1
The value Tmax = 1000 is a timeout code rather than a successful completion tick. The data contain one such event, for the Llama-3.1 composite robot under C2 in round 12. Success rate is reported over all 90 robot–round records for each configuration, while completion-time summaries use only successful records. Because within-configuration records share persistent policy and learning states, these summaries are descriptive rather than estimates based on independent trials. Because the successful finish-tick distributions are rightskewed, the primary efficiency statistic is the median with the interquartile range (IQR). To describe adaptation without selecting a favorable round, the round-wise trace uses the median of the three recorded values, with a timeout retained at Tmax . Early and late performance pool the successful observations from rounds 1–5 and 26–30, respectively. Their relative change is Ic = 100
Mcearly − Mclate Mcearly
,
(25)
where a positive Ic denotes fewer ticks in the final five rounds. Because policy and learning states persist across rounds, this quantity describes change over the evaluation horizon and is not an inferential effect estimate. Following the success-weighted time-step metric used in navigation evaluation [23], lower-bound-normalized success efficiency (LBSE) is ! N K 1 XX k Li LBSEc = yi,c min 1, , (26) k , 1) N K i=1 max(Ti,c k=1 where Li = ⌈d0i /3⌉ is the optimistic movement lower bound already used by the round-level UCB reward. The fixed start coordinates give Li = 20 for all three robots. Hence, LBSEc ∈ [0, 1], with larger values indicating more successful completions closer to this lower bound. Because it averages
the same efficiency signal used internally by UCB, LBSE is an implementation-aligned diagnostic rather than independent validation of the bandit objective. To retain rare slow outcomes, we also report the empirical worst-decile completion cost, a finite-sample upper-tail summary motivated by risk-sensitive RL practice [24]. Let Tc,(1) ≤ · · · ≤ Tc,(N K) be all tick costs in configuration c, including a timeout as Tmax , and let m = N K−⌊0.90N K⌋ = 9. Then NK X 1 Tc,(j) . (27) WDC0.10,c = m j=N K−m+1
Lower values indicate a smaller average cost among the slowest 10% of the 90 observations. This statistic is a finite-sample upper-tail mean, not an estimate of population CVaR. LBSE and WDC0.10,c remain descriptive summaries of the recorded configuration-level data; neither supplies an independent-sample significance test. V. R ESULTS AND A NALYSIS The four configurations are reported as bundled system ablations. To expose the progression requested for the analysis, this section first examines the local LLM–DQN baseline (C1), then the two cross-LLM configurations with vanilla DQN (C2) or no DQN (C3), and finally the complete policy-conditioned Double DQN configuration (C4). The configurations do not form a strict single-factor sequence: C1 already uses roundlevel robot-local LLM refinement, and C2–C4 differ in more than one implementation choice. The results therefore support configuration-level observations rather than isolated causal effects. Figure 5 uses the same disclosed piecewise-linear vertical scale: the 0–100-tick interval occupies 70% of the plot height, while 100–300 ticks occupy the remaining 30%. The axis break at 100 ticks marks the change of scale, and observations above 300 ticks retain their actual numerical labels. A. C1: Local LLM–DQN Baseline C1 disables the shared board and UCB while retaining one robot-local LLM and a five-dimensional vanilla DQN. All 90 robot–round records were successful. Its successful completion-tick median was 69, with a mean of 91.5 and a P90 of 105.8 ticks. Figure 5(a) shows that the early rounds contain several high-tick observations, including the largest successful value in the dataset (644 ticks). The pooled median decreased from 98 ticks in rounds 1–5 to 60 ticks in rounds 26–30. Thus, the local history and DQN state were sufficient to produce a faster late segment, but the configuration retained a pronounced upper tail. B. C2: Cross-LLM Communication, UCB, and Vanilla DQN C2 enables the shared board and robot-local UCB refinement-mode selection while retaining the fivedimensional vanilla DQN. It recorded 89 successes and one timeout, corresponding to a success rate of 98.9%. Among successful observations, the median was 56 ticks,
IEEE JOURNAL MANUSCRIPT
9
(a) C1: LLM (no shared board) + DQN.
(b) C2: LLM with shared board communication + UCB + DQN.
(c) C3: LLM with shared board communication + UCB but without DQN.
(d) C4: LLM with shared board communication + UCB + Double DQN.
Fig. 5. Completion ticks over 30 rounds for the four bundled configurations. Each curve corresponds to one fixed composite heterogeneous robot, and the dashed line is the pooled median. All panels use the same piecewise-linear vertical scale, which expands 0–100 ticks and compresses 100–300 ticks. The break marks the change of scale; values above 300 ticks are placed at the upper boundary and labeled with their actual values, while the C2 timeout is marked as TO.
the mean was 72.7, and the P90 was 89.2. The early and late medians were 87 and 48 ticks, respectively. As shown in Fig. 5(b), most observations moved toward the lower-tick region, although isolated slow rounds remained and the Llama-3.1 composite robot timed out in round 12. Relative to C1, the lower median is associated with the combined addition of cross-LLM communication and UCB; the present configuration does not separate those two components. C. C3: Cross-LLM Communication and UCB Without DQN C3 preserves the round-level cross-LLM and UCB components but executes the current regional rule directly. All 90 robot–round records for C3 reached the goal. Its pooled median was 57 ticks, close to C2, while its mean and P90 were 64.0 and 89.0 ticks. Its temporal profile differed from the DQN-enabled configurations: the early median was 46 ticks, but the late median increased to 78 ticks. Figure 5(c) shows that the slowest observations are concentrated near the end of the horizon, with final-round values of 56, 146, and 184 ticks for the three robots. The no-DQN configuration was therefore
Fig. 6. Successful robot–round completion ticks by configuration. Boxes span the IQR, horizontal bars show the median, whiskers span the 5th–95th percentiles, and lightly jittered points show individual successful observations. The C2 timeout is excluded here and retained in Table III and the worst-decile cost.
competitive in the pooled median but did not exhibit the same late-round reduction observed in C1, C2, and C4.
IEEE JOURNAL MANUSCRIPT
10
TABLE III OVERALL PERFORMANCE OVER 30 ROUNDS . C OMPLETION - TIME STATISTICS EXCLUDE TIMEOUTS ; P90 IS THE 90 TH PERCENTILE OF SUCCESSFUL COMPLETION TICKS . E ARLY AND LATE MEDIANS POOL SUCCESSFUL OBSERVATIONS FROM ROUNDS 1–5 AND 26–30. C1 C2 C3 C4 90/90 90/90 89/90 90/90 Success (SR %) (100.0) (98.9) (100.0) (100.0) 69.0 42.0 56.0 57.0 Median [IQR] [49.3,87.8] [48.0,66.0] [51.3,64.0] [40.0,59.8] Mean 91.5 72.7 64.0 56.3 P90 105.8 89.2 89.0 73.2 Early 98 87 46 57 Late 60 48 78 41 Ic (%) 38.8 44.8 −69.6 28.1
TABLE V S UCCESSFUL COMPLETION - TICK MEDIANS BY COMPOSITE ROBOT. Config. Llama-3.3 Phi-4 Llama-3.1 C1 57.5 80.0 46.5 C2 55.0 57.0 48.0† C3 56.5 57.0 56.0 C4 46.5 35.0 54.5
Metric
TABLE IV C OMPLEMENTARY EFFICIENCY AND SLOW- TAIL SUMMARIES . Config. C1 C2 C3 C4
LBSE ↑ 0.310 0.339 0.339 0.431
WDC0.10 (ticks) ↓ 330.0 313.3 127.3 151.8
D. C4: Complete Multi-Timescale Configuration C4 combines cross-LLM communication, robot-local UCB advice, and policy-conditioned local Double DQN control. All 90 robot–round records for C4 reached the goal, and the configuration had the lowest observed median (42 ticks), mean (56.3 ticks), and P90 (73.2 ticks). Its early median of 57 ticks decreased to 41 ticks in the final five rounds. Figure 5(d) also exposes non-monotonic behavior: the Llama3.3 and Llama-3.1 robots contain isolated values of 232 and 288 ticks, respectively. The Phi-4 composite robot had the lowest within-configuration median (35 ticks) and the smallest IQR (7 ticks). These are agent-level observations under fixed start poses, policies, and learners; they do not establish an intrinsic ranking of the three LLM backends. E. Cross-Configuration Comparison Table III and Fig. 6 compare the four bundled ablations over the complete horizon. C4 reduced the successful-observation median by 39.1%, 25.0%, and 26.3% relative to C1, C2, and C3, respectively. Its P90 was 17.8% below the nextlowest value (89.0 ticks in C3), indicating that its difference was present in the upper portion of the observed distribution as well as at the median. The success rates differ by only 1.1 percentage points: C1, C3, and C4 reached 90/90 goals, whereas C2 reached 89/90. Because the success-rate bars would add little information beyond Table III, Fig. 6 displays only the completion-tick distributions. C2 and C3 had similar medians, but their early-to-late trajectories moved in opposite directions. This distinction would be hidden by a best-round or pooled-median-only comparison. Table IV separates typical success-weighted efficiency from rare slow behavior. C4 had the highest LBSE, 39.0%, 27.3%, and 27.2% above C1, C2, and C3, respectively. C3 instead had the lowest worst-decile cost. Its value of 127.3 ticks was 16.1% below C4’s 151.8 ticks because C4 retained isolated 232- and 288-tick rounds. The complete configuration therefore had the
† Median over 29 successful observations; the remaining observation timed
out.
strongest aggregate normalized efficiency in this run, but it did not dominate every slow-tail criterion. The agent-level medians in Table V show that the aggregate C4 ordering was not uniform: C1 had the lowest median for the Llama-3.1 composite robot. Each column combines a fixed LLM backend, start pose, policy history, and local controller state. Accordingly, the table characterizes the instantiated composite heterogeneous robots rather than the isolated language models. The comparisons answer the evaluation questions only at the configuration level. C1–C2 changes communication and UCB jointly. C2–C3 removes learned tick-level control, but C3 also executes a different action-selection mechanism. C2– C4 changes the state representation, Q-learning rule, discount factor, loss, exploration schedule, reward, and prior bias. The four groups are therefore informative ablations of implemented systems, not single-factor estimates. F. Evidence Scope The reported results are descriptive configuration-level summaries. The 90 robot–round records for each configuration share persistent policies, UCB statistics, replay buffers, network weights, exploration states, and shared-board information across rounds, and are therefore treated as repeated measurements rather than independent replicates. These results do not support significance tests or causal attribution. Their scope is limited to the fixed NetLogo world, start assignments, goal, and LLM-to-robot mapping in Section IV-C. VI. C ONCLUSION This paper presented a decentralized multi-timescale system for composite heterogeneous robots, combining robot-local policy ownership, UCB-guided round-level refinement, and tick-level control. Cross-LLM communication is performed at round boundaries to support robot-specific policy refinement, whereas physical actions are selected at the tick level without repeated LLM calls. In the complete configuration, the local Double DQN conditions its value estimates on navigation state, the active LLM-generated policy, and the corresponding regional-action prior. Reinforcement learning therefore assists the LLM policy layer while leaving policy generation and robot-specific reasoning at the round level. Across the four configurations, C4 combined complete observed success with the lowest pooled median and P90, indicating better aggregate central and upper-tail completion behavior in the fixed task. C3 remained competitive in pooled median completion time but worsened over the evaluation
IEEE JOURNAL MANUSCRIPT
11
horizon, whereas agent-level medians were not uniformly lowest under C4. The strongest observed advantage was therefore configuration-level rather than uniform across robots or rounds. The 90 robot–round records for each configuration share persistent policy and learning states across rounds and are therefore interpreted as repeated measurements rather than independent replicates. The four configurations also contain bundled implementation differences, so the present results neither isolate the causal contribution of every module nor demonstrate generalization to other environments or physical robots. A. Future Work Future experiments will introduce independent seeded repetitions and separate evaluation rollouts to quantify uncertainty independently of ongoing exploration. Single-factor ablations will isolate cross-LLM communication, UCB advice, the policy-conditioned state, Double DQN, reward design, and the LLM-prior bias. An explicit RL-only baseline and a direct pertick LLM-control baseline will complete the progression from local reinforcement learning to round-level LLM assistance and cross-LLM communication. The fixed UCB coefficient will be subjected to sensitivity analysis. The evaluation will also record LLM latency, token consumption, response validity, and fallback frequency. Further studies will vary LLM-to-start assignments, communication quality, obstacles, and task geometry, and will test physically heterogeneous platforms. These extensions are required before making claims about statistical reliability, computational efficiency, scalability, or real-world deployment. R EFERENCES [1] Z. Mandi, S. Jain, and S. Song, “RoCo: Dialectic multi-robot collaboration with large language models,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024, pp. 286–299, doi: 10.1109/ICRA57147.2024.10610855. [2] Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan, “Scalable multi-robot collaboration with large language models: Centralized or decentralized systems?” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024, pp. 4311–4317, doi: 10.1109/ICRA57147.2024.10610676. [3] H. Zhang, W. Du, J. Shan, Q. Zhou, Y. Du, J. B. Tenenbaum, T. Shu, and C. Gan, “Building cooperative embodied agents modularly with large language models,” in Proc. Int. Conf. Learn. Representations (ICLR), 2024. [4] S. S. Kannan, V. L. N. Venkatesh, and B.-C. Min, “SMART-LLM: Smart multi-agent robot task planning using large language models,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2024, pp. 12140–12147, doi: 10.1109/IROS58592.2024.10802322. [5] W. Yu, J. Peng, Y. Ying, S. Li, J. Ji, and Y. Zhang, “MHRC: Closed-loop decentralized multi-heterogeneous robot collaboration with large language models,” arXiv:2409.16030, 2024, doi: 10.48550/arXiv.2409.16030. [6] K. Liu, Z. Tang, D. Wang, Z. Wang, X. Li, and B. Zhao, “COHERENT: Collaboration of heterogeneous multi-robot system with large language models,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025, pp. 10208–10214, doi: 10.1109/ICRA55743.2025.11127808. [7] H. Ben Amor, L. Graesser, A. Iscen, D. D’Ambrosio, S. Abeyruwan, A. Bewley, Y. Zhou, K. Kalirathinam, S. Mishra, and P. Sanketi, “SASPrompt: Large language models as numerical optimizers for robot selfimprovement,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025, pp. 10087–10094, doi: 10.1109/ICRA55743.2025.11127882. [8] C. daCunha, G. Giardini and M. Turqueti,“LLM-based Reinforcement Learning for Controlling Robot Swarms,”in 2026 7th International Conference on Artificial Intelligence, Robotics, and Control (AIRC), 2026, pp. 236-241, doi: 10.1109/AIRC69745.2026.11631479.
[9] W. Ji et al., “GenSwarm: Scalable multi-robot code-policy generation and deployment via language models,” npj Robot., vol. 4, Art. no. 5, 2026, doi: 10.1038/s44182-025-00065-w. [10] R. Mon-Williams et al., “Embodied large language models enable robots to complete complex tasks in unpredictable environments,” Nat. Mach. Intell., vol. 7, pp. 592–601, 2025, doi: 10.1038/s42256-025-01005-x. [11] J. Chen, C. Yu, X. Zhou, T. Xu, Y. Mu, M. Hu, W. Shao, Y. Wang, G. Li, and L. Shao, “EMOS: Embodiment-aware heterogeneous multirobot operating system with LLM agents,” in Proc. Int. Conf. Learn. Representations (ICLR), 2025. [12] Y. Zhuang, Y. Shen, Z. Zhang, Y. Chen, and F. Miao, “YOLO-MARL: You only LLM once for multi-agent reinforcement learning,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2025, pp. 11283–11290, doi: 10.1109/IROS60139.2025.11245945. [13] S. Morad, A. Shankar, J. Blumenkamp, and A. Prorok, “Languageconditioned offline RL for multi-robot navigation,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025, pp. 14984–14991, doi: 10.1109/ICRA55743.2025.11127288. [14] G. Zhu, R. Zhou, W. Ji, and S. Zhao, “LAMARL: LLM-aided multi-agent reinforcement learning for cooperative policy generation,” IEEE Robot. Autom. Lett., vol. 10, no. 7, pp. 7476–7483, 2025, doi: 10.1109/LRA.2025.3577527. [15] T. Yoshida and Y. Sueoka, “Communication-free adaptive swarm robotic system: LLM-based decision making and MARL-based multi-policy control,” J. Robot. Mechatron., vol. 37, no. 6, pp. 1452–1460, 2025, doi: 10.20965/jrm.2025.p1452. [16] J. He, S. Chen, F. Zhang, and Z. Yang, “From words to actions: Unveiling the theoretical underpinnings of LLM-driven autonomous systems,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), PMLR, vol. 235, 2024, pp. 17807–17841. [17] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, 2015, doi: 10.1038/nature14236. [18] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with Double Q-learning,” in Proc. AAAI Conf. Artif. Intell., vol. 30, no. 1, 2016, pp. 2094–2100, doi: 10.1609/aaai.v30i1.10295. [19] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” Mach. Learn., vol. 47, nos. 2–3, pp. 235– 256, 2002, doi: 10.1023/A:1013689704352. [20] A. Grattafiori et al., “The Llama 3 herd of models,” arXiv:2407.21783, 2024, doi: 10.48550/arXiv.2407.21783. [21] Meta, “Llama 3.3 model card,” Hugging Face, Dec. 2024. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct [22] M. Abdin et al., “Phi-4 technical report,” arXiv:2412.08905, 2024, doi: 10.48550/arXiv.2412.08905. [23] X. Wang, Y. Liu, X. Song, Y. Liu, S. Zhang, and S. Jiang, “An interactive navigation method with effect-oriented affordance,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 16446–16456, doi: 10.1109/CVPR52733.2024.01556. [24] X. Ni, G. Liu, and L. Lai, “Risk-sensitive reward-free reinforcement learning with CVaR,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), PMLR, vol. 235, 2024, pp. 37999–38017. [Online]. Available: https://proceedings.mlr.press/v235/ni24c.html