©2026 IEEE. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.
Potential-Field Action Representation for Reinforcement Learning in Contact-Rich Manipulation
arXiv:2609.21609v1 [cs.RO] 18 Sep 2026
Xinyu Liu, Gökhan Solak and Arash Ajoudani Abstract— Model-free reinforcement learning can acquire contact-rich robotic manipulation skills through trial-error interaction, but it often requires the policy to learn both task strategy and low-level motion generation from data. In this setting, the action representation is critical because it determines how policy outputs are converted into robot motion and therefore influences both exploration and physical execution. Direct Cartesian motion-command interfaces are widely used, but they require the policy to generate motion commands at every decision step, coupling task-level adaptation with continuous low-level execution. This increases the learning burden, since the policy must also discover commands that are smooth, bounded, and physically suitable for compliant interaction. We propose reinforcement learning with artificial potential-fields as action representation (PA-RL). Instead of commanding motion directly, the policy adapts the parameters of an energy-like potential field which provides a state-dependent guidance direction that is executed through a Cartesian impedance controller. We evaluate the PA-RL framework on peg-in-hole insertion, a representative contact-rich task with nonlinear robot dynamics and discontinuous contact transitions. Simulation experiments compare PA-RL with direct Cartesian velocity, Cartesian pose, and variable-impedance action spaces trained with the same RL algorithm. PA-RL is the first and only method train a policy to reach a 100% evaluation success rate within given time while the best baseline reaches 92.6%. Compared with the bestperforming baseline, PA-RL reduced the joint-torque variation by 55.4% and Cartesian acceleration variation by 70.8%, respectively, without explicit motion quality penalties in the reward. The simulation-trained policy further completed all 9/9 real-robot insertions without policy fine-tuning, demonstrating deployment feasibility of the learned potential-field interface.
I. INTRODUCTION Humanoid robots are expected to operate in humancentered environments and perform diverse manipulation tasks that involve sustained physical interaction, including assembly, maintenance, and tool use. [1], [2] Such contact-rich manipulation is challenging because robot motion is affected by constrained geometry, nonlinear dynamics, discontinuous contact transitions, and uncertainty in the task state [3], [4]. Successful execution therefore requires the robot to adapt its high-level strategy to the task and contact conditions while This paper was supported by the European Union Horizon Projects TORNADO (Grant GA 101189557) and by the Italian Ministry of University and Research (MUR) under the Fondo Italiano per la Scienza (FIS), call FIS 3, project EPIC with code FIS-2024-02654. Xinyu Liu is with Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, 16121 Genoa, Italy, and also with the Ph.D. Program of National Interest in Robotics and Intelligent Machines (DRIM), Università di Genova, 16126 Genoa, Italy. Gökhan Solak and Arash Ajoudani is with the Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, 16163 Genoa, Italy.
: Gaussian basis
: Linear basis
: Current tool position : Attractor position : Vector field : Resultant velocity
Fig. 1. Overview of the PA-RL framework for the peg-in-hole task. The policy action consists of the positions and parameters of several attractors with linear and gaussian bases, defining a state-dependent artificial potential field over the goal-relative workspace that is illustrated as a vector field (green) along its gradient. The velocity reference is generated following the negative gradient of the potential field.
maintaining precise and compliant motion during physical interaction. Robot control methods, such as operational space control and impedance control, provide effective structures for precise and compliant motion during physical interaction. However, the task-level reference trajectory and interaction strategies are commonly designed manually and require substantial adaptation when task conditions change. Reinforcement learning (RL) provides a complementary approach by allowing robots to acquire difficult-to-engineer behaviors through trial-and-error interaction. Nevertheless, the learning algorithm and reward function alone do not determine how a learned policy interacts with the robot. The action representation is critical: it defines how policy decisions are translated into executable motion and influences both exploration and physical execution [5]–[7]. Many RL methods use direct motion-command action representations, including joint commands, pose increments, and velocities. These representations are general and flexible, but they couple tasklevel decision making with low-level motion generation: the same policy output must both select an interaction strategy and produce an executable command. Consequently, variations between consecutive policy decisions can be directly reflected in the robot motion, which may reduce exploration
efficiency and execution smoothness during contact. For contact-rich manipulation, this motivates action representations that encode a spatial motion structure rather than an isolated step-wise motion command. Artificial potential fields provide a particularly relevant representation because they encode motion as an energy-like landscape over the task space. Their negative gradient acts as a virtual force, yielding guidance that is physically interpretable, spatially structured, and smooth over the task space rather than defined only as isolated step-wise commands. However, conventional artificial potential fields [8], [9] are commonly designed manually and remain fixed during execution. Their behavior can therefore depend strongly on the selected field parameters, limiting their adaptability. Therefore, this paper proposes PA-RL that combines reinforcement learning and artificial potential-fields. The core idea is to use a parameterized artificial potential field as the action representation of the RL policy. Based on goal-relative state and force–torque observations, the PA-RL policy adapts the positions, weights, and widths of a set of attractors. These parameters define a state-dependent potential field, as shown in Fig. 1, whose negative gradient is used to generate a bounded reference trajectory tracked by a fixed-gain Cartesian impedance controller. Consequently, the policy adapts the spatial structure of the motion, while continuous execution is retained within the potential-field and impedance control structure. The proposed method is evaluated on peg-in-hole, a representative contact-rich task with nonlinear dynamics and discontinuous contact transitions [10], [11]. Practically, the precise hole pose is difficult to obtain because of calibration errors, sensing uncertainty, and manufacturing tolerances. Therefore, successful insertion requires the policy to use not only the estimated geometric state but also contactrelated observations to correct its motion during execution. These characteristics make peg-in-hole insertion a suitable benchmark for evaluating adaptive policy interfaces. PA-RL is evaluated against three representative actionspace baselines: end-effector velocity (TCP-Vel), incremental end-effector pose (TCP-Pose) and VICES-style variableimpedance that outputs an incremental pose together with stiffness and damping parameters. All methods use the same RL algorithm and reward, isolating the role of the action representation. The results show that the PA-RL improves the learning efficiency, task performance, and motion quality, and that the simulation-trained policy transfers to the real robot without fine-tuning. The main contributions of this work are: • The PA-RL framework that uses a state-dependent artificial potential field as RL action representation. The framework decouples learned task-level adaptation from continuous reference generation and execution, providing a suitable framework for robot control. • A systematic comparison of PA-RL with TCP-Vel, TCPPose, and a VICES-style variable-impedance baseline with the same RL algorithm. The evaluation examines learning efficiency, task performance, and motion qual-
•
ity. A real-robot deployment of the simulation-trained PARL policy on a Franka Emika Panda without policy finetuning, demonstrating transfer feasibility and contactdependent adaptation in peg-in-hole insertion. II. R ELATED W ORK
A. Action Representations for Contact-Rich Manipulation RL has been applied to contact-rich manipulation using a wide range of action representations. Common choices include joint torques, joint positions, Cartesian poses, pose increments, and velocities [3], [4]. These representations are general and can be used with standard RL algorithms. Control-inspired action representations incorporate additional structure into this action representation. Variableimpedance approaches allow the policy to regulate task-space motion together with stiffness and damping parameters. For example, VICES studies end-effector variable-impedance control as an action space for constrained and contact-rich tasks [5]. Other methods embed energy-shaping, passivity, or stability-oriented controller structures into learned policies [7], [12], [13]. These approaches demonstrate that incorporating robot control structure into the policy representation can influence both learning and physical execution. Structured dynamical-system representations provide another alternative to independent step-wise commands. Dynamic movement primitives (DMPs) represent motion through attractor dynamics and a phase-dependent forcing term, providing a compact representation of temporally evolving trajectories [14], [15]. Neural Dynamic Policies further embed such differential-equation structures into deep policies for reinforcement and imitation learning, allowing the policy to predict a trajectory level representation rather than raw time-step-wise actions [6]. PA-RL emphasizes a different form of structure. Rather than encoding a trajectory through a prescribed phase progression, the policy parameterizes a spatial potential field that is evaluated at the current robot state and updated from geometric and contact-related observations. B. Artificial Potential Fields for Robot Control Artificial potential fields were introduced as a reactive approach to robot motion generation and obstacle avoidance [8]. A scalar potential is defined over the task space, and its negative gradient determines a state-dependent motion direction. Attractive and repulsive potential components can be combined to express task objectives and environmental constraints. Their spatial formulation and real-time performance have made artificial potential fields widely applicable to reactive robot motion. Artificial potential fields can be integrated into robot control systems in different ways. Given a potential function (U (x)), the negative gradient may be interpreted as a virtual force: F(x) = −∇x U (x).
(1)
It could be converted into a velocity or position reference, or combined with operational-space and impedance controllers.Potential fields have also been incorporated into structured trajectory representations; for example, potentialfield terms have been used to provide obstacle avoidance within DMP-generated motion [16]. It also embedded within manually structured task procedure like the approach stage in assembly task which allow different motion-generation mechanisms [17]. The behavior of an artificial potential field depends on the selected potential functions, their spatial locations, and their relative contributions. Classical formulations typically specify these elements manually for a given task and environment. Learning-based methods have therefore investigated stable dynamical systems and potential functions learned from demonstrations [18]. Such methods can encode demonstrated motion and compliant behavior within a learned spatial structure. In this study, PA-RL treats the parameters of the artificial potential field as the online action space of an RL policy. The field is reshaped during task execution according to the current state including force–torque observations, rather than learned once as a fixed motion model. The artificial potential field therefore acts as a bridge between adaptive task-level decisions and continuous robot execution.
A. Artifical Potential Field An artificial potential field (APF) defines a scalar function over the Cartesian task space whose negative gradient induces a motion direction. Let x ∈ R3 denote a controlled Cartesian point, and let pai ∈ R3 denote the center of the i-th basis. We construct the APF as a weighted combination of linear and Gaussian radial bases, X 1 Ui (x). (2) U (x) = P i |λi | i where λi is the signed weight of basis i. The normalization prevents the overall field magnitude from scaling directly with the absolute sum of learned weights. For a linear basis, the potential is defined as 1 λi ∥x − pai ∥2 . 2 which yields the vector-field contribution Uilin (x) =
filin (x) = −∇x Uilin (x) = λi (pai − x).
(3)
(4)
To increase the expressiveness of the field for contact-rich manipulation, we also include localized nonlinear bases. For a Gaussian radial basis, we use r π ∥x − pai ∥ gau √ erf . (5) Ui (x) = λi σi 2 2σi which gives
III. M ETHODOLOGY In the PA-RL framework, as illustrated in Fig. 2, the policy maps observations to attractor parameters that define a statedependent artificial potential field. Then, based on the field gradient, a Cartesian reference is generated and tracked by a Cartesian impedance controller. This decouples learned field adaptation from continuous motion generation and robot control. Cartesian Impedance Controller
Robot Environment
ξt
RL Policy
Observation
wt pc,t pa1,t
100 Hz
at
APF Compiler
Attractor Action
FAPF
Potential Field
Motion Planner
xd
Cartesian Target
pa1,t λa1,t σa1,t
pa2,t p̂g,t
pa2,t λa2,t σa2,t
Here, σi controls the spatial width of the Gaussian basis and ϵ > 0 avoids numerical singularities near the basis center. Using fi (x) to denote either the linear filin (x) or the Gaussian figau (x) basis function, the resulting APF vector field is P fi (x) FAPF (x) = Pi . (7) i |λi | In the proposed method, the APF parameters are not manually fixed. Instead, the RL policy outputs the basis parameters as described in the next section.
1000 Hz 0.5 Hz
figau (x) = −∇x Uigau (x) (6) x − pai ∥x − pai ∥2 . = −λi exp − 2 2σi ∥x − pai ∥ + ϵ
ẋt
pa0,t λa0,t
Fig. 2. Overview of the PA-RL pipeline. The upper diagram shows the closed-loop execution architecture and the lower panels illustrate a representative peg-in-hole example. Here, pc,t and pg,t are peg-tip position and goal position, and force–torque sensor wrench wt . The action specifies the attractor offsets pai ,t , weights λai ,t , and Gaussian sigma σai ,t .
B. Reinforcement Learning (RL) The RL policy adapts the APF parameters from the current task context. We use one linear attractor near the observed goal and N Gaussian attractors, resulting in N + 1 APF attractors in total. At each environment step t, the policy receives an observation composed of the goal-relative Cartesian state, the measured interaction wrench, and the current attractor offsets: ⊤ , (8) ξ t = ∆p⊤ wt⊤ ∆p⊤ A,t g,t where b g,t − pc,t ∆pg,t = p
(9)
denotes the displacement from the controlled Cartesian point b g,t . The vector wt ∈ R6 pc,t to the observed task goal p
is the measured force–torque signal. The stacked attractoroffset vector is ⊤ ⊤ ⊤ ∆pA,t = ∆p⊤ , (10) a0 ,t , ∆pa1 ,t , . . . , ∆paN ,t
The target is projected into the admissible workspace and constrained to remain within a bounded lead distance from the measured robot position,
where a0 denotes the linear attractor and a1 , . . . , aN denote the Gaussian attractors. Thus, the observation dimension is dim(ξ t ) = 3 + 6 + 3(N + 1). The policy outputs a normalized action i⊤ h ⊤ ⊤ at = ∆p⊤ , λ , σ , (11) A,t t t
Thus, the APF-policy module produces a bounded moving equilibrium rather than a direct torque, force, or stiffness command. The generated target is tracked by a Cartesian impedance controller with fixed positive-definite stiffness and damping,
where ∆pA,t ∈ R3(N +1) contains the attractor offsets, λt ∈ RN +1 contains the signed attractor weights, and σ t ∈ RN contains the Gaussian widths. The linear attractor does not require a Gaussian width parameter. Therefore, at ∈ R5N +4 . All policy actions are normalized to [−1, 1] and mapped to predefined physical ranges before constructing the APF. The Cartesian position of attractor k is computed as b g,t + ∆pak ,t , pak ,t = p
k ∈ {0, . . . , N }.
(12)
The decoded offsets, signed weights, and Gaussian widths define the APF vector field FAPF (.), which is then converted into a bounded Cartesian reference by the trajectory generation module. The policy is trained using Soft Actor–Critic (SAC) [19], an off-policy maximum-entropy actor–critic algorithm. SAC optimizes a stochastic policy by maximizing the expected discounted return while encouraging exploration through an entropy term: " T # X t J(π) = Eπ γ (rt + αH (π(·|ξ t ))) , (13) t=0
where rt is the task reward, γ is the discount factor, α is the entropy temperature, and H(π(·|ξt )) denotes the policy entropy. Thus, the learned policy does not directly command torques, forces, or impedance gains. It selects APF parameters, while the generated field is subsequently transformed into a bounded Cartesian reference and executed by the lowlevel controller. C. Motion Planning and Control The APF defined above is not applied directly as a Cartesian force. Instead, it is used as a bounded reference generator for the equilibrium point of a Cartesian impedance controller. Given the APF vector field FAPF (.), the reference velocity is computed as ẋd,t = satvmax (gv FAPF,t (pc,t )) ,
(14)
where gv is a velocity gain and satvmax (·) limits the velocity norm by vmax . This saturation decouples the magnitude of the commanded Cartesian motion from abrupt changes in the learned APF parameters. The Cartesian target is then obtained by integrating the bounded reference velocity, xd,t+∆t = xd,t + ∆t ẋd,t .
(15)
xd,t ∈ X ,
∥xd,t − pc,t ∥ ≤ ℓmax .
Fimp,t = Kp (xd,t − pc,t ) + Dp (ẋd,t − ṗc,t ).
(16)
(17)
The corresponding joint torques are computed through the manipulator Jacobian, τ t = Jt⊤ Fimp,t + τ null,t + τ bias,t .
(18)
where τ null,t denotes the null-space posture regulation term and τ bias,t accounts for model compensation. Therefore, the learned policy shapes the APF used for trajectory generation, while the robot remains controlled through a bounded reference update and a fixed damped impedance law. D. Stability and Boundedness The learned APF enters the robot controller only through the commanded Cartesian equilibrium xd,t . From (14)–(16), the reference velocity and its displacement from the controlled point pc,t are bounded. Thus, abrupt policy-induced APF changes cannot generate unbounded reference jumps. Consider the Cartesian impedance storage function 1 1 Mx (qt )ṗc,t + (pc,t − xd,t )⊤ Kp (pc,t − xd,t ). Vt = ṗ⊤ 2 c,t 2 (19) For fixed positive-definite Kp , Dp , the damping term in (17) dissipates energy. The energy injected by the moving reference is bounded by (pc,t − xd,t )⊤ Kp ẋd,t ≤ ℓmax ∥Kp ∥vmax ,
(20)
and the elastic interaction is bounded by ∥Kp (xd,t − pc,t )∥ ≤ ∥Kp ∥ℓmax .
(21)
Therefore, the learned APF does not act as an unbounded force source; it acts as a bounded reference input to a damped impedance system. Under bounded workspace operation and fixed positive-definite impedance gains, the closed-loop system is practically bounded with respect to the commanded Cartesian equilibrium. This argument does not imply global asymptotic convergence to the task goal, since local APF minima and obstacle-induced detours may occur. It instead establishes that policy-induced APF changes cannot generate unbounded reference motion or unbounded impedance deflection. IV. E XPERIMENTS The PA-RL framework is evaluated on the peg-in-hole task with three baselines and they are presented consistently in the order PA-RL, TCP-Vel, TCP-Pose, and VICES. The experiments are organized around learning efficiency, task performance, and motion quality. Finally the PA-RL policy transferred to real-robot to assess deployment feasibility.
A. Task and Training Setup The task is performed by a Franka Emika Panda robot equipped with a force–torque sensor and a cylindrical peg, as shown in Fig. 3. The controlled Cartesian frame is attached to the peg tip (pc,t ≡ ppeg,t ). The peg and hole diameters are 20 mm and 23.6 mm, respectively. An episode terminates with success when the peg remains inserted for 2 second, or with failure when the timeout is reached. The simulation environment is implemented in MuJoCo. At the beginning of each training episode, the true insertion goal pg is randomized independently along the x-y axes. The policy does not observe pg directly, instead, it receives an estimated goal position p̂g,t with an episode-independent estimation error. Additional noise is applied to the goalrelative position and measured force–torque observations. The randomization ranges are summarized in Table I. TABLE I TASK RANDOMIZATION AND OBSERVATION - NOISE SETTINGS . Parameter
Distribution or range
Initial hole offset [∆x, ∆y] Goal-pos. estimation error [ex , ey ] Goal-distance noise [nx , ny , nz ] Force observation noise Torque observation noise
[−50, 50] mm per axis N 0, (1.5 mm)2 I 2 N 0, diag(0.5, 0.5, 0.3) mm 2 N 0, (0.2 N) I N 0, (0.02 N m)2 I
1) Training Configuration: The physics simulation and the Cartesian impedance controller run at 1000 Hz. PARL updates its attractor-field parameters at every 2 second, while the field is re-evaluated at 100 Hz to produce a bounded Cartesian reference; because this reference is a state-dependent feedback field, the trajectory keeps adapting to the current peg-tip state even when the parameters are held fixed. The TCP-Vel, TCP-Pose, and VICES baselines instead output local motion or impedance parameter and therefore act at the control rate of 62.5 Hz. These rates follow from the temporal semantics of each action interface, not from any computational limitation. All policies were trained with Soft Actor–Critic using twohidden-layer 256 ReLU networks, a batch size of 256, a target smoothing coefficient τ = 0.005, and one gradient step per environment step. Since the four methods use fundamentally different action representations, a single shared hyperparameter set would favour one method and understate the others; we therefore keep the task, reward, and success criterion identical and tune only the learning-side hyperparameters per method, evaluating PA-RL and baselines at its best-performing configuration so the comparison reflects each representation’s maximum attainable performance. PARL used a learning rate of 5×10−4 , γ = 0.6, a replay buffer of 6 × 104 policy step, and 888 warm-up steps; the baselines used 3 × 10−4 , γ = 0.995, a buffer of 2 × 105 , and 3 × 104 warm-up steps. 2) Action Space: For PA-RL, the APF action representation contains one linear attractor and two Gaussian attractors. The policy outputs their goal-relative offsets, weights, and Gaussian sigma, resulting in a 14-dimensional action.
z
z y x
y
x
(a)
(b)
Fig. 3. Simulation and real-robot setups for the peg-in-hole task. (a) MuJoCo environment with a Franka Emika Panda, cylindrical peg, and fixed hole. (b) Physical setup with the same robot and peg–hole geometry, instrumented with an ATI force–torque sensor.
Attractor offsets are expressed in normalized coordinates and mapped to the goal relative operational space which is defined from the range of initial peg-tip positions and the target insertion region The offsets range are based on the operation space while weights and sigmas range preserve the linear attractor as the primary component while allowing the Gaussian attractors to locally modify the field. The broadest attractor region covers approximately 0.8 of this envelope, while the other two regions provide complementary spatial resolutions within the same workspace rather than the full robot workspace. The linear attractor weight is bounded by w1 ∈ [0.20, 0.80], while the Gaussian attractor weights satisfy w2 , w3 ∈ [0, 0.12]. Their Gaussian sigmas are σ2 ∈ [20, 80] and σ3 ∈ [8, 30]. These bounds retain the linear attractor as the principal goaldirected component while allowing the Gaussian attractors to reshape the field online from geometric and force–torque observations. TCP-Vel directly outputs a three-dimensional peg-tip velocity, bounded by ∥vt ∥2 ≤ 0.03 m/s. TCP-Pose outputs a three-dimensional Cartesian position increment, with each component constrained to [−2, 2] mm. The VICES-style variable-impedance baseline extends the position increment with translational stiffness and damping-ratio commands, resulting in a nine-dimensional action. Its translational stiffness is bounded by [200, 200, 250]⊤ and [800, 800, 900]⊤ N/m, while the damping ratio is bounded by [0.70, 0.70, 0.80]⊤ and [1.60, 1.60, 1.85]⊤ . The rotational impedance remains fixed. 3) Observation Space: All policies receive the goalrelative position p̂g,t − ppeg,t and the measured force– torque signal. PA-RL additionally observes the current attractor offsets, producing an 18-dimensional observation. TCPVel and TCP-Pose additionally observe the peg-tip velocity and previous action, producing 15-dimensional observations. VICES further observes its current normalized translational stiffness and damping ratio, producing a 27-dimensional observation. The observation dimensions are not artificially equalized because each interface requires access to its own
execution state. 4) Training and Evaluation: All policies are trained using SAC algorithm with three random seeds. They share the same MuJoCo model, task reward, insertion-success criterion, and force–torque observation model. The configured training budgets are 5 × 106 environment steps for all the methods. Let dt = ∥ppeg,t − pg ∥2 (22) denote the distance from the peg tip to the true insertion goal. The shared reward is rt = 10 (1 − tanh(45dt )) + 60 I[st = 1],
(23)
where st is the insertion-success indicator. The true insertion goal is used only for reward computation and evaluation. Policies are evaluated periodically on a common fixed 3×3 grid, with horizontal goal offsets x, y ∈ {−50, 0, 50} mm. Each position is evaluated three times, resulting in 27 episodes per checkpoint. The true goal position remains fixed across the three trials at each grid point, while the goal-position estimation error is resampled at every episode and observation noise is resampled independently during execution. The configured random seeds are shared across methods.
Fig. 4. Learning efficiency over three seeds. Solid lines and shaded regions denote the across-seed mean and one standard deviation. The endpoint distribution expressed as the final peg-tip XY error relative to the true insertion goal.
B. Experiment Results We report various metrics to evaluate the learning efficiency, task performance and motion quality, as described below. Learning efficiency is assessed using episode return, cumulative successful episodes, and the episode endpoint distribution as presented in Fig. 4. PA-RL exhibits an rapid increase in episode return and maintains a higher return throughout most of the common training range. Its cumulative-success curve also grows substantially faster, indicating that successful insertion behavior is acquired earlier and reproduced more frequently during training. In contrast, the baseline methods improve more gradually and show greater variation in their return curves. The endpoint distribution provides complementary spatial evidence. PA-RL produces a compact endpoint cluster around the true hole center, whereas the baseline distributions are more dispersed endpoints. These distant samples are consistent with incomplete approaches, residual misalignment, or termination before insertion. These results indicate earlier episode-level acquisition of task-relevant behavior. Task performance is assessed using evaluation success rate and the mean final peg-tip-to-goal distance with the N =27 fixed-grid evaluation episodes:
(i)
Lpath =
Ti X
(i)
(i)
ppeg,t − ppeg,t−1
t=2
(24)
where si is the success indicator, Ti is the final sample of (i) episode i, and pg is the corresponding true insertion goal. As shown in Fig. 5, PA-RL is the first and only method to reach a 100% evaluation success rate. The final peg-tipto-goal distance to approximately 2 mm. In contrast, the
. 2
(25)
The peak measured interaction force is defined as (i)
fpeak = max
1≤t≤Ti
(i)
ft
, 2
(i)
N
1 X (i) p − p(i) , dfinal = g N i=1 peg,Ti 2
baseline methods occasionally reach high success rates at individual checkpoints but do not sustain this performance consistently. Their success curves exhibit stronger fluctuations and larger cross-seed variation, accompanied by repeated spikes in the final-distance curves. This pattern indicates that their failures are not merely caused by narrowly missing the success threshold. Instead, a considerable proportion of episodes terminate with incomplete insertion or substantial residual misalignment. Motion quality is assessed using peak measured interaction force, RMS inter-step joint-torque variation, path length, and RMS Cartesian acceleration. Motion quality metrics are averaged over all 27 evaluation episodes, rather than successful episodes only. For evaluation episode i, the Cartesian path length is
(26)
where ft ∈ R3 is the force component of the measured force–torque signal. RMS inter-step joint-torque variation is computed as v Ti u J 2 X u1 X 1 (i) (i) t τt,j − τt−1,j , (27) Dτ(i) = Ti − 1 t=2 J j=1 where J = 7 is the number of robot joints.
RL-APF
TCP-Vel
TCP-Pose
VICES
Evaluation success rate
(a)
Peak contact force
0.75
0.25
RMS Δτ [N m]
0.50
10
5
0.00 0
0.005
0.000 RL
(c)
150 0 6
12
18
24
30
36
Evaluation index
RMS Cartesian acceleration (velocity roughness) is computed from consecutive peg-tip velocities as v u 2 (i) (i) Ti u 1 X ṗc,t − ṗc,t−1 (i) t . (28) ARMS = Ti − 1 t=2 ∆t 2
Each motion-quality result reported in Fig. 6 is the mean of the corresponding per-episode metric over all N = 27 fixed-grid evaluation episodes: N
0.15 0.10 0.05 0.00
Fig. 5. Fixed-grid evaluation task performance. Each evaluation process contains 27 episodes with nine hole positions on a 3×3 grid and three trials per position under resampled goal-position estimation error and observation noise. Solid lines and shaded regions denote the across-seed mean and one standard deviation.
1 X (i) m̄ = m , N i=1
0.20
n o (i) (i) (i) m(i) ∈ Lpath , fpeak , Dτ(i) , ARMS .
(29) The results in Fig. 6 reflect distinct execution behaviors induced by the policy action representation. PA-RL maintained a comparable peak measured interaction force of 9.10 ± 0.45 N, while producing lower RMS inter-step jointtorque variation (0.00226 ± 0.00016 N m) and Cartesian acceleration (0.0599 ± 0.0016 m/s2 ) than the baselines. Its trajectories exhibited continuous goal-directed motion and gradual contact correction, whereas the direct motioncommand interfaces more frequently revised their commands near the hole. Although TCP-Pose achieved a similar path length, its execution contained larger local variations, showing that path length alone does not capture motion regularity. Unsuccessful baseline rollouts typically terminated near the hole after repeated correction or sustained contact without completing insertion. As the common reward contains no explicit motion quality penalties, these behaviors are consistent with the
F -Vel ose S P VICE -AP RL TCP TCP-
F S el se -AP CP-V P-Po VICE T TC
Velocity roughness
RMS accel. [m/s2 ]
Cartesian path length
300
(b)
0.010
F -Vel ose S P VICE -AP RL TCP TCP-
(b)
Path length [m]
Final distance [mm]
Final peg-tip distance 450
Joint-torque variation 0.015
(a) Peak force [N]
Success rate
1.00
different structures introduced by the policy interfaces rather than method-specific reward shaping.
(d) 0.6 0.4 0.2 0.0 RL
F S el se -AP CP-V P-Po VICE T TC
Fig. 6. Motion quality at the selected evaluation experiment. For each method and seed, the checkpoint with the highest evaluation success rate is selected. Metrics are computed over all 27 evaluation episodes. Bars and error bars denote the across-seed mean and one standard deviation; markers denote individual seeds.
Overall, PA-RL achieved faster learning, stronger task performance under added estimation error, and smoother robot motion. C. Real-world Experiment The simulation-trained PA-RL policy was deployed on the physical Franka robot (Fig. 3(b)) without policy fine-tuning. Across the 3×3 start-offset grid, it completed 9/9 peg-inhole insertions and the results are shown in Fig. 7. The mean final peg-tip-to-goal distance was 2.66 mm, with a range of 1.4–3.8 mm. The mean peak measured interaction force was 9.3 N (2.0–15.4 N), and the mean completion time was 7.18 s (4.7–18.5 s). The top-view trajectories approach the insertion goal from different lateral directions and converge within the hole region rather than following a single prescribed path. The side-view trajectories in Fig. 7(c) show lateral correction during approach followed by descent along the insertion axis, showing the trajectory variations that are caused by the different initial offsets and contact sequences. The real-robot experiments demonstrate deployment feasibility. Qualitatively, the trajectories and the supplementary video show contact-dependent adaptation. The policy does not simply execute a fixed downward insertion motion; instead, it performs lateral corrections around the hole and, in some trials, briefly moves upward and reattempts insertion multiple times. This suggests that the learned APF representation can exploit force–torque feedback and peg–hole alignment cues
Insertion success 9/9 +
2.7 mm 13.0 N 7.2 s
1.4 mm 10.1 N 4.9 s
2.9 mm 6.6 N 4.8 s
0
2.2 mm 10.8 N 5.9 s
2.3 mm 5.2 N 4.7 s
3.3 mm 2.0 N 4.7 s
−
2.5 mm 9.9 N 8.6 s
2.8 mm 10.7 N 5.3 s
3.8 mm 15.4 N 18.5 s
−
0
+
20
y − yhole [mm]
y start offset
(a)
Trajectories (top view) (b)
10
R EFERENCES
0 −10 −20 −20
0
20
x − xhole [mm]
x start offset
Trajectories (side view) 150
(c)
z [mm]
125 100 75 50 −20
−10
rameterizations, broader manipulation scenarios, systematic ablations, stronger convergence analysis, improved sensory representations, and larger-scale real-robot evaluations.
0
10
20
x − xhole [mm] Fig. 7. Real-robot deployment over a 3×3 grid. (a) Per-trial task performance with final peg-tip-to-goal distance, peak measured contact force, and completion time. (b) Top-view peg-tip trajectories relative to the true insertion goal and (c) Side-view trajectories. Colors indicate the different initial relative peg–hole positions.
to guide exploratory sliding near the hole before successful insertion. Thus, the real-robot experiment demonstrates both deployment feasibility and adaptive interaction behavior. V. C ONCLUSION This work introduced PA–RL, a framework that allows an RL policy to adapt an APF as an interpretable and spatially structured motion representation. The method uses modelfree RL for task-level adaptation, while a bounded APFbased reference generator and a fixed Cartesian impedance controller handle continuous low-level motion generation and tracking. This separates learned field shaping from direct motion-command execution, reducing the burden on the policy. The results show that PA–RL improves learning efficiency, task performance, and motion quality compared with direct motion-command and variable-impedance baselines. In simulation, PA–RL achieved faster learning, stronger task performance under goal-position estimation error, and smoother robot motion. The simulation-trained policy also completed all real-robot insertions without policy fine-tuning, suggesting robustness to the sim-to-real gap and supporting the feasibility of the learned potential-field interface. At the same time, the present work should be viewed as an early instantiation rather than a general solution. While APFs are broadly applicable, the current parameterization includes task-relevant components, such as goal-relative attractors, that are well suited to peg-in-hole insertion. Moreover, the current formulation does not guarantee global convergence to the task goal. Future work will investigate richer APF pa-
[1] Z. Gu, J. Li, W. Shen, W. Yu, Z. Xie, S. McCrory, X. Cheng, A. Shamsah, R. Griffin, C. K. Liu, A. Kheddar, X. B. Peng, Y. Zhu, G. Shi, Q. Nguyen, G. Cheng, H. Gao, and Y. Zhao, “Humanoid locomotion and manipulation: Current progress and challenges in control, planning, and learning,” IEEE/ASME Transactions on Mechatronics, vol. 31, no. 2, pp. 2300–2330, 2026. [2] R. Riener and A. Rabezzana, “Do robots outperform humans in human-centered domains?” Frontiers in Robotics and AI, vol. 10, p. 1223946, 2023. [3] Í. Elguea-Aguinaco, A. Serrano-Muñoz, D. Chrysostomou, I. InziarteHidalgo, S. Bøgh, and N. Arana-Arexolaleiba, “A review on reinforcement learning for contact-rich robotic manipulation tasks,” Robotics and Computer-Integrated Manufacturing, vol. 81, p. 102517, 2023. [4] H. Zhang, R. Dai, G. Solak, P. Zhou, Y. She, and A. Ajoudani, “Safe learning for contact-rich robot tasks: A survey from classical learning-based methods to safe foundation models,” arXiv preprint arXiv:2512.11908, 2026. [5] R. Martı́n-Martı́n, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg, “Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks,” in 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2019, pp. 1010–1017. [6] S. Bahl, M. Mukadam, A. Gupta, and D. Pathak, “Neural dynamic policies for end-to-end sensorimotor learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 5058–5069, 2020. [7] H. Zhang, G. Solak, G. J. G. Lahr, and A. Ajoudani, “Srl-vic: A variable stiffness-based safe reinforcement learning for contact-rich robotic tasks,” IEEE Robotics and Automation Letters, vol. 9, no. 6, pp. 5631–5638, 2024. [8] O. Khatib, “The potential field approach and operational space formulation in robot control,” in Adaptive and Learning Systems: Theory and Applications. Springer, 1986, pp. 367–377. [9] H. Huang, W. Zheng, J. Duan, K. Huang, J. Guo, and C. Yang, “Learning globally stable neural imitation policies via reshaped energy functions,” IEEE/ASME Transactions on Mechatronics, pp. 1–11, 2026. [10] X. Liu, A. Kramberger, and L. Bodenhagen, “A general peg-in-hole assembly policy based on domain randomized reinforcement learning,” in International Conference on Robotics in Alpe-Adria Danube Region. Springer, 2025, pp. 28–36. [11] X. Liu, C. Zeng, C. Yang, and J. Zhang, “Reinforcement learningbased sequential control policy for multiple peg-in-hole assembly,” CAAI Artificial Intelligence Research, vol. 3, p. 9150043, 2024. [12] S. A. Khader, H. Yin, P. Falco, and D. Kragic, “Stability-guaranteed reinforcement learning for contact-rich manipulation,” IEEE Robotics and Automation Letters, vol. 6, no. 1, pp. 1–8, 2020. [13] ——, “Learning stable normalizing-flow control for robotic manipulation,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021. [14] A. J. Ijspeert, J. Nakanishi, P. Pastor, H. Hoffmann, and S. Schaal, “Dynamical movement primitives: Learning attractor models for motor behaviors,” Neural Computation, vol. 25, no. 2, pp. 328–373, 2013. [15] T. Davchev, K. S. Luck, M. Burke, F. Meier, S. Schaal, and S. Ramamoorthy, “Residual learning from demonstration: Adapting dmps for contact-rich manipulation,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6858–6865, 2022. [16] D.-H. Park, H. Hoffmann, P. Pastor, and S. Schaal, “Movement reproduction and obstacle avoidance with dynamic movement primitives and potential fields,” in Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots, 2008, pp. 91–98. [17] J. Zhang, Z. Jin, Z. Zhao, and C. Yang, “A novel robotic skill learning approach for assembly task with dynamical system and broad learning,” IEEE Transactions on Industrial Electronics, 2025. [18] S. M. Khansari-Zadeh and A. Billard, “Learning stable non-linear dynamical systems with gaussian mixture models,” IEEE Transactions on Robotics, vol. 27, no. 5, pp. 943–957, 2011. [19] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, and P. Abbeel, “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905, 2018.