Abstract Muscle-driven locomotion provides a physically grounded approach to generating realistic human movement. However, achieving both physiological plausibility and adaptability to changes in musculoskeletal capacity and external disturbances remains a fundamental challenge. To address this limitation, we propose a Reflex-Informed Neuromuscular Reinforcement Learning framework for muscle-driven locomotion. Within this framework, a fixed phase-dependent reflex controller serves as the underlying neuromuscular control mechanism, while the reinforcement learning policy produces four biomechanically meaningful residual parameters to modulate key reflex gains and thresholds associated with hip swing, knee support, and ankle propulsion according to the current state. Experimental results demonstrate that the proposed framework generates physiologically plausible locomotion with improved kinematic accuracy and dynamic consistency, as well as better bilateral symmetry and stride-to-stride consistency under nominal walking conditions. The learned policy remains robust under muscle weakness and external perturbations without retraining.

Keywords: Muscle-driven locomotion, musculoskeletal simulation, reflex-based control, residual reinforcement learning, neuromuscular control, physics-based character animation

Introduction

Muscle-driven simulation plays an integral role in physically based character animation and the modeling of human locomotion Sun et al. (2024); Schumacher et al. (2025). Unlike approaches that directly generate motion through joint trajectories or joint torques, muscle-driven models allow movement to emerge naturally from neuromuscular control, muscle dynamics, and physical interactions between the body and the environment, resulting in motion with greater physical consistency and biological plausibility. However, realistic muscle-driven locomotion requires not only physiological plausibility, but also the ability to adapt naturally to changes in the musculoskeletal condition and the environment. Therefore, achieving both physiological plausibility and adaptability remains a key challenge in muscle-driven character animation. Existing approaches to muscle-driven locomotion generally fall into two categories. One category employs structured neuromuscular controllers, such as reflex-based control, central pattern generators (CPGs), and muscle synergies, by explicitly embedding biological control principles into the controller. Such controllers naturally generate stable and physiologically meaningful locomotion. However, they are typically designed manually or optimized offline, and 1

their control policies remain fixed during deployment, limiting their ability to adapt to changes in musculoskeletal conditions or the environment. The other category formulates locomotion as a reinforcement learning problem by directly learning muscle actions. Since muscle activations are generated online according to the current state, these methods exhibit strong adaptability. However, directly learning muscle actions requires the policy to discover effective neuromuscular coordination within a high-dimensional and redundant action space. Task-level rewards specify the desired locomotion objective but do not explicitly determine how muscle activity should be organized, making it difficult to ensure physiologically plausible movement and dynamics. Consequently, existing approaches often face a trade-off between physiological plausibility and adaptability: structured neuromuscular controllers preserve biological organization but lack state-dependent adaptation, whereas direct muscle learning remains adaptive but must rediscover complex neuromuscular coordination from scratch. From the perspective of biological motor control, locomotion is not generated by independently controlling individual muscles. Instead, muscle activity is organized through existing neuromuscular mechanisms involving sensory feedback, spinal regulation, and muscle dynamics. In other words, existing neuromuscular mechanisms already organize how sensory feedback regulates muscle activity during locomotion. If these control principles already exist, should reinforcement learning still rediscover them, or should it instead build upon them? This question motivates us to rethink the role of reinforcement learning in muscle-driven locomotion. Rather than directly learning muscle actions, we formulate learning as the state-dependent regulation of existing neuromuscular control mechanisms. In this formulation, neuromuscular control is responsible for coordinating muscle activity, while reinforcement learning adjusts these control mechanisms according to the current state, enabling adaptation without relearning muscle coordination from scratch. Motivated by this insight, we propose a Reflex-Informed Neuromuscular Learning framework for muscle-driven locomotion, where a fixed phase-dependent reflex controller provides the underlying neuromuscular control mechanism. Rather than directly outputting 18-dimensional muscle actions, the reinforcement learning policy produces four biomechanically meaningful residual signals at each control step to modulate the gains and thresholds associated with hip swing, knee support, and ankle propulsion in a state-dependent manner. This provides a compact, interpretable, and biologically meaningful learning interface for muscle-driven locomotion. To verify that the proposed Reflex-Informed Neuromuscular Learning framework provides both physiological plausibility and adaptability, we conduct complementary experiments under nominal walking, muscle weakness, and external perturbation conditions. Under nominal walking conditions, physiological plausibility is evaluated through human reference kinematics, ground reaction forces, bilateral symmetry, and stride-to-stride consistency. Muscle weakness and external perturbation experiments further evaluate the policy’s ability to adapt to changes in musculoskeletal conditions and the environment without retraining. The main contributions of this work are summarized as follows: 1. We propose a Reflex-Informed Neuromuscular Reinforcement Learning framework that reformulates reinforcement learning as the state-dependent regulation of existing neuromuscular control mechanisms, rather than directly learning muscle actions. 2. We introduce a compact and interpretable learning interface that modulates key pathways of a fixed reflex controller through four biomechanically meaningful residual parameters, enabling neuromuscular regulation without directly controlling individual muscles. 3. We demonstrate that the proposed framework improves physiological plausibility while maintaining adaptability across different locomotion conditions. Extensive experiments show more human-like joint kinematics, ground reaction forces, and gait consistency across multiple walking speeds, while the same trained policy remains effective under muscle weakness and external perturbations without retraining. 2

Related Work

2.1

Muscle-Driven Character Simulation

Musculoskeletal models driven by muscle–tendon dynamics have become a standard framework for studying and generating human locomotion by explicitly modeling muscle activation, muscle– tendon dynamics, skeletal dynamics, and body–environment interaction. In biomechanics, these models have been widely used to generate and analyze healthy and pathological gait, investigate locomotor strategies, study energetic objectives, and evaluate the effects of ageing and musculoskeletal impairments Anderson and Pandy (2001); Ackermann and van den Bogert (2010); Falisse et al. (2019); Song and Geyer (2018); De Groote and Falisse (2021); Uchida and Delp (2020); Ezati et al. (2019); Dembia et al. (2020). In computer graphics, muscle-driven simulation has been adopted for physically based character animation, enabling realistic locomotion through muscle-actuated simulation and control Wang et al. (2010, 2012); Geijtenbeek et al. (2013); Lee et al. (2019). Together, these studies establish muscle-driven simulation as a mature foundation for generating realistic human locomotion. Consequently, recent research has increasingly focused on how to design effective controllers for these musculoskeletal systems.

2.2

Structured Neuromuscular Control

Biologically inspired controllers introduce prior knowledge of human motor control directly into the control architecture. Representative approaches include central pattern generators (CPGs), muscle synergies, and reflex-based controllers. Ijspeert reviewed CPG-based locomotion control in animals and robots, where rhythmic movement is generated through coupled oscillatory networks Ijspeert (2008). d’Avella et al. and Meyer et al. modeled muscle coordination using muscle synergies, representing muscle activation with a reduced set of coordinated control signals d’Avella et al. (2003); Meyer et al. (2016). Geyer and Herr proposed a reflex-based controller that generates human walking through physiologically motivated sensory feedback Geyer and Herr (2010), which was later extended by Song and Geyer to generate diverse locomotion behaviors across different walking speeds Song and Geyer (2015). Dzeladini et al. further combined CPGs and reflexes within a unified neuromuscular controller Dzeladini et al. (2014). Among these approaches, reflex-based control is particularly relevant to muscle-driven locomotion because it directly maps muscle sensory feedback to muscle stimulation. Reflex-based controllers have been shown to generate stable locomotion in humans Geyer and Herr (2010); Song and Geyer (2015), adaptive walking across a wide range of speeds Koseki et al. (2024), and robust locomotion in bipedal robots Batts et al. (2015). However, most existing reflex controllers rely on manually designed or offline-optimized parameters. Although muscle stimulation changes continuously with sensory feedback, the underlying reflex gains and thresholds are not updated online, limiting adaptation to changes in musculoskeletal conditions and the environment.

2.3

Learning-Based Muscle Control

Deep reinforcement learning has enabled physics-based characters to acquire locomotion skills directly through interaction with the environment, while prior work has shown that the action representation and reference-motion objective strongly affect the learned behavior Peng and van de Panne (2017); Peng et al. (2018). For muscle-driven characters, existing approaches typically formulate muscle activations as the policy output, allowing muscle control signals to be generated online according to the current state Weng et al. (2021); de Vree and Carloni (2021); Ogum et al. (2024); Song et al. (2021). Compared with manually designed controllers, these methods improve adaptability and can learn complex locomotion behaviors without explicitly specifying muscle coordination. Recent studies have further improved direct muscle-control reinforcement learning from different perspectives. Bio-inspired reward functions encourage more physiologically realistic 3

gait patterns Nowakowski et al. (2021); Schumacher et al. (2025). Model-based reinforcement learning incorporates musculoskeletal dynamics into policy optimization Su and GutierrezFarewik (2023). DEP-RL improves exploration in overactuated musculoskeletal systems through embodied exploratory feedback Schumacher et al. (2023). More recently, AI-CPG combines a CPG-based feedforward controller with a learned reflex network to improve adaptive locomotion Li et al. (2024). Overall, these studies indicate that existing approaches cannot simultaneously provide physiological plausibility and adaptability. Structured neuromuscular controllers can generate physiologically plausible movement but struggle to adapt to changes in the musculoskeletal model or the environment, whereas reinforcement learning-based muscle control improves adaptation in muscle-driven locomotion but often generates gait patterns that deviate from human kinematics and dynamics.

Method

3.1

Framework Overview

Figure 1 illustrates the proposed Reflex-Informed Neuromuscular Reinforcement Learning framework. A fixed phase-dependent reflex controller provides the underlying neuromuscular control mechanism, while the reinforcement learning policy generates four-dimensional residual actions that regulate selected reflex parameters according to the current musculoskeletal state. The resulting muscle stimulation drives the musculoskeletal simulation, which produces the next state and reward. During learning, transitions (st , at , rt , st+1 ) are stored in the replay buffer D. Maximum a Posteriori Policy Optimization (MPO) updates the policy through an off-policy actor–critic framework using replayed transitions Abdolmaleki et al. (2018).

3.2 3.2.1

Musculoskeletal Model Simulation platform and character.

The physiological environment module (b) is implemented in the Hyfydy musculoskeletal simulation engine Geijtenbeek (2021) through the Python-based SCONE Gym interface Geijtenbeek (2019). We use the sagittal-plane H0918v2j human model, which consists of a pelvis, a rigidly attached torso, and bilateral thigh, shank, and foot segments with nine generalized coordinates. The character is actuated by 18 Hill-type muscle–tendon units representing nine muscle groups on each leg, including tibialis anterior (TA), soleus (SOL), gastrocnemius (GAS), vasti (VAS), biceps femoris short head (BIF), rectus femoris (FEM), iliopsoas (ILI), hamstrings (HAM), and gluteus maximus (GLU). Each foot interacts with the ground through spherical heel and toe contact elements, and the controller is updated at 0.005 s intervals. Figure 2(a) illustrates the musculoskeletal character, while Fig. 2(b) summarizes the arrangement of the modeled muscle–tendon units and the joints they span. 3.2.2

Muscle-driven dynamics.

Within module (b), reflex-generated stimulation is applied to the character through activation dynamics, muscle–tendon mechanics, and forward dynamics. For muscle i, neural stimulation ui is first filtered by activation dynamics to obtain activation αi , after which the Hill-type muscle model computes the corresponding muscle–tendon force α̇i = Ai (αi , ui ),

FiMT = Fi (αi , li , l˙i ),

(1)

where Ai denotes the activation dynamics and Fi denotes the Hill-type muscle model. Muscle forces are then transformed into generalized joint moments through configuration-dependent 4

Figure 1: Overview of the proposed Residual-Reflex RL framework. (a) The rollout policy maps the musculoskeletal state st to a four-dimensional residual action at and collects transitions (st , at , rt , st+1 ). (b) The physiological environment combines the muscle-driven character with a phase-dependent residual-reflex-informed controller. In the enlarged lower row, panel (b1) on the left shows the sparse phase-dependent reflex topology, while panel (b2) on the right shows the proprioceptive control loop from muscle sensing to stimulation, activation, muscle–tendon force, skeletal motion, and updated sensory feedback. Magenta entries in (b1) denote the four reflex parameters modulated by at . Superscripts (all), (st), and (sw) identify parameters shared across all five phases, Early Stance–Late Stance, and Liftoff–Swing–Landing, respectively. The notation i ← j indicates that feedback from source muscle j modulates target muscle i; entries without a source are self-feedback pathways. (c) The collected transitions are stored in D; the critic q(s, a; ω) evaluates candidate actions and MPO updates the rollout policy from π(s; θ now ) to π(s; θ new ). muscle moment arms, τmus,k =

ri,k (q)FiMT ,

(2)

i∈Mk

where ri,k (q) is the configuration-dependent muscle moment arm of muscle i about generalized coordinate k, and Mk denotes the set of muscles spanning that coordinate. Finally, the generalized muscle moments drive the musculoskeletal system through forward dynamics, M(q)q̈ + h(q, q̇) = τ mus + JT c fc ,

(3)

where q, q̇, and q̈ are the generalized coordinates, velocities, and accelerations, respectively; M(q) is the mass matrix; h(q, q̇) collects gravitational, Coriolis, and centrifugal terms; τ mus denotes the generalized joint moments generated by the muscles; Jc is the contact Jacobian; and fc denotes the foot–ground contact forces.

3.3 3.3.1

Neuromuscular Control Mechanism Physiological feedback

The reflex controller computes muscle stimulation from delayed muscle sensory feedback and posture feedback. Muscle length and velocity approximate muscle-spindle feedback, whereas 5

Figure 2: Muscle-driven character model. (a) Front and sagittal views of the H0918v2j character used in the experiments. (b) Schematic arrangement and primary joint functions of the nine bilateral muscle–tendon groups. The complete character is actuated by 18 Hill-type muscle– tendon units, with one instance of each modeled group on each leg. muscle force approximates Golgi tendon organ feedback Grillner and El Manira (2020). These sensory signals form the physiological inputs to the reflex controller. The corresponding gains, thresholds, source muscles, target muscles, and active gait phases are defined in the following sections. As illustrated in Fig. 1(b2), the resulting muscle stimulation drives activation and muscle–tendon force, while the updated muscle states provide sensory feedback for the next control step. 3.3.2

Phase-dependent topology.

A finite-state controller divides the gait cycle into five phases: Early Stance, Late Stance, Liftoff, Swing, and Landing. Reflex pathways are organized into an all-phase group, a stance group, and a swing-related group: Φall = {ES, LS, LF, SW, LA}, Φst = {ES, LS}, Φsw = {LF, SW, LA}. Parameters are shared among phases within the same group but remain independent across groups. For example, the BIF force-feedback gain used during Early Stance and Late Stance differs from that used during Liftoff, Swing, and Landing, allowing the same sensory feedback to support different functions, including stance support, push-off, and swing control, without requiring a separate parameter for every gait phase. Fig. 1(b1) summarizes the resulting topology across the nine modeled muscle groups. Monoarticular pathways involve TA and SOL at the ankle, VAS and BIF at the knee, and ILI and GLU at the hip. The biarticular muscles GAS, FEM, and HAM span two adjacent joints and therefore contribute to the control of both. The active parameter-sharing groups are determined by the current gait phase. During Early Stance and Late Stance, the controller activates the all-phase and stance-specific pathways, whereas during Liftoff, Swing, and Landing, it activates the all-phase and swing-related pathways. For convenience, let ϕ(t) denote the current gait phase and Γ(ϕ) the corresponding set of active parameter-sharing groups, such that Γ(ϕ) = {all, st} during Early Stance and Γ(ϕ) = {all, sw} during Liftoff, Swing, and Landing. Each active parameter-sharing group g ∈ Γ(ϕ) defines a sparse directed feedback graph Gg . A connection i ← j uses sensory feedback measured from source muscle j to modulate 6

the stimulation of target muscle i. Connections with i = j implement self-feedback, whereas connections with i ̸= j encode inter-muscle coordination. Reflex gains and thresholds are indexed by the active parameter-sharing group, while inactive pathways have zero gain during the corresponding gait phases. The sign and magnitude of each gain determine how strongly the corresponding sensory feedback facilitates or suppresses the target muscle stimulation. 3.3.3

Reflex control laws.

For an active group g ∈ Γ(ϕ(t)), muscle stimulation is computed from muscle length, velocity, and force feedback. (g)

(g)

(4)

uV,i←j (t) = KV,i←j max(0, ṽj (t − tD ) − ṽ0,i←j ),

(g)

(5)

(g) (g) (g) uF,i←j (t) = KF,i←j (f˜j (t − tD ) − f˜0,i←j ),

(6)

uL,i←j (t) = KL,i←j max(0, ˜lj (t − tD ) − ˜l0,i←j ),

where ˜lj , ṽj , and f˜j denote delayed normalized muscle length, velocity, and force measured from source muscle j, respectively. The corresponding gains KL , KV , and KF and thresholds ˜l0 , ṽ0 , and f˜0 are indexed by the active parameter-sharing group g, and tD is the sensorimotor delay. A reflex-like postural feedback term is also used to couple trunk pitch errors to selected lower-limb muscles. (g)

(g)

uPD,i (t) = KP,i (θ(t − tD ) − θ0,i ) + KD,i θ̇(t − tD ).

(7) (g)

where θ and θ̇ are the delayed trunk pitch angle and angular velocity, θ0,i is the target pitch (g)

(g)

offset for muscle i, and KP,i and KD,i are the proportional and derivative postural-feedback gains for the active group. The total muscle stimulation of muscle i is obtained by summing the constant background stimulation, muscle sensory feedback and postural feedback contributions from all active parametersharing groups. "

ui (t) =

(g)

uC,i + uPD,i (t)

g∈Γ(ϕ(t))

X (g)

(g)

uL,i←j (t) + uV,i←j (t) + uF,i←j (t)

(8)

j∈Mg (g)

where uC,i is the group-dependent constant background stimulation and Mg is the set of source muscles connected to target i in the corresponding active group. The resulting muscle stimulation drives the activation dynamics and musculoskeletal model described in Sec. 3.2.

3.4 3.4.1

Residual Neuromuscular Regulation Residual formulation

Rather than directly generating muscle stimulation, the reinforcement learning policy regulates the underlying neuromuscular controller through a four-dimensional residual action at ∈ R4 . Each action component modulates one selected reflex parameter around its nominal value, pi (t) = pi,0 (1 + λai (t)),

(9)

where pi,0 is the nominal reflex parameter and λ is the action scale. In the present implementation, λ = 0.5. When ai (t) = 0, the nominal reflex controller is recovered exactly, whereas nonzero actions produce state-dependent modulation of the selected reflex parameters. The resulting muscle stimulation is then computed by the underlying neuromuscular controller using the current muscle sensory feedback, gait phase, and the regulated reflex parameters. 7

Table 1: Selected Residual-Reflex Actions Action Parameter a1 a2 a3 a4

3.4.2

(sw)

KL,HAM ˜l(st) 0,ILI (all) KF,SOL (st) KF,VAS

Pathway

Functional role

S11100.hamstrings.KL Hip extension / knee flexion S00011.iliopsoas.L0 Hip flexion / leg advancement S11111.soleus.KF Ankle support / propulsion S00011.vasti.KF Knee extension / stance support

Regulated reflex parameters

The four regulated parameters are highlighted in bright magenta in Fig. 1(b1), and their corresponding pathways are summarized in Table 1. In Table 1, pathway labels follow the naming convention of the controller implementation, while the superscripts (all), (st), and (sw) explicitly indicate the corresponding phase groups used in this paper. Specifically, a1 modulates the swing-related hamstrings length gain to regulate hip extension and knee flexion during leg swing, a2 adjusts the stance-phase iliopsoas length threshold for hip flexion and leg advancement, a3 modulates the all-phase soleus force-feedback gain for ankle support and propulsion, and a4 regulates the stance-phase vasti force-feedback gain for knee extension and stance support. These four parameters were selected because they regulate gait-critical functions across the major lower-limb joints while directly influencing the key biomechanical roles required for locomotion, including leg swing, stance support, and propulsion. By restricting policy outputs to these biomechanically meaningful reflex parameters, the proposed formulation concentrates policy exploration on neuromuscular mechanisms that are directly related to locomotor function while maintaining a compact and interpretable action space.

3.5

Policy Learning with MPO

Policy learning follows an off-policy actor–critic framework optimized with Maximum a Posteriori Policy Optimization (MPO) Abdolmaleki et al. (2018). As illustrated in Fig. 1(c), the actor receives the current musculoskeletal observation and outputs a four-dimensional residual action, while the critic evaluates the corresponding state–action pair. Let st ∈ R121 denote the observation at time t, (10) st = [lt , l̇t , ft , et , αt , pf eet,t , qt , q̇t ]. Here l, l̇, and f are muscle-fiber lengths, velocities, and forces; e and α are muscle excitations and activations; pf eet represents foot positions relative to the pelvis; q and q̇ denote the pelvis and joint degrees of freedom and their corresponding velocities, respectively. The actor and critic are implemented as multilayer perceptrons with two hidden layers of 256 ReLU units. The actor π(s; θ now ) predicts the mean and standard deviation of a Gaussian policy, from which a four-dimensional residual action is sampled during training. During evaluation, the deterministic policy uses the action mean. The critic q(s, a; ω) estimates the corresponding action-value function Q(s, a). Transitions (st , at , rt , st+1 ) are stored in a replay buffer D. MPO updates the critic from replayed transitions and optimizes the actor under a KL-divergence constraint, enabling stable off-policy learning while reusing collected experience. We follow the standard MPO update procedure and therefore omit the derivation here. Policy optimization is driven by the following reward function, which consists of a target-speed reward rv together with penalty terms for excessive contact loading (cgrf ), temporal excitation variation (ce ), unnecessary muscle recruitment (cm ), joint-limit loading (cq ), pelvis instability (cθ , cθ̇ , ccom ), and rapid changes in the residual parameter increments (c∆p )

rt = wv rv − λgrf cgrf − λe ce − λm cm − λq cq − λθ cθ − λθ̇ cθ̇ − λcom ccom − λ∆p c∆p ,

(11)

The target-speed reward is defined as (

rv =

exp[−(vx − v ∗ )2 ], 1,

vx < v ∗ , vx ≥ v ∗ .

(12)

where vx and v ∗ denote the forward center-of-mass speed and the commanded speed, respectively. The individual penalty terms are defined as cgrf = [Lc − 1.2]+ , 1 ce = ∥et − et−1 ∥22 , N N 1 X I[αi > 0.15] , cm = da i=1 cq =

(13)

J 1X . mean τ lim j J j=1

where Lc is the total contact load normalized by body weight; N is the number of muscles; da is the action-space dimension; J is the number of joints; τ lim is the joint-limit torque of joint j; j I[·] is the indicator function; and [x]+ = max(0, x). cθ = (θt − θ∗ )2 , cθ̇ = θ̇t2 , ccom = (vcom,y )2 ,

(14) δpt = λp0 ⊙ at , 1 c∆p = ∥δpt − δpt−1 ∥22 . 4 ∗ where θ is the target pelvis pitch; vcom,y is the vertical center-of-mass velocity; at denotes the four-dimensional residual action produced by the policy; p0 collects the corresponding nominal reflex parameters; ⊙ denotes element-wise multiplication; and δpt is the vector of residual parameter increments applied at time t.

Evaluation

4.1

Evaluation Protocol

4.1.1

Experimental Setup

The simulator, musculoskeletal character, muscle-driven dynamics, and foot–ground contact model are described in Sec. 3.2. The same character and simulation settings were used for all controllers unless a plantarflexor weakness condition was explicitly introduced. The evaluation includes four control approaches: two direct muscle-control reinforcement learning methods, E2E-RL Weng et al. (2021); de Vree and Carloni (2021); Ogum et al. (2024) and DEP-RL Schumacher et al. (2023); the proposed Residual-Reflex RL method; and a reflex controller whose parameters were optimized offline using CMA-ES. Table 2 summarizes the policy interfaces of the three reinforcement learning methods. The CMA-ES baseline has no learned policy output and keeps its optimized reflex parameters fixed during evaluation. All reinforcement learning policies were trained using the same MPO framework and musculoskeletal model. Training was performed on a workstation equipped with an AMD Ryzen Threadripper PRO 9965WX CPU and 256 GB RAM. 9

Table 2: Policy interfaces of the reinforcement learning methods. Method

Action dim. Policy output

E2E-RL DEP-RL

18 18

Residual-Reflex RL

4.1.2

Direct muscle-level control actions. Direct muscle-level control with DEP-style embodied exploration. Residual modulation of selected reflex parameters.

Tasks and Evaluation Metrics

The evaluation consists of three complementary experiments designed to assess both physiological plausibility and adaptability. Nominal walking evaluates gait quality under controlled targetspeed tasks, plantarflexor weakness evaluates adaptation to changes in musculoskeletal capacity, and external push perturbation evaluates recovery from environmental disturbances. For nominal walking, policies were evaluated at target walking speeds of 0.8, 1.0, and 1.2 m/s. Each policy was simulated for 25 s, and the resulting trajectories were exported as STO files for post-hoc gait analysis. All detected gait cycles were normalized to 0–100% of the gait cycle. The comparison includes E2E-RL, DEP-RL, and Residual-Reflex RL. Four groups of metrics are used to evaluate task performance, physiological plausibility, and gait consistency. Speed tracking and stride measures To evaluate task performance, we report realized walking speed, absolute speed error |v̂ − vtarget |, stride length, stride time, and the number of analyzed cycles. The realized walking speed and speed error quantify how accurately the controller follows the commanded locomotion task, whereas stride length and stride time characterize the gait strategy adopted to achieve the target speed. Human-reference kinematic gait score To evaluate kinematic plausibility, we compare the generated gait-cycle profiles with human-reference kinematic data for the hip, knee, and ankle joints. These reference profiles are used only for offline evaluation and are not included in the reinforcement learning reward. For each joint, left and right gait cycles were time-normalized and averaged to obtain a mean simulated profile ȳ(x), where x denotes the normalized gait-cycle percentage. Let ymin (xk ) and ymax (xk ) denote the lower and upper human-reference bounds at sample point xk . The normalized reference-range violation is e=

N [ymin (xk ) − ȳ(xk )]+ + [ȳ(xk ) − ymax (xk )]+ 1 X , N k=1 max (0.01, ymax (xk ) − ymin (xk ))

(15)

where [z]+ = max(z, 0). The corresponding score is S = 100 · clip(1 − e, 0, 1).

(16)

Higher scores indicate closer agreement with the human-reference gait profiles. A score of 100 indicates that the averaged trajectory remains entirely within the reference range. The overall kinematic score is computed as the average of the hip, knee, and ankle scores. Vertical GRF human-reference score To evaluate dynamic plausibility, we analyze the vertical ground reaction force (GRF), which directly reflects body loading and push-off behavior. Individual left and right GRF gait cycles are visualized to assess stance-phase loading patterns, double-support and propulsion behavior, and stride-to-stride consistency. The numerical GRF score is computed using the human-reference range metric in Eq. 15, applied to the vertical GRF profile, whereas the waveform plots provide a qualitative comparison of left–right symmetry and stride-to-stride repeatability. 10

Symmetry and repeatability score To evaluate gait consistency and repeatability, we quantify bilateral symmetry and stride-to-stride repeatability. Bilateral symmetry is measured as the root-mean-square (RMS) difference between the average left and right normalized gait-cycle waveforms. For a gait-cycle quantity y, this waveform difference is v u N u1 X Dy = t (ȳL (xk ) − ȳR (xk ))2 ,

N k=1

(17)

where y denotes a normalized gait-cycle waveform, xk is the kth gait-cycle sample, N is the number of samples in the normalized gait cycle, and ȳL and ȳR are the average left- and right-side profiles computed over the analyzed cycles. In the results, lower Dy indicates more symmetric left–right motion. Stride-to-stride repeatability is quantified using the coefficient of variation (CV) of stride time and stride length, CVz = 100

σz , µz

(18)

where z denotes either stride time or stride length, µz is the mean value of z, and σz is its standard deviation over the analyzed gait cycles. The factor 100 expresses the coefficient of variation as a percentage. Lower CVz indicates more repeatable gait cycles. In the reported results, the Hip, Knee, Ankle, and GRF columns correspond to Dy for the respective left–right waveform pairs, whereas the Time CV and Length CV columns report CVz for stride time and stride length, respectively.

4.2

Nominal Gait Generation

This experiment evaluates whether the proposed controller generates physiologically plausible locomotion under nominal walking conditions. Three controllers (E2E-RL, DEP-RL, and Residual-Reflex RL) are compared at target walking speeds of 0.8, 1.0, and 1.2 m/s. The evaluation proceeds from task performance to kinematic plausibility, dynamic plausibility, gait consistency, and qualitative motion analysis. 4.2.1

Task Performance

The speed tracking and stride measures show that all three controllers successfully generated sustained forward walking under the commanded speed tasks. As summarized in Table 3, Residual-Reflex RL achieved the most accurate speed tracking at all target speeds, with absolute speed errors of 0.01, 0.05, and 0.01 m/s at 0.8, 1.0, and 1.2 m/s, respectively. E2E-RL and DEP-RL also completed all walking tasks, but exhibited larger speed errors, particularly at higher target speeds where both methods tended to overshoot the commanded velocity. These results establish that the subsequent comparisons are performed among controllers that all accomplish the locomotion task, rather than between successful and failed walking trials. The stride measures further reveal different gait strategies. E2E-RL tends to use shorter and faster steps at lower walking speeds, whereas DEP-RL and Residual-Reflex RL adopt longer strides. The following analyses therefore investigate whether successful task execution is also accompanied by human-like kinematics, dynamic consistency, and gait consistency. 4.2.2

Joint Kinematic Plausibility

The human-reference kinematic scores reveal a clear separation between the three controllers. As summarized in Table 4, Residual-Reflex RL achieved the highest joint-kinematic score at all target walking speeds, with an average score of 88.0% across the 0.8, 1.0, and 1.2 m/s evaluations. The improvement is most pronounced in the knee and ankle joints, where the 11

Table 3: Speed tracking and stride measures from the analyzed walking trajectories. Speed error is the absolute difference between realized and target speed. Here v ∗ and v̂ are the target and realized speeds, L and T are stride length and stride time, and N is the number of analyzed cycles. v∗

Method

v̂

|∆v|

0.8

E2E-RL DEP-RL Residual-Reflex RL

0.87 0.83 0.79

0.07 0.03 0.01

0.78 1.09 0.99

0.89 1.31 1.26

50 33 35

1.0

E2E-RL DEP-RL Residual-Reflex RL

1.18 1.14 1.05

0.18 0.14 0.05

1.39 1.29 1.23

1.18 1.13 1.17

37 39 37

1.2

E2E-RL DEP-RL Residual-Reflex RL

1.26 1.30 1.21

0.06 0.10 0.01

1.23 1.37 1.34

0.97 1.06 1.11

46 42 40

Table 4: Human-reference kinematic scores in controlled walking validation. Scores are computed from normalized hip, knee, and ankle gait-cycle profiles. The kinematic average (Avg.) is the mean of the hip, knee, and ankle scores; v ∗ and v̂ are the target and realized speeds. v∗

Method

0.8

v̂

Avg.

Hip

Knee

Ankle

E2E-RL DEP-RL Residual-Reflex RL

0.87 0.83 0.79

39.0 44.1 81.3

98.3 72.7 96.6

18.6 50.8 68.9

0.0 8.7 78.5

1.0

E2E-RL DEP-RL Residual-Reflex RL

1.18 1.14 1.05

81.9 58.5 90.5

81.2 62.5 100.0

72.7 64.2 82.6

91.8 48.7 88.9

1.2

E2E-RL DEP-RL Residual-Reflex RL

1.26 1.30 1.21

75.6 40.8 92.3

88.4 44.5 99.6

52.4 33.4 88.7

85.9 44.5 88.6

proposed controller consistently preserves human-reference joint coordination. In contrast, E2E-RL maintains relatively high hip scores but exhibits less consistent knee and ankle motion, whereas DEP-RL performs consistently worse, particularly at 1.2 m/s. Fig. 3 summarizes these quantitative comparisons across all target speeds. Scores close to zero indicate that the averaged joint trajectory remains outside the humanreference range over most of the gait cycle, rather than exhibiting only localized deviations. This distinction is important because controllers can successfully complete the walking task while still producing joint motions that differ substantially from human gait. The waveform comparisons in Figs. 4–6 further explain these quantitative results and illustrate how the controllers behave across different target speeds. At 0.8 m/s, E2E-RL produced a reasonable hip trajectory but failed to maintain human-reference knee and ankle profiles, whereas DEP-RL exhibited larger left–right differences and stride-to-stride variability. At 1.0 m/s, all controllers produced gait patterns closer to the reference trajectories, but Residual-Reflex RL still generated the most compact waveform distribution and the highest overall score. At 1.2 m/s, where faster stance-to-swing transitions are required, the difference became more pronounced. Residual-Reflex RL maintained knee and ankle trajectories clustered around the human-reference 12

Figure 3: Human-reference kinematic gait scores across target speeds. Numeric labels indicate the score of each bar. The Residual-Reflex RL controller achieves the highest kinematic-average score and especially improves knee and ankle components. ranges, whereas E2E-RL and DEP-RL showed larger deviations and more scattered gait cycles. These observations are consistent with the quantitative scores reported in Table 4 and further explain why Residual-Reflex RL produces more repeatable locomotion. The narrower distributions of the left and right gait cycles indicate more consistent joint coordination across both successive gait cycles and bilateral limbs. 4.2.3

Dynamic Plausibility

The vertical GRF scores further distinguish the three controllers. As summarized in Table 5, Residual-Reflex RL consistently achieved the highest vertical GRF scores across all target walking speeds, indicating the closest agreement with the human-reference loading patterns. In contrast, both E2E-RL and DEP-RL produced substantially lower GRF morphology scores, with DEP-RL showing near-zero agreement at 1.0 and 1.2 m/s. The corresponding GRF waveforms are shown in Fig. 7. Residual-Reflex RL generated smoother vertical GRF profiles with a more human-reference double-support pattern throughout stance. In contrast, E2E-RL exhibited irregular loading despite producing visually plausible joint trajectories in some cases, whereas DEP-RL produced pronounced early-stance impact peaks and larger stride-to-stride variations, particularly at higher walking speeds. A GRF score of 0 indicates that the averaged force profile remains outside the human-reference range after normalization, rather than reflecting only localized deviations. This observation highlights an important distinction between kinematic and dynamic evaluation. Although E2ERL and DEP-RL can generate visually plausible joint motion, their foot–ground interactions

Figure 4: Human-reference kinematic gait-cycle visualization at target speed 0.8 m/s for E2E-RL, DEP-RL, and Residual-Reflex RL. Blue and red curves denote left and right gait cycles, respectively; the black curve is the cycle average and the gray band is the human-reference range. Horizontal axes show normalized gait cycle percentage. Table 5: Vertical GRF human-reference scores in controlled walking validation. Higher values indicate closer agreement with the human-reference vertical GRF range. Method E2E-RL DEP-RL Residual-Reflex RL

0.8 m/s

1.0 m/s

1.2 m/s

35.9 39.8 76.7

27.6 0.0 89.8

21.5 0.0 89.1

remain substantially less consistent with human walking dynamics. The smoother GRF profiles produced by Residual-Reflex RL therefore demonstrate that the proposed controller improves not only joint kinematics but also dynamic consistency during stance. 4.2.4

Gait Consistency

The symmetry and repeatability scores demonstrate that Residual-Reflex RL consistently produces more symmetric and repeatable gait cycles than E2E-RL and DEP-RL. As summarized in Table 6, Residual-Reflex RL achieved lower left–right waveform differences and lower stride time and stride length coefficients of variation across all target walking speeds. The Hip, Knee, Ankle, and GRF columns report the left–right waveform difference Dy , whereas the Time CV and Length CV columns report the coefficient of variation of stride time and stride length, 14

Figure 5: Human-reference kinematic gait-cycle visualization at target speed 1.0 m/s for E2E-RL, DEP-RL, and Residual-Reflex RL. Blue and red curves denote left and right gait cycles, respectively; the black curve is the cycle average and the gray band is the human-reference range. Horizontal axes show normalized gait cycle percentage. respectively. The largest improvements are observed in ankle symmetry and stride variability, where E2E-RL and DEP-RL exhibit substantially less repeatable gait cycles. These quantitative results are consistent with the waveform visualizations, indicating that residual-reflex modulation improves bilateral coordination as well as stride-to-stride repeatability. For example, at 1.2 m/s, Residual-Reflex RL achieved stride time and stride length coefficients of variation of only 0.70% and 0.74%, respectively, indicating that the standard deviations of these quantities were below 1% of their corresponding mean values. In contrast, E2E-RL and DEP-RL exhibited considerably larger cycle-to-cycle variability. These results indicate that the proposed controller not only produces a physiologically plausible average gait cycle, but also maintains consistent locomotion over successive steps. 4.2.5

Qualitative Motion Analysis

The qualitative gait snapshots in Fig. 9 provide a visual interpretation of the quantitative results presented in the preceding sections. Representative frames sampled at 0%, 25%, 50%, 75%, and 100% of the gait cycle illustrate how the different controllers organize stance, push-off, and swing throughout a complete walking cycle. Residual-Reflex RL produces the most natural-looking gait sequence. The body remains upright throughout the gait cycle, while the transition between stance and swing is smooth and well coordinated. These observations are consistent with its higher joint-kinematic, GRF, and 15

Figure 6: Human-reference kinematic gait-cycle visualization at target speed 1.2 m/s for E2E-RL, DEP-RL, and Residual-Reflex RL. Blue and red curves denote left and right gait cycles, respectively; the black curve is the cycle average and the gray band is the human-reference range. Horizontal axes show normalized gait cycle percentage. gait-consistency scores, indicating that the proposed controller achieves both physiologically plausible joint motion and stable foot–ground interaction. E2E-RL also generates sustained walking and maintains visually reasonable postures throughout the gait cycle. However, when interpreted together with the quantitative results, the motion exhibits less coordinated limb movement and less consistent weight transfer, explaining its lower GRF and gait-consistency scores despite relatively good joint-kinematic performance in some conditions. DEP-RL exhibits the largest qualitative deviations. Around 25% of the gait cycle, the body leans further forward while the swing leg adopts a less natural knee–ankle configuration. Similar deviations remain visible near 75% of the gait cycle, resulting in less coordinated stance-toswing transitions. These observations are consistent with its lower joint-kinematic, GRF, and gait-consistency scores. The accompanying supplementary video further illustrates these differences over continuous motion. Compared with the sampled snapshots, the video more clearly reveals the temporal characteristics of weight transfer, limb coordination, and stride-to-stride consistency throughout the gait cycle.

Figure 7: Vertical GRF morphology across target speeds for E2E-RL, DEP-RL, and ResidualReflex RL. Blue and red thin curves show individual left and right normalized gait cycles, respectively; the black curve denotes the cycle mean, the gray band denotes the human-reference range used for post-hoc evaluation, and the panel annotations report the GRF human-reference score. Horizontal axes show normalized gait cycle percentage.

4.3

Adaptation to Plantarflexor Weakness

Beyond nominal walking, this experiment evaluates whether the trained Residual-Reflex RL controller adapts to changes in musculoskeletal capacity without retraining. Plantarflexor weakness was simulated by reducing the maximum isometric force of the soleus muscles while keeping the trained 1.2 m/s policy fixed. Soleus strength was varied from 83% to 100% of the nominal value. Residual-Reflex RL was compared with the CMA-ES-optimized reflex controller under the same weakened musculoskeletal models. To analyze the adaptation strategy, we additionally report the root-mean-square (RMS) magnitude of each residual-action channel, v u T u1 X RMS t A = a (t)2 , i

T t=1

(19)

where ai (t) is the ith residual action and T is the number of evaluated policy steps. The RMS magnitude quantifies how strongly each residual channel is utilized regardless of sign.

Table 6: Cycle-level left–right waveform differences and stride variability. Panel (a) reports Dy , the RMS difference between the average left and right gait-cycle waveforms. Panel (b) reports 100σz /µz for stride time and stride length. Lower values indicate more symmetric or more repeatable gait. (a) Left–right waveform differences Dy v∗

Method

Hip

Knee

Ankle

GRF

0.8

E2E-RL DEP-RL Residual-Reflex RL

2.92 11.20 0.80

5.75 11.73 1.03

6.36 11.59 0.89

0.10 0.12 0.04

1.0

E2E-RL DEP-RL Residual-Reflex RL

6.12 4.54 1.78

10.61 10.12 1.85

6.69 16.27 0.97

0.07 0.28 0.03

1.2

E2E-RL DEP-RL Residual-Reflex RL

3.43 8.33 0.87

3.00 5.83 0.87

9.84 6.31 0.46

0.04 0.34 0.02

(b) Stride variability (%) v∗

Method

Time CV

Length CV

0.8

E2E-RL DEP-RL Residual-Reflex RL

9.05 7.96 1.98

14.88 11.04 5.66

1.0

E2E-RL DEP-RL Residual-Reflex RL

5.03 2.62 1.51

5.63 5.27 1.86

1.2

E2E-RL DEP-RL Residual-Reflex RL

3.87 2.29 0.70

5.83 4.78 0.74

4.3.1

Survival comparison

The survival results immediately distinguish the two controllers. As summarized in Table 7, Residual-Reflex RL completed the full 25 s evaluation at all tested weakness levels, whereas the CMA-ES-optimized reflex controller terminated early for every weakened model and remained stable only under the nominal condition. These results demonstrate that online residual-reflex modulation substantially improves robustness to changes in musculoskeletal capacity compared with fixed offline-optimized reflex parameters. 4.3.2

Speed adaptation analysis

The speed adaptation results further reveal how the learned controller responds to reduced plantarflexor capacity. As shown in Table 8 and Fig. 10, the realized walking speed decreased gradually from 1.20 m/s under the nominal model to 0.95 m/s at 83% soleus strength. Rather than maintaining the commanded speed at the expense of stability, the controller adopted a slower but stable gait as muscle capacity decreased. This behavior resembles a common compensatory strategy observed in human walking, where reduced plantarflexor strength is accompanied by a reduction in preferred walking speed. Importantly, the slower gait was not merely a degraded survival behavior. As shown in Fig. 12, the generated joint trajectories 18

Figure 8: Left–right waveform differences and stride variability across target speeds. Lower values indicate more symmetric and more repeatable gait cycles. remained close to the human-reference profiles across all tested weakness levels, yielding a joint-kinematic score of 89.0% even at 83% soleus strength. These results demonstrate that the proposed controller accommodates substantial changes in muscle capacity while preserving physiologically plausible locomotion without retraining. The accompanying supplementary video further illustrates the representative 83% weakness condition. Residual-Reflex RL maintains a recognizable stance–swing sequence and upright progression, whereas the CMA-ES-optimized reflex controller loses balance and terminates. The synchronized gait-cycle curves further show the evolution of the hip, knee, ankle, and vertical GRF profiles during the two rollouts. 4.3.3

Residual-action analysis

The residual-action RMS values further explain the adaptation strategy adopted by the learned controller. Channel definitions are given in Table 1. Here we focus on how their modulation changes with soleus capacity. The soleus-related channel a3 exhibits the largest RMS magnitude throughout the tested weakness range, indicating that ankle support and propulsion remain strongly regulated as plantarflexor capacity decreases. The policy does not compensate by monotonically increasing a3 . Instead, it accepts a lower walking speed while redistributing modulation across the remaining pathways. The stronger response of a4 under greater weakness is consistent with increased reliance on knee-extension support when ankle push-off capacity is reduced, whereas a1 and a2 change more gradually. Together, the paired trends of (a1 ,a2 ) and (a3 ,a4 ) indicate coordinated redistribution across swing-related and stance-support pathways rather than single-parameter compensation. These results demonstrate that the four-dimensional 19

Figure 9: Qualitative gait snapshots over one detected gait cycle at 1.2 m/s for (a) E2E-RL, (b) DEP-RL, and (c) Residual-Reflex RL. Within each panel, frames were sampled at 0%, 25%, 50%, 75%, and 100% of the gait cycle detected from the corresponding STO trajectory and synchronized video. residual action provides a compact adaptive interface for online neuromuscular regulation, enabling the same trained policy to generalize across weakened musculoskeletal models without retraining.

4.4

Recovery from External Pushes

This experiment evaluates whether the trained Residual-Reflex RL controller can recover from external disturbances without retraining. The same 1.2 m/s policy used in the nominal walking

Table 7: Survival comparison under soleus weakness. Full episode duration is 25 s. The CMA-ES-optimized reflex controller remains stable only in the nominal 100% model, whereas Residual-Reflex RL completes the episode across all tested weakness levels without retraining. Soleus

Residual duration (s)

Residual full

CMA-ES duration (s)

CMA-ES full

83% 87% 90% 93% 95% 97% 100%

25.0 25.0 25.0 25.0 25.0 25.0 25.0

Yes Yes Yes Yes Yes Yes Yes

1.27 1.28 1.28 1.29 1.48 1.47 25.0

No No No No No No Yes

Table 8: Residual-Reflex RL under soleus weakness. The target speed was 1.2 m/s. All listed residual rollouts reached 1000 steps. Channel a3 is the soleus force-feedback modulation defined in Table 1. Soleus strength

Realized speed (m/s)

Speed error (m/s)

RMS a3

83% 87% 90% 93% 95% 97% 100%

0.949 1.002 1.063 1.112 1.137 1.167 1.201

0.251 0.198 0.137 0.088 0.063 0.033 0.001

0.846 0.851 0.878 0.885 0.889 0.895 0.898

and plantarflexor-weakness experiments was retained without further learning. Backward force pulses were applied to the torso for 0.1 s starting at 3.0 s. Perturbation magnitudes of 75 N and 100 N were evaluated under both single-push and repeated-push (every 5 s) conditions. Residual-Reflex RL was compared with the CMA-ES-optimized reflex controller under identical perturbation settings. As summarized in Table 9, Residual-Reflex RL successfully recovered from both 75 N and 100 N single backward pushes and maintained walking for the full 25 s evaluation. Under repeated perturbations, the controller also completed the full evaluation under the 75 N condition, whereas the more challenging 100 N repeated pushes resulted in termination after 20.1 s. In contrast, the CMA-ES-optimized reflex controller consistently failed after approximately 4.5–4.6 s under all perturbation conditions. These results demonstrate that the learned residual-reflex policy substantially improves recovery capability beyond fixed offline-optimized reflex gains. Figure 13 provides a qualitative comparison of the recovery process under the representative 100 N single-push condition. Before the perturbation, both controllers produce stable walking. Following the force pulse, the CMA-ES-optimized reflex controller progressively loses balance and terminates, whereas Residual-Reflex RL reorganizes the subsequent steps, restores upright walking, and completes the full evaluation. The matched frames therefore distinguish sustained recovery from trajectories that merely delay failure. The accompanying supplementary video first illustrates the single-push recovery process by comparing Residual-Reflex RL under 75 N and 100 N backward pushes with the CMA-ESoptimized reflex controller under a 100 N backward push. Residual-Reflex RL reorganizes foot placement and resumes periodic walking after the disturbance, whereas the CMA-ES-optimized

Figure 10: Speed adaptation under plantarflexor weakness. The same trained Residual-Reflex RL policy remains stable across weakened soleus models but adopts a slower realized speed as force capacity is reduced. Table 9: External backward push perturbation results. Full episode duration is 25 s. Method

ResidualReflex RL

CMA-ESoptimized Reflex

Perturbation

Force (N)

Duration (s)

Speed (m/s)

Single push

75 100

25.0 25.0

1.188 1.176

Every 5 s

75 100

25.0 20.1

1.129 1.091

Single push

75 100

4.5 4.6

1.098 1.069

Every 5 s

75 100

4.5 4.6

1.098 1.069

reflex controller fails to regain a stable gait. The video then compares Residual-Reflex RL under repeated 75 N and 100 N backward pushes applied every 5 s. The controller remains stable throughout the 75 N trial but terminates after 20.1 s under repeated 100 N pushes, illustrating the boundary of its recovery capability. Together with the plantarflexor-weakness experiments, these results demonstrate that residual-reflex modulation provides a flexible online adaptation mechanism for changes in both musculoskeletal capacity and external disturbances.

Figure 11: Residual-action modulation under plantarflexor weakness. RMS values show that the trained policy continues to adjust all four selected reflex channels. Channel a3 modulates soleus force feedback for ankle support/propulsion, whereas a4 modulates vasti force feedback for knee extension and stance support.

Figure 12: Human-reference kinematic profiles under soleus weakness. Colored curves show the mean normalized gait-cycle profiles generated by the same trained Residual-Reflex RL policy under different soleus strength levels; the gray bands denote the human-reference ranges.

Discussion

The experimental results demonstrate that regulating an existing neuromuscular controller through reinforcement learning provides a practical way to improve both physiological plausibility

Figure 13: Time-aligned response to a 100 N backward torso push applied for 0.1 s at t = 3.0 s. Columns show synchronized frames from 2.8 s to 6.0 s. The CMA-ES-optimized reflex controller (top) progressively loses upright balance and terminates at 4.57 s. Residual-Reflex RL (bottom) modifies the following steps and continues walking for the full 25 s rollout under the same perturbation. Red arrows indicate the applied force. and adaptability in muscle-driven locomotion. Under nominal walking conditions, ResidualReflex RL consistently achieved better joint kinematics, more human-like ground reaction forces, and higher gait consistency than end-to-end muscle-control policies. The weakness and perturbation experiments further showed that the same trained policy remained effective under changes in musculoskeletal capacity and external disturbances without retraining. Together, these results indicate that reinforcement learning can benefit from regulating neuromuscular control mechanisms rather than directly generating muscle actions. The observed improvements in joint kinematics, GRF morphology, and gait consistency can be understood from the learning formulation itself. In end-to-end muscle control, reinforcement learning directly operates in a high-dimensional and redundant muscle action space, where both muscle activation and the functional relationships among muscles must be learned from task rewards. In the proposed framework, the underlying neuromuscular topology is fixed by the phase-dependent reflex controller, such that the functional organization of sensory feedback pathways is explicitly defined. Reinforcement learning therefore no longer needs to construct this neuromuscular organization from scratch, but instead learns how to regulate a small set of biomechanically meaningful reflex parameters according to the current state. This allows learning to focus on state-dependent adaptation while preserving the existing neuromuscular organization. The weakness and perturbation experiments further clarify why this formulation improves adaptability. The CMA-ES-optimized reflex controller also relies on physiological feedback, but its parameters remain fixed once optimization is completed. In contrast, Residual-Reflex RL preserves the same neuromuscular control structure while continuously adjusting the selected reflex parameters according to the current musculoskeletal state. This adaptation mechanism becomes particularly important when muscle capacity changes or external disturbances occur, 24

because the controller can continuously regulate the underlying reflex parameters instead of relying on a single fixed parameter set. These findings demonstrate that reflex-informed neuromuscular regulation provides an effective formulation for improving both physiological plausibility and adaptability in muscle-driven locomotion.

Conclusion

We presented a Reflex-Informed Neuromuscular Reinforcement Learning framework for adaptive muscle-driven locomotion. Instead of directly generating muscle excitations, the proposed framework learns to regulate four biomechanically meaningful residual parameters of a phasedependent reflex controller. This formulation preserves the underlying neuromuscular control structure while enabling reinforcement learning to adapt reflex behavior online through a compact and interpretable control interface. Experiments in a Hyfydy-based musculoskeletal simulation demonstrated that the proposed framework consistently improved physiological plausibility under nominal walking conditions. Compared with end-to-end muscle-control policies, Residual-Reflex RL generated more humanlike joint kinematics, ground reaction forces, and gait consistency across walking speeds of 0.8, 1.0, and 1.2 m/s. Beyond nominal walking, the same trained policy remained effective under plantarflexor weakness and external perturbations without retraining, demonstrating improved adaptability while preserving physiologically plausible locomotion. These results demonstrate that integrating reinforcement learning with an existing neuromuscular control structure provides an effective formulation for simultaneously improving physiological plausibility and adaptability in muscle-driven locomotion. The proposed framework offers a compact and biologically meaningful learning interface that is applicable to muscle-driven character animation, computational studies of human locomotion, and bio-inspired locomotor control. Future work will extend the framework to continuous speed transitions, broader musculoskeletal morphologies, and more diverse physical interactions.

References Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. Marko Ackermann and Antonie J. van den Bogert. Optimality principles for model-based prediction of human gait. Journal of Biomechanics, 43(6):1055–1060, 2010. Frank C. Anderson and Marcus G. Pandy. Dynamic optimization of human walking. Journal of Biomechanical Engineering, 123(5):381–390, 2001. Zachary Batts, Seungmoon Song, and Hartmut Geyer. Toward a virtual neuromuscular control for robust walking in bipedal robots. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 6318–6323, 2015. Andrea d’Avella, Philippe Saltiel, and Emilio Bizzi. Combinations of muscle synergies in the construction of a natural motor behavior. Nature Neuroscience, 6(3):300–308, 2003. Friedl De Groote and Antoine Falisse. Perspective on musculoskeletal modelling and predictive simulations of human movement to assess the neuromechanics of gait. Proceedings of the Royal Society B, 288(1946), 2021. doi: 10.1098/rspb.2020.2432. Roy de Vree and Raffaella Carloni. Deep reinforcement learning for physics-based musculoskeletal simulations of healthy subjects and transfemoral prostheses’ users during normal walking. 25

IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:607–618, 2021. doi: 10.1109/TNSRE.2021.3063015. Christopher L. Dembia, Nicholas A. Bianco, Antoine Falisse, Jennifer L. Hicks, and Scott L. Delp. OpenSim Moco: Musculoskeletal optimal control. PLOS Computational Biology, 16 (12), 2020. doi: 10.1371/journal.pcbi.1008493. Florin Dzeladini, Jesse van den Kieboom, and Auke Jan Ijspeert. The contribution of a central pattern generator in a reflex-based neuromuscular model. Frontiers in Human Neuroscience, 8, 2014. Milad Ezati, Borna Ghannadi, and John McPhee. A review of simulation methods for human movement dynamics with emphasis on gait. Multibody System Dynamics, 47:265–292, 2019. Antoine Falisse, Giordano Serrancolí, Christopher L. Dembia, Jan Gillis, Ilse Jonkers, and Friedl De Groote. Rapid predictive simulations with complex musculoskeletal models suggest that diverse healthy and pathological human gaits can emerge from similar control strategies. Journal of the Royal Society Interface, 16(157), 2019. doi: 10.1098/rsif.2019.0402. Thomas Geijtenbeek. SCONE: Open source software for predictive simulation of biological motion. Journal of Open Source Software, 4(38), 2019. Thomas Geijtenbeek. The Hyfydy simulation software. Online, November 2021. URL https: //hyfydy.com. Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. Flexible musclebased locomotion for bipedal creatures. ACM Transactions on Graphics, 32(6), 2013. doi: 10.1145/2508363.2508399. Hartmut Geyer and Hugh Herr. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18(3):263–273, 2010. Sten Grillner and Abdeljabbar El Manira. Current principles of motor control, with special reference to vertebrate locomotion. Physiological Reviews, 100(1):271–320, 2020. Auke Jan Ijspeert. Central pattern generators for locomotion control in animals and robots: A review. Neural Networks, 21(4):642–653, 2008. S. Koseki, M. Hayashibe, and D. Owaki. Identifying essential factors for energy-efficient walking control across a wide range of velocities in reflex-based musculoskeletal systems. PLOS Computational Biology, 20(1), 2024. doi: 10.1371/journal.pcbi.1011771. Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Transactions on Graphics, 38(4), 2019. Guoping Li, Auke Jan Ijspeert, and Mitsuhiro Hayashibe. AI-CPG: Adaptive imitated central pattern generators for bipedal locomotion learned through reinforced reflex neural networks. IEEE Robotics and Automation Letters, 9(6):5190–5197, 2024. Andrew J. Meyer, Ilya Eskinazi, John N. Jackson, Anil V. Rao, Carolynn Patten, and Benjamin J. Fregly. Muscle synergies facilitate computational prediction of subject-specific walking motions. Frontiers in Bioengineering and Biotechnology, 4, 2016. K. Nowakowski et al. Human locomotion with reinforcement learning using bioinspired reward reshaping strategies. Medical & Biological Engineering & Computing, 59:243–256, 2021.

Brown N. Ogum, Lambert R. B. Schomaker, and Raffaella Carloni. Learning to walk with deep reinforcement learning: Forward dynamic simulation of a physics-based musculoskeletal model of an osseointegrated transfemoral amputee. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 32:431–441, 2024. doi: 10.1109/TNSRE.2024.3352416. Xue Bin Peng and Michiel van de Panne. Learning locomotion skills using DeepRL: Does the choice of action space matter? In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2017. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. DeepMimic: Exampleguided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics, 37(4), 2018. Pierre Schumacher, Daniel F. B. Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. DEP-RL: Embodied exploration for reinforcement learning in overactuated and musculoskeletal systems. In International Conference on Learning Representations, 2023. Pierre Schumacher, Thomas Geijtenbeek, Vittorio Caggiano, Vikash Kumar, Syn Schmitt, Georg Martius, and Daniel F. B. Haeufle. Emergence of natural and robust bipedal walking by learning from biologically plausible objectives. iScience, 28, 2025. Seungmoon Song and Hartmut Geyer. A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotion. The Journal of Physiology, 593(16): 3493–3511, 2015. Seungmoon Song and Hartmut Geyer. Predictive neuromechanical simulations indicate why walking performance declines with ageing. The Journal of Physiology, 596(7):1199–1210, 2018. Seungmoon Song, Łukasz Kidziński, Xue Bin Peng, Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G. Atkeson, and Scott L. Delp. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. Journal of NeuroEngineering and Rehabilitation, 18, 2021. L. Su and Elena M. Gutierrez-Farewik. Simulating human walking: A model-based reinforcement learning approach with musculoskeletal modeling. Frontiers in Neurorobotics, 17, 2023. doi: 10.3389/fnbot.2023.1244417. L. Sun, Y. Wang, and W. Qin. A language-directed virtual human motion generation approach based on musculoskeletal models. Computer Animation and Virtual Worlds, 35(3), 2024. Thomas K. Uchida and Scott L. Delp. Biomechanics of Movement: The Science of Sports, Robotics, and Rehabilitation. MIT Press, Cambridge, MA, USA, 2020. Jack M. Wang, David J. Fleet, and Aaron Hertzmann. Optimizing walking controllers for uncertain inputs and environments. ACM Transactions on Graphics, 29(4), 2010. Jack M. Wang, Samuel R. Hamner, Scott L. Delp, and Vladlen Koltun. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics, 31(4), 2012. Jun Weng, Ehsan Hashemi, and Arash Arami. Natural walking with musculoskeletal models using deep reinforcement learning. IEEE Robotics and Automation Letters, 6(2):4156–4162, 2021.

Record · ID 673513 · SHA-256 a03f010dfcc9a157
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.