Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction

1

Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction

arXiv:2610.37441v1 [cs.RO] 27 Oct 2026

Yifei Yuan, Jakob Wolf, Ghaith Androwis, and Xianlian Zhou

Abstract—Objective: This study aims to determine whether a hip-exoskeleton policy trained in simulation, by co-adapting with a musculoskeletal human model under a metabolic-energy reward, reduces the metabolic cost of walking in new users without per-user tuning, and how mechanical-power shaping affects the learned assistance. Methods: We developed MARLExo, a multi-agent reinforcement learning framework in which a full-body human model (304 muscle actuators; randomized anatomy, gait, and speed) and a bilateral hip-exoskeleton policy are trained concurrently in two stages. The exoskeleton observes only delayed hip kinematics and its past commands; its reward includes muscle metabolic power from a physiologically based energetics model, alone (Metabolic) or with mechanical-power shaping (Metabolic + Power). The Metabolic policy was deployed and tested without retraining in eight healthy adults walking at 1.25 m/s, using indirect calorimetry. Results: In simulation, the Metabolic policy reduced modeled muscle metabolic power by 13.8% relative to no-device walking, versus 8.8% for Metabolic + Power. On hardware, MARL-Exo reduced net metabolic rate by 18.3 ± 6.6% relative to Passive in all eight participants. Relative to No-Exo, the mean reduction was 7.4 ± 10.0%; six participants more than offset the 13.1% device-wearing penalty, walking up to 24.2% below their NoExo cost. Conclusion: A metabolically informed policy coadapted in simulation transferred to users without per-user tuning and consistently reduced the metabolic cost of walking with the device, while explicit power shaping reduced the simulated benefit. Significance: Unlike per-user human-in-theloop optimization, this approach yields a single population-level controller and allows assistance objectives to be screened in simulation before human testing. Index Terms—Exoskeletons, physical human–robot interaction, reinforcement learning, co-adaptation, metabolic cost, wearable robotics.

I. I NTRODUCTION

P

OWERED lower-limb exoskeletons are being developed both for gait rehabilitation and for augmenting healthy walking [1], [2]. Among them, hip exoskeletons are especially Yifei Yuan, Jakob Wolf, and Xianlian Zhou are with the Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA (e-mail: yy72@njit.edu;jw768@njit.edu;alexzhou@njit.edu). Ghaith Androwis is with Kessler Foundation, West Orange, NJ 07052, USA (e-mail: GAndrowis@kesslerfoundation.org). This work was partially supported by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funded Rehabilitation Engineering Research Center Grant 90REGE0025-01-00 and NSF Award #2524089. The human study was approved by the Institutional Review Board under Protocol No. 2305033091R003 and conducted in accordance with the Declaration of Helsinki. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

attractive because the hip musculature performs a large fraction of the positive work of level locomotion, so assistance there can offload muscles that do much of the work of walking. Delivering that benefit efficiently is challenging because the wearer is not a passive plant. Once a device applies torque at a joint, the mechanics of gait change, and the neuromuscular system responds by recruiting muscles differently and re-coordinating the limbs [3]. Each side of the interaction thus changes the operating conditions of the other, and the controller is tuned against a wearer who is still adapting. The problem is compounded by the choice of objective: active devices can shape the assistance profile flexibly, but the benefit they deliver depends strongly on what the controller is optimized for, and on how well that objective reflects the wearer’s response. Human-in-the-loop optimization tunes the objective directly on the wearer, but it demands long peruser experiments that must be repeated for every user and condition [4], [5]. Learning the controller in musculoskeletal simulation offers a way around those experiments, since the wearer’s response can be simulated rather than measured. A neural network control policy for a full lower limb exoskeleton coupled to a musculoskeletal model was first used to learn robust assistance for walking [6], and a similar idea was used for joint human–exoskeleton training with sim-to-real transfer that produced a control policy for a hip exoskeleton that lowered metabolic cost without subject-specific tuning [7]. The paradigm has since been extended to speed- and slopeaware assistance through muscle-synergy priors [8], to reflexbased human models paired with end-to-end hip policies [9], to human-aligned simulation of assistance parameters [10], and to asymmetric gait [11], squatting [12], sit-to-stand [13], and running [14]. Two gaps remain in this literature. First, the mutual adaptation of human and device is mostly left implicit: when both policies are optimized at once, each shifts the data distribution seen by the other, and training can drift toward oscillatory torque, mistimed assistance, or divergence. Second, the effort terms used in training are typically proxies such as muscle activation or work. Luo et al. [7] explicitly chose activation over a metabolic-rate term because of calibration and rewardsmoothness concerns, and other controllers use kinematic tracking or activation-based effort [8], [12]. What a wearer ultimately experiences is metabolic energy, which a musculoskeletal simulation can compute directly but which, to our knowledge, has not been used as a term of the exoskeleton

2

policy reward. The closest examples are of two kinds. ExoPlore rewards a simplified activation-weighted energy expression rather than a physiological muscle-energetics model [10]. Barati et al. [9] optimize their reflex-based human model with Umberger’s energetics as the objective; however, this objective was not incorporated into the exoskeleton policy reward, which instead used an activation-based effort term. Curriculum learning is a natural remedy for the first gap, since decomposing a hard learning problem into ordered stages improves stability [15] and physiological plausibility [16], [17]. We previously proposed Staged Multi-Agent Training (SMAT), a four-stage curriculum trained on a compact 26-muscle model [18], [19]. Here we ask two further questions. Can co-adaptive training remain stable, and transfer to real users, when the human is a far richer full-body model with a population of body sizes and gaits? And how does physiological energetics, with and without explicit mechanicalpower shaping, influence the assistance learned through human–exoskeleton co-adaptation, and can the resulting policy transfer to human users? This study addresses both questions with MARL-Exo, a multi-agent framework built on a full-body model with 304 Hill-type muscle actuators. Its human policy outputs residual joint targets adapted to a reference gait trajectory, which a learned muscle-activation layer converts into muscle activations. Its exoskeleton policy sees only intentionally delayed hip kinematics and its own recent commands, so that it can run on inexpensive wearable sensing. Training uses a twostage schedule (human-only gait, then joint co-adaptation), and the exoskeleton reward includes the metabolic power of the hip muscles, evaluated with a muscle-energetics model, either alone (Metabolic) or together with explicit mechanicalpower shaping (Metabolic + Power). The Metabolic policy was deployed on the same hardware and eight-participant cohort of the SMAT evaluation [19]. The contributions are as follows: 1) the use of a physiologically based muscle-energetics model as a term of the exoskeleton control policy’s reward, within a two-stage multi-agent framework that couples a full-body human model and a hip exoskeleton, through regularized constraints and conditions the policy on randomized body and gait parameters; 2) a simulation comparison showing that, with the metabolic term retained, removing explicit positivepower shaping lowered modeled metabolic power across the analyzed muscle set by 13.8 % relative to no-device walking, against 8.8 % when power shaping was included; and 3) a human validation in eight healthy adults, in which the deployed policy lowered net metabolic rate by 18.3 % relative to the passive device in all participants (adjusted p < 0.001), more than offset the device’s 13.1 % wearing penalty in six of eight participants (mean 7.4 % and up to 24.2 % below no-device walking), enlarged hip range of motion by 36.8 %, and delivered predominantly positive estimated assistance power (98.5 %). The rest of this paper is organized as follows. Section II describes the simulation model and training framework, Sec-

tion III the hardware and human experiment, Section IV the simulation and experimental results, and Sections V and VI discuss and conclude. II. C OOPERATIVE M ULTI -AGENT T RAINING A. Human–Exoskeleton Coupling The human and the hip exoskeleton are modeled as two separate multibody systems in a shared physical simulation world simulated with the DART engine [20]; each has its own segment, joint, and inertia definitions. The human musculoskeletal model has 50 joint degrees of freedom (DOFs) (excluding the six global or root DOFs) and is driven by 304 Hill-type muscle actuators (line-of-action elements, several of which can represent one anatomical muscle) [21], [22]. The hip exoskeleton weighs 5.94 kg and has 4 DOFs, including two passive abduction/adduction DOFs and two actuated flexion/extension DOFs. The hip exoskeleton is rigidly attached to the human model through three permanent bilateral constraints resolved by the constraint solver: a weld fusing the pelvis to the exo backpack (all six DOFs locked) and a ball joint pinning each femur to the corresponding exo thigh segment (the three translational degrees of freedom locked). The constraints are regularized rather than ideally rigid: DART enforces them through finite constraint-force-mixing and errorreduction parameters, so the interface carries solver-level compliance while remaining much stiffer and cheaper to solve than explicit elastic or bushing connections [23], [6]. Because the ball joints leave hip rotation free, the wearer’s hip degrees of freedom are not over-constrained; we expect the regularization to affect mainly transient load transmission rather than the net sagittal-plane assistive torque. Exoskeleton motor torques (peak at ±18 N m) act on the device’s hip joints and are transmitted to the human pelvis and thighs through these constraints, altering the muscle effort required to produce the gait. B. Reinforcement Learning Framework Hip assistance is posed as a multi-agent reinforcement learning (MARL) problem [24], [25] with two cooperating agents that share one simulated world (Fig. 1): a human policy πH that drives the musculoskeletal model and an exoskeleton policy πE that commands hip torque. Each agent has its own observation, reward, value function (critic), and PPO update [26], and the two learn from one shared batch of rollouts. This framework shares its multi-agent formulation with SMAT [18] but differs from it in the human model (304 rather than 26 muscle actuators, with randomized anatomy and gait), the action pathway, the curriculum (two stages rather than four), the exoskeleton reward, and separate agentspecific rewards and critics (Table I). Observations. The human observation sH t is a full-body proprioceptive vector: the center-of-mass (COM) height; each body node’s COM position relative to the whole-body COM (∈ R3 ) and orientation (first two rows of its world rotation matrix, ∈ R6 ); each body node’s linear velocity relative to the COM velocity, the COM linear velocity, and each

3

(b) Deployment

(a) Training in simulation Human policy \(\pi_H\)

hip-muscle states

reward \(r_t^E\)

state \(s_t^H\), reward \(r_t^H\)

action \(a_t^H\)

SPD

\( tau_{\mathrm{des} \)

Muscle net \(\Phi^{\psi}\)

Shared physics simulation muscle activations \(boldsymbol{\alpha}\)

\(\psi\)

Supervised regression

\(\theta_H\)

Trained exo policy \(\pi_E\) on-board Raspberry Pi 5

Metabolic energy model Exo policy \(\pi_E\)

state \(s_t^E\)

sim-to-real

Stage 2: human-exo co-adaptation (adds \(\pi_E\) and the device) Stage 1: human-only gait learning

hip torque

IMU hip kinematics

(DART engine)

musculoskeletal human + bilateral hip exoskeleton

\(J^\top\!A\)

action \(a_t^E\)

rollout batch \( mathcal{D}\)

Multi-agent PPO (separate actor-critic per agent)

no subject-specific tuning

\(\theta_E\)

Human validation (8 adults; indirect calorimetry)

Fig. 1. Overview of MARL-Exo. (a) Training in simulation: a human policy πH and an exoskeleton policy πE act on a shared physics simulation; human actions pass through stable proportional-derivative (SPD) control and an auxiliary muscle network Φψ , and the exoskeleton reward is computed by a muscle metabolic energy model. Stage 1 trains πH alone, and Stage 2 adds the exoskeleton and co-adapts both policies. Multi-agent PPO updates both policies from a shared rollout batch D , while Φψ is trained by supervised regression to reproduce the desired joint torques τdes through the muscle moment-arm mapping J ⊤ A (line 22 of Algorithm 1). (b) The trained πE is deployed on a Raspberry Pi 5 without subject-specific tuning and validated in eight adults.

TABLE I M AIN DIFFERENCES BETWEEN MARL-E XO ( THIS WORK ) AND SMAT [18], [19]. SMAT

MARL-Exo

Human model

26 muscles (13 per side), 50-DOF body, 304 Hillimitation reward type muscle actuators, gait reward Human action Muscle excitations Residual joint targets on a reference trajectory → PD torque → learned muscle network Randomization None Anatomy, strength, gait, speed in observation Curriculum Four stages Two stages Critic Shared by both actors One per agent Exo reward Power sign and smooth- Smoothness, magnitude, ness hip-muscle metabolic power, and weak humanreward coupling (deployed policy) Online filtering None 3 Hz angle and velocity Experimental data Same cohort and session; rest, No-Exo, and Passive recordings shared; active trials order randomized

angular velocity (scaled by 0.1); the gait phase ϕt (with the local phase appended when the time-warp is active); the longitudinal COM-to-next-foothold distance; the scaled joint state (current and target angles, ×0.008); and the sampled or randomized anatomical parameters (gait, skeleton-scale, and muscle-force factors) within their sampled ranges. These parameters include whole-body and per-segment skeleton scales (±10% global, ±5% per segment), the maximum isometric force of each muscle (±25%), and gait stride and cadence (±25%). During training, the target walking speed was sampled from 0.65 m/s to 2.0 m/s, and the height and weight of the human model were randomized between [53.6 kg, 1.55 m] and [97.1 kg, 1.94 m]. Since these sampled parameters are appended to the policy’s observation, the controller is explicitly conditioned on the body and gait it drives rather than trained to be invariant. This parameterconditioned domain randomization enables a single training

to produce appropriate gaits across the sampled population. The resulting human observation has 649 dimensions, and the human policy outputs 51 actions: 50 residual joint-target offsets and one phase time-warp term, low-pass filtered with a 6 Hz Butterworth filter at the 50 Hz control rate. The residuals are added to a reference trajectory consisting of a single symmetrized, height-calibrated level-walking cycle which acts as a motion prior; no motion-imitation reward is used. The exo observation sE t is compact and delayed: for each of the two hip motors it stacks the joint angle and angular velocity over the last nhist =3 frames, delayed to two control frames (i.e. 40 ms) and low-pass filtered with a second-order Butterworth filter at 3 Hz, concatenated with the last na =3 exo actions (18 dimensions total). It outputs normalized bilateral hip torque commands û ∈ [−1, 1]2 , scaled by torque limits τmax to produce physical assistance, through an MLP with hidden sizes [128, 64]. In all exoskeleton training, an auxiliary symmetry loss (weight 10) added to the PPO objective penalized differences between the policy’s output and its mirrored output for left–right mirrored observations.

C. Two-Stage Training Training both agents jointly from scratch can often become unstable, as the exoskeleton optimizes against a human policy that has not yet acquired stable gait. We therefore separate training into two sequential stages: locomotion is first learned without the device, and the mechanically coupled exoskeleton is introduced only in the second stage, where both policies are optimized jointly. Algorithm 1 describes the training loop used in both stages; the stages differ in the presence of the exoskeleton and in the active reward terms. Human Baseline Gait Learning. Stage 1 trains the human policy alone, without the device, to walk stably at varied speeds, and the resulting policy is the starting point for coadaptation. With a gait-reward gain wg =2, the human reward multiplies locomotion, velocity, and foot-placement terms and

4

Algorithm 1 Cooperative MARL training of a human policy and a hip-exoskeleton policy with an auxiliary muscle actuator net Φψ . Require: PPO hyperparameter configurations; iterations I ; workers N ; muscle moment-arm matrix J ⊤A; exo motor torque limit τmax ; reference joint trajectory q ref (ϕ) (gait phase ϕ) for the residual policy; action scale ca = 0.04 θ θ 1: Init human policy πHH and exo policy πEE ψ 2: Init muscle net Φ ; broadcast ψ to all workers 3: for i = 1 to I do 4: for worker n = 1 to N do in parallel 5: while episode not terminated do 6: sH t ← GET H UMAN S TATE 7: sE t ← GET E XO S TATE H E E 8: aH t ∼ πH (· | st ); at ∼ πE (· | st ) target ref ← qt (ϕ) + ca aH 9: target angles qt t target 10: desired human joint torque τdes ← SPD(qt ) 11: muscle activations α ← Φψ (J ⊤A, τdes ) ▷ applied each substep 12: exo torque τE ← aE t × τmax 13: ENV. STEP ▷ advance one control step (ϕ) (nsub physics substeps, and advance phase(ϕ)) 14: rtH ← GET H UMAN R EWARD 15: rtE ← GET E XO R EWARD E E E H H 16: store (sH t , at , rt ), (st , at , rt ) in D ; cache

(J ⊤A, τdes )

terminate on end-of-episode / NaN end while 19: end for 20: compute per-agent GAE advantages from D clip clip 21: θH ← θH + η ∇θH JH ; θE ← θE + η ∇θE JE // Auxiliary supervised muscle learning 22: α ← Φψ (J ⊤A, τdes ); ψ ← arg minψ ∥(J ⊤A) α − τdes ∥2 + λ∥α∥2 (Adam); broadcast ψ 23: r̄H , r̄E ← mean episode reward of the human and exo agent over the rollouts of iteration i 24: end for θ θ 25: return πHH , πEE , Φψ 17: 18:

adds an activation-based effort term plus auxiliary regularizers: P k rtH = wg rloco ravg rstep + weff reff + k raux , (1) 2

2

rloco = e−4∥∆ẋhead ∥ e−4 ∆θhead , 2

rstep = e−2∥∆foot ∥ /n ,

∗

2

ravg = e−6∥v̄com −v ẑ∥ , 2

reff = e−8 ∥α∥ /Nm ,

(2)

where ∆ẋhead and ∆θhead are the head linear-acceleration and orientation deviations, v̄com is the current COM velocity, v ∗ = ℓstride /Tstride is the target COM speed, ẑ is the forward direction vector, ∆foot is the dead-zone-clipped swing-foot error, ∥α∥2 is the mean squared muscle activation over the control interval (Nm muscles), and weff =0.2. The term reff is an activation proxy for human effort and is distinct from the physiological energetics term of the exoskeleton reward bek low. The auxiliary terms raux penalize joint-limit violations and insufficient foot clearance and reward smooth joint torques (formulas and constants in Supplementary Section I).

Human–Exoskeleton Co-Adaptation. This stage begins the joint training of the human and exoskeleton policies. The human control policy is loaded from the human baseline training. The exo reward sums an action-smoothness penalty, an action-magnitude penalty, an optional joint-power term, a muscle metabolic-power term, and a small coupling to the human reward:

rtE = − 0.2∥∆aE ∥2 + ∥∆2 aE ∥2 − 0.01∥aE ∥2 + wp rpower + renergy + 0.02 rtH , X  rpower = max(0, Pj ) − 5 max(0, −Pj ) , j

  ĖUchida , −5 , renergy = max 1 − Ė0

(3)

(4)

where ∆aE , ∆2 aE are the first/second temporal differences of the exo action, Pj = τjE q̇j is the mechanical power at hip motor j (so positive/assistive power is rewarded and negative/resistive power penalized 5×), and ĖUchida (W) is the Uchida-model metabolic power of the hip muscles detailed below, normalized by a constant Ė0 = 150 W; the floor at −5 (i.e., ĖUchida ≥ 6Ė0 ) guards against extreme values caused by simulation instabilities. Two exoskeleton controllers were trained whose rewards differ only in wp : Metabolic + Power (wp = 0.005) and Metabolic (wp = 0). The names refer to the terms being compared, not the entire objective; both retain the smoothness, magnitude, metabolic, and humancoupling terms. Both controllers were obtained within the coadaptation stage. The Metabolic + Power controller was coadapted from the Stage-1 human policy until its exoskeleton reward plateaued. The Metabolic controller was then initialized from the converged best checkpoint and co-adaptation continued with the power term removed, so the deployed policy is a reward refinement within Stage 2 rather than an independently trained controller. The Metabolic controller is the one deployed on hardware. Physiologically Based Metabolic Energy Model. ĖUchida follows the muscle-energetics model of Umberger et al. [27] as implemented by Uchida et al. [28], summed over the muscle actuators that span the hip flexion/extension degree of freedom of either leg, i.e., all actuators with a nonzero moment arm about that axis. In the present model these are 46 actuators per side (92 in total), representing adductor brevis, longus, and magnus, gluteus maximus, medius, and minimus, iliacus, psoas major, pectineus, tensor fasciae latae, sartorius, gracilis, rectus femoris, the hamstrings (biceps femoris long head, semimembranosus, semitendinosus), and the short external rotators (piriformis, gemelli, obturators, quadratus femoris). Each muscle’s metabolic rate is the sum of a heat rate and a mechanical work rate, scaled by the mass mi of muscle i: X

ĖUchida =

Ḣi + Ẇi mi ,

i

(5)

Ḣi = max ḢAM,i + ḢS,i , 1 , where Ḣi (in W/kg, floored at 1) combines the activation– maintenance heat ḢAM and the shortening/lengthening heat

5

TABLE II PPO TRAINING HYPERPARAMETERS . T HE SAME CONFIGURATION IS SHARED BY THE HUMAN POLICY πH AND THE EXOSKELETON POLICY πE ; THE AUXILIARY MUSCLE NETWORK IS TRAINED BY SUPERVISED REGRESSION . Parameter

Value

Policy optimization (PPO, per agent) Learning rate 1×10−4 PPO clip range ϵ 0.2 Value-function clip 100 Value loss coefficient 1.0 Discount γ 0.99 GAE λ 0.99 SGD iters per update 4 Train batch size 8192 Minibatch size 512 KL coefficient (initial) 0.01 Target KL 0.01 Entropy coefficient 0 Gradient clipping none Batch mode truncate episodes Horizon 10000 Parallel workers (1 env each) 16 Auxiliary muscle network (supervised) Learning rate 1×10−4 Epochs per update 10 Minibatch size 512

ḢS , and Ẇ is the mechanical work rate. With muscle activation a ∈ [0, 1], active force–length factor Fiso , normalized fiber velocity ṽ = vce /ℓ0m , where vce is the muscle lengthening rate, ℓ0m is the optimal muscle fiber length, and aerobic scale φ = 1.5, the components are (muscle index omitted) ḢAM = φ a0.6 ζ(Fiso ), ḢS = φ ap σ(ṽ), (6) F act vce Ẇ = − , m where p=2 for shortening (ṽ ≤ 0) and p=1 for lengthening, F act = a f0 Fiso is the active fiber force (f0 : maximum isometric force), and ζ, σ are the fiber-length and fiber-velocity heat scalings, which also depend on the recruited fiber-type composition. The PPO hyperparameters, shared by both agents and both stages, are listed in Table II. The physics ran at 500 Hz and both policies at 50 Hz (10 physics substeps per control step), with human joint SPD (Stable Proportional Derivative [21]) gains kp = 200 and kv = 40. Training ran on a workstation with two 16-core Intel Xeon Gold 5218 CPUs (2.3 GHz) and an NVIDIA Quadro RTX 5000 GPU, using 16 rollout workers; the co-adaptation runs took about 24 h for 62.6 M control steps for the Metabolic + Power training and 14 h for 38.9 M control steps for the Metabolic training until convergence. D. Simulation Evaluation Simulations used a full-body musculoskeletal model with a total body mass of 72.9 kg. Walking was simulated at approximately 1.14 m/s with a mean gait-cycle duration of ∼1.10 s. Three conditions were analyzed: unassisted walking without the device (NoExo, a human-only model); an active hip exoskeleton whose controller was trained with both the positive power term rpower and the metabolic term renergy (Metabolic + Power, labeled PowerExo); and an active hip

exoskeleton whose controller was trained with rpower removed while retaining renergy (Metabolic, labeled EnergyExo). The EnergyExo controller is the policy deployed on hardware and is referred to as MARL-Exo in the human experiment. The analysis covers 31 muscle actuators per leg, representing 15 anatomical muscles of the hip, knee, and ankle. They were grouped by joint and direction into six functional groups (hip extension and flexion, knee flexion and extension, ankle plantar flexion and dorsiflexion) and by joint into three joint groups (hip, knee, and ankle, each summing both directions). Group membership is listed in Supplementary Section II (Table S2). Biarticular muscles belong to groups at more than one joint. For each condition we computed muscle-contributed joint torques at the lower limb joints; cycle-averaged muscle activation; and metabolic power per muscle (Watts), aggregated by functional group and summed over all analyzed muscles (W and W/kg), reported per leg as the left–right average. Group activation is the unweighted mean over the member actuators, whereas group metabolic power is their sum. Each condition was evaluated in one deterministic episode at the default body and gait parameters with about 18 complete gait cycles. All quantities are phase-normalized to the gait cycle (0–100%), reported bilaterally (left–right combined), and shown as the cycle mean with a ±1 SD band across cycles. The analyzed muscle set is sagittal-plane only, so absolute metabolic values underestimate whole-body cost. Because the same muscle set is used throughout, the conditions can be compared consistently on this partial-muscle total. III. H UMAN E XPERIMENT A. Hip Exoskeleton Hardware and Deployment The trained MARL-Exo control policy was deployed to a custom bilateral hip exoskeleton, whose mechanical design, size adjustments, and full specifications are given in [19]; only the key elements needed are repeated here. The device weighs 5.94 kg and provides one actuated sagittal flexion/extension degree of freedom per side (MyActuator X825, 25 N·m peak) together with a passive abduction/adduction hinge for fitting and lateral compliance. Bilateral hip angle was taken as the pelvis–femur difference measured by three wireless IMUs (LPMS-B2, LP-Research, Tokyo, Japan) mounted on the pelvis and both thighs and zeroed in a standing posture before each trial, and hip angular velocity as the corresponding gyroscope difference about the flexion axis. The trained MARL-Exo policy ran on an onboard Raspberry Pi 5 and reproduced the simulated observation and action interfaces exactly. At each 50 Hz control step the actor consumed the 18dimensional observation defined in Section II-B and emitted a normalized torque in [−1, 1] per hip, which was scaled by the preset per-hip torque limit and transmitted to the actuators over the CAN bus. A second-order Butterworth low-pass filter with a 3 Hz cutoff, designed for the 50 Hz loop rate, was applied online to the measured hip angle and angular velocity, matching the observation filter used in training. Trials were run under a software torque limit and an emergency stop, with an operator supervising at all times.

6

B. Human Validation Framework Eight healthy adults (5 male, 3 female; age 27.0 ± 7.8 yr, range 18–44; mass 70.4 ± 15.0 kg; height 1.73 ± 0.06 m) participated under IRB approval (Protocol No. 2305033091R003) with written informed consent. These participants and this session also supplied the data of the SMAT evaluation [19]. The standing-rest, no-exoskeleton, and passive recordings are therefore common to both studies and are reused here as the reference conditions (Table IV and Fig. 5); the MARL-Exo trials and every analysis involving them are new. Before data collection, each participant completed a treadmill familiarization session, walking for 5 min with the device worn in passive mode (zero commanded torque) and for 10 min under each of the two active controllers (MARLExo and SMAT), after which the fit was left unchanged for the remainder of the experiment. Data collection followed a symmetric protocol with randomized active-controller order, designed to separate the metabolic penalty of wearing the device from the benefit of its active assistance: standing rest → no-exoskeleton walking → passive walking → two active trials (MARL-Exo and SMAT, order randomized; MARLExo first in 3 of 8 participants) → passive walking → noexoskeleton walking → standing rest (Fig. 2). The passive condition retained the full device mass with commanded torque held at zero and the actuators left back-drivable, isolating the mechanical burden of the hardware from the assistance it delivers. Both active trials used identical hardware and differed only in the deployed policy. MARL-Exo is the controller referred to as Ctrl-B in [19], where it was run in the same session but not analyzed. MARL-Exo was commanded at a 12 N·m torque limit. The SMAT policy saturates its normalized output near 0.8 rather than unity [19] and was therefore commanded at 15 N·m, so that its realized peak (0.8 × 15 ≈ 12 N·m) matched that of MARL-Exo; both controllers thus operated at a common ∼12 N·m level of assistance. Metabolic and kinematic data were recorded identically in both active trials. Placing the no-exoskeleton, passive, and standing-rest conditions both before and after the active block helps assess fatigue and drift over the session, and the two repeats of each were averaged before analysis; averaging reduces, but does not eliminate, the influence of nonlinear drift or carryover between active trials. Every walking trial lasted approximately 6 min on a treadmill at 1.25 m/s, and expired gases were sampled breath-by-breath with a portable indirect calorimetry system (COSMED K5, COSMED, Italy). Between conditions, participants rested seated for at least 5 min. Metabolic rate was computed from breath-by-breath oxygen uptake V̇O2 and carbon-dioxide output V̇CO2 (mL/min) with the Brockway relation [29]. The respiratory exchange ratio (RER, V̇CO2 /V̇O2 ) was checked in every walking trial, with values below 1 confirming predominantly aerobic metabolism. Each condition was represented by the mean of the last 2 min of its trial, the two repeats of the No-Exo, Passive, and rest conditions were averaged, and subtracting the standing-rest value gave the net rate Ėnet . The contrasts of interest are the device penalty (Passive vs. No-Exo) and the assistance

benefit (MARL-Exo vs. Passive and No-Exo). C. Biomechanical and Statistical Analysis Hip angle, angular velocity, and commanded torque were logged at 50 Hz, and analysis used a steady-state window of the last 2 min of each trial. The logged velocity was refiltered offline with a zero-phase fourth-order Butterworth low-pass at 6 Hz, which adds no further phase delay; it cannot, however, remove any delay already present in a causally filtered logged signal (Section III-A). Cycles were segmented at peak hip flexion, since no ground-reaction-force sensors were used. Hip range of motion (ROM) was the difference between peak flexion and peak extension in each cycle. Cadence was computed from the mean cycle duration, and stride length was derived from it at the fixed belt speed, so the two are reciprocal by construction. Kinematic metrics were computed per cycle, then averaged within each subject and across the two hips. Mechanical output was summarized with the metrics of [30], also used in [19]: RMS torque (τRMS ), mean peak torque ext flex ), and the mean positive and negative power over , τMAX (τMAX the n samples of a cycle, MPP =

1 X τi q̇i , n

MNP =

τi q̇i >0

1 X τi q̇i , n

(7)

τi q̇i <0

with P = τ q̇ the estimated assistance power, computed from the commanded torque and the IMU-derived hip angular velocity converted to rad/s. The positive-power ratio Rpos = MPP/(MPP + |MNP|) is the positive estimated work divided by the total absolute estimated work over the same interval; it describes the timing of assistance relative to hip motion and is not a measure of metabolic efficiency. These were averaged like the kinematic metrics. Percentage changes in net metabolic rate were computed per subject and then averaged, and comparisons used twotailed paired t-tests (n = 8) after Shapiro–Wilk screening of the paired differences, Bonferroni-corrected within each outcome family at α = 0.05. The metabolic family has three planned comparisons (Passive vs. No-Exo, MARL-Exo vs. Passive, MARL-Exo vs. No-Exo), and the kinematic family has six (MARL-Exo vs. No-Exo for hip ROM, peak flexion, peak extension, cadence, and stride length, and Passive vs. No-Exo for hip ROM). Confidence intervals are unadjusted paired 95 % intervals. The participant was the inferential unit throughout; gait cycles were averaged within participants before testing. IV. R ESULTS A. Simulation Evaluation Results 1) Hip joint kinematics and kinetics: In the simulation evaluation, both active controllers (PowerExo and EnergyExo) reduced peak hip extension relative to NoExo, with PowerExo producing the larger reduction, and slightly deepened peak hip flexion (most under PowerExo), consistent with the device delivering a flexion assist during swing initiation (Fig. 3). Overlaying the biological reference (Fig. 3) shows qualitative

7

Standing Rest 1

No-Exo 1

Passive 1

MARL-Exo / SMAT

MARL-Exo / SMAT

Passive 2

No-Exo 2

Standing Rest 2

Hip angle (deg)

20 0 20

Assist torque Muscle torque (N·m) (N·m/kg)

0.5 0.0 0.5

Assist power (W)

Fig. 2. Experimental protocol. Each participant completed standing rest, no-exoskeleton walking (No-Exo), and passive walking (Passive), then the two active controllers (MARL-Exo and SMAT) in randomized order, followed by the same reference conditions in reverse order. Each walking trial lasted approximately 6 min on a treadmill at 1.25 m/s, with metabolic rate measured by indirect calorimetry throughout.

50 25 0

15 0 15

0

20

NoExo

EnergyExo

40

60

Gait Cycle (%)

PowerExo

80

100

Experiment

Fig. 3. Hip joint angles, muscle-contributed joint moment (mass-normalized), assistance torque, and assistance power over the gait cycle for the simulated conditions (PowerExo: Metabolic + Power; EnergyExo: Metabolic) and an independent biological reference (Experiment, teal dashed, mean ± 1 SD) [31]. Muscle-contributed moments are not identical to inverse-dynamics net moments.

agreement in hip kinematics: the simulated hip trajectory and range follow the biological curve. At the hip, both controllers reduced the peak flexion torque, with EnergyExo producing the larger reduction (Fig. 3); hip-torque range was reduced by both. Plotted in mass-normalized units, the simulated joint moments agree with the biological reference in overall shape and timing at the hip joint: both show the hip extensor-toflexor transition through stance. Fig. 3 also shows the gaitcycle-averaged assistance torque and power. Compared to the PowerExo assistance, the EnergyExo assistance provided slightly higher τRMS (13.62 vs. 13.12 N·m) but slightly lower MPP (19.98 vs. 20.64 W) with τmax = 18 N·m, both computed from the phase-averaged assistance curve in the figure. 2) Muscle activation and metabolic energy by functional group: Total cycle-averaged metabolic power summed over the analyzed muscles fell under both controllers, with En-

ergyExo achieving the larger reduction (−13.8%, 1.830 → 1.578 W/kg per leg) and PowerExo an intermediate reduction (−8.8%, → 1.670 W/kg per leg) (Table III). Summed by joint, the savings were concentrated proximally: metabolic power of the hip muscles fell by 29.8% under EnergyExo and 16.0% under PowerExo, that of the knee muscles by 12.8% and 11.9%, and that of the ankle muscles by only 3.9% and 2.5%. The two controllers therefore differed mainly at the hip, where EnergyExo used 4.2 W less than PowerExo, whereas their knee and ankle totals differed by less than 1.3 W. Because biarticular muscles are counted at every joint they span, the joint rows overlap, and each controller’s hip and knee savings together exceed its total reduction. Within each joint, the savings were unevenly distributed between directions. At the hip, both directions contributed, with larger reductions in Hip Extension (EnergyExo −35.4%, PowerExo −19.2%) than in Hip Flexion (−24.0%, −12.6%). At the knee, the reduction came almost entirely from Knee Extension (−36.2%, −26.5%), while Knee Flexion changed little (−4.3%, −6.6%). At the ankle, the plantar-flexor group was preserved near baseline (−1.7%, +0.6%), and the small ankle reduction came mainly from the dorsiflexor (−12.7%, −14.9%). Both controllers thus secured broad proximal savings while leaving ankle plantar-flexor energy essentially unchanged. The activation curves (Fig. 4, top row) show that both controllers reduce hip-extensor and knee-extensor activation while hip-flexor activation rises and falls during the gait cycle. The energy curves (Fig. 4, bottom row) show that all three groups lose metabolic energy under both controllers, with EnergyExo generally below PowerExo. The hip flexors thus show an activation–energy divergence: group mean activation rises while summed group energy falls (−24.0%, −12.6%; Table III). Because activation is an unweighted group mean, whereas metabolic power is a sum weighted by muscle mass (Eq. 5; see also [10]), this divergence may partly reflect aggregation and redistribution among member muscles. B. Metabolic Cost of Walking from Experiment Net metabolic rate across the three conditions is summarized in Fig. 5 and Table IV. The RER, averaged over the same final 2 min of each trial, remained below unity in all 40 analyzed walking trials (five per participant: two No-Exo, two Passive, and one MARL-Exo; overall 0.838 ± 0.032, range 0.785–0.916; No-Exo: 0.831 ± 0.035, Passive: 0.839 ± 0.030, MARL-Exo: 0.847 ± 0.029), so the Brockway relation was

8

TABLE III C YCLE - AVERAGED METABOLIC POWER BY FUNCTIONAL GROUP ( SUM ACROSS MEMBER MUSCLES ) AND TOTAL ACROSS ALL INVOLVED MUSCLE ACTUATORS , ALL VALUES PER LEG ( LEFT– RIGHT AVERAGED ; A PARTIAL - MUSCLE SUM , NOT WHOLE - BODY METABOLISM ). P OWER E XO : M ETABOLIC + P OWER REWARD ; E NERGY E XO : M ETABOLIC REWARD ( DEPLOYED ). ∆ IN WATTS ; PERCENTAGES SIGNED VS . N O E XO. G ROUPS OVERLAP THROUGH BIARTICULAR MUSCLES , SO THE SIX GROUP TOTALS EXCEED THE TOTAL ; THE TOTAL ROW IS THE TRUE NON - OVERLAPPING SUM . T HE H IP, K NEE , AND A NKLE ROWS SUM BOTH DIRECTIONS AT EACH JOINT ( NO MUSCLE BELONGS TO BOTH DIRECTIONS OF THE SAME JOINT ); BIARTICULAR MUSCLES STILL APPEAR IN MORE THAN ONE JOINT ROW. G ROUP MEMBERSHIP IS GIVEN IN S UPPLEMENTARY TABLE S2. W/ KG IS THE PER - LEG POWER DIVIDED BY BODY MASS (72.9 KG ).

Muscle group

NoExo W

PowerExo W

EnergyExo W

NoExo W/kg

PowerExo−NoExo ∆ (%)

EnergyExo−NoExo ∆ (%)

Hip Extension Hip Flexion Knee Flexion Knee Extension Ankle PlantarFlex Ankle DorsiFlex

15.47 14.84 56.07 20.32 69.60 17.31

12.50 12.97 52.39 14.94 70.02 14.73

9.99 11.27 53.64 12.97 68.40 15.12

0.212 0.204 0.769 0.279 0.955 0.237

−2.97 (−19.2%) −1.87 (−12.6%) −3.68 (−6.6%) −5.38 (−26.5%) +0.42 (+0.6%) −2.58 (−14.9%)

−5.48 (−35.4%) −3.57 (−24.0%) −2.43 (−4.3%) −7.35 (−36.2%) −1.20 (−1.7%) −2.19 (−12.7%)

Hip (Ext. + Flex.) Knee (Flex. + Ext.) Ankle (PF + DF)

30.30 76.40 86.91

25.47 67.34 84.75

21.26 66.61 83.52

0.416 1.048 1.192

−4.84 (−16.0%) −9.06 (−11.9%) −2.16 (−2.5%)

−9.04 (−29.8%) −9.79 (−12.8%) −3.39 (−3.9%)

TOTAL

133.43

121.75

115.01

1.830

−11.68 (−8.8%)

−18.42 (−13.8%)

Hip Extension

Energy rate (W) (group summed)

Activation (group mean)

0.2

Hip Flexion

Knee Extension

0.4

0.2

0.1

0.2

0.1

0.0

0.0 60

0.0

20

30

40

0

0

40

0

20

40

60

Gait Cycle (%)

80

100

80

0 0

NoExo

20

40

60

80

100

Gait Cycle (%) EnergyExo PowerExo

0

20

40

60

Gait Cycle (%)

80

100

Fig. 4. Muscle activation (top row) and metabolic energy rate (bottom row) over the gait cycle for the three functional groups where the controllers produce their largest reductions: Hip Extension, Hip Flexion, and Knee Extension (group mean activation; group summed energy; mean ± 1 SD). Hip-flexor activation rises during stance phase but falls during swing phase while hip-flexor energy mostly falls under both controllers.

applied throughout within its aerobic range. The two repeats of the no-exoskeleton and passive conditions differed by 6.9 ± 5.2 % and 3.5 ± 2.7 % of gross metabolic rate respectively, with no consistent direction across subjects, so we found no evidence of a systematic trend over the session; averaging the repeats reduces, but does not remove, this variability. Carrying the unpowered device raised net metabolic rate by

13.1 ± 4.9 % above unassisted walking (t7 = 7.80, corrected p < 0.001), a penalty consistent with the added mass and the constraint the structure imposes, although the present data do not separate these causes. This penalized condition, rather than unassisted walking, isolates the contribution of the assistance itself. Against that baseline, active MARL-Exo assistance lowered net metabolic rate by 18.3 ± 6.6 %, from 3.514 ± 0.414 to 2.888 ± 0.538 W/kg (t7 = −9.22, corrected p < 0.001). The effect was present in every participant, with individual

reductions spanning 7.2 % to 27.7 %, so the assistance more than recovered the cost of wearing the hardware. Relative to walking without the device, six of the eight participants walked at a lower net cost with the assisted device than without any device, with individual reductions of up to 24.2 % and above 20 % in two participants; in these six, the assistance recovered between 1.1 and 7.5 times the metabolic penalty of wearing the hardware. Across the cohort the mean reduction was 7.4 ± 10.0 % (paired difference −0.221 W/kg, unadjusted 95 % CI [−0.474, +0.033], t7 = −2.05, corrected p = 0.24), with the interval reflecting the between-subject spread in an eight-participant cohort. C. Hip Biomechanics and Assistance Power Delivery Fig. 6 compares hip kinematics and delivered assistance power across the No-Exo, Passive, and MARL-Exo conditions. Hip kinematics were derived from the three body-mounted IMUs, whose standing-posture zero offsets each subject’s

9

TABLE IV P ER - SUBJECT NET METABOLIC RATE AND REDUCTIONS UNDER MARL-E XO AND SMAT (n = 8). Net metabolic rate (W/kg)

MARL-Exo reduction (%)

SMAT reduction (%)

Subject

No-Exo

Passive

MARL-Exo

SMAT

vs Passive

vs No-Exo

vs Passive

vs No-Exo

S1 S2 S3 S4 S5 S6 S7 S8

3.008 3.575 2.941 2.794 3.734 2.876 2.852 3.088

3.326 4.081 3.051 3.243 4.222 3.301 3.444 3.445

2.405 3.376 2.228 2.747 3.916 2.678 2.884 2.870

2.738 3.221 2.368 2.711 3.571 2.458 2.802 2.744

27.7 17.3 27.0 15.3 7.2 18.9 16.3 16.7

20.1 5.6 24.2 1.7 −4.9 6.9 −1.1 7.1

17.7 21.1 22.4 16.4 15.4 25.5 18.6 20.3

9.0 9.9 19.5 3.0 4.4 14.5 1.8 11.2

3.11 ± 0.35

3.51 ± 0.41

2.89 ± 0.54

2.83 ± 0.39

18.3 ± 6.6

7.4 ± 10.0

19.7 ± 3.3

9.1 ± 6.0

Mean ± SD

Reduction = (Ėref − Ėactive )/Ėref , positive when the active condition costs less, computed per subject and then averaged. The SMAT data are from [19].

n=8

Net Metabolic Rate (W kg 1)

5.0 4.5 4.0 3.5 3.0 2.5 2.0

No Exo No Exo

Passive Passive

MARL-Exo MARL-Exo

Fig. 5. Net metabolic rate (W kg−1 ) across conditions (n = 8). Half-violin distributions, box plots, and individual subjects (connected lines) for NoExo, Passive, and MARL-Exo. The No-Exo and Passive data are shared with [19]. Brackets show the group-mean percentage change between conditions (+13.1 %, −18.3 %, −7.4 %). Paired t-test with Bonferroni correction: ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001; n.s. p ≥ 0.05. All eight subjects reduced net cost under assistance relative to the passive condition.

absolute joint angles. Range of motion is taken as the primary kinematic outcome, while the individual flexion and extension peaks are reported as descriptive quantities. The paired differences satisfied the Shapiro–Wilk normality check for every variable except peak flexion (W = 0.809, p = 0.036), for which a Wilcoxon signed-rank test is reported alongside the parametric result. Active assistance markedly enlarged hip range of motion, from 44.0 ± 4.3◦ without the device to 60.2 ± 6.6◦ under MARL-Exo, an increase of 36.8 % (t7 = 5.97, corrected p = 0.003), with the passive condition falling between the two (48.7 ± 4.0◦ , +10.6 %, t7 = 3.93, corrected p = 0.034). The enlargement was carried almost entirely by the flexion phase: peak flexion rose by 20.1◦ , from 33.6 ± 8.2◦ to 53.6 ± 6.9◦ , a 59.8 % increase (t7 = 5.38, corrected p = 0.006), whereas peak extension was statistically unchanged (−10.5 ± 6.9◦ to −6.6 ± 8.4◦ , t7 = 1.19, corrected p > 0.99). Both range of motion and peak flexion increased in all eight participants without exception. For peak flexion, whose paired differences were not normally distributed, the nonparametric test agreed in direction and remained significant after correction (W = 0,

exact p = 0.008, corrected p = 0.047); with n = 8 the exact test cannot resolve below p = 0.008, so this value reflects the sample size rather than the magnitude of the effect, which the unanimous direction and the 20.1◦ shift characterize more directly. The spatiotemporal structure of the gait shifted far less than its amplitude. Cadence rose from 110.3 ± 8.2 to 115.7 ± 6.3 steps/min under assistance, a 4.8 % increase that did not survive correction (t7 = 2.92, corrected p = 0.13); at the fixed belt speed this is equivalent to a shortening of stride length from 1.366±0.098 to 1.300±0.072 m, the two being reciprocal by construction. Assistance thus amplified hip excursion, while the changes in cadence and stride length were smaller and not significant after correction. The exoskeleton delivered torque in both directions of the stride, with a flexion-assist peak in late swing and an extension-assist phase through mid-stance. Table V reports the per-subject mechanical output. Mean per-cycle peak torque was 11.79±0.39 N·m in flexion and −11.81±0.42 N·m in extension, magnitudes that did not differ significantly (t7 = 1.07, p = 0.32), indicating that the policy saturated symmetrically against the 12 N·m commanded limit in both directions; RMS torque was 9.45 ± 0.45 N·m. Mean positive estimated power was 19.19 ± 2.74 W while mean negative estimated power remained near zero (−0.30±0.14 W), giving a positive-power ratio Rpos of 0.985 ± 0.006. Because the deployed policy was trained without the explicit power term, this predominantly positive power emerged from the metabolic and regularization terms rather than being directly rewarded. The spread across subjects was small in all four metrics, indicating that the single deployed policy produced consistent mechanical assistance without subject-specific tuning. V. D ISCUSSION A central design choice of this work is to include a physiologically detailed metabolic model in the exoskeleton reward. In simulation, with the metabolic term retained in both controllers, removing explicit positive-power shaping lowered modeled metabolic power by a further 5 percentage points (−13.8 % vs. −8.8 %) and changed where the savings fell. Further training of the Metabolic + Power controller over a comparable number of steps did not improve it, which points to the reward change, not the additional training, as the source

10

40 20 0 0

20

40

60

Gait cycle (%)

80

+36.8%

+23.7%

50 25 0 No Exo

Passive

110

200 0 200

Hip Exo Torque

100

10

Stride Length Stride length (m)

Hip angle (deg)

+10.6%

120

100

Peak Hip Flexion and Extension 75

= +6.1% = +4.8%

130

1.4 1.2

MARL-Exo

No Exo

No Exo

Passive

Passive

0 10

= -6.0% = -4.8%

1.6

Power (W)

20

Hip Angular Velocity

Torque (N·m)

Cadence (steps/min)

Hip angle (deg)

Cadence 140

Ang. velocity (deg/s)

Hip Angle

60

MARL-Exo

MARL-Exo

Hip Exo Power

60 40 20 0 0

20

40

60

Gait cycle (%)

80

100

Fig. 6. Treadmill walking at 1.25 m/s under No-Exo (black), Passive (blue), and MARL-Exo (green) (n = 8; per-subject means of both hips). Left: gait-cycle hip angle (flexion positive) with peak flexion and extension; labels give the change in ROM. Middle: cadence and stride length. Right: hip angular velocity together with the exoskeleton torque and mechanical power delivered by MARL-Exo. Bands and error bars show mean ±1 SD across subjects. The No-Exo and Passive data are shared with [19].

TABLE V P ER - SUBJECT EXOSKELETON MECHANICAL OUTPUT, MARL-E XO CONDITION (n = 8, BOTH HIPS AVERAGED ). Subject S1 S2 S3 S4 S5 S6 S7 S8 Mean ± SD

τRMS (N·m)

flex τMAX (N·m)

ext τMAX (N·m)

MPP (W)

MNP (W)

Rpos

9.62 9.76 9.36 10.10 9.42 9.68 8.75 8.89

11.94 11.97 11.92 11.99 11.97 11.80 10.84 11.86

−11.97 −11.98 −11.94 −11.99 −11.97 −11.96 −10.78 −11.88

24.04 21.02 16.79 21.53 18.40 18.26 16.16 17.29

−0.54 −0.24 −0.20 −0.33 −0.39 −0.05 −0.29 −0.33

0.978 0.989 0.988 0.985 0.979 0.997 0.982 0.981

9.45 ± 0.45

11.79 ± 0.39

−11.81 ± 0.42

19.19 ± 2.74

−0.30 ± 0.14

0.985 ± 0.006

flex ext τRMS : RMS torque; τMAX , τMAX : mean per-cycle peak torque in the flexion-

and extension-assist directions; MPP/MNP: mean positive/negative power (Eq. 7); Rpos = MPP/(MPP + |MNP|): positive-power ratio. All metrics were computed per gait cycle and averaged across cycles within each subject, with the two hips averaged.

of the gain. Power shaping thus steers the learned strategy away from energetically cheaper solutions that the metabolic term alone finds. The divergence of hip-flexor activation and energy noted above suggests a mechanism: an energeticsbased reward can favor strategies that an activation-based reward would penalize. Because energy models disagree with one another [32], the simulated ranking of controllers and the savings by muscle group are specific to the Uchida model used here and may differ under alternatives such as that of Bhargava et al. [33]. The reward otherwise follows trajectory-optimization practice, where energy is one term of a composite cost because energy alone can predict unrealistic movement and its rate expressions are non-smooth [34], [35]. On hardware, the central result is that assistance beat the unpowered device in every one of the eight participants (−18.3 ± 6.6 %, p < 0.001), with reductions from 7.2 % to 27.7 %, even though the policy was neither retrained nor tuned for any of them. Because the passive condition carries the same hardware, this contrast isolates what the controller itself contributes, and we treat it as the primary outcome. On a common scale, active assistance removed 0.63 W/kg relative to the passive device, about 1.5 times the 0.41 W/kg

that wearing the unpowered device added. The net result relative to walking without any device therefore depends on the hardware as much as on the controller. Six of eight participants walked at a lower net cost than without the device, two by more than 20 %, and the cohort mean was −7.4 ± 10.0 %. The savings relative to no-device walking reported for other simulation-trained hip controllers, 24.3 % [7] and 15.2 % [9], combine the benefit of each controller with the penalty of its particular device. Isolated from the hardware by the passive condition, the benefit of MARL-Exo (18.3 %) exceeds the nodevice saving reported in [9], and it was achieved under less favorable hardware conditions than either study. Our device weighs 5.94 kg, about twice the 3.2 kg [7] and 2.9 kg [9] of those devices, and it was operated at a 12 N·m limit, lower than the peak torque reported in [7]. The smaller no-device saving is therefore consistent with the 13.1 % penalty of our hardware rather than with an inefficient controller. Accordingly, a lighter device and a higher torque limit would be expected to shift the no-device contrast in favor of assistance. The delivered assistance is consistent with this outcome. Torque was applied in both directions of the stride, with matching peaks in flexion and extension, and almost all of the estimated assistance power acted with the wearer’s hip motion (Table V), the mechanical signature of well-timed assistance. Hip range of motion grew markedly, almost entirely through flexion, while cadence and stride length did not change significantly after correction. This pattern is consistent with an assistance profile that helps swing initiation, and with the simulated reduction of hip-flexor and hip-extensor energy. The larger excursion at the same belt speed is an altered gait response, so part of the metabolic benefit may come from the changed gait as well as from reduced muscle workload. Because the two deployed controllers (SMAT and MARLExo) were tested in the same participants and session at a matched ∼12 N·m peak, they can be compared directly. Relative to the passive device, SMAT lowered net metabolic

11

rate by 19.7 % and MARL-Exo by 18.3 %, and per subject MARL-Exo was on average 0.061 W/kg higher in net cost (unadjusted 95 % CI [−0.116, +0.239] W/kg; t7 = 0.82, uncorrected p = 0.44). Both controllers thus reduced cost relative to the passive device by similar amounts, and this exploratory comparison detected no difference between them. The mechanical pattern was also nearly identical: flexion assistance in late swing and extension assistance through stance, the same positive-power ratio (0.985 vs. 0.984), similar mean positive power (19.2 vs. 18.1 W), and similar gains in hip range of motion (+36.8 % vs. +40.8 %). MARLExo had a slightly higher RMS torque (9.45 vs. 8.84 N·m) with a slightly lower mean peak (11.79 vs. 12.39 N·m), which suggests a flatter profile held near the limit for longer. The similar metabolic benefit may partly reflect the shared hardware limit. Both controllers drove the same device, saturated against the same ∼12 N·m cap, and delivered similar positive mechanical power, and in a rigid hip exoskeleton the metabolic saving is expected to scale with the positive work returned to the wearer. Under a common torque ceiling, two policies that both learn to time assistance with hip motion may therefore converge on similar benefit, and the flatter, higher-RMS profile of MARL-Exo did not lower cost further, as a broad optimum or a ceiling would predict. Testing both controllers at higher torque limits, on lighter hardware, and at more than one speed would show where their differences in training become visible. Several sim-to-real details deserve mention. The observation was built from IMU-derived hip angle and velocity, so it did not depend on motor encoders, and it was low-pass filtered online at 3 Hz, as in training. The policy also ran at 50 Hz on a Raspberry Pi 5, and the observation latency modeled in simulation (40 ms) is what allows such a modest processor to be used. Soft-tissue compliance and strap looseness still make IMU-derived angles an approximation of true hip motion, and the phase alignment of assistance might be improved by more accurate thigh and pelvis sensing [36]. Finally, hardware was commanded at a 12 N·m limit, lower than the ±18 N·m simulated peak, to keep the device comfortable and to avoid wobble and excessive relative movement between the exoskeleton and the wearer, which the rigidly constrained simulation does not capture. The deployed assistance is therefore weaker than the one that produced the simulated savings, and higher torque with a better-fitting interface may yield larger benefit. The scope of the study defines the next steps. The human study evaluated eight healthy adults walking on a treadmill at a single speed after 10 min of familiarization with each controller, with reference conditions shared with our SMAT study. Larger cohorts would resolve the contrast with unassisted walking (corrected p = 0.24 here), longer or multiday exposure would show whether the benefit grows as wearers adapt to the assistance, and testing across speeds, slopes, and gait-impaired populations would exercise the body and gait randomization already present in training. Adding EMG and full lower-limb motion capture to the hip kinematics recorded here would allow the simulated redistribution of muscle effort, and the gait changes that accompany assistance, to

be checked in humans. Finally, a lighter device with a betterfitting interface would reduce the measured wearing penalty and allow the higher torque used in training to be applied without relative motion at the cuffs. VI. C ONCLUSION We presented MARL-Exo, a two-stage multi-agent reinforcement learning framework in which a full-body musculoskeletal human policy and a hip-exoskeleton policy are coadapted in simulation, with a physiologically based muscleenergetics term in the exoskeleton reward. Co-adaptation remained stable with a 304-actuator human model and randomized anatomy, gait, and speed, and the choice of reward shaped the learned strategy: with the metabolic term retained, omitting explicit positive-power shaping lowered modeled muscle metabolic power by 13.8 % rather than 8.8 %, with the additional savings concentrated at the hip and knee. The learned policy was deployed on a hip exoskeleton in eight healthy adults, without per-user tuning, and it reduced net metabolic rate by 18.3 ± 6.6 % relative to the passive device, enlarged hip range of motion by 36.8 %, and delivered almost entirely positive estimated assistance power. Six of eight participants walked below their no-device cost, by up to 24.2 %, with a cohort-mean reduction of 7.4 ± 10.0 %. The benefit was close to that of our earlier SMAT controller at the same torque limit, despite a different human model, curriculum, and reward. These results show that physiologically informed coadaptive training can produce an exoskeleton controller that transfers from simulation to human users without per-user tuning, and that candidate reward designs can be compared in simulation before human testing. Larger and more diverse cohorts, lighter hardware, and evaluation across speeds, slopes, and gait-impaired populations are the next steps toward population-level assistance. ACKNOWLEDGMENT This work was partially supported by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funded Rehabilitation Engineering Research Center Grant 90REGE0025-01-00 and NSF Award #2524089. R EFERENCES [1] A. Rodrı́guez-Fernández et al., “Systematic review on wearable lowerlimb exoskeletons for gait training in neuromuscular impairments,” Journal of NeuroEngineering and Rehabilitation, vol. 18, no. 1, p. 22, 2021. [2] R. Baud et al., “Review of control strategies for lower-limb exoskeletons to assist gait,” Journal of NeuroEngineering and Rehabilitation, vol. 18, no. 1, p. 119, 2021. [3] K. L. Poggensee and S. H. Collins, “How adaptation, training, and customization contribute to benefits from exoskeleton assistance,” Science Robotics, vol. 6, no. 58, p. eabf1078, 2021. [4] J. Zhang et al., “Human-in-the-loop optimization of exoskeleton assistance during walking,” Science, vol. 356, no. 6344, pp. 1280–1284, 2017. [5] Y. Ding et al., “Human-in-the-loop optimization of hip assistance with a soft exosuit during walking,” Science Robotics, vol. 3, no. 15, p. eaar5438, 2018. [6] S. Luo et al., “Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning,” Journal of NeuroEngineering and Rehabilitation, vol. 20, no. 1, p. 34, 2023.

12

[7] S. Luo et al., “Experiment-free exoskeleton assistance via learning in simulation,” Nature, vol. 630, no. 8016, pp. 353–359, 2024. [8] I. Park et al., “Learning hip exoskeleton control policy via predictive neuromusculoskeletal simulation,” arXiv preprint arXiv:2603.04166, 2026. [9] H. Barati et al., “End-to-end policy learning for hip exoskeleton via reinforcement learning and reflex-based musculoskeletal simulation,” IEEE Robotics and Automation Letters, 2026. [10] G. Leem et al., “Exo-Plore: Exploring exoskeleton control space through human-aligned simulation,” International Conference on Learning Representations (ICLR) Proceedings., 2026. [11] Y. Yuan et al., “Gait asymmetry from unilateral weakness and improvement with ankle assistance: A reinforcement learning based simulation study,” Proceedings of 2026 11th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), 2026. [12] N. Ratnakumar et al., “Reinforcement-learning-based assistance reduces squat effort with a modular hip–knee exoskeleton,” Proceedings of 2026 11th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), 2026. [13] N. Ratnakumar et al., “Predicting sit-to-stand motions with a deep reinforcement learning based controller under idealized exoskeleton assistance,” Multibody System Dynamics, vol. 66, no. 4, pp. 835–852, 2026. [14] M. Simos et al., “Reinforcement learning-based motion imitation for physiologically plausible musculoskeletal motor control,” arXiv preprint arXiv:2503.14637, 2025. [15] Y. Bengio et al., “Curriculum learning,” in Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 41–48. [16] A. S. Chiappa et al., “Acquiring musculoskeletal skills with curriculumbased reinforcement learning,” Neuron, vol. 112, no. 23, pp. 3969–3983, 2024. [17] V. Caggiano et al., “MyoSuite–A contact-rich simulation suite for musculoskeletal motor control,” arXiv preprint arXiv:2205.13600, 2022. [18] Y. Yuan et al., “SMAT: Staged multi-agent training for co-adaptive exoskeleton control,” Proceedings of 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems, arXiv preprint arXiv:2603.07618, 2026. [19] Y. Yuan et al., “Staged multi-agent training for human–exoskeleton coadaptation: Simulation-based learning and real-world metabolic and biomechanical validation,” Manuscript under review, arXiv preprint arXiv:2608.00715, 2026. [20] J. Lee et al., “DART: Dynamic animation and robotics toolkit,” The Journal of Open Source Software, vol. 3, no. 22, p. 500, 2018. [21] J. Park et al., “Generative GaitNet,” in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1–9. [22] J. Park et al., “Bidirectional GaitNet: A bidirectional prediction model of human gait and anatomical conditions,” in ACM SIGGRAPH 2023 Conference Proceedings, 2023, pp. 1–9. [23] X. Zhou and X. Chen, “Design and evaluation of torque compensation controllers for a lower extremity exoskeleton,” Journal of Biomechanical Engineering, vol. 143, no. 1, p. 011007, 2021. [24] C. K. Tan et al., “MyoAssist 0.1: MyoSuite for dexterity and agility in bionic humans,” in 2025 International Conference on Rehabilitation Robotics (ICORR). IEEE, 2025, pp. 437–442. [25] R. Lowe et al., “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in Neural Information Processing Systems, vol. 30, 2017. [26] J. Schulman et al., “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [27] B. R. Umberger et al., “A model of human muscle energy expenditure,” Computer Methods in Biomechanics and Biomedical Engineering, vol. 6, no. 2, pp. 99–111, 2003. [28] T. K. Uchida et al., “Stretching your energetic budget: how tendon compliance affects the metabolic cost of running,” PLOS ONE, vol. 11, no. 3, p. e0150378, 2016. [29] J. Brockway, “Derivation of formulae used to calculate energy expenditure in man.” Human Nutrition. Clinical Nutrition, vol. 41, no. 6, pp. 463–471, 1987. [30] B. Lim et al., “Parametric delayed output feedback control for versatile human-exoskeleton interactions during walking and running,” IEEE Robotics and Automation Letters, vol. 8, no. 8, pp. 4497–4504, 2023. [31] J. Camargo et al., “A comprehensive, open-source dataset of lower limb biomechanics in multiple conditions of stairs, ramps, and level-ground ambulation and transitions,” Journal of Biomechanics, p. 110320, 2021. [32] R. H. Miller, “A comparison of muscle energy models for simulating human walking in three dimensions,” Journal of Biomechanics, vol. 47, no. 6, pp. 1373–1381, 2014.

[33] L. J. Bhargava et al., “A phenomenological model for estimating metabolic energy consumption in muscle contraction,” Journal of Biomechanics, vol. 37, no. 1, pp. 81–88, 2004. [34] M. Ackermann and A. J. van den Bogert, “Optimality principles for model-based prediction of human gait,” Journal of Biomechanics, vol. 43, no. 6, pp. 1055–1060, 2010. [35] A. D. Koelewijn et al., “A metabolic energy expenditure model with a continuous first derivative and its application to predictive simulations of gait,” Computer Methods in Biomechanics and Biomedical Engineering, vol. 21, no. 8, pp. 521–531, 2018. [36] T. Seel et al., “IMU-based joint angle measurement for gait analysis,” Sensors, vol. 14, no. 4, pp. 6891–6909, 2014.

13

Supplementary Material Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction Yifei Yuan, Jakob Wolf, Ghaith Androwis, and Xianlian Zhou

I. AUXILIARY T ERMS OF THE H UMAN R EWARD k The human reward contains several auxiliary terms k raux that regularize the learned gait. Three terms are used, with identical settings in the human baseline stage and in both co-adaptation runs (Metabolic and Metabolic + Power):

P

X

k raux = rlim + rτ + rclr .

(S1)

k

Their constants are listed in Table S1. A. Knee Joint-Limit Penalty Knee hyperextension is penalized quadratically once the knee angle crosses a soft limit placed a margin δ inside the joint’s lower (extension) limit: X
2 rlim = − klim max 0, (θjmin + δ) − θj , (S2) j∈{L,R}

where θj is the flexion angle of knee j (flexion positive). With the knee lower limit θ min = −0.1 rad and δ = 0.1 rad, the penalty is active for any hyperextension (θj < 0). The upper (flexion) limit is not penalized. B. Joint-Torque Smoothness Smooth joint actuation is encouraged with a bounded term that decreases as the desired joint torque changes more abruptly between control steps: 2 2 2 3 2 rτ = e−c ∥∇τ t ∥ + 0.5 e−c ∥∇ τ t ∥ + 0.2 e−c ∥∇ τ t ∥ , (S3) where τ t is the vector of SPD desired joint torques (N·m) at control step t, c = 10−3 , and ∇n denotes the n-th backward difference over consecutive control steps:

∇τ t = τ t − τ t−1 , ∇2 τ t = τ t − 2τ t−1 + τ t−2 ,

(S4)

3

∇ τ t = τ t − 3τ t−1 + 3τ t−2 − τ t−3 . The term lies in [0, 1.7] and is applied once four control steps of torque history are available. C. Foot-Clearance Penalty Foot scuffing is penalized when a reference point on the foot is below a clearance height while moving horizontally:

rclr = − kclr

X

min 0, yj − hc

2

vjxz ,

(S5)

j∈{L,R}

where yj is the height of a fixed point on foot j (0.15 m along the foot segment from its origin), vjxz is that point’s horizontal velocity, and hc = 0.03 m. The penalty is zero when the point is above hc or stationary on the ground, so it acts mainly during swing. For example, a point 1 cm below hc moving at 1 m/s gives rclr = −0.01.

14

TABLE S1 C ONSTANTS OF THE AUXILIARY HUMAN REWARD TERMS . Term

Symbol

Value

Description

rlim

klim θmin δ

100

penalty gain knee lower (extension) limit soft-limit margin

rτ

c

10−3 (N·m)−2

exponential scale

rclr

kclr hc

100 s m−3 0.03 m

penalty gain clearance height

−0.1 rad 0.1 rad

II. M USCLE G ROUPS U SED IN THE S IMULATION A NALYSIS The simulation analysis of the main text uses 31 muscle actuators per leg (line-of-action elements of the 304-actuator model), representing 15 anatomical muscles that act primarily in the sagittal plane. Each actuator is logged for the left and right legs, and all reported values are left–right averages (per leg). The actuators are assigned to six functional groups by joint and direction: hip extension (HE), hip flexion (HF), knee flexion (KF), knee extension (KE), ankle plantar flexion (PF), and ankle dorsiflexion (DF). Three joint groups combine both directions at each joint: hip (HE + HF), knee (KF + KE), and ankle (PF + DF). Table S2 lists the membership. No actuator belongs to both directions of the same joint, so each joint group is the disjoint union of its two direction groups. Biarticular muscles (biceps femoris long head, semimembranosus, semitendinosus, rectus femoris, and the gastrocnemius) belong to groups at two joints, so the direction groups and the joint groups overlap across joints and their sums exceed the total over all 31 actuators. Group activation is the unweighted mean over member actuators, and group metabolic power is their sum. TABLE S2 M EMBERSHIP OF THE ANALYZED MUSCLE ACTUATORS ( PER LEG ) IN THE FUNCTIONAL AND JOINT GROUPS . • MARKS MEMBERSHIP. T HE BOTTOM ROWS GIVE THE NUMBER OF ACTUATORS IN EACH GROUP. Hip Anatomical muscle

Actuators

HE

Biceps femoris, long head Biceps femoris, short head Gastrocnemius, lateral head Gastrocnemius, medial head Gluteus maximus Iliacus Psoas major Rectus femoris Semimembranosus Semitendinosus Soleus Tibialis anterior Vastus intermedius Vastus lateralis Vastus medialis

1 2 1 1 5 3 3 2 2 1 2 1 2 2 3

•

Actuators per direction group Actuators per joint group

31

9

HF

Knee KF KE

Ankle PF DF

• • • •

• •

• • • • • •

• • • • • • • •

8 17

8

9 17

4

1 5

Related documents

• Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction#1469044
• Prevalencia de obesidad, hipertensión arterial e hiperglucemia en choferes de ambulancia de áreas remotas y su relación con factores laborales#228390
• Hip External Rotation Physical Therapy Trial#1044618
• Agent-Based Computational Economics: Overview and Brief History#724523
• Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents#1457275