|
|
Learning stable and energy-efficient locomotion for quadruped robots remains challenging due to the long-horizon nature of gait optimization and the need for robust, high-frequency stabilization under disturbances. This work proposes a hierarchical reinforcement learning (HRL) framework that decomposes quadruped locomotion into complementary high-level and low-level control policies operating at different temporal scales. The high-level policy operates at a low frequency (2 Hz) and focuses on energy-efficient gait generation and smooth gait transitions by selecting locomotion primitives and reference motion parameters that adapt to task demands while minimizing energy consumption. The low-level policy runs at a higher control frequency (50 Hz) and is responsible for tracking high-level commands while ensuring dynamic stability, balance, and robustness to modeling uncertainties and external perturbations.
By separating long-horizon energy optimization from short-horizon stabilization, the proposed hierarchy enables efficient exploration of gait strategies while constraining execution to dynamically feasible behaviors. Both policies are trained in simulation using reinforcement learning with decoupled reward formulations tailored to their respective objectives. Experimental results in simulation demonstrate that the proposed HRL framework achieves reduced energy consumption, smoother gait transitions, and improved robustness compared to monolithic reinforcement learning baselines. These results highlight the effectiveness of hierarchical policy decomposition for scalable quadruped locomotion and provide a foundation for future sim-to-real transfer and deployment on real robotic platforms.
We propose a hierarchical reinforcement learning (HRL) framework for stable and energy-efficient quadruped locomotion that explicitly separates long-horizon gait optimization from short-horizon stabilization. The system decomposes control into a high-level (HL) gait generation policy and a low-level (LL) locomotion controller, operating at different temporal scales.
The HL policy operates in a compact, structured, and physically interpretable gait parameter space, and learns to modulate gait timing, inter-leg coordination, stance geometry, body posture, and foot trajectory parameters in response to commanded velocities and proprioceptive feedback. By reasoning over abstract gait parameters rather than joint-level actions, the HL policy focuses on smooth gait transitions and energy-efficient locomotion across a range of speeds.
The LL policy is trained separately using Proximal Policy Optimization (PPO) in Isaac Gym and is responsible for robust, dynamically consistent motion execution. It receives gait parameters from the HL policy and converts them into joint-space targets, which are subsequently tracked using a PD controller to generate joint torques. The LL policy is trained with domain randomization and temporal observation histories to improve robustness to modeling errors and external disturbances.
During HL training, the LL policy is frozen, allowing the HL to efficiently explore gait strategies while relying on the LL to ensure joint-level stability and balance. This hierarchical separation enables scalable learning, reduces training complexity, and facilitates transferability across operating conditions. The resulting system achieves stable, smooth, and energy-efficient locomotion on the AlienGo quadruped platform in simulation.
In the final stage, the high-level policy is fine-tuned to improve velocity tracking accuracy, gait consistency, and locomotion stability. This refinement helps reduce tracking error and prepares the hierarchical controller for more reliable evaluation and deployment.
All the below experiments were conducted by reproducing the Walk These Ways paper for the Aliengo robot (originally implemented on the Go1 platform), incorporating numerous parameter modifications and enhancements over the original implementation. The evaluations were performed at xvel = 1.0 m/s, yvel = 0.0 m/s, yawvel = 0.0 rad/s, body height = 0.0, step frequency = 3.0 Hz, foot swing height = 0.08 m, pitch = 0.0, roll = 0.0 ↓
Pacing Gait ([0, 0, 0.5])
Trotting Gait ([0.5, 0, 0])
Pronking Gait ([0, 0, 0])
Bounding Gait ([0, 0.5, 0])
Galloping Gait ([0.25, 0, 0])
All experiments were conducted by reproducing the methodology proposed in Adaptive Energy Regularization for Autonomous Gait Transition and Energy-Efficient Quadruped Locomotion, adapted to the Aliengo robot platform. The work demonstrates autonomous gait discovery through energy-centric reward shaping, it explicitly zeros out high-level gait modulation commands to isolate and evaluate energy-efficient locomotion behavior under a constrained control setting. In the left experiment, the policy is evaluated at a constant command of vx = 1.0 m/s, vy = 0.0 m/s, and ψ̇ = 0.0 rad/s. In the right experiment, the forward velocity command is linearly ramped from 0 to 1.2 m/s over 350 simulation steps, while other gait‑related parameters (body height, step frequency, phase, offset, bound, duration, foot‑swing height, pitch, roll, and stance width) are held at zero, thereby emphasizing energy-aware locomotion with minimal handcrafted gait biases. ↓
The experiment below is conducted in simulation on flat terrain, where five quadruped robots execute different gait patterns under identical control and environmental conditions. For each commanded forward velocity vx∈[0,1.8] m/s (10 evenly spaced values), five quadruped instances execute distinct, fixed gait patterns and are simulated for 500 steps. The mean cost of transport E/(mgd) over the steady‑state portion of each rollout is then computed per gait. The left panel shows the resulting cost-velocity curves for purely forward motion (vy=0, ψ̇=0), while the right panel shows the same analysis when a lateral velocity component is added (vy=0.3 m/s, ψ̇=0). The results show a clear dependence of energy efficiency on gait choice, with pacing exhibiting the lowest cost of transport across the tested velocity range, followed in order by trotting, galloping, pronking, and bounding. ↓
(a) vx ∈ [0, 1.8] m/s, vy = 0.0 m/s, ψ̅ = 0 rad/s.
(b) vx ∈ [0, 1.8] m/s, vy = 0.3 m/s, ψ̅ = 0 rad/s.
The following experiment is conducted in simulation using the adaptive energy regularization (AER) baseline reproduced on the AlienGo quadruped. The robot is commanded to track a continuously varying forward velocity, during which the end-to-end AER policy autonomously adapts its foot–ground contact patterns without any explicit gait scheduling or predefined phase structure. The accompanying gait diagram illustrates the speed-dependent reorganization of stance and swing phases across the velocity ramp, reflecting how the learned policy modulates its locomotion strategy to maintain stable and energy-efficient motion.
Prior work on AER has shown that end-to-end reinforcement learning can give rise to trotting-like gaits with diagonal limb coordination, as demonstrated on the Unitree Go1 robot. When reproducing AER on AlienGo, we similarly observe the emergence of trotting-like behavior, characterized by clear diagonal synchronization between the front-left and rear-right limbs. However, the complementary diagonal pair (front-right and rear-left) exhibits weaker synchronization and mild phase asymmetries, resulting in a partially symmetric trot rather than a strictly periodic and balanced trotting gait. Such asymmetries are expected and are likely attributable to differences in robot morphology, mass distribution, and actuation characteristics between platforms. ↓
Gait switching under different command velocities. The policy is generated when energy coefficient was set to 0.75. As the command velocity increases, the policy shows automatic gait transition.
Comparison of mean Cost of Transport (CoT) on the Unitree Go1 across baseline methods—fixed gait controller (Walk These Ways), Adaptive Energy Regularization (AER, α = 1), and the proposed hierarchical RL approach, evaluated over a range of forward velocities.
Comparison of mean Cost of Transport (CoT) on the Unitree Aliengo for the proposed hierarchical RL policy across flat, uneven, and rough terrains.
Gait diagrams corresponding to different commanded forward velocities.
(a) Trapezoidal forward velocity tracking.
(b) Ramped forward velocity tracking.
Forward velocity tracking performance of the fine-tuned hierarchical locomotion policy on the Unitree AlienGo robot. (a) Tracking response under a trapezoidal velocity command, where the commanded velocity increases to approximately 1~m/s, remains constant, and then decreases. (b) Tracking response under a ramped velocity command from 0 to approximately 2~m/s. The solid blue curve denotes the measured forward velocity, the dashed orange curve denotes the commanded velocity, and the shaded region represents one standard deviation across evaluation rollouts.
Cost of Transport (CoT) of the proposed controller in the MuJoCo environment across varying forward velocities and terrain conditions, including flat, inclined, and rough terrains.
Velocity tracking performance of the proposed controller in the MuJoCo environment across commanded forward speeds ranging from 0 to 1.6 m/s.
Emergence of Pacing at Low Velocities (HRL) ↓