Quadruped Fall Recovery · Reinforcement Learning · Isaac Gym
Learning a proprioceptive recovery policy that transitions a fallen quadruped from arbitrary poses to a stable, locomotion-ready stand.
|
|
Autonomous fall recovery is a critical capability for legged robots operating outside controlled environments. A quadruped may fall because of slips, impacts, unstable terrain interaction, or failed locomotion transitions; without self-recovery, the robot requires manual reset and cannot continue the mission. This project trains a reinforcement-learning-based fall-recovery controller that maps proprioceptive observations to joint-level actions and learns to recover from fallen configurations to a stable standing posture.
The controller is trained in Isaac Gym using a stage-wise procedure. Initial training focuses on discovering large recovery motions, such as rolling and reorienting the base. Later stages refine the terminal behavior: lifting the body, placing the feet under the base, reducing residual velocity, and reaching a posture suitable for handoff to a locomotion policy. This staged curriculum is used because the task is not a single homogeneous motion; it consists of distinct subskills: reorientation, body lifting, foot placement, and stabilization.
The current project page documents the training pipeline, reward design, success criteria, checkpoint progression, qualitative videos, failure cases, and evaluation protocol for autonomous quadruped fall recovery.
The target behavior is not merely to rotate upright once. The policy must reach a stable terminal state from which a locomotion controller can safely take over.
The fall-recovery controller uses an asymmetric actor-critic architecture with a history-based adaptation module. The deployable policy consists of the adaptation module and actor network: observation history is encoded into an estimated privileged latent, which is concatenated with the current proprioceptive observation and passed to the actor. During training, a separate critic receives true privileged simulation information to improve value estimation.
The reward is designed to guide the policy through a sequence of recovery subskills: reorientation, body lifting, foot support formation, and terminal stabilization. A sparse success bonus alone is insufficient because the robot must first discover intermediate behaviors before reaching a stable standing posture.
The table reports the raw reward or penalty term before multiplication by the stage-specific reward scale. Positive-scale terms act as rewards, while negative-scale terms act as penalties.
Let \(g_z\) be the projected-gravity z-component, \(h\) the base height, \(q\) the joint vector, \(q_0\) the nominal standing joint vector, \(a_t\) the policy action, \(q^{tar}_t\) the target joint command, \(\tau\) the joint torque vector, \(\omega_b\) the base angular velocity, \(F^z_i\) the vertical contact force of foot \(i\), and \(v^{xy}_i\) the horizontal velocity of foot \(i\).
Common gates: \[ U_{\sigma}=\exp\left(-\frac{(g_z+1)^2}{2\sigma^2}\right), \quad H=\mathrm{clip}\left(\frac{h-h_{\min}}{h_{\mathrm{target}}-h_{\min}},0,1\right), \quad H_T=\mathrm{clip}\left(\frac{h-0.26}{0.07},0,1\right) \] \[ G_u(\alpha,\beta)=\mathrm{clip}\left(\frac{-g_z-\alpha}{\beta},0,1\right), \quad G_h(\alpha,\beta)=\mathrm{clip}\left(\frac{h-\alpha}{\beta},0,1\right) \] \[ L_i=\mathrm{clip}\left(\frac{F^z_i-3.0}{20.0},0,1\right), \quad N_i=\exp\left(-8\lVert v^{xy}_i\rVert\right) \]
Here, \(U_{\sigma}\) measures uprightness, \(H\) and \(H_T\) measure height recovery, \(G_u\) and \(G_h\) are soft activation gates, \(L_i\) measures how much load a foot carries, and \(N_i\) measures whether a loaded foot is not sliding.
| Reward term | Raw formulation | Interpretation |
|---|---|---|
recovery_progress |
\[ r = U_{\sigma_{\mathrm{soft}}}\,H_T\, \mathrm{clip}\left(\frac{1}{3}\sum_i L_iN_i,0,1\right) \] | Dense recovery signal. Rewards the robot when it is upright, near terminal standing height, and supported by loaded feet that are not sliding. |
recovery_bonus |
\[ r = \mathbf{1}_{\mathrm{recovered}}\, \mathbf{1}_{t \geq t_{\mathrm{delay}}} \] | Sparse success bonus. In terminal stages, the delay prevents the policy from exploiting near-stand resets. |
upright_orientation |
\[ r = U_{\sigma_{\mathrm{strict}}} \left( 0.30 +0.35H +0.35\,\mathrm{clip} \left( \frac{1}{4}\sum_i \mathbf{1}_{F^z_i \gt 1.0},0,1 \right) \right) \] | Encourages the body to become upright, but gives more value when the robot is also raised and supported. |
height_alignment |
\[ r = \mathrm{clip}(-g_z,0,1)\,G_{\mathrm{loaded}}\,H \] | Encourages standing-height recovery only when the robot is upright and has loaded foot support. |
posture |
\[ r = G_u(0.70,0.25)\, G_h(0.28,0.05)\, G_{\mathrm{stable}}\, \exp\left(-0.6\lVert q-q_0\rVert\right) \] | Rewards joint angles close to the nominal standing posture, but mainly after the robot is upright, high, and supported. |
feet_on_ground |
\[ r = G_u(0.65,0.30)\, G_h(0.26,0.07)\, \mathrm{clip}\left(\frac{\sum_i C_i-1}{2},0,1\right) \] \[ C_i=\mathrm{clip}\left(\frac{F^z_i-1.0}{10.0},0,1\right) \] | Rewards useful foot contact after the body is upright and raised. Disabled in later stages when stricter support rewards are used. |
loaded_foot_support |
\[ r = G_u(0.65,0.30)\, G_h(0.25,0.08)\, \mathrm{clip} \left( \frac{ \sum_i \mathrm{clip}\left(\frac{F^z_i-3.0}{17.0},0,1\right) -1 }{2}, 0,1 \right) \] | Rewards feet that carry real vertical load, not merely light ground contact. |
stable_foot_support |
\[ r = G_u(0.55,0.35)\, G_h(0.25,0.08)\, G_{\mathrm{stance}}\, G_{\mathrm{body}}\, \left(\frac{1}{4}\sum_i L_iN_i\right) \] | Rewards loaded, non-slipping support while discouraging invalid stance geometry and body-contact cheating. |
feet_slip |
\[ p = \sum_i \mathbf{1}_{\mathrm{contact},i} \lVert v^{xy}_i\rVert^2 \] | Penalizes horizontal foot sliding whenever the foot is in contact. |
loaded_foot_slip |
\[ p = G_u(0.70,0.25)\, G_h(0.27,0.06) \sum_i L_i \max\left(\lVert v^{xy}_i\rVert-v_{\mathrm{slip}},0\right)^2 \] | Penalizes slipping specifically for feet that are carrying load during terminal stabilization. |
support_deficit |
\[ p = G_u(0.70,0.25)\, G_h(0.27,0.06) \left( \frac{\max(3-S,0)}{3} \right)^2 \] \[ S = \sum_i L_i \frac{1}{1+\left(\frac{\lVert v^{xy}_i\rVert}{0.35}\right)^2} \] | Penalizes having fewer than about three useful loaded and non-slipping support feet. |
stance_region |
\[ r = G_u(0.70,0.25)\, G_h(0.26,0.07)\, \exp(-6E_{\mathrm{stance}}) \] | Encourages final foot positions to lie inside valid body-frame stance regions. |
rear_leg_separation |
\[ r = G_u(0.70,0.25)\, G_h(0.26,0.07)\, \exp(-8E_{\mathrm{sep}}) \] | Encourages the rear legs to remain separated instead of collapsing inward. |
rear_leg_crossing |
\[ p = G_u(0.70,0.25)\, G_h(0.26,0.07) \left( 0.5E_{\mathrm{front}}+2.0E_{\mathrm{rear}} \right) \] | Penalizes crossed or interlocked rear-leg configurations. |
front_leg_error |
\[ p = G_u(0.75,0.20)\, G_h(0.28,0.05)\, \mathrm{clip} \left( 1.5E_{\mathrm{width}}^2 +E_{\mathrm{forward}}^2 +0.3E_{\mathrm{center}}^2, 0,4 \right) \] | Penalizes front feet that are too wide, too far forward, or laterally unbalanced during terminal stance. |
base_orientation |
\[ p = \lVert g_{xy}\rVert^2 \] | Penalizes roll/pitch tilt using the horizontal projected-gravity components. |
base_ang_vel |
\[ p = \lVert \omega_b\rVert^2 \] | Penalizes fast body rotation, especially important during terminal stabilization. |
base_contact |
\[ p = G_u(0.55,0.35)\, G_h(0.24,0.08)\, \mathbf{1}_{F_{\mathrm{nonfoot}}\gt0.2} \] | Penalizes non-foot body contact after the robot is becoming upright and raised. |
late_nonfoot_contact |
\[ p = G_u(0.70,0.25)\, G_h(0.25,0.08)\, \mathbf{1}_{F_{\mathrm{nonfoot}}\gt F_{\mathrm{nonfoot,th}}} \] | Stronger late-stage penalty against trunk, thigh, or calf contact being used as support. |
body_slip |
\[ p = \mathbf{1}_{\mathrm{body\ contact}}\, \max\left(\lVert v^{xy}_b\rVert-0.15,0\right)^2 \] | Penalizes body sliding while the trunk or other non-foot body parts are touching the ground. |
action |
\[ p = \sum_i a_i^2 \] | Regularizes large policy actions. |
stand_still_action |
\[ p = G_u(0.80,0.15)\, G_h(0.30,0.04)\, G_{\mathrm{support}} \sum_i a_i^2 \] | Suppresses unnecessary actions once the robot is already close to a stable stand. |
terminal_action_prior |
\[ r = \frac{ G_u(0.45,0.45)\, G_h(0.22,0.10) }{ 1+\frac{1}{n}\sum_i\left(\frac{a_i}{\sigma_a}\right)^2 } \] | Encourages smaller actions near terminal recovery. This was used in an earlier terminal no-slip stage and later disabled. |
torques |
\[ p = \sum_i \tau_i^2 \] | Penalizes excessive actuator effort. |
dof_acc |
\[ p = \sum_i \ddot{q}_i^2 \] | Penalizes jerky joint acceleration. |
dof_vel |
\[ p = \sum_i \dot{q}_i^2 \] | Penalizes high joint velocity. |
dof_pos_limits |
\[ p = \sum_i \mathbf{1}_{q_i \lt q_{i,\min}\ \lor\ q_i \gt q_{i,\max}} \] | Penalizes joint position limit violations. |
joint_vel_limit |
\[ p = \sum_i \max\left(|\dot{q}_i|-0.8,0\right) \] | Penalizes joint velocities above the safety threshold. |
action_smoothness_1 |
\[ p = \left\lVert q^{tar}_t-q^{tar}_{t-1} \right\rVert^2 \] | Penalizes sudden first-order changes in target joint commands. |
action_smoothness_2 |
\[ p = \left\lVert q^{tar}_t -2q^{tar}_{t-1} +q^{tar}_{t-2} \right\rVert^2 \] | Penalizes jerk-like second-order changes in target joint commands. |
base_height |
\[ r = \exp\left(-(h-h_{\mathrm{target}})^2\right) \] | Raw base-height reward. Kept disabled because gated height rewards are safer for fall recovery. |
The videos below show the evolution of the learned fall-recovery behavior on the Unitree Go1 quadruped in Isaac Gym. Each rollout corresponds to a different training or fine-tuning stage, starting from an untrained policy that produces unstable flailing motions and progressing toward a policy that can reorient the body, lift the base, place the feet under the body, and stabilize in a standing posture.
Stage 1: Before training: flailing / no recovery skill.
Stage 1: full recovery exploration. Look for early rolling, limb-contact discovery, and high-amplitude recovery motions.
Stage 2: reorientation emerges. The robot can turn upright, but still struggles with standing height and support geometry.
Stage 3: terminal stand refinement. This isolates the near-recovered crouch → stable stand subskill.
Stage 4: action clipping and stability fine-tuning. The desired change is reduced terminal overshoot and quieter stabilization.
Stage 5: best current policy. This is the checkpoint for deterministic evaluation and locomotion handoff testing.