People walking are constantly catching a body that is trying to fall forward. A humanoid has to solve the same problem. This article traces where biped control started and how it has broadened into reinforcement-learning policies, using original papers and references. Knowing the trade-offs also tells you what to look for when watching a robot’s walking demo.
The starting point: an inverted pendulum on legs
Because the upper body is much heavier and has more inertia than the legs, a walking robot solves something like balancing an inverted pendulum — a mass sitting above its pivot. A common simplification keeps the center of mass at a constant height, the linear inverted pendulum model, whose horizontal motion is:
Here is the horizontal position of the center of mass, the center of pressure where the foot pushes on the ground, and the center-of-mass height. The further the center of mass drifts from the center of pressure, the faster the robot tips, so the controller must step or shift the center of pressure to catch it.
import math
def fall_time_constant(com_height_m, g=9.81):
# time constant of the linear inverted pendulum: how fast a fall grows
return math.sqrt(com_height_m / g)
print(round(fall_time_constant(0.9), 3)) # 0.303 s (assuming a 0.9 m center of mass)
A time constant of about 0.3 seconds means that once the center of mass starts to drift, the robot has a few hundred milliseconds to react. That is why biped control demands fast control loops.
ZMP: a precise condition for not tipping over
The zero moment point is the point where the horizontal inertial and gravity forces sum to zero. Miomir Vukobratović and Davor Juričić introduced the concept in 1968. Planning joint trajectories so the ZMP stays inside the stability region keeps the robot dynamically stable. According to the same source, Honda’s P2 (1996) and ASIMO walked using ZMP feedback.
ZMP-based control is easy to interpret physically and gives a clear stability condition. It does, however, assume a planar contact surface with enough friction to keep the feet from sliding, and it depends on well-planned leg trajectories. Uneven ground and sudden disturbances demand rapid replanning.
Whole-body control: many tasks, solved together by priority
A humanoid has to keep its balance while doing work with its hands and placing its feet where intended. Whole-body control assigns priorities to these tasks and computes all joint commands at once while respecting constraints such as torque limits and contact conditions. Balance usually gets top priority, and hand tasks are carried out only as far as they do not compromise it.
The Walker3 study defined a task and constraint hierarchy for dynamic balance and ran real-time whole-body control on a modular controller. By modeling joint friction in the reducers to improve torque tracking, it kept the robot balanced under external impulses and when the footholds were tilted or shifted. How actuator design affects control performance is covered in Rotary vs linear actuators.
RL locomotion: thousands of robots in simulation at once
Where model-based methods walk using planning and constraints designed by people, reinforcement learning finds a walking policy through trial and error in simulation. Simulation speed made the difference. Rudin et al. (2021) ran thousands of simulated robots in parallel on a single workstation GPU, trained flat-ground walking for the ANYmal quadruped in under four minutes and rough-terrain walking in twenty, and transferred the policies to the real robot.
The same trend reached humanoids. Radosavovic et al. (2023) trained a causal transformer that takes the history of proprioceptive observations and actions and predicts the next action, using model-free RL across randomized simulated environments. Deployed zero-shot on a real humanoid, it walked over varied outdoor terrain and withstood external disturbances. Using history to infer the current environment echoes RMA, which let a quadruped adapt to new terrain in fractions of a second.
What an RL policy optimizes: reward design
The character of an RL walking policy is set by its reward function. Typically the policy is rewarded for tracking a commanded velocity and penalized for large joint torques and accelerations, foot slip, torso tilt and energy use. Getting the balance wrong sends the policy toward strange solutions: penalties that are too weak produce violent motions real motors cannot deliver, and penalties that are too strong produce a timid policy that barely moves.
Exploiting the simulator is another common failure — gliding along by abusing small errors in the contact model, or scoring reward with foot-dragging gaits that would not work on a real floor. Teams therefore watch the learned behavior regularly during training, run short tests on the real robot, and revise reward terms in repeated cycles.
The two approaches compared, and combined
| Criterion | Model-based (ZMP, WBC) | Reinforcement-learning policy |
|---|---|---|
| Principle | Plan with simplified dynamics + optimization | Learn a policy by trial and error in simulation |
| Strengths | Interpretable; constraints enforced explicitly | Handles disturbances and rough terrain; less hand design |
| Weaknesses | Degrades when model assumptions break | Failures hard to interpret; needs sim-to-real transfer |
| Resources needed | Accurate robot model, real-time optimization | Massively parallel simulation, reward design |
Real systems often blend the two: a learned policy handles walking, while a model-based layer enforces joint limits and safe stopping. Techniques for moving simulated policies onto hardware are covered in Closing the Sim-to-Real gap.
What to look for in a walking demo
A walking video alone says little about performance. Check whether the same policy handles slopes, stairs and slippery floors; whether it recovers from a sideways push; whether it keeps its balance while carrying a load; and whether it protects itself when it does fall. Whether repeated success rates under fixed conditions have been published is another useful signal.
It also helps to ask how the demo was produced. A clip may show the best of many attempts, a tethered robot, or a policy tuned for one floor surface. Questions worth putting to a vendor include how many hours the robot has walked without intervention, how often it falls per hour of operation, and whether the policy running in the demo is the same one shipped to customers. Answers to those questions tell you more about readiness for real work than top speed or a single impressive maneuver.