Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking

In short: to cross a box, a hurdle or a staircase the way people do, a humanoid has to use its arms and knees too — and pick the right move from what it sees.

Legged robots trained in simulation with reinforcement learning (trial and error guided by a score) have become remarkably agile, especially four-legged ones. Humanoids are harder: they have more joints, balance on two feet, and their best moves over big obstacles involve the whole body. Think of how you climb onto a high box: hands on the top, a knee up, then stand. Jumping down, you let your hands take part of the landing.

The paper's goal is a humanoid that does exactly this, on a real robot: walk toward a goal, see the obstacles in its path with its onboard LiDAR, and switch between whole-body skills — climbing, jumping down, vaulting, stairs — as the terrain demands.

In short: earlier methods either never discover whole-body moves, can only replay one fixed recording, or need a hand-built pipeline for every new skill.

1 · Learning from a score only

Training a humanoid controller purely by reward shaping — writing a score for "go forward, don't fall, move smoothly" — means searching a huge space of possible motions. Without guidance, the paper notes, controllers often converge to strategies dominated by the lower body, with little coordination between arms and legs. That is fine for walking, not for climbing a 75 cm box.

2 · Copying a recording

Motion tracking (the approach behind DeepMimic and many recent humanoid systems) trains a controller to copy reference motions from people. It transfers beautifully coordinated skills — but a tracker only replays choreography. It has no way to change a climb recorded for a 50 cm box into one for an 80 cm box.

3 · Many experts, merged

Another route trains one expert policy per terrain and then distills them into a single student that picks the right skill from what it sees (as in the concurrent works cited as APEX and PHP). These show promising whole-body traversal, but rely on carefully designed distillation: which expert teaches where, and on what mix of terrain. According to the authors, such methods have not yet shown strong scaling to many skills or to new terrain.

And AI that invents motion?

Generative models trained on large motion collections can compose skills naturally. But their raw output can contain artifacts — feet sliding, sudden jumps between frames — that make a real robot fall. Earlier work paired a motion generator with a tracker (CLoSD, RobotMDM, and PARC, whose design this paper follows), mainly for simulated characters, where slow inference and real-world disturbances are not an issue. This paper brings that idea to a real humanoid.

In short: the team recorded one example of each skill, turned it into physically possible robot motion, then multiplied it by resizing the obstacles.

The starting point is small: about 5 minutes of human motion, from videos the team filmed themselves and from public motion-capture collections (AMASS and data from OmniRetarget). There is one representative motion per skill: climbing onto a 50 cm box, vaulting a 35 cm hurdle, jumping down from a 50 cm box, stairs with steps of about 20 cm, plus walking and turning in all directions.

1. Video → 3-D human pose. For the filmed clips, a tool called GVHMR reconstructs the person's 3-D motion.
2. Human → robot (retargeting). An inverse-kinematics solver that respects contacts converts the motion to the G1's body. The obstacles are then placed by hand to match where the person touched them.
3. Physics check. Rather than trust these trajectories directly, a DeepMimic-style tracking policy learns to perform them in simulation, and the motion it actually produces becomes the training data — so everything in the dataset is physically possible.
4. Augmentation. The obstacles are scaled and small random boxes are added along the path; the motion is re-optimized to fit (penalizing going through the terrain and jerky movement), then passed through the physics check again.

The result is about one hour of motion: box climbs and jump-downs on 35–75 cm boxes, vaults over 25–45 cm hurdles, stairs with 15–20 cm steps, and walking among small boxes.

In short: an AI looks at the ground around the robot and the direction it should go, and sketches the next half second of whole-body motion.

The motion generator is a diffusion model, the same family of AI that generates images from noise, here following the MDM architecture as adapted by PARC. It starts from random numbers and removes noise step by step until a plausible motion appears. What it produces is guided by three inputs:

• the target heading — which way the robot should go (computed from a goal position);
• a height scan — the terrain height around the robot, sampled on a 17 × 11 grid;
• the last two frames of the robot's own motion.

Its output covers the next 0.5 s in 25 frames: where the body's root is and how it is oriented, all 23 joint angles, and the positions of the body's links. Besides matching the training motions, it is trained with extra geometric penalties (on velocities, consistency between joints and body positions, and going through the terrain). The height scans and past frames are perturbed during training so the generator copes with noisy robot sensing.

One important detail: when the generator runs alongside the robot, it is conditioned on the robot's actual last two frames, not on its own previous sketch. That closes the loop — if the robot drifts from the plan, the next sketch starts from where the robot really is.

In short: a second network, trained by trial and error in simulation, turns the sketch into motor commands while keeping the robot balanced.

The motion tracker is a single policy trained with RL (the PPO algorithm, in the IsaacLab simulator, on one RTX 4090 GPU), first on the curated dataset alone. Each moment it sees:

• the reference: target body velocity, 23 joint angles and speeds, and 9 key body positions;
• its own body sense over five recent frames: rotation rate, direction of gravity, joint angles and speeds, and its previous action;
• the 17 × 11 terrain height scan.

It outputs 23 target joint angles. The authors note a prior study found terrain input not strictly needed for accurate demonstrations, but giving the tracker the height scan from the start helps the later fine-tuning stage.

What it is rewarded for (paper Table I)

What this means: the tracker is paid mostly for resemblance to the sketch; only in the last stage does it also get paid for heading the right way.

In short: the tracker is trained further while the frozen generator feeds it live sketches, so it learns to follow imperfect plans and to ignore unsafe ones.

Simply plugging the two trained networks together is not enough. The tracker learned from smooth, physically filtered recordings; the generator's live sketches, built from noisy sensing, can contain jumps between frames and other artifacts. That mismatch makes a naive combination fragile.

So there is a third stage, fine-tuning. The generator is frozen and runs in the loop in simulation, conditioned on the robot's real last two frames, with only 2 denoising steps and extra noise injected into both the generator and the tracker's inputs. Goal directions are randomized, a heading reward is added, and the terrain gets harder and more varied: stairs with 15–25 cm steps, rows of 25–55 cm hurdles, and several boxes and pyramid steps of 30–85 cm with varying widths, yaw and pitch.

After fine-tuning, the authors describe the tracker as a motion filter: it follows the generator's patterns and uses the height scan to suppress unsafe parts. Behaviours appear that were not in the recorded data, such as climbing a box from its corner, turning around on top of a box before jumping down, and vaulting several hurdles in a row.

On the real robot they also saw a kind of local navigation: when the robot was not in a good position to climb, the tracker partly overrode the sketch and walked around the box instead, still reaching the goal.

What this means: the rehearsal stage is not a small polish. Without it the combined system fails more and more often as obstacles grow; with it, it stays reliable across most of the tested heights. (The paper reports this as a chart only, so no exact values are quoted here.)

In short: when obstacles differ from the recording, replaying it breaks down, while the generator-plus-tracker system keeps succeeding.

To test whether live motion generation matters, the authors compare two versions in simulation, using the same fine-tuned tracker:

• Tracker only replays one fixed reference: a climb recorded on a 50 cm box, a vault over a 35 cm high, 20 cm wide hurdle, or stairs with 20 cm high, 30 cm deep steps. Success = reaching the reference's final position.
• Tracker + generator is the full system: the generator sketches references online from the goal direction and the terrain. Success = reaching a goal placed at that same position.

The terrain is then changed: taller or shorter obstacles, or rotated (yaw). Each setting uses 500 robots with random starting poses.

What this means: the fine-tuned tracker alone tolerates small changes (a slightly different height, a rotated obstacle), but once an obstacle needs a different timing or style of movement — an 80 cm box, 25 cm steps, a 55 cm hurdle — only live sketching keeps it working. At 50 cm with ±30° yaw, replay is marginally better (0.988 / 1.000 vs 0.980 / 0.982).

In short: everything — sensing, mapping, sketching and control — runs on the robot, with a fresh sketch four times a second.

• Where am I? DLIO, a LiDAR-inertial odometry method, uses the G1's Livox MID-360 LiDAR and its IMU to estimate the robot's pose.
• What's the ground like? Elevation Mapping CuPy builds a terrain height map from the LiDAR and that pose; the height scans are read from it.
• A wobbly head. The G1's neck joint is passive, so aggressive moves swing the head, which carries the LiDAR, and spoil the pose estimate. The team combines the torso IMU with the LiDAR's IMU to estimate the head's pitch and correct for it.
• Speed. The generator runs with TensorRT on a Jetson Thor mounted on the robot's back, taking about 0.02 s per sketch. The tracker and everything else run on the G1's built-in Jetson Orin.
• Re-planning. A sketch covers 0.5 s. In fine-tuning a new one was made every 0.5 s; on the robot a new one is made every 0.25 s (a "receding horizon"), so the robot reacts faster to what it sees.

What the hardware tests showed

The robot climbed a 75 cm box and jumped down in three ways (straight, after a 90° turn on top, and from the corner), using knees and hands for support going up and hands for cushioning coming down — the style of the human data. It vaulted hurdles of different heights, climbed and descended stairs, and crossed a mixed course of a vaulting obstacle, stairs and a box, switching styles on its own — including combinations such as stairs right after a jump-down that it never saw together during fine-tuning.

In short: pairing a motion generator with a fine-tuned tracker gives a real humanoid human-style, terrain-aware skills without a hand-built pipeline per skill — but it is only as good as its terrain map.

• Map dependence. The system relies on a LiDAR elevation map. When mapping degrades because of sensor noise, the authors report that locomotion can deteriorate substantially. They point to learned ("neural") mapping or a belief encoder that falls back on the body's own sensing as remedies.
• Beyond walking. Extending the approach to loco-manipulation (moving while handling objects) and to harder outdoor environments is left for future work.
• Scope of the numbers. The quantitative comparisons are in simulation (500 robots per setting); the hardware experiments are demonstrations, shown in Fig. 3 and the video, without success rates.