Identifying Terrain Physical Parameters from Vision

In short: the hazards that matter most for a walking robot are often invisible in the shape of the ground, and today the robot only learns about them by stepping on them.

Legged robots can now walk over very rough terrain, and they are being used for search and rescue, planetary exploration and inspection of hazardous sites. The paper points to a different kind of danger: non-geometric hazards. A water-covered floor, a patch of ice, soft soil or deep snow can look completely flat to a depth camera, yet a foot slips or sinks as soon as it touches them.

Walking controllers trained in simulation are already robust to this: during training the simulator randomly changes how grippy (friction) and how hard (stiffness) the ground is. But the controller only notices these properties through contact, by feeling a foot slide or sink. The authors want the robot to anticipate them from its camera, before the foot lands, the way animals judge ice or mud from experience.

In short: earlier “learn-from-footsteps” vision methods predicted scores a simulator can't use; this paper predicts the simulator's own friction and stiffness values instead.

Training entirely in simulation doesn't solve this. You can render photorealistic grass, but it is hard to make simulated grass behave exactly like real grass underfoot. So a camera-to-physics link learned in simulation would not transfer.

The opposite route is self-supervised learning on the real robot: the robot walks, measures something about each footstep and uses it to label the camera images. Previous work did this for traversability (how well the robot kept its commanded speed), learned “ground reaction scores”, or the sound the ground makes. These are useful for choosing where to go, but they are signals a simulator can't reproduce accurately, so a controller trained in simulation can't be given them as inputs.

The paper's move is to make the label the simulation parameter itself. Note what this means: a predicted stiffness of 0.5 is “the simulator setting that makes the robot move as it does here”, not a physical stiffness in N/m. That is exactly what a simulation-trained policy needs.

In short: in a simulated world of random patches the right answer is always known, so a network can learn to read friction and stiffness off the robot's motion, one value per foot.

The data comes from legged gym (a GPU robot simulator), extended with procedurally generated terrain and a soft-contact model so that feet can actually sink. Every terrain patch gets a random shape, a friction between 0 and 1 and a stiffness between 1 and 10. An existing robust walking controller (the lab's 2022 perceptive locomotion policy) drives ANYmal over them with random commands up to 1.3 m/s forward or sideways and 0.3 rad/s turning.

For every time step the label is the value under each foot. While a foot is in the air, its label is the patch directly below it. In total they recorded 18.3 hours of simulated walking for training and 5 hours for testing. Training takes about 30 minutes on one GPU (batch 64, Adam, learning rate 0.001, 100 epochs). Labels are normalised on the fly and weighted so that rare values count as much as common ones.

• friction 0–1 per patch
• stiffness 1–10 per patch
• 18.3 h train · 5 h test
• 1 step = 20 ms

In short: two identical networks (one for friction, one for stiffness) remember the last second of motion and use a learned gate to decide how much to rely on the terrain shape.

Each network sees the same input: the last 50 observations (1 s). One observation contains the robot's proprioception (the walking command, joint positions and velocities, body motion and the phase of each leg) and the terrain geometry, sampled as heights on circles around each foot, the same pattern as the lab's 2022 controller.

Inside, a one-layer GRU (a recurrent “memory” with 100 units) keeps what matters from that history, and self-attention layers refine it. Then the network splits into a proprioception-only path and a proprioception + geometry path, each ending in a small MLP (sizes 64, 32, 4). A sigmoid gate (MLP 64, 64, 4) mixes the two. The output is four numbers, one per foot.

Two differences from the 2022 “belief” decoder that inspired it: this one predicts stiffness as well as friction, and it gives a value for each foot rather than one shared estimate. It is also trained on its own rather than as a side task of the walking policy.

In short: the decoder tells the four feet apart, beats the earlier decoder on friction, estimates stiffness well, and works on the real robot without retraining.

In simulation

When ANYmal walks from a patch with friction 0.95 onto one with 0.05, the front foot's estimate drops first, at step 225, and the hind foot's about 60 steps later, so the network really tracks each foot rather than an average. There is a lag of about 20 steps (0.4 s): a foot has already crossed into the new patch but hasn't yet pressed on it.

One honest detail: on grippy ground the estimate sits near 0.6, not 0.95. Between 0.6 and 1.0 nothing slips, so the robot moves the same way and the exact value simply can't be observed from motion. For adapting a walking or navigation policy, “grippy” is enough.

What this means: friction error is about 29 % lower than the earlier decoder, and stiffness is off by roughly 5 % of its range; both methods still can't tell very grippy values apart.

On the real robot

For friction, ANYmal was driven backwards from the lab floor onto a whiteboard covered with water. The new decoder distinguishes the feet: hind feet on the slippery board drop to about 0.1 while the front feet, still on the floor, stay higher. The baseline gives the same value for all feet. For stiffness, the robot walked over a soft foam board and the estimate dropped on the foam and rose again on the floor.

What this means: on the slippery board the new decoder is almost always inside the “correct” range found by the digital twin (next section); on the floor both are fine because that range is wide (0.3–1.0).

In short: replaying real motion in simulation shows which simulator friction values reproduce reality: 0.0–0.2 for the wet board, 0.3–1.0 for the floor, and the decoder's real-world estimates fall in those ranges.

Showing that estimates go up and down in the right places is not the same as showing the numbers are right. Real ground has no “simulator friction” to compare against. So the authors built a digital twin, a simulated copy of the experiment: they cut the real whiteboard recording into 400 ms snippets, started the simulated robot in exactly the recorded pose (orientation from the onboard IMU, joint positions and velocities), rebuilt the flat floor in simulation, and replayed each snippet with 10 friction values from 0.0 to 0.9.

Three measures compare the replay with reality: joint-position error, body-orientation error, and how far the foot slides between touching down and lifting off. They used 18 whiteboard snippets where the foot really slipped (slips are rare) and 220 floor snippets without slipping. Longer replays drift away from reality, which is why snippets are short.

What this means: the decoder's real-world estimates (≈ 0.1 on the board, above 0.4 on the floor) are values that make the simulator behave like reality, so they can be used as labels. The check currently only works on flat, rigid ground; the authors assume, but have not yet shown, that stiffness transfers the same way.

In short: while the robot walks, each footstep's friction and stiffness are painted into camera images taken earlier, and a small image network learns from them live.

The vision side follows the lab's earlier Wild Visual Navigation system. Every camera image goes through DINOv2, a large pre-trained image model that describes each pixel with a 384-number feature vector. The images (scaled and centre-cropped to 910 × 910) are the only visual input; no depth is needed, because friction and stiffness don't depend much on geometry and the features already carry a lot of it.

Three tasks run in parallel. Inference predicts friction and stiffness for new images. Self-supervision keeps the mission graph: it stores an image's features, camera pose and an empty label mask every 1 m the robot travels (0.2 m indoors), and when the decoder labels a new foothold, projects it into every stored image within 5 m. Learning samples stored images that have labels and updates the network, at about 2 updates per second.

The image network is tiny: an encoder-decoder MLP with layers 128, 32, 128 and 384 + 2. The two extra outputs are friction and stiffness; the 384 outputs try to rebuild the input feature (used in the next section). The loss is 0.9 × rebuild error + 0.1 × prediction error, trained only on labelled pixels.

In short: pixels the network can rebuild well are trusted; a two-bump fit decides the cut-off per image with nothing to tune, and indoors the result matched the real floor with IoU 0.82.

The robot only steps on a small part of the scene, yet the network outputs a friction value for every pixel, walls and trees included. The rebuild output solves this: the network learns to rebuild features of ground it was taught, so taught-like pixels have low rebuild error and everything else high error. As training goes on, the histogram of errors turns from one bump into two.

The earlier system fitted a single bell curve to the labelled pixels and needed a threshold setting. Here a two-component Gaussian Mixture Model is fitted to each image's error histogram, and the cut sits where the two components meet. It needs no tuning across datasets and adapts per image.

What this means: from a couple dozen images, the camera alone gives friction values within the digital-twin ranges and marks the walkable floor with 82 % overlap to hand labels; IoU (intersection over union) is the overlap of predicted and true areas.

The supplementary video also shows the pipeline outdoors: friction on snowy paths and stiffness while walking through snow, with the network adapting online.

In short: the paper builds the camera-to-simulator-physics link; the controllers that would use it are the next step.

• Stability. When the robot stands still, friction estimates drift. A thin 5 cm foam board was not detected as soft on the real robot. The authors suspect too little training data or noisy real terrain geometry, and suggest adding noise during training.
• Validation. The digital twin only covers flat, rigid surfaces and needs manual terrain reconstruction; stiffness has not been checked numerically in the real world yet.
• Scope of the vision tests. Numbers come from an indoor dataset (22 training images) and an 82-image outdoor dataset for the mask; the off-road runs in the video are qualitative.
• The payoff. The goal is walking and navigation policies, trained in simulation with friction and stiffness as inputs, that receive those values from a camera on the real robot. The paper lays the ground for that; the policies themselves are future work.