Reconstructing occluded elevation information in terrain maps with self-supervised learning

In short: robots plan on a height map of the ground, and that map is always full of holes wherever the sensor could not see.

Mobile robots that go somewhere on their own need a map of the ground around them to judge where they can walk or drive and to plan a path. A common and compact choice is a 2.5-D elevation map (the paper says DEM, digital elevation map): a regular grid seen from above in which each cell stores one number, the height of the ground there. Robots build it from depth sensors such as a laser scanner (LiDAR) or stereo cameras, and the robot-centric version used on ANYmal keeps the robot at the centre of the grid as it moves.

For safe and efficient planning you want this map complete and accurate. In practice it never is. Rocks, obstacles and convex terrain such as a crest or a step edge hide patches of ground from the sensor’s viewpoint. On top of that, depth measurements fail on reflections, in dust, when stereo matching fails or on textureless surfaces, which leaves further gaps.

The authors’ observation: people look at such a map and guess what hides in the holes quite well. We know what stairs, rubble and hills usually look like, and we intuitively use the line of sight. Classic hole-filling methods use neither. The goal of the paper is a method that does, and that can learn from the data robots actually collect.

In short: existing fixes either avoid the holes or smear smooth surfaces across them, and learning a better fill needs complete maps that real robots never record.

What do robots do with holes today? The most common answer in navigation work is to avoid them: the planner simply does not plan through cells without height information. That is safe but short-sighted (scene 7). The other answer is to fill them with methods borrowed from image repair:

• Interpolation (linear or cubic) draws a smooth surface between the known edges of a hole.
• Diffusion-based inpainting such as Navier–Stokes and Telea propagates the surrounding values into the hole, following lines of equal value.
• Patch-based methods copy the best-matching patch from elsewhere. For terrain, Kolter et al. (2009) searched an offline library of terrain patches and also enforced line-of-sight constraints.

These hand-tuned rules do fine on smooth ground but know nothing about what terrain usually looks like. On stairs and boxes they blur edges into ramps. In image inpainting, learned networks (usually U-Nets, sometimes with adversarial training) have long been the state of the art, and Qiu et al. used a GAN to fill voids in large geospatial elevation data.

The catch for robot maps is training data. A network learns to fill holes by seeing examples with the right answer, which means complete elevation maps. Those exist in simulation, but real maps always have holes. The closest earlier work, Šalanský et al. (2021), built “ground-truth” maps offline with SLAM, which also uses measurements from later in the run and so fills more of the map. The authors point out that such a target inherits pose-estimation errors and flawed depth measurements. The question this paper answers: can we train on the incomplete real maps directly?

In short: take a real map with holes, hide some more of it, and train the network to restore what you hid; the recorded map is the answer key.

The key idea is a new way of doing self-supervised learning on terrain maps. Instead of asking for a complete map, the method treats the partly occluded map the robot recorded as the target. It then makes the input harder by adding artificial occlusion: extra holes that look like the ones a sensor would produce (scene 4 shows how). The network sees the doubly-holed map and must fill it in.

Grading is the clever part. The network’s output is compared with the target only in the artificially hidden cells, where the robot really did measure the height, plus (with less weight) the cells that were visible anyway. Cells that were already missing in the recording are ignored in the loss, because nobody knows their true height. At test time the network fills all holes, including the original ones.

Because no clean answers are needed, the paper notes that this points towards robots that keep improving from the data they gather while exploring, in the spirit of lifelong learning.

In short: the practice holes come from an imaginary sensor placed at a random spot, and its height is tuned until a sensible share of the map is hidden.

For the trick to work, the practice holes must resemble real occlusion; otherwise the network learns to fill the wrong kind of hole. The authors first tried growing the existing holes (dilation) and deleting random pixels, but neither produced realistic or varied masks. What does: ray casting. They wrote a small, fast C++ component that takes an elevation map and a vantage point and checks, for every cell, whether the straight line from the vantage point to that cell is blocked by higher ground. Blocked cells become holes.

The vantage point is drawn uniformly at random on the map. Its height above the ground, the offset o, decides how much gets hidden: low viewpoints cast long shadows, high ones almost none. An iterative search keeps the occlusion ratio (hidden cells over all cells) between 0.1 % and 50 %:

1. Sample o uniformly from the range [o_min, o_max] and ray cast.
2. Too much hidden (> 50 %)? Raise o_min to this o: only higher viewpoints from now on. Too little (< 0.1 %)? Lower o_max to o.
3. If the range shrinks below 5 cm, widen it again by 5 cm. Stop when the share fits, or after 15 tries.

The same ray caster also builds the synthetic datasets: it takes a complete generated terrain, places a “robot” at a random position (within ±1.25 m of the centre, sensor 0.2–0.5 m up on hills and 0.2–0.3 m on stairs and boxes) and produces the occluded map and its mask. There, the full answer is known, so supervised and self-supervised training can be compared fairly (scene 6).

In short: a small, standard image-repair network takes the holey map plus a “what is missing” mask and outputs a complete map; only its guesses inside the holes are used.

The network is a U-Net, the design most learned image-inpainting methods build on. It has two input channels: the occluded elevation map and a binary occlusion mask (1 = missing, 0 = known). Missing cells are stored as “not a number”, which a network cannot take, so they are replaced by 0.0 on real data (and by linear interpolation on synthetic data, where that worked better in a selection study). Before the network, the mean height of the known cells is subtracted, and it is added back to the output, so the network only has to reason about shape, not absolute height.

Compared with the original U-Net, the authors kept it light: the inputs are only 64 × 64 cells (not 572 × 572 pixels), so they use 3 down-sampling steps instead of 5 and 64, 128 and 256 feature channels in the hidden layers. The shrinking half sees the wider shape of the terrain; the growing half rebuilds full resolution; skip connections carry fine detail straight across.

Training used PyTorch and the Adam optimiser (learning rate 0.0001, weight decay 0.001), for up to 100 epochs or until the validation error on hidden cells had not improved for 50 epochs. At use time the output is composed with the input: every measured cell is kept as it was, and the network’s values are used only where the map had holes.

In short: on real robot maps the error in the filled cells drops to between about half and a fifth of the best classic method, and learning from practice holes is nearly as good as learning from complete answers.

The data

• Synthetic terrain: hills (Perlin noise), regular stairs and randomly stacked boxes; 64 × 64 cells at 4 cm (2.56 m square); 25,000 training and 2,500 each for validation and test.
• ANYmal with a dome LiDAR: maps of 300 × 300 cells at 4 cm, cut into 16 tiles of 75 × 75. ETH Zurich stairs (26,233 tiles), an ETH obstacle course (37,274) and the Gonzen mine in Switzerland (16,459), recorded during subterranean exploration; split 80/10/10.
• Tenerife lunar analogue: the ESA HDPR rover testbed with stereo cameras at Minas de San José on the Teide volcano, June 2017. Maps built with GA SLAM using GNSS poses, 600 × 600 cells cut into 64 tiles; 42,600 / 1,000 / 7,950 samples, the test set from a separate side track never seen in training.

Real-world maps

Real maps have no complete answer, so the test is the training trick itself: add ray-cast holes to unseen test maps and measure the error on those cells. Five random seeds; heights in metres.

What this means: on every real dataset the learned fill beats all four classic methods (linear, cubic, Telea, Navier–Stokes), cutting the squared error by 52–82 %; for 75 % of test tiles the average error was under 5 cm (stairs), 4 cm (obstacles, mine) or 3 cm (Tenerife).

Synthetic maps: practice holes vs full answers

What this means: PSNR in hidden cells, higher is better (three seeds). Self-supervision is only 0.3–2.3 dB behind training on complete answers, and the gain over classic fills is largest on structured terrain such as stairs and boxes; on smooth hills cubic interpolation is already close.

The authors also note a side-effect of learning from real data: because the training targets contain sensor noise, the network learns to reproduce similar noise inside the holes.

Speed. For a full 300 × 300 ANYmal map (16 tiles, each resized to 64 × 64, in two batches of 8), the U-Net runs at 3.5 Hz on one core of a laptop CPU and 30 Hz on an NVIDIA TITAN Xp GPU. Telea and Navier–Stokes run at about 50 Hz but are far less accurate.

In short: in a first simulation test, a planner using the filled-in map could plan further ahead, so the robot stopped less often to wait for a new plan.

As a preliminary study, the authors fed completed maps into motion planning for ANYmal in simulation. The network was trained on the real ETH stairs data, and the paper highlights how well it carried over to the simulated world. The locomotion and navigation software is the same that team CERBERUS used in the DARPA Subterranean Challenge.

The baseline plans on the raw, incomplete map and never plans through cells without height information, which is common in the literature. It cannot plan far ahead and its paths are not optimised over the whole map, so the robot often has to stop and wait for re-planning. With the reconstructed map, paths reach further and only need small corrections when the real measurements of the hidden area arrive.

The lanes in scene 7 are an illustrative animation of this behaviour; the paper reports it qualitatively (Fig. 4 and a side-by-side video), without stop counts.

In short: the method removes the need for complete training maps; knowing how sure it is about each filled cell is the next step.

• Evaluated on artificial holes. On real data, accuracy can only be measured on cells that were hidden on purpose, because the true height under the original holes is unknown.
• Learns the noise too. Training targets are real, noisy maps, so the fill reproduces similar noise.
• No confidence yet. The paper names uncertainty estimation as crucial future work: a planner should trust a filled cell less than a measured one, per cell.
• Planning is a proof of concept in simulation, not a field test.
• Ideas from the authors: learn realistic occlusion patterns with a GAN instead of ray casting, and add a discriminator in place of the perceptual losses that need complete answers.