DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition

In short: robots can learn many ways of walking, but today only experts can tell them which way — answering “which video is better?” could open that up to anyone.

The paper starts from a simple wish: that a legged robot walk the way its user prefers, not just the way a task formula dictates. There are two standard routes, and both are expensive for ordinary users. Reinforcement learning — learning by trial and error to maximise a reward — needs a reward function with several hand-tuned terms, which takes expert knowledge. Imitation learning needs demonstrations, and demonstrating a four-legged gait through a robot interface takes skill and a clear idea of the target motion in advance.

A third route asks much less of the user: show two videos and ask which is closer to what they want. This is preference-based reinforcement learning (PbRL). From the answers, a reward model learns to score motions the way the person would; the robot's controller is then trained on that score, new videos are shown, and the loop repeats. The authors say this is, to their knowledge, the first application of PbRL to quadruped locomotion. The obstacle is the number of questions: PbRL usually needs so many that it is impractical on a real schedule.

In short: when both videos come from the same slowly-changing controller, they often look equally good or equally bad, and the person can only say “can’t decide” — a wasted question.

In a typical PbRL setup, both clips of a question are cut from different runs of one policy. After each batch of answers that policy is updated a little — it is a continuation of the previous one. The paper calls the resulting lack of variety policy bias: successive versions behave alike, the clips they produce look alike, and the updated reward is hard to exploit because the policy barely explores anything new.

The key observation is about the person. A question is easy when one clip is clearly closer to what they want. When both are similarly off, or similarly good, people struggle — “not because the trajectories themselves are indistinguishable, but because their underlying preferences are.” Those questions produce no usable label. The paper names the property that matters preference discriminability, and builds the method around making it high.

Earlier ways to save questions each help, but none looks at this: automatically labelling pairs the reward model is already sure about (SURF), exploring before asking and re-scoring old data after each update (PEBBLE), asking about pairs near the reward model's decision boundary (uncertainty sampling), making learning robust to wrong labels, or choosing questions from a fixed set of pre-recorded clips (active reward learning — which cannot create new, easier-to-compare behaviour).

In short: DAPPER throws away the “keep updating one controller” habit: after each batch of answers it trains a new controller from scratch, and questions compare two different controllers.

Training from scratch means the new policy is not anchored to the old one, so policy bias disappears and behaviours spread out. Each finished policy goes into a library Π. A question then pairs the newest policy with one earlier policy from the library (“policy-to-policy”), instead of two episodes of the same policy. The preference model is rewritten accordingly: the probability that policy i is preferred over j is a softmax of the reward model's score of a trajectory from each.

But variety alone is not enough — two very different robots can still be equally far from what the person wants. That is why the second half of the method, below, estimates discriminability directly.

The paper's Fig. 4(b) shows where the learned policies landed in the “posture” feature space (body height × incline). The Baseline starts low and tilted and creeps along a diagonal toward the target. DAPPER without its discriminability bonus jumps around at first — fresh policies help — but then gets stuck far from the target, at the top of the height range. Full DAPPER scatters widely early on and then collects around the target. The scene above is an illustrative version of that pattern.

In short: the authors measured with real people how big the difference must be before a comparison is answerable, and turned that into a rule their experiments could apply automatically.

Running thousands of questions past real people for every comparison would be impractical, so the experiments use a simulated annotator: a rule that answers like a person would. To make that rule honest, the authors first ran a study. They trained 25 policies on a grid of body heights (29, 37, 45, 53, 61 cm) and incline angles (−8, −4, 0, 4, 8°). Five participants saw a reference video of the target posture — 45 cm high, level body — and two candidate videos, and answered “Left is better”, “Right is better” or “Can't decide”, for the same 105 pairs in random order.

The simulated annotator measures each clip's distance from the target as the sum of its feature errors (each feature scaled to 0–1, divided by √(number of features)). The discriminability of a question is the gap between the two clips' distances. Below a threshold, the answer is “can't decide”; above it, the closer clip wins.

What people did: at the smallest gaps, around 60 % of answers were “can't decide”; the share fell as the gap grew and reached zero above 0.4 — every participant could tell. When people did pick a side, they were right more than 80 % of the time at every level, and always right above a gap of 0.3. From this the authors set three thresholds for the experiments: Small = 0.2, Medium = 0.3, Large = 0.4 (larger means harder questions, since more pairs count as unanswerable). One answer took about 7 s on average.

What this means: the experiments that follow simulate people as the study found them — sometimes unable to decide — rather than as perfect judges.

In short: DAPPER learns, from past answers, which pairs a person can tell apart — and uses that both to choose questions and to steer the next controller toward being easy to compare.

Every answer is stored with its label: 1 or 0 when the person chose, 0.5 for “can't decide”. The chosen answers train the reward model R^H in the usual PbRL way. All answers — sorted into “could tell” and “couldn't” — train the discriminator D, a network that takes trajectories of two policies and outputs the probability that a person could tell them apart. It is symmetric (the order of the pair doesn't matter), uses Monte Carlo dropout to stay stable early on when it has seen few labels, and is re-trained from scratch every round so it doesn't overfit to the first policies.

Use 1 — choosing the question

One side of every question is the newest policy. Its partner is drawn from the library with a probability that grows with D's score for the pair — a softmax with a temperature α. A higher α picks tellable pairs more often; too high, and it keeps asking about the same easy pairs, which starves the reward model of variety. Pairs already asked are never repeated.

Use 2 — a bonus for being distinguishable

While the next policy trains, its reward mixes the person's preferences with D's opinion: at every step, R^D is D's average score for “the trajectory so far” against each earlier policy. So the new controller is pushed to be both liked and easy to tell apart from what came before.

Why not just reward distance from earlier policies? The paper tests exactly that (“Ours-DM”, which maximises feature distance). It explores more but converges worse — being different isn't the same as being easy to judge.

In short: each round trains a fresh controller, asks ten questions and updates two small models; training dominates the time, but far fewer rounds are needed.

The three steps of Fig. 2 repeat until the learned style is close enough to the target (a distance below 0.02):

1. Policy learning. A new policy starts from scratch and trains on the mixed reward r: 300 updates, each from 128 simulated robots walking 128 steps in parallel. Training uses Lagrangian PPO, a reinforcement-learning method that also enforces constraints — here walking forward at 1 m/s (± 0.2), no sideways drift or turning, no body or leg collisions — so every candidate is a sensible walk and the person only judges style.
2. Query collection. Ten questions pair the new policy with sampled partners from the library; answers go into the question dataset.
3. Reward update. The reward model and the discriminator are re-fitted on all answers so far.

Retraining from scratch every round sounds wasteful, so the authors timed it (Fig. 6), on one RTX 4090 GPU and assuming about 7 s per human answer. A round takes 13–14 minutes, 12–13 of them policy training; ten questions take about 70 s; updating the models takes seconds. The Baseline can keep training one policy with fewer updates per round. With 30 updates it cut its own total time to about 40 % of the 300-update version — yet still took about twice as long as DAPPER overall, because it needed many more rounds. With only 10 updates per round, asking questions dominated and total time grew again.

In short: DAPPER needed the fewest questions in every setting, kept working when questions were hardest to answer, made an AI judge far more useful, and produced gaits that ran on the real robot.

All experiments use the ANYmal quadruped in the legged_gym / Isaac Gym simulator; the controller runs at 50 Hz and sends joint targets to 400 Hz joint controllers. DAPPER is compared with the Baseline (classic single-policy PbRL), SURF (auto-labelling) and PEBBLE (pre-exploration and re-scoring), plus two variants of itself: without the discriminability bonus, and with a plain distance bonus (Ours-DM). Each setting was run five times.

Questions needed (Fig. 4)

On the posture task, DAPPER reached the target with the fewest questions and had the highest share of answerable questions at every threshold. At the hardest threshold (Large), only DAPPER and Ours-DM kept producing enough answerable questions: Ours-DM needed more than three times as many questions as DAPPER, and the single-policy methods did not reach the target within the 2,000-question limit. In the Discussion the authors sum up that DAPPER cuts the question count to about one tenth of what previous approaches need — though that is still hundreds of answers.

Across a fine sweep of thresholds (Fig. 5), DAPPER beat the Baseline everywhere. The gap narrows when questions are easy — but even with a threshold of zero, where every question is answerable, the Baseline lags, because policy bias stops it from reaching new behaviour in few rounds. At very high thresholds answerable questions become so rare that DAPPER fails too.

More features (Fig. 9)

Two harder styles add gait timing: “Trot only” (4 features: one leg's cycle time and the timing offsets between legs) and “Normal” (6 features: those plus height and incline). DAPPER converged faster in both; earlier methods found very few answerable questions with six features. The authors note even DAPPER may hit limits as features grow.

An AI as the judge (Table IV)

Could a vision-language model answer instead of a person? The authors gave GPT-5 three frames (at 0.5, 1.5 and 2.5 s) of a reference video and two candidates. Such models rarely admit “can't decide”, so answers where the frame-wise preferences were not significantly different (Welch's t-test) were mapped to “can't decide”. With up to 1,000 questions:

What this means: with the Baseline, the AI couldn't answer half the questions and was barely better than a coin flip on the rest, so almost nothing was learned; DAPPER's easier questions made the same AI useful enough to reach the target style — even with about 15 % of its labels wrong.

On the real robot

A “crawl” style (body height 30 cm, level body, trot timing) was learned in simulation and run on the real ANYmal using the standard sim-to-real recipe of legged_gym — randomising sensor noise and physical parameters in training — without extra tuning. The style features are only needed in simulation, to compute the reward.

In short: DAPPER makes preference learning much cheaper in questions, but the style is still described by a handful of hand-chosen features, and real users in the loop remain the next step.

• Hand-picked features. Styles are described by up to six chosen features (heights, angles, gait timings). Conflicting or noisy features could stall learning; learning the features automatically from sensor data is a stated next step.
• Still hundreds of questions. About a tenth of before is a big cut, but richer, higher-dimensional preferences (smoothness, energy use, style) will need more help. With more features, people may also find more pairs undecidable — and Fig. 5 shows that very high thresholds eventually break the method.
• Compute for retraining. Training from scratch each round is the price of avoiding policy bias; warm starts or meta-learning could lower it.
• Early noise in the discriminator. With few labels its estimates can mislead exploration; Monte Carlo dropout helped here, but larger problems may need more.
• Simulated answers. The large comparisons used the simulated annotator calibrated on five people; the GPT-5 study is a proof of concept. Wrong labels (not just “can't decide”) were tolerated at about 15 %, and combining DAPPER with noise-robust methods is suggested.
• Beyond walking. Only quadruped locomotion was tested; manipulation or humanoids are possible future targets.