Microduck Studio books · Book 3 of 4 · Advanced

Training Microducks

A Microduck's walk comes from reinforcement learning and the settings in Pollen Robotics' training code. This book covers both, and the honest measuring that has to come after. Ten lessons, each with a figure to run.

Who it is for: makers, students and club leaders who can read a little algebra and want to train a policy, or judge one. Code helps but is not required, and Book 2 has the same ideas without the formulas.

States, actions, rewards, return
Value, advantage and GAE
PPO: better, but not too much
61 numbers in, 14 out
The reward is the specification
The actuator is most of the gap
Randomize, then measure
Honest numbers
Learning from people
Ship it, and say what it does

How a lesson works: read the left page and commit to a prediction, then run the figure on the right page one step at a time and check yourself. Each lesson ends with something to try, notes for whoever is teaching it, and the source of every number.

Lesson 1 · The problem

States, actions, rewards, return

Training starts by writing the duck's task as a loop, set up as a Markov decision process: the next moment depends only on this one and the action. Strictly it is a partially observed one, since the policy sees an observation rather than the whole state. Every tick t the policy reads an observation o_t of 61 numbers and chooses an action a_t of 14 joint targets, then physics advances and the environment pays a reward r_t. The policies run at 50 Hz, so a tick, which is also a step here, lasts 20 ms.

So what is the policy trained to raise? Not the next reward but the expected discounted return, every reward still to come, each weighted less the further away it is. With the discount γ = 0.99, a reward one tick further away weighs 0.99 times as much.

G_t = r_t + γr_t+1 + γ²r_t+2 + …, with a horizon of about 1/(1 − γ) ticks.

Predict

With γ = 0.99 at 50 Hz, roughly how far into the future does the policy look?

About 100 ticks, 2 s. Just one tick, 20 ms. To the end of the episode.

Check your prediction

About 100 ticks, 2 s. 1/(1 − γ) = 100 ticks is the policy's working horizon, about 2 s of the future, and a reward 100 ticks away weighs 0.99¹⁰⁰ ≈ 0.37 of one paid now. The horizon is soft, though: a reward 200 ticks away still counts 0.13, but the further off a payoff sits, the less it shapes the policy.

The figure

Ticks. A reward arrives every 20 ms tick, one bar each.
Discount. Each bar shrinks by γ for every tick of distance, so the far ones count for less.
Horizon. A flag at 1/(1 − γ) ticks: 100 ticks, 2 s at 50 Hz, where the weight is 0.37.
Return. The weighted bars stack into one number, the return the policy is trained to raise.

Drag γ to move the horizon. The rewards are made up; γ and the tick are Microduck's.

Big idea: a policy optimizes the near future it can see, about 1/(1 − γ) ticks, which for Microduck is about 2 s.

Try it for real

The discount sits in the walking task's runner settings, beside the other PPO numbers in this book:

gamma=0.99,
lam=0.95,
desired_kl=0.01,

Change it there, train two full runs of microduck_rl, and compare the gaits. A smoke test is no use for this: it only checks that a task builds and runs, and it learns no gait. Every run needs a CUDA GPU or Hugging Face Jobs, even a smoke test.

For teachers and mentors

Have the group compute 0.99^k for k = 10, 50, 100 and 300 in a spreadsheet, then imagine a duck with γ = 0.9, whose horizon is 10 ticks, 0.2 s. What could it never learn to set up?

A horizon is like how far down the road a driver looks: a pothole inside it gets steered around, and one far beyond it barely registers. The horizon belongs to the training objective, not to the network, since the deployed policy just maps an observation to an action.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

γ = 0.99: "gamma=0.99" in the walking task's settings.
50 Hz: policies "are trained here at 50 Hz" (microduck_rl README). 61 and 14: "obs[1,61] → actions[1,14]" (robotd design).
20 ms, 0.37, 100 ticks and 2 s are arithmetic on those two numbers, as are 10 ticks and 0.2 s for γ = 0.9, and 0.13 at 200 ticks.

Lesson 2 · Credit

Value, advantage and GAE

Which earlier action earned a reward? PPO, the method in lesson 3, answers with two networks: the actor is the policy, and the critic learns V(s), the return the actor can expect from state s, so each action is judged against what was expected rather than against zero.

That judgment is the advantage, and one step of it is the temporal-difference (TD) error δ_t. Generalized advantage estimation blends the errors that follow, each weighted by (γλ)^k. Microduck uses λ = 0.95: a low λ leans on the critic's guesses, a high one on what really happened.

The critic may also see more than the actor. In the walking task only the critic sees the trunk's linear velocity, and it gets every signal clean and on time, while in the ball kick only the critic sees the ball.

δ_t = r_t + γV(s_t+1) − V(s_t), and Â_t = Σ_k (γλ)^k δ_t+k

Predict

The ball-kick critic sees the ball and the actor does not. What runs on the robot?

Both networks. Only the critic. Only the actor.

Check your prediction

Only the actor. The critic exists only in training, so it is allowed to cheat, and the exported graph is the actor, 61 inputs to 14 outputs. The critic's privileged inputs never have to exist on the robot, which is exactly why it can be given them.

The figure

Value. Posts along the duck's path rise to the critic's estimate of the return from each spot.
Surprise. Each step leaves a TD error: green where things went better than the critic expected, red where worse.
Blend. For one step, the errors after it are weighted by (γλ)^k and summed into its advantage.
Export. The critic gets an extra eye on the ball. At export it folds away, and only the actor ships.

Drag λ to blend more or less of the future into the advantage. The values and errors are made up; γ and λ are Microduck's.

Big idea: the critic turns "was that good?" into "was that better than expected?", and because it never ships, it may see more than the robot can.

Try it for real

Find the privileged terms in the training code; the walking task gives one to the critic alone:

# Add base_lin_vel to critic only (privileged information)
cfg.observations["critic"].terms["base_lin_vel"] = ...

The ball kick's docstring explains its own choice: the robot has no ball sensing, so the actor is ball-blind and only the critic sees the ball.

For teachers and mentors

Give each pair a row of rewards and a guessed V, and have them compute δ by hand for a few steps, then Â with λ = 0 and with λ = 1. The first is one-step TD, and the second is the sampled return minus V.

An advantage reads a step the way a coach reads a lap time, against the runner's usual rather than against zero. Asymmetric actor-critic is a general trick on top of that: train with what the simulator knows, deploy with what the robot senses.

Each advantage sums at most 24 sampled ticks, one rollout, and the critic's estimate stands in for everything after. That is how the horizon of lesson 1 reaches past the end of a rollout.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

λ = 0.95: "lam=0.95" in the walking task's settings, which also say "Add base_lin_vel to critic only (privileged information)".
The ball: "The CRITIC does see ball pos/vel (asymmetric actor-critic)" (ball kick settings).
61 and 14: robotd design. λ = 0 and 1 are the two ends of the range.

Lesson 3 · The update

PPO: better, but not too much

How far should one batch of experience move a policy? Each PPO iteration collects a batch with the current policy, then makes several passes over it: Microduck's runner takes 24 steps from each of 4,096 environments, 98,304 transitions, and makes 5 epochs over them in 4 minibatches.

Reusing data is risky, because the policy can drift far from the one that collected it. So PPO measures the drift per action with the ratio ρ = π_new(a|o) / π_old(a|o), and its clipped objective stops paying once ρ leaves 1 ± ε in the helpful direction, with ε = 0.2 for Microduck.

L = E[ min( ρÂ, clip(ρ, 1 − ε, 1 + ε) Â ) ], with ε = 0.2

Predict

An action had a positive advantage, and the update has already raised its ratio to 1.3. What does pushing it further earn?

More objective, at the same rate as before. Nothing: past 1.2 the objective is flat. A penalty that pulls the ratio back to 1.2.

Check your prediction

Nothing. For Â > 0 the objective is ρÂ only up to 1 + ε = 1.2 and flat after that, with zero gradient, and for Â < 0 it flattens below 0.8.

The figure

The ratio. The axis is ρ: how much likelier the new policy makes a recorded action. At 1 nothing has changed.
A good action. For Â > 0 the objective climbs with ρ, then goes flat at 1.2, and the ball stops being pushed.
A bad action. For Â < 0 the objective rises as ρ falls, and goes flat below 0.8.
The passes. 5 epochs of 4 minibatches make 20 small steps on one batch, while a meter watches the KL against its 0.01 target.

Drag ε to widen or narrow the clip. The curves are the real objective for Â = ±1; the KL meter's level is drawn.

Big idea: PPO takes the improvement the batch supports a small step at a time, and never bets the policy on any one action.

Try it for real

On a run, plot the KL and the learning rate together, and watch the adaptive schedule back the rate off whenever the KL runs hot. The settings, from the walking task's runner:

clip_param=0.2,
entropy_coef=0.01,
num_learning_epochs=5,
num_mini_batches=4,
learning_rate=1.0e-3,
schedule="adaptive",

Around the clip, the runner sets the rest: the learning rate starts at 0.001 and adapts, cut when the KL divergence runs well above its 0.01 target and raised when it runs well below. The entropy bonus is 0.01, the policy is a Gaussian whose noise starts at a standard deviation of 1.0, and gradients are clipped at norm 1.0.

For teachers and mentors

Draw min(ρÂ, clip(ρ, 0.8, 1.2)Â) for Â = 1 and Â = −1 on squared paper, then ask which side of each curve goes flat, and why that is the safe side.

The clip works like a bonus with a ceiling, set on how far one update moves the policy: once the probability ratio passes 1 ± ε in the helpful direction, moving it further earns nothing, but nothing is clawed back either. The entropy bonus is a reward for staying uncertain, and without it a policy can stop exploring before it has found a gait.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

ε, epochs, minibatches, learning rate, KL target, entropy, starting noise, gradient norm and 24 steps: the runner settings in the walking task.
4,096 environments: "a usable gait at 4096 envs" (microduck_rl README).
98,304 is 4,096 × 24; 20 is 5 × 4; 1.2 and 0.8 are 1 ± 0.2. The 1.3 in the question is an example.

Lesson 4 · The interface

61 numbers in, 14 out

Every Microduck policy has one shape, obs[1,61] → actions[1,14], and the robot's daemon builds one observation for all of them in this order: gyro (3), projected gravity (3), joint positions (14), joint velocities (14), last action (14) and command (13). Because the shape never changes, the runtime can hot-swap walk, stand and trick policies. In training, an action is a joint target, the home pose plus 1.0 × the output, but the daemon's defaults scale walking actions by 0.9 and smooth the head and leg targets, so Pollen's two repositories do not agree on this yet.

Training also copies the robot's flaws on purpose: joint velocity arrives 1 tick late, the IMU (the motion sensor) lags 0 to 1 tick, joint readings carry a fixed bias of up to ±0.015 rad, and the IMU a mounting tilt of up to 6°.

The same 61 inputs and 14 outputs, in the same order, for every policy.

Predict

Your new trick never uses the head command. What do you do with its 4 inputs?

Delete them, to make the network smaller. Keep them, and feed them plain zeros. Keep them, with small sampled values.

Check your prediction

Keep them, with small sampled values. Deleting them breaks the contract, so unused command slots stay in, and Pollen's notes say to sample small values in them from step 0, because a command input that is never non-zero keeps dead weights forever, though the ball kick, for one, feeds plain zeros. Big random noise is no answer either, since it is not small and no robot will ever send it.

The figure

61 in. The observation lights up group by group: 3, 3, 14, 14, 14 and 13.
14 out. In training, the outputs become joint targets, the home pose plus 1.0 × the action, and the duck's joints follow.
Late on purpose. Joint velocity slides a tick behind, the IMU wobbles between 0 and 1 tick late, and the joint readings carry a small bias.
Hot swap. A kick policy slides in where the walk was, same plug, then hands back.

The order and sizes are the robot's own. The signals drawn on the bars are made up.

Big idea: a fixed contract lets policies trade places mid-stride, and every flaw trained in is one the robot cannot surprise it with.

Try it for real

In Microduck Studio, Probe this network lets you change one input of a policy and compare the actions, and each input's z-score shows how far it sits from the training distribution.

On a computer, rehearse the runtime's swaps with Pollen's tool:

uv run scripts/infer_policy.py --walking walk.onnx \
  --standing stand.onnx --sitstand sitstand.onnx \
  --roulade roulade.onnx --new-cmd-obs

For teachers and mentors

Lay the 61 slots out on squared paper with their index ranges. The command is velocity 3, head 4 and body 6, at indices 48 to 60 as the daemon's own notes list it, and joint velocity is a tick old because the servo firmware computes it as a moving average.

The contract works like a wall socket: any policy with the right plug can be swapped in, and one that changes the plug fits nothing. The actor behind the plug is small, with hidden layers of 512, 256 and 128 units and 197,774 weights and biases in all.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

The order, the sizes and indices 48 to 60: "[ gyro(3) | projected_gravity(3) | joint_pos(14) | joint_vel(14) | last_action(14) | command(13) ]" and "command = vel(3) + head(4) + body(6)" (robotd design).
Scale 1.0, the lags, the bias and the tilt: the walking task's settings. The daemon's 0.9: "Unset resolves per mode: 0.9 walking" (robotd.toml). Dead weights: AGENTS.md.
197,774 is 61 × 512 + 512 + 512 × 256 + 256 + 256 × 128 + 128 + 128 × 14 + 14.

Lesson 5 · Rewards

The reward is the specification

RL optimizes the letter of the reward, not its spirit: PPO reads a reward the way a lawyer reads a contract, looking for the cheapest way to comply. Pollen's training notes keep hard-won rules for writing one:

Get the signs right. A cost that returns 0 or more takes a negative weight, and a penalty that is already negative, returning 0 or less, takes a positive one. Mix them up and the policy is paid to misbehave.
No jackpots, and no pay for a bad state. Limit how often a "reach X" reward can pay, and never pay for just being in a bad state: pay the change in a potential instead, a score like cos(tilt), so rising pays and holding still pays zero.
Judge a term by its share of the total. The same weight counts a quarter as much when the task reward is 4 times bigger.

Pay progress, γΦ(s_t+1) − Φ(s_t), not presence.

Predict

A get-up task pays a point every tick the duck is low and trying. What does PPO find?

Getting up as fast as it can. Staying low forever, collecting. Getting up, then falling to do it again.

Check your prediction

Staying low. The cheapest state that qualifies gets parked in and farmed, which is exactly what the reward asked for.

The figure

Gated. The left duck is paid every tick it is low. It finds the cheapest low pose and stays there, and its total keeps climbing.
Potential. The right duck is paid the change in cos(tilt). Rising pays; once it stands, the payments stop.
Double negative. A penalty that is already negative gets a negative weight, and its bar jumps above zero: a prize for the violation.
The check. Flip the sign, and every penalty bar sits at or below zero again. One look catches it, on every run.

Toy totals: the shapes of the two rewards are the point, not their sizes.

Big idea: write the reward so the cheapest way to score is the behaviour you want, and check its signs on every run.

Try it for real

Turn the sign rule into a test before training. microduck_rl's workflow asks for config tests that check "reward weights have the intended sign" and run on a CPU:

uv run --with pytest pytest tests/

Then look at each Episode_Reward/<penalty> curve on every run, because all of them must stay at or below zero.

Then look at the sprint board in Pollen's Arena, where the leaders get a running start in the untimed settle and end in a dive across the line, inside the time the rules give a tilted trunk before they call a fall. Race in Pollen's Arena has the run we read and the rule it meets.

For teachers and mentors

Run a reward-hacking round: each group writes a reward for a trick in plain words, and another group finds the laziest way to score it. The training notes' own catches include "ballistic whip instead of a roll" and "head-tripod instead of standing".

Potential-based shaping pays γΦ(s′) − Φ(s), and with γ in it every path between two states earns the same discounted total, so the best behaviour stays the best (Ng, Harada and Russell, 1999). Pollen's upright reward pays the plain change, which is close when γ is near 1.

One more rule from the same notes is about timing: add smoothness penalties only after the skill exists, or doing nothing wins.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

Every rule here, the zero check and the 4× example are from the reward design section of AGENTS.md.
The figure's totals are invented for the picture.

Lesson 6 · Hardware

The actuator is most of the gap

Pollen's training repository says it plainly: at this scale, "actuator fidelity is most of the sim2real gap." That scale is a robot of about 800 g, whose policies drive 14 small Dynamixel XL330 servos, with a 15th for the beak.

So instead of an ideal PD controller, a spring and damper toward the target that always gets the torque it asks for, training models each servo with BAM's M6 model: the voltage control law, back-EMF, and Coulomb, Stribeck and load-dependent friction. Back-EMF makes torque fall as the motor speeds up, and friction makes a slow joint stick, then break away. Each environment draws its own battery voltage, sag, command delay and friction.

Backlash variants add ±1° of gear play, 2° in all, at every servo joint, read through the play as the real encoder on the output side reads it.

A servo is not an ideal PD controller: its torque falls with speed, and it sticks when slow.

Predict

The battery runs down. What happens to the line of torque the servo can give at each speed?

Nothing: the controller makes up for it. Only the top speed falls; stall torque stays. The whole line drops, at every speed.

Check your prediction

The whole line drops. A lower battery voltage leaves less torque at every speed, so a tired battery makes a weaker servo, where an ideal controller's flat line would not move at all. The actuator is most of the gap, which is why the simulator models the XL330 down to its voltage control law and each environment draws its own battery voltage.

The figure

Ideal. An ideal controller gives the asked torque at any speed: a flat line, and a horn that snaps to where it is told.
Back-EMF. A real motor's torque falls as it speeds up, to nothing at its top speed. A lower battery drops the whole line.
Friction. Near zero speed the joint sticks until enough torque breaks it free.
Backlash. Between motor and output there is play: the output can wobble ±1° while the motor holds still.

The curves show shapes, not the XL330's measured numbers. Drag the battery slider to see what a weaker battery does.

Big idea: model the motor you have, not the one you wish you had, or the policy learns to count on torque that is not there.

Try it for real

Rehearse one policy on a computer with and without the servo model. Pollen's rehearsal tool uses the BAM actuators by default, --no-bam falls back to plain PD actuators, and --vin pins the battery voltage:

uv run scripts/infer_policy.py --walking out.onnx --new-cmd-obs
uv run scripts/infer_policy.py --walking out.onnx --new-cmd-obs --no-bam

Keep --new-cmd-obs on both lines, because without it the tool builds the older 51-number observation and refuses a 61-input policy.

For teachers and mentors

Compare a spring with a heavy box. A spring pushes back the moment you move it, like an ideal controller, but a box does not move at all until you push hard enough, and then it lurches. That is friction, and slow joints live in it.

BAM is Rhoban's open project, "better actuator models".

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

About 800 g, the sim2real sentence, the BAM model, the randomized quantities and the backlash: the microduck_rl README.
14 Dynamixel XL330 servos: AGENTS.md. Fifteen servos in all: robotd design. The 51-number observation: scripts/infer_policy.py.
The figure's curves are drawn to show their shapes only.

Lesson 7 · Robustness

Randomize, then measure

A simulator is wrong in ways nobody can list, so the walking task trains across many wrong worlds. Each duck keeps its own foot friction (×0.7 to 1.3) and trunk mass and inertia (±5%), each reset redraws the trunk's center of mass (up to ±15 mm) and joint friction (×0.9 to 1.1), and pushes of ±0.3 m/s come every 3 to 6 s.

Randomizing is a hypothesis, though, and testing it is a separate step. We tested cards meant to walk one square, played in rows (a row is a program of Duck Cards), with Pollen's v8 policy in duck-sim, under test ranges of our own fixed first. Of the short rows, 95 to 98% landed within 10 cm and 15°, but of rows of 10 cards or more, only 33 to 42%. duck-sim drives plain position servos, not lesson 6's servo model.

Randomize what you cannot know; measure against criteria fixed before looking.

Predict

Why do long rows miss so much more often than short ones?

Where it thinks it is drifts with time. Long rows have more turns to get wrong. The battery runs down as the row goes on.

Check your prediction

Drift. The duck's belief about where it is drifts with time and with distance walked, so a longer row ends further off. We split long rows into chapters of up to 7 cards, with the duck re-seated between them on a nest, the square where the child sets it back down, and that lifted long rows from 42% to 63%. A mat under each nest, printed for the duck's head camera to read as a compass, took them to 97.5% in simulation.

The figure

Shake. Every practice duck gets its own floor grip and its own pushes.
A long row. The duck's belief about where it is, the ghost, drifts away from where it really is.
Chapters. Split the row at nests. Re-seating the duck resets the drift, so it builds over one chapter, not the whole row.
Measure. Long rows that landed, as measured: 37% for the shipped loop, the card player Studio first shipped; 42% with merged cards, a pair of walks run as one; 63% with nests; 97.5% with the mat on the nest.

Ranges from the walking task; the long row's path is drawn. Landing rates from the evidence page: Pollen's duck-sim under Studio's own test ranges, at Studio's 10 Hz state rate.

Big idea: train across worlds you cannot rule out, test on conditions you fixed in advance, and report the part that fails.

Try it for real

Read the evidence page from its criteria to its limits, then follow its reproduce section, since every raw row is published.

In the training code, most randomizations are ENABLE_* switches at the top of the task's settings with their ranges beside them, and the battery, sag and delay ranges live in microduck_constants.py.

For teachers and mentors

Before any test, have the group write down what "walks one square reliably" means, in numbers, then compare with the evidence page's: 95% of rows within 10 cm, 95% within 15°, a fall bound of 1%, and time limits.

Randomization is a flight simulator's trick: a pilot trains in many kinds of weather, because nobody knows which one the real flight will bring. Be clear about what is still open, too: the mat result is simulation only, and closed-loop cards stay off on real ducks.

And note the randomizer rule: a draw that accumulated across resets once degraded every long training run for months.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

Friction, center of mass, mass, joint friction and pushes: the walking task's settings. The accumulating randomizer: AGENTS.md.
Every landing rate, the criteria, the 10 Hz rate and the chapters: the evidence page.

Lesson 8 · Evidence

Honest numbers

No falls in 100 rows (a row is a program of Duck Cards) does not mean a fall rate of zero, so how many clean rows does it take to bound the rate at 1%? The Wilson 95% interval is a range that should hold the true rate 95 times in 100, and with 0 falls, that takes 381 rows.

Rows are rarely independent, either. On the evidence page each condition's physics is drawn once per session of 12 rows, "so 400 rows are about 34 ducks", and intervals that count rows come out too narrow, so later candidates were judged by a permutation test over sessions.

And decide what counts before you look: those criteria were written down on 2026-10-15, "before any result was in, and not changed since."

For 0 events in n trials, the Wilson 95% upper bound is z²/(n + z²), with z = 1.96.

Predict

No falls in 100 rows. What is the upper end of the Wilson 95% interval on the fall rate?

About 1.0%. About 1.9%. About 3.7%.

Check your prediction

About 3.7%. 1.96² / (100 + 1.96²) = 0.037, so the interval runs up to 3.7% at n = 100, while the rule of three, 3/n, a one-sided bound, says 3%.

The figure

No falls. 100 rows, every one clean.
The interval. The fall rate could plausibly be anywhere from 0 up to 3.7%.
More rows. The count climbs to 381, and the bar shrinks until it touches the 1% line.
Sessions. The rows clump into sessions of 12 that share a duck: 400 rows are about 34 ducks. Counted by duck, no falls bounds the rate at about 10%. The honest interval lies between the two.

Drag n to move the bound. The bound is the real Wilson formula; the sessions are the evidence page's.

Big idea: report the interval, count what is really independent, and decide what counts before you look.

Try it for real

The bound for zero events is one line of Python:

z = 1.96
for n in (100, 381, 1000):
    print(n, z * z / (n + z * z))

Then read how the evidence page compares candidates session by session rather than row by row.

Pollen's Arena shares one part of this: a single run is not a result there either. It times every entry over several starts, counts a fall as the time limit, and replays the median run, not the best. Race in Pollen's Arena shows what one bad start does to a score.

For teachers and mentors

Before an experiment, have the group write the criteria and the analysis on paper, seal it, and open it afterward: pre-registration is that envelope.

The rule of three is a quick check for zero events, and Wilson is what the evidence page uses.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

"the upper end of the Wilson 95% interval on the fall rate is at most 1% (with zero falls, that takes 381 rows)", the sessions of 12, the 400 rows and 34 ducks, the permutation test and the date: the evidence page.
3.7%, 0.037, 3% and the 10% for 34 ducks are arithmetic; 1.96 is the normal quantile a two-sided 95% interval uses; 1000 in the snippet is an example.

Lesson 9 · People

Learning from people

Some of what makes a duck good company will not fit in a formula, so people supply it.

Episodes. Kith, a web game by this site's author, records an agent's own view five rows a second: what it sensed, what it did next, and its reward under a published schedule. It can release the walks people give as an open LeRobot dataset, though so far its release holds only scripted walks.

Picks. Studio's Compare plays two runs side by side and you pick the better, which the docs call "the human-feedback half of RLHF", reinforcement learning from human feedback. Picks can train a scorer learned from picks, which can then shape a policy's reward.

Poses. Mimic puts the duck in the pose of whoever stands in front of the camera.

P(A picked over B) = σ(S_A − S_B), where S is a run's score and σ is the logistic curve, which turns a gap in score into a chance.

Predict

What do your picks in Compare train on your phone?

Nothing; the picks are only labels. A taste that re-weights Pollen's reward. A walking policy tuned to your taste.

Check your prediction

A taste that re-weights Pollen's reward. With enough picks, Studio's Train from your picks fits your taste on the phone as a simple scorer learned from picks, then scales Pollen's reward terms up or down, within bounds, for a training run off the phone. So it re-weights the reward Pollen wrote rather than replacing it, and Settings decides whether the picks may be shared.

The figure

Episodes. A walk is recorded as rows, five a second: what the duck sensed, what it did, what it earned.
Picks. Two runs side by side. A person picks one, and the pick drops onto a pile of labels.
A scorer. On the phone, the labels fit a curve: the more one run outscores the other, the likelier it is picked.
Poses. A person strikes a pose, and the duck copies it.

Five rows a second is Kith's. The scorer curve is the standard logistic one, not fitted to real picks.

Big idea: people can teach what a reward cannot say: by walking along, by choosing, and by showing.

Try it for real

Walk with a duck in Kith, whose published rewards name the skill each one is meant for, and several say no policy exists yet. Then judge two moves in Studio's Compare, and make one with Mimic.

For teachers and mentors

Run a pick tournament: two moves on a screen, a show of hands, a tally. Then ask what the tally would teach a scorer, and what it would miss.

A scorer learned from picks inherits the pickers' blind spots, and a policy trained against it can exploit them, exactly as it would a hand-written reward.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

"a table of your own view at five rows a second" and the open LeRobot dataset: Kith's notes for agents. The scripted first release: Kith's dataset notes.
Compare, picks and Settings: Studio's docs. Mimic: Duck Cards.

Lesson 10 · Shipping

Ship it, and say what it does

What has to happen between a config file and a robot? These gates, each one cheap next to a wasted run or a fallen robot:

Smoke test: 64 environments, 5 iterations; it "catches ~95% of config errors for cents".
Config tests on a CPU: indices, signs, gates.
Train and read the run: penalties ≤ 0, the task term growing.
Export with scripts/export.py, normalizer baked into the ONNX.
Rehearse in scripts/infer_policy.py, swaps and all.
Publish: checked, then uploaded, private until you say so.
Measure where runs can be reset and scored, and share the EvalLog, one file that keeps the results beside the whole specification.

Rollouts are the policy's real runs: "Report what rollouts actually show." Whoever asked decides when it is good enough.

Predict

Publishing refuses your policy. What could the reason be?

It walks slower than Pollen's walk. Some of its outputs are NaN. Nobody has run it on a real robot.

Check your prediction

NaN outputs. Publishing checks the shape, [1,61] → [1,14], and refuses a file with NaN or constant output, but whether the policy is any good is a measurement, an EvalLog, not a publishing check.

The figure

Smoke test. 64 small ducks run 5 iterations, and a config error is caught before it costs a night.
Train and read. The run's chart: every penalty below zero, the main task term rising.
Export and rehearse. The policy passes through the normalizer into one ONNX file, then rehearses its swaps on a computer.
Publish. The gate checks the shape and the outputs. A broken file is turned away; the good one goes up, private.
Measure. The run is scored and written to an EvalLog that anyone can check.

The gates and their numbers are microduck_rl's and Microduck Studio's; the run's chart is drawn.

Big idea: every gate is a cheap question asked before an expensive answer.

Try it for real

The whole path, with Pollen's tools:

uv run train <TASK_ID> --env.scene.num-envs 64 --agent.max_iterations 5
uv run --with pytest pytest tests/
uv run train <TASK_ID> --env.scene.num-envs 4096
uv run scripts/export.py <TASK_ID> --wandb-run-path <...>
uv run scripts/infer_policy.py --walking output.onnx --new-cmd-obs
uv run publish --onnx output.onnx --repo <user>/microduck-<name> \
  --kind perpetual --slot walk --dry-run

With --dry-run, the last line shows what would be uploaded without touching the Hugging Face Hub, and without it the same line publishes a walk. A trick publishes with --kind episodic --duration-s 4.0 instead. With a duck, sudo robotctl policy load walk puts a gait in the walk slot and sudo robotctl policy add <name> <repo> installs a trick, which robotctl robot do <name> runs once the robot is driving; without one, Studio's evaluations score a network on a bench and write an EvalLog.

For teachers and mentors

Put the gates on the wall as a checklist, the way pilots keep one for take-off, and make "what did the rollout actually show?" the first question at every review, before anyone says "it works".

An EvalLog carries its whole specification beside its results, so a number can be traced to the run that made it.

Something in this lesson unclear, or wrong? Tell us about it.

Where these numbers come from

64 environments, 5 iterations and ~95%: "A 5-iteration smoke test at 64 envs catches ~95% of config errors for cents." "Report what rollouts actually show" and the smoke-test command are from the same AGENTS.md.
[1,61] → [1,14], NaNs, constant outputs, private repositories, the commands and their numbers: the microduck_rl README.
EvalLog: Studio's docs.

Next: Book 4, How a Policy Became, traces real training runs to why they turned out the way they did. Before this: Book 2, How a Microduck Learns, has the same ideas without the formulas, and Book 1, Meet the Microduck, starts from a body, senses and a brain. All four books are on the shelf.