Microduck StudioIndependent project

Microduck Studio books · Book 4 of 4 · Casebook

How a Policy Became

Why does a trained duck do what it does? Real training runs answer from their notes and logs. Ten lessons, each with a figure to run.

Who it is for: readers of Book 3, and anyone who wants to know why a trained policy turned out the way it did. The runs are ours, made with duckbatch, an open project that fine-tunes, in simulation, Pollen's kick and a walk distilled from Pollen's, and none of them has run on a physical duck.

  1. The walk that learned to stand
  2. Pay for going
  3. More practice, more standing
  4. Read the run, not the menu
  5. One change at a time
  6. The teacher's pull
  7. Seven lines of eight
  8. Aim, not distance
  9. A new sense on the bus
  10. The contract travels

How a lesson works: read the left page and commit to a prediction, then run the figure on the right page one step at a time and check yourself. Each lesson ends with something to try, notes for whoever is teaching it, and the source of every number.

Lesson 1 · b003, the kernel

The walk that learned to stand

Ask either walker for 0.10 m/s and it stands still: Pollen's walker moves at 0.002 m/s, and duckbatch's 26,254-parameter student at 0.007. Asked to turn on the spot at 1.0 rad/s, the student manages 0.211. Our first fine-tune, b003, set out to close this dead band.

The walking task pays for tracking its command with a kernel, exp(−e2/σ2) of the gap e, and at Pollen's width, σ = 0.316 m/s, standing still when asked for 0.10 collects about 90% of the pay. So b003 narrowed σ to 0.1, narrowed turning's width too, and fine-tuned for 1,500 iterations.

The duck stood more: 0.060 m/s when asked for 0.30, against the student's 0.108, and 0.017 rad/s of turning, yet its logged tracking reward rose over the run. At σ = 0.1, managing 0.10 of a 0.30 command earns exp(−4) ≈ 0.018 of the pay, and standing exp(−9) ≈ 0.0001, so a first step paid nothing while other terms still paid for standing.

pay = w · exp(−e2/σ2): the narrower σ, the less a step toward the command pays.

Predict

What did b003's duck do when asked for 0.30 m/s?

Check your prediction

Moved at 0.060, slower than the student. A narrow kernel pays almost nothing until the duck is near the command, so its first steps earned nothing and PPO followed the terms that pay for standing. As for the rising log, the close note's reading is that near-zero commands dominate it, and standing meets them.

The figure

  1. Ask. Three ducks are asked for 0.30 m/s. One stands, one manages 0.10, one walks the command.
  2. Wide kernel. At σ = 0.316 the pay climbs all the way: 0.41 for standing, 0.67 for 0.10, all of it at 0.30.
  3. Narrow kernel. At σ = 0.1 the slope is gone: 0.0001 for standing, 0.018 for 0.10.
  4. What PPO did. The student's speeds and b003's, command by command: slower everywhere, and hardly turning.
  5. What the log said. The kernel lies down and b003's own record of its tracking pay stands up: rising over the run, while the final duck was slower at every command.
Drag σ to reshape the kernel. The widths, speeds and log are b003's; the ducks are drawn.

Big idea: a reward teaches through its slope. Take away the pay for partial progress, and the cheapest thing left to learn is standing still.

Lesson 2 · b003b, a progress term

Pay for going

b003's note named the fix: keep the kernel's slope, and stop paying for standing. So b003b put Pollen's widths back and added two progress terms at weight 1.0 each: the share of the commanded speed, and of the commanded turn, that the duck achieves. Standing when told to move earns 0 from them; half the command earns 0.5.

We ran a 100-iteration pilot first, to check that the terms run. They did, but it was worse than the student, 0.96 falls a minute against 0.36. The note called that the usual early cost of a warm start, though not yet known to be only that, and the pre-registered full 1,500 went ahead.

The full run turned on the spot at 0.456 rad/s, 2.2× the student, and reached 0.183 m/s when asked for 0.40, which is past Pollen's walker. At 0.10 m/s it still stood, at 0.008 against the student's 0.007. At 0.30 it reached 0.1199 against a line of 0.12, inside the run-to-run noise, and the note recorded a fail.

progress = clip(v · ĉ, 0, |c|) / |c|: pay in proportion to the share of the command achieved, and nothing for standing.

Predict

What did b003b do at 0.10 m/s?

Check your prediction

Moved at 0.008, about the student's speed. A reward for going slowly was there, but PPO did not find a slow gait. Above the dead band everything improved and falls stayed the same, and the note credits the progress terms for that, though no run tried Pollen's widths without them.

The figure

  1. Two terms. For a 0.30 m/s command: tracking at Pollen's width, weight 2.0, and the new progress ramp, weight 1.0, which pays nothing for standing.
  2. Pilot. 100 iterations to check the terms run: they do, and falls jump to 0.96 a minute.
  3. Full run. The student against b003b: turning on the spot more than doubles, and 0.40 m/s passes Pollen's walker.
  4. The dead band. At 0.10 m/s both stand: 0.007 and 0.008. Nothing changed there.
  5. The line. 0.1199 against 0.12: inside the noise, and still a fail.
Drag the speed to see what each term pays for it. The weights, speeds and lines are b003b's; the duck is drawn.

Big idea: pay for partial progress gives learning a slope to climb, but only where the duck can already take a step.

Lesson 3 · b003c, the practice mix

More practice, more standing

Was the duck short of practice at walking slowly? Pollen's task draws most walking commands evenly from ±0.4 m/s forward and ±0.3 sideways, and a fifth straight ahead at 0.3 or more, so only about 12% land under 0.15 m/s. b003c changed one thing from b003b: 35% of the envs neither standing nor turning in place got a slow command, 0.05 to 0.25 m/s.

The duck stood more: at 0.10 m/s it moved at −0.001, turning on the spot fell from 0.456 to 0.299 rad/s, and walking at 0.30 m/s from 0.120 to 0.066.

The close note did the sums: at a 0.10 command the tracking term, weight 2.0, pays exp(−0.01/0.1) ≈ 0.90 of itself for standing still, which is 1.8 a step against at most 3.0 for walking perfectly, so more practice in the band was more practice at a choice standing nearly wins.

At 0.10 m/s, standing earns 2.0 × 0.90 = 1.8 a step; walking it perfectly earns 2.0 + 1.0 = 3.0, before any risk.

Predict

b003c gave the dead band more practice. What did the duck do at 0.10 m/s?

Check your prediction

Stood: −0.001 m/s. The note's verdict was that the dead band is not short of practice, but that standing is too well paid there. b003b's turning and fast walking were lost too, which the note put down, as far as its numbers went, to less practice above the band.

The figure

  1. The draw. Of the envs asked to walk, about 12% get a command under 0.15 m/s.
  2. More in the band. b003c gives 35% of them a slow command instead, 0.05 to 0.25 m/s.
  3. The sums. At 0.10 m/s, standing earns 1.8 a step and walking perfectly 3.0, if it does not fall.
  4. The result. b003b against b003c: slower at 0.10 and 0.30, and turning lost.
The shares, sums and speeds are b003c's; the practice envs are drawn.

Big idea: more practice at a choice does not change what the choice earns. Work out what each choice earns, and what the practice really holds, before adding more.

Lesson 4 · What ran

Read the run, not the menu

Each run leaves its records beside its verdict: the menu that asked for it, a TensorBoard log of every reward term and of the settings the curriculum manages, a checkpoint every 250 iterations, and the exported network with its measurements. Where the log records a setting, it says what actually ran.

The menus of b003, b003b and b003c each set rel_standing_envs: 0.05, 5% of the practice told to stand still, but b003d's design note found that the log's Curriculum/standing_envs read 0.25 at every iteration. Pollen's walking task has a curriculum for that share, which sets it at every reset, so the menu's value never took effect.

So about a fifth of all practice had been "do not move", in runs meant to teach moving slowly. We kept 0.25 on purpose in b003d, b003e and b003f, and said so, so that each still changed one thing, and b003g, in lesson 7, finally ran the 5%.

Where the log records a setting, the log is what ran, not the menu. Read those values before the results.

Predict

The menus asked for 5%. What did the log say ran?

Check your prediction

25%, at every iteration. The task's curriculum set the share at every reset, over the menu's value, and nothing failed: the menu still said 5%, and the log is where it showed. Turning in place then took a quarter of the envs, which left about 19% standing.

The figure

  1. The folder. One run's records: the menu, the log, the checkpoints, the network and its measurements.
  2. Asked. The menu: rel_standing_envs, 0.05.
  3. Ran. The log's Curriculum/standing_envs: 0.25, at every one of 1,500 iterations.
  4. The mix. Of every hundred envs, 25 are drawn standing. Turning in place takes a quarter and frees some of them, which leaves about 19.
The menu line and the log are b003's own; the folder and the envs are drawn.

Big idea: the menu is a request. Before you read a run's results, read in its log what it actually did.

Lesson 5 · b003d and b003e, the stride

One change at a time

From b003c on, we planned each run in a design note first: one change from a named earlier run, usually b003b, the best so far; the series' pass lines; and an expectation to beat. b003d judged tracking relative to the command, so standing at 0.10 m/s paid 0.20 of the term instead of 0.90. Its note said, "I expect it to pass." It moved at 0.004.

Its close note found a candidate: the tracking error is taken on the duck's instantaneous velocity, and a stride sways the body 0.10 to 0.15 m/s, mostly sideways, which is more than the command. So under b003d's width, stepping in place, the first move away from standing, paid less than standing. The first step was downhill.

b003e judged the velocity averaged over 0.5 s, where the sway falls to 0.03 to 0.05 m/s, and its design added a weaker bar, 0.02 m/s at 0.10, for whether the idea took at all. It did not, at 0.003, but it turned on the spot at 0.525 rad/s, the first run to pass that line.

One change per run, the same lines, and an expectation written down first: then even a failure says something.

Predict

Without the sway, did b003e walk at 0.10 m/s?

Check your prediction

No: 0.003, though it turned better than any run before. A diagnosis can be right and still not be the blocker: with the dip gone and the band still shut, the reward was no longer the likeliest suspect, as b003e's design had said in advance.

The figure

  1. The stride. A walking duck's body sways 0.10 to 0.15 m/s within each stride, mostly sideways.
  2. The instant. Judged at each instant under b003d's width, stepping in place pays 0.09 to 0.17 against standing's 0.20: downhill.
  3. The average. Over 0.5 s the sway falls to 0.03 to 0.05, and stepping in place pays 0.21 to 0.25: uphill from the start.
  4. The result. b003e at 0.10 m/s: 0.003. The band stayed shut, and turning passed at 0.525.
The sway, the pay ranges and the speeds are from b003d's and b003e's notes; the duck's stride is drawn.

Big idea: find a cause, remove it, and measure again. If nothing moves, what you found was real, but it was not what held the band shut.

Lesson 6 · b002 and b003f, the teacher

The teacher's pull

Every walking run started from duckbatch's student: 61 inputs, two layers of 128, 14 outputs and 26,254 parameters, 7.5× smaller than Pollen's 197,774-parameter walker. It was distilled: the student drove, and the teacher labelled every state its mistakes led to. It tracks speed within 5% of the teacher and turning within 11%, but falls about three times as often under pushes.

Fine-tuning kept it tied to the teacher by an anchor: after every PPO update, a pass that pulls the student's actions toward the teacher's, at weight 1.0 on fallen frames and 0.2 on upright frames commanded to walk at 0.35 m/s or more, or to stand. The dead band was left free.

b003f set the upright weight to 0. Turning on the spot went from 0.456 to 0.806 rad/s, 2.6× Pollen's walker, with recovery at 99.6%. The dead band did not move: 0.004 m/s. The note's reading: the teacher had been a ceiling on turning, not the lock on walking slowly.

anchor = w · mean((π(o) − πteacher(o))2): it keeps the teacher's skills, and its limits.

Predict

Without the upright pull, what happened to turning?

Check your prediction

It rose to 0.806 rad/s. The pull never touched a turn-in-place frame, but it held the shared network to the teacher on standing and fast-walking frames, and the teacher turns at about 0.31. The pull on fallen frames stayed, so getting up held.

The figure

  1. Distilled. The student drives and the teacher labels what it sees: 26,254 parameters learning from 197,774.
  2. The anchor. After each update, a pull back toward the teacher: 1.0 when fallen, 0.2 when walking fast or told to stand, none in the dead band.
  3. Let go upright. b003f sets the upright pull to 0, and keeps the pull on fallen frames.
  4. Turning. From 0.456 to 0.806 rad/s, far past the teacher's 0.312, once nothing held the network to the teacher while upright.
The weights, sizes and speeds are duckbatch's; the ducks and the ropes are drawn.

Big idea: a teacher you are tied to is a floor and a ceiling. Keep the rope where you need its skill, and cut it where you need to pass it.

Lesson 7 · b003g, the scorecard

Seven lines of eight

b003g made the menus' 5% standing real, and its log read 0.0500 throughout, so the run counted. By the letter of its lines it passed seven of eight: 0.057 m/s at a 0.10 command and 0.038 at 0.05, turning at 0.699 rad/s, 0.136 at 0.30, and the best falls of any fine-tune, 0.22 a minute. Only planar error failed.

Then the note reported what the lines did not ask. On straight commands the duck turned at about −0.21 rad/s at every speed, so it walked clockwise circles, about 0.27 m in radius at 0.10 m/s. Told to stand, it kept stepping, with a foot in the air 38% of the time against the student's 8%, and drifted at 0.106 m/s.

So the band had opened by stepping all the time on a curve, not by learning a slow straight walk. The verdict: this network is nobody's slow walker. Later runs are to add two lines, for yaw on a straight command and for drift on a stand.

A pass line measures what it asks and nothing more. For every way a duck could pass and still be wrong, write another line.

Predict

b003g passed its slow-walk lines. What else was true?

Check your prediction

Clockwise circles, and no standing still. Its falls were the series' best, so falling was not the problem, and the speed lines asked only for forward speed. Nothing asked about yaw on a straight command, and standing showed only inside the averaged planar error, the one line that failed.

The figure

  1. The card. Eight lines, written before the run. Seven pass; planar error fails.
  2. Straight? Asked to walk straight at 0.10 m/s, the duck turns clockwise at about 0.21 rad/s: a circle about 0.27 m in radius.
  3. Stand. Told to stand, it keeps stepping and drifts at 0.106 m/s.
  4. Two more lines. Yaw on a straight command and drift on a stand, at most 0.05 each.
The lines, the yaw, the radius and the drift are b003g's; the duck and its path are drawn.

Big idea: a scorecard can pass a duck that does the wrong thing. Look at what it does before you trust what it scored.

Lesson 8 · k001, the kick

Aim, not distance

On average, Pollen's right kick sends the ball 5.064 m, 8.0° off line. For shooting, aim matters more than distance. k001 kept Pollen's task and reward and added one term: the ball's speed across the kick line, at weight −12.0, the rate Pollen pays for speed along it, up to 1.0 m/s.

Its lines guarded the trade: 4.0° or straighter, at least 0.95 of Pollen's distance, falls and hopping held, and installable with Pollen's own checks. After 1,000 iterations from Pollen's kick it went 3.4° off line with 0.999 of the distance and falls unchanged, and every line passed.

It held on the backlash twin too, an actuator model it never trained on, at 3.7° against Pollen's 8.6°. One number moved that no line covered, the ball's peak speed along the line, from 1.34 to 1.28 m/s. The notes do not test why.

pay = 12 · min(valong, 1.0) − 4 · max(0, valong − 1.0) − 12 · |vacross|: speed across always costs; along pays only up to 1.0 m/s.

Predict

One term against sideways ball speed, 1,000 iterations. How far did the kick go?

Check your prediction

As far: 0.999 of Pollen's, at 3.4° off line. Nudging was the risk the design named, and the distance line was there to catch it. It did not have to: the kick kept its distance and lost most of its angle.

The figure

  1. Pollen's kick. 5.064 m, 8.0° off line on average.
  2. One term. Speed across the line costs 12.0 per m/s, the rate Pollen pays for speed along it, up to 1.0 m/s.
  3. k001. 3.4° off line after 1,000 iterations, and 0.999 of the distance.
  4. Backlash. On an actuator model it never trained on: 3.7°, against Pollen's 8.6°.
Drag θ to see what a ball at 1.0 m/s, that far off line, earns a step. The angles and distances are k001's; the pitch and the paths are drawn.

Big idea: aim at the one quantity you want, weigh it against what it trades with, and write lines that guard the trade.

Lesson 9 · The Qwiic port

A new sense on the bus

The walk reads 61 numbers, though the duck senses more: a camera in its head and, on some ducks, an 8×8 depth sensor. Pollen has said in its Discord that the duck's board has an unused Qwiic connector: a 3.3 V I²C socket, the plug Adafruit calls STEMMA QT.

Pollen's open board, the Robot HAT, has four, J5 to J8. J5 shares the main I²C bus, header pins 3 and 5, with the board's IMU and audio codec, and the others' buses depend on the computer. On Zero 3W ducks the depth sensor, where fitted, plugs into J5, on a 400 kHz bus with the codec at 0x18 and the sensor at 0x29, so a new device needs its own address.

So how does a new sense reach behaviour? As a command, a program reads it and steers the walk, changing no policy; or as an observation, a network is trained with more than 61 inputs. Either way, the module's weight changes the body the walk learned.

A sense joins as a command, through a program, or as an observation, through retraining.

Predict

What is the quickest way to let a gas sensor steer?

Check your prediction

A program that reads it and steers the walk. The walk's inputs are fixed and it never sees the sensor, but a program that turns readings into commands changes no policy. Then weigh the module and measure the walk again, because a heavier head is a different body.

The figure

  1. The board. Pollen's open Robot HAT: four Qwiic sockets, J5 to J8.
  2. The bus. J5 shares the main bus with the IMU and the codec. On Zero 3W ducks it runs at 400 kHz, with the codec at 0x18 and the depth sensor, where fitted, at 0x29.
  3. A new sensor. On J5's bus, at an address nothing else there answers to. The other sockets may be on buses the duck never switches on.
  4. Two ways in. As a command, a program steers the walk. As an observation, a network learns to read it.
The sockets, the bus and its addresses are from Pollen's open board and robot code. The final ducks carry an improved board, so check yours.

Big idea: the walk only knows its 61 numbers. A new sense earns its place through a program that steers, or a network that is trained to read it.

Lesson 10 · Other simulators

The contract travels

Every run in this book trained in mjlab, which joins Isaac Lab's way of writing tasks to MuJoCo Warp's GPU physics, and others are open: MuJoCo Playground, on MuJoCo MJX; Isaac Lab itself; and Open Duck Playground, for the Open Duck Mini, which Hugging Face and Pollen sponsored.

The duck checks only the contract, an ONNX file taking obs[1,61] and returning actions[1,14], 50 times a second, usually beside a manifest, and it refuses other sizes, in the file or its manifest. But it cannot see a policy that reads its 61 numbers in another order, unit or scale: it loads, runs, and is a different policy.

So check a port twice. The export: the trained network and its file must agree on the same inputs, as b003d's did, with a largest difference of 0 on 256 random inputs. The meaning: your trainer and Pollen's must build the same 61 numbers from the same recorded states. A swapped input passes the first check and fails the second.

Prove the file and the 61 numbers, not just the shapes.

Predict

Beyond its shapes, what must a port match?

Check your prediction

Every input's and output's order, units and scale. Shapes are what the robot can check, and meaning is what you check: the same recorded states, turned into 61 numbers by your trainer and by Pollen's, must agree.

The figure

  1. The plug. 61 numbers in, 14 out, 50 times a second.
  2. Other workshops. mjlab, MuJoCo Playground, Isaac Lab: each can export a network, and it fits only if it reads the duck's 61 numbers, in order.
  3. Same shape, wrong order. Two inputs swapped: it loads, it runs, and it is not the same policy.
  4. The check. The same states through both trainers: the 61 numbers must match, and the swap shows.
The sizes and the tick are the robot's; the workshops and the plugs are drawn.

Big idea: a policy has to keep its contract whatever trained it. Prove the file and the 61 numbers, and another simulator can make a file your duck will load; whether it walks depends on how well that simulator models the duck's body and servos.

Before this: Book 3, Training Microducks, explains the methods these runs use: returns, PPO, reward terms and honest measuring. All four books are on the shelf, and Train your first duck gets a training running on your own machine.