Blackforge
Robot Lab

Lesson I.1

Swap the model, keep the test

FLUX 3 Action's weights are non-commercial, and on our task it scored 0 real successes: every success its detector reported overnight was fake. We replaced it with NVIDIA's GR00T N1.7 (3B parameters), which has an official SO-101 / LeRobot fine-tuning path. Its model card says NVIDIA Open Model License, but the LICENSE file shipped with the checkpoint is the non-commercial NVIDIA License; check which applies before any commercial use. The eval did not change: same objects, same seeds, same strict rule.

• Strict success = the task's success term fires and the object was lifted at least 3 cm and the bowl stayed upright (≤ 20°) within 3 cm of where it started and no physics blow-up. Layouts with the object starting within 10 cm of the bowl are re-seeded.
• Check the harness first. Recorded expert demos replayed through the same eval path: 31 of 32 strict on night 2. If that number were low, every policy number would be suspect.
• Faster to plan: about 120 ms per action chunk against 0.9–1.2 s for FLUX.

Lesson I.2

It never saw itself let go

Night 1 fine-tuned GR00T for 25,000 steps on the 909 demos from the FLUX era. Final: 8/72 strict (12 episodes per object). Lifting worked: the blue block was lifted in 10 of 12 episodes. But in 48 of the 59 episodes where the object was lifted and not placed, the policy never commanded the gripper open at all.

The cause was the demo recorder. The environment reset the instant the object landed in the bowl, so each demo held about one release frame in roughly 145. And the last label of every demo was the reset pose, not the release. No amount of training teaches a behaviour that is missing from the data.

Lesson I.3

Record the release, then weight it

Night 2 fixed the recorder: labels are read before the reset, and each kept demo includes a 0.5–1.3 s tail of release, retreat and settle. That gave 862 demos with a median of 42 release frames each. The fine-tune put a 5× loss weight on frames where the gripper opens or closes, from the base model, same recipe, 25,000 steps. Evaluation doubled to 24 episodes per object.

• Training objects 61/96, held-out objects 12/48. The held-out pair was never in the demos.
• Weighting is a blunt tool. At the halfway checkpoint the 5× weight had overshot: on the blue block the gripper was commanded open most of the time and the policy lifted in only 1 of 12 episodes (halfway total 17/96). The final checkpoint recovered both grasp and release.
• Look at every success. The report links a frame strip for each one; look at them before quoting a number.

cd /mnt/work/AI/robot-lab                       # code: code/robot-lab/tools in this repo
bash tools/overnight_gr00t_v2.sh tiny            # gate: every stage on a few demos, minutes
bash tools/overnight_gr00t_v2.sh full            # record -> convert -> fine-tune -> strict eval -> report
# output: overnight-gr00t-v2/full-*/MORNING_REPORT.md, eval/*/results_*.json, *_strict_strip.png

The policy grasps and lifts but never opens the gripper. Before touching the model, count release frames per demo. Fix the recorder so it reads each step's label before any auto-reset and keeps a release + retreat + settle tail after success. Re-record, convert to GR00T's LeRobot layout, replay a few recorded demos through the strict eval (expect nearly all strict), then fine-tune from the base model with extra loss weight on gripper open/close frames. Evaluate on fixed per-episode seeds with the strict rule, and attach a frame strip to every success.

Lesson I.4

Can a local VLM tell whether it worked?

After an attempt, we asked a local vision-language model "is the object in the bowl?" and scored its answers against the simulator's ground truth. Frames taken at the instant of release are ambiguous, with the fingers still around the object: Gemma 4 12B missed 24–40 of 45 real successes there. So the arm retreats and waits about a second before the picture is taken.

Then a 4-bit QLoRA fine-tune of Gemma (Unsloth, one epoch, 7,614 labelled frames mined from our recordings, one training seed per arm). It fixed the error we trained against and created another:

• Not deployable yet. The tuned models answer "the gripper is still closed" when it has just opened around an object already in the bowl. Our training frames had 805 "hovering over the bowl" failures against 117 release-instant successes; that imbalance is the leading hypothesis, untested.
• Two harnesses, two baselines. The first table is the served models (Gemma on vLLM, Qwen on NInfer); the second is the fine-tuning harness (Transformers). Compare within a table, not across.
• Code: code/robot-lab/verifier (frame builder, QLoRA training, eval against sim truth).

Lesson I.5

Or learn when to let go

The other route keeps night 1's model, the one that grasps but never releases, frozen, and trains a small PPO policy on top of it with residual reinforcement learning. Every action chunk, the policy adds a correction of at most 4° to the arm and decides whether to open the gripper now. Reward comes from the same strict checks. About five hours of training on one GPU, 185 waves of 64 parallel environments; two of 34 training rounds crashed (an environment ended during its settle step) and were skipped.

Then a paired test on sealed seeds never used in training: for each layout, GR00T alone and GR00T with the gate see the same scene and the same model noise.

• Read it as "GR00T + release gate", not "GR00T learned to release": the gate is a separate learned policy, and the opening itself is a fixed jaw-open action.
• Not comparable to night 2's 73/144: different seeds, and the gate sits on night 1's model. The gate has not been tried on night 2's model yet.
• Look at the strips. We watched gated successes on the blue block and the held-out cracker box (in the bowl, gripper open) and a base failure that holds the box over the bowl.

cd /mnt/work/AI/robot-lab/rl                    # code: code/robot-lab/rl in this repo
/mnt/weights/ai/isaac/IsaacLab/.venv/bin/python tests/test_core.py   # CPU tests
RUN=/mnt/weights/ai/robot-lab-data/gr00t-rl/runs/<name> GR00T_CKPT=<checkpoint> \
  TRAIN_END=$(date -d '+5 hours' +%s) GPU=1 bash scripts/run_rl.sh   # train, sealed eval, REPORT.md

Freeze the imitation policy. Train a small PPO actor that, each action chunk, adds a bounded joint correction (a few degrees) and a Bernoulli "release now" decision; reward reach, lift, carry over the bowl, release inside the bowl footprint and the strict success, and penalise knocking the bowl. Batch many environments per object. Keep training, development and sealed evaluation seed ranges disjoint and assert it in code. Evaluate paired: base and base+residual on identical layouts and identical policy noise, with videos and strips.

Lesson 9.1

Build the scorer before the thing it scores

Milestone M0 had no simulator at all: event rules (perimeter crossing, restricted zone, a fall, a vehicle in the wrong place, a gate left open, an obstacle on the route), a scorer, and scenario files that stage those incidents at known times. Only then did Isaac Sim join, in M1: Spot on NVIDIA's pretrained walking policy, walking a generated compound, with the rules reading facts back from the simulated stage.

• Ground truth comes from what the scenario staged, never from what the rules detected. Scoring rules against their own output always gives 100%, so the scorer refuses rule output as truth, and a test checks that.
• Test that the tester can fail. Drop a staged incident, add a phantom one, shift the times: the scorer must report each.
• The M1 gate: both base scenarios pass in Isaac (all six incident types, 0 false alerts, full coverage), then 10 of 10 randomised variants, with every actor's timing shifted and its staged truth shifted with it.

cd /mnt/work/AI/patrol-lab          # private: github.com/matlowai/patrol-lab
uv sync; uv run pytest -q           # rules, scorer, scenarios, planner (no GPU)
uv run python tools/run_scenario.py scenarios/library/SCN-M0-001.yaml --validate
CUDA_VISIBLE_DEVICES=0 OMNI_KIT_ACCEPT_EULA=YES /mnt/weights/ai/isaac/IsaacLab/.venv/bin/python \
  tools/run_scenario_isaac.py scenarios/library/SCN-M0-001.yaml --video

Before any simulator: write event rules over a per-frame fact stream (tracks, zones, asset states), a scorer that matches detected incidents to incidents staged in a scenario file within a tolerance, and mutation tests that prove the scorer fails when an incident is dropped, invented or shifted. Then drive the same rules from Isaac Sim by rebuilding each 10 Hz frame from the stage itself (prim poses, gate hinge angles), never from the scenario file.

Lesson 10.1

An avoidance result means nothing without a control

Spot got a PhysX ray-cast lidar and a dynamic-window planner: sense, track, predict, choose a velocity. Against six oblivious book readers it finished the patrol with no near misses. The number only counts because the control run, the same patrol with avoidance switched off, walked into three of them and was knocked over.

Every planner bug showed up first in a 2-D world with circles for people, which runs a whole patrol in seconds:

• Stalling short of the goal: scoring progress at the end of a rollout punishes the final hop. Score the closest approach.
• Getting run over: treating each scan as frozen makes a head-on walker look static. Track velocities across scans.
• Flip-flopping left and right: equal scores flip every tick. Commit to a side for a few seconds.
• Phantom motion on walls: grazing-angle lidar hits slide along a wall as you move. Trust velocities only from clusters.

Lesson 11.1

Long jobs get their own cgroup

The overnight plan said "keep 20 GB free". The launcher checked free memory, then started six Isaac processes in the same second, and a seventh smoke test was already running. Every check saw the same free memory. All of them were children of the coding session inside VS Code, so when memory pressure hit, systemd-oomd killed the whole VS Code scope: 59.5 GB, the editor, the session, the orchestrator and every run.

• nohup doesn't help. The out-of-memory daemon acts on the whole scope, not the process.
• The relaunch ran each job as its own memory-capped unit and started Isaac processes one at a time, 90 s apart, only with 29 GB free. It ran all day with RAM never below 20 GB free.

systemd-run --user --unit=patrol-night --collect -p MemoryHigh=44G -p MemoryMax=50G \
  --working-directory=$PWD uv run python tools/overnight_patrol.py --stages s1,s2 --parallel 4
journalctl --user --since "-1h" | grep -i oom        # what oomd did, if anything

Lesson 12.1

Watch the video, not just the table

The RL policy looked "stuck" at the end of a three-way comparison video. It was parked: it had finished the patrol 40 seconds early. The book readers were keyframed to a timetable fitted to the no-avoidance robot, which stops and turns in place. The planner and RL turn while walking, ran 10–13% ahead, and outran their encounters. The "collisions" at the end were late readers walking into a robot that had already gone home.

Lesson 12.2

Opponents that react to the thing under test

The fix was a closed-loop crowd of about 50 people: wanderers reading books, loiterers standing on the route, and hunters posted beside every patrol leg. A hunter sees Spot (range, vision cone, line of sight that buildings block), locks on, predicts where it's going and walks to intercept. Inside 2.5–4.5 m it commits to a straight line and stops re-aiming. A pursuer that re-aims until contact at walking speed can't be escaped, and then there's nothing to learn.

Both contenders now get the same exposure. The cheap world was then checked against Isaac on the same 8 seeds: 6.1% hits per commit in Isaac, 7.9% in 2-D. Isaac adds what 2-D can't express: 3 of the planner's 7 hits knocked Spot over.

cd /mnt/work/AI/patrol-lab
uv run python -m benchmarks.avoidance.sim2d --seeds 1-40 --controllers control,heuristic --workers 40 --live hard
uv run pytest -q tests/test_pedestrians.py     # commit rule: standing still is hit, a timely sidestep evades

Write one closed-loop crowd model used by every consumer (2-D benchmark, Isaac runner, torch RL env): wanderers, loiterers and hunters posted beside each patrol leg. Hunters lock on when the robot is in range, inside a vision cone and not hidden by buildings; they aim at a constant-velocity intercept, re-aim every 0.5 s, and commit to a straight line inside commit_m. Score per lock (hit or evaded) and report lock-ons per run next to every hit rate. Port the hunter state machine to torch and prove parity with the numpy reference over a scripted robot path.

Lesson 12.3

Report the denominator

Hit-per-commit is conditional. A contender that's slower meets more hunters; one that falls early meets fewer; one that bumps ordinary people doesn't show up in it at all. On 80 held-out hard-tier seeds:

Longer lidar memory helped against hunters and made RL much faster, but RL still bumps ordinary people twice as often. Its training world had 4–8 people per scene; the benchmark has about 50.

Lesson 12.4

Privileged information leaks through the scene

Hunters had debug beacons that turned magenta when locked on. Spot's head camera, which records the training data for a vision policy, could see those beacons from about 4 m. The data would have told the model each hunter's hidden intent. Debug visuals are now invisible whenever the head camera records, and a check counts forbidden pixels: 0 in 600 frames that included 5 lock-ons.

Lesson 15.1

One change per arm, and a script that killed three runs

Contact forensics had shown that RL bumps people it had seen for about 3 seconds while it kept walking at full speed. Its forward-speed action had collapsed into a go/stop switch. So the next night ran four pre-registered arms, one change each. Checkpoints were picked on validation seeds, each pick ran once on 80 sealed seeds, and the win rule was written down before launch.

The speed fix worked as a mechanism: the action came back to life, with 15% of steps at in-between speeds. But contacts didn't fall, so speed wasn't the bottleneck. Three of the arms almost didn't report at all. While they were running, I edited the shared launch script for the fourth. Bash reads a script lazily, so seven hours later they resumed at a shifted offset and died with unexpected EOF. A parse error skips the error trap, so their status said "running" forever.

Lesson 15.2

Can a network even copy the planner?

Six 7-hour fixes had all changed the reward or the behaviour. None asked whether the policy's input contained the cue. The policy saw three lidar snapshots 0.3 s apart, and in that time a crossing walker moves less than one lidar sector. The cheap test is imitation: clone the planner from the candidate input, change nothing else, and see whether the copy yields. If imitation can't recover a behaviour from an input, RL almost certainly can't either.

The missing ingredient was which way people are moving. The test cost 40 minutes of CPU. The planner's code was refactored to expose its tracker, and the change was checked by reproducing the 40 reference patrols row for row.

cd /mnt/work/AI/patrol-lab
systemd-run --user -p MemoryMax=26G --working-directory=$PWD env GPU= tools/clone_probe.sh avoid-v2t

Before another RL run, test whether the policy's observation can express the expert's behaviour at all. Record the expert (our hand-written planner) driving sim2d patrols with the candidate observation, behaviour-clone it, run 3 DAgger rounds (the clone drives, the expert labels), and score round 3 (fixed in advance) on validation seeds only. Change only the observation between arms. Commit the reading rule first: what result means "the input is the bottleneck".

Lesson 15.3

Give it velocities, and check the emulation

The GPU training world can't run the real tracker, so it emulates it from the true positions. People must be hit by at least two rays, velocities are smoothed and only trusted from the third scan, and 0.2 m/s of noise is added. Before training, the real tracker was run on the training world's own scans and compared direction by direction: 98.5% agreement on what is moving, direction cosine 0.996. At deployment and in every score, the real tracker supplies the velocities.

Every arm improved, on every measure. It still wasn't the planner. The literature said the same: in DRL-VO, a policy with pedestrian velocity inputs but only distance penalties stayed below a hand-written planner until a predicted-conflict reward was added.

Lesson 15.4

A separate price for bumping bystanders

One −10 contact penalty covered hunters and bystanders alike, and 7 of 8 training contacts were hunters. The best move against a hunter (keep speed, sidestep) is exactly wrong for someone crossing your path (slow down, let them go). So the reward stayed the same, and a separate cost was added for ordinary people only. It charges for driving into where each walker will be in the next few seconds, for closing speed when near, and for contact. Constrained RL (PPO-Lagrangian) gives the cost its own critic, and a multiplier that rises automatically until the cost is under budget. Nobody hand-tunes the weight.

How far ahead should it look? The question came from driving: you track cars 20 s out, and things that loom grab your attention. A 12 m lidar at walking speeds gives about 4–6 s of warning, so we tested 2 s against 5 s, with two training seeds each.

In plain words: once it could see where people were heading and had a separate price for crowding bystanders, the learned robot bumped people about as often as the hand-written one. It dodged the people chasing it 2.5× better and finished patrols about 10% faster.

Two of four runs beat the planner on every criterion. The pre-registered demo pick (the best validation score, 5 s seed 0) did not, and naming a winner after seeing the test would be cherry-picking. 2 s versus 5 s couldn't be separated: the random seed moved results by 6–10 runs, as much as the look-ahead did.

cd /mnt/work/AI/patrol-lab
PYTHONPATH=. python -m benchmarks.avoidance.rl.train --out runs/L0 --hours 7 --preset hard-dense --obs avoid-v2t \
  --set w_space=0 --set track_noise_mps=0.2 --squash clip --log-std-max 0 --bound-coef 0.01 --lr 1e-4 \
  --init-ckpt <v5 V3 final> --init-critic --critic-warmup 100 --set ambient_cost=true \
  --set cost_horizon_steps=10 --set cost_dt=0.5 --set cost_decay=0.85 --cost-limit 0.0173 --seed 0

Add a per-step cost that covers only the agents the benchmark scores as bystanders (never the adversaries). It should charge predicted intrusion into each one's constant-velocity path over a tunable horizon, closing speed inside 0.4 m, and contact. Report it in info["cost"] and never in the reward (test that rewards are bit-identical with it on). Train with PPO-Lagrangian: a separate cost critic, advantage (A_r − λA_c)/(1+λ), and λ updated from the rollout's mean cost against a limit set in advance at half the starting policy's cost.

Lesson 15.5

The winner's curse

Every RL run looked better on the 40 validation patrols than on the 80 sealed ones, and I first wrote that the sealed seeds were "harder for RL". They weren't. Each run kept the lowest of 14–18 checkpoint scores, and each score was a noisy estimate from 40 patrols. The minimum of many noisy numbers is mostly the luckiest one.

Averaged over checkpoints, validation matches the test. It even ranked the last runs backwards: the best-validation run tied worst on the sealed seeds. My first explanation leaned on one anecdote ("that pick was the last checkpoint, so it isn't selection"), and the statistic overturned it. The next protocol picks on far more patrols, or fixes the checkpoint in advance.

Lesson 15.6

Before you train a robot to share space with people

These are the questions that would have saved the most time, in order:

1. Can the policy perceive the cue? List what a good human or a hand-written controller uses (here: which way each person is moving), and check it's in the input at the rate the decision needs. Test it by imitation before any RL.
2. Does the reward price each failure separately? Chasers and bystanders need opposite responses. A shared penalty is set by whichever is more common.
3. Does the signal arrive early enough to act on? Contact is too late. Predicted conflicts and closing speed arrive seconds earlier, while braking still helps.
4. Does training contain the situation, as often as the robot meets it? Count what's near the learner, by kind, per step, not headcount per scene.
5. Can the policy actually act? Watch the action distribution. A saturated output can turn "speed" into an on/off switch.
6. Is the evaluation honest? Opponents that react to the contender, exposure reported beside rates, a control run, sealed seeds run once, enough data to pick checkpoints, more than one training seed, and the reading rule committed before you look.
7. Will the pipeline tell you when it fails? Frozen launch copies, resumable stages, deadlines with fallbacks, memory caps, and no status that can say "running" forever.