Skip to content
Effector Lab

Results

Given the same simulated budget of 40 finger designs per run, does NVIDIA Nemotron 3 Super, reading the measured results of earlier designs, reach a finger that seats and centres the part in at least 37 of 48 training grasps in fewer designs than equal-budget random search? Each simulated design stands in for one print-and-test cycle.

Pre-registered confirmatory result. 32 runs per arm, seeds 1001 to 1032, run as Nebius Serverless Jobs on uk-south2. The protocol was frozen and tagged before any confirmatory run; nothing was re-run or tuned after the results were seen.

Protocol: the prereg-v1 tag (docs/PREREG.md). Full write-up with limits and deviations: docs/RESULTS.md.

PRIMARY ENDPOINT

S1: speed to a good design

A run's count is the number of its first design, counted from design 9, that seats and centres the part in at least 37 of 48 training grasps. Designs 1 to 8 are round 0 (the hand-designed entrants and seed designs), the same for both arms at each seed, so they do not count. A run that never gets there within 40 designs counts as 41. Fixed in the protocol before the prereg-v1 tag.

S1 per arm
MeasureRandom search (A)Nemotron with feedback (C)
Runs that reached the target within 40 designs11 of 3223 of 32
Median designs to the targetnot reached (above 40)22.5
Mean designs to the target (not reached counted as 41)34.525.6

Nemotron reached the target in 23 of 32 runs, random search in 11 of 32.

  • Difference in means (A minus C): 8.97 designs, 95% interval 3.53 to 14.16.
  • One-sided exact permutation over C(64,32) splits: p = 0.0011.
  • Sensitivity, permutation log-rank: p = 0.0011.

Best training success so far

Median best training success so far. Random search (A): 74% after 40 designs; Nemotron with feedback (C): 85% after 40 designs. Threshold 37 of 48.

0%25%50%75%100%
Round 0, not counted
threshold 37 of 48
110203040
Simulated designs (budget B = 40)
  • Random search (A), median of 32 runs
  • Nemotron with feedback (C), median of 32 runs

Thin lines are runs, heavy lines each arm's median. A run whose round 0 already reached the target still needs one of its own designs (from design 9) to count.

Designs needed to reach the target

Random search (A)11 of 32 reached · median not reached
Nemotron with feedback (C)23 of 32 reached · median 22.5
110203040not reached
Each dot is one run; the bar marks the arm's median. Fewer designs is better. Hollow dots past the dashed line never reached the threshold within B = 40 (censored).

Exploratory, lift-only scoring (the part only has to come off the table): no difference between arms (p = 1.0). Both arms find a finger that lifts the part almost at once. The advantage is in seating and centring the part, which is what the next step in a work cell depends on.

SECONDARY ENDPOINTS (HOLM FAMILY)

Held-out success and search efficiency

Each run's frozen top-1 design was scored on suites neither arm saw during search (Nebius job aijob-e05nvfme46ss0q4wp4).

All three reject the null at alpha 0.05 after Holm adjustment.

H2 by object width (planned split, reported outside the Holm family): objects 50 mm or narrower across the jaw (18 objects) 19.9% vs 24.1% (p = 0.012); wider than 50 mm (46 objects) 22.8% vs 28.8% (p = 0.001).

Real scanned objects are hard for every design in this space: the best arm seats them about a quarter of the time. The held-out real-object result says the Nemotron arm's picks generalise better, not that the fingers are ready for arbitrary objects.

Limits

Show the jobs behind these numbers
Where the search runs ran. Model calls go to Nemotron on Token Factory and are billed separately from the job.
ArmJobWhereRunsRun timeModel calls
Random search (A)aijob-e05yxb1awgg2d68hq1Nebius · cpu-d3 · uk-south23228 minnone
Nemotron with feedback (C)aijob-e05s3f8ka6r508p4mkNebius · cpu-d3 · uk-south23284 min473

Image: sha256:c6162fa01d41beaab64e1431bc10e07535ce7bd59045fcef7bf0e074857b4b89

Analysis outputs: runs/analysis/s1-confirmatory/s1_v2.json, runs/analysis/secondary-confirmatory/secondary_v2.json (released with the run data).