Skip to content
Effector Lab

Evidence

Every number on this site comes from a committed result file. These are the tests behind the claims, how they were run, and what they cost.

Studio benchmark, use-case benchmark v1 (top-down wrist, 19 Oct), pre-declared

Does the Studio's pick finish the job more often than the best single tool?

74 of 96held-out runs finished by the Studio's pick, against0by the best single tool.

Use caseStudio pickBest singlep (Holm)
Returns tote32 of 320 of 327.0e-10
Pipe coupling32 of 320 of 327.0e-10
Equipment cabinet10 of 320 of 329.8e-4

exact one-sided paired sign-flip permutation test (exact McNemar), b > a; each condition is a new seed, friction and hidden placement error. Product mode: not part of the pre-registered experiment.

A dated note on this result follows below. Read the note.

One Nebius job, aijob-e05e55n905qj9va2fh, uk-south2 cpu-d3, 206 s, about $0.06.

Full resultsProtocolAnalysis file

Finger designer, pre-registered (prereg-v1)

Does a language-model designer with feedback find a good finger in fewer designs?

8.97fewer designs on average than equal-budget random search

95% interval 3.53 to 14.16; exact permutation over C(64,32) splits, p = 0.0011. Sensitivity (permutation log-rank, 20000 relabellings): p = 0.0011.

Results pagePre-registrationDeviations

Note on use-case benchmark v1, added 26 Oct 2026

What v1 could not test

v1 used a top-down wrist; v2 adds wrist tilt and honest tolerances; v1 results stand as published.

  • Returns tote: This result came from a simulated wrist that could only point straight down (no pitch). Under that wrist only soft arms could open the lid, so the soft arm in the recommendation says as much about the wrist as about the task.
  • Pipe coupling: The insert step counts as done at a depth of at most 4.0 mm, a test equal to the thread start. The published passes sat 6 µm inside it, so a small change to the physics could flip them.

The tagged protocol, code and numbers stay as published; a note never changes a result.

Studio benchmark, use-case benchmark v2

v2 pre-registration: results pending

Next: six real jobs, a wrist that tilts, success tests the physics can meet

The next benchmark will ask the same question on six jobs from the use-case atlas. It will fix both v1 limits: the simulated wrist will tilt, and every success test will have a tolerance the physics can meet. Its protocol, arms and secret will be frozen and tagged before any held-out run, and its results will replace this card.

Methods, runs and compute