Exercise the Bench contract with deterministic mock results.
The API creates runs, queues work, and returns stable report-shaped fixtures. It does not execute a robot, simulator, GPU job, or real policy evaluation.
- Mock
- runner state
- JSON
- report artifact
- 0
- real evaluators
from openbot import Client
ob = Client() # reads OPENBOT_API_KEY
# 4-step kitchen handover on a Franka Panda
run = ob.bench.rollout(
policy="openvla-7b",
embodiment="franka_panda",
task="open_drawer → pick_mug → pour → handover",
rollouts=200,
seeds=10,
sim="isaac_sim",
real_hw=True,
edge_target="jetson_orin",
)
result = run.wait() # poll until the run finishes
print(result.task_success) # e.g. 0.73
print(result.subtask["handover"]) # e.g. 0.60 ← bottleneck
print(result.sim_to_real_gap) # e.g. -0.29
print(result.intervention_rate) # e.g. 0.14What the mock runner can verify.
Use it to integrate the lifecycle and report schema—not to judge a model.
- 01
Run lifecycle contract
Create, queue, poll, and retrieve a stable terminal result.
- 02
Deterministic fixtures
The same request produces a stable mock-shaped metrics payload.
- 03
Input provenance
Preserve policy, embodiment, task, rollout, seed, and target labels.
- 04
Schema rehearsal
Exercise edge-metric and subtask fields before a real runner exists.
- 05
JSON report artifacts
Retrieve mock report and failure-cluster JSON through authenticated routes.
- 06
Terminal delivery path
Exercise the wired webhook lifecycle without treating mock metrics as evidence.
A deterministic contract fixture.
These values are generated by deterministic_mock_v1. They test the report shape; they do not evaluate a policy.
kitchen_handover · open_drawer → pick_mug → pour → handover
Subtask success
request metadata · real_hw flag: true- open_drawer98%
- pick_mug91%
- pour82%
- handover60%
Success across 10 seeds
73% ± 5.2
A real runner could use this report shape to flag the handover step. This fixture cannot support a ship verdict.
Fields accepted by the mock contract.
Integrate the contract without mistaking it for evaluation.
Read the API recipe or tell us what a future real runner must prove.