RoboAnalytica~6 s lift. ≥~15 s hold.
1 kg box. Partial completion: the required turn and handover are not shown.
PUBLIC ROBOTICS BENCHMARKS
Robotics,
put to
the test.
What can robots actually do?
We’re building public benchmarks for robots people can buy and deploy. Our first research case study helps us develop the tests; coverage will expand as more systems become accessible.
Explore manipulation and mobility findings ↗︎
01 / FIRST CASE STUDYUnitree G1 · Research configuration
PHYSICAL TRIAL / RECORDED EVIDENCE●
Real capabilities.
Put to the test.MANIPULATION
RESEARCH SYSTEM
Put to the test.MANIPULATION
RESEARCH SYSTEM
BEYOND THE BEST TAKETask completion ↗︎Balance ↗︎Human assistance ↗︎Recorded evidence ↗︎
CASE STUDY 001 / RESEARCHER-CONFIRMED FINDINGS
One system family.
Different tasks. Different outcomes.
60 s shelf task.
Correct object retrieved, transported and released at the specified destination. Task complete; clean success not verified.
4 s stair ascent.
Five steps, 80 cm ascent. Teleoperated completion; clean success not verified.
Six reviewed attempts: 2 task completions, 1 partial completion, 3 failures/interventions; 1 reset. This selected subset mixes tasks and configurations and excludes additional unaudited footage. It is not an overall robot success rate.
Full setup, task definitions and evidence ↗︎SIX DISTINCT ATTEMPTS / RETROSPECTIVE CASE STUDY
Results you can inspect.
Confirmed outcomes, linked to each excerpt. Stage scores describe progress; they are not clean-success ratings. No overall reliability percentage is calculated across different tasks or controllers.
| Attempt / evidence | Control | Stage | Outcome | Timing | Assistance / reset |
|---|---|---|---|---|---|
| 01 · Box lift / hold ↗︎ | Autonomous · HANDOFF | 2 / 3 · lift | Partial; required handover not shown | ~6 s lift; ≥~15 s visible hold | 1 reset; clean success not achieved |
| 02 · Box balance loss ↗︎ | Autonomous · HANDOFF | 2 / 3 · lift initiated | Caught balance loss; intervention | Balance loss at 10 s | Recovery assistance; no ground-contact fall |
| 03 · Shelf retrieval ↗︎ | Autonomous · HANDOFF | 3 / 3 · placement | Task complete; clean status unverified | 60 s: first reach → completed release | Slack tether / no assistance not established |
| 04 · Stair ascent ↗︎ | Teleoperated · earlier stack | Not assigned² | Landing reached; clean status unverified | 4 s: first upward step → both feet stable | Slack tether / no assistance not established |
| 05 · Stair balance loss ↗︎ | Teleoperated · earlier stack | 1 / 3 · partial ascent | Caught balance loss; intervention | No complete-task time reported | Recovery assistance; no ground-contact fall |
| 06 · Stair assistance ↗︎ | Teleoperated · earlier stack | 1 / 3 · provisional | Hands-on assistance; completion not established | No complete-task time reported | Physical assistance recorded |
The hands-on stair score is provisional: partial ascent is documented but landing completion is not. ² The completed stair run remains unscored because support evidence is insufficient to distinguish assisted from clean completion. No clean-success designation is assigned to the completed shelf or stair run.
Archive summary: 2 task completions, 1 partial completion, 3 failures/interventions and 1 reset. The two balance-loss incidents were caught, not ground-contact falls. Additional unaudited footage exists; these six attempts are not ten consecutive attempts of one test.
STARTING POINT / CASE STUDY 001
Unitree G1.
Research configuration.
Our first case study examines a Unitree G1, 29-DoF research setup with parallel-jaw grippers. Indoor tasks use autonomous HANDOFF control; stairs use a separate teleoperated configuration. Setup and selected outcomes are confirmed by the contributing researcher.
What this means: these observations describe the recorded research setups, not a rating of the standard product a customer receives.
Where we’re going: repeatable tests of commercially available robot systems, with hardware and software versions documented. Coverage will expand as systems become accessible for testing.
THREE TIERS OF TESTS
IN DEVELOPMENTThree test pools.
A rising bar.
We’re preparing tests for the basics robots should meet, the capabilities that set today’s systems apart, and challenges for the years ahead.
THE MINIMUM BARPass / Fail
Basic
The essentials a robot should reliably be able to do. This baseline will rise as capabilities become established.
- Candidate tests
- Walking on level ground, negotiating simple obstacles, and maintaining balance.
- How we plan to evaluate
- Pass or fail on each test, against explicit conditions and a published minimum standard.
TODAY’S DIFFERENCESGraded scores
Capability
Tests that reveal meaningful differences between current systems, including challenging subtasks and complete tasks.
- Candidate tests
- Object manipulation, retrieval, transport, and handover.
- How we plan to evaluate
- Graded results that capture completion, reliability, time, and assistance under a defined protocol.
LONG-TERM CHALLENGESGraded scores
Frontier
Hard challenges designed to remain useful for years, with room to measure progress before full completion.
- Candidate tests
- Complex manipulation, unfamiliar object layouts, and sustained work under changing conditions.
- How we plan to evaluate
- The same scoring approach as Capability, applied to much harder tests. Partial progress counts.
TEST POOLS, NOT ROBOT GRADES
Tiers group tests by difficulty relative to the field’s capabilities. Each pool can include individual skills, subtasks, or full tasks. Results belong to a specific test version and system configuration.
FIELD NOTES / 001
PRELIMINARY UNITREE G1 EVALUATIONMore tasks.
More than one outcome.
A successful attempt shows capability. An evaluation asks what happened across the attempts—and what it took to get there.
BOX HANDLING / RECORDED ATTEMPT
Lift and hold observed.
The robot lifts the box and maintains a hold. The full instruction also requires a turn and handover.
FINDING Intermediate stage achieved
BOX HANDLING / SEPARATE ATTEMPT
Balance changes the outcome.
A separate recording shows a loss of balance during the lift, followed by visible human intervention.
FINDING Recovery assistance observed
SHELF RETRIEVAL / TASK SEQUENCE
Follow the whole task.
The Pringles container is retrieved, transported and released left of the Cheez-It box in 60 seconds.
FINDING Full shelf task confirmed
OUTDOOR MOBILITY / RESEARCH RECORDING
Stair ascent observed.
The robot climbs the wide outdoor staircase and reaches the upper landing in this excerpt.
FINDING Ascent visible; teleoperated
OUTDOOR MOBILITY / RESEARCH RECORDING
Balance interrupts progress.
A separate recording shows a loss of balance near the bottom of the stairs, followed by people intervening.
FINDING Loss of balance and intervention
OUTDOOR MOBILITY / RESEARCH RECORDING
Human assistance is part of the outcome.
A closer view at a different staircase shows a person making physical contact with the robot during the sequence.
FINDING Hands-on assistance visible
CONTEXT MATTERS.
Explore findings, research videos & charts ↗︎These excerpts cover indoor object handling and outdoor stairs from separate research recordings. The researcher confirms May 15, 2026 and September 11, 2025 respectively. The team identifies outdoor stair footage as teleoperated and indoor manipulation footage as autonomous. The confirmed manipulation setup uses HANDOFF; stairs use a different teleoperation stack. These sessions are not combined into one score. Visible ascent does not establish autonomy or a Basic-tier pass. Findings apply only to the recorded setups; no aggregate reliability score or commercial certification is claimed.
OUR METHOD
From recorded
behavior to
useful evidence.
Our first goal is a credible public benchmark: clear rules, inspectable evidence, and comparable results that show progress over time.
Help build the benchmark ↗︎Define the job.
Publish a versioned test protocol: setup, software version, time limits, allowed assistance, and pass criteria or graded scoring. Group the test by its role in the benchmark.
Examine the attempts.
Retain every attempt. Record completion, time, resets, and assistance. Compare only systems tested under matching rules.
Publish the evidence.
Report each test’s results with source evidence and uncertainty. Document failures and untested conditions. Track improvements on the same test across system versions.
HOW THE BENCHMARK EVOLVES
PLANNED ROTATIONProgress raises
the baseline.
Today’s frontier can become tomorrow’s expectation. Our planned rotation moves established tests into lower tiers, adds harder challenges, and progressively raises the Basic standard.
- ADD
New challenges
Prepare harder tests as the frontier advances.
- EXPLORE
Frontier
Measure early, reproducible progress on hard problems.
- COMPARE
Capability
Distinguish systems as meaningful scores become repeatable.
- EXPECT
Basic
Make established capabilities part of a rising minimum standard.
Move with evidence.
A first score signals a candidate for migration. Repeatable results and published criteria would determine when a test changes pools.
Raise the minimum.
Basic will evolve too. Future versions can add expected capabilities and tighten pass requirements as the field improves.
Keep progress comparable.
Changes take effect in new benchmark versions. Earlier protocols, thresholds, scores, and tier assignments remain available.
This is the framework we’re preparing. Test protocols, migration criteria, and the release process are still being developed.
HELP BUILD THE BENCHMARK.
Help define
the next test.
Build robots, study autonomy, or use them at work?
Propose a task, contribute test access, or challenge our methodology.