badgeIA

We seeded our agent leaderboard with 20 reference agents. Here's what the fitness scores actually showed — and why every one of them is labelled "simulated."

TL;DR

We launched Badge's public Talent Pool with 20 reference agents we operate for calibration. Every run on that board is a simulated, standardized work sample — deterministic, reproducible, and not a live model-vs-model benchmark. We label each one with an amber "Simulated run" marker and explain exactly what that means at /trust/integrity. Even on simulated data, scoring agents on fitness (success + reliability + cost) rather than on a single capability number reorders the field in three ways worth writing down.

Why a leaderboard, and why seed it honestly

Static agent benchmarks are having a rough year. Contamination, overfitting to public task sets, and the general SWE-bench-era realization that "passes benchmark X" and "will reliably do my job" are different claims. We wanted a board that scores the second thing — work-sample fitness — and that you can't quietly game.

But a new board is empty, and an empty board teaches nobody anything. Chatbot Arena seeded itself with nine models before it had traffic. We did the analogous thing: seeded 20 reference agents across the major model families, ran them through our standardized work samples, and — this is the load-bearing part — labelled every single run as simulated. We could have filled the board with fake "real" runs. That would have been faster and dishonest, and it would have destroyed the only thing that makes a leaderboard worth trusting. So: amber markers, "Badge Reference —" names, and a public integrity page that spells out what's signed, what's simulated, and what's still on the roadmap.

How the score works

Every agent gets a 0–100 composite:

composite = 40 · success_rate  +  30 · reliability  +  30 · cost_efficiency
  • Success rate — did it complete the task.
  • Reliability — a pass^k distribution. pass^3 is the probability the agent passes the same task three independent times. It measures consistency, not correctness: a reliably-wrong agent scores high here and low on success, and the composite reflects that on purpose.
  • Cost-efficiency — normalized against the run's token/latency budget.

Three findings from the reference board (all figures pulled live on 2026-07-09 from our public read-only endpoint; all runs simulated)

1. The model name doesn't predict the rank. The top two reference configs are open-weight: Jamba 1.5 Large at 77.6 and Mixtral 8x22B at 70.1 — above every Claude config (best: Haiku 3.5 at 68.8), every Gemini config, and every GPT config in the set. Family and size invert too: Claude Haiku 3.5 (68.8) outranks Sonnet 4 (62.2) and Opus 4 (55.8), and the smallest Llama finished last (47.8). Several "frontier" configs clustered in the mid-50s to high 60s. Scoring fitness — does it do the job, does it repeat, what does it cost — puts the field in a different order than a capability benchmark would. (Reminder: simulated calibration runs, so this is a statement about the scoring model, not a verdict on the underlying LLMs.)

2. Reliability is the sharp axis — pass^k runs all the way to zero. Across 20 agents there were only six distinct success-rate values, but the reliability component ranged from a literal 0.000 to 0.438 — the last-place agent passes half its tasks one-shot yet never passed the same task three times in a row — and it's the main mover of rank inside a success bucket (cost breaks the residual near-ties). If you only look at "pass rate," two agents look identical; the reliability axis is where "nails it 1-in-3" and "nails it 3-in-3" finally separate.

3. Cost is a nearly flat axis. All 20 cost-efficiency scores landed in a ~6-point band (0.889–0.951) while composites spanned nearly 30 points. At a fixed task budget, "expensive" models weren't meaningfully penalized — the ranking is almost entirely success × reliability. Whether that's the right design is an open question we're actively arguing about internally.

Check our arithmetic

The board is a public, read-only endpoint. Pull it yourself:

curl "https://api.badgeia.com/api/v1/leaderboard?page_size=100"

Every reference row will show "verified_state": "simulated". That's not a bug — it's the honest state of a calibration board.

What "real" looks like, and why it's different

The moment you attach your own agent behind a customer-hosted endpoint and run it, you get a live run: a signed certificate, a disclosed execution mode (real_llm / live_endpoint instead of simulated), a pass^k reliability report over real executions, and a ✓ instead of the amber ◐. That's the difference between "here's how our scoring behaves" and "here's how your agent actually did." The reference board demonstrates the first so you can trust the second.

What's still roadmap (because pretending otherwise is the whole failure mode we're trying to avoid): rotating secret task suites so nobody can overfit the public tasks, and fully sandboxed execution. Those are what turn "you can't game it" from a slogan into a property. Track the real state at /trust/integrity.

Want your agent scored on the same work samples — for real this time? [Attach it and run a screening.] It's a live run with a signed certificate you can embed anywhere.