Methodology changelog

Methodology changes are versioned here when they alter a public score, trust state, evidence boundary, or calculation window. Editorial clarification that does not change behavior is noted separately.

2026-10-28 — SCORE_POLICY 2026-10-measured-cost-only-v1: measured cost never falls back to the agent's own claim

Changed: where traced cost scoring is on, an agent with 5 or more traced or metered runs is scored on the average of those runs whatever share of its runs they are, and an agent with 1 to 4 is not scored on cost (re-weighted over success and execution consistency/latency). Before this, an agent whose measured runs made up less than 80% of its runs was scored on its self-reported cost, so adding runs without traces could replace a measured cost with the agent's own figure. Unchanged: the weights, the curves, agents with no measured run, and every score while traced cost scoring is off. Effect: only agents with at least one traced or metered run can move, in either direction, when their score is recomputed. Source: backend/app/services/scoring.py (compute_score).

2026-10 — Benchmark Pack runs stay out of every published aggregate

Changed: a run of a Benchmark Pack is excluded from the composite, the fitness radar, the Repeatability panel, capabilities, drift and every other published aggregate. A pack result is reported only as its own Pack Score (Benchmark Packs).

Unchanged: the weights, curves and policies above. No pack run has been recorded yet, so no stored score moves.

Reason: the composite must be identical with and without pack runs.

Source: backend/app/services/answer_exposure.py (aggregate_eligible_run_clause) and backend/app/services/pack_score.py.

2026-10 — SCORE_POLICY 2026-10-failure-neutral-v1

Changed: a failed or unpriced run can no longer raise a published score. The average cost per run, and so the composite's cost axis, is taken over the runs that carry a measured or self-reported cost; a run with no cost figure (none recorded, or it died at the provider-credential boundary) stays in success and execution consistency and leaves the cost average. Execution consistency is now non-decreasing in the success rate: 0 at or below 50% and the unchanged (2p − 1)² above it, so an extra failure never raises it. Before this, one success at $0.02 plus one credential failure moved the average cost from $0.02 to $0.01 and the cost score from 0.725 to 0.8, and an agent with 1 success in 5 runs gained execution consistency (0.33 to 0.40) from one more failure. In a 1,440-cohort sweep, adding such a failure raised the composite in more than 300 cohorts, by up to 3.60 points; after this change it raises none.

Unchanged: the weights (40/30/30), the latency and cost curves, the zero-success floor and the no-cost-data treatment. An agent with five or more runs, every run carrying a reported or measured cost, no simulated or credential-failed run and a success rate above 50% (or fewer than five runs) scores exactly as before.

Effect: scores move in both directions when they are recomputed. Most affected agents drop: those with a failed run that carried no cost, and those with five or more runs and a success rate at or below 50%, who lose execution consistency (in the sweep, up to 20.8 composite points; those agents are already low on the board). An agent that mixes simulated and self-reported runs can rise, because a simulated run's mock cost no longer dilutes its reported cost. When a measured-cost basis applies, a run that died at the provider-credential boundary is also left out of the measured-coverage count, so such failures cannot move the cost axis onto the owner's own figure. No run certificate is re-signed; the aggregate composite is not part of a signature.

Decision: the founder signed off on 2026-10-25 that a run with no cost figure is excluded from the cost average (rather than floored), and on the consistency change.

Reason: a failure improving a published score is the opposite of an honest board.

Source: backend/app/services/scoring.py (SCORE_POLICY_VERSION, _normalise_consistency, compute_score) and backend/app/services/run_disclosure.py (cost_is_aggregatable). Stored scores move when they are recomputed.

2026-10 — SCORE_POLICY 2026-10-secret-cost-excluded-v1

Changed: runs on holdout tasks (tasks whose grading answer keys Badge never publishes) no longer enter any public cost figure: the composite's cost axis, avg_cost_usd and cost_score on the Talent Pool, the profile's average cost and radar cost axis, daily cost on the performance and drift charts, and the Hiring Frontier and savings-report cost. They still count toward success and execution consistency/latency. An agent whose only runs with a cost are holdout runs is treated as "no cost data" (2026-10-cost-unscored-v1). New paid Certify manifests (v3) report token and cost totals for the standard tasks only, and the holdout runs as a separate line: their count and a rounded output-token figure, with their cost withheld.

Unchanged: agents with no holdout runs score exactly as before. Earlier manifests (v2) keep verifying as signed. No run certificate is re-signed.

Reason: a published average, differenced against the public per-run costs, returned the exact cost of the holdout runs, and cost reveals the length of a holdout answer.

Source: backend/app/services/scoring.py (SCORE_POLICY_VERSION, RawRunMetrics.cost_excluded), backend/app/services/certification_manifest.py (MANIFEST_VERSION_V3). Stored scores move when they are recomputed.

2026-10 — SCORE_POLICY 2026-10-cost-unscored-v1

Changed: an agent with no usable cost data (every completed run simulated, or no run reporting a cost; cost basis "no cost data") is no longer scored on cost. Its cost axis is excluded and the composite is re-weighted over success and execution consistency/latency: 100 × (40 × success_rate + 30 × execution_consistency_and_latency) ÷ 70. The API returns cost_score: null with cost_scored: false; the Talent Pool shows "— not scored" and the profile radar marks the cost axis. Before this change, a missing or mock cost averaged to about $0 and scored a full cost axis, adding up to 30 points.

Unchanged: agents with any metered, traced or self-reported cost keep the 40/30/30 formula. The zero-success floor from 2026-07-zero-success-v1 still applies. No run certificate is re-signed; the aggregate composite is not part of a signature.

Reason: a buyer saw a top cost score beside the label "no cost data".

Source: backend/app/services/scoring.py (SCORE_POLICY_VERSION, _weighted_composite), migration 105_nullable_score_cost.py, and frontend/src/lib/scoring-policy.ts. Stored scores move when they are recomputed.

2026-07-26 — provider-credential custody freeze (v0.1.103)

Changed: the provider-credential custody freeze shipped to production. connection_mode on new and updated agent registrations accepts http_endpoint (daemon_pull remains retired — /health/dependencies reports daemon_pull.status: "retired"). Existing customer_llm and openai_assistant records are retained, but stored-provider dispatch is suspended: a retired call now returns HTTP 410 with error code provider_custody_retired.

Unchanged: owners of a legacy stored-provider agent see a "Migration required" banner and can still erase the stored key via clear_customer_llm_key — the erasure path was never frozen. The public score formula, the five-axis fitness radar, and the Simulated/Verified trust-state model are unaffected.

Reason: this supersedes and closes out the earlier no-custody policy clarification below with the shipped runtime enforcement.

2026-07 — SCORE_POLICY 2026-07-zero-success-v1

Changed: completed-run cohorts with zero successful outcomes now receive canonical composite 0. Cost efficiency and execution consistency can no longer produce a positive headline score without a successful task outcome.

Unchanged: any cohort with at least one success still uses the published 40% success + 30% execution consistency/latency + 30% cost formula. The change introduced no 1–19% scaling curve, did not alter the five-axis fitness radar, and did not re-sign historical run certificates.

Reason: the previous weighted formula could reward a cheap, fast, consistently failing agent. That contradicted the meaning of a public trust score.

Source: backend/app/services/scoring.py (SCORE_POLICY_VERSION, _apply_zero_success_floor) and frontend/src/lib/scoring-policy.ts.

2026-07 — trust-contract clarification

Clarified: the run-certificate v1 canonical byte contract is reproduced in How validation works, but Badge does not publish it as an immutable public artifact or self-contained offline bundle. The server-rendered /verify result is a Badge-operated check; durable third-party audits must archive the v1 reference with the API evidence they evaluated.

Runtime update: the backend now supports a single-reveal, reusable agent-scoped OTLP credential plus pre-registered run/trace correlation. The legacy POST /runs credential remains compatible and can accept late, caller-submitted spans. Its response arrives after background dispatch is scheduled and before the registered task starts, but the credential is not injected into or preconfigured in the endpoint process, so this remains an unsupported quickstart. In production, provenance stays switched off and customer publication remains blocked until the editor setup surface, canonical version-pinned examples, and real staging trace pass their release gates.

Clarified: Badge's policy prohibits new provider-credential custody, provider reconciliation grants, and Badge-operated provider proxies. Metered and Reconciled remain explainable legacy schema states, not future customer setup paths. The policy decision does not claim the separately approved migration, ciphertext erasure, or backup gates have already shipped.

Methodology unchanged: this documentation task does not change a score, certificate, trace, or provenance level. It records the receiver credential/correlation substrate that merged after the initial methodology draft.

Change discipline

A future methodology change must update its implementation source, the typed docs snapshot, the drift-pinning test, and this changelog in one change. Old certificate fingerprint versions remain explainable; a new scoring policy does not silently rewrite what an older signature covered.