06 / Benchmark Harness

BACKEND BENCHMARK RUNNER

LAB IDLE

The backend instruments itself: the same four workload tasks run through competing systems and every run emits a BenchmarkRecord — the common measurement language. GRAVITY-Static pins the kernel OFF, so the delta between the two GRAVITY rows IS the architecture's contribution.

Why spend frontier-level intelligence when lower-cost computation is sufficient? Same task · same input · same criterion — measured evidence only.

openrouter · not setdeepseek · not set

CLASSES8 combos

JEV excluded — not wired · sign in to run · leaderboard free for allruns / classseed

SHIP GATES · LIVE SELF-ASSESSMENT

○ OPEN

Instrumentation

run once to emit records

○ OPEN

Provider pinning

no frozen config yet

○ OPEN

Deterministic A/B/D

regex + rule checks, no LLM judge

○ OPEN

Dual-mode pair

run GRAVITY + STATIC together

○ OPEN

Cost decomposition

no measured cost yet

● PASS

No fabricated cells

unmeasured renders —, never 0 — enforced in code

○ OPEN

Reproducibility

run once to freeze config

○ OPEN

Provenance

no records yet

NO MEASUREMENTS YET

Pick systems + classes above, then press RUN.

Every cell below fills only from real runs — never placeholders.

1 · 1 system × 1 class ≈ 1 min2 · GRAVITY + STATIC × A–D ≈ 8 min3 · Export the slide PDF

COST STORY · CORRECT AMOUNT OF INTELLIGENCE

Worked example — L0 python $0.000 · stats $0.001 · small LLM $0.008 · verify $0.002 · orchestration $0.001 = $0.012 vs one frontier call $0.06–0.10.

Run GRAVITY + a frontier baseline to print the live multiple.

Deterministic success criteria — no LLM-vibes judging