A quick, intentionally straightforward programming test. Build a working pipeline engine, then refactor it against new requirements. Early mistakes make the second stage difficult.
Assignment 2 is revealed only after Assignment 1 is complete. The test measures adaptation, not advance planning for a disclosed second brief.
MagLev scored 95 with Opus 5, GPT-5.6-SOL, and Grok 4.6. All three shared the top score with Grok Terminal.
Opus 5: 95 vs 94 · GPT-5.6-SOL: 95 vs 92 · Grok 4.6: 95 vs 95 · Gemini 3.1 Pro: 93 vs 0 (DNF)
Native harnesses: Claude Code, Codex, Grok Terminal, and Gemini CLI, respectively. Gemini CLI did not submit an artifact; its zero is a non-completion, not a completed-software grade. Quality scores are the reported Benchmark 1 rubric results, not a new evaluation. The V1 arms were not externally graded and are not inferred here.
The mean of the three metered non-MagLev runs against the mean of the three metered MagLev runs. Equal weight for every run. V1 endpoint results follow below; they are not mixed into this progress curve.
Each run is independently normalized to 0–100% progress, with linear interpolation onto a 5% grid. Progress indicates position within the recorded session—not verified feature completion. No curve smoothing is applied.
Includes all four MagLev and five non-MagLev arms, including OpenRouter Auto. This is an unequal-model-mix descriptive average, not a paired causal comparison. Some source exports lack reliable progress identifiers, so no intermediate progress points are invented.
Seven direct comparisons, with the same underlying model on each side. Cumulative progress curves are shown first; V1 completion totals follow. Lower cumulative values mean less work.
Primary decode KV traffic and the separate uncached counterfactual. Both are token-derived proxies, not measured hardware FLOPs. Decode KV traffic charges context size against every decoded token, so no cache policy can move it.
Primary decode KV traffic and the separate uncached counterfactual. Both are token-derived proxies, not measured hardware FLOPs. Decode KV traffic charges context size against every decoded token, so no cache policy can move it.
Primary decode KV traffic and the separate uncached counterfactual. Both are token-derived proxies, not measured hardware FLOPs. Decode KV traffic charges context size against every decoded token, so no cache policy can move it.
The main curves show causal attention pairs. Expand each comparison to view primary decode KV traffic (S3S) and the uncached scenario (S4) side by side; the latter is counterfactual, not actual hardware work.
Completion totals only; intermediate progress is not established by the supplied export.
Completion totals only; intermediate progress is not established by the supplied export.
Completion totals only; intermediate progress is not established by the supplied export.
Completion totals only; intermediate progress is not established by the supplied export.
KV-cache and total modeled memory-traffic energy, compared at the same normalized progress. Dashed lines are KV-cache energy; solid lines are total modeled energy. Both curves start at zero and accumulate, so the widening gap is the saving.
At completion: 56.5% less KV-cache energy and 38.7% less total modeled energy.
At completion: 82.1% less KV-cache energy and 64.0% less total modeled energy.
At completion: 32.0% less KV-cache energy and 1.2% less total modeled energy.
Total modeled energy includes KV-cache and decode weight traffic. These are estimates under identical fixed assumptions—not measured device or datacenter electricity.
KV-cache savings: 84.8%. Total modeled savings: 70.8%. Negative savings mean higher energy.
KV-cache savings: 63.0%. Total modeled savings: 33.5%. Negative savings mean higher energy.
KV-cache savings: -23.2%. Total modeled savings: -30.7%. Negative savings mean higher energy.
KV-cache savings: 55.0%. Total modeled savings: 24.5%. Negative savings mean higher energy.
Every figure above is a session total, and a total punishes the system that produced more software. These surfaces divide the same metered work by the output tokens that work produced. Same logs, same formulas, corrected units — and the gap widens.
Mean across these pairings. Work divided by the output it produced, so a longer answer is never charged as a loss.
What the end of the session costs — not the average of an easy first hour and an expensive last one.
How much the prompt grows from the first decile of calls to the last. Dimensionless, so it compares across every vendor.
269 output tokens per second against 147, at each arm's own mean context.
| Pairing | Per delivered token | Terminal rate | Escalation | Sessions / card | Decode rate |
|---|---|---|---|---|---|
| MagLev (Opus 5) vs Claude Code (Opus 5) · V1 | 11.7× | 2.44× | 3.10× | 2.96× | 2.96× |
| MagLev (GPT-5.6-SOL) vs Codex (GPT-5.6-SOL) · V1 | 4.84× | 26.9× | 1.06× | 2.04× | 2.04× |
| MagLev (Gemini 3.1 Pro) vs Gemini Terminal (Gemini 3.1 Pro) · V1 | 1.64× | 1.25× | 2.14× | 1.13× | 1.13× |
| MagLev (Grok 4.6) vs Grok Build Terminal (Grok 4.6 Build) · V1 | 5.55× | 509× | 6.14× | 1.85× | 1.85× |
| MagLev (Opus 5) vs Claude Code (Opus 5) · V2 | 2.31× | 1.96× | 1.77× | 1.89× | 1.89× |
| MagLev (GPT-5.6-SOL) vs Codex (GPT-5.6-SOL) · V2 | 3.83× | 2.44× | 1.59× | 2.01× | 2.01× |
| MagLev (Grok 4.6) vs Grok Terminal (Grok 4.6) · V2 | 2.51× | 17.3× | 5.05× | 1.68× | 1.68× |
A model call is bounded by how much KV state has to be streamed from memory for every token it writes. Context size therefore converts directly into how many of these sessions fit on one card — here, at each arm's own mean context. The strongest terminal result on this page is MagLev (Grok 4.6) vs Grok Build Terminal (Grok 4.6 Build) · V1 at 509×.
Ratios are MagLev advantage: how many times more compute the native harness spent to deliver the same unit of output. Sessions per card and decode rate assume one 80 GB H100 SXM at 3.35 TB/s with decode bound by KV traffic; the constants are fixed and identical on both sides. Terminal rate uses the final decile of model calls, with a minimum window so a short arm reports no rate rather than a single-sample one.
Compute and energy cover Opus 5, GPT-5.6-SOL, and Grok 4.6 only. Gemini quality results are included above but are outside this metered dataset. MagLev / Grok includes unobserved usage records that are excluded from compute and energy, not treated as measured zero; the reported totals therefore omit that unknown work. Grok Terminal does not separately identify cache creation, but all fresh input remains charged. Console reconciliation has not been established.
The same per-record definitions and energy assumptions are used on both sides. Sources: Flight 3 row-level compute metrics, compute-energy comparison records, and the B1 full-field grading report. This page reports results without describing MagLev’s internal implementation.
MagLev matched or exceeded paired quality while using less reported attention compute and modeled energy in all three graded model pairings. The V1 runs add four further lower-traffic results.
← Back to all benchmarks