MagLev.
Research & evaluation
Benchmark 04
← Back to all benchmarks
Benchmark 04 · Autonomous product build

Eleven prompts in.
A working enterprise application out.

Build DealCommander: a sell-side M&A deal room for an advisor at the highest level. Eleven navigation areas, six mandatory keystone contracts stated up front and tested at the end, a cloud database schema, optional AI drafting that must degrade to deterministic local templates, and an in-app diagnostics suite that proves its own requirements. Apple-grade design quality. No system was shown a prompt before its turn, and no prompt could be re-explained.

This is the benchmark where the output stops being a test answer and becomes a product. Not a script, not a toy, not a demo harness — data tables, a live cloud backend, a real AI API integration, and a diagnostics suite that audits the brief it was built from. You can open the MagLev deliverable on this page and use it.

11 sequential prompts6 keystone contracts tested at the end3 metered compute pairings, 2 flightsZero human intervention beyond the prompts
What this benchmark tests
  • Shipping a real application: Eleven navigation areas covering dashboard, buyer universe, process timeline, bid tracker, diligence Q&A, board memo, partner brief, AI assistant, import and export, cloud setup, and diagnostics.
  • Honouring contracts stated once, at the start: An exact global API surface, a fixed state shape, seeded data, defined domain math, a storage contract, and diagnostics — all declared in the first prompt and verified in the last.
  • Dispatch-only state: Every mutation must travel through one reducer, with undo and restart behaving predictably.
  • Graceful degradation: AI features must be optional and must fall back to deterministic local templates with no key present.
  • Security discipline: No hardcoded credentials anywhere in the source tree, and a cloud schema with row-level policies.
  • Self-verification: An in-app diagnostics suite, including a map from each brief requirement to the checks that cover it, that fails any requirement no check touches.
  • Runnable by a person: The deliverable must open and work by the canonical path, with no build step required.
  • Code discipline at scale: Durable structure over fragile complexity, across a codebase of real size.
13,521
Lines of product JavaScript

Across 107 files, with 1,820 lines of CSS and 822 lines of SQL — written autonomously from eleven prompts.

0
Product files over 500 lines

Median file 108 lines, largest 351. Its counterpart on the identical model shipped 7 files over 500, the largest at 2,865.

97.0
Graded quality, Opus 5

Against 88.5 for the native harness on the identical model. MagLev won all four graded pairings in this benchmark.

6.8×
Its counterpart’s context per call

350,498 tokens on every call against MagLev’s 51,608, and it ended at 609,663 while MagLev ended at 42,218.

Every compute figure on this page is computed per request from each system’s own metered logs, using identical formulas on both sides. Open the software MagLev built →

01 / The delivered product

The deliverable is the evidence.

Eleven prompts went in. What came out is a sell-side M&A deal room with full data tables, a Supabase cloud backend, and a live AI API integration for drafting and communication. Other than the eleven prompts, there was no human interaction — no correction, no steering, no repair round.

Codebase

107 files

13,521 lines of product JavaScript, organised across nine modules: core, domain, state, storage, api, ai, ui, diagnostics, and seed data.

Cloud backend

8 tables

822 lines of SQL — schema, row-level security policies, and seed data — for an instant connection to a Supabase project.

AI integration

Real API

A live provider-agnostic AI client for drafting and communication, with no credential anywhere in the source tree and a deterministic local-template fallback when no key is present.

Self-verification

Requirement map

An in-app diagnostics suite that maps every requirement in the brief to the checks covering it, and fails any requirement no check touches.

You do not have to take the grade on faith. The application MagLev built is published on this site. Open it, click through all eleven areas, run its diagnostics, and leave when you are done.

Spotlight: the software it built → Launch DealCommander

Counts are measured directly from the delivered source tree: product files only, excluding test harnesses and tooling (6 files, 2,029 lines). Code volume is not a quality score; it is reported here because the structural comparison below is about how that volume is organised.

02 / Graded quality and structure

Four graded pairings. Four MagLev wins.

Each deliverable was graded against the same rubric built from the brief. In every pairing, the MagLev arm and its comparison system ran the same model.

ModelMagLevComparison systemDifference
Opus 59788.5 · Claude Code+8.5
Grok 4.69283 · Grok Terminal+9
GPT-5.6-SOL88.576 · Codex+12.5
Gemini 3.1 Pro6562 · Gemini CLI+3

The structural comparison is where the difference becomes concrete. Both sides of the Opus 5 pairing produced a working application of similar scale. Only one of them produced a codebase a team could take over on Monday.

MeasureMagLev (Opus 5)Claude Code (Opus 5)
Product JavaScript files10718
Product JavaScript lines13,52110,949
Median file length108246
Largest product file3512,865
Product files over 500 lines07
Lines of SQL delivered822844
In-app diagnostic checks10640
Requirement-to-check coverage mapIncludedNot reported

Same model. Same brief. Same eleven prompts. One side produced 107 files with a median length of 108 lines and nothing over 500. The other produced 18 files, seven of them over 500 lines and one at 2,865. Both compile; only one is maintainable, and no prompt in the brief asked for either outcome.

Product-source counts exclude test harnesses and tooling on both sides. The apparent extras in the comparison codebase were audited against the brief and found to be unrequested scope, including one database table with no call sites in its own source. Scores are compared within this study, not averaged across rubrics.

03 / Context carried on every call

Three lines hold flat. Three climb until something has to give.

This is the size of the prompt on each individual model call, from the first call of the build to the last. It is the one chart that explains every other number on this page — and it is where the two approaches stop resembling each other.

ArmMean tokens / callMedianPeakFinal call
Claude Code (Opus 5) · V2350,498364,569609,663609,663
Claude Code (Opus 5) · V1561,071585,316966,00738,251
Codex (GPT-5.6-SOL)116,937122,242202,21658,254
MagLev (Opus 5) · V251,60846,327129,72342,218
MagLev (Opus 5) · V146,59443,616107,68135,022
MagLev (GPT-5.6-SOL)26,31625,77668,92118,638

In V2, the native harness on Opus 5 averages 350,498 tokens per call and ends at its own peak of 609,663 — still climbing when the work runs out. In V1 it climbed to 966,007 and then shed 927,756 tokens in a single step, down to 38,251. Codex did the same thing at 91.4% complete, falling from 202,216 to 31,826. MagLev on the identical model held 51,608 and 46,594 in the two runs, and never exceeded 129,723 in either.

That vertical drop is not a saving. It is compaction: the harness reached the edge of the window, and the session it had been accumulating all build was summarised and thrown away. The cost was already paid to carry it, and the detail does not come back. Every requirement from the early prompts that mattered at the end was inside it.

MagLev never reaches that wall, because it was never walking toward it. Persistent memory means the project is remembered rather than re-read, so the eleventh prompt is answered with the same discipline — and the same cost — as the first. Nothing has to be summarised away, because nothing was being hoarded in the first place. MagLev stays on track like a maglev: the thread is never lost, and it never has to be reconstructed.

Prompt tokens per call as reported by each provider’s own meter, interpolated onto the shared progress grid. Cached and uncached input are both included: this is the size of the package the model was handed, before any billing discount. MagLev arms also show call-to-call variation — they operate from a low ceiling by design, not by truncation at a limit.

04 / Average cumulative quadratic attention

2.77× less attention compute at completion.

The mean of the metered non-MagLev arms against the mean of the metered MagLev arms. Equal weight for every arm; both curves start at zero and accumulate across the build.

At completion: 840 billion attention pairs across 2 comparison arms against 303 billion across 2 MagLev arms — a 63.9% reduction. One metered pairing — Codex against MagLev on GPT-5.6-SOL, V1 — is excluded from this average and disclosed rather than diluted into a mean: on that one small-scale pairing MagLev accumulated more attention pairs, having done several times the work volume and produced several times the output. Each arm is independently normalized to 0–100% progress, with linear interpolation onto a 5% grid. Progress indicates position within the recorded session — not verified feature completion. No curve smoothing is applied.

05 / Per-pairing energy savings

Less modeled memory-traffic energy, in every pairing.

Cumulative modeled memory-traffic energy at the same normalized progress. Dashed lines are KV-cache energy; solid lines are total modeled energy.

Claude Code (Opus 5) · V2 vs MagLev (Opus 5) · V2

Opus 5 · V2

47.8% KV energy saved

At completion: 47.8% less KV cache energy.

Claude Code (Opus 5) · V1 vs MagLev (Opus 5) · V1

Opus 5 · V1

73.8% KV energy saved

At completion: 73.8% less KV cache energy.

Codex (GPT-5.6-SOL) vs MagLev (GPT-5.6-SOL)

GPT-5.6-SOL · same model

64.6% KV energy saved

At completion: 64.6% less KV cache energy.

Total modeled energy includes KV-cache and decode weight traffic, under identical fixed assumptions on both sides. These are estimates derived from measured token surfaces — not measured device or datacenter electricity. Where a total-energy surface is not shown, the MagLev arm was higher on it: in three of these four pairings MagLev generated substantially more output, and output tokens carry decode weight traffic regardless of how small the prompt is.

06 / Compute per delivered token

Divide by the work actually delivered.

Every figure above is a session total, and a total punishes the system that produced more software. These surfaces divide the same metered work by the output tokens that work produced. Same logs, same formulas, corrected units — and the gap widens.

15.4×
Compute per delivered token

Mean across these pairings. Work divided by the output it produced, so a longer answer is never charged as a loss.

38.2×
Terminal rate, final decile

What the end of the session costs — not the average of an easy first hour and an expensive last one.

5.91×
Less context escalation

How much the prompt grows from the first decile of calls to the last. Dimensionless, so it compares across every vendor.

4.52×
Decode throughput

283 output tokens per second against 63, at each arm's own mean context.

PairingPer delivered tokenTerminal rateEscalationSessions / cardDecode rate
MagLev (Opus 5) vs Claude Code (Opus 5) · V141.0×146×8.63×12.0×12.0×
MagLev (GPT-5.6-SOL) vs Codex (GPT-5.6-SOL) · V19.69×2.23×1.14×4.44×4.44×
MagLev (Gemini 3.1 Pro) vs Gemini Terminal (Gemini 3.1 Pro) · V17.80×15.4×4.77×5.40×5.40×
MagLev (Grok 4.6) vs Grok Terminal (Grok 4.6) · V213.5×34.5×4.38×3.37×3.37×
MagLev (Grok 4.6) vs Grok Terminal (Grok 4.6) · V118.6×27.3×4.00×3.67×3.67×
MagLev (Opus 5) vs OpenRouter (Auto Mode) · V13.35×12.9×13.4×1.97×1.97×
MagLev (Opus 5) vs Claude Code (Opus 5) · V210.8×40.5×7.95×6.79×6.79×
MagLev (GPT-5.6-SOL) vs Claude Code (Opus 5) · V318.7×27.6×2.95×7.45×7.45×

The capacity statement.

A model call is bounded by how much KV state has to be streamed from memory for every token it writes. Context size therefore converts directly into how many of these sessions fit on one card — here, at each arm's own mean context. The strongest terminal result on this page is MagLev (Opus 5) vs Claude Code (Opus 5) · V1 at 146×.

Ratios are MagLev advantage: how many times more compute the native harness spent to deliver the same unit of output. Sessions per card and decode rate assume one 80 GB H100 SXM at 3.35 TB/s with decode bound by KV traffic; the constants are fixed and identical on both sides. Terminal rate uses the final decile of model calls, with a minimum window so a short arm reports no rate rather than a single-sample one. This section publishes only pairings where MagLev finished ahead on every surface shown. Held back and named here instead: MagLev (GPT-5.6-SOL) vs Codex (GPT-5.6-SOL) · V2 (context escalation at 0.56×).

Eleven prompts.
No human touched it again.

A sell-side M&A deal room with full data tables, a live cloud backend, and a real AI API connection — built autonomously, graded highest in its field, and delivered in 107 files with nothing over 500 lines. On the identical model, with the identical prompts, the native harness scored lower, shipped a 2,865-line file, and ran its context up to the wall until the session had to be thrown away.

All of it comes from one property: persistent memory. MagLev remembers the project instead of re-reading it, so the thread is never lost, the prompt never swells, and the eleventh prompt costs what the first one cost. That is what MagLev does — it takes the frustration out of creation.

Spotlight: the software it built → Launch DealCommander ← Back to all benchmarks