MCP server against files: the full report.

• Does an agent connected only to the Amaca MCP server apply the system at least as well as one given files, at a lower cost?
• Does it call check_compliance on its own before handing over?
• Of the routes that use files, which works best?

Four models from one vendor ran as subagents in a cloud coding session, one model at a time, after a pilot. Each condition gave the agent a different way into the system; the tools each subagent could use were restricted per condition, and every tool call was logged. Model names are left out of this report on purpose.

Each output was scored 0 to 100 on token discipline, use of the registry, the gap protocol, variants, the 85/10/5 law, the accessibility floor and valid output. The repository's own compliance scan counted errors at handover, and a context share measured how many classes and var() in the output resolve to Amaca. Tokens, time and tool calls came from the session.

In one sentence. At the same quality, around 93, the server route uses a third of the skill's context and half of DESIGN.md's. The short files cost little but stop at 76 to 79: they name the components, not their parts, and the agent invents the sub-parts.

Averaged over the seven common conditions, the four models scored 77.2, 78.1, 77.8 and 74.0. The last is incomplete: ten of its runs were skipped at a usage limit, and some of its cells have two runs instead of three.

Under M1, check_compliance was called before the final answer in 34 of 83 runs, 41%. In 24 of them the code was already clean. The average hides the models: 3 of 21, 7 of 21, 11 of 21 and 13 of 20. The rate is a property of the model, not of the system.

The matrix of outputs turned up two false positives in check_compliance. The 0.01ms override inside prefers-reduced-motion, which the system prescribes, accounted for all 94 raw-duration findings outside C0. The skip link parked off screen, also prescribed by DESIGN.md, hit 19 runs of C2b and C4. One agent wrote animation: none in place of the prescribed kill switch and said it did so to avoid tripping the verifier: the false positive was pushing it out of the contract.

Both were fixed and shipped in v5.3.0. Clean outputs from this run and mutants with one violation each now guard the scan, so a rule that starts flagging what the system prescribes fails before release.

• C0 is not a blank baseline. Every subagent received the server's instructions, so C0 means "no file", not "no context". One C0 subagent tried to call get_component and was refused.
• One model in four did not receive the server's instructions, only the tool descriptions. M1b was added for it. This is a fact about clients.
• The scorer has limits. The gap criterion is rule-based (33 runs are marked for a reread by hand); validity of HTML is checked leniently; the registry criterion gives half a point to plain Tailwind.
• Leaks: 0 in 598 runs; reading under downloads/ injects downloads/CLAUDE.md, so C1 and C2a used copies placed elsewhere.
• Budget: 27.6M tokens against 20.7M projected from the pilot; the gap comes from one model on C2b and C4, at 150k to 190k tokens per run. Two usage limits interrupted 12 runs: 6 were redone, 10 skipped.
• Not reported here: a blind comparison of one output per condition on task 7, still waiting for the second judge. Not run: the tests on real chat clients and in Figma.

• v5.3.0: the two false positives fixed; a regression check on the scan.
• v6.0.0: the server names what is off-system and tells the agent to stop and ask; the short files carry each component's root class, parts and variants; the scan sees a px inside a transform, a token used but never declared, and a token that does not exist.
• Next: the same run on 6.0.0, to see whether those changes close the gaps they were written for.

Back to the case study