In the claude-sonnet-4-6 → claude-sonnet-5 release, tasks that
require raw JSON output began arriving wrapped in a Markdown code fence
(```json) even when the prompt explicitly says no markdown. The JSON
inside is correct — but a strict JSON.parse / json.loads on the raw
response now fails where it previously succeeded. Flip rate: 5/5 on structured-output
tasks, deterministic. The two competing releases we measured in the same window
showed no comparable change.
The diff
Fixed battery, six production-shaped archetypes, N=5 trials per task, temperature 0, deterministic checkers, pinned seed. We report pass rate by category only — never the underlying prompts.
| Archetype | claude 4-6 → 5 | gpt 5.5 → 5.6-sol | gemini 3fp → 3.5 |
|---|---|---|---|
| Structured output (JSON) | 25/25 → 0/25 | 25/25 → 25/25 | 25/25 → 25/25 |
| Extraction | 24/25 → 18/25 | 25/25 → 24/25 | 25/25 → 25/25 |
| Multilingual (JSON) | 25/25 → 14/25 | 25/25 → 25/25 | 25/25 → 25/25 |
| Instruction adherence | 24/25 → 25/25 | 25/25 → 24/25 | 21/24 → 20/25 |
| Tool / function calling | 25/25 → 25/25 | 25/25 → 25/25 | 25/25 → 25/25 |
| Agentic multi-step | 25/25 → 25/25 | 25/25 → 25/25 | 25/25 → 25/25 |
| Total | 99% → 71% | 100% → 99% | 98% → 97% |
We flag a regression only when ≥3 of 5 trials fail on a task that previously passed ≥4 of 5 — so single-trial wobble (the 1-point dips for gpt and gemini) does not count. By that rule, eight tasks regressed, all on one model, all from one behavior.
What changed
Every regressed task shares a single, isolable cause: the model now prepends
```json and appends ``` around JSON responses that were requested as
raw objects with an explicit "no markdown" instruction. It is not a
content-quality drop — field values, types, and structure inside the fence are correct. It is a
change in envelope: the model treats a raw-JSON instruction as a request for a fenced
code block.
The practical effect is narrow but real: any consumer that strictly parses the raw completion
— json.loads(resp), JSON.parse(text), a Pydantic
model_validate_json, a downstream schema gate — now throws on the first character.
Consumers that already strip fences, or that use the provider's structured-output / tool
interface rather than free-text JSON, are unaffected.
What did not change
Just as important for anyone deciding whether to worry:
- Tool / function calling: 25/25 → 25/25. JSON delivered through the tool interface is untouched; the fence behavior is specific to free-text output.
- Agentic multi-step: 25/25 → 25/25. Plan-execute and state tracking held.
- Instruction adherence (non-JSON): 24/25 → 25/25. General instruction-following did not degrade — it nudged up. This is the control that matters: the change is surgically the JSON-envelope behavior, not a broad regression.
Did this hit you?
You're exposed if, on Sonnet, you ask for raw JSON in the prompt and
json.loads / JSON.parse the completion directly, and
do not strip a leading/trailing code fence before parsing. If that's your path, this update
fails your parse silently on structured, extraction, and multilingual-JSON workloads. Two ways
out, both small:
- Strip the fence before parsing — trim a leading
```json/```and trailing```(a three-line guard), or - Move JSON to the tool / structured-output interface, which this release leaves at 25/25.
Reproducing this yourself costs one full battery — six archetypes × five trials × the current-and-previous version of every provider you ship on — every time a model updates. That's the point of subscribing: we run it the day a version ships and send you the diff.
Method (public at category level)
- Fixed battery, six archetypes, each five tasks. Every task is generated from a template with randomized surface details per run, so raw inputs are never a reusable artifact (contamination-resistant) while runs stay reproducible from the seed.
- N = 5 trials per task, temperature 0 (or provider default where 0 is rejected), pinned seed — identical across every model and version, so a diff is drift, not a different draw.
- Deterministic checkers — no LLM-as-judge. A JSON task passes only on a clean parse, exact key set, and matching values/types.
- Infrastructure errors (rate limits, 5xx, truncation) are separated from behavior and excluded from pass rate; this run had one such trial (gemini predecessor), excluded.
- We publish the diff and the category. We never publish the task inputs.
Caveats
- N = 5 is a screen, not a proof. It reliably catches deterministic, high-flip changes like this one; it will miss rare intermittent regressions. The ≥3/5-on-≥4/5 rule is deliberately conservative.
- Single-trial movements on gpt-5.6-sol and gemini-3.5-flash are within noise and are not claimed as regressions.
- This measures behavior on our battery, not your exact workload. The paid tier maps these archetypes onto your specific workflows.
- Not affiliated with any model provider. We measure and report; we do not use any provider's output to tune or advantage another.