Constrained Decoding (Structured Outputs).

🐛The bug report

Extraction pipeline, 40k docs/day. 98.7% clean. About 500/day blow up:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ValidationError: price — expected float, got "1,299 USD"

You already set response_format={"type":"json_object"}. So why is it still breaking?

🔍Step 1 — log the raw string, not the exception

Print the completion BEFORE parsing. Three different bugs are hiding behind one stack trace:

A) ``json\n{...}\n`` → fence leaked in. That's the "char 0" crash.

B) {"price": "1,299 USD"} → perfectly valid JSON, wrong types.

C) {"item":"widget","price":12.5 → truncated, never closed.

🧠Step 2 — the concept

Two very different mechanisms both get called "JSON mode":

• Prompt-level ("please reply in JSON"): the model usually complies. Failure A lives here.
• Constrained decoding: at every decode step the sampler builds a mask over the whole vocabulary and zeroes out any token that could not continue a valid parse. A } that would close an unopened object gets probability 0 — not "unlikely", impossible.

Your JSON Schema is compiled into a state machine that walks beside the decoder. Illegal tokens are unemittable, so A (and drift-style C) disappear by construction, not by luck.

🎯Step 3 — what it does NOT fix (this is the intermediate part)

• Semantics. The grammar enforces "a number", not "the right number". {"price": 0} is schema-perfect and wrong. B survives.
• Truncation. Hitting max_tokens mid-object still yields unbalanced JSON. Check finish_reason == "length" before parsing — a grammar can't grant extra token budget.
• Quality. Constraining isn't free. Schema field order IS generation order, so a schema that puts the answer first forces the model to commit before it reasons. Fix:

  {"reasoning": "...", "price": 12.5}
Scratchpad field first, answer after. Measurable accuracy gap on hard extraction.

• Unsupported keywords. Many providers quietly ignore pattern, minimum/maximum, or oneOf. A silently-dropped constraint feels safe and isn't. Verify yours.

🔧The fix, as policy

1. Use real strict-schema structured outputs, not bare json_object. Mask-time enforcement vs. vibes.
2. Still pipe the parsed object through Pydantic/zod. Grammar = syntax. Your validator = meaning.
3. Check finish_reason first. Always.

📉If your provider/local model can't do it

Feed the parser error back and retry ("your output failed: <err>; return only JSON"). Recovers roughly 80% of A and C — but costs a full extra round trip on every failure, and latency is worst exactly when you're already degraded. Constrained decoding costs ~0.

🧪Your exercise

Take a schema you already ship. Build two variants: one with the answer field first, one with a "reasoning" string field first. Run the same 50 hard examples through both at temperature 0 and report the accuracy gap — plus whether latency actually got worse.

Bonus: add {"type":"string","pattern":"^[A-Z]{3}$"} to a currency field and find out whether your provider enforces it or silently drops it. Most people never check.