Structured extraction can fail before content scoring even starts

You asked the model for JSON. It returned JSON-adjacent prose wrapped in markdown. Your eval parser choked silently. You logged a score of 0.0. The pipeline continued.

This is a separate failure class from an incorrect answer. The model may understand the task and still produce output that a downstream JSON parser or schema validator rejects. The rate depends heavily on the model, schema, prompt, and retry strategy, so there is no honest universal percentage.

When your evaluator silently returns 0.0 on a parse failure, you're not measuring model quality. You're measuring a mix of model quality and the model's willingness to comply with output format instructions on any given day — which is itself non-deterministic. The failure is invisible unless you instrument for it specifically.

This post covers what causes extraction failures, how to detect them, and how to build evals that are robust to them.

What "Structured Extraction" Means Here

Any task where you need the model's output to conform to a defined structure:

JSON with specific fields
A Pydantic model (typed Python object)
A fixed schema (product name, price, category, etc.)
Tool call arguments
Numbered lists, headers, YAML

All of these have the same failure mode: the model produces output that looks right to a human but fails programmatic parsing.

The Four Failure Modes

1. Markdown wrapping

```json
{"title": "Quarterly Report", "score": 0.87}
```

A bare json.loads() fails on this. Your eval logs 0.0. The model actually got it right.

2. Extra prose before or after

Sure! Here's the JSON you asked for:

{"title": "Quarterly Report", "score": 0.87}

Let me know if you need anything else!

Again, parse fails. Common with instruction-tuned models that are optimized for conversational helpfulness.

3. Type coercion errors

{"title": "Quarterly Report", "score": "0.87"}

The model returned a string "0.87" instead of a float 0.87. Valid JSON, wrong type. A Pydantic model will reject it. A JSON Schema validator with "type": "number" will reject it.

4. Missing required fields

{"title": "Quarterly Report"}

The model omitted a field. Sometimes it's hallucinating absent data and refuses to include it. Sometimes it misread the schema. Either way, your downstream pipeline breaks silently.

How Standard Evals Miss This

Most eval frameworks check the model's answer against a reference. If the model returns "0.87" (string) when you expected 0.87 (float), a string-matching evaluator says "wrong answer." A numeric comparison crashes. Neither tells you why it failed.

The deeper problem: if your evaluator returns 0.0 on a parse failure and 0.0 on a genuine wrong answer, you lose signal. You can't distinguish "the model doesn't know the answer" from "the model knows the answer but can't format its response."

This distinction matters. Format failures are fixable with better prompt engineering or retry logic. Knowledge failures require a different model or retrieval strategy.

Schema Evaluation: Catching Format Failures Separately

The fix is to evaluate structure and content as separate concerns:

from pydantic import BaseModel
from multivon_eval import EvalSuite, SchemaEvaluator, Faithfulness

class DocumentSummary(BaseModel):
    title: str
    score: float
    tags: list[str]
    confidence: float

suite = EvalSuite("Document Intelligence")
suite.add_cases(cases)
suite.add_evaluators(
    SchemaEvaluator(DocumentSummary),  # structure first
    Faithfulness(),                     # content second
)

SchemaEvaluator does four things:

Strips markdown code fences — so ```json ... ``` is parsed correctly
Validates types — "0.87" fails when float is expected, with a clear error message
Checks required fields — reports which fields are missing
Reports per-field errors — not just "schema invalid" but exactly what went wrong

Sample output when structure fails:

Schema validation failed:
  score: Input should be a valid number, unable to parse string as a float
  tags: Field required

You see the structure error separate from any content evaluation. Your faithfulness score (which would be meaningless on unparsed output) only runs when the schema check passes.

Why the rate is workload-specific

Measure the rate on your own schemas and traffic. Common factors include:

the number of required fields;
nesting and arrays;
numeric and enum coercion;
whether the provider enforces structured output;
retries, repair passes, and multi-model handoffs.
Multi-model pipelines (extraction → summarization → classification): failures compound

For illustration, if a five-step pipeline independently succeeds 90% of the time at each step, its end-to-end success rate is 0.9^5 ≈ 59%. That is a hypothetical compounding example, not a measured universal model rate.

Testing extraction robustness explicitly — not just "does the model return the right answer" but "does the model return a parseable, valid-schema answer consistently" — is the key shift.

JSON Schema for Non-Python Pipelines

If you're not using Python or Pydantic, SchemaEvaluator also accepts a JSON Schema dict:

from multivon_eval import SchemaEvaluator

suite.add_evaluators(SchemaEvaluator({
    "type": "object",
    "required": ["title", "score", "tags"],
    "additionalProperties": False,
    "properties": {
        "title": {"type": "string", "minLength": 1},
        "score": {"type": "number", "minimum": 0, "maximum": 1},
        "tags": {
            "type": "array",
            "items": {"type": "string"},
            "minItems": 1
        }
    }
}))

additionalProperties: False catches models that invent extra fields — often a sign that the model is hallucinating structure it wasn't asked for.

Connecting to Non-Determinism

Structured extraction failures interact with the non-determinism problem from the previous post. If a case fails due to a parse error on run 1, it might succeed on run 3. This looks like flakiness, but it's actually format compliance variance.

Running each case 5 times and taking majority vote helps here: if the model returns valid JSON 4 out of 5 times, the case is a robust pass with one format hiccup. If it returns valid JSON 2 out of 5 times, you have a systemic format compliance problem worth fixing.

suite = EvalSuite("Document Intelligence")
suite.add_cases(cases)
suite.add_evaluators(SchemaEvaluator(DocumentSummary), Faithfulness())

# 5 runs per case — separates format flakiness from content failure
report = suite.run(my_pipeline, runs=5)

The flakiness report will separate cases where format was the root cause from cases where the model's answer itself was inconsistent.

What to Actually Do

If you're building a structured extraction pipeline:

Add SchemaEvaluator to every eval suite that involves JSON output
Use Pydantic models — they give you per-field error messages, which are much more useful for debugging than JSON Schema's generic errors
Set strict=True if you want to catch models that add extra fields

If you're evaluating document intelligence, data extraction, or anything that produces structured output:

Separate structure evaluation from content evaluation
Run at least 5 trials per case — format compliance is non-deterministic
Check for type coercion failures specifically — these are the most common and least visible

If you inherit an eval suite without schema validation:

Add a schema check as a pass/fail gate before any content evaluators
Look at your historical 0.0 scores — many of them are probably parse failures, not genuine wrong answers