It puts a confidence interval on every number — a Wilson 95% CI on pass rate and a bootstrap 95% CI on average score — so you can tell whether a change actually moved the metric or just moved the noise. 44 evaluators across 7 tiers, one API. Judge thresholds are calibrated against human-labeled data, not guessed. Runs locally against Ollama, LM Studio, vLLM, or any OpenAI-compatible server. No account, no telemetry.
No signup · Local-first · First eval in ~3 min
$ pip install multivon-eval
$ python -m multivon_eval demo
multivon-eval demo · customer support bot
6 cases · deterministic tier · no API key
Pass Rate: 83.3% [44%–97% Wilson 95% CI]
Avg Score: 0.83 [0.50–1.00 bootstrap 95% CI]
Score dist: p10 0.50 · p50 1.00 · p90 1.00
⚡ Power warning: 6 cases — min detectable change
at 80% power is ~74%. Add cases to catch less.κ ≈ −0.01 · 58% disagreement between multivon-eval and DeepEval (RAGAS, earlier run: 40% / 20% vs each).
Same judge, same dataset, same seed (n=50). The worst pair agrees barely better than chance; all three disagree more than you’d hope.
Read the benchmark →F1 0.830 [0.70–0.92] on held-out HaluEval-Sum at the calibrated threshold.
Wilson + bootstrap CIs on every number, judge thresholds calibrated against human labels.
See multivon-eval →Pixels-only nearly inverts the PDF Hell leaderboard: GPT-5 94.7→67.6, Opus 4-7 79.4→85.9.
Same 170 cases rasterised locally at 150 dpi — no embedded text layer — and the PDF leader drops 27.1 points while Opus takes the lead.
Per-model deltas →An eval vendor should be auditable. Every entry below links to an artifact we can’t edit after the fact — PyPI release history, GitHub issues, dated posts.
multivon-eval, DeepEval, and RAGAS — same judge, same dataset, same seed (n=50) — disagree on 56% of cases; the worst pair agrees barely better than chance. A result that indicts the category we sell in, published anyway.
A provider temperature bug had been silently scored as wrong answers; the “0% on all seven trap families” claim was an eval artifact, not a model failure. Retracted, leaderboard re-run and corrected, notice left permanently in the repo README.
A peer review caught the 0.9.4 “held-out” claim being in-distribution — the evaluator's threshold was calibrated on the same dataset it was tested on. 0.9.5 corrected the claim, 0.9.6 fixed runtime bugs the same review caught in the generated bootstrap template, 0.9.7 fixed a threshold-vs-default mismatch in the held-out reproducer. All four releases left on PyPI — yanking them would erase the record.
Scanner v3 (0.10.1) measured 278 prompt call sites across aider, gpt-researcher, open-interpreter, letta, and pr-agent: 20.9% statically resolvable, below the 50% gate we set ourselves. Published with the per-repo table on the epic; the runtime recorder was promoted to the priority path past the static ceiling.
Opt-in pytest --record-prompts captures rendered prompt fingerprints at the same three SDK surfaces the static scanner reads. Three trust tiers, never collapsed: static scan proves prompt text; runtime recordings prove only the k-of-N renderings observed; templates stay honestly deferred. Case-to-site bindings are propose-only — observation replaced fabrication.
The cross-modality run exposed that two autoresearch trap families (zero_width_space_split, unicode_confusable_total) rendered visible tofu boxes where they claimed visual normality — the “visually identical” premise was false. Both were redesigned in pdfhell 0.6.1 (adjacent-text-run fragmentation; a digit-zero T0TAL confusable) and a sixth glyph_clean validation gate now pins the invariant. Alongside it, multivon-eval 0.11.1 shipped scanner v4 with an explicit UNSCANNABLE tier — honest UNKNOWN over confident wrong.
Each package solves a piece of the AI-evaluation problem and composes with the others. All five are Apache 2.0 — three on PyPI, eval-action and eval-framework-benchmark on GitHub; multivon-guard is in private beta. No telemetry.
the SDK
44 evaluators across deterministic, LLM-judge (QAG), agent-trace, conversation, compliance, and multimodal tiers, plus a new bootstrap CLI that proposes a tuned eval suite from your product description + sample traces. Calibrated thresholds, hash-chained audit logs, pytest plugin.
the benchmark
510 adversarial PDFs across 17 trap families that fool AI document readers — like an invoice whose visible total differs from its hidden text layer. The correct answer comes from code, so no AI is grading another AI. Drops straight into CI.
the agent surface
MCP server exposing 22 evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or OpenCode and your AI coding agent calls evals directly mid-edit.
the CI gate
GitHub Action that runs multivon-eval on PRs, posts a diff comment with regressions, and gates merges on safety-class failures.
the receipts
Head-to-head benchmark vs DeepEval + RAGAS on hallucination detection. Same judge, same dataset, same seed, fully reproducible. multivon-eval F1 0.79 vs DeepEval 0.0 at defaults.
the runtime gatekeeper
Evaluation alone isn't enough. multivon-guard catches what your AI is about to send before it hits the wire, secrets, PII, API keys. Local proxy, auto-detects Claude Code / Codex / Cursor / Aider / Continue. Now in private beta.
Bring your own API key. Local-first by default. Hash-chained audit logs ship with every run.
What's new across the stack
Persona simulator + gated case generation (0.12.0), the runtime prompt recorder (0.11.0), prompt-drift staleness (0.10.0), and the intelligent-eval layer underneath. See the full roadmap →
multivon-eval simulate drives your bot through persona-driven, adaptive multi-turn conversations: a simulator LLM plays the user, reacts to each reply, and stops at a hard budget ceiling. Honesty contract built in — every transcript carries provenance and is labeled simulated, not real traffic, so simulated results can never blend into production metrics.
Read more →bootstrap --n-seed-cases scales seed-case generation to 500 behind duplicate and hardness gates, and the generation report states exactly what survived — "generated 500, accepted 431 — no silent caps." Rejected cases are counted and named, never silently dropped, so a thin suite can't masquerade as a big one.
Read more →Opt-in pytest --record-prompts captures rendered prompt fingerprints at call time — the honest path past the 20.9% static-resolvability ceiling. Three trust tiers, never collapsed: static scan proves prompt text, runtime recordings prove only the k-of-N renderings observed, templates stay honestly deferred. Case-to-site bindings are propose-only — observation, never fabrication.
Read more →multivon-eval staleness diffs a live scan of your repo's prompt call sites against a committed prompt_baseline.json: CHANGED (with before/after fingerprints and the cases bound to that prompt), REMOVED, ADDED, and UNKNOWN for dynamic prompts static analysis can't read — honestly unknown, never fake-fresh. staleness stamp binds cases to the prompts they exercise; bootstrap writes the baseline automatically. Every report opens with a determinacy headline and closes with a blind-spots footer. --fail-on gates CI.
Read more →multivon-eval bootstrap --product PRODUCT.md --traces TRACES.jsonl emits a tuned EvalSuite + adversarial seed cases + calibrated thresholds + a forwardable DISCOVERY_REPORT.md. Single LLM call, ~$0.12 per bootstrap. The fastest path from "I don't know what to eval" to a runnable suite — and 0.12.0's --n-seed-cases scales it to 500 gated cases.
Read more →The 0.10-era substrate under bootstrap, in one card: auto_evaluators(case) ranks evaluators heuristically in microseconds (zero LLM cost); generate_adversarial_cases targets 10 failure modes with stress_test routing metadata; an N-shot judge-noise filter keeps only validated-hard cases (+0.80 mean failure-rate separation); a local PII/secret scan redacts before any trace leaves your machine; thresholds calibrate from your own traces (p25 of baseline scores).
One-command installer for the three bundled Claude Code skills (eval-bootstrap, eval-audit, eval-explain). Symlinks SKILL.md files from the wheel into ~/.claude/skills/ so pip install -U multivon-eval propagates skill edits without re-running the installer.
Read more →An autoresearch loop (Karpathy pattern) where a rotation of Opus 4-7, GPT-5, and Gemini 2.5 Pro propose adversarial PDF traps against an 8-model eval panel. Six validation gates filter candidates before any vision-eval spend (glyph_clean added in 0.6.1 after the tofu-box bug). $88 total across two overnight runs produced 11 surviving trap families — now in mini-v4 (510 cases). The agent does not merge its own work; a human curator promotes from keep/.
Read more →The full mini-v4 suite is 510 cases across 17 trap families; the public leaderboard runs the 170-case mini-v4-sample (first 10 seeds/family). The sample surfaces three real per-trap blind spots: GPT-4o on hidden-OCR invoices (0/10), Anthropic's premium tier on a 3.5pt-footnote trap (0/10), and three models on text-run fragmentation (0/10, measured pre-0.6.1-redesign). The 2026-06-12 pixels runs traced all three to PDF ingestion, not vision.
Read more →PDF Hell is an adversarial benchmark — 510 procedurally-generated cases across 17 trap families that break the assumptions document-AI vendors silently rely on. Six hand-authored, 11 discovered by an autoresearch loop. Code-based ground truth, not LLM-as-judge. The current leaderboard exposes per-trap blind spots in GPT-4o (hidden-OCR text-layer trust), Anthropic's premium tier (a 3.5pt-footnote trap that inverts on locally-rasterised pixels — a PDF-ingestion failure, not vision), and three different models (text-run fragmentation).
multivon-eval ships Faithfulness, ContextPrecision, and ContextRecall as QAG-graded evaluators with calibrated thresholds per judge. Pass-rate gates your CI without you babysitting an LLM judge. Bootstrap CLI generates a runnable suite + 30 adversarial seed cases + calibrated thresholds from your own traces in one command — and when your prompts change later, the staleness report tells you which cases were authored against the old ones.
8 agent-native evaluators — ToolCallAccuracy, ToolCallNecessity, TrajectoryEfficiency, StepFaithfulness, AgentMemoryEval. LangGraph + OpenAI Agents SDK tracers wired natively. Plus 22 MCP-callable tools so Claude Code / Cursor / Cline run the eval mid-edit.
Hash-chained NDJSON, SHA-256 manifest, EU AI Act / NIST AI RMF / HIPAA / DPDP India mappings. Generate one from any pdfhell or multivon-eval run — free, no signup. JudgeConfig(base_url=…) routes judging through any OpenAI-compatible endpoint (Ollama, vLLM, on-prem) so production data never leaves the VPC.
multivon-guard is a local proxy that intercepts your agent's outbound traffic and redacts secrets before they leave your machine. Auto-detects Claude Code, Codex, Cursor, Aider, Continue. Now in private beta.
Request early access →The five public packages above are Apache 2.0 and free forever. They share the same engine (multivon-eval) and the same methodology (QAG + code-based ground truth). multivon-guard is in private beta — email for access.
Frameworks that show a single number lie by omission. Multivon’s default output ships with confidence intervals, power analysis, and a tamper-evident audit trail — so a shipping decision actually corresponds to real signal.
0.74vs 0.04
multivon-eval · DeepEval · F1 on ragtruth-sum (n=100, gpt-4o-mini judge)
Same judge, same dataset, same seed, side-by-side. DeepEval’s default threshold flags almost nothing — F1 0.04(recall 0.02). Multivon’s defaults hit F1 0.744; at best-tuned thresholds it’s 0.83 vs 0.63. The gap is the default threshold: DeepEval ships an uncalibrated one, multivon’s is calibrated against human labels. Code in the repo — re-run it yourself.
Read the benchmark →Default outputA pass rate of 84% with CI [62%, 95%] tells you it’s indistinguishable from 75% — and the framework tells you so. When N is too small to detect a 10-point shift, stderr warns before the run, not after the wrong release.
Statistical rigor docs →Compliance--audit-pack out.zip bundles every test case, answer key, run JSON, JUnit XML, and a SHA-256 manifest. Hash-chained NDJSON inside; suite-hash + library-version locked into every report.
See the audit format →Pick the ones your use case needs, ignore the rest. All Apache 2.0, all available via Python SDK and MCP.
No LLM judge needed. Cheap, fast, reproducible.
QAG decomposition. Wilson 95% CIs on every aggregate. Thresholds calibrated against human labels; inter-judge agreement κ 0.60-0.80.
Trajectory-level scoring for multi-step agents.
Procurement-grade checks. PII redaction + schema enforcement.
Image-grounded faithfulness + document QA scoring.
Multi-turn dialog quality, not just single-turn answers.
Run-to-run stability across N samples.
Full list: docs.multivon.ai/evaluators · agent-callable via MCP
One real run on the left tab — GPT-4o vs the current pdfhell mini-v4-sample suite (170 cases, 17 trap families) — plus three illustrative shapes for faithfulness, experiment comparison, and plain-English checks. Real numbers and reproducer commands for every model we've scored live on the leaderboard.
JUnit XML output renders natively in every CI runner. A 10-line YAML wires PDF Hell into a pull-request gate. multivon-eval’s pytest plugin slots into your existing test suite without a special harness.
# .github/workflows/pdfhell.yml
name: PDF Hell
on: [pull_request]
jobs:
pdfhell:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
- run: uvx pdfhell run \
--model anthropic:claude-sonnet-4-6 \
--suite mini \
--junit results.xml \
--audit-pack audit.zip \
--fail-threshold 0.85
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: pdfhell
path: |
results.xml
audit.zipEvery public package is on PyPI today. Stars and issues welcome on GitHub. For custom adversarial trap families, on-prem deployment, or paid integration support, see /commercial — inbound only, no fake tiers.
Run the current pdfhell mini-v4-sample suite (170 procedurally generated PDFs, 17 trap families) against gpt-4o. It passes most of the suite at 81% overall, but never catches the hidden-OCR mismatch — the visible page text says one number, the buried OCR layer says another, and the model returns the OCR layer every time. The numbers below are a real run — same suite, same seeds, reproducible.