The track record
An eval vendor should be auditable. Every entry links to an artifact we can’t edit after the fact: PyPI release history, GitHub issues, dated posts. We append to this list. We do not rewrite it.
- 2026-05-13
v1 pilot published: cross-framework disagreement, κ = 0.03
The v1 pilot (HaluEval-Sum, n=50): multivon-eval, DeepEval, and RAGAS — same judge, same dataset, same seed — disagreed on 56% of cases (κ = 0.03), the worst pair barely above chance. A result that indicts the category we sell in, published anyway. (The current headline uses the later committed RAGTruth-Sum n=100 run, κ ≈ 0.04.)
- 2026-05-24
pdfhell Opus 4-7 headline finding retracted
A provider temperature bug had been silently scored as wrong answers; the “0% on all seven trap families” claim was an eval artifact, not a model failure. Retracted, leaderboard re-run and corrected, notice left permanently in the repo README.
- 2026-06-03
Four releases in one day: 0.9.4 → 0.9.7
A peer review caught the 0.9.4 “held-out” claim being in-distribution — the evaluator's threshold was calibrated on the same dataset it was tested on. 0.9.5 corrected the claim, 0.9.6 fixed runtime bugs the same review caught in the generated bootstrap template, 0.9.7 fixed a threshold-vs-default mismatch in the held-out reproducer. All four releases left on PyPI — yanking them would erase the record.
- 2026-06-11
Determinacy gate run on real repos — failed at 20.9%
Scanner v3 (0.10.1) measured 278 prompt call sites across aider, gpt-researcher, open-interpreter, letta, and pr-agent: 20.9% statically resolvable, below the 50% gate we set ourselves. Published with the per-repo table on the epic; the runtime recorder was promoted to the priority path past the static ceiling.
- 2026-06-11
Runtime recorder shipped (0.11.0) — the answer to the 20.9% ceiling
Opt-in pytest --record-prompts captures rendered prompt fingerprints at the same three SDK surfaces the static scanner reads. Each source keeps its own trust tier: the static scan proves prompt text, recordings prove only the k-of-N renderings observed, and template prompts stay out of scope. Case-to-site bindings are propose-only.
- 2026-06-12
Our own pixels modality caught a bug in our own benchmark
The cross-modality run exposed that two autoresearch trap families (zero_width_space_split, unicode_confusable_total) rendered visible tofu boxes where they claimed visual normality — the “visually identical” premise was false. Both were redesigned in pdfhell 0.6.1 (adjacent-text-run fragmentation; a digit-zero T0TAL confusable) and a sixth glyph_clean validation gate now pins the invariant. Alongside it, multivon-eval 0.11.1 shipped scanner v4 with an explicit UNSCANNABLE tier — honest UNKNOWN over confident wrong.
- 2026-06-26
Benchmark re-measured on 0.15.1 — headline moved 0.744 → 0.729
Re-running the head-to-head on current versions moved our F1 from 0.744 (0.9.8) to 0.729, and we shipped the lower number. RAGAS — which had errored on every case in the prior harness — was restored to the comparison on 0.4.3, where it completes. The run is pinned under a git tag so the snapshot can't drift.
- 2026-07-13
Our own judge parser had a yes-bias
A hedged judge reply — “I cannot say yes or no with certainty” — parsed as YES; per-question judge exceptions were scored as silent False votes and failed claim extraction returned a 0.0 quality score indistinguishable from a model failure; a CI gate could show green at a 90% judge-error rate. All three found by a three-model verification review of our own release candidate, fixed in 0.16.0 with ~40 tests: unparseable verdicts are now UNKNOWN, judge errors surface as error statuses excluded from pass rate, and max_error_rate puts a budget on the gate.
- 2026-07-13
Three frontier models voted 0.16.0 not ready — so it wasn't released
The pre-release verification round — three independent model reviews — voted unanimously NOT ready, on four major defects including a crash that fired after the paid model run but before results were saved. All four were fixed with ~40 tests and the release re-verified to a unanimous green in round 2 before tagging. The vote and its outcome are recorded in the release notes.
- 2026-07-13
Preregistered study: our own power gate failed at n=300
Before any test API call, the cross-framework study's power gate failed at the planned n=300 — 0.38–0.39 minimum power on the original grid, 0.55 on the refined confirmatory flip-rate endpoint, all below the committed 0.80 bar. Per the preregistered rule we escalated the RAGTruth-Sum test split to n=500 (the sole preregistered escalation) and published the gate failure in the addendum before spending on the test split.
- 2026-07-14
In the study's strong-judge ablation, the framework that broke was ours
Against a reasoning-tier judge, multivon-eval errored on 71 of 150 cases (47.3%) in its ablation cell — a hardcoded max_tokens=512 in the internal claims call, exhausted by reasoning tokens — while the other four frameworks ran clean. Documented in the public addendum §11 and kept as data with no repair pass, per the preregistered errors-as-failures rule.
- 2026-08-16
The 47% error rate we published against ourselves is fixed in 0.16.1
A reasoning model spends part of its output budget thinking before it answers. Our judge calls capped that budget at 100 tokens for a yes/no verdict and 512 for claim extraction, so the judge was cut off mid-thought and returned an empty verdict — the 47.3% error rate we published in the study's ablation instead of quietly repairing. 0.16.1 floors the per-call ceiling at 2048 tokens for reasoning-tier judges. Every other judge is byte-identical, and an explicit larger max_tokens is never lowered.