Roadmap

Persona-driven adaptive multi-turn eval: a simulator LLM plays a configured user persona against your bot, adapting each turn to the previous reply, under a hard budget ceiling. Honesty contract built in — every transcript carries provenance and is labeled simulated, not real traffic, so simulated results never blend into production metrics.

bootstrap --n-seed-cases scales seed-case generation to 500 behind duplicate and hardness gates, with a generation report that states exactly what survived — "generated 500, accepted 431 — no silent caps." Rejected candidates are counted and named, never silently dropped.

Robustness hardening on the staleness scanner: call sites the scanner cannot read surface as an explicit UNSCANNABLE tier instead of disappearing from the report — honest UNKNOWN over confident wrong, the same principle the determinacy gate was built on.

multivon-eval staleness diffs a live scan of every prompt call site against a committed prompt_baseline.json: CHANGED (with before/after fingerprints, bound cases, and a git diff pointer), REMOVED (always with the three-way caveat: feature removed / renamed+edited / moved beyond static reach), ADDED (new prompts with no covering cases), UNKNOWN (dynamic prompts — never guessed at). staleness baseline writes the snapshot; staleness stamp binds hand-written cases to the prompt call sites they exercise; bootstrap stamps generated cases and writes the baseline automatically. Matching is content-first — line numbers and git SHAs are display-only, so a whitespace refactor or rebase produces zero false staleness. Every report opens with a determinacy headline and closes with a blind-spots footer. Deliberately deferred: propose-and-review case refresh (sync) and the eval-action CI gate — tracked below on the epic.

Before claiming drift coverage, we measured how much real-world prompt traffic static analysis can actually read. Scanner v2's first pass reported zero call sites on 4 of 5 real repos — blind to aliased litellm imports (pr-agent), **kwargs-unpacked calls (aider), and messages=<variable>. v3 detects all three; what it still can't read surfaces as honest UNKNOWN records instead of vanishing. Re-measured: 278 call sites across aider, gpt-researcher, open-interpreter, letta, and pr-agent — 20.9% statically resolvable, below the 50% gate. The gate failed, the result is published with the per-repo table on the epic, and the runtime recorder was promoted to priority.

One-command installer for the three bundled Claude Code skills. Symlinks eval-bootstrap, eval-audit, and eval-explain from the wheel into ~/.claude/skills/, so pip install -U multivon-eval propagates SKILL.md edits without re-running install.

Hallucination evaluator calibrated on HaluEval-QA (threshold 0.55, explicit JudgeConfig), tested without re-tuning on HaluEval-Sum. Calibration set strictly disjoint from test set. Result: F1 0.830 [0.70–0.92] on n=60.

benchmarks/_add_cis.py walks every results JSON and writes Wilson CIs on precision/recall plus bootstrap CIs (1000 resamples, seed 20260603) on F1. Idempotent. Closes the "framework preaches CIs but doesn't ship them on its own numbers" dogfood violation.

18 calibration entries across 6 judges × 3 evaluators in _calibration_data/v2.json. Six previously-null F1 cells (opus, gpt-4o, gpt-5.5 across faithfulness/hallucination/relevance) filled via a $15–20 sweep on real held-out data.

eval-bootstrap (cold-start eval generator), eval-audit (suite review), and eval-explain (judge-output interpreter). SKILL.md files ship inside the wheel under multivon_eval/_skills/. No separate marketplace install needed.

Four same-day PyPI releases responding to public peer review: contamination fix on the headline held-out claim, runtime bug in the generated bootstrap template, threshold-vs-default mismatch in the held-out reproducer. All four releases left published as the audit trail.

AST-aware scan of prompt sources in a repo, structured diff between two refs, markdown rendering. Public API: multivon_eval.attribution.scan(repo_root), diff_records(base, head), render_markdown(diffs). Descriptive only — causal attribution intentionally deferred to Phase 2. Scanner v2 (0.10.0) added one-hop module-level constant resolution and loose fingerprints; this scan is now the substrate the staleness drift report runs on.

The 2026-06-11 determinacy gate measured 20.9% static resolvability across 278 call sites on five real repos — most real-world prompt traffic is dynamic construction, beyond what static analysis can read. The recorder is the honest path past that ceiling: opt-in, local-only capture of prompt fingerprints at call time (pytest --record-prompts), runtime fingerprints labeled separately from static ones, and case binding via observation — never fabricated. Promoted from deferred to priority by the failed gate; shipped in 0.11.0 the same week.

Long-running process that re-runs the configured suite on file or git changes in a watched directory. Debounced. Reuses the existing EvalSuite and CostTracker. First step toward continuous eval that doesn't require a CI trigger. Re-scoped after 0.10.0: the watcher will run the staleness report on file change, so prompt drift surfaces in your terminal before any suite re-run spends judge tokens.

Consumes the staleness JSON report and proposes case-file diffs for human review when bound prompts change. Never auto-commits — confidently-wrong refreshes would poison the suite, so every proposed diff requires explicit approval. The deliberate other half of the 0.10.0 read-only report.

Surfaces the staleness report in the eval-action PR workflow — warn-only $GITHUB_STEP_SUMMARY line first (works today as a documented one-liner with --format markdown), per-category fail-on as an action input later. Gate-by-default is deliberately avoided: failing PRs on ADDED prompts punishes adoption.

Pluggable sink so every EvalReport can be streamed to a destination of your choice. Three built-in adapters: file (JSONL append), stdout (rich-rendered), Datadog (gauge per evaluator + count per status). No hosted sink — bring your own observability.

First-class adapters for LangSmith, Sentry, and S3 so traces stored across systems can be loaded into a single eval run without you writing custom ETL. Extends the existing load_traces aliasing work (LangFuse, Phoenix already covered).

Fourth bundled skill wrapping the watch daemon. Slash command that starts the daemon scoped to the current repo, surfaces failures in chat, and proposes fixes when a regression appears. Sibling to /eval-bootstrap, /eval-audit, /eval-explain.

Proof-of-concept that coordinates three named subagents (auditor, attributor, explainer) against a repo and surfaces a unified report. Skeleton only — the real swarm is the long-term thesis below; this is the first walking version.

HaluEval-QA and HaluEval-Sum share CNN/DailyMail structure. Adding TruthfulQA and FaithBench as held-out targets gives a genuinely cross-corpus generalization figure. Will also re-run at n=500–1000 to tighten CIs (current held-out n=60 is honestly wide).

Adds a non-prompt-change signal (git diff classifier over pyproject, model configs, dependencies) ANDed against Haiku majority-voted attribution. When the sidecar fires `present`, attribution refuses rather than confidently misattributing a mixed-cause regression. Design doc: feature_prompt_attribution_phase2_sidecar_design.md.

QAG evaluators (Faithfulness, Hallucination, ContextPrecision) currently don't override aevaluate, so async paths fall back to sync. Closes a known limitation flagged in iter-2 peer review. Pairs with batch judging via the existing OpenAI/Anthropic batch APIs for 50% cost reduction on long runs.

Cosign signature on the published container image, SBOM (CycloneDX) attached to each tagged release, immutable @v1.x.y tags alongside the mutable @v1. Closes r/MLOps procurement-tier concerns from launch review.

Today multivon-eval compare pairs cases by sequential index within an input. Adding a pluggable baseline source (git ref, JSON file, S3 bucket, MetricsSink read-back) lets compare answer "did my prompt change help vs the version on main 30 days ago" without manual artifact wrangling.

The underlying thesis. Every roadmap item above is a step toward one thing: a swarm of eval agents that goes into your system, reads your code and your data and your logs wherever they're stored, and continuously updates eval metrics without a human writing or maintaining the eval suite. The watch daemon is the trigger. MetricsSink is the egress. Ingest adapters are how the swarm sees your traces. The Phase 2 attribution sidecar is how it stays honest about causality. Swarm coordination is the orchestration layer. Once those pieces are mature, the eval suite stops being a thing you write — it becomes a thing your repo grows.

Honesty principle: a managed-SaaS eval dashboard is not on the roadmap. A hosted eval-database is not on the roadmap. Fine-tuning-as-a-service is not on the roadmap. multivon-eval is and will remain Apache-2.0, runs on your machine, sinks metrics to observability tools you already own. If the right answer for you is a hosted dashboard, the answer is Datadog or Phoenix or your own — not us. The commercial surface (if any) will be support and integration consulting, not a tier above the OSS.