Research notes from building evaluation tools.
Benchmarks, failure reports, and practical guidance for teams measuring LLM applications. The current featured finding: three evaluation frameworks agree barely above chance on which answers are hallucinated.
From a readable receipt to a correct ledger entry
A small workflow study using CORD, pdfhell and Inspect found wrong amounts, missing writes, and limits in our own task definition.
Three open-source eval frameworks disagree on 56% of cases. Cohen's κ = 0.03.
Same judge, same dataset, same seed. multivon-eval, DeepEval, and RAGAS produce different verdicts on more than half the cases. The detection-prompt gap is small; the shipped-calibration gap is huge. Methodology and circularity disclosure included.
Cross-dataset calibration: F1=0.787 on data it never trained on
v2 benchmark: re-ran the head-to-head on RAGTruth — a dataset multivon-eval's calibration has never seen. At our HaluEval-derived threshold, F1=0.787 on RAGTruth, higher than the in-distribution 0.63. Removes the v1 circularity caveat with real numbers.
Structured extraction can fail before content scoring even starts
A correct-looking answer is still unusable when it does not satisfy the schema. Here is how to separate format failures from content quality.
Single-run LLM evals routinely misrank models (NAACL 2025)
Single-run evaluation scores are so noisy they routinely misrank models and miss regressions. Here's the research behind it, and what to do instead.
QAG vs LLM-as-Judge: Why We Score With Questions, Not Numbers
Asking a model to rate output 1-10 introduces its own hallucination risk. There's a more reliable way: generate yes/no questions and score by the fraction answered correctly.