On recursive resolution: Scalable ground truth for self-improving AI systems

Compound retrieval-augmented question-answering (QA) systems present a fundamental evaluation challenge: manual annotation does not scale, yet automated evaluation lacks the ground truth necessary for calibration. We introduce a self-improving evaluation architecture that addresses this circular dependency through three contributions. First, iterative consensus synthesis: an algorithm that treats LLM-human agreement as a free signal, escalating only disagreements to additional annotators and achieving multi-annotator reliability at 15.95× lower cost. Second, a tiered Wilson interval framework that transforms noisy binary judgments into confidence-bounded evaluation results, correctly identifying regressions that raw failure rates obscure. Third, recursive resolution: an architecture where evaluation thresholds update based on remedy effectiveness, enabling systems to improve their own detection criteria over time. We validate on a compound retrieval-augmented QA system across 13 quality metrics and three locale-specific test suites. The system completed full improvement cycles detecting a 44% content coverage gap during retrieval, validating a targeted fix through shadow evaluation, and generalizing the pattern to prevent similar regressions on other locale-specific test sets. These results demonstrate that scalable, self-improving evaluation of compound AI systems is achievable without sacrificing annotation quality.