Confidence-aware multi-agent orchestration for evaluating multimodal rule compliance

Evaluating rule compliance in industry requires assessing products against complex regulatory standards using multimodal data sources—a task where both correctness and trustworthiness of automated judgments are critical. Existing approaches either rely on costly human audits, supervised classifiers that demand large-scale labeled training data, or monolithic multimodal models that apply uniform reasoning without mechanisms for quantifying uncertainty or resolving cross-modal conflicts. These limitations raise fundamental questions about how to evaluate and ensure trustworthiness in agentic systems operating over heterogeneous, noisy data. We propose a confidence-aware multi-agent orchestration framework that addresses these evaluation challenges through three mechanisms: (1) a Rule Generation module that produces interpretable, auditable compliance specifications; (2) modality-specialized Base Agents with a perception-then-judgment protocol that separates observation from evaluation, enabling fine-grained assessment of evidence faithfulness; and (3) an Orchestrator Agent that performs confidence-aware triage and generates targeted follow-up queries to resolve inter-agent conflicts, providing an explicit audit trail of its reasoning process. Our framework demonstrates how multi-agent orchestration can be designed for trustworthiness by construction—through confidence-aware decision routing, evidence-grounded adjudication, and iterative uncertainty reduction. On a product industry guideline compliance task with 216 human-validated samples, the framework achieves 89.8% accuracy and 92.5% F1 score, outperforming the strongest baseline (77.3% accuracy, 83.5% F1) by 12.5 and 9.0 absolute points respectively. Detailed ablation analysis reveals that cross-modal conflict resolution contributes 13.9% absolute F1 improvement over naive multi-modal fusion, providing quantitative evidence for the value of confidence-aware orchestration in trustworthy agentic evaluation.