Beyond Task Success: An eight-metric tiered evaluation protocol for LLM agents over fragmented operational data

Single-metric task-success scores systematically misrank LLM agents in domain-specific deployments: two configurations can achieve identical end-to-end accuracy while exhibiting structurally different production behaviors — silent argument hallucination, infinite tool loops, redundant retries, semantically-equivalent-but-syntactically-divergent queries. We propose a behavioral evaluation protocol that decomposes agent quality into an aligned eight-dimensional score vector across three layers: answer (Accuracy, Semantic F1, Cosine Sim), trajectory (Set Match, Order Match, Args Match, Tool Coverage), and execution (Avg Failures/Traj). The protocol's diagnostic value comes not from any single metric but from the joint reading of metrics across layers — specific gap patterns localize to specific behavioral failure modes (resilient self-correction, compliant-but-fragile planning, formatting-only failures, tool-misrouting, execution-environment failures). We instantiate the protocol on five reasoning configurations (zero-shot and DSPy-Simba-optimized variants of Chain-of-Thought and ReAct, plus a CodeAgent baseline) over the ToolHop multi-hop tool-use benchmark and over a domain-expert-curated production set of 50 operational queries from a real enterprise deployment. Optimized ReAct attains 65% on ToolHop and 86% on the production set; the eight-metric breakdown reveals that DSPy optimization patches CoT's syntactic compliance without unlocking adaptive reasoning, exposes a structural 'glass ceiling' for static planners, and surfaces a Resilient behavioral signature (high Set Match, low Order Match, non-zero declining Avg Failures/Traj) that transfers from synthetic benchmarks to live operational queries. The protocol is implemented as a stateful metric callable plus a trace-extraction harness that decouples measurement from deployment, enabling retroactive re-scoring of logged traffic against evolving metric definitions; the present paper describes both in sufficient detail for independent reimplementation.