Does the optimal hallucination detector for agentic tool calls depend on model scale?
2026
An AI agent that hallucinates a tool call typically shows no overt signs of failure: the call is syntactically correct, its arguments are well-formed, and the associated token probabilities appear routine. In these cases, the execution pipeline receives no indication that anything is amiss. The call executes, downstream components consume its output, and the erroneous transaction completes without human scrutiny.
Standard uncertainty signals offer little protection against this failure mode. Token log-probabilities and small-N sampling consensus (N = 5) deliver only 0.51–0.62 AUC across seven models, while incurring roughly a fivefold increase in inference cost. In response, recent work has proposed two single-pass alternatives: spectral analyses of the attention routing graph and hidden-state probes over designated token-role positions. Both families outperform conventional uncertainty metrics, but it remains an open question which detector class is preferable under realistic deployment constraints.
We answer it across seven models from 1B to 14B on identical data, splits, and labels, and find the optimal detector depends on scale non-monotonically. Our Layerwise Multi-Metric (LMM) probe on attention alone reaches AUC 0.957 at 3B, exceeding the hidden-state probe (0.947); at 8B, hallucination signal diffuses across 32 layers and trajectory aggregation (SpRich) takes the lead (0.888); at 14B, layer specialization re-concentrates the signal and LMM recovers (0.917 vs. 0.882). Merging the two signal families never helps in our experiments. A controlled within-family study on Llama-3.x (1B/3B/8B) confirms that scale alone drives monotonic improvement in spectral detectors while hidden-state probes saturate near-ceiling at 1B, directly disentangling scale from architectural variation. The deployment rule is to match the guardrail to depth and attention type, not to combine.
Code: https://github.com/vcnoel/spectral-tool-use.
Research areas