Diagnostic knowledge graphs: Automated benchmark construction and deterministic evaluation for multi-step reasoning agents

Evaluating multi-step diagnostic reasoning in LLM agents remains an open problem. When cause labels are extracted from resolved operational cases (customer-service tickets, incident reports, clinical notes), the resulting gold standards exhibit extreme vocabulary explosion—5,076 unique cause strings from 2,196 tickets on a single symptom, 92% appearing only once—making LLM-as-judge protocols variance-prone (±2–3pp inter-run) and longitudinal monitoring impossible. We argue that building reproducible diagnostic benchmarks and building effective diagnostic agents are dual problems solved by the same artifact—a canonical knowledge structure that normalizes evaluation gold-standards and constrains agent hypothesis spaces simultaneously. We instantiate this duality as Diagnostic Knowledge Graphs (DKGs): hierarchical cause trees with frequency priors, built automatically from resolved tickets via LLM extraction, two-pass BERTopic clustering, and LLM merge, with optional per-node SQL grounding against operational data. The pipeline compresses 5,076 cause strings to 73 canonical clusters (97% coverage) and scales to 284 symptom families from 14,953 tickets without manual curation. A domain-expert validation shows the resulting normalization achieves 92% accuracy—exceeding the LLM judge's 80% agreement with the same expert—while enabling deterministic scoring (σ=0 inter-run variance). Under a fair semantic-judge protocol (scoring against raw gold chains, so DKG agents gain no vocabulary advantage), the DKG yields a +32pp action-accuracy lift over an unstructured baseline (73% vs. 41%). A five-condition ablation reveals that the cause menu and frequency priors alone—without SQL—account for the dominant share of this gain (78–86% of the total lift), establishing that the evaluation infrastructure itself is the primary source of agent improvement. Pre-validated SQL adds a directionally positive but non-significant benefit at n=100, bounded by data sparsity rather than a method ceiling.