DI-Bench: Systematically generating in-domain benchmarks for data intelligence agents

Evaluating LLM-based agents on enterprise data intelligence tasks, where correct answers require both retrieving information from unstructured documents and computing over structured databases, remains challenging due to the cost of manual benchmark construction. We present DI-Bench, a domain-transferable pipeline that generates evaluation benchmarks from any existing database by inverting the traditional approach: gold answers are computed first via programmatic SQL execution, then natural language questions are generated backward from the answers. A typed graph over tables, metrics, policies, and dimensions ensures systematic task diversity and reliability, and guarantees that tasks cannot be answered correctly without retrieving the appropriate knowledge. We demonstrate the pipeline on two public datasets (Brazilian e-commerce and Czech financial), producing 752 validated tasks across 8 analysis types (489 BR, 263 CZ). Evaluation of 4 models shows clear capability separation, with accuracies ranging from 32–57% depending on model and domain. The pipeline cuts manual benchmark construction effort by over 90%.