Toward engineering AGI: Benchmarking the engineering design capabilities of LLMs

Benchmark design plays a critical role in advancing large language models (LLMs) by providing systematic frameworks to measure, compare, and improve their capabilities across diverse tasks. While existing benchmarks have driven progress in areas such as factual question answering, code synthesis, and scientific problem solving, there remains a notable lack of benchmarks tailored to evaluate LLMs in real-world engineering design tasks. In this work, we introduce EngDesign, an Engineering Design benchmark that systematically assesses the proficiency of LLMs in solving domain-specific engineering design challenges. Unlike existing benchmarks that focus on factual recall or question answering, EngDesign uniquely emphasizes LLMs' ability to synthesize domain knowledge, reason under constraints, and generate functional, objective-oriented designs. Each task in EngDesign represents a real-world engineering design problem, accompanied by a detailed task description specifying design goals, constraints, and performance requirements. Crucially, instead of relying on static gold-standard answers, EngDesign introduces a novel simulation-based evaluation framework: for each task, we provide an executable evaluation pipeline that programmatically tests LLM-generated designs through domain-specific simulations (e.g., control system validation, circuit simulation). EngDesign proposes a new benchmark paradigm that shifts evaluation from static answer checking to dynamic, simulation-driven functional verification, enabling robust, reproducible, and performance-grounded assessment of LLM design capabilities.