BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints

Lyle Regenwetter, Yazan Abu Obaideh, Fabien M. C. Chiotti, Ioanna Lykourentzou, Faez Ahmed

Keywords: Benchmark, LLM, Optimization, Synthetic Data, CAD, Generative Models, Constraints, Bicycle, Design

TL;DR: We introduce a bicycle design benchmark that evaluates multiphysics performance, constraint satisfaction, and adherence to human preferences, and we benchmark LLMs, tabular generative models, and design optimization algorithms side by side.

Abstract: We introduce BikeBench, an engineering design benchmark for evaluating generative models on problems with multiple real-world objectives and constraints. As generative AI's reach continues to grow, evaluating its capability to understand physical laws, human guidelines, and hard constraints grows increasingly important. Engineering product design lies at the intersection of these difficult tasks, providing new challenges for AI capabilities. BikeBench evaluates AI models' capabilities to generate bicycle designs that not only resemble the dataset, but meet specific performance objectives and constraints. To do so, BikeBench quantifies a variety of human-centered and multiphysics performance characteristics, such as aerodynamics, ergonomics, structural mechanics, human-rated usability, and similarity to subjective text or image prompts. Supporting the benchmark are several datasets of simulation results, a dataset of 10,000 human-rated bicycle assessments, and a synthetically generated dataset of 1.6M designs, each with a parametric, CAD/XML, SVG, and PNG representation. BikeBench is uniquely configured to evaluate tabular generative models, large language models (LLMs), design optimization, and hybrid algorithms side-by-side. Our experiments indicate that LLMs and tabular generative models fall short of hybrid GenAI+optimization algorithms in design quality, constraint satisfaction, and similarity scores, suggesting significant room for improvement. We hope that BikeBench, a first-of-its-kind benchmark, will help catalyze progress in generative AI for constrained multi-objective engineering design problems. We provide code, data, an interactive leaderboard, and other resources at https://github.com/Lyleregenwetter/BikeBench.

Dataset URL: https://doi.org/10.7910/DVN/BSJSM6

Code URL: https://github.com/Lyleregenwetter/BikeBench

Primary Area: Datasets & Benchmarks illustrating Different Deep learning Scenarios (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 1941