LEMUR Neural Network Dataset: Towards Seamless AutoML

Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai Dhameliya, Zofia Antonina Bentyn, Dmitry Yu. Ignatov, Radu Timofte

Abstract:

Neural networks are the backbone of modern AI, yet designing, evaluating, and comparing them remains labor-intensive. We introduce LEMUR, an open-source dataset of training runs and accompanying framework that simplifies this process by unifying training and evaluation for a broad collection of PyTorch models across classification, segmentation, detection, and NLP tasks. Unlike typical AutoML or neural architecture search (NAS) toolkits that treat training as a black box, LEMUR exposes and standardizes it: models share a common template, datasets and metrics are modular, and experiments run with minimal configuration while remaining fully transparent. A core feature of LEMUR is end-to-end experiment logging. Every run stores a code-linked training trace comprising the task, dataset, metric and its code, model and its code, epoch, accuracy, duration, and all hyperparameters in a lightweight SQLite database, yielding a structured, queryable corpus of training runs for large-scale meta-analysis and LLM-ready controller learning. LEMUR integrates Optuna for automated hyperparameter search, includes analysis and visualization utilities, and offers a simple API for programmatic access. Its modular design makes adding new models, datasets, or metrics straightforward while preserving comparability, accelerating research and enabling fair benchmarking at scale. The core and plugins are released under the MIT license at:
\href{https://github.com/ABrain-One/nn-dataset}{https://github.com/ABrain-One/nn-dataset},
\href{https://github.com/ABrain-One/nn-plots}{https://github.com/ABrain-One/nn-plots}, and
\href{https://github.com/ABrain-One/nn-vr}{https://github.com/ABrain-One/nn-vr}.