QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design

Rui Yang, Ziruo Wang, Yuntian Gu, Yitao Liang, Tongyang Li

Keywords: AI for science, quantum computing, quantum algorithm design

Abstract:

Quantum computing is an emerging field recognized for the significant speedup it offers over classical computing through quantum algorithms. However, designing and implementing quantum algorithms pose challenges due to the complex nature of quantum mechanics and the necessity for precise control over quantum states. Despite the significant advancements in AI, there has been a lack of datasets specifically tailored for this purpose. In this work, we introduce QCircuitBench, the first benchmark dataset designed to evaluate AI's capability in designing and implementing quantum algorithms in the form of quantum circuit codes. Unlike using AI for writing traditional codes, this task is fundamentally more complicated due to highly flexible design space.

Our key contributions include:
1. A general framework which formulates the key features of quantum algorithm design task for Large Language Models.
2. Implementation for quantum algorithms from basic primitives to advanced applications, spanning 3 task suites, 25 algorithms, and 120,290 data points.
3. Automatic validation and verification functions, allowing for iterative and interactive evaluation without human inspection.
4. Promising potential as a training dataset through primitive fine-tuning results.

We observed several interesting experimental phenomena: fine-tuning does not always outperform few-shot learning, and LLMs tend to exhibit consistent error patterns. QCircuitBench provides a comprehensive benchmark for AI-driven quantum algorithm design, while also revealing some limitations of LLMs in this domain.

Dataset URL: https://doi.org/10.7910/DVN/ZC4PNI

Primary Area: Other (please use sparingly, only use the keyword field for more details)

Submission Number: 1519