ChipVQA: Benchmarking Visual Language Models for Chip Design

Haoyu Yang, Qijing Huang, Nathaniel Pinckney, Walker Turner, Wenfei Zhou, Yanqing Zhang, Chia-Tung Ho, Chen-Chia Chang, Haoxing Ren

Keywords: Multimodal LLM; Chip Design and Manufacturing; VQA

TL;DR: We present a benchmark suite for VLM on chip design and manufacturing knowledge

Abstract:

Large-language models (LLMs) have exhibited great potential to as-
sist chip designs and analysis. Recent research and efforts are mainly
focusing on text-based tasks including general QA, debugging, design
tool scripting, and so on. However, chip design and implementa-
tion workflow usually require a visual understanding of diagrams,
flow charts, graphs, schematics, waveforms, etc, which demands
the development of multi-modality foundation models. In this paper, we propose ChipVQA, a benchmark designed to evaluate the
capability of visual language models for chip design. ChipVQA includes 142 carefully designed and collected VQA questions covering five chip design disciplines: Digital Design, Analog Design, Architecture, Physical Design and Semiconductor Manufacturing. Un-
like existing VQA benchmarks, ChipVQA questions are carefully
designed by chip design experts and require in-depth domain knowledge and reasoning to solve. We conduct comprehensive evaluations
on both open-source and proprietary multi-modal models that are
greatly challenged by the benchmark suit. ChipVQA examples are available at
https://anonymous.4open.science/r/chipvqa-2079/.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12186