IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation

Xiaoran Yang, Yuyang Du, Kexin Chen, Soung Chang Liew, Jiamin Lu, Ziyu Guo, Xiaoyan Liu, Qun Yang, Shiqi XU, Xingyu Fan, Yuchen Pan, Taoyong Cui, Hongyu Deng, Boris Düdder, Jianzhang Pan, Qun Fang, Pheng-Ann Heng

Keywords: LLM Benchmark, Industrial Automation, General Computer Control

TL;DR: IndusGCC introduces the first dataset and benchmark for LLM-based GUI automation in industrial settings, spanning 448 tasks across seven domains, with a new evaluation framework revealing both the promise and limitations of LLM-driven automation.

Abstract: As Industry 4.0 progresses, flexible manufacturing has become a cornerstone of modern industrial systems, with equipment automation playing a pivotal role. However, existing control software for industrial equipment, typically reliant on graphical user interfaces (GUIs) that require human interactions such as mouse clicks or screen touches, poses significant barriers to the adoption of code-based equipment automation. Recently, Large Language Model-based General Computer Control (LLM-GCC) has emerged as a promising approach to automate GUI-based operations. However, industrial settings pose unique challenges, including visually diverse, domain-specific interfaces and mission-critical tasks demanding high precision. This paper introduces IndusGCC, the first dataset and benchmark tailored to LLM-GCC in industrial environments, encompassing 448 real-world tasks across seven domains, from robotic arm control to production line configuration. IndusGCC features multimodal human interaction data with the equipment software, providing robust supervision for GUI-level code generation. Additionally, we propose a novel evaluation framework with functional and structural metrics to assess LLM-generated control scripts. Experimental results on mainstream LLMs demonstrate both the potential of LLM-GCC and the challenges it faces, establishing a strong foundation for future research toward fully automated factories. Our data and code are publicly available at: https://github.com/Golden-Arc/IndusGCC.

Archival Option: The authors of this submission want it to appear in the archival proceedings.

Submission Number: 89