UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs

Yi Gui, Yao Wan, Zhen Li, Zhongyi Zhang, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, Xiangliang Zhang

Track: Web mining and content analysis

Keywords: UICopilot, Webpage Synthesis, UI Generation, Code Generation

TL;DR: We propose UICopilot, a framework that decouples HTML structure and style generation to improve efficiency and effectiveness in webpage code generation.

Abstract:

Automating the synthesis of User Interface (UI) plays an important role in enhancing productivity, ensuring design consistency, and
expediting the development lifecycle. Recently, the rapid development of Multimodal Large Language Model (MLLM) has made it
possible to generate front-end Hypertext Markup Language (HTML) code directly from webpage designs. However, real-world web-
pages encompass not only a diverse array of HTML tags but also complex stylesheets, resulting in significantly lengthy code. The
lengthy code challenges the performance and efficiency of MLLMs, especially in capturing UI’s structure information. To address this
challenge, we propose UICopilot, a structure-aware HTML code generation framework from webpage designs via hierarchy code
generation. Our framework introduces a structure model and a code agent, decoupling the generation of the HTML code’s hierarchical
structure from its fine-grained details, thereby significantly reducing the model’s burden in producing lengthy code. We evaluate our
framework on real-world test datasets, and the experimental results demonstrate that it significantly outperforms existing baselines in
both automatic metrics and human evaluations. Specifically, statistical analysis reveals that the majority of human annotators prefer
the webpages generated by our framework over those produced by GPT-4V.

Submission Number: 613