Leveraging Human Revisions for Improving Text-to-Layout Models

Amber Xie, Chin-Yi Cheng, Forrest Huang, Yang Li

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: human feedback, reinforcement learning, generative models, UI layouts

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We use human revisions as a new form of feedback to better align text-to-layout models.

Abstract:

Learning from human feedback has shown success in aligning large, pretrained models with human values. However, prior works have mostly focused on using high-level labels, such as preferences between pairs of model outputs. On the other hand, many domains could benefit from more involved, detailed feedback, such as corrections, explanations, and reasoning of human users. Our work proposes using nuanced feedback through the form of human revisions for stronger alignment. In this paper, we ask expert designers to fix layouts generated from a generative layout model that is pretrained on a large-scale dataset of mobile screens. Then, we train a reward model based on how human designers revise these generated layouts. With the learned reward model, we optimize our model with reinforcement learning from human feedback (RLHF). Our method, Revision-Aware Reward Models (RARE), allows a generative model to produce more modern, designer-aligned layouts, showing the potential for utilizing human corrections and stronger forms of feedback in improving generative models.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4564