UniAR: A Unified model for predicting human Attention and Responses on visual content

Peizhao Li, Junfeng He, Gang Li, Rachit Bhargava, Shaolei Shen, NACHIAPPAN VALLIAPPAN, Youwei Liang, Hongxiang Gu, Venky Ramachandran, Golnaz farhadi, Yang Li, Kai J Kohlhoff, Vidhya Navalpakkam

Keywords: attention, saliency, scanpath, aesthetics

Abstract: Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.

Primary Area: Neuroscience and cognitive science (neural coding, brain-computer interfaces)

Flagged For Ethics Review: true

Submission Number: 13099