VisualLens: Personalization through Task-Agnostic Visual History

Wang Bill Zhu, Deqing Fu, Kai Sun, Yi Lu, Zhaojiang Lin, Seungwhan Moon, Kanika Narang, MUSTAFA CANIM, Yue Liu, Anuj Kumar, Xin Luna Dong

Keywords: Recommendation, Personalization, LLM, Multimodal LLM

TL;DR: We propose VisualLens, which leverages task-agnostic user's visual record to enhance the understanding of individual preferences and deliver more personalized MLLM recommendations.

Abstract:

Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals.
However, item-based histories are not always accessible and generalizable for multimodal recommendation.
We hypothesize that a user's visual history --- comprising images from daily life --- can offer rich, task-agnostic insights into their interests and preferences, and thus be leveraged for effective personalization.
To this end, we propose VisualLens, a novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task-agnostic visual history.
VisualLens extracts, filters, and refines a spectrum user profile from the visual history to support personalized recommendation.
We created two new benchmarks, Google-Review-V and Yelp-V, with task-agnostic visual histories, and show that VisualLens improves over state-of-the-art item-based multimodal recommendations by 5-10\% on Hit@3, and outperforms GPT-4o by 2-5\%.
Further analysis shows that VisualLens is robust across varying history lengths and excels at adapting to both longer histories and unseen content categories.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 18582