PMG : Personalized Multimodal Response Generation with Large Language Models

Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, Xi Xiao

Keywords: Multimodal, Large Language Model, Response Generation, Personalization

Abstract:

The emergence of large language models (LLMs) has revolutionized the capabilities of text comprehension and generation. While previous research has focused on multimodal understanding using LLMs, there is little work on personalized generation, particularly in the context of recommender systems. This paper proposes the first method for personalized multimodal response generation using LLMs, showcases its applications and validates its performance via an extensive experimental study on two datasets. The proposed method, Personalized Multimodal Generator (PMG for short) first converts user behaviors (e.g., clicks in recommender systems or conversations with a virtual assistant) into natural language to facilitate LLM understanding and extract user preference descriptions. Such user preferences are then fed into a generator, such as a multimodal LLM or diffusion model, to produce personalized responses. To capture user preferences comprehensively and accurately, we propose to let the LLM output a combination of explicit keywords and implicit embeddings to represent user preferences. Then the combination of keywords and embeddings are used as prompts to condition the generator. We optimize a weighted sum of the accuracy score and preference score so that the generated responses have a good balance between them. Compared to a baseline method without personalization, PMG has a significant improvement on personalization for up to 8\% in terms of LPIPS while retaining the accuracy of generated responses.

This paper proposes a method for generating multimodal responses, which is prevalent in web applications.

Track: User Modeling and Recommendation

Submission Guidelines Scope: Yes

Submission Guidelines Blind: Yes

Submission Guidelines Format: Yes

Submission Guidelines Limit: Yes

Submission Guidelines Authorship: Yes

Student Author: Yes

Submission Number: 1913