Item Language Model
Keywords: large language models, interpretability methods, representation, recommender systems
TL;DR: We adapt Querying transformer with a domain-specific contrastive loss to learn language aligned representation for entities from a new domain and show improved interpretability of the domain entities by large language models (LLMs).
Abstract: Embeddings are extensively used in many domains to represent information about domain entities in a compressed manner. In recommendation systems, these embeddings are trained to extract meaningful information about an item/user from collaborative filtering data consisting users ratings or implicit feedback on items. These behavioral embeddings are usually not trained on data from language domain, but they encode very useful behavioral information which cannot be described using language. In contrast, in large language models (LLM) this collaborative data and behavioral entities(users/items) are not well represented as they are not textual and are specific to the recommendation system/product. Bridging this gap between behavioral understanding and language understanding can enable new item and language interleaved tasks. In our work we show how we can efficiently adapt rich behavioral embeddings as an additional behavioral input representation in pre-trained LLMs. To achieve this we adapt Querying Transformer technique with a new item contrastive loss and show improved item-text joint understanding in PALM2. Finally, we also demonstrate improved capabilities in recommendation domain over using the behavioral embeddings directly as input to PALM2.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12450
Loading