Preferences Influence Deeply: Enhancing Interaction with Large Language Models via Inference-time Alignment

ACL ARR 2025 May Submission1752 Authors

Abstract:

User-LLM interaction history contains rich user preferences, which may guide the model in generating more personalized responses. There has not been a systematic exploration of whether current LLMs can infer and align these preferences automatically, and to what extent they can.
To fill this gap, we have conducted this study on the capabilities of the current LLMs.
We begin by formalizing the task and introducing the \textbf{\ourbench} benchmark for evaluation.
This benchmark includes:
1) A set of interaction histories that contains different preferences, constructed through real histories we collected from a self-built temporary website.
2) A systematic evaluation tool kit.

We tested the performance of over 20 open-sourced and proprietary LLMs across various scenarios, including the bare model, hand-crafted prompts, human-designed workflows, and fine-tuning.
Our findings reveal that this task is an overlooked capability in current LLM alignment. Furthermore, by comparing different models and analyzing the failure cases, we provide insights for enhancing model performance in the future.
We demonstrate that fine-tuning on \ourbench can make LLM consider more preferences. This exploration paves the way for the development of future powerful personalized AI assistants.
The project can be accessed at \url{https://anonymous.4open.science/r/InterPref}.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: applications

Contribution Types: NLP engineering experiment, Data resources, Data analysis

Languages Studied: English, Chinese

Submission Number: 1752