Essay.08-FeiyangXie-2100017837

Feiyang Xie

Keywords: preference; RL; utility

Abstract: Rooted in the intersection of philosophy, economics, and game theory, the concept of utility stands as a foundational principle within modern decision theory. In this framework, an agent's rational decision-making is guided by the pursuit of maximizing its expected utility, encapsulated by the principle of maximum expected utility. In the realm of daily life, individuals not only shape their current behaviors based on utility functions but also employ them to decipher the actions and intentions of others, as well as discern cause-and-effect relationships within their environment. The acquisition and representation of human utility have emerged as a pivotal research area, particularly within fields such as computer vision (CV) and reinforcement learning (RL). Given the intricate nature of human utility functions, direct modeling proves challenging. Consequently, the predominant approach involves learning from human preferences. In the context of RL, the assimilation of human preferences is often conceptualized as the acquisition of a reward function, aligning with the agent's overarching objective to maximize cumulative rewards. This essay endeavors to explore the representation of human preference, delving into various computational frameworks for utility learning while scrutinizing their respective advantages and disadvantages. Subsequently, it will delve into the intricate landscape of human utility within large language models (LLMs) and elucidate the limitations inherent in preference-based methodologies.

Submission Number: 176