Large Language Models as superpositions of cultural perspectives

Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, Pierre-Yves Oudeyer

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Large Language Models, context-dependence, controllability, cultural values, personal values, personality traits, societal considerations, Shalom H Schwartz, Geert Hofstede, Big Five

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Conceptualization of LLMs as superpositions, empirical analyses of unexpected context-depentant value changes, and systematic analyses of different models on the controlability of expressed cultural, personal values, and personality.

Abstract: Large language models (LLMs) are sometimes viewed as if they were individuals, with given values, personality, knowledge and abilities. We argue that this ”LLM as an individual” metaphor misrepresents their nature. As opposed to humans, they exhibit highly context-dependent values and personality traits. We propose a new metaphor, ”LLM as a superposition of perspectives” : LLMs simulate a multiplicity of behaviors, e.g. expressing values, which can be triggered by a given context. As a case study, we conduct experiments on how values vary as a function of context using psychology questionnaires. Crucially, we demonstrate that changes in the context that are unrelated to the topic of questionnaires - varying articles, simulated conversations on other topics, and textual formats - all result in significant unwanted, hard-to-predict changes in the expressed values. We refer to this as the unexpected perspective shift effect. We discuss how this questions the interpretations of studies using psychology questionnaires (and more generally benchmarks) to draw general conclusions about LLMs’ values, knowledge and abilities. Indeed, expressing some values on a questionnaire says little about which values a model would express in other contexts. Instead, models should be studied in terms of how the expressed values change over contexts in both expected and unexpected ways. Following this insight, we introduce the concept of perspective controllability - a model’s affordance to adopt various perspectives. We conduct a systematic comparison of the controllability of 16 different models over three questionnaires (PVQ, VSM, IPIP) and different methods for inducing perspectives. We conclude by examining the broader implications of our work and outline a variety of associated scientific questions.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9417