The Human Visual System Can Inspire New Interaction Paradigms for LLMs

Diana Robinson, Neil D Lawrence

Keywords: envisioning alignment, challenges and future directions, large language models, human-computer interaction

TL;DR: We propose that the similarities underlying human visual perception and LLM information processing can inform new research directions for new ways of exploring and understanding LLMs.

Abstract: The dominant metaphor of LLMs-as-minds leads to misleading conceptions of machine agency and is limited in its ability to help both users and developers build the right degree of trust and understanding for outputs from LLMs. It makes it harder to disentangle hallucinations from useful model interactions. This position paper argues that there are fundamental similarities between visual perception and the way LLMs process and present language. These similarities inspire a metaphor for LLMs which could open new avenues for research into interaction paradigms and shared representations. Our visual system metaphor introduces possibilities for addressing these challenges by understanding the information landscape assimilated by LLMs.

Submission Type: Long Paper (9 Pages)

Archival Option: This is an archival submission

Presentation Venue Preference: ICLR 2025

Submission Number: 64