Clusters Emerge in Transformer-based Causal Language Models

Xinbo Wu, Lav R. Varshney

Track: Extended abstract

Keywords: transformer, language models

Abstract: Even though large language models (LLMs) have demonstrated remarkable capability in solving various natural language tasks, the capability of an LLM to follow human instructions is still an area of active development. Recent works \citep{RLHF, DPO, IT_survey} have shown great improvements in instruction-following capability through additional training for instruction-following tasks. However, the mechanisms responsible for effective instruction-following capabilities remain inadequately understood. Here, we introduce a simplified instruction-following task and use synthetic datasets to analyze a Transformer-based causal language model. Our findings suggest that the model learns task-specific information by clustering data within its hidden space, with this clustering process evolving dynamically during learning.

Submission Number: 57