Heads up! Large Language Models Can Perform Tasks Without Your Instruction via Selective Attention Head Masking

Senyu Han, Hongchuan Zeng, Kai Yu, Lu Chen

TL;DR: By selectively masking attention heads and allowing only certain heads to remain active in LLMs, they can directly exhibit specific task functionalities without the presence of instruction prompts.

Abstract: Large language models (LLMs) consist of numerous Transformer modules, and while the models can perform various functions, it remains an open question of how these modules are combined to elicit distinct inherent functionalities. In this paper, we investigate the modules inside LLMs and demonstrate that, by simply masking or retaining specific attention heads during inference, LLMs can exhibit specific task functionalities without requiring explicit instructions or modifications to the model parameters. Experiments across various models and tasks reveal that LLMs inherently encode ``functional pathways'', the structured groups of interdependent attention heads that are crucial for executing specific tasks. These pathways not only govern the model's functional behaviors but also enhance parameter efficiency, as suppressing attention heads outside the pathway can improve task performance. The code is available in this repository: [https://github.com/OpenDFM/HeadsUp](https://github.com/OpenDFM/HeadsUp).

Lay Summary: Researchers are trying to understand how large language models (LLMs)—like those that power AI chatbots—perform different tasks using the same architecture. The core issue is that while these models consist of many parts called “attention heads,” it’s unclear how specific combinations of them contribute to different functions. In this study, we found that you can control which functions the model performs by turning on or off certain attention heads, without changing the model’s settings or giving it special instructions. Through experiments, we discovered that these models contain hidden “functional pathways”—groups of attention heads that naturally work together to handle specific tasks. Surprisingly, removing parts of the model that aren’t in use for a particular task can actually improve performance. This insight shows that LLMs are more organized internally than previously thought. Our findings could help make AI models more efficient, faster, and easier to understand.

Link To Code: https://github.com/OpenDFM/HeadsUp

Primary Area: Deep Learning->Large Language Models

Keywords: Large Language Model, Attention Mechanism, Interpretability, Deep Learning

Submission Number: 3834