Collaborative Alignment of NLP Models

Fereshte Khani, Marco Tulio Ribeiro

Keywords: alignment, collaborative alignment, debugging, nlp, interference, multi-user interaction

TL;DR: We enable experts/users to interact with the model and align it with their values without interfering with each other.

Abstract:

Despite substantial advancements, Natural Language Processing (NLP) models often require post-training adjustments to enforce business rules, rectify undesired behavior, and align with user values.
These adjustments involve operationalizing "concepts"—dictating desired model responses to certain inputs.
However, it's difficult for a single entity to enumerate and define all possible concepts, indicating a need for a multi-user, collaborative model alignment framework.
Moreover, the exhaustive delineation of a concept is challenging, and an improper approach can create shortcuts or interfere with original data or other concepts.

To address these challenges, we introduce CoAlign, a framework that enables multi-user interaction with the model, thereby mitigating individual limitations.
CoAlign aids users in operationalizing their concepts using Large Language Models, and relying on the principle that NLP models exhibit simpler behaviors in local regions.
Our main insight is learning a \emph{local} model for each concept, and a \emph{global} model to integrate the original data with all concepts.
We then steer a large language model to generate instances within concept boundaries where local and global disagree.
Our experiments show CoAlign is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.

Submission Number: 2126