Latent Space Attribute Disentanglement for Attribute-Based Controllable Text Generation with Large Language Models

ACL ARR 2025 May Submission6421 Authors

Abstract: Attribute‑based controllable text generation (CTG) aims to produce sentences that satisfy user‑specified properties while retaining fluency. Existing approaches either bias the decoding logit vectors or fine‑tune small adapter layers. However, they implicitly assume that the latent space of the model already provides clear and linearly separable directions for every attribute. In reality, heterogeneous training corpora induce highly overlapping attribute distributions, entangle latent features, and interfere with text generation with specific attributes. We introduce Latent Space Attribute Disentanglement, a lightweight but practical framework that explicitly factorizes the latent space into orthogonal subspaces, one for each attribute. Concretely, we attach gated LoRA experts to every transformer block; the gating mechanism learns to capture attribute‑specific patterns. These experts are optimized with two complementary objectives, domain alignment and subspace independence, enforced by additional loss terms. During decoding, our method generates text that precisely exhibits the desired attributes; extensive experiments demonstrate that the proposed framework delivers consistent and significant gains on attribute‑specific generation tasks.

Paper Type: Long

Research Area: Generation

Research Area Keywords: Controllable text generation, Large Language Model

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 6421