Semantic Fusion with Fuzzy-Membership Features for Controllable Language Modelling

ICLR 2026 Conference Submission14295 Authors

Keywords: Semantic fusion, Fuzzy memberships, controllable language modelling

TL;DR: We propose a lightweight, interpretable add-on to Transformer LMs that injects fuzzy token-semantics, improving perplexity and enabling precise, user-controlled sentiment with minimal overhead for language generation.

Abstract: We propose \textit{semantic fusion}, a lightweight scheme that augments a Transformer language model (LM) with a parallel, fuzzy-membership feature channel that encodes token-level semantics. Each token is represented by a vector of interpretable features (e.g. part-of-speech cues, shallow roles, boundary flags, sentiment polarity and strength) whose values are graded degrees from differentiable membership functions (e.g. power kernels). These per-token vectors form a sentence-level semantic matrix fused via a gated adapter into the LM. Training uses standard next-token prediction, an auxiliary loss that reconstructs the semantic features from hidden states, and a lightweight “uniformizer” that regularizes adjective-class distributions. On a synthetic two-clause corpus with held-out adjectives for out-of-distribution (OOD) control, semantic fusion improves perplexity and enables precise, user-controllable generation of polarity and punctuation while maintaining model simplicity. This approach adds only small overhead, remains fully compatible with tied input-output embeddings, and provides an interpretable pathway for conditioned natural language generation.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14295