The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure

ACL ARR 2025 May Submission4172 Authors

Abstract: Embedding-based similarity metrics between text sequences can be influenced not just by the content dimensions we most care about, but can also be biased by spurious attributes like the text's source or language. These document confounders cause problems for many applications, but especially those that need to pool texts from different corpora. This paper shows that a debiasing algorithm that removes information about observed confounders from the encoder representations substantially reduces these biases at a minimal computational cost. Document similarity and clustering metrics improve across every embedding variant and task we evaluate---often dramatically. Interestingly, performance on out-of-distribution benchmarks is not impacted, indicating that the embeddings are not otherwise degraded.

Paper Type: Long

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: NLP tools for social analysis, multilingual evaluation, semantic textual similarity, phrase/sentence embedding

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Data resources

Languages Studied: English, German, French, Italian, Spanish, Chinese

Submission Number: 4172