SLIM-LLMs: Low-Rank Models of Linguistic Style
Keywords: Linguistic Style, Sensorial Linguistics, LIWC, SLIM-LLMs
Abstract: Linguistic style encompasses a range of dimensions, including sensorial language as well as traditional stylistic features (represented using LIWC features). While these dimensions of linguistic style have been studied independently, relationships between the different dimensions, particularly between sensorial style and traditional stylistic features, remain understudied. This paper introduces a novel approach to model this interaction and tests it across a diverse set of texts.
In particular, we propose using a Reduced-Rank Ridge Regression (R4) to model low-rank latent relationships between LIWC-based stylistic features and sensorial language features. We find that compared to the full LIWC feature set ($r = 74$), its low-dimensional latent representations ($r = 24$) effectively capture stylistic information relevant to sensorial language prediction.
Based on our results, we propose Stylometrically Lean Interpretable Models (SLIM-LLMs) — dimensionality-reduced LLMs that model the non-linear relationships between these two major dimensions of style. We evaluate SLIM-LLMs on the ability to predict sensorial language (the actual sensorial words used) in five text genres: business reviews, novels, song lyrics, advertisements, and informative articles. Results show that SLIM-LLMs augmented with low-rank style features consistently outperform baseline models. These SLIM-LLMs approach the performance of full-scale language models while using significantly fewer parameters (up to 80\% reduction).
Primary Area: interpretability and explainable AI
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12691
Loading