Exploring Task-Specific Dimensions in Word Embeddings Through Automatic Rule Learning

Liyuan Gao, Huixin Zhan, Victor S. Sheng

Abstract: Word embeddings are fundamental in natural language processing tasks. However, understanding and interpreting the individual dimensions within these embeddings remains a challenge. Traditional methods have largely concentrated on visualization techniques and the training of interpretable vectors. In this paper, we introduce an innovative methodology to find out task-related dimensions within word embeddings, leveraging automatic rule learning to extract dimensions critical to specific tasks. Our approach involves strategically nullifying the values of identified dimensions and evaluating their impact on the performance of the modified embeddings. The experiments are conducted on three widely-used pre-trained word embeddings, applied across two distinct datasets. Each dataset is designed to evaluate two specific tasks: one for gender classification and medical specialty classification, and the other for gender classification and sentiment analysis. Notably, the results reveal that the removal of gender-related dimensions significantly affects gender classification performance while having minimal impact on other tasks. This highlights that there exist different related dimensions for different tasks in word embeddings. Significantly, by removing the values of these dimensions for a specific task, the modified embeddings still work well on other tasks, which opens avenues for creating more tailored and efficient text analysis tools.