One-Hot Encoding Strikes Back: Fully Orthogonal Coordinate-Aligned Class Representations
Abstract: Representation learning via embeddings has become a central component in many machine learning tasks. This featurization process has gotten gradually less interpretable from each coordinating having a specific meaning (e.g., one-hot encodings) to learned distributed representations where meaning is entangled across all coordinates. In this paper, we provide a new mechanism that converts state-of-the-art embedded representations and carefully augments them to allocate some of the coordinates for specific meaning. We focus on applications in multi-class image processing applications, where our method Iterative Class Rectification (ICR) makes the representation of each class completely orthogonal, and then changes the basis to be on coordinate axes. This allows these representations to regain their long-lost interpretability, and demonstrating that classification accuracy is about the same or in some cases slightly improved.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Andriy_Mnih1
Submission Number: 2173
Loading