Embedding semantic relationships in hidden representations via label smoothing
Keywords: deep learning, hierarchical modeling, neural network analysis
Abstract: In recent years, neural networks have demonstrated their ability to learn previously intractable problems, particularly in the field of computer vision. Although classification accuracy continues to increase for the most challenging benchmark datasets, model efficacy evaluations typically focus on raw accuracy results without a consideration for the types of errors being made. Further, most networks are trained such that classes are treated as separate, unrelated categories without any notion of higher order semantic relationships. This work shows a simple approach for embedding semantic relationships into learned representations via category-aware label smoothing. Using MNIST experiments, we demonstrate that preferable error profiles can be enforced this way and that underparameterized networks without the capacity to achieve reasonable raw accuracy are still capable of learning an effective model when relative error cost is taken into consideration. Additionally, we embed hierarchical information into cifar10 class labels to show that it is possible to enforce arbitrary semantic hierarchies using this method. Further, we use a new method for analyzing class hierarchy in hidden representations, Neurodynamical Agglomerative Analyisis (NAA), to show that latent class relationships in this analysis model tend toward the relationships of the label vectors as the data is projected deeper into the network.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
One-sentence Summary: We show a novel approach to embedding semantic information into deep networks via label smoothing and show its impact on latent representations using multiple analysis techniques.
Reviewed Version (pdf): https://openreview.net/references/pdf?id=HnWnXfdLS4
5 Replies
Loading