Semantic Similarity Based Label Augmentation for Visual Classification

Yu Cao

Keywords: Label Augmentation, CLIP, Semantic Similarity, Visual Classification

TL;DR: How semantic similarity can be used to augment labels for training visual classification models.

Abstract: Real-world applications may present visual categories for which examples are many but a definition is elusive. When data augmentation helps little and hand-crafted heuristics fail to warrant weak supervision, similarity remains a simple but effective guide for augmenting training labels. This paper showcases, for recognizing clutters in e-commerce photography, how similarity between visual representations pre-trained to capture semantics provides quality supervision. Our approach significantly improves on the label propagation baseline, with classification precision and recall above 0.8 in most cases.