Fine-tuning Deep Neural Networks by Interactively Refining the 2D Latent Space of Ambiguous Images

Jiafu Wei, Haoran Xie, Chia-Ming Chang, Xi Yang

Abstract: Deep neural networks (DNNs) have achieved excellent results currently in classification, while they may still suffer from ambiguous images which are similar across classes. By contrast, humans have a relatively good ability to distinguish these categories of images. Therefore, we propose a human-in-the-loop solution to assist the network to better classify the images by leveraging human knowledge. To achieve this, we project the high-dimensional latent space trained by the network onto a two-dimensional workspace. The users can interactively modify the projected coordinates of inputs on the workspace using our designed tools, then the modified information will be fed back to the network to fine-tune it, which in turn affects the network's classification results, thereby improving the accuracy of network classification.