Virtual Adversarial Training for Semi-Supervised Text Classification

Takeru Miyato, Andrew M. Dai, Ian Goodfellow

Abstract:

Adversarial training provides a means of regularizing supervised learning algorithms
while virtual adversarial training is able to extend supervised learning algorithms
to the semi-supervised setting. However, both methods require making
small perturbations to numerous entries of the input vector, which is inappropriate
for sparse high-dimensional inputs such as one-hot word representations. We
extend adversarial and virtual adversarial training to the text domain by applying
perturbations to the word embeddings in a recurrent neural network rather than
to the original input itself. The proposed method achieves state of the art results
on multiple benchmark semi-supervised and purely supervised tasks. We provide
visualizations and analysis showing that the learned word embeddings have improved
in quality and that while training, the model is less prone to overfitting.

Recommender: Takeru Miyato