Update Frequently, Update Fast: Retraining Semantic Parsing Systems in a Fraction of Time

Anonymous

Abstract:

Currently used semantic parsing systems deployed in voice assistants can require weeks to train.
Datasets for these models often receive small and frequent updates, data patches. Each patch requires training a new model. To reduce training time, one can fine-tune the previously trained model on each patch, but naive fine-tuning exhibits catastrophic forgetting -- degradation of the model performance on the data not represented in the data patch.

In this work, we propose a simple method that alleviates catastrophic forgetting and show that it is possible to match the performance of a model trained from scratch in less than 10\% of a time via fine-tuning. The key to achieving this is supersampling and EWC regularization. We demonstrate the effectiveness of our method on multiple splits of the Facebook TOP and SNIPS datasets.