D2U: Distance-to-Uniform Learning for Out-of-Scope Detection

Anonymous

Abstract: Supervised models trained for single-label classification tasks with cross-entropy loss are implicitly enforced to produce probability distributions that follow a discrete delta distribution in training. Model predictions in test time are expected to be similar to delta distributions given that the classifier determines the class of an input correctly. However, the shape of the predicted probability distribution becomes similar to the uniform distribution when the model cannot infer properly. We exploit this observation for detecting out-of-scope (OOS) utterances in conversational systems. Specifically, we propose a zero-shot post-processing step, called Distance-to-Uniform (D2U), exploiting not only the classification confidence score, but the shape of the entire output distribution. We also introduce a learning procedure that uses D2U for loss calculation in the supervised setup. We conduct experiments using six publicly available datasets. Experimental results show that the performance of out-of-scope detection is improved with our post-processing when there is no OOS training data, as well as with D2U learning procedure when OOS training data is available.