Multi-Site Data Collection and Evaluation in Spoken Language Understanding

Lynette Hirschman, Madeleine Bates, Deborah A. Dahl, William M. Fisher, John S. Garofolo, David S. Pallett, Kate Hunicke-Smith, Patti Price, Alexander I. Rudnicky, Evelyne Tzoukermann

Abstract: The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development. The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods. Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MADCOW). We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.