DOE and ANOVA based Performance Influencing Factor Analysis for Evaluation of Speech Recognition Systems

Xiangdong Wang, Feng Xie, Shouxun Lin, Yueliang Qian, Qun Liu

Abstract: In this paper, a framework of performance influencing factor analysis (PIFA) is proposed for evaluation of speech recognition systems. For each system under evaluation, the influence of various data properties (e. g. accent, SNR, and speaking rate) on system performance is analyzed to show corresponding characteristics of the system. The main idea of this approach is to design test data by means of Design of Experiments (DOE) methods and perform analysis of variance (ANOVA) and other statistical analyses on the performance of each system. To use ANOVA which is far more powerful than rank tests generally used in performance evaluation, a method is proposed for generating performance measurement data satisfying the two basic assumptions of ANOVA. The PIFA approach is applied to the 2004 HTRDP ASR Evaluations, and analysis results are reported and interpreted, which are considered helpful for pointing out virtues and deficiencies of each system and accelerating improvement.