Reducing Parsing Complexity by Intra-Sentence Segmentation based on Maximum Entropy Model

Sung Dong Kim, Byoung-Tak Zhang, Yung Tack Kim

Abstract: Long sentence analysis has been a critical problem because of high complexity. This paper addresses the reduction of parsing complexity by intra-sentence segmentation, and presents maximum entropy model for determining segmentation positions. The model features lexical contexts of segmentation positions, giving a probability to each potential position. Segmentation coverage and accuracy of the proposed method are 96% and 88% respectively. The parsing efficiency is improved by 77% in time and 71% in space.