Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective

Tianyu Liu, Afra Amini, Mrinmaya Sachan, Ryan Cotterell

Submission Type: Regular Long Paper

Submission Track: Syntax, Parsing and their Applications

Keywords: structured prediction, dependency parsing, coreference resolution

TL;DR: We show that most structured prediction problems in NLP can be solved in linear time and space by considering them as partial orderings of the tokens in the input string.

Abstract:

Tasks that model the relation between pairs of tokens in a string are a vital part of understanding natural language.
Such tasks, in general, require exhaustive pair-wise comparisons of tokens, thus having a quadratic runtime complexity in the length of the string.
We show that these exhaustive comparisons can be avoided, and, moreover, the complexity of such tasks can be reduced to linear by casting the relation between tokens as a partial order over the string.
Our method predicts real numbers for each token in a string in parallel and sorts the tokens accordingly, resulting in total orders of the tokens in the string.
Each total order implies a set of arcs oriented from smaller to greater tokens, sorted by their predicted numbers.
The intersection of total orders results in a partial order over the set of tokens in the string, which is then decoded into a directed graph representing the desired linguistic structure.
Our experiments on dependency parsing and coreference resolution show that our method achieves state-of-the-art or comparable performance.
Moreover, the linear complexity and parallelism of our method double the speed of graph-based coreference resolution models, and bring a 10-times speed-up over graph-based dependency parsers.

Submission Number: 305