Moment Distributionally Robust Tree Structured Prediction

Yeshu Li, Danyal Saeed, Xinhua Zhang, Brian D Ziebart, Kevin Gimpel

Keywords: structured prediction, robustness, arborescence, projection, dependency parsing

TL;DR: We propose a distributionally robust method for structured prediction of tree-shaped objects with consistency and generalization guarantees.

Abstract: Structured prediction of tree-shaped objects is heavily studied under the name of syntactic dependency parsing. Current practice based on maximum likelihood or margin is either agnostic to or inconsistent with the evaluation loss. Risk minimization alleviates the discrepancy between training and test objectives but typically induces a non-convex problem. These approaches adopt explicit regularization to combat overfitting without probabilistic interpretation. We propose a moment-based distributionally robust optimization approach for tree structured prediction, where the worst-case expected loss over a set of distributions within bounded moment divergence from the empirical distribution is minimized. We develop efficient algorithms for arborescences and other variants of trees. We derive Fisher consistency, convergence rates and generalization bounds for our proposed method. We evaluate its empirical effectiveness on dependency parsing benchmarks.