The Impact of State Merging on Predictive Accuracy in Probabilistic Tree Automata: Dietze's Conjecture Revisited

Johanna Björklund

Abstract: Dietze’s conjecture concerns the problem of equipping a tree automaton M with weights to make it probabilistic, in such a way that the resulting automaton N predicts a given corpus \(\mathcal {C}\) as accurately as possible. The conjecture states that the accuracy cannot increase if the states in M are merged with respect to an equivalence relation \(\sim \) so that the result is a smaller automaton \(M^\sim \). Put differently, merging states can never improve predictions. This is under the assumption that both M and \(M^\sim \) are bottom-up deterministic and accept every tree in \(\mathcal {C}\). We prove that the conjecture holds, using a construction that turns any probabilistic version \(N^\sim \) of \(M^\sim \) into a probabilistic version N of M, such that N assigns at least as great a weight to each tree in \(\mathcal {C}\) as \(N^\sim \) does.