Exact Paired-Permutation Testing for Structured Test Statistics

Anonymous

Paper Link: https://openreview.net/forum?id=cst0-LIBwku

Paper Type: Short paper (up to four pages of content + unlimited references and appendices)

Abstract: Significance testing—especially the paired-permutation test—has played a vital role in developing NLP systems to provide confidence that the difference in performance between two systems (i.e., the test statistic) is not due to luck. However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm. In this paper, we provide an efficient exact algorithm for the paired-permutation test for a family of structured test statistics. Our algorithm runs in $\mathcal{O}(G N (\log GN )(\log N))$ time where $N$ is the dataset size and $G$ is the range of the test statistic. We found that our exact algorithm was $10$x faster than the Monte Carlo approximation with $20000$ samples on a common dataset

Presentation Mode: This paper will be presented virtually

Virtual Presentation Timezone: UTC+1

Copyright Consent Signature (type Name Or NA If Not Transferrable): Ran Zmigrod

Copyright Consent Name And Address: Ran Zmigrod, 15 JJ Thomson Ave, Cambridge CB3 0FD, United Kingdom