Familywise error rate control by interactive unmasking

Boyan Duan, Aaditya Ramdas, Larry Wasserman

Abstract:

We propose a method for multiple hypothesis testing with familywise error rate (FWER) control,
called the i-FWER test. Most testing methods are predefined algorithms that do not allow modifications
after observing the data. However, in practice, analysts tend to choose a promising algorithm after
observing the data; unfortunately, this violates the validity of the conclusion. The i-FWER test allows
much flexibility: a human (or a computer program acting on the human’s behalf) may adaptively guide
the algorithm in a data-dependent manner. We prove that our test controls FWER if the analysts adhere
to a particular protocol of “masking” and “unmasking”. We demonstrate via numerical experiments the
power of our test under structured non-nulls, and then explore new forms of masking