TestAug: A Framework for Augmenting Capability-based NLP Tests

Anonymous

Abstract: The recently proposed capability-based NLP tests go beyond the traditional heldout evaluation paradigm, allowing model developers to test the different linguistic capabilities of a model. However, existing work on capability-based testing requires the (semi-)manual creation of the test suites (templates); such approach thus heavily relies on the linguistic expertise and domain expertise of the developers. In this paper, we investigate an automatic approach for generating and augmenting the test suites by prompting the GPT-3 engine. Our experiments show that our approach can generate diverse test suites which has a better coverage than the existing approaches using templates. The augmented test suites can also be used to detect more errors compared to existing work. Our test suites can be downloaded at https://anonymous-researcher-nlp.github.io/testaug/.

Paper Type: long