Task Discovery: Finding the Tasks that Neural Networks Generalize onDownload PDF

Published: 31 Oct 2022, Last Modified: 22 Oct 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: Generalization, Understanding Neural Networks, Deep Learning
Abstract: When developing deep learning models, we usually decide what task we want to solve then search for a model that generalizes well on the task. An intriguing question would be: what if, instead of fixing the task and searching in the model space, we fix the model and search in the task space? Can we find tasks that the model generalizes on? How do they look, or do they indicate anything? These are the questions we address in this paper. We propose a task discovery framework that automatically finds examples of such tasks via optimizing a generalization-based quantity called agreement score. We demonstrate that one set of images can give rise to many tasks on which neural networks generalize well. These tasks are a reflection of the inductive biases of the learning framework and the statistical patterns present in the data, thus they can make a useful tool for analyzing the neural networks and their biases. As an example, we show that the discovered tasks can be used to automatically create ''adversarial train-test splits'' which make a model fail at test time, without changing the pixels or labels, but by only selecting how the datapoints should be split between the train and test sets. We end with a discussion on human-interpretability of the discovered tasks.
TL;DR: We propose a framework to find the tasks that a neural network can generalize on. We investigate some of the intriguing properties of such tasks which are a reflection of the inductive biases of the network and the statistical patterns in the data.
Supplementary Material: zip
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/arxiv:2212.00261/code)
18 Replies

Loading