Neural Coherence
Keywords: Out-of-Distribution, Meta-Learning, Transfer Learning, Few-Shot Learning, Domain Adaptation, Model Selection
Abstract: Many important machine learning problem settings involve adapting a large pre-trained model to a limited number of examples. To create a good pre-trained model a number of choices must be made, from determining when to stop training the model, to which dataset(s) to train the model upon. In this work we develop a principle that we shall refer to as Neural Coherence, which is based on characterizing the way in which a neural network behaves on different types of inputs using the statistics of activation functions across different sets of inputs. Our experiments show that these measures of Neural Coherence can be used to formulate a general approach for inferring and improving the generalization of a model to a downstream problem. We show how Neural Coherence can be used to make decisions for early-stopping, and to infer which dataset to train on, given a target dataset. Overall, our experiments indicate that our approach to using Neural Coherence for model selection tasks can significantly improve the performance of deep learning models to out-of-distribution downstream problems.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5923
Loading