How Does Data Diversity Shape The Weight Landscape of Neural Networks?

Yang Ba, Michelle V Mancenido, Rong Pan

Keywords: data diversity, regularization, data augmentation, synthetic data, transfer learning

TL;DR: Using Random Matrix Theory (RMT), we find that data diversity shapes the weight matrix similarly to dropout, improving model generalization. We also explore how synthetic data can further enhance training diversity.

Abstract:

To enhance the generalization of machine learning models to unseen data, techniques such as dropout, weight decay (L2 regularization), and noise augmentation
are commonly employed. While regularization methods (i.e., dropout and weight
decay) are geared toward adjusting model parameters to prevent overfitting, data
augmentation increases the diversity of the input training set, a method purported
to improve accuracy and calibration error. In this paper, we investigate the impact of each of these techniques on the parameter space of neural networks, with
the goal of understanding how they alter the weight landscape in transfer learning
scenarios. To accomplish this, we employ Random Matrix Theory to analyze the
eigenvalue distributions of pre-trained models, fine-tuned using these techniques
but using different levels of data diversity, for the same downstream tasks. We
observe that diverse data influences the weight landscape in a similar fashion as
dropout. Additionally, we compare commonly used data augmentation methods
with synthetic data created by generative models. We conclude that synthetic data
can bring more diversity into real input data, resulting in a better performance on
out-of-distribution test instances.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12304