Heterogeneity of Regularization between adjacent periods

SangHwa Hong, Bongseok Kim, Min Kyu Sim

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Regularization, Heterogeneity, Periodic Regularization, Reinforcement Learning, Transfer Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper proposes Periodic Regularization and integrates this method with Reinforcement Learning or Transfer Learning to establish regularization techniques that consistently demonstrates remarkable performance across multiple datasets

Abstract: Since the inception of deep learning, regularization techniques have been developed for the purpose of preventing the overfitting phenomenon. Regularization is typically accomplished in two ways: incorporating randomness (e.g., injecting noise into data, activating nodes, or using dropout) or heterogeneity (e.g., data augmentation). These approaches are known to lead to better generalization and, consequently, improved performance. In the case of introducing heterogeneity by adjusting the hyperparameter during the training process, such as the drop rate of dropout, experiments have shown that tuning hyperparameters after a period, which consists of a certain number of forward propagations, is more effective than either uniformly sustaining hyperparameters or tuning them during every propagation. Therefore, this paper proposes a novel regularization technique named Periodic Regularization that introduces periodicity into the dynamic hyperparameter tuning of other regularization methods. Furthermore, this paper suggests combining Periodic Regularization and other learning techniques such as Reinforcement Learning and Transfer Learning. This approach, particularly when combining dropout and reinforcement learning, shows significant improvement in empirical testing across various popular datasets. This is notably evident in Facial Expression Recognition (FER) tasks, where conventional methods, such as noise injection and dropout, have proven ineffective. Our proposed periodic regularization method not only can fill the research gap found in traditional regularization techniques but also can be a cornerstone for further research where the concept of periodic regularization is combined with diverse vanilla regularization techniques and learning techniques.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3476