Unveiling Multiple Descents in Unsupervised Autoencoders

Kobi Rahimi, Yehonathan Refael, Tom Tirer, Ofir Lindenbaum

Abstract: The phenomenon of double descent has challenged the traditional bias-variance trade-off in supervised learning but remains unexplored in unsupervised learning, with some studies arguing for its absence. In this study, we first demonstrate analytically that double descent does not occur in linear unsupervised autoencoders (AEs). In contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the critical role of bottleneck size in shaping the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. Our findings indicate that over-parameterized models not only improve reconstruction but also enhance performance in downstream tasks such as anomaly detection and domain adaptation, highlighting their practical value in complex real-world scenarios.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have carefully addressed all the suggestions of reviewers p8vq, RR8M, CF33, and incorporated the required changes into the camera-ready version of the paper.
Regarding reviewer p8vq, we added additional steps to the mathematical proof to improve clarity and summarized which experiments follow the standard ML practice of using the same distribution and which do not.

Following reviewer RR8M suggestions, we clarified the term “SNR” within the main paper to make it self-contained, included a brief description of the datasets, and provided justifications for their selection in the Data Model section. In the Conclusion section, we added a discussion on the broader implications and potential impact of our work, as well as several open research questions arising from our findings.

Additionally, we restructured Sections 3.2-3.5 to improve readability, clarified the definition of “undercomplete FCN” (now referred to as “undercomplete AE FCN”), and addressed all minor concerns raised by the reviewers.

Following the recommendations provided by reviewer CF33, we applied the suggested refinements, including enlarging figure legends, improving figure placement, reducing excess whitespace, and consolidating the double descent figures under consistent conditions to better highlight the source of the observed phenomenon.

Assigned Action Editor: ~Tatiana_Likhomanenko1

Submission Number: 4970