Response Patterns to Rotation Angle in a Rotation Pretext Task Vary Across Datasets and Architectures: An Observation and a Negative Result

Paul Yan, Amy Saranchuk, Michael Guerzhoy

Keywords: self-supervised pretraining, contrastive learning, rotation augmentation, periodicity, representation learning, medical imaging, HoG, SVM

TL;DR: Weird patterns emerge when varying the rotation angle in the rotation pretext task in SSP, and we don't know why

Abstract:

We show that the response to rotation angle $\theta$ in a rotation-based pretext task in self-supervised pretraining (SSP) via contrastive learning interacts in systematic, dataset-dependent, and architecture-dependent ways that produce unique ``signature'' curves of performance versus $\theta$. We perform a comprehensive $16\times 16$ experiment, pre-training eight encoder architectures on 16 diverse image datasets using both SimCLR and MoCo v2, with $\theta$ swept from $0^\circ$ to $360^\circ$ in $0.1^\circ$ increments. Each of the resulting 256 accuracy-versus-$\theta$ plots exhibits a distinct periodic pattern. A simple classifier trained on these curves can predict the originating dataset and encoder–method pair with high accuracy, confirming patterns specific to both datasets and architectures.

In a preliminary experiment on three medical imaging datasets (BraTS, Lung Mask, Kvasir-SEG), we measure Dice scores between ground-truth masks and saliency maps from ResNet-50, ConvNeXt-Tiny, and ViT-B/16 encoders pre-trained at fixed $\theta$, observing clear dataset-specific oscillations. We report a negative result: Histogram-of-Gradients (HoG) features do not explain the phenomenon. We find a fascinating and previously undocumented ``fingerprinting'' effect linking augmentation choices to data and architecture and a negative finding about a mechanistic explanation for it.

Submission Number: 69