Dictionary Contrastive Learning for Efficient Local Supervision without Auxiliary Networks

Suhwan Choi, Myeongho Jeon, Yeonjung Hwang, Jeonglyul Oh, Sungjun Lim, Joonseok Lee, Myungjoo Kang

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Contrastive learning, Forward learning, Local learning, Image classification, Efficient learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a simple and efficient local contrastive learning objective that directly compares local features with label embeddings.

Abstract:

While backpropagation (BP) has achieved widespread success in deep learning, it
faces two prominent challenges: computational inefficiency and biological implausibility.
In response to these challenges, local supervision, encompassing Local
Learning (LL) and Forward Learning (FL), has emerged as a promising research
direction. LL employs module-wise BP to achieve competitive results yet relies on
module-wise auxiliary networks, which increase memory and parameter demands.
Conversely, FL updates layer weights without BP and auxiliary networks but falls
short of BP’s performance. This paper proposes a simple yet effective objective
within a contrastive learning framework for local supervision without auxiliary
networks. Given the insight that the existing contrastive learning framework for
local supervision is susceptible to task-irrelevant information without auxiliary
networks, we present DICTIONARY CONTRASTIVE LEARNING (DCL) that optimizes
the similarity between local features and label embeddings. Our method
using static label embeddings yields substantial performance improvements in the
FL scenario, outperforming state-of-the-art FL approaches. Moreover, our method
using adaptive label embeddings closely approaches the performance achieved by
LL while achieving superior memory and parameter efficiency.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 4912