LAuReL: Learned Augmented Residual Layer
Keywords: model efficiency, transformer, residual connection, skip connection, llm, efficient deep learning, efficient machine learning
TL;DR: We introduce a novel alternative to the residual connection called 'LAuReL', which augments the residual connection with learnable light-weight improvements, outperforming the baseline on non-trivial vision and language tasks..
Abstract: One of the core pillars of efficient deep learning methods is architectural improvements such as the residual/skip connection (He et al.), which has led to significantly better model convergence and quality. Since then the residual connection has become ubiquitous in not just convolutional neural networks but also transformer-based architectures, the backbone of LLMs.
In this paper we introduce a Learned Augmented Residual Layer (LAUREL)—a novel generalization of the canonical residual connection—with the goal to be an in-situ replacement of the latter while outperforming on both model quality and footprint metrics. Our experiments show that using LAUREL can help boost performance for both vision and language models. For example, on the ResNet-50, ImageNet 1K task, it achieves 60% of the gains from adding an extra layer, while only adding 0.003% more parameters, and matches it while adding 2.6x fewer parameters
Submission Number: 56
Loading