Dynamic Scale for Transformer

Xiaoya Tang, Xiwen Li, Tolga Tasdizen

Keywords: Multi-scale, Dynamic CLS token, ECG Classification

TL;DR: We propose a dynamic CLS token that aggregates information from CLS tokens across all layers, each embedded with varying receptive fields.

Abstract: To enhance the hierarchical transformer with fixed embedding sizes, we propose a dynamic CLS token that aggregates information from CLS tokens across all layers, each embedded with varying receptive fields, by leveraging a squeeze-and-excitation module. This architecture offers a more flexible approach to utilizing multi-scale features in transformers. Code is available at: https://github.com/xiaoyatang/DynamicCLS.git.

Submission Number: 91