Expressive yet Efficient Feature Expansion with Adaptive Cross-Hadamard Products

Xuyang Zhang, Xi Zhang, Liang Chen, Hao Shi, Qingshan Guo

Keywords: Efficient Deep Learning, Neural Architecture Search, Hadamard-based Operators, Feature Expansion and Reuse

TL;DR: We propose the Adaptive Cross-Hadamard (ACH) module to enable efficient, non-linear feature expansion via learnable Hadamard products and dynamic softsign normalization.

Abstract: Recent theoretical advances reveal that the Hadamard product induces nonlinear representations and implicit high-dimensional mappings for the field of deep learning, yet their practical deployment in resource-constrained vision models remains largely unexplored. To address this gap, we introduce the Adaptive Cross-Hadamard (ACH) module, a novel operator that embeds learnability through differentiable discrete sampling and dynamic softsign normalization. This facilitates highly efficient feature reuse without incurring additional convolutional parameters, while ensuring stable gradient flow. Integrated into Hadaptive-Net (Hadamard Adaptive Network) via neural architecture search, our approach achieves unprecedented efficiency. Comprehensive experiments demonstrate state-of-the-art accuracy/speed trade-offs on image classification tasks, establishing Hadamard operations as specific building blocks for efficient vision models.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 6821