Embeddings for Numerical Features Using tanh Activation

Bingyan Liu, Charles Elkan, Anil N. Hirani

Keywords: Tree-based inductive bias, Tanh activation, Numerical features embedding, Tabular data, Neural networks, Neural network initialization

TL;DR: A tanh-based embedding method and corresponding neural network initialization for numerical features in tabular data, leveraging tanh's ability to mimic both linear and decision tree-like behaviors.

Abstract: Recent advances in tabular deep learning have demonstrated the importance of embeddings for numerical features, where scalar values are mapped to high-dimensional spaces before being processed by the main model. Here, we propose an embedding method using the hyperbolic tangent (tanh) activation function that allows neural networks to achieve better accuracy on tabular data via an inductive bias similar to that of decision trees. To make training with the new embedding method reliable and efficient, we additionally propose a principled initialization method. Experiments demonstrate that the new approach improves upon or matches accuracy results from previously proposed embedding methods across multiple tabular datasets and model architectures.

Include In Proceedings: Yes

Submission Number: 26