Tensor Product Neural Networks for Functional ANOVA Model

Seokhun Park, Insung Kong, yongchan Choi, Chanmoo Park, Yongdai Kim

TL;DR: In this paper, we propose a novel neural network which guarantees a unique functional ANOVA decomposition and thus is able to estimate each component stably and accurately.

Abstract:

Interpretability for machine learning models is becoming more and more important as machine learning models become more complex.
The functional ANOVA model, which decomposes a high-dimensional function into a sum of lower dimensional functions (commonly referred to as components), is one of the most popular tools for interpretable AI, and recently, various neural networks have been developed for estimating each component in the functional ANOVA model.
However, such neural networks are highly unstable when estimating each component since the components themselves are not uniquely defined.
That is, there are multiple functional ANOVA decompositions for a given function.
In this paper, we propose a novel neural network which guarantees a unique functional ANOVA decomposition and thus is able to estimate each component stably.
We call our proposed neural network ANOVA Tensor Product Neural Network (ANOVA-TPNN) since
it is motivated by the tensor product basis expansion.
Theoretically, we prove that ANOVA-TPNN can approximate any smooth function well.
Empirically, we show that ANOVA-TPNN provide much more stable estimation of each component and thus much more stable interpretation when training data and initial values of the model parameters vary than existing neural networks do.
Our source code is released at https://github.com/ParkSeokhun/ANOVA-TPNN

Lay Summary:

As machine learning models become more complex, it is becoming increasingly important to understand how they work.
One common approach to making these models more interpretable is to break them down into smaller, more understandable parts.
A widely used method for doing this is called functional ANOVA, which analyzes the effect of each interaction separately by decomposing the model into components.

However, existing AI models that implement this method often produce unstable and inconsistent results.
This is because there are many ways to perform such a decomposition, and traditional models cannot guarantee a unique or consistent breakdown of the model.

To address this issue, we developed a new AI model called ANOVA-TPNN.
This model ensures the uniqueness of decomposition, so that the effect of each component can be reliably understood.
We have proven mathematically that ANOVA-TPNN can represent a wide range of smooth functions, and our experiments show that it provides much more stable and trustworthy results compared to existing models.

Link To Code: https://github.com/ParkSeokhun/ANOVA-TPNN

Primary Area: Social Aspects->Accountability, Transparency, and Interpretability

Keywords: Interpretability, Trustworthy AI, Functional ANOVA model, Generalized additive models, Tensor product neural network

Submission Number: 13951