Principal Components for Neural Network Initialization: A Novel Approach to Explainability and Efficiency

Thu Nguyen, Thanh Nhan Phan, Uyen Dang, Michael Alexander Riegler, Pål Halvorsen

Keywords: intialization, XAI, explainability

Abstract:

Principal Component Analysis (PCA) is a commonly used tool for dimension reduction and denoising. Therefore, it is also widely used on the data prior to training a neural network. However, this approach can complicate the explanation of eXplainable Artificial Intelligence (XAI) methods for the decision of the model. In this work, we analyze the potential issues with this approach and propose Principal Components-based Initialization (PCsInit), a strategy to avoid this complexity by initializing the first layer of a neural network with the principal components. We will illustrate that when this first layer (which is initialized by the principal components) is frozen, PCsInit is equivalent to applying PCA to the input and then training on the principal components, while being simpler to explain. In addition, we propose two variants PCsInit-Act (to incorporate nonlinearity) and PCsInit-Sub (for a more scalable approach),
and show that the proposed techniques possess desirable theoretical properties. Moreover, as will be illustrated in the experiments, such training strategies can also allow further improvement of training via backpropagation compared to training neural networks on principal components.

Primary Area: interpretability and explainable AI

Submission Number: 8032