Compute-efficient and backpropagation-free pseudoinverse learning for neural networks: A comprehensive survey

Mohammed A. B. Mahmoud

Abstract:

The pseudoinverse learning algorithm is a non-gradient, efficient learning scheme originally designed for
training single hidden layer feedforward neural networks. It has been developed into various variants and
successfully applied across numerous fields. This paper provides a systematic review of the fundamental
theories of the pseudoinverse learning algorithm and its major variants, outlining different types of neural
networks and learning system architectures based on the pseudoinverse learning scheme. Furthermore, we
summarize the fundamental ideas and methodologies of applying the pseudoinverse learning scheme to various
learning tasks such as classification, representation learning, time series forecasting, incremental learning,
automated machine learning, and content generation. We also summarize and compare the performance of
pseudoinverse learning with representative competing baselines on several commonly used data sets based on
existing literature reports. The results demonstrate that PIL exhibits significant efficiency advantages over
gradient-based approaches (training time was reduced by 72.73% to 99.37%), aligning with its inherent
gradient-free nature. Notably, recent PIL variants maintain this computational superiority while achieving
enhanced performance compared to other gradient-free algorithms. In addition, we briefly introduce the
representative applications of pseudoinverse learning in various fields. To the best of our knowledge, this is the
first comprehensive review in this field to encompass all aforementioned aspects. It facilitates the synthesis and
integration of existing knowledge from disparate studies. By highlighting limitations in prior works including
the computational complexity in large-scale pseudoinverse computation, potential numerical instability for
ill-conditioned matrices, risk of overfitting, and constraints in modeling multidimensional patterns, this paper
also recommends directions for future research in this area.