SONG: Self-Organizing Neural Graphs

Łukasz Struski, Tomasz Danel, Marek Śmieja, Jacek Tabor, Bartosz Michał Zieliński

Keywords: decision graphs, self-organizing models, neural networks, Markov processes, decision trees, interpretability

Abstract: Recent years have seen a surge in research on combining deep neural networks with other methods, including decision trees and graphs. There are at least three advantages of incorporating decision trees and graphs: they are easy to interpret since they are based on sequential decisions, they can make decisions faster, and they provide a hierarchy of classes. However, one of the well-known drawbacks of decision trees, as compared to decision graphs, is that decision trees cannot reuse the decision nodes. Nevertheless, decision graphs were not commonly used in deep learning due to the lack of efficient gradient-based training techniques. In this paper, we fill this gap and provide a general paradigm based on Markov processes, which allows for efficient training of the special type of decision graphs, which we call Self-Organizing Neural Graphs (SONG). We provide an extensive theoretical study of SONG, complemented by experiments conducted on Letter, Connect4, MNIST, CIFAR, and TinyImageNet datasets, showing that our method performs on par or better than existing decision models.

One-sentence Summary: We introduce Self-Organizing Neural Graphs (SONGs), a new paradigm of end-to-end decision graph training based on Markov processes that simultaneously learn the optimal graph structure and transition probabilities.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2107.13214/code)