Training Deep Morphological Neural Networks as Universal Approximators

TMLR Paper7769 Authors

Abstract: We investigate deep morphological neural networks (DMNNs), studying how changes in algebraic structure affect the expressiveness and trainability of deep architectures. We show that despite the inherent non-linearity of morphological operations, existing deep morphological architectures fail to be universal approximators and exhibit optimization limitations. To address these issues, we introduce architectures incorporating constrained "linear" activations between morphological layers. In the first two architectures, only $O(N)$ parameters (or learnable parameters) per layer of size $N$ belong to the activations, with the remaining parameters constrained to morphological operations. We prove universal approximation results for the proposed architectures and show empirically that they can be successfully trained on standard image classification tasks. Residual connections and weight dropout further improve generalization. Our experiments show that our networks are trainable, without requiring substantially larger parameter counts than comparable linear networks despite the imposed architectural restrictions. Finally, we propose a hybrid linear/morphological architecture and observe accelerated convergence of gradient descent under large batches.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Stanislaw_Kamil_Jastrzebski1

Submission Number: 7769