Regression with Deep Neural Networks: Generalization Error Guarantees, Learning Algorithms, and Regularizers

Jaweria Amjad, Zhaoyan Lyu, Miguel R. D. Rodrigues

Abstract: We present new data-dependent characterizations of the generalization capability of deep neural networks based data representations within the context of regression tasks. In particular, we propose new generalization error bounds that depend on various elements associated with the learning problem such as the complexity of the data space, the cardinality of the training set, and the input-output Jacobian of the deep neural network. Moreover, building upon our bounds, we propose new regularization strategies constraining the network Lipschitz properties through norms of the network gradient. Experimental results show that our newly proposed regularization techniques can deliver state-of-the-art performance in comparison to established weight-based regularization.