Task dependent model complexity: shallow vs deep network

Abir Mohammad Hadi, Kwanghee Won, Sung Shin

Abstract: Deeper network models have opened a world unknown to many before. With the help of deeper networks, high dimensional data problems are solved. Yet the need for compact networks is growing as the end devices are still trying to harness the computational power in a small form factor. With the current state, finding the optimal architecture for a problem is one of the primary concern for edge deployment. Deeper networks come with immense power but a vast amount of it remain unused for simple problems. In this study we tried to show whether task dependencies have any correlation with the need of deeper or shallower network and how the network compression behaves with the evolving parameters. For the task dependency, we tried to restrict a DCNN to recognize a fixed scale feature and discard the others. On the way of building the experiments we have seen some intriguing phenomena such as how and where the filters responsible for recognizing a certain scale of the dataset grows. We used unstructured pruning without retraining for the deeper networks that uniformly discarded the filters having rank lower than a set threshold. All the experiments were performed on a binary classifier and the best result from a deeper network is 97.31% while the accuracy of the shallow network is 94.59% on the scale dependent task with a computable parameter ratio of 119:1.