Does local pruning offer task-specific models to learn effectively ?

Abhishek Kumar Mishra

Abstract:

The need to deploy large-scale pre-trained
models on edge devices under limited com-
putational resources has led to substantial re-
search to compress these large models. How-
ever, less attention has been given to com-
press the task-specific models. In this work,
we investigate the different methods of un-
structured pruning on task-specific models
for Aspect-based Sentiment Analysis (ABSA)
tasks. Specifically, we analyze differences in
the learning dynamics of pruned models by us-
ing the standard pruning techniques to achieve
high-performing sparse networks. We develop
a hypothesis to demonstrate the effectiveness
of local pruning over global pruning consider-
ing a simple CNN model. Later, we utilize the
hypothesis to demonstrate the efficacy of the
pruned state-of-the-art model compared to the
over-parameterized state-of-the-art model un-
der two settings, the first considering the base-
lines for the same task used for generating the
hypothesis, i.e., aspect extraction and the sec-
ond considering a different task, i.e., sentiment
analysis. We also provide discussion related to
the generalization of the pruning hypothesis.