A Quantum Annealing Instance Selection Approach for Efficient and Effective Transformer Fine-Tuning

Andrea Pasin, Washington Cunha, Marcos André Gonçalves, Nicola Ferro

Keywords: Instance Selection, Quantum Computing, Quantum Annealing, Deep Learning, Text Classification, Transformers

TL;DR: We introduce a novel Quantum Annealing (QA) Instance Selection (IS) approach, reducing training set size, maintaining effectiveness, and speeding up training. Our solution is the first to apply QA to the IS problem and offers a new QUBO formulation.

Abstract: Deep Learning approaches have become pervasive in recent years. In fact, they allow for solving tasks that were thought to be too complex a few decades ago, sometimes with superhuman effectiveness. However, these models require huge datasets to be properly trained and to provide a good generalization. This translates into high training and fine-tuning time, even several days for the most complex models and large datasets. In this work, we present a novel quantum IS approach that allows to significantly reduce the size of the training datasets (by up to 28%) while maintaining the model's effectiveness, thus promoting (training) speedups and scalability. Our solution is innovative in the sense that it exploits a different computing paradigm -- QA -- a specific Quantum Computing paradigm that can be used to tackle practical optimization problems. To the best of our knowledge, there have been no prior attempts to tackle the IS problem using QA. Furthermore, we propose a new QUBO formulation specific for the IS problem, which is a contribution in itself. Through an extensive set of experiments with several ATC benchmarks, we empirically demonstrate both the feasibility of our quantum solution and its competitiveness with the current state-of-the-art IS solutions.

Submission Number: 19