On the Concept of Resource-Efficiency in NLP

Luise Dürlich, Evangelia Gogoulou, Joakim Nivre

Keywords: resource-efficiency, natural language processing, amortized efficiency

TL;DR: We discuss different conceptualizations of resource-efficiency in NLP, propose a novel notion of amortized efficiency, and illustrate different conceptualizations in a case study on fine-tuning and knowledge distillation.

Abstract: Resource-efficiency is a growing concern in the NLP community. But what are the resources we care about and why? How do we measure efficiency in a way that is reliable and relevant? And how do we balance efficiency and other important concerns? Based on a review of the emerging literature on the subject, we discuss different ways of conceptualizing efficiency in terms of product and cost, using a simple case study on fine-tuning and knowledge distillation for illustration. We propose a novel metric of amortized efficiency that is better suited for life-cycle analysis than existing metrics.

Student Paper: Yes, the first author is a student