A Thorough Examination of Decoding Methods in the Era of LLMs

Anonymous

Abstract:

Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. Prior research on decoding methods, primarily focusing on task-specific models, may not extend to the current era of general-purpose large language models (LLMs). Moreover, the recent influx of decoding strategies has further complicated this landscape.
This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of LLMs, evaluating their performance, robustness to hyperparameter changes, and decoding speeds across a wide range of tasks, models, and deployment environments. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization. Intriguingly, sensitivity analysis exposes that certain methods achieve superior performance at the cost of extensive hyperparameter tuning, highlighting the trade-off between attaining optimal results and the practicality of implementation in varying contexts.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment

Languages Studied: English

Preprint Status: There is a non-anonymous preprint (URL specified in the next question).

A1: yes

A1 Elaboration For Yes Or No: Limitation section

A2: n/a

A3: yes

A3 Elaboration For Yes Or No: section abstract and introduction

B: yes

B1: yes

B1 Elaboration For Yes Or No: Experimental settings

B2: n/a

B3: n/a

B4: n/a

B5: n/a

B6: yes

B6 Elaboration For Yes Or No: Experimental settings and Appendix

C: yes

C1: yes

C1 Elaboration For Yes Or No: Experimental settings

C2: yes

C2 Elaboration For Yes Or No: Experimental settings and results and Appendix

C3: yes

C3 Elaboration For Yes Or No: Results and Analysis

C4: yes

C4 Elaboration For Yes Or No: Experimental settings

D: no

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: no

E1: n/a