Go to ICLR 2024 Workshop LLMAgents homepageExpressing and Exploiting Parallelism in Language Model Decoding
For autoregressive language models, decoding naturally occurs sequentially, generating tokens one after another. Recent attempts to introduce parallelism require a pre-determined structure to implement parallel generation, such as generating an outline and dividing the responses into parallel sub-tasks. In this work we explore a new technique to automate parallel generation by dynamically exploiting various parallel structure in the semantics of the language model response. Specifically, we introduce a simple annotation language MSG that allows language models to express parallelism in their outputs. We then develop an interpreter for MSG that performs on-the-fly parallel generation during decoding, exploiting the parallelism expressed in the MSG-annotated outputs. We demonstrate that our approach can improve tokens generated per second by 21% while maintaining the same quality of output.