A Critical Assessment of LLMs for Solving Multi-step Problems: Preliminary Results

Vincent Hsiao, Morgan Fine-Morris, Mak Roberts, Leslie N Smith, Laura M. Hiatt

Keywords: Hierarchical Task Network, Hybrid LLM systems, Agentic LLM systems, Planning

Abstract: Large Language Models (LLMs) have shown impressive performance on various language processing tasks, but often struggle with complex, multi-step tasks such as travel planning. To address this challenge, extensions like LLM-modulo systems and agentic approaches have been proposed, each with its own strengths and limitations. This paper examines the unique strengths and limitations of these approaches, using the Travel Planner benchmark as a case study. We analyze the results and propose a new hybrid task planner approach to address the challenges of solving multi-step tasks with LLMs, highlighting implications for future research in this area.

Submission Number: 10