ACCORD: Autoregressive Constraint-satisfying Generation for COmbinatorial Optimization with Routing and Dynamic attention

Henrik Abgaryan, Ararat Harutyunyan, Tristan Cazenave

Keywords: Large Language Models, Combinatorial Optimization, NP-hard Problems, Autoregressive Generation, Constraint Satisfaction, Dynamic Attention, LoRA, ACCORD-90k, TSP, VRP, Knapsack, FlowShop, JSSP, BinPacking

TL;DR: ACCORD uses LLMs with dynamic constraints and attention to solve NP-hard optimization tasks, leveraging a 90k-example dataset across 6 problems, and outperforms larger models like GPT-4.

Abstract: Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, yet their direct application to NP-hard combinatorial problems (CPs) remains underexplored. In this work, we systematically investigate the reasoning abilities of LLMs on a variety of NP-hard combinatorial optimization tasks and introduce \textbf{ACCORD}: \textbf{A}utoregressive \textbf{C}onstraint-satisfying generation for \textbf{CO}mbinatorial optimization with \textbf{R}outing and \textbf{D}ynamic attention. ACCORD features a novel dataset representation and model architecture that leverage the autoregressive nature of LLMs to dynamically enforce feasibility constraints, coupled with attention-based routing to activate problem-specific LoRA modules. We also present the ACCORD-90k supervised dataset, covering six NP-hard combinatorial problems: TSP, VRP, Knapsack, FlowShop, JSSP, and BinPacking. Extensive experiments demonstrate that our ACCORD model, built on an 8B-parameter Llama backbone, consistently outperforms standard prompting and input-output methods, even when compared to much larger LLMs, such as gpt-4. Ablation studies further show that our output structure enhances solution feasibility. To the best of our knowledge, this is the first large-scale, end-to-end framework for exploring the applications of LLMs to a broad spectrum of combinatorial optimization problems. The codes are publicly available at \footnote{https://github.com/starjob42/ACCORD}

Submission Number: 23