Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options

Lakshmi Nair, Ian Trase, J. Mark Kim

Abstract: We present a novel reasoning approach called Flow-of-Options (FoO), designed to address intrinsic biases in Large Language Models (LLMs). Flow-of-Options enables LLMs to systematically explore a diverse range of possibilities in their reasoning, as demonstrated by an FoO-based agentic framework developed for autonomously solving Machine Learning (ML) tasks. FoO enforces diversity in LLM solutions through compressed and interpretable task representations, resulting in improvements of 38.2% - 69.2% on standard data science tasks, and 37.4% - 47.9% on therapeutic chemistry tasks, as compared to state-of-the-art baselines. With an overall operation cost under $1 per task, our framework is well-suited for cost-sensitive applications. Going beyond tabular classification and regression, we show the broader applicability of our FoO-based agentic system to tasks such as reinforcement learning and image generation. Our code is open-sourced at: https://github.com/flagshippioneering/Flow-of-Options.

Lay Summary: Large Language Models (LLMs) often default to familiar patterns from their training data, limiting their ability to explore diverse or creative solutions. To tackle this limitation, we developed a new strategy called Flow-of-Options (FoO), that pushes LLMs to systematically consider multiple alternative approaches before deciding how to proceed. FoO explicitly represents each task as a network of possible options, encouraging the model to thoroughly explore different solutions rather than defaulting to biased or familiar paths. We demonstrate FoO’s practical benefits through extensive experiments on standard machine learning (ML) tasks in data science and therapeutic chemistry. Our results show that FoO significantly outperforms existing systems — improving performance by up to 69.2%, while maintaining low operational costs. We also show that FoO is versatile and can extend to a broad range of ML problems, making FoO a practical tool for enhancing LLM-based ML automation.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/flagshippioneering/Flow-of-Options

Primary Area: Deep Learning->Large Language Models

Keywords: LLM Agents, AutoML, LLM reasoning

Submission Number: 12168