syftr: Pareto-Optimal Generative AI

Alexander Conway, Debadeepta Dey, Stefan Hackmann, Matthew Hausknecht, Michael Douglas Schmidt, Mark Lewis Steadman, Nick Volynets

Confirmation: our paper adheres to reproducibility best practices. In particular, we confirm that all important details required to reproduce results are described in the paper,, the authors agree to the paper being made available online through OpenReview under a CC-BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), and, the authors have read and commit to adhering to the AutoML 2025 Code of Conduct (https://2025.automl.cc/code-of-conduct/).

TL;DR: syftr efficiently optimizes RAG pipelines for accuracy, latency, and cost using Bayesian Optimization.

Abstract:

Retrieval-Augmented Generation (RAG) pipelines are central to applying large language
models (LLMs) to proprietary or dynamic data. However, building effective RAG flows
is complex, requiring careful selection among vector databases, embedding models, text
splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic
paradigms. Modules like verifiers, rewriters, and rerankers—each with intricate hyperparam-
eter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy,
and cost becomes increasingly difficult in performance-sensitive applications.

We introduce syftr, a framework that performs efficient multi-objective search over a broad
space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr
discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early-
stopping mechanism further improves efficiency by pruning clearly suboptimal candidates.
Across multiple RAG benchmarks, syftr finds flows which are on average≈9×cheaper
while preserving most of the accuracy of the most accurate flows on the Pareto-frontier.
Furthermore, syftr’s ability to design and optimize also allows integrating new modules,
making it even easier and faster to realize high-performing generative AI pipelines. syftr is fully open source: https://github.com/datarobot/syftr

Submission Number: 39