Neo: A Learned Query Optimizer

Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil

Abstract: Query optimization is one of the most challenging problems indatabase systems. Despite the progress made over the past decades,query optimizers remain extremely complex components that re-quire a great deal of hand-tuning for specific workloads and datasets.Motivated by this shortcoming and inspired by recent advances inapplying machine learning to data management challenges, we in-troduceNeo(Neural Optimizer), a novel learning-based query op-timizer that relies on deep neural networks to generate query exe-cutions plans. Neo bootstraps its query optimization model fromexisting optimizers and continues to learn from incoming queries,building upon its successes and learning from its failures. Further-more, Neo naturally adapts to underlying data patterns and is robustto estimation errors. Experimental results demonstrate that Neo,even when bootstrapped from a simple optimizer like PostgreSQL,can learn a model that offers similar performance to state-of-the-artcommercial optimizers, and in some cases even surpass them.