How Predictable is AI Progress?

David Mayo, Abdulrahman Alabdulkareem, Albert Eaton Shaw, Colin Conwell, Andrey Gizdov, Andrei Barbu, Boris Katz, Brian Cheung

Keywords: Deep learning

Abstract: Benchmarks only serve to measure what models are capable of now, not what they will be capable of in the future. We find that the ordering of acquired capabilities is remarkably consistent across large populations of AI models, which begs the question of whether one can forecast which specific examples and capabilities future models will solve next. We propose formalizing this problem into a new evaluation task called progress prediction: Can we forecast which unsolved problems will be solved next as future models improve? We find that progress follows a structured pattern, a sign of potential predictability. Through an empirical study of hundreds of millions of predictions made by 1,000+ vision models and 1,500+ language models, we find that this predictability may be possible due to the consistent order in which models acquire capabilities across architectures, datasets, and modalities.

Submission Number: 104