Rethinking Neural Operations for Diverse Tasks

Nicholas Carl Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Re, Ameet Talwalkar

Keywords: automated machine learning, neural architecture search, partial differential equations

TL;DR: A general-purpose search space for neural operations that work well on diverse domains, including PDE solving, protein folding, and music modeling.

Abstract: An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users to discover the right neural operations given data from their specific domain. We introduce a search space of operations called XD-Operations that mimic the inductive bias of standard multi-channel convolutions while being much more expressive: we prove that it includes many named operations across multiple application areas. Starting with any standard backbone such as ResNet, we show how to transform it into a search space over XD-operations and how to traverse the space using a simple weight sharing scheme. On a diverse set of tasks—solving PDEs, distance prediction for protein folding, and music modeling—our approach consistently yields models with lower error than baseline networks and often even lower error than expert-designed domain-specific approaches.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/nick11roberts/XD

Community Implementations: [ 9 code implementations](https://www.catalyzex.com/paper/arxiv:2103.15798/code)