A cookbook for hardware-friendly implicit learning on static data

Maxence Ernoult, Rasmus Høier, Jack Kendall

Keywords: implicit differentiation, hardware, bilevel optimization, multilevel optimization, zeroth-order optimization, neuromorphic computing, energy-based models, Hopfield models, biologically plausible credit assignment

Abstract: The following aims to be a pragmatic introduction to hardware-friendly learning of implicit models, which encompass a broad class of models from feedforward nets to physical systems, taking static data as inputs. Starting from first principles, we present a minimal hierarchy of independent concepts to circumvent some problems inherent to the hardware implementation of standard differentiation. This way, we avoid entangling essential ingredients with arbitrary design choices by naively listing existing algorithms and instead propose the draft of a “cookbook” to help the exploration of many possible combinations of these independent mechanisms.

Submission Number: 54