Attention Beats Concatenation for Conditioning Neural Fields
Abstract: Neural fields model signals by mapping coordinate inputs to sampled values. They are becoming an increasingly important backbone architecture across many fields from vision and graphics to biology and astronomy. In this paper, we explore the differences between common conditioning mechanisms within these networks, an essential ingredient in shifting neural fields from memorization of signals to generalization, where the set of signals lying on a manifold is modelled jointly. In particular, we are interested in the scaling behaviour of these mechanisms to increasingly high-dimensional conditioning variables. As we show in our experiments, high-dimensional conditioning is key to modelling complex data distributions, thus it is important to determine what architecture choices best enable this when working on such problems. To this end, we run experiments modelling 2D, 3D, and 4D signals with neural fields, employing concatenation, hyper-network, and attention-based conditioning strategies -- a necessary but laborious effort that has not been performed in the literature.
We find that attention-based conditioning outperforms other approaches in a variety of settings.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: - Added an additional section to the appendix with further architectural details.
- Added clarification about use of token-level mapping in auto-decoders.
- Minor grammar fixes.
Assigned Action Editor: ~Zhe_Gan1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 455
Loading