On the Conditioning Consistency Gap in Conditional Neural Processes

TMLR Paper7142 Authors

Abstract: Neural processes are meta-learning models that map context sets to predictive distributions. While inspired by stochastic processes, NPs do not generally satisfy the Kolmogorov consistency conditions required to define a valid stochastic process. This inconsistency is widely acknowledged but poorly understood. Practitioners note that NPs work well despite the violation, without quantifying what this means. We address this gap by defining the conditioning consistency gap, a KL divergence measuring how much a CNP's predictions change when a point is added to the context versus conditioned upon. Our main results show that for CNPs with bounded encoders and Lipschitz decoders, the consistency gap is $O(1/n^2)$ in context size $n$, and that this rate is tight. These bounds explain why CNPs behave approximately consistently for moderate context sizes while potentially exhibiting inconsistency in the few-shot regime.

Submission Type: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=9KJi1Z4bwQ

Changes Since Last Submission: fix formatting

Assigned Action Editor: ~Manuel_Haussmann1

Submission Number: 7142