Characterizing Probabilistic Structure in Learning Using Information Sufficiency

Victor Faraggi, Jorge F. Silva, Camilo Ramírez, Alvaro Egaña, Eduardo Pavez

Abstract: We use the concept of information sufficiency (IS) to represent probabilistic structures in machine learning (ML). Our main result provides a functional expression that charac-terizes the class of probabilistic models consistent with an IS encoder-decoder latent predictive structure. This result formally justifies the encoder-decoder forward stages many modern ML architectures adopt to learn latent (compressed) representations in data. To illustrate IS as a realistic and rele-vant model assumption, we revisit some known ML concepts and present some interesting new examples: invariant, robust, sparse, and digital models.