Understanding encoder-decoder structures in machine learning using information measures

Jorge F. Silva, Victor Faraggi, Camilo Ramírez, Alvaro Egaña, Eduardo Pavez

Abstract: Highlights•A new theory of representation learning to understand encoder–decoder design.•Information sufficiency to model and characterize the predictive structures in learning.•Shannon’s information loss proposes to measure the encoder’s lack of expressiveness.•New results for universal cross-entropy learning.•On the appropriateness of digital encoders and information bottleneck for learning.