Universal Learning of Individual Data

Yaniv Fogel, Meir Feder

Abstract: Universal supervised learning of individual data is considered from an information theoretic point of view in the standard supervised “batch” learning where prediction is done on a test sample once the entire training data is observed. In this individual setting the features and labels, both in the training and the test, are specific individual, deterministic quantities. Prediction loss is naturally measured by the log-loss. The presented results provide a minimax universal learning scheme, termed the Predictive Normalized Maximum Likelihood (pNML) that competes with a “genie” (or reference) that knows the true test label. In addition, a pointwise learnability measure associated with the pNML, for the specific training and test, is provided. This measure may also indicate the performance of the commonly used Empirical Risk Minimizer (ERM) learner.