The principle of maximum entropy as a tool for unstructured data processing

Anton V. Kvasnov, Sergey V Kuleshov

Keywords: Maximum entropy, MaxEnt distribution, unstructured data processing, central tendency, supervised learning, image processing, Lebesgue measure

Abstract: The maximum entropy principle (MaxEnt) offers several advantages that make it suitable for use in unstructured data processing. However, finding the MaxEnt distribution requires solving an optimization problem using Lagrange multipliers with minimal prior data. While most studies rely on the well-known moment problem, we examine only first- and second-order moments. From the perspective of the calculus of variations, it has been shown that the maximum entropy distribution for a known first-order moment (the mathematical expectation) is a Gibbs distribution with an associated exponential function. For the second-order moment (the variance), MaxEnt is a convex function whose extremum region achieves the greatest information "saturation." This study demonstrates the effectiveness of this approach in digital image processing for identifying color contrast zones that most accurately capture silhouettes and object data. Knowledge of central moments provides additional information about texture, reproduced objects, and image elements. The effectiveness of the maximum entropy principle is demonstrated in applications to supervised learning. Further research focuses on generalizing the principle to f-entropy (specifically, with respect to moment problems for Rényi and Tsallis entropy), and on applied evaluation of the principle's effectiveness on time series in comparison with recurrent artificial neural network technologies.

Submission Number: 23