A Theoretical and Experimental Study for Dependent Learned Rate-Distortion Optimization

Yingwen Zhang, Meng Wang, Junru Li, Kai Zhang, Li Zhang, Shiqi Wang

Abstract: Recent advancements in learned rate-distortion optimization (RDO) showcase that by making the intra coding decisions based on a learned measure, the encoding can be significantly accelerated without incurring much coding loss. Despite great progress in complexity reduction, the dependency issue has been largely neglected in the current learned RDO research. In this study, aiming to tap the full potential of dependent learned RDO, we first derive a probabilistic RDO framework for theoretical analysis, under which the classic and the learned RDO problems are equivalent to the maximum a posteriori (MAP) inference and the distribution imitation, respectively. Subsequently, we probabilistically revisit dependency considerations in the intra RDO research. Our key finding is that the existing learned RDO scheme can only produce a measure that indicates the local “goodness” of coding decisions. We therefore further discuss the opportunities for learning a dependent measure that is more optimal in the long run. Finally, as learning an accurate measure for the full decision space could be extremely challenging, taking the High Efficiency Video Coding (HEVC) intra coding as a case study, we experimentally identify that the prediction decision accounts for the majority of the dependent optimization gain and is of the utmost value to be learned, paving the way for future research on dependent learned RDO.

External IDs:doi:10.1109/tcsvt.2025.3555152