Adaptive integer-precision Lagrange multiplier selection for high performance AVS video coding

Hai Bing Yin, Bingqian Zhou, Chuang Zhu, Huizhu Jia

Abstract: In AVS and H.264/AVC, Lagrangian Rate distortion (RD) optimization techniques are widely adopted for coding mode selection and displacement vector estimation. The optimal Lagrange multipliers in these two cases are both floating-point values. If RD optimized video encoder is implemented on computation-constrained fixed-point platform such as FPGA and ASIC, fixed-point Lagrange multiplier selection is an important problem to trade-off the RD performance and computation complexity. This work focuses on fixed-point Lagrange multiplier selection for RD mode decision. Adaptive scaling matrix is used to trade-off complexity and RD performance. Also, intensive simulation results and analysis on precision, hardware cost, and RD performance are given. The proposed approach is also well-suited for RD optimized motion estimation for computation-constrained video coding.