CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming

Vignesh V. Menon, Hadi Amirpour, Mohammed Ghanbari, Christian Timmerer

Abstract: Ultra High Definition Television (UHDTV) offers a better immersive audiovisual experience than HDTV by improving the aesthetic sense of the content [1]. How-ever, it may lead to an increase of both encoding time complexity and compression artifacts at lower bitrates. To address this challenge, a low-latency pre-processing algorithm named COntent-aware frame Dropping Algorithm (CODA) is proposed to predict the optimized framerate per video segment in streaming scenarios. The optimized framerate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\hat{f})$</tex> for every video segment at each target bitrate is modelled as an exponential decay (increasing) function whose decay rate is directly proportional to the temporal characteristics <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(h)$</tex> [2] [3] of the video and the target bitrate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(b)$</tex> , and inversely proportional to the spatial characteristics <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(E)$</tex> of the video. The encoding is carried out with the predicted framerate, saving encoding time and improving visual quality at lower bitrates. At the decoder side, the video is upscaled in the temporal domain to the original framerate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(f_{max})$</tex> for display.