Deepfovea: neural reconstruction for foveated rendering and video compression using learned natural video statistics

Anton Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, Gizem Rufo

Abstract: Recent advances in head-mounted displays (HMDs) provide new levels of immersion by delivering imagery straight to human eyes. The high spatial and temporal resolution requirements of these displays pose a tremendous challenge for real-time rendering and video compression. Since the eyes rapidly decrease in spatial acuity with increasing eccentricity, providing high resolution to peripheral vision is unnecessary. Upcoming VR displays provide real-time estimation of gaze, enabling gaze-contingent rendering and compression methods that take advantage of this acuity falloff. In this setting, special care must be given to avoid visible artifacts such as a loss of contrast or addition of flicker.