Controllable Shadow Generation Using Pixel Height Maps

Yichen Sheng, Yifan liu, Jianming Zhang, Wei Yin, Cengiz Oztireli, HE Zhang, Zhe Lin, Eli Shechtman, Bedrich Benes

Abstract:

Shadows are essential for realistic image compositing from
2D image cutouts. Physics-based shadow rendering methods require 3D
geometries, which are not always available. Deep learning-based shadow
synthesis methods learn a mapping from the light information to an
object’s shadow without explicitly modeling the shadow geometry. Still,
they lack control and are prone to visual artifacts. We introduce “Pixel
Height”, a novel geometry representation that encodes the correlations
between objects, ground, and camera pose. The Pixel Height can be
calculated from 3D geometries, manually annotated on 2D images, and
can also be predicted from a single-view RGB image by a supervised
approach. It can be used to calculate hard shadows in a 2D image
based on the projective geometry, providing precise control of the shadows’
direction and shape. Furthermore, we propose a data-driven soft
shadow generator to apply softness to a hard shadow based on a softness
input parameter. Qualitative and quantitative evaluations demonstrate
that the proposed Pixel Height significantly improves the quality of the
shadow generation while allowing for controllability.