Synthesis of Environment Maps for Mixed Reality

David R. Walton, Diego Thomas, Anthony Steed, Akihiro Sugimoto

Abstract:

When rendering virtual objects in a mixed reality application, it
is helpful to have access to an environment map that captures the
appearance of the scene from the perspective of the virtual object.
It is straightforward to render virtual objects into such maps, but
capturing and correctly rendering the real components of the scene
into the map is much more challenging. This information is often
recovered from physical light probes, such as reflective spheres or
fisheye cameras, placed at the location of the virtual object in the
scene. For many application areas, however, real light probes would
be intrusive or impractical.
Ideally, all of the information necessary to produce detailed environment
maps could be captured using a single device. We introduce
a method using an RGBD camera and a small fisheye camera,
contained in a single unit, to create environment maps at any location
in an indoor scene. The method combines the output from
both cameras to correct for their limited field of view and the displacement
from the virtual object, producing complete environment
maps suitable for rendering the virtual content in real time. Our
method improves on previous probeless approaches by its ability to
recover high-frequency environment maps. We demonstrate how
this can be used to render virtual objects which shadow, reflect and
refract their environment convincingly.