A Pipeline for Transparency Estimation of Glass and Plastic Bottle Images for Neural Scanning

Floris Erich, Jerome Susgin, Noriaki Ando, Yusuke Yoshiyasu

Abstract: Reduction of the gap between simulation and reality (sim2real gap) is essential for robots to learn how to manipulate objects in real scenarios. Estimation of an alpha value for transparent pixels is necessary to render novel views of common objects such as bottles and cups. While it is straightforward to estimate an alpha value for transparent objects, many practical objects have a mixture of transparent areas and opaque areas. In this paper we present a pipeline for automatically segmenting bottles into label, cap and body and estimating an alpha value for the body. We train a segmentation network (Detectron2) for the task of transparent object segmentation, based on a bottle dataset distilled from the PACO dataset. We combine the segmentation masks into a trimap, which is then used as input for an off-the-shelf matting deep neural network (ViTMatte). In our experiments, we show that the per-pixel error for transparent pixels can be reduced by 44% using our pipeline, compared to the baseline of not applying transparency estimation.