PeRFception: Perception using Radiance Fields

Yoonwoo Jeong, Seungjoo Shin, Junha Lee, Chris Choy, Anima Anandkumar, Minsu Cho, Jaesik Park

Keywords: dataset, neural radiance field, image classification, 3D shape classification, 3D semantic segmentation

TL;DR: We propose a new dataset, PeRFception dataset, that is a new unified radiance field dataset for the 2D image classification, 3D shape classification, and 3D semantic segmentation.

Abstract: The recent progress in implicit 3D representation, i.e., Neural Radiance Fields (NeRFs), has made accurate and photorealistic 3D reconstruction possible in a differentiable manner. This new representation can effectively convey the information of hundreds of high-resolution images in one compact format and allows photorealistic synthesis of novel views. In this work, using the variant of NeRF called Plenoxels, we create the first large-scale radiance fields datasets for perception tasks, called the PeRFception, which consists of two parts that incorporate both object-centric and scene-centric scans for classification and segmentation. It shows a significant memory compression rate (96.4\%) from the original dataset, while containing both 2D and 3D information in a unified form. We construct the classification and segmentation models that directly take this radiance fields format as input and also propose a novel augmentation technique to avoid overfitting on backgrounds of images. The code and data are publicly available in "https://postech-cvlab.github.io/PeRFception/".

Author Statement: Yes

URL: https://postech-cvlab.github.io/PeRFception/

Dataset Url:

PeRFception-CO3D - Data Chunks 1: https://1drv.ms/u/s!As9A9EbDsoWcbnHoOoqWmIB6RLs?e=SYGC03
PeRFception-CO3D - Data Chunks 2: https://1drv.ms/u/s!AgY2evoYo6FgiwomlG1QUiLg7wqy?e=ReG5Yp
PeRFception-ScanNet: https://1drv.ms/u/s!AgY2evoYo6FghYVw3MLYwq743fsoUw?e=ylF8KX

Dataset Embargo:

We have publicly released the data and benchmarked models in our project page and GitHub.
- Project Page: https://postech-cvlab.github.io/PeRFception/
- GitHub: https://github.com/POSTECH-CVLab/PeRFception

License: The PeRFception datasets are released under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license\footnote{\url{https://creativecommons.org/licenses/by-sa/4.0/}}, which allows anybody to use them for remix, transform, or build upon the material.

Contribution Process Agreement: Yes

In Person Attendance: Yes

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2208.11537/code)