InFlux: A Benchmark for Self-Calibration of Dynamic Intrinsics of Video Cameras

Erich Liang, Roma Bhattacharjee, Sreemanti Dey, Rafael Moschopoulos, Caitlin Wang, Michel Liao, Grace Tan, Andrew Wang, Karhan Kayan, Stamatis Alexandropoulos, Jia Deng

Keywords: Camera intrinsics, dynamic camera intrinsics, video intrinsics, camera calibration, large FOV calibration, intrinsics estimation, per-frame calibration, lens metadata, computer vision, 3D vision, 3D reconstruction, SfM, cameras, drones, Blender, real-world, dataset, benchmark, evaluation

TL;DR: InFlux is the first real-world benchmark that provides per-frame ground truth camera intrinsics for videos with dynamic intrinsics, and current baselines struggle to predict accurate intrinsics on our benchmark.

Abstract: Accurately tracking camera intrinsics is crucial for achieving 3D understanding from 2D video. However, most 3D algorithms assume that camera intrinsics stay constant throughout a video, which is often not true for many real-world in-the-wild videos. A major obstacle in this field is a lack of dynamic camera intrinsics benchmarks--existing benchmarks typically offer limited diversity in scene content and intrinsics variation, and none provide per-frame intrinsic changes for consecutive video frames. In this paper, we present Intrinsics in Flux (InFlux), a real-world benchmark that provides per-frame ground truth intrinsics annotations for videos with dynamic intrinsics. Compared to prior benchmarks, InFlux captures a wider range of intrinsic variations and scene diversity, featuring 143K+ annotated frames from 386 high-resolution indoor and outdoor videos with dynamic camera intrinsics. To ensure accurate per-frame intrinsics, we build a comprehensive lookup table of calibration experiments and extend the Kalibr toolbox to improve its accuracy and robustness. Using our benchmark, we evaluate existing baseline methods for predicting camera intrinsics and find that most struggle to achieve accurate predictions on videos with dynamic intrinsics. For the dataset, code, videos, and submission, please visit https://influx.cs.princeton.edu/.

Dataset URL: https://huggingface.co/datasets/princeton-vl/InFlux

Code URL: https://github.com/princeton-vl/InFlux/tree/main

Primary Area: Datasets & Benchmarks for applications in computer vision

Submission Number: 1071