Unified 2D-3D Discrete Priors for Noise-Robust and Calibration-Free Multiview 3D Human Pose Estimation

Geng Chen, Pengfei Ren, Xufeng Jian, Haifeng Sun, Menghao Zhang, Qi Qi, Zirui Zhuang, Jing Wang, Jianxin Liao, Jingyu Wang

Keywords: 3d human pose estimation; pose estimation; VQ-VAE

TL;DR: We introduce a calibration-free 3D pose estimator that uses a discrete prior learned via VQ-VAE and integrates it through proposed discrete-continuous attention for robust, accurate prediction.

Abstract:

Multi-view 3D human pose estimation (HPE) leverages complementary information across views to improve accuracy and robustness. Traditional methods rely on camera calibration to establish geometric correspondences, which is sensitive to calibration accuracy and lacks flexibility in dynamic settings. Calibration-free approaches address these limitations by learning adaptive view interactions, typically leveraging expressive and flexible continuous representations. However, as the multiview interaction relationship is learned entirely from data without constraint, they are vulnerable to noisy input, which can propagate, amplify and accumulate errors across all views, severely corrupting the final estimated pose.
To mitigate this, we propose a novel framework that integrates a noise-resilient discrete prior into the continuous representation-based model. Specifically, we introduce the \textit{UniCodebook}, a unified, compact, robust, and discrete representation complementary to continuous features, allowing the model to benefit from robustness to noise while preserving regression capability.
Furthermore, we further propose an attribute-preserving and complementarity-enhancing Discrete-Continuous Spatial Attention (DCSA) mechanism to facilitate interaction between discrete priors and continuous pose features.
Extensive experiments on three representative datasets demonstrate that our approach outperforms both calibration-required and calibration-free methods, achieving state-of-the-art performance.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Flagged For Ethics Review: true

Submission Number: 12941