ESCA: Enabling Seamless Codec Avatar Execution
through Algorithm and Hardware Co-Optimization for Virtual Reality

Supplementary Material for NeurIPS Submission 19446.

Abstract

Photorealistic Codec Avatars (PCA), which enable high-fidelity human face rendering, are increasingly adopted in AR/VR applications to support immersive communication and interaction via deep learning-based generative models. However, these models impose significant computational demands, making real-time inference challenging on resource-constrained AR/VR devices such as head-mounted displays (HMDs), where latency and power efficiency are critical. To address this challenge, we propose an efficient post-training quantization (PTQ) method tailored for Codec Avatar models, enabling low-precision execution without compromising output quality. In addition, we design a custom hardware accelerator that can be integrated into the system-on-chip (SoC) of AR/VR devices to further enhance processing efficiency. Building on these components, we introduce ESCA, a full-stack optimization framework that accelerates PCA inference on edge AR/VR platforms. Experimental results demonstrate that ESCA boosts FovVideoVDP quality scores by up to +0.39 over the best 4-bit baseline, delivers up to 3.36× latency reduction, and sustains a rendering rate of 100 frames per second in end-to-end tests, satisfying real-time VR requirements. These results demonstrate the feasibility of deploying high-fidelity codec avatars on resource-constrained devices, opening the door to more immersive and portable VR experiences.

Comparison of Avatars with Full and Quantized Models

Left: Avatar rendered using the full precision MultiFace model (CVPR 2023-3DMV)
Middle: Degraded avatar showing noise and jitter artifacts from state-of-the-art post-training quantization (INT4)
Right: Clean, stable avatar achieved through the proposed ESCA quantization method (INT4)

Quantization Pipeline

Main Contributions

• Input Channel-wise Activation Smoothing (ICAS): We introduce a novel input channel-wise smoothing module inserted during training to alleviate extreme inter-channel activation disparities in the VAE decoder. By reducing outlier activations, ICAS diminishes quantization error and prevents aberrations when the model is later quantized to low bit-widths.
• Facial-Feature-Aware Smoothing (FFAS): We develop a region-aware smoothing strategy that uses facial masks to identify key areas like the eyes and mouth. Based on the activation variance in these regions, FFAS selectively skips smoothing for the channels most critical to fine details, preserving important textures while still smoothing less sensitive regions.
• UV-weighted Hessian-Based Weight Quantization: We propose a weight quantization scheme guided by a UV-weighted Hessian matrix of the decoder's loss. This method computes second-order sensitivity and weights it by the UV importance of each face region, thereby prioritizing the precision of weights that most affect critical facial features. This results a low-bit model that maintains high reconstruction fidelity in salient areas of the face.
• Customized DNN Hardware Accelerator: We co-design a specialized hardware accelerator to support our quantized Codec Avatar model with high-throughput 4-bit and 8-bit operations. The accelerator features an input-combining mechanism to exploit the structured sparsity of the activation matrix. Moreover, an optimized end-to-end pipeline is applied to deliver over 100 FPS inference on an AR/VR headset ensuring smooth, real-time avatar rendering.

VDP scores

Inference Latency