DepthSense+DP: Adaptive Learning for Robust and Differential Private Silent Speech Recognition

Rong Fu, weizhi Tang, Simon James Fong

Keywords: Adaptive Learning, Silent Speech Recognition, Privacy Aware, Depth Sensing, Biometric securiny, Zero Shot Generalization, Multimodal Fusion

TL;DR: DepthSense+DP bridges depth-sensed lip movements and ultrasound tongue dynamics through adaptive neural learning, enabling private, robust silent speech recognition across wearables and ambient devices.

Abstract: DepthSense+DP is a privacy-preserving framework for silent speech recognition from dynamic 3D depth point clouds. It integrates calibrated input perturbation, feature-level differential privacy, and geometry-preserving alignment within a lightweight P4DConv front end and Conformer encoder to ensure robust cross-user and cross-device generalization under formal DP guarantees. A dual-stage DP pipeline injects noise at point and feature levels while maintaining articulatory geometry, aided by an adaptive DAD gate for improved privacy–utility trade-off. The co-designed architecture enables efficient on-device inference. Experiments on a large multi-location corpus show near-baseline accuracy with significant reductions in membership, inversion, and attribute-inference risks, supported by full DP accounting and attack evaluations.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 2621