Content-Aware Cross-Modal Stream Transmission

Zhe Yuan, Xin Wei, Liang Zhou, Weihua Zhuang

Abstract: Multi-modal services, integrating audio, video, and haptic streams, have shown their great potential to improve user immersive experience. However, due to their distinct requirements, simultaneous transmission of streams from different modalities is a significant challenge. To address the challenge, we propose a content-aware cross-modal stream transmission scheme by leveraging content correlations to connect haptic preemptive scheduling and video signal restoration. Specifically, we firstly formulate a general cross-modal stream transmission problem as video utility maximization under the haptic requirement constraint. Then, an online content-aware cross-modal resource allocation algorithm is designed to solve the cross-modal stream transmission problem by scheduling haptic streams to preempt highly correlated video streams in a dynamic environment. Finally, simulation results show that our scheme improves the video throughput by 11.7% as compared with other popular schemes while ensuring low latency and high reliability of haptic streams.