Measuring Objective Visual Quality of Real-time Communication Systems in the Wild

Chih-Fan Hsu, Yu-Lun Chang, Xin Liu

Abstract: During the outbreak of epidemic diseases, the demand for real-time communication (RTC) systems has dramatically increased. People use RTC systems to communicate with each other, attend online courses, present projects, and share videos. An open question, how to choose an appropriate system for high-quality communication with different network conditions and usage scenarios, comes RTC users. In this paper, we propose a scheme to systematically measure the visual quality of RTC systems. Our scheme does not need to modify systems and thus can be easily adapted to other RTC systems and video transmission systems, i.e., streaming systems. Based on this scheme, we measure six commercial RTC systems in four common usage scenarios. We measure the received video quality (graphical quality and frame rate) at the receiver and the upload bitrate at the sender. Furthermore, we propose a novel metric, area loss, to measure the system ability to handle insufficient bandwidth using video freezing and frame delay. Our detailed analysis reveals the advantages and disadvantages of each system. We expect the measurement scheme, the metric, and our findings can help the future development of RTC systems.