Enhancing Real-time Inference Performance for Time-Critical Software-Defined Vehicles

Sumaiya, Reza Jafarpourmarzouni, Sidi Lu, Zheng Dong

Abstract: In the rapidly evolving landscape of vehicle computing, the efficiency and reliability of real-time responses are paramount. The primary rationale lies in the dynamic and unpredictable nature of road environments, where swift and accurate recognition of obstacles, pedestrians, and other vehicles is essential for safe navigation. Faster inference time ensures minimal latency in decision-making, allowing for immediate responses to sudden changes in the driving scenario, such as unexpected pedestrian movements or the rapid approach of other vehicles. This rapid processing capability is indispensable for preventing accidents and enhancing passenger safety. In response to these challenges, our research presents an experimental investigation aimed at accelerating inference time and maximizing throughput. We conducted a comparative analysis of four different workflows using the mainstream object detection models on TensorRT for Full Precision (FP32), Half Precision (FP16), and Integer Precision (INT8). Our results showcase the inference performance of each workflow and observations with their respective accuracy levels. This paper provides a detailed guide for selecting an appropriate workflow based on specific requirements for inference performance and accuracy, offering valuable insights for advancements in the domain of software-defined vehicles and other real-time systems.