RTO-LLI: Robust Real-Time Image Orientation Method With Rapid Multilevel Matching and Third-Times Optimizations for Low-Overlap Large-Format UAV Images

Xiongwu Xiao, Zihao Zhang, Gui-Song Xia, Zhenfeng Shao, Jianya Gong, Deren Li

Abstract: Unmanned aerial vehicle (UAV) real-time photogrammetry is important to promote the rapid generation of photogrammetry 4-D product, intelligent information extraction and rapid remote sensing mapping, and efficient large-scale 3-D modeling. However, for real-time processing of low-overlap large-format image sequence, there remain two challenges: 1) large-format images result in greater data volume and computational load, posing challenges for real-time online processing on regular-performance computing units, requiring more efficient algorithms and 2) low-overlap images make it difficult for matching correspondences to cover the entire overlapping area at real time, leading to significant challenges for real-time and robust relative orientation. Therefore, this article proposes a robust real-time orientation method for low-overlap large-format UAV image (RTO-LLI), which can robustly handle this kind of data in real time. First, a robust initialization method for real-time processing of low-overlap large-format images was designed to ensure a high success rate of simultaneous localization and mapping (SLAM) initialization. Second, constant velocity hypothesis tracking enables fast orientation during constant-speed flight. Third, when the second step is false, use a real-time pose estimation method based on multilevel matching and coarse-to-fine optimization to robustly solve the precision image pose. Fourth, the final (third-level) pose optimization method is based on the iteratively reweighted least squares (IRLS) algorithm with a suitable search area, which can compute higher precision image pose in real time. Finally, real-time mapping based on parallel processing for low-overlap images can generate high-precision 3-D point maps and complete feature extraction for the next frame in real time. Experiments conducted on several different types of scenes show that: 1) the processing speed of RTO-LLI significantly surpasses traditional offline methods: PhotoScan, OpenMVG, and Colmap. RTO-LLI can handle large-format UAV image sequence (single imagery has 20 million pixels) at a speed of 1.5 frames/s, meeting the demands of real-time UAV photogrammetry tasks; 2) RTO-LLI is the only method that has successfully completed real-time tasks in all 50-time repeated experiments for four different types of scenes, demonstrating robustness far superior to other classical SLAM solutions; and 3) the displacement error of the estimated pose by RTO-LLI is less than 1/2000 of the trajectory length, and the average reprojection error is less than 1.5 pixels, almost as well as traditional offline methods. The RTO-LLI method meets the efficiency, robustness, and accuracy requirements of real-time photogrammetry for low-overlap large-format UAV images.