Acquisition of 3-D trajectories with labeling support for multi-species insects under unconstrained flying conditions

Abdul Nasir, Muhammad Obaid Ullah, Muhammad Haroon Yousaf, Muhammad Asif Aziz

Abstract: Highlights • Acquisition of multimodal data set (about 1900 IR, RGB and, depth videos) having flying instances of honey bee, Apis mellifera and Vespa spp. • Introduction of a new technique for detection and tracking of flying insects using only valid depth map data. • A new trajectory defragmentation scheme based upon flying insects' spatio-temporal correlation, speed, and physical size. • Object localization in color frames for trajectory labeling based upon analysis of insects' flying kinematics and 3-D location coordinates. • Implementation of a novel data validation strategy to estimate error upper bound in trajectory measurements. Abstract In the work presented here, a new technique based upon stereo vision is proposed to acquire three-dimensional time-resolved trajectories with labeling support for multi-species insects under unconstrained flying conditions. A low-cost, off-the-shelf depth camera is used for stereo vision which is equipped with two wide-angle global-shutter IR imagers to compute depth map and a separate narrow-angle color imager to have color and texture information. Two novel strategies are employed to tackle the challenges imposed by the small size of insects, fast movements, and natural daylight, etc.; (i) introduction of a simple but robust technique for detection and tracking of insects using only the valid depth map data and subsequently to acquire 3-D trajectories (ii) object localization scheme for the insects in acquired trajectories in allied color frames based upon the analysis of insects' flying kinematics and their 3-D location coordinates. The object localization of flying insects in allied color frames provides support to label the multi-spp. trajectories with respective types of the species. The proposed technique provides completely generalized solution to acquire 3-D trajectories of the multi-spp. insects in natural outdoor conditions and in the presented work it is applied to acquire the trajectories of honey bees (Apis mellifera) and invasive hornets (Vespa spp.) near beehives. These hornets are serious honey bee stressors and cause severe damage to the hive's foraging force. The trajectory patterns of honey bees together with Vespa spp. can be analyzed for early detection of Vespa spp. near beehives as well as to estimate stress levels on the honey bees. Acquired 3-D trajectory data are validated for trajectory measurements as well as for the labeling support information. A mean error upper bound of 5 mm is estimated in trajectory measurements whereas 100% inter-spp. and up to 94% intra-spp. labeling support accuracies are recorded.