Towards Depth Fusion into Object Detectors for Improved Benthic Species Classification

Matthew Dawkins, Jon Crall, Matthew J. Leotta, Tasha O'Hara, Liese Siemann

Abstract: Coonamessett Farm Foundation (CFF) conducts one of the optical surveys of the sea scallop resource using a HabCam towed vehicle. The CFF HabCam v3 collects 5+ images per second, providing a continuous track of imagery as it is towed over scallop grounds. Annotations from HabCam images are translated into biomass estimates through a multi-step process that relies on adequate image subsampling and accurate scallop counts and measurements. Annotating images is the most time-consuming part of generating these reliable scallop biomass estimates. Reliably automating this process has the potential to increase annotation rates and improve biomass estimation, addressing questions about scallop patchiness and distribution over small-to-large scales on scallop grounds. Furthermore, optical survey images provide a wealth of raw data that is not consistently utilized. An additional high priority goal is to provide data for improving flounder stock assessments because the incidental bycatch of non-target species could have negative impacts on the long-term sustainability of the fishery. Kitware, Inc developed the Video and Image Analytics for Marine Environments (VIAME) system for analysis of underwater imagery with support from the NOAA Automated Image Analysis Strategic Initiative. CFF collaborated with Kitware to develop improved automated detectors for scallop classes (live scallops, swimming scallops, and clappers) and flounder by incorporating depth disparity information obtained from stereo image pairs collected during surveys. The accuracy and precision of these new detectors, with depth information incorporated, was contrasted against that of single-image detectors when utilized upgraded deep learning networks. A few methods were investigated, including early and late fusion of the depth information.