Deep Learning for HABs Prediction with Multimodal Fusion

Fei Zhao, Chengcui Zhang

Abstract: Harmful Algal Blooms (HABs) present significant environmental and public health threats. Recent machine learning-based HABs monitoring methods often rely solely on unimodal data, e.g., satellite imagery, overlooking crucial environmental factors such as temperature. Moreover, existing multi-modal approaches grapple with real-time applicability and generalizability challenges due to the use of ensemble methodologies and hard-coded geolocation clusters. Addressing these gaps, this paper presents a novel deep learning model using a single-model-based multi-task framework. This framework is designed to segment water bodies and predict HABs severity levels concurrently, enabling the model to focus on areas of interest, thereby enhancing prediction accuracy. Our model integrates multimodal inputs, i.e., satellite imagery, elevation data, temperature readings, and geolocation details, via a dual-branch architecture: the Satellite-Elevation (SE) branch and the Temperature-Geolocation (TG) branch. Satellite and elevation data in the SE branch, being spatially coherent, assist in water area detection and feature extraction. Meanwhile, the TG branch, using sequential temperature data and geolocation information, captures temporal algal growth patterns and adjusts for temperature variations influenced by regional climatic differences, ensuring the model's adaptability across different geographic regions. Additionally, we propose a geometric multimodal focal loss to further enhance representation learning. On the Tick-Tick Bloom (TTB) dataset, our approach outperforms the SOTA methods by 15.65%.