Amodal Memory-Based Hyperspectral Object Tracking

Anthony Medellin, David Grabowsky, Albert Ma, Dariusz Mikulski, Reza Langari

Abstract: We propose the use of Segment Anything Model 2 (SAM2) to perform object tracking with amodal annotations by using monocular depth estimation and Kalman filter-based object re-initialization. The tiny SAM2 model utilizes a 3 channel false image, generated from the most informative layers of a hyperspectral image, to perform object tracking. This approach leads to decreased computational cost and increased object tracking speed, compared to utilization of the entire hyperspectral image, without degradation in performance. Moreover, this approach is easily extensible from hyperspectral images with 15 channels to 200+ channels, without re-training a model or fine-tuning. This is due to the fact that a foundational model and an analytical-based re-initialization method is used. The results show that the SAM2 model with the incorporation of amodal annotation and re-initalization data outperform current state-of-the-art hyperspectral-based object trackers.