SAM2Flow: Interactive Optical Flow Estimation with Dual Memory for in vivo Microcirculation Analysis

Luojie Huang, Ryan Zhang, Marisa M Morakis, Michaela Taylor-Williams, Gregory N. McKay, Nicholas J. Durr

Keywords: Optical Flow Estimation, Microscopic Blood Cell Analysis, Segment Anything Model

TL;DR: This work presents SAM2Flow, an interactive optical flow model for analyzing microvascular flow in OBM videos, leveraging SAM2-inspired prompting and a novel dual motion+context memory mechanism for efficient long-sequence flow estimation.

Abstract:

Analysis of noninvasive microvascular blood flow can improve the diagnosis, prognosis, and management of many medical conditions, including cardiovascular, peripheral vascular, and sickle cell disease. This paper introduces SAM2Flow, an interactive optical flow estimation model to analyze long Oblique Back-illumination Microscopy (OBM) videos of *in vivo* microvascular flow. Inspired by the Segment Anything Model (SAM2), SAM2Flow enables users to specify regions of interest through user prompts for focused flow estimation.
SAM2Flow also incorporates a dual memory attention mechanism, comprising both motion and context memory, to achieve efficient and stable flow estimations over extended video sequences. According to our experiments, SAM2Flow achieves SOTA accuracy in foreground optical flow estimation on both microvascular flow and public datasets, with a fast inference speed of over $20$ fps on $512\times512$ inputs. Based on the temporally robust flow estimation, SAM2Flow demonstrated superior performance in downstream physiological applications compared to existing models. The code and dataset will be published with this paper.

Primary Area: Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)

Submission Number: 3465