% Title: Learning Vortex Dynamics for Fluid\\ Inference and Prediction

% ArXiv ID: 2301.11494





\maketitle

\begin{abstract}
We propose a novel \emph{differentiable vortex particle} (DVP) method to infer and predict fluid dynamics from a single video.
Lying at its core is a particle-based latent space to encapsulate the \textit{hidden}, Lagrangian vortical evolution underpinning the \textit{observable}, Eulerian flow phenomena. Our \textit{differentiable vortex particles} are coupled with a learnable, vortex-to-velocity dynamics mapping to effectively capture the complex flow features in a physically-constrained, low-dimensional space. This representation facilitates the learning of a fluid simulator tailored to the input video that can deliver robust, long-term future predictions.
The value of our method is twofold: first, our learned simulator enables the inference of hidden physics quantities (\textit{e.g.}, velocity field) purely from visual observation; secondly, it also supports future prediction, constructing the input video's \textit{sequel} along with its future dynamics evolution.
We compare our method with a range of existing methods on both synthetic and real-world videos, demonstrating improved reconstruction quality, visual plausibility, and physical integrity.\footnote{Our video results, code, and data can be found at our project website: \url{https://yitongdeng.github.io/vortex_learning_webpage}.}

\end{abstract}
 
\section{Introduction}


As small as thin soap films, and as large as atmospheric eddies observable from outer space, fluid systems can exhibit intricate dynamic features on different mediums and scales. 
However, despite recent progress, it remains an open problem for scientific machine learning to effectively represent these flow features, identify the underlying dynamics system, and predict the future evolution, due to the noisy data, imperfect modeling, and unavailable, \textit{hidden} physics quantities. 
 
Here, we identify three fundamental challenges that currently hinder the success of such endeavors.
First, \emph{flow features are difficult to represent}.
Traditional methods learn the fluid dynamics by storing velocity fields either using regularly-spaced grids or smooth neural networks. These approaches have demonstrated promising results for fluid phenomena that are relatively damped and laminar (\textit{e.g.}, \citealp{chu2022physics}), but for fluid systems that can exhibit turbulent features on varying scales, these methods fall short due to the problem's curse of dimensionality (high-resolution space and time), local non-smoothness, and hidden constraints.
As a result, more compact and structured representation spaces and data structures are called for.

Secondly, 
\emph{hidden flow dynamics is hard to learn.}
Fluid systems as prescribed by the Navier-Stokes equations tightly couple multiple physical quantities
(\textit{i.e.}, velocity, pressure, and density), and yet, only the density information can be accessibly measured. Due to the system's complexity, ambiguity, and non-linearity, directly learning the underlying dynamics from the observable density space is infeasible; and successful learning usually relies on velocity or pressure supervision, a requirement that distances these methods from deployment in real-world scenarios. 

Exciting recent progress has been made in hidden dynamics inference by PDE-based frameworks such as \citet{HFMScience}, which uncover the underlying physics variables solely from density observations. However, this type of methods encounter the third fundamental challenge, which is that \textit{performing future prediction is difficult.} As we will demonstrate, although strong results are obtained for interpolating inside the observation window provided by the training data, these methods cannot extrapolate into the future, profoundly limiting their usage.

In this paper, we propose the \emph{differentiable vortex particle} (DVP) method, a novel fluid learning model to tackle the three aforementioned challenges in a unified framework. 
In particular, harnessing the physical insights developed for the \textit{vortex methods} in the computational fluid dynamics (CFD) literature, we design a novel, data-driven Lagrangian vortex system to serve as a compact and structured \textit{latent representation} of the flow dynamics. 
We learn the complex dynamical system underneath the high-dimensional image space by learning a surrogate, low-dimensional model on the latent vortex space, and use a physics-based, learnable mechanism: the vortex-to-velocity module, to \textit{decode} the latent-space dynamics back to the image space.
Leveraging this physics-based representation, we design an end-to-end training pipeline that learns from a single video containing only density information. Our DVP method enables accurate inference of hidden physics quantities and robust long-term future predictions, as shown in Figure~\ref{fig:teaser}.  


To examine the efficacy of our method, we compare our method's performance on both \textit{motion inference} and \textit{future prediction} against various state-of-the-art methods along with their extensions. We conduct benchmark testing on synthetic videos generated using high-order numeric simulation schemes as well as real-world videos in the wild. Evaluation is carried out both quantitatively through exhaustive numerical analysis, and qualitatively by generating a range of realistic visual effects. We compare the uncovered velocities in terms of both reconstruction quality and physical integrity, and the predicted visual results in terms of both pixel-level and perceptual proximity.  Results indicate that our proposed method provides enhanced abilities on both fronts, inferring hidden quantities at higher accuracy, and predicting future evolution with higher plausibility.



\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/teaser2.pdf}
    \vspace{-0.6cm}
    \caption{Our goal is to learn vortex dynamics for fluid inference and prediction. The 3 frames on the left are observed from a real video of a soap film on a circular metal rim, where the red ink is spreading. The 3 frames on the right are future prediction results produced by our method.}
    \label{fig:teaser}
    \vspace{-0.3cm}
\end{figure}


In summary, the main technical contributions of our framework align with the three challenges regarding flow representation, dynamics learning, and simulator synthesis.
(1) We devise a novel representation for fluid learning, the \emph{differentiable vortex particles} (DVP), to drastically reduce the learning problem's dimensionality on complex flow fields. Motivated by the vortex methods in CFD, we establish the vorticity-carrying fluid particles as a new type of learning primitive to transform the existing PDE-constrained optimization problem %
to a particle trajectory (ODE) learning problem.
(2) We design a novel particle-to-field paradigm for learning the Lagrangian vortex dynamics. Instead of learning the interaction among particles (\textit{e.g.}, \citealp{sanchez2020learning}), our model learns the continuous vortex-to-velocity induction mapping to naturally connect the vortex particle dynamics in the latent space with the fluid phenomena captured in the image space. 
(3) We develop an \emph{end-to-end differentiable pipeline} composed of two network models to synthesize data-driven simulators based on single, short RGB videos. 




\section{Related Work}
\emph{Hidden Dynamics Inference.} 
The problem of inferring dynamical systems based on noisy or incomplete observations has been addressed using a variety of techniques, including symbolic regression \citep{bongard2007automated, schmidt2009distilling}, dynamic mode decomposition \citep{schmid2010dynamic, kutz2016dynamic}, sparse regression \citep{brunton2016discovering, rudy2017data}, Gaussian process regression \citep{raissi2017machine, raissi2018hidden}, and neural networks \citep{raissi2019physics, yang2020physics, jin2021nsfnets, chu2022physics}. Among these inspiring advancements, the ``hidden fluid mechanics'' (HFM) method proposed in \citet{HFMScience} is particularly noteworthy, as it uncovers the continuous solutions of fluid flow using only images (the transport of smoke or ink). 


\emph{Data-driven Simulation.}
Recently, growing interests are cast on learning numerical simulators according to data supervision, which has shown promise to reduce computation time \citep{ladicky2015data,guo2016convolutional, wiewel2019latent, pfaff2020learning,sanchez2020learning,tompson2017accelerating}, increase simulation realism \citep{chu2017data, xie2018tempogan}, enable stylized control \citep{kim2020lagrangian}, estimate dynamic quantities such as viscosity and energy \citep{chang2016compositional, battaglia2016interaction, ummenhofer2019lagrangian}, and facilitate the training of control policies \citep{sanchez2018graph, li2018learning}. Akin to \citet{watters2017visual}, our system takes images as inputs and performs dynamics simulation on a low-dimensional latent space; but 
our method learns purely from the input video and performs future rollout in the image space. 
Our method is also related to \citet{guan2022neurofluid}, which infers Lagrangian fluid simulation from observed images. We propose sparse neural vortices as our representation while they use dense material points.
 
\emph{Vortex Methods.}
The underlying physical prior incorporated in our machine learning system is rooted in the family of vortex methods that are rigorously derived, analyzed, and tested in the computational fluid dynamics (CFD) \citep{leonard1980vortex,PERLMAN1985200, beale1985high, WINCKELMANS1993247, mimeau2021review} and computer graphics (CG) \citep{selle2005vortex, park2005vortex, weissmann2010filament, brochu2012linear} communities.
\citet{xiong2020neural} is pioneering for combining the Discrete Vortex Method with neural networks, but its proposed method relies on a large set of ground truth velocity sequences, whereas our method learns from single videos without needing the ground truth velocity.

\section{Physical Model}
\label{sec:physics}

We consider the velocity-vorticity form of the Navier--Stokes equations (obtained by taking the curl operator of both sides of the momentum equation, see \cite{cottet2000vortex} for details):
\begin{align}
    \frac{D\bm{\omega}}{Dt} = \frac{\partial{\bm{\omega}}}{\partial{t}} + \bm{u}\cdot\nabla\bm{\omega} &= \bm{\omega}\cdot \nabla\bm{u} + \nu\nabla^2\bm{\omega} + \nabla \times\bm{b},\\
    \bm{u} = \nabla \times \bm{\phi} &,\ \ \  \nabla^2 \bm{\phi} = -\bm{\omega},
\end{align}
where $\bm{\omega}$ denotes the vorticity, $\bm{u}$ the velocity, $\bm{b}$ the conservative body force, $\nu$ the kinematic viscosity, and $\bm\phi$ the streamfunction.
If we ignore the viscosity and stretching terms (inviscid 2D flow), we obtain $D\bm{\omega} / Dt = \bm{0}$, which directly conveys the Lagrangian conservative nature of vorticity (\textit{i.e.}, a particle's vorticity will not change during its advection).

If we assume the fluid domain has an open boundary, we can further obtain the vortex-to-velocity induction formula, which is derived by solving Poisson's equation on $\bm\phi$ using Green's method (also known as the Biot-Savart Law in fluid mechanics):

\begin{align}
    \bm{u}(\bm{x}) &= \int{K(\bm{x} - \bm{x}')\bm\omega(\bm{x}')d\bm{x}'},\label{eq:B-S}
\end{align}
The kernel $K$ exhibits a type-II singularity at $\bm{0}$ and causes numerical instabilities, therefore in CFD practices, $K$ is replaced by various mollified versions $K_\delta$ to improve the simulation accuracy \citep{beale1985high}. We note that the mollified version $K_\delta$ is not unique, and can be customized and tuned in different numerical schemes per human heuristics. Different types and parameters for the mollification bring about significantly different simulation results.

\emph{Takeaways.}
The mathematical models above provide two central physical insights guiding the design of our vortex-based learning framework\rev{:}
(1) The Lagrangian conservation of vorticity $\bm{\omega}$ suggests the suitability of adopting Lagrangian data structures (\textit{e.g.}, particles as opposed to grids) to capture the dynamics. 
Since the tracked variable $\bm{\omega}$ remains temporally invariant for each Lagrangian vortex, the evolution of the continuous flow field is embodied fully by the movement of these vortices, which significantly alleviates the difficulty in learning.
(2) Equation~\ref{eq:B-S} presents an induction mapping from the vorticity $\bm{\omega}$, a Lagrangian quantity carried by particles, to the velocity $\bm{u}$, an Eulerian variable that can be queried continuously at an arbitrary location $\bm{x}$. This lends the possibility for the Lagrangian method to be used in conjunction with Eulerian data structures (\textit{e.g.}, a grid) for learning from the widely available video data.
Furthermore, such a mapping can benefit from data-driven learning, as we can replace the human heuristics by learning a mollified kernel $K_\delta$ to minimize the discrepancy between the simulated and observed flow phenomena.

\section{Method}
\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/representation_diagram.pdf}
    \caption{Illustration of our differentiable vortex particle (DVP) method. Given an input RGB image sequence (top row), we learn a dynamical system on a low-dimensional vortex space (bottom row), whose motion is decoded into the motion of the high-dimensional image space to explain the observed fluid phenomena.}
    \label{fig:overview_illustration}
\end{figure}

\emph{System Overview.}
Following the physics insight conveyed in Section~\ref{sec:physics}, we design a learning system whose workflow is illustrated in Figure~\ref{fig:overview_illustration}. 
As shown on the top row, our system takes as input a single RGB video that captures the vortical flow phenomena. 
As shown on the bottom row, our method learns and outputs a dynamical simulator --- not on the image space itself, but on a latent space consisting of discrete vortices. 
Learning the latent dynamics in the vortex space would only be useful and feasible if we can tie it back to the image space, because it is the image space that we want to perform future prediction on, and we have no ground truth values for the vortex particles to begin with. The bridge to tie the vortex space with the image space derives from Equation~\ref{eq:B-S}, which supplies the core insight that there exists a learnable mapping from vortex particles to the continuous velocity field at arbitrary positions. This mapping is modeled by our learned dynamics module $\mathcal{D}$, %
which gives rise to the intermediate velocity space, as shown in the middle row of Figure~\ref{fig:overview_illustration}.

\subsection{Differentiable Vortex Particles}
\rev{We track a collection $\mathcal{V}$ of $n$ vortex particles, \textit{i.e.}, $\mathcal{V} \coloneqq [V_1, \dots, V_n]$. We define each vortex $V_i$ as the 3-tuple $(\bm{x}_i, \omega_i, \delta_i)$, where $\bm{x}$ represents the position, $\omega$ the vortex strength, and $\delta$ the size. The number of particles $n$ is a hyperparameter which we set to 16 for all our results. Further discussions and experiments regarding the choice of $n$ can be found in Appendix~\ref{append: Num_P}. We also note that, since we are concerned with 2D inviscid incompressible flow, the size $\delta$ of a vortex does not change in time due to incompressibility, and the vortex strength $\omega$ does not change in time due to Kelvin's circulation theorem (see \citet{Hald} for a thorough discussion).}


\emph{Learning Particle Trajectory.}
As shown in Figure~\ref{fig:synthetic_plot}, we learn a particle trajectory module: a query function $\mathcal{T}$ such that $\mathcal{V}_t = \mathcal{T}(t)$, which predicts the configuration of all the vortices at any time $t \in [0, t_E]$ where $t_E$ represents the end time of the input video. As described above, predicting $\mathcal{V}_t$ boils down to determining two time-invariant components: (1) $[\omega_1,\dots,\omega_n]$, (2) $[\delta_1,\dots,\delta_n]$, and one time-varying component: $[(\bm{x}_1)_t, \dots, (\bm{x}_n)_t]$. For the two time-invariant components, we introduce two trainable $n \times 1$ vectors $\Delta$ and $\Omega$ to represent $\delta$ and $\omega$ respectively, such that $[\omega_1, \dots, \omega_n] = \sin(\Omega)$ and $[\delta_1, \dots, \delta_n] = \text{sigmoid}(\Delta) + \eps$ ($\eps$ is a hyperparameter we set to 0.03). \rev{The vortex size $\Delta$ and strength $\Omega$ are optimized to fit the motion depicted by the input RGB video.} For the time-varying component, we use a neural network $N_1(t)$ to encode $N_1(t) = [(\bm{x}_1)_t, \dots, (\bm{x}_n)_t]$, and the particle velocities $\frac{dN_1}{dt}$ can be extracted using automatic differentiation. \rev{We note that learning the full particle trajectory, rather than the initial particle configuration, allows the aggregation of dynamics information throughout the input video for better inference and prediction. We provide further discussion on this design in Appendix~\ref{append: learn_trajectory}}.


\rev{\emph{Trajectory Initialization.} As discussed above, the trajectory $\mathcal{T}$ has three learnable components: $\Delta$, $\Omega$ and $N_1$. We initialize $\Delta$ and $\Omega$ as zero vectors, which gives $\delta_i = 0.5 + \epsilon$ and $\omega_i = 0$ for all $i$. Conceptually, these vortices are initialized as large blobs with no vortex strength, which learn to alter their sizes and grow their strengths to better recreate the eddies seen in the video. The initial positions $[(\bm{x}_1)_{0}, \dots, (\bm{x}_n)_{0}]$ are regularly spaced points to populate the entire domain. We initialize the 16 particles to lie at the grid centers of a $4 \times 4$ grid. To do so, we simply pretrain $N_1$ so that $N_1(0)$ evaluates to the grid centers. The details regarding pretraining are given in Appendix~\ref{append: imp}.}

\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/trajectory_and_flow.pdf}
    \caption{We encapsulate the motion of a continuous field by the motion of discrete particles. The blue trajectory is encoded by a neural network $N_1$, corresponding to the input video; while the red trajectory is unrolled using our learned dynamics module and a numeric integrator, corresponding to the future prediction.}
    \label{fig:synthetic_plot}
\end{figure}

\emph{Learning the Vortex-to-Velocity Mapping.}
The vortex-to-velocity mapping is performed by our dynamics module $\mathcal{D}$, which predicts the velocity $\bm{u}$ at an arbitrary query point $\bm{x}$ given the collection of vortices $\mathcal{V} = [(\bm{x}_1, \omega_1, \delta_1), \dots, (\bm{x}_n, \omega_n, \delta_n)]$. Following the physical insight conveyed in Section~\ref{sec:physics}, $\mathcal{D}$ should evaluate the integration:
\begin{align}
    \bm u(\bm{x}) = \int{K_\delta(\bm{x} - \bm{x}')\bm\omega(\bm{x}')d\bm{x}'},
\end{align}
which replaces the kernel $K$ in Equation~\ref{eq:B-S} by a learnable mapping $K_\delta: \mathbb{R}^d \rightarrow \mathbb{R}^d$, with $d$ representing the spatial dimension. Rather than directly using a neural network to model this $\mathbb{R}^d \rightarrow \mathbb{R}^d$ mapping, we incorporate further physical insights by analyzing the structure of $K_\delta$. As derived in \citet{beale1985high}, the kernel $K_\delta$ for 2-dimensional flow exhibits the following form: 
\begin{align}
    K_\delta(\bm{z}) &= \frac{1}{2\pi r}M(r,\delta)R_{\frac{2}{\pi}}(\bm{z}),\ r = |\bm{z}|
\end{align}
where $R_{\frac{2}{\pi}}(\bm z)$ computes the unit direction of the cross product of $\bm{z}$ and the out-of-plane unit vector; and $M(r, \delta)$ is the human heuristic term that varies by choice. Hence, we opt to replace $\frac{1}{2\pi r}M(r,\delta)$ by a $\mathbb{R}^2 \rightarrow \mathbb{R}$ neural network function $N_2(r, \delta)$ so that:
\begin{align}
    \bm u(\bm{x}) &= \int{N_2(|\bm{x} - \bm{x}'|, \delta_i)R_{\frac{2}{\pi}}(\bm{x} - \bm{x}')\bm\omega(\bm{x}')d\bm{x}'}\\
    &\ \approx \sum_{i=1}^n {N_2(|\bm{x} - \bm{x}_i|, \delta_i)R_{\frac{2}{\pi}}(\bm{x} - \bm{x}_i){\omega}_i}
    = \mathcal{D}(\mathcal{V})(\bm{x}).\label{eq:discrete_bs}
\end{align}
\rev{Learning this induction kernel $N_2(r, \delta)$ instead of using heuristics-based kernels allows for more accurate fluid learning and prediction from input videos. We discuss more on this in Appendix~\ref{append: Learn_Kernel}}. 

\subsection{End-to-end training}
As previously mentioned, the dynamics on the latent vortex space is bridged to the evolution of the image space through the differentiable, dynamics module $\mathcal{D}$. Hence, we can optimize the vortex representation $\mathcal{V}_t = \mathcal{T}(t)$ at time $t$ using images as supervision. First, we select $m+1$ frames: $[I_t, \dots, I_{t+m}]$ from the video. Then, we compute $\bm{u}_t = \mathcal{D}(\mathcal{V}_t)$. After that, $(\bm{u}_t, I_t)$ is fed into an integrator on the Eulerian grid to predict $\tilde{I}_{t+1}$. Simultaneously, $(\bm{u}_t, \mathcal{V}_t)$ is fed into an integrator on the Lagrangian particles to predict $\tilde{\mathcal{V}}_{t+1}$. The process is then repeated, using $\tilde{I}_{t+1}$ in place of $I_t$ and $\tilde{\mathcal{V}}_{t+1}$ in place of $\mathcal{V}_t$, to generate $\tilde{I}_{t+2}$ and $\tilde{\mathcal{V}}_{t+2}$, and so on. Eventually, we would obtain $[\tilde{I}_{t+1}, \dots, \tilde{I}_{t+m}]$, which are the predicted outcome starting at time $t$. We optimize $\mathcal{T}$ and $\mathcal{D}$ jointly by minimizing the difference between $[\tilde{I}_{t+1}, \dots, \tilde{I}_{t+m}]$ and $[I_{t+1}, \dots, I_{t+m}]$ in an end-to-end fashion. 

\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/two_more_reals.pdf}
    \caption{Applied to real-world videos, our DVP method can create more realistic future predictions over long periods of time compared to existing methods (and their extensions).}
    \label{fig:real3andreal10}
\end{figure}

By picking different values of $t$ in each training iteration to cover $[0, t_E]$, we optimize $\mathcal{T}$ and $\mathcal{D}$ to fit the input video.
There remains one more caveat --- which is that the trajectories encoded with $\mathcal{T}$ are not enforced to be consistent with $\mathcal{D}$, because each frame of $\mathcal{V}_t$ is optimized individually. In other words, if we evaluate the particle velocities $[\dot{\bm{x_1}}$, \dots, $\dot{\bm{x_n}}] = \frac{dN_1}{dt}$ as prescribed by $\mathcal{T}$, it should coincide with $[\mathcal{D}(\mathcal{V})(\bm{x_1}), \dots, \mathcal{D}(\mathcal{V})(\bm{x_n})]$, as prescribed by $\mathcal{D}$. Hence, in training, another loss is computed between $\frac{d{N}_1}{dt}$ and $[\mathcal{D}(\mathcal{V})(\bm{x_1}), \dots, \mathcal{D}(\mathcal{V})(\bm{x_n})]$ to align the vortex trajectory and the predicted velocity.

\emph{Deployment.} After successful training, our learned system performs two important tasks. First, using our query function $\mathcal{T}(t)$, we are able to temporally interpolate for $\mathcal{V}_t$, which then uncovers the hidden velocity field $\bm{u} = \mathcal{D}(\mathcal{V}_t)$ at arbitrary resolutions, providing the same functionality as \citet{raissi2018hidden}, but using vorticity instead of pressure as the secondary variable. Moreover, with the dynamics module $\mathcal{D}$, we can perform future prediction to unroll the input video, a feature unsupported by previous methods. As shown in Figure~\ref{fig:real3andreal10}, since our method is forward-simulating by nature, it can provide more realistic and robust future predictions than existing methods or their extensions.
\rev{Further implementation details of our method, including hyperparameters, network architectures, training schemes, and computational costs can be found in Appendix~\ref{append: imp}.} 


\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/inference_illustration.pdf}
    \caption{Hidden motion inference compared with existing methods on a synthetic video. Our DVP method uncovers the underlying velocity field at higher accuracy.}
    \vspace{-0.2cm}
    \label{fig:synthetic_illustration}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/combined_revised.pdf}
    \caption{Error analysis on a synthetic video. The top row plots the inference errors of velocity, vorticity, and compressibility. The bottom row plots the future prediction errors, which consider both the dynamics error of the velocity and the perceptual error of the generated image sequence.}
    \label{fig:synthetic_curves}
\end{figure}

\section{Experiments}
We evaluate our method's ability to perform motion inference and future prediction on both synthetic and real videos, comparing against existing methods.

\emph{Baselines.}
For motion inference, we compare our method against \citet{raissi2018hidden} (HFM) and \citet{zhang2022learning} (E-R). We reimplement the HFM method as prescribed in the paper, making only the modification that instead of using only a single concentration variable $c$ and its corresponding variable $d:=1-c$, we create three $(c,d)$ pairs for each of the RGB channel for the support of colored videos. The E-R method is evaluated using the published pretrained models. We further compare against an ablated version of our proposed method, termed ``UNet'', which essentially replaces the Lagrangian components of the system with a UNet architecture \citep{UNet}, a classic method for learning field-to-field mappings. The UNet baseline takes two images $I_t$ and $I_{t+1}$ and predicts a velocity field $\bm{u}_{t+1}$ to predict $I_{t+2}$ using the same Eulerian integrator as our method. 
For future prediction, there do not exist previous methods that operate in comparable settings, so we extend the inference methods in a few ways to support future prediction in a logical and straightforward manner. First, since HFM offers a query function parameterized by $t$, we test its future prediction behavior by simply extrapolating with $t>t_E$; this is referred to as ``HFM extp.''. Since both \citet{raissi2018hidden} and \citet{zhang2022learning} uncover the time-varying velocity field, we use a UNet to learn the evolution from $\bm{u}_t$ to $\bm{u}_{t+1}$, and use this velocity update mechanism to perform future prediction. The two baselines thus obtained are referred to as ``HFM+UNet'' and ``E-R+UNet'' respectively. Our method's ablation ``UNet'' supports future prediction intrinsically.

\subsection{Synthetic Video}
The synthetic video for vortical flow is generated using the Discrete Vortex Method with a first-order Gaussian mollifying kernel $M$ \citep{beale1985high}. The high-fidelity BFECC advection scheme \citep{kim2005flowfixer} with Runge-Kutta-3 time integration is deployed. The simulation advects a background grid of size $256 \times 256$, with a time step $dt = 0.01$ to create 300 simulated frames. Only the first 100 frames will be disclosed to train all methods, and future predictions are tested and examined on the following 200 frames.


\emph{Motion Inference.} The results for the uncovering of hidden dynamic variables are illustrated in Figure~\ref{fig:synthetic_illustration} and Figure~\ref{fig:synthetic_curves}. Shown in Figure~\ref{fig:synthetic_illustration} are the velocities uncovered by all 4 methods against the ground truth, at frame 55 of the synthetic video with 100 observed frames. The velocity is visualized in the forms of colors (top row) and streamlines (middle row), while the velocity residue, measured in end-point error (EPE), is depicted on the bottom row. It can be seen that HFM, UNet, and our method achieve agreeing results, all matching the ground truth values at high accuracy. On the bottom row, it can be seen that as compared to HFM and UNet, our method generates the inference velocity that best matches the unseen ground truth.

The inference results over the full 100 frames are depicted at the top of Figure~\ref{fig:synthetic_curves}. We evaluate the velocity with four metrics: the average end-point error (AEPE), average angular error (AAE), vorticity RMSE and compressibility RMSE. From all 4 metrics, it can be seen that our method outperforms the baselines consistently. The time-averaged data for all four metrics are shown on the left of Table~\ref{tab: synthetic_table}, which deems our method favorable for all metrics used.

\emph{Future Prediction.} In Figure~\ref{fig:synthetic_future_pred}, we visually compare the future prediction results (from frame 100 to frame 299) using our method and the 4 benchmarks against the ground truth. It can be seen that the sequence generated by our method best matches the ground truth video, capturing the vortical flow structures, while the other baselines either quickly diffuse or generate unnatural, hard-edged patterns. Numerical analysis confirms these visual observations, as we compare the 200 future frames in terms of both velocity and visual similarity. The velocity analysis inherits the same 4 metrics, and the visual similarity is gauged using the pixel-level RMSE and the VGG feature reconstruction loss \citep{perceptual_loss}. The time-averaged results of all 6 metrics are documented on the right of Table~\ref{tab: synthetic_table}, and the time-varying results are plotted on the bottom of Figure~\ref{fig:synthetic_curves}. It can be concluded from the visual and numerical evidence that our method outperforms the baselines in this case.

\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/future_pred_comparison.pdf}
    \caption{Future prediction of our DVP method compared to baselines methods. Our method accurately predicts the unseen, future sequence that is twice as long as the seen sequence.}
    \label{fig:synthetic_future_pred}
\end{figure}

\begin{table}
\centering\small
\begin{tabularx}{\textwidth}{Y | Y Y Y Y || Y Y Y Y Y Y Y}
\hlineB{3}
\multicolumn{5}{c}{Time-averaged Inference Errors} & \multicolumn{7}{c}{Time-averaged Prediction Errors}\\
\hlineB{2.5}
& AEPE & AAE & Vort. & Div. & & VGG & RMSE & AEPE &AAE&Vort.&Div.\\
\hlineB{2}
E-R & 0.505 &  1.393 & 8.470 & 2.319& +UNet & 4.346 & 0.205 & 0.631 &1.424&12.84&6.580\\
\hlineB{2}
\multirow{2}{*}{HFM} & \multirow{2}{*}{0.100} & \multirow{2}{*}{0.212} & \multirow{2}{*}{3.949} & \multirow{2}{*}{0.202} & +UNet & 4.258 & 0.205 & 0.720 & 1.062 & 36.73 & 10.41\\
\cline{6-12}
&&&&& Extp. & 4.080 & 0.285 & 0.541 & 1.464 & 7.761 & 4.315\\
\hlineB{2}
UNet & 0.048 & 0.100 & 1.799 & 1.145 &  & 4.530 & 0.211 & 0.424 &1.159&7.334&3.017\\
\hlineB{2}
Ours & \textbf{0.020} & \textbf{0.041} & \textbf{0.976} & \textbf{0.053} &  & \textbf{2.010} & \textbf{0.080} & \textbf{0.048} &\textbf{0.096}&\textbf{1.621}&\textbf{0.043}\\
\hlineB{3}
\end{tabularx}
\vspace{5pt}
\captionof{table}{Error analysis of benchmark testing on a synthetic dataset.}
\label{tab: synthetic_table}
\end{table}

\subsection{Real Video}
A similar numerical analysis is carried out on a real video published on YouTube, as shown in Figure~\ref{fig:real4_plot}. The video has 150 frames: the first 100 frames will be used for training, while the remaining 50 frames will be reserved for testing. Since the ground truth velocities for the real video are intractable, we will only analyze the future-predicting performance. For all methods, we perform future prediction for 150 frames; among these, the first 50 frames will be compared with the original video, and the rest (100 frames) will be evaluated visually and qualitatively. Since only part of the video is fluid (within the circular rim), we pre-generate a signed distance field for all methods, so that only the fluid regions are considered in learning and simulation. The same boundary condition is employed for all methods (except for ``HFM extp.'' which requires no advection).

The numerical analysis for the first 50 predicted frames is documented and plotted in Table~\ref{fig: real_data_table} and Figure~\ref{fig: real_data_curves}. We compare our method against the baselines based on the VGG perceptual loss for visual plausibility, and the velocity divergence (which should in theory be 0 for incompressible fluid) for physical integrity. It can be seen that our method prevails on all metrics used. For prediction results that exceed the duration of the real video, qualitative observations can be made: our method preserves the vortical structures and generates smooth visualizations over the entire time horizon, while other methods end up yielding glitchy patterns.





\begin{figure}[t]
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/real3.pdf}
    \caption{Future prediction of our DVP method compared to baseline methods on a real video sequence. Our method generates a predicted sequence that best matches the input video within its duration, and remains visually plausible way beyond its duration.}
    \label{fig:real4_plot}
\end{figure}

\begin{figure}[t!]
  \begin{minipage}[b]{0.49\textwidth}
    \centering
    \includegraphics[width=1.\textwidth]{actual_results/real_plots.pdf}
    \captionof{figure}{Error plots corresponding to
    Figure~\ref{fig:real4_plot}.}
    \label{fig: real_data_curves}
  \end{minipage}
  \hfill
  \begin{minipage}[b]{0.5\textwidth}
    \centering\small
\begin{tabular}{
  p{\dimexpr.3\linewidth-2\tabcolsep-1.3333\arrayrulewidth}%
  |
  p{\dimexpr.2\linewidth-2\tabcolsep-1.3333\arrayrulewidth}%
  p{\dimexpr.2\linewidth-2\tabcolsep-1.3333\arrayrulewidth}%
  p{\dimexpr.2\linewidth-2\tabcolsep-1.3333\arrayrulewidth}%
  }
  \hlineB{3}
  \centering  & \centering VGG (avg.)  & VGG (final) & \centering\arraybackslash Div. (avg)    \\ \hlineB{2.5}
  E-R         & 2.095        & 2.205 & 2.046        \\ \hline
  HFM+UNet         & 2.151        & 2.231 &0.940        \\ \hline
  HFM Extp.         & 2.980        & 3.271 &1.922        \\ \hline
  UNet         & 2.111        & 2.088 &1.447       \\ \hline
  Ours         & \textbf{2.093}        & \textbf{2.045}   &\textbf{0.318}      \\ \hlineB{3}
\end{tabular}
        \vspace{10pt}
      \captionof{table}{Time-averaged errors for Figure~\ref{fig:real4_plot}.}
      \label{fig: real_data_table}
    \end{minipage}
\end{figure}

\rev{We perform additional quantitative benchmark testings in Appendix~\ref{append: Diff_Fluid} against a differentiable grid-based simulator on real and synthetic videos; and in Appendix~\ref{append: Additional} against 4 baselines on another synthetic video featuring different visual and dynamical distributions.}

\section{Conclusion \& Limitations}
In this work, we propose a novel data-driven system to perform fluid hidden dynamics inference and future prediction from single RGB videos, leveraging a novel, vortex latent space. The success of our method in synthetic and real data, both qualitatively and quantitatively, suggests the potential for embedding Lagrangian structures for fluid learning. Our method has several limitations. First, \rev{our} vortex model is \rev{currently} limited to 2D \rev{inviscid flow}. \rev{Extending to 3D, viscous flow is an exciting direction, which can be enabled by allowing vortex strengths and sizes to evolve in time \citep{mimeau2021review}.} Secondly, our vortex evolution did not take into account the boundary conditions in a physically-based manner, hence it cannot accurately predict flow details around a solid boundary. \rev{Incorporating learning-based boundary modeling may be an interesting exploration.} Thirdly, scaling our method to handle turbulence with multi-scale vortices remains to be explored. We consider two \rev{additional} directions for future work. First, we plan to explore the numerical accuracy of our neural vortex representation to improve the current vortex particle methods for scientific computing. Secondly, we plan to combine our differentiable simulator with neural rendering methods to \rev{synthesize} visually appealing simulations from 3D videos. 



\subsubsection*{Acknowledgments}
We thank all the anonymous reviewers for their constructive feedback. This work is in part supported by ONR MURI N00014-22-1-2740, NSF RI \#2211258, \#1919647, \#2106733, \#2144806, \#2153560, the Stanford Institute for Human-Centered AI (HAI), Google, and Qualcomm.




