\section{Experiments}
\label{sec:experiment}

In this section, we evaluate the performance of {\name} in learning various types of watermarks across diverse image distributions. Importantly, 
{\name} employs a simpler $G$ to acquire information about \textit{reconstruction hardness} for guiding watermark optimization, and learned watermarks are capable of resisting more advanced watermark removal techniques. These results confirmed the versatility of {\name}. 
% that have substantial differences to the backbone used in {\name}.



\subsection{Experiment Setup}



\textbf{{\name} Setup.}
Following~\citet{whang21solve,liu2023aipo}, we use representative normalizing flow model RealNVP~\citep{dinh2016density} as pre-trained on CelebA~\citep{liu2015faceattributes} as a reliable generative prior $G$ \citep{liu2020empirical}. 
% We followed~\citet{whang21solve} to pre-train $G$, to which we refer the readers for
More training details can be found in~\citet{whang21solve}. 
% more information about model architecture and technical details. 
% For similarity $s$, we 
We adopt peak-signal-to-ratio (PSNR) to measure similarity between reconstruction $\Tilde{\xv}$ and ground truth $\xv_T$ that {\name} seeks to minimize in Eq \eqref{eq:opt-wm}.


\textbf{Learnable Watermarks.}
We consider two families of learnable watermarks for empirical study. 
The \textbf{logo-styled} watermarks are simulated by MNIST digits~\citep{lecun1998gradient}, and we use all digits 0-9.
The \textbf{initial-styled} watermarks, on the other hand, are constructed from handwritten English letters~\citep{cohen2017emnist}, 
and we choose two randomly selected initials, ``NJ'' and ``OS''. 
All watermark generators are implemented by lightweight variational auto-encoder (VAE,~\citet{kingma2013auto}) using fully-connected layers and can be trained with CPU only. 
We provide more details in Appendix \ref{app:wm-details}. 



\textbf{Image Datasets.}
We consider three image sets to protect.
The \textbf{In-distribution} set is a validation subset of CelebA whereon $G$ was trained. 
This dataset helps understand the scenario where $G$ can be maintained by the copyright owners. 
For further evaluations of {\name} in scenarios where copyrighted images are not allowed to be used for training $G$, 
we consider two \textbf{out-of-distribution} sets:
a validation subset of ImageNet~\citep{deng2009imagenet}, and 10 manually selected Cartoon images. 
Due to budget constraints, on CelebA and ImageNet we randomly choose 100 images respectively, see Appendix \ref{app:data} for more details. 








\textbf{Watermark Removal Methods.}
After constructing \textit{hard-to-remove} watermark $\mv$, 
we conduct two classes of watermark removal methods. 
The first \textit{worst-case} class have access to the ground truth location of watermarks (i.e., exact $\Amat_m$ is assumed known) and remove them by solving inverse problems. To this end, \textbf{Flow-R} uses the same flow-based model $G$ to solve the inpainting task with random initialized $\xv$~\citep{liu2023aipo}, and \textbf{RePaint} is a representative diffusion model-based inpainting method~\citep{lugmayr2022repaint}. 
The second \textit{Blind-case} class contains \textbf{SLBR} \citep{liang2021visible} and \textbf{DeNet} \citep{sun2023denet}, which are blind watermark removal models that are pretrained on diverse images and watermarks to \textit{locate-and-remove} watermark in an end-to-end manner. 
% RePaint and SLBR use their official implementations along with pre-trained checkpoints. 
Notably, \textit{{\name} is not optimized for any of these methods.}
% In addition, we study \textbf{worst-case scenarios} on inpainting-based Flow-R and RePaint: they have access to the ground truth location of watermarks, 
% \footnote{We didn't find a way to provide watermark location to SLBR, so we used the orginal form.}.

% Notably, \textbf{Flow-R} can be thought gray-box 



\textbf{Evaluation Metrics.}
We evaluate the performance of {\name} based on the reconstruction quality of $\Tilde{\xv}(\mv_0)$ and $\Tilde{\xv}(\mv_T)$, where $\mv_0$ and $\mv_T$ denotes the initial and learned watermarks respectively. Following the literature~\citep{liang2021visible,lugmayr2022repaint}, the reconstruction quality is measured by peak-signal-to-ratio (PSNR),  structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS) \citep{zhang2018unreasonable}.
As will be detailed shortly, we manipulate the three metrics to make sure that 
higher indicates better reconstruction, and thus weaker copyright protection. 



\subsection{Quantitative Evaluation of {\name}}


When conducting watermark removal, 
We noted all of the four methods suffered from notable performance degradation in challenging scenarios. 
As an extreme case, SLBR and DeNet failed to recognize watermarks, and produced reconstruction nearly identical to the watermarked observation, as shown in Fig \ref{fig:main}.
Consequently, a direct comparison of PSNR and other metrics may fail to correctly measures the effectiveness of {\name}:
when a reconstruction $\Tilde{\xv}$ is identical to the observation $\yv$, metric $\text{PSNR}(\Tilde{\xv}, \xv_T) = \text{PSNR}(\yv, \xv_T)$ 
in essence quantifies \textit{how much watermark $\mv$ distorts the image}, other than \textit{how difficult it is to be removed}. 
% as these metrics quantify \textit{how much the watermark distorted the image}, other than \textit{how difficult it is to be removed}. 

To avoid this misleading evaluation, we check \textit{to what extent a reconstructed image is better than the watermarked observation} by computing how much PSNR or SSIM from a reconstruction to the ground truth is higher than from the observation. 
Specifically, we defined 
$v_\text{PSNR}(\xv) \triangleq \text{PSNR}(\Tilde{\xv}, \xv_T) - \text{PSNR}(\yv, \xv_T)$
as a measure of how good reconstruction $\Tilde{\xv}$ is in terms of PSNR, the measures of SSIM and LPIPS are defined similarly\footnote{As lower LPIPS implies higher similarity, we flip its subtraction order to make larger $v_\text{LPIPS}$ indicate better reconstruction. }. 
We report these results (mean$\pm$se) in Tab \ref{tab:quant}. 
Due to page limit, we defer the results from SLBR that failed on our watermarks to App \ref{app:results}. Original metrics are also reported in Tab \ref{tab:raw} in App \ref{app:results} for more comprehensive evaluation. 



From Tab \ref{tab:quant}, {\name} successfully learned watermarks resisting both flow- and diffusion-based worst-case methods,
Flow-R and RePaint, in all cases. 
Blind-case methods failed to identify added watermarks,
possibly due to the substantial style and semantic difference between our learned watermarks and their pre-trained data. This highlights the limitation of blind-case methods. 


When comparing the defense performance against the two worst-case methods, 
{\name} exhibited better performance on Flow-R than on RePaint. 
We hypothesize that this superior performance can be attributed to the fact that {\name} and Flow-R share the same generative prior $G$ and employ a similar \textit{maximum-a-posteriori} Bayesian optimization framework.
% We conjecture that this better performance was attributed to the fact that {\name} and Flow-R shared the same generative prior $G$, and a similar \textit{maximum-a-posterior} Bayesian formulation for optimization. 
In contrast, images reconstructed by RePaint undergo a significantly different optimization process. 
Conceptually, this distinction is similar to attacking a \textit{gray-box} model versus \textit{black-box} model~\citep{papernot2017practical}.
Furthermore, {\name} shows strong transferability in both scenarios, which are challenging for traditional adversarial attacks~\citep{demontis2019adversarial}.
We attribute this success to the fact that {\name}'s target, the \textit{hard-to-reconstruct region} of image $\xv_T$, 
is an intrinsic characteristic of the real $\xv_T$. Consequently, any generative model pre-trained on real images will inherently reflect this property.
% We consider this success is due to the fact that {\name}'s target, \textit{hard-to-reconstruct} region of image $\xv_T$, is a characteristic of the real $\xv_T$, therefore, any generative models pre-trained on real images will reflect this knowledge.
As a result, {\name} offers a general protection. 
In contrast, previous adversarial attack-based protections targeted on models-specific shortcuts that are not shared across different models, often resulting in unsatisfactory transferability~\citep{huang2020metapoison}. 


% Conceptually, This is  similar to gray-box attack in adversarial attack
% This trend echoes well with previous adversarial attack-based defense~\citep{}. 

As further evidence, although the generative prior $G$ was trained on CelebA, 
{\name} still offers comparable defense performance on out-of-distribution ImageNet and Cartoon datasets. 
As pointed in previous studies~\citep{asim2020invertible, whang21solve}, 
Flow as a generative prior provides a certain degree of generalizability across different image distributions for measuring the likelihood of an image. Our results further demonstrate that this flexibility can be leveraged to identify the \textit{hard-to-reconstruct region} in out-of-distribution images as well. 

% In conclusion, these quantitative results confirmed the effectiveness of {\name} for adding robust watermarks to images from a variety of different sources, that are hard to remove by existing watermark removing tools.


\input{subfiles/4_main_table}





\subsection{Qualitative Evaluation of {\name}}


We conclude this section by providing careful analysis on how {\name} learns watermarks in order to make them hard-to-remove. We visualize
reconstructions generated using different methods applied to random and {\name} learned watermarks. Results are shown in Fig \ref{fig:main}. 
Due to space constraints, one sample is presented for each watermark.	
% Due to page limit, one sample is included for each watermark. 

From Fig \ref{fig:main}, {\name} increased the difficulty of watermark removal while simultaneously preserving both image and watermark readability.
To achieve this, it selected regions with abundant visual details as {hard-to-reconstruct regions} to place watermarks. 
Importantly, we found these details are likely overlooked even by human readers.
For example, in three out of four CelebA images, {\name} placed watermarks along the boundaries between human hair and the background. 
These placements caused both Flow-R and RePaint to fail in accurately reconstructing the textures. 
Similarly, watermarks were put on leafy backgrounds on ImageNet images, leading to further failures of the two models. 

Interestingly, in column 8, both model failed to reconstruct the smaller lizard masked by the digit logo watermark. Furthermore, they both misinterpreted this lizard as part of the larger one. 
Given that Flow-R and RePaint employed generative priors of different architectures trained on distinct datasets, 
this ``coincidence'' can be considered as concrete evidence that the \textit{hard-to-reconstruct region} is an intrinsic characteristic of the image, learned by different generative priors trained in diverse scenes. 
By targeting this intrinsic characteristic, {\name} shows strong transferability.
% As a result, {\name} targeting on this characteristic of the image is able to provide strong transferability. 



\ExplSyntaxOn
% #1: prefix
% #2: args of \includegraphics
% #3: start index
% #4: num of images
\cs_set:Npn \loadimage #1#2#3#4 {
    \seq_clear:N \l_tmpa_seq
    \int_step_inline:nnn {#3} {#3 + #4 - 1} {
        \seq_put_right:Nx \l_tmpa_seq {
            \exp_not:N \includegraphics
                [\exp_not:n {#2}]
                {#1/##1.png}
        }
    }
    \seq_use:Nn \l_tmpa_seq {&}
}

% #1: image width
% #2: num of col
\cs_set:Npn \calctotalwidth #1#2 {
    \fp_eval:n {
        (#2) * (#1) + (#2 - 1) * (\tabcolsep)
    } pt
}
\ExplSyntaxOff



\begin{figure*}[htb!]
\centering
\resizebox{0.75\textwidth}{!}{
\renewcommand{\tabcolsep}{1.5pt}
\def\figwidth{0.07\linewidth}%
\newcommand{\authornote}[1]{
\adjustbox{rotate=90}{\parbox{\figwidth}{\small \bf \centering #1}}
}

\begin{tabular}{*{15}{c}}

% \toprule

\toprule[0.4ex]
& \multicolumn{4}{c}{\bf CelebA Samples} \vline
& \multicolumn{4}{c}{\bf ImageNet Samples} \vline
& \multicolumn{4}{c}{\bf Cartoon Samples} \\

% \cmidrule[0.2ex]{2-13}

% \cmidrule[0.2ex]{2-5} \cmidrule[0.2ex]{7-11} \cmidrule[0.2ex]{13-16}

% & \multicolumn{4}{c}{\fbox{\parbox{\calctotalwidth{\figwidth}{4}}{\footnotesize h\centering {CelebA Samples}}}} 
% & \multicolumn{4}{c}{\fbox{\parbox{\calctotalwidth{\figwidth}{4}}{\footnotesize \centering {ImageNet Samples}}}}
% & \multicolumn{4}{c}{\fbox{\parbox{\calctotalwidth{\figwidth}{4}}{\footnotesize \centering {Cartoon Samples}}}} \\
\noalign{\vskip 0.5ex}

\authornote{Clean}     &  
\loadimage{figures/main}{width=\figwidth}{0}{12}\\
\cmidrule[0.15ex]{2-13}
\multicolumn{13}{c}{\bf On Random Watermark}  \\
\cmidrule[0.15ex]{2-13}
% \multicolumn{13}{c}{\fbox{\parbox{\calctotalwidth{\figwidth}{13}}{\centering {Before Running {\name}, Random Watermark}}}} \\

\authornote{Obs}     &  
\loadimage{figures/main}{width=\figwidth}{12}{12}\\
\authornote{Inpaint}     &  
\loadimage{figures/main}{width=\figwidth}{24}{12}\\
\authornote{Flow-R}     &  
\loadimage{figures/main}{width=\figwidth}{36}{12}\\
\authornote{RePaint}     &  
\loadimage{figures/main}{width=\figwidth}{48}{12}\\
\authornote{SLBR}     &  
\loadimage{figures/main}{width=\figwidth}{60}{12}\\
\authornote{DeNet}     &  
\loadimage{figures/main}{width=\figwidth}{72}{12}\\

\cmidrule[0.15ex]{2-13}
\multicolumn{13}{c}{\bf On {\name} Watermark}  \\
\cmidrule[0.15ex]{2-13}

\authornote{Obs}     &  
% \loadimage{figures/main}{width=\figwidth}{84}{12}\\
\loadimage{figures/main}{width=\figwidth}{96}{12}\\
\authornote{Inpaint}     &  
\loadimage{figures/main}{width=\figwidth}{108}{12}\\
\authornote{Flow-R}     &  
\loadimage{figures/main}{width=\figwidth}{120}{12}\\
\authornote{RePaint}     &  
\loadimage{figures/main}{width=\figwidth}{132}{12}\\
\authornote{SLBR}     &  
\loadimage{figures/main}{width=\figwidth}{144}{12}\\
\authornote{DeNet}     &  
\loadimage{figures/main}{width=\figwidth}{156}{12}\\

% \cmidrule[0.2ex]{2-13}

& \multicolumn{10}{c}{\bf Digit WM} \vline
& \multicolumn{2}{c}{\bf Initial WM} \\


% & \multicolumn{10}{c}{\fbox{\parbox{\calctotalwidth{\figwidth}{10.24}}{\bf \footnotesize  \centering {DIGIT WM}}}}
% & \multicolumn{2}{c}{\fbox{\parbox{\calctotalwidth{\figwidth}{1.9}}{\bf  \footnotesize \centering {INITIAL WM}}}} \\

\bottomrule[0.4ex]


\end{tabular}
}
\caption{
Watermark removal performance of \textit{worse-case} Flow-R and RePaint, and \textit{blind-case} SLBR and DeNet. 
``Obs''and ``Inpaint'' show watermarked and surrogate inpainting images respectively.  
% Two worst-case methods have access to ground truth watermark locations. 
}
\label{fig:main}
\end{figure*}
