\documentclass[accepted]{uai2022}  % Accepted
%\documentclass[accepted]{uai2022} % after acceptance, for a revised
                                    % version; also before submission to
                                    % see how the non-anonymous paper
                                    % would look like
%% There is a class option to choose the math font
% \documentclass[mathfont=ptmx]{uai2022} % ptmx math instead of Computer
                                         % Modern (has noticable issues)
% \documentclass[mathfont=newtx]{uai2022} % newtx fonts (improves upon
                                          % ptmx; less tested, no support)
% NOTE: Only keep *one* line above as appropriate, as it will be replaced
%       automatically for papers to be published. Do not make any other
%       change above this note for an accepted version.

%% Choose your variant of English; be consistent
\usepackage[american]{babel}
% \usepackage[british]{babel}

%% Some suggested packages, as needed:
\usepackage{natbib} % has a nice set of citation styles and commands
    \bibliographystyle{plainnat}
    \renewcommand{\bibsection}{\subsubsection*{References}}
\usepackage{mathtools} 		% amsmath with fixes and additions
% \usepackage{siunitx} 		% for proper typesetting of numbers and units
\usepackage{booktabs} 		% commands to create good-looking tables
\usepackage{xr-hyper}		% References to main paper
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{amsfonts}		% more math
\usepackage{graphicx}		% figures
\usepackage{caption}		% tables
\usepackage{longtable}		% tables
\usepackage[ruled, linesnumbered, commentsnumbered]{algorithm2e}

%% Provided macros
% \smaller: Because the class footnote size is essentially LaTeX's \small,
%           redefining \footnotesize, we provide the original \footnotesize
%           using this macro.
%           (Use only sparingly, e.g., in drawings, as it is quite small.)

%% Self-defined macros
\newcommand{\swap}[3][-]{#3#1#2} % just an example

% Taken from Overleaf: 
% https://www.overleaf.com/learn/how-to/Cross_referencing_with_the_xr_package_in_Overleaf
\makeatletter
\newcommand*{\addFileDependency}[1]{% argument=file name and extension
  \typeout{(#1)}
  \@addtofilelist{#1}
  \IfFileExists{#1}{}{\typeout{No file #1.}}
}
\makeatother

\newcommand*{\myexternaldocument}[1]{%
    \externaldocument{#1}%
    \addFileDependency{#1.tex}%
    \addFileDependency{#1.aux}%
}

\externaldocument{kamulete_5}

\title{Test for non-negligible adverse shifts: Supplementary material}

% The standard author block has changed for UAI 2022 to provide
% more space for long author lists and allow for complex affiliations
%
% All author information is authomatically removed by the class for the
% anonymous submission version of your paper, so you can already add your
% information below.
%
% Add authors
\author{
	Vathy M. Kamulete
	\href{https://orcid.org/0000-0002-4451-3743}{\includegraphics[scale=0.06]{images/orcid.png}}
	\\
	Enterprise Model Risk Management \\
	Royal Bank of Canada \\
	Toronto, Canada \\
	\texttt{\href{mailto:vathy.kamulete@rbccm.com}{\nolinkurl{vathy.kamulete@rbccm.com}}} \\
}

  
  \begin{document}
  \onecolumn \maketitle



\section{Experiment Details}\label{setup}

For the empirical experiments, we use isolation forest from the \texttt{isotree}
package \citep{cortes2020package}. We take the default hyperparamters from
\texttt{isotree} as given, but increase the ensemble size to 500. To investigate
other notions of outlyingness, we use random forests from the \texttt{ranger}
package \citep{wright2015ranger}. We take its default hyperparameters from
\citet{probst2019tunability}.

All experiments were run on a commodity desktop computer with a 12-core
Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz processor and 64 GB RAM in R
version 3.6.1 (2019-07-05). We stress that no hyperparameter tuning was
performed - we set the hyperparameters to reasonable defaults as
previously discussed. To avoid ambiguity, we explicitly say when the
results pertain to \texttt{DSOS-SS}, \texttt{DSOS-PT} or
\texttt{DSOS-CV}.

\subsection{Simulated shifts}\label{gmm-simulation}

We simulate distribution shifts from a two-component multivariate Gaussian
mixture model (GMM). The training and test set are drawn from:

\begin{align*} \label{eq:simulated}
	X^{tr} \sim \phi^{tr} \cdot \mathcal{N}_{d}(\mathbf{\mu_1^{tr}}, \mathbf{\Sigma_1^{tr}}) + (1 - \phi^{tr}) \cdot \mathcal{N}_{d}(\mathbf{\mu_2^{tr}}, \mathbf{\Sigma_2^{tr}}) \\
	X^{te} \sim \phi^{te} \cdot \mathcal{N}_{d}(\mathbf{\mu_1^{te}}, \mathbf{\Sigma_1^{te}}) + (1 - \phi^{te}) \cdot \mathcal{N}_{d}(\mathbf{\mu_2^{te}}, \mathbf{\Sigma_2^{te}})
\end{align*}

Omitting subscripts and superscripts for brevity, \(\phi \in [0, 1]\),
\(\mathbf{\mu}\) and \(\mathbf{\Sigma}\) are the component weight, mean
vector and covariance matrix respectively. The baseline specifies the
training and test sample size
\(n^{tr} = n^{te} \in \{400, 800, 1600\}\), the number of dimensions
\(d \in \{4, 8, 16\}\), the component weights
\(\phi^{tr} = \phi^{te} = 0.5\), the mean vectors
\(\mathbf{\mu_1^{tr}} = \mathbf{\mu_1^{te}} = \mathbf{1}_{d}\) and
\(\mathbf{\mu_2^{tr}} = \mathbf{\mu_2^{te}} = -\mathbf{1}_{d}\) and the
covariance matrices
\(\mathbf{\Sigma_1^{tr}} = \mathbf{\Sigma_1^{te}} = \mathbf{\Sigma_2^{tr}} = \mathbf{\Sigma_2^{te}} = \mathbf{I}_{d}\).
\(\mathbf{I}_{d}\) is the \(d \times d\) identity matrix and
\(\mathbf{1}_{d}\) is the \(d\)-dimensional all-ones vector. The
baseline configurations enforce that training and test set are drawn
from the same distribution, i.e.~no shift. There is a total of 9 such
configurations (3 dimensions times 3 sample sizes).

We generally shift the distribution so that the dimension of change
\(d^{*} = 1\) is fixed as the ambient dimension \(d\) increases. The
power of multivariate tests based on kernels and distances decays with
increasing dimension when differences only exist along a few intrinsic
dimensions \(d^{*} \ll d\). We vary one or more
parameters, namely \(\phi^{tr}\), \(\phi^{te}\), \(\mathbf{\mu_2^{te}}\)
and \(\mathbf{\Sigma_2^{te}}\), to simulate the desired shifts, all else
constant. We change the following parameters to pre-set intensity
levels:

\begin{enumerate}
	\def\labelenumi{\arabic{enumi}.}
	\item
	Label (prior) shift -- We flip the weights so that
	\(\phi^{tr} \in \{0.49, 0.47, 0.45\}\) goes with
	\(\phi^{te} = 1 - \phi^{tr}\). The majority component in training
	becomes the minority in the test sample.
	\item
	Corrupted sample -- We draw a fraction
	\(\omega \in \{0.01, 0.02, 0.04\}\) of examples in the test set from
	the component that is absent in training such that \(\phi^{tr} = 1\)
	and \(\phi^{te} = 1 - \omega\).
	\item
	Mean shift -- We change the mean vector in the test set so that
	\(\mathbf{\mu_2^{te}} = [-\frac{\kappa}{10}, -\mathbf{1}_{d-1}]\),
	where \(\kappa \in \{11, 12, 14\}\).
	\item
	Noise shift -- We change the covariance matrix in the test set so that
	\(diag(\mathbf{\Sigma_1^{te}}) = [\frac{\theta}{10}, \mathbf{1}_{d-1}]\),
	where \(\theta\in \{11, 12, 14\}\) and \(diag(\cdot)\) is the
	assignment operator for the diagonal elements of the \(d\)-by-\(d\)
	covariance matrix.
	\item
	Dependency shift -- We induce a positive relationship between the
	first two covariates. We change the covariance structure in the test
	set so that
	\(\mathbf{\Sigma_2^{te}}[1, 2] = \mathbf{\Sigma_2^{te}}[2, 1] = \gamma\),
	where \(\gamma \in \{0.1, 0.2, 0.4\}\).
\end{enumerate}

There is a total of 27 configurations for each shift type. For each shift type
and shift intensity, we repeat the experiment 500 times as noted in the main
text.

\subsection{Partition-induced shifts}\label{cv-shift}

For each OpenML-CC18 dataset, we perform stratified 10-fold
cross-validation, repeated twice. We end up with 20 train-test splits
per task. In total, we run 3720 tests of no adverse shifts (62 datasets,
20 random splits, and 3 tests). Summary statistics for these datasets
and granular test results are bundled together in the submission package
for this paper. We expect the statistical tests to be correlated within but
not across datasets. To formalize this setup, we use the following model:

\begin{equation} \label{eq:cc18}
	\log s_i^j \sim \mathcal{N}(\mu_i^j, \Sigma_i^j)
\end{equation}

where for each dataset \(i = 1, 2, \cdots, 62\) and test
\(j = 1, 2, 3\), the \(s-\)value \(s_i^j\) is lognormally distributed:
positive and skewed to the right. The \(s-\)value \(s_i^j\) consists of
a dataset-specific (fixed) effect \(\mu_i^j\), subject to noise in
\(\Sigma_i^j\); \(\Sigma_i^j\) accounts for within-dataset covariance.

We fit the model in \ref{eq:cc18} using the \texttt{clubSandwich}
package to obtain robust estimates of the fixed effects \(\mu_i^j\) even
with arbitrary covariance structure in \(\Sigma_i^j\) left unspecified
\citep{pustejovsky2018small}. These fixed effects (means) \(\mu_i^j\)
in \ref{eq:cc18} measure how sensitive on average a dataset is to these
partition-induced shifts. The higher the value, the more susceptible a
dataset is to adverse shifts caused by sampling variation. As mentioned
in the paper, the exponentiated means \(\mu_i^j\) are on the \(s\)-value
scale and can be interpreted  as the strength of evidence against the null
of no adverse shift.

\bibliography{kamulete_5.bib}

\end{document}
