\section{Scalar Mechanisms}

We first consider Problem~\ref{prob:fl} when each input $x_i$ is a scalar in the interval $[0. 1]$, and the statistic $T$ is the average $\frac{1}{n}\sum_{i=1}^{n} x_i$. For simplicity, our server side aggregation protocol will also output an average of the client responses. Our goal now is to design a client-side mechanism $M$ that is $\epsilon$-local DP, unbiased, and can be encoded in $b$ bits. 

\subsection{Some Notation}

We begin with some notation. The inputs to our algorithms are a continuous value $x \in [0, 1]$, a privacy parameter $\epsilon$ and a communication budget $b$. We use the notation $B = 2^b$. The output of the client-side mechanism $M$ is a number $i \in \{0, \ldots, B - 1 \}$ which can be represented as a sequence of $b$ bits. Additionally, we have an alphabet $A = \{ a_0, \ldots, a_{B - 1}\}$ shared between the clients and server; a number $i$ transmitted by a client is decoded as the letter $a_i$ in $A$. The purpose of the alphabet $A$ is to ensure unbiasedness and hence asymptotic consistency.

Our proposed algorithms will have two phases. The first phase will use dithering to round $x$ into the grid $\{0, \frac{1}{B-1}, \ldots, 1\}$ while maintaining unbiasedness. The second phase will use a local differentially private mechanism to privately transmit the rounded value. 

We first observe that two existing local DP mechanisms -- Randomized Response~\cite{werner65} and RAPPOR~\cite{rappor} -- can be modified to adapt to this setting. We call the modified algorithms Unbiased Multiple Randomized Response and Unbiased RAPPOR. 

%\kc{Move this earlier to Sec 2.4?}
%The dithering procedure works as follows. If $x \in [\frac{k}{B-1}, \frac{k+1}{B-1}]$ where $0 \leq k \leq B - 2$, then, we select $Z = \frac{k}{B-1}$ with probability $(B-1)(x - \frac{k}{B-1})$, and $Z = \frac{k+1}{B-1}$ with probability $1 - (B-1)(x - \frac{k}{B-1})$. It is not too hard to see that this ensures that $\bbE[Z] = x$. Hence-forward, for the second phase of each algorithm, we will assume that the input lies in the set $\{0, \frac{1}{B-1}, \ldots, 1\}$. 

\subsection{Unbiased Randomized Response}

\paragraph{Randomized Response.} Randomized Response is one of the simplest local differentially private mechanisms that was proposed by~\cite{werner65} to sanitize a single $0/1$ response obtained in a survey. Given a bit $y \in \{ 0, 1\}$, the Randomized Response mechanism outputs the bit $y$ with some probability $p$ and the flipped bit $1 - y$ with probability $1 - p$. It can be shown that when $p = \frac{1}{1 + e^{-\epsilon}}$, then the response is $\epsilon$-local DP. 

\paragraph{Unbiased Multiple Randomized Response Mechanism.} The basic Randomized Response mechanism does not directly apply to our task for two reasons -- first, it is not unbiased, and second, it only applies to one bit. Fortunately, both limitations can be addressed quite easily. To address the first, we use an output alphabet of $\{ - \frac{1}{e^{\epsilon} - 1}, \frac{e^{\epsilon}}{e^{\epsilon} - 1} \}$ instead of $\{0, 1\}$, and to address the second, we repeat the mechanism $b$ times on each bit of $x$, with a privacy budget of $\epsilon/b$ each time. The full algorithm is in Algorithm~\ref{alg:umrr}.

\begin{algorithm}[t]
\caption{Unbiased Multiple Randomized Response}
\label{alg:umrr}

\begin{algorithmic}[1]
\STATE {\bf{Input:}} $x \in [0, 1]$, privacy budget $\epsilon$, communication budget $b$.
\STATE Let $B = 2^b$.
\STATE $z = \text{Dither}(x, \{ 0, 1/(B-1), \ldots, 1\})$.
\FOR{$j = 1, \ldots, b$}
	\STATE $z_j$ be bit $j$ of $(B-1)z$.
	\STATE Set $y_j = z_j$ with probability $ \frac{1}{1 + e^{-\epsilon/b}}$, $y_j = 1 - z_j$ otherwise.
	\STATE Set $t_j = a_0 + z (a_1 - a_0)$ where  $a_0 =  - \frac{1}{e^{\epsilon/b} - 1}$ and $a_1 =  \frac{e^{\epsilon/b}}{e^{\epsilon/b} - 1}$.
\ENDFOR
\STATE {\textbf{Return}} $(t_1, t_2, \ldots, t_b)$. 
\end{algorithmic}
\end{algorithm}

The following theorem, proved in the Appendix, illustrates that it satisfies $\epsilon$-local DP; additionally, it is unbiased, thus ensuring asymptotic consistency when used along with a server that averages the client responses.

\begin{theorem}\label{thm:umrr}
Algorithm~\ref{alg:umrr} satisfies $\epsilon$-local DP and is unbiased.
\end{theorem}

%Here, UnbiasedRR is the following unbiased version of the Randomized Response mechanism.

%\begin{enumerate}
%\item {\bf{Inputs:}} A bit $y \in {0, 1}$. Privacy parameter $\epsilon$.
%\item \begin{eqnarray*}
%z & = y, & \text{with probability} \frac{1}{1 + e^{-\epsilon}} \\
%& = 1 - y, & \text{otherwise}
%\end{eqnarray*}
%\item Output $a_0 + z (a_1 - a_0)$ where $a_0 =  - \frac{1}{e^{\epsilon} - 1}$ and $a_1 =  \frac{e^{\epsilon}}{e^{\epsilon} - 1}$.
%\end{enumerate}


\subsection{Unbiased RAPPOR}

\paragraph{RAPPOR.} RAPPOR is a simple local differentially private mechanism for transmitting a categorical value $x \in \{1, \ldots, K\}$. The main idea is to sanitize $x$ by transmitting it with some probability $p$, and a draw from an uniform distribution over $\{1, \ldots, K\}$ with probability $1 - p$. It can be shown that the algorithm satisfies $\epsilon$-differential privacy when $p = \frac{e^{\epsilon} - 1}{K + e^{\epsilon} - 1}$. 

\paragraph{Unbiased RAPPOR.} A plausible mechanism is to round the input $x$ by dithering to the grid $\{ 0, \frac{1}{B-1}, \ldots, 1\}$, and then transmit the resulting value using RAPPOR. However, the main problem with this is that the result will not be unbiased. To address this, we propose changing the alphabet of RAPPOR to $A = \{ a_0, a_1, \ldots, a_{B-1}\}$ such that unbiasedness is maintained. 

Specifically, to ensure unbiasedness, for any $i \in \{ 0, \ldots, B - 1 \}$, we need that when the input is $\frac{i}{B-1}$, the expected output is also $\frac{i}{B-1}$. Noting that here the parameter $K = B$, this reduces to the following equation:
\begin{equation} \label{eqn:airappor}
 a_i \cdot \frac{e^{\epsilon} - 1}{B + e^{\epsilon} - 1} + \sum_{j = 0}^{B-1} a_j \cdot \frac{B}{ B + e^{\epsilon} - 1} = \frac{i}{B-1} 
\end{equation}
Writing this down for each $i$ gives $B$ linear equations, solving which will give us the values of $a_0, \ldots, a_{B-1}$. The complete algorithm is shown in Algorithm~\ref{alg:rappor}. Theorem~\ref{thm:rappor} establishes its privacy and unbiasedness properties. 

\begin{algorithm}
\caption{Unbiased RAPPOR}
	\label{alg:rappor}
\begin{algorithmic}[1]
\STATE {\bf{Inputs:}} $x \in [0, 1]$, privacy budget $\epsilon$, communication budget $b$.
\STATE $z = \text{Dither}(x, \{ 0, 1/(B-1), \ldots, 1\})$.
\STATE Calculate $a_0, \ldots, a_{B-1}$ by solving Equation~\eqref{eqn:airappor}. 
\STATE Draw $z'$ from a mixture of $\delta_{(B-1)z}$ and $Unif\{0, 1, \ldots, B - 1\}$ with mixing weights $\frac{e^{\epsilon} - 1}{B + e^{\epsilon} - 1}$ and $\frac{B}{B + e^{\epsilon} - 1}$. 
\STATE {\bf{Return}}  $z'$.
\end{algorithmic}
\end{algorithm}

\begin{theorem}\label{thm:rappor}
Algorithm~\ref{alg:rappor} satisfies $\epsilon$-local DP and is unbiased.
\end{theorem}


\subsection{The \qp\ Mechanism}

A challenge with Unbiased Multiple RR and Unbiased RAPPOR is that both algorithms are not intrinsically designed for ordinal or numerical values, which may result in poor accuracy. We next propose a new method which attempts to improve the estimation accuracy by reducing the variance of each client's output while still retaining unbiasedness and hence asymptotic consistency. 

Our proposed method attempts to directly minimize the variance of the client's output; this is done by solving an optimization problem to find both an alphabet and a randomization mechanism. As usual, we begin with the standard dithering step, after which we have a value $z \in \{ 0, \frac{1}{B-1}, \ldots, 1 \}$. 

The client mechanism will output a letter from the alphabet $A = \{ a_0, a_1, \ldots, a_{B-1} \}$ where the values of $a_i$ are to be determined. Additionally, if $z = i (B-1)$, then it outputs $a_j$ with probability $p_{j, i}$. This gives us $B^2 + B$ variables -- the $a_j$s and the $p_{j, i}$s. Probability constraints ensure that for all $i$, we have:
\begin{equation} \label{eqn:qpprob}
	\sum_{j=0}^{B-1} p_{j, i} = 1, \quad p_{j, i} \geq 0 
\end{equation}
Additionally, differential privacy requires that for all $i \neq i'$ and for all $j$,
\begin{equation} \label{eqn:qpdp}
 p_{j, i'} e^{-\epsilon} \leq p_{j, i} \leq p_{j, i'} e^{\epsilon} 
\end{equation}
Finally, we require that the output is unbiased; in other words, when $z = \frac{i}{B-1}$, the expected value of the output is also $\frac{i}{B-1}$. This constraint can be encoded as the following equation that applies to every $i \in \{ 0, \ldots, B-1\}$. 
\begin{equation} \label{eqn:qpexp}
 \sum_{j=0}^{B-1} a_j p_{j, i} = \frac{i}{B-1}
\end{equation}
Any set of $a_j$s and $p_{j, i}$s that satisfy these constraints represent a feasible mechanism. To obtain the best one out of the feasible set, we propose to minimize the variance, written as follows:
\begin{equation} \label{eqn:qpvar}
\sum_{i=0}^{B-1} \sum_{j=0}^{B-1} p_{j, i} \left( \frac{i}{B-1} - a_j \right)^2 
\end{equation}
The final algorithm is stated in Algorithm~\ref{alg:qp}. Observe that by construction, it is $\epsilon$-local DP, and is unbiased.  

\begin{algorithm}
\label{alg:qp}
\caption{The \qp\ Mechanism}
	\begin{algorithmic}[1]
\STATE {\bf{Inputs:}} $x \in [0, 1]$, privacy budget $\epsilon$, communication budget $b$.
\STATE Let $B = 2^b$.
\STATE $z = \text{Dither}(x, \{ 0, 1/(B-1), \ldots, 1\})$.
\STATE Solve the optimization problem with objective~\eqref{eqn:qpvar} and constraints~\eqref{eqn:qpprob},\eqref{eqn:qpdp} and~\eqref{eqn:qpexp} to calculate $a_0, \ldots, a_{B-1}$ and probabilities $p_{j, i}$. 
\STATE If $z = \frac{i}{B-1}$ then set $z' = a_j$ with probability $p_{j, i}$.
\STATE {\bf{Return}} $z'$.
	\end{algorithmic}
\end{algorithm}

\paragraph{Solving the \qp.} \kc{paragraph to add for Mike}

%\paragraph{Metric DP.} The metric DP constraint is a slight modification of Equation \ref{eqn:qpdp}:
%\begin{equation} \label{eqn:qpdp_metric}
% p_{j, i'} e^{-\epsilon |i-i'| / (B-1)} \leq p_{j, i} \leq p_{j, i'} e^{\epsilon |i-i'| / (B-1)},
%\end{equation}
%where the level of privacy protection varies depending on the distance in input space. Alternatively, we can use a quadratic privacy penalty instead of the absolute penalty in Equation \ref{eqn:qpdp_metric}:
%\begin{equation} \label{eqn:qpdp_qmetric}
% p_{j, i'} e^{-\epsilon (i-i')^2 / (B-1)^2} \leq p_{j, i} \leq p_{j, i'} e^{\epsilon (i-i')^2 / (B-1)^2}.
%\end{equation}
%Doing so allows us to handle vector-valued inputs at a given $L_2$-sensitivity.
%Suppose that $\bx^{(0)}, \bx^{(1)} \in \{0,\frac{1}{B-1},\ldots,1\}^d$ with $\| \bx^{(0)} - \bx^{(1)} \|_2 \leq \Delta$ and let $\bz$ be the randomized output according to the optimal mechanism with (quadratic) metric DP condition (Equation \ref{eqn:qpdp_qmetric}). Then:
%\begin{align*}
%    \frac{\mathbb{P}(\bz | \bx^{(1)})}{\mathbb{P}(\bz | \bx^{(0)})} &= \prod_{i=1}^d \frac{\mathbb{P}(\bz_i | \bx^{(1)})}{\mathbb{P}(\bz_i | \bx^{(0)})} \\
%    &\leq \prod_{i=1}^d \exp(\epsilon (\bx^{(0)}_i - \bx^{(1)}_i)^2) \\
%    &= \exp \left( \epsilon \sum_{i=1}^d (\bx^{(0)}_i - \bx^{(1)}_i)^2 \right) \leq \exp(\epsilon \Delta^2),
%\end{align*}
%and similarly for the lower bound $\exp(-\epsilon \Delta^2) \leq \mathbb{P}(\bz | \bx^{(1)}) / \mathbb{P}(\bz | \bx^{(0)})$. Thus if the optimal mechanism is (quadratic) metric $\epsilon$-DP with respect to each coordinate of the vector $\bx$ then it is $\epsilon \Delta^2$-DP with respect to the vector $\bx$ at $L_2$-sensitivity $\Delta$.

\subsection{Extension to Metric-DP}

Metric DP is a variation of the differential privacy definition that 


\section{Extending to Vectors}

