

\section{Methodology}
The ideal goal is to make the model privacy-safe with no or little generalizability loss. To achieve the goal, we consider of employing RelaxLoss and CenterLoss. RelaxLoss \cite{chen2022relaxloss} can mitigate MIAs by reducing distinguishability between the member and non-member loss distributions. CenterLoss \cite{wen2016centerloss} can enhance the discriminative power of the deeply learned features. The benefits of incorporating both of them are i) it helps to keep the discriminative power of deep features while ``relaxing'' the model; ii) both of them do not require additional training information or data sources; iii) low training costs and no modifications to the model's inference in the evaluation phase make them easy to apply in most scenarios.

As we describe above, they bring exclusive advantages. However, those advantages are not utilized by simply combining RelaxLoss and CenterLoss. Hence, we propose CRL to utilize their advantages harmoniously.


\begin{figure}[t]
     \centering
     \includegraphics[width=1.\linewidth]{figs/d2b_d2o_4_0.png}
\caption{Relationship between the distance to the origin and the distance to the decision boundary in Cross entropy, RelaxLoss, and our proposed approach, CRL. The blue points are Member data while the green points are non-member data. We compute the distance to the decision boundary by subtracting 2nd largest confidence from the 1st largest confidence. Therefore, more overlap between Member and Non-member is better for privacy.}
\label{fig:scatter}
\end{figure}


\subsection{Mechanism of RelaxLoss}
The insight of relaxed loss function (RelaxLoss) \cite{chen2022relaxloss} is to adjust the fitting degree of the model on the training set through mini-batches. RelaxLoss sets three stages for achieving this goal. The algorithm can be formulated as follows:
\begin{equation}
\begin{aligned}
%\footnotesize
%\mathcal{L}_{rce} = 
\mathtt{Relax} &\mathtt{Loss} (y, p, \alpha_{rce}, epoch) \\ = 
& \begin{cases}
\mathcal{L}_{ce}, & \text{if } \mathcal{L}_{ce}>\alpha_{rce}, \\
|\mathcal{L}_{ce} - \alpha_{rce}|, & \text{else if epoch}  \% 2 = 0, \\
\mathcal{L}_{sce}, & \text{otherwise}\\
\end{cases}
\end{aligned}
\end{equation}
where $\mathcal{L}_{ce}$ denotes cross-entropy loss function, $\alpha_{rce}$ denotes the threshold hyper-parameter, $y$ denotes the ground-truth, $p$ denotes the prediction probabilities, and $\mathcal{L}_{sce}$ denotes the soft cross-entropy loss function formulated later in Eq.~\ref{eq:sce} (without logits normalization). RelaxLoss sets a threshold, $\alpha_{rce}$, to judge if the model should become more fitting on the mini-batch sample. When $\mathcal{L}_{ce}$ is below $\alpha_{rce}$, RelaxLoss relieves the fitting degree in two ways. The first case, which only happens in an even number of epochs, reverses the cross-entropy's gradient direction so the model degenerates to the threshold. 
%The other case takes the soft cross-entropy loss function to fine-tune the training samples' prediction distribution to prevent the model from merely returning to the state before further fit.
The other case takes the soft cross-entropy loss function to enable the model to further fine-tune its prediction distribution. The combination of these two cases prevents the model from merely returning to the state before further fitting.





\subsection{Mechanism of Center Loss}
As an auxiliary loss function, the center loss \cite{wen2016centerloss} does not act directly on the output layer. Instead, it assumes that a classification model can summarize each class' general features. In detail, each class has a feature vector or a feature map so that the sample is similar to the feature vector or map corresponding to its class after the model processes the sample. To find a set of class centers, we randomly generate a set of vectors, $\{c\}_{i=1}^{C}$, for $C$ classes. Then, we compute the distance between the sample's output of the intermediate layer (usually the second or the global pooling layer) and the corresponding center is,
\begin{equation}
%\small
    \mathcal{L}_{ct} = -\frac{1}{\mathcal{|M|}}\sum_{(x,y)\in \mathcal{M}}^{}\frac{1}{2}\|f_e(x;\theta_e) - c_{y}\|_2^2
\end{equation}
where $\|\cdot\|_2$ is the function of euclidean distance, $\mathcal{M}$ denotes the set of a mini-batch, and $y$ is the ground-truth of the input $x$. And $f_e(\cdot;\theta_e)$ is a model without the classifier part. Representations of samples from the same class are encouraged to approach each other during training. 
The intuitive idea of why we choose center loss is that the model is more likely to make similar predictions when features in the representation space are closer to their class members. 
%If two inputs are close to each other in the representation space, they would be able to get similar predictions from the model.
In other words, we try to bring this mechanism to privacy protection so that the member and non-member data of the model overlap more in the representation space, shown in \texttt{CRL} in Fig.~\ref{fig:scatter}, to obtain more consistent predictions.
However, it does not mitigate the problem that the model is too confident in predicting training samples. We improve it in the next subsection to make it capable of this problem.


\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figs/relaxcenterloss.pdf}
    \caption{An overview of the proposed relaxed center loss function. The mini-batch determines which scenario to execute.}
    \label{fig:relax_center_loss}
\end{figure}






\subsection{Center-Based Relaxed Learning Method (\texttt{CRL})}
%We notice that RelaxLoss has a conspicuous flaw: it cannot change the shortcoming that some predictions of training samples are easy to be overconfident. 
%As a current SOTA method that does not increase model size or data costs, 
RelaxLoss shows outstanding performance considering tradeoffs between models' generalizability and the prediction distribution gap between training data and testing data, which leads to the behavioral differences of the model on the training members and non-members.
However, we observe that the cross-entropy loss function is prone to make the model overconfident in some training samples. To ameliorate the impact of the issue, as a part of our approach, \texttt{CRL}, we propose an improved relaxed loss (\texttt{ImpRelaxLoss}), which is inspired by \cite{wei2022logitnorm}. 
First, we define the model's function without softmax as $f(\cdot;\theta)$, and the parameters $\theta$ is a superset of $\theta_e$. 
Then, we revisit the softmax probabilities and define the normalized probabilities:
\begin{equation}
%\small
p_i = \frac{e^{g_i}}{\sum_{j=1}^{C} e^{g_j}}, \qquad p_{i, norm} = \frac{e^{g_i / (1 + \tau_{rce} \|g\|_2 )}}{\sum_{j=1}^{C} e^{g_j / (1 + \tau_{rce} \|g\|_2 )}}
\end{equation}
%\begin{equation}
%\small
%p_{i, norm} = \frac{e^{g_i / (1 + \tau_{rce} \|g_i\|_2 )}}{\sum_{j=1}^{C} e^{g_j / (1 + \tau_{rce} \|g_j\|_2 )}}
%\end{equation}
where $g = f(x;\theta)$, and $\tau_{rce}$ is a scaling factor to control the degree of how much the predicting probabilities are normalized. One different point is that we add $1$ to $\|g\|_2$ to ensure the denominator is always greater than or equal to $1$. Normalization can amplify the loss of difficult samples more so that the model preferentially focuses on difficult samples, intensifying the goal of RelaxLoss.
Next, we compute the cross-entropy loss with logit-normalized probabilities: 
\begin{equation}
%\small
\mathcal{L}_{lce} = -\frac{1}{\mathcal{|M|}}\sum_{(x,y)\in \mathcal{M}}^{}\sum_{i=1}^{C} y_i \mathrm{log}(p_{i, norm}) 
%= -\frac{1}{B}\sum_{i=1}^{B}  \mathrm{log}(p_{i, norm})
\end{equation}
where $\mathcal{M}$ is the set of a mini-batch, and $p_{i, norm}$ is the normalized probability of the $i$-th class for each sample $y$ in the mini-batch.
Afterward, we compute the soft label for the loss function. 
We change the non-normalized probabilities to produce soft label, $p_{tar}$. The probabilities are averaged except the probabilities of the corresponding class: %we keep the ground-truth class's corresponding probabilities and average the of non-positive classes.
\begin{equation}
%\small
p_{i, tar} = 
\begin{cases}
p_y, & i = y, \\
(1-p_y)/(C-1), & i\neq y 
\end{cases}
\end{equation}
where $p_{i, tar}$ is the $i$-th class probability of the soft label $p_{tar}$.
Then, we compute the soft cross-entropy loss as follows:
\begin{equation}
%\small
    \mathcal{L}_{sce} = -\frac{1}{\mathcal{|M|}}\sum_{(x,y)\in \mathcal{M}}^{}\sum_{i=1}^{C} p_{i, tar} \mathrm{log}(p_{i, norm})
    %-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{C}p_{j_i, tar} \mathrm{log}(p_{i, norm})
\label{eq:sce}
\end{equation}


\noindent According to the size difference of $\mathcal{L}_{lce}$ and $\alpha_{rce}$, the \texttt{ImpRelaxLoss} function is formulated as follows:
\begin{equation}
\begin{aligned}
%\footnotesize
%\mathcal{L}_{rce} = 
\mathtt{ImpRelax} &\mathtt{Loss} (y, p, \alpha_{rce}, \tau_{rcl}, epoch) \\ = 
& \begin{cases}
|\mathcal{L}_{lce} - \alpha_{rce}|, & \text{if epoch}  \% 2 = 0, \\
\mathcal{L}_{lce}, & \text{else if } \mathcal{L}_{lce}>\alpha_{rce}, \\
\mathcal{L}_{sce}, & \text{otherwise}\\
\end{cases}
\end{aligned}
\end{equation}
Then, \texttt{ImpRelaxLoss} is assigned to $L_{rce}$ (Line 20 in Algorithm~\ref{alg:CRLalgorithm}.)



\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{figs/crl.pdf}
    \caption{An overview of CRL's training and testing phases. The parameters of centers are a part of the loss function but not a part of the model.}
    \label{fig:crl}
\end{figure}


For the next part of \texttt{CRL}, we introduce the \emph{Relaxed center loss} function of which overview is described in Fig.~\ref{fig:relax_center_loss}. Similar to \texttt{RelaxLoss}, there are three scenarios in the Relaxed center loss. To determine which scenario to execute, we use the epoch index and distance to the class center (similar to what vanilla center loss does) at the mini-batch level as a metric. 



\begin{algorithm}[t!]
\small
\caption{\makebox[.73\linewidth][s]{Center-Based Relaxed Learning method (\texttt{CRL})}}
\label{alg:CRLalgorithm}
\textbf{Input}: 
Training Dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N}$ in a random order, training epochs $E$, model learning rates $\tau$, class centers learning rates $\tau_c$, mini-batch size $B$, number of output classes $C$, improved relaxed loss function's threshold value $\alpha_{rce}$, relaxed center loss function's threshold value $\alpha_{rcl}$, normalized factor $\tau_{rce}$ and $\tau_{rcl}$, the joint loss adjustment coefficient $\lambda$;
\\
\textbf{Parameter}: 
Model's encoder part parameters $\theta_{e}$, classifier part parameters $\theta_{c}$, class centers' parameter $\{c\}_{i=1}^{C}$;
\\
\textbf{Output}: Model $f(\cdot;\theta)$ (inclusive of encoder $f_e(\cdot;\theta_e)$ and classifier $f_c(\cdot;\theta_c)$) with parameters $\theta$ (both $\theta_{e}$ and $\theta_{c}$ are inclusive);
\begin{algorithmic}[1] %[1] enables line numbers
\STATE Randomly initialize the model's parameters $\theta$ and class centers' parameters $\{c\}_{i=1}^{C}$
\FOR{$epoch$ \textbf{in} $\{1, 2, \cdots, E\}$}
\REPEAT
\STATE Sample a mini-batch $\{(x_j, y_j)\}_{j=1}^{B}$ from $\mathcal{D}$
\STATE /* Perform forward pass */ 
\STATE $q_j = f_e(x_j;\theta_e)$, $p_j = f_c(q_j;\theta_c)$
\STATE $q_{j,norm} = \frac{q_{j}}{1 + \tau_{rcl}\|q_j\|_2}$, $c_{y_j,norm} = \frac{c_{y_j}}{1 + \tau_{rcl}\|c_{y_j}\|_2}$
\STATE /* Compute relaxed center loss */
\STATE $\mathcal{L}_{ct} = \sum_{j=1}^{B}\|q_{j,norm} - c_{y_j,norm}\|_2^2 / 2B$
\IF{$epoch \% 2 = 0$}
\STATE $\mathcal{L}_{rcl} = |\mathcal{L}_{ct} - \alpha_{rcl}|$
\ELSIF{$\mathcal{L}_{ct} > \alpha_{rcl}$}
\STATE $\mathcal{L}_{rcl} = \mathcal{L}_{ct}$
\ELSE
\STATE $t_{j, y} = p_{y_j}$ \hfill // confidence of the true class
\STATE $t_{j, o} = 1 - p_{y_j}$ 
\STATE $\mathcal{L}_{rcl} = \sum_{j=1}^{B}[t_{j, y}\|q_{j,norm} - c_{y_j,norm}\|_2^2 +t_{j, o}\| q_{j,norm} \|_2^2 ]/2B$
\ENDIF
\STATE /* Compute improved relaxed loss */ 
\STATE $\mathcal{L}_{rce} = \mathrm{ImpRelaxLoss}(y_j, p_j, \alpha_{rce}, \tau_{rcl}, epoch)$
\STATE /* Compute total loss */ 
\STATE $\mathcal{L} = \mathcal{L}_{rce} + \lambda \mathcal{L}_{rcl}$
\STATE /* Update model's and centers' parameters */
\STATE $c_{y_j} \leftarrow c_{y_j} - \tau_c \nabla \mathcal{L}_{rcl}$
\STATE $\theta \leftarrow \theta - \tau \nabla \mathcal{L}$
\UNTIL{all training samples are sampled in this $epoch$}
\ENDFOR
\end{algorithmic}
\end{algorithm}


First, we set a distance boundary $\alpha_{rcl}$. The boundary is a hypersphere since we use Euclid distance as a measuring metric.
The simplest scenario is shown in Fig.~\ref{fig:relax_center_loss}~(a): when the average distance of the mini-batch to corresponding centers is larger than the boundary, the centers and samples are driven to close each other.
The second scenario illustrated in Fig.~\ref{fig:relax_center_loss}~(b) occurs only when the index of the current epoch is even. In this scenario, a boundary of distance to centers is defined as a hyperplane in which all points have the same specific distance to the corresponding class centers. It aims to keep the samples around the boundary of distance to their class centers to prevent the collapse of the classifier caused by excessive relaxation when merely relaxing cross entropy. %the ineffectiveness of RelaxLoss .
The last scenario shown in Fig.~\ref{fig:relax_center_loss}~(c) prevents the center and samples from staying too far from the zero point. 

\emph{Unlike} RelaxLoss (Cross-Entropy part), relaxed center loss encourages the samples to stay around the connection line between the class center and the origin in this scenario, which implicitly reconstructs the training sample representation's magnitude and direction, helping more relaxed magnitude and narrower angle. In the relaxation process, narrowing the angle helps better generalization \cite{liu2016largemargin}. As shown in Fig.~\ref{fig:scatter}, scenario 3 helps the model's member and non-member samples distribution become sharper than the other two methods.
The details are presented in Algorithm~\ref{alg:CRLalgorithm}.



As a result, \texttt{ImpRelaxLoss} and Relaxed center loss compose \texttt{CRL}. The overview of \texttt{CRL} is shown in Fig.~\ref{fig:crl} and the algorithm is described in Algorithm~\ref{alg:CRLalgorithm}. %Inputs $x$ are fed into the model. 
Through the encoder part of the model, the input gets deep features, then the deep features and their corresponding class centers are normalized. Afterward, the logits produced by the model are also normalized. We then use the logits to compute $\mathcal{L}_{rce}$ and $\mathcal{L}_{rcl}$, respectively. Hyper-parameter $\lambda$ controls the balance between the two losses. Another feature of CRL is that the two components do not execute the corresponding scenarios simultaneously since they have respective thresholds $\alpha_{rl}$ and $\alpha_{rcl}$. In other words, the model can relax the cross entropy loss while maintaining a certain degree of inter-class aggregation in the representation space so that the model can better keep the model's generalizability.


