\section{Introduction}
\label{sec:intro}
% Since the residual block forwarding structure \cite{he2016resnet} has been proposed, the neural network has ushered in rapid advancement. 
% %The most performant neural networks have reached the trillion level parameters \cite{openai2023gpt4}. 
% As it gets too mature, merely upgrading the neural network operator module or a number of neurons may not significantly further improve the neural network's performance. 
% Besides, the training and deployment of gigantic-scale neural networks require a lot of computing resources, energy, and maintenance costs. This situation will soon become unsustainable due to growing energy consumption. 
% This has encouraged researchers to turn to the cornerstone of neural networks — the way models are trained. 
% Although \cite{hinton2022forward} tried to use twice-forward passes instead of backpropagation to train the model, we may still rely on backpropagation for the long term.
% This encourages us not to ignore the various potential flaws (e.g., redundancy, overfitting, learned shortcut features) generated during model training.

Recently, machine learning has been increasingly questioned with regard to data privacy issues due to the increasing incidents of data leakage in practice. %In real life, one main concern is big data leakage. 
Some studies \cite{fredrikson2015model,song2017machine,carlini2019secret} have addressed that machine learning models tend to memorize the training data, and some techniques are able to even reconstruct those data \cite{salem2020updates, niv2022reconstructdata}. Research on deploying privacy protection solutions in machine learning models has become a pressing need to play a better role in privacy-sensitive applications. 

In machine learning, Membership Inference Attack (MIA) \cite{shokri2017membership} is one of the most important data inference attacks. In membership inference attacks, an attacker tries to determine whether a sample is a member of a target model's training set. MIAs try to develop a proxy to help the attacker distinguish if a sample is a `member' or `non-member.' Depending on specific MIAs' policies, the proxy can be a model or a threshold. In general, the attack's difficulty of MIAs depends on the learning task's difficulty of the target model. 

We observed that it is the \emph{discrepancy} in the prediction distribution of the model on member data and non-member data that leads to the leakage of privacy. Therefore, we conjecture that if the two prediction distributions coincide, the model will no longer leak membership information. 
In theory, a perfect model achieves perfect confidence and accuracy in both the training and testing sets.
However, it is challenging to train a model that perfectly conforms to the above conception under the current state of the art and resources. Hence, we slightly relax the constraint: while maintaining generalization ability, we would like to develop a model that tries to make the distributions of prediction on the training and testing sets as close as possible.

To achieve this goal, in this paper, we introduce a new training paradigm that can be effective against MIAs by enhancing model generalizability. Our approach is based on the insight that learning models usually show under-confidence and overconfidence in non-member and member data, respectively. Accordingly, our design goal is intuitive: making two distributions close to each other so that the model becomes neither overconfident nor underconfident. Additionally, our approach can easily be applied to train any classification model.
In summary, this paper makes the following contributions:
\begin{enumerate}
    \item We propose \texttt{CRL},  a novel defense mechanism, helping classification models gain more robust privacy protection capabilities.
    \item To the best of our knowledge, our approach is the first to enhance and maintain models' prediction confidence in nonmember data while mitigating overconfidence in the models' prediction distribution on member data.
    \item Through extensive evaluations, we empirically show that our approach outperforms existing defense mechanisms.
\end{enumerate}