{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Trusted-AI/AIF360/blob/master/examples/sklearn/demo_reject_option_classification.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reject Option based Classification\n",
    "\n",
    "Reject Option based Classification (ROC) is a simple but powerful post-processing method which works by simply assigning favorable outcomes to members of the unprivileged group and unfavorable outcomes to members of the privileged groups when the prediction probability from an estimator is within a \"critical region\" of ambiguity. We parameterize this critical region with a threshold (center) and margin (radius). We can then run a naive search over all combinations of these parameters to find the best tradeoff between predictive performance and fairness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Install AIF360\n",
    "!pip install 'aif360' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import tri\n",
    "from mpl_toolkits import mplot3d\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from aif360.sklearn.postprocessing import RejectOptionClassifierCV, PostProcessingMeta\n",
    "from aif360.sklearn.datasets import fetch_adult\n",
    "from aif360.algorithms.postprocessing.reject_option_classification import _get_pareto_frontier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fetch Adult dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y, sample_weight = fetch_adult()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encode protected attributes, labels, and categorical features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.index = pd.MultiIndex.from_arrays(X.index.codes, names=X.index.names)\n",
    "y.index = pd.MultiIndex.from_arrays(y.index.codes, names=y.index.names)\n",
    "y = pd.Series(y.factorize(sort=True)[0], index=y.index)\n",
    "X = pd.get_dummies(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine our ROC grid search with a logistic regression classifier using the meta-estimator class to complete the whole process in one step. Note: `RejectOptionClassifierCV` does not actually perform true cross-validation since this algorithm does not \"learn\" anything but this class will perform the grid search and, optionally, \"refit\" the post-processor using the optimal threshold and margin (best balanced accuracy given fairness score >= 0.8 for disparate impact or >= -0.1 for all other scorers). The `n_jobs` argument specifies the number of parallel processes to use (-1 means use all CPUs). By using the meta-estimator, we ensure the data used to fit the post-processor is a held-out validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'margin': 0.12000000000000001, 'prot_attr': 'sex', 'threshold': 0.18}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metric = 'disparate_impact'\n",
    "pp = PostProcessingMeta(LogisticRegression(solver='liblinear'),\n",
    "        RejectOptionClassifierCV('sex', scoring=metric, step=0.02, n_jobs=-1))\n",
    "pp.fit(X, y)\n",
    "pp.postprocessor_.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the results of the grid search in 2D and look at the Pareto front (cyan) for the tradeoff between balanced accuracy and disparate impact. The starred point represents the best params chosen by the refitting protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24112 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = pd.DataFrame(pp.postprocessor_.cv_results_)\n",
    "ax = res.plot.scatter('mean_test_'+metric, 'mean_test_bal_acc', c='gray', marker='.')\n",
    "\n",
    "all_pts = res[['mean_test_'+metric, 'mean_test_bal_acc']].to_numpy()\n",
    "pareto_idx = _get_pareto_frontier(all_pts)\n",
    "ax.scatter(all_pts[pareto_idx, 0], all_pts[pareto_idx, 1], c='cyan')\n",
    "\n",
    "refit_idx = pp.postprocessor_.best_index_\n",
    "ax.scatter(all_pts[refit_idx, 0], all_pts[refit_idx, 1], s=100, c='magenta', marker='*')\n",
    "\n",
    "ba = res.mean_test_bal_acc\n",
    "ax.fill_betweenx([ba.min()-0.05, ba.max()+0.05], 1, 0.8, color='g', alpha=0.1)\n",
    "ax.fill_betweenx([ba.min()-0.05, ba.max()+0.05], 0.8, color='r', alpha=0.1)\n",
    "ax.margins(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also visualize the 4D surface of our choices of threshold and margin and their corresponding balanced accuracy and disparate impact by plotting in 3D and using color to represent the fourth dimension. Again, the Pareto front points and \"best\" choice are shown in cyan and magenta, respectively (although they may be covered up by the surface plot from some angles). We can see that increasing the margin improves fairness but past a certain point it begins to overcompensate. We can also see that, in this case, lower thresholds generally lead to better balanced accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 61900 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f = plt.figure()\n",
    "ax = f.add_subplot(projection='3d')\n",
    "surf = ax.plot_trisurf(res.param_margin, res.param_threshold, res.mean_test_bal_acc, cmap='PiYG', alpha=0.75)\n",
    "triangles = tri.Triangulation(res.param_margin, res.param_threshold).triangles\n",
    "colors = res.mean_test_disparate_impact.to_numpy().take(triangles).max(axis=1)\n",
    "surf.set_array(colors)\n",
    "f.colorbar(surf, shrink=0.75, label='disparate impact');\n",
    "\n",
    "pareto_pts = res.iloc[pareto_idx]\n",
    "ax.scatter(pareto_pts.param_margin, pareto_pts.param_threshold, pareto_pts.mean_test_bal_acc, c='cyan')\n",
    "\n",
    "best_pt = res.loc[pp.postprocessor_.best_index_]\n",
    "ax.scatter(best_pt.param_margin, best_pt.param_threshold, best_pt.mean_test_bal_acc,\n",
    "           c='magenta', marker='*', s=100)\n",
    "\n",
    "ax.set_xlabel('margin')\n",
    "ax.set_ylabel('threshold')\n",
    "ax.set_zlabel('bal acc')\n",
    "ax.view_init(30, 30)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "d0c5ced7753e77a483fec8ff7063075635521cce6e0bd54998c8f174742209dd"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
