{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SANITY CHECK 1: MATCH MWU MATRIX WITH COMPARE RESULTS\n",
    "\n",
    "# load the related files\n",
    "import numpy as np\n",
    "import json\n",
    "\n",
    "eval_pair = \"0_0_1_2\"\n",
    "\n",
    "MWU_matrix = np.load('dating_app/evaluators/potential_date/MWU_matrix.npy')\n",
    "with open(f'dating_app/evaluators/potential_date/compare_results/{eval_pair}.json', 'r') as f:\n",
    "    compare_results = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'indices': [0, 0, 1, 2],\n",
       " 'responses': ['Reasoning: Both of these scenarios make Eric sound like a fun and interesting guy. However, I personally find Scenario 1 more appealing. In it, Eric comes across as more sophisticated and thoughtful with his words, showing a little bit more depth of character. He is also more open about his love for animals, his pursuit of education, and his view of life. His sense of humor is subtle, hinting at a multifaceted personality. This way, it feels like a more serious attempt at seeking connection, while still maintaining a light, playful tone. I do like tennis and animals, and the thought of a leisurely walk or friendly competition on the court appeals to me. Moreover, his invitation to explore life full of surprises sounds intriguing.\\n\\nAnswer: Scenario 1',\n",
       "  \"Reasoning:\\nBoth scenarios paint a pretty appealing picture. However, I would be more inclined to swipe right on scenario 1. In this scenario, Eric sounds more like a well-spoken, reflective, and dedicated individual who is very genuine about his love for tennis, university, and his canine companion. His interest in education and his preference for nature over city life seem to detail a personality that values intellectual growth and tranquility, both of which are attractive traits. Also, I appreciate Eric's mention of what he wants in a partner: someone who shares his enthusiasm for life, valuing education, participating in outdoor activities, and loving animals. This creates clarity about what he's seeking, which removes any ambiguity about his intentions.\\n\\nAnswer: Scenario 1\",\n",
       "  \"Answer: Scenario 1\\n\\nReasoning: The first scenario offers a considerably more detailed and well-rounded portrayal of Eric's personality, interests, and what he seeks in a potential romantic partner. He communicates an enthusiastic love for learning, sports, and animals, and also includes delightful anecdotes about his dog. His bio's humor and light tone make him appear more personable and approachable. He is specific about his interests and what he's looking for, which makes it easier to imagine what being involved with him might be like. The invitation at the end is quite thoughtful and open, serving as a soft launch into potential conversation. Therefore, this profile facilitates a more personal connection, increasing my interest and leading to a swiping right.\"],\n",
       " 'flipped': True,\n",
       " 'system_prompt': {'role': 'system',\n",
       "  'content': \"You are a young woman in your early 20s using a dating app, open to the possibility of forming a romantic relationship. You come across Eric's profile, an undergraduate student who likes tennis, hanging out with friends, and has a soft spot for animals. You are reading his bio and looking at his profile pictures, deciding whether to swipe right or left.\"},\n",
       " 'user_prompt': {'role': 'user',\n",
       "  'content': 'Scenario 1:\\n\"Hey there, I\\'m Eric! Currently busy flexing my brain in much-loved study as an undergraduate student at a beautiful, bustling state university. When I\\'m not buried in the books, you\\'ll probably find me either chasing a fluorescent yellow ball on the tennis court or chasing after my beagle, Scott, who has a knack for finding some real sniff-worthy spots. \\n\\nBeing quite the animal lover, I\\'d rather choose the sound of playful barks and chirps over the city\\'s hustle-and-bustle, and there\\'s nothing more relaxing than a leisurely walk with Scott.\\n\\nI\\'m looking for someone who shares my zest for life, appreciation for education, love of outdoor activities, and a soft spot for four-legged friends. If you can handle friendly competition on the tennis court and won\\'t mind having a fetching fur companion tagging along, then I\\'m eager to meet you. \\n\\nLife\\'s full of adventure and surprises, so why not explore it with me? And hey, let\\'s be honest, aren\\'t you curious to see if I\\'m as cute as my dog?\"\\n\\nScenario 2:\\n\"Hey there! I\\'m Eric, an undergraduate student at a big state university. I\\'m a big fan of whacking fuzzy green balls across the net (read: I love tennis), so if you\\'ve got a mean backhand or love cheering from the sidelines, we\\'ll get along just fine!\\n\\nBut that\\'s not all I\\'m about. I\\'m also an animal lover; you can often find me taking long walks with my beagle, Scott. He\\'s pretty good company, but I\\'m on the hunt for some human companionship - someone to share in my dog walking adventures. I hope I can find someone special to teach the secret tricks to my dog\\'s heart (caramel ice-cream, but you didn\\'t hear it from me 😉).\\n\\nIn case you were wondering, I\\'m about average in looks and height - but more than make up for it with my charm and sense of humor. Let\\'s grab a tennis match or coffee sometime and see if we hit it off!\"\\n\\nIn which scenario are you more likely to swipe right and potentially match with Eric? Please think step by step and explain your reasoning. \\nPut your response in the following format:\\n[reasoning text]\\n\\nAnswer: Scenario [1 or 2]'},\n",
       " 'parsed_answers': [1, 1, 1]}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compare_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MWU_matrix[compare_results['indices'][0], compare_results['indices'][1], \\\n",
    "           compare_results['indices'][2], compare_results['indices'][3]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notes: \n",
    "0_1 is better than 0_0 for all 3 ratings in potential_date\n",
    "\n",
    "1_1_2_1: XYZ no fakes, paving my way thanks for stopping by\n",
    "1_1_1_2: flipped ; XYZ no fakes, paving my way thanks for stopping by\n",
    "\n",
    "0_1_1_2: flipped; XYZ no fakes, fuzzy green balls\n",
    "\n",
    "\n",
    "\n",
    "duplicate queries -- it's not a bug, it's just being overwritten: 1_1_2_1 0.667 overwrites 1_1_1_2's 0 with 0.333"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5015697354995683"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# SANITY CHECK 2: CHECK POSITIONAL BIAS\n",
    "import os\n",
    "\n",
    "evaluators = ['dating_app/evaluators/potential_date', 'dating_app/evaluators/eric_himself', \\\n",
    "              'plane_crash/evaluators/general_public', 'plane_crash/evaluators/victims_families/', \\\n",
    "              'barista/evaluators/the_customer/', 'bargaining/evaluators/artisan/', \\\n",
    "              'product_launch/evaluators/companys_sales_and_marketing_team/', 'product_launch/evaluators/media_outlets/', \\\n",
    "              'product_launch/evaluators/potential_consumers']\n",
    "\n",
    "all_answers = []\n",
    "\n",
    "for evaluator in evaluators:\n",
    "    path = evaluator + '/compare_results/'\n",
    "    for compare_result in os.listdir(path):\n",
    "        with open(path + compare_result, 'r') as f:\n",
    "            compare_results = json.load(f)\n",
    "        \n",
    "        answers = compare_results['parsed_answers']\n",
    "        # if compare_results['flipped']:\n",
    "        #     answers = [1 - answer for answer in answers]\n",
    "        \n",
    "        for answer in answers:\n",
    "            all_answers.append(answer)\n",
    "\n",
    "np.mean(all_answers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "there is no positional bias between first and second scenarios. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4328752642706131\n",
      "0.4174242424242424\n"
     ]
    }
   ],
   "source": [
    "# RE-SANITY CHECK 2: CHECK POSITIONAL BIAS\n",
    "import os\n",
    "\n",
    "evaluators = ['plane_crash/evaluators/general_public', 'plane_crash/evaluators/victims_families/']\n",
    "\n",
    "all_single_answers = []\n",
    "all_redone_answers = []\n",
    "\n",
    "for evaluator in evaluators:\n",
    "    path = evaluator + '/compare_results/'\n",
    "    for compare_result in os.listdir(path):\n",
    "        with open(path + compare_result, 'r') as f:\n",
    "            compare_results = json.load(f)\n",
    "        \n",
    "        answers = compare_results['parsed_answers']\n",
    "        flipped = compare_results['flipped']\n",
    "\n",
    "        if type(flipped) == type(True):\n",
    "            if flipped:\n",
    "                answers = [1 - answer for answer in answers]\n",
    "            \n",
    "            for answer in answers:\n",
    "                all_single_answers.append(answers)\n",
    "\n",
    "        else:\n",
    "            if flipped[0]:\n",
    "                answers_first_three = [1 - answer for answer in answers[:3]]\n",
    "                answers_last_three = answers[3:]\n",
    "            else:\n",
    "                answers_first_three = answers[:3]\n",
    "                answers_last_three = [1 - answer for answer in answers[3:]]\n",
    "            \n",
    "            for answer in answers_first_three:\n",
    "                all_redone_answers.append(answer)\n",
    "            for answer in answers_last_three:\n",
    "                all_redone_answers.append(answer)\n",
    "\n",
    "print(np.mean(all_single_answers))\n",
    "print(np.mean(all_redone_answers))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
