{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Elq1CKh3N5IE"
   },
   "source": [
    "# Demo VirtualHome"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fr8V5spCN5IH"
   },
   "source": [
    "This is a demo of how to run VirtualHome UnitySimulator. The demo will walk though how to start an environment and visualize it, how to prepare it to perform activities and finally how to perform activities in them.\n",
    "\n",
    "<img src=https://raw.githubusercontent.com/xavierpuigf/virtualhome_unity/master/doc/assets/banner.gif />\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-RPaEKyXN9Ae"
   },
   "source": [
    "# Setup\n",
    "The code below is only needed if you are running the demo in colab. It installs a package to stream content on Google Colab.\n",
    "\n",
    "**Note:** Make sure you have GPU enabled if you are running in colab. \n",
    "\n",
    "Select Runtime > Change runtime type > Hardware accelerator: GPU\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LvYTf6n9N_gO",
    "outputId": "bfb50121-b4b7-4c86-a0a6-fa5829f3dd8a"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "if 'google.colab' in str(get_ipython()):\n",
    "    print('Running on CoLab')\n",
    "    osname = \"linux\"\n",
    "    !pip install git+https://github.com/xavierpuigf/colabstreamer\n",
    "    import colabstreamer\n",
    "    colabstreamer.config_all()\n",
    "    _xorg = colabstreamer.open_xorg()\n",
    "    # Clone VirtualHome\n",
    "    !git clone https://github.com/xavierpuigf/virtualhome.git\n",
    "    %cd /content/virtualhome\n",
    "    !pip install -r requirements.txt\n",
    "else:\n",
    "    %cd ..\n",
    "    from sys import platform\n",
    "    if platform == \"darwin\":\n",
    "        osname = \"macos\"\n",
    "    elif platform == \"linux\":\n",
    "        osname = \"linux\"\n",
    "    elif platform == \"windows\" or platform == 'win32':\n",
    "        osname = \"windows\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HdsOviOROaBf"
   },
   "source": [
    "## Download the simulator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eJssGx5kOZgu",
    "outputId": "ea6b6b44-9847-4eb8-dc38-0cb950ec06a6"
   },
   "outputs": [],
   "source": [
    "if not os.path.isfile(f\"{osname}_exec.zip\"):\n",
    "    ! wget http://virtual-home.org/release/simulator/v2.0/v2.3.0/linux_exec.zip\n",
    "    ! unzip -q linux_exec.zip\n",
    "%cd demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i4AG_awMOyLG"
   },
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UvwB2FFMOzbe"
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "import IPython.display\n",
    "import glob\n",
    "from utils_demo import *\n",
    "from sys import platform\n",
    "import sys\n",
    "from PIL import Image\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm import tqdm\n",
    "\n",
    "import virtualhome\n",
    "from unity_simulator.comm_unity import UnityCommunication\n",
    "from unity_simulator import utils_viz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jab1StRnN5II"
   },
   "source": [
    "# Starting communication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B0mcWBA9N5IJ"
   },
   "source": [
    "The first step is to start a communication with the simulator. Make sure before you run this that you have downloaded the simulator, and placed it under the `simulation` folder. You will be interacting with the simulator with the communication `comm` created here. You can include the file name of the simulator or just call `UnityCommunication()` and manually open the executable.\n",
    "\n",
    "Select `manual` if you are opening the executable separately, and `auto` if the unity executable is still not open.\n",
    "\n",
    "Remember that if you are running this in a headless server, you will need to start a display in a separate terminal using: \n",
    "\n",
    "```\n",
    "sudo python helper_scripts/startx $DISPLAY_NUM\n",
    "```\n",
    "\n",
    "This is not needed if you are running a colab notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "V1e_kN_sN5IJ",
    "outputId": "2645a019-c730-46e0-effa-f0634de34f68"
   },
   "outputs": [],
   "source": [
    "mode = 'auto' # auto / manual\n",
    "if mode == 'auto':\n",
    "    if platform == 'darwin':\n",
    "        exec_file = '../macos_exec*'\n",
    "    else:\n",
    "        exec_file = '../linux_exec*.x86_64'\n",
    "    file_names = glob.glob(exec_file)\n",
    "    if len(file_names) > 0:\n",
    "        file_name = file_names[0]\n",
    "        comm = UnityCommunication(file_name=file_name, port=\"8082\", x_display=\"0\")\n",
    "    else:\n",
    "        print(\"Error: executable path not found.\")\n",
    "else:\n",
    "    comm = UnityCommunication()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LPouJsE1N5IJ"
   },
   "source": [
    "# Starting and Visualizing Scenes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gDOAeyljN5IK"
   },
   "source": [
    "After initalizing the simulation. We can interact with the environments provided in VirtualHome. The simulator is composed of 50 human designed apartments, a sample of environments can be seen here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 258
    },
    "id": "W1aqfXaPN5IK",
    "outputId": "8034b867-3f59-4525-916e-86f4ca6909fb",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# The environments are numbered 0 to 50, let's visualize a few\n",
    "views = []\n",
    "for scene_id in tqdm(range(10)):\n",
    "    comm.reset(scene_id)\n",
    "    \n",
    "    # We will go over the line below later\n",
    "    comm.remove_terrain()\n",
    "    top_view = get_scene_cameras(comm, [-1])\n",
    "    views += top_view\n",
    "    \n",
    "IPython.display.display(display_grid_img(views, nrows=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GC5FTvmGN5IL"
   },
   "source": [
    "# Procedural Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DVuvEXewN5IL"
   },
   "source": [
    "VirtualHome also has support for procedural generation where we can generate completely new environments during runtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oUZ2UgaFN5IL",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "views = []\n",
    "for proc_gen_seed in tqdm(range(10)):\n",
    "    comm.procedural_generation(proc_gen_seed)\n",
    "    \n",
    "    # We will go over the line below later\n",
    "    comm.remove_terrain()\n",
    "    top_view = get_scene_cameras(comm, [-1])\n",
    "    views += top_view\n",
    "    \n",
    "IPython.display.display(display_grid_img(views, nrows=2)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-_sFFUhBN5IM"
   },
   "source": [
    "## Scene start and display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JcKOGV6ON5IM"
   },
   "source": [
    "We will start scene number 4 and visualize it from different views. We start it by calling reset. Scenes are numbered from 0 to 49."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "41DAoBDwN5IM",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comm.reset(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SWBghzyyN5IM"
   },
   "source": [
    "Each scene has multiple cameras, we will take screenshots for some of the cameras in this scene, specified by indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T-QyzDseN5IN",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "indices = [3, 32, -5, -1, -20, 15, 48, -8, 50, 17]\n",
    "img_final = display_scene_cameras(comm, indices, nrows=2)\n",
    "IPython.display.display(img_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I21p9ms8N5IN"
   },
   "source": [
    "## VirtualHome supports multiple modalities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1eJiJ9wnN5IN"
   },
   "source": [
    "The cameras can also display other modalities, such as semantic segmentation, depth, instance segmentation or optical flow when playing videos. We will display a few of those here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "D9FLzs_SN5IN"
   },
   "outputs": [],
   "source": [
    "indices = [-20, 1, 48, -8, 17]\n",
    "img_final = display_scene_modalities(\n",
    "    comm, \n",
    "    indices, \n",
    "    modalities=['normal', 'seg_class', 'seg_inst', 'depth', 'surf_normals'], nrows=5)\n",
    "IPython.display.display(img_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1g-769jJN5IO"
   },
   "source": [
    "## Including cameras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jMLmfDZnN5IO"
   },
   "source": [
    "You can also add new cameras in the scene and get screenshots from those"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RZdHuQ_aN5IO"
   },
   "outputs": [],
   "source": [
    "# specify the position and rotation of the camera\n",
    "comm.add_camera(position=[-3,1.8,-4], rotation=[20, 120, 0], field_view=60)\n",
    "\n",
    "# Get the last camera\n",
    "s, c = comm.camera_count()\n",
    "img_final = display_scene_cameras(comm, [c-1], nrows=1)\n",
    "IPython.display.display(img_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aG3PDQH7N5IO"
   },
   "source": [
    "We can also update existing cameras, here we will update the camera we just added"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4PhTHqeIN5IO"
   },
   "outputs": [],
   "source": [
    "# specify the position and rotation of the camera\n",
    "comm.update_camera(c-1, position=[-3,1.8,-4], rotation=[20, 120, 0], field_view=80)\n",
    "\n",
    "# Get the last camera\n",
    "s, c = comm.camera_count()\n",
    "img_final = display_scene_cameras(comm, [c-1], nrows=1)\n",
    "IPython.display.display(img_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ppePPEg-N5IQ"
   },
   "source": [
    "## Visualizing the scene as a graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D0-ZN01aN5IQ"
   },
   "source": [
    "Each scene in VirtualHome can be visualized as a graph, allowing to query the objects appearing, and their relationships. We start by obtaining the graph from the current scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S0rEAV2yN5IQ"
   },
   "outputs": [],
   "source": [
    "s, graph = comm.environment_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yWaCZEDRN5IQ"
   },
   "source": [
    "The graph is a dictionary with `nodes` and `edges`. Each node corresponds to an object and contains information such as.\n",
    "- class_name: the object_name\n",
    "- states: in which state the object is\n",
    "- id: a number you can use to perform actions over the object "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G-PyrMKYN5IR"
   },
   "source": [
    "Let's print one of the nodes, to see more of the information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rK0U8eqSN5IR"
   },
   "outputs": [],
   "source": [
    "graph['nodes'][140]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V54CKD49N5IR"
   },
   "source": [
    "The edges connect object ids with spatial relationships, such as `INSIDE`, `ON`, `CLOSE`. You can check more of them in the `simulation` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6RMKoOiVN5IR"
   },
   "outputs": [],
   "source": [
    "graph['edges'][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X3TpzZtcN5IR"
   },
   "source": [
    "The graph also contains bounding box and center information, which may be useful to reason about the environment layout."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xnzYR9sEN5IR"
   },
   "source": [
    "# Modifying your environment and preparing for activities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vZvRC_N4N5IR"
   },
   "source": [
    "In the previous section we viewed how to read and visualize the environment. Now we are interested in modifying the environment to perform activities in them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iIepBZWhN5IS"
   },
   "source": [
    "## Get default environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JH1-Ow5sN5IS"
   },
   "source": [
    "All the environments have a default setting. We can go to this setting by calling reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "x8hBRTi5N5IS"
   },
   "outputs": [],
   "source": [
    "comm.reset(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06sZZIwiN5IS"
   },
   "source": [
    "## Adding Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U8-yquU5N5IS"
   },
   "source": [
    "We will start by adding objects to interact with in the environments. We can start by adding a cat in the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MyViMtDrN5IS"
   },
   "source": [
    "### Adding a cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "atjSQFBEN5IS"
   },
   "source": [
    "We first want to make sure that the cat will be added in the current environment. Let's say that we want to add it in one of the sofas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mMO-9UqkN5IS"
   },
   "outputs": [],
   "source": [
    "imgs_prev = get_scene_cameras(comm, [-4])\n",
    "display_grid_img(imgs_prev, nrows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BFEL6SIbN5IT"
   },
   "source": [
    "We start by reading the graph and looking for one of the sodas in the scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TRpsdOeJN5IT"
   },
   "outputs": [],
   "source": [
    "success, graph = comm.environment_graph();\n",
    "sofa = find_nodes(graph, class_name='sofa')[-2]\n",
    "print(sofa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K8a745N9N5IT"
   },
   "source": [
    "We now add one node with id `1000` of type cat, and an edge between the sofa node and the cat, specifying that the cat should be on the sofa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gPNFt930N5IT"
   },
   "outputs": [],
   "source": [
    "add_node(graph, {'class_name': 'cat', \n",
    "                   'category': 'Animals', \n",
    "                   'id': 1000, \n",
    "                   'properties': [], \n",
    "                   'states': []})\n",
    "add_edge(graph, 1000, 'ON', sofa['id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o8o2_SeZN5IT"
   },
   "source": [
    "#### Update environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LMvZ0f3YN5IT"
   },
   "source": [
    "The graph is now updated, but now we have to call the simulator so that the environment gets updated with the graph. Let's do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W29Kocw-N5IT"
   },
   "outputs": [],
   "source": [
    "success, message = comm.expand_scene(graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "779G93RHN5IU"
   },
   "source": [
    "You can now take an image of the environment, you will see how there has been a cat added in the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bhzWIX5CN5IU"
   },
   "outputs": [],
   "source": [
    "imgs_final = get_scene_cameras(comm, [-4])\n",
    "display_grid_img(imgs_prev+imgs_final, nrows=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Xgv6yixUN5IU"
   },
   "outputs": [],
   "source": [
    "imgs_prev = imgs_final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ol-mDuAZN5IU"
   },
   "source": [
    "### Opening fridge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tn_nS_IDN5IU"
   },
   "source": [
    "We may not want to add any new object, but just to change the state of the current objects. We can do this very similarly, by changing the environment graph. Let's say we want to open the fridge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HyV7-2TyN5IU"
   },
   "source": [
    "We read again the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AwevJDHBN5IU",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "success, graph = comm.environment_graph();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sN_gfPN1N5IV"
   },
   "source": [
    "We find the node `fridge` and change its `states` to open. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kn0oa26HN5IV"
   },
   "outputs": [],
   "source": [
    "fridge = find_nodes(graph, class_name='fridge')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "84U8D0Y8N5IV"
   },
   "outputs": [],
   "source": [
    "fridge['states'] = ['OPEN']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i9hjys1dN5IV"
   },
   "source": [
    "We finally expand the graph, as we did before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g2wtU7a4N5IV"
   },
   "outputs": [],
   "source": [
    "success = comm.expand_scene(graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LUhaf6klN5IV"
   },
   "outputs": [],
   "source": [
    "imgs_final = get_scene_cameras(comm, [-4])\n",
    "display_grid_img(imgs_prev+imgs_final, nrows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H7pbG2qDN5IV"
   },
   "source": [
    "### Appliances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SP2Sb65XN5IV"
   },
   "source": [
    "We will use the same method as before to change the state of some appliances. Again just by modifying the state of the graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_XT6E3fzN5IV"
   },
   "source": [
    "We take a picture of the apartment to see how it looks before doing any change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NzzBo1r1N5IW"
   },
   "outputs": [],
   "source": [
    "indices = [0]\n",
    "imgs_prev = get_scene_cameras(comm, indices)\n",
    "display_grid_img(imgs_prev, nrows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IqrCi7bxN5IW"
   },
   "source": [
    "We now get the graph of the environment, and select a TV and a light"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "k7K69BNNN5IW"
   },
   "outputs": [],
   "source": [
    "success, graph = comm.environment_graph()\n",
    "prev_graph = graph\n",
    "tv_node = [x for x in graph['nodes'] if x['class_name'] == 'tv'][0]\n",
    "light_node = [x for x in graph['nodes'] if x['class_name'] == 'lightswitch'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rvUKDsp2N5IW"
   },
   "source": [
    "We change the state and modify the scene with the new graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "D1bupD9nN5IW"
   },
   "outputs": [],
   "source": [
    "tv_node['states'] = ['ON']\n",
    "light_node['states'] = ['OFF']\n",
    "_ = comm.expand_scene(graph)\n",
    "last_graph = graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bsgoRwzGN5IW"
   },
   "source": [
    "We visualize the final scene"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7_uO-2zAN5IW",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "imgs_final = get_scene_cameras(comm, indices)\n",
    "display_grid_img(imgs_prev+imgs_final, nrows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3khYQqbtN5IW"
   },
   "source": [
    "### Setting up time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "djcJnIuaN5IW"
   },
   "source": [
    "VirtualHome also includes a real-time management system based on the 24 hour system and assign tasks to agents depending on the time of day. Furthermore the time also affects the sun's position in the sky which has a direct effect on the outdoor lighting and indoor lighting through the windows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gq05DwrPN5IW"
   },
   "outputs": [],
   "source": [
    "comm.reset(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FW0pGJodN5IX"
   },
   "outputs": [],
   "source": [
    "views = []\n",
    "s, message = comm.add_camera(position=[-9.2,1.3,-3], rotation=[15, 130, 0], field_view=60)\n",
    "cam_id = int(message.split(':')[1])\n",
    "# Set time to 05:30 \n",
    "comm.set_time(hours=10, minutes=30, seconds=0)\n",
    "morning_view = get_scene_cameras(comm, [cam_id])\n",
    "views += morning_view\n",
    "\n",
    "# Set time to 8:30 \n",
    "comm.set_time(hours=15, minutes=30, seconds=0)\n",
    "day_view = get_scene_cameras(comm, [cam_id])\n",
    "views += day_view\n",
    "\n",
    "# Set time to 21:00 \n",
    "comm.set_time(hours=21, minutes=0, seconds=0)\n",
    "night_view = get_scene_cameras(comm, [cam_id])\n",
    "views += night_view\n",
    "    \n",
    "IPython.display.display(display_grid_img(views, nrows=1)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lVkeXDrQN5IX"
   },
   "source": [
    "We can also deactivate the time and the forests outside, if we want"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2Iztc6dzN5IX"
   },
   "outputs": [],
   "source": [
    "comm.reset(0)\n",
    "comm.remove_terrain()\n",
    "no_day_view = get_scene_cameras(comm, [17])\n",
    "IPython.display.display(display_grid_img(no_day_view, nrows=1)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ydMBom8LN5IX"
   },
   "source": [
    "# Generating Scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DfJm-yIgN5IX"
   },
   "source": [
    "We now can start scenes, visualize them and modify them. The last step is to perform activities in them. We do this by defining scripts: Lists of instructions that will be executed in sequence. Each instruction contains an action, an object, and an id. The id should match with the `id` of each of the nodes in the environment graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "npYF5X7WN5IX"
   },
   "source": [
    "You can check the list of actions currently implemented [here](https://github.com/xavierpuigf/virtualhome/tree/master/simulation#actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sc1rCBGdN5IX"
   },
   "source": [
    "### Adding a character"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WjfiAWOXN5IX"
   },
   "source": [
    "The first step is to add agents in the environment, that will be performing the activity. You can specify which agent you want to add and the room where you want to add it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "71ctIkMNN5IX"
   },
   "outputs": [],
   "source": [
    "comm.reset(4)\n",
    "tv_node['states'] = ['OFF']\n",
    "comm.expand_scene(prev_graph)\n",
    "comm.add_character('chars/Female2', initial_room='kitchen')\n",
    "s, g = comm.environment_graph()\n",
    "cat_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'cat'][0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FcqAWdIJN5IY"
   },
   "source": [
    "If you count the number of cameras, you will see that a few new cameras have been added into the scene. These are cameras attached to the character. When the character moves, the cameras will move as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1-yZ_SQkN5IY"
   },
   "outputs": [],
   "source": [
    "s, nc = comm.camera_count()\n",
    "indices = range(nc - 6, nc)\n",
    "imgs_prev = get_scene_cameras(comm, indices)\n",
    "display_grid_img(imgs_prev, nrows=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nQbdm9W1N5IY"
   },
   "source": [
    "## Generating the first script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NcLd8kosN5IY"
   },
   "source": [
    "Let's start by interacting witht the cat and the sofa that we set up before. The cat had id 1000. The sofa was stored in a variable `sofa` containing that node. We can query its id directly. This sequence will make the agent walk to the sofa, grab the cat and sit in the sofa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1EY9nbZ8N5IY"
   },
   "outputs": [],
   "source": [
    "script = ['<char0> [Walk] <sofa> ({})'.format(sofa['id']),\n",
    "          '<char0> [Find] <cat> ({})'.format(cat_id),\n",
    "          '<char0> [Grab] <cat> ({})'.format(cat_id),\n",
    "          '<char0> [Sit] <sofa> ({})'.format(sofa['id'])]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZM7Q12vIN5IY"
   },
   "source": [
    "We now want to execute the script in the environment. We do that through render_script. Notice that we can specify a file name, which will be used to save a video with the activity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GXE7EYKJN5IY"
   },
   "outputs": [],
   "source": [
    "success, message = comm.render_script(script=script,\n",
    "                                      processing_time_limit=60,\n",
    "                                      find_solution=False,\n",
    "                                      image_width=320,\n",
    "                                      image_height=240,  \n",
    "                                      skip_animation=False,\n",
    "                                      recording=True,\n",
    "                                      save_pose_data=True,\n",
    "                                      file_name_prefix='relax')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lleTVLeIN5Ia"
   },
   "source": [
    "This saves the frames of the video into the `Output/relax` folder, which should be where you had your executable. Let's generate a video from the frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gjUmQCOBN5Ia"
   },
   "outputs": [],
   "source": [
    "# Enter here the path to the video, it should be in the same location where you stored your executable \n",
    "path_video = \"./Output/\"\n",
    "utils_viz.generate_video(input_path=path_video, prefix='relax', output_path='.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8q7YTLwIN5Ia"
   },
   "outputs": [],
   "source": [
    "display_vid('./video_normal.mp4')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "65XofHrwN5Ib"
   },
   "source": [
    "You can specify camera names, for cameras that will follow the character or camera indices, for static cameras. To get the list of camera names, call character_cameras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "k9887wX5N5Ib"
   },
   "outputs": [],
   "source": [
    "comm.character_cameras()[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XAv1XsYRN5Ib"
   },
   "source": [
    "## Generating underspecified videos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zFuDX6rhN5Ic"
   },
   "source": [
    "If we do not care which objects the simulator should interact with, we can also let it decide. If we use the flag `find_solution=True` we can start enumerating objects by `1` instead of following the graph ids. Unity will try to find a solution. Note that if we want to interact with 2 objects of the same kind, they will need to have different ids (i.e. 1 and 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1Md-o4P2N5Ic",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "comm.reset(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "v2ZfmPdpN5Ic"
   },
   "outputs": [],
   "source": [
    "s, g = comm.environment_graph()\n",
    "[node for node in g['nodes'] if node['class_name'] == 'salmon']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CJGstzheN5Ic"
   },
   "outputs": [],
   "source": [
    "comm.add_character()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Jv3feYdpN5Ic"
   },
   "outputs": [],
   "source": [
    "script = ['<char0> [walk] <salmon> (1)',\n",
    "          '<char0> [grab] <salmon> (1)',\n",
    "          '<char0> [walk] <microwave> (1)',\n",
    "          '<char0> [open] <microwave> (1)',\n",
    "          '<char0> [putin] <salmon> (1) <microwave> (1)',\n",
    "          '<char0> [close] <microwave> (1)',\n",
    "          '<char0> [switchon] <microwave> (1)']\n",
    "          \n",
    "success, message = comm.render_script(script=script, \n",
    "                                      find_solution=True,\n",
    "                                      processing_time_limit=80,\n",
    "                                      frame_rate=15,\n",
    "                                      image_width=512, image_height=320,\n",
    "                                      skip_animation=False,\n",
    "                                      image_synthesis=['normal'],\n",
    "                                      camera_mode=['PERSON_FROM_BACK'],\n",
    "                                      recording=True,\n",
    "                                      file_name_prefix='milk')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5efHIy_HN5Ic"
   },
   "source": [
    "# Multi-agent Actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BYW6D8FVN5Ic"
   },
   "source": [
    "You can also generate actions with multiple agents in them, simply add more agents, and specify which agents should do which action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WayXfhd2N5Ic"
   },
   "outputs": [],
   "source": [
    "# Reset the scene\n",
    "comm.reset(0)\n",
    "s, g = comm.environment_graph()\n",
    "# Add two agents this time\n",
    "comm.add_character('Chars/Male2', initial_room='kitchen')\n",
    "comm.add_character('Chars/Female4', initial_room='bedroom')\n",
    "\n",
    "# Get nodes for salmon and microwave, glass, faucet and sink\n",
    "salmon_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'salmon'][0]\n",
    "microwave_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'microwave'][0]\n",
    "glass_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'waterglass'][-1]\n",
    "sink_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'sink'][0]\n",
    "faucet_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'faucet'][-1]\n",
    "\n",
    "\n",
    "# Put salmon in microwave\n",
    "script = [\n",
    "    '<char0> [walk] <salmon> ({}) | <char1> [walk] <glass> ({})'.format(salmon_id, glass_id),\n",
    "    '<char0> [grab] <salmon> ({}) | <char1> [grab] <glass> ({})'.format(salmon_id, glass_id),\n",
    "    '<char0> [open] <microwave> ({}) | <char1> [walk] <sink> ({})'.format(microwave_id, sink_id),\n",
    "    '<char0> [putin] <salmon> ({}) <microwave> ({}) | <char1> [putback] <glass> ({}) <sink> ({})'.format(salmon_id, microwave_id, glass_id, sink_id),\n",
    "    '<char0> [close] <microwave> ({}) | <char1> [switchon] <faucet> ({})'.format(microwave_id, faucet_id)\n",
    "]\n",
    "comm.render_script(script, frame_rate=10, camera_mode=[\"PERSON_FROM_BACK\"], recording=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZabO3ZjDN5Ic"
   },
   "source": [
    "# Interactive agents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "euFGRIbmN5Ic"
   },
   "source": [
    "So far we have seen how to generate videos, but we can use the same command to deploy or train agents in the environment. You can execute the previous instructions one by one, and get an observation or graph at every step. or that, you don't need to generate videos or have animations, since it will slow down your agents. Use `skip_animation=True` to generate actions without animating them. Remember to turn off the recording mode as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JMlhdT73N5Id"
   },
   "outputs": [],
   "source": [
    "# Reset the scene\n",
    "comm.reset(0)\n",
    "\n",
    "# Add two agents this time\n",
    "comm.add_character('Chars/Male2', initial_room='kitchen')\n",
    "comm.add_character('Chars/Female4', initial_room='bedroom')\n",
    "\n",
    "# Get nodes for salmon and microwave, glass, faucet and sink\n",
    "salmon_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'salmon'][0]\n",
    "microwave_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'microwave'][0]\n",
    "glass_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'waterglass'][-1]\n",
    "sink_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'sink'][0]\n",
    "faucet_id = [node['id'] for node in g['nodes'] if node['class_name'] == 'faucet'][-1]\n",
    "\n",
    "\n",
    "# Put salmon in microwave\n",
    "script = [\n",
    "    '<char0> [walk] <salmon> ({}) | <char1> [walk] <glass> ({})'.format(salmon_id, glass_id),\n",
    "    '<char0> [grab] <salmon> ({}) | <char1> [grab] <glass> ({})'.format(salmon_id, glass_id),\n",
    "    '<char0> [open] <microwave> ({}) | <char1> [walk] <sink> ({})'.format(microwave_id, sink_id),\n",
    "    '<char0> [putin] <salmon> ({}) <microwave> ({}) | <char1> [putback] <glass> ({}) <sink> ({})'.format(salmon_id, microwave_id, glass_id, sink_id),\n",
    "    '<char0> [close] <microwave> ({}) | <char1> [switchon] <faucet> ({})'.format(microwave_id, faucet_id)\n",
    "]\n",
    "\n",
    "s, cc = comm.camera_count()\n",
    "images = []\n",
    "for script_instruction in script:\n",
    "    print(script_instruction)\n",
    "    comm.render_script([script_instruction], recording=False, skip_animation=True)\n",
    "    # Here you can get an observation, for instance\n",
    "    s, im = comm.camera_image([cc-8], image_width=300, image_height=300)\n",
    "    images.append(im[0])\n",
    "    \n",
    "display_grid_img(images, nrows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iZoLqoeVN5Id"
   },
   "source": [
    "# Finer control"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xXnOjakTN5Id"
   },
   "source": [
    "We can also have finer control where objects go via action modifiers, the demo below shows how to place an object in different parts of the table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "46zuAup4N5Id"
   },
   "outputs": [],
   "source": [
    "comm.reset(0)\n",
    "s, g = comm.environment_graph()\n",
    "# Remove stuff from the table and add the character\n",
    "table = [node for node in g['nodes'] if node['class_name'] == 'kitchentable'][0]\n",
    "mouse = [node for node in g['nodes'] if node['class_name'] == 'mouse'][0]\n",
    "mouse_id = mouse['id']\n",
    "table_id = table['id']\n",
    "\n",
    "objects_on_table = [edge['from_id'] for edge in g['edges'] \n",
    "                    if edge['to_id'] == table['id'] and edge['relation_type'] == 'ON']\n",
    "new_graph = {\n",
    "    'nodes': [node for node in g['nodes'] if node['id'] not in objects_on_table],\n",
    "    'edges': [edge for edge in g['edges'] if edge['from_id'] not in objects_on_table and \n",
    "                                             edge['to_id'] not in objects_on_table]\n",
    "}\n",
    "\n",
    "comm.expand_scene(new_graph)\n",
    "comm.add_character()\n",
    "\n",
    "# This is to get a camera view on top of the character\n",
    "cc = comm.camera_count()[1] - 7\n",
    "\n",
    "\n",
    "# Start grabbing the mouse and go to the table\n",
    "script1 = [f'<char0> [grab] <mouse> ({mouse_id})', \n",
    "           f'<char0> [walk] <kitchentable> ({table_id})']\n",
    "s, m = comm.render_script(script1, skip_animation=True)\n",
    "\n",
    "images = []\n",
    "\n",
    "# Place the mouse in some position\n",
    "posx, posy = -1.1, -5.5\n",
    "script2 = [f'<char0> [putback] <mouse> ({mouse_id}) <kitchentable> ({table_id}) <position> {posx},{posy}']\n",
    "s, m = comm.render_script(script2, skip_animation=True)\n",
    "s, im1 = comm.camera_image([cc])\n",
    "images.append(im1[0])\n",
    "\n",
    "# Try a different positiom\n",
    "posx, posy = -1.3, -5.1\n",
    "script2 = [f'<char0> [grab] <mouse> ({mouse_id})',\n",
    "           f'<char0> [putback] <mouse> ({mouse_id}) <kitchentable> ({table_id}) <position> {posx},{posy}']\n",
    "s, m = comm.render_script(script2, skip_animation=True)\n",
    "s, im1 = comm.camera_image([cc])\n",
    "images.append(im1[0])\n",
    "\n",
    "display_grid_img(images, nrows=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WTDxT1PoN5Id"
   },
   "source": [
    "# Physics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GmcKofDAN5Id"
   },
   "source": [
    "VirtualHome is also able to simulate physics in the environments, and you can use the following api call to alter the gravitational force experienced in the environemnt. By activating physics, all objects behave as expected, just like in the real world. However you can also alter the \"g-force\" to make the agent feel like they are a astronaut in space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rC-ME5wEN5Id"
   },
   "outputs": [],
   "source": [
    "# Reset the scene and get the graph\n",
    "comm.reset(0)\n",
    "s, g = comm.environment_graph()\n",
    "\n",
    "# Add a agent \n",
    "comm.add_character('Chars/female2', initial_room='kitchen')\n",
    "\n",
    "# Get nodes for apple, desk, kitchen\n",
    "apple = [node['id'] for node in g['nodes'] if node['class_name'] == 'apple'][0]\n",
    "desk = [node['id'] for node in g['nodes'] if node['class_name'] == 'desk'][1]\n",
    "kitchen = [node['id'] for node in g['nodes'] if node['class_name'] == 'kitchen'][0]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "z6-KsiPpN5Id"
   },
   "outputs": [],
   "source": [
    "# Activate gravity\n",
    "comm.activate_physics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F5zYKoMcN5Ie"
   },
   "outputs": [],
   "source": [
    "# Put apple on desk\n",
    "script = [\n",
    "          '<char0> [grab] <apple> ({})'.format(apple),\n",
    "          '<char0> [put] <apple> ({}) <desk> ({})'.format(apple, desk),\n",
    "          '<char0> [walk] <kitchen> ({})'.format(kitchen),\n",
    "          '<char0> [grab] <apple> ({})'.format(apple),\n",
    "          '<char0> [put] <apple> ({}) <desk> ({}  <position> 0,-3)'.format(apple, desk),\n",
    "         ]\n",
    "success, message = comm.render_script(script=script[:2], \n",
    "                                      find_solution=True,\n",
    "                                      processing_time_limit=80,\n",
    "                                      frame_rate=15,\n",
    "                                      image_width=512, image_height=320,\n",
    "                                      skip_animation=False,\n",
    "                                      image_synthesis=['normal'],\n",
    "                                      camera_mode=['58'],\n",
    "                                      recording=True,\n",
    "                                      file_name_prefix='gravity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_video = \"./Output/\"\n",
    "utils_viz.generate_video(input_path=path_video, prefix='gravity', output_path='.')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "unity_demo.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
