{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7f5e2756-dae8-4791-b69e-7dad3d0b8da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224911bc-facf-4cfb-87d6-86a31b737e47",
   "metadata": {},
   "source": [
    "# Proof-of-Concept: Pretraining Language Models with Predicted and Fully Frozen Token Embedding Weights\n",
    "\n",
    "This notebook and accompanying code present a systematic empirical study of large language models (LLMs) trained with **predicted and entirely frozen input embedding weights** throughout the entire training phase. \n",
    "\n",
    "The key goal of this work is to demonstrate, as a proof-of-concept, that it is possible to train powerful transformer-based language models even when the token embedding matrix is:\n",
    "- **not initialized randomly or pretrained in the standard way**\n",
    "- but is instead **pre-computed by external means** (in our case: using Unicode visual features, subword patterns, or aggregated token text features)\n",
    "- and **remains absolutely frozen during all of training and finetuning**\n",
    "\n",
    "## Experimental Setup\n",
    "\n",
    "- **Pretrain dataset:** 9 billion tokens sampled from diverse English, Russian, and Chinese corpora (including Wikipedia, SQuAD_v2, TriviaQA, HotpotQA, NQ-Open, BoolQ, CommonsenseQA, AI2 ARC).\n",
    "- **Model architectures:** Transformer models trained with frozen predicted embeddings as well as baseline models with learnable (standard) embeddings. All other hyperparameters and datasets are equivalent.\n",
    "- **SFT (\"instruction-tuning\") data:** Only ~10% of the pretraining batches included SFT-style examples. The models remain primarily pretraining LMs, not instruction-tuned models.\n",
    "- **Frozen embeddings:** The token embeddings were constructed from external sources and kept frozen for 100% of pretraining and evaluation.\n",
    "- **Unfrozen baselines:** For comparison, models were trained from identical data with fully learnable embeddings in the classic manner.\n",
    "\n",
    "## Scientific Contribution\n",
    "\n",
    "The core scientific value of these experiments is to **provide the first strong empirical demonstration** that:\n",
    "- Large transformer LMs can learn and generalize even if the input embedding space contains no semantic or distributional information and is not updated during training;\n",
    "- The higher-level semantic and abstraction capabilities emerge in the transformer layers themselves, not primarily in the learned embedding table;\n",
    "- The gap in performance between frozen, predicted-embedding models and conventional (baseline) LMs is nonzero but limited, which has ramifications for modularity, model merging/fusion, and scaling.\n",
    "\n",
    "To our knowledge, **this is the first large-scale direct comparison of fully-frozen, externally-predicted embeddings vs. standard learnable embeddings on a modern LLM pretrain task** with all other factors held equal.\n",
    "\n",
    "These models are **pretrain-only** (not strong SFT/instruction-tuned LMs). \n",
    "Downstream/benchmarked metrics such as BLEU, MMLU, ARC, etc., should be interpreted accordingly: as indications of base model potential, **not as the upper bound of possible finetuned or instruction-following performance**.\n",
    "\n",
    "For context: Performance metrics of our frozen-embedding LMs are broadly comparable to strong GPT-2-class (100M–400M) baselines under similar pretrain-only settings.\n",
    "\n",
    "## What this is **not:**\n",
    "- Not a claim for SOTA on any downstream task.\n",
    "- Not an argument that no further improvement is possible with more compute or advanced tricks.\n",
    "- Not a replacement for full SFT/finetuning in LLM pipelines.\n",
    "\n",
    "## What this **is:**  \n",
    "**A proof-of-concept and key ablation establishing that high-capacity LMs can learn to encode language and world knowledge using only externally-predicted, fully-frozen token embeddings.**\n",
    "\n",
    "**All code, models, and metrics are provided for community validation and further research.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8da091ae-35cf-4aef-888e-33dee366081a",
   "metadata": {},
   "outputs": [],
   "source": [
    "prefixes = ['pro_bvv_en', \n",
    "            'pro_bvv_unfrozen',\n",
    "            #'pro_bvv_ru',\n",
    "            #'pro_bvv_zh', \n",
    "            'best_bvv_ru', \n",
    "            'best_bvv_unfrozen_ru', \n",
    "            'best_bvv_zh', \n",
    "            'best_bvv_unfrozen_zh',\n",
    "            #'best_bvv_moe', \n",
    "            'max_bvv_ru', \n",
    "            'max_bvv_zh', \n",
    "            #'max_bvv_moe', \n",
    "            'nemo_bvv_ru', \n",
    "            'nemo_bvv_zh', \n",
    "            #'nemo_bvv_moe'\n",
    "           ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5fe124df-5c1f-4d0f-b851-a01736de84dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_folder = '_results/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b0db1e2e-0398-4c16-a879-d2f5593869c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "files = [f for f in os.listdir(results_folder)\n",
    "         if any(f.endswith(fix+'.txt') for fix in prefixes)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "476b3330-3d5a-48a0-a01b-2e02a8fef265",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_benchmark_file(filepath):\n",
    "    with open(filepath, 'r', encoding='utf-8') as f:\n",
    "        text = f.read()\n",
    "\n",
    "    m = re.search(r'(^\\S+).*Total parameters:\\s*([0-9.]+[MB])', text, re.MULTILINE)\n",
    "    if m:\n",
    "        model_name = f\"{m.group(1)} [{m.group(2)}]\"\n",
    "    else:\n",
    "        model_name = os.path.basename(filepath)[:-4]\n",
    "\n",
    "    regex = re.compile(\n",
    "        r\"\\s\\S+\\s([A-Z\\-]+)(?: \\[([^\\]]+)\\])?:\\s+([0-9.]+)%\", re.MULTILINE)\n",
    "    data = []\n",
    "    for m in regex.finditer(text):\n",
    "        benchmark = m.group(1)\n",
    "        topic = m.group(2)\n",
    "        percent = float(m.group(3))\n",
    "        shortname = f'{benchmark}{\" [\"+topic+\"]\" if topic else \"\"}'\n",
    "        data.append({'model': model_name, 'benchmark': shortname, 'score': percent})\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "231bb76e-9730-47fe-a5e2-8213a8cb2e3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_results = []\n",
    "for file in files:\n",
    "    filepath = os.path.join(results_folder, file)\n",
    "    all_results.extend(parse_benchmark_file(filepath))\n",
    "\n",
    "df = pd.DataFrame(all_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0d0bd993-ca24-4046-be07-39d485abd490",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_comparison(df, benchmark_keyword='MMLU'):\n",
    "    # Можно фильтровать по нужному набору бенчмарков или задач\n",
    "    subdf = df[df['benchmark'].str.startswith(benchmark_keyword)]\n",
    "    plt.figure(figsize=(12, max(5, len(subdf['benchmark'].unique())*0.35)))\n",
    "    sns.barplot(\n",
    "        data=subdf,\n",
    "        x='score', y='benchmark', hue='model',\n",
    "        orient='h'\n",
    "    )\n",
    "    plt.xlabel(\"Score (%)\")\n",
    "    plt.ylabel(\"Benchmark/Task\")\n",
    "    plt.title(f\"{benchmark_keyword} comparison\")\n",
    "    plt.legend(title=\"Model\", bbox_to_anchor=(1.05,1), loc='upper left')\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d56d6465-a7d4-421b-8902-a8753c513002",
   "metadata": {},
   "outputs": [],
   "source": [
    "#plot_comparison(df, benchmark_keyword='BLEU')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7019c5eb-646f-4bf6-8dd3-bcc43867fdf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_heatmap(df, benchmark_startswith='MMLU'):\n",
    "    pivot = df[df['benchmark'].str.startswith(benchmark_startswith)]\n",
    "    table = pivot.pivot(index='benchmark', columns='model', values='score')\n",
    "    plt.figure(figsize=(1+1.8*len(table.columns), 1+0.42*len(table)))\n",
    "    sns.heatmap(table, annot=True, fmt=\".1f\", cmap='YlGnBu', linewidths=0.5,\n",
    "                cbar_kws=dict(label='Score (%)'), vmin=0, vmax=np.nanmax(table.values))\n",
    "    plt.title(f'{benchmark_startswith} heatmap')\n",
    "    plt.ylabel(\" \")#plt.ylabel(\"Benchmark/Task\")\n",
    "    plt.xlabel(\"Model\")\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "78ac7a5b-43c3-4e64-a719-126fb3d5fb7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 733112 characters]",
      "text/plain": [
       "<Figure size 1900x2536 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_heatmap(df, benchmark_startswith='MMLU')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a1075db3-124d-4c6c-80a4-ebc00afff3f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_heatmap_best_subtasks(df, score_threshold=26.0):\n",
    "    # Группируем по [benchmark, model], берём максимальный score по модели (на самом деле по файлу у тебя одна строка на пару model+subtask)\n",
    "    pivot = df.pivot(index='benchmark', columns='model', values='score')\n",
    "    # Фильтруем: выбрать те строки, где максимальное значение по строке превышает threshold\n",
    "    high_score_rows = pivot[pivot.max(axis=1) > score_threshold]\n",
    "    if high_score_rows.empty:\n",
    "        print(f\"Нет задач, где хотя бы одна модель >{score_threshold}%\")\n",
    "        return\n",
    "    plt.figure(figsize=(1+1.8*len(high_score_rows.columns), 1+0.42*len(high_score_rows)))\n",
    "    sns.heatmap(high_score_rows, annot=True, fmt=\".1f\", cmap='YlGnBu', linewidths=0.5,\n",
    "                cbar_kws=dict(label='Score (%)'), vmin=0, vmax=np.nanmax(high_score_rows.values))\n",
    "    plt.title(f\"Best results >{score_threshold}%\")\n",
    "    plt.ylabel(\" \")#plt.ylabel(\"Benchmark [Task]\")\n",
    "    plt.xlabel(\"Model\")\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fdbcec11-9548-4f14-91e0-15c652e3294a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 294868 characters]",
      "text/plain": [
       "<Figure size 1900x940 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_heatmap_best_subtasks(df, score_threshold=27.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "03feda24-7998-497a-bd5f-57fdd1df4b52",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_table_best_subtasks(df, score_threshold=27.0):\n",
    "    \"\"\"\n",
    "    Вывести таблицу (текстом), где максимальный score по хотя бы одной модели > threshold\n",
    "    \"\"\"\n",
    "    import pandas as pd\n",
    "    # Группировка и получение таблицы\n",
    "    pivot = df.pivot(index='benchmark', columns='model', values='score')\n",
    "    high_score_rows = pivot[pivot.max(axis=1) > score_threshold]\n",
    "    if high_score_rows.empty:\n",
    "        print(f\"Нет задач, где хотя бы одна модель >{score_threshold}%\")\n",
    "        return\n",
    "    \n",
    "    # Форматируем значения до 1 знака после запятой (как на heatmap)\n",
    "    formatted = high_score_rows.applymap(lambda x: f\"{x:.1f}\" if pd.notnull(x) else \"\")\n",
    "    \n",
    "    #try:\n",
    "    #    from tabulate import tabulate\n",
    "    #    print(\"Таблица лучших результатов (> {:.1f}%):\".format(score_threshold))\n",
    "    #    print(tabulate(formatted, headers='keys', tablefmt='pretty'))\n",
    "    #except ImportError:\n",
    "    #    print(\"tabulate не установлен, вот обычный вывод Pandas:\")\n",
    "        \n",
    "    print(formatted.to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8cbedcc9-add8-4745-bd5e-7e28ca06d468",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model                                      best_bvv_ru [0.5B] best_bvv_unfrozen_ru [0.5B] best_bvv_unfrozen_zh [0.5B] best_bvv_zh [0.5B] max_bvv_ru [0.4B] max_bvv_zh [0.4B] nemo_bvv_ru [0.4B] nemo_bvv_zh [0.4B] pro_bvv_en [0.2B] pro_bvv_unfrozen [0.2B]\n",
      "benchmark                                                                                                                                                                                                                                                   \n",
      "MMLU [astronomy]                                         22.6                         9.9                        15.2               20.3              27.1              27.3                8.0                5.1              23.0                    13.4\n",
      "MMLU [clinical_knowledge]                                25.7                        14.8                        20.6               23.0              26.6              28.1               16.0                8.3              24.6                    17.8\n",
      "MMLU [college_chemistry]                                 27.3                        13.5                        16.6               20.7              33.2              35.9                8.6                6.5              27.7                    11.9\n",
      "MMLU [college_computer_science]                          23.0                        14.8                        11.5               18.1              27.8              27.2                6.5                4.8              25.2                    13.2\n",
      "MMLU [college_mathematics]                               24.8                        11.8                        17.4               15.7              23.7              28.3                5.3                4.4              22.3                    12.0\n",
      "MMLU [college_medicine]                                  23.1                        13.1                        18.4               21.0              29.5              27.3               13.1                7.2              24.7                    15.4\n",
      "MMLU [college_physics]                                   28.2                        12.4                        11.6               15.7              28.4              28.8                5.7                5.3              27.6                    14.4\n",
      "MMLU [high_school_biology]                               27.2                        13.0                        20.0               23.0              29.5              29.2               12.4                8.8              25.9                    16.3\n",
      "MMLU [high_school_chemistry]                             26.4                        11.4                        16.8               19.4              28.0              25.7                9.2                6.1              23.6                    13.0\n",
      "MMLU [high_school_geography]                             26.6                        14.4                        17.4               24.3              29.0              28.0                9.9                8.3              25.7                    14.9\n",
      "MMLU [high_school_government_and_politics]               27.8                        14.1                        12.3               23.2              26.0              28.1                9.3                6.6              23.8                    12.9\n",
      "MMLU [high_school_macroeconomics]                        29.4                        19.3                        19.9               22.6              31.9              30.7               10.2                6.9              26.1                    16.0\n",
      "MMLU [high_school_microeconomics]                        26.5                        15.1                        19.0               21.3              30.1              31.1               11.2                7.9              26.2                    17.1\n",
      "MMLU [high_school_physics]                               24.9                        12.4                        14.2               17.4              29.6              28.3                7.0                4.4              23.8                    13.5\n",
      "MMLU [high_school_psychology]                            26.9                        14.5                        19.2               20.8              30.2              31.1               10.5                9.8              25.4                    15.8\n",
      "MMLU [high_school_statistics]                            31.9                        11.4                        14.1               21.9              36.0              33.5                8.8                6.2              25.7                    11.3\n",
      "MMLU [human_sexuality]                                   20.3                        11.5                        17.5               18.5              24.5              27.4               11.2               10.2              25.0                    15.4\n",
      "MMLU [management]                                        25.6                        14.0                        12.0               22.2              30.5              32.1                8.3                6.8              28.4                    13.6\n",
      "MMLU [nutrition]                                         24.4                        13.6                        17.9               20.2              27.6              26.7               10.7                8.7              23.4                    15.7\n",
      "MMLU [professional_medicine]                             32.3                        11.1                        16.1               20.7              37.9              33.3                9.0                7.9              25.8                    11.5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3430565/3970745291.py:14: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
      "  formatted = high_score_rows.applymap(lambda x: f\"{x:.1f}\" if pd.notnull(x) else \"\")\n"
     ]
    }
   ],
   "source": [
    "print_table_best_subtasks(df, score_threshold=27.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6124007c-ec7a-4deb-8fa6-01e8a78df3e3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "0bc568c1-f90c-4c87-a105-2241e84a2c74",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_overview_table(df):\n",
    "    # Если нужно включить и BLEU, и любые другие \"групповые\" метрики:\n",
    "    mask = (~df['benchmark'].str.contains(r'^MMLU \\[')) & (~df['benchmark'].str.startswith(\"BLEU\"))\n",
    "    overview = df[mask].copy()\n",
    "    overview = overview.sort_values(['benchmark', 'model'])\n",
    "    return overview\n",
    "    \n",
    "overview = get_overview_table(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "20844f34-34bb-46fe-b190-e857c8571062",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_overview_heatmap(overview):\n",
    "    table = overview.pivot(index='benchmark', columns='model', values='score')\n",
    "    plt.figure(figsize=(1+1.8*len(table.columns), 1+0.60*len(table)))\n",
    "    sns.heatmap(table, annot=True, fmt=\".2f\", cmap='YlGnBu', linewidths=0.5,\n",
    "                cbar_kws=dict(label='Score (%)'), vmin=0, vmax=np.nanmax(table.values))\n",
    "    plt.title(\"Benchmark by model\")\n",
    "    plt.ylabel(\" \")#\"Benchmark!\"\n",
    "    plt.xlabel(\"Model\")\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "df33bcdf-00ce-4afa-b224-4ad539ef536b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3185119/3660134052.py:9: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all Axes decorations.\n",
      "  plt.tight_layout()\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 57648 characters]",
      "text/plain": [
       "<Figure size 1180x280 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_overview_heatmap(overview)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c6920cb4-fdf9-4db3-8415-12614bf91f72",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_overview_table(overview):\n",
    "    \"\"\"\n",
    "    Вывести текстовую таблицу (без heatmap), аналогичную overview heatmap.\n",
    "    \"\"\"\n",
    "    import pandas as pd\n",
    "    # Формируем таблицу, как в heatmap\n",
    "    table = overview.pivot(index='benchmark', columns='model', values='score')\n",
    "    # Форматируем значения до 2 знаков после запятой (как на heatmap)\n",
    "    formatted = table.applymap(lambda x: f\"{x:.2f}\" if pd.notnull(x) else \"\")\n",
    "    \n",
    "    try:\n",
    "        from tabulate import tabulate\n",
    "        print(\"Benchmark by model:\")\n",
    "        print(tabulate(formatted, headers='keys', tablefmt='pretty'))\n",
    "    except ImportError:\n",
    "        print(\"tabulate не установлен, вот обычный вывод Pandas:\")\n",
    "        print(formatted.to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9633cd4f-140f-458a-88a2-b101c68fc1fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Benchmark by model:\n",
      "+--------------+--------------------+-----------------------------+-----------------------------+--------------------+-------------------+-------------------+--------------------+--------------------+-------------------+-------------------+-------------------------+-------------------+\n",
      "|  benchmark   | best_bvv_ru [0.5B] | best_bvv_unfrozen_ru [0.5B] | best_bvv_unfrozen_zh [0.5B] | best_bvv_zh [0.5B] | max_bvv_ru [0.4B] | max_bvv_zh [0.4B] | nemo_bvv_ru [0.4B] | nemo_bvv_zh [0.4B] | pro_bvv_en [0.2B] | pro_bvv_ru [0.2B] | pro_bvv_unfrozen [0.2B] | pro_bvv_zh [0.2B] |\n",
      "+--------------+--------------------+-----------------------------+-----------------------------+--------------------+-------------------+-------------------+--------------------+--------------------+-------------------+-------------------+-------------------------+-------------------+\n",
      "| BLEU [en-ru] |        6.44        |            8.40             |            4.29             |        3.15        |       8.65        |       1.28        |        4.68        |        2.05        |       0.37        |       6.14        |          0.68           |       2.82        |\n",
      "| BLEU [en-zh] |        0.95        |            0.77             |            1.65             |        1.41        |       0.52        |       2.36        |        0.67        |        0.87        |       0.28        |       0.68        |          0.16           |       1.32        |\n",
      "| BLEU [ru-en] |        8.80        |            7.96             |            2.90             |        3.04        |       8.77        |       3.47        |        5.71        |        2.31        |       0.45        |       8.07        |          0.81           |       2.26        |\n",
      "| BLEU [zh-en] |        0.74        |            1.09             |            5.93             |        7.78        |       0.81        |       7.44        |        0.89        |        4.05        |       0.06        |       1.07        |          0.42           |       4.65        |\n",
      "|   C-SENSE    |       20.13        |            18.79            |            18.98            |       19.62        |       19.65       |       20.21       |       19.51        |       19.27        |       19.54       |       20.26       |          19.76          |       18.51       |\n",
      "|     MMLU     |       22.29        |            11.37            |            14.03            |       19.42        |       23.58       |       24.72       |        8.80        |        6.62        |       23.68       |       22.63       |          14.00          |       17.96       |\n",
      "|    SQUAD     |       14.84        |            13.55            |            13.52            |       15.12        |       18.95       |       17.66       |        6.80        |        6.05        |       9.61        |       10.94       |          13.28          |       5.59        |\n",
      "+--------------+--------------------+-----------------------------+-----------------------------+--------------------+-------------------+-------------------+--------------------+--------------------+-------------------+-------------------+-------------------------+-------------------+\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3123075/1875485470.py:9: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
      "  formatted = table.applymap(lambda x: f\"{x:.2f}\" if pd.notnull(x) else \"\")\n"
     ]
    }
   ],
   "source": [
    "print_overview_table(overview)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cb38caa-ed06-45fa-a5fb-cf4b680b546d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3_11",
   "language": "python",
   "name": "python3_11"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
