{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c078adf3-b73f-4d4a-a90a-c39743f7c93a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import tiktoken\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from datasets import get_dataset_config_names, load_dataset\n",
    "from transformers import AutoTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d021511c-d452-4136-a092-9af5e3e49eb9",
   "metadata": {},
   "source": [
    "## Introduction: The bvv241* Tokenizer Architecture\n",
    "\n",
    "This work introduces the **bvv241** tokenizer family, a set of open-source multilingual tokenization schemes designed to maximize transparency, Unicode coverage, and extensibility. All implementations are available on HuggingFace for public access and reproducibility:\n",
    "\n",
    "- [bvv241-2-3](https://huggingface.co/Bochkov/bvv241-2-3)\n",
    "- [bvv241-max](https://huggingface.co/Bochkov/bvv241-max)\n",
    "- [bvv241-nemo](https://huggingface.co/Bochkov/bvv241-nemo)\n",
    "- [bvv241-abs](https://huggingface.co/Bochkov/bvv241-abs)\n",
    "\n",
    "The core approach is based on bijective mapping of Unicode codepoints to individual tokens, whenever possible, ensuring a 1:1 relationship between characters and tokens across a wide range of scripts. Reserved, surrogate, and private Unicode ranges are systematically exploited to store custom and multi-character tokens, allowing for efficient representation of commonly occurring substrings and entire words—particularly beneficial for high-frequency terms shared across models and languages.\n",
    "\n",
    "The tokenizer’s hybrid design draws from frequent multi-character patterns identified in top-performing, state-of-the-art models: approximately 19,000 such tokens are incorporated. This structure enables robust handling of diverse languages and scripts, mitigating the fragmentation common in byte-pair encoding (BPE) and unigram tokenizers—especially for underrepresented scripts and code-mixed text.\n",
    "\n",
    "All bvv241* tokenizer variants, vocabularies, and evaluation tools are publicly available, promoting transparency and collaborative research in multilingual NLP scenarios.\n",
    "\n",
    "Demonstration Variant: bvv241-nemo\n",
    "- [bvv241-nemo](https://huggingface.co/Bochkov/bvv241-nemo)\n",
    "\n",
    "The bvv241-nemo variant is provided as a demonstration of the flexibility of the proposed methodology. While the underlying Unicode-centric mapping scheme is applicable to any tokenizer, bvv241-nemo specifically replicates the vocabulary, token structure, and exact token enumeration of the SOTA [Mistral Nemo tokenizer](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407), using its public vocabulary files as a template. This showcases how the approach can serve as a drop-in replacement or augmentation for existing BPE or unigram schemes.\n",
    "\n",
    "Furthermore, since all token indices and their corresponding Unicode/codepoint representations are deterministic and exhaustively specified, this methodology enables the direct creation of a pre-initialized embedding matrix of shape vocab_size × n_embed. Such a matrix can be either used as a fixed (non-trainable) embedding or as a high-quality cold start, reducing the need for extensive embedding pre-training.\n",
    "\n",
    "The code and trained vocabularies for bvv241-nemo are publicly available for reproducibility and to foster further experimentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29899f02-83da-466b-a4ff-38253e240a35",
   "metadata": {},
   "source": [
    "## The code below demonstrates how to benchmark tokenizer efficiency and generate comparative visualizations for different languages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "175664d7-6fb8-425c-9398-2e67edd5219e",
   "metadata": {},
   "outputs": [],
   "source": [
    "languages =     [\"en\", \"ru\", \"fr\", \"de\", \"zh\", \"he\", \"it\", \"es\", \"ar\", \"pt\", \"ko\", \"hu\", \"sa\", \"la\", \"ja\", \"el\",     \"sv\", \"nl\", \"pl\", \"vi\", \"fa\", \"no\", \"tr\", \"fi\", \"cs\", \"hy\", \"da\", \"bn\", \"az\", \"ka\", \"hi\", \"id\"]\n",
    "\n",
    "lang_name_map = {\n",
    "    \"en\": \"English\",\n",
    "    \"ru\": \"Russian\",\n",
    "    \"fr\": \"French\",\n",
    "    \"de\": \"German\",\n",
    "    \"zh\": \"Chinese\",\n",
    "    \"he\": \"Hebrew\",\n",
    "    \"it\": \"Italian\",\n",
    "    \"es\": \"Spanish\",\n",
    "    \"ar\": \"Arabic\",\n",
    "    \"pt\": \"Portuguese\",\n",
    "    \"ko\": \"Korean\",\n",
    "    \"hu\": \"Hungarian\",\n",
    "    \"sa\": \"Sanskrit\",\n",
    "    \"la\": \"Latin\",\n",
    "    \"ja\": \"Japanese\",\n",
    "    \"el\": \"Greek\",\n",
    "\n",
    "    \"sv\": \"Swedish\",\n",
    "    \"nl\": \"Dutch\",\n",
    "    \"pl\": \"Polish\",\n",
    "    \"vi\": \"Vietnamese\",\n",
    "    \"fa\": \"Persian\",\n",
    "    \"no\": \"Norwegian\",\n",
    "    \"tr\": \"Turkish\",\n",
    "    \"fi\": \"Finnish\",\n",
    "    \"cs\": \"Czech\",\n",
    "    \"hy\": \"Armenian\",\n",
    "    \"da\": \"Danish\",\n",
    "    \"bn\": \"Bangla\",\n",
    "    \"az\": \"Azerbaijani\",\n",
    "    \"ka\": \"Georgian\",\n",
    "    \"hi\": \"Hindi\",\n",
    "    \"id\": \"Indonesian\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c645ad23-8810-4b02-a3b5-b999c552a16a",
   "metadata": {},
   "outputs": [],
   "source": [
    "tiktoken_gpt2_tok = tiktoken.get_encoding(\"gpt2\")\n",
    "tiktoken_cl100k_tok = tiktoken.get_encoding(\"cl100k_base\")\n",
    "\n",
    "hf_tokenizers = {\n",
    "    \"bvv241-2-3\":  AutoTokenizer.from_pretrained(\"Bochkov/bvv241-2-3\"),                             #vocab_size 65536\n",
    "    #\"bvv241-abs\"==\"bvv241-max\":  AutoTokenizer.from_pretrained(\"Bochkov/bvv241-abs\"),              #vocab_size 131072\n",
    "    \"bvv241-nemo\": AutoTokenizer.from_pretrained(\"Bochkov/bvv241-nemo\"),                            #vocab_size 131072\n",
    "    \"bvv241-max\":  AutoTokenizer.from_pretrained(\"Bochkov/bvv241-max\"),                             #vocab_size 131072\n",
    "    \"Mistral\":     AutoTokenizer.from_pretrained('mistralai/Mistral-Nemo-Instruct-2407'),           #vocab_size 131072\n",
    "    \"Qwen\":        AutoTokenizer.from_pretrained(\"Qwen/QwQ-32B\", trust_remote_code=True),           #vocab_size 152064\n",
    "    \"DeepSeek-R1\": AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1\", trust_remote_code=True),#vocab_size 129280\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2d83e8a9-28d0-4b4c-a323-8464e839a2b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "wikipedia_dataset_id = \"wikimedia/wikipedia\"\n",
    "wikisource_dataset_id = \"wikimedia/wikisource\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6d3513c9-a53e-4f13-bcb5-4fe726dcfa3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "configs_wikipedia = get_dataset_config_names(wikipedia_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "109d6824-6e80-47a8-a60b-a23cf7e7bb6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "configs_wikisource = get_dataset_config_names(wikisource_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a9fffd38-f74a-44ad-ad8b-35bbc4403d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "def analyze_tokenizer(texts, tokenizer, tokenizer_name=\"unknown\", is_tiktoken=False):\n",
    "    char_counts = []\n",
    "    token_counts = []\n",
    "    for text in texts:\n",
    "        chars = len(text)\n",
    "        tokens = tokenizer.encode(text) if is_tiktoken else tokenizer.encode(text, add_special_tokens=False)\n",
    "        num_tokens = len(tokens) if isinstance(tokens, list) else len(tokens[\"input_ids\"])\n",
    "        if num_tokens == 0:\n",
    "            continue\n",
    "        char_counts.append(chars)\n",
    "        token_counts.append(num_tokens)\n",
    "    ratio = np.array(char_counts) / np.array(token_counts)\n",
    "    return ratio.mean(), ratio.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9564e35b-2bd5-4934-998a-bbed512f5ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_pct = 0.05 # 5% of wikipedia & wikisource:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49ac1692-3fe3-42d8-ad7d-af89fa2a2738",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "932231f255e645e49d4875b3c2aa2f49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Resolving data files:   0%|          | 0/41 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd84ada6580b44ca966505494ff5c6ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading dataset shards:   0%|          | 0/41 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lang= en  ( English ) len(ds wikipedia)  = 6407814\n",
      "lang= en  ( English ) len(ds wikisource) = 208279\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "\n",
    "for lang in languages:\n",
    "    try:\n",
    "        texts = []\n",
    "        try:\n",
    "            configs = configs_wikipedia \n",
    "            latest_config = sorted([c for c in configs if c.endswith(f\".{lang}\")])[-1]\n",
    "            ds = load_dataset(wikipedia_dataset_id, latest_config, split=\"train\") \n",
    "            total_len = len(ds)\n",
    "            sample_size = int(total_len * top_pct)\n",
    "            ds_sub = ds.shuffle(seed=42).select(range(sample_size))\n",
    "            print('lang=', lang, ' (', lang_name_map[lang], ') len(ds wikipedia)  =',len(ds))\n",
    "            texts = [x['text'] for x in ds_sub]\n",
    "        except Exception as e:\n",
    "            print(f\"❌ Error processing wikipedia_dataset_id, language '{lang}': {e}\")\n",
    "            pass\n",
    "\n",
    "        if lang != 'ka':\n",
    "            try:\n",
    "                configs = configs_wikisource \n",
    "                latest_config = sorted([c for c in configs if c.endswith(f\".{lang}\")])[-1]\n",
    "                ds = load_dataset(wikisource_dataset_id, latest_config, split=\"train\") \n",
    "                total_len = len(ds)\n",
    "                sample_size = int(total_len * top_pct)\n",
    "                ds_sub = ds.shuffle(seed=42).select(range(sample_size))\n",
    "                print('lang=', lang, ' (', lang_name_map[lang], ') len(ds wikisource) =',len(ds))\n",
    "                texts = texts + [x['text'] for x in ds_sub]\n",
    "            except Exception as e:\n",
    "                print(f\"❌ Error processing wikisource_dataset_id, language '{lang}': {e}\")\n",
    "                pass\n",
    "\n",
    "        # GPT-2 (tiktoken)\n",
    "        avg, std = analyze_tokenizer(texts, tiktoken_gpt2_tok, \"GPT-2\", is_tiktoken=True)\n",
    "        results.append({\"Language\": lang, \"Tokenizer\": \"GPT-2\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "        print({\"Language\": lang, \"Tokenizer\": \"GPT-2\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "\n",
    "        # GPT-4 (cl100k)\n",
    "        avg, std = analyze_tokenizer(texts, tiktoken_cl100k_tok, \"GPT-4\", is_tiktoken=True)\n",
    "        results.append({\"Language\": lang, \"Tokenizer\": \"GPT-4\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "        print({\"Language\": lang, \"Tokenizer\": \"GPT-4\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "\n",
    "        for name, tok in hf_tokenizers.items():\n",
    "            avg, std = analyze_tokenizer(texts, tok, name, is_tiktoken=False)\n",
    "            results.append({\"Language\": lang, \"Tokenizer\": name, \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "            print({\"Language\": lang, \"Tokenizer\": name, \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "        \n",
    "    except Exception as e:\n",
    "        print(f\"❌ Error processing language '{lang}': {e}\")\n",
    "        continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8fe5da5-0de7-4ab5-8621-12966dda40a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(results)\n",
    "\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "\n",
    "df_pivot = df.pivot(index=\"Language\", columns=\"Tokenizer\", values=\"MeanCharsPerToken\")\n",
    "\n",
    "df[\"LanguageName\"] = df[\"Language\"].map(lang_name_map)\n",
    "\n",
    "df_pivot = df.pivot_table(index=\"LanguageName\", columns=\"Tokenizer\", values=\"MeanCharsPerToken\")\n",
    "\n",
    "df_pivot = df_pivot.sort_index()\n",
    "\n",
    "df_pivot.to_csv(\"_tokenizer-benchmarking.csv\")\n",
    "\n",
    "df_pivot.round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8d41fc90-60e0-4dac-8a1e-6538c3251808",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d3efb76f-d7bf-4aac-9ab9-8cbff29aeea5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 2000x900 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_pivot = pd.read_csv('_tokenizer-benchmarking.csv', index_col=0)\n",
    "\n",
    "plt.figure(figsize=(20, 9))\n",
    "sns.set(style=\"whitegrid\")\n",
    "\n",
    "selected_tokenizers = [\n",
    "    \"DeepSeek-R1\", \"GPT-2\", \"GPT-4\", \"Mistral\", \"Qwen\", \n",
    "    \"bvv241-2-3\", \"bvv241-max\", \"bvv241-nemo\"\n",
    "]\n",
    "\n",
    "df_long = df_pivot[selected_tokenizers].reset_index().melt(\n",
    "    id_vars='LanguageName', var_name='Tokenizer', value_name='Chars per Token'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cf24e9da-70f0-42b4-a3e7-1caeea27b8de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 166388 characters]",
      "text/plain": [
       "<Figure size 2200x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_order = df_pivot.mean(axis=1).sort_values().index\n",
    "\n",
    "plt.figure(figsize=(22, 10))\n",
    "sns.barplot(\n",
    "    data=df_long, \n",
    "    x='LanguageName', y='Chars per Token', \n",
    "    hue='Tokenizer',\n",
    "    order=lang_order\n",
    ")\n",
    "plt.xticks(rotation=65, ha='right')\n",
    "plt.title('Average Characters per Token Across Languages and Tokenizers')\n",
    "plt.tight_layout()\n",
    "plt.legend(bbox_to_anchor=(1, 1))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9a1724d8-1f96-4838-b243-112a3ce57d8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 348448 characters]",
      "text/plain": [
       "<Figure size 2000x1200 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(20,12))\n",
    "sns.heatmap(df_pivot[selected_tokenizers], annot=True, fmt=\".2f\", cmap=\"coolwarm\")\n",
    "plt.title(\"Heatmap: Symbols per token\")\n",
    "plt.ylabel(\"Language\")\n",
    "plt.xlabel(\"Tokenizer\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6df08532-0549-4042-8941-eac2977d10c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_pivot[selected_tokenizers] = df_pivot[selected_tokenizers].round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "646bf790-c200-45cd-b260-75588c6e80a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "important_languages = [\n",
    "    \"English\",\n",
    "    \"Russian\",\n",
    "    \"Chinese\",\n",
    "    \"Korean\",\n",
    "    \"German\",\n",
    "    \"French\",\n",
    "    \"Arabic\",\n",
    "    \"Japanese\"\n",
    "]\n",
    "\n",
    "# Предположение: индексы DataFrame — названия языков на английском\n",
    "filtered_df = df_pivot.loc[important_languages]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "889c553b-e728-417f-aa52-98db0da58335",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DeepSeek-R1</th>\n",
       "      <th>GPT-2</th>\n",
       "      <th>GPT-4</th>\n",
       "      <th>Mistral</th>\n",
       "      <th>Qwen</th>\n",
       "      <th>bvv241-2-3</th>\n",
       "      <th>bvv241-max</th>\n",
       "      <th>bvv241-nemo</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LanguageName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>4.21</td>\n",
       "      <td>4.07</td>\n",
       "      <td>4.15</td>\n",
       "      <td>3.96</td>\n",
       "      <td>3.95</td>\n",
       "      <td>1.74</td>\n",
       "      <td>2.27</td>\n",
       "      <td>3.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>2.81</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1.97</td>\n",
       "      <td>2.87</td>\n",
       "      <td>2.46</td>\n",
       "      <td>1.57</td>\n",
       "      <td>1.85</td>\n",
       "      <td>2.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>1.51</td>\n",
       "      <td>0.56</td>\n",
       "      <td>0.86</td>\n",
       "      <td>1.09</td>\n",
       "      <td>1.36</td>\n",
       "      <td>1.11</td>\n",
       "      <td>1.20</td>\n",
       "      <td>1.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>1.43</td>\n",
       "      <td>0.56</td>\n",
       "      <td>1.03</td>\n",
       "      <td>1.59</td>\n",
       "      <td>1.37</td>\n",
       "      <td>1.14</td>\n",
       "      <td>1.24</td>\n",
       "      <td>1.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>3.40</td>\n",
       "      <td>2.66</td>\n",
       "      <td>3.23</td>\n",
       "      <td>3.59</td>\n",
       "      <td>3.10</td>\n",
       "      <td>1.77</td>\n",
       "      <td>2.19</td>\n",
       "      <td>3.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>3.45</td>\n",
       "      <td>2.79</td>\n",
       "      <td>3.30</td>\n",
       "      <td>3.65</td>\n",
       "      <td>3.22</td>\n",
       "      <td>1.72</td>\n",
       "      <td>2.10</td>\n",
       "      <td>3.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>2.46</td>\n",
       "      <td>1.03</td>\n",
       "      <td>1.43</td>\n",
       "      <td>2.83</td>\n",
       "      <td>2.37</td>\n",
       "      <td>1.44</td>\n",
       "      <td>1.72</td>\n",
       "      <td>2.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>1.43</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.99</td>\n",
       "      <td>1.32</td>\n",
       "      <td>1.34</td>\n",
       "      <td>1.21</td>\n",
       "      <td>1.37</td>\n",
       "      <td>1.32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              DeepSeek-R1  GPT-2  GPT-4  Mistral  Qwen  bvv241-2-3  \\\n",
       "LanguageName                                                         \n",
       "English              4.21   4.07   4.15     3.96  3.95        1.74   \n",
       "Russian              2.81   0.98   1.97     2.87  2.46        1.57   \n",
       "Chinese              1.51   0.56   0.86     1.09  1.36        1.11   \n",
       "Korean               1.43   0.56   1.03     1.59  1.37        1.14   \n",
       "German               3.40   2.66   3.23     3.59  3.10        1.77   \n",
       "French               3.45   2.79   3.30     3.65  3.22        1.72   \n",
       "Arabic               2.46   1.03   1.43     2.83  2.37        1.44   \n",
       "Japanese             1.43   0.76   0.99     1.32  1.34        1.21   \n",
       "\n",
       "              bvv241-max  bvv241-nemo  \n",
       "LanguageName                           \n",
       "English             2.27         3.96  \n",
       "Russian             1.85         2.87  \n",
       "Chinese             1.20         1.09  \n",
       "Korean              1.24         1.59  \n",
       "German              2.19         3.59  \n",
       "French              2.10         3.65  \n",
       "Arabic              1.72         2.83  \n",
       "Japanese            1.37         1.32  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_df "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eac7d4fb-3f02-463a-8328-410279aac982",
   "metadata": {},
   "source": [
    "# Strengths of the bvv241* Tokenizer Family\n",
    "## Transparent Unicode Mapping: \n",
    "Where possible, one Unicode codepoint maps directly to one token, resulting in maximally predictable tokenization across all languages.\n",
    "Efficient Integration of Special Tokens: Unused and private Unicode ranges are systematically repurposed for custom and multi-character tokens, in contrast to the more chaotic allocation typical of BPE/Unigram.\n",
    "Broad Language Coverage: Minimal token fragmentation is observed for rare and non-Latin scripts (e.g., Armenian, Georgian, Tibetan, ideographic systems).\n",
    "## Multi-character Tokens from SOTA Models: \n",
    "Incorporation of ~19,000 multi-character tokens, derived from top-performing SOTA tokenizers, enables the compression of frequent substrings and words, reducing token counts in long texts while maintaining transparency.\n",
    "# Weaknesses\n",
    "## Less Aggressive Compression for Latin Scripts: \n",
    "In languages with short words and frequent morphemes (e.g., English, German, Spanish), traditional tokenizers (especially GPT-style BPEs) achieve stronger compression; as a result, bvv241* tokenizers produce 2x or more character per tokens for such languages.\n",
    "\n",
    "# Analytical Perspective\n",
    "A Unicode/character-oriented tokenizer is optimal for truly multilingual systems—as demonstrated by consistently low token inflation and minimal Unicode fragmentation for nearly all non-Latin scripts in the benchmarks.\n",
    "\n",
    "In English and similar languages, these advantages become less significant: highly optimized Latin-centric BPEs (e.g., GPT-2/4) achieve stronger text compression, yielding tangible savings in memory and computation. However, for code-switched or multilingual datasets, symbol-based tokenization ensures robustness and reliability.\n",
    "\n",
    "The inclusion of frequent multi-character tokens represents a beneficial compromise for diverse tasks, especially with numerous low-resource languages, ideographics, and emojis.\n",
    "\n",
    "# Summary Recommendation:\n",
    "\n",
    "A hybrid Unicode/character tokenizer as described here excels when the target model is truly multilingual (supporting many scripts), when predictable Unicode handling is desirable, or when compatibility with user input and special-symbol processing is critical. For fully English-centric tasks, classic BPE approaches remain more efficient.\n",
    "\n",
    "# Final Conclusions\n",
    "Combining strict Unicode mapping with multi-character tokens, the bvv241 family achieves highly stable performance for Asian, Arabic, and underrepresented scripts—where differences from SOTA are minimal.\n",
    "\n",
    "For Indo-European (Latin and partially Cyrillic) languages, compression is less aggressive, but the model gains in predictability, recoverability, and linguistic inclusivity.\n",
    "\n",
    "For general-purpose, multilingual applications, a high-frequency/hybrid character tokenizer of this type can surpass many current solutions in universality.\n",
    "\n",
    "For highly English-focused tasks (e.g., pure Self-Instruct or similar), GPT-style BPE models still have an edge—but for mixed/multilingual semantic applications, this approach offers significant advantages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41b105e1-1f99-45e2-bc3a-63a7bc32ca30",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3_11",
   "language": "python",
   "name": "python3_11"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
