{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c078adf3-b73f-4d4a-a90a-c39743f7c93a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import tiktoken\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from datasets import get_dataset_config_names, load_dataset\n",
    "from transformers import AutoTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d021511c-d452-4136-a092-9af5e3e49eb9",
   "metadata": {},
   "source": [
    "## Introduction: The bvv241* Tokenizer Architecture\n",
    "\n",
    "This work introduces the **bvv241** tokenizer family, a set of open-source multilingual tokenization schemes designed to maximize transparency, Unicode coverage, and extensibility. All implementations are available on HuggingFace for public access and reproducibility:\n",
    "\n",
    "- [bvv241-2-3](https://huggingface.co/Bochkov/bvv241-2-3)\n",
    "- [bvv241-max](https://huggingface.co/Bochkov/bvv241-max)\n",
    "- [bvv241-nemo](https://huggingface.co/Bochkov/bvv241-nemo)\n",
    "- [bvv241-abs](https://huggingface.co/Bochkov/bvv241-abs)\n",
    "\n",
    "The core approach is based on bijective mapping of Unicode codepoints to individual tokens, whenever possible, ensuring a 1:1 relationship between characters and tokens across a wide range of scripts. Reserved, surrogate, and private Unicode ranges are systematically exploited to store custom and multi-character tokens, allowing for efficient representation of commonly occurring substrings and entire words—particularly beneficial for high-frequency terms shared across models and languages.\n",
    "\n",
    "The tokenizer’s hybrid design draws from frequent multi-character patterns identified in top-performing, state-of-the-art models: approximately 19,000 such tokens are incorporated. This structure enables robust handling of diverse languages and scripts, mitigating the fragmentation common in byte-pair encoding (BPE) and unigram tokenizers—especially for underrepresented scripts and code-mixed text.\n",
    "\n",
    "All bvv241* tokenizer variants, vocabularies, and evaluation tools are publicly available, promoting transparency and collaborative research in multilingual NLP scenarios.\n",
    "\n",
    "Demonstration Variant: bvv241-nemo\n",
    "- [bvv241-nemo](https://huggingface.co/Bochkov/bvv241-nemo)\n",
    "\n",
    "The bvv241-nemo variant is provided as a demonstration of the flexibility of the proposed methodology. While the underlying Unicode-centric mapping scheme is applicable to any tokenizer, bvv241-nemo specifically replicates the vocabulary, token structure, and exact token enumeration of the SOTA [Mistral Nemo tokenizer](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407), using its public vocabulary files as a template. This showcases how the approach can serve as a drop-in replacement or augmentation for existing BPE or unigram schemes.\n",
    "\n",
    "Furthermore, since all token indices and their corresponding Unicode/codepoint representations are deterministic and exhaustively specified, this methodology enables the direct creation of a pre-initialized embedding matrix of shape vocab_size × n_embed. Such a matrix can be either used as a fixed (non-trainable) embedding or as a high-quality cold start, reducing the need for extensive embedding pre-training.\n",
    "\n",
    "The code and trained vocabularies for bvv241-nemo are publicly available for reproducibility and to foster further experimentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29899f02-83da-466b-a4ff-38253e240a35",
   "metadata": {},
   "source": [
    "## The code below demonstrates how to benchmark tokenizer efficiency and generate comparative visualizations for different languages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "175664d7-6fb8-425c-9398-2e67edd5219e",
   "metadata": {},
   "outputs": [],
   "source": [
    "languages =     [\"en\", \"ru\", \"fr\", \"de\", \"zh\", \"he\", \"it\", \"es\", \"ar\", \"pt\", \"ko\", \"hu\", \"sa\", \"la\", \"ja\", \"el\",     \"sv\", \"nl\", \"pl\", \"vi\", \"fa\", \"no\", \"tr\", \"fi\", \"cs\", \"hy\", \"da\", \"bn\", \"az\", \"ka\", \"hi\", \"id\"]\n",
    "\n",
    "lang_name_map = {\n",
    "    \"en\": \"English\",\n",
    "    \"ru\": \"Russian\",\n",
    "    \"fr\": \"French\",\n",
    "    \"de\": \"German\",\n",
    "    \"zh\": \"Chinese\",\n",
    "    \"he\": \"Hebrew\",\n",
    "    \"it\": \"Italian\",\n",
    "    \"es\": \"Spanish\",\n",
    "    \"ar\": \"Arabic\",\n",
    "    \"pt\": \"Portuguese\",\n",
    "    \"ko\": \"Korean\",\n",
    "    \"hu\": \"Hungarian\",\n",
    "    \"sa\": \"Sanskrit\",\n",
    "    \"la\": \"Latin\",\n",
    "    \"ja\": \"Japanese\",\n",
    "    \"el\": \"Greek\",\n",
    "\n",
    "    \"sv\": \"Swedish\",\n",
    "    \"nl\": \"Dutch\",\n",
    "    \"pl\": \"Polish\",\n",
    "    \"vi\": \"Vietnamese\",\n",
    "    \"fa\": \"Persian\",\n",
    "    \"no\": \"Norwegian\",\n",
    "    \"tr\": \"Turkish\",\n",
    "    \"fi\": \"Finnish\",\n",
    "    \"cs\": \"Czech\",\n",
    "    \"hy\": \"Armenian\",\n",
    "    \"da\": \"Danish\",\n",
    "    \"bn\": \"Bangla\",\n",
    "    \"az\": \"Azerbaijani\",\n",
    "    \"ka\": \"Georgian\",\n",
    "    \"hi\": \"Hindi\",\n",
    "    \"id\": \"Indonesian\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c645ad23-8810-4b02-a3b5-b999c552a16a",
   "metadata": {},
   "outputs": [],
   "source": [
    "tiktoken_gpt2_tok = tiktoken.get_encoding(\"gpt2\")\n",
    "tiktoken_cl100k_tok = tiktoken.get_encoding(\"cl100k_base\")\n",
    "\n",
    "hf_tokenizers = {\n",
    "    \"bvv241-2-3\":  AutoTokenizer.from_pretrained(\"Bochkov/bvv241-2-3\"),                             #vocab_size 65536\n",
    "    #\"bvv241-abs\"==\"bvv241-max\":  AutoTokenizer.from_pretrained(\"Bochkov/bvv241-abs\"),              #vocab_size 131072\n",
    "    \"bvv241-nemo\": AutoTokenizer.from_pretrained(\"Bochkov/bvv241-nemo\"),                            #vocab_size 131072\n",
    "    \"bvv241-max\":  AutoTokenizer.from_pretrained(\"Bochkov/bvv241-max\"),                             #vocab_size 131072\n",
    "    \"Mistral\":     AutoTokenizer.from_pretrained('mistralai/Mistral-Nemo-Instruct-2407'),           #vocab_size 131072\n",
    "    \"Qwen\":        AutoTokenizer.from_pretrained(\"Qwen/QwQ-32B\", trust_remote_code=True),           #vocab_size 152064\n",
    "    \"DeepSeek-R1\": AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1\", trust_remote_code=True),#vocab_size 129280\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2d83e8a9-28d0-4b4c-a323-8464e839a2b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "wikipedia_dataset_id = \"wikimedia/wikipedia\"\n",
    "wikisource_dataset_id = \"wikimedia/wikisource\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6d3513c9-a53e-4f13-bcb5-4fe726dcfa3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "configs_wikipedia = get_dataset_config_names(wikipedia_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "109d6824-6e80-47a8-a60b-a23cf7e7bb6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "configs_wikisource = get_dataset_config_names(wikisource_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a9fffd38-f74a-44ad-ad8b-35bbc4403d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "def analyze_tokenizer(texts, tokenizer, tokenizer_name=\"unknown\", is_tiktoken=False):\n",
    "    char_counts = []\n",
    "    token_counts = []\n",
    "    for text in texts:\n",
    "        chars = len(text)\n",
    "        tokens = tokenizer.encode(text) if is_tiktoken else tokenizer.encode(text, add_special_tokens=False)\n",
    "        num_tokens = len(tokens) if isinstance(tokens, list) else len(tokens[\"input_ids\"])\n",
    "        if num_tokens == 0:\n",
    "            continue\n",
    "        char_counts.append(chars)\n",
    "        token_counts.append(num_tokens)\n",
    "    ratio = np.array(char_counts) / np.array(token_counts)\n",
    "    return ratio.mean(), ratio.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9564e35b-2bd5-4934-998a-bbed512f5ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_pct = 0.05 # 5% of wikipedia & wikisource:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49ac1692-3fe3-42d8-ad7d-af89fa2a2738",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "932231f255e645e49d4875b3c2aa2f49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Resolving data files:   0%|          | 0/41 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd84ada6580b44ca966505494ff5c6ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading dataset shards:   0%|          | 0/41 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lang= en  ( English ) len(ds wikipedia)  = 6407814\n",
      "lang= en  ( English ) len(ds wikisource) = 208279\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "\n",
    "for lang in languages:\n",
    "    try:\n",
    "        texts = []\n",
    "        try:\n",
    "            configs = configs_wikipedia \n",
    "            latest_config = sorted([c for c in configs if c.endswith(f\".{lang}\")])[-1]\n",
    "            ds = load_dataset(wikipedia_dataset_id, latest_config, split=\"train\") \n",
    "            total_len = len(ds)\n",
    "            sample_size = int(total_len * top_pct)\n",
    "            ds_sub = ds.shuffle(seed=42).select(range(sample_size))\n",
    "            print('lang=', lang, ' (', lang_name_map[lang], ') len(ds wikipedia)  =',len(ds))\n",
    "            texts = [x['text'] for x in ds_sub]\n",
    "        except Exception as e:\n",
    "            print(f\"❌ Error processing wikipedia_dataset_id, language '{lang}': {e}\")\n",
    "            pass\n",
    "\n",
    "        if lang != 'ka':\n",
    "            try:\n",
    "                configs = configs_wikisource \n",
    "                latest_config = sorted([c for c in configs if c.endswith(f\".{lang}\")])[-1]\n",
    "                ds = load_dataset(wikisource_dataset_id, latest_config, split=\"train\") \n",
    "                total_len = len(ds)\n",
    "                sample_size = int(total_len * top_pct)\n",
    "                ds_sub = ds.shuffle(seed=42).select(range(sample_size))\n",
    "                print('lang=', lang, ' (', lang_name_map[lang], ') len(ds wikisource) =',len(ds))\n",
    "                texts = texts + [x['text'] for x in ds_sub]\n",
    "            except Exception as e:\n",
    "                print(f\"❌ Error processing wikisource_dataset_id, language '{lang}': {e}\")\n",
    "                pass\n",
    "\n",
    "        # GPT-2 (tiktoken)\n",
    "        avg, std = analyze_tokenizer(texts, tiktoken_gpt2_tok, \"GPT-2\", is_tiktoken=True)\n",
    "        results.append({\"Language\": lang, \"Tokenizer\": \"GPT-2\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "        print({\"Language\": lang, \"Tokenizer\": \"GPT-2\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "\n",
    "        # GPT-4 (cl100k)\n",
    "        avg, std = analyze_tokenizer(texts, tiktoken_cl100k_tok, \"GPT-4\", is_tiktoken=True)\n",
    "        results.append({\"Language\": lang, \"Tokenizer\": \"GPT-4\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "        print({\"Language\": lang, \"Tokenizer\": \"GPT-4\", \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "\n",
    "        for name, tok in hf_tokenizers.items():\n",
    "            avg, std = analyze_tokenizer(texts, tok, name, is_tiktoken=False)\n",
    "            results.append({\"Language\": lang, \"Tokenizer\": name, \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "            print({\"Language\": lang, \"Tokenizer\": name, \"MeanCharsPerToken\": avg, \"Std\": std})\n",
    "        \n",
    "    except Exception as e:\n",
    "        print(f\"❌ Error processing language '{lang}': {e}\")\n",
    "        continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8fe5da5-0de7-4ab5-8621-12966dda40a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(results)\n",
    "\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "\n",
    "df_pivot = df.pivot(index=\"Language\", columns=\"Tokenizer\", values=\"MeanCharsPerToken\")\n",
    "\n",
    "df[\"LanguageName\"] = df[\"Language\"].map(lang_name_map)\n",
    "\n",
    "df_pivot = df.pivot_table(index=\"LanguageName\", columns=\"Tokenizer\", values=\"MeanCharsPerToken\")\n",
    "\n",
    "df_pivot = df_pivot.sort_index()\n",
    "\n",
    "df_pivot.to_csv(\"_tokenizer-benchmarking.csv\")\n",
    "\n",
    "df_pivot.round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8d41fc90-60e0-4dac-8a1e-6538c3251808",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d3efb76f-d7bf-4aac-9ab9-8cbff29aeea5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 2000x900 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_pivot = pd.read_csv('_tokenizer-benchmarking.csv', index_col=0)\n",
    "\n",
    "plt.figure(figsize=(20, 9))\n",
    "sns.set(style=\"whitegrid\")\n",
    "\n",
    "selected_tokenizers = [\n",
    "    \"DeepSeek-R1\", \"GPT-2\", \"GPT-4\", \"Mistral\", \"Qwen\", \n",
    "    \"bvv241-2-3\", \"bvv241-max\", \"bvv241-nemo\"\n",
    "]\n",
    "\n",
    "df_long = df_pivot[selected_tokenizers].reset_index().melt(\n",
    "    id_vars='LanguageName', var_name='Tokenizer', value_name='Chars per Token'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "cf24e9da-70f0-42b4-a3e7-1caeea27b8de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 166388 characters]",
      "text/plain": [
       "<Figure size 2200x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_order = df_pivot.mean(axis=1).sort_values().index\n",
    "\n",
    "plt.figure(figsize=(22, 10))\n",
    "sns.barplot(\n",
    "    data=df_long, \n",
    "    x='LanguageName', y='Chars per Token', \n",
    "    hue='Tokenizer',\n",
    "    order=lang_order\n",
    ")\n",
    "plt.xticks(rotation=65, ha='right')\n",
    "plt.title('Average Characters per Token Across Languages and Tokenizers')\n",
    "plt.tight_layout()\n",
    "plt.legend(bbox_to_anchor=(1, 1))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9a1724d8-1f96-4838-b243-112a3ce57d8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 348448 characters]",
      "text/plain": [
       "<Figure size 2000x1200 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(20,12))\n",
    "sns.heatmap(df_pivot[selected_tokenizers], annot=True, fmt=\".2f\", cmap=\"coolwarm\")\n",
    "plt.title(\"Heatmap: Symbols per token\")\n",
    "plt.ylabel(\"Language\")\n",
    "plt.xlabel(\"Tokenizer\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eac7d4fb-3f02-463a-8328-410279aac982",
   "metadata": {},
   "source": [
    "# Strengths of the bvv241* Tokenizer Family\n",
    "## Transparent Unicode Mapping: \n",
    "Where possible, one Unicode codepoint maps directly to one token, resulting in maximally predictable tokenization across all languages.\n",
    "Efficient Integration of Special Tokens: Unused and private Unicode ranges are systematically repurposed for custom and multi-character tokens, in contrast to the more chaotic allocation typical of BPE/Unigram.\n",
    "Broad Language Coverage: Minimal token fragmentation is observed for rare and non-Latin scripts (e.g., Armenian, Georgian, Tibetan, ideographic systems).\n",
    "## Multi-character Tokens from SOTA Models: \n",
    "Incorporation of ~19,000 multi-character tokens, derived from top-performing SOTA tokenizers, enables the compression of frequent substrings and words, reducing token counts in long texts while maintaining transparency.\n",
    "# Weaknesses\n",
    "## Less Aggressive Compression for Latin Scripts: \n",
    "In languages with short words and frequent morphemes (e.g., English, German, Spanish), traditional tokenizers (especially GPT-style BPEs) achieve stronger compression; as a result, bvv241* tokenizers produce 2x or more character per tokens for such languages.\n",
    "\n",
    "# Analytical Perspective\n",
    "A Unicode/character-oriented tokenizer is optimal for truly multilingual systems—as demonstrated by consistently low token inflation and minimal Unicode fragmentation for nearly all non-Latin scripts in the benchmarks.\n",
    "\n",
    "In English and similar languages, these advantages become less significant: highly optimized Latin-centric BPEs (e.g., GPT-2/4) achieve stronger text compression, yielding tangible savings in memory and computation. However, for code-switched or multilingual datasets, symbol-based tokenization ensures robustness and reliability.\n",
    "\n",
    "The inclusion of frequent multi-character tokens represents a beneficial compromise for diverse tasks, especially with numerous low-resource languages, ideographics, and emojis.\n",
    "\n",
    "# Summary Recommendation:\n",
    "\n",
    "A hybrid Unicode/character tokenizer as described here excels when the target model is truly multilingual (supporting many scripts), when predictable Unicode handling is desirable, or when compatibility with user input and special-symbol processing is critical. For fully English-centric tasks, classic BPE approaches remain more efficient.\n",
    "\n",
    "# Final Conclusions\n",
    "Combining strict Unicode mapping with multi-character tokens, the bvv241 family achieves highly stable performance for Asian, Arabic, and underrepresented scripts—where differences from SOTA are minimal.\n",
    "\n",
    "For Indo-European (Latin and partially Cyrillic) languages, compression is less aggressive, but the model gains in predictability, recoverability, and linguistic inclusivity.\n",
    "\n",
    "For general-purpose, multilingual applications, a high-frequency/hybrid character tokenizer of this type can surpass many current solutions in universality.\n",
    "\n",
    "For highly English-focused tasks (e.g., pure Self-Instruct or similar), GPT-style BPE models still have an edge—but for mixed/multilingual semantic applications, this approach offers significant advantages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41b105e1-1f99-45e2-bc3a-63a7bc32ca30",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3_11",
   "language": "python",
   "name": "python3_11"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
