{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7e54c78-9f34-4fbe-9480-f5731f591ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "from datetime import timedelta\n",
    "from datetime import datetime\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import gc\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM\n",
    "\n",
    "from _models_benchmarking import load_opus_samples, load_squad_v2_samples, load_mmlu_samples, load_boolq_samples, load_commonsenseqa_samples, load_ai2arc_e_samples, load_ai2arc_c_samples\n",
    "from _models_benchmarking import evaluate_with_stats, save_list_to_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b88ca98-7764-4983-a306-baba9c43e382",
   "metadata": {},
   "source": [
    "# Frozen Embeddings in Language Modeling: Proof-of-Concept Study\n",
    "\n",
    "This repository presents code and results from a series of experiments studying language models trained with **predicted and fully frozen token embedding weights** throughout the entire pretraining phase.\n",
    "\n",
    "*Our work provides a fundamental proof-of-concept : it is in principle possible to train competitive language models with embeddings that were never updated during training.*\n",
    "\n",
    "---\n",
    "\n",
    "## Background and Motivation\n",
    "\n",
    "Traditional transformer language models (e.g. GPT-2, GPT-3) initialize token embeddings randomly and **allow these vectors to be updated** jointly with all other model parameters during training. It is commonly believed that a major part of the language model's semantic capacity is stored in the embedding layer.\n",
    "\n",
    "Here, we challenge this paradigm:  \n",
    "- We **predict** (precompute) token embeddings using external criteria (e.g., visual or Unicode composition),  \n",
    "- Then **fully freeze** those embeddings,\n",
    "- And train all other model parameters on standard large-scale text corpora.\n",
    "\n",
    "We compare such **frozen-embedding models** directly with classical models (where embeddings are trainable), using the same tokenizers, architectures, and training setup.\n",
    "\n",
    "---\n",
    "\n",
    "## Experimental Setup\n",
    "\n",
    "- **Training data:** All models were pretrained on a 9-billion-token English (and optionally multilingual) corpus.  \n",
    "- **Pretraining only:** No large-scale supervised finetuning was performed.  \n",
    "- **Instruct/SFT \"sprinkling\":** To enable validation on reasoning benchmarks (MMLU, SQuAD, etc.), a small fraction (~10%) of the training data consisted of instruct/SFT-formatted examples (\"instruct data infusion\"), but the vast majority (90%) of tokens were standard unsupervised text.\n",
    "- **Baselines:** For each experiment, we compare:\n",
    "    - Models with **frozen, predicted embeddings** (never updated during training)\n",
    "    - Models with **trainable (classic) embeddings** (all weights optimizable)\n",
    "\n",
    "All other hyperparameters (number of layers, heads, model dimension, batch size, LR schedule, etc.) were kept identical for both types.\n",
    "\n",
    "---\n",
    "\n",
    "## Summary of What is Provided\n",
    "\n",
    "- Reproducible code for data processing, model definition, and training (PyTorch).\n",
    "- Scripts/notebooks for running evaluation benchmarks (MMLU, SQuAD, ARC, BLEU, etc).\n",
    "- All raw results and code for analysis in this repo.\n",
    "\n",
    "---\n",
    "\n",
    "## Scientific Contribution\n",
    "\n",
    "* To our knowledge, this is the first systematic direct comparison of large-scale transformer language models trained with (i) fully frozen, externally-predicted embedding layers and (ii) standard learned embeddings, under identical conditions.\n",
    "* We demonstrate that competitive language modeling is possible even with completely static and non-semantic embedding weights; the main semantic structure of the model emerges in the transformer blocks, not in the embedding table.\n",
    "* This opens up directions for language model modularity, standardization of tokenization, and parameter fusion (see MoE experiments).\n",
    "\n",
    "**Note:** This is a \"proof-of-concept\" (POC) study. The models are not expected to match or exceed the performance of advanced instruction-tuned models; all results should be interpreted in the context of unsupervised pretraining only. Metrics (MMLU, etc.) will naturally be lower than state-of-the-art because only generic pretraining was performed.\n",
    "\n",
    "---\n",
    "\n",
    "## Comparison to GPT-2 Baseline\n",
    "\n",
    "Pretraining scale is roughly comparable to the original GPT-2-medium / large papers (300-700M parameters, up to 5B tokens trained-on).  \n",
    "You may interpret the results as a fair indicator of \"GPT-2 class\" LM performance, given the data and compute.\n",
    "\n",
    "---\n",
    "\n",
    "## Transparency and Reproducibility\n",
    "\n",
    "**All reported results in the main paper were obtained using the code and scripts in this repository.**\n",
    "\n",
    "We encourage other researchers to:\n",
    "- Reproduce and extend the experiments,\n",
    "- Explore different embedding generation strategies,\n",
    "- And contribute to further clarity on the role of embedding layers in language modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "38a120c0-c23f-44a8-a36b-dc0f30c149ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "057dbd7d-4207-4169-b828-a2bc39401a13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3fd081e-c5f5-4247-9ed1-18f729e6fe49",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0a410f09-2f97-43ca-b031-fd3096d9b366",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_subjects = ['high_school_european_history', 'business_ethics', 'clinical_knowledge', 'medical_genetics', 'high_school_us_history', 'high_school_physics', 'high_school_world_history', 'virology', 'high_school_microeconomics', 'econometrics', 'college_computer_science', 'high_school_biology', 'abstract_algebra', 'professional_accounting', 'philosophy', 'professional_medicine', 'nutrition', 'global_facts', 'machine_learning', 'security_studies', 'public_relations', 'professional_psychology', 'prehistory', 'anatomy', 'human_sexuality', 'college_medicine', 'high_school_government_and_politics', 'college_chemistry', 'logical_fallacies', 'high_school_geography', 'elementary_mathematics', 'human_aging', 'college_mathematics', 'high_school_psychology', 'formal_logic', 'high_school_statistics', 'international_law', 'high_school_mathematics', 'high_school_computer_science', 'conceptual_physics', 'miscellaneous', 'high_school_chemistry', 'marketing', 'professional_law', 'management', 'college_physics', 'jurisprudence', 'world_religions', 'sociology', 'us_foreign_policy', 'high_school_macroeconomics', 'computer_security', 'moral_scenarios', 'moral_disputes', 'electrical_engineering', 'astronomy', 'college_biology']\n",
    "\n",
    "mmlu_samples = load_mmlu_samples(all_subjects, n_questions=-1)\n",
    "boolq_samples = load_boolq_samples(n_questions=3270)\n",
    "commonsense_qa_samples = load_commonsenseqa_samples(n_questions=1221)\n",
    "ai2arc_e_qa_samples = load_ai2arc_e_samples(n_questions=570)\n",
    "ai2arc_c_qa_samples = load_ai2arc_c_samples(n_questions=299)\n",
    "squad_v2_samples = load_squad_v2_samples(n_questions=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3fcc5e57-5fcd-4499-b634-9b5eb24d4878",
   "metadata": {},
   "outputs": [],
   "source": [
    "mmlu_samples_by_subj = {}\n",
    "for subject in all_subjects:\n",
    "    mmlu_samples_by_subj[subject] = load_mmlu_samples([subject], n_questions=-1)\n",
    "    #print(len(mmlu_samples_by_subj[subject]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d96fb5a5-e09c-43fd-b944-bb3d0159d4e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bochkov/nemo_bvv_ru\n",
      "nemo_bvv_ru Total parameters:     0.4B\n"
     ]
    }
   ],
   "source": [
    "model_name = 'nemo_bvv_ru'\n",
    "model_id = \"Bochkov/\" + model_name\n",
    "print(model_id)\n",
    "model = AutoModelForCausalLM.from_pretrained(model_id, device_map=device, trust_remote_code=True)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id) \n",
    "reports = []\n",
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "status = model_name + ' ' + f\"Total parameters:     {total_params / 1e9:.1f}B\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2c644963-aca4-4707-94ec-b636ecd2a583",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nemo_bvv_ru MMLU [high_school_european_history]: 5.76% ± 1.03% (σ=1.65%)\n",
      "nemo_bvv_ru MMLU [business_ethics]: 10.30% ± 1.90% (σ=3.07%)\n",
      "nemo_bvv_ru MMLU [clinical_knowledge]: 16.00% ± 0.86% (σ=1.38%)\n",
      "nemo_bvv_ru MMLU [medical_genetics]: 9.90% ± 1.70% (σ=2.74%)\n",
      "nemo_bvv_ru MMLU [high_school_us_history]: 5.15% ± 1.04% (σ=1.67%)\n",
      "nemo_bvv_ru MMLU [high_school_physics]: 7.02% ± 1.44% (σ=2.32%)\n",
      "nemo_bvv_ru MMLU [high_school_world_history]: 5.99% ± 0.67% (σ=1.08%)\n",
      "nemo_bvv_ru MMLU [virology]: 11.39% ± 1.42% (σ=2.29%)\n",
      "nemo_bvv_ru MMLU [high_school_microeconomics]: 11.18% ± 0.99% (σ=1.60%)\n",
      "nemo_bvv_ru MMLU [econometrics]: 7.89% ± 1.58% (σ=2.54%)\n",
      "nemo_bvv_ru MMLU [college_computer_science]: 6.50% ± 0.64% (σ=1.02%)\n",
      "nemo_bvv_ru MMLU [high_school_biology]: 12.42% ± 1.17% (σ=1.88%)\n",
      "nemo_bvv_ru MMLU [abstract_algebra]: 6.50% ± 0.93% (σ=1.50%)\n",
      "nemo_bvv_ru MMLU [professional_accounting]: 7.73% ± 1.00% (σ=1.61%)\n",
      "nemo_bvv_ru MMLU [philosophy]: 15.59% ± 1.30% (σ=2.09%)\n",
      "nemo_bvv_ru MMLU [professional_medicine]: 9.04% ± 0.60% (σ=0.96%)\n",
      "nemo_bvv_ru MMLU [nutrition]: 10.65% ± 1.01% (σ=1.63%)\n",
      "nemo_bvv_ru MMLU [global_facts]: 4.30% ± 1.52% (σ=2.45%)\n",
      "nemo_bvv_ru MMLU [machine_learning]: 6.88% ± 1.31% (σ=2.11%)\n",
      "nemo_bvv_ru MMLU [security_studies]: 8.45% ± 0.88% (σ=1.43%)\n",
      "nemo_bvv_ru MMLU [public_relations]: 9.45% ± 1.34% (σ=2.16%)\n",
      "nemo_bvv_ru MMLU [professional_psychology]: 9.59% ± 0.65% (σ=1.04%)\n",
      "nemo_bvv_ru MMLU [prehistory]: 13.67% ± 1.02% (σ=1.65%)\n",
      "nemo_bvv_ru MMLU [anatomy]: 13.04% ± 1.54% (σ=2.48%)\n",
      "nemo_bvv_ru MMLU [human_sexuality]: 11.22% ± 1.71% (σ=2.75%)\n",
      "nemo_bvv_ru MMLU [college_medicine]: 13.12% ± 1.08% (σ=1.74%)\n",
      "nemo_bvv_ru MMLU [high_school_government_and_politics]: 9.27% ± 0.77% (σ=1.24%)\n",
      "nemo_bvv_ru MMLU [college_chemistry]: 8.60% ± 1.39% (σ=2.24%)\n",
      "nemo_bvv_ru MMLU [logical_fallacies]: 9.69% ± 1.64% (σ=2.64%)\n",
      "nemo_bvv_ru MMLU [high_school_geography]: 9.90% ± 0.69% (σ=1.11%)\n",
      "nemo_bvv_ru MMLU [elementary_mathematics]: 6.24% ± 0.52% (σ=0.85%)\n",
      "nemo_bvv_ru MMLU [human_aging]: 7.94% ± 0.87% (σ=1.40%)\n",
      "nemo_bvv_ru MMLU [college_mathematics]: 5.30% ± 1.21% (σ=1.95%)\n",
      "nemo_bvv_ru MMLU [high_school_psychology]: 10.46% ± 0.76% (σ=1.23%)\n",
      "nemo_bvv_ru MMLU [formal_logic]: 6.83% ± 1.27% (σ=2.05%)\n",
      "nemo_bvv_ru MMLU [high_school_statistics]: 8.75% ± 0.50% (σ=0.81%)\n",
      "nemo_bvv_ru MMLU [international_law]: 8.26% ± 1.92% (σ=3.09%)\n",
      "nemo_bvv_ru MMLU [high_school_mathematics]: 5.56% ± 1.06% (σ=1.71%)\n",
      "nemo_bvv_ru MMLU [high_school_computer_science]: 6.80% ± 1.38% (σ=2.23%)\n",
      "nemo_bvv_ru MMLU [conceptual_physics]: 9.62% ± 1.14% (σ=1.84%)\n",
      "nemo_bvv_ru MMLU [miscellaneous]: 8.75% ± 0.34% (σ=0.55%)\n",
      "nemo_bvv_ru MMLU [high_school_chemistry]: 9.16% ± 1.72% (σ=2.78%)\n",
      "nemo_bvv_ru MMLU [marketing]: 12.22% ± 1.11% (σ=1.78%)\n",
      "nemo_bvv_ru MMLU [professional_law]: 6.56% ± 0.41% (σ=0.66%)\n",
      "nemo_bvv_ru MMLU [management]: 8.35% ± 1.87% (σ=3.01%)\n",
      "nemo_bvv_ru MMLU [college_physics]: 5.69% ± 1.76% (σ=2.83%)\n",
      "nemo_bvv_ru MMLU [jurisprudence]: 10.65% ± 1.61% (σ=2.59%)\n",
      "nemo_bvv_ru MMLU [world_religions]: 7.02% ± 1.40% (σ=2.26%)\n",
      "nemo_bvv_ru MMLU [sociology]: 9.40% ± 1.04% (σ=1.67%)\n",
      "nemo_bvv_ru MMLU [us_foreign_policy]: 7.20% ± 0.91% (σ=1.47%)\n",
      "nemo_bvv_ru MMLU [high_school_macroeconomics]: 10.23% ± 0.69% (σ=1.11%)\n",
      "nemo_bvv_ru MMLU [computer_security]: 5.90% ± 1.31% (σ=2.12%)\n",
      "nemo_bvv_ru MMLU [moral_scenarios]: 6.30% ± 0.41% (σ=0.66%)\n",
      "nemo_bvv_ru MMLU [moral_disputes]: 9.65% ± 0.96% (σ=1.54%)\n",
      "nemo_bvv_ru MMLU [electrical_engineering]: 10.41% ± 1.58% (σ=2.55%)\n",
      "nemo_bvv_ru MMLU [astronomy]: 7.96% ± 1.42% (σ=2.29%)\n",
      "nemo_bvv_ru MMLU [college_biology]: 10.69% ± 1.38% (σ=2.22%)\n"
     ]
    }
   ],
   "source": [
    "for subject in all_subjects:\n",
    "    mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'MMLU', mmlu_samples_by_subj[subject], all_subjects, device=device, n_runs=10, p_source_lang='', p_target_lang='')\n",
    "    status = model_name + ' ' + f\"MMLU [{subject}]: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "    print(status)\n",
    "    reports.append(status)\n",
    "    save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e4f81049-8276-4bf8-86fe-44e7e18de500",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nemo_bvv_ru MMLU: 8.80% ± 0.20% (σ=0.32%)\n"
     ]
    }
   ],
   "source": [
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'MMLU', mmlu_samples, all_subjects, device=device, n_runs=10, p_source_lang='', p_target_lang='')\n",
    "status = model_name + ' ' + f\"MMLU: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7ff419c5-f6e8-4ffa-a1a5-12724fa5c814",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nemo_bvv_ru ARC-e: 19.53% ± 1.11% (σ=1.80%)\n"
     ]
    }
   ],
   "source": [
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'ARC-e', ai2arc_e_qa_samples, [], device=device, n_runs=10, p_source_lang='', p_target_lang='')\n",
    "status = model_name + ' ' + f\"ARC-e: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f5010b15-49f0-4bd6-9ce5-5cf30e459af6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nemo_bvv_ru ARC-c: 21.34% ± 1.18% (σ=1.91%)\n"
     ]
    }
   ],
   "source": [
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'ARC-c', ai2arc_c_qa_samples, [], device=device, n_runs=10, p_source_lang='', p_target_lang='')\n",
    "status = model_name + ' ' + f\"ARC-c: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ffe8fe47-a8ff-471f-9689-325c4cd2b5c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nemo_bvv_ru C-SENSE: 19.51% ± 0.28% (σ=0.44%)\n"
     ]
    }
   ],
   "source": [
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'C-SENSE', commonsense_qa_samples, [], device=device, n_runs=10, p_source_lang='', p_target_lang='')\n",
    "status = model_name + ' ' + f\"C-SENSE: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "27416483-e755-4926-83a5-7a41fd809beb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nemo_bvv_ru SQUAD: 6.80% ± 0.74% (σ=1.20%)\n"
     ]
    }
   ],
   "source": [
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'SQUAD', squad_v2_samples, [], device=device, n_runs=10, p_source_lang='', p_target_lang='')\n",
    "status = model_name + ' ' + f\"SQUAD: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "940f262d-96c9-46f8-a173-c571451a0a36",
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = 'en' \n",
    "lang_second = 'ru' \n",
    "lang_lang = lang + '-' + lang_second\n",
    "opus_src_tgt_samples = load_opus_samples(lang, lang_second, lang_lang, n_questions=128)\n",
    "opus_tgt_src_samples = load_opus_samples(lang_second, lang, lang_lang, n_questions=128)\n",
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'BLEU', opus_src_tgt_samples, [], device=device, n_runs=10, p_source_lang=lang, p_target_lang=lang_second)\n",
    "status = model_name + ' ' + f\"BLEU [{lang}-{lang_second}]: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')\n",
    "\n",
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'BLEU', opus_tgt_src_samples, [], device=device, n_runs=10, p_source_lang=lang_second, p_target_lang=lang)\n",
    "status = model_name + ' ' + f\"BLEU [{lang_second}-{lang}]: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18c2c873-4d8f-48b9-9837-bacfe685cb36",
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = 'en' \n",
    "lang_second = 'zh' \n",
    "lang_lang = lang + '-' + lang_second\n",
    "opus_src_tgt_samples = load_opus_samples(lang, lang_second, lang_lang, n_questions=128)\n",
    "opus_tgt_src_samples = load_opus_samples(lang_second, lang, lang_lang, n_questions=128)\n",
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'BLEU', opus_src_tgt_samples, [], device=device, n_runs=10, p_source_lang=lang, p_target_lang=lang_second)\n",
    "status = model_name + ' ' + f\"BLEU [{lang}-{lang_second}]: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')\n",
    "\n",
    "mean, std, ci95 = evaluate_with_stats(model, tokenizer, 'BLEU', opus_tgt_src_samples, [], device=device, n_runs=10, p_source_lang=lang_second, p_target_lang=lang)\n",
    "status = model_name + ' ' + f\"BLEU [{lang_second}-{lang}]: {mean:.2f}% ± {ci95:.2f}% (σ={std:.2f}%)\"\n",
    "print(status)\n",
    "reports.append(status)\n",
    "save_list_to_file(reports, '_models_benchmarking_' + model_name + '.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a38a6fc4-6142-47e2-b2f1-854fcb7c20e7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
