Quantitative Certification of Knowledge Comprehension in LLMs

Isha Chaudhary, Vedaant V Jain, Gagandeep Singh

Keywords: Large Language Models, Knowledge comprehension, Formal certification

TL;DR: We develop a formal certifier for knowledge comprehension capability in LLMs.

Abstract: Large Language Models (LLMs) have demonstrated impressive performance on several benchmarks. However, traditional studies do not provide formal guarantees on the performance of LLMs. In this work, we propose a novel certification framework for LLMs, wherein we formally certify the knowledge-comprehension capabilities of popular LLMs. Our certificates are quantitative — they consist of high-confidence, tight bounds on the probability that the target LLM gives the correct answer on any relevant knowledge comprehension prompt. Our certificates for the Llama, Vicuna, and Mistral LLMs indicate that the knowledge comprehension capability improves with an increase in the number of parameters and that the Mistral model is less performant than the rest on this axis of evaluation.

Submission Number: 107