Decoding Intelligence: A Framework for Certifying Knowledge Comprehension in LLMs
Keywords: Large Language Models, Reasoning, Information Extraction, Certification
TL;DR: We certify LLMs for their knowledge comprehension capabilities.
Abstract: Knowledge comprehension capability is an important aspect of human intelligence. As Large Language Models (LLMs) are being envisioned as superhuman
agents, it is crucial for them to be proficient at knowledge comprehension. However, existing benchmarking studies do not provide consistent, generalizable, and
formal guarantees on the knowledge comprehension capabilities of LLMs. In
this work, we propose the first framework to certify knowledge comprehension in
LLMs with formal probabilistic guarantees. Our certificates are quantitative -
they consist of high-confidence, tight bounds on the probability that a target LLM
gives the correct answer on any knowledge comprehension prompt sampled from
a distribution. We design and certify novel specifications that precisely represent
distributions of knowledge comprehension prompts leveraging knowledge graphs.
We certify SOTA LLMs for specifications over the Wikidata5m knowledge graph.
We find that knowledge comprehension improves with increasing model size.
Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8329
Loading