What Matters in Learning Facts in Language Models? Multifaceted Knowledge Probing with Diverse Multi-Prompt Datasets
Abstract: Large language models (LLMs) face issues in handling factual knowledge, making it vital to evaluate their true ability to understand facts.
In this study, we introduce knowledge probing frameworks, BELIEF(-ICL), to evaluate the knowledge understanding ability of both encoder-based and decoder-based pre-trained LMs (PLMs) from diverse perspectives.
BELIEFs utilize a multi-prompt dataset to evaluate PLM's accuracy, consistency, and reliability in factual knowledge understanding.
To enable a more reliable evaluation with BELIEFs, we semi-automatically create MyriadLAMA, which has massively diverse prompts.
We validate the effectiveness of BELIEFs in correctly and comprehensively evaluating PLM's factual understanding ability via extensive evaluations with recent LLMs.
We then investigate key factors in learning facts in LLMs, and reveal the limitation of the prompt-based knowledge probing.
The dataset is anonymously publicized.
Paper Type: Long
Research Area: Interpretability and Analysis of Models for NLP
Research Area Keywords: Interpretability and Analysis of Models for NLP,Resources and Evaluation,Question Answering
Contribution Types: Model analysis & interpretability, Data resources, Data analysis
Languages Studied: English
Submission Number: 3628
Loading