Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

Open Webpage

Jiyeon Kim, Hyunji Lee, Hyowon Cho, Joel Jang, Hyeonbin Hwang, Seungpil Won, Youbin Ahn, Dohaeng Lee, Minjoon Seo

Published: 01 Jan 2025, Last Modified: 05 Nov 2025ICLR 2025EveryoneRevisionsBibTeXCC BY-SA 4.0