The Early Bird Catches the Worm: A Positional Decay Reweighting Approach to Membership Inference in Large Language Models

Jinhan Liu, Yibo Yang, Ruiying Lu, Piotr Piękos, Yimeng Chen, Peng Wang, Dandan Guo

Keywords: Pretrain Data Detection; Large Language Model;

Abstract: Membership inference attacks (MIAs) against large language models (LLMs) aim to detect whether a specific data point was included in the training dataset. While existing likelihood-based MIA methods have shown promise, they typically aggregate token-level scores using uniform weights (e.g., via simple averaging). We argue that this uniform aggregation is suboptimal because it fails to explicitly account for the decaying nature of memorization signals. Inspired by the information-theoretic principle that conditioning reduces uncertainty, we hypothesize that the memorization signal is strongest at the beginning of a sequence—where model uncertainty is highest—and generally decays with token position. To leverage this insight, we introduce Positional Decay Reweighting (PDR), a simple and lightweight plug-and-play method. PDR applies decay functions to explicitly re-weight token-level scores from existing likelihood-based MIA methods, systematically amplifying the strong signals from early tokens while attenuating noise from later ones. Extensive experiments show that PDR consistently enhances a wide range of advanced methods across multiple benchmarks.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 20062