The Early Bird Catches the Worm: A Positional Decay Reweighting Approach to Membership Inference in Large Language Models

ICLR 2026 Conference Submission20062 Authors

Keywords: Pretrain Data Detection; Large Language Model;

Abstract:

Membership inference attack (MIA) against large language models (LLMs) aim to detect whether a specific data point was
included in the training dataset of LLMs, which have become increasingly critical in many scenarios.
Existing likelihood-based MIA methods against LLMs treat all token-level scores as equally important, with a latent assumption that the memorization signal is position-agnostic. We argue, however, that this signal is not uniformly distributed.
Inspired by the information-theoretic principle that conditioning reduces uncertainty, we hypothesize that the memorization signal is not uniformly distributed. Instead, it tends to be strongest at the beginning of a sequence—where model uncertainty is highest—and generally decays with token position. To leverage this insight, we introduce Positional Decay Reweighting (PDR), a simple and lightweight plug-and-play method. PDR applies decay functions to re-weight token-level scores from existing likelihood-based MIA methods, systematically amplifying the strong signals from early tokens while attenuating noise from later ones. Extensive experiments show that PDR consistently enhances a wide range of advanced methods across multiple benchmarks.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 20062