K-PGD: Fast Discrete Projected Gradient Descent with K-Means Acceleration on GPT

Xuanzhou Chen

Keywords: Projected Gradient Descent (PGD), k-means clustering, Adversarial attacks, NLP

Abstract: Projected Gradient Descent (PGD) is a workhorse for optimization over discrete sets, but with large vocabularies the projection step becomes the runtime bottleneck. We present K-PGD, a $k$-means–accelerated variant that replaces exhaustive projection with a centroid-based shortlist followed by a restricted search. The approach provides simple per-iteration certificates that quantify approximation error and yield convergence guarantees for PGD with approximate projections. Our theory connects cluster geometry to certificate strength and gives iteration bounds under bounded accumulated error. In a GPT-2 token-substitution case study, K-PGD reduces projection cost while preserving attack success and solution quality, showing that clustering can substantially accelerate discrete PGD without compromising rigor.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 11522