From Queries to Clones: A Systematic Study of Encoder Stealing Attacks

Abstract: Model stealing is a growing threat for online ML services. Encoder models are more vulnerable to stealing than classifiers because their high-dimensional embeddings reveal substantially richer information than class logits. This risk is amplified by Encoder-as-a-Service platforms that expose foundation encoders. Prior work mostly studied encoder stealing attacks in isolation with inconsistent setups, so practical trade-offs and failure modes across attacks remain unclear. We present a comprehensive benchmark and comparative study of a representative set of encoder stealing attacks on two widely used encoders, CLIP and DINO. We consider three threat scenarios with increasing realism: (i) the attacker has access to the victim’s training data, (ii) the attacker knows the victim’s training distribution and uses disjoint data, and (iii) the attacker has no reliable knowledge of the victim’s training distribution. We also evaluate a novel setting where the attacker uses queries from multiple datasets to steal a more generalizable surrogate. Finally, we vary data and query budgets, surrogate capacity, and resource constraints to understand practical attack scenarios. Across our settings, we observe that high utility on the stealing distribution does not necessarily translate to high utility on the victim’s training distribution under shift. We find that contrastive objectives with strong augmentations are the most reliable, conventional methods can be brittle, and prototype alignment is query-efficient but shifts cost to local compute and memory. In our experiments, mixed-source queries reveal a data density-diversity trade-off, and DINO is consistently easier to steal than CLIP, with cross-modal text guidance partially narrowing the gap. Overall, our results map practical attack operating points and highlight vulnerabilities relevant to the foundation model era.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Amartya_Sanyal1

Submission Number: 8359