When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

ACL ARR 2026 January Submission5629 Authors

Keywords: LLM Agents, Knowledge Distillation, Behavioral Similarity, Tool Use, Model Evaluation

Abstract:

Model distillation is a primary driver behind the rapid progress of LLM agents, yet it often leads to behavioral homogenization.
Many emerging agents share nearly identical reasoning steps and failure modes, suggesting they may be distilled echoes of a few dominant teachers.
To ensure ecosystem robustness, it is critical to quantify this alignment.
However, existing metrics fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences.
In this work, we quantify agent distillation by isolating non-mandatory behavioral patterns via two complementary metrics: $\textbf{Response Pattern Similarity (RPS)}$ for verbal alignment and $\textbf{Action Graph Similarity (AGS)}$ for tool-use habits modeled as directed graphs.
We evaluate 18 models from 8 providers on $\tau$-Bench and $\tau^2$-Bench using Claude 4.5 Sonnet as the reference.
Experimental results reveal three key insights: (1) Anthropic models exhibit high internal consistency, validating our metrics; (2) Kimi-K2 shows the highest structural similarity to the teacher among non-Anthropic models, particularly in tool-dependency patterns; and (3) RPS and AGS capture distinct behavioral dimensions.
By disentangling behavioral mimicry from task-driven necessity, our work provides a systematic tool to improve the transparency and independent development of the agent ecosystem.

Paper Type: Long

Research Area: AI/LLM Agents

Research Area Keywords: LLM/AI agents, safety and alignment, robustness

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 5629