Toward User Comprehension Supports for LLM Agent Skill Specifications

Zikai Alex Wen

Keywords: LLM agents, agent skills, usable security, specification design, user comprehension

TL;DR: Study of 878 cybersecurity skills finds most specs lack examples and full context, making capabilities hard to understand, so specs should be treated as clear user facing disclosures rather than just instruction containers.

Abstract: Users often interpret and select agent skills through their $\texttt{SKILL.md}$ specifications. To protect users, existing audits focus mainly on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we coded four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Operational basis was common, but only 19.0\% included an example task, sample, fixture, or expected output, and only 2.3\% provided all four anchors. We further examined a DNS/C2 telemetry subset to showcase why missing examples matters. Examples made first local checks easier to construct from the specification, while no-example skills often required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions.

Presentation Mode: Yes, at least one author will attend and present in person.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 68