From Observed Reasoning to Stable Skills: A Memory Substrate for Skill Graduation

Sazan Khalid, Amit Arora

Keywords: agent skills, procedural knowledge, memory systems, trajectory mining, coding agents, LLM observability, SKILL.md graduation

TL;DR: CMM is a memory substrate that captures reasoning from coding agent sessions and graduates validated patterns into stable SKILL.md files. On a real bug-fix task, the CMM-augmented agent used 61% fewer messages than baseline.

Abstract: The Agent Skills format gives LLM-based agents portable procedural knowledge through SKILL.md files. Skills are powerful when curated and harmful when misauthored: a recent benchmark reports a 16.2-point average pass-rate improvement for curated Skills, yet self-generated Skills degrade performance in 31% of evaluated tasks. We argue that this gap reflects a missing observation layer: Skills are authored without empirical evidence about which procedural patterns actually recur and pay off across real agent sessions. We present the Cognitive Memory Manager (CMM), a memory system that observes coding agent execution, extracts reasoning structure as a directed acyclic graph, accumulates patterns across sessions and developers under a human-approval gate, and serves them to future sessions. On top of this substrate we propose and implement a Skill graduation mechanism: criteria under which a memory pattern, once validated by repeated observation and successful retrieval, is promoted to a stable SKILL.md. We evaluate CMM with a case study A/B comparison on a real bug in a mid-sized enterprise open-source codebase. With CMM-derived knowledge loaded at session start, an agent reduced assistant messages by 61% (119→46), files modified by 71% (7→2), and reached the correct root cause 11 messages earlier than the baseline; limitations of this single-run evaluation are discussed in the Limitations section. The system, including extraction pipeline, distributed sync layer, graduation command, and 216-test harness, is released under Apache 2.0.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 50