Mind the Metrics: Patterns for Telemetry-Aware In-IDE AI Application Development using Model Context Protocol (MCP)

TMLR Paper4923 Authors

Abstract: Modern AI-driven development environments are destined to evolve into observability-first platforms by integrating real-time telemetry and feedback loops directly into the developer workflow. This paper introduces telemetry-aware IDEs driven by Model Context Protocol (MCP), a new paradigm for building software. We articulate how an IDE (integrated development environment), enhanced with an MCP client/server, can unify prompt engineering with live metrics, traces, and evaluations to enable iterative optimization and robust monitoring. We present a progression of design patterns: from local large language model (LLM) coding with immediate metrics feedback, to continuous integration (CI) pipelines that automatically refine prompts, to autonomous agents that monitor and adapt prompts based on telemetry. Instead of focusing on any single optimizer, we emphasize a general architecture (exemplified by the Model Context Protocol and illustrated through a reference MCP server implementation) that consolidates prompt and agent telemetry for the future integration of various optimization techniques. We survey related work in prompt engineering, AI observability, and optimization (e.g., Prompts-as-Programs, DSPy's MIPRO, Microsoft's PromptWizard) to position this approach within the emerging AI developer experience. This theoretical systems perspective highlights new design affordances and workflows for AI-first software development, laying a foundation for future benchmarking and empirical studies on optimization in these environments.

Submission Length: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Yutian_Chen1

Submission Number: 4923