CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimization

ICLR 2026 Conference Submission15875 Authors

Keywords: Large Language Models, Code Generation, CUDA Kernel Optimization, Multi-Agent Systems

TL;DR: CudaForge is a training-free multi-agent system that uses hardware feedback to optimize CUDA kernels, achieving 97.6% correctness and 1.68× speedup at low cost, generalizing across GPUs and models.

Abstract:

Developing efficient CUDA kernels is increasingly critical for AI applications such as large-scale LLM training. However, manual kernel design is both costly and time-consuming, motivating automatic approaches that leverage LLMs for code generation. Existing methods for automatic kernel generation, however, often produce low-efficiency kernels, incur high computational overhead, and fail to generalize across settings.

In this work, we propose CudaForge, a training-free multi-agent workflow for CUDA kernel generation and optimization. Our workflow is inspired by the iterative workflow of human experts, which contains steps such as developing initial kernels, testing correctness, analyzing hardware feedback, and iterative improvement.
More specifically, CudaForge employs two LLM agents -- a Coder and a Judge -- that iteratively generate, correct, and optimize CUDA kernels, while integrating hardware feedback such as Nsight Compute (NCU) metrics. In our extensive evaluations, we show that CudaForge, by leveraging base models like OpenAI-o3, achieves 97.6\% correctness of generated kernels and an average 1.68$\times$ speedup over PyTorch baselines, substantially surpassing state-of-the-art models including OpenAI-o3 and Kevin on KernelBench. Beyond accuracy and speed, CudaForge demonstrates strong generalization across GPUs (A100, RTX 6000, 4090, 3090) and base models (OpenAI-o3, GPT5, gpt-oss-120B, Claude-Sonnet-4, QwQ-32B), while maintaining high efficiency. In particular, generating an optimized kernel takes about $25$ minutes on one RTX6000 and incurs \$0.30 API cost. Our results highlight that multi-agent, training-free workflows can enable cost-effective, generalizable, and high-performance CUDA kernel optimization.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 15875