**Summary of "HipKittens: Making AMD AI Kernel Development Fast and Accessible"**
Artificial intelligence (AI) has seen rapid advances in recent years, with much of this progress powered by improvements in hardware—specifically, the graphics processing units (GPUs) that underpin modern machine learning. However, the AI landscape has been dominated by a single hardware vendor, NVIDIA, whose CUDA software ecosystem has become essential for high-performance AI workloads. This dominance has created what many in the industry call a "CUDA moat": a situation where software and hardware innovations are tightly coupled, making it difficult for alternative hardware providers to compete.
Despite this, AMD has emerged as a promising challenger. Its latest GPUs offer state-of-the-art compute capabilities and memory bandwidth that, on paper, rival or even surpass those of NVIDIA. Large-scale deployments of AMD hardware, including those at the gigawatt scale, suggest that the physical infrastructure is in place. However, these performance gains are largely inaccessible to AI practitioners because of gaps in the AMD software ecosystem. The tools necessary to write high-performance machine learning kernels on AMD hardware are either immature, underperforming, or too complex for widespread adoption.
### The AMD Software Landscape: Challenges and Gaps
AMD’s software stack for AI development includes a handful of key tools and libraries:
- **AITER:** A high-performance AI kernel library. - **PyTorch:** The popular AI framework, with some support for AMD. - **Compilers:** Such as Triton, Mojo, and TileLang, which aim to bridge the gap between high-level AI code and low-level hardware instructions. - **Composable Kernel (CK):** AMD’s C++-based programming model for kernel development.
Despite these offerings, the AMD ecosystem remains fragile. Many of these tools suffer from performance bottlenecks, lack of features, or complex dependencies. For example, Mojo’s multi-head attention (MHA) kernel achieves only around half of the theoretical peak performance on AMD's MI355X GPUs due to issues like bank conflicts. TileLang, while competitive with PyTorch, lags behind in terms of available features and is restricted to the latest AMD architectures, further complicating adoption.
Triton, a compiler designed for easier GPU programming, faces hurdles on AMD hardware; it struggles with register management and fails to optimize memory access patterns, resulting in subpar performance on basic operations like matrix multiplication. While these compilers make kernel development more approachable, they often force developers to choose between ease-of-use and peak hardware performance.
This fragmented landscape leaves a critical question for the AI community: What is the best way to achieve high-performance, cross-platform AI kernel development? The answer is not yet clear, but what is certain is that relying on raw assembly code (the lowest-level hardware programming) is not sustainable for the broader community.
### Learning from NVIDIA: The Rise of Opinionated Primitives
It is worth noting that developing high-performance kernels for NVIDIA GPUs was, until recently, equally difficult.