A major shift is underway in the world of AI hardware.
For years, the story of AI progress has been centered on more powerful GPUs. But as AI models, especially those handling very long texts (long-context inference), have grown, they've hit a frustrating bottleneck. It’s not about raw computing power anymore; it’s about data access. Even the most advanced GPUs, like NVIDIA's H100, spend a significant amount of time waiting for data to be fetched from memory. This is often called the 'memory wall', and it's a major hurdle for efficiency.
To break through this wall, the industry is turning to a new architecture: HBM-PNM (High Bandwidth Memory with Processing-Near-Memory). The core idea is simple but profound. Instead of moving massive amounts of data between the memory chips and the GPU, why not perform some of the calculations right where the data is stored? HBM-PNM achieves this by placing small, specialized processing units directly on the logic die at the base of the HBM stack. This is a fundamental change from simply making memory faster; it makes the memory itself 'smart'.
This isn't just a theoretical concept. A recent research paper called 'AMMA', co-authored by a team from Samsung, NVIDIA, and top universities, provided compelling evidence. Their HBM-PNM design demonstrated a 15.5x reduction in attention latency and a 6.9x improvement in energy efficiency compared to a top-tier H100 GPU for long-context tasks. These numbers show that targeting the memory bottleneck directly yields significant performance gains.
Two key trends are making this shift possible right now. First, the arrival of HBM4 technology. Advanced manufacturing processes, like TSMC's 5nm node for the HBM base die, finally provide enough sophistication and space to integrate meaningful logic and processing. Second, the entire AI supply chain is strained. Persistent bottlenecks in advanced chip packaging (CoWoS) and manufacturing capacity mean that simply building more or bigger GPUs isn't a sustainable solution. Improving efficiency within existing components, like memory, becomes a much more attractive path.
This evolution could rebalance the power dynamic in the semiconductor industry. While GPU designers like NVIDIA will remain vital, memory suppliers such as Samsung, SK hynix, and Micron are positioned to capture more value. As HBM becomes an active computational component rather than just a passive storage unit, their strategic importance and pricing power are likely to increase, heralding a new, memory-centric era of AI acceleration.
- HBM-PNM (Processing-Near-Memory): A memory architecture where processing units are placed on the logic die of an HBM stack, allowing computation to happen closer to the data, reducing data movement and latency.
- Long-Context Inference: The process of an AI model making predictions based on a very long piece of input text, such as a large document or an entire book. This is highly demanding on memory bandwidth.
- KV Cache: In transformer-based AI models, this is a type of memory cache that stores key (K) and value (V) states from previous calculations to speed up the generation of the next token. Accessing this cache is a major bottleneck in long-context scenarios.