The landscape of AI infrastructure is undergoing a fundamental change, shifting the primary bottleneck away from GPUs.
For the past few years, the AI race was all about who could get the most GPUs. Now, the focus is shifting to other critical components. The CEO of Rabelup, Shin Jung-kyu, recently stated, "The new bottleneck is the CPU, not the GPU." This signals a major transition in how we build and scale AI systems, driven by the rise of more sophisticated AI applications.
So, what caused this change? The primary driver is the transition to Agentic AI. Unlike older AI models that performed a single task, agentic AIs can handle complex, multi-step processes. They can process extremely long inputs, use external tools, and orchestrate various tasks. This requires immense processing power not just from the GPU, but heavily from the CPU and memory for tasks like pre-processing data, managing states, and coordinating actions. Evidence of this shift is overwhelming. For instance, Google reported a nearly 50-fold increase in token traffic over a year, indicating a massive rise in these CPU-intensive orchestration tasks.
Hardware trends and research confirm this bottleneck shift. First, NVIDIA's latest Grace Blackwell (GB200) chip tightly integrates a powerful CPU with its GPUs, acknowledging that both are crucial. Second, Intel has noted that for AI inference and agent workloads, the required CPU-to-GPU ratio is tightening from 7:1 to as low as 1:1. Third, a recent academic paper demonstrated that a lack of CPU resources in multi-GPU setups can severely slow down performance, creating GPU idle time and increasing response latency (TTFT).
This has created a ripple effect across the entire supply chain. Prices for server memory (DDR5) have surged by up to 50%, and high-bandwidth memory (HBM) from suppliers like Micron and SK hynix is sold out for 2025 and much of 2026. This isn't just a GPU cycle anymore; it's an entire infrastructure upgrade cycle.
Ultimately, the most fundamental bottleneck is emerging: power. Data center electricity demand in the U.S. is projected to jump nearly 18% in just one year. Grid operators are struggling to keep up, with some, like in Denmark, pausing new connections. In response, tech giants like Meta and AWS are making massive deals to secure gigawatts of power directly from nuclear power plants, signaling that the future of AI growth depends as much on energy as it does on silicon.
- Agentic AI: AI systems that can proactively and autonomously perform complex, multi-step tasks by using tools, reasoning, and planning to achieve a goal.
- Time to First Token (TTFT): A measure of AI model responsiveness. It's the time it takes for the model to start generating the first piece (token) of its response after receiving a prompt.
- Orchestration: In AI, this refers to the process of coordinating and managing multiple AI models, tools, and data sources to execute a complex task.