Chip Industry Explores Optical Links to Separate GPUs and HBM, Aiming to Break Memory Bottlenecks

The current method of placing HBM next to GPUs is hitting a physical limit called the 'shoreline bottleneck,' which restricts memory expansion.

As a solution, the industry is seriously considering separating the GPU and HBM and connecting them with high-speed optical links, allowing for much greater memory capacity and bandwidth.

This discussion is gaining momentum due to recent large-scale investments in optical technology, the establishment of industry standards, and the worsening HBM supply shortage.

The idea of physically separating GPUs from HBM memory and connecting them with light is no longer a distant dream, but a tangible solution being actively discussed in the industry today.

At the heart of this discussion is the 'shoreline bottleneck.' In current 2.5D packaging, HBM memory stacks must be placed right next to the GPU, but the chip's perimeter—its 'shoreline'—is limited. This physically restricts how many HBM stacks can be added. While the industry has cleverly stacked memory chips higher and higher (12, 16, and even 20 layers), this vertical approach is also approaching its limits due to issues with height, heat, and manufacturing yield.

So, what changed to make optical links a serious contender? Three key developments have converged recently. First is standardization. The formation of the Optical Compute Interconnect MSA (OCI-MSA) by major players like NVIDIA, AMD, and Microsoft creates a common framework. This allows different companies to build components that work together, significantly lowering the risk and cost of adoption.

Second, there's a massive flow of capital. NVIDIA's recent investments, totaling over $4 billion in optical component makers like Lumentum and Coherent, are a powerful signal. They are proactively building and securing a supply chain for a future where data moves on beams of light, not just electrical wires.

Third is the undeniable demand pressure. SK Hynix recently reported that its HBM order book is full for the next three years, exceeding its production capacity. This highlights that the current methods of expanding memory bandwidth simply cannot keep up with the explosive growth of AI.

However, the transition will be gradual. The first commercial applications of this technology, called Co-Packaged Optics (CPO), will appear in network switches connecting server racks. From there, optical I/O (OIO) will move 'inward' onto the circuit board to connect different chips. The final, most complex step will be integrating these optical links directly inside the chip package to connect a GPU and its HBM. This phased approach allows the technology to mature before tackling its most challenging application.

Shoreline Bottleneck: The physical limit on the number of electrical connections that can be made at the edge (the 'shoreline') of a chip. This restricts how many HBM stacks can be placed around a GPU.
Co-Packaged Optics (CPO): A technology that integrates optical components directly with electronic chips (like network switches) in the same package to increase bandwidth and reduce power consumption.
Optical I/O (OIO): Using light (photons) instead of electricity to transmit data between chips. It offers much higher bandwidth and efficiency, especially over distances longer than a few millimeters.

Chip Industry Explores Optical Links to Separate GPUs and HBM, Aiming to Break Memory Bottlenecks

The current method of placing HBM next to GPUs is hitting a physical limit called the 'shoreline bottleneck,' which restricts memory expansion.

As a solution, the industry is seriously considering separating the GPU and HBM and connecting them with high-speed optical links, allowing for much greater memory capacity and bandwidth.

This discussion is gaining momentum due to recent large-scale investments in optical technology, the establishment of industry standards, and the worsening HBM supply shortage.

The idea of physically separating GPUs from HBM memory and connecting them with light is no longer a distant dream, but a tangible solution being actively discussed in the industry today.

Shoreline Bottleneck: The physical limit on the number of electrical connections that can be made at the edge (the 'shoreline') of a chip. This restricts how many HBM stacks can be placed around a GPU.
Co-Packaged Optics (CPO): A technology that integrates optical components directly with electronic chips (like network switches) in the same package to increase bandwidth and reduce power consumption.
Optical I/O (OIO): Using light (photons) instead of electricity to transmit data between chips. It offers much higher bandwidth and efficiency, especially over distances longer than a few millimeters.