The Ai Bandwidth Wall And Co Packaged Optics

TITLE: The AI Bandwidth Wall & Co-Packaged Optics CHANNEL: Asianometry DATE: 2025-08-10 ---TRANSCRIPT---

Over the past two decades, the semiconductor industry has successfully pushed the limits to deliver more raw compute. A wild success, really. A 2025 Nvidia B200 GPU has 178,000 times more Floating
Point Operations Per Second, or FLOPs than a top-of-the-line Intel CPU in the late 1990s. Unfortunately, we don’t fully benefit thanks to limits - or “walls”,
so to say - that keep us from taking full advantage of all that new raw power. One such wall is the memory wall

• where semiconductor memories
  like DRAM have not scaled as fast as the compute has. I have covered this before. But in recent years, another bottleneck has presented itself: One relating to IO bandwidth.
  In response, the industry has been working on a silicon photonic solution. In today’s video, we talk about a technology trend
  that has been over a decade in the making: Co-packaged optics.

Beginnings

When it comes to data transfers out of a rack, there is an equation with two parts - silicon switch chips and the optics IO. A switch chip directs data coming in and out of the server,
forwarding them at high speed. It’s the heart of something like an Ethernet switch. Think of the switch chip as like an airport’s terminal - with tens
of thousands of passengers entering and leaving en route to their destination. Then we have the optics IO, which is where the data actually enters and exits. Today, the data inside the server rack travels as electric signals through thin
copper wire. But signals sent over a distance greater than 2 meters are better sent using
fiber. Fiber can carry more data over further distances with fewer signal integrity issues. The transition between electric and light is managed using a silicon photonics product
called a pluggable optical transceiver. We plug them into the switchgear’s faceplate
on each server rack, and they translate between digital optical and digital electrical signals. Inside a transceiver, there is a laser transmitter and modulator to encode and
send outgoing light signals, as well as a photodiode receiver to receive and interpret
incoming light signals. These together are referred to as the “optical engine”. The transceiver will also have components handling the electrical signals coming in from the server’s
switch chips. These are collectively referred to as the “electrical engine”. So all in all, if the switch silicon is like the airport terminal, then you can
think of the optical IO as like that terminal’s airport gates and roads. If the terminal is
too small - or if its roads or gates are too easily congested - then we get traffic jams.

Light and Silicon

Now switch chips are made from CMOS silicon, and so benefit from Moore’s Law. In recent years, switch chips have massively scaled up their
capacity as foundries like TSMC have shipped ever smaller nodes. But optical IO transceivers - despite having gone from 10 gigabits per second
20 years ago to 800 gigabits per second today - have grown nowhere near as fast.
Creating an obvious bottleneck because you can only plug so many into a switchgear. The industry is working on next-generation transceivers capable of doing 1.6 terabits
per second. Those are rolling out right now. After that,
the industry is going to spend a few years trying to earn back their investment. Meanwhile, switch chips in 2023 have capacities of 51.2 terabits per second.
And Broadcom is already seeding a switch chip with capacity of 102.4 terabits per second. And that number will very likely double again in another two years. The hard
truth of the matter is that Moore’s Law - even today - goes far faster than optics.

Faster and Faster

Why do we have this underperformance by the optical IO side? One of the bigger reasons is the Serdes. What is that? The data coming from those switches
is sent over in parallel - multiple data streams sent at lower speeds. But data can only be sent through the optical fiber serially. So we
need something to turn those parallel data streams into
a serial one. That something is called the serializer/deserializer, or Serdes. Imagine multiple streams of people approaching an
airport gate from different directions. The serializer is like the gate agent. They are arranging all these passengers into an orderly line that can get checked in through the
gate, that corridor bridge thing, and finally into their seat on the plane. The deserializer does the same thing but in the opposite direction. It
receives a serial signal that it converts into many parallel ones. At 800 gigabits per second, the electrical signals going
to the transceivers through the copper wires start experiencing serious integrity issues. So the Serdes needs to compensate by either using more sophisticated algorithms to pack
more bits or boosting weak signals via equalization. Both cost power, thus making the Serdes one of the
fastest growing consumers of power in the whole building. In 2013-14, Huawei estimated that in a 28 nanometer switch, the Serdes took up about 10-15%
of the switch’s area and power. Ten years later, Serdes now take up over twice that proportion. Nvidia’s VP of hyperscale Ian Buck told IEEE Spectrum in a 2025 interview that
pluggable optics now consume 10% of a GPU’s total compute power. Spread across thousands of these Serdes links, all that power consumed turns into
a real problem. Data center operators must work within power budgets pre-defined by
their fixed infrastructure. Power spent on networking cannot be used for GPUs.

Co-Packaged Optics

The general response has been to bring the electric and optical signals closer together. The Holy Grail is a small, manufacturable silicon photonics laser integrated right
into the chip itself to get data out as fast as possible. But this dream remains a dream. Silicon cannot easily generate or modulate laser
light. So the alternative has been to put the optic modules
like the transceiver as close as possible to the switch silicon. With the goal of bringing it inside the package itself - making it “Co-packaged” so to use the
term. As with anything relating to packaging, Co-Packaged Optics is a broad term that can mean
anything. There is a lot less standardization than in the mainline silicon industry. Co-packaged optics generally speaking means to package the switch silicon and optical modules
like the transceiver components together in the same package and on the same substrate. Again, the key idea is to shift away from these pluggable transceivers and
bring the photonic transition as close to the silicon as possible.
The less copper wiring all that data has to travel through, the better. Other benefits is that moving optical IO off the faceplate grants us more physical space. There are also some chances to eliminate redundant parts and reduce costs.

CPO Challenges

As with anything related to advanced packaging, co-packaged optics is no cakewalk. The single biggest concern is heat. Thermal issues in advanced packaging
has been a serious concern since the days of multi-chip modules. As we all know,
heat can warp or damage the packaging materials. But with optics, heat has an additional negative component. Light travels around
a photonic circuit through these thin channels called “waveguides”. Heat shifts the waveguides’
resonant wavelengths, causing them to stop working correctly at the laser’s
wavelength. So heat dissipation becomes doubly important here. And integrating the light source itself brings up crucial questions. Where
will the light source be? Off or on-chip? Having the light source be off-chip lowers
thermal risk and makes the whole thing more repairable, but introduces coupling issues. Coupling refers to the fact that the waveguides are very thin,
perhaps a few hundred nanometers across. But light arrives to the chip via optical
fibers that are about 10 micrometers wide

• so maybe ten times larger or thicker. This is an obvious mismatch, and is like channeling water from a firehose into a
  bundle of straws. The delicate work of onboarding and offboarding that light is called “coupling”.
  Achieving the most precise alignment of the fibers is hard, and errors lead to light losses.

OBO and NBO

The technical challenges are formidable,
but can be overcome. The thing that’s harder to overcome is inertia. Generally speaking, the industry is conservative. The faceplate pluggable
transceiver has been used for years. They are cheap. They are easy to replace. And
there are well-established standards, giving you a broad range of vendors to pick from. Getting customers to adopt something so different from before is always a
pain in the butt. So like as with 3D stacking packaging,
the path to Co-Packaged Optics adoption has been long and winding. In 2018, an Microsoft-led industry consortium introduced On-Board Optics or Board-Mounted
Optics solutions. As the name implies, On-Board Optics remove the pluggable
transceivers and bring the optical engine directly onto the PCB with the switch chips. The data coming out of the switch goes down into the board and then through that to the
optical engine. Since it’s a shorter distance, there is better latency and some power savings. Then in 2020, companies started introducing the idea of “Near-Package Optics”. Here we
insert a high speed silicon interposer in between the packaged chip and optical engine. This is a middle step towards full Co-Packaged Optics. Kind of like how ASE and the Taiwanese
packaging industry put forth 2.5D integration as a middle step towards full 3D stacked logic.

TSMC and COUPE

In 2021, TSMC presented a co-packaged optics technology,
called COmpact Universal Photonics Engine or COUPE. COUPE 3D-stacks an electrical Integrated Circuit or IC on top of another “photonic IC”. Laser light from an outside diode is guided into that photonic IC where
it can be modulated before being sent out to the optical fiber. TSMC argues certain significant benefits with this. First,
it lets you make the electrical IC with an advanced node while making the photonic IC
with something less advanced like 65 nanometers. This saves on cost. Second, stacking the electrical IC on top of the photonic IC shortens the length of
the wires connecting them. This reduces parasitic capacitance and signal loss,
issues that might arise if the two ICs are sitting side-by-side on an interposer. TSMC claimed that COUPE improved the photonic engine’s power consumption
by 40% and raised its speed by 170% compared to other ways of connecting. So,
not a bad way to finally make that leap towards co-packaged optics. The issue again, however, was that nobody had a
reason to take that leap. Judging by the number of people who wrote
about COUPE when it was first announced - basically none - the same issues applied. TSMC’s 2022 annual report mentioned in a single,
brief paragraph that a few companies taped out COUPE chips in 2022 as a test. Same thing with its 2023 annual report.
TSMC mentioned that the number of test chips made with COUPE achieved expectations. Which,
again, makes me think that the customer reception was pretty mid.

AI Blows Up

That changed when ChatGPT and generative AI became the hottest thing since Mala and boba. Which you should not mix together. And as Nvidia began its march to becoming the world’s most valuable company. In 2023 and 2024, tech giants rushed to build billion-dollar AI factories
across the world. With those factories consuming unprecedented amounts of power,
vendors suddenly had a real economic technical reason to finally adopt Co-Packaged Optics. TSMC rushed to build out COUPE’s roadmap and tie it into the rest of their advanced packaging
ecosystem. At their 2024 North American Technology Symposium, they discussed a 3-stage roadmap that
first brings COUPE to pluggable transceivers. So a familiar, evolutionary step forward to start. Next, TSMC integrates COUPE into their flagship advanced packaging platform,
CoWoS. This lets vendors make electrical-photonic COUPE
chiplets and put them right next to switch silicon. Or even later on, CPU/GPU silicon. This dual-approach might sound wishy-washy on the surface,
but I reckon it lets companies dip their toe into this complicated Co-Packaged
optics solution without feeling like they are crossing some bridge of no return. Nvidia has been COUPE’s tip of the spear. At GTC 2025, Nvidia announced
two new switches - the Quantum-X InfiniBand platform and Spectrum-X Ethernet platform. While they have been in the switch industry since 2023,
these are their first switch products with co-packaged optics. Nvidia says that replacing pluggables with co-packaged optics grants their
equipment 3.5 times better power efficiency,
10 times better network resilience, and 1.3 times faster deployment.

Broadcom, GF, and the Startups

Of course, Nvidia and TSMC are not the only ones working on these type solutions for AI. Broadcom has their own sizable AI chip business, including making
semi-customized AI ASIC hardware for hyperscalers like the Google TPU chip. They are a technical pioneer,
having first brought Co-Packaged Optics to market with the 51.2 terabit Tomahawk 5. And as I mentioned earlier, they are relentlessly
pushing the market forward with faster and faster switch chips. And it has long been known that TSMC definitely has ground to make up in the
silicon photonics space behind other foundries like GlobalFoundries and Tower Semiconductor. The former especially is quite strong. Their GF Fotonix platform is now in its
third generation with a good PDK, manufacturing scale, and a strong
cluster of silicon photonic customers like Ayar Labs, Ranovus and Lightmatter. Ayar Labs in particular seems to be already doing what COUPE is planning to do over the next few
years. They have received investment money from both AMD and Intel (imagine that boardroom). They recently announced a new “optical interconnect chiplet”. With up to 8
terabits per second bandwidth and compatibility with the Universal
Chiplet Interconnect Express standard, it sounds like a pretty compelling product.

Conclusion

Co-Packaged Optics is another step forward in bringing light and electrons ever closer together. But does that mean we have a situation ahead where light does a complete takeover? As in a
pure photonic AI chip? The thought occasionally pokes its head in the news from time to time. Again, the issue as I see it involves both technology and economic inertia. Economically,
we have been doing silicon and electrons for 70 years now, and it is hard to move away from that. And technically. Many of the current heat and coupling issues get worse with
an all-photonic chip. Add to that a lack of scalable on-chip light
sources and issues relating to achieving certain logics with all-optical gates. And then, the electrical IC folks can also stay very competitive by adopting strong evolutionary
improvements like replacing some copper interconnects with Ruthenium for instance. So that all-photonic dream remains for now a dream. But the long-awaited rise of Co-Packaged
Optics is a marvelous step, and a real sign of how AI is remaking the world’s silicon supply chains.