SK hynix Highlights Co-Packaged Optics: Why AI Infrastructure Is Moving Beyond GPUs

Short Answer

AI infrastructure is no longer just a GPU procurement question. As models and clusters grow, the cost of *moving* data between processors, memory and networking becomes as important as the cost of computing on it.

SK hynix has highlighted a technology roadmap for co-packaged optics (CPO), positioning it as a foundation technology for next-generation AI infrastructure. Co-packaged optics places optical engines in the same package as the switch or processor silicon, shortening the electrical path that data must travel before it becomes light.

The strategic point for enterprises is simpler than the physics: faster compute does not help if the interconnect cannot keep it fed.

Why Data Movement Became the Constraint

Compute performance has scaled faster than the interconnect and memory bandwidth around it for over a decade. Training and serving large models makes this gap visible because the workload is fundamentally collective — thousands of accelerators must repeatedly exchange gradients, activations and key-value cache state, and each step waits for the slowest participant.

The result is a familiar and expensive symptom: accelerator utilisation well below what the hardware datasheet implies. We routinely see enterprise GPU fleets running at 30–50% effective utilisation. Almost never because the chips are slow. Usually because they are waiting — on the network, on storage, on host memory, on a scheduler, or on a data pipeline that was never designed for this throughput.

A full AI system depends on all of the following, and the weakest link sets the pace:

• Compute (GPUs and AI accelerators)
• Memory capacity and bandwidth (HBM and system memory)
• Interconnect between accelerators inside a node
• Network fabric between nodes
• Storage throughput for training data and checkpoints
• Power delivery
• Cooling capacity

What Co-Packaged Optics Actually Is

Traditional data-centre networking converts electrical signals to optical ones in a pluggable transceiver at the faceplate of the switch. Data therefore travels a comparatively long electrical distance across the board before it becomes light.

Co-packaged optics moves the optical engine into the same package as the switch ASIC or processor. At a simplified level:

`Compute → short electrical path → optical engine in package → optical fibre → optical engine in package → Compute`

Because the electrical run is dramatically shorter, the approach targets three problems simultaneously: bandwidth per package, energy consumed per bit transmitted, and signal integrity at very high data rates.

The energy figure is the one that changes financial models. In large AI clusters, interconnect power is a material share of total facility power, and every watt spent moving bits is a watt unavailable for computation and a watt that must be cooled. At cluster scale, an efficiency improvement in data movement translates directly into more usable compute inside the same power envelope — which, as operators are discovering, is the genuinely scarce resource.

SK hynix's interest is also a memory story. As HBM stacks grow and accelerator packages become more complex, the boundary between memory, interconnect and networking blurs. Advanced packaging is where those disciplines now meet.

Why AI Workloads Drive This

Consider what happens during large-scale distributed training:

`Forward pass → backward pass → gradient exchange across all accelerators → parameter update → synchronise → repeat`

That gradient exchange is a collective operation. It happens thousands of times per training run, and its duration is bounded by fabric bandwidth and tail latency, not by compute. Inference at scale creates a different but related pressure: long-context serving moves large volumes of key-value cache between memory tiers and nodes, so memory bandwidth and interconnect quality directly determine tokens per second and cost per token.

This is why the industry conversation has widened from "how many FLOPS" to bandwidth, latency, energy per bit, network topology and cooling.

The AI Infrastructure Stack, Layer by Layer

Read that right-hand column carefully. In enterprise deployments, the layer that limits results is rarely the one that received the budget.

What This Means for Enterprises That Are Not Hyperscalers

Almost no enterprise will procure co-packaged optics directly. It matters because it changes what your cloud provider can offer, at what price, and it reframes how you should evaluate AI infrastructure options.

Practical implications:

• Benchmark systems, not chips. Compare achieved throughput and latency on your actual workload, not accelerator specifications.
• Measure utilisation before buying capacity. If your fleet sits at 35% utilisation, the cheapest capacity increase is fixing the bottleneck.
• Treat data pipelines as infrastructure. Storage throughput and preprocessing frequently limit training more than GPUs do.
• Include egress and inter-region transfer in cost models. Data movement has a cloud invoice line as well as a physics cost.
• Check instance-level networking, not just GPU type. Two instances with identical accelerators can differ substantially in fabric performance.
• Right-size the model. A smaller, well-evaluated model often removes an infrastructure problem entirely.

Cloud Versus On-Premises

For AI workloads, the decision hinges on utilisation and predictability rather than ideology.

Cloud suits bursty training, uncertain workloads, rapid experimentation and teams without data-centre operations capability. You are effectively renting someone else's power, cooling and networking investment — including advances like CPO — without a capital commitment.

On-premises or colocation suits sustained high-utilisation inference, strict data residency requirements and organisations with genuinely predictable long-term demand. The economics only work above a fairly high, sustained utilisation threshold, and the hidden cost is operations: capacity planning, thermal management, hardware lifecycle and staffing.

Hybrid is the common landing point. Steady-state inference where it is cheapest and most controlled, elastic training and experimentation in cloud.

Technology Options

Open source. Kubernetes, the NVIDIA GPU Operator, Slurm, Ray, vLLM, Prometheus, Grafana and OpenTelemetry cover scheduling, serving and the observability needed to see where time is actually being spent. Without that observability layer, bottleneck analysis is guesswork.

Commercial. AWS, Microsoft Azure and Google Cloud AI infrastructure, NVIDIA compute and networking platforms, and specialised GPU cloud providers each make different trade-offs on availability, fabric quality, region coverage and price.

The correct architecture follows from workload characteristics — utilisation profile, latency requirement, data gravity and residency — not from a vendor preference stated in advance.

How We Approach This

An AI infrastructure assessment should produce a bottleneck answer, not a shopping list. In practice that means measuring accelerator utilisation and idle causes, network throughput and tail latency during collective operations, memory bandwidth saturation, storage throughput during checkpoints, cloud spend against actual utilisation, and the power and cooling headroom you have left.

The recommendation that follows is often unwelcome and usually correct: fix the pipeline, right-size the model and raise utilisation before adding hardware. That work sits at the intersection of our [cloud engineering solutions](/solutions) and [AI cost optimisation](/ai-cost) practice.

Conclusion

The next phase of AI infrastructure will not be decided by model size or peak FLOPS alone. It will be decided by whether the whole system can move data quickly, efficiently and economically enough to keep expensive compute busy.

SK hynix's co-packaged optics roadmap is another signal that competition has moved into memory, packaging and networking. For enterprises, the lesson is not to buy optics. It is to stop evaluating AI infrastructure one component at a time.

Frequently Asked Questions

What is co-packaged optics (CPO)?

Co-packaged optics integrates optical engines into the same package as a switch ASIC or processor, replacing long electrical paths to faceplate transceivers. It targets higher bandwidth per package, lower energy per bit and better signal integrity at high data rates.

Why is co-packaged optics important for AI?

Large AI clusters spend a significant share of time and power moving data between accelerators. Reducing energy per bit and improving interconnect bandwidth frees power and time for computation, which raises effective cluster throughput.

Does my enterprise need to care about CPO?

Not directly — you will not buy it. It matters because it shapes what cloud providers can offer and reinforces that AI infrastructure decisions should be made at system level rather than by comparing accelerator specifications.

Why is our GPU utilisation low?

Usually data movement: storage throughput during training, network tail latency during collective operations, memory bandwidth saturation, inefficient batching or scheduler gaps. Instrument the pipeline before adding capacity.

Is cloud or on-premises better for AI workloads?

Cloud suits bursty and uncertain workloads; on-premises or colocation can win for sustained high-utilisation inference with strict residency needs. Most enterprises end up hybrid, and the deciding variable is sustained utilisation.

What should we measure before buying more AI infrastructure?

Accelerator utilisation and idle causes, network throughput and tail latency, memory bandwidth headroom, storage throughput during checkpoints, cost per successful request and available power and cooling capacity.