Building Flexibility Into AI Infrastructure From the Start

The question that matters most for AI system architects right now is how much flexibility can be designed in before the first silicon exists. Training and inference both run on AI hardware, but they load it differently, and a single fixed architecture is increasingly hard to justify across both.

My read is that the answer is shifting toward giving designers real choices across memory, compute, connectivity and packaging, and then doing the harder work of making those pieces interoperate.

Why one architecture doesn't fit both jobs

Training and inference often share much of the same underlying infrastructure, but the pressures they put on a system are different. Training leans heavily on memory and on communication between accelerators. Inference cares about a different mix: memory capacity, latency, throughput and efficiency.

That split changes the hardware decisions. A memory technology that suits one workload can be the wrong fit for the other, and the same logic applies to interconnects and networking. Here's the thing: a design tuned for peak training bandwidth may carry cost and power that inference deployments don't need, and an inference-optimised design may leave training throughput on the table.

The result is a more heterogeneous approach. Architects are picking different technologies for different functions based on what the system has to do, instead of forcing every workload through one template.

Fig. 1: Heterogeneous applications demand distinct solutions for optimized implementations.

Open die-to-die standards change the packaging conversation

Open standards are what make that mixing practical at the package level. UCIe is the clearest example in the source material: it defines a common approach to die-to-die connectivity and is aimed at supporting an open chiplet ecosystem. Instead of tying a design to one vendor's die-to-die implementation, it gives the industry a standardised way to connect dies inside a package.

That matters because architects are working across a wider range of process technologies, packaging approaches and functions than before. A chiplet-based approach lets compute, I/O, memory interfaces and other blocks be partitioned into separate dies and then integrated in one package. UCIe supplies the common die-to-die link for that arrangement, so designers can test different partitioning choices without redesigning the interface every time.

The payoff is choice. Functions built on different process nodes or by different suppliers can share a package, provided they speak the same interface. That freedom becomes especially useful when the trade-offs are performance, power, cost, packaging and process technology all at once, which is the normal situation for AI silicon.

Interoperability is where the real work sits

An open standard doesn't guarantee that every implementation will cooperate. Open standards give a common foundation for bringing technologies from different suppliers and foundries into one system, and they help designers reason about how those implementations will behave together. But the standard is the starting point, not the finish line.

Verification carries much of the load here. As systems add more functions, more interfaces and more technologies, the number of combinations that need to be checked grows quickly. Teams in this position typically find that interface compliance is only one layer; timing, power delivery and thermal behaviour across dies can surface problems that a spec sheet doesn't show. That said, a shared interface still shrinks the verification problem compared with a fully bespoke link.

The same logic runs through the rest of the AI factory

The need for choice doesn't stop at the package boundary. An AI system is built from several kinds of connectivity, and each one covers a different part of the architecture. The source material names PCIe, CXL, UCIe, UALink and Ethernet-based technologies, and memory adds its own set of decisions on top.

Fig. 2: Different connectivity, different scope, same goal: economical, rapid data transfer.

What each link needs depends on where the data is going. HBM4E and LPDDR6 address different memory needs. PCIe handles high-speed host and device connectivity. UALink is aimed at accelerator-to-accelerator communication. 224G high-speed SerDes provides the underlying signalling for demanding connectivity applications, and UCIe connects dies within packages. Each of those sits at a different distance and bandwidth tier, and mixing them up is an expensive mistake.

So the useful questions are concrete: what is being connected, how much data has to move, and what the workload actually demands. Looking at those answers together gives a fuller picture of the system than tuning each interface in isolation. In practice, that shifts the design review from "is this link fast enough" to "does this link fit the traffic pattern of the whole machine."

Designing for the parts you haven't picked yet

AI infrastructure will keep changing as workloads, memory technologies, accelerators and interconnects advance. A design locked to one generation of each will age quickly. Building flexibility into the architecture from the start gives teams room to swap in new memory, add a chiplet or change a network tier without rethinking the whole system.

Open interfaces and interoperability are what make that room usable. They let designers choose each technology for what the system needs, rather than accepting a single vendor's approach across the board. Watch how quickly the die-to-die and accelerator-link specifications mature in the next few design cycles, because that will decide how much mixing is realistic in production rather than only in reference designs.