Why Memory Is Becoming the Bottleneck in Physical AI Design

Most AI hardware discussions still start with the accelerator. That makes sense, since accelerators are easy to compare and their headline numbers are easy to quote. But for systems that sense, decide and act in the physical world, the limiting factor often sits somewhere else. A robot or vehicle is only as responsive as its slowest data path, and in physical AI that path runs through memory.

Data movement now sets the pace

A physical AI device rarely runs one workload. It handles sensor streams, model weights, activations, temporary buffers, control loops and safety functions at the same time, and all of them compete for the same memory system. When memory access becomes unpredictable, compute engines stall and sit idle. Every extra trip across the memory hierarchy also costs power. Poorly planned buffering adds latency that can build up over time and is hard to claw back later.

That's why the useful question has shifted. It's no longer just how fast the processor is. Architects now need to know how predictably the whole system behaves when several real-world workloads are active at once.

Keeping data close to compute

Physical AI systems typically pair a small amount of fast on-chip memory with a much larger external pool. LPDDR or GDDR holds model weights, sensor data, activations and other working sets. On-chip SRAM and cache hold the slices that the NPU, CPU, DSP, ISP or other engines need right now. Data moves from external memory into local memory when the pipeline needs it, and the design wins when that local copy gets reused many times before it's evicted.

The constraint is silicon. SRAM and cache take up die area, so local capacity is limited. Anything that doesn't fit, or isn't touched often, stays in external DRAM until it's needed. The goal isn't to eliminate external memory traffic, which isn't realistic. It's to keep the right data near compute for as long as it's useful and avoid moving it more often than the workload requires. A well-planned dataflow coordinates the compute engines, local memory, external DRAM, buffering and interconnect around the workload, so the processors stay busy and external bandwidth and capacity get used efficiently.

Each device type stresses memory differently

Humanoid robots, drones, autonomous vehicles, smart cameras, inspection robots, industrial robots and autonomous mobile robots all run local intelligence. Each one puts a different load on memory and its interfaces, and that difference determines what stays on-chip and what lives in higher-capacity external memory.

A humanoid combines cameras, depth sensing, audio, inertial data, balance control, object recognition, path planning and actuator feedback inside a tight power and thermal envelope. If perception data arrives late, control decisions slip. If model weights and intermediate results get reloaded too often, sustained performance drops and power climbs. In this case, memory efficiency directly shapes how smoothly the robot can perceive, decide and move.

Drones and autonomous mobile robots face a different mix: visual navigation, obstacle avoidance, stabilization, tracking, wireless communication and video processing, all on a limited power and thermal budget. Unnecessary external accesses cut operating time, add heat, or push latency into the control loop. For these devices, bandwidth per watt matters as much as raw bandwidth.

Autonomous vehicles and ADAS sit at the demanding end. Cameras, radar, lidar, ultrasonic sensors, telemetry, map data and control functions all reach for memory. Some streams need steady throughput, while others must be fused across sensor domains. Data also has to stay resident long enough to support planning. Here, memory behavior affects both throughput and how predictable the response is.

Pick the memory after you understand the workload

The sensible starting point is the workload's behavior, not the fastest DRAM on the datasheet. Architects need to size the full hierarchy together: local storage, external capacity, bandwidth, power, buffering and integration. The inputs are the device's data streams, reuse patterns, latency targets and physical limits.

LPDDR and GDDR serve different priorities. LPDDR is often the stronger fit when bandwidth per watt and capacity are needed to sustain local inference. When continuous image or sensor-stream bandwidth dominates, GDDR can be viable because its higher bandwidth can meet the requirement with fewer DRAM devices. Cadence, for example, lists LPDDR6/LPDDR5X PHY and Controller IP supporting data rates up to 14.4Gbps, and GDDR7 PHY IP supporting data rates up to 36Gbps, for high-bandwidth applications. Those options let the external interface follow the workload rather than fitting every design to one template.

In practice, a humanoid might lean toward LPDDR to get power-efficient access to weights, perception data and control information. A smart camera or vehicle perception processor may push harder on sustained vision-stream bandwidth. Either way, the most reused and latency-sensitive data should stay close to compute, and the external interface should be sized for what's left. That avoids both bottlenecks and wasteful over-provisioning.

Five decisions to settle early

Before committing to an external memory interface, it's worth working through five questions tied to the device's dataflow and operating limits:

• Data locality: Which data must stay near compute to meet latency, reuse and energy targets, and which can live in external memory?
• Traffic contention: Where do sensor, inference, software, safety and control traffic collide for memory access?
• External memory fit: Which option best matches the device's bandwidth, power and integration needs?
• Buffering strategy: How much local storage is needed to reuse data well without pushing unnecessary traffic off-chip?
• Latency predictability: Can latency stay bounded when several workloads run simultaneously?

Predictability is the real target

Physical AI puts intelligence inside devices that run continuously, handle real-world signals and must respond within tight power, thermal, latency and reliability limits. Meeting those limits means treating compute, memory and data movement as one coordinated system. Keep the most-used data near the engines that need it, use external bandwidth carefully, and make sure the system stays predictable as workloads overlap. My read is that the teams who get this right will be the ones who profile their dataflow early, before the processor choice locks in the memory options. Cadence's white paper, "Designing Physical AI Systems Under Real-World Constraints", covers the design side of that work in more depth.