Microsoft Explores Shared Memory to Ease AI Shortage

Memory is the scarcest resource in AI data centers, and many hyperscalers are trying new ways to hand it out. The shortage forces graphics cards and other processors to sit idle while waiting on memory and storage. Microsoft’s pathfinding team is looking at system‑level technologies that can turn memory into a shared, dynamically assigned resource across Azure, says Samir Rajadnya, principal architect at Microsoft. “Memory was a servant to the CPU or GPU, but now it is being seen as another key resource,” he said. The goal is to use the limited supply more efficiently, so that work is not stranded behind unused memory.

The core idea behind composable memory is to pool capacity beyond individual servers and allocate it on demand. This approach lets a workload that needs extra memory reach it even if the server it runs on has limited local memory. Microsoft is evaluating how to make that possible at scale, and the company has been active in the Open Compute Project, where it shares design work for large‑scale AI infrastructure. Rajadnya noted that innovation in a single component is less useful unless the whole ecosystem can adopt it. AI inference, the most common workload, is limited by two ceilings: power and memory.

Rajadnya said if he could design a memory system with no constraints, he would connect it over long distances with the lowest possible power and latency. He added that enterprises can handle the problem more easily by focusing on the top two or three workloads and building systems tailored to them. “We need vertical integration of workload, system, software, silicon, all in one,” he said. Making composable memory work depends on the links between memory and processors, including the specialized accelerators that Rajadnya calls XPUs. Once memory leaves the server chassis, the distance data must travel determines whether copper wires suffice or optical connections are required.

Rajadnya emphasized that optical interconnects will play a crucial role, as data must travel long distances and optical links may be necessary to connect memory across server chassis.