When scaling modern distributed systems—whether training trillion-parameter Large Language Models (LLMs), building high-throughput inference engines, or architecting enterprise storage arrays—traditional server designs hit a wall. Memory bandwidth limits, CPU bottlenecks, and network latency become the primary constraints.
To build scalable infrastructure, modern systems rely on a tightly integrated hardware and software stack: high-speed interconnects, disaggregated storage, and distributed memory management.
This guide breaks down these core building blocks—from physical interconnects like RDMA and CXL up to the inference orchestration layer with vLLM, NIXL, and KVBM.
Table of Contents
1. Physical & Network Interconnects
RDMA (Remote Direct Memory Access)
Traditional TCP/IP networking requires the operating system and CPU to package, unpackage, and copy data multiple times between application space and kernel memory. RDMA bypasses the host OS kernel and CPU entirely, allowing network interface cards (RNICs) to read and write directly between application memory on different host machines.
Key Benefits: Sub-microsecond latency, zero-copy transfers, near-zero CPU utilization.
Flavors: InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP.
Primary Use Case: GPU cluster synchronization (AllReduce operations) during distributed LLM training.
CXL (Compute Express Link)
While RDMA handles network transfers between separate servers, CXL handles high-speed, cache-coherent interconnects inside the box or across adjacent rack chassis over physical PCIe 5.0/6.0 lanes.
CXL allows CPUs, GPUs, and expansion cards to share memory with sub-100 nanosecond latency using standard CPU Load/Store instructions. It runs via three sub-protocols:
CXL.io: Standard PCIe signaling and device discovery.
CXL.cache: Low-latency caching for attached accelerators.
CXL.mem: Allows host CPUs to treat remote expansion memory cards as local DRAM.
FAM (Fabric-Attached Memory)
Fabric-Attached Memory (FAM) shifts server design from CPU-centric to memory-centric. Instead of locking DRAM inside individual CPU sockets, large pools of memory sit directly on a shared CXL 3.0 or fabric switch. Any attached compute node (CPU, GPU, or DPU) can dynamically borrow byte-addressable memory from the shared pool, eliminating "stranded memory" across the data center.
2. Disaggregated Storage Architecture
NVMe-oF & JBOF
To scale enterprise and AI storage without forcing hosts to pack dozens of internal drives, modern systems decouple compute from physical flash:
JBOF (Just a Box of Flash): A passive, high-density 2U rack enclosure housing 24–32 enterprise NVMe SSDs with high-speed SmartNICs or DPUs—without running a bloated server OS.
NVMe-oF (NVMe over Fabrics): The protocol layer extending native NVMe commands over network fabrics (RoCEv2, InfiniBand, or TCP).
How they work together: Host compute nodes talk to the remote JBOF enclosure over NVMe-oF using RDMA, making remote cloud drives appear to local Linux kernels as standard, low-latency /dev/nvme block devices.
3. The AI Data-Plane & Distributed Transfer Layer
NVIDIA NIXL (Inference Transfer Library)
Transferring prompt context and attention states across heterogeneous memory types (GPU HBM, Host DRAM, NVMe SSDs, remote nodes) requires complex networking code.
NIXL provides an open-source, vendor-agnostic unified API for data movement in distributed inference clusters. It decouples transfer orchestration (the Conductor) from hardware execution (the Transfer Agent), automatically selecting the fastest transport backend—whether GPUDirect Storage, RoCE RDMA, or NVMe-oF—for zero-copy tensor movement.
4. LLM Inference Runtimes & Memory Orchestration
KV Cache & LMCache
During LLM generation, computing self-attention ($Q \times K^T \times V$) across past tokens is computationally expensive.
KV Cache: Caches calculated Key ($K$) and Value ($V$) tensors in GPU VRAM after the initial prompt prefill phase, reducing token generation complexity from quadratic $O(N^2)$ to linear $O(N)$.
LMCache: An open-source distributed storage layer that pools and offloads KV caches out of expensive GPU HBM down to host DRAM or NVMe drives. It allows different GPU nodes to share pre-computed KV cache states across the network, skipping redundant prefill compute.
NVIDIA KVBM (Key-Value Block Manager)
Operating inside distributed engines like NVIDIA Dynamo, KVBM manages multi-tier KV cache memory (GPU HBM $\rightarrow$ CPU DRAM $\rightarrow$ NVMe SSDs $\rightarrow$ Remote Storage). Using NIXL underneath, KVBM enables seamless Prefill-Decode (PD) Disaggregation, where dedicated prefill worker nodes process long prompts and pass the KV cache blocks straight to decode worker nodes over RDMA.
vLLM & PagedAttention
vLLM is the open-source serving engine that revolutionized LLM inference by introducing PagedAttention.
By adapting virtual memory paging from operating systems, PagedAttention partitions KV cache tensors into non-contiguous physical memory blocks. This eliminates internal memory fragmentation, reducing GPU memory waste from over 60% down to under 4%, allowing inference servers to run at 2x–4x higher request concurrency.
Key Latency Metrics: TTFT
When measuring LLM serving performance, TTFT (Time To First Token) tracks the duration (in milliseconds) from when a user submits a prompt to when the model streams its very first token back. TTFT measures system responsiveness, covering network transit, queueing delays, and prompt prefill computation.
5. Frontier Open Models: GLM-5.0
Frontier open-weight architectures like GLM-5 demonstrate why these infrastructure advances are necessary.
Developed by Zhipu AI / Z.ai, GLM-5 is a 744B total parameter (40B active) Mixture-of-Experts (MoE) foundation model. Built for autonomous agentic engineering, repository-scale code generation, and multi-file editing, GLM-5 leverages DeepSeek Sparse Attention (DSA) and a native 1-Million-token context window—demanding high-bandwidth RDMA networks and efficient KV cache managers like LMCache and KVBM to serve at scale.
6. Summary: The Modern AI Infrastructure Stack
When building high-throughput AI platforms, these technologies stack together logically from metal to software application:
| Layer | Primary Technologies | Core Purpose |
| Application / Agent | Claude Code, Cursor, OpenClaw | User interaction and task execution |
| Model Weight Architecture | GLM-5, DeepSeek-V3, Llama 3 | Base reasoning engine |
| Serving Runtime | vLLM (PagedAttention), TensorRT-LLM | High-throughput GPU execution & local memory allocation |
| KV Memory Orchestration | KVBM, LMCache | Multi-tier, cross-node KV cache sharing & offloading |
| Data Movement Library | NVIDIA NIXL | Transport abstraction for zero-copy tensor transfers |
| Physical Interconnects | RDMA (RoCE/InfiniBand), CXL 3.0, NVMe-oF | Sub-microsecond hardware networking and memory pooling |
| Hardware | GPUs (HBM), JBOF SSD Enclosures, FAM Pools | Physical compute and storage medium |
Posted by Madhukar Rupakumar on
