The Memory Fragmentation Problem in LLM Inference Clusters
In multi-tenant LLM serving clusters, allocating static VRAM blocks for varying prompt context lengths results in severe memory fragmentation. While some worker nodes run out of VRAM due to long-context sessions, neighboring nodes sit idle with 40% unallocated memory. Static allocation caps overall cluster concurrency and inflates infrastructure costs.
InferCluster Distributed KV-Cache Memory Architecture
InferCluster introduces a unified, cluster-wide KV-cache memory pool. Utilizing high-speed InfiniBand fabrics, worker nodes can store and retrieve overflow KV-cache blocks from remote GPU VRAM or pooled CXL-attached memory tiers on neighboring cluster nodes with microsecond access latencies.
PagedAttention over Fabrics (PoF)
Extending local PagedAttention principles across network boundaries, InferCluster implements PagedAttention over Fabrics (PoF). KV-cache blocks are organized into standardized memory pages mapped across cluster nodes. When an attention forward pass requires historical KV tokens stored on a remote node, the local GPU fetches the blocks via RDMA on-demand during attention calculation.
Cache-Aware Load Balancing and Locality Optimization
To minimize network hops, InferCluster's scheduler pairs distributed memory pooling with cache-aware request routing. Incoming queries are preferentially routed to nodes holding prompt prefix KV-caches locally, falling back to remote cluster memory pooling only when local VRAM saturation is reached.
