# NUMA-Torch: Compiler-Driven NUMA-Aware LLM Prefill on Multi-Socket CPUs

DevFeed: [NUMA-Torch: Compiler-Driven NUMA-Aware LLM Prefill on Multi-Socket CPUs](<https://devfeed.tech/articles/numa-torch-compiler-driven-numa-aware-llm-prefill-on-multi-socket-cpus-121261.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07105>)

Author: Thisara Rathnayaka Weerakoon (Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka), Ishan Dewpura (Department of Computer Science & Engineering, University of Moratuwa, Sri

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [NUMA](<https://devfeed.tech/topics/numa.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [affinity](<https://devfeed.tech/tags/affinity.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [banks](<https://devfeed.tech/tags/banks.md>), [bert](<https://devfeed.tech/tags/bert.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cpu-affinity](<https://devfeed.tech/tags/cpu-affinity.md>), [cpus](<https://devfeed.tech/tags/cpus.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [data](<https://devfeed.tech/tags/data.md>), [data-locality](<https://devfeed.tech/tags/data-locality.md>), [dram](<https://devfeed.tech/tags/dram.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [llm](<https://devfeed.tech/tags/llm.md>), [numa](<https://devfeed.tech/tags/numa.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [socket](<https://devfeed.tech/tags/socket.md>), [torch](<https://devfeed.tech/tags/torch.md>)

## AI overview

NUMA-Torch targets multi-socket CPU LLM prefill bottlenecks caused by weights being accessed across NUMA sockets, which reduces effective DRAM bandwidth. Its NUMA-aware Tensor Parallelism treats each socket as a partition, shards weights across sockets, and uses CPU affinity to keep thread pools and memory allocations local. The abstract reports evaluations on GPT-2, BERT, and Gemma 3 270M, with speedups of up to 2.63x on dual-socket Haswell, 1.87x on dual-socket Cascade Lake, and 2.27x on four-socket Haswell-EX. These are reported maxima; the excerpt does not provide per-model results or broader evaluation details.

## Source excerpt

arXiv:2611.07105v1 Announce Type: new Abstract: For sufficiently large transformer-based Large Language Models (LLMs), whose parameters cannot be retained in on-chip caches, prefill throughput on multi-socket CPUs is often limited by sustainable DRAM bandwidth rather than peak floating-point throughput. In the standard PyTorch compilation (i.e., TorchInductor) path, model weights are allocated without considering the Non-Uniform Memory Access (NUMA) topology: specifically, the affinity between memory banks and compute cores is disregarded. This causes a substantial fraction of weight accesses to traverse the inter-socket interconnect, severely degrading the effective memory bandwidth available to each socket. We address this bottleneck through a compiler-level solution that makes TorchInductor NUMA-aware. Treating each NUMA socket as an independent tensor-parallel partition, we introduce NUMA-aware Tensor Parallelism (NUMA-TP). This approach transparently shards weights across sockets and schedules compute kernels such that thread pools and memory allocations remain strictly local to their respective sockets using CPU affinity. We evaluate the approach on GPT-2 (117M, 345M, 762M) and BERT (Small, Large), which exercise standard multi-head attention and MLP, as well as Gemma 3 270M, which combines Multi-Query Attention with a GeGLU-gated MLP. By strictly enforcing data locality, our approach successfully mitigates cross-socket traffic, achieving prefill speedups of up to 2.63x on a dual-socket Haswell platform, up to 1.87x on a dual-socket Cascade Lake platform, and up to 2.27x on a four-socket Haswell-EX platform.