Weight Placement Is Not Enough: Synchronization in Multi-Node CPU Decoding of Ternary LLMs

Ternary LLMs read only a few gigabytes per decoded token, yet on dual-socket servers with four NUMA nodes the fastest ternary runtime we found uses less than half of the memory bandwidth. Placing each weight on the node of the thread that reads it is necessary but not enough: in a decode engine with