The 30 percent energy hole hiding in the operating system

What the work claims

The paper claims that automatic Non-Uniform Memory Access (NUMA) balancing, a Linux kernel feature meant to reduce remote memory accesses on multi-socket servers, can degrade the performance and energy efficiency of large spiking network simulations by a large margin. The headline number is that turning it off may reduce energy consumption by 30 percent. The authors call this cost-effectiveness advantage larger than many attempts to increase data-center energy efficiency through hardware or cooling improvements.¹

The effect is shown on the multi-area model (MAM) of the macaque visual cortex, a large-scale spiking network comprising one square millimeter of each of 32 visual cortical areas, simulated with NEST 3.10 on JURECA-DC. The benchmark used 16 MPI processes with 64 threads each on 8 compute nodes. Each JURECA-DC node has two 64-core AMD EPYC 7742 processors and 512 GiB of memory. With automatic NUMA balancing on and the jemalloc allocator, the maximum observed simulation-cycle time was 222.84 milliseconds; with NUMA balancing off, it dropped to 99.98 milliseconds. The standard deviation of cycle times fell from 0.83 milliseconds to 0.42 milliseconds. The reduction in real-time factor amounts to approximately 30 percent, and even in the second half of the simulation the reduction is 28 percent.¹

Importantly, the authors show that the long-tail cycle times are not caused by network dynamics. Plotting cycle time against the spike count of the previous cycle reveals no correlation for the longest cycles when NUMA balancing is on. Spike count accounts for only 17 percent of cycle-time variability when NUMA balancing is off. The cause is therefore at the machine level, outside the simulation code itself.¹

How it works

A NEST simulation cycle has two phases with different memory-access patterns. Spike delivery and neuron update are fully thread-parallel across all cores of a socket. Collocation and global spike communication are single-threaded, executed by the master thread of each MPI process. The master thread reads spike-buffer data that were written in parallel by threads across the four NUMA domains of the socket.¹

Automatic NUMA balancing periodically unmaps memory pages and records which NUMA domain faults on each page, then migrates pages toward the domain that accesses them most. The scan period ranges from one to sixty seconds, while one simulation cycle lasts only a few milliseconds. Because the simulation alternates between fully parallel and single-threaded phases, the phase in which a page is first accessed after an unmap is largely coincidental. The kernel therefore migrates pages based on a single access that may not reflect the phase dominating execution time, incurring overhead without lasting benefit.¹

The authors' new diagnostic is a time- and compute-node-resolved heatmap of cycle times, which exposes an initial transient of elevated cycle times and a persistent horizontal two-tone stripe pattern across MPI processes. The initial transient matches the adaptive scan period of NUMA balancing, which starts frequent and then throttles back. The persistent asymmetry between processes remains not fully understood. The team has since added a Slurm option, #SBATCH --numa-balancing=0, that lets users disable automatic NUMA balancing on a per-job basis without administrator intervention.¹

Where a skeptic should push

The 30 percent figure is a single-system result on one supercomputer, JURECA-DC, with one benchmark model and one software stack. The authors note that it remains to be investigated how widespread the phenomenon is among scientific codes. NEST's communication pattern, with frequent collective communication between nodes and symmetrical parallel action of all threads in between, may be an extreme case. So the headline number should be read as an upper bound for this class of simulation, not a universal constant for all spiking network code.¹

The persistent two-tone process asymmetry is unexplained. The authors acknowledge this and say it deserves further study. That means part of the energy loss is not yet tied to a mechanism, so the 30 percent reduction may not be fully actionable on every system even if NUMA balancing is disabled.

Second, the study is about simulation on conventional supercomputers, not about the intrinsic efficiency of neuromorphic hardware. The authors use it to argue that conventional simulation technology is still far from hardware limits, which is fair, but a reader should not conflate this with a claim that CPUs beat neuromorphic chips. The point is that the reference simulations used to design and validate neuromorphic architectures are themselves wasteful in an avoidable way.

Third, the model has known stresses: some brain areas show higher synchronization than in nature, which increases communication load. Improved brain-scale models might ease this and reduce the absolute savings. The direction of the effect is robust across node counts and random seeds in the strong-scaling data, but the magnitude could shift.

What it changes for the cost of validating organoid architectures

Organoid intelligence lives in a research pipeline that is mostly simulated. Architectures for stimulating, reading, and training living neural tissue are first prototyped in silico, benchmarked against large spiking network models like the cortical microcircuit or the multi-area model, and only later mapped to wetware. This paper says that a large slice of the energy bill for that pipeline is a software-configuration bug, not a physics limit. If the field's reference simulations are running 30 percent slower than necessary, then the carbon and dollar cost of every architecture search, every surrogate-gradient study, and every closed-loop policy optimization is inflated by the same fraction.¹

The non-obvious implication is institutional. Supercomputing centers generally tune for dense matrix workloads, where NUMA balancing is often helpful. Spiking network simulation has the opposite memory-access signature: sparse, event-driven, with rapid alternation between parallel and serial phases. A facility optimized for dense AI may be systematically misconfigured for neuroscience simulation. For organoid-intelligence groups that buy time on shared clusters, this means the default queue settings may silently penalize their workloads. The paper's per-job Slurm switch is a immediate fix; the deeper fix is making HPC procurement and configuration sensitive to sparse, spiking workloads.

The threat is complacency. If a 30 percent energy win has been sitting in the kernel for years, unnoticed because simulation results were correct and fast enough, then other large wins are likely hiding in the software stack. The paper itself says variability in cycle times remains on the order of the mean even after NUMA balancing is disabled, suggesting further room for improvement. For a field that often justifies organoid computing by silicon energy limits, ignoring known software inefficiencies undermines the comparison. A fair accounting of wetware's energy advantage must start from an optimized conventional baseline, and this paper shows the baseline is not yet optimized.

The opportunity is methodological. The cycle-resolved timers and heatmap diagnostics introduced here could be applied directly to organoid-interface simulators. If a team is training a closed-loop controller for a dish using a surrogate model, the same NUMA and variability effects may be present. Measuring cycle times per process and per simulation step, rather than only end-to-end wall time, is now a proven way to catch operating-system interference that would otherwise be invisible.

The bottom line

Established: on JURECA-DC, disabling automatic NUMA balancing removes a long tail in NEST simulation-cycle times and reduces the real-time factor of a macaque multi-area model by about 30 percent, with the effect persisting across node counts and random seeds. The cause is an interaction between the kernel's page-migration heuristic and the alternating parallel/serial memory-access pattern of the NEST simulation cycle, not network dynamics. Open: how widespread the effect is across other supercomputers, other spiking simulators, and other models, and whether the persistent process asymmetry can be eliminated.

For organoid intelligence, the message is that the in-silico reference pipeline is more expensive than it needs to be. What would confirm the relevance: a reproduction on other HPC systems used by the field, and a measurement showing that the same setting speeds up organoid-controller training workloads. What would limit it: if the effect is specific to NEST's hybrid MPI/OpenMP structure and large all-to-all spike communication, then the savings may not transfer to smaller-scale or GPU-based organoid simulations.