Sparse event-driven language models cut MACs by up to 76%

What the work claims

The authors claim that activation sparsity can be induced across all linear projections of a quantized linear-attention language model, not just the feed-forward down-projection, and that the resulting sparse model is a natural fit for event-driven neuromorphic hardware. Their method uses a learnable two-sided threshold: activations within plus-or-minus delta are zeroed, high-magnitude outliers are preserved, and a smooth surrogate gradient keeps training stable.¹

For the 370M-parameter MMFreeLM model, the sparsified variant at sparsity penalty lambda equals 1 uses 118M effective MACs versus 307M for the dense baseline, a 61.6% reduction, and achieves an average zero-shot accuracy of 41.24% on ARC-Easy, HellaSwag, OpenBookQA, and PIQA, compared with 41.88% for the dense model. At lambda equals 2, MACs fall to 95M, a 69.1% reduction, and average accuracy falls to 39.74%. For the 2.7B model, lambda equals 1 reduces MACs from 2.32B to 1.01B, a 56.5% reduction, with average accuracy dropping from 50.59% to 48.50%.¹

Projected onto a multi-chip Loihi 2 deployment, the 370M sparse model is estimated to generate tokens at 224.1 per second at 75.0 millijoules per token, compared with 41.5 per second at 405 millijoules per token for the dense MMFreeLM baseline on the same hardware. The authors also compare with edge-GPU deployments of transformer models of comparable downstream performance and project up to 37 times higher throughput and 16 times lower power.¹

This is a methods-and-benchmarking paper, combining a sparsification technique, a sensitivity analysis, and an extrapolated hardware performance model. The Loihi 2 sparse numbers are projections from measured dense results, not fresh silicon measurements.

How it works

The base architecture is MMFreeLM, a MatMul-Free Language Model that replaces self-attention with a MatMul-Free Linear-GRU, or MLGRU, block and pushes quantization aggressively: ternary weights of minus one, zero, or plus one, and 8-bit fixed-point activations. Dense layers therefore become BitLinear layers in which multiplication reduces to addition or subtraction. The model already runs on GPUs, edge GPUs, and Loihi 2, making it a useful testbed for neuromorphic deployment.¹

The authors first run a sensitivity study by forcing top-k sparsity on each projection type and each layer of a pretrained 370M model. They find that projections involved in the hidden-state transition, conventionally labeled I and F, are the most sensitive to sparsity, while output projections based on the current state and input are more robust. The large down-projection in the feed-forward block is the most tolerant. Early and late layers are more sensitive than middle layers. This sensitivity map guides where to push sparsity hardest.¹

The sparsification method itself is a pre-activation gate. For each linear projection, activations inside a learnable range plus-or-minus delta are set to zero, while activations outside the range are shifted by the threshold and kept. The threshold is learned per projection. To make the zero count differentiable, the authors add an L0 surrogate loss based on an exponential decay around zero; this estimates the fraction of zero activations without blocking gradients. The penalty weight is ramped up linearly over the first 5% of training steps to avoid premature sparsity.¹

Training continues the pretrained 370M and 2.7B MMFreeLM models on 4B tokens of FineWeb-Edu with a reduced learning rate. The sparse models are then evaluated on zero-shot reasoning tasks and their per-projection activation densities are measured. These densities feed a multi-chip Loihi 2 performance model that accounts for two effects of sparsity: skipping zero MACs inside each core, and suppressing zero-valued packets across the event-driven interconnect between chips.¹

Where a skeptic should push

The most load-bearing assumption is that the Loihi 2 sparse results are projections, not measurements. The authors start from measured dense throughput on a 24-chip Loihi 2 system and scale it by modeled intra-chip MAC density and inter-chip communication density. The model is plausible, but real sparse deployment on a message-passing many-core system can suffer from load imbalance, serialization, and synchronization overhead that analytical models miss. The 5.4 times generation speedup and 37 times edge-GPU advantage should be treated as best-case estimates until silicon runs are reported.

Second, the accuracy evaluation is on a narrow set of English reasoning benchmarks: ARC-Easy, HellaSwag, OpenBookQA, PIQA, and WinoGrande, with Table III reporting only ARC-Easy, HellaSwag, OpenBookQA, and PIQA averages. A model that preserves average accuracy on these tasks may still degrade on long-context recall, coding, or multilingual tasks. The paper does not test those.

Third, the comparison with edge GPUs is not head-to-head on identical models. The Loihi 2 sparse numbers are for MMFreeLM, a specialized linear-attention architecture, while the edge-GPU comparisons use Qwen2 500M and Alireo 400M, different model families with different capabilities. The throughput and power ratios therefore mix architecture effects, sparsity effects, and hardware effects.

Fourth, the sparsity penalty is a tuning knob with a real cost. At lambda equals 2 the 370M model's average accuracy drops by about two percentage points. That may be acceptable for some applications, but it is not free, and the trade-off will be task-dependent.

Billion-parameter sparse models set the bar for wetware

The non-obvious implication for organoid intelligence is that event-driven computation is no longer a niche idea for small networks. The paper projects that a 370M-parameter language model, sparsified and quantized, can run with average MAC density near 0.20 and generate tokens at 224.1 per second on neuromorphic hardware. That is a billion-parameter-class computation happening in an event-driven substrate. For organoid computing, this raises the bar for what a living neural network would have to match to be competitive as a general-purpose processor.

The opportunity is methodological. The paper provides a concrete recipe for co-designing a model with its target hardware: measure per-projection sensitivity, learn a sparse activation distribution, then project the gain onto an event-driven platform. Organoid interfaces currently lack this level of co-design. Stimulus patterns and readout weights are usually chosen by hand or with simple optimization, not by jointly tuning the input encoding, the living network, and the decoding layer for a task. The same sensitivity-analysis plus learnable-sparsity framework could, in principle, be adapted to find stimulation patterns that drive an organoid into sparse, task-representative activity while keeping its metabolic cost low.

The threat is that neuromorphic silicon is approaching the efficiency regime that organoid computing hoped to own. If a 370M-parameter sparse model can run at 75 millijoules per token and 224 tokens per second on Loihi 2, then the practical argument for biological computing must shift from raw efficiency to something only living tissue can provide. Plausible candidates include continuous learning from few examples, robustness to distribution shift, or the ability to integrate chemical and electrical signaling. The paper does not test any of these, but it frames the question sharply: organoids must demonstrate a capability gap, not just an efficiency dream.

There is also a warning about metrics. The authors report MAC reduction, throughput, and energy per token. Organoid research often reports spike rates, firing synchrony, or classification accuracy, but rarely the full cost-accounting that would allow comparison with silicon. If organoid intelligence wants to claim a real advantage, it needs to report energy or metabolic cost per useful operation, latency distributions, and failure modes with the same rigor shown here. Otherwise the field will keep comparing beautiful biology against engineering systems that already publish their numbers.

The bottom line

Established: a learnable per-projection activation sparsity gate can reduce effective MACs in quantized MMFreeLM by up to 76% while holding zero-shot reasoning accuracy within a few points of the dense baseline. The 370M model at lambda equals 1 achieves 41.24% average accuracy on the reported benchmarks with 118M MACs, down from 41.88% with 307M MACs. Projected Loihi 2 deployment gives 224.1 tokens per second at 75.0 millijoules per token in generation, a 5.4 times improvement over the dense baseline on the same hardware.

For organoid intelligence, the work is both a benchmark and a methodological template. It shows the scale and efficiency that event-driven hardware can already reach, and it demonstrates a systematic way to co-design sparsity with a platform. The honest next question is whether living neural tissue, interfaced with comparable co-design, can do something that sparse silicon cannot.