Research analysis · Spiking reinforcement learning

Modules without backpropagation, and what they are really for

Biological circuits are recurrent and modular; the spiking networks we train to act in the world are usually feed-forward stacks tuned by backpropagation. Ramavarapu, Udupi and Gautam introduce Community Recurrent SNNs, in which a recurrent network organizes itself into a controller community, latent communities and one community per action, all trained by local three-factor plasticity with no weight transport. The architecture matches or beats its feed-forward ancestor on three control benchmarks, and a pre-registered ablation of 240 trained models delivers the uncomfortable part: the latent modules, the pieces that look most like internal computation, turn out to be sinks, not computers.

Source: Recurrent Modular Structure in Spiking Neural Network Reinforcement Learning, arXiv:2611.14262v1, 10 November 2026. Primary source. Read: the full PDF, including methods, Tables 1 and 3 to 5, the modularity sweep, and the appendix statistics.

What the work claims

This is a primary results paper, a collaboration between the University of Illinois Urbana-Champaign and Oak Ridge National Laboratory, and it makes two claims that deserve to be weighed separately.1 The first is architectural: a recurrent spiking network partitioned into communities with fixed roles, a controller that receives the stimulus, latent communities that carry internal state, and one action community per possible action, can be trained to control standard reinforcement-learning tasks using only local learning rules, and can match or exceed the corresponding feed-forward actor trained the same way. The action is read out as a race between action communities' spike counts rather than a softmax, and information flows as a walk between communities rather than layer by layer.

The second claim is about how the modules should be obtained. The authors introduce Map-STDP, a three-factor rule in which a global TD reward signal gates local eligibility traces, as in the TD-STDP actor-critic they build on, plus a structural term derived from the map equation, an information-theoretic measure of the description length of a random walk over the network. Minimizing that description length during learning pushes weight into dense intra-community connections and sparse inter-community links, encouraging modular structure to form rather than imposing it. They compare this dynamic regime against partial modularity, a stochastic block model initialization with a cross-community connection probability of 0.05, and full modularity, in which cross-community synapses are sealed entirely.

The headline numbers, all at 20 seeds per condition: on CartPole, adding the map term to the standalone spiking agents improves mean training return by 65 to 87 points across the three spiking neuron models, and matches fully sealed modularity on two of the three. Dropped into the existing TD-STDP actor-critic of Chung and Kozma, the recurrent action communities beat the original feed-forward actor on CartPole by 25 to 45 AUC points and on LunarLander by 36 to 41 points of final return, and match it on Acrobot. Energy and compressibility move together: the effective synaptic operations needed per decision correlate with the description length of the observed spike walk at Spearman rho of 0.61 to 0.95 across engines, so more compressible activity is measurably cheaper activity.

How it works

The map equation assigns any partition of a graph a code length for describing a random walk on that graph, and the partition that best contains the walk minimizes it. Map-STDP turns that global objective into a local weight update: each synapse adjusts according to whether strengthening it moves the network's activity toward a more compressible community structure, with a weighting lambda of 0.05 in the dynamic regime, while the TD term keeps the network solving the task. Nothing in the rule needs a global view of the network: each synapse sees presynaptic and postsynaptic spikes, an eligibility trace, and two broadcast scalar signals. That locality is the entire point, because it is the property backpropagation lacks and the property any learning rule running on neuromorphic hardware, or in living tissue, would require.

The evaluation is unusually careful about what would count as evidence. The agents are built on four neuron engines, three spiking (quadratic integrate-and-fire, leaky integrate-and-fire, and a branching process model) plus one flow engine with no spikes at all, so a conclusion that only held for one spike model would be exposed. Statistical comparisons use two-sided Mann-Whitney tests with Holm adjustment across the four engines. The inter-module connectivity sweep spans cross-community probabilities from 0 to 0.5 at 7 values, with and without the map term, 140 final networks in total, and finds the optimum at or near zero for all three tasks, with a small exception: LunarLander peaks at 0.025 with the map term on, which the authors read as actions that depend on each other benefiting from a trickle of cross-module traffic.

The ablation is the paper's real contribution to methodology. Every trained network from the standalone experiments, 240 models across engines and regimes, was evaluated intact and under three lesions: cutting only the latents' outputs, removing the latents entirely with outflow renormalized, and cutting the direct controller-to-action synapses to leave only the latent path. Evaluations ran 50 episodes per model with common random numbers, so each model is its own control, and the design was pre-registered. The result overturns the intuitive reading of the architecture. With only the latent path available, every configuration collapses to roughly random play, returns of 10 to 32 against 22.8 for a random policy. The policy lives in the direct controller-to-action routing. The latents instead act as a sink: they absorb 51 to 92 percent of the controller's drive in the trained networks, and removing them reroutes that drive straight into the action communities, degrading the policy. In some regimes cutting latent output actually improved return, meaning it removed interference.

Where a skeptic should push

The single most load-bearing assumption is that a random walk is a good abstraction of inference in these networks. The authors flag it themselves: random walks are inherently sequential, so computing the walk statistics that drive Map-STDP becomes a bottleneck as networks scale, and the flow engine, the one whose dynamics most literally match the walk abstraction, also learns best. That correlation cuts both ways. It may mean the learning rule transfers gracefully to spiking hardware that approximates walk statistics, or it may mean part of the reported gain is the map term steering networks toward the regime where the approximation is most valid, a subtle form of self-consistency that a spiking-chip replication would settle.

Second, the description-length story is more modest than it first appears. Compressibility largely followed modularity itself: the fully sealed regime, which has no map term at all, lands within 0.2 bits of the dynamic regime on two engines. The authors conclude honestly that objectives rewarding modularity directly might do the same job without the information-theoretic machinery, which demotes Map-STDP from a fundamental principle to one working instance of a broader class. Third, the benchmarks are small classical control tasks, and LunarLander in particular pooled 10,000 episodes over 16 parallel environment copies, a training budget that flatters sample efficiency claims. Fourth, the latent communities, the element with the most biological romance, do not yet compute anything the policy needs; they regulate how much drive reaches the actions. The paper's own discussion says the field should train them to carry memory before reading cortical parallels into them. Finally, everything is simulation in NumPy and Numba; no neuromorphic hardware was involved, so the energy story rests on effective synaptic operation counts, not silicon.

Modularity claims in tissue need the lesion, not the map

For organoid intelligence, the first implication is about the training channel, not the architecture. If living neural tissue is ever to be trained on a task rather than merely read out, the learning signal available to it will look like Map-STDP and nothing like backpropagation: local synaptic eligibility, a delayed broadcast modulatory factor, and a reward or prediction-error term that arrives seconds after the spikes it must credit. This paper demonstrates that such a channel is not a crippled approximation; with a structural term added, it grows functional organization that a backpropagation-free system could never be handed by construction. That is the strongest simulation-side evidence to date that a neuromodulatory closed loop over an organoid, the kind already feasible with dopaminergic drugs or patterned stimulation, is a plausible training interface rather than a metaphor for one.

The second implication is the uncomfortable one, and it should be printed on every organoid functional-mapping paper. The CoRe networks were designed to have latent modules carrying internal state, they demonstrably developed dense-community-sparse-between wiring, compressible dynamics, and improved task performance, and yet when the authors actually lesioned the parts, the function was in the direct routing all along and the latents were a drive-regulating sink that in some regimes hurt. Organoid studies routinely infer function from structure that looks modular, whether that is regional transcriptomic domains, patterned activity clusters on a microelectrode array, or bursts that seem to route through particular tissue sectors. This paper is a worked example of how far wrong that inference can go even in a system you built and can ablate at will, in 240 pre-registered models with common random numbers. In tissue, where lesioning is harder and interpretation is clouded by developmental variability, the distance between an attractive modular activity map and a load-bearing circuit is likely larger, not smaller.

The opportunity and the threat follow directly. The opportunity: the race readout between action communities is equivalent to softmax sampling under constant hazards, and a comparable population-code race is a far more plausible decode-and-stimulate contract with an organoid than a vector readout, because it tolerates the rate variability that living tissue imposes. The description-length metric is also a free audit instrument for the OI field, which currently lacks any standard way to say whether an organoid's activity became more organized across a training run. The threat: Map-STDP's gains are entangled with how well the random-walk abstraction fits, and living tissue violates that abstraction in both directions, with dendritic nonlinearities the model ignores and with variability the model does not have. A neuromodulatory training loop on an organoid could easily reorganize the tissue's activity in ways that raise compressibility without raising computational capacity, exactly the dissociation this paper's own ablation exposed.

The bottom line

Established: recurrent spiking actors with modular communities can be trained by purely local three-factor rules, can match or beat a matched feed-forward actor on classical control tasks at 20 seeds, and show a tight coupling between the compressibility of activity and its energetic cost. Established with caveats: the modules that look like internal computation are currently drive-regulating sinks, and the structural objective mostly amplifies modularity that sealing alone would impose. Not established: hardware results, scalability past small networks, or any demonstration that latent communities can be made to genuinely compute. What would confirm the organoid relevance is a closed-loop neuromodulatory experiment in which an eligibility-trace-compatible signal, delivered over seconds-matched delays, reorganizes organoid activity in the direction of a task, with description length of the spike walk tracked as the audit metric. What would break it is the lesion pattern repeating in tissue: task performance surviving the removal of the regions whose activity maps looked most structured.

Frequently asked questions

What is a Community Recurrent SNN?

A spiking network organized into communities with fixed roles: a controller community receives the stimulus, latent communities carry internal state, and each possible action has its own community. The action is whichever action community wins a race of spike counts, so computation is a walk between communities rather than a feed-forward pass.

What does Map-STDP add to ordinary reward-modulated STDP?

Ordinary three-factor rules gate a synaptic eligibility trace with a global reward signal. Map-STDP adds a structural term derived from the map equation, which measures the description length of a random walk over the network. Minimizing it during learning encourages weights to form dense intra-community and sparse inter-community patterns, so modules emerge from learning rather than being imposed.

Did the latent modules turn out to be doing the computing?

No, and this is the paper's most instructive negative result. Forcing activity through the latent path collapsed every trained network to random play, while cutting the latents' outputs often changed little or even helped. The latents act as a sink that sets how much of the controller's drive reaches the action communities; the policy itself lives in the direct routing.

How strong is the evidence?

Reasonably strong for the claims made: 20 seeds per condition, four neuron models including one non-spiking control, Holm-adjusted Mann-Whitney tests, a pre-registered 240-model ablation with common random numbers, and a 140-network connectivity sweep. The limits are that everything is simulation, the tasks are small classical benchmarks, and no neuromorphic hardware was tested.

Why does this matter for brain organoids?

Two reasons. The training rule is the right kind for living tissue: local eligibility traces plus a delayed broadcast modulator is the only plausible contract a closed loop can offer an organoid, and this paper shows such rules can build functional organization. And the ablation is a warning: even in a designed system, the anatomically obvious modules were not where the computation lived, which raises the evidential bar for functional claims about organoid regions.

What is the map equation?

An information-theoretic measure from network science that gives the code length needed to describe a random walk on a graph, given a partition into communities. It is shortest when communities contain the walk, so it converts the vague goal of modular structure into a quantity that can be optimized.

References

  1. V Ramavarapu, H Udupi, A Gautam. Recurrent Modular Structure in Spiking Neural Network Reinforcement Learning. arXiv:2611.14262. 2026. http://arxiv.org/abs/2611.14262v1. Accessed 2026-11-11.