A spiking actor network matches Soft Actor-Critic on continuous control

What the work claims

The authors claim that a spiking neural network can replace the actor network in Soft Actor-Critic without sacrificing performance on a continuous control task. They keep the critics as conventional networks, convert only the policy actor to spikes, and report near-equivalent mean reward across four hidden-dimension configurations on the Farama Foundation Bipedal Walker environment.¹

How it works

Soft Actor-Critic is an off-policy reinforcement-learning algorithm that learns a stochastic policy, called the actor, and one or more action-value functions, called critics. The actor maps states to distributions over continuous actions, and the critics estimate the expected return of state-action pairs. In SANSAC the two critics remain ordinary deep networks; only the actor is replaced by a spiking neural network built with leaky-integrate-fire neurons.

The actor has three fully connected layers. The hidden layers use standard LIF dynamics governed by a membrane time constant, a resting potential, and a membrane resistance. The output layer uses non-spiking leaky-integrate neurons, which produce a continuous voltage rather than a spike train. That voltage is decoded into the continuous action space, avoiding the awkward step of reading out spike counts for motor commands.

Training uses surrogate gradients. The binary threshold of a spiking neuron is non-differentiable, so the authors replace its true gradient with a sigmoid surrogate during backpropagation through time. Apart from that substitution, the loss functions and entropy regularisation follow the standard SAC formulation. The implementation uses the SpikingJelly library, and the code is public.

The experiments compare SAC and SANSAC on four hidden-dimension pairs: 256/256, 200/200, 128/128, and 64/64. Each configuration is run across 10 random seeds for up to 1200 episodes, with early stopping if no reward improvement occurs for 100 consecutive episodes after the first 50. The task is considered solved when an agent accumulates at least 300 reward points in 1600 steps.

Where a skeptic should push

The most load-bearing assumption is that the Bipedal Walker result generalises to the continuous-control problems where neuromorphic hardware is actually needed. The paper tests only one environment. Bipedal Walker is a useful benchmark, but it is not a robot arm, a drone, or a closed-loop sensory-motor task with partial observability.

Second, the comparison is performed entirely on conventional hardware. The authors note that SANSAC training takes roughly twice as long as SAC training, 61 versus 21 minutes on average across configurations, because temporal unrolling is expensive on a sequential processor. They explicitly state that they do not measure energy on conventional hardware because the reporting was inconsistent. The hoped-for energy benefit therefore remains theoretical.

Third, the statistical equivalence is broad. At the largest hidden dimension of 256/256, SANSAC achieves a mean test reward of 273.9 plus or minus 130.8 standard deviation, while SAC achieves 224.5 plus or minus 162.9. The Mann-Whitney p-value is 0.7913 and Cohen's d is 0.080, which is negligible. But the high variance means neither algorithm is reliably solving the task; success rates are 70 percent for SANSAC and 80 percent for SAC.

Fourth, the actor is only one component of the deployed system. Keeping the critics as deep networks during training means SANSAC is a hybrid training scheme, not a fully neuromorphic training scheme. The final deployed agent discards the critics, but the training pipeline still relies on conventional backpropagation through large networks.

OI implication: control policies for embodied wetware

For organoid intelligence and biological computing, SANSAC is a useful existence proof. It shows that a spiking actor can learn a continuous control policy using standard reinforcement-learning machinery. If the goal is to drive electrodes, microfluidic valves, or mechanical actuators from a living neural culture, the policy network that maps neural readouts to actions could in principle be a spiking network trained in this way.

The opportunity is methodological. Organoid intelligence has devoted much effort to readout decoding and to in-vitro learning rules, but less to the higher-level control layer that would turn neural activity into behaviour. SANSAC offers a ready-made recipe: train a spiking actor with SAC on a conventional machine, then deploy the actor on spiking hardware or interface it to living tissue. The hybrid design, in which critics stay non-spiking during training, is pragmatic and probably necessary while organoid training remains slow and noisy.

The threat is that the result reduces the algorithmic uniqueness of biological control. If a conventional spiking network trained with surrogate gradients can learn Bipedal Walker policies, then an organoid system must show that living tissue provides something beyond what an artificial SNN can provide. Possible candidates include faster adaptation to perturbations, richer temporal coding, or the ability to learn online from a small number of real-world trials. Those claims are not tested here.

The honest conclusion is that SANSAC clears one prerequisite for embodied organoid intelligence, a trainable spiking controller, without requiring biological tissue. It does not demonstrate that organoids are the best place to run such a controller. The field still needs to show that the biological substrate adds enough value to justify the experimental overhead.

The bottom line

The paper establishes that a spiking actor trained with Soft Actor-Critic can match a conventional actor on Bipedal Walker in terms of mean reward and success rate. The result is a simulation-based baseline, not a hardware deployment, and it is limited to one environment. For organoid intelligence, the main takeaway is that continuous-control policies for spiking substrates are now algorithmically tractable. The remaining question is whether living neural tissue can improve on artificial spiking networks once both are placed in a real closed loop.