What an exhaustive timing audit found hiding under a spiking classifier's average accuracy
Around a frozen spiking speech classifier, shifting a single count unit by one 5-millisecond bin makes the average accuracy go up, not down, while 13 of the 84 inputs the model gets right have a neighbor that flips them to a wrong answer. Five utterances carry 93.21% of the harmful moves. Aggregate scoring sees none of this.
Source: Aggregate accuracy conceals concentrated temporal vulnerability in a spiking speech classifier, arXiv (cs.NE), 2 October 2026. Primary source. Read: full HTML text of the preprint.
What the work claims
This is an evaluation-methods paper, and a deliberately exhaustive one. The author takes a trained spiking speech classifier (a SpikeSCR model3 with about 3.3 million parameters, built from spiking self-attention and local convolution blocks), freezes it, and chooses 100 validation utterances from the Spiking Speech Commands dataset2. Around each utterance he defines a precise, finite perturbation set: every distinct input formed by moving one integrated count unit into an adjacent 5 ms bin, within the same transformed feature channel. Then he evaluates every member and keeps every prediction. No sampling, no attack optimization, no early stopping at the first failure: 725,070 unique candidates in total, neighborhoods ranging from 3,633 to 12,014 inputs per utterance, with a mean of 7,250.71.
The central claim is that the resulting local map contains information that no aggregate metric carries. The headline contrast: under equal weighting of the 84 initially correct sources, expected accuracy across the neighborhood rises from 84.00% to 84.54%, driven by 9,031 corrective transitions outweighing 5,157 adverse ones; yet 13 of those 84 sources, 15.48%, have at least one neighbor that makes the model wrong. The harm is concentrated almost absurdly: five sources carry 93.21% of all adverse candidates, and the two most affected carry 68.47%1.
How it works
The perturbation operator is intentionally modest. It does not add noise, delete spikes, or retime events adversarially; it performs the smallest discrete change the input representation admits, shifting one count by one bin, and enumerates the entire reachable set. That choice converts robustness from a search problem into a census: because every candidate is scored, the paper can report exact incidence (how many sources have any adverse neighbor), density (how many adverse candidates each source has), concentration (how the adverse load distributes), and transition direction, separating harmful flips from corrective ones and from lateral wrong-to-different-wrong moves1.
Three further instruments deepen the map. First, source-matched internal traces: of the 699,250 candidates that preserve the predicted class, 88.37% show a nonzero relative change in the final block's activations, with a pooled median change of 0.65, so a stable label usually conceals a changed internal computation. Second, matched clean-activation replacement: supplying the clean internal state at a declared boundary often restores the correct decision (candidate-pooled restoration rates of 41.17% to 77.64% across four branch boundaries), which shows the clean state is frequently sufficient even where the internal trace had diverged. Third, controlled execution experiments: reversing batch order changes 531 validation labels with almost no net accuracy effect, and isolating cross-source coupling in the query/key LIF stages removes the effect entirely and restores bitwise order invariance across all 9,981 validation score vectors1.
Two separately seeded count-readout replicas add a second warning. Their validation accuracies differ by 0.25 percentage points, yet one has 5.21 times as many class-changing neighbors as the other on the same 100 utterance identities, a gap concentrated in just four sources; and 7.94% and 6.41% of changed traces respectively alter the final-layer spike timing while preserving the exact spike-count readout, activity that temporal aggregation erases entirely1.
Where a skeptic should push
The obvious limit is scope, and the author is commendably plain about it: this is one checkpoint, one architecture, one dataset, and a 100-utterance fixed panel, so the source-level percentages describe that panel, not prevalence over all inputs or over training seeds. The bootstrap interval on the mean adverse rate is wide, 0.25% to 1.70%. The count-readout comparison adds two seeds but in a different, less accurate architecture at a different native resolution, so it establishes the existence of aggregation ambiguity, not its size in SpikeSCR specifically1.
The deeper question is whether exhaustive enumeration buys enough over smarter search. The paper's own data say: sometimes. At 8,400 candidate queries, hard focus on the lowest-margin quartile discovers 9.42 of the 13 vulnerable sources in expectation, better than uniform sampling's 7.61; but one rare vulnerable source sits outside the low-margin quartile and receives zero focused queries, and fixed gradient rankings miss different rare cases. Only a post-hoc gradient prefix consuming 461,380 queries, 73.45% of the full census of initially correct sources, finds all 13. Enumeration is not always needed to find witnesses; it is needed to know the distribution you would have been sampling from. That is a real but narrower claim than "always audit everything"1.
Why organoid benchmarks need local audits
Organoid intelligence runs on aggregate numbers. A culture is trained, a decoder is fit, and the field reports mean decoding accuracy or a task score across trials. This paper is the cleanest published demonstration that such numbers can look excellent while the substrate harbors concentrated, input-specific failure modes that no aggregate statistic will ever surface. Worse, the equivalent of the batch-order coupling that silently changed 531 labels here, shared state across nominally independent channels, is not a software bug in a dish. It is the default condition: shared synapses, volume transmission, glial state, and electrode crosstalk couple every channel to every other, and there is no isolation patch. For tissue, execution dependence is not an implementation flaw to fix; it is a property to measure.
The transferable contribution is the reporting protocol. Define the perturbation operator exactly (for a culture: spike-time shifts, burst retiming, or stimulation-pattern jitter inside the closed loop); distinguish candidate-pooled from source-equal rates; report incidence and concentration, not just means; and separate decision stability, representation stability, and trajectory stability, because the replica experiment shows they can rank the same two systems in opposite orders. The finding that near-identical accuracies can hide a 5.21-fold difference in local vulnerability is a direct caution against comparing organoid batches, cell lines, or protocols by mean decoding accuracy alone.
The opportunity cuts the other way too. The audit's most comforting result for biological computing is that internal change and harmful direction decouple: large internal divergence with preserved decisions was common, and clean-state restoration showed much of the internal change was irrelevant to the output. Living tissue is noisy and drifts; this paper gives a measurement framework for asking, quantitatively, which of that drift actually reaches the readout and which is the readout's nullspace. That is exactly the question an organoid engineer needs answered before claiming either that a culture is stable or that it is broken.
The bottom line
Established, within its declared bounded setting: for this frozen classifier and this precisely defined neighborhood, favorable average change coexists with adverse neighbors for 15.48% of initially correct sources, extreme concentration of harm (five sources, 93.21% of adverse moves), label stability that usually conceals internal change, and execution dependence with an identifiable implementation mechanism. Not established: prevalence beyond the panel, generalization across architectures, or that the proposed training remedies (temporal augmentation, consistency penalties) would help at all; the author flags those as untested hypotheses. What would confirm the broader claim: the same census protocol on a second architecture and on non-speech event data, with seed replication. What would break it: evidence that adverse load is typically uniform across sources on larger panels, making concentration a small-sample artifact. For organoid intelligence, the actionable takeaway stands regardless: report the local map, not just the mean.
Frequently asked questions
What exactly was perturbed?
One integrated count unit in the transformed input was moved to an adjacent 5 ms bin, staying within the same feature channel. That is the smallest discrete change the representation allows, and every distinct input reachable that way was evaluated: 725,070 unique candidates around 100 utterances.
How can average accuracy go up while the model has vulnerable inputs?
Because averages mix transition types. In the census there were 9,031 corrective flips (wrong to right) against 5,157 adverse flips (right to wrong), so the net and the mean improved, while 13 individual inputs each sat next to a harmful neighbor. A single number cannot represent that distribution.
Does this mean the classifier is fragile?
Not uniformly. Only 15.48% of initially correct sources had any adverse neighbor, and harm was concentrated: five sources carried 93.21% of adverse candidates, with some vulnerable sources having only one or two harmful moves among thousands of neighbors. Fragility is a property of specific inputs, not the whole model.
Why not just use adversarial attacks instead of enumerating everything?
Attacks are good at finding some witnesses cheaply. The paper shows fixed-budget searches miss rare vulnerable sources that other methods catch, and only an exhaustive post-hoc prefix found all 13. Enumeration is what reveals the underlying distribution: incidence, density, and concentration that sampling cannot estimate reliably.
What was the batch-order effect about?
Reversing the order of inputs inside a padded batch changed 531 validation labels almost without changing net accuracy, because different sources shared temporal LIF lanes in the public implementation. Isolating those lanes removed the effect entirely. It is a warning that the execution contract, not just the weights, is part of the model.
Why should organoid intelligence researchers care about this audit?
Organoid work is scored almost entirely by aggregate decoding accuracy. This study shows such scores can rank two systems identically while their local vulnerability differs five-fold, and that internal drift and readout-relevant change are different things. Tissue, which cannot be patched to isolate its cross-channel coupling, needs this kind of local audit even more than software does.
References
- I. C. Dikmen. Aggregate accuracy conceals concentrated temporal vulnerability in a spiking speech classifier. arXiv:2610.03155 (cs.NE), 2026. https://arxiv.org/abs/2610.03155. Accessed 2026-10-30.
- B. Cramer, Y. Stradmann, J. Schemmel, and F. Zenke. The Heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems 33(7), 2744 to 2757, 2022. https://doi.org/10.1109/TNNLS.2020.3044364. Accessed 2026-10-30.
- J. Wang, L. Yu, L. Huang, C. Zhou, H. Zhang, Z. Song, H. Liu, M. Zhang, Z. Ma, and Z. Zhang. Efficient speech command recognition leveraging spiking neural networks and progressive time-scaled curriculum distillation. Neural Networks 195, 108253, 2026. https://doi.org/10.1016/j.neunet.2025.108253. Accessed 2026-10-30.