TL;DR
A model of primary visual cortex that describes images with a small codebook of discrete features predicted neural responses as well as a standard CNN (fraction of explainable variance 0.86 vs. 0.84). The discrete codes are a step toward finding features that don’t depend on each neuron’s preferred orientation.
The model (top): a visual stimulus is encoded, snapped to a discrete codebook, and then used both to predict neural responses and to reconstruct the image. Validation loss and per-neuron performance against a CNN baseline (bottom). From the poster.
The model (top): a visual stimulus is encoded, snapped to a discrete codebook, and then used both to predict neural responses and to reconstruct the image. Validation loss and per-neuron performance against a CNN baseline (bottom). From the poster.

The question

Neurons in primary visual cortex (V1) each prefer edges at a particular orientation. Recognizing objects, though, requires features that stay the same when an object rotates. How the visual system gets from orientation-tuned neurons to rotation-invariant representations is still unclear.

CNN models of V1 already predict neural responses to natural images well. They capture the idea that many neurons do the same computation at different locations by splitting the final readout into a where part (a spatial mask) and a what part (feature weights). To handle rotation, earlier models built rotated copies of their filters by hand. Could a model instead learn a shared set of features, independent of each neuron’s preferred orientation, by being forced to use a discrete vocabulary?

What we did

We built a model based on a vector-quantized autoencoder (VQ-VAE):

  1. An encoder turns the stimulus into a grid of feature vectors.
  2. Each vector is replaced by its nearest entry in a learned discrete codebook.
  3. The quantized features go down two branches. One decoder reconstructs the image. A factorized readout predicts each neuron’s response, using a spatial mask (where the neuron looks) times feature weights (what it responds to).

The model is trained on three goals at once: predict neural responses, reconstruct the stimulus, and keep the codebook entries close to the encoder’s outputs.

We tested it on a synthetic dataset: 100 model neurons responding linearly to 48 × 48 px Gaussian white-noise images. Performance was measured as the fraction of explainable variance explained (FEV; 1 is perfect) and the correlation between predicted and true responses.

(A) The visual system: V1 feeds the ventral ("what") and dorsal ("where") pathways. (B) Our model: encoder outputs are snapped to the nearest codebook entry, and the quantized code drives both neuron predictions and image reconstruction.
(A) The visual system: V1 feeds the ventral (“what”) and dorsal (“where”) pathways. (B) Our model: encoder outputs are snapped to the nearest codebook entry, and the quantized code drives both neuron predictions and image reconstruction.

What we found

1. Discrete codes predict neurons as well as a CNN

With a fraction of explainable variance of 0.86 and a correlation of 0.936, the discrete-codebook model matched a standard CNN (0.84 and 0.936). Forcing the model through a small vocabulary of features did not cost predictive accuracy.

ModelFEVCorrelation
CNN0.840.936
Ours without factorized readout0.630.838
Ours0.860.936
Fraction of explainable variance for each of the 100 neurons. Our model (purple) sits alongside the CNN (yellow); without the factorized readout (teal), it falls well below.
Fraction of explainable variance for each of the 100 neurons. Our model (purple) sits alongside the CNN (yellow); without the factorized readout (teal), it falls well below.

2. Separating “where” from “what” is essential

Without the factorized readout, performance dropped to an FEV of 0.63. The validation loss also stalled early and slowly rose. With it, the loss kept falling and leveled off alongside the CNN’s. Splitting each neuron’s readout into a spatial mask and feature weights is what lets the shared codebook serve all 100 neurons.

Validation loss during training. Without the factorized readout (teal), the loss stops improving early; the full model (purple) and the CNN (yellow) converge to the same low loss.
Validation loss during training. Without the factorized readout (teal), the loss stops improving early; the full model (purple) and the CNN (yellow) converge to the same low loss.

3. A first look at rotation in the codebook (preliminary)

As a first step toward the rotation question, we embedded the codebook entries for an image and for its rotated versions in two dimensions. This analysis is exploratory, and testing whether the codes are truly rotation-invariant is listed on the poster as future work. Next steps are real V1 recordings and a hierarchical codebook.

t-SNE of codebook entries for an original digit (blue) and its rotated versions (red), with the rotations shown below.
t-SNE of codebook entries for an original digit (blue) and its rotated versions (red), with the rotations shown below.

My take

BibTeX
@misc{li2024visualneural,
  title        = {A Framework of Mapping Visual Stimulus to Neural Responses via
                  Discrete Latent Representation},
  author       = {Li, Chenyi and Ran, Xuming},
  howpublished = {Poster, NITMB Emerging Directions Workshop, Chicago, IL},
  year         = {2024}
}