Noisy models of the ventral stream reveal the impact of recurrence and learned representations on information processing timescales

Abstract

A bstract In vision neuroscience, the temporal dynamics of the sensory stream and of its neural representations are thought to be deeply linked to the function of the hierarchy of cortical areas that deal with object recognition, known as the visual ventral stream. Neural representations that are invariant under identity-preserving object transformations, and therefore allow for efficient learning of object identity, are theorized to emerge from a self-supervised learning process that attempts to extract “temporally stable” features from the sensory input. Conversely, invariance increases along the hierarchy, putatively implying progressively slower codes in higher-level areas. Recent neurophysiological evidence shows that indeed, as one moves along this cortical hierarchy, neural representations of dynamic stimuli become slower, and additionally the temporal scales of the within-trial fluctuations of these representations (called “intrinsic timescales”) increase starkly. However, the network determinants of these timescale hierarchies are not understood in realistic systems, as the classical theory is based on models without noise, recurrence, or adaptive mechanisms. Here we investigate the temporal structure of the neural code in a noisy, recurrent and adaptive model of the ventral visual stream. We show that, surprisingly, the organization of the representation timescales is set by the broad architectural features of the network, regardless of training, while the intrinsic timescales depend on the details of the functions implemented on each layer. Our work underscores the importance of the temporal structure of the neural code as a probe for the link between structure and function in models of the vertebrate visual system.