Temporal Context in Brains and Transformers - Project Summary: Long-term working memory allows us to remember what was said and seen on the slides at the end of a lecture, what was read in a book after long sequences of saccadic eye movements, and the musical progression during a symphony. The interplay between long-term memory retrieval in many cortical areas is involved. However, the neural mechanisms that allow us to integrate new detailed information and keep it fresh in mind over minutes and hours are unknown. Brains extract meaning from the temporal context of words in a sequence. This Pioneer Award aims to discover how global dynamical activity in the cortex and medium-term synaptic plasticity in cortical circuits create temporal context extending over minutes and hours. Transformers in Large Language Models can also keep temporal context over long dialogs and documents by storing them in a long input vector, simultaneously making all past information available. Brains don't have digital memories like computers, but nature could have evolved alternative mechanisms. Multichannel recordings have revealed traveling waves of neural activity in multiple sensory, motor, and cognitive systems. Traveling waves are ubiquitous in the cortex and carry information from the past to the future. My working hypothesis is that cortical traveling waves may be a dynamical version of the long input vector in transformers. Spacetime coding by cortical traveling waves is complementary to the rate code used for well-studied, fast sensorimotor processing. It could be used in sensory and memory systems to organize the global processing and storage of new memories over minutes and their retrieval and maintenance over hours. A cortical neuron that receives direct sensory input from the thalamus also receives indirect inputs from distant neurons in the form of a traveling wave delayed by slow, unmyelinated, long-range horizontal connections. Mixing present and past information creates a new type of spacetime population code that simultaneously represents neural activity occurring within a time window and provides temporal context between past and present inputs. The time windows for spacetime codes will vary over the cortex, from 100 ms in V1 to minutes in the prefrontal cortex. However, another mechanism is needed to extend long-term working memory to hours. The frequencies of cortical waves during awake states range from 10 to 40 Hz, which can trigger the suboptimal induction of spike-timing-dependent synaptic plasticity (STDP) lasting 20-60 minutes. This matches the time window for long-term working memory and has the advantage that it can be reused. Long-term working memory will be simulated and studied in two-dimensional very-large-scale models (VLSMs) of the spiking recurrent cortical networks that support traveling waves and STDP. Experimental collaborators will test predictions from the models in rats, monkeys, and humans. New analytic techniques will be needed to decode the spacetime code from cortical recordings from a wide range of cortical areas.
