Fast and Flexible Conjunction Coding in Biological and Artificial Vision - Project Summary How does the human brain encode visual feature conjunctions? Two influential, yet disparate, research traditions have proposed two different mechanisms. Research in neurophysiology and computer vision has explored static conjunction coding, where feature conjunctions are automatically extracted via a feedforward hierarchy. While efficient, this mechanism may be limited to encoding conjunctions enabled by the hierarchy's learned connectivity. By contrast, research in cognitive psychology has explored dynamic conjunction coding, which sequentially encodes task-relevant conjunctions via attentional selection. While slow, this mechanism is flexible and capable of encoding any feature conjunction. Despite evidence for both mechanisms, their interplay remains unclear. In this project, we leverage advances in deep learning and open neuroimaging datasets to understand how these two mechanisms interact in the human brain, yielding the best of both worlds: fast, but flexible conjunction coding. Through the three complementary Specific Aims, we advance our understanding of these fundamental issues in vision science, and on a practical level, develop approaches that can be used to improve computer vision, aiding the development of useful technologies like autonomous vehicles. In the course of this project, I will master modern approaches in deep learning and computational neuroscience through the mentorship of my sponsor Dr. Kriegeskorte and my co-sponsor Dr. Fusi, equipping me for a career leading a lab that bridges cognitive science, neuroscience, and artificial intelligence. Hypotheses: The human brain implements static conjunction coding via neural populations with “and-like” tuning to feature combinations that emerges via feedforward convergence of neurons tuned to single features, while dy- namic conjunction coding requires recurrent connections. Static conjunction coding can rapidly encode familiar, but not unfamiliar conjunctions, while dynamic conjunction coding can encode any conjunction, but more slowly. Aim 1: We use a massive open-source fMRI dataset to chart the prevalence of static conjunction coding through- out the human visual system, using a method we developed in preliminary analyses. Aim 2: We apply “synthetic neurophysiology” to feedforward artificial neural networks to understand how static conjunction coding emerges in a feedforward hierarchy, beginning by characterizing conjunction-tuned units iden- tified in preliminary analyses, followed by testing influential models for how this occurs in biological vision. Aim 3: We test whether feedforward artificial neural networks exhibit similar limitations on a visual search task compared to known results for human feedforward vision, followed by testing whether introducing recurrent con- nections to networks can overcome these limitations.
