Multiscale Models for Understanding Multi-Animal Interactions and Intent - ABSTRACT Studying multi-animal interactions is crucial for understanding cognitive mechanisms underlying social behaviors and decision-making processes. Observing collective behaviors can reveal the neural basis of social bonding, aggression, and cooperation. However, current methods for automating multi-animal behavior analysis are often too simplistic and may neglect interactions, context, and the complexity of multi-animal behavior. Here we argue that understanding complex animal behaviors requires breaking them down into fundamental units, and then forming an understanding of how these units combine to form more complex natural behaviors. Our approach is inspired by linguistic concepts, where basic elements (syllables) combine according to gram- matical rules (syntax) to convey meaning (intent). We aim to dissect multi-animal behaviors by identifying these fundamental units and their combinations to gain deeper insights into social interactions. To achieve our goals, we will organize the effort along three main aims, progressing from behavioral syllables (Aim 1), to syntax (Aim 2), and finally to intent (Aim 3). In Aim 1, we will develop methods for learning latent representations from multi-animal behavioral time se- ries and segment them into behavioral syllables—brief movements or actions efficiently describing behavioral features. The syllables from multi-animal data are mainly social syllables representing exchanges between indi- viduals that best capture or generate natural behaviors. In Aim 2, we will extract motifs, i.e., longer sequences of interactions, to comprehend complex social behaviors. To solve such a long sequence learning problem, we propose transformers for their ability to capture long-term and intricate patterns in sequential data. We will craft a compositional model that combines behavior syllables to form motifs, effectively revealing behavior syntax. In Aim 3, we will develop a novel framework for understanding intent and rewards in multi-animal behaviors, extending inverse reinforcement learning to include multiple animals. Each animal will be treated as a decision- maker whose state space will be expanded to include others' states and actions. We aim to reveal underlying intents driving social behaviors. The project will produce innovative tools for modeling multi-animal behavior, transforming raw data into frame-level behavioral syllables and constructing abstract representations of behavioral rules, patterns, and in- tent. We will provide accessible software and demos for the research community. The project's impact will be substantial, offering advanced computational tools to deepen insights into the cognitive mechanisms of social in- teractions, cooperative behaviors, and decision-making processes. This framework will enhance the analysis and prediction of complex social behaviors across species, benefit the fields of behavioral and social neuroscience, and contribute to long-term advancements in human health research.
