About
Thomas Serre is a French neuroscientist and computer scientist and the Thomas J. Watson, Sr. Professor of Science and Professor of Cognitive & Psychological Sciences and Computer Science at Brown University. He serves as Faculty Director of the Center for Computation & Visualization and Associate Director of the Center for Computational Brain Science, and is also an affiliate of the Carney Institute for Brain Science, the Center for Theoretical Physics and Innovation, and the Data Science Institute. He holds an International Chair in AI at the Artificial and Natural Intelligence Toulouse Institute (France) and is a Fellow of the European Laboratory for Learning and Intelligent Systems (ELLIS).
His research investigates the computational principles of biological vision to understand how the brain works—and to build AI systems that see and reason more like humans. He studies how recurrent and feedback processes support visual reasoning, develops cognitive-psychology-inspired benchmarks to reveal gaps between human and machine vision, and designs brain-inspired computational models that help bridge neuroscience and artificial intelligence.
Dr. Serre received his Ph.D. in Neuroscience from MIT (2006) and an M.Sc. in Electrical Engineering and Computer Science from Télécom Bretagne, France (2000). His work has been featured by the BBC, The Economist, New Scientist, and Scientific American. He serves as an area chair for premier conferences such as CVPR, ICML, ICLR, and NeurIPS, and as a Senior Editor of Minds, Machines, and Brains (MIT Press).
He has received an NSF Early Career Award, a DARPA Young Faculty Award, a DARPA Director's Award, and Brown's Mid-Career Research Achievement Award (2024), and his team's work on human action recognition earned the IEEE PAMI Helmholtz Prize (2021) and the Mark Everingham Prize (2022).
Research
A model can recognize thousands of object categories yet struggle to trace a contour, judge a relation, or imagine a scene from another viewpoint. We study these discrepancies to identify the computations that support biological vision.
We combine behavioral and neural experiments with computational models of recurrent visual processing and learning from experience. We also develop methods to identify the visual features and representations that AI models use, and we apply computer vision to behavioral and clinical research, where it can quantify behaviors at a scale and level of detail that would be impractical by hand.
Human–AI alignment in vision
As recognition accuracy improves, agreement with human behavior and primate neural recordings can plateau or decline. Our paper on vision-model alignment reports that better performance does not necessarily produce a better model of vision. In our ClickMe game, people mark the parts of an image that help them recognize an object. We use these human importance maps to train networks to base their decisions on the same image regions that people do, a procedure we call harmonization. This improves agreement with human vision without changing the network architecture.
Funded by NSF (IIS-2402875) and featured by NSF.
Cognitive benchmarks for visual reasoning
We design tasks that isolate specific demands on vision. The Pathfinder task tests whether a model can trace a single contour through clutter — the same challenge as the tangled-line puzzles in children's activity books, where you follow one winding string through a jumble of crossing lines to find where it ends. Our recurrent models succeed on conditions that challenge the tested feedforward networks.
The Compositional Visual Relations benchmark tests whether learned visual concepts can be combined in new ways. The 3D-PC benchmark tests visual perspective-taking: reasoning about how a scene appears from another person's viewpoint, an ability that underpins theory of mind. The 3D-PC project page, 3D-PC code, and 3D-PC dataset are available for that benchmark.
In same–different tasks, a model must judge whether two objects match, then apply that relation to objects it has not encountered during training. Comparing people and models on these tasks helps us distinguish successful recognition from the computations needed for grouping and relational reasoning.
Funded by ONR (N00014-24-1-2026).
Cortical feedback and recurrent vision
Beyond the initial feedforward sweep, visual cortex is richly recurrent: horizontal and feedback connections continually propagate signals within and between areas. We study how these interactions reshape a representation over time and what they contribute to perception.
We build recurrent network models of these circuits and test them against behavior and neural data. Networks with horizontal recurrent connections learn to trace long-range contours that much larger feedforward networks struggle with, and related circuit models detect contours in natural images while reproducing contextual illusions such as the tilt illusion. Comparing the two kinds of recurrence shows that horizontal and top-down connections support different forms of perceptual grouping. Because these models compute over time, they also yield reaction-time predictions that can be compared with people's, and their dynamics track the time course of grouping in natural scenes.
Experiments on same–different judgments reveal neural dynamics associated with relational processing.
In collaboration with David Sheinberg, our mental-simulation paper asks how monkeys predict where a ball will land after falling through obstacles, before seeing it move. Their behavior and eye movements are consistent with mentally tracing the ball's path. We compare these observations with recurrent networks that learn to solve the same task.
We also investigate whether synchronized neural activity helps track objects and solve feature binding: keeping the color, shape, and other properties of each object associated with that object rather than mixing them with those of its neighbors. To make these mechanisms practical at larger scales, we develop stable recurrent models and convolutional state space models.
Funded by ONR (N00014-24-1-2026) and ANITI (France 2030, ANR-23-IACL-0002).
Development and embodiment
Harmonization imposes human-like visual strategies through direct supervision. A complementary question is whether alignment can emerge without that supervision, from the way a system learns. Animals learn to see without labels, through experience that unfolds continuously as they move through the world. We study what models can learn from that temporal structure and from acting, and how visual experience and learning objectives determine whether a model comes to see the world as we do.
We collaborate with developmental cognitive scientists to identify the learning principles that support perceptual development, including how intrinsic rewards guide exploration.
Ongoing work uses a newborn chick's first-person visual experience to test whether predictive learning can support grouping and object recognition without labels. The failures of current models on the 3D-PC perspective-taking benchmark motivate work on agents that learn through exploration. In collaboration with Fulvio Domini, we also study the computational mechanisms of depth perception.
Funded by the REPRISM MURI (ONR N00014-24-1-2603), ONR (N00014-24-1-2026), and the NSF AI Research Institute on Interaction for AI Assistants (ARIA; NSF Cooperative Agreement 2433429).
Explainable AI for scientific discovery
A model's predictions can be accurate for the wrong reasons. In our histopathology paper, models trained to predict tumor mutations from tissue images relied on correlations with tissue subtype and failed a more demanding test. Accurate predictions alone did not establish that the models measured the intended biology.
With collaborators at the Artificial and Natural Intelligence Toulouse Institute (ANITI), we develop methods to examine the evidence behind such predictions. CRAFT identifies concepts used by a model, and MACO visualizes its features. These methods are available through Xplique and Horama libraries.
We test explanations with people, finding that more capable vision models are not necessarily more interpretable and comparing which representations people can understand. Applications include identifying plant families from leaf architecture. LeafLENS and ObjectLENS let visitors explore model explanations directly.
Funded by ANITI (France 2030, ANR-23-IACL-0002) and the NSF AI Research Institute on Interaction for AI Assistants (ARIA; NSF Cooperative Agreement 2433429).
Foundation models and multimodal reasoning
Foundation models offer increasingly powerful predictions of behavior and brain activity. Establishing what those predictions explain requires understanding how the models learn and represent information. Our perspective on prediction and understanding develops that distinction.
We ask how the training objective determines which information a model retains. Contrastive learning brings representations of matched inputs closer together while separating mismatched inputs. Predictive learning trains a model to infer one representation from another. Our spectral theory of multimodal losses studies the relationship between these objectives and the representations they produce. We also study shared multimodal workspaces, failures of feature binding in vision–language models, and language-model representations that track human judgments of event plausibility.
Funded by the NSF AI Research Institute on Interaction for AI Assistants (ARIA; NSF Cooperative Agreement 2433429) and ONR (N00014-24-1-2026).
AI for behavioral and clinical science
Studying behavior often requires researchers to watch and label many hours of video. Our automated home-cage phenotyping paper describes a system that learns to recognize mouse behaviors such as eating, grooming, and resting, so researchers can measure when these behaviors occur and how they change over time. We have extended video-based behavioral analysis to worms, zebrafish, and children, and we have used these measurements to characterize mouse models of amyotrophic lateral sclerosis and frontotemporal dementia (ALS-FTD).
ALS-FTD phenotyping was supported by NIH/NINDS (R21 NS112743, 2020–2022).
In a separate collaboration with Taylor Burke at Massachusetts General Hospital, we develop computer-vision methods to analyze photographs of self-inflicted injuries. The models measure visible signs of injury severity. We are testing whether these measurements, combined with clinical assessments, can improve prediction of future suicide attempts. This work, supported by NIH award R01 MH140004, investigates a potential source of information for clinicians assessing suicide risk.
Suicide-risk research is funded by NIH/NIMH (R01 MH140004 and R01 MH143695).
With David Borton's group, we develop machine learning methods to personalize electrical stimulation after spinal cord injury. Our models learn how stimulation settings relate to muscle activity, helping the team find settings that produce a desired response without testing every possible combination. This work contributes to a neuroprosthetic interface that restored muscle control and sensory feedback in three participants with spinal cord injuries.
Spinal-interface research was supported by DARPA (D19AC00015, 2019–2024; PI: David Borton).
Teaching
I teach computational courses at the interface between natural and artificial intelligence, bridging neuroscience, cognitive science, and AI.
-
CPSY 1291: Computational Methods for Mind, Brain & Behavior
A broad introduction to NeuroAI combining lectures with hands-on programming assignments. Students explore computational models of brain and cognition, classical machine learning algorithms, and modern deep learning architectures.
-
CPSY 1950: Deep Learning in Brains, Minds & Machines
A seminar on how biological and artificial systems process information. Students read and discuss recent papers in NeuroAI and evaluate their methods and evidence.
-
CPSY 2107: Graduate Seminar in NeuroAI
An intensive graduate seminar on the joint study of artificial and biological intelligence. Students read work on foundation models and their alignment with neural and behavioral data, and each student presents a paper and leads discussion of others.
Selected recent publications
- Not too generative, not too discriminative: The human alignment sweet spot
- A unified spectral theory of multimodal losses
- Perilesional neuromodulation replaces lost sensorimotor function in persons with spinal cord injury
- Better artificial intelligence does not mean better models of biology
- Feature binding in biological and artificial vision
- Is this just fantasy? Language model representations reflect human judgments of event plausibility
- From prediction to understanding: Will AI foundation models transform brain science?
- The 3D-PC: A benchmark for visual perspective taking in humans and machines
- Deceptive learning in histopathology
- Monkeys engage in visual simulation to solve complex problems
See the lab publications page for a complete list of publications.
Selected talks
A selection of recorded talks. For the full list of invited talks, see my CV (PDF).
-
Scaling laws vs. neural laws: Toward more natural artificial vision
Modern computer vision has advanced by scaling models, data and compute, and deep networks now match or exceed human accuracy on benchmarks such as ImageNet. Yet their alignment with human vision is fragile: on simple probes from the cognitive sciences, even the largest models fall to near chance, and models that reach human accuracy do so with very different visual strategies. This keynote argues that more natural artificial vision will come from the neural laws of biological vision: the developmental principles that shape how brains learn to see, and the recurrent architecture of cortex.
-
Cortical feedback mechanisms in visual reasoning: From perceptual grouping to abstract compositional reasoning
How does cortical feedback contribute to visual reasoning? This talk examines perceptual grouping and same–different judgments, using computational models to test how recurrent processing helps solve these tasks.
Active grants
- High-performance compute cluster for brain science
Supporting acquisition of a high-performance computing cluster for large-scale computational neuroscience modeling and deep learning experiments. The Center for Computation and Visualization runs this instrument as the Carney Condo (RRID:SCR_029056).
- Brain-inspired deep learning models of visual reasoning
Investigating computational principles underlying visual reasoning to develop more capable AI through cognitive benchmarks and brain-inspired architectures.
- Brown Postdoctoral Training Program in Computational Psychiatry
Training postdoctoral fellows at the intersection of computational neuroscience, machine learning, and psychiatry to develop computational methods for understanding psychiatric disorders.
- One vision: Computational alignment of deep neural networks with humans
Developing methods to quantify and improve alignment between deep neural networks and human visual processing using developmental psychology approaches.
- ANITI AI Cluster: C3-PO Synergy Chair (Cobots with Conversation, Cognition and Perception)
Leading the vision and perception track of a Toulouse chair on collaborative robots that combine conversation, cognition and perception.
- NSF AI Research Institute on Interaction for AI Assistants (ARIA)
A national AI research institute developing trustworthy, context-aware AI assistants grounded in human and machine cognition, with mental and behavioral health as a central use case.
- REPRISM: Flexible embodied problem-solving by manipulating the representational prism
Developing vision systems that adaptively change representations based on task demands for flexible and generalizable problem-solving.
- SEA-CROGS: Scalable, efficient and accelerated causal reasoning operators, graphs and spikes for earth and embedded systems
Creating scalable causal reasoning methods combining graph-based representations, spiking neural networks, and causal inference for Earth system science.
- The next generation of operator regression networks: Theory, algorithms, applications
Advancing theoretical foundations and practical applications of operator regression networks for solving partial differential equations and modeling physical systems.
- Enhancing suicide risk detection through computer vision
Developing computer-vision measures of visible injury severity in photographs of self-inflicted injuries, and testing whether those measures add to clinical assessments when predicting future suicide attempts.
- A mechanistic analysis of the transition from nonsuicidal to suicidal self-injury in adolescents
Investigating the mechanistic transition from nonsuicidal to suicidal self-injury in adolescents through combined computational and clinical analysis.