Brain-Score Language

The Brain-Score Language platform enables the discovery of precise accounts of how humans process language. By evaluating computational models on their alignment to brain and behavioral data, Brain-Score synergizes the fields of brain and cognitive sciences and machine learning in an accessible, scalable, and open-source platform.

With the diversity of neuroscientific and behavioral experiments, recorded data typically comes from many different sources in different formats with different details about how the experiment was conducted. For instance, one lab might record the neurally evoked activity of subjects listening to a story while another lab measures the response times for reading word after word. While making sense of this complex data is difficult, previous research had found that language models show surprising similarities to the human language system. But engaging with all this data at scale can be daunting especially for modelers who require large sets of data to plausibly validate the next generation of human-like models and would traditionally have to parse all the data and experimental details themselves. Importantly, data alone is not enough for this research at the intersection of natural and artificial intelligence: a reproducible way of interacting with the data and evaluating model alignment is just as important.

Brain-Score solves this problem by providing accessible benchmarks that already have all the knowledge about the experiment and data intricacies codified in a standard format such that models can be immediately tested. When a model is submitted to the Brain-Score platform, it is evaluated on a battery of linguistic tests such that its scores can reproducibly be compared to other models. Brain-Score thus provides a leaderboard for which model is the most aligned to human language processing and builds a community that combines expertise covering neuroscience, psychophysics, and deep learning, with the shared goal of modeling the human language system.

Brain-Score Language adds to the community of researchers in vision that already actively contribute data and models to the Brain-Score platform, and adds the exciting cognitive aspects of the human language system to the quest for natural intelligence.

The GitHub repo with documentation is here, and the language components (scoreboard, submission options) will soon be available on the Brain-Score website.