New progress in both our understanding of intelligence and new AI models are increasingly driven by large compilations of human behavioral data. Currently, these large volumes of experimental data are difficult, expensive and time-consuming to obtain, and are even more scarce as a resource for researchers. The MIT Quest Intelligence Observatory (IO) is enabling the collection of these data sets by building the platforms necessary to empirically characterize human cognition with precision and at scale.
The Intelligence Observatory will enable the creation of behavioral datasets working with research institutions worldwide and allow researchers to execute specific behavioral studies — on humans, non-human primates, and machine learning models.
The IO will collect data from behavioral experiments, in which human subjects perform various psychological tasks.
In a typical experiment, a participant is briefly shown an image and then asked questions, for example, “Was there a dog in the left side of the picture?” Crucially, human beings (or any vision system) are not always “correct” in answering such questions – they display a unique pattern of behavior. The patterns of behavioral errors reflect the specific structure of that vision system. By asking many participants many such questions with respect to the same visual stimulus, researchers in the IO hope to uncover a precise estimate of how human beings perceive that stimulus; i.e., the human mind's “perceptual contents” for that stimulus. Credit: Bill Chizek photo of the Franklin Delano Roosevelt memorial in Washington, DC.
While such experiments appear simple, behavior is deeply intertwined with the underlying mechanisms of intelligence: it is their final product. Thus, by collecting measurements of behavior — carefully, across a variety of contexts, and at scale — the IO aims to advance the understanding of the underlying mechanisms of intelligence.
Among the activities that will take place in the IO are:
- Tool-building: IO will build scalable technologies for specifying and deploying virtual environments and measuring human behavior in those environments
- Psychophysical Theory: IO will develop theoretically-motivated descriptors of human visual perception and experimental procedures for estimating those descriptors using behavioral data
- Asset curation: IO will curate 3D asset libraries and build tools to access, manipulate, and compute on those assets.
The IO will initially focus on human visual perception in dynamic 3D scenes, and later extend to other domains of intelligence. The Quest is very excited about the Intelligence Observatory, hoping to establish it as the experimental cornerstone for the new science of intelligence and a repository for findings to be shared with the scientific community.