Description
The Embodied Intelligence (EI) Mission studies how beings perceive the world around them and integrate this information to plan and complete tasks. The EI Mission’s research includes
- the interface of perception, planning, and action
- how it is possible to learn efficiently from small data sets
- creation of behavioral benchmark tasks
In alignment with these goals, the Quest Engineering Team collaborated with the EI Mission to develop a system that facilitates experimentation and benchmarking in the Quest Robotics/Embodied Intelligence development cycle that
- easily scales to incorporate relevant natural intelligence benchmarks as the requisite data becomes available
- leverages existing Quest Engineering infrastructure, now and as needed in the future
- extends a unified, integrative approach to model benchmarking across Quest Missions.
QREI procedural experiment creation and benchmarking infrastructure. Two configuration options are shown: 1) remote execution of many experiments in parallel, but no benchmarking, and 2) local execution of a single experiment, where results are evaluated against one or more benchmarks.
-
Former Head of EngineeringMissions -
Software EngineerMissions -
-
Software EngineerMissions -
Quest Missions Project Manager -
Director of Research, MIT Quest for IntelligencePanasonic Professor, Department of Electrical Engineering and Computer ScienceComputer Science and Artificial Intelligence Laboratory- AI Robotics
- Machine Learning
- Computer Vision
Missions
Additional Details
Links to code
The QREI repository functions contiguously with the QR repository, enabling experiment management and benchmarking capabilities for:
- Experiment Creation: Automatically modifying and/or generating .pddl domain and problem files from .yml-based configuration files. Large numbers of files can now be generated with specific elements based on probability distributions, and
- Experiment Running: Automatically launching locally hosted or remote containerized QR test problems based on ROS2 package and launch files and managing the input and output data files.
Resources
The QREI Experiment Creation tool begins with a user populating a .yaml configuration file with environment customizations and goals to evaluate, and ends with the output of problem and domain .pddl (planning domain definition language) files that define the experiment “world”. The configuration file is designed to maximize freedom within experiment creation; users can initiate experiment runs anywhere on a spectrum from running the same experiment many times over to running several types of experiments with different end goals, object types, coordinates, and sizes varying between each. Based on these parameters, an environment can be actualized (either from scratch or by building on preset environments) that contains information on all objects subsumed therein. Jinja, a templating engine, is then used to fill in generic problem and domain .pddl templates based on the novel environment.
The QREI Experiment Running process begins with a user specified .yaml configuration file that allows specification of the relevant pddl models (if any), and customization of multiple different experiment processes, customization of outputs by defining ROS topics, message types, and package names to store, and process management/cleanup. The experiments can be hosted locally, or containerized and run remotely. Given sufficient resources, the container orchestration system will accommodate running many of these procedurally generated experiments in parallel and combine output summaries into a single summary.