A Generalist Agent

Published: 10 Nov 2022, Last Modified: 17 Sept 2024Accepted by TMLREveryoneRevisionsBibTeXCC BY 4.0
Event Certifications: iclr.cc/ICLR/2024/Journal_Track
Abstract: Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: De-anonymized version. Updated Section F.5 as requested.
Certifications: Featured Certification, Outstanding Certification
Assigned Action Editor: ~Laurent_Charlin1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 371
Loading