The Concordia Contest: Advancing the Cooperative Intelligence of Language Agents

Chandler Smith, Rakshit Trivedi, Jesse Clifton, Lewis Hammond, Akbir Khan, Sasha Vezhnevets, John P Agapiou, Edgar A. Duéñez-Guzmán, Jayd Matyas, Danny Karmon, Marwa Abdulhai, Dylan Hadfield-Menell, Natasha Jaques, Joel Z Leibo, Oliver Slumbers, Tim Baarslag, Minsuk Chang

Keywords: Cooperative AI, Language Models, Generalization, Mixed-Motive Games

TL;DR: As a sequel to the Melting Pot Contest at NeurIPS 2023, we introduce a new contest centered on cooperation between Language Model (LM)-based agents in intricate, text-mediated environments reflective of real world scenarios.

Abstract: Building on the success of the Melting Pot contest at NeurIPS 2023, which challenged participants to develop multi-agent reinforcement learning agents capable of cooperation in groups, we are excited to propose a new contest centered on cooperation between language model (LM) agents in intricate, text-mediated environments. Our goal is to advance research on the cooperative intelligence of such LM agents. Of particular interest are the agents capable of using natural language to effectively cooperate with each other in complex environments, even in the face of challenges such as competing interests, differing values, and potential miscommunication. To this end, we will leverage the recently released Concordia framework, an open-source library for defining open-ended environments where LM agents like those of Park et al. (2023) can interact with one another by generating free-form natural text describing what they intend to do or say. Concordia provides a suite of mixed-motive social dilemma scenarios where cooperation is valuable but hard to achieve. The proposed contest will challenge the participants to develop LM agents that exhibit cooperative intelligence in a variety of Concordia scenarios designed to assess multiple distinct skills of cooperation, including promise-keeping, negotiation, reciprocity, reputation, partner choice, compromise, and sanctioning. Participants will be scored based on the ability of their trained agents in executing skillful cooperation, particularly in the presence of new co-players in unforeseen (held-out) scenarios. Given the rapid development of LMs and the anticipated increase in the use of personalised LM agents, we contend that their propensity and ability to cooperate well with a diverse array of other actors (human or machine) will soon be of critical importance.

Competition Timeline:

July 30: Beta version of all necessary resources will be made available.

Aug 26: Official opening of the contest to the public, signifying the start of the warm-up phase. This phase allows participants to familiarize themselves with the Concordia environment, make preliminary submissions, and seek clarifications from the organizers.

Aug 31 - Oct 31: Development phase begins. Submissions during this period contribute towards the leaderboard rankings. This phase is crucial for participants to develop and iterate on their solutions.

Oct 31 - Nov 15: Evaluation phase commences. During this phase, detailed feedback is limited to error reports and final scores. The leaderboard remains confidential until the contest at NeurIPS.

Nov 15 - Nov 20: The organizing committee reviews and verifies the results. The top 10 entries, along with selected others, are invited to provide detailed system descriptions.

Nov 20 - Dec 9: Organizers conduct an in-depth analysis of contest results to prepare for the conference.

Dec 3: Deadline for the submission of artifacts and system descriptions from the top entries.

Dec 9 - Dec 14: The contest culminates in a dedicated session at NeurIPS. Winners are announced, prizes awarded, and high-ranking participants, alongside organizers, present insightful findings

Website: www.cooperativeai.com/contests/concordia-2024

Primary Contact Email: [e-mail]

Participant Contact Email: [e-mail]

Workshop Format: Hybrid (Vancouver + some online speakers)

Preferred Timezone: 12-2 pm Eastern is preferable

Submission Number: 29