Supported data for manuscript "Can LLM-Augmented autonomous agents cooperate?, An evaluation of their cooperative capabilities through Melting Pot"

<p>The repository data corresponds partially to the manuscript titled&nbsp;<strong>"Can LLM-Augmented Autonomous Agents Cooperate? An Evaluation of Their Cooperative Capabilities through Melting Pot,"</strong>&nbsp;submitted to&nbsp;<em>IEEE Transactions on Artificial Intelligence</em>. The dataset comprises experiments conducted with&nbsp;<strong>Large Language Model-Augmented Autonomous Agents (LAAs)</strong>, as implemented in the ["Cooperative Agents" repository](https://github.com/Cooperative-IA/CooperativeGPT/tree/main), using substrates from the Melting Pot framework.</p> <h3>Dataset Scope</h3> <p>This dataset is divided into two main experiment categories:</p> <ol> <li> <p><strong>Personality__experiments</strong>:</p> <ul> <li>These focus on a single scenario (<strong>Commons Harvest</strong>) to assess various agent personalities and their cooperative dynamics.</li> </ul> </li> <li> <p><strong>Comparison_baselines__experiments</strong>:</p> <ul> <li>These experiments include three distinct scenarios designed by Melting Pot: <ul> <li><strong>Commons Harvest Open</strong></li> <li><strong>Externally Mushrooms</strong></li> <li><strong>Coins</strong></li> </ul> </li> </ul> </li> </ol> <p>These scenarios evaluate different cooperative and competitive behaviors among agents and are used to compare&nbsp;<strong>decision-making architectures</strong>&nbsp;of LAAs against reinforcement learning (RL) baselines. Unlike the&nbsp;<strong>Personality__experiments</strong>, these comparisons do not involve bots but exclusively analyze RL and LAA architectures.</p> <h3>Scenarios and Metrics</h3> <p>The metrics and indicators extracted from the experiments depend on the scenario being evaluated:</p> <ol> <li> <p><strong>Commons Harvest Open</strong>:</p> <ul> <li>Focus:&nbsp;<strong>Resource consumption</strong>&nbsp;and&nbsp;<strong>environmental impact</strong>.</li> <li>Metrics include: <ul> <li>Number of apples consumed.</li> <li>Devastation of trees (i.e., depletion of resources).</li> </ul> </li> </ul> </li> <li> <p><strong>Externally Mushrooms</strong>:</p> <ul> <li>Focus:&nbsp;<strong>Self-interest vs. collective benefit</strong>.</li> <li>Agents consume mushrooms with different outcomes: <ul> <li>Mushrooms that benefit the individual.</li> <li>Mushrooms that benefit everyone.</li> <li>Mushrooms that benefit only others.</li> <li>Mushrooms that benefit the individual but penalize others.</li> </ul> </li> <li>Metrics evaluate trade-offs between individual gain and collective welfare.</li> </ul> </li> <li> <p><strong>Coins</strong>:</p> <ul> <li>Focus:&nbsp;<strong>Reciprocity and fairness</strong>.</li> <li>Agents collect coins with two options: <ul> <li>Collect their own color coin for a reward.</li> <li>Collect a different color coin, which grants a reward to the agent but penalizes the other.</li> </ul> </li> <li>Metrics include reciprocity rates and the balance of mutual benefits.</li> </ul> </li> </ol> <h3>Objectives of Comparison Experiments</h3> <p>The&nbsp;<strong>Comparison_baselines__experiments</strong>&nbsp;aim to:</p> <ol> <li>Assess how LAAs compare to RL baselines in cooperative and competitive tasks across diverse scenarios.</li> <li>Compare&nbsp;<strong>decision-making architectures</strong>&nbsp;within LAAs, including chain-of-thought and generative approaches.</li> </ol> <p>These experiments help evaluate the robustness of LAAs in scenarios with varying complexity and social dilemmas, providing insights into their potential applications in real-world cooperative systems.</p> <h3>Simulation Details (Applicable to All Experiments)</h3> <p>In each simulation:</p> <ol> <li> <p><strong>Participants</strong>:</p> <ul> <li>Experiments involve predefined numbers of LAAs or RL agents.</li> <li>No bots are included in&nbsp;<strong>Comparison_baselines__experiments</strong>.</li> </ul> </li> <li> <p><strong>Action Dynamics</strong>:</p> <ul> <li>Each agent performs high-level actions sequentially.</li> <li>Simulations conclude either after reaching a&nbsp;<strong>preset maximum number of rounds</strong>&nbsp;(typically 100) or prematurely if the scenario's resources are fully depleted.</li> </ul> </li> <li> <p><strong>Metrics and Indicators</strong>:</p> <ul> <li>Extracted metrics depend on the scenario and include measures of individual performance, collective outcomes, and agent reciprocity.</li> </ul> </li> </ol> <p>This repository enables reproducibility and serves as a benchmark for advancing research into cooperative and competitive behaviors in LLM-based agents.</p>