Start Time
Type Duration Title & Speaker / Author(s)

7:00 10:00 16:00 Intro 10 min Workshop overview
Viliam Lisy, Martin Schmid, Noam Brown
7:10 10:10 16:10 Mini Tutorial 30 min Noam Brown
7:40 10:40 16:40 Invited Talk 40 min Multiagent learning: from games to sports, and back
Karl Tuyls (DeepMind, University of Liverpool)
8:20 11:20 17:20 Posters 60 min Poster Session #1

Break (20 min)

9:40 12:40 18:40 Talk 20 min Provably Efficient Decentralized Communication for Multi-Agent RL
Justin Lidard, Udari Madhushani and Naomi Leonard
10:00 13:00 19:00 Talk 20 min Subgame solving without common knowledge
Brian Zhang and Tuomas Sandholm
10:20 13:20 19:20 Invited Talk 40 min Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination
Outracing Champion Gran Turismo Drivers with Deep Reinforcement Learning
Peter Stone (The University of Texas at Austin and Sony AI)
11:00 14:00 20:00 HIGC 10 min Hidden Information Games Competition
Michal Sustr

Break (30 min)

11:40 14:40 20:40 Talk 20 min Multi-Agent Learning for Iterative Dominance Elimination: Intrinsic Barriers and New Algorithms
Jibang Wu, Haifeng Xu and Fan Yao
12:00 15:00 21:00 Talk 20 min Computing Strategies of American Football via Counterfactual Regret Minimization
Yuki Shimano, Kenshi Abe, Atsushi Iwasaki and Kazunori Ohkawara
12:20 15:20 21:20 Invited Talk 40 min Reinforcement Learning for Security Games and Beyond
Fei Fang (Carnegie Mellon University)
13:00 16:00 22:00 Posters 60 min Poster Session #2


The workshop is taking place in Blue 1.

Invited Talks

Multiagent learning: from games to sports, and back
Karl Tuyls (DeepMind, University of Liverpool)

Many real-world scenarios can be modelled as multiagent systems, in which multiple autonomous decision makers interact in a single environment, often cast as a multi-player game. Recent progress in reinforcement learning, evolutionary and empirical game theory has made it possible to apply multiagent learning techniques in applications such as complex board games, multi-robot systems and sports. In this talk I'll provide an overview of how some of the theoretical progress on the game-theoretic front paves the way to apply some multiagent learning techniques in more challenging domains like sports and robotics, showing how these techniques allow for new insights and algorithms in this area. At the same time I will review some of the theoretical progress these multiagent learning techniques build on, based on evolutionary dynamics as a tool or means to capture the dynamics of multiple learning agents across different classes of multiagent learning.

Outracing Champion Gran Turismo Drivers with Deep Reinforcement Learning
Peter Stone (The University of Texas at Austin and Sony AI)

Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits. Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world's best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing's important, but under-specified, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world's best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.
Project webpage:

Reinforcement Learning for Security Games and Beyond
Fei Fang (Carnegie Mellon University)

Security game models have led to great success in modeling defender-attacker interaction or alike in security and environmental sustainability domains. Most of the earlier work in security games relies on mathematical programming-based algorithms to solve the game and compute the optimal strategy for the defender. However, for many problems that account for more practical aspects, the game model would be much more complex and mathematical programming-based methods are not applicable. In this talk, we introduce our work that leverages reinforcement learning to handle complex security games, including games with continuous action space, green security games with real-time information, and repeated games with unknown attacker type.

Accepted Posters

Poster Session #1

Making Something Out of Nothing: Monte Carlo Graph Search in Sparse Reward Environments
Marko Tot, Michelangelo Conserva, Sam Devlin and Diego Perez Liebana

Improving Sample Efficiency of Value Based Models Using Attention and Vision Transformers
Amir Ardalan Kalantari Dehaghi, Mohammad Amini, Sarath Chandar and Doina Precup

Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learning
Somjit Nath, Omkar Shelke, Durgesh Kalwar, Hardik Meisheri and Harshad Khadilkar

Detecting Influence Structures in Multi-Agent Reinforcement Learning Systems
Fabian Raoul Pieroth, Katherine Fitch and Lenz Belzner

Godot Reinforcement Learning Agents
Edward Beeching, Jilles Dibangoye, Olivier Simonin and Christian Wolf

Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Games
Jing Tan, Ramin Khalili and Holger Karl

Local Information Based Attentional Opponent Modelling In Multi-agent Reinforcement Learning
Binqiang Chen

Exploiting Opponents under Utility Constraints in Extensive-Form Games
Martino Bernasconi de Luca, Federico Cacciamani, Simone Fioravanti, Alberto Marchesi, Nicola Gatti and Francesco Trovò

The Evolutionary Dynamics of Soft-Max Policy Gradient in Games
Martino Bernasconi de Luca, Federico Cacciamani, Simone Fioravanti, Nicola Gatti and Francesco Trovò

Commonsense Knowledge from Scene Graphs for Textual Environments
Tsunehiko Tanaka, Daiki Kimura and Michiaki Tatsubori

Direct Behavior Specification via Constrained Reinforcement Learning
Julien Roy, Roger Girgis, Joshua Romoff, Pierre-Luc Bacon and Christopher Pal

Equilibrium Computation for Auction Games via Multi-Swarm Optimization
Nils Kohring, Carina Fröhlich, Stefan Heidekrueger and Martin Bichler

Exploring Reward Surfaces in Reinforcement Learning Environments
Ryan Sullivan, J. K. Terry, Benjamin Black and John Dickerson

HiRL: Dealing with Non-stationarity in Hierarchical Reinforcement Learning via High-level Relearning
Yuhang Jiao and Yoshimasa Tsuruoka

Batch Monte Carlo Tree Search
Tristan Cazenave

Computing Distributional Bayes Nash Equilibria in Auction Games via Gradient Dynamics
Maximilian Fichtl, Matthias Oberlechner and Martin Bichler

Fast Payoff Matrix Sparsification Techniques for Structured Extensive-Form Games
Gabriele Farina and Tuomas Sandholm

Dreaming with Transformers
Catherine Zeng, Jordan Docter, Alexander Amini, Igor Gilitschenski, Ramin Hasani and Daniela Rus

Coalitional Negotiation Games with Emergent Communication
Xiaoyang Gao, Siqi Chen, Jie Lin, Yang Yang, Haiying Wu and Jianye Hao

Team Correlated Equilibria in Zero-Sum Extensive-Form Games via Tree Decompositions
Brian Zhang and Tuomas Sandholm

Poster Session #2

Cooperation Learning in Time-Varying Multi-Agent Networks
Vasanth Reddy Baddam, Almuatazbellah Boker and Hoda Eldardiry

Learning Generalizable Behavior via Visual Rewrite Rules
Mingxuan Li, Yiheng Xie, Shangqun Yu and Michael Littman

Graph augmented Deep Reinforcement Learning in the GameRLand3D environment
Edward Beeching, Maxim Peter, Philippe Marcotte, Jilles Debangoye, Olivier Simonin, Joshua Romoff and Christian Wolf

Anytime Optimal PSRO for Two-Player Zero-Sum Games
Stephen McAleer, Kevin Wang, Marc Lanctot, John Lanier, Pierre Baldi and Roy Fox

On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
Andrew Cohen, Ervin Teng, Vincent-Pierre Berges, Ruo-Ping Dong, Hunter Henry, Marwan Mattar, Alexander Zook and Sujoy Ganguly

The Partially Observable History Process
Dustin Morrill, Amy Greenwald and Michael Bowling

A Review for Deep Reinforcement Learning in Atari: Benchmarks, Challenges and Solutions
Jiajun Fan

Continual Depth-limited Responses for Computing Counter-strategies in Extensive-form Games
David Milec and Viliam Lisy

Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
Yantian Zha, Lin Guan and Subbarao Kambhampati

Stackelberg MADDPG: Learning Emergent Behaviors via Information Asymmetry in Competitive Games
Boling Yang, Liyuan Zheng, Lillian Ratliff, Byron Boots and Joshua Smith

Faster No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium
Ioannis Anagnostides, Gabriele Farina, Tuomas Sandholm and Christian Kroer

Connecting Optimal Ex-Ante Collusion in Teams to Extensive-Form Correlation: Faster Algorithms and Positive Complexity Results
Gabriele Farina, Andrea Celli, Nicola Gatti and Tuomas Sandholm

GDI: Rethinking What Makes Reinforcement Learning Different from Supervised Learning
Jiajun Fan, Changnan Xiao and Yue Huang

Where, When & Which Concepts does AlphaZero Learn? Lessons from the Game of Hex
Jessica Forde, Charles Lovering, Ellie Pavlick and Michael Littman

MDP Abstraction with Successor Features
Dongge Han, Michael Wooldridge and Sebastian Tschiatschek

Fast Algorithms for Poker Require Modelling it as a Sequential Bayesian Game
Vojtech Kovarik, David Milec, Michal Sustr, Dominik Seitz and Viliam Lisy

A Bayesian Policy Reuse Approach for Bilateral Negotiation Games
Xiaoyang Gao, Siqi Chen, Yan Zheng and Jianye Hao

An Adaptive State Aggregation Algorithm for Markov Decision Processes
Guanting Chen, Johann Gaebler, Matt Peng, Chunlin Sun and Yinyu Ye

Deep Catan
Brahim Driss and Tristan Cazenave