Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning

[NeurIPS 2024]

Hao-Lun Hsu^* · Weixin Wang^* · Miroslav Pajic · Pan Xu

Duke University (^*indicates equal contribution)

Official implementation of the paper "Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning" with both Perturbed-History Exploration and Langevin Monte Carlo Exploration in a multi-agent setting.

Requirements

Although Explorer is for single-agent settings, the installation process is similar.

• Python (>=3.6)
• PyTorch
• Others: Please check requirements.txt.

The dependency tree of agent classes for N-chain problem

Base Agent
  ├── Vanilla DQN
  |     ├── DQN
  |     |    ├── DDQN
  |     |    ├── NoisyNetDQN
  |     |    ├── BootstrappedDQN
  |     |    └── LSVI-LMC
  |     ├── Maxmin DQN ── LSVI-PHE

Experiments

To train different exploration strategies for N-chain, please change the context in the configuration file in the file "configs/nchain.json" with the corresponding hyper-parameters. Specifically, we can change (1) the value of n in "env" for the length of the states, (2) the agent name, and (3) the optimizer (e.g.,"aSGLD" for LSVI-LMC "Adam" for all other methods). Then we can run an experiment for N-chain problem as python main.py To train different exploration strategies for Super Mario Bros task is more straightforward. Please select the method you are going to use in the script of mario_main.py. For example, algo = "dqn". Then we can run an experiment for Super Mario Bros task as python mario_main.py

Citation

@article{hsu2024randomized,
  title={Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning},
  author={Hsu, Hao-Lun and Wang, Weixin and Pajic, Miroslav and Xu, Pan},
  journal={Advances in Neural Information Processing Systems},
  volume={},
  pages={},
  year={2024}
}