Summary of Sample and Communication Efficient Fully Decentralized Marl Policy Evaluation Via a New Approach: Local Td Update, by Fnu Hairi et al.
Sample and Communication Efficient Fully Decentralized MARL Policy Evaluation via a New Approach: Local TD update
by Fnu Hairi, Zifan Zhang, Jia Liu
First submitted to arxiv on: 23 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Multiagent Systems (cs.MA)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the actor-critic framework for fully decentralized multi-agent reinforcement learning (MARL), specifically focusing on the MARL policy evaluation (PE) problem. The goal is to develop an approach that efficiently evaluates the value function of global states for a given policy while minimizing sample and communication complexities. To achieve this, the authors propose the “natural” idea of performing multiple local TD-update steps between consecutive communication rounds to reduce the frequency of communication. However, this approach raises concerns about agent drift resulting from heterogeneous rewards across agents. The paper makes a first attempt to answer the fundamental question: Can the local TD-update approach entail low sample and communication complexities? The authors explore the setting of MARL-PE with average reward, motivated by multi-agent network optimization problems. Their theoretical and experimental results demonstrate that allowing multiple local TD-update steps effectively reduces the sample and communication complexities compared to consensus-based MARL-PE algorithms. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary In this paper, researchers study a way for machines to learn together without a central controller. This is useful for many real-world situations where multiple agents need to work together, like self-driving cars or drones. The team looks at how to make sure these agents learn efficiently and don’t get stuck in bad situations. They propose an idea that involves agents doing some learning on their own before sharing with others. This can help reduce the amount of communication needed between agents. The authors test this idea and show that it can be very effective. |
Keywords
* Artificial intelligence * Optimization * Reinforcement learning
