Summary of On Centralized Critics in Multi-agent Reinforcement Learning, by Xueguang Lyu et al.
On Centralized Critics in Multi-Agent Reinforcement Learning
by Xueguang Lyu, Andrea Baisero, Yuchen Xiao, Brett Daley, Christopher Amato
First submitted to arxiv on: 26 Aug 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Centralized Training for Decentralized Execution (CTDE) is a widely used approach in Multi-Agent Reinforcement Learning (MARL), where agents are trained offline and execute online. This paper analyzes the use of centralized critics, which have access to global information, including true system state. While CTDE methods perform well, the benefits of using centralized critics are yet to be theoretically or empirically analyzed. The authors formally analyze centralized and decentralized critic approaches and show that centralization is not always beneficial. They also prove that state-based critics can introduce bias and variance compared to history-based critics. The experiments demonstrate practical issues with representation learning in partially observable environments, highlighting the need for theoretical analysis. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper studies how artificial intelligence (AI) agents learn together in complex systems. One common way to train these agents is called Centralized Training for Decentralized Execution. In this approach, agents are trained offline and then work together online without sharing information. The researchers looked at a specific part of this process, where a single “critic” tries to understand the entire system. They found that having one critic isn’t always the best approach. Sometimes it can even make things worse! The study also shows that AI agents have trouble learning in situations where they don’t have all the information. |
Keywords
» Artificial intelligence » Reinforcement learning » Representation learning
