Summary of Parallel Split Learning with Global Sampling, by Mohammad Kohankhaki et al.
Parallel Split Learning with Global Sampling
by Mohammad Kohankhaki, Ahmad Ayad, Mahdi Barhoush, Anke Schmeink
First submitted to arxiv on: 22 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper proposes novel solutions to address challenges in distributed deep learning systems, specifically in parallel split learning on resource-constrained devices. The authors identify three key issues: large effective batch sizes, non-independent and identically distributed data, and the straggler effect. They introduce two new methods, uniform global sampling and Latent Dirichlet Sampling, to mitigate these challenges. Uniform global sampling decouples the effective batch size from the number of clients, reducing mini-batch deviation. Latent Dirichlet Sampling generalizes this approach to balance batch deviation and training time. The authors’ simulations show that their proposed methods enhance model accuracy by up to 34.1% in non-independent settings and reduce training time by up to 62% when stragglers are present. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making it easier for devices with limited power to work together to learn new things. Right now, this process can be slow and inaccurate because of problems like having too much data at once or some devices being slower than others. The researchers came up with two new ways to make this process better: one called uniform global sampling, which helps spread the load more evenly, and another called Latent Dirichlet Sampling, which makes sure that all devices are working together efficiently. They tested these methods and found that they can improve accuracy by up to 34% and reduce training time by up to 62%. |
Keywords
» Artificial intelligence » Deep learning
