Summary of Cross-domain-aware Worker Selection with Training For Crowdsourced Annotation, by Yushi Sun et al.
Cross-domain-aware Worker Selection with Training for Crowdsourced Annotation
by Yushi Sun, Jiachuan Wang, Peng Cheng, Libin Zheng, Lei Chen, Jian Yin
First submitted to arxiv on: 11 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Databases (cs.DB)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel worker selection scheme for annotation tasks through crowdsourcing. Existing approaches only consider workers’ past performance on tasks with ground truth, neglecting two crucial aspects: historical performances in other tasks (cross-domain) and dynamic worker performance as they learn from the ground truth. The proposed cross-domain-aware worker selection with training approach addresses these limitations by introducing two estimation modules: one for statistical analysis of cross-domain correlation and another for simulating workers’ learning gain dynamically. A theoretical framework is provided, along with experimental results on real-world and synthetic datasets, demonstrating the superiority of the proposed method over baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps make it easier to get accurate answers from people who help with tasks like labeling images or videos. Right now, methods only look at how well someone did on a specific task in the past. But what if that task is very different from the new one they’re working on? Or what if the person learns and gets better as they go along? This paper introduces a new way to choose who does which tasks by taking both of these things into account. They tested their approach using real-world data and showed it works better than other methods. |
