Summary of A Closer Look at Auroc and Auprc Under Class Imbalance, by Matthew B. A. Mcdermott (1) et al.
A Closer Look at AUROC and AUPRC under Class Imbalance
by Matthew B. A. McDermott, Haoran Zhang, Lasse Hyldig Hansen, Giovanni Angelotti, Jack Gallifant
First submitted to arxiv on: 11 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Methodology (stat.ME)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper challenges a widely held claim that the area under the precision-recall curve (AUPRC) is superior to the area under the receiver operating characteristic (AUROC) for model comparison in tasks with class imbalance. The authors theoretically demonstrate that AUPRC is not generally superior in these cases and can even be harmful by favoring improvements in subpopulations with more frequent positive labels, exacerbating algorithmic disparities. Experimental results on semi-synthetic and real-world fairness datasets support this theory. A review of over 1.5 million scientific papers reveals the origin of this invalid claim: often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about a common mistake in machine learning. It says that one way to compare models is not as good as another way when there are more of one type than the other. The authors show why this is wrong. They also find out where this idea came from and how it’s been used wrongly many times. This matters because it can make some problems worse. |
Keywords
* Artificial intelligence * Machine learning * Precision * Recall
