Summary of Out-of-distribution Detection Should Use Conformal Prediction (and Vice-versa?), by Paul Novello et al.
Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?)
by Paul Novello, Joseba Dalmau, Léo Andeol
First submitted to arxiv on: 18 Mar 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach utilizes Conformal Prediction (CP) to enhance the efficiency of Out-of-Distribution (OOD) detection. Specifically, it emphasizes the limitations of standard evaluation metrics in OOD benchmark settings due to finite sample sizes. To address this, new conformal AUROC and FRP@TPR95 metrics are defined, providing probabilistic conservativeness guarantees on their variability. The benefits of combining OOD with CP are demonstrated using two reference benchmarks: OpenOOD (Yang et al., 2022) and ADBench (Han et al., 2022). Additionally, the approach is shown to improve upon current CP methods by utilizing OOD scores as non-conformity scores. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research combines Out-of-Distribution (OOD) detection with Conformal Prediction (CP) to make better predictions. Current evaluation metrics for OOD detection can be too optimistic because they only look at a small sample of data. To fix this, the authors create new metrics that give us a range of possible results instead of just one. They test these new metrics on two different datasets and show that combining OOD with CP makes better predictions than using either method alone. |
