Summary of Automatic Outlier Rectification Via Optimal Transport, by Jose Blanchet et al.
Automatic Outlier Rectification via Optimal Transport
by Jose Blanchet, Jiajin Li, Markus Pelger, Greg Zanotti
First submitted to arxiv on: 21 Mar 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Machine Learning (cs.LG); Optimization and Control (math.OC); Methodology (stat.ME)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The novel conceptual framework proposed in this paper uses optimal transport with a concave cost function to detect outliers. The traditional two-stage procedure for outlier detection involves first detecting and removing outliers, then performing estimation on the cleaned data. However, this approach does not consider how outlier removal affects the estimation task, leaving room for improvement. To address this limitation, the authors propose an automatic outlier rectification mechanism that integrates rectification and estimation within a joint optimization framework. The optimal transport distance with a concave cost function is used to construct a rectification set in the space of probability distributions, and the best distribution within the rectification set is selected to perform the estimation task. This approach is demonstrated to be more effective than conventional approaches in simulations and empirical analyses for mean estimation, least absolute regression, and the fitting of option implied volatility surfaces. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper proposes a new way to find outliers using optimal transport with a special cost function. The usual method of finding and removing outliers before doing statistical analysis doesn’t take into account how the outliers affect the analysis. To fix this, the authors came up with a way to automatically remove outliers while also doing the analysis. They use a special distance metric called optimal transport to find the best way to correct for the outliers. This new method is shown to be better than old methods in simulations and real-world tests. |
Keywords
* Artificial intelligence * Optimization * Outlier detection * Probability * Regression
